Executive Status
EnvEnvironment partition: dev, int, stg, or prod. | GoalConfigured success target and run scope for this environment. INT/STG/PROD use all E2E job runs. DEV uses post-good runs plus Tide batches: runs on the final commit of a PR that later merged, or batch retests of PRs that passed their individual e2e-parallel checks. | RunsNumber of job runs in the selected goal definition for this environment. | SuccessSuccess rate on the selected goal definition: (runs - failed runs) / runs * 100. | Gap vs targetDifference in percentage points between current success and the configured target rate. | Change vs prevHow much the success rate changed compared with the immediately preceding equal-length window, using the same goal definition as this row. | Provision successProvisioning here means infrastructure setup before E2E tests start. A run counts as provision-successful if it got past provisioning, even if it later failed during E2E. Runs that failed for 'Other' CI-infrastructure reasons are excluded because those runs never reached the provision step. DEV rows use post-good runs plus Tide batches. INT/STG/PROD show n/a because provisioning is not part of those environments. | Provision change vs prevChange in provision success, in percentage points, compared with the previous equal-length window. Uses the same provision-success definition as the Provision success column. DEV only; INT/STG/PROD show n/a. | E2E successAmong runs that reached E2E execution, this shows how many completed successfully. Provision failures and 'Other' CI-infrastructure failures are excluded because those runs never reached the E2E step. DEV rows use only runs after the final push to a PR that later merged. INT/STG/PROD use all E2E job runs. | E2E success vs prevChange in E2E success, in percentage points, compared with the previous equal-length window. Uses the same E2E-success definition as the E2E success column. |
|---|---|---|---|---|---|---|---|---|---|
| DEV | 95% - Post-good runs + Tide batches | 396 | 40.91% (off track) | -54.09pp | +7.76pp | 91.82% (348/379) | -2.54pp | 46.55% (162/348) | +8.82pp |
| INT | 95% - All E2E job runs | 29 | 13.79% (off track) | -81.21pp | -39.04pp | n/a | n/a | 100.00% (4/4) | +45.10pp |
| STG | 95% - All E2E job runs | 27 | 22.22% (off track) | -72.78pp | +7.94pp | n/a | n/a | 25.00% (6/24) | +7.98pp |
| PROD | 95% - All E2E job runs | 96 | 52.08% (off track) | -42.92pp | +18.15pp | n/a | n/a | 54.95% (50/91) | +19.76pp |
Environment: DEV
Daily Run Outcomes
Daily Run Outcomes for DEV Goal Basis (post-good + batches)
No tests below 95.00% in this window with at least 10 runs.
Up to 50 failure patterns are loaded (minimum 1.00% impact), with 10 shown by default. Click column headers to re-sort.
Jump to the full Failure Patterns view for this window
| Failure patternThe canonical description of a recurring CI failure, extracted and normalized from raw logs. | Failed atThe stage of the job run where this failure occurred: 'provision' (environment setup, DEV only), 'e2e' (test suite execution), or 'other' (CI infrastructure issues that did not produce a failure pattern). | Number of distinct job runs where this failure pattern was detected. | Percentage of all job runs in this environment affected by this failure pattern during the selected window. | Classification of this failure pattern: Regression (likely caused by a specific PR), Flake (intermittent failure spread across days), Noise (low-quality or generic pattern), or Indeterminate. | TrendShows daily activity for this failure pattern in a trailing window anchored to the selected end date. The sparkline covers at least 7 days and at most 14 days, depending on the current window size. | Also inOther environments where the same failure pattern was also detected during the selected window. | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
alert [hcp] KubePodNotReady firedfull failure pattern: alert [hcp] KubePodNotReady fired Signal: FlakeSignal: Flake — present in 4 prior week(s); spread across 13 days | alert | 77 | 7.74%77 of 995 job runs affected | FlakeSignal: Flake — present in 4 prior week(s); spread across 13 days | Show trend detailsOct 12: 5 · Oct 13: 6 · Oct 14: 4 · Oct 15: 10 · Oct 16: 9 · Oct 17: 1 · Oct 18: 2 · Oct 19: 13 · Oct 20: 8 · Oct 21: 11 · Oct 22: 4 · Oct 23: 3 · Oct 24: 0 · Oct 25: 1 | none | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (3)alert fired 34 time(s) (3 unknown, 31 known) Firing 1: State: Resolved Started: 2026-10-25T11:42:42Z Ended: 2026-10-25T11:50:40Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j7524864-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tlp73656llsbjr851ct21vm7ag358hj-cilium-cl", pod="azure-file-csi-driver-controller-66f6c89876-l6fts", region="westus3", severity="warning" Description: Pod ocm-arohcpci01-2tlp73656llsbjr851ct21vm7ag358hj-cilium-cl/azure-file-csi-driver-controller-66f6c89876-l6fts has been in a non-ready state for longer than 5 minutes. Firing 2: State: Resolved Started: 2026-10-25T11:44:40Z Ended: 2026-10-25T11:50:40Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j7524864-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tlp73656llsbjr851ct21vm7ag358hj-cilium-cl", pod="multus-admission-controller-58547c8cf6-lmftl", region="westus3", severity="warning" Description: Pod ocm-arohcpci01-2tlp73656llsbjr851ct21vm7ag358hj-cilium-cl/multus-admission-controller-58547c8cf6-lmftl has been in a non-ready state for longer than 5 minutes. Firing 3: State: Resolved Started: 2026-10-25T11:53:40Z Ended: 2026-10-25T11:59:42Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j7524864-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tlp70no771drpm6jf2csmj08ibs461h-s2r6f9c0k1q1q9w", pod="etcd-0", region="westus3", severity="warning" Description: Pod ocm-arohcpci01-2tlp70no771drpm6jf2csmj08ibs461h-s2r6f9c0k1q1q9w/etcd-0 has been in a non-ready state for longer than 5 minutes. alert fired 1 time(s) Firing 1: State: Resolved Started: 2026-10-23T15:27:17Z Ended: 2026-10-23T15:33:17Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j5062656-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tkibdc5gs5nul5ksrupnsegrco0ubpn-w6p5u2c0q5q4w4r", pod="ignition-server-proxy-bf7d94666-n9xlf", region="westus3", severity="warning" Description: Pod ocm-arohcpci01-2tkibdc5gs5nul5ksrupnsegrco0ubpn-w6p5u2c0q5q4w4r/ignition-server-proxy-bf7d94666-n9xlf has been in a non-ready state for longer than 5 minutes. alert fired 11 time(s) (3 unknown, 8 known) Firing 1: State: Resolved Started: 2026-10-23T13:16:41Z Ended: 2026-10-23T13:26:48Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j6471040-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tkg8fusv1drnq89nr4jr4kc118h003a-d8f2g8w6t8m3b5s", pod="cluster-network-operator-6857fbf8d7-5cgmz", region="centralus", severity="warning" Description: Pod ocm-arohcpci01-2tkg8fusv1drnq89nr4jr4kc118h003a-d8f2g8w6t8m3b5s/cluster-network-operator-6857fbf8d7-5cgmz has been in a non-ready state for longer than 5 minutes. Firing 2: State: Resolved Started: 2026-10-23T13:16:41Z Ended: 2026-10-23T13:26:48Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j6471040-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tkg8fusv1drnq89nr4jr4kc118h003a-d8f2g8w6t8m3b5s", pod="ingress-operator-787469b994-cfnzc", region="centralus", severity="warning" Description: Pod ocm-arohcpci01-2tkg8fusv1drnq89nr4jr4kc118h003a-d8f2g8w6t8m3b5s/ingress-operator-787469b994-cfnzc has been in a non-ready state for longer than 5 minutes. Firing 3: State: Resolved Started: 2026-10-23T13:16:41Z Ended: 2026-10-23T13:26:48Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j6471040-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tkg8fusv1drnq89nr4jr4kc118h003a-d8f2g8w6t8m3b5s", pod="dns-operator-6d4899cf84-7qfzv", region="centralus", severity="warning" Description: Pod ocm-arohcpci01-2tkg8fusv1drnq89nr4jr4kc118h003a-d8f2g8w6t8m3b5s/dns-operator-6d4899cf84-7qfzv has been in a non-ready state for longer than 5 minutes. Contributing tests (1)
Affected runs (77) | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
alert [svc] BackendControllerQueueDepthHigh firedfull failure pattern: alert [svc] BackendControllerQueueDepthHigh fired Signal: FlakeSignal: Flake — present in 4 prior week(s); spread across 11 days | alert | 76 | 7.64%76 of 995 job runs affected | FlakeSignal: Flake — present in 4 prior week(s); spread across 11 days | Show trend detailsOct 12: 8 · Oct 13: 9 · Oct 14: 12 · Oct 15: 11 · Oct 16: 6 · Oct 17: 0 · Oct 18: 0 · Oct 19: 6 · Oct 20: 12 · Oct 21: 8 · Oct 22: 2 · Oct 23: 1 · Oct 24: 1 · Oct 25: 0 | none | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (3)alert fired 2 time(s) Firing 1: State: Resolved Started: 2026-10-24T01:56:07Z Ended: 2026-10-24T02:15:08Z Severity: Sev3 Labels: alertname="BackendControllerQueueDepthHigh", cluster="ci01-j0562432-svc", component="backend", name="csstatedump", region="centralus", severity="warning" Description: Backend controller workqueue csstatedump has had a depth > 10 for more than 15 minutes, indicating work is accumulating faster than it can be processed. Firing 2: State: Resolved Started: 2026-10-24T01:57:06Z Ended: 2026-10-24T02:15:08Z Severity: Sev3 Labels: alertname="BackendControllerQueueDepthHigh", cluster="ci01-j0562432-svc", component="backend", name="clusterresources", region="centralus", severity="warning" Description: Backend controller workqueue clusterresources has had a depth > 10 for more than 15 minutes, indicating work is accumulating faster than it can be processed. alert fired 2 time(s) Firing 1: State: Resolved Started: 2026-10-23T08:40:51Z Ended: 2026-10-23T08:58:52Z Severity: Sev3 Labels: alertname="BackendControllerQueueDepthHigh", cluster="ci01-j1650944-svc", component="backend", name="csstatedump", region="canadacentral", severity="warning" Description: Backend controller workqueue csstatedump has had a depth > 10 for more than 15 minutes, indicating work is accumulating faster than it can be processed. Firing 2: State: Resolved Started: 2026-10-23T08:40:51Z Ended: 2026-10-23T08:57:52Z Severity: Sev3 Labels: alertname="BackendControllerQueueDepthHigh", cluster="ci01-j1650944-svc", component="backend", name="clusterresources", region="canadacentral", severity="warning" Description: Backend controller workqueue clusterresources has had a depth > 10 for more than 15 minutes, indicating work is accumulating faster than it can be processed. alert fired 1 time(s) Firing 1: State: Resolved Started: 2026-10-22T03:40:46Z Ended: 2026-10-22T03:51:47Z Severity: Sev3 Labels: alertname="BackendControllerQueueDepthHigh", cluster="ci01-j6940672-svc", component="backend", name="clusterresources", region="centralus", severity="warning" Description: Backend controller workqueue clusterresources has had a depth > 10 for more than 15 minutes, indicating work is accumulating faster than it can be processed. Contributing tests (1)
Affected runs (76) | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
alert [hcp] KubeDeploymentReplicasMismatch firedfull failure pattern: alert [hcp] KubeDeploymentReplicasMismatch fired Signal: FlakeSignal: Flake — present in 4 prior week(s); spread across 10 days | alert | 54 | 5.43%54 of 995 job runs affected | FlakeSignal: Flake — present in 4 prior week(s); spread across 10 days | Show trend detailsOct 12: 7 · Oct 13: 6 · Oct 14: 11 · Oct 15: 9 · Oct 16: 0 · Oct 17: 0 · Oct 18: 0 · Oct 19: 4 · Oct 20: 9 · Oct 21: 1 · Oct 22: 3 · Oct 23: 3 · Oct 24: 0 · Oct 25: 1 | none | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (3)alert fired 2 time(s) Firing 1: State: Resolved Started: 2026-10-25T14:09:53Z Ended: 2026-10-25T14:23:58Z Severity: Sev3 Labels: alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j2101760-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="capi-provider", endpoint="http", environment="ci01", instance="10.128.64.228:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j2101760/providers/microsoft.monitor/accounts/hcps-j2101760", microsoft.metrics_include_label="service", namespace="ocm-arohcpci01-2tlrehr6k1m1drteiq3dbt6soul2kbq3-v20260901", pod="arohcp-monitor-kube-state-metrics-85d48f46ff-xj2t5", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="centralus", service="arohcp-monitor-kube-state-metrics", severity="warning" Description: Deployment ocm-arohcpci01-2tlrehr6k1m1drteiq3dbt6soul2kbq3-v20260901/capi-provider has not matched the expected number of replicas for longer than 30 minutes. Firing 2: State: Resolved Started: 2026-10-25T14:10:54Z Ended: 2026-10-25T14:19:54Z Severity: Sev3 Labels: alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j2101760-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="capi-provider", endpoint="http", environment="ci01", instance="10.128.64.228:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j2101760/providers/microsoft.monitor/accounts/hcps-j2101760", microsoft.metrics_include_label="service", namespace="ocm-arohcpci01-2tlrehr6k1m1drteiq3dbt6soul2kbq3-v20260901", pod="arohcp-monitor-kube-state-metrics-85d48f46ff-xj2t5", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="centralus", service="arohcp-monitor-kube-state-metrics", severity="warning" Description: Deployment ocm-arohcpci01-2tlrehr6k1m1drteiq3dbt6soul2kbq3-v20260901/capi-provider has not matched the expected number of replicas for longer than 30 minutes. alert fired 2 time(s) Firing 1: State: Resolved Started: 2026-10-23T20:05:44Z Ended: 2026-10-23T20:17:41Z Severity: Sev3 Labels: alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j5932672-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="capi-provider", endpoint="http", environment="ci01", instance="10.128.64.248:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j5932672/providers/microsoft.monitor/accounts/hcps-j5932672", microsoft.metrics_include_label="service", namespace="ocm-arohcpci01-2tkmf8a67er4ddrbv9lckvi84f56n756-w1f1k6c5p9d8s2z", pod="arohcp-monitor-kube-state-metrics-85d48f46ff-hs2b5", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="centralus", service="arohcp-monitor-kube-state-metrics", severity="warning" Description: Deployment ocm-arohcpci01-2tkmf8a67er4ddrbv9lckvi84f56n756-w1f1k6c5p9d8s2z/capi-provider has not matched the expected number of replicas for longer than 30 minutes. Firing 2: State: Resolved Started: 2026-10-23T20:05:45Z Ended: 2026-10-23T20:17:41Z Severity: Sev3 Labels: alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j5932672-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="capi-provider", endpoint="http", environment="ci01", instance="10.128.64.248:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j5932672/providers/microsoft.monitor/accounts/hcps-j5932672", microsoft.metrics_include_label="service", namespace="ocm-arohcpci01-2tkmf8a67er4ddrbv9lckvi84f56n756-w1f1k6c5p9d8s2z", pod="arohcp-monitor-kube-state-metrics-85d48f46ff-hs2b5", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="centralus", service="arohcp-monitor-kube-state-metrics", severity="warning" Description: Deployment ocm-arohcpci01-2tkmf8a67er4ddrbv9lckvi84f56n756-w1f1k6c5p9d8s2z/capi-provider has not matched the expected number of replicas for longer than 30 minutes. alert fired 2 time(s) Firing 1: State: Resolved Started: 2026-10-23T18:28:28Z Ended: 2026-10-23T18:40:28Z Severity: Sev3 Labels: alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j8170368-mgmt-1", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="capi-provider", endpoint="http", environment="ci01", instance="10.128.64.219:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j8170368/providers/microsoft.monitor/accounts/hcps-j8170368", microsoft.metrics_include_label="service", namespace="ocm-arohcpci01-2tkl2ehrrq3bfgsofqiq3vb4ha9jrgif-l1v8c7t2c1o4j9z", pod="arohcp-monitor-kube-state-metrics-85d48f46ff-d4dzl", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning" Description: Deployment ocm-arohcpci01-2tkl2ehrrq3bfgsofqiq3vb4ha9jrgif-l1v8c7t2c1o4j9z/capi-provider has not matched the expected number of replicas for longer than 30 minutes. Firing 2: State: Resolved Started: 2026-10-23T18:28:28Z Ended: 2026-10-23T18:40:28Z Severity: Sev3 Labels: alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j8170368-mgmt-1", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="capi-provider", endpoint="http", environment="ci01", instance="10.128.64.219:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j8170368/providers/microsoft.monitor/accounts/hcps-j8170368", microsoft.metrics_include_label="service", namespace="ocm-arohcpci01-2tkl2ehrrq3bfgsofqiq3vb4ha9jrgif-l1v8c7t2c1o4j9z", pod="arohcp-monitor-kube-state-metrics-85d48f46ff-d4dzl", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning" Description: Deployment ocm-arohcpci01-2tkl2ehrrq3bfgsofqiq3vb4ha9jrgif-l1v8c7t2c1o4j9z/capi-provider has not matched the expected number of replicas for longer than 30 minutes. Contributing tests (1)
Affected runs (54) | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
failed to create nodepool <nodepool>: failed waiting for nodepool="<nodepool>" for cluster "<cluster>" in resourcegroup="<resource-group>" to finish creating: GET <url>full failure pattern: failed to create nodepool <nodepool>: failed waiting for nodepool="<nodepool>" for cluster "<cluster>" in resourcegroup="<resource-group>" to finish creating: GET <url> Signal: FlakeSignal: Flake — present in 4 prior week(s); spread across 11 days | e2e | 41 | 4.12%41 of 995 job runs affected | FlakeSignal: Flake — present in 4 prior week(s); spread across 11 days | Show trend detailsOct 12: 3 · Oct 13: 7 · Oct 14: 5 · Oct 15: 4 · Oct 16: 2 · Oct 17: 0 · Oct 18: 0 · Oct 19: 4 · Oct 20: 6 · Oct 21: 1 · Oct 22: 2 · Oct 23: 6 · Oct 24: 1 · Oct 25: 0 | none | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (3)fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_delete.go:128]: failed to create node pools
Expected success, but got an error:
<*fmt.wrapError | 0x19f71c1da380>:
failed to create NodePool np-deletable: failed waiting for nodepool="np-deletable" for cluster "np-delete-hcp-cluster" in resourcegroup="nodepool-delete-frlx5lsjl64c" to finish creating: GET https://rp.j5387264.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/6eb737a8-b808-43b4-bafc-d691d15a4945
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/6eb737a8-b808-43b4-bafc-d691d15a4945",
"name": "6eb737a8-b808-43b4-bafc-d691d15a4945",
"status": "Failed",
"startTime": "2026-10-24T02:13:35.7654536Z",
"endTime": "2026-10-24T02:32:39.313667809Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine y9f1n7k6s3b3s7k-np-deletable-csm74-d5dxn: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-frlx5lsjl64c--managed/providers/Microsoft.Compute/virtualMachines/y9f1n7k6s3b3s7k-np-deletable-csm74-d5dxn to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine y9f1n7k6s3b3s7k-np-deletable-csm74-d5dxn: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-frlx5lsjl64c--managed/providers/Microsoft.Compute/virtualMachines/y9f1n7k6s3b3s7k-np-deletable-csm74-d5dxn to exist\n"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://rp.j5387264.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/6eb737a8-b808-43b4-bafc-d691d15a4945
correlation ID: b38b110a-9400-47b0-9d30-29c40b6745e7
client request ID: 160823fa-0861-4cd5-b970-f7ba5cee199c
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_delete.go:128]: failed to create node pools
Expected success, but got an error:
<*fmt.wrapError | 0x19f71c1da380>:
failed to create NodePool np-deletable: failed waiting for nodepool="np-deletable" for cluster "np-delete-hcp-cluster" in resourcegroup="nodepool-delete-frlx5lsjl64c" to finish creating: GET https://rp.j5387264.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/6eb737a8-b808-43b4-bafc-d691d15a4945
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/6eb737a8-b808-43b4-bafc-d691d15a4945",
"name": "6eb737a8-b808-43b4-bafc-d691d15a4945",
"status": "Failed",
"startTime": "2026-10-24T02:13:35.7654536Z",
"endTime": "2026-10-24T02:32:39.313667809Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine y9f1n7k6s3b3s7k-np-deletable-csm74-d5dxn: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-frlx5lsjl64c--managed/providers/Microsoft.Compute/virtualMachines/y9f1n7k6s3b3s7k-np-deletable-csm74-d5dxn to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine y9f1n7k6s3b3s7k-np-deletable-csm74-d5dxn: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-frlx5lsjl64c--managed/providers/Microsoft.Compute/virtualMachines/y9f1n7k6s3b3s7k-np-deletable-csm74-d5dxn to exist\n"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://rp.j5387264.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/6eb737a8-b808-43b4-bafc-d691d15a4945
correlation ID: b38b110a-9400-47b0-9d30-29c40b6745e7
client request ID: 160823fa-0861-4cd5-b970-f7ba5cee199cfail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:165]: failed to create node pool
Expected success, but got an error:
<*fmt.wrapError | 0x26e13640a140>:
failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "ingress-cert-le" in resourcegroup="ingress-cert-le-jft4smjjlxxw" to finish creating: GET https://rp.j8222464.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/fb71e438-88d3-4b63-82a0-b820e717c220
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/fb71e438-88d3-4b63-82a0-b820e717c220",
"name": "fb71e438-88d3-4b63-82a0-b820e717c220",
"status": "Failed",
"startTime": "2026-10-24T00:13:46.41509851Z",
"endTime": "2026-10-24T00:32:46.057566699Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift node pool not cached yet"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://rp.j8222464.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/fb71e438-88d3-4b63-82a0-b820e717c220
correlation ID: 97b55a4e-bfeb-47d6-99bd-fc35a2cc6359
client request ID: c7f67463-49ac-4db2-9fc4-28f026c30acf
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:165]: failed to create node pool
Expected success, but got an error:
<*fmt.wrapError | 0x26e13640a140>:
failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "ingress-cert-le" in resourcegroup="ingress-cert-le-jft4smjjlxxw" to finish creating: GET https://rp.j8222464.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/fb71e438-88d3-4b63-82a0-b820e717c220
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/fb71e438-88d3-4b63-82a0-b820e717c220",
"name": "fb71e438-88d3-4b63-82a0-b820e717c220",
"status": "Failed",
"startTime": "2026-10-24T00:13:46.41509851Z",
"endTime": "2026-10-24T00:32:46.057566699Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift node pool not cached yet"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://rp.j8222464.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/fb71e438-88d3-4b63-82a0-b820e717c220
correlation ID: 97b55a4e-bfeb-47d6-99bd-fc35a2cc6359
client request ID: c7f67463-49ac-4db2-9fc4-28f026c30acffail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:165]: failed to create node pool
Expected success, but got an error:
<*fmt.wrapError | 0xa03dff15100>:
failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "ingress-cert-le" in resourcegroup="ingress-cert-le-k98p95t7rwlk" to finish creating: GET https://rp.j3087360.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/1937a216-e9d1-46bb-b9f8-a55a34e91da8
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/1937a216-e9d1-46bb-b9f8-a55a34e91da8",
"name": "1937a216-e9d1-46bb-b9f8-a55a34e91da8",
"status": "Failed",
"startTime": "2026-10-23T20:32:20.327838457Z",
"endTime": "2026-10-23T20:51:29.045056606Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine ingress-cert-le-np-1-2p298-8vgqx: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/ingress-cert-le-k98p95t7rwlk--managed/providers/Microsoft.Compute/virtualMachines/ingress-cert-le-np-1-2p298-8vgqx to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine ingress-cert-le-np-1-2p298-8vgqx: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/ingress-cert-le-k98p95t7rwlk--managed/providers/Microsoft.Compute/virtualMachines/ingress-cert-le-np-1-2p298-8vgqx to exist\n"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://rp.j3087360.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/1937a216-e9d1-46bb-b9f8-a55a34e91da8
correlation ID: f3bdc38e-f649-43a0-824c-a60a987dceb4
client request ID: f8924ec9-c563-43bf-b506-67458e3fbd1f
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:165]: failed to create node pool
Expected success, but got an error:
<*fmt.wrapError | 0xa03dff15100>:
failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "ingress-cert-le" in resourcegroup="ingress-cert-le-k98p95t7rwlk" to finish creating: GET https://rp.j3087360.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/1937a216-e9d1-46bb-b9f8-a55a34e91da8
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/1937a216-e9d1-46bb-b9f8-a55a34e91da8",
"name": "1937a216-e9d1-46bb-b9f8-a55a34e91da8",
"status": "Failed",
"startTime": "2026-10-23T20:32:20.327838457Z",
"endTime": "2026-10-23T20:51:29.045056606Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine ingress-cert-le-np-1-2p298-8vgqx: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/ingress-cert-le-k98p95t7rwlk--managed/providers/Microsoft.Compute/virtualMachines/ingress-cert-le-np-1-2p298-8vgqx to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine ingress-cert-le-np-1-2p298-8vgqx: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/ingress-cert-le-k98p95t7rwlk--managed/providers/Microsoft.Compute/virtualMachines/ingress-cert-le-np-1-2p298-8vgqx to exist\n"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://rp.j3087360.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/1937a216-e9d1-46bb-b9f8-a55a34e91da8
correlation ID: f3bdc38e-f649-43a0-824c-a60a987dceb4
client request ID: f8924ec9-c563-43bf-b506-67458e3fbd1fContributing tests (9)
Affected runs (41) | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
alert [svc] LeaderElectionLeaseStale firedfull failure pattern: alert [svc] LeaderElectionLeaseStale fired Signal: FlakeSignal: Flake — present in 2 prior week(s); spread across 5 days | alert | 36 | 3.62%36 of 995 job runs affected | FlakeSignal: Flake — present in 2 prior week(s); spread across 5 days | Show trend detailsOct 12: 5 · Oct 13: 13 · Oct 14: 8 · Oct 15: 9 · Oct 16: 0 · Oct 17: 0 · Oct 18: 0 · Oct 19: 0 · Oct 20: 1 · Oct 21: 0 · Oct 22: 0 · Oct 23: 0 · Oct 24: 0 · Oct 25: 0 | none | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (3)alert fired 3 time(s) Firing 1: State: Fired (not resolved) Started: 2026-10-20T15:16:39Z Severity: Sev3 Labels: alertname="LeaderElectionLeaseStale", cluster="ci01-j1835008-mgmt-1", component="leader-election", container="kube-state-metrics", endpoint="http", environment="ci01", instance="10.128.65.37:8080", job="kube-state-metrics", lease="work-manager", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j1835008/providers/microsoft.monitor/accounts/services-j1835008", namespace="klusterlet-2tiihjhi9hdv5bk2hdv9n1u5gspnntl2", pod="arohcp-monitor-kube-state-metrics-7988dc648c-ggrf6", prometheus="prometheus/prometheus", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning" Description: Leader election lease work-manager in namespace klusterlet-2tiihjhi9hdv5bk2hdv9n1u5gspnntl2 on cluster ci01-j1835008-mgmt-1 has not been renewed for more than 37 minutes. The leadership election might be broken or the component stopped running. Firing 2: State: Fired (not resolved) Started: 2026-10-20T15:18:39Z Severity: Sev3 Labels: alertname="LeaderElectionLeaseStale", cluster="ci01-j1835008-mgmt-1", component="leader-election", container="kube-state-metrics", endpoint="http", environment="ci01", instance="10.128.65.37:8080", job="kube-state-metrics", lease="governance-policy-framework", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j1835008/providers/microsoft.monitor/accounts/services-j1835008", namespace="klusterlet-2tiihjhi9hdv5bk2hdv9n1u5gspnntl2", pod="arohcp-monitor-kube-state-metrics-7988dc648c-ggrf6", prometheus="prometheus/prometheus", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning" Description: Leader election lease governance-policy-framework in namespace klusterlet-2tiihjhi9hdv5bk2hdv9n1u5gspnntl2 on cluster ci01-j1835008-mgmt-1 has not been renewed for more than 37 minutes. The leadership election might be broken or the component stopped running. Firing 3: State: Fired (not resolved) Started: 2026-10-20T15:19:39Z Severity: Sev3 Labels: alertname="LeaderElectionLeaseStale", cluster="ci01-j1835008-mgmt-1", component="leader-election", container="kube-state-metrics", endpoint="http", environment="ci01", instance="10.128.65.37:8080", job="kube-state-metrics", lease="config-policy-controller", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j1835008/providers/microsoft.monitor/accounts/services-j1835008", namespace="klusterlet-2tiihjhi9hdv5bk2hdv9n1u5gspnntl2", pod="arohcp-monitor-kube-state-metrics-7988dc648c-ggrf6", prometheus="prometheus/prometheus", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning" Description: Leader election lease config-policy-controller in namespace klusterlet-2tiihjhi9hdv5bk2hdv9n1u5gspnntl2 on cluster ci01-j1835008-mgmt-1 has not been renewed for more than 37 minutes. The leadership election might be broken or the component stopped running. alert fired 1 time(s) Firing 1: State: Fired (not resolved) Started: 2026-10-15T21:26:40Z Severity: Sev3 Labels: alertname="LeaderElectionLeaseStale", cluster="ci01-j4315776-mgmt-2", component="leader-election", container="kube-state-metrics", endpoint="http", environment="ci01", instance="10.128.64.179:8080", job="kube-state-metrics", lease="work-manager", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j4315776/providers/microsoft.monitor/accounts/services-j4315776", namespace="klusterlet-2tfeeaom86b60erngar03ad51ehpgqtm", pod="arohcp-monitor-kube-state-metrics-7988dc648c-h4l8s", prometheus="prometheus/prometheus", region="centralus", service="arohcp-monitor-kube-state-metrics", severity="warning" Description: Leader election lease work-manager in namespace klusterlet-2tfeeaom86b60erngar03ad51ehpgqtm on cluster ci01-j4315776-mgmt-2 has not been renewed for more than 37 minutes. The leadership election might be broken or the component stopped running. alert fired 1 time(s) Firing 1: State: Fired (not resolved) Started: 2026-10-15T18:05:49Z Severity: Sev3 Labels: alertname="LeaderElectionLeaseStale", cluster="ci01-j7905664-mgmt-1", component="leader-election", container="kube-state-metrics", endpoint="http", environment="ci01", instance="10.128.65.54:8080", job="kube-state-metrics", lease="work-manager", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j7905664/providers/microsoft.monitor/accounts/services-j7905664", namespace="klusterlet-2tfbjf51so3bj30gni3l73dohrjlrc07", pod="arohcp-monitor-kube-state-metrics-7988dc648c-mz4bk", prometheus="prometheus/prometheus", region="centralus", service="arohcp-monitor-kube-state-metrics", severity="warning" Description: Leader election lease work-manager in namespace klusterlet-2tfbjf51so3bj30gni3l73dohrjlrc07 on cluster ci01-j7905664-mgmt-1 has not been renewed for more than 37 minutes. The leadership election might be broken or the component stopped running. Contributing tests (1)
Affected runs (36) | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
alert [hcp] KubePodCrashLooping firedfull failure pattern: alert [hcp] KubePodCrashLooping fired Signal: FlakeSignal: Flake — present in 4 prior week(s); spread across 9 days | alert | 30 | 3.02%30 of 995 job runs affected | FlakeSignal: Flake — present in 4 prior week(s); spread across 9 days | Show trend detailsOct 12: 4 · Oct 13: 2 · Oct 14: 2 · Oct 15: 2 · Oct 16: 0 · Oct 17: 0 · Oct 18: 0 · Oct 19: 5 · Oct 20: 8 · Oct 21: 0 · Oct 22: 2 · Oct 23: 3 · Oct 24: 0 · Oct 25: 2 | none | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (3)alert fired 2 time(s) Firing 1: State: Resolved Started: 2026-10-25T13:57:52Z Ended: 2026-10-25T14:23:56Z Severity: Sev3 Labels: alertname="KubePodCrashLooping", cluster="ci01-j2101760-mgmt-2", component="kubernetes-infrastructure", container="etcd", endpoint="http", environment="ci01", instance="10.128.64.228:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j2101760/providers/microsoft.monitor/accounts/hcps-j2101760", microsoft.metrics_include_label="service", namespace="ocm-arohcpci01-2tlrehr6k1m1drteiq3dbt6soul2kbq3-v20260901", pod="etcd-2", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", reason="CrashLoopBackOff", region="centralus", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="b19b2a4f-abec-4efb-8fa1-a0f49de6f865" Description: Pod ocm-arohcpci01-2tlrehr6k1m1drteiq3dbt6soul2kbq3-v20260901/etcd-2 (etcd) is in waiting state (reason: "CrashLoopBackOff"). Firing 2: State: Resolved Started: 2026-10-25T13:57:52Z Ended: 2026-10-25T14:23:56Z Severity: Sev3 Labels: alertname="KubePodCrashLooping", cluster="ci01-j2101760-mgmt-2", component="kubernetes-infrastructure", container="etcd", endpoint="http", environment="ci01", instance="10.128.64.228:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j2101760/providers/microsoft.monitor/accounts/hcps-j2101760", microsoft.metrics_include_label="service", namespace="ocm-arohcpci01-2tlrehr6k1m1drteiq3dbt6soul2kbq3-v20260901", pod="etcd-2", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", reason="CrashLoopBackOff", region="centralus", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="b19b2a4f-abec-4efb-8fa1-a0f49de6f865" Description: Pod ocm-arohcpci01-2tlrehr6k1m1drteiq3dbt6soul2kbq3-v20260901/etcd-2 (etcd) is in waiting state (reason: "CrashLoopBackOff"). alert fired 3 time(s) Firing 1: State: Resolved Started: 2026-10-25T11:41:40Z Ended: 2026-10-25T11:51:40Z Severity: Sev3 Labels: alertname="KubePodCrashLooping", cluster="ci01-j7524864-mgmt-2", component="kubernetes-infrastructure", container="manager", endpoint="http", environment="ci01", instance="10.128.64.204:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j7524864/providers/microsoft.monitor/accounts/hcps-j7524864", microsoft.metrics_include_label="service", namespace="ocm-arohcpci01-2tlp722bk3stbjb74e8i42b30lftatrs-ingress-cert-le", pod="capi-provider-6d7bfdf44-24wm6", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", reason="CrashLoopBackOff", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="5602ef00-22ff-45dd-b3cb-39a5ba53e966" Description: Pod ocm-arohcpci01-2tlp722bk3stbjb74e8i42b30lftatrs-ingress-cert-le/capi-provider-6d7bfdf44-24wm6 (manager) is in waiting state (reason: "CrashLoopBackOff"). Firing 2: State: Resolved Started: 2026-10-25T11:41:40Z Ended: 2026-10-25T11:51:41Z Severity: Sev3 Labels: alertname="KubePodCrashLooping", cluster="ci01-j7524864-mgmt-2", component="kubernetes-infrastructure", container="manager", endpoint="http", environment="ci01", instance="10.128.64.204:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j7524864/providers/microsoft.monitor/accounts/hcps-j7524864", microsoft.metrics_include_label="service", namespace="ocm-arohcpci01-2tlp722bk3stbjb74e8i42b30lftatrs-ingress-cert-le", pod="capi-provider-6d7bfdf44-24wm6", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", reason="CrashLoopBackOff", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="5602ef00-22ff-45dd-b3cb-39a5ba53e966" Description: Pod ocm-arohcpci01-2tlp722bk3stbjb74e8i42b30lftatrs-ingress-cert-le/capi-provider-6d7bfdf44-24wm6 (manager) is in waiting state (reason: "CrashLoopBackOff"). Firing 3: State: Resolved Started: 2026-10-25T11:43:40Z Ended: 2026-10-25T11:51:41Z Severity: Sev3 Labels: alertname="KubePodCrashLooping", cluster="ci01-j7524864-mgmt-2", component="kubernetes-infrastructure", container="manager", endpoint="http", environment="ci01", instance="10.128.64.204:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j7524864/providers/microsoft.monitor/accounts/hcps-j7524864", microsoft.metrics_include_label="service", namespace="ocm-arohcpci01-2tlp733dhbjgega0lnvcjvgg1grv9tf4-i6w1d6x7y2t6h2v", pod="capi-provider-8f7b799d8-49zjj", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", reason="CrashLoopBackOff", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="706780f8-3192-4947-b780-05f86e3d1b53" Description: Pod ocm-arohcpci01-2tlp733dhbjgega0lnvcjvgg1grv9tf4-i6w1d6x7y2t6h2v/capi-provider-8f7b799d8-49zjj (manager) is in waiting state (reason: "CrashLoopBackOff"). alert fired 2 time(s) Firing 1: State: Resolved Started: 2026-10-23T19:53:47Z Ended: 2026-10-23T20:19:41Z Severity: Sev3 Labels: alertname="KubePodCrashLooping", cluster="ci01-j5932672-mgmt-2", component="kubernetes-infrastructure", container="etcd", endpoint="http", environment="ci01", instance="10.128.64.248:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j5932672/providers/microsoft.monitor/accounts/hcps-j5932672", microsoft.metrics_include_label="service", namespace="ocm-arohcpci01-2tkmf8a67er4ddrbv9lckvi84f56n756-w1f1k6c5p9d8s2z", pod="etcd-0", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", reason="CrashLoopBackOff", region="centralus", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="ad470ce0-381a-4372-b725-f995a508a24f" Description: Pod ocm-arohcpci01-2tkmf8a67er4ddrbv9lckvi84f56n756-w1f1k6c5p9d8s2z/etcd-0 (etcd) is in waiting state (reason: "CrashLoopBackOff"). Firing 2: State: Resolved Started: 2026-10-23T19:53:47Z Ended: 2026-10-23T20:19:41Z Severity: Sev3 Labels: alertname="KubePodCrashLooping", cluster="ci01-j5932672-mgmt-2", component="kubernetes-infrastructure", container="etcd", endpoint="http", environment="ci01", instance="10.128.64.248:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j5932672/providers/microsoft.monitor/accounts/hcps-j5932672", microsoft.metrics_include_label="service", namespace="ocm-arohcpci01-2tkmf8a67er4ddrbv9lckvi84f56n756-w1f1k6c5p9d8s2z", pod="etcd-0", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", reason="CrashLoopBackOff", region="centralus", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="ad470ce0-381a-4372-b725-f995a508a24f" Description: Pod ocm-arohcpci01-2tkmf8a67er4ddrbv9lckvi84f56n756-w1f1k6c5p9d8s2z/etcd-0 (etcd) is in waiting state (reason: "CrashLoopBackOff"). Contributing tests (1)
Affected runs (30) | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedC...full failure pattern: ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider]; provider Microsoft.RedHatOpenShift Signal: FlakeSignal: Flake — present in 4 prior week(s); spread across 9 days | e2e | 29 | 2.91%29 of 995 job runs affected | FlakeSignal: Flake — present in 4 prior week(s); spread across 9 days | Show trend detailsOct 12: 3 · Oct 13: 6 · Oct 14: 3 · Oct 15: 1 · Oct 16: 0 · Oct 17: 0 · Oct 18: 0 · Oct 19: 5 · Oct 20: 8 · Oct 21: 0 · Oct 22: 3 · Oct 23: 2 · Oct 24: 0 · Oct 25: 1 | INT, PROD, STG | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (3)fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_v20260901preview.go:86]: failed to create HCP cluster "v20260901" via v20260901preview
Unexpected error:
<*fmt.wrapError | 0x209528ec2340>:
failed to create HCP cluster v20260901: failed waiting for cluster="v20260901" in resourcegroup="v20260901preview-ktmjg5k5mr4p" to finish creating: GET https://rp.j2101760.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/cd12106f-dc74-4bb2-88d9-552815761564
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/cd12106f-dc74-4bb2-88d9-552815761564",
"name": "cd12106f-dc74-4bb2-88d9-552815761564",
"status": "Failed",
"startTime": "2026-10-25T13:33:58.83934802Z",
"endTime": "2026-10-25T14:03:01.452063547Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://rp.j2101760.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/cd12106f-dc74-4bb2-88d9-552815761564
correlation ID: a999ca91-1f04-4667-8416-be09ce537e26
client request ID: e9d482d5-6b2f-4cbd-99d8-1b5bb0124d29
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_v20260901preview.go:86]: failed to create HCP cluster "v20260901" via v20260901preview
Unexpected error:
<*fmt.wrapError | 0x209528ec2340>:
failed to create HCP cluster v20260901: failed waiting for cluster="v20260901" in resourcegroup="v20260901preview-ktmjg5k5mr4p" to finish creating: GET https://rp.j2101760.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/cd12106f-dc74-4bb2-88d9-552815761564
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/cd12106f-dc74-4bb2-88d9-552815761564",
"name": "cd12106f-dc74-4bb2-88d9-552815761564",
"status": "Failed",
"startTime": "2026-10-25T13:33:58.83934802Z",
"endTime": "2026-10-25T14:03:01.452063547Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://rp.j2101760.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/cd12106f-dc74-4bb2-88d9-552815761564
correlation ID: a999ca91-1f04-4667-8416-be09ce537e26
client request ID: e9d482d5-6b2f-4cbd-99d8-1b5bb0124d29
occurredfail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:126]: HCP cluster rg-candidate-4-22-hbhgzqp5zds5/cluster-candidate-4-22-9t7f9b should provision
Unexpected error:
<*fmt.wrapError | 0x2655fb8fc880>:
failed to create HCP cluster cluster-candidate-4-22-9t7f9b: failed waiting for cluster="cluster-candidate-4-22-9t7f9b" in resourcegroup="rg-candidate-4-22-hbhgzqp5zds5" to finish creating: GET https://rp.j5932672.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/9cab47e8-7ab4-4765-9ac8-c0235f97fa6d
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/9cab47e8-7ab4-4765-9ac8-c0235f97fa6d",
"name": "9cab47e8-7ab4-4765-9ac8-c0235f97fa6d",
"status": "Failed",
"startTime": "2026-10-23T19:29:36.620172443Z",
"endTime": "2026-10-23T19:58:43.098929016Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://rp.j5932672.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/9cab47e8-7ab4-4765-9ac8-c0235f97fa6d
correlation ID: 91833dfb-2346-485c-9abf-90635cc143aa
client request ID: 8cd6038e-c1e2-4db7-aaf4-36204667b8f5
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:126]: HCP cluster rg-candidate-4-22-hbhgzqp5zds5/cluster-candidate-4-22-9t7f9b should provision
Unexpected error:
<*fmt.wrapError | 0x2655fb8fc880>:
failed to create HCP cluster cluster-candidate-4-22-9t7f9b: failed waiting for cluster="cluster-candidate-4-22-9t7f9b" in resourcegroup="rg-candidate-4-22-hbhgzqp5zds5" to finish creating: GET https://rp.j5932672.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/9cab47e8-7ab4-4765-9ac8-c0235f97fa6d
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/9cab47e8-7ab4-4765-9ac8-c0235f97fa6d",
"name": "9cab47e8-7ab4-4765-9ac8-c0235f97fa6d",
"status": "Failed",
"startTime": "2026-10-23T19:29:36.620172443Z",
"endTime": "2026-10-23T19:58:43.098929016Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://rp.j5932672.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/9cab47e8-7ab4-4765-9ac8-c0235f97fa6d
correlation ID: 91833dfb-2346-485c-9abf-90635cc143aa
client request ID: 8cd6038e-c1e2-4db7-aaf4-36204667b8f5
occurredfail [github.com/Azure/ARO-HCP/test/e2e/admin_api.go:312]: failed to create HCP cluster "sre-hcp-cluster-sc" for serial console test
Unexpected error:
<*fmt.wrapError | 0x29d3447c6480>:
failed to create HCP cluster sre-hcp-cluster-sc: failed waiting for cluster="sre-hcp-cluster-sc" in resourcegroup="admin-api-serialconsole-gbjmnzzdfntv" to finish creating: GET https://rp.j8170368.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/644d90e9-61e3-42ef-a2dd-083bda41c8ae
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/644d90e9-61e3-42ef-a2dd-083bda41c8ae",
"name": "644d90e9-61e3-42ef-a2dd-083bda41c8ae",
"status": "Failed",
"startTime": "2026-10-23T17:53:58.718650875Z",
"endTime": "2026-10-23T18:23:02.815422763Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://rp.j8170368.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/644d90e9-61e3-42ef-a2dd-083bda41c8ae
correlation ID: 6d68ce4a-22a8-4eea-a977-b246a66b35ea
client request ID: 9a628c19-f8fd-4bf6-859e-bb2f0567065e
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/admin_api.go:312]: failed to create HCP cluster "sre-hcp-cluster-sc" for serial console test
Unexpected error:
<*fmt.wrapError | 0x29d3447c6480>:
failed to create HCP cluster sre-hcp-cluster-sc: failed waiting for cluster="sre-hcp-cluster-sc" in resourcegroup="admin-api-serialconsole-gbjmnzzdfntv" to finish creating: GET https://rp.j8170368.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/644d90e9-61e3-42ef-a2dd-083bda41c8ae
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/644d90e9-61e3-42ef-a2dd-083bda41c8ae",
"name": "644d90e9-61e3-42ef-a2dd-083bda41c8ae",
"status": "Failed",
"startTime": "2026-10-23T17:53:58.718650875Z",
"endTime": "2026-10-23T18:23:02.815422763Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://rp.j8170368.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/644d90e9-61e3-42ef-a2dd-083bda41c8ae
correlation ID: 6d68ce4a-22a8-4eea-a977-b246a66b35ea
client request ID: 9a628c19-f8fd-4bf6-859e-bb2f0567065e
occurredContributing tests (24)
Affected runs (29) | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
alert [hcp] KubeStatefulSetReplicasMismatch firedfull failure pattern: alert [hcp] KubeStatefulSetReplicasMismatch fired Signal: FlakeSignal: Flake — present in 4 prior week(s); spread across 9 days | alert | 27 | 2.71%27 of 995 job runs affected | FlakeSignal: Flake — present in 4 prior week(s); spread across 9 days | Show trend detailsOct 12: 3 · Oct 13: 3 · Oct 14: 2 · Oct 15: 1 · Oct 16: 0 · Oct 17: 0 · Oct 18: 0 · Oct 19: 5 · Oct 20: 8 · Oct 21: 0 · Oct 22: 2 · Oct 23: 2 · Oct 24: 0 · Oct 25: 1 | none | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (3)alert fired 2 time(s) Firing 1: State: Resolved Started: 2026-10-25T13:55:59Z Ended: 2026-10-25T14:19:54Z Severity: Sev3 Labels: alertname="KubeStatefulSetReplicasMismatch", cluster="ci01-j2101760-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", endpoint="http", environment="ci01", instance="10.128.64.228:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j2101760/providers/microsoft.monitor/accounts/hcps-j2101760", microsoft.metrics_include_label="service", namespace="ocm-arohcpci01-2tlrehr6k1m1drteiq3dbt6soul2kbq3-v20260901", pod="arohcp-monitor-kube-state-metrics-85d48f46ff-xj2t5", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="centralus", service="arohcp-monitor-kube-state-metrics", severity="warning", statefulset="etcd" Description: StatefulSet ocm-arohcpci01-2tlrehr6k1m1drteiq3dbt6soul2kbq3-v20260901/etcd has not matched the expected number of replicas for longer than 15 minutes. Firing 2: State: Resolved Started: 2026-10-25T14:06:53Z Ended: 2026-10-25T14:19:54Z Severity: Sev3 Labels: alertname="KubeStatefulSetReplicasMismatch", cluster="ci01-j2101760-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", endpoint="http", environment="ci01", instance="10.128.64.228:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j2101760/providers/microsoft.monitor/accounts/hcps-j2101760", microsoft.metrics_include_label="service", namespace="ocm-arohcpci01-2tlrehr6k1m1drteiq3dbt6soul2kbq3-v20260901", pod="arohcp-monitor-kube-state-metrics-85d48f46ff-xj2t5", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="centralus", service="arohcp-monitor-kube-state-metrics", severity="warning", statefulset="etcd" Description: StatefulSet ocm-arohcpci01-2tlrehr6k1m1drteiq3dbt6soul2kbq3-v20260901/etcd has not matched the expected number of replicas for longer than 15 minutes. alert fired 2 time(s) Firing 1: State: Resolved Started: 2026-10-23T19:51:45Z Ended: 2026-10-23T20:15:45Z Severity: Sev3 Labels: alertname="KubeStatefulSetReplicasMismatch", cluster="ci01-j5932672-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", endpoint="http", environment="ci01", instance="10.128.64.248:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j5932672/providers/microsoft.monitor/accounts/hcps-j5932672", microsoft.metrics_include_label="service", namespace="ocm-arohcpci01-2tkmf8a67er4ddrbv9lckvi84f56n756-w1f1k6c5p9d8s2z", pod="arohcp-monitor-kube-state-metrics-85d48f46ff-hs2b5", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="centralus", service="arohcp-monitor-kube-state-metrics", severity="warning", statefulset="etcd" Description: StatefulSet ocm-arohcpci01-2tkmf8a67er4ddrbv9lckvi84f56n756-w1f1k6c5p9d8s2z/etcd has not matched the expected number of replicas for longer than 15 minutes. Firing 2: State: Resolved Started: 2026-10-23T19:51:45Z Ended: 2026-10-23T20:15:45Z Severity: Sev3 Labels: alertname="KubeStatefulSetReplicasMismatch", cluster="ci01-j5932672-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", endpoint="http", environment="ci01", instance="10.128.64.248:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j5932672/providers/microsoft.monitor/accounts/hcps-j5932672", microsoft.metrics_include_label="service", namespace="ocm-arohcpci01-2tkmf8a67er4ddrbv9lckvi84f56n756-w1f1k6c5p9d8s2z", pod="arohcp-monitor-kube-state-metrics-85d48f46ff-hs2b5", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="centralus", service="arohcp-monitor-kube-state-metrics", severity="warning", statefulset="etcd" Description: StatefulSet ocm-arohcpci01-2tkmf8a67er4ddrbv9lckvi84f56n756-w1f1k6c5p9d8s2z/etcd has not matched the expected number of replicas for longer than 15 minutes. alert fired 2 time(s) Firing 1: State: Resolved Started: 2026-10-23T18:14:34Z Ended: 2026-10-23T18:39:29Z Severity: Sev3 Labels: alertname="KubeStatefulSetReplicasMismatch", cluster="ci01-j8170368-mgmt-1", component="kubernetes-infrastructure", container="kube-state-metrics", endpoint="http", environment="ci01", instance="10.128.64.219:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j8170368/providers/microsoft.monitor/accounts/hcps-j8170368", microsoft.metrics_include_label="service", namespace="ocm-arohcpci01-2tkl2ehrrq3bfgsofqiq3vb4ha9jrgif-l1v8c7t2c1o4j9z", pod="arohcp-monitor-kube-state-metrics-85d48f46ff-d4dzl", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", statefulset="etcd" Description: StatefulSet ocm-arohcpci01-2tkl2ehrrq3bfgsofqiq3vb4ha9jrgif-l1v8c7t2c1o4j9z/etcd has not matched the expected number of replicas for longer than 15 minutes. Firing 2: State: Resolved Started: 2026-10-23T18:14:35Z Ended: 2026-10-23T18:39:29Z Severity: Sev3 Labels: alertname="KubeStatefulSetReplicasMismatch", cluster="ci01-j8170368-mgmt-1", component="kubernetes-infrastructure", container="kube-state-metrics", endpoint="http", environment="ci01", instance="10.128.64.219:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j8170368/providers/microsoft.monitor/accounts/hcps-j8170368", microsoft.metrics_include_label="service", namespace="ocm-arohcpci01-2tkl2ehrrq3bfgsofqiq3vb4ha9jrgif-l1v8c7t2c1o4j9z", pod="arohcp-monitor-kube-state-metrics-85d48f46ff-d4dzl", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", statefulset="etcd" Description: StatefulSet ocm-arohcpci01-2tkl2ehrrq3bfgsofqiq3vb4ha9jrgif-l1v8c7t2c1o4j9z/etcd has not matched the expected number of replicas for longer than 15 minutes. Contributing tests (1)
Affected runs (27) | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
ERROR CODE: DeadlineExceeded; detail message node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hype...full failure pattern: ERROR CODE: DeadlineExceeded; detail message node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift node pool not cached yet; provider Microsoft.RedHatOpenShift Signal: FlakeSignal: Flake — present in 2 prior week(s); spread across 10 days | e2e | 23 | 2.31%23 of 995 job runs affected | FlakeSignal: Flake — present in 2 prior week(s); spread across 10 days | Show trend detailsOct 12: 1 · Oct 13: 1 · Oct 14: 4 · Oct 15: 5 · Oct 16: 1 · Oct 17: 0 · Oct 18: 0 · Oct 19: 2 · Oct 20: 6 · Oct 21: 2 · Oct 22: 0 · Oct 23: 5 · Oct 24: 0 · Oct 25: 1 | PROD | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (3)fail [github.com/Azure/ARO-HCP/test/e2e/cluster_pullsecret.go:178]: failed to create node pool np-1 for pull secret cluster
Unexpected error:
<*fmt.wrapError | 0x34e2e46f8680>:
failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "pullsecret-hcp-cluster" in resourcegroup="pullsecret-test-mfmrpz5xbrdj" to finish creating: GET https://rp.j1877504.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/3d1a2520-89cf-4c42-a91c-ec725b89af14
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/3d1a2520-89cf-4c42-a91c-ec725b89af14",
"name": "3d1a2520-89cf-4c42-a91c-ec725b89af14",
"status": "Failed",
"startTime": "2026-10-25T11:04:09.039925979Z",
"endTime": "2026-10-25T11:23:09.310567948Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift node pool not cached yet"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://rp.j1877504.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/3d1a2520-89cf-4c42-a91c-ec725b89af14
correlation ID: 9acdbad6-f09a-46a0-a926-c738c62f3c07
client request ID: 916943ff-0406-46da-9830-2889e6355ff3
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_pullsecret.go:178]: failed to create node pool np-1 for pull secret cluster
Unexpected error:
<*fmt.wrapError | 0x34e2e46f8680>:
failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "pullsecret-hcp-cluster" in resourcegroup="pullsecret-test-mfmrpz5xbrdj" to finish creating: GET https://rp.j1877504.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/3d1a2520-89cf-4c42-a91c-ec725b89af14
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/3d1a2520-89cf-4c42-a91c-ec725b89af14",
"name": "3d1a2520-89cf-4c42-a91c-ec725b89af14",
"status": "Failed",
"startTime": "2026-10-25T11:04:09.039925979Z",
"endTime": "2026-10-25T11:23:09.310567948Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift node pool not cached yet"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://rp.j1877504.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/3d1a2520-89cf-4c42-a91c-ec725b89af14
correlation ID: 9acdbad6-f09a-46a0-a926-c738c62f3c07
client request ID: 916943ff-0406-46da-9830-2889e6355ff3
occurredfail [github.com/Azure/ARO-HCP/test/e2e/cluster_nodepool_active_versions.go:96]: failed to create node pool "np-1" for v20261001 cluster "v20261001"
Unexpected error:
<*fmt.wrapError | 0x3dd71ecc7860>:
failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "v20261001" in resourcegroup="v20261001-gvwnv545c9xf" to finish creating: GET https://rp.j1278720.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/b77ed110-707e-4c38-ab0a-33ad172ee68c
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/b77ed110-707e-4c38-ab0a-33ad172ee68c",
"name": "b77ed110-707e-4c38-ab0a-33ad172ee68c",
"status": "Failed",
"startTime": "2026-10-23T21:19:37.486805843Z",
"endTime": "2026-10-23T21:38:45.647369611Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift node pool not cached yet"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://rp.j1278720.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/b77ed110-707e-4c38-ab0a-33ad172ee68c
correlation ID: 6572f7b9-3e5a-422f-aa65-926c920bbad1
client request ID: bb39b90a-4ada-4996-91b9-6bd38bf61535
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_nodepool_active_versions.go:96]: failed to create node pool "np-1" for v20261001 cluster "v20261001"
Unexpected error:
<*fmt.wrapError | 0x3dd71ecc7860>:
failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "v20261001" in resourcegroup="v20261001-gvwnv545c9xf" to finish creating: GET https://rp.j1278720.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/b77ed110-707e-4c38-ab0a-33ad172ee68c
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/b77ed110-707e-4c38-ab0a-33ad172ee68c",
"name": "b77ed110-707e-4c38-ab0a-33ad172ee68c",
"status": "Failed",
"startTime": "2026-10-23T21:19:37.486805843Z",
"endTime": "2026-10-23T21:38:45.647369611Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift node pool not cached yet"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://rp.j1278720.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/b77ed110-707e-4c38-ab0a-33ad172ee68c
correlation ID: 6572f7b9-3e5a-422f-aa65-926c920bbad1
client request ID: bb39b90a-4ada-4996-91b9-6bd38bf61535
occurredfail [github.com/Azure/ARO-HCP/test/e2e/admin_api.go:328]: failed to create worker nodepool for serial console test
Unexpected error:
<*fmt.wrapError | 0x204486f8560>:
failed to create NodePool worker: failed waiting for nodepool="worker" for cluster "sre-hcp-cluster-sc" in resourcegroup="admin-api-serialconsole-wrxzvkddb94j" to finish creating: GET https://rp.j1278720.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/13760d44-2d42-4c53-9fca-8d97381d23a5
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/13760d44-2d42-4c53-9fca-8d97381d23a5",
"name": "13760d44-2d42-4c53-9fca-8d97381d23a5",
"status": "Failed",
"startTime": "2026-10-23T21:19:36.261324147Z",
"endTime": "2026-10-23T21:38:45.545540127Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift node pool not cached yet"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://rp.j1278720.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/13760d44-2d42-4c53-9fca-8d97381d23a5
correlation ID: 5d55a2cb-56aa-47f7-a5b9-cf7b9a27d226
client request ID: f043a8e7-f2d1-473c-b2e3-11abda25efb3
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/admin_api.go:328]: failed to create worker nodepool for serial console test
Unexpected error:
<*fmt.wrapError | 0x204486f8560>:
failed to create NodePool worker: failed waiting for nodepool="worker" for cluster "sre-hcp-cluster-sc" in resourcegroup="admin-api-serialconsole-wrxzvkddb94j" to finish creating: GET https://rp.j1278720.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/13760d44-2d42-4c53-9fca-8d97381d23a5
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/13760d44-2d42-4c53-9fca-8d97381d23a5",
"name": "13760d44-2d42-4c53-9fca-8d97381d23a5",
"status": "Failed",
"startTime": "2026-10-23T21:19:36.261324147Z",
"endTime": "2026-10-23T21:38:45.545540127Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift node pool not cached yet"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://rp.j1278720.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/13760d44-2d42-4c53-9fca-8d97381d23a5
correlation ID: 5d55a2cb-56aa-47f7-a5b9-cf7b9a27d226
client request ID: f043a8e7-f2d1-473c-b2e3-11abda25efb3
occurredContributing tests (18)
Affected runs (23) | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
cluster <resource-group>/<cluster> entered terminal state "Failed" after role assignment deployment; expected "Succeeded"full failure pattern: cluster <resource-group>/<cluster> entered terminal state "Failed" after role assignment deployment; expected "Succeeded" Signal: FlakeSignal: Flake — present in 3 prior week(s); spread across 5 days | e2e | 23 | 2.31%23 of 995 job runs affected | FlakeSignal: Flake — present in 3 prior week(s); spread across 5 days | Show trend detailsOct 12: 5 · Oct 13: 6 · Oct 14: 7 · Oct 15: 4 · Oct 16: 1 · Oct 17: 0 · Oct 18: 0 · Oct 19: 0 · Oct 20: 0 · Oct 21: 0 · Oct 22: 0 · Oct 23: 0 · Oct 24: 0 · Oct 25: 0 | PROD | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (3)fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:227]: Timed out after 2580.003s.
cluster should eventually succeed after role assignments are created
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:223 with:
cluster delayed-rbac-9jbnpswbqhnc/delayed-rbac-cluster entered terminal state "Failed" after role assignment deployment; expected "Succeeded"
Expected
<armredhatopenshifthcp.ProvisioningState>: Failed
not to equal
<armredhatopenshifthcp.ProvisioningState>: Failed
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:227]: Timed out after 2580.003s.
cluster should eventually succeed after role assignments are created
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:223 with:
cluster delayed-rbac-9jbnpswbqhnc/delayed-rbac-cluster entered terminal state "Failed" after role assignment deployment; expected "Succeeded"
Expected
<armredhatopenshifthcp.ProvisioningState>: Failed
not to equal
<armredhatopenshifthcp.ProvisioningState>: Failedfail [github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:227]: Timed out after 2580.002s.
cluster should eventually succeed after role assignments are created
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:223 with:
cluster delayed-rbac-kzwmzc2svmh2/delayed-rbac-cluster entered terminal state "Failed" after role assignment deployment; expected "Succeeded"
Expected
<armredhatopenshifthcp.ProvisioningState>: Failed
not to equal
<armredhatopenshifthcp.ProvisioningState>: Failed
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:227]: Timed out after 2580.002s.
cluster should eventually succeed after role assignments are created
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:223 with:
cluster delayed-rbac-kzwmzc2svmh2/delayed-rbac-cluster entered terminal state "Failed" after role assignment deployment; expected "Succeeded"
Expected
<armredhatopenshifthcp.ProvisioningState>: Failed
not to equal
<armredhatopenshifthcp.ProvisioningState>: Failedfail [github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:227]: Timed out after 2580.001s.
cluster should eventually succeed after role assignments are created
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:223 with:
cluster delayed-rbac-fs95n68z9x97/delayed-rbac-cluster entered terminal state "Failed" after role assignment deployment; expected "Succeeded"
Expected
<armredhatopenshifthcp.ProvisioningState>: Failed
not to equal
<armredhatopenshifthcp.ProvisioningState>: Failed
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:227]: Timed out after 2580.001s.
cluster should eventually succeed after role assignments are created
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:223 with:
cluster delayed-rbac-fs95n68z9x97/delayed-rbac-cluster entered terminal state "Failed" after role assignment deployment; expected "Succeeded"
Expected
<armredhatopenshifthcp.ProvisioningState>: Failed
not to equal
<armredhatopenshifthcp.ProvisioningState>: FailedContributing tests (1)
Affected runs (23) | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
DNS for route host <route-host> did not resolve: DNS for <route-host> did not resolve within <duration> (last error: lookup <host> on <dns-server>: dial udp <ip>: i/o timeout): co...full failure pattern: DNS for route host <route-host> did not resolve: DNS for <route-host> did not resolve within <duration> (last error: lookup <host> on <dns-server>: dial udp <ip>: i/o timeout): context deadline exceeded Signal: FlakeSignal: Flake — present in 4 prior week(s); spread across 10 days | e2e | 19 | 1.91%19 of 995 job runs affected | FlakeSignal: Flake — present in 4 prior week(s); spread across 10 days | Show trend detailsOct 12: 0 · Oct 13: 0 · Oct 14: 5 · Oct 15: 1 · Oct 16: 1 · Oct 17: 0 · Oct 18: 0 · Oct 19: 3 · Oct 20: 2 · Oct 21: 1 · Oct 22: 2 · Oct 23: 1 · Oct 24: 1 · Oct 25: 2 | INT, PROD | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (3)fail [github.com/Azure/ARO-HCP/test/e2e/cluster_version_backlevel.go:201]: failed to verify simple web app on back-level cluster version 4.20
Unexpected error:
<*fmt.wrapError | 0xd75a95e43e0>:
DNS for route host agnhost-e2e-sample-app-qx6cd.apps.aro.s4p6p0t9k4o2s8h.fx2j.j3671936.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-qx6cd.apps.aro.s4p6p0t9k4o2s8h.fx2j.j3671936.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-qx6cd.apps.aro.s4p6p0t9k4o2s8h.fx2j.j3671936.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded
{
msg: "DNS for route host agnhost-e2e-sample-app-qx6cd.apps.aro.s4p6p0t9k4o2s8h.fx2j.j3671936.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-qx6cd.apps.aro.s4p6p0t9k4o2s8h.fx2j.j3671936.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-qx6cd.apps.aro.s4p6p0t9k4o2s8h.fx2j.j3671936.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
err: <*fmt.wrapError | 0xd75a8fba020>{
msg: "DNS for agnhost-e2e-sample-app-qx6cd.apps.aro.s4p6p0t9k4o2s8h.fx2j.j3671936.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-qx6cd.apps.aro.s4p6p0t9k4o2s8h.fx2j.j3671936.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
err: <context.deadlineExceededError>{},
},
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_version_backlevel.go:201]: failed to verify simple web app on back-level cluster version 4.20
Unexpected error:
<*fmt.wrapError | 0xd75a95e43e0>:
DNS for route host agnhost-e2e-sample-app-qx6cd.apps.aro.s4p6p0t9k4o2s8h.fx2j.j3671936.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-qx6cd.apps.aro.s4p6p0t9k4o2s8h.fx2j.j3671936.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-qx6cd.apps.aro.s4p6p0t9k4o2s8h.fx2j.j3671936.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded
{
msg: "DNS for route host agnhost-e2e-sample-app-qx6cd.apps.aro.s4p6p0t9k4o2s8h.fx2j.j3671936.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-qx6cd.apps.aro.s4p6p0t9k4o2s8h.fx2j.j3671936.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-qx6cd.apps.aro.s4p6p0t9k4o2s8h.fx2j.j3671936.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
err: <*fmt.wrapError | 0xd75a8fba020>{
msg: "DNS for agnhost-e2e-sample-app-qx6cd.apps.aro.s4p6p0t9k4o2s8h.fx2j.j3671936.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-qx6cd.apps.aro.s4p6p0t9k4o2s8h.fx2j.j3671936.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
err: <context.deadlineExceededError>{},
},
}
occurredfail [github.com/Azure/ARO-HCP/test/e2e/cluster_version_backlevel.go:201]: failed to verify simple web app on back-level cluster version 4.20
Unexpected error:
<*fmt.wrapError | 0xc4f5ed30b40>:
DNS for route host agnhost-e2e-sample-app-jfjrs.apps.aro.a7b4y7y9a8k0c1j.bzjc.j0985984.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-jfjrs.apps.aro.a7b4y7y9a8k0c1j.bzjc.j0985984.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-jfjrs.apps.aro.a7b4y7y9a8k0c1j.bzjc.j0985984.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded
{
msg: "DNS for route host agnhost-e2e-sample-app-jfjrs.apps.aro.a7b4y7y9a8k0c1j.bzjc.j0985984.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-jfjrs.apps.aro.a7b4y7y9a8k0c1j.bzjc.j0985984.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-jfjrs.apps.aro.a7b4y7y9a8k0c1j.bzjc.j0985984.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
err: <*fmt.wrapError | 0xc4f5f2da4a0>{
msg: "DNS for agnhost-e2e-sample-app-jfjrs.apps.aro.a7b4y7y9a8k0c1j.bzjc.j0985984.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-jfjrs.apps.aro.a7b4y7y9a8k0c1j.bzjc.j0985984.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
err: <context.deadlineExceededError>{},
},
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_version_backlevel.go:201]: failed to verify simple web app on back-level cluster version 4.20
Unexpected error:
<*fmt.wrapError | 0xc4f5ed30b40>:
DNS for route host agnhost-e2e-sample-app-jfjrs.apps.aro.a7b4y7y9a8k0c1j.bzjc.j0985984.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-jfjrs.apps.aro.a7b4y7y9a8k0c1j.bzjc.j0985984.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-jfjrs.apps.aro.a7b4y7y9a8k0c1j.bzjc.j0985984.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded
{
msg: "DNS for route host agnhost-e2e-sample-app-jfjrs.apps.aro.a7b4y7y9a8k0c1j.bzjc.j0985984.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-jfjrs.apps.aro.a7b4y7y9a8k0c1j.bzjc.j0985984.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-jfjrs.apps.aro.a7b4y7y9a8k0c1j.bzjc.j0985984.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
err: <*fmt.wrapError | 0xc4f5f2da4a0>{
msg: "DNS for agnhost-e2e-sample-app-jfjrs.apps.aro.a7b4y7y9a8k0c1j.bzjc.j0985984.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-jfjrs.apps.aro.a7b4y7y9a8k0c1j.bzjc.j0985984.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
err: <context.deadlineExceededError>{},
},
}
occurredfail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:194]: failed to verify simple web app runs on cluster "cluster-candidate-4-21-zdl7h9"
Unexpected error:
<*fmt.wrapError | 0x7f0cb444140>:
DNS for route host agnhost-e2e-sample-app-lplgk.apps.aro.y6v4u1u9i5a4q3x.e0z9.j9726464.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-lplgk.apps.aro.y6v4u1u9i5a4q3x.e0z9.j9726464.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-lplgk.apps.aro.y6v4u1u9i5a4q3x.e0z9.j9726464.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded
{
msg: "DNS for route host agnhost-e2e-sample-app-lplgk.apps.aro.y6v4u1u9i5a4q3x.e0z9.j9726464.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-lplgk.apps.aro.y6v4u1u9i5a4q3x.e0z9.j9726464.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-lplgk.apps.aro.y6v4u1u9i5a4q3x.e0z9.j9726464.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
err: <*fmt.wrapError | 0x7f0cb818340>{
msg: "DNS for agnhost-e2e-sample-app-lplgk.apps.aro.y6v4u1u9i5a4q3x.e0z9.j9726464.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-lplgk.apps.aro.y6v4u1u9i5a4q3x.e0z9.j9726464.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
err: <context.deadlineExceededError>{},
},
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:194]: failed to verify simple web app runs on cluster "cluster-candidate-4-21-zdl7h9"
Unexpected error:
<*fmt.wrapError | 0x7f0cb444140>:
DNS for route host agnhost-e2e-sample-app-lplgk.apps.aro.y6v4u1u9i5a4q3x.e0z9.j9726464.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-lplgk.apps.aro.y6v4u1u9i5a4q3x.e0z9.j9726464.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-lplgk.apps.aro.y6v4u1u9i5a4q3x.e0z9.j9726464.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded
{
msg: "DNS for route host agnhost-e2e-sample-app-lplgk.apps.aro.y6v4u1u9i5a4q3x.e0z9.j9726464.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-lplgk.apps.aro.y6v4u1u9i5a4q3x.e0z9.j9726464.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-lplgk.apps.aro.y6v4u1u9i5a4q3x.e0z9.j9726464.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
err: <*fmt.wrapError | 0x7f0cb818340>{
msg: "DNS for agnhost-e2e-sample-app-lplgk.apps.aro.y6v4u1u9i5a4q3x.e0z9.j9726464.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-lplgk.apps.aro.y6v4u1u9i5a4q3x.e0z9.j9726464.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
err: <context.deadlineExceededError>{},
},
}
occurredContributing tests (7)
Affected runs (19) | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
alert [svc] KubePodNotReady firedfull failure pattern: alert [svc] KubePodNotReady fired Signal: FlakeSignal: Flake — present in 4 prior week(s); spread across 9 days | alert | 15 | 1.51%15 of 995 job runs affected | FlakeSignal: Flake — present in 4 prior week(s); spread across 9 days | Show trend detailsOct 12: 2 · Oct 13: 1 · Oct 14: 1 · Oct 15: 2 · Oct 16: 1 · Oct 17: 0 · Oct 18: 0 · Oct 19: 0 · Oct 20: 1 · Oct 21: 0 · Oct 22: 4 · Oct 23: 1 · Oct 24: 2 · Oct 25: 0 | none | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (3)alert fired 26 time(s) Firing 1: State: Fired (not resolved) Started: 2026-10-24T17:02:57Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j5468416-svc", component="kubernetes-infrastructure", namespace="aro-hcp-exporter", pod="aro-hcp-exporter-85fb9c778c-pbtkv", region="centralus", severity="warning" Description: Pod aro-hcp-exporter/aro-hcp-exporter-85fb9c778c-pbtkv has been in a non-ready state for longer than 5 minutes. Firing 2: State: Fired (not resolved) Started: 2026-10-24T17:02:57Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j5468416-svc", component="kubernetes-infrastructure", namespace="aro-hcp-exporter", pod="aro-hcp-exporter-85fb9c778c-82j8b", region="centralus", severity="warning" Description: Pod aro-hcp-exporter/aro-hcp-exporter-85fb9c778c-82j8b has been in a non-ready state for longer than 5 minutes. Firing 3: State: Fired (not resolved) Started: 2026-10-24T17:02:58Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j5468416-svc", component="kubernetes-infrastructure", namespace="aro-hcp-exporter", pod="aro-hcp-exporter-85fb9c778c-f27gs", region="centralus", severity="warning" Description: Pod aro-hcp-exporter/aro-hcp-exporter-85fb9c778c-f27gs has been in a non-ready state for longer than 5 minutes. Firing 4: State: Fired (not resolved) Started: 2026-10-24T17:02:58Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j5468416-svc", component="kubernetes-infrastructure", namespace="aro-hcp-exporter", pod="aro-hcp-exporter-85fb9c778c-lwgdc", region="centralus", severity="warning" Description: Pod aro-hcp-exporter/aro-hcp-exporter-85fb9c778c-lwgdc has been in a non-ready state for longer than 5 minutes. Firing 5: State: Fired (not resolved) Started: 2026-10-24T17:02:58Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j5468416-svc", component="kubernetes-infrastructure", namespace="aro-hcp-exporter", pod="aro-hcp-exporter-85fb9c778c-vs5np", region="centralus", severity="warning" Description: Pod aro-hcp-exporter/aro-hcp-exporter-85fb9c778c-vs5np has been in a non-ready state for longer than 5 minutes. Firing 6: State: Fired (not resolved) Started: 2026-10-24T17:02:58Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j5468416-svc", component="kubernetes-infrastructure", namespace="aro-hcp-exporter", pod="aro-hcp-exporter-85fb9c778c-87dfl", region="centralus", severity="warning" Description: Pod aro-hcp-exporter/aro-hcp-exporter-85fb9c778c-87dfl has been in a non-ready state for longer than 5 minutes. Firing 7: State: Fired (not resolved) Started: 2026-10-24T17:02:58Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j5468416-svc", component="kubernetes-infrastructure", namespace="aro-hcp-exporter", pod="aro-hcp-exporter-85fb9c778c-qpl82", region="centralus", severity="warning" Description: Pod aro-hcp-exporter/aro-hcp-exporter-85fb9c778c-qpl82 has been in a non-ready state for longer than 5 minutes. Firing 8: State: Fired (not resolved) Started: 2026-10-24T17:02:58Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j5468416-svc", component="kubernetes-infrastructure", namespace="aro-hcp-exporter", pod="aro-hcp-exporter-85fb9c778c-5q9wz", region="centralus", severity="warning" Description: Pod aro-hcp-exporter/aro-hcp-exporter-85fb9c778c-5q9wz has been in a non-ready state for longer than 5 minutes. Firing 9: State: Fired (not resolved) Started: 2026-10-24T17:02:58Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j5468416-svc", component="kubernetes-infrastructure", namespace="aro-hcp-exporter", pod="aro-hcp-exporter-85fb9c778c-q7t7r", region="centralus", severity="warning" Description: Pod aro-hcp-exporter/aro-hcp-exporter-85fb9c778c-q7t7r has been in a non-ready state for longer than 5 minutes. Firing 10: State: Fired (not resolved) Started: 2026-10-24T17:02:58Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j5468416-svc", component="kubernetes-infrastructure", namespace="aro-hcp-exporter", pod="aro-hcp-exporter-85fb9c778c-7llr5", region="centralus", severity="warning" Description: Pod aro-hcp-exporter/aro-hcp-exporter-85fb9c778c-7llr5 has been in a non-ready state for longer than 5 minutes. Firing 11: State: Fired (not resolved) Started: 2026-10-24T17:02:58Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j5468416-svc", component="kubernetes-infrastructure", namespace="aro-hcp-exporter", pod="aro-hcp-exporter-85fb9c778c-lxj96", region="centralus", severity="warning" Description: Pod aro-hcp-exporter/aro-hcp-exporter-85fb9c778c-lxj96 has been in a non-ready state for longer than 5 minutes. Firing 12: State: Fired (not resolved) Started: 2026-10-24T17:02:58Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j5468416-svc", component="kubernetes-infrastructure", namespace="aro-hcp-exporter", pod="aro-hcp-exporter-85fb9c778c-mcdmv", region="centralus", severity="warning" Description: Pod aro-hcp-exporter/aro-hcp-exporter-85fb9c778c-mcdmv has been in a non-ready state for longer than 5 minutes. Firing 13: State: Fired (not resolved) Started: 2026-10-24T17:02:58Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j5468416-svc", component="kubernetes-infrastructure", namespace="aro-hcp-exporter", pod="aro-hcp-exporter-85fb9c778c-swlsf", region="centralus", severity="warning" Description: Pod aro-hcp-exporter/aro-hcp-exporter-85fb9c778c-swlsf has been in a non-ready state for longer than 5 minutes. Firing 14: State: Fired (not resolved) Started: 2026-10-24T17:02:58Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j5468416-svc", component="kubernetes-infrastructure", namespace="aro-hcp-exporter", pod="aro-hcp-exporter-85fb9c778c-tkgxq", region="centralus", severity="warning" Description: Pod aro-hcp-exporter/aro-hcp-exporter-85fb9c778c-tkgxq has been in a non-ready state for longer than 5 minutes. Firing 15: State: Fired (not resolved) Started: 2026-10-24T17:02:58Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j5468416-svc", component="kubernetes-infrastructure", namespace="aro-hcp-exporter", pod="aro-hcp-exporter-85fb9c778c-92mtz", region="centralus", severity="warning" Description: Pod aro-hcp-exporter/aro-hcp-exporter-85fb9c778c-92mtz has been in a non-ready state for longer than 5 minutes. Firing 16: State: Fired (not resolved) Started: 2026-10-24T17:02:58Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j5468416-svc", component="kubernetes-infrastructure", namespace="aro-hcp-exporter", pod="aro-hcp-exporter-85fb9c778c-7vvmn", region="centralus", severity="warning" Description: Pod aro-hcp-exporter/aro-hcp-exporter-85fb9c778c-7vvmn has been in a non-ready state for longer than 5 minutes. Firing 17: State: Fired (not resolved) Started: 2026-10-24T17:02:58Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j5468416-svc", component="kubernetes-infrastructure", namespace="aro-hcp-exporter", pod="aro-hcp-exporter-85fb9c778c-tb8wn", region="centralus", severity="warning" Description: Pod aro-hcp-exporter/aro-hcp-exporter-85fb9c778c-tb8wn has been in a non-ready state for longer than 5 minutes. Firing 18: State: Fired (not resolved) Started: 2026-10-24T17:02:58Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j5468416-svc", component="kubernetes-infrastructure", namespace="aro-hcp-exporter", pod="aro-hcp-exporter-85fb9c778c-cv95p", region="centralus", severity="warning" Description: Pod aro-hcp-exporter/aro-hcp-exporter-85fb9c778c-cv95p has been in a non-ready state for longer than 5 minutes. Firing 19: State: Fired (not resolved) Started: 2026-10-24T17:02:58Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j5468416-svc", component="kubernetes-infrastructure", namespace="aro-hcp-exporter", pod="aro-hcp-exporter-85fb9c778c-qpvlz", region="centralus", severity="warning" Description: Pod aro-hcp-exporter/aro-hcp-exporter-85fb9c778c-qpvlz has been in a non-ready state for longer than 5 minutes. Firing 20: State: Fired (not resolved) Started: 2026-10-24T17:02:58Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j5468416-svc", component="kubernetes-infrastructure", namespace="aro-hcp-exporter", pod="aro-hcp-exporter-85fb9c778c-zfl7b", region="centralus", severity="warning" Description: Pod aro-hcp-exporter/aro-hcp-exporter-85fb9c778c-zfl7b has been in a non-ready state for longer than 5 minutes. Firing 21: State: Fired (not resolved) Started: 2026-10-24T17:02:58Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j5468416-svc", component="kubernetes-infrastructure", namespace="aro-hcp-exporter", pod="aro-hcp-exporter-85fb9c778c-4wdk9", region="centralus", severity="warning" Description: Pod aro-hcp-exporter/aro-hcp-exporter-85fb9c778c-4wdk9 has been in a non-ready state for longer than 5 minutes. Firing 22: State: Fired (not resolved) Started: 2026-10-24T17:02:58Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j5468416-svc", component="kubernetes-infrastructure", namespace="aro-hcp-exporter", pod="aro-hcp-exporter-85fb9c778c-9hkcm", region="centralus", severity="warning" Description: Pod aro-hcp-exporter/aro-hcp-exporter-85fb9c778c-9hkcm has been in a non-ready state for longer than 5 minutes. Firing 23: State: Fired (not resolved) Started: 2026-10-24T17:02:59Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j5468416-svc", component="kubernetes-infrastructure", namespace="aro-hcp-exporter", pod="aro-hcp-exporter-85fb9c778c-699hx", region="centralus", severity="warning" Description: Pod aro-hcp-exporter/aro-hcp-exporter-85fb9c778c-699hx has been in a non-ready state for longer than 5 minutes. Firing 24: State: Fired (not resolved) Started: 2026-10-24T17:02:59Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j5468416-svc", component="kubernetes-infrastructure", namespace="aro-hcp-exporter", pod="aro-hcp-exporter-85fb9c778c-hdfbt", region="centralus", severity="warning" Description: Pod aro-hcp-exporter/aro-hcp-exporter-85fb9c778c-hdfbt has been in a non-ready state for longer than 5 minutes. Firing 25: State: Fired (not resolved) Started: 2026-10-24T17:02:59Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j5468416-svc", component="kubernetes-infrastructure", namespace="aro-hcp-exporter", pod="aro-hcp-exporter-85fb9c778c-cxrkz", region="centralus", severity="warning" Description: Pod aro-hcp-exporter/aro-hcp-exporter-85fb9c778c-cxrkz has been in a non-ready state for longer than 5 minutes. Firing 26: State: Fired (not resolved) Started: 2026-10-24T17:03:59Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j5468416-svc", component="kubernetes-infrastructure", namespace="aro-hcp-exporter", pod="aro-hcp-exporter-85fb9c778c-4hnj4", region="centralus", severity="warning" Description: Pod aro-hcp-exporter/aro-hcp-exporter-85fb9c778c-4hnj4 has been in a non-ready state for longer than 5 minutes. alert fired 1 time(s) Firing 1: State: Fired (not resolved) Started: 2026-10-24T09:35:35Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j4869376-mgmt-1", component="kubernetes-infrastructure", namespace="open-cluster-management-agent-addon", pod="klusterlet-addon-workmgr-58f4f6fb5b-wj69n", region="centralus", severity="warning" Description: Pod open-cluster-management-agent-addon/klusterlet-addon-workmgr-58f4f6fb5b-wj69n has been in a non-ready state for longer than 5 minutes. alert fired 62 time(s) Firing 1: State: Fired (not resolved) Started: 2026-10-23T14:33:49Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j5183488-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-67995d8d8b-q7xh2", region="canadacentral", severity="warning" Description: Pod monitoring/kube-events-67995d8d8b-q7xh2 has been in a non-ready state for longer than 5 minutes. Firing 2: State: Fired (not resolved) Started: 2026-10-23T14:33:49Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j5183488-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-67995d8d8b-ndlbl", region="canadacentral", severity="warning" Description: Pod monitoring/kube-events-67995d8d8b-ndlbl has been in a non-ready state for longer than 5 minutes. Firing 3: State: Fired (not resolved) Started: 2026-10-23T14:33:49Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j5183488-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-67995d8d8b-7xjfm", region="canadacentral", severity="warning" Description: Pod monitoring/kube-events-67995d8d8b-7xjfm has been in a non-ready state for longer than 5 minutes. Firing 4: State: Fired (not resolved) Started: 2026-10-23T14:33:49Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j5183488-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-67995d8d8b-5dvf5", region="canadacentral", severity="warning" Description: Pod monitoring/kube-events-67995d8d8b-5dvf5 has been in a non-ready state for longer than 5 minutes. Firing 5: State: Fired (not resolved) Started: 2026-10-23T14:33:49Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j5183488-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-67995d8d8b-mb68l", region="canadacentral", severity="warning" Description: Pod monitoring/kube-events-67995d8d8b-mb68l has been in a non-ready state for longer than 5 minutes. Firing 6: State: Fired (not resolved) Started: 2026-10-23T14:33:49Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j5183488-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-67995d8d8b-ssk4m", region="canadacentral", severity="warning" Description: Pod monitoring/kube-events-67995d8d8b-ssk4m has been in a non-ready state for longer than 5 minutes. Firing 7: State: Fired (not resolved) Started: 2026-10-23T14:33:49Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j5183488-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-67995d8d8b-9ljfg", region="canadacentral", severity="warning" Description: Pod monitoring/kube-events-67995d8d8b-9ljfg has been in a non-ready state for longer than 5 minutes. Firing 8: State: Fired (not resolved) Started: 2026-10-23T14:33:49Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j5183488-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-67995d8d8b-hlrms", region="canadacentral", severity="warning" Description: Pod monitoring/kube-events-67995d8d8b-hlrms has been in a non-ready state for longer than 5 minutes. Firing 9: State: Fired (not resolved) Started: 2026-10-23T14:33:49Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j5183488-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-67995d8d8b-7zxmv", region="canadacentral", severity="warning" Description: Pod monitoring/kube-events-67995d8d8b-7zxmv has been in a non-ready state for longer than 5 minutes. Firing 10: State: Fired (not resolved) Started: 2026-10-23T14:33:49Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j5183488-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-67995d8d8b-995kh", region="canadacentral", severity="warning" Description: Pod monitoring/kube-events-67995d8d8b-995kh has been in a non-ready state for longer than 5 minutes. Firing 11: State: Fired (not resolved) Started: 2026-10-23T14:33:49Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j5183488-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-67995d8d8b-jj844", region="canadacentral", severity="warning" Description: Pod monitoring/kube-events-67995d8d8b-jj844 has been in a non-ready state for longer than 5 minutes. Firing 12: State: Fired (not resolved) Started: 2026-10-23T14:33:49Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j5183488-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-67995d8d8b-hk2pc", region="canadacentral", severity="warning" Description: Pod monitoring/kube-events-67995d8d8b-hk2pc has been in a non-ready state for longer than 5 minutes. Firing 13: State: Fired (not resolved) Started: 2026-10-23T14:33:49Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j5183488-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-67995d8d8b-v22fj", region="canadacentral", severity="warning" Description: Pod monitoring/kube-events-67995d8d8b-v22fj has been in a non-ready state for longer than 5 minutes. Firing 14: State: Fired (not resolved) Started: 2026-10-23T14:33:49Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j5183488-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-67995d8d8b-x55nd", region="canadacentral", severity="warning" Description: Pod monitoring/kube-events-67995d8d8b-x55nd has been in a non-ready state for longer than 5 minutes. Firing 15: State: Fired (not resolved) Started: 2026-10-23T14:33:49Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j5183488-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-67995d8d8b-dbxr7", region="canadacentral", severity="warning" Description: Pod monitoring/kube-events-67995d8d8b-dbxr7 has been in a non-ready state for longer than 5 minutes. Firing 16: State: Fired (not resolved) Started: 2026-10-23T14:33:49Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j5183488-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-67995d8d8b-gmslj", region="canadacentral", severity="warning" Description: Pod monitoring/kube-events-67995d8d8b-gmslj has been in a non-ready state for longer than 5 minutes. Firing 17: State: Fired (not resolved) Started: 2026-10-23T14:33:49Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j5183488-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-67995d8d8b-tfmkl", region="canadacentral", severity="warning" Description: Pod monitoring/kube-events-67995d8d8b-tfmkl has been in a non-ready state for longer than 5 minutes. Firing 18: State: Fired (not resolved) Started: 2026-10-23T14:33:49Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j5183488-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-67995d8d8b-wcfrc", region="canadacentral", severity="warning" Description: Pod monitoring/kube-events-67995d8d8b-wcfrc has been in a non-ready state for longer than 5 minutes. Firing 19: State: Fired (not resolved) Started: 2026-10-23T14:33:49Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j5183488-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-67995d8d8b-vz2cv", region="canadacentral", severity="warning" Description: Pod monitoring/kube-events-67995d8d8b-vz2cv has been in a non-ready state for longer than 5 minutes. Firing 20: State: Fired (not resolved) Started: 2026-10-23T14:33:49Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j5183488-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-67995d8d8b-9mcmc", region="canadacentral", severity="warning" Description: Pod monitoring/kube-events-67995d8d8b-9mcmc has been in a non-ready state for longer than 5 minutes. Firing 21: State: Fired (not resolved) Started: 2026-10-23T14:33:49Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j5183488-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-67995d8d8b-d6nhd", region="canadacentral", severity="warning" Description: Pod monitoring/kube-events-67995d8d8b-d6nhd has been in a non-ready state for longer than 5 minutes. Firing 22: State: Fired (not resolved) Started: 2026-10-23T14:33:49Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j5183488-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-67995d8d8b-9txxf", region="canadacentral", severity="warning" Description: Pod monitoring/kube-events-67995d8d8b-9txxf has been in a non-ready state for longer than 5 minutes. Firing 23: State: Fired (not resolved) Started: 2026-10-23T14:33:49Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j5183488-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-67995d8d8b-6b8wc", region="canadacentral", severity="warning" Description: Pod monitoring/kube-events-67995d8d8b-6b8wc has been in a non-ready state for longer than 5 minutes. Firing 24: State: Fired (not resolved) Started: 2026-10-23T14:33:49Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j5183488-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-67995d8d8b-nrgxf", region="canadacentral", severity="warning" Description: Pod monitoring/kube-events-67995d8d8b-nrgxf has been in a non-ready state for longer than 5 minutes. Firing 25: State: Fired (not resolved) Started: 2026-10-23T14:33:49Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j5183488-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-67995d8d8b-sgtgx", region="canadacentral", severity="warning" Description: Pod monitoring/kube-events-67995d8d8b-sgtgx has been in a non-ready state for longer than 5 minutes. Firing 26: State: Fired (not resolved) Started: 2026-10-23T14:33:49Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j5183488-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-67995d8d8b-rqj2v", region="canadacentral", severity="warning" Description: Pod monitoring/kube-events-67995d8d8b-rqj2v has been in a non-ready state for longer than 5 minutes. Firing 27: State: Fired (not resolved) Started: 2026-10-23T14:33:49Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j5183488-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-67995d8d8b-8vhqg", region="canadacentral", severity="warning" Description: Pod monitoring/kube-events-67995d8d8b-8vhqg has been in a non-ready state for longer than 5 minutes. Firing 28: State: Fired (not resolved) Started: 2026-10-23T14:33:49Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j5183488-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-67995d8d8b-rsscm", region="canadacentral", severity="warning" Description: Pod monitoring/kube-events-67995d8d8b-rsscm has been in a non-ready state for longer than 5 minutes. Firing 29: State: Fired (not resolved) Started: 2026-10-23T14:33:49Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j5183488-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-67995d8d8b-t4fln", region="canadacentral", severity="warning" Description: Pod monitoring/kube-events-67995d8d8b-t4fln has been in a non-ready state for longer than 5 minutes. Firing 30: State: Fired (not resolved) Started: 2026-10-23T14:33:49Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j5183488-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-67995d8d8b-tjxjb", region="canadacentral", severity="warning" Description: Pod monitoring/kube-events-67995d8d8b-tjxjb has been in a non-ready state for longer than 5 minutes. Firing 31: State: Fired (not resolved) Started: 2026-10-23T14:33:49Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j5183488-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-67995d8d8b-m5d5c", region="canadacentral", severity="warning" Description: Pod monitoring/kube-events-67995d8d8b-m5d5c has been in a non-ready state for longer than 5 minutes. Firing 32: State: Fired (not resolved) Started: 2026-10-23T14:33:49Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j5183488-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-67995d8d8b-bhfw8", region="canadacentral", severity="warning" Description: Pod monitoring/kube-events-67995d8d8b-bhfw8 has been in a non-ready state for longer than 5 minutes. Firing 33: State: Fired (not resolved) Started: 2026-10-23T14:33:49Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j5183488-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-67995d8d8b-8kc57", region="canadacentral", severity="warning" Description: Pod monitoring/kube-events-67995d8d8b-8kc57 has been in a non-ready state for longer than 5 minutes. Firing 34: State: Fired (not resolved) Started: 2026-10-23T14:33:49Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j5183488-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-67995d8d8b-nmlxd", region="canadacentral", severity="warning" Description: Pod monitoring/kube-events-67995d8d8b-nmlxd has been in a non-ready state for longer than 5 minutes. Firing 35: State: Fired (not resolved) Started: 2026-10-23T14:33:49Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j5183488-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-67995d8d8b-7nz9z", region="canadacentral", severity="warning" Description: Pod monitoring/kube-events-67995d8d8b-7nz9z has been in a non-ready state for longer than 5 minutes. Firing 36: State: Fired (not resolved) Started: 2026-10-23T14:33:49Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j5183488-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-67995d8d8b-rr5tb", region="canadacentral", severity="warning" Description: Pod monitoring/kube-events-67995d8d8b-rr5tb has been in a non-ready state for longer than 5 minutes. Firing 37: State: Fired (not resolved) Started: 2026-10-23T14:33:49Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j5183488-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-67995d8d8b-zgzmg", region="canadacentral", severity="warning" Description: Pod monitoring/kube-events-67995d8d8b-zgzmg has been in a non-ready state for longer than 5 minutes. Firing 38: State: Fired (not resolved) Started: 2026-10-23T14:33:49Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j5183488-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-67995d8d8b-v2xpg", region="canadacentral", severity="warning" Description: Pod monitoring/kube-events-67995d8d8b-v2xpg has been in a non-ready state for longer than 5 minutes. Firing 39: State: Fired (not resolved) Started: 2026-10-23T14:33:49Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j5183488-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-67995d8d8b-v96bl", region="canadacentral", severity="warning" Description: Pod monitoring/kube-events-67995d8d8b-v96bl has been in a non-ready state for longer than 5 minutes. Firing 40: State: Fired (not resolved) Started: 2026-10-23T14:33:49Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j5183488-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-67995d8d8b-j9jl5", region="canadacentral", severity="warning" Description: Pod monitoring/kube-events-67995d8d8b-j9jl5 has been in a non-ready state for longer than 5 minutes. Firing 41: State: Fired (not resolved) Started: 2026-10-23T14:33:49Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j5183488-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-67995d8d8b-b87xs", region="canadacentral", severity="warning" Description: Pod monitoring/kube-events-67995d8d8b-b87xs has been in a non-ready state for longer than 5 minutes. Firing 42: State: Fired (not resolved) Started: 2026-10-23T14:33:49Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j5183488-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-67995d8d8b-pphfm", region="canadacentral", severity="warning" Description: Pod monitoring/kube-events-67995d8d8b-pphfm has been in a non-ready state for longer than 5 minutes. Firing 43: State: Fired (not resolved) Started: 2026-10-23T14:33:49Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j5183488-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-67995d8d8b-hgkjq", region="canadacentral", severity="warning" Description: Pod monitoring/kube-events-67995d8d8b-hgkjq has been in a non-ready state for longer than 5 minutes. Firing 44: State: Fired (not resolved) Started: 2026-10-23T14:33:49Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j5183488-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-67995d8d8b-xd72x", region="canadacentral", severity="warning" Description: Pod monitoring/kube-events-67995d8d8b-xd72x has been in a non-ready state for longer than 5 minutes. Firing 45: State: Fired (not resolved) Started: 2026-10-23T14:33:49Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j5183488-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-67995d8d8b-8jcjf", region="canadacentral", severity="warning" Description: Pod monitoring/kube-events-67995d8d8b-8jcjf has been in a non-ready state for longer than 5 minutes. Firing 46: State: Fired (not resolved) Started: 2026-10-23T14:33:49Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j5183488-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-67995d8d8b-rvdns", region="canadacentral", severity="warning" Description: Pod monitoring/kube-events-67995d8d8b-rvdns has been in a non-ready state for longer than 5 minutes. Firing 47: State: Fired (not resolved) Started: 2026-10-23T14:33:49Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j5183488-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-67995d8d8b-x4kzj", region="canadacentral", severity="warning" Description: Pod monitoring/kube-events-67995d8d8b-x4kzj has been in a non-ready state for longer than 5 minutes. Firing 48: State: Fired (not resolved) Started: 2026-10-23T14:33:49Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j5183488-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-67995d8d8b-b8qd2", region="canadacentral", severity="warning" Description: Pod monitoring/kube-events-67995d8d8b-b8qd2 has been in a non-ready state for longer than 5 minutes. Firing 49: State: Fired (not resolved) Started: 2026-10-23T14:33:49Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j5183488-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-67995d8d8b-qt827", region="canadacentral", severity="warning" Description: Pod monitoring/kube-events-67995d8d8b-qt827 has been in a non-ready state for longer than 5 minutes. Firing 50: State: Fired (not resolved) Started: 2026-10-23T14:33:49Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j5183488-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-67995d8d8b-q8mvg", region="canadacentral", severity="warning" Description: Pod monitoring/kube-events-67995d8d8b-q8mvg has been in a non-ready state for longer than 5 minutes. Firing 51: State: Fired (not resolved) Started: 2026-10-23T14:33:49Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j5183488-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-67995d8d8b-kfjrt", region="canadacentral", severity="warning" Description: Pod monitoring/kube-events-67995d8d8b-kfjrt has been in a non-ready state for longer than 5 minutes. Firing 52: State: Fired (not resolved) Started: 2026-10-23T14:33:49Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j5183488-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-67995d8d8b-hwkjz", region="canadacentral", severity="warning" Description: Pod monitoring/kube-events-67995d8d8b-hwkjz has been in a non-ready state for longer than 5 minutes. Firing 53: State: Fired (not resolved) Started: 2026-10-23T14:33:49Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j5183488-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-67995d8d8b-9td26", region="canadacentral", severity="warning" Description: Pod monitoring/kube-events-67995d8d8b-9td26 has been in a non-ready state for longer than 5 minutes. Firing 54: State: Fired (not resolved) Started: 2026-10-23T14:33:49Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j5183488-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-67995d8d8b-cr96z", region="canadacentral", severity="warning" Description: Pod monitoring/kube-events-67995d8d8b-cr96z has been in a non-ready state for longer than 5 minutes. Firing 55: State: Fired (not resolved) Started: 2026-10-23T14:33:49Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j5183488-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-67995d8d8b-9dqr2", region="canadacentral", severity="warning" Description: Pod monitoring/kube-events-67995d8d8b-9dqr2 has been in a non-ready state for longer than 5 minutes. Firing 56: State: Fired (not resolved) Started: 2026-10-23T14:33:50Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j5183488-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-67995d8d8b-d56c6", region="canadacentral", severity="warning" Description: Pod monitoring/kube-events-67995d8d8b-d56c6 has been in a non-ready state for longer than 5 minutes. Firing 57: State: Fired (not resolved) Started: 2026-10-23T14:33:50Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j5183488-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-67995d8d8b-s89qq", region="canadacentral", severity="warning" Description: Pod monitoring/kube-events-67995d8d8b-s89qq has been in a non-ready state for longer than 5 minutes. Firing 58: State: Fired (not resolved) Started: 2026-10-23T14:33:50Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j5183488-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-67995d8d8b-k2xxx", region="canadacentral", severity="warning" Description: Pod monitoring/kube-events-67995d8d8b-k2xxx has been in a non-ready state for longer than 5 minutes. Firing 59: State: Fired (not resolved) Started: 2026-10-23T14:33:50Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j5183488-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-67995d8d8b-jb7dj", region="canadacentral", severity="warning" Description: Pod monitoring/kube-events-67995d8d8b-jb7dj has been in a non-ready state for longer than 5 minutes. Firing 60: State: Fired (not resolved) Started: 2026-10-23T14:33:50Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j5183488-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-67995d8d8b-l6zq2", region="canadacentral", severity="warning" Description: Pod monitoring/kube-events-67995d8d8b-l6zq2 has been in a non-ready state for longer than 5 minutes. Firing 61: State: Fired (not resolved) Started: 2026-10-23T14:33:50Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j5183488-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-67995d8d8b-hzddl", region="canadacentral", severity="warning" Description: Pod monitoring/kube-events-67995d8d8b-hzddl has been in a non-ready state for longer than 5 minutes. Firing 62: State: Fired (not resolved) Started: 2026-10-23T14:33:50Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j5183488-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-67995d8d8b-k2wfc", region="canadacentral", severity="warning" Description: Pod monitoring/kube-events-67995d8d8b-k2wfc has been in a non-ready state for longer than 5 minutes. Contributing tests (1)
Affected runs (15) | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
alert [svc] KubeDeploymentReplicasMismatch firedfull failure pattern: alert [svc] KubeDeploymentReplicasMismatch fired Signal: FlakeSignal: Flake — present in 4 prior week(s); spread across 7 days | alert | 12 | 1.21%12 of 995 job runs affected | FlakeSignal: Flake — present in 4 prior week(s); spread across 7 days | Show trend detailsOct 12: 0 · Oct 13: 1 · Oct 14: 0 · Oct 15: 1 · Oct 16: 0 · Oct 17: 0 · Oct 18: 0 · Oct 19: 0 · Oct 20: 0 · Oct 21: 1 · Oct 22: 6 · Oct 23: 1 · Oct 24: 1 · Oct 25: 1 | none | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (3)alert fired 3 time(s) Firing 1: State: Resolved Started: 2026-10-25T19:03:55Z Ended: 2026-10-25T19:10:56Z Severity: Sev3 Labels: alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j3671936-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="config-policy-controller", endpoint="http", environment="ci01", instance="10.128.64.162:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j3671936/providers/microsoft.monitor/accounts/services-j3671936", microsoft.metrics_include_label="service", namespace="klusterlet-2tlvp310lq0lqg3o9bn0eeqgoijvib5c", pod="arohcp-monitor-kube-state-metrics-85d48f46ff-7vdl7", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning" Description: Deployment klusterlet-2tlvp310lq0lqg3o9bn0eeqgoijvib5c/config-policy-controller has not matched the expected number of replicas for longer than 30 minutes. Firing 2: State: Resolved Started: 2026-10-25T19:03:55Z Ended: 2026-10-25T19:09:56Z Severity: Sev3 Labels: alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j3671936-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="klusterlet-addon-workmgr", endpoint="http", environment="ci01", instance="10.128.64.162:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j3671936/providers/microsoft.monitor/accounts/services-j3671936", microsoft.metrics_include_label="service", namespace="klusterlet-2tlvp310lq0lqg3o9bn0eeqgoijvib5c", pod="arohcp-monitor-kube-state-metrics-85d48f46ff-7vdl7", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning" Description: Deployment klusterlet-2tlvp310lq0lqg3o9bn0eeqgoijvib5c/klusterlet-addon-workmgr has not matched the expected number of replicas for longer than 30 minutes. Firing 3: State: Resolved Started: 2026-10-25T19:03:55Z Ended: 2026-10-25T19:10:56Z Severity: Sev3 Labels: alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j3671936-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="governance-policy-framework", endpoint="http", environment="ci01", instance="10.128.64.162:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j3671936/providers/microsoft.monitor/accounts/services-j3671936", microsoft.metrics_include_label="service", namespace="klusterlet-2tlvp310lq0lqg3o9bn0eeqgoijvib5c", pod="arohcp-monitor-kube-state-metrics-85d48f46ff-7vdl7", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning" Description: Deployment klusterlet-2tlvp310lq0lqg3o9bn0eeqgoijvib5c/governance-policy-framework has not matched the expected number of replicas for longer than 30 minutes. alert fired 6 time(s) Firing 1: State: Fired (not resolved) Started: 2026-10-24T10:00:39Z Severity: Sev3 Labels: alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j4869376-mgmt-1", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="klusterlet-addon-workmgr", endpoint="http", environment="ci01", instance="10.128.65.11:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j4869376/providers/microsoft.monitor/accounts/services-j4869376", microsoft.metrics_include_label="service", namespace="open-cluster-management-agent-addon", pod="arohcp-monitor-kube-state-metrics-85d48f46ff-kc252", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="centralus", service="arohcp-monitor-kube-state-metrics", severity="warning" Description: Deployment open-cluster-management-agent-addon/klusterlet-addon-workmgr has not matched the expected number of replicas for longer than 30 minutes. Firing 2: State: Fired (not resolved) Started: 2026-10-24T10:00:39Z Severity: Sev3 Labels: alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j4869376-mgmt-1", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="klusterlet-addon-workmgr", endpoint="http", environment="ci01", instance="10.128.65.11:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j4869376/providers/microsoft.monitor/accounts/services-j4869376", microsoft.metrics_include_label="service", namespace="open-cluster-management-agent-addon", pod="arohcp-monitor-kube-state-metrics-85d48f46ff-kc252", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="centralus", service="arohcp-monitor-kube-state-metrics", severity="warning" Description: Deployment open-cluster-management-agent-addon/klusterlet-addon-workmgr has not matched the expected number of replicas for longer than 30 minutes. Firing 3: State: Resolved Started: 2026-10-24T10:09:36Z Ended: 2026-10-24T10:18:36Z Severity: Sev3 Labels: alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j4869376-mgmt-1", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="klusterlet-addon-workmgr", endpoint="http", environment="ci01", instance="10.128.65.11:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j4869376/providers/microsoft.monitor/accounts/services-j4869376", microsoft.metrics_include_label="service", namespace="klusterlet-2tl2qrt5450mmm2tpbrvv2e0lvko6ltl", pod="arohcp-monitor-kube-state-metrics-85d48f46ff-kc252", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="centralus", service="arohcp-monitor-kube-state-metrics", severity="warning" Description: Deployment klusterlet-2tl2qrt5450mmm2tpbrvv2e0lvko6ltl/klusterlet-addon-workmgr has not matched the expected number of replicas for longer than 30 minutes. Firing 4: State: Resolved Started: 2026-10-24T10:09:36Z Ended: 2026-10-24T10:18:36Z Severity: Sev3 Labels: alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j4869376-mgmt-1", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="klusterlet-addon-workmgr", endpoint="http", environment="ci01", instance="10.128.65.11:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j4869376/providers/microsoft.monitor/accounts/services-j4869376", microsoft.metrics_include_label="service", namespace="klusterlet-2tl2qrt5450mmm2tpbrvv2e0lvko6ltl", pod="arohcp-monitor-kube-state-metrics-85d48f46ff-kc252", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="centralus", service="arohcp-monitor-kube-state-metrics", severity="warning" Description: Deployment klusterlet-2tl2qrt5450mmm2tpbrvv2e0lvko6ltl/klusterlet-addon-workmgr has not matched the expected number of replicas for longer than 30 minutes. Firing 5: State: Resolved Started: 2026-10-24T10:10:38Z Ended: 2026-10-24T10:21:36Z Severity: Sev3 Labels: alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j4869376-mgmt-1", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="klusterlet-addon-workmgr", endpoint="http", environment="ci01", instance="10.128.65.11:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j4869376/providers/microsoft.monitor/accounts/services-j4869376", microsoft.metrics_include_label="service", namespace="klusterlet-2tl2qq6lifhtq61o83ao3dcajsdckdi5", pod="arohcp-monitor-kube-state-metrics-85d48f46ff-kc252", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="centralus", service="arohcp-monitor-kube-state-metrics", severity="warning" Description: Deployment klusterlet-2tl2qq6lifhtq61o83ao3dcajsdckdi5/klusterlet-addon-workmgr has not matched the expected number of replicas for longer than 30 minutes. Firing 6: State: Resolved Started: 2026-10-24T10:10:38Z Ended: 2026-10-24T10:21:36Z Severity: Sev3 Labels: alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j4869376-mgmt-1", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="klusterlet-addon-workmgr", endpoint="http", environment="ci01", instance="10.128.65.11:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j4869376/providers/microsoft.monitor/accounts/services-j4869376", microsoft.metrics_include_label="service", namespace="klusterlet-2tl2qq6lifhtq61o83ao3dcajsdckdi5", pod="arohcp-monitor-kube-state-metrics-85d48f46ff-kc252", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="centralus", service="arohcp-monitor-kube-state-metrics", severity="warning" Description: Deployment klusterlet-2tl2qq6lifhtq61o83ao3dcajsdckdi5/klusterlet-addon-workmgr has not matched the expected number of replicas for longer than 30 minutes. alert fired 2 time(s) Firing 1: State: Fired (not resolved) Started: 2026-10-23T12:46:11Z Severity: Sev3 Labels: alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j0830592-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="klusterlet-addon-workmgr", endpoint="http", environment="ci01", instance="10.128.64.252:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0830592/providers/microsoft.monitor/accounts/services-j0830592", microsoft.metrics_include_label="service", namespace="open-cluster-management-agent-addon", pod="arohcp-monitor-kube-state-metrics-85d48f46ff-g9k4t", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning" Description: Deployment open-cluster-management-agent-addon/klusterlet-addon-workmgr has not matched the expected number of replicas for longer than 30 minutes. Firing 2: State: Fired (not resolved) Started: 2026-10-23T12:46:11Z Severity: Sev3 Labels: alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j0830592-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="klusterlet-addon-workmgr", endpoint="http", environment="ci01", instance="10.128.64.252:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0830592/providers/microsoft.monitor/accounts/services-j0830592", microsoft.metrics_include_label="service", namespace="open-cluster-management-agent-addon", pod="arohcp-monitor-kube-state-metrics-85d48f46ff-g9k4t", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning" Description: Deployment open-cluster-management-agent-addon/klusterlet-addon-workmgr has not matched the expected number of replicas for longer than 30 minutes. Contributing tests (1)
Affected runs (12) | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
alert [svc] MaestroAgentMetricsProxyDown firedfull failure pattern: alert [svc] MaestroAgentMetricsProxyDown fired Signal: FlakeSignal: Flake — present in 3 prior week(s); spread across 7 days | alert | 11 | 1.11%11 of 995 job runs affected | FlakeSignal: Flake — present in 3 prior week(s); spread across 7 days | Show trend detailsOct 12: 0 · Oct 13: 1 · Oct 14: 2 · Oct 15: 3 · Oct 16: 2 · Oct 17: 0 · Oct 18: 0 · Oct 19: 0 · Oct 20: 0 · Oct 21: 1 · Oct 22: 1 · Oct 23: 1 · Oct 24: 0 · Oct 25: 0 | none | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (3)alert fired 1 time(s) Firing 1: State: Resolved Started: 2026-10-23T02:40:08Z Ended: 2026-10-23T02:49:10Z Severity: Sev3 Labels: alertname="MaestroAgentMetricsProxyDown", cluster="ci01-j3977984-mgmt-2", component="monitoring-infrastructure", region="canadacentral", severity="3" Description: Prometheus has not scraped maestro-agent /metrics through the metrics-proxy sidecar (namespace=maestro, endpoint=metrics) for 10 minutes, while a kube-state-metrics scrape series still exists for this management cluster. Maestro agent metrics and any alerts that depend on them may be missing. alert fired 2 time(s) Firing 1: State: Fired (not resolved) Started: 2026-10-22T21:15:12Z Severity: Sev3 Labels: alertname="MaestroAgentMetricsProxyDown", cluster="ci01-j3800704-mgmt-2", component="monitoring-infrastructure", region="canadacentral", severity="3" Description: Prometheus has not scraped maestro-agent /metrics through the metrics-proxy sidecar (namespace=maestro, endpoint=metrics) for 10 minutes, while a kube-state-metrics scrape series still exists for this management cluster. Maestro agent metrics and any alerts that depend on them may be missing. Firing 2: State: Fired (not resolved) Started: 2026-10-22T21:24:13Z Severity: Sev3 Labels: alertname="MaestroAgentMetricsProxyDown", cluster="ci01-j3800704-mgmt-1", component="monitoring-infrastructure", severity="3" Description: Prometheus has not scraped maestro-agent /metrics through the metrics-proxy sidecar (namespace=maestro, endpoint=metrics) for 10 minutes, while a kube-state-metrics scrape series still exists for this management cluster. Maestro agent metrics and any alerts that depend on them may be missing. alert fired 2 time(s) Firing 1: State: Fired (not resolved) Started: 2026-10-21T18:12:50Z Severity: Sev3 Labels: alertname="MaestroAgentMetricsProxyDown", cluster="ci01-j5592832-mgmt-1", component="monitoring-infrastructure", region="canadacentral", severity="3" Description: Prometheus has not scraped maestro-agent /metrics through the metrics-proxy sidecar (namespace=maestro, endpoint=metrics) for 10 minutes, while a kube-state-metrics scrape series still exists for this management cluster. Maestro agent metrics and any alerts that depend on them may be missing. Firing 2: State: Fired (not resolved) Started: 2026-10-21T18:12:50Z Severity: Sev3 Labels: alertname="MaestroAgentMetricsProxyDown", cluster="ci01-j5592832-mgmt-2", component="monitoring-infrastructure", region="canadacentral", severity="3" Description: Prometheus has not scraped maestro-agent /metrics through the metrics-proxy sidecar (namespace=maestro, endpoint=metrics) for 10 minutes, while a kube-state-metrics scrape series still exists for this management cluster. Maestro agent metrics and any alerts that depend on them may be missing. Contributing tests (1)
Affected runs (11) | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
ERROR CODE: InternalServerError; detail message cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster still exists; [clusterServic...full failure pattern: ERROR CODE: InternalServerError; detail message cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster still exists; [clusterServiceStatus] state is "uninstalling"; [hostedCluster] HostedCluster still exists; [resourceTeardown] tearing down namespaces; [descendantResources] remaining resources: serviceProviderClusters; provider Microsoft.RedHatOpenShift Signal: FlakeSignal: Flake — present in 2 prior week(s); spread across 5 days | e2e | 10 | 1.01%10 of 995 job runs affected | FlakeSignal: Flake — present in 2 prior week(s); spread across 5 days | Show trend detailsOct 12: 2 · Oct 13: 0 · Oct 14: 2 · Oct 15: 4 · Oct 16: 0 · Oct 17: 0 · Oct 18: 0 · Oct 19: 0 · Oct 20: 1 · Oct 21: 2 · Oct 22: 0 · Oct 23: 0 · Oct 24: 0 · Oct 25: 0 | none | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (3)fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:310]: Unexpected error:
<*errors.joinError | 0x279359bea228>:
resource group "admin-api-version-pin-f5l942lsxnxx": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="sre-version-pin-7g5shb" in resourcegroup="admin-api-version-pin-f5l942lsxnxx" to finish deleting: GET https://rp.j9083904.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/c9617226-f7ce-4606-a2f9-eb711857548d
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: InternalServerError
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/c9617226-f7ce-4606-a2f9-eb711857548d",
"name": "c9617226-f7ce-4606-a2f9-eb711857548d",
"status": "Failed",
"startTime": "2026-10-21T14:39:39.950651125Z",
"endTime": "2026-10-21T15:03:42.224683778Z",
"error": {
"code": "InternalServerError",
"message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2tj7lluppmfrbdhhdr0g5hdh5ec68bse still exists (deletion dispatched at 2026-10-21T14:39:40Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [hostedCluster] HostedCluster still exists; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters; [resourceTeardown] tearing down namespaces (1)"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://rp.j9083904.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/c9617226-f7ce-4606-a2f9-eb711857548d
correlation ID: 48d5a3fe-e637-4aff-9f54-63cebfd278ec
client request ID: d94bc996-f0e4-48a3-ab58-bf75318409cc
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:310]: Unexpected error:
<*errors.joinError | 0x279359bea228>:
resource group "admin-api-version-pin-f5l942lsxnxx": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="sre-version-pin-7g5shb" in resourcegroup="admin-api-version-pin-f5l942lsxnxx" to finish deleting: GET https://rp.j9083904.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/c9617226-f7ce-4606-a2f9-eb711857548d
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: InternalServerError
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/c9617226-f7ce-4606-a2f9-eb711857548d",
"name": "c9617226-f7ce-4606-a2f9-eb711857548d",
"status": "Failed",
"startTime": "2026-10-21T14:39:39.950651125Z",
"endTime": "2026-10-21T15:03:42.224683778Z",
"error": {
"code": "InternalServerError",
"message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2tj7lluppmfrbdhhdr0g5hdh5ec68bse still exists (deletion dispatched at 2026-10-21T14:39:40Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [hostedCluster] HostedCluster still exists; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters; [resourceTeardown] tearing down namespaces (1)"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://rp.j9083904.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/c9617226-f7ce-4606-a2f9-eb711857548d
correlation ID: 48d5a3fe-e637-4aff-9f54-63cebfd278ec
client request ID: d94bc996-f0e4-48a3-ab58-bf75318409cc
occurredfail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:310]: Unexpected error:
<*errors.joinError | 0x2cef6ec040a8>:
resource group "admin-api-breakglass-9q9nnh4t7882": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="sre-hcp-cluster" in resourcegroup="admin-api-breakglass-9q9nnh4t7882" to finish deleting: GET https://rp.j9083904.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/8eaf5e6c-659a-4bc4-8c15-d54c813f8e68
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: InternalServerError
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/8eaf5e6c-659a-4bc4-8c15-d54c813f8e68",
"name": "8eaf5e6c-659a-4bc4-8c15-d54c813f8e68",
"status": "Failed",
"startTime": "2026-10-21T14:29:52.564744569Z",
"endTime": "2026-10-21T14:53:53.054355548Z",
"error": {
"code": "InternalServerError",
"message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2tj7od9dtldg9atesqujpab56gdtktc3 still exists (deletion dispatched at 2026-10-21T14:29:53Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [hostedCluster] HostedCluster still exists; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters; [resourceTeardown] tearing down namespaces (2)"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://rp.j9083904.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/8eaf5e6c-659a-4bc4-8c15-d54c813f8e68
correlation ID: ad5bbe53-529d-4033-95da-f2bd383b8927
client request ID: 1402c96d-416f-418a-a03d-6b48908f76d1
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:310]: Unexpected error:
<*errors.joinError | 0x2cef6ec040a8>:
resource group "admin-api-breakglass-9q9nnh4t7882": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="sre-hcp-cluster" in resourcegroup="admin-api-breakglass-9q9nnh4t7882" to finish deleting: GET https://rp.j9083904.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/8eaf5e6c-659a-4bc4-8c15-d54c813f8e68
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: InternalServerError
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/8eaf5e6c-659a-4bc4-8c15-d54c813f8e68",
"name": "8eaf5e6c-659a-4bc4-8c15-d54c813f8e68",
"status": "Failed",
"startTime": "2026-10-21T14:29:52.564744569Z",
"endTime": "2026-10-21T14:53:53.054355548Z",
"error": {
"code": "InternalServerError",
"message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2tj7od9dtldg9atesqujpab56gdtktc3 still exists (deletion dispatched at 2026-10-21T14:29:53Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [hostedCluster] HostedCluster still exists; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters; [resourceTeardown] tearing down namespaces (2)"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://rp.j9083904.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/8eaf5e6c-659a-4bc4-8c15-d54c813f8e68
correlation ID: ad5bbe53-529d-4033-95da-f2bd383b8927
client request ID: 1402c96d-416f-418a-a03d-6b48908f76d1
occurredfail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:310]: Unexpected error:
<*errors.joinError | 0x17bcf44a2b8>:
resource group "rg-cp-ystream-upgrade-4-22-r2g5kqccwtmj": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cp-ystream-upgrade-4-22-slpchf" in resourcegroup="rg-cp-ystream-upgrade-4-22-r2g5kqccwtmj" to finish deleting: GET https://rp.j1835008.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/ff043c9b-da3d-4c1d-a4b4-8cfcb1832d72
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: InternalServerError
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/ff043c9b-da3d-4c1d-a4b4-8cfcb1832d72",
"name": "ff043c9b-da3d-4c1d-a4b4-8cfcb1832d72",
"status": "Failed",
"startTime": "2026-10-20T14:45:38.289702505Z",
"endTime": "2026-10-20T15:09:42.028079236Z",
"error": {
"code": "InternalServerError",
"message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2tiihmd86lqecaamth0bbejp0vuiupso still exists (deletion dispatched at 2026-10-20T14:45:38Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [hostedCluster] HostedCluster still exists; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters; [resourceTeardown] tearing down namespaces (1)"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://rp.j1835008.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/ff043c9b-da3d-4c1d-a4b4-8cfcb1832d72
correlation ID: 23fd6d56-2117-4837-9a44-03b34bfb8098
client request ID: 7c29a11d-1441-4204-99a1-68643f5d4265
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:310]: Unexpected error:
<*errors.joinError | 0x17bcf44a2b8>:
resource group "rg-cp-ystream-upgrade-4-22-r2g5kqccwtmj": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cp-ystream-upgrade-4-22-slpchf" in resourcegroup="rg-cp-ystream-upgrade-4-22-r2g5kqccwtmj" to finish deleting: GET https://rp.j1835008.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/ff043c9b-da3d-4c1d-a4b4-8cfcb1832d72
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: InternalServerError
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/ff043c9b-da3d-4c1d-a4b4-8cfcb1832d72",
"name": "ff043c9b-da3d-4c1d-a4b4-8cfcb1832d72",
"status": "Failed",
"startTime": "2026-10-20T14:45:38.289702505Z",
"endTime": "2026-10-20T15:09:42.028079236Z",
"error": {
"code": "InternalServerError",
"message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2tiihmd86lqecaamth0bbejp0vuiupso still exists (deletion dispatched at 2026-10-20T14:45:38Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [hostedCluster] HostedCluster still exists; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters; [resourceTeardown] tearing down namespaces (1)"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://rp.j1835008.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/ff043c9b-da3d-4c1d-a4b4-8cfcb1832d72
correlation ID: 23fd6d56-2117-4837-9a44-03b34bfb8098
client request ID: 7c29a11d-1441-4204-99a1-68643f5d4265
occurredContributing tests (11)
Affected runs (10) | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
clusterIssuer not ready: reason=ErrRegisterACMEAccount message=Failed to register ACME account: Get "<url>full failure pattern: clusterIssuer not ready: reason=ErrRegisterACMEAccount message=Failed to register ACME account: Get "<url> Signal: FlakeSignal: Flake — present in 4 prior week(s); spread across 6 days | e2e | 10 | 1.01%10 of 995 job runs affected | FlakeSignal: Flake — present in 4 prior week(s); spread across 6 days | Show trend detailsOct 12: 0 · Oct 13: 2 · Oct 14: 1 · Oct 15: 2 · Oct 16: 1 · Oct 17: 0 · Oct 18: 0 · Oct 19: 0 · Oct 20: 0 · Oct 21: 0 · Oct 22: 0 · Oct 23: 2 · Oct 24: 2 · Oct 25: 0 | INT | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (3)fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:391]: Timed out after 300.000s.
ClusterIssuer should become Ready (ACME account registered)
Expected success, but got an error:
<*errors.errorString | 0x2617933fd6e0>:
clusterIssuer not ready: reason=ErrRegisterACMEAccount message=Failed to register ACME account: Get "https://acme-staging-v02.api.letsencrypt.org/directory": dial tcp: lookup acme-staging-v02.api.letsencrypt.org on 172.30.0.10:53: read udp 10.129.0.10:40244->172.30.0.10:53: read: connection refused
{
s: "clusterIssuer not ready: reason=ErrRegisterACMEAccount message=Failed to register ACME account: Get \"https://acme-staging-v02.api.letsencrypt.org/directory\": dial tcp: lookup acme-staging-v02.api.letsencrypt.org on 172.30.0.10:53: read udp 10.129.0.10:40244->172.30.0.10:53: read: connection refused",
}
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:391]: Timed out after 300.000s.
ClusterIssuer should become Ready (ACME account registered)
Expected success, but got an error:
<*errors.errorString | 0x2617933fd6e0>:
clusterIssuer not ready: reason=ErrRegisterACMEAccount message=Failed to register ACME account: Get "https://acme-staging-v02.api.letsencrypt.org/directory": dial tcp: lookup acme-staging-v02.api.letsencrypt.org on 172.30.0.10:53: read udp 10.129.0.10:40244->172.30.0.10:53: read: connection refused
{
s: "clusterIssuer not ready: reason=ErrRegisterACMEAccount message=Failed to register ACME account: Get \"https://acme-staging-v02.api.letsencrypt.org/directory\": dial tcp: lookup acme-staging-v02.api.letsencrypt.org on 172.30.0.10:53: read udp 10.129.0.10:40244->172.30.0.10:53: read: connection refused",
}fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:391]: Timed out after 300.001s.
ClusterIssuer should become Ready (ACME account registered)
Expected success, but got an error:
<*errors.errorString | 0xc346b690c20>:
clusterIssuer not ready: reason=ErrRegisterACMEAccount message=Failed to register ACME account: Get "https://acme-staging-v02.api.letsencrypt.org/directory": dial tcp: lookup acme-staging-v02.api.letsencrypt.org on 172.30.0.10:53: read udp 10.129.0.10:38043->172.30.0.10:53: read: connection refused
{
s: "clusterIssuer not ready: reason=ErrRegisterACMEAccount message=Failed to register ACME account: Get \"https://acme-staging-v02.api.letsencrypt.org/directory\": dial tcp: lookup acme-staging-v02.api.letsencrypt.org on 172.30.0.10:53: read udp 10.129.0.10:38043->172.30.0.10:53: read: connection refused",
}
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:391]: Timed out after 300.001s.
ClusterIssuer should become Ready (ACME account registered)
Expected success, but got an error:
<*errors.errorString | 0xc346b690c20>:
clusterIssuer not ready: reason=ErrRegisterACMEAccount message=Failed to register ACME account: Get "https://acme-staging-v02.api.letsencrypt.org/directory": dial tcp: lookup acme-staging-v02.api.letsencrypt.org on 172.30.0.10:53: read udp 10.129.0.10:38043->172.30.0.10:53: read: connection refused
{
s: "clusterIssuer not ready: reason=ErrRegisterACMEAccount message=Failed to register ACME account: Get \"https://acme-staging-v02.api.letsencrypt.org/directory\": dial tcp: lookup acme-staging-v02.api.letsencrypt.org on 172.30.0.10:53: read udp 10.129.0.10:38043->172.30.0.10:53: read: connection refused",
}fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:391]: Timed out after 300.000s.
ClusterIssuer should become Ready (ACME account registered)
Expected success, but got an error:
<*errors.errorString | 0x1355449513d0>:
clusterIssuer not ready: reason=ErrRegisterACMEAccount message=Failed to register ACME account: Get "https://acme-staging-v02.api.letsencrypt.org/directory": dial tcp: lookup acme-staging-v02.api.letsencrypt.org on 172.30.0.10:53: read udp 10.128.0.10:58451->172.30.0.10:53: read: connection refused
{
s: "clusterIssuer not ready: reason=ErrRegisterACMEAccount message=Failed to register ACME account: Get \"https://acme-staging-v02.api.letsencrypt.org/directory\": dial tcp: lookup acme-staging-v02.api.letsencrypt.org on 172.30.0.10:53: read udp 10.128.0.10:58451->172.30.0.10:53: read: connection refused",
}
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:391]: Timed out after 300.000s.
ClusterIssuer should become Ready (ACME account registered)
Expected success, but got an error:
<*errors.errorString | 0x1355449513d0>:
clusterIssuer not ready: reason=ErrRegisterACMEAccount message=Failed to register ACME account: Get "https://acme-staging-v02.api.letsencrypt.org/directory": dial tcp: lookup acme-staging-v02.api.letsencrypt.org on 172.30.0.10:53: read udp 10.128.0.10:58451->172.30.0.10:53: read: connection refused
{
s: "clusterIssuer not ready: reason=ErrRegisterACMEAccount message=Failed to register ACME account: Get \"https://acme-staging-v02.api.letsencrypt.org/directory\": dial tcp: lookup acme-staging-v02.api.letsencrypt.org on 172.30.0.10:53: read udp 10.128.0.10:58451->172.30.0.10:53: read: connection refused",
}Contributing tests (4)
Affected runs (10) | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| take 101, Post "<url>": context deadline exceededfull failure pattern: | take 101, Post "<url>": context deadline exceeded Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 9 day(s) | e2e | 71 | 7.14%71 of 995 job runs affected | IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 9 day(s) | Show trend detailsOct 12: 0 · Oct 13: 2 · Oct 14: 26 · Oct 15: 18 · Oct 16: 10 · Oct 17: 1 · Oct 18: 0 · Oct 19: 1 · Oct 20: 3 · Oct 21: 9 · Oct 22: 1 · Oct 23: 0 · Oct 24: 0 · Oct 25: 0 | none | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (3)fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:112]: expected complete healthy SWIFT router, ignition, KAS, DNS and worker checks for /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-mw8mkc7mm29l/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster
Unexpected error:
<*fmt.wrapError | 0x1bbfb1338400>:
VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-mw8mkc7mm29l/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.001s: query SWIFT checks in https://hcp-dev-us-2.eastus2.kusto.windows.net/ServiceLogs: Op(OpQuery): Kind(KHTTPError): With query: let target = "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-mw8mkc7mm29l/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster"; let start = datetime(2026-10-22T13:28:01.8728308Z);
let desires = cosmosResourceSnapshots
| where resourceID startswith strcat(target, "/")
| where resourceType =~ "microsoft.redhatopenshift/hcpopenshiftclusters/readdesires"
| extend resourceID = tolower(resourceID), ts = tolong(content._ts), sourceCluster = cluster
| summarize arg_max(timestamp, *) by environment, region, sourceCluster, resourceID, ts
| summarize arg_max(ts, *) by environment, region, sourceCluster, resourceID
| where isempty(content.deletionTimestamp)
| extend manifest = content.properties.status.kubeContent
| where manifest.kind == "HostedCluster" and isempty(manifest.metadata.deletionTimestamp)
| extend hostedClusterNamespace = tostring(manifest.metadata.namespace), hostedClusterName = tostring(manifest.metadata.name),
managementCluster = tolower(tostring(content.properties.spec.managementCluster))
| where isnotempty(hostedClusterNamespace) and isnotempty(hostedClusterName) and isnotempty(managementCluster)
| project environment, region, sourceCluster, managementCluster, namespace = strcat(hostedClusterNamespace, "-", hostedClusterName);
let fleet = cosmosResourceSnapshots
| where resourceType =~ "microsoft.redhatopenshift/stamps/managementclusters"
| extend managementCluster = tolower(resourceID), ts = tolong(content._ts), sourceCluster = cluster
| join kind=leftsemi desires on environment, region, sourceCluster, managementCluster
| summarize arg_max(timestamp, *) by environment, region, sourceCluster, managementCluster, ts
| summarize arg_max(ts, *) by environment, region, sourceCluster, managementCluster
| where isempty(content.deletionTimestamp)
| extend clusterName = tostring(split(tostring(content.properties.status.aksResourceID), "/")[-1])
| where isnotempty(clusterName)
| project environment, region, sourceCluster, managementCluster, cluster = clusterName;
let hcp = desires
| join kind=inner fleet on environment, region, sourceCluster, managementCluster
| distinct environment, region, cluster, namespace;
let inventory = materialize(kubernetesResourceSnapshots
| where timestamp >= start and objectKind in ("Pod", "Deployment")
| lookup kind=inner hcp on environment, region, cluster, namespace
| summarize arg_max(timestamp, *) by environment, region, cluster, namespace, objectKind, name
| where event != "Delete" and isempty(object.metadata.deletionTimestamp));
let pods = inventory
| where objectKind == "Pod" and object.metadata.labels.app == "private-router" and object.status.phase == "Running"
| project environment, region, cluster, namespace, UID = uid, Node = tostring(object.spec.nodeName);
let counts = pods | summarize Current = count() by environment, region, cluster, namespace;
let deployments = inventory
| where objectKind == "Deployment" and name == "router"
| project environment, region, cluster, namespace, Desired = tolong(object.spec.replicas);
let summaries = containerLogs
| where timestamp >= start and container_name == "swift-recorder"
| where log.controller_name == "swift-router-check" and log.field == "summary"
| extend namespace = tostring(log.pod_namespace), UID = tostring(log.pod_uid)
| lookup kind=inner hcp on environment, region, cluster, namespace
| summarize arg_max(timestamp, *) by environment, region, cluster, namespace, UID
| project environment, region, cluster, namespace, UID, SummaryUID = tostring(log.pod_uid),
SummaryNode = tostring(log.node_name), Pass = tostring(log.pass_id), At = timestamp, Summary = tostring(log.record);
hcp
| join kind=leftouter deployments on environment, region, cluster, namespace
| join kind=leftouter counts on environment, region, cluster, namespace
| join kind=leftouter pods on environment, region, cluster, namespace
| join kind=leftouter summaries on environment, region, cluster, namespace, UID
| project Scope = strcat(environment, "/", region, "/", cluster, "/", namespace),
Desired = coalesce(Desired, 0), Current = coalesce(Current, 0), UID, Node, SummaryUID, SummaryNode, Pass, At, Summary
| order by Scope asc, UID asc
| take 101, Post "https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query": context deadline exceeded
{
msg: "VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-mw8mkc7mm29l/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.001s: query SWIFT checks in https://hcp-dev-us-2.eastus2.kusto.windows.net/ServiceLogs: Op(OpQuery): Kind(KHTTPError): With query: let target = \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-mw8mkc7mm29l/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster\"; let start = datetime(2026-10-22T13:28:01.8728308Z);\nlet desires = cosmosResourceSnapshots\n| where resourceID startswith strcat(target, \"/\")\n| where resourceType =~ \"microsoft.redhatopenshift/hcpopenshiftclusters/readdesires\"\n| extend resourceID = tolower(resourceID), ts = tolong(content._ts), sourceCluster = cluster\n| summarize arg_max(timestamp, *) by environment, region, sourceCluster, resourceID, ts\n| summarize arg_max(ts, *) by environment, region, sourceCluster, resourceID\n| where isempty(content.deletionTimestamp)\n| extend manifest = content.properties.status.kubeContent\n| where manifest.kind == \"HostedCluster\" and isempty(manifest.metadata.deletionTimestamp)\n| extend hostedClusterNamespace = tostring(manifest.metadata.namespace), hostedClusterName = tostring(manifest.metadata.name),\n managementCluster = tolower(tostring(content.properties.spec.managementCluster))\n| where isnotempty(hostedClusterNamespace) and isnotempty(hostedClusterName) and isnotempty(managementCluster)\n| project environment, region, sourceCluster, managementCluster, namespace = strcat(hostedClusterNamespace, \"-\", hostedClusterName);\nlet fleet = cosmosResourceSnapshots\n| where resourceType =~ \"microsoft.redhatopenshift/stamps/managementclusters\"\n| extend managementCluster = tolower(resourceID), ts = tolong(content._ts), sourceCluster = cluster\n| join kind=leftsemi desires on environment, region, sourceCluster, managementCluster\n| summarize arg_max(timestamp, *) by environment, region, sourceCluster, managementCluster, ts\n| summarize arg_max(ts, *) by environment, region, sourceCluster, managementCluster\n| where isempty(content.deletionTimestamp)\n| extend clusterName = tostring(split(tostring(content.properties.status.aksResourceID), \"/\")[-1])\n| where isnotempty(clusterName)\n| project environment, region, sourceCluster, managementCluster, cluster = clusterName;\nlet hcp = desires\n| join kind=inner fleet on environment, region, sourceCluster, managementCluster\n| distinct environment, region, cluster, namespace;\nlet inventory = materialize(kubernetesResourceSnapshots\n| where timestamp >= start and objectKind in (\"Pod\", \"Deployment\")\n| lookup kind=inner hcp on environment, region, cluster, namespace\n| summarize arg_max(timestamp, *) by environment, region, cluster, namespace, objectKind, name\n| where event != \"Delete\" and isempty(object.metadata.deletionTimestamp));\nlet pods = inventory\n| where objectKind == \"Pod\" and object.metadata.labels.app == \"private-router\" and object.status.phase == \"Running\"\n| project environment, region, cluster, namespace, UID = uid, Node = tostring(object.spec.nodeName);\nlet counts = pods | summarize Current = count() by environment, region, cluster, namespace;\nlet deployments = inventory\n| where objectKind == \"Deployment\" and name == \"router\"\n| project environment, region, cluster, namespace, Desired = tolong(object.spec.replicas);\nlet summaries = containerLogs\n| where timestamp >= start and container_name == \"swift-recorder\"\n| where log.controller_name == \"swift-router-check\" and log.field == \"summary\"\n| extend namespace = tostring(log.pod_namespace), UID = tostring(log.pod_uid)\n| lookup kind=inner hcp on environment, region, cluster, namespace\n| summarize arg_max(timestamp, *) by environment, region, cluster, namespace, UID\n| project environment, region, cluster, namespace, UID, SummaryUID = tostring(log.pod_uid),\n SummaryNode = tostring(log.node_name), Pass = tostring(log.pass_id), At = timestamp, Summary = tostring(log.record);\nhcp\n| join kind=leftouter deployments on environment, region, cluster, namespace\n| join kind=leftouter counts on environment, region, cluster, namespace\n| join kind=leftouter pods on environment, region, cluster, namespace\n| join kind=leftouter summaries on environment, region, cluster, namespace, UID\n| project Scope = strcat(environment, \"/\", region, \"/\", cluster, \"/\", namespace),\n Desired = coalesce(Desired, 0), Current = coalesce(Current, 0), UID, Node, SummaryUID, SummaryNode, Pass, At, Summary\n| order by Scope asc, UID asc\n| take 101, Post \"https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query\": context deadline exceeded",
err: <*fmt.wrapError | 0x1bbfb13383e0>{
msg: "query SWIFT checks in https://hcp-dev-us-2.eastus2.kusto.windows.net/ServiceLogs: Op(OpQuery): Kind(KHTTPError): With query: let target = \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-mw8mkc7mm29l/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster\"; let start = datetime(2026-10-22T13:28:01.8728308Z);\nlet desires = cosmosResourceSnapshots\n| where resourceID startswith strcat(target, \"/\")\n| where resourceType =~ \"microsoft.redhatopenshift/hcpopenshiftclusters/readdesires\"\n| extend resourceID = tolower(resourceID), ts = tolong(content._ts), sourceCluster = cluster\n| summarize arg_max(timestamp, *) by environment, region, sourceCluster, resourceID, ts\n| summarize arg_max(ts, *) by environment, region, sourceCluster, resourceID\n| where isempty(content.deletionTimestamp)\n| extend manifest = content.properties.status.kubeContent\n| where manifest.kind == \"HostedCluster\" and isempty(manifest.metadata.deletionTimestamp)\n| extend hostedClusterNamespace = tostring(manifest.metadata.namespace), hostedClusterName = tostring(manifest.metadata.name),\n managementCluster = tolower(tostring(content.properties.spec.managementCluster))\n| where isnotempty(hostedClusterNamespace) and isnotempty(hostedClusterName) and isnotempty(managementCluster)\n| project environment, region, sourceCluster, managementCluster, namespace = strcat(hostedClusterNamespace, \"-\", hostedClusterName);\nlet fleet = cosmosResourceSnapshots\n| where resourceType =~ \"microsoft.redhatopenshift/stamps/managementclusters\"\n| extend managementCluster = tolower(resourceID), ts = tolong(content._ts), sourceCluster = cluster\n| join kind=leftsemi desires on environment, region, sourceCluster, managementCluster\n| summarize arg_max(timestamp, *) by environment, region, sourceCluster, managementCluster, ts\n| summarize arg_max(ts, *) by environment, region, sourceCluster, managementCluster\n| where isempty(content.deletionTimestamp)\n| extend clusterName = tostring(split(tostring(content.properties.status.aksResourceID), \"/\")[-1])\n| where isnotempty(clusterName)\n| project environment, region, sourceCluster, managementCluster, cluster = clusterName;\nlet hcp = desires\n| join kind=inner fleet on environment, region, sourceCluster, managementCluster\n| distinct environment, region, cluster, namespace;\nlet inventory = materialize(kubernetesResourceSnapshots\n| where timestamp >= start and objectKind in (\"Pod\", \"Deployment\")\n| lookup kind=inner hcp on environment, region, cluster, namespace\n| summarize arg_max(timestamp, *) by environment, region, cluster, namespace, objectKind, name\n| where event != \"Delete\" and isempty(object.metadata.deletionTimestamp));\nlet pods = inventory\n| where objectKind == \"Pod\" and object.metadata.labels.app == \"private-router\" and object.status.phase == \"Running\"\n| project environment, region, cluster, namespace, UID = uid, Node = tostring(object.spec.nodeName);\nlet counts = pods | summarize Current = count() by environment, region, cluster, namespace;\nlet deployments = inventory\n| where objectKind == \"Deployment\" and name == \"router\"\n| project environment, region, cluster, namespace, Desired = tolong(object.spec.replicas);\nlet summaries = containerLogs\n| where timestamp >= start and container_name == \"swift-recorder\"\n| where log.controller_name == \"swift-router-check\" and log.field == \"summary\"\n| extend namespace = tostring(log.pod_namespace), UID = tostring(log.pod_uid)\n| lookup kind=inner hcp on environment, region, cluster, namespace\n| summarize arg_max(timestamp, *) by environment, region, cluster, namespace, UID\n| project environment, region, cluster, namespace, UID, SummaryUID = tostring(log.pod_uid),\n SummaryNode = tostring(log.node_name), Pass = tostring(log.pass_id), At = timestamp, Summary = tostring(log.record);\nhcp\n| join kind=leftouter deployments on environment, region, cluster, namespace\n| join kind=leftouter counts on environment, region, cluster, namespace\n| join kind=leftouter pods on environment, region, cluster, namespace\n| join kind=leftouter summaries on environment, region, cluster, namespace, UID\n| project Scope = strcat(environment, \"/\", region, \"/\", cluster, \"/\", namespace),\n Desired = coalesce(Desired, 0), Current = coalesce(Current, 0), UID, Node, SummaryUID, SummaryNode, Pass, At, Summary\n| order by Scope asc, UID asc\n| take 101, Post \"https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query\": context deadline exceeded",
err: <*errors.Error | 0x1bbfb092c1e0>{
Op: 1,
Kind: 7,
Err: <*fmt.wrapError | 0x1bbfb13383c0>{
msg: "With query: let target = \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-mw8mkc7mm29l/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster\"; let start = datetime(2026-10-22T13:28:01.8728308Z);\nlet desires = cosmosResourceSnapshots\n| where resourceID startswith strcat(target, \"/\")\n| where resourceType =~ \"microsoft.redhatopenshift/hcpopenshiftclusters/readdesires\"\n| extend resourceID = tolower(resourceID), ts = tolong(content._ts), sourceCluster = cluster\n| summarize arg_max(timestamp, *) by environment, region, sourceCluster, resourceID, ts\n| summarize arg_max(ts, *) by environment, region, sourceCluster, resourceID\n| where isempty(content.deletionTimestamp)\n| extend manifest = content.properties.status.kubeContent\n| where manifest.kind == \"HostedCluster\" and isempty(manifest.metadata.deletionTimestamp)\n| extend hostedClusterNamespace = tostring(manifest.metadata.namespace), hostedClusterName = tostring(manifest.metadata.name),\n managementCluster = tolower(tostring(content.properties.spec.managementCluster))\n| where isnotempty(hostedClusterNamespace) and isnotempty(hostedClusterName) and isnotempty(managementCluster)\n| project environment, region, sourceCluster, managementCluster, namespace = strcat(hostedClusterNamespace, \"-\", hostedClusterName);\nlet fleet = cosmosResourceSnapshots\n| where resourceType =~ \"microsoft.redhatopenshift/stamps/managementclusters\"\n| extend managementCluster = tolower(resourceID), ts = tolong(content._ts), sourceCluster = cluster\n| join kind=leftsemi desires on environment, region, sourceCluster, managementCluster\n| summarize arg_max(timestamp, *) by environment, region, sourceCluster, managementCluster, ts\n| summarize arg_max(ts, *) by environment, region, sourceCluster, managementCluster\n| where isempty(content.deletionTimestamp)\n| extend clusterName = tostring(split(tostring(content.properties.status.aksResourceID), \"/\")[-1])\n| where isnotempty(clusterName)\n| project environment, region, sourceCluster, managementCluster, cluster = clusterName;\nlet hcp = desires\n| join kind=inner fleet on environment, region, sourceCluster, managementCluster\n| distinct environment, region, cluster, namespace;\nlet inventory = materialize(kubernetesResourceSnapshots\n| where timestamp >= start and objectKind in (\"Pod\", \"Deployment\")\n| lookup kind=inner hcp on environment, region, cluster, namespace\n| summarize arg_max(timestamp, *) by environment, region, cluster, namespace, objectKind, name\n| where event != \"Delete\" and isempty(object.metadata.deletionTimestamp));\nlet pods = inventory\n| where objectKind == \"Pod\" and object.metadata.labels.app == \"private-router\" and object.status.phase == \"Running\"\n| project environment, region, cluster, namespace, UID = uid, Node = tostring(object.spec.nodeName);\nlet counts = pods | summarize Current = count() by environment, region, cluster, namespace;\nlet deployments = inventory\n| where objectKind == \"Deployment\" and name == \"router\"\n| project environment, region, cluster, namespace, Desired = tolong(object.spec.replicas);\nlet summaries = containerLogs\n| where timestamp >= start and container_name == \"swift-recorder\"\n| where log.controller_name == \"swift-router-check\" and log.field == \"summary\"\n| extend namespace = tostring(log.pod_namespace), UID = tostring(log.pod_uid)\n| lookup kind=inner hcp on environment, region, cluster, namespace\n| summarize arg_max(timestamp, *) by environment, region, cluster, namespace, UID\n| project environment, region, cluster, namespace, UID, SummaryUID = tostring(log.pod_uid),\n SummaryNode = tostring(log.node_name), Pass = tostring(log.pass_id), At = timestamp, Summary = tostring(log.record);\nhcp\n| join kind=leftouter deployments on environment, region, cluster, namespace\n| join kind=leftouter counts on environment, region, cluster, namespace\n| join kind=leftouter pods on environment, region, cluster, namespace\n| join kind=leftouter summaries on environment, region, cluster, namespace, UID\n| project Scope = strcat(environment, \"/\", region, \"/\", cluster, \"/\", namespace),\n Desired = coalesce(Desired, 0), Current = coalesce(Current, 0), UID, Node, SummaryUID, SummaryNode, Pass, At, Summary\n| order by Scope asc, UID asc\n| take 101, Post \"https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query\": context deadline exceeded",
err: <*url.Error | 0x1bbfb280d260>{
Op: "Post",
URL: "https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query",
Err: <context.deadlineExceededError>{},
},
},
restErrMsg: nil,
decoded: nil,
permanent: false,
inner: nil,
},
},
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:112]: expected complete healthy SWIFT router, ignition, KAS, DNS and worker checks for /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-mw8mkc7mm29l/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster
Unexpected error:
<*fmt.wrapError | 0x1bbfb1338400>:
VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-mw8mkc7mm29l/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.001s: query SWIFT checks in https://hcp-dev-us-2.eastus2.kusto.windows.net/ServiceLogs: Op(OpQuery): Kind(KHTTPError): With query: let target = "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-mw8mkc7mm29l/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster"; let start = datetime(2026-10-22T13:28:01.8728308Z);
let desires = cosmosResourceSnapshots
| where resourceID startswith strcat(target, "/")
| where resourceType =~ "microsoft.redhatopenshift/hcpopenshiftclusters/readdesires"
| extend resourceID = tolower(resourceID), ts = tolong(content._ts), sourceCluster = cluster
| summarize arg_max(timestamp, *) by environment, region, sourceCluster, resourceID, ts
| summarize arg_max(ts, *) by environment, region, sourceCluster, resourceID
| where isempty(content.deletionTimestamp)
| extend manifest = content.properties.status.kubeContent
| where manifest.kind == "HostedCluster" and isempty(manifest.metadata.deletionTimestamp)
| extend hostedClusterNamespace = tostring(manifest.metadata.namespace), hostedClusterName = tostring(manifest.metadata.name),
managementCluster = tolower(tostring(content.properties.spec.managementCluster))
| where isnotempty(hostedClusterNamespace) and isnotempty(hostedClusterName) and isnotempty(managementCluster)
| project environment, region, sourceCluster, managementCluster, namespace = strcat(hostedClusterNamespace, "-", hostedClusterName);
let fleet = cosmosResourceSnapshots
| where resourceType =~ "microsoft.redhatopenshift/stamps/managementclusters"
| extend managementCluster = tolower(resourceID), ts = tolong(content._ts), sourceCluster = cluster
| join kind=leftsemi desires on environment, region, sourceCluster, managementCluster
| summarize arg_max(timestamp, *) by environment, region, sourceCluster, managementCluster, ts
| summarize arg_max(ts, *) by environment, region, sourceCluster, managementCluster
| where isempty(content.deletionTimestamp)
| extend clusterName = tostring(split(tostring(content.properties.status.aksResourceID), "/")[-1])
| where isnotempty(clusterName)
| project environment, region, sourceCluster, managementCluster, cluster = clusterName;
let hcp = desires
| join kind=inner fleet on environment, region, sourceCluster, managementCluster
| distinct environment, region, cluster, namespace;
let inventory = materialize(kubernetesResourceSnapshots
| where timestamp >= start and objectKind in ("Pod", "Deployment")
| lookup kind=inner hcp on environment, region, cluster, namespace
| summarize arg_max(timestamp, *) by environment, region, cluster, namespace, objectKind, name
| where event != "Delete" and isempty(object.metadata.deletionTimestamp));
let pods = inventory
| where objectKind == "Pod" and object.metadata.labels.app == "private-router" and object.status.phase == "Running"
| project environment, region, cluster, namespace, UID = uid, Node = tostring(object.spec.nodeName);
let counts = pods | summarize Current = count() by environment, region, cluster, namespace;
let deployments = inventory
| where objectKind == "Deployment" and name == "router"
| project environment, region, cluster, namespace, Desired = tolong(object.spec.replicas);
let summaries = containerLogs
| where timestamp >= start and container_name == "swift-recorder"
| where log.controller_name == "swift-router-check" and log.field == "summary"
| extend namespace = tostring(log.pod_namespace), UID = tostring(log.pod_uid)
| lookup kind=inner hcp on environment, region, cluster, namespace
| summarize arg_max(timestamp, *) by environment, region, cluster, namespace, UID
| project environment, region, cluster, namespace, UID, SummaryUID = tostring(log.pod_uid),
SummaryNode = tostring(log.node_name), Pass = tostring(log.pass_id), At = timestamp, Summary = tostring(log.record);
hcp
| join kind=leftouter deployments on environment, region, cluster, namespace
| join kind=leftouter counts on environment, region, cluster, namespace
| join kind=leftouter pods on environment, region, cluster, namespace
| join kind=leftouter summaries on environment, region, cluster, namespace, UID
| project Scope = strcat(environment, "/", region, "/", cluster, "/", namespace),
Desired = coalesce(Desired, 0), Current = coalesce(Current, 0), UID, Node, SummaryUID, SummaryNode, Pass, At, Summary
| order by Scope asc, UID asc
| take 101, Post "https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query": context deadline exceeded
{
msg: "VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-mw8mkc7mm29l/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.001s: query SWIFT checks in https://hcp-dev-us-2.eastus2.kusto.windows.net/ServiceLogs: Op(OpQuery): Kind(KHTTPError): With query: let target = \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-mw8mkc7mm29l/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster\"; let start = datetime(2026-10-22T13:28:01.8728308Z);\nlet desires = cosmosResourceSnapshots\n| where resourceID startswith strcat(target, \"/\")\n| where resourceType =~ \"microsoft.redhatopenshift/hcpopenshiftclusters/readdesires\"\n| extend resourceID = tolower(resourceID), ts = tolong(content._ts), sourceCluster = cluster\n| summarize arg_max(timestamp, *) by environment, region, sourceCluster, resourceID, ts\n| summarize arg_max(ts, *) by environment, region, sourceCluster, resourceID\n| where isempty(content.deletionTimestamp)\n| extend manifest = content.properties.status.kubeContent\n| where manifest.kind == \"HostedCluster\" and isempty(manifest.metadata.deletionTimestamp)\n| extend hostedClusterNamespace = tostring(manifest.metadata.namespace), hostedClusterName = tostring(manifest.metadata.name),\n managementCluster = tolower(tostring(content.properties.spec.managementCluster))\n| where isnotempty(hostedClusterNamespace) and isnotempty(hostedClusterName) and isnotempty(managementCluster)\n| project environment, region, sourceCluster, managementCluster, namespace = strcat(hostedClusterNamespace, \"-\", hostedClusterName);\nlet fleet = cosmosResourceSnapshots\n| where resourceType =~ \"microsoft.redhatopenshift/stamps/managementclusters\"\n| extend managementCluster = tolower(resourceID), ts = tolong(content._ts), sourceCluster = cluster\n| join kind=leftsemi desires on environment, region, sourceCluster, managementCluster\n| summarize arg_max(timestamp, *) by environment, region, sourceCluster, managementCluster, ts\n| summarize arg_max(ts, *) by environment, region, sourceCluster, managementCluster\n| where isempty(content.deletionTimestamp)\n| extend clusterName = tostring(split(tostring(content.properties.status.aksResourceID), \"/\")[-1])\n| where isnotempty(clusterName)\n| project environment, region, sourceCluster, managementCluster, cluster = clusterName;\nlet hcp = desires\n| join kind=inner fleet on environment, region, sourceCluster, managementCluster\n| distinct environment, region, cluster, namespace;\nlet inventory = materialize(kubernetesResourceSnapshots\n| where timestamp >= start and objectKind in (\"Pod\", \"Deployment\")\n| lookup kind=inner hcp on environment, region, cluster, namespace\n| summarize arg_max(timestamp, *) by environment, region, cluster, namespace, objectKind, name\n| where event != \"Delete\" and isempty(object.metadata.deletionTimestamp));\nlet pods = inventory\n| where objectKind == \"Pod\" and object.metadata.labels.app == \"private-router\" and object.status.phase == \"Running\"\n| project environment, region, cluster, namespace, UID = uid, Node = tostring(object.spec.nodeName);\nlet counts = pods | summarize Current = count() by environment, region, cluster, namespace;\nlet deployments = inventory\n| where objectKind == \"Deployment\" and name == \"router\"\n| project environment, region, cluster, namespace, Desired = tolong(object.spec.replicas);\nlet summaries = containerLogs\n| where timestamp >= start and container_name == \"swift-recorder\"\n| where log.controller_name == \"swift-router-check\" and log.field == \"summary\"\n| extend namespace = tostring(log.pod_namespace), UID = tostring(log.pod_uid)\n| lookup kind=inner hcp on environment, region, cluster, namespace\n| summarize arg_max(timestamp, *) by environment, region, cluster, namespace, UID\n| project environment, region, cluster, namespace, UID, SummaryUID = tostring(log.pod_uid),\n SummaryNode = tostring(log.node_name), Pass = tostring(log.pass_id), At = timestamp, Summary = tostring(log.record);\nhcp\n| join kind=leftouter deployments on environment, region, cluster, namespace\n| join kind=leftouter counts on environment, region, cluster, namespace\n| join kind=leftouter pods on environment, region, cluster, namespace\n| join kind=leftouter summaries on environment, region, cluster, namespace, UID\n| project Scope = strcat(environment, \"/\", region, \"/\", cluster, \"/\", namespace),\n Desired = coalesce(Desired, 0), Current = coalesce(Current, 0), UID, Node, SummaryUID, SummaryNode, Pass, At, Summary\n| order by Scope asc, UID asc\n| take 101, Post \"https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query\": context deadline exceeded",
err: <*fmt.wrapError | 0x1bbfb13383e0>{
msg: "query SWIFT checks in https://hcp-dev-us-2.eastus2.kusto.windows.net/ServiceLogs: Op(OpQuery): Kind(KHTTPError): With query: let target = \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-mw8mkc7mm29l/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster\"; let start = datetime(2026-10-22T13:28:01.8728308Z);\nlet desires = cosmosResourceSnapshots\n| where resourceID startswith strcat(target, \"/\")\n| where resourceType =~ \"microsoft.redhatopenshift/hcpopenshiftclusters/readdesires\"\n| extend resourceID = tolower(resourceID), ts = tolong(content._ts), sourceCluster = cluster\n| summarize arg_max(timestamp, *) by environment, region, sourceCluster, resourceID, ts\n| summarize arg_max(ts, *) by environment, region, sourceCluster, resourceID\n| where isempty(content.deletionTimestamp)\n| extend manifest = content.properties.status.kubeContent\n| where manifest.kind == \"HostedCluster\" and isempty(manifest.metadata.deletionTimestamp)\n| extend hostedClusterNamespace = tostring(manifest.metadata.namespace), hostedClusterName = tostring(manifest.metadata.name),\n managementCluster = tolower(tostring(content.properties.spec.managementCluster))\n| where isnotempty(hostedClusterNamespace) and isnotempty(hostedClusterName) and isnotempty(managementCluster)\n| project environment, region, sourceCluster, managementCluster, namespace = strcat(hostedClusterNamespace, \"-\", hostedClusterName);\nlet fleet = cosmosResourceSnapshots\n| where resourceType =~ \"microsoft.redhatopenshift/stamps/managementclusters\"\n| extend managementCluster = tolower(resourceID), ts = tolong(content._ts), sourceCluster = cluster\n| join kind=leftsemi desires on environment, region, sourceCluster, managementCluster\n| summarize arg_max(timestamp, *) by environment, region, sourceCluster, managementCluster, ts\n| summarize arg_max(ts, *) by environment, region, sourceCluster, managementCluster\n| where isempty(content.deletionTimestamp)\n| extend clusterName = tostring(split(tostring(content.properties.status.aksResourceID), \"/\")[-1])\n| where isnotempty(clusterName)\n| project environment, region, sourceCluster, managementCluster, cluster = clusterName;\nlet hcp = desires\n| join kind=inner fleet on environment, region, sourceCluster, managementCluster\n| distinct environment, region, cluster, namespace;\nlet inventory = materialize(kubernetesResourceSnapshots\n| where timestamp >= start and objectKind in (\"Pod\", \"Deployment\")\n| lookup kind=inner hcp on environment, region, cluster, namespace\n| summarize arg_max(timestamp, *) by environment, region, cluster, namespace, objectKind, name\n| where event != \"Delete\" and isempty(object.metadata.deletionTimestamp));\nlet pods = inventory\n| where objectKind == \"Pod\" and object.metadata.labels.app == \"private-router\" and object.status.phase == \"Running\"\n| project environment, region, cluster, namespace, UID = uid, Node = tostring(object.spec.nodeName);\nlet counts = pods | summarize Current = count() by environment, region, cluster, namespace;\nlet deployments = inventory\n| where objectKind == \"Deployment\" and name == \"router\"\n| project environment, region, cluster, namespace, Desired = tolong(object.spec.replicas);\nlet summaries = containerLogs\n| where timestamp >= start and container_name == \"swift-recorder\"\n| where log.controller_name == \"swift-router-check\" and log.field == \"summary\"\n| extend namespace = tostring(log.pod_namespace), UID = tostring(log.pod_uid)\n| lookup kind=inner hcp on environment, region, cluster, namespace\n| summarize arg_max(timestamp, *) by environment, region, cluster, namespace, UID\n| project environment, region, cluster, namespace, UID, SummaryUID = tostring(log.pod_uid),\n SummaryNode = tostring(log.node_name), Pass = tostring(log.pass_id), At = timestamp, Summary = tostring(log.record);\nhcp\n| join kind=leftouter deployments on environment, region, cluster, namespace\n| join kind=leftouter counts on environment, region, cluster, namespace\n| join kind=leftouter pods on environment, region, cluster, namespace\n| join kind=leftouter summaries on environment, region, cluster, namespace, UID\n| project Scope = strcat(environment, \"/\", region, \"/\", cluster, \"/\", namespace),\n Desired = coalesce(Desired, 0), Current = coalesce(Current, 0), UID, Node, SummaryUID, SummaryNode, Pass, At, Summary\n| order by Scope asc, UID asc\n| take 101, Post \"https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query\": context deadline exceeded",
err: <*errors.Error | 0x1bbfb092c1e0>{
Op: 1,
Kind: 7,
Err: <*fmt.wrapError | 0x1bbfb13383c0>{
msg: "With query: let target = \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-mw8mkc7mm29l/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster\"; let start = datetime(2026-10-22T13:28:01.8728308Z);\nlet desires = cosmosResourceSnapshots\n| where resourceID startswith strcat(target, \"/\")\n| where resourceType =~ \"microsoft.redhatopenshift/hcpopenshiftclusters/readdesires\"\n| extend resourceID = tolower(resourceID), ts = tolong(content._ts), sourceCluster = cluster\n| summarize arg_max(timestamp, *) by environment, region, sourceCluster, resourceID, ts\n| summarize arg_max(ts, *) by environment, region, sourceCluster, resourceID\n| where isempty(content.deletionTimestamp)\n| extend manifest = content.properties.status.kubeContent\n| where manifest.kind == \"HostedCluster\" and isempty(manifest.metadata.deletionTimestamp)\n| extend hostedClusterNamespace = tostring(manifest.metadata.namespace), hostedClusterName = tostring(manifest.metadata.name),\n managementCluster = tolower(tostring(content.properties.spec.managementCluster))\n| where isnotempty(hostedClusterNamespace) and isnotempty(hostedClusterName) and isnotempty(managementCluster)\n| project environment, region, sourceCluster, managementCluster, namespace = strcat(hostedClusterNamespace, \"-\", hostedClusterName);\nlet fleet = cosmosResourceSnapshots\n| where resourceType =~ \"microsoft.redhatopenshift/stamps/managementclusters\"\n| extend managementCluster = tolower(resourceID), ts = tolong(content._ts), sourceCluster = cluster\n| join kind=leftsemi desires on environment, region, sourceCluster, managementCluster\n| summarize arg_max(timestamp, *) by environment, region, sourceCluster, managementCluster, ts\n| summarize arg_max(ts, *) by environment, region, sourceCluster, managementCluster\n| where isempty(content.deletionTimestamp)\n| extend clusterName = tostring(split(tostring(content.properties.status.aksResourceID), \"/\")[-1])\n| where isnotempty(clusterName)\n| project environment, region, sourceCluster, managementCluster, cluster = clusterName;\nlet hcp = desires\n| join kind=inner fleet on environment, region, sourceCluster, managementCluster\n| distinct environment, region, cluster, namespace;\nlet inventory = materialize(kubernetesResourceSnapshots\n| where timestamp >= start and objectKind in (\"Pod\", \"Deployment\")\n| lookup kind=inner hcp on environment, region, cluster, namespace\n| summarize arg_max(timestamp, *) by environment, region, cluster, namespace, objectKind, name\n| where event != \"Delete\" and isempty(object.metadata.deletionTimestamp));\nlet pods = inventory\n| where objectKind == \"Pod\" and object.metadata.labels.app == \"private-router\" and object.status.phase == \"Running\"\n| project environment, region, cluster, namespace, UID = uid, Node = tostring(object.spec.nodeName);\nlet counts = pods | summarize Current = count() by environment, region, cluster, namespace;\nlet deployments = inventory\n| where objectKind == \"Deployment\" and name == \"router\"\n| project environment, region, cluster, namespace, Desired = tolong(object.spec.replicas);\nlet summaries = containerLogs\n| where timestamp >= start and container_name == \"swift-recorder\"\n| where log.controller_name == \"swift-router-check\" and log.field == \"summary\"\n| extend namespace = tostring(log.pod_namespace), UID = tostring(log.pod_uid)\n| lookup kind=inner hcp on environment, region, cluster, namespace\n| summarize arg_max(timestamp, *) by environment, region, cluster, namespace, UID\n| project environment, region, cluster, namespace, UID, SummaryUID = tostring(log.pod_uid),\n SummaryNode = tostring(log.node_name), Pass = tostring(log.pass_id), At = timestamp, Summary = tostring(log.record);\nhcp\n| join kind=leftouter deployments on environment, region, cluster, namespace\n| join kind=leftouter counts on environment, region, cluster, namespace\n| join kind=leftouter pods on environment, region, cluster, namespace\n| join kind=leftouter summaries on environment, region, cluster, namespace, UID\n| project Scope = strcat(environment, \"/\", region, \"/\", cluster, \"/\", namespace),\n Desired = coalesce(Desired, 0), Current = coalesce(Current, 0), UID, Node, SummaryUID, SummaryNode, Pass, At, Summary\n| order by Scope asc, UID asc\n| take 101, Post \"https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query\": context deadline exceeded",
err: <*url.Error | 0x1bbfb280d260>{
Op: "Post",
URL: "https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query",
Err: <context.deadlineExceededError>{},
},
},
restErrMsg: nil,
decoded: nil,
permanent: false,
inner: nil,
},
},
}
occurredfail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:112]: expected complete healthy SWIFT router, ignition, KAS, DNS and worker checks for /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-npskt6qgktdr/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster
Unexpected error:
<*fmt.wrapError | 0x3fb9cc54d40>:
VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-npskt6qgktdr/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.003s: query SWIFT checks in https://hcp-dev-us-2.eastus2.kusto.windows.net/ServiceLogs: Op(OpQuery): Kind(KHTTPError): With query: let env = "dev"; let rgn = "centralus"; let cl = "ci01-j4218880-mgmt-1"; let ns = "ocm-arohcpci01-2tjdecblfn866doucbvsck82v6djrsrv-p7h2c2c6w4m5q7f"; let start = datetime(2026-10-21T20:47:23.6514068Z);
let scope = strcat(env, "/", rgn, "/", cl, "/", ns);
let inventory = materialize(kubernetesResourceSnapshots
| where timestamp >= start and environment == env and region == rgn and cluster == cl and namespace == ns
| where objectKind in ("Pod", "Deployment")
| summarize arg_max(timestamp, *) by objectKind, name
| where event != "Delete" and isempty(object.metadata.deletionTimestamp));
let pods = inventory
| where objectKind == "Pod" and object.metadata.labels.app == "private-router" and object.status.phase == "Running"
| project UID = uid, Node = tostring(object.spec.nodeName);
let desired = toscalar(inventory
| where objectKind == "Deployment" and name == "router"
| project tolong(object.spec.replicas));
let current = toscalar(pods | count);
let summaries = containerLogs
| where timestamp >= start and environment == env and region == rgn and cluster == cl
| where container_name == "swift-recorder"
| where log.controller_name == "swift-router-check" and log.field == "summary"
| where tostring(log.pod_namespace) == ns
| extend UID = tostring(log.pod_uid)
| summarize arg_max(timestamp, *) by UID
| project UID, SummaryUID = tostring(log.pod_uid),
SummaryNode = tostring(log.node_name), Pass = tostring(log.pass_id), At = timestamp, Summary = tostring(log.record);
pods
| join kind=leftouter summaries on UID
| extend Scope = scope, Desired = coalesce(desired, tolong(0)), Current = coalesce(current, tolong(0))
| project Scope, Desired, Current, UID, Node, SummaryUID, SummaryNode, Pass, At, Summary
| order by Scope asc, UID asc
| take 101, Post "https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query": context deadline exceeded
{
msg: "VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-npskt6qgktdr/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.003s: query SWIFT checks in https://hcp-dev-us-2.eastus2.kusto.windows.net/ServiceLogs: Op(OpQuery): Kind(KHTTPError): With query: let env = \"dev\"; let rgn = \"centralus\"; let cl = \"ci01-j4218880-mgmt-1\"; let ns = \"ocm-arohcpci01-2tjdecblfn866doucbvsck82v6djrsrv-p7h2c2c6w4m5q7f\"; let start = datetime(2026-10-21T20:47:23.6514068Z);\nlet scope = strcat(env, \"/\", rgn, \"/\", cl, \"/\", ns);\nlet inventory = materialize(kubernetesResourceSnapshots\n| where timestamp >= start and environment == env and region == rgn and cluster == cl and namespace == ns\n| where objectKind in (\"Pod\", \"Deployment\")\n| summarize arg_max(timestamp, *) by objectKind, name\n| where event != \"Delete\" and isempty(object.metadata.deletionTimestamp));\nlet pods = inventory\n| where objectKind == \"Pod\" and object.metadata.labels.app == \"private-router\" and object.status.phase == \"Running\"\n| project UID = uid, Node = tostring(object.spec.nodeName);\nlet desired = toscalar(inventory\n| where objectKind == \"Deployment\" and name == \"router\"\n| project tolong(object.spec.replicas));\nlet current = toscalar(pods | count);\nlet summaries = containerLogs\n| where timestamp >= start and environment == env and region == rgn and cluster == cl\n| where container_name == \"swift-recorder\"\n| where log.controller_name == \"swift-router-check\" and log.field == \"summary\"\n| where tostring(log.pod_namespace) == ns\n| extend UID = tostring(log.pod_uid)\n| summarize arg_max(timestamp, *) by UID\n| project UID, SummaryUID = tostring(log.pod_uid),\n SummaryNode = tostring(log.node_name), Pass = tostring(log.pass_id), At = timestamp, Summary = tostring(log.record);\npods\n| join kind=leftouter summaries on UID\n| extend Scope = scope, Desired = coalesce(desired, tolong(0)), Current = coalesce(current, tolong(0))\n| project Scope, Desired, Current, UID, Node, SummaryUID, SummaryNode, Pass, At, Summary\n| order by Scope asc, UID asc\n| take 101, Post \"https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query\": context deadline exceeded",
err: <*fmt.wrapError | 0x3fb9cc54d20>{
msg: "query SWIFT checks in https://hcp-dev-us-2.eastus2.kusto.windows.net/ServiceLogs: Op(OpQuery): Kind(KHTTPError): With query: let env = \"dev\"; let rgn = \"centralus\"; let cl = \"ci01-j4218880-mgmt-1\"; let ns = \"ocm-arohcpci01-2tjdecblfn866doucbvsck82v6djrsrv-p7h2c2c6w4m5q7f\"; let start = datetime(2026-10-21T20:47:23.6514068Z);\nlet scope = strcat(env, \"/\", rgn, \"/\", cl, \"/\", ns);\nlet inventory = materialize(kubernetesResourceSnapshots\n| where timestamp >= start and environment == env and region == rgn and cluster == cl and namespace == ns\n| where objectKind in (\"Pod\", \"Deployment\")\n| summarize arg_max(timestamp, *) by objectKind, name\n| where event != \"Delete\" and isempty(object.metadata.deletionTimestamp));\nlet pods = inventory\n| where objectKind == \"Pod\" and object.metadata.labels.app == \"private-router\" and object.status.phase == \"Running\"\n| project UID = uid, Node = tostring(object.spec.nodeName);\nlet desired = toscalar(inventory\n| where objectKind == \"Deployment\" and name == \"router\"\n| project tolong(object.spec.replicas));\nlet current = toscalar(pods | count);\nlet summaries = containerLogs\n| where timestamp >= start and environment == env and region == rgn and cluster == cl\n| where container_name == \"swift-recorder\"\n| where log.controller_name == \"swift-router-check\" and log.field == \"summary\"\n| where tostring(log.pod_namespace) == ns\n| extend UID = tostring(log.pod_uid)\n| summarize arg_max(timestamp, *) by UID\n| project UID, SummaryUID = tostring(log.pod_uid),\n SummaryNode = tostring(log.node_name), Pass = tostring(log.pass_id), At = timestamp, Summary = tostring(log.record);\npods\n| join kind=leftouter summaries on UID\n| extend Scope = scope, Desired = coalesce(desired, tolong(0)), Current = coalesce(current, tolong(0))\n| project Scope, Desired, Current, UID, Node, SummaryUID, SummaryNode, Pass, At, Summary\n| order by Scope asc, UID asc\n| take 101, Post \"https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query\": context deadline exceeded",
err: <*errors.Error | 0x3fb9be4c6e0>{
Op: 1,
Kind: 7,
Err: <*fmt.wrapError | 0x3fb9cc54d00>{
msg: "With query: let env = \"dev\"; let rgn = \"centralus\"; let cl = \"ci01-j4218880-mgmt-1\"; let ns = \"ocm-arohcpci01-2tjdecblfn866doucbvsck82v6djrsrv-p7h2c2c6w4m5q7f\"; let start = datetime(2026-10-21T20:47:23.6514068Z);\nlet scope = strcat(env, \"/\", rgn, \"/\", cl, \"/\", ns);\nlet inventory = materialize(kubernetesResourceSnapshots\n| where timestamp >= start and environment == env and region == rgn and cluster == cl and namespace == ns\n| where objectKind in (\"Pod\", \"Deployment\")\n| summarize arg_max(timestamp, *) by objectKind, name\n| where event != \"Delete\" and isempty(object.metadata.deletionTimestamp));\nlet pods = inventory\n| where objectKind == \"Pod\" and object.metadata.labels.app == \"private-router\" and object.status.phase == \"Running\"\n| project UID = uid, Node = tostring(object.spec.nodeName);\nlet desired = toscalar(inventory\n| where objectKind == \"Deployment\" and name == \"router\"\n| project tolong(object.spec.replicas));\nlet current = toscalar(pods | count);\nlet summaries = containerLogs\n| where timestamp >= start and environment == env and region == rgn and cluster == cl\n| where container_name == \"swift-recorder\"\n| where log.controller_name == \"swift-router-check\" and log.field == \"summary\"\n| where tostring(log.pod_namespace) == ns\n| extend UID = tostring(log.pod_uid)\n| summarize arg_max(timestamp, *) by UID\n| project UID, SummaryUID = tostring(log.pod_uid),\n SummaryNode = tostring(log.node_name), Pass = tostring(log.pass_id), At = timestamp, Summary = tostring(log.record);\npods\n| join kind=leftouter summaries on UID\n| extend Scope = scope, Desired = coalesce(desired, tolong(0)), Current = coalesce(current, tolong(0))\n| project Scope, Desired, Current, UID, Node, SummaryUID, SummaryNode, Pass, At, Summary\n| order by Scope asc, UID asc\n| take 101, Post \"https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query\": context deadline exceeded",
err: <*url.Error | 0x3fb9dd08330>{
Op: "Post",
URL: "https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query",
Err: <context.deadlineExceededError>{},
},
},
restErrMsg: nil,
decoded: nil,
permanent: false,
inner: nil,
},
},
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:112]: expected complete healthy SWIFT router, ignition, KAS, DNS and worker checks for /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-npskt6qgktdr/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster
Unexpected error:
<*fmt.wrapError | 0x3fb9cc54d40>:
VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-npskt6qgktdr/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.003s: query SWIFT checks in https://hcp-dev-us-2.eastus2.kusto.windows.net/ServiceLogs: Op(OpQuery): Kind(KHTTPError): With query: let env = "dev"; let rgn = "centralus"; let cl = "ci01-j4218880-mgmt-1"; let ns = "ocm-arohcpci01-2tjdecblfn866doucbvsck82v6djrsrv-p7h2c2c6w4m5q7f"; let start = datetime(2026-10-21T20:47:23.6514068Z);
let scope = strcat(env, "/", rgn, "/", cl, "/", ns);
let inventory = materialize(kubernetesResourceSnapshots
| where timestamp >= start and environment == env and region == rgn and cluster == cl and namespace == ns
| where objectKind in ("Pod", "Deployment")
| summarize arg_max(timestamp, *) by objectKind, name
| where event != "Delete" and isempty(object.metadata.deletionTimestamp));
let pods = inventory
| where objectKind == "Pod" and object.metadata.labels.app == "private-router" and object.status.phase == "Running"
| project UID = uid, Node = tostring(object.spec.nodeName);
let desired = toscalar(inventory
| where objectKind == "Deployment" and name == "router"
| project tolong(object.spec.replicas));
let current = toscalar(pods | count);
let summaries = containerLogs
| where timestamp >= start and environment == env and region == rgn and cluster == cl
| where container_name == "swift-recorder"
| where log.controller_name == "swift-router-check" and log.field == "summary"
| where tostring(log.pod_namespace) == ns
| extend UID = tostring(log.pod_uid)
| summarize arg_max(timestamp, *) by UID
| project UID, SummaryUID = tostring(log.pod_uid),
SummaryNode = tostring(log.node_name), Pass = tostring(log.pass_id), At = timestamp, Summary = tostring(log.record);
pods
| join kind=leftouter summaries on UID
| extend Scope = scope, Desired = coalesce(desired, tolong(0)), Current = coalesce(current, tolong(0))
| project Scope, Desired, Current, UID, Node, SummaryUID, SummaryNode, Pass, At, Summary
| order by Scope asc, UID asc
| take 101, Post "https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query": context deadline exceeded
{
msg: "VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-npskt6qgktdr/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.003s: query SWIFT checks in https://hcp-dev-us-2.eastus2.kusto.windows.net/ServiceLogs: Op(OpQuery): Kind(KHTTPError): With query: let env = \"dev\"; let rgn = \"centralus\"; let cl = \"ci01-j4218880-mgmt-1\"; let ns = \"ocm-arohcpci01-2tjdecblfn866doucbvsck82v6djrsrv-p7h2c2c6w4m5q7f\"; let start = datetime(2026-10-21T20:47:23.6514068Z);\nlet scope = strcat(env, \"/\", rgn, \"/\", cl, \"/\", ns);\nlet inventory = materialize(kubernetesResourceSnapshots\n| where timestamp >= start and environment == env and region == rgn and cluster == cl and namespace == ns\n| where objectKind in (\"Pod\", \"Deployment\")\n| summarize arg_max(timestamp, *) by objectKind, name\n| where event != \"Delete\" and isempty(object.metadata.deletionTimestamp));\nlet pods = inventory\n| where objectKind == \"Pod\" and object.metadata.labels.app == \"private-router\" and object.status.phase == \"Running\"\n| project UID = uid, Node = tostring(object.spec.nodeName);\nlet desired = toscalar(inventory\n| where objectKind == \"Deployment\" and name == \"router\"\n| project tolong(object.spec.replicas));\nlet current = toscalar(pods | count);\nlet summaries = containerLogs\n| where timestamp >= start and environment == env and region == rgn and cluster == cl\n| where container_name == \"swift-recorder\"\n| where log.controller_name == \"swift-router-check\" and log.field == \"summary\"\n| where tostring(log.pod_namespace) == ns\n| extend UID = tostring(log.pod_uid)\n| summarize arg_max(timestamp, *) by UID\n| project UID, SummaryUID = tostring(log.pod_uid),\n SummaryNode = tostring(log.node_name), Pass = tostring(log.pass_id), At = timestamp, Summary = tostring(log.record);\npods\n| join kind=leftouter summaries on UID\n| extend Scope = scope, Desired = coalesce(desired, tolong(0)), Current = coalesce(current, tolong(0))\n| project Scope, Desired, Current, UID, Node, SummaryUID, SummaryNode, Pass, At, Summary\n| order by Scope asc, UID asc\n| take 101, Post \"https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query\": context deadline exceeded",
err: <*fmt.wrapError | 0x3fb9cc54d20>{
msg: "query SWIFT checks in https://hcp-dev-us-2.eastus2.kusto.windows.net/ServiceLogs: Op(OpQuery): Kind(KHTTPError): With query: let env = \"dev\"; let rgn = \"centralus\"; let cl = \"ci01-j4218880-mgmt-1\"; let ns = \"ocm-arohcpci01-2tjdecblfn866doucbvsck82v6djrsrv-p7h2c2c6w4m5q7f\"; let start = datetime(2026-10-21T20:47:23.6514068Z);\nlet scope = strcat(env, \"/\", rgn, \"/\", cl, \"/\", ns);\nlet inventory = materialize(kubernetesResourceSnapshots\n| where timestamp >= start and environment == env and region == rgn and cluster == cl and namespace == ns\n| where objectKind in (\"Pod\", \"Deployment\")\n| summarize arg_max(timestamp, *) by objectKind, name\n| where event != \"Delete\" and isempty(object.metadata.deletionTimestamp));\nlet pods = inventory\n| where objectKind == \"Pod\" and object.metadata.labels.app == \"private-router\" and object.status.phase == \"Running\"\n| project UID = uid, Node = tostring(object.spec.nodeName);\nlet desired = toscalar(inventory\n| where objectKind == \"Deployment\" and name == \"router\"\n| project tolong(object.spec.replicas));\nlet current = toscalar(pods | count);\nlet summaries = containerLogs\n| where timestamp >= start and environment == env and region == rgn and cluster == cl\n| where container_name == \"swift-recorder\"\n| where log.controller_name == \"swift-router-check\" and log.field == \"summary\"\n| where tostring(log.pod_namespace) == ns\n| extend UID = tostring(log.pod_uid)\n| summarize arg_max(timestamp, *) by UID\n| project UID, SummaryUID = tostring(log.pod_uid),\n SummaryNode = tostring(log.node_name), Pass = tostring(log.pass_id), At = timestamp, Summary = tostring(log.record);\npods\n| join kind=leftouter summaries on UID\n| extend Scope = scope, Desired = coalesce(desired, tolong(0)), Current = coalesce(current, tolong(0))\n| project Scope, Desired, Current, UID, Node, SummaryUID, SummaryNode, Pass, At, Summary\n| order by Scope asc, UID asc\n| take 101, Post \"https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query\": context deadline exceeded",
err: <*errors.Error | 0x3fb9be4c6e0>{
Op: 1,
Kind: 7,
Err: <*fmt.wrapError | 0x3fb9cc54d00>{
msg: "With query: let env = \"dev\"; let rgn = \"centralus\"; let cl = \"ci01-j4218880-mgmt-1\"; let ns = \"ocm-arohcpci01-2tjdecblfn866doucbvsck82v6djrsrv-p7h2c2c6w4m5q7f\"; let start = datetime(2026-10-21T20:47:23.6514068Z);\nlet scope = strcat(env, \"/\", rgn, \"/\", cl, \"/\", ns);\nlet inventory = materialize(kubernetesResourceSnapshots\n| where timestamp >= start and environment == env and region == rgn and cluster == cl and namespace == ns\n| where objectKind in (\"Pod\", \"Deployment\")\n| summarize arg_max(timestamp, *) by objectKind, name\n| where event != \"Delete\" and isempty(object.metadata.deletionTimestamp));\nlet pods = inventory\n| where objectKind == \"Pod\" and object.metadata.labels.app == \"private-router\" and object.status.phase == \"Running\"\n| project UID = uid, Node = tostring(object.spec.nodeName);\nlet desired = toscalar(inventory\n| where objectKind == \"Deployment\" and name == \"router\"\n| project tolong(object.spec.replicas));\nlet current = toscalar(pods | count);\nlet summaries = containerLogs\n| where timestamp >= start and environment == env and region == rgn and cluster == cl\n| where container_name == \"swift-recorder\"\n| where log.controller_name == \"swift-router-check\" and log.field == \"summary\"\n| where tostring(log.pod_namespace) == ns\n| extend UID = tostring(log.pod_uid)\n| summarize arg_max(timestamp, *) by UID\n| project UID, SummaryUID = tostring(log.pod_uid),\n SummaryNode = tostring(log.node_name), Pass = tostring(log.pass_id), At = timestamp, Summary = tostring(log.record);\npods\n| join kind=leftouter summaries on UID\n| extend Scope = scope, Desired = coalesce(desired, tolong(0)), Current = coalesce(current, tolong(0))\n| project Scope, Desired, Current, UID, Node, SummaryUID, SummaryNode, Pass, At, Summary\n| order by Scope asc, UID asc\n| take 101, Post \"https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query\": context deadline exceeded",
err: <*url.Error | 0x3fb9dd08330>{
Op: "Post",
URL: "https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query",
Err: <context.deadlineExceededError>{},
},
},
restErrMsg: nil,
decoded: nil,
permanent: false,
inner: nil,
},
},
}
occurredfail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:112]: expected complete healthy SWIFT router, ignition, KAS, DNS and worker checks for /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-jdsxztvcznpr/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster
Unexpected error:
<*fmt.wrapError | 0x2fae15cf05c0>:
VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-jdsxztvcznpr/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.002s: query SWIFT checks in https://hcp-dev-us-2.eastus2.kusto.windows.net/ServiceLogs: Op(OpQuery): Kind(KHTTPError): With query: let target = "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-jdsxztvcznpr/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster"; let start = datetime(2026-10-21T19:48:49.875795Z);
let desires = cosmosResourceSnapshots
| where resourceID startswith strcat(target, "/")
| where resourceType =~ "microsoft.redhatopenshift/hcpopenshiftclusters/readdesires"
| extend resourceID = tolower(resourceID), ts = tolong(content._ts), sourceCluster = cluster
| summarize arg_max(timestamp, *) by environment, region, sourceCluster, resourceID, ts
| summarize arg_max(ts, *) by environment, region, sourceCluster, resourceID
| where isempty(content.deletionTimestamp)
| extend manifest = content.properties.status.kubeContent
| where manifest.kind == "HostedCluster" and isempty(manifest.metadata.deletionTimestamp)
| extend hostedClusterNamespace = tostring(manifest.metadata.namespace), hostedClusterName = tostring(manifest.metadata.name),
managementCluster = tolower(tostring(content.properties.spec.managementCluster))
| where isnotempty(hostedClusterNamespace) and isnotempty(hostedClusterName) and isnotempty(managementCluster)
| project environment, region, sourceCluster, managementCluster, namespace = strcat(hostedClusterNamespace, "-", hostedClusterName);
let fleet = cosmosResourceSnapshots
| where resourceType =~ "microsoft.redhatopenshift/stamps/managementclusters"
| extend managementCluster = tolower(resourceID), ts = tolong(content._ts), sourceCluster = cluster
| join kind=leftsemi desires on environment, region, sourceCluster, managementCluster
| summarize arg_max(timestamp, *) by environment, region, sourceCluster, managementCluster, ts
| summarize arg_max(ts, *) by environment, region, sourceCluster, managementCluster
| where isempty(content.deletionTimestamp)
| extend clusterName = tostring(split(tostring(content.properties.status.aksResourceID), "/")[-1])
| where isnotempty(clusterName)
| project environment, region, sourceCluster, managementCluster, cluster = clusterName;
let hcp = desires
| join kind=inner fleet on environment, region, sourceCluster, managementCluster
| distinct environment, region, cluster, namespace;
let inventory = materialize(kubernetesResourceSnapshots
| where timestamp >= start and objectKind in ("Pod", "Deployment")
| lookup kind=inner hcp on environment, region, cluster, namespace
| summarize arg_max(timestamp, *) by environment, region, cluster, namespace, objectKind, name
| where event != "Delete" and isempty(object.metadata.deletionTimestamp));
let pods = inventory
| where objectKind == "Pod" and object.metadata.labels.app == "private-router" and object.status.phase == "Running"
| project environment, region, cluster, namespace, UID = uid, Node = tostring(object.spec.nodeName);
let counts = pods | summarize Current = count() by environment, region, cluster, namespace;
let deployments = inventory
| where objectKind == "Deployment" and name == "router"
| project environment, region, cluster, namespace, Desired = tolong(object.spec.replicas);
let summaries = containerLogs
| where timestamp >= start and container_name == "swift-recorder"
| where log.controller_name == "swift-router-check" and log.field == "summary"
| extend namespace = tostring(log.pod_namespace), UID = tostring(log.pod_uid)
| lookup kind=inner hcp on environment, region, cluster, namespace
| summarize arg_max(timestamp, *) by environment, region, cluster, namespace, UID
| project environment, region, cluster, namespace, UID, SummaryUID = tostring(log.pod_uid),
SummaryNode = tostring(log.node_name), Pass = tostring(log.pass_id), At = timestamp, Summary = tostring(log.record);
hcp
| join kind=leftouter deployments on environment, region, cluster, namespace
| join kind=leftouter counts on environment, region, cluster, namespace
| join kind=leftouter pods on environment, region, cluster, namespace
| join kind=leftouter summaries on environment, region, cluster, namespace, UID
| project Scope = strcat(environment, "/", region, "/", cluster, "/", namespace),
Desired = coalesce(Desired, 0), Current = coalesce(Current, 0), UID, Node, SummaryUID, SummaryNode, Pass, At, Summary
| order by Scope asc, UID asc
| take 101, Post "https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query": context deadline exceeded
{
msg: "VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-jdsxztvcznpr/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.002s: query SWIFT checks in https://hcp-dev-us-2.eastus2.kusto.windows.net/ServiceLogs: Op(OpQuery): Kind(KHTTPError): With query: let target = \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-jdsxztvcznpr/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster\"; let start = datetime(2026-10-21T19:48:49.875795Z);\nlet desires = cosmosResourceSnapshots\n| where resourceID startswith strcat(target, \"/\")\n| where resourceType =~ \"microsoft.redhatopenshift/hcpopenshiftclusters/readdesires\"\n| extend resourceID = tolower(resourceID), ts = tolong(content._ts), sourceCluster = cluster\n| summarize arg_max(timestamp, *) by environment, region, sourceCluster, resourceID, ts\n| summarize arg_max(ts, *) by environment, region, sourceCluster, resourceID\n| where isempty(content.deletionTimestamp)\n| extend manifest = content.properties.status.kubeContent\n| where manifest.kind == \"HostedCluster\" and isempty(manifest.metadata.deletionTimestamp)\n| extend hostedClusterNamespace = tostring(manifest.metadata.namespace), hostedClusterName = tostring(manifest.metadata.name),\n managementCluster = tolower(tostring(content.properties.spec.managementCluster))\n| where isnotempty(hostedClusterNamespace) and isnotempty(hostedClusterName) and isnotempty(managementCluster)\n| project environment, region, sourceCluster, managementCluster, namespace = strcat(hostedClusterNamespace, \"-\", hostedClusterName);\nlet fleet = cosmosResourceSnapshots\n| where resourceType =~ \"microsoft.redhatopenshift/stamps/managementclusters\"\n| extend managementCluster = tolower(resourceID), ts = tolong(content._ts), sourceCluster = cluster\n| join kind=leftsemi desires on environment, region, sourceCluster, managementCluster\n| summarize arg_max(timestamp, *) by environment, region, sourceCluster, managementCluster, ts\n| summarize arg_max(ts, *) by environment, region, sourceCluster, managementCluster\n| where isempty(content.deletionTimestamp)\n| extend clusterName = tostring(split(tostring(content.properties.status.aksResourceID), \"/\")[-1])\n| where isnotempty(clusterName)\n| project environment, region, sourceCluster, managementCluster, cluster = clusterName;\nlet hcp = desires\n| join kind=inner fleet on environment, region, sourceCluster, managementCluster\n| distinct environment, region, cluster, namespace;\nlet inventory = materialize(kubernetesResourceSnapshots\n| where timestamp >= start and objectKind in (\"Pod\", \"Deployment\")\n| lookup kind=inner hcp on environment, region, cluster, namespace\n| summarize arg_max(timestamp, *) by environment, region, cluster, namespace, objectKind, name\n| where event != \"Delete\" and isempty(object.metadata.deletionTimestamp));\nlet pods = inventory\n| where objectKind == \"Pod\" and object.metadata.labels.app == \"private-router\" and object.status.phase == \"Running\"\n| project environment, region, cluster, namespace, UID = uid, Node = tostring(object.spec.nodeName);\nlet counts = pods | summarize Current = count() by environment, region, cluster, namespace;\nlet deployments = inventory\n| where objectKind == \"Deployment\" and name == \"router\"\n| project environment, region, cluster, namespace, Desired = tolong(object.spec.replicas);\nlet summaries = containerLogs\n| where timestamp >= start and container_name == \"swift-recorder\"\n| where log.controller_name == \"swift-router-check\" and log.field == \"summary\"\n| extend namespace = tostring(log.pod_namespace), UID = tostring(log.pod_uid)\n| lookup kind=inner hcp on environment, region, cluster, namespace\n| summarize arg_max(timestamp, *) by environment, region, cluster, namespace, UID\n| project environment, region, cluster, namespace, UID, SummaryUID = tostring(log.pod_uid),\n SummaryNode = tostring(log.node_name), Pass = tostring(log.pass_id), At = timestamp, Summary = tostring(log.record);\nhcp\n| join kind=leftouter deployments on environment, region, cluster, namespace\n| join kind=leftouter counts on environment, region, cluster, namespace\n| join kind=leftouter pods on environment, region, cluster, namespace\n| join kind=leftouter summaries on environment, region, cluster, namespace, UID\n| project Scope = strcat(environment, \"/\", region, \"/\", cluster, \"/\", namespace),\n Desired = coalesce(Desired, 0), Current = coalesce(Current, 0), UID, Node, SummaryUID, SummaryNode, Pass, At, Summary\n| order by Scope asc, UID asc\n| take 101, Post \"https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query\": context deadline exceeded",
err: <*fmt.wrapError | 0x2fae15cf05a0>{
msg: "query SWIFT checks in https://hcp-dev-us-2.eastus2.kusto.windows.net/ServiceLogs: Op(OpQuery): Kind(KHTTPError): With query: let target = \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-jdsxztvcznpr/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster\"; let start = datetime(2026-10-21T19:48:49.875795Z);\nlet desires = cosmosResourceSnapshots\n| where resourceID startswith strcat(target, \"/\")\n| where resourceType =~ \"microsoft.redhatopenshift/hcpopenshiftclusters/readdesires\"\n| extend resourceID = tolower(resourceID), ts = tolong(content._ts), sourceCluster = cluster\n| summarize arg_max(timestamp, *) by environment, region, sourceCluster, resourceID, ts\n| summarize arg_max(ts, *) by environment, region, sourceCluster, resourceID\n| where isempty(content.deletionTimestamp)\n| extend manifest = content.properties.status.kubeContent\n| where manifest.kind == \"HostedCluster\" and isempty(manifest.metadata.deletionTimestamp)\n| extend hostedClusterNamespace = tostring(manifest.metadata.namespace), hostedClusterName = tostring(manifest.metadata.name),\n managementCluster = tolower(tostring(content.properties.spec.managementCluster))\n| where isnotempty(hostedClusterNamespace) and isnotempty(hostedClusterName) and isnotempty(managementCluster)\n| project environment, region, sourceCluster, managementCluster, namespace = strcat(hostedClusterNamespace, \"-\", hostedClusterName);\nlet fleet = cosmosResourceSnapshots\n| where resourceType =~ \"microsoft.redhatopenshift/stamps/managementclusters\"\n| extend managementCluster = tolower(resourceID), ts = tolong(content._ts), sourceCluster = cluster\n| join kind=leftsemi desires on environment, region, sourceCluster, managementCluster\n| summarize arg_max(timestamp, *) by environment, region, sourceCluster, managementCluster, ts\n| summarize arg_max(ts, *) by environment, region, sourceCluster, managementCluster\n| where isempty(content.deletionTimestamp)\n| extend clusterName = tostring(split(tostring(content.properties.status.aksResourceID), \"/\")[-1])\n| where isnotempty(clusterName)\n| project environment, region, sourceCluster, managementCluster, cluster = clusterName;\nlet hcp = desires\n| join kind=inner fleet on environment, region, sourceCluster, managementCluster\n| distinct environment, region, cluster, namespace;\nlet inventory = materialize(kubernetesResourceSnapshots\n| where timestamp >= start and objectKind in (\"Pod\", \"Deployment\")\n| lookup kind=inner hcp on environment, region, cluster, namespace\n| summarize arg_max(timestamp, *) by environment, region, cluster, namespace, objectKind, name\n| where event != \"Delete\" and isempty(object.metadata.deletionTimestamp));\nlet pods = inventory\n| where objectKind == \"Pod\" and object.metadata.labels.app == \"private-router\" and object.status.phase == \"Running\"\n| project environment, region, cluster, namespace, UID = uid, Node = tostring(object.spec.nodeName);\nlet counts = pods | summarize Current = count() by environment, region, cluster, namespace;\nlet deployments = inventory\n| where objectKind == \"Deployment\" and name == \"router\"\n| project environment, region, cluster, namespace, Desired = tolong(object.spec.replicas);\nlet summaries = containerLogs\n| where timestamp >= start and container_name == \"swift-recorder\"\n| where log.controller_name == \"swift-router-check\" and log.field == \"summary\"\n| extend namespace = tostring(log.pod_namespace), UID = tostring(log.pod_uid)\n| lookup kind=inner hcp on environment, region, cluster, namespace\n| summarize arg_max(timestamp, *) by environment, region, cluster, namespace, UID\n| project environment, region, cluster, namespace, UID, SummaryUID = tostring(log.pod_uid),\n SummaryNode = tostring(log.node_name), Pass = tostring(log.pass_id), At = timestamp, Summary = tostring(log.record);\nhcp\n| join kind=leftouter deployments on environment, region, cluster, namespace\n| join kind=leftouter counts on environment, region, cluster, namespace\n| join kind=leftouter pods on environment, region, cluster, namespace\n| join kind=leftouter summaries on environment, region, cluster, namespace, UID\n| project Scope = strcat(environment, \"/\", region, \"/\", cluster, \"/\", namespace),\n Desired = coalesce(Desired, 0), Current = coalesce(Current, 0), UID, Node, SummaryUID, SummaryNode, Pass, At, Summary\n| order by Scope asc, UID asc\n| take 101, Post \"https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query\": context deadline exceeded",
err: <*errors.Error | 0x2fae15532370>{
Op: 1,
Kind: 7,
Err: <*fmt.wrapError | 0x2fae15cf0580>{
msg: "With query: let target = \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-jdsxztvcznpr/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster\"; let start = datetime(2026-10-21T19:48:49.875795Z);\nlet desires = cosmosResourceSnapshots\n| where resourceID startswith strcat(target, \"/\")\n| where resourceType =~ \"microsoft.redhatopenshift/hcpopenshiftclusters/readdesires\"\n| extend resourceID = tolower(resourceID), ts = tolong(content._ts), sourceCluster = cluster\n| summarize arg_max(timestamp, *) by environment, region, sourceCluster, resourceID, ts\n| summarize arg_max(ts, *) by environment, region, sourceCluster, resourceID\n| where isempty(content.deletionTimestamp)\n| extend manifest = content.properties.status.kubeContent\n| where manifest.kind == \"HostedCluster\" and isempty(manifest.metadata.deletionTimestamp)\n| extend hostedClusterNamespace = tostring(manifest.metadata.namespace), hostedClusterName = tostring(manifest.metadata.name),\n managementCluster = tolower(tostring(content.properties.spec.managementCluster))\n| where isnotempty(hostedClusterNamespace) and isnotempty(hostedClusterName) and isnotempty(managementCluster)\n| project environment, region, sourceCluster, managementCluster, namespace = strcat(hostedClusterNamespace, \"-\", hostedClusterName);\nlet fleet = cosmosResourceSnapshots\n| where resourceType =~ \"microsoft.redhatopenshift/stamps/managementclusters\"\n| extend managementCluster = tolower(resourceID), ts = tolong(content._ts), sourceCluster = cluster\n| join kind=leftsemi desires on environment, region, sourceCluster, managementCluster\n| summarize arg_max(timestamp, *) by environment, region, sourceCluster, managementCluster, ts\n| summarize arg_max(ts, *) by environment, region, sourceCluster, managementCluster\n| where isempty(content.deletionTimestamp)\n| extend clusterName = tostring(split(tostring(content.properties.status.aksResourceID), \"/\")[-1])\n| where isnotempty(clusterName)\n| project environment, region, sourceCluster, managementCluster, cluster = clusterName;\nlet hcp = desires\n| join kind=inner fleet on environment, region, sourceCluster, managementCluster\n| distinct environment, region, cluster, namespace;\nlet inventory = materialize(kubernetesResourceSnapshots\n| where timestamp >= start and objectKind in (\"Pod\", \"Deployment\")\n| lookup kind=inner hcp on environment, region, cluster, namespace\n| summarize arg_max(timestamp, *) by environment, region, cluster, namespace, objectKind, name\n| where event != \"Delete\" and isempty(object.metadata.deletionTimestamp));\nlet pods = inventory\n| where objectKind == \"Pod\" and object.metadata.labels.app == \"private-router\" and object.status.phase == \"Running\"\n| project environment, region, cluster, namespace, UID = uid, Node = tostring(object.spec.nodeName);\nlet counts = pods | summarize Current = count() by environment, region, cluster, namespace;\nlet deployments = inventory\n| where objectKind == \"Deployment\" and name == \"router\"\n| project environment, region, cluster, namespace, Desired = tolong(object.spec.replicas);\nlet summaries = containerLogs\n| where timestamp >= start and container_name == \"swift-recorder\"\n| where log.controller_name == \"swift-router-check\" and log.field == \"summary\"\n| extend namespace = tostring(log.pod_namespace), UID = tostring(log.pod_uid)\n| lookup kind=inner hcp on environment, region, cluster, namespace\n| summarize arg_max(timestamp, *) by environment, region, cluster, namespace, UID\n| project environment, region, cluster, namespace, UID, SummaryUID = tostring(log.pod_uid),\n SummaryNode = tostring(log.node_name), Pass = tostring(log.pass_id), At = timestamp, Summary = tostring(log.record);\nhcp\n| join kind=leftouter deployments on environment, region, cluster, namespace\n| join kind=leftouter counts on environment, region, cluster, namespace\n| join kind=leftouter pods on environment, region, cluster, namespace\n| join kind=leftouter summaries on environment, region, cluster, namespace, UID\n| project Scope = strcat(environment, \"/\", region, \"/\", cluster, \"/\", namespace),\n Desired = coalesce(Desired, 0), Current = coalesce(Current, 0), UID, Node, SummaryUID, SummaryNode, Pass, At, Summary\n| order by Scope asc, UID asc\n| take 101, Post \"https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query\": context deadline exceeded",
err: <*url.Error | 0x2fae16f952c0>{
Op: "Post",
URL: "https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query",
Err: <context.deadlineExceededError>{},
},
},
restErrMsg: nil,
decoded: nil,
permanent: false,
inner: nil,
},
},
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:112]: expected complete healthy SWIFT router, ignition, KAS, DNS and worker checks for /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-jdsxztvcznpr/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster
Unexpected error:
<*fmt.wrapError | 0x2fae15cf05c0>:
VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-jdsxztvcznpr/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.002s: query SWIFT checks in https://hcp-dev-us-2.eastus2.kusto.windows.net/ServiceLogs: Op(OpQuery): Kind(KHTTPError): With query: let target = "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-jdsxztvcznpr/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster"; let start = datetime(2026-10-21T19:48:49.875795Z);
let desires = cosmosResourceSnapshots
| where resourceID startswith strcat(target, "/")
| where resourceType =~ "microsoft.redhatopenshift/hcpopenshiftclusters/readdesires"
| extend resourceID = tolower(resourceID), ts = tolong(content._ts), sourceCluster = cluster
| summarize arg_max(timestamp, *) by environment, region, sourceCluster, resourceID, ts
| summarize arg_max(ts, *) by environment, region, sourceCluster, resourceID
| where isempty(content.deletionTimestamp)
| extend manifest = content.properties.status.kubeContent
| where manifest.kind == "HostedCluster" and isempty(manifest.metadata.deletionTimestamp)
| extend hostedClusterNamespace = tostring(manifest.metadata.namespace), hostedClusterName = tostring(manifest.metadata.name),
managementCluster = tolower(tostring(content.properties.spec.managementCluster))
| where isnotempty(hostedClusterNamespace) and isnotempty(hostedClusterName) and isnotempty(managementCluster)
| project environment, region, sourceCluster, managementCluster, namespace = strcat(hostedClusterNamespace, "-", hostedClusterName);
let fleet = cosmosResourceSnapshots
| where resourceType =~ "microsoft.redhatopenshift/stamps/managementclusters"
| extend managementCluster = tolower(resourceID), ts = tolong(content._ts), sourceCluster = cluster
| join kind=leftsemi desires on environment, region, sourceCluster, managementCluster
| summarize arg_max(timestamp, *) by environment, region, sourceCluster, managementCluster, ts
| summarize arg_max(ts, *) by environment, region, sourceCluster, managementCluster
| where isempty(content.deletionTimestamp)
| extend clusterName = tostring(split(tostring(content.properties.status.aksResourceID), "/")[-1])
| where isnotempty(clusterName)
| project environment, region, sourceCluster, managementCluster, cluster = clusterName;
let hcp = desires
| join kind=inner fleet on environment, region, sourceCluster, managementCluster
| distinct environment, region, cluster, namespace;
let inventory = materialize(kubernetesResourceSnapshots
| where timestamp >= start and objectKind in ("Pod", "Deployment")
| lookup kind=inner hcp on environment, region, cluster, namespace
| summarize arg_max(timestamp, *) by environment, region, cluster, namespace, objectKind, name
| where event != "Delete" and isempty(object.metadata.deletionTimestamp));
let pods = inventory
| where objectKind == "Pod" and object.metadata.labels.app == "private-router" and object.status.phase == "Running"
| project environment, region, cluster, namespace, UID = uid, Node = tostring(object.spec.nodeName);
let counts = pods | summarize Current = count() by environment, region, cluster, namespace;
let deployments = inventory
| where objectKind == "Deployment" and name == "router"
| project environment, region, cluster, namespace, Desired = tolong(object.spec.replicas);
let summaries = containerLogs
| where timestamp >= start and container_name == "swift-recorder"
| where log.controller_name == "swift-router-check" and log.field == "summary"
| extend namespace = tostring(log.pod_namespace), UID = tostring(log.pod_uid)
| lookup kind=inner hcp on environment, region, cluster, namespace
| summarize arg_max(timestamp, *) by environment, region, cluster, namespace, UID
| project environment, region, cluster, namespace, UID, SummaryUID = tostring(log.pod_uid),
SummaryNode = tostring(log.node_name), Pass = tostring(log.pass_id), At = timestamp, Summary = tostring(log.record);
hcp
| join kind=leftouter deployments on environment, region, cluster, namespace
| join kind=leftouter counts on environment, region, cluster, namespace
| join kind=leftouter pods on environment, region, cluster, namespace
| join kind=leftouter summaries on environment, region, cluster, namespace, UID
| project Scope = strcat(environment, "/", region, "/", cluster, "/", namespace),
Desired = coalesce(Desired, 0), Current = coalesce(Current, 0), UID, Node, SummaryUID, SummaryNode, Pass, At, Summary
| order by Scope asc, UID asc
| take 101, Post "https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query": context deadline exceeded
{
msg: "VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-jdsxztvcznpr/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.002s: query SWIFT checks in https://hcp-dev-us-2.eastus2.kusto.windows.net/ServiceLogs: Op(OpQuery): Kind(KHTTPError): With query: let target = \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-jdsxztvcznpr/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster\"; let start = datetime(2026-10-21T19:48:49.875795Z);\nlet desires = cosmosResourceSnapshots\n| where resourceID startswith strcat(target, \"/\")\n| where resourceType =~ \"microsoft.redhatopenshift/hcpopenshiftclusters/readdesires\"\n| extend resourceID = tolower(resourceID), ts = tolong(content._ts), sourceCluster = cluster\n| summarize arg_max(timestamp, *) by environment, region, sourceCluster, resourceID, ts\n| summarize arg_max(ts, *) by environment, region, sourceCluster, resourceID\n| where isempty(content.deletionTimestamp)\n| extend manifest = content.properties.status.kubeContent\n| where manifest.kind == \"HostedCluster\" and isempty(manifest.metadata.deletionTimestamp)\n| extend hostedClusterNamespace = tostring(manifest.metadata.namespace), hostedClusterName = tostring(manifest.metadata.name),\n managementCluster = tolower(tostring(content.properties.spec.managementCluster))\n| where isnotempty(hostedClusterNamespace) and isnotempty(hostedClusterName) and isnotempty(managementCluster)\n| project environment, region, sourceCluster, managementCluster, namespace = strcat(hostedClusterNamespace, \"-\", hostedClusterName);\nlet fleet = cosmosResourceSnapshots\n| where resourceType =~ \"microsoft.redhatopenshift/stamps/managementclusters\"\n| extend managementCluster = tolower(resourceID), ts = tolong(content._ts), sourceCluster = cluster\n| join kind=leftsemi desires on environment, region, sourceCluster, managementCluster\n| summarize arg_max(timestamp, *) by environment, region, sourceCluster, managementCluster, ts\n| summarize arg_max(ts, *) by environment, region, sourceCluster, managementCluster\n| where isempty(content.deletionTimestamp)\n| extend clusterName = tostring(split(tostring(content.properties.status.aksResourceID), \"/\")[-1])\n| where isnotempty(clusterName)\n| project environment, region, sourceCluster, managementCluster, cluster = clusterName;\nlet hcp = desires\n| join kind=inner fleet on environment, region, sourceCluster, managementCluster\n| distinct environment, region, cluster, namespace;\nlet inventory = materialize(kubernetesResourceSnapshots\n| where timestamp >= start and objectKind in (\"Pod\", \"Deployment\")\n| lookup kind=inner hcp on environment, region, cluster, namespace\n| summarize arg_max(timestamp, *) by environment, region, cluster, namespace, objectKind, name\n| where event != \"Delete\" and isempty(object.metadata.deletionTimestamp));\nlet pods = inventory\n| where objectKind == \"Pod\" and object.metadata.labels.app == \"private-router\" and object.status.phase == \"Running\"\n| project environment, region, cluster, namespace, UID = uid, Node = tostring(object.spec.nodeName);\nlet counts = pods | summarize Current = count() by environment, region, cluster, namespace;\nlet deployments = inventory\n| where objectKind == \"Deployment\" and name == \"router\"\n| project environment, region, cluster, namespace, Desired = tolong(object.spec.replicas);\nlet summaries = containerLogs\n| where timestamp >= start and container_name == \"swift-recorder\"\n| where log.controller_name == \"swift-router-check\" and log.field == \"summary\"\n| extend namespace = tostring(log.pod_namespace), UID = tostring(log.pod_uid)\n| lookup kind=inner hcp on environment, region, cluster, namespace\n| summarize arg_max(timestamp, *) by environment, region, cluster, namespace, UID\n| project environment, region, cluster, namespace, UID, SummaryUID = tostring(log.pod_uid),\n SummaryNode = tostring(log.node_name), Pass = tostring(log.pass_id), At = timestamp, Summary = tostring(log.record);\nhcp\n| join kind=leftouter deployments on environment, region, cluster, namespace\n| join kind=leftouter counts on environment, region, cluster, namespace\n| join kind=leftouter pods on environment, region, cluster, namespace\n| join kind=leftouter summaries on environment, region, cluster, namespace, UID\n| project Scope = strcat(environment, \"/\", region, \"/\", cluster, \"/\", namespace),\n Desired = coalesce(Desired, 0), Current = coalesce(Current, 0), UID, Node, SummaryUID, SummaryNode, Pass, At, Summary\n| order by Scope asc, UID asc\n| take 101, Post \"https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query\": context deadline exceeded",
err: <*fmt.wrapError | 0x2fae15cf05a0>{
msg: "query SWIFT checks in https://hcp-dev-us-2.eastus2.kusto.windows.net/ServiceLogs: Op(OpQuery): Kind(KHTTPError): With query: let target = \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-jdsxztvcznpr/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster\"; let start = datetime(2026-10-21T19:48:49.875795Z);\nlet desires = cosmosResourceSnapshots\n| where resourceID startswith strcat(target, \"/\")\n| where resourceType =~ \"microsoft.redhatopenshift/hcpopenshiftclusters/readdesires\"\n| extend resourceID = tolower(resourceID), ts = tolong(content._ts), sourceCluster = cluster\n| summarize arg_max(timestamp, *) by environment, region, sourceCluster, resourceID, ts\n| summarize arg_max(ts, *) by environment, region, sourceCluster, resourceID\n| where isempty(content.deletionTimestamp)\n| extend manifest = content.properties.status.kubeContent\n| where manifest.kind == \"HostedCluster\" and isempty(manifest.metadata.deletionTimestamp)\n| extend hostedClusterNamespace = tostring(manifest.metadata.namespace), hostedClusterName = tostring(manifest.metadata.name),\n managementCluster = tolower(tostring(content.properties.spec.managementCluster))\n| where isnotempty(hostedClusterNamespace) and isnotempty(hostedClusterName) and isnotempty(managementCluster)\n| project environment, region, sourceCluster, managementCluster, namespace = strcat(hostedClusterNamespace, \"-\", hostedClusterName);\nlet fleet = cosmosResourceSnapshots\n| where resourceType =~ \"microsoft.redhatopenshift/stamps/managementclusters\"\n| extend managementCluster = tolower(resourceID), ts = tolong(content._ts), sourceCluster = cluster\n| join kind=leftsemi desires on environment, region, sourceCluster, managementCluster\n| summarize arg_max(timestamp, *) by environment, region, sourceCluster, managementCluster, ts\n| summarize arg_max(ts, *) by environment, region, sourceCluster, managementCluster\n| where isempty(content.deletionTimestamp)\n| extend clusterName = tostring(split(tostring(content.properties.status.aksResourceID), \"/\")[-1])\n| where isnotempty(clusterName)\n| project environment, region, sourceCluster, managementCluster, cluster = clusterName;\nlet hcp = desires\n| join kind=inner fleet on environment, region, sourceCluster, managementCluster\n| distinct environment, region, cluster, namespace;\nlet inventory = materialize(kubernetesResourceSnapshots\n| where timestamp >= start and objectKind in (\"Pod\", \"Deployment\")\n| lookup kind=inner hcp on environment, region, cluster, namespace\n| summarize arg_max(timestamp, *) by environment, region, cluster, namespace, objectKind, name\n| where event != \"Delete\" and isempty(object.metadata.deletionTimestamp));\nlet pods = inventory\n| where objectKind == \"Pod\" and object.metadata.labels.app == \"private-router\" and object.status.phase == \"Running\"\n| project environment, region, cluster, namespace, UID = uid, Node = tostring(object.spec.nodeName);\nlet counts = pods | summarize Current = count() by environment, region, cluster, namespace;\nlet deployments = inventory\n| where objectKind == \"Deployment\" and name == \"router\"\n| project environment, region, cluster, namespace, Desired = tolong(object.spec.replicas);\nlet summaries = containerLogs\n| where timestamp >= start and container_name == \"swift-recorder\"\n| where log.controller_name == \"swift-router-check\" and log.field == \"summary\"\n| extend namespace = tostring(log.pod_namespace), UID = tostring(log.pod_uid)\n| lookup kind=inner hcp on environment, region, cluster, namespace\n| summarize arg_max(timestamp, *) by environment, region, cluster, namespace, UID\n| project environment, region, cluster, namespace, UID, SummaryUID = tostring(log.pod_uid),\n SummaryNode = tostring(log.node_name), Pass = tostring(log.pass_id), At = timestamp, Summary = tostring(log.record);\nhcp\n| join kind=leftouter deployments on environment, region, cluster, namespace\n| join kind=leftouter counts on environment, region, cluster, namespace\n| join kind=leftouter pods on environment, region, cluster, namespace\n| join kind=leftouter summaries on environment, region, cluster, namespace, UID\n| project Scope = strcat(environment, \"/\", region, \"/\", cluster, \"/\", namespace),\n Desired = coalesce(Desired, 0), Current = coalesce(Current, 0), UID, Node, SummaryUID, SummaryNode, Pass, At, Summary\n| order by Scope asc, UID asc\n| take 101, Post \"https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query\": context deadline exceeded",
err: <*errors.Error | 0x2fae15532370>{
Op: 1,
Kind: 7,
Err: <*fmt.wrapError | 0x2fae15cf0580>{
msg: "With query: let target = \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-jdsxztvcznpr/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster\"; let start = datetime(2026-10-21T19:48:49.875795Z);\nlet desires = cosmosResourceSnapshots\n| where resourceID startswith strcat(target, \"/\")\n| where resourceType =~ \"microsoft.redhatopenshift/hcpopenshiftclusters/readdesires\"\n| extend resourceID = tolower(resourceID), ts = tolong(content._ts), sourceCluster = cluster\n| summarize arg_max(timestamp, *) by environment, region, sourceCluster, resourceID, ts\n| summarize arg_max(ts, *) by environment, region, sourceCluster, resourceID\n| where isempty(content.deletionTimestamp)\n| extend manifest = content.properties.status.kubeContent\n| where manifest.kind == \"HostedCluster\" and isempty(manifest.metadata.deletionTimestamp)\n| extend hostedClusterNamespace = tostring(manifest.metadata.namespace), hostedClusterName = tostring(manifest.metadata.name),\n managementCluster = tolower(tostring(content.properties.spec.managementCluster))\n| where isnotempty(hostedClusterNamespace) and isnotempty(hostedClusterName) and isnotempty(managementCluster)\n| project environment, region, sourceCluster, managementCluster, namespace = strcat(hostedClusterNamespace, \"-\", hostedClusterName);\nlet fleet = cosmosResourceSnapshots\n| where resourceType =~ \"microsoft.redhatopenshift/stamps/managementclusters\"\n| extend managementCluster = tolower(resourceID), ts = tolong(content._ts), sourceCluster = cluster\n| join kind=leftsemi desires on environment, region, sourceCluster, managementCluster\n| summarize arg_max(timestamp, *) by environment, region, sourceCluster, managementCluster, ts\n| summarize arg_max(ts, *) by environment, region, sourceCluster, managementCluster\n| where isempty(content.deletionTimestamp)\n| extend clusterName = tostring(split(tostring(content.properties.status.aksResourceID), \"/\")[-1])\n| where isnotempty(clusterName)\n| project environment, region, sourceCluster, managementCluster, cluster = clusterName;\nlet hcp = desires\n| join kind=inner fleet on environment, region, sourceCluster, managementCluster\n| distinct environment, region, cluster, namespace;\nlet inventory = materialize(kubernetesResourceSnapshots\n| where timestamp >= start and objectKind in (\"Pod\", \"Deployment\")\n| lookup kind=inner hcp on environment, region, cluster, namespace\n| summarize arg_max(timestamp, *) by environment, region, cluster, namespace, objectKind, name\n| where event != \"Delete\" and isempty(object.metadata.deletionTimestamp));\nlet pods = inventory\n| where objectKind == \"Pod\" and object.metadata.labels.app == \"private-router\" and object.status.phase == \"Running\"\n| project environment, region, cluster, namespace, UID = uid, Node = tostring(object.spec.nodeName);\nlet counts = pods | summarize Current = count() by environment, region, cluster, namespace;\nlet deployments = inventory\n| where objectKind == \"Deployment\" and name == \"router\"\n| project environment, region, cluster, namespace, Desired = tolong(object.spec.replicas);\nlet summaries = containerLogs\n| where timestamp >= start and container_name == \"swift-recorder\"\n| where log.controller_name == \"swift-router-check\" and log.field == \"summary\"\n| extend namespace = tostring(log.pod_namespace), UID = tostring(log.pod_uid)\n| lookup kind=inner hcp on environment, region, cluster, namespace\n| summarize arg_max(timestamp, *) by environment, region, cluster, namespace, UID\n| project environment, region, cluster, namespace, UID, SummaryUID = tostring(log.pod_uid),\n SummaryNode = tostring(log.node_name), Pass = tostring(log.pass_id), At = timestamp, Summary = tostring(log.record);\nhcp\n| join kind=leftouter deployments on environment, region, cluster, namespace\n| join kind=leftouter counts on environment, region, cluster, namespace\n| join kind=leftouter pods on environment, region, cluster, namespace\n| join kind=leftouter summaries on environment, region, cluster, namespace, UID\n| project Scope = strcat(environment, \"/\", region, \"/\", cluster, \"/\", namespace),\n Desired = coalesce(Desired, 0), Current = coalesce(Current, 0), UID, Node, SummaryUID, SummaryNode, Pass, At, Summary\n| order by Scope asc, UID asc\n| take 101, Post \"https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query\": context deadline exceeded",
err: <*url.Error | 0x2fae16f952c0>{
Op: "Post",
URL: "https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query",
Err: <context.deadlineExceededError>{},
},
},
restErrMsg: nil,
decoded: nil,
permanent: false,
inner: nil,
},
},
}
occurredContributing tests (1)
Affected runs (71) | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
ERROR CODE: DeadlineExceeded; detail message node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hype...full failure pattern: ERROR CODE: DeadlineExceeded; detail message node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is <count>, want <count>; node pool condition AllNodesHealthy is False: <count> of <count> machines are not healthy Machine <node>: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.Compute/virtualMachines/<node> to exist; node pool condition AllMachinesReady is False: <count> of <count> machines are not ready Machine <node>: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.Compute/virtualMachines/<node> to exist; provider Microsoft.RedHatOpenShift Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 11 day(s) | e2e | 38 | 3.82%38 of 995 job runs affected | IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 11 day(s) | Show trend detailsOct 12: 2 · Oct 13: 3 · Oct 14: 2 · Oct 15: 7 · Oct 16: 2 · Oct 17: 0 · Oct 18: 0 · Oct 19: 3 · Oct 20: 7 · Oct 21: 3 · Oct 22: 5 · Oct 23: 2 · Oct 24: 2 · Oct 25: 0 | PROD, STG | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (3)fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:165]: failed to create node pool npupgrade-4-20 with version 4.20.43
Unexpected error:
<*fmt.wrapError | 0x1f6cce913900>:
failed to create NodePool npupgrade-4-20: failed waiting for nodepool="npupgrade-4-20" for cluster "np-version-upgrade-cluster-b559ft" in resourcegroup="rg-np-version-upgrade-b559ft-j4qdwfs6szxx" to finish creating: GET https://rp.j3761152.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/489dec27-e4f4-49eb-a330-6052b204ad8e
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/489dec27-e4f4-49eb-a330-6052b204ad8e",
"name": "489dec27-e4f4-49eb-a330-6052b204ad8e",
"status": "Failed",
"startTime": "2026-10-24T05:53:26.952860579Z",
"endTime": "2026-10-24T06:12:28.455904126Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine p8m7i5l8y6g1n1v-npupgrade-4-20-kdgcn-qg9p4: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-version-upgrade-b559ft-j4qdwfs6szxx-np-upgrade-b5-5fe002ad/providers/Microsoft.Compute/virtualMachines/p8m7i5l8y6g1n1v-npupgrade-4-20-kdgcn-qg9p4 to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine p8m7i5l8y6g1n1v-npupgrade-4-20-kdgcn-qg9p4: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-version-upgrade-b559ft-j4qdwfs6szxx-np-upgrade-b5-5fe002ad/providers/Microsoft.Compute/virtualMachines/p8m7i5l8y6g1n1v-npupgrade-4-20-kdgcn-qg9p4 to exist\n"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://rp.j3761152.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/489dec27-e4f4-49eb-a330-6052b204ad8e
correlation ID: d5af14b9-8e48-4974-af98-abb8c215a6d4
client request ID: 1283d9b2-c265-4f0a-97ea-52c82d8cd670
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:165]: failed to create node pool npupgrade-4-20 with version 4.20.43
Unexpected error:
<*fmt.wrapError | 0x1f6cce913900>:
failed to create NodePool npupgrade-4-20: failed waiting for nodepool="npupgrade-4-20" for cluster "np-version-upgrade-cluster-b559ft" in resourcegroup="rg-np-version-upgrade-b559ft-j4qdwfs6szxx" to finish creating: GET https://rp.j3761152.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/489dec27-e4f4-49eb-a330-6052b204ad8e
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/489dec27-e4f4-49eb-a330-6052b204ad8e",
"name": "489dec27-e4f4-49eb-a330-6052b204ad8e",
"status": "Failed",
"startTime": "2026-10-24T05:53:26.952860579Z",
"endTime": "2026-10-24T06:12:28.455904126Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine p8m7i5l8y6g1n1v-npupgrade-4-20-kdgcn-qg9p4: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-version-upgrade-b559ft-j4qdwfs6szxx-np-upgrade-b5-5fe002ad/providers/Microsoft.Compute/virtualMachines/p8m7i5l8y6g1n1v-npupgrade-4-20-kdgcn-qg9p4 to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine p8m7i5l8y6g1n1v-npupgrade-4-20-kdgcn-qg9p4: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-version-upgrade-b559ft-j4qdwfs6szxx-np-upgrade-b5-5fe002ad/providers/Microsoft.Compute/virtualMachines/p8m7i5l8y6g1n1v-npupgrade-4-20-kdgcn-qg9p4 to exist\n"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://rp.j3761152.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/489dec27-e4f4-49eb-a330-6052b204ad8e
correlation ID: d5af14b9-8e48-4974-af98-abb8c215a6d4
client request ID: 1283d9b2-c265-4f0a-97ea-52c82d8cd670
occurredfail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:321]: failed to create nodepool npnoedge-4-20 at version 4.20.42
Unexpected error:
<*fmt.wrapError | 0x298d72d44040>:
failed to create NodePool npnoedge-4-20: failed waiting for nodepool="npnoedge-4-20" for cluster "np-noedge-gw598c" in resourcegroup="rg-np-noedge-gw598c-2zsqpwzqbmb8" to finish creating: GET https://rp.j9730176.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/ea64491d-d0fd-400e-88eb-8c8bd1cb256b
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/ea64491d-d0fd-400e-88eb-8c8bd1cb256b",
"name": "ea64491d-d0fd-400e-88eb-8c8bd1cb256b",
"status": "Failed",
"startTime": "2026-10-24T05:54:16.071407678Z",
"endTime": "2026-10-24T06:13:22.793053391Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine u8s0r4g9w5s8j0u-npnoedge-4-20-q9dz8-xgzd2: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-noedge-gw598c-2zsqpwzqbmb8-np-ne-gw598c--managed/providers/Microsoft.Compute/virtualMachines/u8s0r4g9w5s8j0u-npnoedge-4-20-q9dz8-xgzd2 to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine u8s0r4g9w5s8j0u-npnoedge-4-20-q9dz8-xgzd2: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-noedge-gw598c-2zsqpwzqbmb8-np-ne-gw598c--managed/providers/Microsoft.Compute/virtualMachines/u8s0r4g9w5s8j0u-npnoedge-4-20-q9dz8-xgzd2 to exist\n"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://rp.j9730176.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/ea64491d-d0fd-400e-88eb-8c8bd1cb256b
correlation ID: 149b615d-592f-486c-b354-94d297e4fa6d
client request ID: eb91eb91-fd3e-493e-b389-b15c2dca94ef
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:321]: failed to create nodepool npnoedge-4-20 at version 4.20.42
Unexpected error:
<*fmt.wrapError | 0x298d72d44040>:
failed to create NodePool npnoedge-4-20: failed waiting for nodepool="npnoedge-4-20" for cluster "np-noedge-gw598c" in resourcegroup="rg-np-noedge-gw598c-2zsqpwzqbmb8" to finish creating: GET https://rp.j9730176.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/ea64491d-d0fd-400e-88eb-8c8bd1cb256b
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/ea64491d-d0fd-400e-88eb-8c8bd1cb256b",
"name": "ea64491d-d0fd-400e-88eb-8c8bd1cb256b",
"status": "Failed",
"startTime": "2026-10-24T05:54:16.071407678Z",
"endTime": "2026-10-24T06:13:22.793053391Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine u8s0r4g9w5s8j0u-npnoedge-4-20-q9dz8-xgzd2: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-noedge-gw598c-2zsqpwzqbmb8-np-ne-gw598c--managed/providers/Microsoft.Compute/virtualMachines/u8s0r4g9w5s8j0u-npnoedge-4-20-q9dz8-xgzd2 to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine u8s0r4g9w5s8j0u-npnoedge-4-20-q9dz8-xgzd2: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-noedge-gw598c-2zsqpwzqbmb8-np-ne-gw598c--managed/providers/Microsoft.Compute/virtualMachines/u8s0r4g9w5s8j0u-npnoedge-4-20-q9dz8-xgzd2 to exist\n"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://rp.j9730176.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/ea64491d-d0fd-400e-88eb-8c8bd1cb256b
correlation ID: 149b615d-592f-486c-b354-94d297e4fa6d
client request ID: eb91eb91-fd3e-493e-b389-b15c2dca94ef
occurredfail [github.com/Azure/ARO-HCP/test/e2e/cluster_pullsecret.go:178]: failed to create node pool np-1 for pull secret cluster
Unexpected error:
<*fmt.wrapError | 0x1d7f216af3c0>:
failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "pullsecret-hcp-cluster" in resourcegroup="pullsecret-test-sw22pzbqp5qg" to finish creating: GET https://rp.j4094592.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/57d3456e-a011-478a-861a-14e4fe67f873
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/57d3456e-a011-478a-861a-14e4fe67f873",
"name": "57d3456e-a011-478a-861a-14e4fe67f873",
"status": "Failed",
"startTime": "2026-10-23T09:12:07.672503322Z",
"endTime": "2026-10-23T09:31:10.853698527Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine j9s2r8n0s7j5m0g-np-1-s7snv-swvmp: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/pullsecret-test-sw22pzbqp5qg--managed/providers/Microsoft.Compute/virtualMachines/j9s2r8n0s7j5m0g-np-1-s7snv-swvmp to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine j9s2r8n0s7j5m0g-np-1-s7snv-swvmp: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/pullsecret-test-sw22pzbqp5qg--managed/providers/Microsoft.Compute/virtualMachines/j9s2r8n0s7j5m0g-np-1-s7snv-swvmp to exist\n"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://rp.j4094592.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/57d3456e-a011-478a-861a-14e4fe67f873
correlation ID: 62c61aed-7617-418e-886e-be6374e7f22a
client request ID: eb9543dc-eb4b-4b68-a493-4efa1c23921e
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_pullsecret.go:178]: failed to create node pool np-1 for pull secret cluster
Unexpected error:
<*fmt.wrapError | 0x1d7f216af3c0>:
failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "pullsecret-hcp-cluster" in resourcegroup="pullsecret-test-sw22pzbqp5qg" to finish creating: GET https://rp.j4094592.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/57d3456e-a011-478a-861a-14e4fe67f873
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/57d3456e-a011-478a-861a-14e4fe67f873",
"name": "57d3456e-a011-478a-861a-14e4fe67f873",
"status": "Failed",
"startTime": "2026-10-23T09:12:07.672503322Z",
"endTime": "2026-10-23T09:31:10.853698527Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine j9s2r8n0s7j5m0g-np-1-s7snv-swvmp: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/pullsecret-test-sw22pzbqp5qg--managed/providers/Microsoft.Compute/virtualMachines/j9s2r8n0s7j5m0g-np-1-s7snv-swvmp to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine j9s2r8n0s7j5m0g-np-1-s7snv-swvmp: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/pullsecret-test-sw22pzbqp5qg--managed/providers/Microsoft.Compute/virtualMachines/j9s2r8n0s7j5m0g-np-1-s7snv-swvmp to exist\n"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://rp.j4094592.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/57d3456e-a011-478a-861a-14e4fe67f873
correlation ID: 62c61aed-7617-418e-886e-be6374e7f22a
client request ID: eb9543dc-eb4b-4b68-a493-4efa1c23921e
occurredContributing tests (14)
Affected runs (38) | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
registry.redhat.io credentials not found in pull-secret filefull failure pattern: registry.redhat.io credentials not found in pull-secret file Signal: IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | e2e | 21 | 2.11%21 of 995 job runs affected | IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | Show trend detailsOct 12: 0 · Oct 13: 0 · Oct 14: 0 · Oct 15: 0 · Oct 16: 0 · Oct 17: 0 · Oct 18: 0 · Oct 19: 21 · Oct 20: 0 · Oct 21: 0 · Oct 22: 0 · Oct 23: 0 · Oct 24: 0 · Oct 25: 0 | none | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/e2e/cluster_pullsecret.go:197]: registry.redhat.io credentials not found in pull-secret file
Expected
<bool>: false
to be true
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_pullsecret.go:197]: registry.redhat.io credentials not found in pull-secret file
Expected
<bool>: false
to be trueContributing tests (1)
Affected runs (21) | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
alert [svc] KubePodCrashLooping firedfull failure pattern: alert [svc] KubePodCrashLooping fired Signal: IndeterminateSignal: Indeterminate — no prior history; active 5 day(s) — New failure pattern — no prior history | alert | 17 | 1.71%17 of 995 job runs affected | IndeterminateSignal: Indeterminate — no prior history; active 5 day(s) — New failure pattern — no prior history | Show trend detailsOct 12: 0 · Oct 13: 0 · Oct 14: 0 · Oct 15: 0 · Oct 16: 0 · Oct 17: 0 · Oct 18: 0 · Oct 19: 0 · Oct 20: 0 · Oct 21: 2 · Oct 22: 6 · Oct 23: 3 · Oct 24: 2 · Oct 25: 4 | none | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (3)alert fired 4 time(s) Firing 1: State: Fired (not resolved) Started: 2026-10-25T18:47:54Z Severity: Sev3 Labels: alertname="KubePodCrashLooping", cluster="ci01-j3671936-mgmt-2", component="kubernetes-infrastructure", container="hypershift-addon-agent", endpoint="http", environment="ci01", instance="10.128.64.162:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j3671936/providers/microsoft.monitor/accounts/services-j3671936", microsoft.metrics_include_label="service", namespace="open-cluster-management-agent-addon", pod="hypershift-addon-agent-76ffb6fdb8-bfwrz", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", reason="CrashLoopBackOff", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="97b76fc8-3f29-44d9-83c4-855c4cca2cf8" Description: Pod open-cluster-management-agent-addon/hypershift-addon-agent-76ffb6fdb8-bfwrz (hypershift-addon-agent) is in waiting state (reason: "CrashLoopBackOff"). Firing 2: State: Fired (not resolved) Started: 2026-10-25T18:47:54Z Severity: Sev3 Labels: alertname="KubePodCrashLooping", cluster="ci01-j3671936-mgmt-1", component="kubernetes-infrastructure", container="hypershift-addon-agent", endpoint="http", environment="ci01", instance="10.128.64.215:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j3671936/providers/microsoft.monitor/accounts/services-j3671936", microsoft.metrics_include_label="service", namespace="open-cluster-management-agent-addon", pod="hypershift-addon-agent-76ffb6fdb8-mftk6", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", reason="CrashLoopBackOff", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="c4f6f9ce-1370-407b-a328-72082e9577a0" Description: Pod open-cluster-management-agent-addon/hypershift-addon-agent-76ffb6fdb8-mftk6 (hypershift-addon-agent) is in waiting state (reason: "CrashLoopBackOff"). Firing 3: State: Fired (not resolved) Started: 2026-10-25T18:50:58Z Severity: Sev3 Labels: alertname="KubePodCrashLooping", cluster="ci01-j3671936-mgmt-1", component="kubernetes-infrastructure", container="hypershift-addon-agent", endpoint="http", environment="ci01", instance="10.128.64.215:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j3671936/providers/microsoft.monitor/accounts/services-j3671936", microsoft.metrics_include_label="service", namespace="open-cluster-management-agent-addon", pod="hypershift-addon-agent-76ffb6fdb8-mftk6", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", reason="CrashLoopBackOff", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="c4f6f9ce-1370-407b-a328-72082e9577a0" Description: Pod open-cluster-management-agent-addon/hypershift-addon-agent-76ffb6fdb8-mftk6 (hypershift-addon-agent) is in waiting state (reason: "CrashLoopBackOff"). Firing 4: State: Fired (not resolved) Started: 2026-10-25T18:50:58Z Severity: Sev3 Labels: alertname="KubePodCrashLooping", cluster="ci01-j3671936-mgmt-2", component="kubernetes-infrastructure", container="hypershift-addon-agent", endpoint="http", environment="ci01", instance="10.128.64.162:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j3671936/providers/microsoft.monitor/accounts/services-j3671936", microsoft.metrics_include_label="service", namespace="open-cluster-management-agent-addon", pod="hypershift-addon-agent-76ffb6fdb8-bfwrz", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", reason="CrashLoopBackOff", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="97b76fc8-3f29-44d9-83c4-855c4cca2cf8" Description: Pod open-cluster-management-agent-addon/hypershift-addon-agent-76ffb6fdb8-bfwrz (hypershift-addon-agent) is in waiting state (reason: "CrashLoopBackOff"). alert fired 4 time(s) Firing 1: State: Fired (not resolved) Started: 2026-10-25T13:52:18Z Severity: Sev3 Labels: alertname="KubePodCrashLooping", cluster="ci01-j8072320-mgmt-2", component="kubernetes-infrastructure", container="hypershift-addon-agent", endpoint="http", environment="ci01", instance="10.128.65.54:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j8072320/providers/microsoft.monitor/accounts/services-j8072320", microsoft.metrics_include_label="service", namespace="open-cluster-management-agent-addon", pod="hypershift-addon-agent-76ffb6fdb8-jdq5f", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", reason="CrashLoopBackOff", region="centralus", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="9c6d2039-7988-4be2-8890-d290ed7d2d20" Description: Pod open-cluster-management-agent-addon/hypershift-addon-agent-76ffb6fdb8-jdq5f (hypershift-addon-agent) is in waiting state (reason: "CrashLoopBackOff"). Firing 2: State: Fired (not resolved) Started: 2026-10-25T13:54:18Z Severity: Sev3 Labels: alertname="KubePodCrashLooping", cluster="ci01-j8072320-mgmt-2", component="kubernetes-infrastructure", container="hypershift-addon-agent", endpoint="http", environment="ci01", instance="10.128.65.54:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j8072320/providers/microsoft.monitor/accounts/services-j8072320", microsoft.metrics_include_label="service", namespace="open-cluster-management-agent-addon", pod="hypershift-addon-agent-76ffb6fdb8-jdq5f", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", reason="CrashLoopBackOff", region="centralus", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="9c6d2039-7988-4be2-8890-d290ed7d2d20" Description: Pod open-cluster-management-agent-addon/hypershift-addon-agent-76ffb6fdb8-jdq5f (hypershift-addon-agent) is in waiting state (reason: "CrashLoopBackOff"). Firing 3: State: Fired (not resolved) Started: 2026-10-25T13:55:20Z Severity: Sev3 Labels: alertname="KubePodCrashLooping", cluster="ci01-j8072320-mgmt-1", component="kubernetes-infrastructure", container="hypershift-addon-agent", endpoint="http", environment="ci01", instance="10.128.64.141:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j8072320/providers/microsoft.monitor/accounts/services-j8072320", microsoft.metrics_include_label="service", namespace="open-cluster-management-agent-addon", pod="hypershift-addon-agent-76ffb6fdb8-h6zb9", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", reason="CrashLoopBackOff", region="centralus", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="8720bf8b-027c-410b-9181-eafea5b14401" Description: Pod open-cluster-management-agent-addon/hypershift-addon-agent-76ffb6fdb8-h6zb9 (hypershift-addon-agent) is in waiting state (reason: "CrashLoopBackOff"). Firing 4: State: Fired (not resolved) Started: 2026-10-25T13:55:20Z Severity: Sev3 Labels: alertname="KubePodCrashLooping", cluster="ci01-j8072320-mgmt-1", component="kubernetes-infrastructure", container="hypershift-addon-agent", endpoint="http", environment="ci01", instance="10.128.64.141:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j8072320/providers/microsoft.monitor/accounts/services-j8072320", microsoft.metrics_include_label="service", namespace="open-cluster-management-agent-addon", pod="hypershift-addon-agent-76ffb6fdb8-h6zb9", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", reason="CrashLoopBackOff", region="centralus", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="8720bf8b-027c-410b-9181-eafea5b14401" Description: Pod open-cluster-management-agent-addon/hypershift-addon-agent-76ffb6fdb8-h6zb9 (hypershift-addon-agent) is in waiting state (reason: "CrashLoopBackOff"). alert fired 8 time(s) Firing 1: State: Resolved Started: 2026-10-25T11:31:30Z Ended: 2026-10-25T11:58:30Z Severity: Sev3 Labels: alertname="KubePodCrashLooping", cluster="ci01-j7524864-mgmt-2", component="kubernetes-infrastructure", container="manager", endpoint="http", environment="ci01", instance="10.128.64.204:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j7524864/providers/microsoft.monitor/accounts/services-j7524864", microsoft.metrics_include_label="service", namespace="secret-sync-controller", pod="secrets-store-sync-controller-manager-7f4b9bc5df-czwqq", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", reason="CrashLoopBackOff", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="76e86419-4829-44c0-a136-817aadd188d4" Description: Pod secret-sync-controller/secrets-store-sync-controller-manager-7f4b9bc5df-czwqq (manager) is in waiting state (reason: "CrashLoopBackOff"). Firing 2: State: Resolved Started: 2026-10-25T11:31:30Z Ended: 2026-10-25T11:58:31Z Severity: Sev3 Labels: alertname="KubePodCrashLooping", cluster="ci01-j7524864-mgmt-2", component="kubernetes-infrastructure", container="manager", endpoint="http", environment="ci01", instance="10.128.64.204:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j7524864/providers/microsoft.monitor/accounts/services-j7524864", microsoft.metrics_include_label="service", namespace="secret-sync-controller", pod="secrets-store-sync-controller-manager-7f4b9bc5df-czwqq", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", reason="CrashLoopBackOff", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="76e86419-4829-44c0-a136-817aadd188d4" Description: Pod secret-sync-controller/secrets-store-sync-controller-manager-7f4b9bc5df-czwqq (manager) is in waiting state (reason: "CrashLoopBackOff"). Firing 3: State: Resolved Started: 2026-10-25T11:37:32Z Ended: 2026-10-25T11:48:32Z Severity: Sev3 Labels: alertname="KubePodCrashLooping", cluster="ci01-j7524864-mgmt-2", component="kubernetes-infrastructure", container="governance-policy-propagator", endpoint="http", environment="ci01", instance="10.128.64.204:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j7524864/providers/microsoft.monitor/accounts/services-j7524864", microsoft.metrics_include_label="service", namespace="multicluster-engine", pod="grc-policy-propagator-5fc7fb4677-67tll", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", reason="CrashLoopBackOff", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="e5593f36-ff37-4a56-9501-050ee9a5f489" Description: Pod multicluster-engine/grc-policy-propagator-5fc7fb4677-67tll (governance-policy-propagator) is in waiting state (reason: "CrashLoopBackOff"). Firing 4: State: Resolved Started: 2026-10-25T11:37:32Z Ended: 2026-10-25T11:48:32Z Severity: Sev3 Labels: alertname="KubePodCrashLooping", cluster="ci01-j7524864-mgmt-2", component="kubernetes-infrastructure", container="governance-policy-propagator", endpoint="http", environment="ci01", instance="10.128.64.204:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j7524864/providers/microsoft.monitor/accounts/services-j7524864", microsoft.metrics_include_label="service", namespace="multicluster-engine", pod="grc-policy-propagator-5fc7fb4677-67tll", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", reason="CrashLoopBackOff", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="e5593f36-ff37-4a56-9501-050ee9a5f489" Description: Pod multicluster-engine/grc-policy-propagator-5fc7fb4677-67tll (governance-policy-propagator) is in waiting state (reason: "CrashLoopBackOff"). Firing 5: State: Resolved Started: 2026-10-25T11:39:32Z Ended: 2026-10-25T11:48:32Z Severity: Sev3 Labels: alertname="KubePodCrashLooping", cluster="ci01-j7524864-mgmt-2", component="kubernetes-infrastructure", container="governance-policy-propagator", endpoint="http", environment="ci01", instance="10.128.64.204:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j7524864/providers/microsoft.monitor/accounts/services-j7524864", microsoft.metrics_include_label="service", namespace="multicluster-engine", pod="grc-policy-propagator-5fc7fb4677-j5hzx", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", reason="CrashLoopBackOff", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="eabaed55-b955-4173-b78d-fec0366c15f2" Description: Pod multicluster-engine/grc-policy-propagator-5fc7fb4677-j5hzx (governance-policy-propagator) is in waiting state (reason: "CrashLoopBackOff"). Firing 6: State: Resolved Started: 2026-10-25T11:39:32Z Ended: 2026-10-25T11:48:32Z Severity: Sev3 Labels: alertname="KubePodCrashLooping", cluster="ci01-j7524864-mgmt-2", component="kubernetes-infrastructure", container="governance-policy-propagator", endpoint="http", environment="ci01", instance="10.128.64.204:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j7524864/providers/microsoft.monitor/accounts/services-j7524864", microsoft.metrics_include_label="service", namespace="multicluster-engine", pod="grc-policy-propagator-5fc7fb4677-j5hzx", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", reason="CrashLoopBackOff", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="eabaed55-b955-4173-b78d-fec0366c15f2" Description: Pod multicluster-engine/grc-policy-propagator-5fc7fb4677-j5hzx (governance-policy-propagator) is in waiting state (reason: "CrashLoopBackOff"). Firing 7: State: Resolved Started: 2026-10-25T11:41:31Z Ended: 2026-10-25T11:52:33Z Severity: Sev3 Labels: alertname="KubePodCrashLooping", cluster="ci01-j7524864-mgmt-2", component="kubernetes-infrastructure", container="ocm-controller", endpoint="http", environment="ci01", instance="10.128.64.204:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j7524864/providers/microsoft.monitor/accounts/services-j7524864", microsoft.metrics_include_label="service", namespace="multicluster-engine", pod="ocm-controller-859484fc49-pj8f8", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", reason="CrashLoopBackOff", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="a3ea24fa-145a-42ea-812b-1fe46f10bfd1" Description: Pod multicluster-engine/ocm-controller-859484fc49-pj8f8 (ocm-controller) is in waiting state (reason: "CrashLoopBackOff"). Firing 8: State: Resolved Started: 2026-10-25T11:41:31Z Ended: 2026-10-25T11:52:33Z Severity: Sev3 Labels: alertname="KubePodCrashLooping", cluster="ci01-j7524864-mgmt-2", component="kubernetes-infrastructure", container="ocm-controller", endpoint="http", environment="ci01", instance="10.128.64.204:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j7524864/providers/microsoft.monitor/accounts/services-j7524864", microsoft.metrics_include_label="service", namespace="multicluster-engine", pod="ocm-controller-859484fc49-pj8f8", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", reason="CrashLoopBackOff", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="a3ea24fa-145a-42ea-812b-1fe46f10bfd1" Description: Pod multicluster-engine/ocm-controller-859484fc49-pj8f8 (ocm-controller) is in waiting state (reason: "CrashLoopBackOff"). Contributing tests (1)
Affected runs (17) | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
failed to roll out Helm release: failed post-install: resource Job/default-sc/unannotate-default-sc not ready. status: Failed, message: Job Failed. failed: 1/1full failure pattern: failed to roll out Helm release: failed post-install: resource Job/default-sc/unannotate-default-sc not ready. status: Failed, message: Job Failed. failed: 1/1 Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 4 day(s) | provision | 17 | 1.71%17 of 995 job runs affected | IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 4 day(s) | Show trend detailsOct 12: 0 · Oct 13: 0 · Oct 14: 0 · Oct 15: 2 · Oct 16: 2 · Oct 17: 0 · Oct 18: 0 · Oct 19: 9 · Oct 20: 4 · Oct 21: 0 · Oct 22: 0 · Oct 23: 0 · Oct 24: 0 · Oct 25: 0 | none | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (3)time=2026-10-20T07:36:27.039Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=storageclass stamp=2 description="Step storageclass\n Kind: Helm\n" time=2026-10-20T07:36:27.554Z level=INFO msg="Resolved input values." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=storageclass stamp=2 values="map[skuName:PremiumV2_LRS unannotateJobImage:mcr.microsoft.com/aks/command/runtime@sha256:5c16384cf45be47415214887ce360da268c4e7dbd157914665ad1244e8982595]" time=2026-10-20T07:36:27.554Z level=INFO msg="Applying namespaces." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=storageclass stamp=2 time=2026-10-20T07:36:28.125Z level=INFO msg="Namespace applied successfully." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=storageclass stamp=2 namespace=default-sc time=2026-10-20T07:36:28.391Z level=INFO msg="Rolling out Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=storageclass stamp=2 dryRun=false time=2026-10-20T07:36:28.391Z level=INFO msg="Searching for release history..." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=storageclass stamp=2 time=2026-10-20T07:36:28.520Z level=INFO msg="No release history found, running the first release..." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=storageclass stamp=2 time=2026-10-20T07:37:32.195Z level=ERROR msg="Failed to roll out the Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=storageclass stamp=2 err="failed post-install: resource Job/default-sc/unannotate-default-sc not ready. status: Failed, message: Job Failed. failed: 1/1" time=2026-10-20T07:37:32.196Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=storageclass stamp=2 err="stamp 2: error running Helm release deployment Step, failed to deploy helm release: failed to roll out Helm release: failed post-install: resource Job/default-sc/unannotate-default-sc not ready. status: Failed, message: Job Failed. failed: 1/1" time=2026-10-20T04:38:22.120Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=storageclass stamp=2 description="Step storageclass\n Kind: Helm\n" time=2026-10-20T04:38:22.853Z level=INFO msg="Resolved input values." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=storageclass stamp=2 values="map[skuName:PremiumV2_LRS unannotateJobImage:mcr.microsoft.com/aks/command/runtime@sha256:5c16384cf45be47415214887ce360da268c4e7dbd157914665ad1244e8982595]" time=2026-10-20T04:38:22.853Z level=INFO msg="Applying namespaces." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=storageclass stamp=2 time=2026-10-20T04:38:25.274Z level=INFO msg="Namespace applied successfully." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=storageclass stamp=2 namespace=default-sc time=2026-10-20T04:38:25.609Z level=INFO msg="Rolling out Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=storageclass stamp=2 dryRun=false time=2026-10-20T04:38:25.609Z level=INFO msg="Searching for release history..." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=storageclass stamp=2 time=2026-10-20T04:38:25.740Z level=INFO msg="No release history found, running the first release..." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=storageclass stamp=2 time=2026-10-20T04:39:19.309Z level=ERROR msg="Failed to roll out the Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=storageclass stamp=2 err="failed post-install: resource Job/default-sc/unannotate-default-sc not ready. status: Failed, message: Job Failed. failed: 1/1" time=2026-10-20T04:39:19.309Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=storageclass stamp=2 err="stamp 2: error running Helm release deployment Step, failed to deploy helm release: failed to roll out Helm release: failed post-install: resource Job/default-sc/unannotate-default-sc not ready. status: Failed, message: Job Failed. failed: 1/1" time=2026-10-20T02:55:35.748Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=storageclass stamp=1 description="Step storageclass\n Kind: Helm\n" time=2026-10-20T02:55:36.227Z level=INFO msg="Resolved input values." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=storageclass stamp=1 values="map[skuName:PremiumV2_LRS unannotateJobImage:mcr.microsoft.com/aks/command/runtime@sha256:5c16384cf45be47415214887ce360da268c4e7dbd157914665ad1244e8982595]" time=2026-10-20T02:55:36.227Z level=INFO msg="Applying namespaces." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=storageclass stamp=1 time=2026-10-20T02:55:36.416Z level=INFO msg="Namespace applied successfully." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=storageclass stamp=1 namespace=default-sc time=2026-10-20T02:55:36.603Z level=INFO msg="Rolling out Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=storageclass stamp=1 dryRun=false time=2026-10-20T02:55:36.603Z level=INFO msg="Searching for release history..." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=storageclass stamp=1 time=2026-10-20T02:55:36.660Z level=INFO msg="No release history found, running the first release..." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=storageclass stamp=1 time=2026-10-20T02:56:29.064Z level=ERROR msg="Failed to roll out the Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=storageclass stamp=1 err="failed post-install: resource Job/default-sc/unannotate-default-sc not ready. status: Failed, message: Job Failed. failed: 1/1" time=2026-10-20T02:56:29.064Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=storageclass stamp=1 err="stamp 1: error running Helm release deployment Step, failed to deploy helm release: failed to roll out Helm release: failed post-install: resource Job/default-sc/unannotate-default-sc not ready. status: Failed, message: Job Failed. failed: 1/1" Contributing tests (2)
Affected runs (17) | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
failed to roll out Helm release: resource Deployment/secret-sync-controller/secrets-store-sync-controller-manager not ready. status: InProgress, message: Available: 0/1 context de...full failure pattern: failed to roll out Helm release: resource Deployment/secret-sync-controller/secrets-store-sync-controller-manager not ready. status: InProgress, message: Available: 0/1 context deadline exceeded Signal: IndeterminateSignal: Indeterminate — no prior history; active 3 day(s) — New failure pattern — no prior history | provision | 13 | 1.31%13 of 995 job runs affected | IndeterminateSignal: Indeterminate — no prior history; active 3 day(s) — New failure pattern — no prior history | Show trend detailsOct 12: 0 · Oct 13: 0 · Oct 14: 0 · Oct 15: 0 · Oct 16: 0 · Oct 17: 0 · Oct 18: 0 · Oct 19: 0 · Oct 20: 0 · Oct 21: 0 · Oct 22: 3 · Oct 23: 6 · Oct 24: 0 · Oct 25: 4 | none | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (3)time=2026-10-25T10:27:22.014Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.SecretSyncController resourceGroup=management step=deploy stamp=2 description="Step deploy\n Kind: Helm\n" time=2026-10-25T10:27:22.644Z level=INFO msg="Resolved input values." serviceGroup=Microsoft.Azure.ARO.HCP.SecretSyncController resourceGroup=management step=deploy stamp=2 values="map[controllerName:secrets-store-sync-controller-manager image:arohcpsvcdev.azurecr.io/secrets-store-sync/controller@sha256:b98f9f1c3c603f0e3ee70cd3d70746d8265b8d425df5b71f23c4b0dcb3eb979c imagePullPolicy:IfNotPresent metricsPort:8085 providerContainer:[map[args:[--endpoint=unix:///provider/azure.sock --construct-pem-chain=true --healthz-port=8989 --healthz-path=/healthz --healthz-timeout=5s] image:mcr.microsoft.com/oss/v2/azure/secrets-store/provider-azure@sha256:b451542dd17a9b1b1b400fa1ca768a492720b87b696f373f0494217d855aee8c imagePullPolicy:IfNotPresent livenessProbe:map[failureThreshold:3 httpGet:map[path:/healthz port:8989] initialDelaySeconds:5 periodSeconds:30 timeoutSeconds:10] name:provider-azure-installer ports:[map[containerPort:8898 name:metrics protocol:TCP]] resources:map[limits:map[memory:unlimited] requests:map[cpu:50m memory:100Mi]] securityContext:map[allowPrivilegeEscalation:false capabilities:map[drop:[ALL]] readOnlyRootFilesystem:true runAsUser:0] volumeMounts:[map[mountPath:/provider name:providervol]]]] tokenRequestAudience:[map[audience:<nil>]] validatingAdmissionPolicies:map[allowedSecretTypes:[Opaque kubernetes.io/basic-auth bootstrap.kubernetes.io/token kubernetes.io/dockerconfigjson kubernetes.io/dockercfg kubernetes.io/ssh-auth kubernetes.io/tls] applyPolicies:true deniedSecretTypes:[kubernetes.io/service-account-token]]]" time=2026-10-25T10:27:22.644Z level=INFO msg="Applying namespaces." serviceGroup=Microsoft.Azure.ARO.HCP.SecretSyncController resourceGroup=management step=deploy stamp=2 time=2026-10-25T10:27:22.918Z level=INFO msg="Namespace applied successfully." serviceGroup=Microsoft.Azure.ARO.HCP.SecretSyncController resourceGroup=management step=deploy stamp=2 namespace=secret-sync-controller time=2026-10-25T10:27:23.279Z level=INFO msg="Rolling out Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.SecretSyncController resourceGroup=management step=deploy stamp=2 dryRun=false time=2026-10-25T10:27:23.279Z level=INFO msg="Searching for release history..." serviceGroup=Microsoft.Azure.ARO.HCP.SecretSyncController resourceGroup=management step=deploy stamp=2 time=2026-10-25T10:27:23.445Z level=INFO msg="No release history found, running the first release..." serviceGroup=Microsoft.Azure.ARO.HCP.SecretSyncController resourceGroup=management step=deploy stamp=2 time=2026-10-25T10:32:28.984Z level=ERROR msg="Failed to roll out the Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.SecretSyncController resourceGroup=management step=deploy stamp=2 err="resource Deployment/secret-sync-controller/secrets-store-sync-controller-manager not ready. status: InProgress, message: Available: 0/1\ncontext deadline exceeded" time=2026-10-25T10:32:28.985Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.SecretSyncController resourceGroup=management step=deploy stamp=2 err="stamp 2: error running Helm release deployment Step, failed to deploy helm release: failed to roll out Helm release: resource Deployment/secret-sync-controller/secrets-store-sync-controller-manager not ready. status: InProgress, message: Available: 0/1\ncontext deadline exceeded" time=2026-10-25T08:14:44.299Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.SecretSyncController resourceGroup=management step=deploy stamp=1 description="Step deploy\n Kind: Helm\n" time=2026-10-25T08:14:45.051Z level=INFO msg="Resolved input values." serviceGroup=Microsoft.Azure.ARO.HCP.SecretSyncController resourceGroup=management step=deploy stamp=1 values="map[controllerName:secrets-store-sync-controller-manager image:arohcpsvcdev.azurecr.io/secrets-store-sync/controller@sha256:b98f9f1c3c603f0e3ee70cd3d70746d8265b8d425df5b71f23c4b0dcb3eb979c imagePullPolicy:IfNotPresent metricsPort:8085 providerContainer:[map[args:[--endpoint=unix:///provider/azure.sock --construct-pem-chain=true --healthz-port=8989 --healthz-path=/healthz --healthz-timeout=5s] image:mcr.microsoft.com/oss/v2/azure/secrets-store/provider-azure@sha256:b451542dd17a9b1b1b400fa1ca768a492720b87b696f373f0494217d855aee8c imagePullPolicy:IfNotPresent livenessProbe:map[failureThreshold:3 httpGet:map[path:/healthz port:8989] initialDelaySeconds:5 periodSeconds:30 timeoutSeconds:10] name:provider-azure-installer ports:[map[containerPort:8898 name:metrics protocol:TCP]] resources:map[limits:map[memory:unlimited] requests:map[cpu:50m memory:100Mi]] securityContext:map[allowPrivilegeEscalation:false capabilities:map[drop:[ALL]] readOnlyRootFilesystem:true runAsUser:0] volumeMounts:[map[mountPath:/provider name:providervol]]]] tokenRequestAudience:[map[audience:<nil>]] validatingAdmissionPolicies:map[allowedSecretTypes:[Opaque kubernetes.io/basic-auth bootstrap.kubernetes.io/token kubernetes.io/dockerconfigjson kubernetes.io/dockercfg kubernetes.io/ssh-auth kubernetes.io/tls] applyPolicies:true deniedSecretTypes:[kubernetes.io/service-account-token]]]" time=2026-10-25T08:14:45.051Z level=INFO msg="Applying namespaces." serviceGroup=Microsoft.Azure.ARO.HCP.SecretSyncController resourceGroup=management step=deploy stamp=1 time=2026-10-25T08:14:45.565Z level=INFO msg="Namespace applied successfully." serviceGroup=Microsoft.Azure.ARO.HCP.SecretSyncController resourceGroup=management step=deploy stamp=1 namespace=secret-sync-controller time=2026-10-25T08:14:45.984Z level=INFO msg="Rolling out Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.SecretSyncController resourceGroup=management step=deploy stamp=1 dryRun=false time=2026-10-25T08:14:45.984Z level=INFO msg="Searching for release history..." serviceGroup=Microsoft.Azure.ARO.HCP.SecretSyncController resourceGroup=management step=deploy stamp=1 time=2026-10-25T08:14:46.156Z level=INFO msg="No release history found, running the first release..." serviceGroup=Microsoft.Azure.ARO.HCP.SecretSyncController resourceGroup=management step=deploy stamp=1 time=2026-10-25T08:19:54.254Z level=ERROR msg="Failed to roll out the Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.SecretSyncController resourceGroup=management step=deploy stamp=1 err="resource Deployment/secret-sync-controller/secrets-store-sync-controller-manager not ready. status: InProgress, message: Available: 0/1\ncontext deadline exceeded" time=2026-10-25T08:19:54.254Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.SecretSyncController resourceGroup=management step=deploy stamp=1 err="stamp 1: error running Helm release deployment Step, failed to deploy helm release: failed to roll out Helm release: resource Deployment/secret-sync-controller/secrets-store-sync-controller-manager not ready. status: InProgress, message: Available: 0/1\ncontext deadline exceeded" time=2026-10-25T06:58:33.435Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.SecretSyncController resourceGroup=management step=deploy stamp=1 description="Step deploy\n Kind: Helm\n" time=2026-10-25T06:58:33.964Z level=INFO msg="Resolved input values." serviceGroup=Microsoft.Azure.ARO.HCP.SecretSyncController resourceGroup=management step=deploy stamp=1 values="map[controllerName:secrets-store-sync-controller-manager image:arohcpsvcdev.azurecr.io/secrets-store-sync/controller@sha256:b98f9f1c3c603f0e3ee70cd3d70746d8265b8d425df5b71f23c4b0dcb3eb979c imagePullPolicy:IfNotPresent metricsPort:8085 providerContainer:[map[args:[--endpoint=unix:///provider/azure.sock --construct-pem-chain=true --healthz-port=8989 --healthz-path=/healthz --healthz-timeout=5s] image:mcr.microsoft.com/oss/v2/azure/secrets-store/provider-azure@sha256:b451542dd17a9b1b1b400fa1ca768a492720b87b696f373f0494217d855aee8c imagePullPolicy:IfNotPresent livenessProbe:map[failureThreshold:3 httpGet:map[path:/healthz port:8989] initialDelaySeconds:5 periodSeconds:30 timeoutSeconds:10] name:provider-azure-installer ports:[map[containerPort:8898 name:metrics protocol:TCP]] resources:map[limits:map[memory:unlimited] requests:map[cpu:50m memory:100Mi]] securityContext:map[allowPrivilegeEscalation:false capabilities:map[drop:[ALL]] readOnlyRootFilesystem:true runAsUser:0] volumeMounts:[map[mountPath:/provider name:providervol]]]] tokenRequestAudience:[map[audience:<nil>]] validatingAdmissionPolicies:map[allowedSecretTypes:[Opaque kubernetes.io/basic-auth bootstrap.kubernetes.io/token kubernetes.io/dockerconfigjson kubernetes.io/dockercfg kubernetes.io/ssh-auth kubernetes.io/tls] applyPolicies:true deniedSecretTypes:[kubernetes.io/service-account-token]]]" time=2026-10-25T06:58:33.964Z level=INFO msg="Applying namespaces." serviceGroup=Microsoft.Azure.ARO.HCP.SecretSyncController resourceGroup=management step=deploy stamp=1 time=2026-10-25T06:58:34.148Z level=INFO msg="Namespace applied successfully." serviceGroup=Microsoft.Azure.ARO.HCP.SecretSyncController resourceGroup=management step=deploy stamp=1 namespace=secret-sync-controller time=2026-10-25T06:58:34.349Z level=INFO msg="Rolling out Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.SecretSyncController resourceGroup=management step=deploy stamp=1 dryRun=false time=2026-10-25T06:58:34.350Z level=INFO msg="Searching for release history..." serviceGroup=Microsoft.Azure.ARO.HCP.SecretSyncController resourceGroup=management step=deploy stamp=1 time=2026-10-25T06:58:34.404Z level=INFO msg="No release history found, running the first release..." serviceGroup=Microsoft.Azure.ARO.HCP.SecretSyncController resourceGroup=management step=deploy stamp=1 time=2026-10-25T07:03:38.507Z level=ERROR msg="Failed to roll out the Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.SecretSyncController resourceGroup=management step=deploy stamp=1 err="resource Deployment/secret-sync-controller/secrets-store-sync-controller-manager not ready. status: InProgress, message: Available: 0/1\ncontext deadline exceeded" time=2026-10-25T07:03:38.507Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.SecretSyncController resourceGroup=management step=deploy stamp=1 err="stamp 1: error running Helm release deployment Step, failed to deploy helm release: failed to roll out Helm release: resource Deployment/secret-sync-controller/secrets-store-sync-controller-manager not ready. status: InProgress, message: Available: 0/1\ncontext deadline exceeded" Contributing tests (2)
Affected runs (13) | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
failed to run istio upgrade step: failed to start canary: start-canary: ARM operation timed out or failed: context deadline exceededfull failure pattern: failed to run istio upgrade step: failed to start canary: start-canary: ARM operation timed out or failed: context deadline exceeded Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 7 day(s) | provision | 12 | 1.21%12 of 995 job runs affected | IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 7 day(s) | Show trend detailsOct 12: 0 · Oct 13: 0 · Oct 14: 0 · Oct 15: 0 · Oct 16: 1 · Oct 17: 0 · Oct 18: 1 · Oct 19: 1 · Oct 20: 0 · Oct 21: 2 · Oct 22: 3 · Oct 23: 2 · Oct 24: 2 · Oct 25: 0 | none | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (3)time=2026-10-24T16:56:57.345Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=istio-upgrade description="Step istio-upgrade\n Kind: IstioUpgrade\n AKSCluster: {{ svc.aks.name }}\n"
time=2026-10-24T16:56:58.695Z level=INFO msg="Istio upgrade -- cluster state" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=istio-upgrade cluster=ci01-j5468416-svc versions=asm-1-30 logger=istio-upgrade k8sVersion=1.35 provisioningState=Succeeded installedRevisions=[asm-1-29] availableUpgrades=[asm-1-30] upgradeInProgress=false
time=2026-10-24T16:56:58.695Z level=INFO msg="Upgrading to svc.istio.versions target" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=istio-upgrade cluster=ci01-j5468416-svc versions=asm-1-30 logger=istio-upgrade from=asm-1-29 to=asm-1-30 k8sVersion=1.35
time=2026-10-24T16:56:59.317Z level=INFO msg="Istio upgrade -- mesh state" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=istio-upgrade cluster=ci01-j5468416-svc versions=asm-1-30 logger=istio-upgrade namespaces=0 controlPlanes=asm-1-29(2/2) ingressGateways=1 webhooks=1 configMaps=2
time=2026-10-24T16:56:59.317Z level=INFO msg="Starting canary -- installing target alongside current" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=istio-upgrade cluster=ci01-j5468416-svc versions=asm-1-30 logger=istio-upgrade
time=2026-10-24T16:56:59.317Z level=INFO msg="Starting ARM operation" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=istio-upgrade stepID=Microsoft.Azure.ARO.HCP.Service.Infra/service/istio-upgrade operation=start-canary cluster=ci01-j5468416-svc
time=2026-10-24T17:12:01.792Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=istio-upgrade err="failed to run istio upgrade step: failed to start canary: start-canary: ARM operation timed out or failed: context deadline exceeded"time=2026-10-24T12:42:08.055Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=istio-upgrade description="Step istio-upgrade\n Kind: IstioUpgrade\n AKSCluster: {{ svc.aks.name }}\n"
time=2026-10-24T12:42:09.448Z level=INFO msg="Istio upgrade -- cluster state" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=istio-upgrade cluster=ci01-j1222656-svc versions=asm-1-30 logger=istio-upgrade k8sVersion=1.35 provisioningState=Succeeded installedRevisions=[asm-1-29] availableUpgrades=[asm-1-30] upgradeInProgress=false
time=2026-10-24T12:42:09.448Z level=INFO msg="Upgrading to svc.istio.versions target" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=istio-upgrade cluster=ci01-j1222656-svc versions=asm-1-30 logger=istio-upgrade from=asm-1-29 to=asm-1-30 k8sVersion=1.35
time=2026-10-24T12:42:10.162Z level=INFO msg="Istio upgrade -- mesh state" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=istio-upgrade cluster=ci01-j1222656-svc versions=asm-1-30 logger=istio-upgrade namespaces=0 controlPlanes=asm-1-29(2/2) ingressGateways=1 webhooks=1 configMaps=2
time=2026-10-24T12:42:10.162Z level=INFO msg="Starting canary -- installing target alongside current" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=istio-upgrade cluster=ci01-j1222656-svc versions=asm-1-30 logger=istio-upgrade
time=2026-10-24T12:42:10.162Z level=INFO msg="Starting ARM operation" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=istio-upgrade stepID=Microsoft.Azure.ARO.HCP.Service.Infra/service/istio-upgrade operation=start-canary cluster=ci01-j1222656-svc
time=2026-10-24T12:57:12.942Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=istio-upgrade err="failed to run istio upgrade step: failed to start canary: start-canary: ARM operation timed out or failed: context deadline exceeded"time=2026-10-23T10:04:28.865Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=istio-upgrade description="Step istio-upgrade\n Kind: IstioUpgrade\n AKSCluster: {{ svc.aks.name }}\n"
time=2026-10-23T10:04:30.857Z level=INFO msg="Istio upgrade -- cluster state" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=istio-upgrade cluster=ci01-j4942976-svc versions=asm-1-30 logger=istio-upgrade k8sVersion=1.35 provisioningState=Succeeded installedRevisions=[asm-1-29] availableUpgrades=[asm-1-30] upgradeInProgress=false
time=2026-10-23T10:04:30.857Z level=INFO msg="Upgrading to svc.istio.versions target" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=istio-upgrade cluster=ci01-j4942976-svc versions=asm-1-30 logger=istio-upgrade from=asm-1-29 to=asm-1-30 k8sVersion=1.35
time=2026-10-23T10:04:32.118Z level=INFO msg="Istio upgrade -- mesh state" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=istio-upgrade cluster=ci01-j4942976-svc versions=asm-1-30 logger=istio-upgrade namespaces=0 controlPlanes=asm-1-29(2/2) ingressGateways=1 webhooks=1 configMaps=2
time=2026-10-23T10:04:32.118Z level=INFO msg="Starting canary -- installing target alongside current" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=istio-upgrade cluster=ci01-j4942976-svc versions=asm-1-30 logger=istio-upgrade
time=2026-10-23T10:04:32.118Z level=INFO msg="Starting ARM operation" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=istio-upgrade stepID=Microsoft.Azure.ARO.HCP.Service.Infra/service/istio-upgrade operation=start-canary cluster=ci01-j4942976-svc
time=2026-10-23T10:19:36.095Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=istio-upgrade err="failed to run istio upgrade step: failed to start canary: start-canary: ARM operation timed out or failed: context deadline exceeded"Contributing tests (1)
Affected runs (12) | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
ERROR CODE: DeadlineExceeded; detail message node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hype...full failure pattern: ERROR CODE: DeadlineExceeded; detail message node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is <count>, want <count>; node pool condition AllNodesHealthy is False: <count> of <count> machines are not healthy Machine <node>: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.Compute/virtualMachines/<node> to exist Machine <node>: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.Compute/virtualMachines/<node> to exist; node pool condition AllMachinesReady is False: <count> of <count> machines are not ready Machine <node>: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.Compute/virtualMachines/<node> to exist Machine <node>: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.Compute/virtualMachines/<node> to exist; provider Microsoft.RedHatOpenShift Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 7 day(s) | e2e | 11 | 1.11%11 of 995 job runs affected | IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 7 day(s) | Show trend detailsOct 12: 2 · Oct 13: 1 · Oct 14: 0 · Oct 15: 4 · Oct 16: 1 · Oct 17: 0 · Oct 18: 0 · Oct 19: 0 · Oct 20: 0 · Oct 21: 1 · Oct 22: 0 · Oct 23: 1 · Oct 24: 0 · Oct 25: 1 | INT, PROD | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (3)fail [github.com/Azure/ARO-HCP/test/e2e/cluster_autoscaling.go:139]: failed to create nodepool "np-1" for autoscaling cluster
Unexpected error:
<*fmt.wrapError | 0x2a5fafb1fdc0>:
failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "autoscaling-hcp-cluster" in resourcegroup="autoscaling-cluster-dhxpqc2757nk" to finish creating: GET https://rp.j9438720.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/995ad93d-44a1-4aae-9f3c-cef4914b4415
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/995ad93d-44a1-4aae-9f3c-cef4914b4415",
"name": "995ad93d-44a1-4aae-9f3c-cef4914b4415",
"status": "Failed",
"startTime": "2026-10-25T11:09:27.17231924Z",
"endTime": "2026-10-25T11:28:33.403436193Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine w1u6b4w7n6r4n6j-np-1-tvg6z-kjbw8: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/autoscaling-cluster-dhxpqc2757nk--managed/providers/Microsoft.Compute/virtualMachines/w1u6b4w7n6r4n6j-np-1-tvg6z-kjbw8 to exist\nMachine w1u6b4w7n6r4n6j-np-1-tvg6z-njbxt: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/autoscaling-cluster-dhxpqc2757nk--managed/providers/Microsoft.Compute/virtualMachines/w1u6b4w7n6r4n6j-np-1-tvg6z-njbxt to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine w1u6b4w7n6r4n6j-np-1-tvg6z-kjbw8: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/autoscaling-cluster-dhxpqc2757nk--managed/providers/Microsoft.Compute/virtualMachines/w1u6b4w7n6r4n6j-np-1-tvg6z-kjbw8 to exist\nMachine w1u6b4w7n6r4n6j-np-1-tvg6z-njbxt: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/autoscaling-cluster-dhxpqc2757nk--managed/providers/Microsoft.Compute/virtualMachines/w1u6b4w7n6r4n6j-np-1-tvg6z-njbxt to exist\n"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://rp.j9438720.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/995ad93d-44a1-4aae-9f3c-cef4914b4415
correlation ID: d01faed1-7c94-4749-8892-78ade8753a54
client request ID: 5d2be0a3-7126-4924-b3ac-5c1c229ea51d
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_autoscaling.go:139]: failed to create nodepool "np-1" for autoscaling cluster
Unexpected error:
<*fmt.wrapError | 0x2a5fafb1fdc0>:
failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "autoscaling-hcp-cluster" in resourcegroup="autoscaling-cluster-dhxpqc2757nk" to finish creating: GET https://rp.j9438720.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/995ad93d-44a1-4aae-9f3c-cef4914b4415
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/995ad93d-44a1-4aae-9f3c-cef4914b4415",
"name": "995ad93d-44a1-4aae-9f3c-cef4914b4415",
"status": "Failed",
"startTime": "2026-10-25T11:09:27.17231924Z",
"endTime": "2026-10-25T11:28:33.403436193Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine w1u6b4w7n6r4n6j-np-1-tvg6z-kjbw8: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/autoscaling-cluster-dhxpqc2757nk--managed/providers/Microsoft.Compute/virtualMachines/w1u6b4w7n6r4n6j-np-1-tvg6z-kjbw8 to exist\nMachine w1u6b4w7n6r4n6j-np-1-tvg6z-njbxt: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/autoscaling-cluster-dhxpqc2757nk--managed/providers/Microsoft.Compute/virtualMachines/w1u6b4w7n6r4n6j-np-1-tvg6z-njbxt to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine w1u6b4w7n6r4n6j-np-1-tvg6z-kjbw8: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/autoscaling-cluster-dhxpqc2757nk--managed/providers/Microsoft.Compute/virtualMachines/w1u6b4w7n6r4n6j-np-1-tvg6z-kjbw8 to exist\nMachine w1u6b4w7n6r4n6j-np-1-tvg6z-njbxt: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/autoscaling-cluster-dhxpqc2757nk--managed/providers/Microsoft.Compute/virtualMachines/w1u6b4w7n6r4n6j-np-1-tvg6z-njbxt to exist\n"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://rp.j9438720.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/995ad93d-44a1-4aae-9f3c-cef4914b4415
correlation ID: d01faed1-7c94-4749-8892-78ade8753a54
client request ID: 5d2be0a3-7126-4924-b3ac-5c1c229ea51d
occurredfail [github.com/Azure/ARO-HCP/test/e2e/nodepool_labels_taints.go:141]: failed to create node pool np-1 with initial labels and taints
Unexpected error:
<*fmt.wrapError | 0x35e0b0894420>:
failed waiting for nodepool="np-1" for cluster "cluster-np-labels-taints" in resourcegroup="rg-np-labels-taints-9pzwhwthgd8h" to finish creating: GET https://rp.j2135936.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/4735d84e-eeda-4991-9a00-15bdfb0112df
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/4735d84e-eeda-4991-9a00-15bdfb0112df",
"name": "4735d84e-eeda-4991-9a00-15bdfb0112df",
"status": "Failed",
"startTime": "2026-10-23T22:59:37.48096278Z",
"endTime": "2026-10-23T23:18:39.961096304Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine r0o3j9w3o6f8u8z-np-1-x89sx-s9cv2: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-labels-taints-9pzwhwthgd8h--managed/providers/Microsoft.Compute/virtualMachines/r0o3j9w3o6f8u8z-np-1-x89sx-s9cv2 to exist\nMachine r0o3j9w3o6f8u8z-np-1-x89sx-xbh47: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-labels-taints-9pzwhwthgd8h--managed/providers/Microsoft.Compute/virtualMachines/r0o3j9w3o6f8u8z-np-1-x89sx-xbh47 to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine r0o3j9w3o6f8u8z-np-1-x89sx-s9cv2: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-labels-taints-9pzwhwthgd8h--managed/providers/Microsoft.Compute/virtualMachines/r0o3j9w3o6f8u8z-np-1-x89sx-s9cv2 to exist\nMachine r0o3j9w3o6f8u8z-np-1-x89sx-xbh47: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-labels-taints-9pzwhwthgd8h--managed/providers/Microsoft.Compute/virtualMachines/r0o3j9w3o6f8u8z-np-1-x89sx-xbh47 to exist\n"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://rp.j2135936.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/4735d84e-eeda-4991-9a00-15bdfb0112df
correlation ID: e810e9b2-3fca-4c24-b421-0b45eb552e0e
client request ID: 0af33e2e-2abb-488f-a355-27d28c97d1c5
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_labels_taints.go:141]: failed to create node pool np-1 with initial labels and taints
Unexpected error:
<*fmt.wrapError | 0x35e0b0894420>:
failed waiting for nodepool="np-1" for cluster "cluster-np-labels-taints" in resourcegroup="rg-np-labels-taints-9pzwhwthgd8h" to finish creating: GET https://rp.j2135936.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/4735d84e-eeda-4991-9a00-15bdfb0112df
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/4735d84e-eeda-4991-9a00-15bdfb0112df",
"name": "4735d84e-eeda-4991-9a00-15bdfb0112df",
"status": "Failed",
"startTime": "2026-10-23T22:59:37.48096278Z",
"endTime": "2026-10-23T23:18:39.961096304Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine r0o3j9w3o6f8u8z-np-1-x89sx-s9cv2: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-labels-taints-9pzwhwthgd8h--managed/providers/Microsoft.Compute/virtualMachines/r0o3j9w3o6f8u8z-np-1-x89sx-s9cv2 to exist\nMachine r0o3j9w3o6f8u8z-np-1-x89sx-xbh47: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-labels-taints-9pzwhwthgd8h--managed/providers/Microsoft.Compute/virtualMachines/r0o3j9w3o6f8u8z-np-1-x89sx-xbh47 to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine r0o3j9w3o6f8u8z-np-1-x89sx-s9cv2: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-labels-taints-9pzwhwthgd8h--managed/providers/Microsoft.Compute/virtualMachines/r0o3j9w3o6f8u8z-np-1-x89sx-s9cv2 to exist\nMachine r0o3j9w3o6f8u8z-np-1-x89sx-xbh47: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-labels-taints-9pzwhwthgd8h--managed/providers/Microsoft.Compute/virtualMachines/r0o3j9w3o6f8u8z-np-1-x89sx-xbh47 to exist\n"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://rp.j2135936.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/4735d84e-eeda-4991-9a00-15bdfb0112df
correlation ID: e810e9b2-3fca-4c24-b421-0b45eb552e0e
client request ID: 0af33e2e-2abb-488f-a355-27d28c97d1c5
occurredfail [github.com/Azure/ARO-HCP/test/e2e/cluster_fips_mode.go:103]: failed to create node pool "np-1" for fips-enabled cluster "fips-enabled-cluster"
Unexpected error:
<*fmt.wrapError | 0x2f37765b920>:
failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "fips-enabled-cluster" in resourcegroup="fips-enabled-7xsbkjfw6rsz" to finish creating: GET https://rp.j2736896.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/a4c6551c-33dc-4004-bb01-0a2aa772eecc
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/a4c6551c-33dc-4004-bb01-0a2aa772eecc",
"name": "a4c6551c-33dc-4004-bb01-0a2aa772eecc",
"status": "Failed",
"startTime": "2026-10-21T06:55:23.209858283Z",
"endTime": "2026-10-21T07:14:31.602256912Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine l0l3p9l4t2w4v4g-np-1-rqp7k-l24nh: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/fips-enabled-7xsbkjfw6rsz--managed/providers/Microsoft.Compute/virtualMachines/l0l3p9l4t2w4v4g-np-1-rqp7k-l24nh to exist\nMachine l0l3p9l4t2w4v4g-np-1-rqp7k-mhsz6: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/fips-enabled-7xsbkjfw6rsz--managed/providers/Microsoft.Compute/virtualMachines/l0l3p9l4t2w4v4g-np-1-rqp7k-mhsz6 to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine l0l3p9l4t2w4v4g-np-1-rqp7k-l24nh: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/fips-enabled-7xsbkjfw6rsz--managed/providers/Microsoft.Compute/virtualMachines/l0l3p9l4t2w4v4g-np-1-rqp7k-l24nh to exist\nMachine l0l3p9l4t2w4v4g-np-1-rqp7k-mhsz6: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/fips-enabled-7xsbkjfw6rsz--managed/providers/Microsoft.Compute/virtualMachines/l0l3p9l4t2w4v4g-np-1-rqp7k-mhsz6 to exist\n"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://rp.j2736896.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/a4c6551c-33dc-4004-bb01-0a2aa772eecc
correlation ID: 6ad3d41d-5643-43b0-aa39-e8dd37fe508a
client request ID: 2b4396d2-9eda-4e0c-b6dc-3f33c00bc172
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_fips_mode.go:103]: failed to create node pool "np-1" for fips-enabled cluster "fips-enabled-cluster"
Unexpected error:
<*fmt.wrapError | 0x2f37765b920>:
failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "fips-enabled-cluster" in resourcegroup="fips-enabled-7xsbkjfw6rsz" to finish creating: GET https://rp.j2736896.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/a4c6551c-33dc-4004-bb01-0a2aa772eecc
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/a4c6551c-33dc-4004-bb01-0a2aa772eecc",
"name": "a4c6551c-33dc-4004-bb01-0a2aa772eecc",
"status": "Failed",
"startTime": "2026-10-21T06:55:23.209858283Z",
"endTime": "2026-10-21T07:14:31.602256912Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine l0l3p9l4t2w4v4g-np-1-rqp7k-l24nh: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/fips-enabled-7xsbkjfw6rsz--managed/providers/Microsoft.Compute/virtualMachines/l0l3p9l4t2w4v4g-np-1-rqp7k-l24nh to exist\nMachine l0l3p9l4t2w4v4g-np-1-rqp7k-mhsz6: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/fips-enabled-7xsbkjfw6rsz--managed/providers/Microsoft.Compute/virtualMachines/l0l3p9l4t2w4v4g-np-1-rqp7k-mhsz6 to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine l0l3p9l4t2w4v4g-np-1-rqp7k-l24nh: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/fips-enabled-7xsbkjfw6rsz--managed/providers/Microsoft.Compute/virtualMachines/l0l3p9l4t2w4v4g-np-1-rqp7k-l24nh to exist\nMachine l0l3p9l4t2w4v4g-np-1-rqp7k-mhsz6: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/fips-enabled-7xsbkjfw6rsz--managed/providers/Microsoft.Compute/virtualMachines/l0l3p9l4t2w4v4g-np-1-rqp7k-mhsz6 to exist\n"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://rp.j2736896.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/a4c6551c-33dc-4004-bb01-0a2aa772eecc
correlation ID: 6ad3d41d-5643-43b0-aa39-e8dd37fe508a
client request ID: 2b4396d2-9eda-4e0c-b6dc-3f33c00bc172
occurredContributing tests (8)
Affected runs (11) | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
ERROR CODE: DeploymentFailed; detail code AzureKeyVaultKmsValidateIdentityPermissionCustomerError; detail message The identity does not have keys encrypt/decrypt permission on key...full failure pattern: ERROR CODE: DeploymentFailed; detail code AzureKeyVaultKmsValidateIdentityPermissionCustomerError; detail message The identity does not have keys encrypt/decrypt permission on key vault "<url>"; provider Microsoft.ContainerService Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 2 day(s) | provision | 10 | 1.01%10 of 995 job runs affected | IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 2 day(s) | Show trend detailsOct 12: 0 · Oct 13: 0 · Oct 14: 0 · Oct 15: 9 · Oct 16: 1 · Oct 17: 0 · Oct 18: 0 · Oct 19: 0 · Oct 20: 0 · Oct 21: 0 · Oct 22: 0 · Oct 23: 0 · Oct 24: 0 · Oct 25: 0 | none | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (3)time=2026-10-16T10:37:16.131Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster description="Step cluster\n Kind: ARM\n Template: templates/svc-cluster.bicep\n Parameters: configurations/svc-cluster.tmpl.bicepparam"
time=2026-10-16T10:37:17.576Z level=DEBUG+2 msg="Starting ARM deployment" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster
time=2026-10-16T10:37:18.879Z level=DEBUG+3 msg="Deployment started" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster deployment=182dddaef1eb2eeeaeb1c74f26cc60d61fef94e53fb090c30b83b72986af948d portal=https://ms.portal.azure.com/#view/Microsoft_Azure_Resources/DeploymentDetails.MenuView/~/overview/id/%2Fsubscriptions%2FXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX%2FresourceGroups%2Fhcp-underlay-ci01-j6554112-svc%2Fproviders%2FMicrosoft.Resources%2Fdeployments%2F182dddaef1eb2eeeaeb1c74f26cc60d61fef94e53fb090c30b83b72986af948d
time=2026-10-16T10:39:19.372Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster err="failed to run ARM step: failed to poll deployment: failed to wait for deployment completion: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j6554112-svc/providers/Microsoft.Resources/deployments/182dddaef1eb2eeeaeb1c74f26cc60d61fef94e53fb090c30b83b72986af948d/operationStatuses/08584094598472503244\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n \"status\": \"Failed\",\n \"error\": {\n \"code\": \"DeploymentFailed\",\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n \"details\": [\n {\n \"code\": \"Conflict\",\n \"message\": \"{\\r\\n \\\"status\\\": \\\"Failed\\\",\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j6554112-svc/providers/Microsoft.Resources/deployments/cluster-ismqifijm7koe\\\",\\r\\n \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j6554112-svc/providers/Microsoft.ContainerService/managedClusters/ci01-j6554112-svc\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"AzureKeyVaultKmsValidateIdentityPermissionCustomerError\\\",\\r\\n \\\"message\\\": \\\"The identity does not have keys encrypt/decrypt permission on key vault \\\\\\\"https://ah-ci01-se-j6554112-1.vault.azure.net\\\\\\\"\\\"\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n }\\r\\n}\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n"time=2026-10-15T22:37:03.691Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster description="Step cluster\n Kind: ARM\n Template: templates/svc-cluster.bicep\n Parameters: configurations/svc-cluster.tmpl.bicepparam"
time=2026-10-15T22:37:05.354Z level=DEBUG+2 msg="Starting ARM deployment" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster
time=2026-10-15T22:37:06.762Z level=DEBUG+3 msg="Deployment started" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster deployment=bee630fb16cd45492e943a7ab4f37f451fa3266ef665a6d74f650467bd60800f portal=https://ms.portal.azure.com/#view/Microsoft_Azure_Resources/DeploymentDetails.MenuView/~/overview/id/%2Fsubscriptions%2FXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX%2FresourceGroups%2Fhcp-underlay-ci01-j9396992-svc%2Fproviders%2FMicrosoft.Resources%2Fdeployments%2Fbee630fb16cd45492e943a7ab4f37f451fa3266ef665a6d74f650467bd60800f
time=2026-10-15T22:40:07.453Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster err="failed to run ARM step: failed to poll deployment: failed to wait for deployment completion: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j9396992-svc/providers/Microsoft.Resources/deployments/bee630fb16cd45492e943a7ab4f37f451fa3266ef665a6d74f650467bd60800f/operationStatuses/08584095030595181402\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n \"status\": \"Failed\",\n \"error\": {\n \"code\": \"DeploymentFailed\",\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n \"details\": [\n {\n \"code\": \"Conflict\",\n \"message\": \"{\\r\\n \\\"status\\\": \\\"Failed\\\",\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j9396992-svc/providers/Microsoft.Resources/deployments/cluster-kpgg4tugptwaw\\\",\\r\\n \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j9396992-svc/providers/Microsoft.ContainerService/managedClusters/ci01-j9396992-svc\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"AzureKeyVaultKmsValidateIdentityPermissionCustomerError\\\",\\r\\n \\\"message\\\": \\\"The identity does not have keys encrypt/decrypt permission on key vault \\\\\\\"https://ah-ci01-se-j9396992-1.vault.azure.net\\\\\\\"\\\"\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n }\\r\\n}\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n"time=2026-10-15T20:48:50.569Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster description="Step cluster\n Kind: ARM\n Template: templates/svc-cluster.bicep\n Parameters: configurations/svc-cluster.tmpl.bicepparam"
time=2026-10-15T20:48:52.496Z level=DEBUG+2 msg="Starting ARM deployment" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster
time=2026-10-15T20:48:54.249Z level=DEBUG+3 msg="Deployment started" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster deployment=8276bf3e2ac415cdaf52404f9c09132aafef64d9fdc663f4e7aca3248fabbce7 portal=https://ms.portal.azure.com/#view/Microsoft_Azure_Resources/DeploymentDetails.MenuView/~/overview/id/%2Fsubscriptions%2FXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX%2FresourceGroups%2Fhcp-underlay-ci01-j3682688-svc%2Fproviders%2FMicrosoft.Resources%2Fdeployments%2F8276bf3e2ac415cdaf52404f9c09132aafef64d9fdc663f4e7aca3248fabbce7
time=2026-10-15T20:51:24.632Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster err="failed to run ARM step: failed to poll deployment: failed to wait for deployment completion: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j3682688-svc/providers/Microsoft.Resources/deployments/8276bf3e2ac415cdaf52404f9c09132aafef64d9fdc663f4e7aca3248fabbce7/operationStatuses/08584095095521617911\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n \"status\": \"Failed\",\n \"error\": {\n \"code\": \"DeploymentFailed\",\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n \"details\": [\n {\n \"code\": \"Conflict\",\n \"message\": \"{\\r\\n \\\"status\\\": \\\"Failed\\\",\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j3682688-svc/providers/Microsoft.Resources/deployments/cluster-a22genk7v3fe6\\\",\\r\\n \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j3682688-svc/providers/Microsoft.ContainerService/managedClusters/ci01-j3682688-svc\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"AzureKeyVaultKmsValidateIdentityPermissionCustomerError\\\",\\r\\n \\\"message\\\": \\\"The identity does not have keys encrypt/decrypt permission on key vault \\\\\\\"https://ah-ci01-se-j3682688-1.vault.azure.net\\\\\\\"\\\"\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n }\\r\\n}\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n"Contributing tests (1)
Affected runs (10) | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Environment: INT
Daily Run Outcomes
No tests below 95.00% in this window with at least 10 runs.
Up to 50 failure patterns are loaded (minimum 1.00% impact), with 10 shown by default. Click column headers to re-sort.
Jump to the full Failure Patterns view for this window
| Failure patternThe canonical description of a recurring CI failure, extracted and normalized from raw logs. | Failed atThe stage of the job run where this failure occurred: 'provision' (environment setup, DEV only), 'e2e' (test suite execution), or 'other' (CI infrastructure issues that did not produce a failure pattern). | Number of distinct job runs where this failure pattern was detected. | Percentage of all job runs in this environment affected by this failure pattern during the selected window. | Classification of this failure pattern: Regression (likely caused by a specific PR), Flake (intermittent failure spread across days), Noise (low-quality or generic pattern), or Indeterminate. | TrendShows daily activity for this failure pattern in a trailing window anchored to the selected end date. The sparkline covers at least 7 days and at most 14 days, depending on the current window size. | Also inOther environments where the same failure pattern was also detected during the selected window. | |||||||||||||||||||||||||||||||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
tls: failed to verify certificate: x509: certificate signed by unknown authorityfull failure pattern: tls: failed to verify certificate: x509: certificate signed by unknown authority Signal: IndeterminateSignal: Indeterminate — no prior history; active 2 day(s) — New failure pattern — no prior history | unknown | 12 | 41.38%12 of 29 job runs affected | IndeterminateSignal: Indeterminate — no prior history; active 2 day(s) — New failure pattern — no prior history | Show trend detailsOct 12: 0 · Oct 13: 0 · Oct 14: 0 · Oct 15: 0 · Oct 16: 0 · Oct 17: 0 · Oct 18: 0 · Oct 19: 0 · Oct 20: 0 · Oct 21: 0 · Oct 22: 0 · Oct 23: 5 · Oct 24: 7 · Oct 25: 0 | none | |||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (3)fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:194]: failed to verify simple web app runs on cluster "cluster-candidate-4-23-l96rfk"
Unexpected error:
<*fmt.wrapError | 0x11cfecaf8760>:
[strict TLS verification] route was never reachable: Get "https://agnhost-e2e-sample-app-4vg4t.apps.aro.o6e4m4u5u4t4c4u.sbp8.westus3.aroapp-hcp.azure-test.net": tls: failed to verify certificate: x509: certificate signed by unknown authority
{
msg: "[strict TLS verification] route was never reachable: Get \"https://agnhost-e2e-sample-app-4vg4t.apps.aro.o6e4m4u5u4t4c4u.sbp8.westus3.aroapp-hcp.azure-test.net\": tls: failed to verify certificate: x509: certificate signed by unknown authority",
err: <*url.Error | 0x11cfee5e70e0>{
Op: "Get",
URL: "https://agnhost-e2e-sample-app-4vg4t.apps.aro.o6e4m4u5u4t4c4u.sbp8.westus3.aroapp-hcp.azure-test.net",
Err: <*tls.CertificateVerificationError | 0x11cfee5e70b0>{
UnverifiedCertificates: [
{
Raw: [48, 130, 4, 1, 48, 130, 2, 233, 160, 3, 2, 1, 2, 2, 8, 122, 194, 216, 171, 111, 107, 55, 112, 48, 13, 6, 9, 42, 134, 72, 134, 247, 13, 1, 1, 11, 5, 0, 48, 38, 49, 18, 48, 16, 6, 3, 85, 4, 11, 19, 9, 111, 112, 101, 110, 115, 104, 105, 102, 116, 49, 16, 48, 14, 6, 3, 85, 4, 3, 19, 7, 114, 111, 111, 116, 45, 99, 97, 48, 30, 23, 13, 50, 54, 49, 48, 50, 52, 50, 50, 50, 56, 48, 49, 90, 23, 13, 50, 55, 49, 48, 50, 52, 50, 50, 50, 56, 48, 49, 90, 48, 48, 49, 18, 48, 16, 6, 3, 85, 4, 10, 19, 9, 111, 112, 101, 110, 115, 104, 105, 102, 116, 49, 26, 48, 24, 6, 3, 85, 4, 3, 19, 17, 111, 112, 101, 110, 115, 104, 105, 102, 116, 45, 105, 110, 103, 114, 101, 115, 115, 48, 130, 1, 34, 48, 13, 6, 9, 42, 134, 72, 134, 247, 13, 1, 1, 1, 5, 0, 3, 130, 1, 15, 0, 48, 130, 1, 10, 2, 130, 1, 1, 0, 178, 55, 149, 119, 219, 63, 47, 5, 249, 2, 142, 64, 226, 181, 25, 19, 163, 170, 217, 26, 124, 161, 29, 250, 162, 212, 197, 31, 10, 179, 100, 47, 54, 215, 80, 7, 78, 121, 50, 101, 254, 19, 69, 126, 231, 240, 34, 4, 205, 235, 198, 28, 58, 139, 209, 113, 230, 105, 161, 163, 221, 192, 72, 27, 82, 64, 156, 231, 230, 222, 216, 170, 66, 81, 109, 14, 218, 138, 180, 42, 46, 37, 12, 20, 42, 46, 254, 75, 14, 239, 200, 146, 1, 44, 249, 109, 167, 6, 174, 142, 209, 55, 87, 205, 253, 244, 165, 78, 181, 174, 211, 88, 111, 195, 22, 44, 169, 29, 62, 197, 215, 209, 73, 64, 144, 182, 37, 230, 141, 3, 209, 58, 41, 137, 165, 79, 15, 206, 195, 232, 245, 252, 140, 232, 188, 179, 203, 150, 101, 191, 186, 47, 212, 214, 37, 240, 83, 70, 206, 45, 54, 33, 211, 136, 55, 189, 43, 255, 208, 60, 224, 146, 245, 105, 10, 7, 134, 154, 110, 155, 81, 215, 26, 170, 120, 141, 123, 32, 172, 156, 94, 207, 62, 225, 195, 61, 155, 50, 127, 235, 92, 32, 207, 220, 201, 84, 27, 1, 117, 152, 204, 160, 80, 33, 0, 199, 209, 234, 168, 90, 184, 171, 178, 224, 8, 13, 41, 49, 94, 136, 52, 225, 159, 183, 109, 30, 220, 54, 37, 247, 46, 253, 174, 26, 197, 35, 142, 122, 171, 24, 218, 246, 148, 31, 58, 235, 2, 3, 1, 0, 1, 163, 130, 1, 39, 48, 130, 1, 35, 48, 14, 6, 3, 85, 29, 15, 1, 1, 255, 4, 4, 3, 2, 5, 160, 48, 29, 6, 3, 85, 29, 37, 4, 22, 48, 20, 6, 8, 43, 6, 1, 5, 5, 7, 3, 2, 6, 8, 43, 6, 1, 5, 5, 7, 3, 1, 48, 12, 6, 3, 85, 29, 19, 1, 1, 255, 4, 2, 48, 0, 48, 73, 6, 3, 85, 29, 14, 4, 66, 4, 64, 197, 98, 86, 250, 137, 217, 107, 201, 120, 11, 250, 28, 8, 159, 50, 58, 147, 2, 127, 27, 39, 46, 176, 228, 89, 113, 119, 156, 18, 254, 186, 225, 239, 236, 201, 198, 127, 141, 38, 73, 33, 245, 173, 2, 193, 175, 61, 245, 132, 223, 97, 52, 79, 55, 120, 201, 63, 76, 209, 109, 45, 189, 34, 122, 48, 75, 6, 3, 85, 29, 35, 4, 68, 48, 66, 128, 64, 248, 50, 11, 119, 242, 227, 194, 167, 149, 215, 190, 179, 176, 143, 74, 56, 221, 252, 111, 212, 224, 237, 204, 6, 0, 197, 97, 180, 119, 93, 170, 15, 222, 152, 153, 8, 168, 38, 140, 186, 14, 89, 130, 247, 23, 168, 143, 191, 156, 56, 163, 177, 216, 127, 102, 115, 15, 90, 73, 84, 30, 110, 35, 172, 48, 76, 6, 3, 85, 29, 17, 4, 69, 48, 67, 130, 65, 42, 46, 97, 112, 112, 115, 46, 97, 114, 111, 46, 111, 54, 101, 52, 109, 52, 117, 53, 117, 52, 116, 52, 99, 52, 117, 46, 115, 98, 112, 56, 46, 119, 101, 115, 116, 117, 115, 51, 46, 97, 114, 111, 97, 112, 112, 45, 104, 99, 112, 46, 97, 122, 117, 114, 101, 45, 116, 101, 115, 116, 46, 110, 101, 116, 48, 13, 6, 9, 42, 134, 72, 134, 247, 13, 1, 1, 11, 5, 0, 3, 130, 1, 1, 0, 50, 25, 231, 199, 177, 69, 58, 16, 51, 91, 243, 219, 210, 199, 138, 155, 53, 245, 245, 106, 127, 215, 113, 116, 53, 212, 199, 147, 106, 103, 247, 189, 33, 171, 252, 21, 143, 150, 56, 31, 85, 0, 41, 147, 65, 24, 201, 140, 203, 150, 3, 31, 110, 137, 35, 75, 171, 175, 235, 209, 21, 26, 164, 179, 71, 67, 21, 133, 105, 14, 253, 71, 51, 45, 32, 43, 100, 94, 84, 47, 64, 244, 56, 58, 136, 157, 46, 87, 211, 40, 31, 180, 2, 165, 196, 210, 222, 204, 152, 251, 130, 232, 9, 182, 120, 239, 235, 22, 89, 58, 198, 78, 82, 139, 120, 162, 101, 8, 223, 107, 59, 223, 193, 151, 252, 30, 79, 0, 15, 40, 112, 104, 59, 157, 68, 154, 197, 178, 218, 209, 164, 56, 94, 93, 22, 22, 98, 191, 126, 106, 4, 194, 233, 190, 68, 193, 123, 232, 86, 187, 65, 26, 135, 65, 91, 129, 105, 54, 92, 78, 247, 245, 172, 13, 38, 206, 186, 5, 136, 153, 211, 200, 140, 153, 205, 176, 93, 247, 111, 230, 227, 13, 56, 73, 251, 143, 236, 121, 113, 183, 124, 61, 169, 206, 40, 146, 37, 166, 252, 132, 4, 153, 168, 84, 39, 23, 232, 218, 136, 80, 210, 143, 97, 106, 58, 193, 56, 253, 19, 202, 243, 199, 49, 235, 83, 136, 105, 217, 57, 123, 238, 192, 105, 226, 211, 224, 255, 115, 130, 221, 177, 40, 19, 51, 31, 249],
RawTBSCertificate: [48, 130, 2, 233, 160, 3, 2, 1, 2, 2, 8, 122, 194, 216, 171, 111, 107, 55, 112, 48, 13, 6, 9, 42, 134, 72, 134, 247, 13, 1, 1, 11, 5, 0, 48, 38, 49, 18, 48, 16, 6, 3, 85, 4, 11, 19, 9, 111, 112, 101, 110, 115, 104, 105, 102, 116, 49, 16, 48, 14, 6, 3, 85, 4, 3, 19, 7, 114, 111, 111, 116, 45, 99, 97, 48, 30, 23, 13, 50, 54, 49, 48, 50, 52, 50, 50, 50, 56, 48, 49, 90, 23, 13, 50, 55, 49, 48, 50, 52, 50, 50, 50, 56, 48, 49, 90, 48, 48, 49, 18, 48, 16, 6, 3, 85, 4, 10, 19, 9, 111, 112, 101, 110, 115, 104, 105, 102, 116, 49, 26, 48, 24, 6, 3, 85, 4, 3, 19, 17, 111, 112, 101, 110, 115, 104, 105, 102, 116, 45, 105, 110, 103, 114, 101, 115, 115, 48, 130, 1, 34, 48, 13, 6, 9, 42, 134, 72, 134, 247, 13, 1, 1, 1, 5, 0, 3, 130, 1, 15, 0, 48, 130, 1, 10, 2, 130, 1, 1, 0, 178, 55, 149, 119, 219, 63, 47, 5, 249, 2, 142, 64, 226, 181, 25, 19, 163, 170, 217, 26, 124, 161, 29, 250, 162, 212, 197, 31, 10, 179, 100, 47, 54, 215, 80, 7, 78, 121, 50, 101, 254, 19, 69, 126, 231, 240, 34, 4, 205, 235, 198, 28, 58, 139, 209, 113, 230, 105, 161, 163, 221, 192, 72, 27, 82, 64, 156, 231, 230, 222, 216, 170, 66, 81, 109, 14, 218, 138, 180, 42, 46, 37, 12, 20, 42, 46, 254, 75, 14, 239, 200, 146, 1, 44, 249, 109, 167, 6, 174, 142, 209, 55, 87, 205, 253, 244, 165, 78, 181, 174, 211, 88, 111, 195, 22, 44, 169, 29, 62, 197, 215, 209, 73, 64, 144, 182, 37, 230, 141, 3, 209, 58, 41, 137, 165, 79, 15, 206, 195, 232, 245, 252, 140, 232, 188, 179, 203, 150, 101, 191, 186, 47, 212, 214, 37, 240, 83, 70, 206, 45, 54, 33, 211, 136, 55, 189, 43, 255, 208, 60, 224, 146, 245, 105, 10, 7, 134, 154, 110, 155, 81, 215, 26, 170, 120, 141, 123, 32, 172, 156, 94, 207, 62, 225, 195, 61, 155, 50, 127, 235, 92, 32, 207, 220, 201, 84, 27, 1, 117, 152, 204, 160, 80, 33, 0, 199, 209, 234, 168, 90, 184, 171, 178, 224, 8, 13, 41, 49, 94, 136, 52, 225, 159, 183, 109, 30, 220, 54, 37, 247, 46, 253, 174, 26, 197, 35, 142, 122, 171, 24, 218, 246, 148, 31, 58, 235, 2, 3, 1, 0, 1, 163, 130, 1, 39, 48, 130, 1, 35, 48, 14, 6, 3, 85, 29, 15, 1, 1, 255, 4, 4, 3, 2, 5, 160, 48, 29, 6, 3, 85, 29, 37, 4, 22, 48, 20, 6, 8, 43, 6, 1, 5, 5, 7, 3, 2, 6, 8, 43, 6, 1, 5, 5, 7, 3, 1, 48, 12, 6, 3, 85, 29, 19, 1, 1, 255, 4, 2, 48, 0, 48, 73, 6, 3, 85, 29, 14, 4, 66, 4, 64, 197, 98, 86, 250, 137, 217, 107, 201, 120, 11, 250, 28, 8, 159, 50, 58, 147, 2, 127, 27, 39, 46, 176, 228, 89, 113, 119, 156, 18, 254, 186, 225, 239, 236, 201, 198, 127, 141, 38, 73, 33, 245, 173, 2, 193, 175, 61, 245, 132, 223, 97, 52, 79, 55, 120, 201, 63, 76, 209, 109, 45, 189, 34, 122, 48, 75, 6, 3, 85, 29, 35, 4, 68, 48, 66, 128, 64, 248, 50, 11, 119, 242, 227, 194, 167, 149, 215, 190, 179, 176, 143, 74, 56, 221, 252, 111, 212, 224, 237, 204, 6, 0, 197, 97, 180, 119, 93, 170, 15, 222, 152, 153, 8, 168, 38, 140, 186, 14, 89, 130, 247, 23, 168, 143, 191, 156, 56, 163, 177, 216, 127, 102, 115, 15, 90, 73, 84, 30, 110, 35, 172, 48, 76, 6, 3, 85, 29, 17, 4, 69, 48, 67, 130, 65, 42, 46, 97, 112, 112, 115, 46, 97, 114, 111, 46, 111, 54, 101, 52, 109, 52, 117, 53, 117, 52, 116, 52, 99, 52, 117, 46, 115, 98, 112, 56, 46, 119, 101, 115, 116, 117, 115, 51, 46, 97, 114, 111, 97, 112, 112, 45, 104, 99, 112, 46, 97, 122, 117, 114, 101, 45, 116, 101, 115, 116, 46, 110, 101, 116],
RawSubjectPublicKeyInfo: [48, 130, 1, 34, 48, 13, 6, 9, 42, 134, 72, 134, 247, 13, 1, 1, 1, 5, 0, 3, 130, 1, 15, 0, 48, 130, 1, 10, 2, 130, 1, 1, 0, 178, 55, 149, 119, 219, 63, 47, 5, 249, 2, 142, 64, 226, 181, 25, 19, 163, 170, 217, 26, 124, 161, 29, 250, 162, 212, 197, 31, 10, 179, 100, 47, 54, 215, 80, 7, 78, 121, 50, 101, 254, 19, 69, 126, 231, 240, 34, 4, 205, 235, 198, 28, 58, 139, 209, 113, 230, 105, 161, 163, 221, 192, 72, 27, 82, 64, 156, 231, 230, 222, 216, 170, 66, 81, 109, 14, 218, 138, 180, 42, 46, 37, 12, 20, 42, 46, 254, 75, 14, 239, 200, 146, 1, 44, 249, 109, 167, 6, 174, 142, 209, 55, 87, 205, 253, 244, 165, 78, 181, 174, 211, 88, 111, 195, 22, 44, 169, 29, 62, 197, 215, 209, 73, 64, 144, 182, 37, 230, 141, 3, 209, 58, 41, 137, 165, 79, 15, 206, 195, 232, 245, 252, 140, 232, 188, 179, 203, 150, 101, 191, 186, 47, 212, 214, 37, 240, 83, 70, 206, 45, 54, 33, 211, 136, 55, 189, 43, 255, 208, 60, 224, 146, 245, 105, 10, 7, 134, 154, 110, 155, 81, 215, 26, 170, 120, 141, 123, 32, 172, 156, 94, 207, 62, 225, 195, 61, 155, 50, 127, 235, 92, 32, 207, 220, 201, 84, 27, 1, 117, 152, 204, 160, 80, 33, 0, 199, 209, 234, 168, 90, 184, 171, 178, 224, 8, 13, 41, 49, 94, 136, 52, 225, 159, 183, 109, 30, 220, 54, 37, 247, 46, 253, 174, 26, 197, 35, 142, 122, 171, 24, 218, 246, 148, 31, 58, 235, 2, 3, 1, 0, 1],
RawSubject: [48, 48, 49, 18, 48, 16, 6, 3, 85, 4, 10, 19, 9, 111, 112, 101, 110, 115, 104, 105, 102, 116, 49, 26, 48, 24, 6, 3, 85, 4, 3, 19, 17, 111, 112, 101, 110, 115, 104, 105, 102, 116, 45, 105, 110, 103, 114, 101, 115, 115],
RawIssuer: [48, 38, 49, 18, 48, 16, 6, 3, 85, 4, 11, 19, 9, 111, 112, 101, 110, 115, 104, 105, 102, 116, 49, 16, 48, 14, 6, 3, 85, 4, 3, 19, 7, 114, 111, 111, 116, 45, 99, 97],
Signature: [50, 25, 231, 199, 177, 69, 58, 16, 51, 91, 243, 219, 210, 199, 138, 155, 53, 245, 245, 106, 127, 215, 113, 116, 53, 212, 199, 147, 106, 103, 247, 189, 33, 171, 252, 21, 143, 150, 56, 31, 85, 0, 41, 147, 65, 24, 201, 140, 203, 150, 3, 31, 110, 137, 35, 75, 171, 175, 235, 209, 21, 26, 164, 179, 71, 67, 21, 133, 105, 14, 253, 71, 51, 45, 32, 43, 100, 94, 84, 47, 64, 244, 56, 58, 136, 157, 46, 87, 211, 40, 31, 180, 2, 165, 196, 210, 222, 204, 152, 251, 130, 232, 9, 182, 120, 239, 235, 22, 89, 58, 198, 78, 82, 139, 120, 162, 101, 8, 223, 107, 59, 223, 193, 151, 252, 30, 79, 0, 15, 40, 112, 104, 59, 157, 68, 154, 197, 178, 218, 209, 164, 56, 94, 93, 22, 22, 98, 191, 126, 106, 4, 194, 233, 190, 68, 193, 123, 232, 86, 187, 65, 26, 135, 65, 91, 129, 105, 54, 92, 78, 247, 245, 172, 13, 38, 206, 186, 5, 136, 153, 211, 200, 140, 153, 205, 176, 93, 247, 111, 230, 227, 13, 56, 73, 251, 143, 236, 121, 113, 183, 124, 61, 169, 206, 40, 146, 37, 166, 252, 132, 4, 153, 168, 84, 39, 23, 232, 218, 136, 80, 210, 143, 97, 106, 58, 193, 56, 253, 19, 202, 243, 199, 49, 235, 83, 136, 105, 217, 57, 123, 238, 192, 105, 226, 211, 224, 255, 115, 130, 221, 177, 40, 19, 51, 31, 249],
SignatureAlgorithm: 4,
PublicKeyAlgorithm: 1,
PublicKey: <*rsa.PublicKey | 0x11cfee823170>{
N: {
neg: false,
abs: [12328844732508617451, 3386053924852305530, 11508787352079574519, 580165219016783073, 15126087406076015328, 8473747687808762055, 6638534397921598209, 4531117369475629035, 8686734636532653775, 9699186360867822250, 15005114980278798855, 3900631334283717631, 15336487295176330797, 13597435546774256175, 1139063161098112232, 10161195232013165903, 15551291529941624294, 8053304939740217029, 18299432943387333464, 12035498982612883405, 1076299365375342957, 3325077180629581387, 4778720590180299818, 5926909629444315306, 16602979226450151451, 14838171221845856625, 18308053321550668292, 3951715190352458341, 11733219666321171503, 11793477282006375930, 17943060274730047763, 12841897204482256645],
},
E: 65537,
},
Version: 3,
SerialNumber: {
neg: false,
abs: [8845870848929380208],
},
Issuer: {
Country: nil,
Organization: nil,
OrganizationalUnit: ["openshift"],
Locality: nil,
Province: nil,
StreetAddress: nil,
PostalCode: nil,
SerialNumber: "",
CommonName: "root-ca",
Names: [
{
Type: [2, 5, 4, 11],
Value: <string>"openshift",
},
{
Type: [2, 5, 4, 3],
Value: <string>"root-ca",
},
],
ExtraNames: nil,
},
Subject: {
Country: nil,
Organization: ["openshift"],
OrganizationalUnit: nil,
Locality: nil,
Province: nil,
StreetAddress: nil,
PostalCode: nil,
SerialNumber: "",
CommonName: "openshift-ingress",
Names: [
{
Type: [2, 5, 4, 10],
Value: <string>"openshift",
},
{
Type: [2, 5, 4, 3],
Value: <string>"openshift-ingress",
},
],
ExtraNames: nil,
},
NotBefore: {wall: 0, ext: 63928477681, loc: nil},
NotAfter: {wall: 0, ext: 63960013681, loc: nil},
KeyUsage: 5,
Extensions: [
{
Id: [2, 5, 29, 15],
Critical: true,
Value: [3, 2, 5, 160],
},
{
Id: [2, 5, 29, 37],
Critical: false,
Value: [48, 20, 6, 8, 43, 6, 1, 5, 5, 7, 3, 2, 6, 8, 43, 6, 1, 5, 5, 7, 3, 1],
},
{Id: [2, 5, 29, 19], Critical: true, Value: [48, 0]},
{
Id: [2, 5, 29, 14],
Critical: false,
Value: [4, 64, 197, 98, 86, 250, 137, 217, 107, 201, 120, 11, 250, 28, 8, 159, 50, 58, 147, 2, 127, 27, 39, 46, 176, 228, 89, 113, 119, 156, 18, 254, 186, 225, 239, 236, 201, 198, 127, 141, 38, 73, 33, 245, 173, 2, 193, 175, 61, 245, 132, 223, 97, 52, 79, 55, 120, 201, 63, 76, 209, 109, 45, 189, 34, 122],
},
{
Id: [2, 5, 29, 35],
Critical: false,
Value: [48, 66, 128, 64, 248, 50, 11, 119, 242, 227, 194, 167, 149, 215, 190, 179, 176, 143, 74, 56, 221, 252, 111, 212, 224, 237, 204, 6, 0, 197, 97, 180, 119, 93, 170, 15, 222, 152, 153, 8, 168, 38, 140, 186, 14, 89, 130, 247, 23, 168, 143, 191, 156, 56, 163, 177, 216, 127, 102, 115, 15, 90, 73, 84, 30, 110, 35, 172],
},
{
Id: [2, 5, 29, 17],
Critical: false,
Value: "0C\x82A*.apps.aro.o6e4m4u5u4t4c4u.sbp8.westus3.aroapp-hcp.azure-test.net",
},
],
ExtraExtensions: nil,
UnhandledCriticalExtensions: nil,
ExtKeyUsage: [2, 1],
UnknownExtKeyUsage: nil,
BasicConstraintsValid: true,
IsCA: false,
MaxPathLen: -1,
MaxPathLenZero: false,
SubjectKeyId: [197, 98, 86, 250, 137, 217, 107, 201, 120, 11, 250, 28, 8, 159, 50, 58, 147, 2, 127, 27, 39, 46, 176, 228, 89, 113, 119, 156, 18, 254, 186, 225, 239, 236, 201, 198, 127, 141, 38, 73, 33, 245, 173, 2, 193, 175, 61, 245, 132, 223, 97, 52, 79, 55, 120, 201, 63, 76, 209, 109, 45, 189, 34, 122],
AuthorityKeyId: [248, 50, 11, 119, 242, 227, 194, 167, 149, 215, 190, 179, 176, 143, 74, 56, 221, 252, 111, 212, 224, 237, 204, 6, 0, 197, 97, 180, 119, 93, 170, 15, 222, 152, 153, 8, 168, 38, 140, 186, 14, 89, 130, 247, 23, 168, 143, 191, 156, 56, 163, 177, 216, 127, 102, 115, 15, 90, 73, 84, 30, 110, 35, 172],
OCSPServer: nil,
IssuingCertificateURL: nil,
DNSNames: [
"*.apps.aro.o6e4m4u5u4t4c4u.sbp8.westus3.aroapp-hcp.azure-test.net",
],
EmailAddresses: nil,
IPAddresses: nil,
URIs: nil,
PermittedDNSDomainsCritical: false,
PermittedDNSDomains: nil,
ExcludedDNSDomains: nil,
PermittedIPRanges: nil,
ExcludedIPRanges: nil,
PermittedEmailAddresses: nil,
ExcludedEmailAddresses: nil,
PermittedURIDomains: nil,
ExcludedURIDomains: nil,
CRLDistributionPoints: nil,
PolicyIdentifiers: nil,
Policies: nil,
InhibitAnyPolicy: 0,
InhibitAnyPolicyZero: false,
InhibitPolicyMapping: 0,
InhibitPolicyMappingZero: false,
RequireExplicitPolicy: 0,
RequireExplicitPolicyZero: false,
PolicyMappings: nil,
},
],
Err: <x509.UnknownAuthorityError>{
Cert: 0x11cfed040008 (cyclic reference),
hintErr: nil,
hintCert: nil,
},
},
},
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:194]: failed to verify simple web app runs on cluster "cluster-candidate-4-23-l96rfk"
Unexpected error:
<*fmt.wrapError | 0x11cfecaf8760>:
[strict TLS verification] route was never reachable: Get "https://agnhost-e2e-sample-app-4vg4t.apps.aro.o6e4m4u5u4t4c4u.sbp8.westus3.aroapp-hcp.azure-test.net": tls: failed to verify certificate: x509: certificate signed by unknown authority
{
msg: "[strict TLS verification] route was never reachable: Get \"https://agnhost-e2e-sample-app-4vg4t.apps.aro.o6e4m4u5u4t4c4u.sbp8.westus3.aroapp-hcp.azure-test.net\": tls: failed to verify certificate: x509: certificate signed by unknown authority",
err: <*url.Error | 0x11cfee5e70e0>{
Op: "Get",
URL: "https://agnhost-e2e-sample-app-4vg4t.apps.aro.o6e4m4u5u4t4c4u.sbp8.westus3.aroapp-hcp.azure-test.net",
Err: <*tls.CertificateVerificationError | 0x11cfee5e70b0>{
UnverifiedCertificates: [
{
Raw: [48, 130, 4, 1, 48, 130, 2, 233, 160, 3, 2, 1, 2, 2, 8, 122, 194, 216, 171, 111, 107, 55, 112, 48, 13, 6, 9, 42, 134, 72, 134, 247, 13, 1, 1, 11, 5, 0, 48, 38, 49, 18, 48, 16, 6, 3, 85, 4, 11, 19, 9, 111, 112, 101, 110, 115, 104, 105, 102, 116, 49, 16, 48, 14, 6, 3, 85, 4, 3, 19, 7, 114, 111, 111, 116, 45, 99, 97, 48, 30, 23, 13, 50, 54, 49, 48, 50, 52, 50, 50, 50, 56, 48, 49, 90, 23, 13, 50, 55, 49, 48, 50, 52, 50, 50, 50, 56, 48, 49, 90, 48, 48, 49, 18, 48, 16, 6, 3, 85, 4, 10, 19, 9, 111, 112, 101, 110, 115, 104, 105, 102, 116, 49, 26, 48, 24, 6, 3, 85, 4, 3, 19, 17, 111, 112, 101, 110, 115, 104, 105, 102, 116, 45, 105, 110, 103, 114, 101, 115, 115, 48, 130, 1, 34, 48, 13, 6, 9, 42, 134, 72, 134, 247, 13, 1, 1, 1, 5, 0, 3, 130, 1, 15, 0, 48, 130, 1, 10, 2, 130, 1, 1, 0, 178, 55, 149, 119, 219, 63, 47, 5, 249, 2, 142, 64, 226, 181, 25, 19, 163, 170, 217, 26, 124, 161, 29, 250, 162, 212, 197, 31, 10, 179, 100, 47, 54, 215, 80, 7, 78, 121, 50, 101, 254, 19, 69, 126, 231, 240, 34, 4, 205, 235, 198, 28, 58, 139, 209, 113, 230, 105, 161, 163, 221, 192, 72, 27, 82, 64, 156, 231, 230, 222, 216, 170, 66, 81, 109, 14, 218, 138, 180, 42, 46, 37, 12, 20, 42, 46, 254, 75, 14, 239, 200, 146, 1, 44, 249, 109, 167, 6, 174, 142, 209, 55, 87, 205, 253, 244, 165, 78, 181, 174, 211, 88, 111, 195, 22, 44, 169, 29, 62, 197, 215, 209, 73, 64, 144, 182, 37, 230, 141, 3, 209, 58, 41, 137, 165, 79, 15, 206, 195, 232, 245, 252, 140, 232, 188, 179, 203, 150, 101, 191, 186, 47, 212, 214, 37, 240, 83, 70, 206, 45, 54, 33, 211, 136, 55, 189, 43, 255, 208, 60, 224, 146, 245, 105, 10, 7, 134, 154, 110, 155, 81, 215, 26, 170, 120, 141, 123, 32, 172, 156, 94, 207, 62, 225, 195, 61, 155, 50, 127, 235, 92, 32, 207, 220, 201, 84, 27, 1, 117, 152, 204, 160, 80, 33, 0, 199, 209, 234, 168, 90, 184, 171, 178, 224, 8, 13, 41, 49, 94, 136, 52, 225, 159, 183, 109, 30, 220, 54, 37, 247, 46, 253, 174, 26, 197, 35, 142, 122, 171, 24, 218, 246, 148, 31, 58, 235, 2, 3, 1, 0, 1, 163, 130, 1, 39, 48, 130, 1, 35, 48, 14, 6, 3, 85, 29, 15, 1, 1, 255, 4, 4, 3, 2, 5, 160, 48, 29, 6, 3, 85, 29, 37, 4, 22, 48, 20, 6, 8, 43, 6, 1, 5, 5, 7, 3, 2, 6, 8, 43, 6, 1, 5, 5, 7, 3, 1, 48, 12, 6, 3, 85, 29, 19, 1, 1, 255, 4, 2, 48, 0, 48, 73, 6, 3, 85, 29, 14, 4, 66, 4, 64, 197, 98, 86, 250, 137, 217, 107, 201, 120, 11, 250, 28, 8, 159, 50, 58, 147, 2, 127, 27, 39, 46, 176, 228, 89, 113, 119, 156, 18, 254, 186, 225, 239, 236, 201, 198, 127, 141, 38, 73, 33, 245, 173, 2, 193, 175, 61, 245, 132, 223, 97, 52, 79, 55, 120, 201, 63, 76, 209, 109, 45, 189, 34, 122, 48, 75, 6, 3, 85, 29, 35, 4, 68, 48, 66, 128, 64, 248, 50, 11, 119, 242, 227, 194, 167, 149, 215, 190, 179, 176, 143, 74, 56, 221, 252, 111, 212, 224, 237, 204, 6, 0, 197, 97, 180, 119, 93, 170, 15, 222, 152, 153, 8, 168, 38, 140, 186, 14, 89, 130, 247, 23, 168, 143, 191, 156, 56, 163, 177, 216, 127, 102, 115, 15, 90, 73, 84, 30, 110, 35, 172, 48, 76, 6, 3, 85, 29, 17, 4, 69, 48, 67, 130, 65, 42, 46, 97, 112, 112, 115, 46, 97, 114, 111, 46, 111, 54, 101, 52, 109, 52, 117, 53, 117, 52, 116, 52, 99, 52, 117, 46, 115, 98, 112, 56, 46, 119, 101, 115, 116, 117, 115, 51, 46, 97, 114, 111, 97, 112, 112, 45, 104, 99, 112, 46, 97, 122, 117, 114, 101, 45, 116, 101, 115, 116, 46, 110, 101, 116, 48, 13, 6, 9, 42, 134, 72, 134, 247, 13, 1, 1, 11, 5, 0, 3, 130, 1, 1, 0, 50, 25, 231, 199, 177, 69, 58, 16, 51, 91, 243, 219, 210, 199, 138, 155, 53, 245, 245, 106, 127, 215, 113, 116, 53, 212, 199, 147, 106, 103, 247, 189, 33, 171, 252, 21, 143, 150, 56, 31, 85, 0, 41, 147, 65, 24, 201, 140, 203, 150, 3, 31, 110, 137, 35, 75, 171, 175, 235, 209, 21, 26, 164, 179, 71, 67, 21, 133, 105, 14, 253, 71, 51, 45, 32, 43, 100, 94, 84, 47, 64, 244, 56, 58, 136, 157, 46, 87, 211, 40, 31, 180, 2, 165, 196, 210, 222, 204, 152, 251, 130, 232, 9, 182, 120, 239, 235, 22, 89, 58, 198, 78, 82, 139, 120, 162, 101, 8, 223, 107, 59, 223, 193, 151, 252, 30, 79, 0, 15, 40, 112, 104, 59, 157, 68, 154, 197, 178, 218, 209, 164, 56, 94, 93, 22, 22, 98, 191, 126, 106, 4, 194, 233, 190, 68, 193, 123, 232, 86, 187, 65, 26, 135, 65, 91, 129, 105, 54, 92, 78, 247, 245, 172, 13, 38, 206, 186, 5, 136, 153, 211, 200, 140, 153, 205, 176, 93, 247, 111, 230, 227, 13, 56, 73, 251, 143, 236, 121, 113, 183, 124, 61, 169, 206, 40, 146, 37, 166, 252, 132, 4, 153, 168, 84, 39, 23, 232, 218, 136, 80, 210, 143, 97, 106, 58, 193, 56, 253, 19, 202, 243, 199, 49, 235, 83, 136, 105, 217, 57, 123, 238, 192, 105, 226, 211, 224, 255, 115, 130, 221, 177, 40, 19, 51, 31, 249],
RawTBSCertificate: [48, 130, 2, 233, 160, 3, 2, 1, 2, 2, 8, 122, 194, 216, 171, 111, 107, 55, 112, 48, 13, 6, 9, 42, 134, 72, 134, 247, 13, 1, 1, 11, 5, 0, 48, 38, 49, 18, 48, 16, 6, 3, 85, 4, 11, 19, 9, 111, 112, 101, 110, 115, 104, 105, 102, 116, 49, 16, 48, 14, 6, 3, 85, 4, 3, 19, 7, 114, 111, 111, 116, 45, 99, 97, 48, 30, 23, 13, 50, 54, 49, 48, 50, 52, 50, 50, 50, 56, 48, 49, 90, 23, 13, 50, 55, 49, 48, 50, 52, 50, 50, 50, 56, 48, 49, 90, 48, 48, 49, 18, 48, 16, 6, 3, 85, 4, 10, 19, 9, 111, 112, 101, 110, 115, 104, 105, 102, 116, 49, 26, 48, 24, 6, 3, 85, 4, 3, 19, 17, 111, 112, 101, 110, 115, 104, 105, 102, 116, 45, 105, 110, 103, 114, 101, 115, 115, 48, 130, 1, 34, 48, 13, 6, 9, 42, 134, 72, 134, 247, 13, 1, 1, 1, 5, 0, 3, 130, 1, 15, 0, 48, 130, 1, 10, 2, 130, 1, 1, 0, 178, 55, 149, 119, 219, 63, 47, 5, 249, 2, 142, 64, 226, 181, 25, 19, 163, 170, 217, 26, 124, 161, 29, 250, 162, 212, 197, 31, 10, 179, 100, 47, 54, 215, 80, 7, 78, 121, 50, 101, 254, 19, 69, 126, 231, 240, 34, 4, 205, 235, 198, 28, 58, 139, 209, 113, 230, 105, 161, 163, 221, 192, 72, 27, 82, 64, 156, 231, 230, 222, 216, 170, 66, 81, 109, 14, 218, 138, 180, 42, 46, 37, 12, 20, 42, 46, 254, 75, 14, 239, 200, 146, 1, 44, 249, 109, 167, 6, 174, 142, 209, 55, 87, 205, 253, 244, 165, 78, 181, 174, 211, 88, 111, 195, 22, 44, 169, 29, 62, 197, 215, 209, 73, 64, 144, 182, 37, 230, 141, 3, 209, 58, 41, 137, 165, 79, 15, 206, 195, 232, 245, 252, 140, 232, 188, 179, 203, 150, 101, 191, 186, 47, 212, 214, 37, 240, 83, 70, 206, 45, 54, 33, 211, 136, 55, 189, 43, 255, 208, 60, 224, 146, 245, 105, 10, 7, 134, 154, 110, 155, 81, 215, 26, 170, 120, 141, 123, 32, 172, 156, 94, 207, 62, 225, 195, 61, 155, 50, 127, 235, 92, 32, 207, 220, 201, 84, 27, 1, 117, 152, 204, 160, 80, 33, 0, 199, 209, 234, 168, 90, 184, 171, 178, 224, 8, 13, 41, 49, 94, 136, 52, 225, 159, 183, 109, 30, 220, 54, 37, 247, 46, 253, 174, 26, 197, 35, 142, 122, 171, 24, 218, 246, 148, 31, 58, 235, 2, 3, 1, 0, 1, 163, 130, 1, 39, 48, 130, 1, 35, 48, 14, 6, 3, 85, 29, 15, 1, 1, 255, 4, 4, 3, 2, 5, 160, 48, 29, 6, 3, 85, 29, 37, 4, 22, 48, 20, 6, 8, 43, 6, 1, 5, 5, 7, 3, 2, 6, 8, 43, 6, 1, 5, 5, 7, 3, 1, 48, 12, 6, 3, 85, 29, 19, 1, 1, 255, 4, 2, 48, 0, 48, 73, 6, 3, 85, 29, 14, 4, 66, 4, 64, 197, 98, 86, 250, 137, 217, 107, 201, 120, 11, 250, 28, 8, 159, 50, 58, 147, 2, 127, 27, 39, 46, 176, 228, 89, 113, 119, 156, 18, 254, 186, 225, 239, 236, 201, 198, 127, 141, 38, 73, 33, 245, 173, 2, 193, 175, 61, 245, 132, 223, 97, 52, 79, 55, 120, 201, 63, 76, 209, 109, 45, 189, 34, 122, 48, 75, 6, 3, 85, 29, 35, 4, 68, 48, 66, 128, 64, 248, 50, 11, 119, 242, 227, 194, 167, 149, 215, 190, 179, 176, 143, 74, 56, 221, 252, 111, 212, 224, 237, 204, 6, 0, 197, 97, 180, 119, 93, 170, 15, 222, 152, 153, 8, 168, 38, 140, 186, 14, 89, 130, 247, 23, 168, 143, 191, 156, 56, 163, 177, 216, 127, 102, 115, 15, 90, 73, 84, 30, 110, 35, 172, 48, 76, 6, 3, 85, 29, 17, 4, 69, 48, 67, 130, 65, 42, 46, 97, 112, 112, 115, 46, 97, 114, 111, 46, 111, 54, 101, 52, 109, 52, 117, 53, 117, 52, 116, 52, 99, 52, 117, 46, 115, 98, 112, 56, 46, 119, 101, 115, 116, 117, 115, 51, 46, 97, 114, 111, 97, 112, 112, 45, 104, 99, 112, 46, 97, 122, 117, 114, 101, 45, 116, 101, 115, 116, 46, 110, 101, 116],
RawSubjectPublicKeyInfo: [48, 130, 1, 34, 48, 13, 6, 9, 42, 134, 72, 134, 247, 13, 1, 1, 1, 5, 0, 3, 130, 1, 15, 0, 48, 130, 1, 10, 2, 130, 1, 1, 0, 178, 55, 149, 119, 219, 63, 47, 5, 249, 2, 142, 64, 226, 181, 25, 19, 163, 170, 217, 26, 124, 161, 29, 250, 162, 212, 197, 31, 10, 179, 100, 47, 54, 215, 80, 7, 78, 121, 50, 101, 254, 19, 69, 126, 231, 240, 34, 4, 205, 235, 198, 28, 58, 139, 209, 113, 230, 105, 161, 163, 221, 192, 72, 27, 82, 64, 156, 231, 230, 222, 216, 170, 66, 81, 109, 14, 218, 138, 180, 42, 46, 37, 12, 20, 42, 46, 254, 75, 14, 239, 200, 146, 1, 44, 249, 109, 167, 6, 174, 142, 209, 55, 87, 205, 253, 244, 165, 78, 181, 174, 211, 88, 111, 195, 22, 44, 169, 29, 62, 197, 215, 209, 73, 64, 144, 182, 37, 230, 141, 3, 209, 58, 41, 137, 165, 79, 15, 206, 195, 232, 245, 252, 140, 232, 188, 179, 203, 150, 101, 191, 186, 47, 212, 214, 37, 240, 83, 70, 206, 45, 54, 33, 211, 136, 55, 189, 43, 255, 208, 60, 224, 146, 245, 105, 10, 7, 134, 154, 110, 155, 81, 215, 26, 170, 120, 141, 123, 32, 172, 156, 94, 207, 62, 225, 195, 61, 155, 50, 127, 235, 92, 32, 207, 220, 201, 84, 27, 1, 117, 152, 204, 160, 80, 33, 0, 199, 209, 234, 168, 90, 184, 171, 178, 224, 8, 13, 41, 49, 94, 136, 52, 225, 159, 183, 109, 30, 220, 54, 37, 247, 46, 253, 174, 26, 197, 35, 142, 122, 171, 24, 218, 246, 148, 31, 58, 235, 2, 3, 1, 0, 1],
RawSubject: [48, 48, 49, 18, 48, 16, 6, 3, 85, 4, 10, 19, 9, 111, 112, 101, 110, 115, 104, 105, 102, 116, 49, 26, 48, 24, 6, 3, 85, 4, 3, 19, 17, 111, 112, 101, 110, 115, 104, 105, 102, 116, 45, 105, 110, 103, 114, 101, 115, 115],
RawIssuer: [48, 38, 49, 18, 48, 16, 6, 3, 85, 4, 11, 19, 9, 111, 112, 101, 110, 115, 104, 105, 102, 116, 49, 16, 48, 14, 6, 3, 85, 4, 3, 19, 7, 114, 111, 111, 116, 45, 99, 97],
Signature: [50, 25, 231, 199, 177, 69, 58, 16, 51, 91, 243, 219, 210, 199, 138, 155, 53, 245, 245, 106, 127, 215, 113, 116, 53, 212, 199, 147, 106, 103, 247, 189, 33, 171, 252, 21, 143, 150, 56, 31, 85, 0, 41, 147, 65, 24, 201, 140, 203, 150, 3, 31, 110, 137, 35, 75, 171, 175, 235, 209, 21, 26, 164, 179, 71, 67, 21, 133, 105, 14, 253, 71, 51, 45, 32, 43, 100, 94, 84, 47, 64, 244, 56, 58, 136, 157, 46, 87, 211, 40, 31, 180, 2, 165, 196, 210, 222, 204, 152, 251, 130, 232, 9, 182, 120, 239, 235, 22, 89, 58, 198, 78, 82, 139, 120, 162, 101, 8, 223, 107, 59, 223, 193, 151, 252, 30, 79, 0, 15, 40, 112, 104, 59, 157, 68, 154, 197, 178, 218, 209, 164, 56, 94, 93, 22, 22, 98, 191, 126, 106, 4, 194, 233, 190, 68, 193, 123, 232, 86, 187, 65, 26, 135, 65, 91, 129, 105, 54, 92, 78, 247, 245, 172, 13, 38, 206, 186, 5, 136, 153, 211, 200, 140, 153, 205, 176, 93, 247, 111, 230, 227, 13, 56, 73, 251, 143, 236, 121, 113, 183, 124, 61, 169, 206, 40, 146, 37, 166, 252, 132, 4, 153, 168, 84, 39, 23, 232, 218, 136, 80, 210, 143, 97, 106, 58, 193, 56, 253, 19, 202, 243, 199, 49, 235, 83, 136, 105, 217, 57, 123, 238, 192, 105, 226, 211, 224, 255, 115, 130, 221, 177, 40, 19, 51, 31, 249],
SignatureAlgorithm: 4,
PublicKeyAlgorithm: 1,
PublicKey: <*rsa.PublicKey | 0x11cfee823170>{
N: {
neg: false,
abs: [12328844732508617451, 3386053924852305530, 11508787352079574519, 580165219016783073, 15126087406076015328, 8473747687808762055, 6638534397921598209, 4531117369475629035, 8686734636532653775, 9699186360867822250, 15005114980278798855, 3900631334283717631, 15336487295176330797, 13597435546774256175, 1139063161098112232, 10161195232013165903, 15551291529941624294, 8053304939740217029, 18299432943387333464, 12035498982612883405, 1076299365375342957, 3325077180629581387, 4778720590180299818, 5926909629444315306, 16602979226450151451, 14838171221845856625, 18308053321550668292, 3951715190352458341, 11733219666321171503, 11793477282006375930, 17943060274730047763, 12841897204482256645],
},
E: 65537,
},
Version: 3,
SerialNumber: {
neg: false,
abs: [8845870848929380208],
},
Issuer: {
Country: nil,
Organization: nil,
OrganizationalUnit: ["openshift"],
Locality: nil,
Province: nil,
StreetAddress: nil,
PostalCode: nil,
SerialNumber: "",
CommonName: "root-ca",
Names: [
{
Type: [2, 5, 4, 11],
Value: <string>"openshift",
},
{
Type: [2, 5, 4, 3],
Value: <string>"root-ca",
},
],
ExtraNames: nil,
},
Subject: {
Country: nil,
Organization: ["openshift"],
OrganizationalUnit: nil,
Locality: nil,
Province: nil,
StreetAddress: nil,
PostalCode: nil,
SerialNumber: "",
CommonName: "openshift-ingress",
Names: [
{
Type: [2, 5, 4, 10],
Value: <string>"openshift",
},
{
Type: [2, 5, 4, 3],
Value: <string>"openshift-ingress",
},
],
ExtraNames: nil,
},
NotBefore: {wall: 0, ext: 63928477681, loc: nil},
NotAfter: {wall: 0, ext: 63960013681, loc: nil},
KeyUsage: 5,
Extensions: [
{
Id: [2, 5, 29, 15],
Critical: true,
Value: [3, 2, 5, 160],
},
{
Id: [2, 5, 29, 37],
Critical: false,
Value: [48, 20, 6, 8, 43, 6, 1, 5, 5, 7, 3, 2, 6, 8, 43, 6, 1, 5, 5, 7, 3, 1],
},
{Id: [2, 5, 29, 19], Critical: true, Value: [48, 0]},
{
Id: [2, 5, 29, 14],
Critical: false,
Value: [4, 64, 197, 98, 86, 250, 137, 217, 107, 201, 120, 11, 250, 28, 8, 159, 50, 58, 147, 2, 127, 27, 39, 46, 176, 228, 89, 113, 119, 156, 18, 254, 186, 225, 239, 236, 201, 198, 127, 141, 38, 73, 33, 245, 173, 2, 193, 175, 61, 245, 132, 223, 97, 52, 79, 55, 120, 201, 63, 76, 209, 109, 45, 189, 34, 122],
},
{
Id: [2, 5, 29, 35],
Critical: false,
Value: [48, 66, 128, 64, 248, 50, 11, 119, 242, 227, 194, 167, 149, 215, 190, 179, 176, 143, 74, 56, 221, 252, 111, 212, 224, 237, 204, 6, 0, 197, 97, 180, 119, 93, 170, 15, 222, 152, 153, 8, 168, 38, 140, 186, 14, 89, 130, 247, 23, 168, 143, 191, 156, 56, 163, 177, 216, 127, 102, 115, 15, 90, 73, 84, 30, 110, 35, 172],
},
{
Id: [2, 5, 29, 17],
Critical: false,
Value: "0C\x82A*.apps.aro.o6e4m4u5u4t4c4u.sbp8.westus3.aroapp-hcp.azure-test.net",
},
],
ExtraExtensions: nil,
UnhandledCriticalExtensions: nil,
ExtKeyUsage: [2, 1],
UnknownExtKeyUsage: nil,
BasicConstraintsValid: true,
IsCA: false,
MaxPathLen: -1,
MaxPathLenZero: false,
SubjectKeyId: [197, 98, 86, 250, 137, 217, 107, 201, 120, 11, 250, 28, 8, 159, 50, 58, 147, 2, 127, 27, 39, 46, 176, 228, 89, 113, 119, 156, 18, 254, 186, 225, 239, 236, 201, 198, 127, 141, 38, 73, 33, 245, 173, 2, 193, 175, 61, 245, 132, 223, 97, 52, 79, 55, 120, 201, 63, 76, 209, 109, 45, 189, 34, 122],
AuthorityKeyId: [248, 50, 11, 119, 242, 227, 194, 167, 149, 215, 190, 179, 176, 143, 74, 56, 221, 252, 111, 212, 224, 237, 204, 6, 0, 197, 97, 180, 119, 93, 170, 15, 222, 152, 153, 8, 168, 38, 140, 186, 14, 89, 130, 247, 23, 168, 143, 191, 156, 56, 163, 177, 216, 127, 102, 115, 15, 90, 73, 84, 30, 110, 35, 172],
OCSPServer: nil,
IssuingCertificateURL: nil,
DNSNames: [
"*.apps.aro.o6e4m4u5u4t4c4u.sbp8.westus3.aroapp-hcp.azure-test.net",
],
EmailAddresses: nil,
IPAddresses: nil,
URIs: nil,
PermittedDNSDomainsCritical: false,
PermittedDNSDomains: nil,
ExcludedDNSDomains: nil,
PermittedIPRanges: nil,
ExcludedIPRanges: nil,
PermittedEmailAddresses: nil,
ExcludedEmailAddresses: nil,
PermittedURIDomains: nil,
ExcludedURIDomains: nil,
CRLDistributionPoints: nil,
PolicyIdentifiers: nil,
Policies: nil,
InhibitAnyPolicy: 0,
InhibitAnyPolicyZero: false,
InhibitPolicyMapping: 0,
InhibitPolicyMappingZero: false,
RequireExplicitPolicy: 0,
RequireExplicitPolicyZero: false,
PolicyMappings: nil,
},
],
Err: <x509.UnknownAuthorityError>{
Cert: 0x11cfed040008 (cyclic reference),
hintErr: nil,
hintCert: nil,
},
},
},
}
occurredfail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:194]: failed to verify simple web app runs on cluster "cluster-candidate-4-23-2rcw86"
Unexpected error:
<*fmt.wrapError | 0x2b2e578984a0>:
[strict TLS verification] route was never reachable: Get "https://agnhost-e2e-sample-app-w86zx.apps.aro.a2s9b8m2e4p7p0g.0mu8.westus3.aroapp-hcp.azure-test.net": tls: failed to verify certificate: x509: certificate signed by unknown authority
{
msg: "[strict TLS verification] route was never reachable: Get \"https://agnhost-e2e-sample-app-w86zx.apps.aro.a2s9b8m2e4p7p0g.0mu8.westus3.aroapp-hcp.azure-test.net\": tls: failed to verify certificate: x509: certificate signed by unknown authority",
err: <*url.Error | 0x2b2e57370780>{
Op: "Get",
URL: "https://agnhost-e2e-sample-app-w86zx.apps.aro.a2s9b8m2e4p7p0g.0mu8.westus3.aroapp-hcp.azure-test.net",
Err: <*tls.CertificateVerificationError | 0x2b2e57370750>{
UnverifiedCertificates: [
{
Raw: [48, 130, 4, 1, 48, 130, 2, 233, 160, 3, 2, 1, 2, 2, 8, 124, 87, 70, 122, 130, 6, 100, 82, 48, 13, 6, 9, 42, 134, 72, 134, 247, 13, 1, 1, 11, 5, 0, 48, 38, 49, 18, 48, 16, 6, 3, 85, 4, 11, 19, 9, 111, 112, 101, 110, 115, 104, 105, 102, 116, 49, 16, 48, 14, 6, 3, 85, 4, 3, 19, 7, 114, 111, 111, 116, 45, 99, 97, 48, 30, 23, 13, 50, 54, 49, 48, 50, 52, 50, 49, 48, 56, 48, 56, 90, 23, 13, 50, 55, 49, 48, 50, 52, 50, 49, 48, 56, 48, 56, 90, 48, 48, 49, 18, 48, 16, 6, 3, 85, 4, 10, 19, 9, 111, 112, 101, 110, 115, 104, 105, 102, 116, 49, 26, 48, 24, 6, 3, 85, 4, 3, 19, 17, 111, 112, 101, 110, 115, 104, 105, 102, 116, 45, 105, 110, 103, 114, 101, 115, 115, 48, 130, 1, 34, 48, 13, 6, 9, 42, 134, 72, 134, 247, 13, 1, 1, 1, 5, 0, 3, 130, 1, 15, 0, 48, 130, 1, 10, 2, 130, 1, 1, 0, 184, 6, 158, 64, 94, 24, 96, 40, 160, 154, 141, 196, 128, 87, 218, 59, 39, 71, 199, 234, 79, 240, 146, 24, 254, 106, 245, 45, 201, 136, 214, 64, 254, 110, 56, 158, 251, 103, 159, 125, 70, 9, 9, 192, 157, 185, 88, 45, 237, 209, 14, 238, 22, 11, 156, 42, 61, 172, 248, 175, 205, 29, 87, 44, 249, 69, 148, 194, 201, 88, 31, 36, 152, 175, 207, 120, 70, 231, 234, 239, 27, 120, 11, 188, 47, 126, 21, 187, 83, 61, 218, 253, 69, 150, 57, 242, 31, 129, 90, 78, 122, 162, 115, 193, 66, 231, 239, 171, 129, 241, 209, 248, 79, 198, 197, 15, 5, 20, 97, 114, 135, 82, 183, 39, 26, 12, 204, 167, 245, 131, 126, 229, 14, 152, 249, 66, 23, 145, 179, 201, 129, 84, 180, 47, 247, 191, 198, 219, 235, 14, 55, 141, 41, 250, 184, 135, 128, 115, 246, 220, 161, 224, 164, 119, 161, 199, 4, 69, 138, 127, 197, 71, 128, 167, 90, 212, 150, 2, 13, 143, 216, 191, 58, 228, 209, 93, 22, 58, 9, 137, 127, 203, 163, 74, 40, 217, 253, 186, 51, 153, 40, 52, 238, 73, 60, 57, 57, 108, 128, 2, 48, 134, 115, 108, 142, 65, 158, 113, 75, 25, 249, 189, 39, 130, 154, 24, 112, 209, 120, 229, 129, 223, 190, 147, 169, 232, 115, 83, 39, 47, 230, 201, 87, 216, 98, 161, 27, 95, 39, 237, 157, 102, 109, 222, 170, 149, 2, 3, 1, 0, 1, 163, 130, 1, 39, 48, 130, 1, 35, 48, 14, 6, 3, 85, 29, 15, 1, 1, 255, 4, 4, 3, 2, 5, 160, 48, 29, 6, 3, 85, 29, 37, 4, 22, 48, 20, 6, 8, 43, 6, 1, 5, 5, 7, 3, 2, 6, 8, 43, 6, 1, 5, 5, 7, 3, 1, 48, 12, 6, 3, 85, 29, 19, 1, 1, 255, 4, 2, 48, 0, 48, 73, 6, 3, 85, 29, 14, 4, 66, 4, 64, 43, 61, 184, 58, 190, 246, 117, 145, 150, 149, 52, 148, 50, 70, 171, 24, 140, 56, 164, 203, 105, 63, 65, 146, 183, 66, 114, 129, 170, 239, 140, 79, 13, 223, 192, 116, 206, 195, 234, 70, 154, 102, 164, 7, 77, 72, 8, 99, 67, 120, 61, 224, 48, 163, 84, 4, 250, 254, 27, 88, 242, 13, 99, 21, 48, 75, 6, 3, 85, 29, 35, 4, 68, 48, 66, 128, 64, 145, 115, 72, 248, 147, 3, 210, 10, 72, 90, 51, 130, 36, 255, 101, 123, 45, 151, 78, 158, 148, 254, 229, 16, 121, 180, 193, 131, 183, 127, 36, 88, 142, 33, 22, 114, 228, 61, 202, 29, 164, 17, 211, 221, 107, 244, 245, 206, 36, 16, 177, 96, 20, 74, 22, 17, 184, 132, 100, 119, 214, 222, 88, 253, 48, 76, 6, 3, 85, 29, 17, 4, 69, 48, 67, 130, 65, 42, 46, 97, 112, 112, 115, 46, 97, 114, 111, 46, 97, 50, 115, 57, 98, 56, 109, 50, 101, 52, 112, 55, 112, 48, 103, 46, 48, 109, 117, 56, 46, 119, 101, 115, 116, 117, 115, 51, 46, 97, 114, 111, 97, 112, 112, 45, 104, 99, 112, 46, 97, 122, 117, 114, 101, 45, 116, 101, 115, 116, 46, 110, 101, 116, 48, 13, 6, 9, 42, 134, 72, 134, 247, 13, 1, 1, 11, 5, 0, 3, 130, 1, 1, 0, 144, 63, 1, 9, 63, 203, 104, 158, 72, 190, 218, 173, 218, 174, 171, 175, 249, 99, 198, 163, 154, 183, 80, 70, 18, 90, 143, 16, 110, 40, 74, 15, 152, 104, 155, 119, 3, 189, 92, 182, 181, 193, 112, 193, 179, 171, 177, 195, 218, 45, 246, 105, 209, 209, 4, 147, 233, 109, 252, 165, 52, 200, 179, 47, 64, 202, 66, 155, 170, 48, 184, 165, 51, 160, 106, 229, 210, 44, 72, 109, 204, 32, 9, 38, 222, 130, 253, 176, 6, 246, 50, 104, 240, 45, 124, 77, 233, 84, 164, 109, 154, 110, 171, 77, 164, 253, 134, 225, 58, 131, 116, 48, 78, 1, 85, 150, 81, 149, 67, 171, 102, 216, 126, 229, 90, 56, 26, 33, 171, 124, 161, 209, 140, 187, 61, 34, 212, 215, 17, 134, 62, 6, 13, 208, 221, 55, 5, 187, 22, 63, 93, 164, 76, 124, 166, 239, 104, 210, 84, 78, 146, 114, 146, 113, 215, 97, 33, 13, 186, 135, 110, 182, 194, 44, 64, 215, 36, 43, 178, 21, 170, 171, 251, 180, 100, 40, 239, 189, 24, 174, 192, 195, 98, 108, 222, 96, 135, 172, 56, 114, 81, 213, 239, 84, 199, 27, 118, 56, 193, 239, 243, 191, 178, 97, 106, 201, 66, 252, 31, 190, 254, 107, 226, 184, 19, 188, 113, 44, 99, 24, 245, 52, 154, 97, 249, 178, 221, 232, 203, 63, 51, 4, 109, 167, 92, 246, 143, 188, 32, 32, 87, 98, 186, 5, 69, 17],
RawTBSCertificate: [48, 130, 2, 233, 160, 3, 2, 1, 2, 2, 8, 124, 87, 70, 122, 130, 6, 100, 82, 48, 13, 6, 9, 42, 134, 72, 134, 247, 13, 1, 1, 11, 5, 0, 48, 38, 49, 18, 48, 16, 6, 3, 85, 4, 11, 19, 9, 111, 112, 101, 110, 115, 104, 105, 102, 116, 49, 16, 48, 14, 6, 3, 85, 4, 3, 19, 7, 114, 111, 111, 116, 45, 99, 97, 48, 30, 23, 13, 50, 54, 49, 48, 50, 52, 50, 49, 48, 56, 48, 56, 90, 23, 13, 50, 55, 49, 48, 50, 52, 50, 49, 48, 56, 48, 56, 90, 48, 48, 49, 18, 48, 16, 6, 3, 85, 4, 10, 19, 9, 111, 112, 101, 110, 115, 104, 105, 102, 116, 49, 26, 48, 24, 6, 3, 85, 4, 3, 19, 17, 111, 112, 101, 110, 115, 104, 105, 102, 116, 45, 105, 110, 103, 114, 101, 115, 115, 48, 130, 1, 34, 48, 13, 6, 9, 42, 134, 72, 134, 247, 13, 1, 1, 1, 5, 0, 3, 130, 1, 15, 0, 48, 130, 1, 10, 2, 130, 1, 1, 0, 184, 6, 158, 64, 94, 24, 96, 40, 160, 154, 141, 196, 128, 87, 218, 59, 39, 71, 199, 234, 79, 240, 146, 24, 254, 106, 245, 45, 201, 136, 214, 64, 254, 110, 56, 158, 251, 103, 159, 125, 70, 9, 9, 192, 157, 185, 88, 45, 237, 209, 14, 238, 22, 11, 156, 42, 61, 172, 248, 175, 205, 29, 87, 44, 249, 69, 148, 194, 201, 88, 31, 36, 152, 175, 207, 120, 70, 231, 234, 239, 27, 120, 11, 188, 47, 126, 21, 187, 83, 61, 218, 253, 69, 150, 57, 242, 31, 129, 90, 78, 122, 162, 115, 193, 66, 231, 239, 171, 129, 241, 209, 248, 79, 198, 197, 15, 5, 20, 97, 114, 135, 82, 183, 39, 26, 12, 204, 167, 245, 131, 126, 229, 14, 152, 249, 66, 23, 145, 179, 201, 129, 84, 180, 47, 247, 191, 198, 219, 235, 14, 55, 141, 41, 250, 184, 135, 128, 115, 246, 220, 161, 224, 164, 119, 161, 199, 4, 69, 138, 127, 197, 71, 128, 167, 90, 212, 150, 2, 13, 143, 216, 191, 58, 228, 209, 93, 22, 58, 9, 137, 127, 203, 163, 74, 40, 217, 253, 186, 51, 153, 40, 52, 238, 73, 60, 57, 57, 108, 128, 2, 48, 134, 115, 108, 142, 65, 158, 113, 75, 25, 249, 189, 39, 130, 154, 24, 112, 209, 120, 229, 129, 223, 190, 147, 169, 232, 115, 83, 39, 47, 230, 201, 87, 216, 98, 161, 27, 95, 39, 237, 157, 102, 109, 222, 170, 149, 2, 3, 1, 0, 1, 163, 130, 1, 39, 48, 130, 1, 35, 48, 14, 6, 3, 85, 29, 15, 1, 1, 255, 4, 4, 3, 2, 5, 160, 48, 29, 6, 3, 85, 29, 37, 4, 22, 48, 20, 6, 8, 43, 6, 1, 5, 5, 7, 3, 2, 6, 8, 43, 6, 1, 5, 5, 7, 3, 1, 48, 12, 6, 3, 85, 29, 19, 1, 1, 255, 4, 2, 48, 0, 48, 73, 6, 3, 85, 29, 14, 4, 66, 4, 64, 43, 61, 184, 58, 190, 246, 117, 145, 150, 149, 52, 148, 50, 70, 171, 24, 140, 56, 164, 203, 105, 63, 65, 146, 183, 66, 114, 129, 170, 239, 140, 79, 13, 223, 192, 116, 206, 195, 234, 70, 154, 102, 164, 7, 77, 72, 8, 99, 67, 120, 61, 224, 48, 163, 84, 4, 250, 254, 27, 88, 242, 13, 99, 21, 48, 75, 6, 3, 85, 29, 35, 4, 68, 48, 66, 128, 64, 145, 115, 72, 248, 147, 3, 210, 10, 72, 90, 51, 130, 36, 255, 101, 123, 45, 151, 78, 158, 148, 254, 229, 16, 121, 180, 193, 131, 183, 127, 36, 88, 142, 33, 22, 114, 228, 61, 202, 29, 164, 17, 211, 221, 107, 244, 245, 206, 36, 16, 177, 96, 20, 74, 22, 17, 184, 132, 100, 119, 214, 222, 88, 253, 48, 76, 6, 3, 85, 29, 17, 4, 69, 48, 67, 130, 65, 42, 46, 97, 112, 112, 115, 46, 97, 114, 111, 46, 97, 50, 115, 57, 98, 56, 109, 50, 101, 52, 112, 55, 112, 48, 103, 46, 48, 109, 117, 56, 46, 119, 101, 115, 116, 117, 115, 51, 46, 97, 114, 111, 97, 112, 112, 45, 104, 99, 112, 46, 97, 122, 117, 114, 101, 45, 116, 101, 115, 116, 46, 110, 101, 116],
RawSubjectPublicKeyInfo: [48, 130, 1, 34, 48, 13, 6, 9, 42, 134, 72, 134, 247, 13, 1, 1, 1, 5, 0, 3, 130, 1, 15, 0, 48, 130, 1, 10, 2, 130, 1, 1, 0, 184, 6, 158, 64, 94, 24, 96, 40, 160, 154, 141, 196, 128, 87, 218, 59, 39, 71, 199, 234, 79, 240, 146, 24, 254, 106, 245, 45, 201, 136, 214, 64, 254, 110, 56, 158, 251, 103, 159, 125, 70, 9, 9, 192, 157, 185, 88, 45, 237, 209, 14, 238, 22, 11, 156, 42, 61, 172, 248, 175, 205, 29, 87, 44, 249, 69, 148, 194, 201, 88, 31, 36, 152, 175, 207, 120, 70, 231, 234, 239, 27, 120, 11, 188, 47, 126, 21, 187, 83, 61, 218, 253, 69, 150, 57, 242, 31, 129, 90, 78, 122, 162, 115, 193, 66, 231, 239, 171, 129, 241, 209, 248, 79, 198, 197, 15, 5, 20, 97, 114, 135, 82, 183, 39, 26, 12, 204, 167, 245, 131, 126, 229, 14, 152, 249, 66, 23, 145, 179, 201, 129, 84, 180, 47, 247, 191, 198, 219, 235, 14, 55, 141, 41, 250, 184, 135, 128, 115, 246, 220, 161, 224, 164, 119, 161, 199, 4, 69, 138, 127, 197, 71, 128, 167, 90, 212, 150, 2, 13, 143, 216, 191, 58, 228, 209, 93, 22, 58, 9, 137, 127, 203, 163, 74, 40, 217, 253, 186, 51, 153, 40, 52, 238, 73, 60, 57, 57, 108, 128, 2, 48, 134, 115, 108, 142, 65, 158, 113, 75, 25, 249, 189, 39, 130, 154, 24, 112, 209, 120, 229, 129, 223, 190, 147, 169, 232, 115, 83, 39, 47, 230, 201, 87, 216, 98, 161, 27, 95, 39, 237, 157, 102, 109, 222, 170, 149, 2, 3, 1, 0, 1],
RawSubject: [48, 48, 49, 18, 48, 16, 6, 3, 85, 4, 10, 19, 9, 111, 112, 101, 110, 115, 104, 105, 102, 116, 49, 26, 48, 24, 6, 3, 85, 4, 3, 19, 17, 111, 112, 101, 110, 115, 104, 105, 102, 116, 45, 105, 110, 103, 114, 101, 115, 115],
RawIssuer: [48, 38, 49, 18, 48, 16, 6, 3, 85, 4, 11, 19, 9, 111, 112, 101, 110, 115, 104, 105, 102, 116, 49, 16, 48, 14, 6, 3, 85, 4, 3, 19, 7, 114, 111, 111, 116, 45, 99, 97],
Signature: [144, 63, 1, 9, 63, 203, 104, 158, 72, 190, 218, 173, 218, 174, 171, 175, 249, 99, 198, 163, 154, 183, 80, 70, 18, 90, 143, 16, 110, 40, 74, 15, 152, 104, 155, 119, 3, 189, 92, 182, 181, 193, 112, 193, 179, 171, 177, 195, 218, 45, 246, 105, 209, 209, 4, 147, 233, 109, 252, 165, 52, 200, 179, 47, 64, 202, 66, 155, 170, 48, 184, 165, 51, 160, 106, 229, 210, 44, 72, 109, 204, 32, 9, 38, 222, 130, 253, 176, 6, 246, 50, 104, 240, 45, 124, 77, 233, 84, 164, 109, 154, 110, 171, 77, 164, 253, 134, 225, 58, 131, 116, 48, 78, 1, 85, 150, 81, 149, 67, 171, 102, 216, 126, 229, 90, 56, 26, 33, 171, 124, 161, 209, 140, 187, 61, 34, 212, 215, 17, 134, 62, 6, 13, 208, 221, 55, 5, 187, 22, 63, 93, 164, 76, 124, 166, 239, 104, 210, 84, 78, 146, 114, 146, 113, 215, 97, 33, 13, 186, 135, 110, 182, 194, 44, 64, 215, 36, 43, 178, 21, 170, 171, 251, 180, 100, 40, 239, 189, 24, 174, 192, 195, 98, 108, 222, 96, 135, 172, 56, 114, 81, 213, 239, 84, 199, 27, 118, 56, 193, 239, 243, 191, 178, 97, 106, 201, 66, 252, 31, 190, 254, 107, 226, 184, 19, 188, 113, 44, 99, 24, 245, 52, 154, 97, 249, 178, 221, 232, 203, 63, 51, 4, 109, 167, 92, 246, 143, 188, 32, 32, 87, 98, 186, 5, 69, 17],
SignatureAlgorithm: 4,
PublicKeyAlgorithm: 1,
PublicKey: <*rsa.PublicKey | 0x2b2e57843d70>{
N: {
neg: false,
abs: [2877128800215149205, 16629919685921545055, 13732506504615110447, 11103748926260740575, 11416989105297106818, 9223988340828442177, 2897202458616478060, 11766261893188694937, 15086238765284949963, 10809216967110507236, 9979912210168109780, 11664503868611822661, 3024932991847888604, 17852206095763650445, 1698336212534998063, 17691123283498957122, 9751057521779199143, 5748498642685419890, 4821085446021960184, 2270194980282856385, 5998191060056422898, 1979344939107882427, 11002240530182040303, 17961926253159653156, 4444200366252054316, 17136494472856312874, 5046575580329891885, 18333653388545728381, 18332734810168546880, 2830450700003086872, 11572718067588258363, 13260460152132755496],
},
E: 65537,
},
Version: 3,
SerialNumber: {
neg: false,
abs: [8959707475658302546],
},
Issuer: {
Country: nil,
Organization: nil,
OrganizationalUnit: ["openshift"],
Locality: nil,
Province: nil,
StreetAddress: nil,
PostalCode: nil,
SerialNumber: "",
CommonName: "root-ca",
Names: [
{
Type: [2, 5, 4, 11],
Value: <string>"openshift",
},
{
Type: [2, 5, 4, 3],
Value: <string>"root-ca",
},
],
ExtraNames: nil,
},
Subject: {
Country: nil,
Organization: ["openshift"],
OrganizationalUnit: nil,
Locality: nil,
Province: nil,
StreetAddress: nil,
PostalCode: nil,
SerialNumber: "",
CommonName: "openshift-ingress",
Names: [
{
Type: [2, 5, 4, 10],
Value: <string>"openshift",
},
{
Type: [2, 5, 4, 3],
Value: <string>"openshift-ingress",
},
],
ExtraNames: nil,
},
NotBefore: {wall: 0, ext: 63928472888, loc: nil},
NotAfter: {wall: 0, ext: 63960008888, loc: nil},
KeyUsage: 5,
Extensions: [
{
Id: [2, 5, 29, 15],
Critical: true,
Value: [3, 2, 5, 160],
},
{
Id: [2, 5, 29, 37],
Critical: false,
Value: [48, 20, 6, 8, 43, 6, 1, 5, 5, 7, 3, 2, 6, 8, 43, 6, 1, 5, 5, 7, 3, 1],
},
{Id: [2, 5, 29, 19], Critical: true, Value: [48, 0]},
{
Id: [2, 5, 29, 14],
Critical: false,
Value: [4, 64, 43, 61, 184, 58, 190, 246, 117, 145, 150, 149, 52, 148, 50, 70, 171, 24, 140, 56, 164, 203, 105, 63, 65, 146, 183, 66, 114, 129, 170, 239, 140, 79, 13, 223, 192, 116, 206, 195, 234, 70, 154, 102, 164, 7, 77, 72, 8, 99, 67, 120, 61, 224, 48, 163, 84, 4, 250, 254, 27, 88, 242, 13, 99, 21],
},
{
Id: [2, 5, 29, 35],
Critical: false,
Value: [48, 66, 128, 64, 145, 115, 72, 248, 147, 3, 210, 10, 72, 90, 51, 130, 36, 255, 101, 123, 45, 151, 78, 158, 148, 254, 229, 16, 121, 180, 193, 131, 183, 127, 36, 88, 142, 33, 22, 114, 228, 61, 202, 29, 164, 17, 211, 221, 107, 244, 245, 206, 36, 16, 177, 96, 20, 74, 22, 17, 184, 132, 100, 119, 214, 222, 88, 253],
},
{
Id: [2, 5, 29, 17],
Critical: false,
Value: "0C\x82A*.apps.aro.a2s9b8m2e4p7p0g.0mu8.westus3.aroapp-hcp.azure-test.net",
},
],
ExtraExtensions: nil,
UnhandledCriticalExtensions: nil,
ExtKeyUsage: [2, 1],
UnknownExtKeyUsage: nil,
BasicConstraintsValid: true,
IsCA: false,
MaxPathLen: -1,
MaxPathLenZero: false,
SubjectKeyId: [43, 61, 184, 58, 190, 246, 117, 145, 150, 149, 52, 148, 50, 70, 171, 24, 140, 56, 164, 203, 105, 63, 65, 146, 183, 66, 114, 129, 170, 239, 140, 79, 13, 223, 192, 116, 206, 195, 234, 70, 154, 102, 164, 7, 77, 72, 8, 99, 67, 120, 61, 224, 48, 163, 84, 4, 250, 254, 27, 88, 242, 13, 99, 21],
AuthorityKeyId: [145, 115, 72, 248, 147, 3, 210, 10, 72, 90, 51, 130, 36, 255, 101, 123, 45, 151, 78, 158, 148, 254, 229, 16, 121, 180, 193, 131, 183, 127, 36, 88, 142, 33, 22, 114, 228, 61, 202, 29, 164, 17, 211, 221, 107, 244, 245, 206, 36, 16, 177, 96, 20, 74, 22, 17, 184, 132, 100, 119, 214, 222, 88, 253],
OCSPServer: nil,
IssuingCertificateURL: nil,
DNSNames: [
"*.apps.aro.a2s9b8m2e4p7p0g.0mu8.westus3.aroapp-hcp.azure-test.net",
],
EmailAddresses: nil,
IPAddresses: nil,
URIs: nil,
PermittedDNSDomainsCritical: false,
PermittedDNSDomains: nil,
ExcludedDNSDomains: nil,
PermittedIPRanges: nil,
ExcludedIPRanges: nil,
PermittedEmailAddresses: nil,
ExcludedEmailAddresses: nil,
PermittedURIDomains: nil,
ExcludedURIDomains: nil,
CRLDistributionPoints: nil,
PolicyIdentifiers: nil,
Policies: nil,
InhibitAnyPolicy: 0,
InhibitAnyPolicyZero: false,
InhibitPolicyMapping: 0,
InhibitPolicyMappingZero: false,
RequireExplicitPolicy: 0,
RequireExplicitPolicyZero: false,
PolicyMappings: nil,
},
],
Err: <x509.UnknownAuthorityError>{
Cert: 0x2b2e55ba2008 (cyclic reference),
hintErr: nil,
hintCert: nil,
},
},
},
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:194]: failed to verify simple web app runs on cluster "cluster-candidate-4-23-2rcw86"
Unexpected error:
<*fmt.wrapError | 0x2b2e578984a0>:
[strict TLS verification] route was never reachable: Get "https://agnhost-e2e-sample-app-w86zx.apps.aro.a2s9b8m2e4p7p0g.0mu8.westus3.aroapp-hcp.azure-test.net": tls: failed to verify certificate: x509: certificate signed by unknown authority
{
msg: "[strict TLS verification] route was never reachable: Get \"https://agnhost-e2e-sample-app-w86zx.apps.aro.a2s9b8m2e4p7p0g.0mu8.westus3.aroapp-hcp.azure-test.net\": tls: failed to verify certificate: x509: certificate signed by unknown authority",
err: <*url.Error | 0x2b2e57370780>{
Op: "Get",
URL: "https://agnhost-e2e-sample-app-w86zx.apps.aro.a2s9b8m2e4p7p0g.0mu8.westus3.aroapp-hcp.azure-test.net",
Err: <*tls.CertificateVerificationError | 0x2b2e57370750>{
UnverifiedCertificates: [
{
Raw: [48, 130, 4, 1, 48, 130, 2, 233, 160, 3, 2, 1, 2, 2, 8, 124, 87, 70, 122, 130, 6, 100, 82, 48, 13, 6, 9, 42, 134, 72, 134, 247, 13, 1, 1, 11, 5, 0, 48, 38, 49, 18, 48, 16, 6, 3, 85, 4, 11, 19, 9, 111, 112, 101, 110, 115, 104, 105, 102, 116, 49, 16, 48, 14, 6, 3, 85, 4, 3, 19, 7, 114, 111, 111, 116, 45, 99, 97, 48, 30, 23, 13, 50, 54, 49, 48, 50, 52, 50, 49, 48, 56, 48, 56, 90, 23, 13, 50, 55, 49, 48, 50, 52, 50, 49, 48, 56, 48, 56, 90, 48, 48, 49, 18, 48, 16, 6, 3, 85, 4, 10, 19, 9, 111, 112, 101, 110, 115, 104, 105, 102, 116, 49, 26, 48, 24, 6, 3, 85, 4, 3, 19, 17, 111, 112, 101, 110, 115, 104, 105, 102, 116, 45, 105, 110, 103, 114, 101, 115, 115, 48, 130, 1, 34, 48, 13, 6, 9, 42, 134, 72, 134, 247, 13, 1, 1, 1, 5, 0, 3, 130, 1, 15, 0, 48, 130, 1, 10, 2, 130, 1, 1, 0, 184, 6, 158, 64, 94, 24, 96, 40, 160, 154, 141, 196, 128, 87, 218, 59, 39, 71, 199, 234, 79, 240, 146, 24, 254, 106, 245, 45, 201, 136, 214, 64, 254, 110, 56, 158, 251, 103, 159, 125, 70, 9, 9, 192, 157, 185, 88, 45, 237, 209, 14, 238, 22, 11, 156, 42, 61, 172, 248, 175, 205, 29, 87, 44, 249, 69, 148, 194, 201, 88, 31, 36, 152, 175, 207, 120, 70, 231, 234, 239, 27, 120, 11, 188, 47, 126, 21, 187, 83, 61, 218, 253, 69, 150, 57, 242, 31, 129, 90, 78, 122, 162, 115, 193, 66, 231, 239, 171, 129, 241, 209, 248, 79, 198, 197, 15, 5, 20, 97, 114, 135, 82, 183, 39, 26, 12, 204, 167, 245, 131, 126, 229, 14, 152, 249, 66, 23, 145, 179, 201, 129, 84, 180, 47, 247, 191, 198, 219, 235, 14, 55, 141, 41, 250, 184, 135, 128, 115, 246, 220, 161, 224, 164, 119, 161, 199, 4, 69, 138, 127, 197, 71, 128, 167, 90, 212, 150, 2, 13, 143, 216, 191, 58, 228, 209, 93, 22, 58, 9, 137, 127, 203, 163, 74, 40, 217, 253, 186, 51, 153, 40, 52, 238, 73, 60, 57, 57, 108, 128, 2, 48, 134, 115, 108, 142, 65, 158, 113, 75, 25, 249, 189, 39, 130, 154, 24, 112, 209, 120, 229, 129, 223, 190, 147, 169, 232, 115, 83, 39, 47, 230, 201, 87, 216, 98, 161, 27, 95, 39, 237, 157, 102, 109, 222, 170, 149, 2, 3, 1, 0, 1, 163, 130, 1, 39, 48, 130, 1, 35, 48, 14, 6, 3, 85, 29, 15, 1, 1, 255, 4, 4, 3, 2, 5, 160, 48, 29, 6, 3, 85, 29, 37, 4, 22, 48, 20, 6, 8, 43, 6, 1, 5, 5, 7, 3, 2, 6, 8, 43, 6, 1, 5, 5, 7, 3, 1, 48, 12, 6, 3, 85, 29, 19, 1, 1, 255, 4, 2, 48, 0, 48, 73, 6, 3, 85, 29, 14, 4, 66, 4, 64, 43, 61, 184, 58, 190, 246, 117, 145, 150, 149, 52, 148, 50, 70, 171, 24, 140, 56, 164, 203, 105, 63, 65, 146, 183, 66, 114, 129, 170, 239, 140, 79, 13, 223, 192, 116, 206, 195, 234, 70, 154, 102, 164, 7, 77, 72, 8, 99, 67, 120, 61, 224, 48, 163, 84, 4, 250, 254, 27, 88, 242, 13, 99, 21, 48, 75, 6, 3, 85, 29, 35, 4, 68, 48, 66, 128, 64, 145, 115, 72, 248, 147, 3, 210, 10, 72, 90, 51, 130, 36, 255, 101, 123, 45, 151, 78, 158, 148, 254, 229, 16, 121, 180, 193, 131, 183, 127, 36, 88, 142, 33, 22, 114, 228, 61, 202, 29, 164, 17, 211, 221, 107, 244, 245, 206, 36, 16, 177, 96, 20, 74, 22, 17, 184, 132, 100, 119, 214, 222, 88, 253, 48, 76, 6, 3, 85, 29, 17, 4, 69, 48, 67, 130, 65, 42, 46, 97, 112, 112, 115, 46, 97, 114, 111, 46, 97, 50, 115, 57, 98, 56, 109, 50, 101, 52, 112, 55, 112, 48, 103, 46, 48, 109, 117, 56, 46, 119, 101, 115, 116, 117, 115, 51, 46, 97, 114, 111, 97, 112, 112, 45, 104, 99, 112, 46, 97, 122, 117, 114, 101, 45, 116, 101, 115, 116, 46, 110, 101, 116, 48, 13, 6, 9, 42, 134, 72, 134, 247, 13, 1, 1, 11, 5, 0, 3, 130, 1, 1, 0, 144, 63, 1, 9, 63, 203, 104, 158, 72, 190, 218, 173, 218, 174, 171, 175, 249, 99, 198, 163, 154, 183, 80, 70, 18, 90, 143, 16, 110, 40, 74, 15, 152, 104, 155, 119, 3, 189, 92, 182, 181, 193, 112, 193, 179, 171, 177, 195, 218, 45, 246, 105, 209, 209, 4, 147, 233, 109, 252, 165, 52, 200, 179, 47, 64, 202, 66, 155, 170, 48, 184, 165, 51, 160, 106, 229, 210, 44, 72, 109, 204, 32, 9, 38, 222, 130, 253, 176, 6, 246, 50, 104, 240, 45, 124, 77, 233, 84, 164, 109, 154, 110, 171, 77, 164, 253, 134, 225, 58, 131, 116, 48, 78, 1, 85, 150, 81, 149, 67, 171, 102, 216, 126, 229, 90, 56, 26, 33, 171, 124, 161, 209, 140, 187, 61, 34, 212, 215, 17, 134, 62, 6, 13, 208, 221, 55, 5, 187, 22, 63, 93, 164, 76, 124, 166, 239, 104, 210, 84, 78, 146, 114, 146, 113, 215, 97, 33, 13, 186, 135, 110, 182, 194, 44, 64, 215, 36, 43, 178, 21, 170, 171, 251, 180, 100, 40, 239, 189, 24, 174, 192, 195, 98, 108, 222, 96, 135, 172, 56, 114, 81, 213, 239, 84, 199, 27, 118, 56, 193, 239, 243, 191, 178, 97, 106, 201, 66, 252, 31, 190, 254, 107, 226, 184, 19, 188, 113, 44, 99, 24, 245, 52, 154, 97, 249, 178, 221, 232, 203, 63, 51, 4, 109, 167, 92, 246, 143, 188, 32, 32, 87, 98, 186, 5, 69, 17],
RawTBSCertificate: [48, 130, 2, 233, 160, 3, 2, 1, 2, 2, 8, 124, 87, 70, 122, 130, 6, 100, 82, 48, 13, 6, 9, 42, 134, 72, 134, 247, 13, 1, 1, 11, 5, 0, 48, 38, 49, 18, 48, 16, 6, 3, 85, 4, 11, 19, 9, 111, 112, 101, 110, 115, 104, 105, 102, 116, 49, 16, 48, 14, 6, 3, 85, 4, 3, 19, 7, 114, 111, 111, 116, 45, 99, 97, 48, 30, 23, 13, 50, 54, 49, 48, 50, 52, 50, 49, 48, 56, 48, 56, 90, 23, 13, 50, 55, 49, 48, 50, 52, 50, 49, 48, 56, 48, 56, 90, 48, 48, 49, 18, 48, 16, 6, 3, 85, 4, 10, 19, 9, 111, 112, 101, 110, 115, 104, 105, 102, 116, 49, 26, 48, 24, 6, 3, 85, 4, 3, 19, 17, 111, 112, 101, 110, 115, 104, 105, 102, 116, 45, 105, 110, 103, 114, 101, 115, 115, 48, 130, 1, 34, 48, 13, 6, 9, 42, 134, 72, 134, 247, 13, 1, 1, 1, 5, 0, 3, 130, 1, 15, 0, 48, 130, 1, 10, 2, 130, 1, 1, 0, 184, 6, 158, 64, 94, 24, 96, 40, 160, 154, 141, 196, 128, 87, 218, 59, 39, 71, 199, 234, 79, 240, 146, 24, 254, 106, 245, 45, 201, 136, 214, 64, 254, 110, 56, 158, 251, 103, 159, 125, 70, 9, 9, 192, 157, 185, 88, 45, 237, 209, 14, 238, 22, 11, 156, 42, 61, 172, 248, 175, 205, 29, 87, 44, 249, 69, 148, 194, 201, 88, 31, 36, 152, 175, 207, 120, 70, 231, 234, 239, 27, 120, 11, 188, 47, 126, 21, 187, 83, 61, 218, 253, 69, 150, 57, 242, 31, 129, 90, 78, 122, 162, 115, 193, 66, 231, 239, 171, 129, 241, 209, 248, 79, 198, 197, 15, 5, 20, 97, 114, 135, 82, 183, 39, 26, 12, 204, 167, 245, 131, 126, 229, 14, 152, 249, 66, 23, 145, 179, 201, 129, 84, 180, 47, 247, 191, 198, 219, 235, 14, 55, 141, 41, 250, 184, 135, 128, 115, 246, 220, 161, 224, 164, 119, 161, 199, 4, 69, 138, 127, 197, 71, 128, 167, 90, 212, 150, 2, 13, 143, 216, 191, 58, 228, 209, 93, 22, 58, 9, 137, 127, 203, 163, 74, 40, 217, 253, 186, 51, 153, 40, 52, 238, 73, 60, 57, 57, 108, 128, 2, 48, 134, 115, 108, 142, 65, 158, 113, 75, 25, 249, 189, 39, 130, 154, 24, 112, 209, 120, 229, 129, 223, 190, 147, 169, 232, 115, 83, 39, 47, 230, 201, 87, 216, 98, 161, 27, 95, 39, 237, 157, 102, 109, 222, 170, 149, 2, 3, 1, 0, 1, 163, 130, 1, 39, 48, 130, 1, 35, 48, 14, 6, 3, 85, 29, 15, 1, 1, 255, 4, 4, 3, 2, 5, 160, 48, 29, 6, 3, 85, 29, 37, 4, 22, 48, 20, 6, 8, 43, 6, 1, 5, 5, 7, 3, 2, 6, 8, 43, 6, 1, 5, 5, 7, 3, 1, 48, 12, 6, 3, 85, 29, 19, 1, 1, 255, 4, 2, 48, 0, 48, 73, 6, 3, 85, 29, 14, 4, 66, 4, 64, 43, 61, 184, 58, 190, 246, 117, 145, 150, 149, 52, 148, 50, 70, 171, 24, 140, 56, 164, 203, 105, 63, 65, 146, 183, 66, 114, 129, 170, 239, 140, 79, 13, 223, 192, 116, 206, 195, 234, 70, 154, 102, 164, 7, 77, 72, 8, 99, 67, 120, 61, 224, 48, 163, 84, 4, 250, 254, 27, 88, 242, 13, 99, 21, 48, 75, 6, 3, 85, 29, 35, 4, 68, 48, 66, 128, 64, 145, 115, 72, 248, 147, 3, 210, 10, 72, 90, 51, 130, 36, 255, 101, 123, 45, 151, 78, 158, 148, 254, 229, 16, 121, 180, 193, 131, 183, 127, 36, 88, 142, 33, 22, 114, 228, 61, 202, 29, 164, 17, 211, 221, 107, 244, 245, 206, 36, 16, 177, 96, 20, 74, 22, 17, 184, 132, 100, 119, 214, 222, 88, 253, 48, 76, 6, 3, 85, 29, 17, 4, 69, 48, 67, 130, 65, 42, 46, 97, 112, 112, 115, 46, 97, 114, 111, 46, 97, 50, 115, 57, 98, 56, 109, 50, 101, 52, 112, 55, 112, 48, 103, 46, 48, 109, 117, 56, 46, 119, 101, 115, 116, 117, 115, 51, 46, 97, 114, 111, 97, 112, 112, 45, 104, 99, 112, 46, 97, 122, 117, 114, 101, 45, 116, 101, 115, 116, 46, 110, 101, 116],
RawSubjectPublicKeyInfo: [48, 130, 1, 34, 48, 13, 6, 9, 42, 134, 72, 134, 247, 13, 1, 1, 1, 5, 0, 3, 130, 1, 15, 0, 48, 130, 1, 10, 2, 130, 1, 1, 0, 184, 6, 158, 64, 94, 24, 96, 40, 160, 154, 141, 196, 128, 87, 218, 59, 39, 71, 199, 234, 79, 240, 146, 24, 254, 106, 245, 45, 201, 136, 214, 64, 254, 110, 56, 158, 251, 103, 159, 125, 70, 9, 9, 192, 157, 185, 88, 45, 237, 209, 14, 238, 22, 11, 156, 42, 61, 172, 248, 175, 205, 29, 87, 44, 249, 69, 148, 194, 201, 88, 31, 36, 152, 175, 207, 120, 70, 231, 234, 239, 27, 120, 11, 188, 47, 126, 21, 187, 83, 61, 218, 253, 69, 150, 57, 242, 31, 129, 90, 78, 122, 162, 115, 193, 66, 231, 239, 171, 129, 241, 209, 248, 79, 198, 197, 15, 5, 20, 97, 114, 135, 82, 183, 39, 26, 12, 204, 167, 245, 131, 126, 229, 14, 152, 249, 66, 23, 145, 179, 201, 129, 84, 180, 47, 247, 191, 198, 219, 235, 14, 55, 141, 41, 250, 184, 135, 128, 115, 246, 220, 161, 224, 164, 119, 161, 199, 4, 69, 138, 127, 197, 71, 128, 167, 90, 212, 150, 2, 13, 143, 216, 191, 58, 228, 209, 93, 22, 58, 9, 137, 127, 203, 163, 74, 40, 217, 253, 186, 51, 153, 40, 52, 238, 73, 60, 57, 57, 108, 128, 2, 48, 134, 115, 108, 142, 65, 158, 113, 75, 25, 249, 189, 39, 130, 154, 24, 112, 209, 120, 229, 129, 223, 190, 147, 169, 232, 115, 83, 39, 47, 230, 201, 87, 216, 98, 161, 27, 95, 39, 237, 157, 102, 109, 222, 170, 149, 2, 3, 1, 0, 1],
RawSubject: [48, 48, 49, 18, 48, 16, 6, 3, 85, 4, 10, 19, 9, 111, 112, 101, 110, 115, 104, 105, 102, 116, 49, 26, 48, 24, 6, 3, 85, 4, 3, 19, 17, 111, 112, 101, 110, 115, 104, 105, 102, 116, 45, 105, 110, 103, 114, 101, 115, 115],
RawIssuer: [48, 38, 49, 18, 48, 16, 6, 3, 85, 4, 11, 19, 9, 111, 112, 101, 110, 115, 104, 105, 102, 116, 49, 16, 48, 14, 6, 3, 85, 4, 3, 19, 7, 114, 111, 111, 116, 45, 99, 97],
Signature: [144, 63, 1, 9, 63, 203, 104, 158, 72, 190, 218, 173, 218, 174, 171, 175, 249, 99, 198, 163, 154, 183, 80, 70, 18, 90, 143, 16, 110, 40, 74, 15, 152, 104, 155, 119, 3, 189, 92, 182, 181, 193, 112, 193, 179, 171, 177, 195, 218, 45, 246, 105, 209, 209, 4, 147, 233, 109, 252, 165, 52, 200, 179, 47, 64, 202, 66, 155, 170, 48, 184, 165, 51, 160, 106, 229, 210, 44, 72, 109, 204, 32, 9, 38, 222, 130, 253, 176, 6, 246, 50, 104, 240, 45, 124, 77, 233, 84, 164, 109, 154, 110, 171, 77, 164, 253, 134, 225, 58, 131, 116, 48, 78, 1, 85, 150, 81, 149, 67, 171, 102, 216, 126, 229, 90, 56, 26, 33, 171, 124, 161, 209, 140, 187, 61, 34, 212, 215, 17, 134, 62, 6, 13, 208, 221, 55, 5, 187, 22, 63, 93, 164, 76, 124, 166, 239, 104, 210, 84, 78, 146, 114, 146, 113, 215, 97, 33, 13, 186, 135, 110, 182, 194, 44, 64, 215, 36, 43, 178, 21, 170, 171, 251, 180, 100, 40, 239, 189, 24, 174, 192, 195, 98, 108, 222, 96, 135, 172, 56, 114, 81, 213, 239, 84, 199, 27, 118, 56, 193, 239, 243, 191, 178, 97, 106, 201, 66, 252, 31, 190, 254, 107, 226, 184, 19, 188, 113, 44, 99, 24, 245, 52, 154, 97, 249, 178, 221, 232, 203, 63, 51, 4, 109, 167, 92, 246, 143, 188, 32, 32, 87, 98, 186, 5, 69, 17],
SignatureAlgorithm: 4,
PublicKeyAlgorithm: 1,
PublicKey: <*rsa.PublicKey | 0x2b2e57843d70>{
N: {
neg: false,
abs: [2877128800215149205, 16629919685921545055, 13732506504615110447, 11103748926260740575, 11416989105297106818, 9223988340828442177, 2897202458616478060, 11766261893188694937, 15086238765284949963, 10809216967110507236, 9979912210168109780, 11664503868611822661, 3024932991847888604, 17852206095763650445, 1698336212534998063, 17691123283498957122, 9751057521779199143, 5748498642685419890, 4821085446021960184, 2270194980282856385, 5998191060056422898, 1979344939107882427, 11002240530182040303, 17961926253159653156, 4444200366252054316, 17136494472856312874, 5046575580329891885, 18333653388545728381, 18332734810168546880, 2830450700003086872, 11572718067588258363, 13260460152132755496],
},
E: 65537,
},
Version: 3,
SerialNumber: {
neg: false,
abs: [8959707475658302546],
},
Issuer: {
Country: nil,
Organization: nil,
OrganizationalUnit: ["openshift"],
Locality: nil,
Province: nil,
StreetAddress: nil,
PostalCode: nil,
SerialNumber: "",
CommonName: "root-ca",
Names: [
{
Type: [2, 5, 4, 11],
Value: <string>"openshift",
},
{
Type: [2, 5, 4, 3],
Value: <string>"root-ca",
},
],
ExtraNames: nil,
},
Subject: {
Country: nil,
Organization: ["openshift"],
OrganizationalUnit: nil,
Locality: nil,
Province: nil,
StreetAddress: nil,
PostalCode: nil,
SerialNumber: "",
CommonName: "openshift-ingress",
Names: [
{
Type: [2, 5, 4, 10],
Value: <string>"openshift",
},
{
Type: [2, 5, 4, 3],
Value: <string>"openshift-ingress",
},
],
ExtraNames: nil,
},
NotBefore: {wall: 0, ext: 63928472888, loc: nil},
NotAfter: {wall: 0, ext: 63960008888, loc: nil},
KeyUsage: 5,
Extensions: [
{
Id: [2, 5, 29, 15],
Critical: true,
Value: [3, 2, 5, 160],
},
{
Id: [2, 5, 29, 37],
Critical: false,
Value: [48, 20, 6, 8, 43, 6, 1, 5, 5, 7, 3, 2, 6, 8, 43, 6, 1, 5, 5, 7, 3, 1],
},
{Id: [2, 5, 29, 19], Critical: true, Value: [48, 0]},
{
Id: [2, 5, 29, 14],
Critical: false,
Value: [4, 64, 43, 61, 184, 58, 190, 246, 117, 145, 150, 149, 52, 148, 50, 70, 171, 24, 140, 56, 164, 203, 105, 63, 65, 146, 183, 66, 114, 129, 170, 239, 140, 79, 13, 223, 192, 116, 206, 195, 234, 70, 154, 102, 164, 7, 77, 72, 8, 99, 67, 120, 61, 224, 48, 163, 84, 4, 250, 254, 27, 88, 242, 13, 99, 21],
},
{
Id: [2, 5, 29, 35],
Critical: false,
Value: [48, 66, 128, 64, 145, 115, 72, 248, 147, 3, 210, 10, 72, 90, 51, 130, 36, 255, 101, 123, 45, 151, 78, 158, 148, 254, 229, 16, 121, 180, 193, 131, 183, 127, 36, 88, 142, 33, 22, 114, 228, 61, 202, 29, 164, 17, 211, 221, 107, 244, 245, 206, 36, 16, 177, 96, 20, 74, 22, 17, 184, 132, 100, 119, 214, 222, 88, 253],
},
{
Id: [2, 5, 29, 17],
Critical: false,
Value: "0C\x82A*.apps.aro.a2s9b8m2e4p7p0g.0mu8.westus3.aroapp-hcp.azure-test.net",
},
],
ExtraExtensions: nil,
UnhandledCriticalExtensions: nil,
ExtKeyUsage: [2, 1],
UnknownExtKeyUsage: nil,
BasicConstraintsValid: true,
IsCA: false,
MaxPathLen: -1,
MaxPathLenZero: false,
SubjectKeyId: [43, 61, 184, 58, 190, 246, 117, 145, 150, 149, 52, 148, 50, 70, 171, 24, 140, 56, 164, 203, 105, 63, 65, 146, 183, 66, 114, 129, 170, 239, 140, 79, 13, 223, 192, 116, 206, 195, 234, 70, 154, 102, 164, 7, 77, 72, 8, 99, 67, 120, 61, 224, 48, 163, 84, 4, 250, 254, 27, 88, 242, 13, 99, 21],
AuthorityKeyId: [145, 115, 72, 248, 147, 3, 210, 10, 72, 90, 51, 130, 36, 255, 101, 123, 45, 151, 78, 158, 148, 254, 229, 16, 121, 180, 193, 131, 183, 127, 36, 88, 142, 33, 22, 114, 228, 61, 202, 29, 164, 17, 211, 221, 107, 244, 245, 206, 36, 16, 177, 96, 20, 74, 22, 17, 184, 132, 100, 119, 214, 222, 88, 253],
OCSPServer: nil,
IssuingCertificateURL: nil,
DNSNames: [
"*.apps.aro.a2s9b8m2e4p7p0g.0mu8.westus3.aroapp-hcp.azure-test.net",
],
EmailAddresses: nil,
IPAddresses: nil,
URIs: nil,
PermittedDNSDomainsCritical: false,
PermittedDNSDomains: nil,
ExcludedDNSDomains: nil,
PermittedIPRanges: nil,
ExcludedIPRanges: nil,
PermittedEmailAddresses: nil,
ExcludedEmailAddresses: nil,
PermittedURIDomains: nil,
ExcludedURIDomains: nil,
CRLDistributionPoints: nil,
PolicyIdentifiers: nil,
Policies: nil,
InhibitAnyPolicy: 0,
InhibitAnyPolicyZero: false,
InhibitPolicyMapping: 0,
InhibitPolicyMappingZero: false,
RequireExplicitPolicy: 0,
RequireExplicitPolicyZero: false,
PolicyMappings: nil,
},
],
Err: <x509.UnknownAuthorityError>{
Cert: 0x2b2e55ba2008 (cyclic reference),
hintErr: nil,
hintCert: nil,
},
},
},
}
occurredfail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:194]: failed to verify simple web app runs on cluster "cluster-candidate-4-23-wk4gjj"
Unexpected error:
<*fmt.wrapError | 0x30fb7a9c6d80>:
[strict TLS verification] route was never reachable: Get "https://agnhost-e2e-sample-app-25lgf.apps.aro.c9p0v7j9q7j4a1q.afld.westus3.aroapp-hcp.azure-test.net": tls: failed to verify certificate: x509: certificate signed by unknown authority
{
msg: "[strict TLS verification] route was never reachable: Get \"https://agnhost-e2e-sample-app-25lgf.apps.aro.c9p0v7j9q7j4a1q.afld.westus3.aroapp-hcp.azure-test.net\": tls: failed to verify certificate: x509: certificate signed by unknown authority",
err: <*url.Error | 0x30fb7cb9b320>{
Op: "Get",
URL: "https://agnhost-e2e-sample-app-25lgf.apps.aro.c9p0v7j9q7j4a1q.afld.westus3.aroapp-hcp.azure-test.net",
Err: <*tls.CertificateVerificationError | 0x30fb7cb9b2f0>{
UnverifiedCertificates: [
{
Raw: [48, 130, 4, 1, 48, 130, 2, 233, 160, 3, 2, 1, 2, 2, 8, 110, 83, 80, 98, 163, 186, 156, 29, 48, 13, 6, 9, 42, 134, 72, 134, 247, 13, 1, 1, 11, 5, 0, 48, 38, 49, 18, 48, 16, 6, 3, 85, 4, 11, 19, 9, 111, 112, 101, 110, 115, 104, 105, 102, 116, 49, 16, 48, 14, 6, 3, 85, 4, 3, 19, 7, 114, 111, 111, 116, 45, 99, 97, 48, 30, 23, 13, 50, 54, 49, 48, 50, 52, 49, 55, 50, 48, 48, 48, 90, 23, 13, 50, 55, 49, 48, 50, 52, 49, 55, 50, 48, 48, 48, 90, 48, 48, 49, 18, 48, 16, 6, 3, 85, 4, 10, 19, 9, 111, 112, 101, 110, 115, 104, 105, 102, 116, 49, 26, 48, 24, 6, 3, 85, 4, 3, 19, 17, 111, 112, 101, 110, 115, 104, 105, 102, 116, 45, 105, 110, 103, 114, 101, 115, 115, 48, 130, 1, 34, 48, 13, 6, 9, 42, 134, 72, 134, 247, 13, 1, 1, 1, 5, 0, 3, 130, 1, 15, 0, 48, 130, 1, 10, 2, 130, 1, 1, 0, 175, 180, 66, 13, 148, 254, 185, 112, 159, 180, 48, 58, 217, 14, 123, 23, 229, 28, 224, 241, 179, 210, 115, 71, 123, 94, 152, 56, 200, 179, 120, 80, 246, 68, 78, 194, 165, 17, 161, 64, 168, 40, 31, 243, 155, 207, 171, 17, 238, 179, 159, 199, 90, 32, 61, 243, 133, 233, 153, 229, 126, 69, 220, 234, 188, 36, 185, 6, 114, 77, 215, 59, 104, 34, 80, 95, 66, 69, 104, 166, 84, 17, 159, 234, 101, 109, 164, 153, 172, 156, 240, 134, 16, 200, 61, 179, 79, 164, 99, 227, 229, 46, 174, 192, 70, 194, 161, 207, 5, 78, 18, 21, 247, 101, 35, 19, 102, 159, 29, 233, 182, 86, 199, 53, 244, 175, 208, 228, 116, 49, 45, 21, 118, 178, 72, 104, 19, 188, 1, 131, 254, 127, 37, 139, 95, 60, 129, 254, 7, 18, 184, 51, 199, 217, 71, 107, 203, 113, 33, 123, 83, 246, 30, 234, 39, 214, 99, 207, 167, 116, 247, 66, 241, 15, 133, 22, 9, 238, 99, 105, 166, 167, 72, 142, 182, 229, 22, 51, 127, 84, 32, 214, 8, 185, 111, 104, 100, 152, 181, 219, 81, 90, 133, 133, 127, 114, 2, 134, 80, 248, 53, 178, 96, 3, 247, 147, 225, 186, 125, 75, 198, 237, 177, 202, 247, 71, 49, 31, 123, 47, 222, 47, 12, 75, 219, 147, 5, 27, 92, 161, 76, 204, 58, 211, 26, 120, 156, 19, 51, 136, 52, 116, 18, 166, 216, 205, 2, 3, 1, 0, 1, 163, 130, 1, 39, 48, 130, 1, 35, 48, 14, 6, 3, 85, 29, 15, 1, 1, 255, 4, 4, 3, 2, 5, 160, 48, 29, 6, 3, 85, 29, 37, 4, 22, 48, 20, 6, 8, 43, 6, 1, 5, 5, 7, 3, 2, 6, 8, 43, 6, 1, 5, 5, 7, 3, 1, 48, 12, 6, 3, 85, 29, 19, 1, 1, 255, 4, 2, 48, 0, 48, 73, 6, 3, 85, 29, 14, 4, 66, 4, 64, 184, 27, 111, 100, 185, 3, 60, 74, 232, 82, 187, 169, 146, 147, 185, 95, 74, 1, 30, 137, 225, 41, 223, 249, 128, 141, 254, 150, 25, 225, 242, 3, 235, 207, 147, 83, 107, 148, 237, 0, 226, 240, 42, 175, 47, 113, 123, 148, 128, 190, 12, 146, 119, 216, 235, 223, 47, 29, 124, 171, 8, 133, 245, 128, 48, 75, 6, 3, 85, 29, 35, 4, 68, 48, 66, 128, 64, 107, 35, 93, 140, 120, 74, 68, 242, 166, 39, 191, 159, 156, 255, 152, 139, 116, 160, 21, 220, 61, 249, 246, 136, 64, 148, 57, 252, 107, 221, 102, 126, 167, 207, 36, 223, 46, 112, 92, 229, 162, 35, 131, 186, 43, 139, 91, 12, 221, 184, 234, 124, 29, 20, 52, 38, 121, 7, 144, 253, 45, 84, 166, 41, 48, 76, 6, 3, 85, 29, 17, 4, 69, 48, 67, 130, 65, 42, 46, 97, 112, 112, 115, 46, 97, 114, 111, 46, 99, 57, 112, 48, 118, 55, 106, 57, 113, 55, 106, 52, 97, 49, 113, 46, 97, 102, 108, 100, 46, 119, 101, 115, 116, 117, 115, 51, 46, 97, 114, 111, 97, 112, 112, 45, 104, 99, 112, 46, 97, 122, 117, 114, 101, 45, 116, 101, 115, 116, 46, 110, 101, 116, 48, 13, 6, 9, 42, 134, 72, 134, 247, 13, 1, 1, 11, 5, 0, 3, 130, 1, 1, 0, 62, 37, 28, 104, 244, 85, 128, 159, 33, 132, 56, 114, 149, 155, 76, 150, 56, 190, 182, 145, 23, 80, 230, 251, 69, 191, 95, 185, 206, 12, 203, 223, 159, 176, 108, 31, 228, 30, 29, 147, 188, 224, 159, 222, 209, 80, 26, 111, 48, 5, 218, 228, 214, 243, 4, 174, 170, 173, 102, 5, 47, 112, 16, 26, 249, 76, 134, 179, 187, 39, 158, 241, 110, 27, 4, 56, 253, 35, 167, 85, 21, 75, 203, 60, 52, 83, 13, 153, 94, 185, 69, 153, 196, 55, 251, 226, 149, 80, 157, 57, 244, 94, 195, 179, 137, 174, 166, 85, 12, 161, 202, 212, 188, 14, 213, 25, 59, 74, 197, 148, 98, 52, 206, 23, 41, 197, 32, 214, 165, 60, 202, 205, 67, 25, 42, 38, 206, 141, 201, 237, 101, 166, 209, 131, 101, 208, 225, 123, 244, 235, 82, 50, 143, 145, 70, 217, 191, 27, 3, 130, 194, 10, 227, 129, 5, 165, 150, 45, 143, 154, 59, 167, 145, 196, 78, 140, 76, 209, 96, 166, 23, 54, 227, 85, 65, 210, 246, 64, 194, 155, 249, 210, 255, 119, 224, 229, 103, 89, 68, 18, 164, 119, 9, 207, 73, 245, 219, 68, 214, 3, 155, 42, 94, 212, 192, 93, 148, 231, 125, 114, 77, 118, 234, 177, 58, 169, 209, 85, 145, 75, 243, 82, 234, 209, 51, 198, 131, 119, 63, 14, 40, 242, 219, 78, 215, 35, 237, 91, 234, 2, 142, 86, 121, 121, 249, 150],
RawTBSCertificate: [48, 130, 2, 233, 160, 3, 2, 1, 2, 2, 8, 110, 83, 80, 98, 163, 186, 156, 29, 48, 13, 6, 9, 42, 134, 72, 134, 247, 13, 1, 1, 11, 5, 0, 48, 38, 49, 18, 48, 16, 6, 3, 85, 4, 11, 19, 9, 111, 112, 101, 110, 115, 104, 105, 102, 116, 49, 16, 48, 14, 6, 3, 85, 4, 3, 19, 7, 114, 111, 111, 116, 45, 99, 97, 48, 30, 23, 13, 50, 54, 49, 48, 50, 52, 49, 55, 50, 48, 48, 48, 90, 23, 13, 50, 55, 49, 48, 50, 52, 49, 55, 50, 48, 48, 48, 90, 48, 48, 49, 18, 48, 16, 6, 3, 85, 4, 10, 19, 9, 111, 112, 101, 110, 115, 104, 105, 102, 116, 49, 26, 48, 24, 6, 3, 85, 4, 3, 19, 17, 111, 112, 101, 110, 115, 104, 105, 102, 116, 45, 105, 110, 103, 114, 101, 115, 115, 48, 130, 1, 34, 48, 13, 6, 9, 42, 134, 72, 134, 247, 13, 1, 1, 1, 5, 0, 3, 130, 1, 15, 0, 48, 130, 1, 10, 2, 130, 1, 1, 0, 175, 180, 66, 13, 148, 254, 185, 112, 159, 180, 48, 58, 217, 14, 123, 23, 229, 28, 224, 241, 179, 210, 115, 71, 123, 94, 152, 56, 200, 179, 120, 80, 246, 68, 78, 194, 165, 17, 161, 64, 168, 40, 31, 243, 155, 207, 171, 17, 238, 179, 159, 199, 90, 32, 61, 243, 133, 233, 153, 229, 126, 69, 220, 234, 188, 36, 185, 6, 114, 77, 215, 59, 104, 34, 80, 95, 66, 69, 104, 166, 84, 17, 159, 234, 101, 109, 164, 153, 172, 156, 240, 134, 16, 200, 61, 179, 79, 164, 99, 227, 229, 46, 174, 192, 70, 194, 161, 207, 5, 78, 18, 21, 247, 101, 35, 19, 102, 159, 29, 233, 182, 86, 199, 53, 244, 175, 208, 228, 116, 49, 45, 21, 118, 178, 72, 104, 19, 188, 1, 131, 254, 127, 37, 139, 95, 60, 129, 254, 7, 18, 184, 51, 199, 217, 71, 107, 203, 113, 33, 123, 83, 246, 30, 234, 39, 214, 99, 207, 167, 116, 247, 66, 241, 15, 133, 22, 9, 238, 99, 105, 166, 167, 72, 142, 182, 229, 22, 51, 127, 84, 32, 214, 8, 185, 111, 104, 100, 152, 181, 219, 81, 90, 133, 133, 127, 114, 2, 134, 80, 248, 53, 178, 96, 3, 247, 147, 225, 186, 125, 75, 198, 237, 177, 202, 247, 71, 49, 31, 123, 47, 222, 47, 12, 75, 219, 147, 5, 27, 92, 161, 76, 204, 58, 211, 26, 120, 156, 19, 51, 136, 52, 116, 18, 166, 216, 205, 2, 3, 1, 0, 1, 163, 130, 1, 39, 48, 130, 1, 35, 48, 14, 6, 3, 85, 29, 15, 1, 1, 255, 4, 4, 3, 2, 5, 160, 48, 29, 6, 3, 85, 29, 37, 4, 22, 48, 20, 6, 8, 43, 6, 1, 5, 5, 7, 3, 2, 6, 8, 43, 6, 1, 5, 5, 7, 3, 1, 48, 12, 6, 3, 85, 29, 19, 1, 1, 255, 4, 2, 48, 0, 48, 73, 6, 3, 85, 29, 14, 4, 66, 4, 64, 184, 27, 111, 100, 185, 3, 60, 74, 232, 82, 187, 169, 146, 147, 185, 95, 74, 1, 30, 137, 225, 41, 223, 249, 128, 141, 254, 150, 25, 225, 242, 3, 235, 207, 147, 83, 107, 148, 237, 0, 226, 240, 42, 175, 47, 113, 123, 148, 128, 190, 12, 146, 119, 216, 235, 223, 47, 29, 124, 171, 8, 133, 245, 128, 48, 75, 6, 3, 85, 29, 35, 4, 68, 48, 66, 128, 64, 107, 35, 93, 140, 120, 74, 68, 242, 166, 39, 191, 159, 156, 255, 152, 139, 116, 160, 21, 220, 61, 249, 246, 136, 64, 148, 57, 252, 107, 221, 102, 126, 167, 207, 36, 223, 46, 112, 92, 229, 162, 35, 131, 186, 43, 139, 91, 12, 221, 184, 234, 124, 29, 20, 52, 38, 121, 7, 144, 253, 45, 84, 166, 41, 48, 76, 6, 3, 85, 29, 17, 4, 69, 48, 67, 130, 65, 42, 46, 97, 112, 112, 115, 46, 97, 114, 111, 46, 99, 57, 112, 48, 118, 55, 106, 57, 113, 55, 106, 52, 97, 49, 113, 46, 97, 102, 108, 100, 46, 119, 101, 115, 116, 117, 115, 51, 46, 97, 114, 111, 97, 112, 112, 45, 104, 99, 112, 46, 97, 122, 117, 114, 101, 45, 116, 101, 115, 116, 46, 110, 101, 116],
RawSubjectPublicKeyInfo: [48, 130, 1, 34, 48, 13, 6, 9, 42, 134, 72, 134, 247, 13, 1, 1, 1, 5, 0, 3, 130, 1, 15, 0, 48, 130, 1, 10, 2, 130, 1, 1, 0, 175, 180, 66, 13, 148, 254, 185, 112, 159, 180, 48, 58, 217, 14, 123, 23, 229, 28, 224, 241, 179, 210, 115, 71, 123, 94, 152, 56, 200, 179, 120, 80, 246, 68, 78, 194, 165, 17, 161, 64, 168, 40, 31, 243, 155, 207, 171, 17, 238, 179, 159, 199, 90, 32, 61, 243, 133, 233, 153, 229, 126, 69, 220, 234, 188, 36, 185, 6, 114, 77, 215, 59, 104, 34, 80, 95, 66, 69, 104, 166, 84, 17, 159, 234, 101, 109, 164, 153, 172, 156, 240, 134, 16, 200, 61, 179, 79, 164, 99, 227, 229, 46, 174, 192, 70, 194, 161, 207, 5, 78, 18, 21, 247, 101, 35, 19, 102, 159, 29, 233, 182, 86, 199, 53, 244, 175, 208, 228, 116, 49, 45, 21, 118, 178, 72, 104, 19, 188, 1, 131, 254, 127, 37, 139, 95, 60, 129, 254, 7, 18, 184, 51, 199, 217, 71, 107, 203, 113, 33, 123, 83, 246, 30, 234, 39, 214, 99, 207, 167, 116, 247, 66, 241, 15, 133, 22, 9, 238, 99, 105, 166, 167, 72, 142, 182, 229, 22, 51, 127, 84, 32, 214, 8, 185, 111, 104, 100, 152, 181, 219, 81, 90, 133, 133, 127, 114, 2, 134, 80, 248, 53, 178, 96, 3, 247, 147, 225, 186, 125, 75, 198, 237, 177, 202, 247, 71, 49, 31, 123, 47, 222, 47, 12, 75, 219, 147, 5, 27, 92, 161, 76, 204, 58, 211, 26, 120, 156, 19, 51, 136, 52, 116, 18, 166, 216, 205, 2, 3, 1, 0, 1],
RawSubject: [48, 48, 49, 18, 48, 16, 6, 3, 85, 4, 10, 19, 9, 111, 112, 101, 110, 115, 104, 105, 102, 116, 49, 26, 48, 24, 6, 3, 85, 4, 3, 19, 17, 111, 112, 101, 110, 115, 104, 105, 102, 116, 45, 105, 110, 103, 114, 101, 115, 115],
RawIssuer: [48, 38, 49, 18, 48, 16, 6, 3, 85, 4, 11, 19, 9, 111, 112, 101, 110, 115, 104, 105, 102, 116, 49, 16, 48, 14, 6, 3, 85, 4, 3, 19, 7, 114, 111, 111, 116, 45, 99, 97],
Signature: [62, 37, 28, 104, 244, 85, 128, 159, 33, 132, 56, 114, 149, 155, 76, 150, 56, 190, 182, 145, 23, 80, 230, 251, 69, 191, 95, 185, 206, 12, 203, 223, 159, 176, 108, 31, 228, 30, 29, 147, 188, 224, 159, 222, 209, 80, 26, 111, 48, 5, 218, 228, 214, 243, 4, 174, 170, 173, 102, 5, 47, 112, 16, 26, 249, 76, 134, 179, 187, 39, 158, 241, 110, 27, 4, 56, 253, 35, 167, 85, 21, 75, 203, 60, 52, 83, 13, 153, 94, 185, 69, 153, 196, 55, 251, 226, 149, 80, 157, 57, 244, 94, 195, 179, 137, 174, 166, 85, 12, 161, 202, 212, 188, 14, 213, 25, 59, 74, 197, 148, 98, 52, 206, 23, 41, 197, 32, 214, 165, 60, 202, 205, 67, 25, 42, 38, 206, 141, 201, 237, 101, 166, 209, 131, 101, 208, 225, 123, 244, 235, 82, 50, 143, 145, 70, 217, 191, 27, 3, 130, 194, 10, 227, 129, 5, 165, 150, 45, 143, 154, 59, 167, 145, 196, 78, 140, 76, 209, 96, 166, 23, 54, 227, 85, 65, 210, 246, 64, 194, 155, 249, 210, 255, 119, 224, 229, 103, 89, 68, 18, 164, 119, 9, 207, 73, 245, 219, 68, 214, 3, 155, 42, 94, 212, 192, 93, 148, 231, 125, 114, 77, 118, 234, 177, 58, 169, 209, 85, 145, 75, 243, 82, 234, 209, 51, 198, 131, 119, 63, 14, 40, 242, 219, 78, 215, 35, 237, 91, 234, 2, 142, 86, 121, 121, 249, 150],
SignatureAlgorithm: 4,
PublicKeyAlgorithm: 1,
PublicKey: <*rsa.PublicKey | 0x30fb7a9895b0>{
N: {
neg: false,
abs: [3713275565900749005, 5533862720488119315, 886043176200789153, 17818264461995138607, 16265450768615453130, 5834472357489801107, 5862144673391444614, 628656117149906395, 13178964295004987606, 715618696217184398, 12066541168516629782, 6050057140458382287, 14400619811794133371, 6862502860246136883, 1422013248764323211, 8372522752467421288, 13138907997452357860, 17826693266249031145, 5098815638656127509, 5738821655630622400, 12438080729488309683, 6057798802862875801, 7503648299216627878, 13557164215630681915, 9649412887563066602, 17200267080417426931, 12116969928591911697, 17745395029646614848, 8889710084128733264, 16509317662742049607, 11507875977146301207, 12660817078547036528],
},
E: 65537,
},
Version: 3,
SerialNumber: {
neg: false,
abs: [7949786151822990365],
},
Issuer: {
Country: nil,
Organization: nil,
OrganizationalUnit: ["openshift"],
Locality: nil,
Province: nil,
StreetAddress: nil,
PostalCode: nil,
SerialNumber: "",
CommonName: "root-ca",
Names: [
{
Type: [2, 5, 4, 11],
Value: <string>"openshift",
},
{
Type: [2, 5, 4, 3],
Value: <string>"root-ca",
},
],
ExtraNames: nil,
},
Subject: {
Country: nil,
Organization: ["openshift"],
OrganizationalUnit: nil,
Locality: nil,
Province: nil,
StreetAddress: nil,
PostalCode: nil,
SerialNumber: "",
CommonName: "openshift-ingress",
Names: [
{
Type: [2, 5, 4, 10],
Value: <string>"openshift",
},
{
Type: [2, 5, 4, 3],
Value: <string>"openshift-ingress",
},
],
ExtraNames: nil,
},
NotBefore: {wall: 0, ext: 63928459200, loc: nil},
NotAfter: {wall: 0, ext: 63959995200, loc: nil},
KeyUsage: 5,
Extensions: [
{
Id: [2, 5, 29, 15],
Critical: true,
Value: [3, 2, 5, 160],
},
{
Id: [2, 5, 29, 37],
Critical: false,
Value: [48, 20, 6, 8, 43, 6, 1, 5, 5, 7, 3, 2, 6, 8, 43, 6, 1, 5, 5, 7, 3, 1],
},
{Id: [2, 5, 29, 19], Critical: true, Value: [48, 0]},
{
Id: [2, 5, 29, 14],
Critical: false,
Value: [4, 64, 184, 27, 111, 100, 185, 3, 60, 74, 232, 82, 187, 169, 146, 147, 185, 95, 74, 1, 30, 137, 225, 41, 223, 249, 128, 141, 254, 150, 25, 225, 242, 3, 235, 207, 147, 83, 107, 148, 237, 0, 226, 240, 42, 175, 47, 113, 123, 148, 128, 190, 12, 146, 119, 216, 235, 223, 47, 29, 124, 171, 8, 133, 245, 128],
},
{
Id: [2, 5, 29, 35],
Critical: false,
Value: [48, 66, 128, 64, 107, 35, 93, 140, 120, 74, 68, 242, 166, 39, 191, 159, 156, 255, 152, 139, 116, 160, 21, 220, 61, 249, 246, 136, 64, 148, 57, 252, 107, 221, 102, 126, 167, 207, 36, 223, 46, 112, 92, 229, 162, 35, 131, 186, 43, 139, 91, 12, 221, 184, 234, 124, 29, 20, 52, 38, 121, 7, 144, 253, 45, 84, 166, 41],
},
{
Id: [2, 5, 29, 17],
Critical: false,
Value: "0C\x82A*.apps.aro.c9p0v7j9q7j4a1q.afld.westus3.aroapp-hcp.azure-test.net",
},
],
ExtraExtensions: nil,
UnhandledCriticalExtensions: nil,
ExtKeyUsage: [2, 1],
UnknownExtKeyUsage: nil,
BasicConstraintsValid: true,
IsCA: false,
MaxPathLen: -1,
MaxPathLenZero: false,
SubjectKeyId: [184, 27, 111, 100, 185, 3, 60, 74, 232, 82, 187, 169, 146, 147, 185, 95, 74, 1, 30, 137, 225, 41, 223, 249, 128, 141, 254, 150, 25, 225, 242, 3, 235, 207, 147, 83, 107, 148, 237, 0, 226, 240, 42, 175, 47, 113, 123, 148, 128, 190, 12, 146, 119, 216, 235, 223, 47, 29, 124, 171, 8, 133, 245, 128],
AuthorityKeyId: [107, 35, 93, 140, 120, 74, 68, 242, 166, 39, 191, 159, 156, 255, 152, 139, 116, 160, 21, 220, 61, 249, 246, 136, 64, 148, 57, 252, 107, 221, 102, 126, 167, 207, 36, 223, 46, 112, 92, 229, 162, 35, 131, 186, 43, 139, 91, 12, 221, 184, 234, 124, 29, 20, 52, 38, 121, 7, 144, 253, 45, 84, 166, 41],
OCSPServer: nil,
IssuingCertificateURL: nil,
DNSNames: [
"*.apps.aro.c9p0v7j9q7j4a1q.afld.westus3.aroapp-hcp.azure-test.net",
],
EmailAddresses: nil,
IPAddresses: nil,
URIs: nil,
PermittedDNSDomainsCritical: false,
PermittedDNSDomains: nil,
ExcludedDNSDomains: nil,
PermittedIPRanges: nil,
ExcludedIPRanges: nil,
PermittedEmailAddresses: nil,
ExcludedEmailAddresses: nil,
PermittedURIDomains: nil,
ExcludedURIDomains: nil,
CRLDistributionPoints: nil,
PolicyIdentifiers: nil,
Policies: nil,
InhibitAnyPolicy: 0,
InhibitAnyPolicyZero: false,
InhibitPolicyMapping: 0,
InhibitPolicyMappingZero: false,
RequireExplicitPolicy: 0,
RequireExplicitPolicyZero: false,
PolicyMappings: nil,
},
],
Err: <x509.UnknownAuthorityError>{
Cert: 0x30fb7a8f2c08 (cyclic reference),
hintErr: nil,
hintCert: nil,
},
},
},
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:194]: failed to verify simple web app runs on cluster "cluster-candidate-4-23-wk4gjj"
Unexpected error:
<*fmt.wrapError | 0x30fb7a9c6d80>:
[strict TLS verification] route was never reachable: Get "https://agnhost-e2e-sample-app-25lgf.apps.aro.c9p0v7j9q7j4a1q.afld.westus3.aroapp-hcp.azure-test.net": tls: failed to verify certificate: x509: certificate signed by unknown authority
{
msg: "[strict TLS verification] route was never reachable: Get \"https://agnhost-e2e-sample-app-25lgf.apps.aro.c9p0v7j9q7j4a1q.afld.westus3.aroapp-hcp.azure-test.net\": tls: failed to verify certificate: x509: certificate signed by unknown authority",
err: <*url.Error | 0x30fb7cb9b320>{
Op: "Get",
URL: "https://agnhost-e2e-sample-app-25lgf.apps.aro.c9p0v7j9q7j4a1q.afld.westus3.aroapp-hcp.azure-test.net",
Err: <*tls.CertificateVerificationError | 0x30fb7cb9b2f0>{
UnverifiedCertificates: [
{
Raw: [48, 130, 4, 1, 48, 130, 2, 233, 160, 3, 2, 1, 2, 2, 8, 110, 83, 80, 98, 163, 186, 156, 29, 48, 13, 6, 9, 42, 134, 72, 134, 247, 13, 1, 1, 11, 5, 0, 48, 38, 49, 18, 48, 16, 6, 3, 85, 4, 11, 19, 9, 111, 112, 101, 110, 115, 104, 105, 102, 116, 49, 16, 48, 14, 6, 3, 85, 4, 3, 19, 7, 114, 111, 111, 116, 45, 99, 97, 48, 30, 23, 13, 50, 54, 49, 48, 50, 52, 49, 55, 50, 48, 48, 48, 90, 23, 13, 50, 55, 49, 48, 50, 52, 49, 55, 50, 48, 48, 48, 90, 48, 48, 49, 18, 48, 16, 6, 3, 85, 4, 10, 19, 9, 111, 112, 101, 110, 115, 104, 105, 102, 116, 49, 26, 48, 24, 6, 3, 85, 4, 3, 19, 17, 111, 112, 101, 110, 115, 104, 105, 102, 116, 45, 105, 110, 103, 114, 101, 115, 115, 48, 130, 1, 34, 48, 13, 6, 9, 42, 134, 72, 134, 247, 13, 1, 1, 1, 5, 0, 3, 130, 1, 15, 0, 48, 130, 1, 10, 2, 130, 1, 1, 0, 175, 180, 66, 13, 148, 254, 185, 112, 159, 180, 48, 58, 217, 14, 123, 23, 229, 28, 224, 241, 179, 210, 115, 71, 123, 94, 152, 56, 200, 179, 120, 80, 246, 68, 78, 194, 165, 17, 161, 64, 168, 40, 31, 243, 155, 207, 171, 17, 238, 179, 159, 199, 90, 32, 61, 243, 133, 233, 153, 229, 126, 69, 220, 234, 188, 36, 185, 6, 114, 77, 215, 59, 104, 34, 80, 95, 66, 69, 104, 166, 84, 17, 159, 234, 101, 109, 164, 153, 172, 156, 240, 134, 16, 200, 61, 179, 79, 164, 99, 227, 229, 46, 174, 192, 70, 194, 161, 207, 5, 78, 18, 21, 247, 101, 35, 19, 102, 159, 29, 233, 182, 86, 199, 53, 244, 175, 208, 228, 116, 49, 45, 21, 118, 178, 72, 104, 19, 188, 1, 131, 254, 127, 37, 139, 95, 60, 129, 254, 7, 18, 184, 51, 199, 217, 71, 107, 203, 113, 33, 123, 83, 246, 30, 234, 39, 214, 99, 207, 167, 116, 247, 66, 241, 15, 133, 22, 9, 238, 99, 105, 166, 167, 72, 142, 182, 229, 22, 51, 127, 84, 32, 214, 8, 185, 111, 104, 100, 152, 181, 219, 81, 90, 133, 133, 127, 114, 2, 134, 80, 248, 53, 178, 96, 3, 247, 147, 225, 186, 125, 75, 198, 237, 177, 202, 247, 71, 49, 31, 123, 47, 222, 47, 12, 75, 219, 147, 5, 27, 92, 161, 76, 204, 58, 211, 26, 120, 156, 19, 51, 136, 52, 116, 18, 166, 216, 205, 2, 3, 1, 0, 1, 163, 130, 1, 39, 48, 130, 1, 35, 48, 14, 6, 3, 85, 29, 15, 1, 1, 255, 4, 4, 3, 2, 5, 160, 48, 29, 6, 3, 85, 29, 37, 4, 22, 48, 20, 6, 8, 43, 6, 1, 5, 5, 7, 3, 2, 6, 8, 43, 6, 1, 5, 5, 7, 3, 1, 48, 12, 6, 3, 85, 29, 19, 1, 1, 255, 4, 2, 48, 0, 48, 73, 6, 3, 85, 29, 14, 4, 66, 4, 64, 184, 27, 111, 100, 185, 3, 60, 74, 232, 82, 187, 169, 146, 147, 185, 95, 74, 1, 30, 137, 225, 41, 223, 249, 128, 141, 254, 150, 25, 225, 242, 3, 235, 207, 147, 83, 107, 148, 237, 0, 226, 240, 42, 175, 47, 113, 123, 148, 128, 190, 12, 146, 119, 216, 235, 223, 47, 29, 124, 171, 8, 133, 245, 128, 48, 75, 6, 3, 85, 29, 35, 4, 68, 48, 66, 128, 64, 107, 35, 93, 140, 120, 74, 68, 242, 166, 39, 191, 159, 156, 255, 152, 139, 116, 160, 21, 220, 61, 249, 246, 136, 64, 148, 57, 252, 107, 221, 102, 126, 167, 207, 36, 223, 46, 112, 92, 229, 162, 35, 131, 186, 43, 139, 91, 12, 221, 184, 234, 124, 29, 20, 52, 38, 121, 7, 144, 253, 45, 84, 166, 41, 48, 76, 6, 3, 85, 29, 17, 4, 69, 48, 67, 130, 65, 42, 46, 97, 112, 112, 115, 46, 97, 114, 111, 46, 99, 57, 112, 48, 118, 55, 106, 57, 113, 55, 106, 52, 97, 49, 113, 46, 97, 102, 108, 100, 46, 119, 101, 115, 116, 117, 115, 51, 46, 97, 114, 111, 97, 112, 112, 45, 104, 99, 112, 46, 97, 122, 117, 114, 101, 45, 116, 101, 115, 116, 46, 110, 101, 116, 48, 13, 6, 9, 42, 134, 72, 134, 247, 13, 1, 1, 11, 5, 0, 3, 130, 1, 1, 0, 62, 37, 28, 104, 244, 85, 128, 159, 33, 132, 56, 114, 149, 155, 76, 150, 56, 190, 182, 145, 23, 80, 230, 251, 69, 191, 95, 185, 206, 12, 203, 223, 159, 176, 108, 31, 228, 30, 29, 147, 188, 224, 159, 222, 209, 80, 26, 111, 48, 5, 218, 228, 214, 243, 4, 174, 170, 173, 102, 5, 47, 112, 16, 26, 249, 76, 134, 179, 187, 39, 158, 241, 110, 27, 4, 56, 253, 35, 167, 85, 21, 75, 203, 60, 52, 83, 13, 153, 94, 185, 69, 153, 196, 55, 251, 226, 149, 80, 157, 57, 244, 94, 195, 179, 137, 174, 166, 85, 12, 161, 202, 212, 188, 14, 213, 25, 59, 74, 197, 148, 98, 52, 206, 23, 41, 197, 32, 214, 165, 60, 202, 205, 67, 25, 42, 38, 206, 141, 201, 237, 101, 166, 209, 131, 101, 208, 225, 123, 244, 235, 82, 50, 143, 145, 70, 217, 191, 27, 3, 130, 194, 10, 227, 129, 5, 165, 150, 45, 143, 154, 59, 167, 145, 196, 78, 140, 76, 209, 96, 166, 23, 54, 227, 85, 65, 210, 246, 64, 194, 155, 249, 210, 255, 119, 224, 229, 103, 89, 68, 18, 164, 119, 9, 207, 73, 245, 219, 68, 214, 3, 155, 42, 94, 212, 192, 93, 148, 231, 125, 114, 77, 118, 234, 177, 58, 169, 209, 85, 145, 75, 243, 82, 234, 209, 51, 198, 131, 119, 63, 14, 40, 242, 219, 78, 215, 35, 237, 91, 234, 2, 142, 86, 121, 121, 249, 150],
RawTBSCertificate: [48, 130, 2, 233, 160, 3, 2, 1, 2, 2, 8, 110, 83, 80, 98, 163, 186, 156, 29, 48, 13, 6, 9, 42, 134, 72, 134, 247, 13, 1, 1, 11, 5, 0, 48, 38, 49, 18, 48, 16, 6, 3, 85, 4, 11, 19, 9, 111, 112, 101, 110, 115, 104, 105, 102, 116, 49, 16, 48, 14, 6, 3, 85, 4, 3, 19, 7, 114, 111, 111, 116, 45, 99, 97, 48, 30, 23, 13, 50, 54, 49, 48, 50, 52, 49, 55, 50, 48, 48, 48, 90, 23, 13, 50, 55, 49, 48, 50, 52, 49, 55, 50, 48, 48, 48, 90, 48, 48, 49, 18, 48, 16, 6, 3, 85, 4, 10, 19, 9, 111, 112, 101, 110, 115, 104, 105, 102, 116, 49, 26, 48, 24, 6, 3, 85, 4, 3, 19, 17, 111, 112, 101, 110, 115, 104, 105, 102, 116, 45, 105, 110, 103, 114, 101, 115, 115, 48, 130, 1, 34, 48, 13, 6, 9, 42, 134, 72, 134, 247, 13, 1, 1, 1, 5, 0, 3, 130, 1, 15, 0, 48, 130, 1, 10, 2, 130, 1, 1, 0, 175, 180, 66, 13, 148, 254, 185, 112, 159, 180, 48, 58, 217, 14, 123, 23, 229, 28, 224, 241, 179, 210, 115, 71, 123, 94, 152, 56, 200, 179, 120, 80, 246, 68, 78, 194, 165, 17, 161, 64, 168, 40, 31, 243, 155, 207, 171, 17, 238, 179, 159, 199, 90, 32, 61, 243, 133, 233, 153, 229, 126, 69, 220, 234, 188, 36, 185, 6, 114, 77, 215, 59, 104, 34, 80, 95, 66, 69, 104, 166, 84, 17, 159, 234, 101, 109, 164, 153, 172, 156, 240, 134, 16, 200, 61, 179, 79, 164, 99, 227, 229, 46, 174, 192, 70, 194, 161, 207, 5, 78, 18, 21, 247, 101, 35, 19, 102, 159, 29, 233, 182, 86, 199, 53, 244, 175, 208, 228, 116, 49, 45, 21, 118, 178, 72, 104, 19, 188, 1, 131, 254, 127, 37, 139, 95, 60, 129, 254, 7, 18, 184, 51, 199, 217, 71, 107, 203, 113, 33, 123, 83, 246, 30, 234, 39, 214, 99, 207, 167, 116, 247, 66, 241, 15, 133, 22, 9, 238, 99, 105, 166, 167, 72, 142, 182, 229, 22, 51, 127, 84, 32, 214, 8, 185, 111, 104, 100, 152, 181, 219, 81, 90, 133, 133, 127, 114, 2, 134, 80, 248, 53, 178, 96, 3, 247, 147, 225, 186, 125, 75, 198, 237, 177, 202, 247, 71, 49, 31, 123, 47, 222, 47, 12, 75, 219, 147, 5, 27, 92, 161, 76, 204, 58, 211, 26, 120, 156, 19, 51, 136, 52, 116, 18, 166, 216, 205, 2, 3, 1, 0, 1, 163, 130, 1, 39, 48, 130, 1, 35, 48, 14, 6, 3, 85, 29, 15, 1, 1, 255, 4, 4, 3, 2, 5, 160, 48, 29, 6, 3, 85, 29, 37, 4, 22, 48, 20, 6, 8, 43, 6, 1, 5, 5, 7, 3, 2, 6, 8, 43, 6, 1, 5, 5, 7, 3, 1, 48, 12, 6, 3, 85, 29, 19, 1, 1, 255, 4, 2, 48, 0, 48, 73, 6, 3, 85, 29, 14, 4, 66, 4, 64, 184, 27, 111, 100, 185, 3, 60, 74, 232, 82, 187, 169, 146, 147, 185, 95, 74, 1, 30, 137, 225, 41, 223, 249, 128, 141, 254, 150, 25, 225, 242, 3, 235, 207, 147, 83, 107, 148, 237, 0, 226, 240, 42, 175, 47, 113, 123, 148, 128, 190, 12, 146, 119, 216, 235, 223, 47, 29, 124, 171, 8, 133, 245, 128, 48, 75, 6, 3, 85, 29, 35, 4, 68, 48, 66, 128, 64, 107, 35, 93, 140, 120, 74, 68, 242, 166, 39, 191, 159, 156, 255, 152, 139, 116, 160, 21, 220, 61, 249, 246, 136, 64, 148, 57, 252, 107, 221, 102, 126, 167, 207, 36, 223, 46, 112, 92, 229, 162, 35, 131, 186, 43, 139, 91, 12, 221, 184, 234, 124, 29, 20, 52, 38, 121, 7, 144, 253, 45, 84, 166, 41, 48, 76, 6, 3, 85, 29, 17, 4, 69, 48, 67, 130, 65, 42, 46, 97, 112, 112, 115, 46, 97, 114, 111, 46, 99, 57, 112, 48, 118, 55, 106, 57, 113, 55, 106, 52, 97, 49, 113, 46, 97, 102, 108, 100, 46, 119, 101, 115, 116, 117, 115, 51, 46, 97, 114, 111, 97, 112, 112, 45, 104, 99, 112, 46, 97, 122, 117, 114, 101, 45, 116, 101, 115, 116, 46, 110, 101, 116],
RawSubjectPublicKeyInfo: [48, 130, 1, 34, 48, 13, 6, 9, 42, 134, 72, 134, 247, 13, 1, 1, 1, 5, 0, 3, 130, 1, 15, 0, 48, 130, 1, 10, 2, 130, 1, 1, 0, 175, 180, 66, 13, 148, 254, 185, 112, 159, 180, 48, 58, 217, 14, 123, 23, 229, 28, 224, 241, 179, 210, 115, 71, 123, 94, 152, 56, 200, 179, 120, 80, 246, 68, 78, 194, 165, 17, 161, 64, 168, 40, 31, 243, 155, 207, 171, 17, 238, 179, 159, 199, 90, 32, 61, 243, 133, 233, 153, 229, 126, 69, 220, 234, 188, 36, 185, 6, 114, 77, 215, 59, 104, 34, 80, 95, 66, 69, 104, 166, 84, 17, 159, 234, 101, 109, 164, 153, 172, 156, 240, 134, 16, 200, 61, 179, 79, 164, 99, 227, 229, 46, 174, 192, 70, 194, 161, 207, 5, 78, 18, 21, 247, 101, 35, 19, 102, 159, 29, 233, 182, 86, 199, 53, 244, 175, 208, 228, 116, 49, 45, 21, 118, 178, 72, 104, 19, 188, 1, 131, 254, 127, 37, 139, 95, 60, 129, 254, 7, 18, 184, 51, 199, 217, 71, 107, 203, 113, 33, 123, 83, 246, 30, 234, 39, 214, 99, 207, 167, 116, 247, 66, 241, 15, 133, 22, 9, 238, 99, 105, 166, 167, 72, 142, 182, 229, 22, 51, 127, 84, 32, 214, 8, 185, 111, 104, 100, 152, 181, 219, 81, 90, 133, 133, 127, 114, 2, 134, 80, 248, 53, 178, 96, 3, 247, 147, 225, 186, 125, 75, 198, 237, 177, 202, 247, 71, 49, 31, 123, 47, 222, 47, 12, 75, 219, 147, 5, 27, 92, 161, 76, 204, 58, 211, 26, 120, 156, 19, 51, 136, 52, 116, 18, 166, 216, 205, 2, 3, 1, 0, 1],
RawSubject: [48, 48, 49, 18, 48, 16, 6, 3, 85, 4, 10, 19, 9, 111, 112, 101, 110, 115, 104, 105, 102, 116, 49, 26, 48, 24, 6, 3, 85, 4, 3, 19, 17, 111, 112, 101, 110, 115, 104, 105, 102, 116, 45, 105, 110, 103, 114, 101, 115, 115],
RawIssuer: [48, 38, 49, 18, 48, 16, 6, 3, 85, 4, 11, 19, 9, 111, 112, 101, 110, 115, 104, 105, 102, 116, 49, 16, 48, 14, 6, 3, 85, 4, 3, 19, 7, 114, 111, 111, 116, 45, 99, 97],
Signature: [62, 37, 28, 104, 244, 85, 128, 159, 33, 132, 56, 114, 149, 155, 76, 150, 56, 190, 182, 145, 23, 80, 230, 251, 69, 191, 95, 185, 206, 12, 203, 223, 159, 176, 108, 31, 228, 30, 29, 147, 188, 224, 159, 222, 209, 80, 26, 111, 48, 5, 218, 228, 214, 243, 4, 174, 170, 173, 102, 5, 47, 112, 16, 26, 249, 76, 134, 179, 187, 39, 158, 241, 110, 27, 4, 56, 253, 35, 167, 85, 21, 75, 203, 60, 52, 83, 13, 153, 94, 185, 69, 153, 196, 55, 251, 226, 149, 80, 157, 57, 244, 94, 195, 179, 137, 174, 166, 85, 12, 161, 202, 212, 188, 14, 213, 25, 59, 74, 197, 148, 98, 52, 206, 23, 41, 197, 32, 214, 165, 60, 202, 205, 67, 25, 42, 38, 206, 141, 201, 237, 101, 166, 209, 131, 101, 208, 225, 123, 244, 235, 82, 50, 143, 145, 70, 217, 191, 27, 3, 130, 194, 10, 227, 129, 5, 165, 150, 45, 143, 154, 59, 167, 145, 196, 78, 140, 76, 209, 96, 166, 23, 54, 227, 85, 65, 210, 246, 64, 194, 155, 249, 210, 255, 119, 224, 229, 103, 89, 68, 18, 164, 119, 9, 207, 73, 245, 219, 68, 214, 3, 155, 42, 94, 212, 192, 93, 148, 231, 125, 114, 77, 118, 234, 177, 58, 169, 209, 85, 145, 75, 243, 82, 234, 209, 51, 198, 131, 119, 63, 14, 40, 242, 219, 78, 215, 35, 237, 91, 234, 2, 142, 86, 121, 121, 249, 150],
SignatureAlgorithm: 4,
PublicKeyAlgorithm: 1,
PublicKey: <*rsa.PublicKey | 0x30fb7a9895b0>{
N: {
neg: false,
abs: [3713275565900749005, 5533862720488119315, 886043176200789153, 17818264461995138607, 16265450768615453130, 5834472357489801107, 5862144673391444614, 628656117149906395, 13178964295004987606, 715618696217184398, 12066541168516629782, 6050057140458382287, 14400619811794133371, 6862502860246136883, 1422013248764323211, 8372522752467421288, 13138907997452357860, 17826693266249031145, 5098815638656127509, 5738821655630622400, 12438080729488309683, 6057798802862875801, 7503648299216627878, 13557164215630681915, 9649412887563066602, 17200267080417426931, 12116969928591911697, 17745395029646614848, 8889710084128733264, 16509317662742049607, 11507875977146301207, 12660817078547036528],
},
E: 65537,
},
Version: 3,
SerialNumber: {
neg: false,
abs: [7949786151822990365],
},
Issuer: {
Country: nil,
Organization: nil,
OrganizationalUnit: ["openshift"],
Locality: nil,
Province: nil,
StreetAddress: nil,
PostalCode: nil,
SerialNumber: "",
CommonName: "root-ca",
Names: [
{
Type: [2, 5, 4, 11],
Value: <string>"openshift",
},
{
Type: [2, 5, 4, 3],
Value: <string>"root-ca",
},
],
ExtraNames: nil,
},
Subject: {
Country: nil,
Organization: ["openshift"],
OrganizationalUnit: nil,
Locality: nil,
Province: nil,
StreetAddress: nil,
PostalCode: nil,
SerialNumber: "",
CommonName: "openshift-ingress",
Names: [
{
Type: [2, 5, 4, 10],
Value: <string>"openshift",
},
{
Type: [2, 5, 4, 3],
Value: <string>"openshift-ingress",
},
],
ExtraNames: nil,
},
NotBefore: {wall: 0, ext: 63928459200, loc: nil},
NotAfter: {wall: 0, ext: 63959995200, loc: nil},
KeyUsage: 5,
Extensions: [
{
Id: [2, 5, 29, 15],
Critical: true,
Value: [3, 2, 5, 160],
},
{
Id: [2, 5, 29, 37],
Critical: false,
Value: [48, 20, 6, 8, 43, 6, 1, 5, 5, 7, 3, 2, 6, 8, 43, 6, 1, 5, 5, 7, 3, 1],
},
{Id: [2, 5, 29, 19], Critical: true, Value: [48, 0]},
{
Id: [2, 5, 29, 14],
Critical: false,
Value: [4, 64, 184, 27, 111, 100, 185, 3, 60, 74, 232, 82, 187, 169, 146, 147, 185, 95, 74, 1, 30, 137, 225, 41, 223, 249, 128, 141, 254, 150, 25, 225, 242, 3, 235, 207, 147, 83, 107, 148, 237, 0, 226, 240, 42, 175, 47, 113, 123, 148, 128, 190, 12, 146, 119, 216, 235, 223, 47, 29, 124, 171, 8, 133, 245, 128],
},
{
Id: [2, 5, 29, 35],
Critical: false,
Value: [48, 66, 128, 64, 107, 35, 93, 140, 120, 74, 68, 242, 166, 39, 191, 159, 156, 255, 152, 139, 116, 160, 21, 220, 61, 249, 246, 136, 64, 148, 57, 252, 107, 221, 102, 126, 167, 207, 36, 223, 46, 112, 92, 229, 162, 35, 131, 186, 43, 139, 91, 12, 221, 184, 234, 124, 29, 20, 52, 38, 121, 7, 144, 253, 45, 84, 166, 41],
},
{
Id: [2, 5, 29, 17],
Critical: false,
Value: "0C\x82A*.apps.aro.c9p0v7j9q7j4a1q.afld.westus3.aroapp-hcp.azure-test.net",
},
],
ExtraExtensions: nil,
UnhandledCriticalExtensions: nil,
ExtKeyUsage: [2, 1],
UnknownExtKeyUsage: nil,
BasicConstraintsValid: true,
IsCA: false,
MaxPathLen: -1,
MaxPathLenZero: false,
SubjectKeyId: [184, 27, 111, 100, 185, 3, 60, 74, 232, 82, 187, 169, 146, 147, 185, 95, 74, 1, 30, 137, 225, 41, 223, 249, 128, 141, 254, 150, 25, 225, 242, 3, 235, 207, 147, 83, 107, 148, 237, 0, 226, 240, 42, 175, 47, 113, 123, 148, 128, 190, 12, 146, 119, 216, 235, 223, 47, 29, 124, 171, 8, 133, 245, 128],
AuthorityKeyId: [107, 35, 93, 140, 120, 74, 68, 242, 166, 39, 191, 159, 156, 255, 152, 139, 116, 160, 21, 220, 61, 249, 246, 136, 64, 148, 57, 252, 107, 221, 102, 126, 167, 207, 36, 223, 46, 112, 92, 229, 162, 35, 131, 186, 43, 139, 91, 12, 221, 184, 234, 124, 29, 20, 52, 38, 121, 7, 144, 253, 45, 84, 166, 41],
OCSPServer: nil,
IssuingCertificateURL: nil,
DNSNames: [
"*.apps.aro.c9p0v7j9q7j4a1q.afld.westus3.aroapp-hcp.azure-test.net",
],
EmailAddresses: nil,
IPAddresses: nil,
URIs: nil,
PermittedDNSDomainsCritical: false,
PermittedDNSDomains: nil,
ExcludedDNSDomains: nil,
PermittedIPRanges: nil,
ExcludedIPRanges: nil,
PermittedEmailAddresses: nil,
ExcludedEmailAddresses: nil,
PermittedURIDomains: nil,
ExcludedURIDomains: nil,
CRLDistributionPoints: nil,
PolicyIdentifiers: nil,
Policies: nil,
InhibitAnyPolicy: 0,
InhibitAnyPolicyZero: false,
InhibitPolicyMapping: 0,
InhibitPolicyMappingZero: false,
RequireExplicitPolicy: 0,
RequireExplicitPolicyZero: false,
PolicyMappings: nil,
},
],
Err: <x509.UnknownAuthorityError>{
Cert: 0x30fb7a8f2c08 (cyclic reference),
hintErr: nil,
hintCert: nil,
},
},
},
}
occurredContributing tests (1)
Affected runs (12)
| |||||||||||||||||||||||||||||||||||||||||||||||||||||
CreateOrUpdateExternalAuthAndWait timed out after <minutes> minutes; context deadline exceededfull failure pattern: CreateOrUpdateExternalAuthAndWait timed out after <minutes> minutes; context deadline exceeded Signal: IndeterminateSignal: Indeterminate — no prior history; active 3 day(s) — New failure pattern — no prior history | unknown | 4 | 13.79%4 of 29 job runs affected | IndeterminateSignal: Indeterminate — no prior history; active 3 day(s) — New failure pattern — no prior history | Show trend detailsOct 12: 0 · Oct 13: 0 · Oct 14: 0 · Oct 15: 0 · Oct 16: 0 · Oct 17: 0 · Oct 18: 0 · Oct 19: 0 · Oct 20: 0 · Oct 21: 1 · Oct 22: 2 · Oct 23: 1 · Oct 24: 0 · Oct 25: 0 | PROD, STG | |||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (3)fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:181]: failed to create external auth config external-auth
Unexpected error:
<*fmt.wrapErrors | 0x1977dbbad170>:
failed waiting for external auth "external-auth" in resourcegroup="external-auth-cluster-c7xfsrnzff2g" for cluster="ea-cluster" to finish creating or updating, caused by: timeout '15.000000' minutes exceeded during CreateOrUpdateExternalAuthAndWait for external auth external-auth in cluster ea-cluster in resource group external-auth-cluster-c7xfsrnzff2g, error: context deadline exceeded
{
msg: "failed waiting for external auth \"external-auth\" in resourcegroup=\"external-auth-cluster-c7xfsrnzff2g\" for cluster=\"ea-cluster\" to finish creating or updating, caused by: timeout '15.000000' minutes exceeded during CreateOrUpdateExternalAuthAndWait for external auth external-auth in cluster ea-cluster in resource group external-auth-cluster-c7xfsrnzff2g, error: context deadline exceeded",
errs: [
<*errors.errorString | 0x1977dcaa2220>{
s: "timeout '15.000000' minutes exceeded during CreateOrUpdateExternalAuthAndWait for external auth external-auth in cluster ea-cluster in resource group external-auth-cluster-c7xfsrnzff2g",
},
<context.deadlineExceededError>{},
],
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:181]: failed to create external auth config external-auth
Unexpected error:
<*fmt.wrapErrors | 0x1977dbbad170>:
failed waiting for external auth "external-auth" in resourcegroup="external-auth-cluster-c7xfsrnzff2g" for cluster="ea-cluster" to finish creating or updating, caused by: timeout '15.000000' minutes exceeded during CreateOrUpdateExternalAuthAndWait for external auth external-auth in cluster ea-cluster in resource group external-auth-cluster-c7xfsrnzff2g, error: context deadline exceeded
{
msg: "failed waiting for external auth \"external-auth\" in resourcegroup=\"external-auth-cluster-c7xfsrnzff2g\" for cluster=\"ea-cluster\" to finish creating or updating, caused by: timeout '15.000000' minutes exceeded during CreateOrUpdateExternalAuthAndWait for external auth external-auth in cluster ea-cluster in resource group external-auth-cluster-c7xfsrnzff2g, error: context deadline exceeded",
errs: [
<*errors.errorString | 0x1977dcaa2220>{
s: "timeout '15.000000' minutes exceeded during CreateOrUpdateExternalAuthAndWait for external auth external-auth in cluster ea-cluster in resource group external-auth-cluster-c7xfsrnzff2g",
},
<context.deadlineExceededError>{},
],
}
occurredfail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:276]: failed to create external auth config "ext-auth-cidr"
Unexpected error:
<*fmt.wrapErrors | 0x2cc075309380>:
failed waiting for external auth "ext-auth-cidr" in resourcegroup="e2e-cidr-connectivity-rhgbbsmxx2nr" for cluster="cidr-connectivity-test" to finish creating or updating, caused by: timeout '15.000000' minutes exceeded during CreateOrUpdateExternalAuthAndWait for external auth ext-auth-cidr in cluster cidr-connectivity-test in resource group e2e-cidr-connectivity-rhgbbsmxx2nr, error: context deadline exceeded
{
msg: "failed waiting for external auth \"ext-auth-cidr\" in resourcegroup=\"e2e-cidr-connectivity-rhgbbsmxx2nr\" for cluster=\"cidr-connectivity-test\" to finish creating or updating, caused by: timeout '15.000000' minutes exceeded during CreateOrUpdateExternalAuthAndWait for external auth ext-auth-cidr in cluster cidr-connectivity-test in resource group e2e-cidr-connectivity-rhgbbsmxx2nr, error: context deadline exceeded",
errs: [
<*errors.errorString | 0x2cc074e74090>{
s: "timeout '15.000000' minutes exceeded during CreateOrUpdateExternalAuthAndWait for external auth ext-auth-cidr in cluster cidr-connectivity-test in resource group e2e-cidr-connectivity-rhgbbsmxx2nr",
},
<context.deadlineExceededError>{},
],
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:276]: failed to create external auth config "ext-auth-cidr"
Unexpected error:
<*fmt.wrapErrors | 0x2cc075309380>:
failed waiting for external auth "ext-auth-cidr" in resourcegroup="e2e-cidr-connectivity-rhgbbsmxx2nr" for cluster="cidr-connectivity-test" to finish creating or updating, caused by: timeout '15.000000' minutes exceeded during CreateOrUpdateExternalAuthAndWait for external auth ext-auth-cidr in cluster cidr-connectivity-test in resource group e2e-cidr-connectivity-rhgbbsmxx2nr, error: context deadline exceeded
{
msg: "failed waiting for external auth \"ext-auth-cidr\" in resourcegroup=\"e2e-cidr-connectivity-rhgbbsmxx2nr\" for cluster=\"cidr-connectivity-test\" to finish creating or updating, caused by: timeout '15.000000' minutes exceeded during CreateOrUpdateExternalAuthAndWait for external auth ext-auth-cidr in cluster cidr-connectivity-test in resource group e2e-cidr-connectivity-rhgbbsmxx2nr, error: context deadline exceeded",
errs: [
<*errors.errorString | 0x2cc074e74090>{
s: "timeout '15.000000' minutes exceeded during CreateOrUpdateExternalAuthAndWait for external auth ext-auth-cidr in cluster cidr-connectivity-test in resource group e2e-cidr-connectivity-rhgbbsmxx2nr",
},
<context.deadlineExceededError>{},
],
}
occurredfail [github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:181]: failed to create external auth config external-auth
Unexpected error:
<*fmt.wrapErrors | 0x32cbbfa08480>:
failed waiting for external auth "external-auth" in resourcegroup="external-auth-cluster-s8rx2m7kc8lm" for cluster="ea-cluster" to finish creating or updating, caused by: timeout '15.000000' minutes exceeded during CreateOrUpdateExternalAuthAndWait for external auth external-auth in cluster ea-cluster in resource group external-auth-cluster-s8rx2m7kc8lm, error: context deadline exceeded
{
msg: "failed waiting for external auth \"external-auth\" in resourcegroup=\"external-auth-cluster-s8rx2m7kc8lm\" for cluster=\"ea-cluster\" to finish creating or updating, caused by: timeout '15.000000' minutes exceeded during CreateOrUpdateExternalAuthAndWait for external auth external-auth in cluster ea-cluster in resource group external-auth-cluster-s8rx2m7kc8lm, error: context deadline exceeded",
errs: [
<*errors.errorString | 0x32cbbf682f10>{
s: "timeout '15.000000' minutes exceeded during CreateOrUpdateExternalAuthAndWait for external auth external-auth in cluster ea-cluster in resource group external-auth-cluster-s8rx2m7kc8lm",
},
<context.deadlineExceededError>{},
],
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:181]: failed to create external auth config external-auth
Unexpected error:
<*fmt.wrapErrors | 0x32cbbfa08480>:
failed waiting for external auth "external-auth" in resourcegroup="external-auth-cluster-s8rx2m7kc8lm" for cluster="ea-cluster" to finish creating or updating, caused by: timeout '15.000000' minutes exceeded during CreateOrUpdateExternalAuthAndWait for external auth external-auth in cluster ea-cluster in resource group external-auth-cluster-s8rx2m7kc8lm, error: context deadline exceeded
{
msg: "failed waiting for external auth \"external-auth\" in resourcegroup=\"external-auth-cluster-s8rx2m7kc8lm\" for cluster=\"ea-cluster\" to finish creating or updating, caused by: timeout '15.000000' minutes exceeded during CreateOrUpdateExternalAuthAndWait for external auth external-auth in cluster ea-cluster in resource group external-auth-cluster-s8rx2m7kc8lm, error: context deadline exceeded",
errs: [
<*errors.errorString | 0x32cbbf682f10>{
s: "timeout '15.000000' minutes exceeded during CreateOrUpdateExternalAuthAndWait for external auth external-auth in cluster ea-cluster in resource group external-auth-cluster-s8rx2m7kc8lm",
},
<context.deadlineExceededError>{},
],
}
occurredContributing tests (2)
Affected runs (4)
| |||||||||||||||||||||||||||||||||||||||||||||||||||||
DNS for route host <route-host> did not resolve: DNS for <route-host> did not resolve within <duration> (last error: lookup <host> on <dns-server>: dial udp <ip>: i/o timeout): co...full failure pattern: DNS for route host <route-host> did not resolve: DNS for <route-host> did not resolve within <duration> (last error: lookup <host> on <dns-server>: dial udp <ip>: i/o timeout): context deadline exceeded Signal: IndeterminateSignal: Indeterminate — no prior history; active 3 day(s) — New failure pattern — no prior history | unknown | 3 | 10.34%3 of 29 job runs affected | IndeterminateSignal: Indeterminate — no prior history; active 3 day(s) — New failure pattern — no prior history | Show trend detailsOct 12: 0 · Oct 13: 0 · Oct 14: 0 · Oct 15: 0 · Oct 16: 0 · Oct 17: 0 · Oct 18: 0 · Oct 19: 0 · Oct 20: 0 · Oct 21: 0 · Oct 22: 1 · Oct 23: 1 · Oct 24: 1 · Oct 25: 0 | DEV, PROD | |||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (3)fail [github.com/Azure/ARO-HCP/test/e2e/cluster_nodepool_active_versions.go:149]: failed to verify simple web app runs on v20261001 cluster "v20261001"
Unexpected error:
<*fmt.wrapError | 0xdad4da81e80>:
DNS for route host agnhost-e2e-sample-app-4gnm9.apps.aro.v20261001.bhnx.westus3.aroapp-hcp.azure-test.net did not resolve: DNS for agnhost-e2e-sample-app-4gnm9.apps.aro.v20261001.bhnx.westus3.aroapp-hcp.azure-test.net did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-4gnm9.apps.aro.v20261001.bhnx.westus3.aroapp-hcp.azure-test.net on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded
{
msg: "DNS for route host agnhost-e2e-sample-app-4gnm9.apps.aro.v20261001.bhnx.westus3.aroapp-hcp.azure-test.net did not resolve: DNS for agnhost-e2e-sample-app-4gnm9.apps.aro.v20261001.bhnx.westus3.aroapp-hcp.azure-test.net did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-4gnm9.apps.aro.v20261001.bhnx.westus3.aroapp-hcp.azure-test.net on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
err: <*fmt.wrapError | 0xdad4d9b41a0>{
msg: "DNS for agnhost-e2e-sample-app-4gnm9.apps.aro.v20261001.bhnx.westus3.aroapp-hcp.azure-test.net did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-4gnm9.apps.aro.v20261001.bhnx.westus3.aroapp-hcp.azure-test.net on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
err: <context.deadlineExceededError>{},
},
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_nodepool_active_versions.go:149]: failed to verify simple web app runs on v20261001 cluster "v20261001"
Unexpected error:
<*fmt.wrapError | 0xdad4da81e80>:
DNS for route host agnhost-e2e-sample-app-4gnm9.apps.aro.v20261001.bhnx.westus3.aroapp-hcp.azure-test.net did not resolve: DNS for agnhost-e2e-sample-app-4gnm9.apps.aro.v20261001.bhnx.westus3.aroapp-hcp.azure-test.net did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-4gnm9.apps.aro.v20261001.bhnx.westus3.aroapp-hcp.azure-test.net on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded
{
msg: "DNS for route host agnhost-e2e-sample-app-4gnm9.apps.aro.v20261001.bhnx.westus3.aroapp-hcp.azure-test.net did not resolve: DNS for agnhost-e2e-sample-app-4gnm9.apps.aro.v20261001.bhnx.westus3.aroapp-hcp.azure-test.net did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-4gnm9.apps.aro.v20261001.bhnx.westus3.aroapp-hcp.azure-test.net on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
err: <*fmt.wrapError | 0xdad4d9b41a0>{
msg: "DNS for agnhost-e2e-sample-app-4gnm9.apps.aro.v20261001.bhnx.westus3.aroapp-hcp.azure-test.net did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-4gnm9.apps.aro.v20261001.bhnx.westus3.aroapp-hcp.azure-test.net on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
err: <context.deadlineExceededError>{},
},
}
occurredfail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:194]: failed to verify simple web app runs on cluster "cluster-candidate-5-1-5f9zxt"
Unexpected error:
<*fmt.wrapError | 0x3db0320ca900>:
DNS for route host agnhost-e2e-sample-app-8tqcz.apps.aro.x0a9b9z6x4r0r3x.a2p1.westus3.aroapp-hcp.azure-test.net did not resolve: DNS for agnhost-e2e-sample-app-8tqcz.apps.aro.x0a9b9z6x4r0r3x.a2p1.westus3.aroapp-hcp.azure-test.net did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-8tqcz.apps.aro.x0a9b9z6x4r0r3x.a2p1.westus3.aroapp-hcp.azure-test.net on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded
{
msg: "DNS for route host agnhost-e2e-sample-app-8tqcz.apps.aro.x0a9b9z6x4r0r3x.a2p1.westus3.aroapp-hcp.azure-test.net did not resolve: DNS for agnhost-e2e-sample-app-8tqcz.apps.aro.x0a9b9z6x4r0r3x.a2p1.westus3.aroapp-hcp.azure-test.net did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-8tqcz.apps.aro.x0a9b9z6x4r0r3x.a2p1.westus3.aroapp-hcp.azure-test.net on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
err: <*fmt.wrapError | 0x3db031a2bd20>{
msg: "DNS for agnhost-e2e-sample-app-8tqcz.apps.aro.x0a9b9z6x4r0r3x.a2p1.westus3.aroapp-hcp.azure-test.net did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-8tqcz.apps.aro.x0a9b9z6x4r0r3x.a2p1.westus3.aroapp-hcp.azure-test.net on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
err: <context.deadlineExceededError>{},
},
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:194]: failed to verify simple web app runs on cluster "cluster-candidate-5-1-5f9zxt"
Unexpected error:
<*fmt.wrapError | 0x3db0320ca900>:
DNS for route host agnhost-e2e-sample-app-8tqcz.apps.aro.x0a9b9z6x4r0r3x.a2p1.westus3.aroapp-hcp.azure-test.net did not resolve: DNS for agnhost-e2e-sample-app-8tqcz.apps.aro.x0a9b9z6x4r0r3x.a2p1.westus3.aroapp-hcp.azure-test.net did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-8tqcz.apps.aro.x0a9b9z6x4r0r3x.a2p1.westus3.aroapp-hcp.azure-test.net on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded
{
msg: "DNS for route host agnhost-e2e-sample-app-8tqcz.apps.aro.x0a9b9z6x4r0r3x.a2p1.westus3.aroapp-hcp.azure-test.net did not resolve: DNS for agnhost-e2e-sample-app-8tqcz.apps.aro.x0a9b9z6x4r0r3x.a2p1.westus3.aroapp-hcp.azure-test.net did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-8tqcz.apps.aro.x0a9b9z6x4r0r3x.a2p1.westus3.aroapp-hcp.azure-test.net on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
err: <*fmt.wrapError | 0x3db031a2bd20>{
msg: "DNS for agnhost-e2e-sample-app-8tqcz.apps.aro.x0a9b9z6x4r0r3x.a2p1.westus3.aroapp-hcp.azure-test.net did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-8tqcz.apps.aro.x0a9b9z6x4r0r3x.a2p1.westus3.aroapp-hcp.azure-test.net on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
err: <context.deadlineExceededError>{},
},
}
occurredfail [github.com/Azure/ARO-HCP/test/e2e/cluster_nodepool_active_versions.go:149]: failed to verify simple web app runs on v20261001 cluster "v20261001"
Unexpected error:
<*fmt.wrapError | 0x1342e25441a0>:
DNS for route host agnhost-e2e-sample-app-6g5k9.apps.aro.v20261001.1f89.westus3.aroapp-hcp.azure-test.net did not resolve: DNS for agnhost-e2e-sample-app-6g5k9.apps.aro.v20261001.1f89.westus3.aroapp-hcp.azure-test.net did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-6g5k9.apps.aro.v20261001.1f89.westus3.aroapp-hcp.azure-test.net on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded
{
msg: "DNS for route host agnhost-e2e-sample-app-6g5k9.apps.aro.v20261001.1f89.westus3.aroapp-hcp.azure-test.net did not resolve: DNS for agnhost-e2e-sample-app-6g5k9.apps.aro.v20261001.1f89.westus3.aroapp-hcp.azure-test.net did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-6g5k9.apps.aro.v20261001.1f89.westus3.aroapp-hcp.azure-test.net on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
err: <*fmt.wrapError | 0x1342e251c060>{
msg: "DNS for agnhost-e2e-sample-app-6g5k9.apps.aro.v20261001.1f89.westus3.aroapp-hcp.azure-test.net did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-6g5k9.apps.aro.v20261001.1f89.westus3.aroapp-hcp.azure-test.net on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
err: <context.deadlineExceededError>{},
},
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_nodepool_active_versions.go:149]: failed to verify simple web app runs on v20261001 cluster "v20261001"
Unexpected error:
<*fmt.wrapError | 0x1342e25441a0>:
DNS for route host agnhost-e2e-sample-app-6g5k9.apps.aro.v20261001.1f89.westus3.aroapp-hcp.azure-test.net did not resolve: DNS for agnhost-e2e-sample-app-6g5k9.apps.aro.v20261001.1f89.westus3.aroapp-hcp.azure-test.net did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-6g5k9.apps.aro.v20261001.1f89.westus3.aroapp-hcp.azure-test.net on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded
{
msg: "DNS for route host agnhost-e2e-sample-app-6g5k9.apps.aro.v20261001.1f89.westus3.aroapp-hcp.azure-test.net did not resolve: DNS for agnhost-e2e-sample-app-6g5k9.apps.aro.v20261001.1f89.westus3.aroapp-hcp.azure-test.net did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-6g5k9.apps.aro.v20261001.1f89.westus3.aroapp-hcp.azure-test.net on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
err: <*fmt.wrapError | 0x1342e251c060>{
msg: "DNS for agnhost-e2e-sample-app-6g5k9.apps.aro.v20261001.1f89.westus3.aroapp-hcp.azure-test.net did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-6g5k9.apps.aro.v20261001.1f89.westus3.aroapp-hcp.azure-test.net on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
err: <context.deadlineExceededError>{},
},
}
occurredContributing tests (2)
Affected runs (3)
| |||||||||||||||||||||||||||||||||||||||||||||||||||||
VerifySimpleWebApp failed: DNS for route host <route-host> did not resolve: DNS for <route-host> did not resolve within <duration> (last error: lookup <host> on <dns-server>: dial...full failure pattern: VerifySimpleWebApp failed: DNS for route host <route-host> did not resolve: DNS for <route-host> did not resolve within <duration> (last error: lookup <host> on <dns-server>: dial udp <ip>: i/o timeout): context deadline exceeded Signal: IndeterminateSignal: Indeterminate — no prior history; active 3 day(s) — New failure pattern — no prior history | unknown | 3 | 10.34%3 of 29 job runs affected | IndeterminateSignal: Indeterminate — no prior history; active 3 day(s) — New failure pattern — no prior history | Show trend detailsOct 12: 0 · Oct 13: 0 · Oct 14: 0 · Oct 15: 0 · Oct 16: 0 · Oct 17: 0 · Oct 18: 0 · Oct 19: 0 · Oct 20: 0 · Oct 21: 0 · Oct 22: 0 · Oct 23: 1 · Oct 24: 1 · Oct 25: 1 | DEV, PROD | |||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (3)fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_cni_cilium.go:186]: failed to run simple web app and connectivity check app with cilium CNI
Unexpected error:
<*errors.joinError | 0x248e56e12588>:
VerifySimpleWebApp failed: DNS for route host agnhost-e2e-sample-app-84lh2.apps.aro.cilium-cl.t19t.westus3.aroapp-hcp.azure-test.net did not resolve: DNS for agnhost-e2e-sample-app-84lh2.apps.aro.cilium-cl.t19t.westus3.aroapp-hcp.azure-test.net did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-84lh2.apps.aro.cilium-cl.t19t.westus3.aroapp-hcp.azure-test.net on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded
{
errs: [
<*fmt.wrapError | 0x248e56f14360>{
msg: "VerifySimpleWebApp failed: DNS for route host agnhost-e2e-sample-app-84lh2.apps.aro.cilium-cl.t19t.westus3.aroapp-hcp.azure-test.net did not resolve: DNS for agnhost-e2e-sample-app-84lh2.apps.aro.cilium-cl.t19t.westus3.aroapp-hcp.azure-test.net did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-84lh2.apps.aro.cilium-cl.t19t.westus3.aroapp-hcp.azure-test.net on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
err: <*fmt.wrapError | 0x248e5757aa00>{
msg: "DNS for route host agnhost-e2e-sample-app-84lh2.apps.aro.cilium-cl.t19t.westus3.aroapp-hcp.azure-test.net did not resolve: DNS for agnhost-e2e-sample-app-84lh2.apps.aro.cilium-cl.t19t.westus3.aroapp-hcp.azure-test.net did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-84lh2.apps.aro.cilium-cl.t19t.westus3.aroapp-hcp.azure-test.net on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
err: <*fmt.wrapError | 0x248e57018e60>{
msg: "DNS for agnhost-e2e-sample-app-84lh2.apps.aro.cilium-cl.t19t.westus3.aroapp-hcp.azure-test.net did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-84lh2.apps.aro.cilium-cl.t19t.westus3.aroapp-hcp.azure-test.net on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
err: <context.deadlineExceededError>{},
},
},
},
],
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_cni_cilium.go:186]: failed to run simple web app and connectivity check app with cilium CNI
Unexpected error:
<*errors.joinError | 0x248e56e12588>:
VerifySimpleWebApp failed: DNS for route host agnhost-e2e-sample-app-84lh2.apps.aro.cilium-cl.t19t.westus3.aroapp-hcp.azure-test.net did not resolve: DNS for agnhost-e2e-sample-app-84lh2.apps.aro.cilium-cl.t19t.westus3.aroapp-hcp.azure-test.net did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-84lh2.apps.aro.cilium-cl.t19t.westus3.aroapp-hcp.azure-test.net on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded
{
errs: [
<*fmt.wrapError | 0x248e56f14360>{
msg: "VerifySimpleWebApp failed: DNS for route host agnhost-e2e-sample-app-84lh2.apps.aro.cilium-cl.t19t.westus3.aroapp-hcp.azure-test.net did not resolve: DNS for agnhost-e2e-sample-app-84lh2.apps.aro.cilium-cl.t19t.westus3.aroapp-hcp.azure-test.net did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-84lh2.apps.aro.cilium-cl.t19t.westus3.aroapp-hcp.azure-test.net on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
err: <*fmt.wrapError | 0x248e5757aa00>{
msg: "DNS for route host agnhost-e2e-sample-app-84lh2.apps.aro.cilium-cl.t19t.westus3.aroapp-hcp.azure-test.net did not resolve: DNS for agnhost-e2e-sample-app-84lh2.apps.aro.cilium-cl.t19t.westus3.aroapp-hcp.azure-test.net did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-84lh2.apps.aro.cilium-cl.t19t.westus3.aroapp-hcp.azure-test.net on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
err: <*fmt.wrapError | 0x248e57018e60>{
msg: "DNS for agnhost-e2e-sample-app-84lh2.apps.aro.cilium-cl.t19t.westus3.aroapp-hcp.azure-test.net did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-84lh2.apps.aro.cilium-cl.t19t.westus3.aroapp-hcp.azure-test.net on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
err: <context.deadlineExceededError>{},
},
},
},
],
}
occurredfail [github.com/Azure/ARO-HCP/test/e2e/cluster_autoscaling.go:203]: failed to verify cluster workloads and pods after autoscaling update
Unexpected error:
<*errors.joinError | 0x22d4bdfbc060>:
VerifySimpleWebApp failed: DNS for route host agnhost-e2e-sample-app-mcq7p.apps.aro.m0y7f5l6s2i7t2m.fbfu.westus3.aroapp-hcp.azure-test.net did not resolve: DNS for agnhost-e2e-sample-app-mcq7p.apps.aro.m0y7f5l6s2i7t2m.fbfu.westus3.aroapp-hcp.azure-test.net did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-mcq7p.apps.aro.m0y7f5l6s2i7t2m.fbfu.westus3.aroapp-hcp.azure-test.net on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded
{
errs: [
<*fmt.wrapError | 0x22d4bdf9c1a0>{
msg: "VerifySimpleWebApp failed: DNS for route host agnhost-e2e-sample-app-mcq7p.apps.aro.m0y7f5l6s2i7t2m.fbfu.westus3.aroapp-hcp.azure-test.net did not resolve: DNS for agnhost-e2e-sample-app-mcq7p.apps.aro.m0y7f5l6s2i7t2m.fbfu.westus3.aroapp-hcp.azure-test.net did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-mcq7p.apps.aro.m0y7f5l6s2i7t2m.fbfu.westus3.aroapp-hcp.azure-test.net on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
err: <*fmt.wrapError | 0x22d4bdee0f00>{
msg: "DNS for route host agnhost-e2e-sample-app-mcq7p.apps.aro.m0y7f5l6s2i7t2m.fbfu.westus3.aroapp-hcp.azure-test.net did not resolve: DNS for agnhost-e2e-sample-app-mcq7p.apps.aro.m0y7f5l6s2i7t2m.fbfu.westus3.aroapp-hcp.azure-test.net did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-mcq7p.apps.aro.m0y7f5l6s2i7t2m.fbfu.westus3.aroapp-hcp.azure-test.net on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
err: <*fmt.wrapError | 0x22d4bdf9da20>{
msg: "DNS for agnhost-e2e-sample-app-mcq7p.apps.aro.m0y7f5l6s2i7t2m.fbfu.westus3.aroapp-hcp.azure-test.net did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-mcq7p.apps.aro.m0y7f5l6s2i7t2m.fbfu.westus3.aroapp-hcp.azure-test.net on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
err: <context.deadlineExceededError>{},
},
},
},
],
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_autoscaling.go:203]: failed to verify cluster workloads and pods after autoscaling update
Unexpected error:
<*errors.joinError | 0x22d4bdfbc060>:
VerifySimpleWebApp failed: DNS for route host agnhost-e2e-sample-app-mcq7p.apps.aro.m0y7f5l6s2i7t2m.fbfu.westus3.aroapp-hcp.azure-test.net did not resolve: DNS for agnhost-e2e-sample-app-mcq7p.apps.aro.m0y7f5l6s2i7t2m.fbfu.westus3.aroapp-hcp.azure-test.net did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-mcq7p.apps.aro.m0y7f5l6s2i7t2m.fbfu.westus3.aroapp-hcp.azure-test.net on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded
{
errs: [
<*fmt.wrapError | 0x22d4bdf9c1a0>{
msg: "VerifySimpleWebApp failed: DNS for route host agnhost-e2e-sample-app-mcq7p.apps.aro.m0y7f5l6s2i7t2m.fbfu.westus3.aroapp-hcp.azure-test.net did not resolve: DNS for agnhost-e2e-sample-app-mcq7p.apps.aro.m0y7f5l6s2i7t2m.fbfu.westus3.aroapp-hcp.azure-test.net did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-mcq7p.apps.aro.m0y7f5l6s2i7t2m.fbfu.westus3.aroapp-hcp.azure-test.net on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
err: <*fmt.wrapError | 0x22d4bdee0f00>{
msg: "DNS for route host agnhost-e2e-sample-app-mcq7p.apps.aro.m0y7f5l6s2i7t2m.fbfu.westus3.aroapp-hcp.azure-test.net did not resolve: DNS for agnhost-e2e-sample-app-mcq7p.apps.aro.m0y7f5l6s2i7t2m.fbfu.westus3.aroapp-hcp.azure-test.net did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-mcq7p.apps.aro.m0y7f5l6s2i7t2m.fbfu.westus3.aroapp-hcp.azure-test.net on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
err: <*fmt.wrapError | 0x22d4bdf9da20>{
msg: "DNS for agnhost-e2e-sample-app-mcq7p.apps.aro.m0y7f5l6s2i7t2m.fbfu.westus3.aroapp-hcp.azure-test.net did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-mcq7p.apps.aro.m0y7f5l6s2i7t2m.fbfu.westus3.aroapp-hcp.azure-test.net on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
err: <context.deadlineExceededError>{},
},
},
},
],
}
occurredfail [github.com/Azure/ARO-HCP/test/e2e/cluster_autoscaling.go:203]: failed to verify cluster workloads and pods after autoscaling update
Unexpected error:
<*errors.joinError | 0x1b9fbf4865b8>:
VerifySimpleWebApp failed: DNS for route host agnhost-e2e-sample-app-6gx6w.apps.aro.v0o5x2p3z8t0f8t.72eg.westus3.aroapp-hcp.azure-test.net did not resolve: DNS for agnhost-e2e-sample-app-6gx6w.apps.aro.v0o5x2p3z8t0f8t.72eg.westus3.aroapp-hcp.azure-test.net did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-6gx6w.apps.aro.v0o5x2p3z8t0f8t.72eg.westus3.aroapp-hcp.azure-test.net on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded
{
errs: [
<*fmt.wrapError | 0x1b9fbd955860>{
msg: "VerifySimpleWebApp failed: DNS for route host agnhost-e2e-sample-app-6gx6w.apps.aro.v0o5x2p3z8t0f8t.72eg.westus3.aroapp-hcp.azure-test.net did not resolve: DNS for agnhost-e2e-sample-app-6gx6w.apps.aro.v0o5x2p3z8t0f8t.72eg.westus3.aroapp-hcp.azure-test.net did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-6gx6w.apps.aro.v0o5x2p3z8t0f8t.72eg.westus3.aroapp-hcp.azure-test.net on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
err: <*fmt.wrapError | 0x1b9fbd298140>{
msg: "DNS for route host agnhost-e2e-sample-app-6gx6w.apps.aro.v0o5x2p3z8t0f8t.72eg.westus3.aroapp-hcp.azure-test.net did not resolve: DNS for agnhost-e2e-sample-app-6gx6w.apps.aro.v0o5x2p3z8t0f8t.72eg.westus3.aroapp-hcp.azure-test.net did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-6gx6w.apps.aro.v0o5x2p3z8t0f8t.72eg.westus3.aroapp-hcp.azure-test.net on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
err: <*fmt.wrapError | 0x1b9fbe030640>{
msg: "DNS for agnhost-e2e-sample-app-6gx6w.apps.aro.v0o5x2p3z8t0f8t.72eg.westus3.aroapp-hcp.azure-test.net did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-6gx6w.apps.aro.v0o5x2p3z8t0f8t.72eg.westus3.aroapp-hcp.azure-test.net on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
err: <context.deadlineExceededError>{},
},
},
},
],
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_autoscaling.go:203]: failed to verify cluster workloads and pods after autoscaling update
Unexpected error:
<*errors.joinError | 0x1b9fbf4865b8>:
VerifySimpleWebApp failed: DNS for route host agnhost-e2e-sample-app-6gx6w.apps.aro.v0o5x2p3z8t0f8t.72eg.westus3.aroapp-hcp.azure-test.net did not resolve: DNS for agnhost-e2e-sample-app-6gx6w.apps.aro.v0o5x2p3z8t0f8t.72eg.westus3.aroapp-hcp.azure-test.net did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-6gx6w.apps.aro.v0o5x2p3z8t0f8t.72eg.westus3.aroapp-hcp.azure-test.net on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded
{
errs: [
<*fmt.wrapError | 0x1b9fbd955860>{
msg: "VerifySimpleWebApp failed: DNS for route host agnhost-e2e-sample-app-6gx6w.apps.aro.v0o5x2p3z8t0f8t.72eg.westus3.aroapp-hcp.azure-test.net did not resolve: DNS for agnhost-e2e-sample-app-6gx6w.apps.aro.v0o5x2p3z8t0f8t.72eg.westus3.aroapp-hcp.azure-test.net did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-6gx6w.apps.aro.v0o5x2p3z8t0f8t.72eg.westus3.aroapp-hcp.azure-test.net on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
err: <*fmt.wrapError | 0x1b9fbd298140>{
msg: "DNS for route host agnhost-e2e-sample-app-6gx6w.apps.aro.v0o5x2p3z8t0f8t.72eg.westus3.aroapp-hcp.azure-test.net did not resolve: DNS for agnhost-e2e-sample-app-6gx6w.apps.aro.v0o5x2p3z8t0f8t.72eg.westus3.aroapp-hcp.azure-test.net did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-6gx6w.apps.aro.v0o5x2p3z8t0f8t.72eg.westus3.aroapp-hcp.azure-test.net on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
err: <*fmt.wrapError | 0x1b9fbe030640>{
msg: "DNS for agnhost-e2e-sample-app-6gx6w.apps.aro.v0o5x2p3z8t0f8t.72eg.westus3.aroapp-hcp.azure-test.net did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-6gx6w.apps.aro.v0o5x2p3z8t0f8t.72eg.westus3.aroapp-hcp.azure-test.net on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
err: <context.deadlineExceededError>{},
},
},
},
],
}
occurredContributing tests (2)
Affected runs (3)
| |||||||||||||||||||||||||||||||||||||||||||||||||||||
ERROR CODE: DeadlineExceeded; detail code SubscriptionRequestsThrottled; detail message Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor '<u...full failure pattern: ERROR CODE: DeadlineExceeded; detail code SubscriptionRequestsThrottled; detail message Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor '<uuid>' exceeded. Please try again after '<seconds>' seconds after additional tokens are available.; provider Microsoft.RedHatOpenShift Signal: IndeterminateSignal: Indeterminate — no prior history; active 2 day(s) — New failure pattern — no prior history | unknown | 2 | 6.90%2 of 29 job runs affected | IndeterminateSignal: Indeterminate — no prior history; active 2 day(s) — New failure pattern — no prior history | Show trend detailsOct 12: 0 · Oct 13: 0 · Oct 14: 0 · Oct 15: 0 · Oct 16: 0 · Oct 17: 0 · Oct 18: 0 · Oct 19: 0 · Oct 20: 0 · Oct 21: 2 · Oct 22: 0 · Oct 23: 0 · Oct 24: 0 · Oct 25: 1 | DEV | |||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (3)fail [github.com/Azure/ARO-HCP/test/e2e/gpu_nodepools_create_delete.go:118]: failed to create default nodepool np-1
Unexpected error:
<*fmt.wrapError | 0x3cdcfbca5680>:
failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "cluster-gpu-np" in resourcegroup="rg-gpu-nodepool-8vtwfvfssx9k" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/907ac069-8701-4021-b410-cf10181345c0
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/907ac069-8701-4021-b410-cf10181345c0",
"name": "907ac069-8701-4021-b410-cf10181345c0",
"status": "Failed",
"startTime": "2026-10-25T05:14:31.336281353Z",
"endTime": "2026-10-25T05:33:33.578871475Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine cluster-gpu-np-np-1-8v7xk-22mlx: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-gpu-nodepool-8vtwfvfssx9k--managed/providers/Microsoft.Compute/virtualMachines/cluster-gpu-np-np-1-8v7xk-22mlx to exist\nMachine cluster-gpu-np-np-1-8v7xk-p7hpl: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-gpu-nodepool-8vtwfvfssx9k--managed/providers/Microsoft.Compute/virtualMachines/cluster-gpu-np-np-1-8v7xk-p7hpl to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine cluster-gpu-np-np-1-8v7xk-p7hpl: Failed: interfaces failed to create or update. err: failed to get existing resource rg-gpu-nodepool-8vtwfvfssx9k--managed/cluster-gpu-np-np-1-8v7xk-p7hpl-nic (service: interfaces): GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-gpu-nodepool-8vtwfvfssx9k--managed/providers/Microsoft.Network/networkInterfaces/cluster-gpu-np-np-1-8v7xk-p7hpl-nic\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"SubscriptionRequestsThrottled\",\n \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor '6096f642-63a0-4dd6-958d-715a87f33535' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to ... message truncated\nMachine cluster-gpu-np-np-1-8v7xk-22mlx: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-gpu-nodepool-8vtwfvfssx9k--managed/providers/Microsoft.Compute/virtualMachines/cluster-gpu-np-np-1-8v7xk-22mlx to exist\n"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/907ac069-8701-4021-b410-cf10181345c0
correlation ID: 1c529a86-2bdd-43d7-ab0e-4267cd216305
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/gpu_nodepools_create_delete.go:118]: failed to create default nodepool np-1
Unexpected error:
<*fmt.wrapError | 0x3cdcfbca5680>:
failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "cluster-gpu-np" in resourcegroup="rg-gpu-nodepool-8vtwfvfssx9k" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/907ac069-8701-4021-b410-cf10181345c0
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/907ac069-8701-4021-b410-cf10181345c0",
"name": "907ac069-8701-4021-b410-cf10181345c0",
"status": "Failed",
"startTime": "2026-10-25T05:14:31.336281353Z",
"endTime": "2026-10-25T05:33:33.578871475Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine cluster-gpu-np-np-1-8v7xk-22mlx: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-gpu-nodepool-8vtwfvfssx9k--managed/providers/Microsoft.Compute/virtualMachines/cluster-gpu-np-np-1-8v7xk-22mlx to exist\nMachine cluster-gpu-np-np-1-8v7xk-p7hpl: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-gpu-nodepool-8vtwfvfssx9k--managed/providers/Microsoft.Compute/virtualMachines/cluster-gpu-np-np-1-8v7xk-p7hpl to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine cluster-gpu-np-np-1-8v7xk-p7hpl: Failed: interfaces failed to create or update. err: failed to get existing resource rg-gpu-nodepool-8vtwfvfssx9k--managed/cluster-gpu-np-np-1-8v7xk-p7hpl-nic (service: interfaces): GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-gpu-nodepool-8vtwfvfssx9k--managed/providers/Microsoft.Network/networkInterfaces/cluster-gpu-np-np-1-8v7xk-p7hpl-nic\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"SubscriptionRequestsThrottled\",\n \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor '6096f642-63a0-4dd6-958d-715a87f33535' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to ... message truncated\nMachine cluster-gpu-np-np-1-8v7xk-22mlx: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-gpu-nodepool-8vtwfvfssx9k--managed/providers/Microsoft.Compute/virtualMachines/cluster-gpu-np-np-1-8v7xk-22mlx to exist\n"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/907ac069-8701-4021-b410-cf10181345c0
correlation ID: 1c529a86-2bdd-43d7-ab0e-4267cd216305
client request ID: unavailable
occurredfail [github.com/Azure/ARO-HCP/test/e2e/cluster_fips_mode.go:103]: failed to create node pool "np-1" for fips-enabled cluster "fips-enabled-cluster"
Unexpected error:
<*fmt.wrapError | 0x3c4c027a02e0>:
failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "fips-enabled-cluster" in resourcegroup="fips-enabled-6xshnph7ptd9" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/363f69bc-d1f2-4956-842d-980aba640091
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/363f69bc-d1f2-4956-842d-980aba640091",
"name": "363f69bc-d1f2-4956-842d-980aba640091",
"status": "Failed",
"startTime": "2026-10-21T21:02:55.100233745Z",
"endTime": "2026-10-21T21:21:58.935771192Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine j1s0i5w0t2h9v3h-np-1-br8d2-5c77c: Failed: virtualmachine failed to create or update. err: failed to get existing resource fips-enabled-6xshnph7ptd9--managed/j1s0i5w0t2h9v3h-np-1-br8d2-5c77c (service: virtualmachine): GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/fips-enabled-6xshnph7ptd9--managed/providers/Microsoft.Compute/virtualMachines/j1s0i5w0t2h9v3h-np-1-br8d2-5c77c\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"SubscriptionRequestsThrottled\",\n \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor '6096f642-63a0-4dd6-958d-715a87f33535' exceeded. Please try again after '1' seconds after additional tokens are available. Refer to https:... message truncated\n"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/363f69bc-d1f2-4956-842d-980aba640091
correlation ID: 90349399-701b-4ce9-affd-54978e664c51
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_fips_mode.go:103]: failed to create node pool "np-1" for fips-enabled cluster "fips-enabled-cluster"
Unexpected error:
<*fmt.wrapError | 0x3c4c027a02e0>:
failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "fips-enabled-cluster" in resourcegroup="fips-enabled-6xshnph7ptd9" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/363f69bc-d1f2-4956-842d-980aba640091
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/363f69bc-d1f2-4956-842d-980aba640091",
"name": "363f69bc-d1f2-4956-842d-980aba640091",
"status": "Failed",
"startTime": "2026-10-21T21:02:55.100233745Z",
"endTime": "2026-10-21T21:21:58.935771192Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine j1s0i5w0t2h9v3h-np-1-br8d2-5c77c: Failed: virtualmachine failed to create or update. err: failed to get existing resource fips-enabled-6xshnph7ptd9--managed/j1s0i5w0t2h9v3h-np-1-br8d2-5c77c (service: virtualmachine): GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/fips-enabled-6xshnph7ptd9--managed/providers/Microsoft.Compute/virtualMachines/j1s0i5w0t2h9v3h-np-1-br8d2-5c77c\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"SubscriptionRequestsThrottled\",\n \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor '6096f642-63a0-4dd6-958d-715a87f33535' exceeded. Please try again after '1' seconds after additional tokens are available. Refer to https:... message truncated\n"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/363f69bc-d1f2-4956-842d-980aba640091
correlation ID: 90349399-701b-4ce9-affd-54978e664c51
client request ID: unavailable
occurredfail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_v20260901preview.go:102]: failed to create node pool "np-1" for v20260901preview cluster "v20260901"
Unexpected error:
<*fmt.wrapError | 0x16dd5f842340>:
failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "v20260901" in resourcegroup="v20260901preview-h4vdzmhsx8jj" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/214af887-1dd9-4618-9147-3df5b406eba9
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/214af887-1dd9-4618-9147-3df5b406eba9",
"name": "214af887-1dd9-4618-9147-3df5b406eba9",
"status": "Failed",
"startTime": "2026-10-21T21:02:58.414383168Z",
"endTime": "2026-10-21T21:21:59.161334523Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine v20260901-np-1-4x84k-cvs95: Failed: interfaces failed to create or update. err: failed to get existing resource v20260901preview-h4vdzmhsx8jj--managed/v20260901-np-1-4x84k-cvs95-nic (service: interfaces): GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/v20260901preview-h4vdzmhsx8jj--managed/providers/Microsoft.Network/networkInterfaces/v20260901-np-1-4x84k-cvs95-nic\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"SubscriptionRequestsThrottled\",\n \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor '6096f642-63a0-4dd6-958d-715a87f33535' exceeded. Please try again after '2' seconds after additional tokens are available. Refer to https://aka.ms... message truncated\n"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/214af887-1dd9-4618-9147-3df5b406eba9
correlation ID: 6a1aa67d-c740-4581-b0a0-1116fc9156bb
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_v20260901preview.go:102]: failed to create node pool "np-1" for v20260901preview cluster "v20260901"
Unexpected error:
<*fmt.wrapError | 0x16dd5f842340>:
failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "v20260901" in resourcegroup="v20260901preview-h4vdzmhsx8jj" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/214af887-1dd9-4618-9147-3df5b406eba9
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/214af887-1dd9-4618-9147-3df5b406eba9",
"name": "214af887-1dd9-4618-9147-3df5b406eba9",
"status": "Failed",
"startTime": "2026-10-21T21:02:58.414383168Z",
"endTime": "2026-10-21T21:21:59.161334523Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine v20260901-np-1-4x84k-cvs95: Failed: interfaces failed to create or update. err: failed to get existing resource v20260901preview-h4vdzmhsx8jj--managed/v20260901-np-1-4x84k-cvs95-nic (service: interfaces): GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/v20260901preview-h4vdzmhsx8jj--managed/providers/Microsoft.Network/networkInterfaces/v20260901-np-1-4x84k-cvs95-nic\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"SubscriptionRequestsThrottled\",\n \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor '6096f642-63a0-4dd6-958d-715a87f33535' exceeded. Please try again after '2' seconds after additional tokens are available. Refer to https://aka.ms... message truncated\n"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/214af887-1dd9-4618-9147-3df5b406eba9
correlation ID: 6a1aa67d-c740-4581-b0a0-1116fc9156bb
client request ID: unavailable
occurredContributing tests (3)
Affected runs (2)
| |||||||||||||||||||||||||||||||||||||||||||||||||||||
DNS for route host <route-host> did not resolve: DNS for <route-host> did not resolve within <duration> (last error: lookup <host> on <dns-server>: no such host): context deadline...full failure pattern: DNS for route host <route-host> did not resolve: DNS for <route-host> did not resolve within <duration> (last error: lookup <host> on <dns-server>: no such host): context deadline exceeded Signal: IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | unknown | 1 | 3.45%1 of 29 job runs affected | IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | Show trend detailsOct 12: 0 · Oct 13: 0 · Oct 14: 0 · Oct 15: 0 · Oct 16: 0 · Oct 17: 0 · Oct 18: 0 · Oct 19: 0 · Oct 20: 0 · Oct 21: 0 · Oct 22: 0 · Oct 23: 0 · Oct 24: 1 · Oct 25: 0 | DEV, PROD | |||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_v20260901preview.go:121]: failed to verify simple web app runs on v20260901preview cluster "v20260901"
Unexpected error:
<*fmt.wrapError | 0x3f57e0004820>:
DNS for route host agnhost-e2e-sample-app-g99mf.apps.aro.v20260901.9ggg.westus3.aroapp-hcp.azure-test.net did not resolve: DNS for agnhost-e2e-sample-app-g99mf.apps.aro.v20260901.9ggg.westus3.aroapp-hcp.azure-test.net did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-g99mf.apps.aro.v20260901.9ggg.westus3.aroapp-hcp.azure-test.net on 172.30.0.10:53: no such host): context deadline exceeded
{
msg: "DNS for route host agnhost-e2e-sample-app-g99mf.apps.aro.v20260901.9ggg.westus3.aroapp-hcp.azure-test.net did not resolve: DNS for agnhost-e2e-sample-app-g99mf.apps.aro.v20260901.9ggg.westus3.aroapp-hcp.azure-test.net did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-g99mf.apps.aro.v20260901.9ggg.westus3.aroapp-hcp.azure-test.net on 172.30.0.10:53: no such host): context deadline exceeded",
err: <*fmt.wrapError | 0x3f57dfc6e000>{
msg: "DNS for agnhost-e2e-sample-app-g99mf.apps.aro.v20260901.9ggg.westus3.aroapp-hcp.azure-test.net did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-g99mf.apps.aro.v20260901.9ggg.westus3.aroapp-hcp.azure-test.net on 172.30.0.10:53: no such host): context deadline exceeded",
err: <context.deadlineExceededError>{},
},
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_v20260901preview.go:121]: failed to verify simple web app runs on v20260901preview cluster "v20260901"
Unexpected error:
<*fmt.wrapError | 0x3f57e0004820>:
DNS for route host agnhost-e2e-sample-app-g99mf.apps.aro.v20260901.9ggg.westus3.aroapp-hcp.azure-test.net did not resolve: DNS for agnhost-e2e-sample-app-g99mf.apps.aro.v20260901.9ggg.westus3.aroapp-hcp.azure-test.net did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-g99mf.apps.aro.v20260901.9ggg.westus3.aroapp-hcp.azure-test.net on 172.30.0.10:53: no such host): context deadline exceeded
{
msg: "DNS for route host agnhost-e2e-sample-app-g99mf.apps.aro.v20260901.9ggg.westus3.aroapp-hcp.azure-test.net did not resolve: DNS for agnhost-e2e-sample-app-g99mf.apps.aro.v20260901.9ggg.westus3.aroapp-hcp.azure-test.net did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-g99mf.apps.aro.v20260901.9ggg.westus3.aroapp-hcp.azure-test.net on 172.30.0.10:53: no such host): context deadline exceeded",
err: <*fmt.wrapError | 0x3f57dfc6e000>{
msg: "DNS for agnhost-e2e-sample-app-g99mf.apps.aro.v20260901.9ggg.westus3.aroapp-hcp.azure-test.net did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-g99mf.apps.aro.v20260901.9ggg.westus3.aroapp-hcp.azure-test.net on 172.30.0.10:53: no such host): context deadline exceeded",
err: <context.deadlineExceededError>{},
},
}
occurredContributing tests (1)
Affected runs (1)
| |||||||||||||||||||||||||||||||||||||||||||||||||||||
DNS for route host agnhost-e2e-sample-app-<id>full failure pattern: DNS for route host agnhost-e2e-sample-app-<id> Signal: IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | unknown | 1 | 3.45%1 of 29 job runs affected | IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | Show trend detailsOct 12: 0 · Oct 13: 0 · Oct 14: 0 · Oct 15: 0 · Oct 16: 0 · Oct 17: 0 · Oct 18: 0 · Oct 19: 0 · Oct 20: 0 · Oct 21: 0 · Oct 22: 0 · Oct 23: 0 · Oct 24: 1 · Oct 25: 0 | DEV, PROD | |||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:528]: sample application must serve the exact customer-specified certificate with a valid Let's Encrypt chain
Expected success, but got an error:
<*fmt.wrapError | 0x7f30d39c620>:
DNS for route host agnhost-e2e-sample-app-gt2zw.apps.aro.ingress-cert-le.msqo.westus3.aroapp-hcp.azure-test.net did not resolve: DNS for agnhost-e2e-sample-app-gt2zw.apps.aro.ingress-cert-le.msqo.westus3.aroapp-hcp.azure-test.net did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-gt2zw.apps.aro.ingress-cert-le.msqo.westus3.aroapp-hcp.azure-test.net on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded
{
msg: "DNS for route host agnhost-e2e-sample-app-gt2zw.apps.aro.ingress-cert-le.msqo.westus3.aroapp-hcp.azure-test.net did not resolve: DNS for agnhost-e2e-sample-app-gt2zw.apps.aro.ingress-cert-le.msqo.westus3.aroapp-hcp.azure-test.net did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-gt2zw.apps.aro.ingress-cert-le.msqo.westus3.aroapp-hcp.azure-test.net on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
err: <*fmt.wrapError | 0x7f30d39c600>{
msg: "DNS for agnhost-e2e-sample-app-gt2zw.apps.aro.ingress-cert-le.msqo.westus3.aroapp-hcp.azure-test.net did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-gt2zw.apps.aro.ingress-cert-le.msqo.westus3.aroapp-hcp.azure-test.net on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
err: <context.deadlineExceededError>{},
},
}
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:528]: sample application must serve the exact customer-specified certificate with a valid Let's Encrypt chain
Expected success, but got an error:
<*fmt.wrapError | 0x7f30d39c620>:
DNS for route host agnhost-e2e-sample-app-gt2zw.apps.aro.ingress-cert-le.msqo.westus3.aroapp-hcp.azure-test.net did not resolve: DNS for agnhost-e2e-sample-app-gt2zw.apps.aro.ingress-cert-le.msqo.westus3.aroapp-hcp.azure-test.net did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-gt2zw.apps.aro.ingress-cert-le.msqo.westus3.aroapp-hcp.azure-test.net on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded
{
msg: "DNS for route host agnhost-e2e-sample-app-gt2zw.apps.aro.ingress-cert-le.msqo.westus3.aroapp-hcp.azure-test.net did not resolve: DNS for agnhost-e2e-sample-app-gt2zw.apps.aro.ingress-cert-le.msqo.westus3.aroapp-hcp.azure-test.net did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-gt2zw.apps.aro.ingress-cert-le.msqo.westus3.aroapp-hcp.azure-test.net on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
err: <*fmt.wrapError | 0x7f30d39c600>{
msg: "DNS for agnhost-e2e-sample-app-gt2zw.apps.aro.ingress-cert-le.msqo.westus3.aroapp-hcp.azure-test.net did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-gt2zw.apps.aro.ingress-cert-le.msqo.westus3.aroapp-hcp.azure-test.net on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
err: <context.deadlineExceededError>{},
},
}Contributing tests (1)
Affected runs (1)
| |||||||||||||||||||||||||||||||||||||||||||||||||||||
ERROR CODE: DeadlineExceeded; detail code SubscriptionRequestsThrottled; detail message Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor '<u...full failure pattern: ERROR CODE: DeadlineExceeded; detail code SubscriptionRequestsThrottled; detail message Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor '<uuid>' exceeded. Ple... message truncated; provider Microsoft.RedHatOpenShift Signal: IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | unknown | 1 | 3.45%1 of 29 job runs affected | IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | Show trend detailsOct 12: 0 · Oct 13: 0 · Oct 14: 0 · Oct 15: 0 · Oct 16: 0 · Oct 17: 0 · Oct 18: 0 · Oct 19: 0 · Oct 20: 0 · Oct 21: 0 · Oct 22: 0 · Oct 23: 0 · Oct 24: 1 · Oct 25: 0 | none | |||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade_blocked_pod_creation.go:201]: failed to create node pool "np-deny-pods" for cluster "cp-ystream-deny-pods-5-1-fjj5cf"
Unexpected error:
<*fmt.wrapError | 0x3b484fe8ea00>:
failed to create NodePool np-deny-pods: failed waiting for nodepool="np-deny-pods" for cluster "cp-ystream-deny-pods-5-1-fjj5cf" in resourcegroup="rg-cp-ystream-deny-pods-5-1-j5b76vjznvvh" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/cf1ae438-bf01-4f86-8ab5-784122b7cb5a
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/cf1ae438-bf01-4f86-8ab5-784122b7cb5a",
"name": "cf1ae438-bf01-4f86-8ab5-784122b7cb5a",
"status": "Failed",
"startTime": "2026-10-24T03:36:53.315230765Z",
"endTime": "2026-10-24T03:56:02.496643822Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [hypershiftNodePool] node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine z4z5b9p8d5g2j9j-np-deny-pods-8gxvs-vs65j: Failed: interfaces failed to create or update. err: failed to get existing resource rg-cp-ystream-deny-pods-5-1-j5b76vjznvvh-cp-ystream-den-2ba76655/z4z5b9p8d5g2j9j-np-deny-pods-8gxvs-vs65j-nic (service: interfaces): GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cp-ystream-deny-pods-5-1-j5b76vjznvvh-cp-ystream-den-2ba76655/providers/Microsoft.Network/networkInterfaces/z4z5b9p8d5g2j9j-np-deny-pods-8gxvs-vs65j-nic\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"SubscriptionRequestsThrottled\",\n \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor '6096f642-63a0-4dd6-958d-715a87f33535' exceeded. Ple... message truncated\n"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/cf1ae438-bf01-4f86-8ab5-784122b7cb5a
correlation ID: 3b16cd17-d78f-4b89-a83b-76ef2e36cf89
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade_blocked_pod_creation.go:201]: failed to create node pool "np-deny-pods" for cluster "cp-ystream-deny-pods-5-1-fjj5cf"
Unexpected error:
<*fmt.wrapError | 0x3b484fe8ea00>:
failed to create NodePool np-deny-pods: failed waiting for nodepool="np-deny-pods" for cluster "cp-ystream-deny-pods-5-1-fjj5cf" in resourcegroup="rg-cp-ystream-deny-pods-5-1-j5b76vjznvvh" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/cf1ae438-bf01-4f86-8ab5-784122b7cb5a
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/cf1ae438-bf01-4f86-8ab5-784122b7cb5a",
"name": "cf1ae438-bf01-4f86-8ab5-784122b7cb5a",
"status": "Failed",
"startTime": "2026-10-24T03:36:53.315230765Z",
"endTime": "2026-10-24T03:56:02.496643822Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [hypershiftNodePool] node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine z4z5b9p8d5g2j9j-np-deny-pods-8gxvs-vs65j: Failed: interfaces failed to create or update. err: failed to get existing resource rg-cp-ystream-deny-pods-5-1-j5b76vjznvvh-cp-ystream-den-2ba76655/z4z5b9p8d5g2j9j-np-deny-pods-8gxvs-vs65j-nic (service: interfaces): GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cp-ystream-deny-pods-5-1-j5b76vjznvvh-cp-ystream-den-2ba76655/providers/Microsoft.Network/networkInterfaces/z4z5b9p8d5g2j9j-np-deny-pods-8gxvs-vs65j-nic\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"SubscriptionRequestsThrottled\",\n \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor '6096f642-63a0-4dd6-958d-715a87f33535' exceeded. Ple... message truncated\n"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/cf1ae438-bf01-4f86-8ab5-784122b7cb5a
correlation ID: 3b16cd17-d78f-4b89-a83b-76ef2e36cf89
client request ID: unavailable
occurredContributing tests (1)
Affected runs (1)
| |||||||||||||||||||||||||||||||||||||||||||||||||||||
ERROR CODE: DeadlineExceeded; detail code SubscriptionRequestsThrottled; detail message Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor '<u...full failure pattern: ERROR CODE: DeadlineExceeded; detail code SubscriptionRequestsThrottled; detail message Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor '<uuid>' exceeded. Please try again after '10' se... message truncated; provider Microsoft.RedHatOpenShift Signal: IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | unknown | 1 | 3.45%1 of 29 job runs affected | IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | Show trend detailsOct 12: 0 · Oct 13: 0 · Oct 14: 0 · Oct 15: 0 · Oct 16: 0 · Oct 17: 0 · Oct 18: 0 · Oct 19: 0 · Oct 20: 0 · Oct 21: 1 · Oct 22: 0 · Oct 23: 0 · Oct 24: 0 · Oct 25: 0 | DEV | |||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_ingress.go:123]: failed to create node pool "np-1" for private ingress cluster "private-ingress"
Unexpected error:
<*fmt.wrapError | 0x14b6bdca84e0>:
failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "private-ingress" in resourcegroup="private-ingress-gxdr6w65gk4m" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/f8d1a217-26ee-4eb3-92f9-7a398b20fbf3
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/f8d1a217-26ee-4eb3-92f9-7a398b20fbf3",
"name": "f8d1a217-26ee-4eb3-92f9-7a398b20fbf3",
"status": "Failed",
"startTime": "2026-10-21T21:02:30.132303735Z",
"endTime": "2026-10-21T21:21:38.466559997Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine private-ingress-np-1-z68fd-hf6g2: Failed: availabilitysets failed to create or update. err: failed to get existing resource private-ingress-gxdr6w65gk4m--managed/2tjdfde52u2f4iccs1dt8sgim461diee_private-ingress-np-1-as (service: availabilitysets): GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-ingress-gxdr6w65gk4m--managed/providers/Microsoft.Compute/availabilitySets/2tjdfde52u2f4iccs1dt8sgim461diee_private-ingress-np-1-as\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"SubscriptionRequestsThrottled\",\n \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor '6096f642-63a0-4dd6-958d-715a87f33535' exceeded. Please try again after '10' se... message truncated\n"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/f8d1a217-26ee-4eb3-92f9-7a398b20fbf3
correlation ID: 3e8f1379-1d07-482e-b286-68789ae9a346
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_ingress.go:123]: failed to create node pool "np-1" for private ingress cluster "private-ingress"
Unexpected error:
<*fmt.wrapError | 0x14b6bdca84e0>:
failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "private-ingress" in resourcegroup="private-ingress-gxdr6w65gk4m" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/f8d1a217-26ee-4eb3-92f9-7a398b20fbf3
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/f8d1a217-26ee-4eb3-92f9-7a398b20fbf3",
"name": "f8d1a217-26ee-4eb3-92f9-7a398b20fbf3",
"status": "Failed",
"startTime": "2026-10-21T21:02:30.132303735Z",
"endTime": "2026-10-21T21:21:38.466559997Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine private-ingress-np-1-z68fd-hf6g2: Failed: availabilitysets failed to create or update. err: failed to get existing resource private-ingress-gxdr6w65gk4m--managed/2tjdfde52u2f4iccs1dt8sgim461diee_private-ingress-np-1-as (service: availabilitysets): GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-ingress-gxdr6w65gk4m--managed/providers/Microsoft.Compute/availabilitySets/2tjdfde52u2f4iccs1dt8sgim461diee_private-ingress-np-1-as\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"SubscriptionRequestsThrottled\",\n \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor '6096f642-63a0-4dd6-958d-715a87f33535' exceeded. Please try again after '10' se... message truncated\n"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/f8d1a217-26ee-4eb3-92f9-7a398b20fbf3
correlation ID: 3e8f1379-1d07-482e-b286-68789ae9a346
client request ID: unavailable
occurredContributing tests (1)
Affected runs (1)
| |||||||||||||||||||||||||||||||||||||||||||||||||||||
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedC...full failure pattern: ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider]; provider Microsoft.RedHatOpenShift Signal: IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | unknown | 1 | 3.45%1 of 29 job runs affected | IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | Show trend detailsOct 12: 0 · Oct 13: 0 · Oct 14: 0 · Oct 15: 0 · Oct 16: 0 · Oct 17: 0 · Oct 18: 0 · Oct 19: 0 · Oct 20: 0 · Oct 21: 0 · Oct 22: 0 · Oct 23: 0 · Oct 24: 1 · Oct 25: 0 | DEV, PROD, STG | |||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:125]: failed to create HCP cluster "cluster-zstream-4-22-njchjb" with version 4.22.18 on candidate channel
Unexpected error:
<*fmt.wrapError | 0x2294f905a340>:
failed to create HCP cluster cluster-zstream-4-22-njchjb: failed waiting for cluster="cluster-zstream-4-22-njchjb" in resourcegroup="rg-zstream-upgrade-4-22-5bksdkff6tvq" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/b7f42e30-e443-47e9-982e-3fe8abc06d9a
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/b7f42e30-e443-47e9-982e-3fe8abc06d9a",
"name": "b7f42e30-e443-47e9-982e-3fe8abc06d9a",
"status": "Failed",
"startTime": "2026-10-24T22:19:20.094796575Z",
"endTime": "2026-10-24T22:48:29.265123983Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/b7f42e30-e443-47e9-982e-3fe8abc06d9a
correlation ID: c6c76fd1-9a00-45d4-a377-4b7ed8ffb192
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:125]: failed to create HCP cluster "cluster-zstream-4-22-njchjb" with version 4.22.18 on candidate channel
Unexpected error:
<*fmt.wrapError | 0x2294f905a340>:
failed to create HCP cluster cluster-zstream-4-22-njchjb: failed waiting for cluster="cluster-zstream-4-22-njchjb" in resourcegroup="rg-zstream-upgrade-4-22-5bksdkff6tvq" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/b7f42e30-e443-47e9-982e-3fe8abc06d9a
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/b7f42e30-e443-47e9-982e-3fe8abc06d9a",
"name": "b7f42e30-e443-47e9-982e-3fe8abc06d9a",
"status": "Failed",
"startTime": "2026-10-24T22:19:20.094796575Z",
"endTime": "2026-10-24T22:48:29.265123983Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/b7f42e30-e443-47e9-982e-3fe8abc06d9a
correlation ID: c6c76fd1-9a00-45d4-a377-4b7ed8ffb192
client request ID: unavailable
occurredContributing tests (1)
Affected runs (1)
| |||||||||||||||||||||||||||||||||||||||||||||||||||||
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet complete...full failure pattern: ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version <version> is Partial (want Completed), started <duration> ago; hosted cluster degraded: UnavailableReplicas: [router]; provider Microsoft.RedHatOpenShift Signal: IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | unknown | 1 | 3.45%1 of 29 job runs affected | IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | Show trend detailsOct 12: 0 · Oct 13: 0 · Oct 14: 0 · Oct 15: 0 · Oct 16: 0 · Oct 17: 0 · Oct 18: 0 · Oct 19: 0 · Oct 20: 0 · Oct 21: 0 · Oct 22: 0 · Oct 23: 0 · Oct 24: 1 · Oct 25: 0 | DEV | |||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:208]: failed to create HCP cluster "agg-cluster" with aggregated settings
Unexpected error:
<*fmt.wrapError | 0x9134b719420>:
failed waiting for cluster="agg-cluster" in resourcegroup="feature-aggregation-c2zb7x4tvbjz" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/2d4481c2-5c8d-4e2e-a526-9b051ca6139f
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/2d4481c2-5c8d-4e2e-a526-9b051ca6139f",
"name": "2d4481c2-5c8d-4e2e-a526-9b051ca6139f",
"status": "Failed",
"startTime": "2026-10-24T17:16:34.851278228Z",
"endTime": "2026-10-24T17:45:35.221810696Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.39 is Partial (want Completed), started 22m57s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/2d4481c2-5c8d-4e2e-a526-9b051ca6139f
correlation ID: 4e6d7a8e-ea3e-4773-abae-818d5133da86
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:208]: failed to create HCP cluster "agg-cluster" with aggregated settings
Unexpected error:
<*fmt.wrapError | 0x9134b719420>:
failed waiting for cluster="agg-cluster" in resourcegroup="feature-aggregation-c2zb7x4tvbjz" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/2d4481c2-5c8d-4e2e-a526-9b051ca6139f
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/2d4481c2-5c8d-4e2e-a526-9b051ca6139f",
"name": "2d4481c2-5c8d-4e2e-a526-9b051ca6139f",
"status": "Failed",
"startTime": "2026-10-24T17:16:34.851278228Z",
"endTime": "2026-10-24T17:45:35.221810696Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.39 is Partial (want Completed), started 22m57s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/2d4481c2-5c8d-4e2e-a526-9b051ca6139f
correlation ID: 4e6d7a8e-ea3e-4773-abae-818d5133da86
client request ID: unavailable
occurredContributing tests (1)
Affected runs (1)
| |||||||||||||||||||||||||||||||||||||||||||||||||||||
ERROR CODE: DeadlineExceeded; detail message node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hype...full failure pattern: ERROR CODE: DeadlineExceeded; detail message node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is <count>, want <count>; node pool condition AllNodesHealthy is False: <count> of <count> machines are not healthy Machine <node>: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.Compute/virtualMachines/<node> to exist Machine <node>: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.Compute/virtualMachines/<node> to exist; node pool condition AllMachinesReady is False: <count> of <count> machines are not ready Machine <node>: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.Compute/virtualMachines/<node> to exist Machine <node>: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.Compute/virtualMachines/<node> to exist; provider Microsoft.RedHatOpenShift Signal: IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | unknown | 1 | 3.45%1 of 29 job runs affected | IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | Show trend detailsOct 12: 0 · Oct 13: 0 · Oct 14: 0 · Oct 15: 0 · Oct 16: 0 · Oct 17: 0 · Oct 18: 0 · Oct 19: 0 · Oct 20: 0 · Oct 21: 0 · Oct 22: 1 · Oct 23: 0 · Oct 24: 0 · Oct 25: 0 | DEV, PROD | |||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/e2e/simple_negative_cases.go:159]: failed to create node pool np-1
Unexpected error:
<*fmt.wrapError | 0x1e6e8804d3c0>:
failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "negative-tests-cluster" in resourcegroup="rg-negative-tests-m5vrcvff2ctf" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/75b8b1f9-f718-45b2-b1ee-a3009f824cf8
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/75b8b1f9-f718-45b2-b1ee-a3009f824cf8",
"name": "75b8b1f9-f718-45b2-b1ee-a3009f824cf8",
"status": "Failed",
"startTime": "2026-10-22T14:53:42.149827787Z",
"endTime": "2026-10-22T15:12:50.02702561Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine c2g2u9m5x1z4p0m-np-1-bfdzc-5kdm2: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-negative-tests-m5vrcvff2ctf--managed/providers/Microsoft.Compute/virtualMachines/c2g2u9m5x1z4p0m-np-1-bfdzc-5kdm2 to exist\nMachine c2g2u9m5x1z4p0m-np-1-bfdzc-kbcws: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-negative-tests-m5vrcvff2ctf--managed/providers/Microsoft.Compute/virtualMachines/c2g2u9m5x1z4p0m-np-1-bfdzc-kbcws to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine c2g2u9m5x1z4p0m-np-1-bfdzc-5kdm2: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-negative-tests-m5vrcvff2ctf--managed/providers/Microsoft.Compute/virtualMachines/c2g2u9m5x1z4p0m-np-1-bfdzc-5kdm2 to exist\nMachine c2g2u9m5x1z4p0m-np-1-bfdzc-kbcws: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-negative-tests-m5vrcvff2ctf--managed/providers/Microsoft.Compute/virtualMachines/c2g2u9m5x1z4p0m-np-1-bfdzc-kbcws to exist\n"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/75b8b1f9-f718-45b2-b1ee-a3009f824cf8
correlation ID: e990e88e-110c-474c-9878-aa84f73234da
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/simple_negative_cases.go:159]: failed to create node pool np-1
Unexpected error:
<*fmt.wrapError | 0x1e6e8804d3c0>:
failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "negative-tests-cluster" in resourcegroup="rg-negative-tests-m5vrcvff2ctf" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/75b8b1f9-f718-45b2-b1ee-a3009f824cf8
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/75b8b1f9-f718-45b2-b1ee-a3009f824cf8",
"name": "75b8b1f9-f718-45b2-b1ee-a3009f824cf8",
"status": "Failed",
"startTime": "2026-10-22T14:53:42.149827787Z",
"endTime": "2026-10-22T15:12:50.02702561Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine c2g2u9m5x1z4p0m-np-1-bfdzc-5kdm2: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-negative-tests-m5vrcvff2ctf--managed/providers/Microsoft.Compute/virtualMachines/c2g2u9m5x1z4p0m-np-1-bfdzc-5kdm2 to exist\nMachine c2g2u9m5x1z4p0m-np-1-bfdzc-kbcws: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-negative-tests-m5vrcvff2ctf--managed/providers/Microsoft.Compute/virtualMachines/c2g2u9m5x1z4p0m-np-1-bfdzc-kbcws to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine c2g2u9m5x1z4p0m-np-1-bfdzc-5kdm2: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-negative-tests-m5vrcvff2ctf--managed/providers/Microsoft.Compute/virtualMachines/c2g2u9m5x1z4p0m-np-1-bfdzc-5kdm2 to exist\nMachine c2g2u9m5x1z4p0m-np-1-bfdzc-kbcws: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-negative-tests-m5vrcvff2ctf--managed/providers/Microsoft.Compute/virtualMachines/c2g2u9m5x1z4p0m-np-1-bfdzc-kbcws to exist\n"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/75b8b1f9-f718-45b2-b1ee-a3009f824cf8
correlation ID: e990e88e-110c-474c-9878-aa84f73234da
client request ID: unavailable
occurredContributing tests (1)
Affected runs (1)
| |||||||||||||||||||||||||||||||||||||||||||||||||||||
ERROR CODE: DeploymentFailed; detail code DeploymentScriptACIProvisioningTimeout; detail message The supporting resource: 'Azure Container Instance' did not provision during the e...full failure pattern: ERROR CODE: DeploymentFailed; detail code DeploymentScriptACIProvisioningTimeout; detail message The supporting resource: 'Azure Container Instance' did not provision during the expected time. Correlation Id: <uuid>. Please try again later, if the issue persists contact technical support for further investigation. Signal: IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | unknown | 1 | 3.45%1 of 29 job runs affected | IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | Show trend detailsOct 12: 0 · Oct 13: 0 · Oct 14: 0 · Oct 15: 0 · Oct 16: 0 · Oct 17: 0 · Oct 18: 0 · Oct 19: 0 · Oct 20: 0 · Oct 21: 0 · Oct 22: 1 · Oct 23: 0 · Oct 24: 0 · Oct 25: 0 | none | |||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/e2e/cluster_etcd_mhsm.go:106]: failed to deploy and activate Managed HSM
Unexpected error:
<*fmt.wrapError | 0x563a226e760>:
failed waiting for deployment "mhsm-deploy-etcd-mhsm" in resourcegroup="etcd-mhsm-blgzqwztp2gh" to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/etcd-mhsm-blgzqwztp2gh/providers/Microsoft.Resources/deployments/mhsm-deploy-etcd-mhsm/operationStatuses/08584089270308456271
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeploymentFailed
--------------------------------------------------------------------------------
{
"status": "Failed",
"error": {
"code": "DeploymentFailed",
"message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.",
"details": [
{
"code": "Conflict",
"message": "{\r\n \"status\": \"failed\",\r\n \"error\": {\r\n \"code\": \"ResourceDeploymentFailure\",\r\n \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'failed'.\",\r\n \"details\": [\r\n {\r\n \"code\": \"DeploymentScriptACIProvisioningTimeout\",\r\n \"message\": \"The supporting resource: 'Azure Container Instance' did not provision during the expected time. Correlation Id: c1661afd-ea85-487b-8e90-65a3f8664855. Please try again later, if the issue persists contact technical support for further investigation.\"\r\n }\r\n ]\r\n }\r\n}"
}
]
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/etcd-mhsm-blgzqwztp2gh/providers/Microsoft.Resources/deployments/mhsm-deploy-etcd-mhsm/operationStatuses/08584089270308456271
correlation ID: 989b1e9c-002b-494d-ae3b-825cc5f4adf2
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_etcd_mhsm.go:106]: failed to deploy and activate Managed HSM
Unexpected error:
<*fmt.wrapError | 0x563a226e760>:
failed waiting for deployment "mhsm-deploy-etcd-mhsm" in resourcegroup="etcd-mhsm-blgzqwztp2gh" to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/etcd-mhsm-blgzqwztp2gh/providers/Microsoft.Resources/deployments/mhsm-deploy-etcd-mhsm/operationStatuses/08584089270308456271
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeploymentFailed
--------------------------------------------------------------------------------
{
"status": "Failed",
"error": {
"code": "DeploymentFailed",
"message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.",
"details": [
{
"code": "Conflict",
"message": "{\r\n \"status\": \"failed\",\r\n \"error\": {\r\n \"code\": \"ResourceDeploymentFailure\",\r\n \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'failed'.\",\r\n \"details\": [\r\n {\r\n \"code\": \"DeploymentScriptACIProvisioningTimeout\",\r\n \"message\": \"The supporting resource: 'Azure Container Instance' did not provision during the expected time. Correlation Id: c1661afd-ea85-487b-8e90-65a3f8664855. Please try again later, if the issue persists contact technical support for further investigation.\"\r\n }\r\n ]\r\n }\r\n}"
}
]
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/etcd-mhsm-blgzqwztp2gh/providers/Microsoft.Resources/deployments/mhsm-deploy-etcd-mhsm/operationStatuses/08584089270308456271
correlation ID: 989b1e9c-002b-494d-ae3b-825cc5f4adf2
client request ID: unavailable
occurredContributing tests (1)
Affected runs (1)
| |||||||||||||||||||||||||||||||||||||||||||||||||||||
ERROR CODE: NotFound; detail message The resource record '*.e2e-dns-c2hldx' does not exist in resource group '<resource-group>' of subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX...full failure pattern: ERROR CODE: NotFound; detail message The resource record '*.e2e-dns-c2hldx' does not exist in resource group '<resource-group>' of subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX'.; provider Microsoft.Network Signal: IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | unknown | 1 | 3.45%1 of 29 job runs affected | IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | Show trend detailsOct 12: 0 · Oct 13: 0 · Oct 14: 0 · Oct 15: 0 · Oct 16: 0 · Oct 17: 0 · Oct 18: 0 · Oct 19: 0 · Oct 20: 0 · Oct 21: 0 · Oct 22: 0 · Oct 23: 0 · Oct 24: 0 · Oct 25: 1 | none | |||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:246]: failed to verify cluster operators on cluster "cluster-candidate-5-1-g8n765"
Unexpected error:
<*errors.joinError | 0x254aca410498>:
VerifyIngressDNSRecord(aro.u3d0c7m0f4m7m7j.sg1c.westus3.aroapp-hcp.azure-test.net) failed: VerifyIngressDNSRecord(aro.u3d0c7m0f4m7m7j.sg1c.westus3.aroapp-hcp.azure-test.net) timed out after 10m0.008s: DNS A record "*.e2e-dns-c2hldx" not yet present in zone rg-candidate-5-1-jqlpsk27qjbp-candidate-g8n765--managed/aro.u3d0c7m0f4m7m7j.sg1c.westus3.aroapp-hcp.azure-test.net: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-5-1-jqlpsk27qjbp-candidate-g8n765--managed/providers/Microsoft.Network/dnsZones/aro.u3d0c7m0f4m7m7j.sg1c.westus3.aroapp-hcp.azure-test.net/A/*.e2e-dns-c2hldx
--------------------------------------------------------------------------------
RESPONSE 404: 404 Not Found
ERROR CODE: NotFound
--------------------------------------------------------------------------------
{
"code": "NotFound",
"message": "The resource record '*.e2e-dns-c2hldx' does not exist in resource group 'rg-candidate-5-1-jqlpsk27qjbp-candidate-g8n765--managed' of subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX'."
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-5-1-jqlpsk27qjbp-candidate-g8n765--managed/providers/Microsoft.Network/dnsZones/aro.u3d0c7m0f4m7m7j.sg1c.westus3.aroapp-hcp.azure-test.net/A/*.e2e-dns-c2hldx
correlation ID: 192cc89e-15d5-4e23-8532-45b3cbb07059
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:246]: failed to verify cluster operators on cluster "cluster-candidate-5-1-g8n765"
Unexpected error:
<*errors.joinError | 0x254aca410498>:
VerifyIngressDNSRecord(aro.u3d0c7m0f4m7m7j.sg1c.westus3.aroapp-hcp.azure-test.net) failed: VerifyIngressDNSRecord(aro.u3d0c7m0f4m7m7j.sg1c.westus3.aroapp-hcp.azure-test.net) timed out after 10m0.008s: DNS A record "*.e2e-dns-c2hldx" not yet present in zone rg-candidate-5-1-jqlpsk27qjbp-candidate-g8n765--managed/aro.u3d0c7m0f4m7m7j.sg1c.westus3.aroapp-hcp.azure-test.net: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-5-1-jqlpsk27qjbp-candidate-g8n765--managed/providers/Microsoft.Network/dnsZones/aro.u3d0c7m0f4m7m7j.sg1c.westus3.aroapp-hcp.azure-test.net/A/*.e2e-dns-c2hldx
--------------------------------------------------------------------------------
RESPONSE 404: 404 Not Found
ERROR CODE: NotFound
--------------------------------------------------------------------------------
{
"code": "NotFound",
"message": "The resource record '*.e2e-dns-c2hldx' does not exist in resource group 'rg-candidate-5-1-jqlpsk27qjbp-candidate-g8n765--managed' of subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX'."
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-5-1-jqlpsk27qjbp-candidate-g8n765--managed/providers/Microsoft.Network/dnsZones/aro.u3d0c7m0f4m7m7j.sg1c.westus3.aroapp-hcp.azure-test.net/A/*.e2e-dns-c2hldx
correlation ID: 192cc89e-15d5-4e23-8532-45b3cbb07059
client request ID: unavailable
occurredContributing tests (1)
Affected runs (1)
| |||||||||||||||||||||||||||||||||||||||||||||||||||||
ERROR CODE: NotFound; detail message The resource record '*.e2e-dns-jgd8x5' does not exist in resource group '<resource-group>' of subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX...full failure pattern: ERROR CODE: NotFound; detail message The resource record '*.e2e-dns-jgd8x5' does not exist in resource group '<resource-group>' of subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX'.; provider Microsoft.Network Signal: IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | unknown | 1 | 3.45%1 of 29 job runs affected | IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | Show trend detailsOct 12: 0 · Oct 13: 0 · Oct 14: 0 · Oct 15: 0 · Oct 16: 0 · Oct 17: 0 · Oct 18: 0 · Oct 19: 0 · Oct 20: 0 · Oct 21: 0 · Oct 22: 0 · Oct 23: 0 · Oct 24: 0 · Oct 25: 1 | none | |||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:246]: failed to verify cluster operators on cluster "cluster-candidate-4-22-652kdm"
Unexpected error:
<*errors.joinError | 0x282c0f144570>:
VerifyIngressDNSRecord(aro.u8o6a8v1u0z8q8o.bssq.westus3.aroapp-hcp.azure-test.net) failed: VerifyIngressDNSRecord(aro.u8o6a8v1u0z8q8o.bssq.westus3.aroapp-hcp.azure-test.net) timed out after 10m0.006s: DNS A record "*.e2e-dns-jgd8x5" not yet present in zone rg-candidate-4-22-b89xsggl4sd5-candidate-652kdm--managed/aro.u8o6a8v1u0z8q8o.bssq.westus3.aroapp-hcp.azure-test.net: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-22-b89xsggl4sd5-candidate-652kdm--managed/providers/Microsoft.Network/dnsZones/aro.u8o6a8v1u0z8q8o.bssq.westus3.aroapp-hcp.azure-test.net/A/*.e2e-dns-jgd8x5
--------------------------------------------------------------------------------
RESPONSE 404: 404 Not Found
ERROR CODE: NotFound
--------------------------------------------------------------------------------
{
"code": "NotFound",
"message": "The resource record '*.e2e-dns-jgd8x5' does not exist in resource group 'rg-candidate-4-22-b89xsggl4sd5-candidate-652kdm--managed' of subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX'."
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-22-b89xsggl4sd5-candidate-652kdm--managed/providers/Microsoft.Network/dnsZones/aro.u8o6a8v1u0z8q8o.bssq.westus3.aroapp-hcp.azure-test.net/A/*.e2e-dns-jgd8x5
correlation ID: a77d314a-799e-4560-8840-6738610cf0af
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:246]: failed to verify cluster operators on cluster "cluster-candidate-4-22-652kdm"
Unexpected error:
<*errors.joinError | 0x282c0f144570>:
VerifyIngressDNSRecord(aro.u8o6a8v1u0z8q8o.bssq.westus3.aroapp-hcp.azure-test.net) failed: VerifyIngressDNSRecord(aro.u8o6a8v1u0z8q8o.bssq.westus3.aroapp-hcp.azure-test.net) timed out after 10m0.006s: DNS A record "*.e2e-dns-jgd8x5" not yet present in zone rg-candidate-4-22-b89xsggl4sd5-candidate-652kdm--managed/aro.u8o6a8v1u0z8q8o.bssq.westus3.aroapp-hcp.azure-test.net: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-22-b89xsggl4sd5-candidate-652kdm--managed/providers/Microsoft.Network/dnsZones/aro.u8o6a8v1u0z8q8o.bssq.westus3.aroapp-hcp.azure-test.net/A/*.e2e-dns-jgd8x5
--------------------------------------------------------------------------------
RESPONSE 404: 404 Not Found
ERROR CODE: NotFound
--------------------------------------------------------------------------------
{
"code": "NotFound",
"message": "The resource record '*.e2e-dns-jgd8x5' does not exist in resource group 'rg-candidate-4-22-b89xsggl4sd5-candidate-652kdm--managed' of subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX'."
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-22-b89xsggl4sd5-candidate-652kdm--managed/providers/Microsoft.Network/dnsZones/aro.u8o6a8v1u0z8q8o.bssq.westus3.aroapp-hcp.azure-test.net/A/*.e2e-dns-jgd8x5
correlation ID: a77d314a-799e-4560-8840-6738610cf0af
client request ID: unavailable
occurredContributing tests (1)
Affected runs (1)
| |||||||||||||||||||||||||||||||||||||||||||||||||||||
ERROR CODE: NotFound; detail message The resource record '*.e2e-dns-rg99mn' does not exist in resource group '<resource-group>' of subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX...full failure pattern: ERROR CODE: NotFound; detail message The resource record '*.e2e-dns-rg99mn' does not exist in resource group '<resource-group>' of subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX'.; provider Microsoft.Network Signal: IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | unknown | 1 | 3.45%1 of 29 job runs affected | IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | Show trend detailsOct 12: 0 · Oct 13: 0 · Oct 14: 0 · Oct 15: 0 · Oct 16: 0 · Oct 17: 0 · Oct 18: 0 · Oct 19: 0 · Oct 20: 0 · Oct 21: 0 · Oct 22: 0 · Oct 23: 0 · Oct 24: 0 · Oct 25: 1 | none | |||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:246]: failed to verify cluster operators on cluster "cluster-candidate-4-22-4fhvgp"
Unexpected error:
<*errors.joinError | 0x3188fb29e5a0>:
VerifyIngressDNSRecord(aro.d8c1z3j0o2p5g9d.3jyi.westus3.aroapp-hcp.azure-test.net) failed: VerifyIngressDNSRecord(aro.d8c1z3j0o2p5g9d.3jyi.westus3.aroapp-hcp.azure-test.net) timed out after 10m0.005s: DNS A record "*.e2e-dns-rg99mn" not yet present in zone rg-candidate-4-22-sp7fr6hfrw9x-candidate-4fhvgp--managed/aro.d8c1z3j0o2p5g9d.3jyi.westus3.aroapp-hcp.azure-test.net: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-22-sp7fr6hfrw9x-candidate-4fhvgp--managed/providers/Microsoft.Network/dnsZones/aro.d8c1z3j0o2p5g9d.3jyi.westus3.aroapp-hcp.azure-test.net/A/*.e2e-dns-rg99mn
--------------------------------------------------------------------------------
RESPONSE 404: 404 Not Found
ERROR CODE: NotFound
--------------------------------------------------------------------------------
{
"code": "NotFound",
"message": "The resource record '*.e2e-dns-rg99mn' does not exist in resource group 'rg-candidate-4-22-sp7fr6hfrw9x-candidate-4fhvgp--managed' of subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX'."
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-22-sp7fr6hfrw9x-candidate-4fhvgp--managed/providers/Microsoft.Network/dnsZones/aro.d8c1z3j0o2p5g9d.3jyi.westus3.aroapp-hcp.azure-test.net/A/*.e2e-dns-rg99mn
correlation ID: cfd4563d-f15e-486d-ad09-c0f336987b6a
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:246]: failed to verify cluster operators on cluster "cluster-candidate-4-22-4fhvgp"
Unexpected error:
<*errors.joinError | 0x3188fb29e5a0>:
VerifyIngressDNSRecord(aro.d8c1z3j0o2p5g9d.3jyi.westus3.aroapp-hcp.azure-test.net) failed: VerifyIngressDNSRecord(aro.d8c1z3j0o2p5g9d.3jyi.westus3.aroapp-hcp.azure-test.net) timed out after 10m0.005s: DNS A record "*.e2e-dns-rg99mn" not yet present in zone rg-candidate-4-22-sp7fr6hfrw9x-candidate-4fhvgp--managed/aro.d8c1z3j0o2p5g9d.3jyi.westus3.aroapp-hcp.azure-test.net: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-22-sp7fr6hfrw9x-candidate-4fhvgp--managed/providers/Microsoft.Network/dnsZones/aro.d8c1z3j0o2p5g9d.3jyi.westus3.aroapp-hcp.azure-test.net/A/*.e2e-dns-rg99mn
--------------------------------------------------------------------------------
RESPONSE 404: 404 Not Found
ERROR CODE: NotFound
--------------------------------------------------------------------------------
{
"code": "NotFound",
"message": "The resource record '*.e2e-dns-rg99mn' does not exist in resource group 'rg-candidate-4-22-sp7fr6hfrw9x-candidate-4fhvgp--managed' of subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX'."
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-22-sp7fr6hfrw9x-candidate-4fhvgp--managed/providers/Microsoft.Network/dnsZones/aro.d8c1z3j0o2p5g9d.3jyi.westus3.aroapp-hcp.azure-test.net/A/*.e2e-dns-rg99mn
correlation ID: cfd4563d-f15e-486d-ad09-c0f336987b6a
client request ID: unavailable
occurredContributing tests (1)
Affected runs (1)
| |||||||||||||||||||||||||||||||||||||||||||||||||||||
Internal error occurred: failed calling webhook "webhookfull failure pattern: Internal error occurred: failed calling webhook "webhook Signal: IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | unknown | 1 | 3.45%1 of 29 job runs affected | IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | Show trend detailsOct 12: 0 · Oct 13: 0 · Oct 14: 0 · Oct 15: 0 · Oct 16: 0 · Oct 17: 0 · Oct 18: 0 · Oct 19: 0 · Oct 20: 0 · Oct 21: 0 · Oct 22: 0 · Oct 23: 0 · Oct 24: 1 · Oct 25: 0 | DEV | |||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:372]: Timed out after 120.003s.
ClusterIssuer CRD should be installed and the resource creatable
Expected success, but got an error:
<*errors.StatusError | 0x11d151799220>:
Internal error occurred: failed calling webhook "webhook.cert-manager.io": failed to call webhook: Post "https://cert-manager-webhook.cert-manager.svc:443/validate?timeout=30s": no endpoints available for service "cert-manager-webhook"
{
ErrStatus: {
TypeMeta: {Kind: "Status", APIVersion: "v1"},
ListMeta: {
SelfLink: "",
ResourceVersion: "",
Continue: "",
RemainingItemCount: nil,
ShardInfo: nil,
},
Status: "Failure",
Message: "Internal error occurred: failed calling webhook \"webhook.cert-manager.io\": failed to call webhook: Post \"https://cert-manager-webhook.cert-manager.svc:443/validate?timeout=30s\": no endpoints available for service \"cert-manager-webhook\"",
Reason: "InternalError",
Details: {
Name: "",
Group: "",
Kind: "",
UID: "",
Causes: [
{
Type: "",
Message: "failed calling webhook \"webhook.cert-manager.io\": failed to call webhook: Post \"https://cert-manager-webhook.cert-manager.svc:443/validate?timeout=30s\": no endpoints available for service \"cert-manager-webhook\"",
Field: "",
},
],
RetryAfterSeconds: 0,
},
Code: 500,
},
}
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:372]: Timed out after 120.003s.
ClusterIssuer CRD should be installed and the resource creatable
Expected success, but got an error:
<*errors.StatusError | 0x11d151799220>:
Internal error occurred: failed calling webhook "webhook.cert-manager.io": failed to call webhook: Post "https://cert-manager-webhook.cert-manager.svc:443/validate?timeout=30s": no endpoints available for service "cert-manager-webhook"
{
ErrStatus: {
TypeMeta: {Kind: "Status", APIVersion: "v1"},
ListMeta: {
SelfLink: "",
ResourceVersion: "",
Continue: "",
RemainingItemCount: nil,
ShardInfo: nil,
},
Status: "Failure",
Message: "Internal error occurred: failed calling webhook \"webhook.cert-manager.io\": failed to call webhook: Post \"https://cert-manager-webhook.cert-manager.svc:443/validate?timeout=30s\": no endpoints available for service \"cert-manager-webhook\"",
Reason: "InternalError",
Details: {
Name: "",
Group: "",
Kind: "",
UID: "",
Causes: [
{
Type: "",
Message: "failed calling webhook \"webhook.cert-manager.io\": failed to call webhook: Post \"https://cert-manager-webhook.cert-manager.svc:443/validate?timeout=30s\": no endpoints available for service \"cert-manager-webhook\"",
Field: "",
},
],
RetryAfterSeconds: 0,
},
Code: 500,
},
}Contributing tests (1)
Affected runs (1)
| |||||||||||||||||||||||||||||||||||||||||||||||||||||
VerifyImagePulled(acrpullxt7r5cdx2wbm.azurecr.io/ubi-minimal) timed out after 2m0.005s: failed to list pods in namespace acr-pull-test-day2: context deadline exceededfull failure pattern: VerifyImagePulled(acrpullxt7r5cdx2wbm.azurecr.io/ubi-minimal) timed out after 2m0.005s: failed to list pods in namespace acr-pull-test-day2: context deadline exceeded Signal: IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | unknown | 1 | 3.45%1 of 29 job runs affected | IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | Show trend detailsOct 12: 0 · Oct 13: 0 · Oct 14: 0 · Oct 15: 0 · Oct 16: 0 · Oct 17: 0 · Oct 18: 0 · Oct 19: 0 · Oct 20: 0 · Oct 21: 1 · Oct 22: 0 · Oct 23: 0 · Oct 24: 0 · Oct 25: 0 | none | |||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/e2e/cluster_container_registry.go:346]: [day 2] pod in namespace acr-pull-test-day2 never pulled image ubi-minimal from private ACR acrpullxt7r5cdx2wbm.azurecr.io
Unexpected error:
<*fmt.wrapError | 0xfec25889960>:
VerifyImagePulled(acrpullxt7r5cdx2wbm.azurecr.io/ubi-minimal) timed out after 2m0.005s: failed to list pods in namespace acr-pull-test-day2: context deadline exceeded
{
msg: "VerifyImagePulled(acrpullxt7r5cdx2wbm.azurecr.io/ubi-minimal) timed out after 2m0.005s: failed to list pods in namespace acr-pull-test-day2: context deadline exceeded",
err: <*fmt.wrapError | 0xfec25889880>{
msg: "failed to list pods in namespace acr-pull-test-day2: context deadline exceeded",
err: <context.deadlineExceededError>{},
},
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_container_registry.go:346]: [day 2] pod in namespace acr-pull-test-day2 never pulled image ubi-minimal from private ACR acrpullxt7r5cdx2wbm.azurecr.io
Unexpected error:
<*fmt.wrapError | 0xfec25889960>:
VerifyImagePulled(acrpullxt7r5cdx2wbm.azurecr.io/ubi-minimal) timed out after 2m0.005s: failed to list pods in namespace acr-pull-test-day2: context deadline exceeded
{
msg: "VerifyImagePulled(acrpullxt7r5cdx2wbm.azurecr.io/ubi-minimal) timed out after 2m0.005s: failed to list pods in namespace acr-pull-test-day2: context deadline exceeded",
err: <*fmt.wrapError | 0xfec25889880>{
msg: "failed to list pods in namespace acr-pull-test-day2: context deadline exceeded",
err: <context.deadlineExceededError>{},
},
}
occurredContributing tests (1)
Affected runs (1)
| |||||||||||||||||||||||||||||||||||||||||||||||||||||
clusterIssuer not ready: reason=ErrRegisterACMEAccount message=Failed to register ACME account: Get "<url>full failure pattern: clusterIssuer not ready: reason=ErrRegisterACMEAccount message=Failed to register ACME account: Get "<url> Signal: IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | unknown | 1 | 3.45%1 of 29 job runs affected | IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | Show trend detailsOct 12: 0 · Oct 13: 0 · Oct 14: 0 · Oct 15: 0 · Oct 16: 0 · Oct 17: 0 · Oct 18: 0 · Oct 19: 0 · Oct 20: 0 · Oct 21: 0 · Oct 22: 0 · Oct 23: 1 · Oct 24: 0 · Oct 25: 0 | DEV | |||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:391]: Timed out after 300.002s.
ClusterIssuer should become Ready (ACME account registered)
Expected success, but got an error:
<*errors.errorString | 0x1e28491f1160>:
clusterIssuer not ready: reason=ErrRegisterACMEAccount message=Failed to register ACME account: Get "https://acme-staging-v02.api.letsencrypt.org/directory": dial tcp: lookup acme-staging-v02.api.letsencrypt.org on 172.30.0.10:53: read udp 10.129.0.21:39416->172.30.0.10:53: read: connection refused
{
s: "clusterIssuer not ready: reason=ErrRegisterACMEAccount message=Failed to register ACME account: Get \"https://acme-staging-v02.api.letsencrypt.org/directory\": dial tcp: lookup acme-staging-v02.api.letsencrypt.org on 172.30.0.10:53: read udp 10.129.0.21:39416->172.30.0.10:53: read: connection refused",
}
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:391]: Timed out after 300.002s.
ClusterIssuer should become Ready (ACME account registered)
Expected success, but got an error:
<*errors.errorString | 0x1e28491f1160>:
clusterIssuer not ready: reason=ErrRegisterACMEAccount message=Failed to register ACME account: Get "https://acme-staging-v02.api.letsencrypt.org/directory": dial tcp: lookup acme-staging-v02.api.letsencrypt.org on 172.30.0.10:53: read udp 10.129.0.21:39416->172.30.0.10:53: read: connection refused
{
s: "clusterIssuer not ready: reason=ErrRegisterACMEAccount message=Failed to register ACME account: Get \"https://acme-staging-v02.api.letsencrypt.org/directory\": dial tcp: lookup acme-staging-v02.api.letsencrypt.org on 172.30.0.10:53: read udp 10.129.0.21:39416->172.30.0.10:53: read: connection refused",
}Contributing tests (1)
Affected runs (1)
| |||||||||||||||||||||||||||||||||||||||||||||||||||||
console fixture namespace "<namespace>": configure Console ExternalAuth through RP: context deadline exceeded; Dex namespace="console-lrpv2qd"; Deployment/dex condition="Available...full failure pattern: console fixture namespace "<namespace>": configure Console ExternalAuth through RP: context deadline exceeded; Dex namespace="console-lrpv2qd"; Deployment/dex condition="Available" status="True" reason="MinimumReplicasAvailable"; Deployment/dex Signal: IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | unknown | 1 | 3.45%1 of 29 job runs affected | IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | Show trend detailsOct 12: 0 · Oct 13: 0 · Oct 14: 0 · Oct 15: 0 · Oct 16: 0 · Oct 17: 0 · Oct 18: 0 · Oct 19: 0 · Oct 20: 0 · Oct 21: 0 · Oct 22: 0 · Oct 23: 1 · Oct 24: 0 · Oct 25: 0 | none | |||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:197]: failed to set up Console login fixture for cluster "cluster-candidate-4-20-4x46vw"
Unexpected error:
<*fmt.wrapError | 0x3b1760358220>:
console fixture namespace "console-lrpv2qd": configure Console ExternalAuth through RP: context deadline exceeded; Dex namespace="console-lrpv2qd"; Deployment/dex condition="Available" status="True" reason="MinimumReplicasAvailable"; Deployment/dex condition="Progressing" status="True" reason="NewReplicaSetAvailable"; Deployment/dex generation=1 observed=1 replicas=1 ready=1 available=1; Dex Pods=1; Dex ReplicaSets=1; Pod/dex-78f4dcc67d-zhbqp condition="ContainersReady" status="True" reason=""; Pod/dex-78f4dcc67d-zhbqp condition="Initialized" status="True" reason=""; Pod/dex-78f4dcc67d-zhbqp condition="PodReadyToStartContainers" status="True" reason=""; Pod/dex-78f4dcc67d-zhbqp condition="PodScheduled" status="True" reason=""; Pod/dex-78f4dcc67d-zhbqp condition="Ready" status="True" reason=""; Pod/dex-78f4dcc67d-zhbqp container="dex" ready=true restarts=0; Pod/dex-78f4dcc67d-zhbqp phase="Running" reason=""
{
msg: "console fixture namespace \"console-lrpv2qd\": configure Console ExternalAuth through RP: context deadline exceeded; Dex namespace=\"console-lrpv2qd\"; Deployment/dex condition=\"Available\" status=\"True\" reason=\"MinimumReplicasAvailable\"; Deployment/dex condition=\"Progressing\" status=\"True\" reason=\"NewReplicaSetAvailable\"; Deployment/dex generation=1 observed=1 replicas=1 ready=1 available=1; Dex Pods=1; Dex ReplicaSets=1; Pod/dex-78f4dcc67d-zhbqp condition=\"ContainersReady\" status=\"True\" reason=\"\"; Pod/dex-78f4dcc67d-zhbqp condition=\"Initialized\" status=\"True\" reason=\"\"; Pod/dex-78f4dcc67d-zhbqp condition=\"PodReadyToStartContainers\" status=\"True\" reason=\"\"; Pod/dex-78f4dcc67d-zhbqp condition=\"PodScheduled\" status=\"True\" reason=\"\"; Pod/dex-78f4dcc67d-zhbqp condition=\"Ready\" status=\"True\" reason=\"\"; Pod/dex-78f4dcc67d-zhbqp container=\"dex\" ready=true restarts=0; Pod/dex-78f4dcc67d-zhbqp phase=\"Running\" reason=\"\"",
err: <*fmt.wrapError | 0x3b1760b23ba0>{
msg: "configure Console ExternalAuth through RP: context deadline exceeded",
err: <context.deadlineExceededError>{},
},
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:197]: failed to set up Console login fixture for cluster "cluster-candidate-4-20-4x46vw"
Unexpected error:
<*fmt.wrapError | 0x3b1760358220>:
console fixture namespace "console-lrpv2qd": configure Console ExternalAuth through RP: context deadline exceeded; Dex namespace="console-lrpv2qd"; Deployment/dex condition="Available" status="True" reason="MinimumReplicasAvailable"; Deployment/dex condition="Progressing" status="True" reason="NewReplicaSetAvailable"; Deployment/dex generation=1 observed=1 replicas=1 ready=1 available=1; Dex Pods=1; Dex ReplicaSets=1; Pod/dex-78f4dcc67d-zhbqp condition="ContainersReady" status="True" reason=""; Pod/dex-78f4dcc67d-zhbqp condition="Initialized" status="True" reason=""; Pod/dex-78f4dcc67d-zhbqp condition="PodReadyToStartContainers" status="True" reason=""; Pod/dex-78f4dcc67d-zhbqp condition="PodScheduled" status="True" reason=""; Pod/dex-78f4dcc67d-zhbqp condition="Ready" status="True" reason=""; Pod/dex-78f4dcc67d-zhbqp container="dex" ready=true restarts=0; Pod/dex-78f4dcc67d-zhbqp phase="Running" reason=""
{
msg: "console fixture namespace \"console-lrpv2qd\": configure Console ExternalAuth through RP: context deadline exceeded; Dex namespace=\"console-lrpv2qd\"; Deployment/dex condition=\"Available\" status=\"True\" reason=\"MinimumReplicasAvailable\"; Deployment/dex condition=\"Progressing\" status=\"True\" reason=\"NewReplicaSetAvailable\"; Deployment/dex generation=1 observed=1 replicas=1 ready=1 available=1; Dex Pods=1; Dex ReplicaSets=1; Pod/dex-78f4dcc67d-zhbqp condition=\"ContainersReady\" status=\"True\" reason=\"\"; Pod/dex-78f4dcc67d-zhbqp condition=\"Initialized\" status=\"True\" reason=\"\"; Pod/dex-78f4dcc67d-zhbqp condition=\"PodReadyToStartContainers\" status=\"True\" reason=\"\"; Pod/dex-78f4dcc67d-zhbqp condition=\"PodScheduled\" status=\"True\" reason=\"\"; Pod/dex-78f4dcc67d-zhbqp condition=\"Ready\" status=\"True\" reason=\"\"; Pod/dex-78f4dcc67d-zhbqp container=\"dex\" ready=true restarts=0; Pod/dex-78f4dcc67d-zhbqp phase=\"Running\" reason=\"\"",
err: <*fmt.wrapError | 0x3b1760b23ba0>{
msg: "configure Console ExternalAuth through RP: context deadline exceeded",
err: <context.deadlineExceededError>{},
},
}
occurredContributing tests (1)
Affected runs (1)
| |||||||||||||||||||||||||||||||||||||||||||||||||||||
console fixture namespace "<namespace>": configure Console ExternalAuth through RP: context deadline exceeded; Dex namespace="console-mr8rvbh"; Deployment/dex condition="Available...full failure pattern: console fixture namespace "<namespace>": configure Console ExternalAuth through RP: context deadline exceeded; Dex namespace="console-mr8rvbh"; Deployment/dex condition="Available" status="True" reason="MinimumReplicasAvailable"; Deployment/dex Signal: IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | unknown | 1 | 3.45%1 of 29 job runs affected | IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | Show trend detailsOct 12: 0 · Oct 13: 0 · Oct 14: 0 · Oct 15: 0 · Oct 16: 0 · Oct 17: 0 · Oct 18: 0 · Oct 19: 0 · Oct 20: 0 · Oct 21: 0 · Oct 22: 1 · Oct 23: 0 · Oct 24: 0 · Oct 25: 0 | none | |||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:194]: failed to set up Console login fixture for cluster "cluster-candidate-5-0-6rxg4b"
Unexpected error:
<*fmt.wrapError | 0x12de498cec20>:
console fixture namespace "console-mr8rvbh": configure Console ExternalAuth through RP: context deadline exceeded; Dex namespace="console-mr8rvbh"; Deployment/dex condition="Available" status="True" reason="MinimumReplicasAvailable"; Deployment/dex condition="Progressing" status="True" reason="NewReplicaSetAvailable"; Deployment/dex generation=1 observed=1 replicas=1 ready=1 available=1; Dex Pods=1; Dex ReplicaSets=1; Pod/dex-77657bcfc5-zhr7j condition="ContainersReady" status="True" reason=""; Pod/dex-77657bcfc5-zhr7j condition="Initialized" status="True" reason=""; Pod/dex-77657bcfc5-zhr7j condition="PodReadyToStartContainers" status="True" reason=""; Pod/dex-77657bcfc5-zhr7j condition="PodScheduled" status="True" reason=""; Pod/dex-77657bcfc5-zhr7j condition="Ready" status="True" reason=""; Pod/dex-77657bcfc5-zhr7j container="dex" ready=true restarts=0; Pod/dex-77657bcfc5-zhr7j phase="Running" reason=""
{
msg: "console fixture namespace \"console-mr8rvbh\": configure Console ExternalAuth through RP: context deadline exceeded; Dex namespace=\"console-mr8rvbh\"; Deployment/dex condition=\"Available\" status=\"True\" reason=\"MinimumReplicasAvailable\"; Deployment/dex condition=\"Progressing\" status=\"True\" reason=\"NewReplicaSetAvailable\"; Deployment/dex generation=1 observed=1 replicas=1 ready=1 available=1; Dex Pods=1; Dex ReplicaSets=1; Pod/dex-77657bcfc5-zhr7j condition=\"ContainersReady\" status=\"True\" reason=\"\"; Pod/dex-77657bcfc5-zhr7j condition=\"Initialized\" status=\"True\" reason=\"\"; Pod/dex-77657bcfc5-zhr7j condition=\"PodReadyToStartContainers\" status=\"True\" reason=\"\"; Pod/dex-77657bcfc5-zhr7j condition=\"PodScheduled\" status=\"True\" reason=\"\"; Pod/dex-77657bcfc5-zhr7j condition=\"Ready\" status=\"True\" reason=\"\"; Pod/dex-77657bcfc5-zhr7j container=\"dex\" ready=true restarts=0; Pod/dex-77657bcfc5-zhr7j phase=\"Running\" reason=\"\"",
err: <*fmt.wrapError | 0x12de4a14f120>{
msg: "configure Console ExternalAuth through RP: context deadline exceeded",
err: <context.deadlineExceededError>{},
},
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:194]: failed to set up Console login fixture for cluster "cluster-candidate-5-0-6rxg4b"
Unexpected error:
<*fmt.wrapError | 0x12de498cec20>:
console fixture namespace "console-mr8rvbh": configure Console ExternalAuth through RP: context deadline exceeded; Dex namespace="console-mr8rvbh"; Deployment/dex condition="Available" status="True" reason="MinimumReplicasAvailable"; Deployment/dex condition="Progressing" status="True" reason="NewReplicaSetAvailable"; Deployment/dex generation=1 observed=1 replicas=1 ready=1 available=1; Dex Pods=1; Dex ReplicaSets=1; Pod/dex-77657bcfc5-zhr7j condition="ContainersReady" status="True" reason=""; Pod/dex-77657bcfc5-zhr7j condition="Initialized" status="True" reason=""; Pod/dex-77657bcfc5-zhr7j condition="PodReadyToStartContainers" status="True" reason=""; Pod/dex-77657bcfc5-zhr7j condition="PodScheduled" status="True" reason=""; Pod/dex-77657bcfc5-zhr7j condition="Ready" status="True" reason=""; Pod/dex-77657bcfc5-zhr7j container="dex" ready=true restarts=0; Pod/dex-77657bcfc5-zhr7j phase="Running" reason=""
{
msg: "console fixture namespace \"console-mr8rvbh\": configure Console ExternalAuth through RP: context deadline exceeded; Dex namespace=\"console-mr8rvbh\"; Deployment/dex condition=\"Available\" status=\"True\" reason=\"MinimumReplicasAvailable\"; Deployment/dex condition=\"Progressing\" status=\"True\" reason=\"NewReplicaSetAvailable\"; Deployment/dex generation=1 observed=1 replicas=1 ready=1 available=1; Dex Pods=1; Dex ReplicaSets=1; Pod/dex-77657bcfc5-zhr7j condition=\"ContainersReady\" status=\"True\" reason=\"\"; Pod/dex-77657bcfc5-zhr7j condition=\"Initialized\" status=\"True\" reason=\"\"; Pod/dex-77657bcfc5-zhr7j condition=\"PodReadyToStartContainers\" status=\"True\" reason=\"\"; Pod/dex-77657bcfc5-zhr7j condition=\"PodScheduled\" status=\"True\" reason=\"\"; Pod/dex-77657bcfc5-zhr7j condition=\"Ready\" status=\"True\" reason=\"\"; Pod/dex-77657bcfc5-zhr7j container=\"dex\" ready=true restarts=0; Pod/dex-77657bcfc5-zhr7j phase=\"Running\" reason=\"\"",
err: <*fmt.wrapError | 0x12de4a14f120>{
msg: "configure Console ExternalAuth through RP: context deadline exceeded",
err: <context.deadlineExceededError>{},
},
}
occurredContributing tests (1)
Affected runs (1)
| |||||||||||||||||||||||||||||||||||||||||||||||||||||
console fixture namespace "<namespace>": configure Console ExternalAuth through RP: context deadline exceeded; Dex namespace="console-n2bkhw2"; Deployment/dex condition="Available...full failure pattern: console fixture namespace "<namespace>": configure Console ExternalAuth through RP: context deadline exceeded; Dex namespace="console-n2bkhw2"; Deployment/dex condition="Available" status="True" reason="MinimumReplicasAvailable"; Deployment/dex Signal: IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | unknown | 1 | 3.45%1 of 29 job runs affected | IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | Show trend detailsOct 12: 0 · Oct 13: 0 · Oct 14: 0 · Oct 15: 0 · Oct 16: 0 · Oct 17: 0 · Oct 18: 0 · Oct 19: 0 · Oct 20: 0 · Oct 21: 0 · Oct 22: 0 · Oct 23: 1 · Oct 24: 0 · Oct 25: 0 | none | |||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:197]: failed to set up Console login fixture for cluster "cluster-candidate-5-1-qx8d44"
Unexpected error:
<*fmt.wrapError | 0x39e6141ab040>:
console fixture namespace "console-n2bkhw2": configure Console ExternalAuth through RP: context deadline exceeded; Dex namespace="console-n2bkhw2"; Deployment/dex condition="Available" status="True" reason="MinimumReplicasAvailable"; Deployment/dex condition="Progressing" status="True" reason="NewReplicaSetAvailable"; Deployment/dex generation=1 observed=1 replicas=1 ready=1 available=1; Dex Pods=1; Dex ReplicaSets=1; Pod/dex-77657bcfc5-nrx7d condition="ContainersReady" status="True" reason=""; Pod/dex-77657bcfc5-nrx7d condition="Initialized" status="True" reason=""; Pod/dex-77657bcfc5-nrx7d condition="PodReadyToStartContainers" status="True" reason=""; Pod/dex-77657bcfc5-nrx7d condition="PodScheduled" status="True" reason=""; Pod/dex-77657bcfc5-nrx7d condition="Ready" status="True" reason=""; Pod/dex-77657bcfc5-nrx7d container="dex" ready=true restarts=0; Pod/dex-77657bcfc5-nrx7d phase="Running" reason=""
{
msg: "console fixture namespace \"console-n2bkhw2\": configure Console ExternalAuth through RP: context deadline exceeded; Dex namespace=\"console-n2bkhw2\"; Deployment/dex condition=\"Available\" status=\"True\" reason=\"MinimumReplicasAvailable\"; Deployment/dex condition=\"Progressing\" status=\"True\" reason=\"NewReplicaSetAvailable\"; Deployment/dex generation=1 observed=1 replicas=1 ready=1 available=1; Dex Pods=1; Dex ReplicaSets=1; Pod/dex-77657bcfc5-nrx7d condition=\"ContainersReady\" status=\"True\" reason=\"\"; Pod/dex-77657bcfc5-nrx7d condition=\"Initialized\" status=\"True\" reason=\"\"; Pod/dex-77657bcfc5-nrx7d condition=\"PodReadyToStartContainers\" status=\"True\" reason=\"\"; Pod/dex-77657bcfc5-nrx7d condition=\"PodScheduled\" status=\"True\" reason=\"\"; Pod/dex-77657bcfc5-nrx7d condition=\"Ready\" status=\"True\" reason=\"\"; Pod/dex-77657bcfc5-nrx7d container=\"dex\" ready=true restarts=0; Pod/dex-77657bcfc5-nrx7d phase=\"Running\" reason=\"\"",
err: <*fmt.wrapError | 0x39e614e48260>{
msg: "configure Console ExternalAuth through RP: context deadline exceeded",
err: <context.deadlineExceededError>{},
},
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:197]: failed to set up Console login fixture for cluster "cluster-candidate-5-1-qx8d44"
Unexpected error:
<*fmt.wrapError | 0x39e6141ab040>:
console fixture namespace "console-n2bkhw2": configure Console ExternalAuth through RP: context deadline exceeded; Dex namespace="console-n2bkhw2"; Deployment/dex condition="Available" status="True" reason="MinimumReplicasAvailable"; Deployment/dex condition="Progressing" status="True" reason="NewReplicaSetAvailable"; Deployment/dex generation=1 observed=1 replicas=1 ready=1 available=1; Dex Pods=1; Dex ReplicaSets=1; Pod/dex-77657bcfc5-nrx7d condition="ContainersReady" status="True" reason=""; Pod/dex-77657bcfc5-nrx7d condition="Initialized" status="True" reason=""; Pod/dex-77657bcfc5-nrx7d condition="PodReadyToStartContainers" status="True" reason=""; Pod/dex-77657bcfc5-nrx7d condition="PodScheduled" status="True" reason=""; Pod/dex-77657bcfc5-nrx7d condition="Ready" status="True" reason=""; Pod/dex-77657bcfc5-nrx7d container="dex" ready=true restarts=0; Pod/dex-77657bcfc5-nrx7d phase="Running" reason=""
{
msg: "console fixture namespace \"console-n2bkhw2\": configure Console ExternalAuth through RP: context deadline exceeded; Dex namespace=\"console-n2bkhw2\"; Deployment/dex condition=\"Available\" status=\"True\" reason=\"MinimumReplicasAvailable\"; Deployment/dex condition=\"Progressing\" status=\"True\" reason=\"NewReplicaSetAvailable\"; Deployment/dex generation=1 observed=1 replicas=1 ready=1 available=1; Dex Pods=1; Dex ReplicaSets=1; Pod/dex-77657bcfc5-nrx7d condition=\"ContainersReady\" status=\"True\" reason=\"\"; Pod/dex-77657bcfc5-nrx7d condition=\"Initialized\" status=\"True\" reason=\"\"; Pod/dex-77657bcfc5-nrx7d condition=\"PodReadyToStartContainers\" status=\"True\" reason=\"\"; Pod/dex-77657bcfc5-nrx7d condition=\"PodScheduled\" status=\"True\" reason=\"\"; Pod/dex-77657bcfc5-nrx7d condition=\"Ready\" status=\"True\" reason=\"\"; Pod/dex-77657bcfc5-nrx7d container=\"dex\" ready=true restarts=0; Pod/dex-77657bcfc5-nrx7d phase=\"Running\" reason=\"\"",
err: <*fmt.wrapError | 0x39e614e48260>{
msg: "configure Console ExternalAuth through RP: context deadline exceeded",
err: <context.deadlineExceededError>{},
},
}
occurredContributing tests (1)
Affected runs (1)
| |||||||||||||||||||||||||||||||||||||||||||||||||||||
console fixture namespace "<namespace>": configure Console ExternalAuth through RP: context deadline exceeded; Dex namespace="console-r7gptrj"; Deployment/dex condition="Available...full failure pattern: console fixture namespace "<namespace>": configure Console ExternalAuth through RP: context deadline exceeded; Dex namespace="console-r7gptrj"; Deployment/dex condition="Available" status="True" reason="MinimumReplicasAvailable"; Deployment/dex Signal: IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | unknown | 1 | 3.45%1 of 29 job runs affected | IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | Show trend detailsOct 12: 0 · Oct 13: 0 · Oct 14: 0 · Oct 15: 0 · Oct 16: 0 · Oct 17: 0 · Oct 18: 0 · Oct 19: 0 · Oct 20: 0 · Oct 21: 0 · Oct 22: 0 · Oct 23: 1 · Oct 24: 0 · Oct 25: 0 | none | |||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:197]: failed to set up Console login fixture for cluster "cluster-candidate-5-0-glc2bs"
Unexpected error:
<*fmt.wrapError | 0x2d9cfabb2a80>:
console fixture namespace "console-r7gptrj": configure Console ExternalAuth through RP: context deadline exceeded; Dex namespace="console-r7gptrj"; Deployment/dex condition="Available" status="True" reason="MinimumReplicasAvailable"; Deployment/dex condition="Progressing" status="True" reason="NewReplicaSetAvailable"; Deployment/dex generation=1 observed=1 replicas=1 ready=1 available=1; Dex Pods=1; Dex ReplicaSets=1; Pod/dex-77657bcfc5-mwmw6 condition="ContainersReady" status="True" reason=""; Pod/dex-77657bcfc5-mwmw6 condition="Initialized" status="True" reason=""; Pod/dex-77657bcfc5-mwmw6 condition="PodReadyToStartContainers" status="True" reason=""; Pod/dex-77657bcfc5-mwmw6 condition="PodScheduled" status="True" reason=""; Pod/dex-77657bcfc5-mwmw6 condition="Ready" status="True" reason=""; Pod/dex-77657bcfc5-mwmw6 container="dex" ready=true restarts=0; Pod/dex-77657bcfc5-mwmw6 phase="Running" reason=""
{
msg: "console fixture namespace \"console-r7gptrj\": configure Console ExternalAuth through RP: context deadline exceeded; Dex namespace=\"console-r7gptrj\"; Deployment/dex condition=\"Available\" status=\"True\" reason=\"MinimumReplicasAvailable\"; Deployment/dex condition=\"Progressing\" status=\"True\" reason=\"NewReplicaSetAvailable\"; Deployment/dex generation=1 observed=1 replicas=1 ready=1 available=1; Dex Pods=1; Dex ReplicaSets=1; Pod/dex-77657bcfc5-mwmw6 condition=\"ContainersReady\" status=\"True\" reason=\"\"; Pod/dex-77657bcfc5-mwmw6 condition=\"Initialized\" status=\"True\" reason=\"\"; Pod/dex-77657bcfc5-mwmw6 condition=\"PodReadyToStartContainers\" status=\"True\" reason=\"\"; Pod/dex-77657bcfc5-mwmw6 condition=\"PodScheduled\" status=\"True\" reason=\"\"; Pod/dex-77657bcfc5-mwmw6 condition=\"Ready\" status=\"True\" reason=\"\"; Pod/dex-77657bcfc5-mwmw6 container=\"dex\" ready=true restarts=0; Pod/dex-77657bcfc5-mwmw6 phase=\"Running\" reason=\"\"",
err: <*fmt.wrapError | 0x2d9cfac24920>{
msg: "configure Console ExternalAuth through RP: context deadline exceeded",
err: <context.deadlineExceededError>{},
},
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:197]: failed to set up Console login fixture for cluster "cluster-candidate-5-0-glc2bs"
Unexpected error:
<*fmt.wrapError | 0x2d9cfabb2a80>:
console fixture namespace "console-r7gptrj": configure Console ExternalAuth through RP: context deadline exceeded; Dex namespace="console-r7gptrj"; Deployment/dex condition="Available" status="True" reason="MinimumReplicasAvailable"; Deployment/dex condition="Progressing" status="True" reason="NewReplicaSetAvailable"; Deployment/dex generation=1 observed=1 replicas=1 ready=1 available=1; Dex Pods=1; Dex ReplicaSets=1; Pod/dex-77657bcfc5-mwmw6 condition="ContainersReady" status="True" reason=""; Pod/dex-77657bcfc5-mwmw6 condition="Initialized" status="True" reason=""; Pod/dex-77657bcfc5-mwmw6 condition="PodReadyToStartContainers" status="True" reason=""; Pod/dex-77657bcfc5-mwmw6 condition="PodScheduled" status="True" reason=""; Pod/dex-77657bcfc5-mwmw6 condition="Ready" status="True" reason=""; Pod/dex-77657bcfc5-mwmw6 container="dex" ready=true restarts=0; Pod/dex-77657bcfc5-mwmw6 phase="Running" reason=""
{
msg: "console fixture namespace \"console-r7gptrj\": configure Console ExternalAuth through RP: context deadline exceeded; Dex namespace=\"console-r7gptrj\"; Deployment/dex condition=\"Available\" status=\"True\" reason=\"MinimumReplicasAvailable\"; Deployment/dex condition=\"Progressing\" status=\"True\" reason=\"NewReplicaSetAvailable\"; Deployment/dex generation=1 observed=1 replicas=1 ready=1 available=1; Dex Pods=1; Dex ReplicaSets=1; Pod/dex-77657bcfc5-mwmw6 condition=\"ContainersReady\" status=\"True\" reason=\"\"; Pod/dex-77657bcfc5-mwmw6 condition=\"Initialized\" status=\"True\" reason=\"\"; Pod/dex-77657bcfc5-mwmw6 condition=\"PodReadyToStartContainers\" status=\"True\" reason=\"\"; Pod/dex-77657bcfc5-mwmw6 condition=\"PodScheduled\" status=\"True\" reason=\"\"; Pod/dex-77657bcfc5-mwmw6 condition=\"Ready\" status=\"True\" reason=\"\"; Pod/dex-77657bcfc5-mwmw6 container=\"dex\" ready=true restarts=0; Pod/dex-77657bcfc5-mwmw6 phase=\"Running\" reason=\"\"",
err: <*fmt.wrapError | 0x2d9cfac24920>{
msg: "configure Console ExternalAuth through RP: context deadline exceeded",
err: <context.deadlineExceededError>{},
},
}
occurredContributing tests (1)
Affected runs (1)
| |||||||||||||||||||||||||||||||||||||||||||||||||||||
console fixture namespace "<namespace>": configure Console ExternalAuth through RP: context deadline exceeded; Dex namespace="console-xzp4sq4"; Deployment/dex condition="Available...full failure pattern: console fixture namespace "<namespace>": configure Console ExternalAuth through RP: context deadline exceeded; Dex namespace="console-xzp4sq4"; Deployment/dex condition="Available" status="True" reason="MinimumReplicasAvailable"; Deployment/dex Signal: IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | unknown | 1 | 3.45%1 of 29 job runs affected | IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | Show trend detailsOct 12: 0 · Oct 13: 0 · Oct 14: 0 · Oct 15: 0 · Oct 16: 0 · Oct 17: 0 · Oct 18: 0 · Oct 19: 0 · Oct 20: 0 · Oct 21: 0 · Oct 22: 1 · Oct 23: 0 · Oct 24: 0 · Oct 25: 0 | none | |||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:194]: failed to set up Console login fixture for cluster "cluster-candidate-4-20-8dwfln"
Unexpected error:
<*fmt.wrapError | 0x22821099b6a0>:
console fixture namespace "console-xzp4sq4": configure Console ExternalAuth through RP: context deadline exceeded; Dex namespace="console-xzp4sq4"; Deployment/dex condition="Available" status="True" reason="MinimumReplicasAvailable"; Deployment/dex condition="Progressing" status="True" reason="NewReplicaSetAvailable"; Deployment/dex generation=1 observed=1 replicas=1 ready=1 available=1; Dex Pods=1; Dex ReplicaSets=1; Pod/dex-78f4dcc67d-swzhr condition="ContainersReady" status="True" reason=""; Pod/dex-78f4dcc67d-swzhr condition="Initialized" status="True" reason=""; Pod/dex-78f4dcc67d-swzhr condition="PodReadyToStartContainers" status="True" reason=""; Pod/dex-78f4dcc67d-swzhr condition="PodScheduled" status="True" reason=""; Pod/dex-78f4dcc67d-swzhr condition="Ready" status="True" reason=""; Pod/dex-78f4dcc67d-swzhr container="dex" ready=true restarts=0; Pod/dex-78f4dcc67d-swzhr phase="Running" reason=""
{
msg: "console fixture namespace \"console-xzp4sq4\": configure Console ExternalAuth through RP: context deadline exceeded; Dex namespace=\"console-xzp4sq4\"; Deployment/dex condition=\"Available\" status=\"True\" reason=\"MinimumReplicasAvailable\"; Deployment/dex condition=\"Progressing\" status=\"True\" reason=\"NewReplicaSetAvailable\"; Deployment/dex generation=1 observed=1 replicas=1 ready=1 available=1; Dex Pods=1; Dex ReplicaSets=1; Pod/dex-78f4dcc67d-swzhr condition=\"ContainersReady\" status=\"True\" reason=\"\"; Pod/dex-78f4dcc67d-swzhr condition=\"Initialized\" status=\"True\" reason=\"\"; Pod/dex-78f4dcc67d-swzhr condition=\"PodReadyToStartContainers\" status=\"True\" reason=\"\"; Pod/dex-78f4dcc67d-swzhr condition=\"PodScheduled\" status=\"True\" reason=\"\"; Pod/dex-78f4dcc67d-swzhr condition=\"Ready\" status=\"True\" reason=\"\"; Pod/dex-78f4dcc67d-swzhr container=\"dex\" ready=true restarts=0; Pod/dex-78f4dcc67d-swzhr phase=\"Running\" reason=\"\"",
err: <*fmt.wrapError | 0x228210e20900>{
msg: "configure Console ExternalAuth through RP: context deadline exceeded",
err: <context.deadlineExceededError>{},
},
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:194]: failed to set up Console login fixture for cluster "cluster-candidate-4-20-8dwfln"
Unexpected error:
<*fmt.wrapError | 0x22821099b6a0>:
console fixture namespace "console-xzp4sq4": configure Console ExternalAuth through RP: context deadline exceeded; Dex namespace="console-xzp4sq4"; Deployment/dex condition="Available" status="True" reason="MinimumReplicasAvailable"; Deployment/dex condition="Progressing" status="True" reason="NewReplicaSetAvailable"; Deployment/dex generation=1 observed=1 replicas=1 ready=1 available=1; Dex Pods=1; Dex ReplicaSets=1; Pod/dex-78f4dcc67d-swzhr condition="ContainersReady" status="True" reason=""; Pod/dex-78f4dcc67d-swzhr condition="Initialized" status="True" reason=""; Pod/dex-78f4dcc67d-swzhr condition="PodReadyToStartContainers" status="True" reason=""; Pod/dex-78f4dcc67d-swzhr condition="PodScheduled" status="True" reason=""; Pod/dex-78f4dcc67d-swzhr condition="Ready" status="True" reason=""; Pod/dex-78f4dcc67d-swzhr container="dex" ready=true restarts=0; Pod/dex-78f4dcc67d-swzhr phase="Running" reason=""
{
msg: "console fixture namespace \"console-xzp4sq4\": configure Console ExternalAuth through RP: context deadline exceeded; Dex namespace=\"console-xzp4sq4\"; Deployment/dex condition=\"Available\" status=\"True\" reason=\"MinimumReplicasAvailable\"; Deployment/dex condition=\"Progressing\" status=\"True\" reason=\"NewReplicaSetAvailable\"; Deployment/dex generation=1 observed=1 replicas=1 ready=1 available=1; Dex Pods=1; Dex ReplicaSets=1; Pod/dex-78f4dcc67d-swzhr condition=\"ContainersReady\" status=\"True\" reason=\"\"; Pod/dex-78f4dcc67d-swzhr condition=\"Initialized\" status=\"True\" reason=\"\"; Pod/dex-78f4dcc67d-swzhr condition=\"PodReadyToStartContainers\" status=\"True\" reason=\"\"; Pod/dex-78f4dcc67d-swzhr condition=\"PodScheduled\" status=\"True\" reason=\"\"; Pod/dex-78f4dcc67d-swzhr condition=\"Ready\" status=\"True\" reason=\"\"; Pod/dex-78f4dcc67d-swzhr container=\"dex\" ready=true restarts=0; Pod/dex-78f4dcc67d-swzhr phase=\"Running\" reason=\"\"",
err: <*fmt.wrapError | 0x228210e20900>{
msg: "configure Console ExternalAuth through RP: context deadline exceeded",
err: <context.deadlineExceededError>{},
},
}
occurredContributing tests (1)
Affected runs (1)
| |||||||||||||||||||||||||||||||||||||||||||||||||||||
console fixture namespace "<namespace>": configure Console ExternalAuth through RP: context deadline exceeded; Dex namespace="console-z2wqbfw"; Deployment/dex condition="Available...full failure pattern: console fixture namespace "<namespace>": configure Console ExternalAuth through RP: context deadline exceeded; Dex namespace="console-z2wqbfw"; Deployment/dex condition="Available" status="True" reason="MinimumReplicasAvailable"; Deployment/dex Signal: IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | unknown | 1 | 3.45%1 of 29 job runs affected | IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | Show trend detailsOct 12: 0 · Oct 13: 0 · Oct 14: 0 · Oct 15: 0 · Oct 16: 0 · Oct 17: 0 · Oct 18: 0 · Oct 19: 0 · Oct 20: 0 · Oct 21: 0 · Oct 22: 1 · Oct 23: 0 · Oct 24: 0 · Oct 25: 0 | none | |||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:194]: failed to set up Console login fixture for cluster "cluster-candidate-4-20-9js6f6"
Unexpected error:
<*fmt.wrapError | 0x10354e044ea0>:
console fixture namespace "console-z2wqbfw": configure Console ExternalAuth through RP: context deadline exceeded; Dex namespace="console-z2wqbfw"; Deployment/dex condition="Available" status="True" reason="MinimumReplicasAvailable"; Deployment/dex condition="Progressing" status="True" reason="NewReplicaSetAvailable"; Deployment/dex generation=1 observed=1 replicas=1 ready=1 available=1; Dex Pods=1; Dex ReplicaSets=1; Pod/dex-78f4dcc67d-6mptx condition="ContainersReady" status="True" reason=""; Pod/dex-78f4dcc67d-6mptx condition="Initialized" status="True" reason=""; Pod/dex-78f4dcc67d-6mptx condition="PodReadyToStartContainers" status="True" reason=""; Pod/dex-78f4dcc67d-6mptx condition="PodScheduled" status="True" reason=""; Pod/dex-78f4dcc67d-6mptx condition="Ready" status="True" reason=""; Pod/dex-78f4dcc67d-6mptx container="dex" ready=true restarts=0; Pod/dex-78f4dcc67d-6mptx phase="Running" reason=""
{
msg: "console fixture namespace \"console-z2wqbfw\": configure Console ExternalAuth through RP: context deadline exceeded; Dex namespace=\"console-z2wqbfw\"; Deployment/dex condition=\"Available\" status=\"True\" reason=\"MinimumReplicasAvailable\"; Deployment/dex condition=\"Progressing\" status=\"True\" reason=\"NewReplicaSetAvailable\"; Deployment/dex generation=1 observed=1 replicas=1 ready=1 available=1; Dex Pods=1; Dex ReplicaSets=1; Pod/dex-78f4dcc67d-6mptx condition=\"ContainersReady\" status=\"True\" reason=\"\"; Pod/dex-78f4dcc67d-6mptx condition=\"Initialized\" status=\"True\" reason=\"\"; Pod/dex-78f4dcc67d-6mptx condition=\"PodReadyToStartContainers\" status=\"True\" reason=\"\"; Pod/dex-78f4dcc67d-6mptx condition=\"PodScheduled\" status=\"True\" reason=\"\"; Pod/dex-78f4dcc67d-6mptx condition=\"Ready\" status=\"True\" reason=\"\"; Pod/dex-78f4dcc67d-6mptx container=\"dex\" ready=true restarts=0; Pod/dex-78f4dcc67d-6mptx phase=\"Running\" reason=\"\"",
err: <*fmt.wrapError | 0x10354ed78880>{
msg: "configure Console ExternalAuth through RP: context deadline exceeded",
err: <context.deadlineExceededError>{},
},
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:194]: failed to set up Console login fixture for cluster "cluster-candidate-4-20-9js6f6"
Unexpected error:
<*fmt.wrapError | 0x10354e044ea0>:
console fixture namespace "console-z2wqbfw": configure Console ExternalAuth through RP: context deadline exceeded; Dex namespace="console-z2wqbfw"; Deployment/dex condition="Available" status="True" reason="MinimumReplicasAvailable"; Deployment/dex condition="Progressing" status="True" reason="NewReplicaSetAvailable"; Deployment/dex generation=1 observed=1 replicas=1 ready=1 available=1; Dex Pods=1; Dex ReplicaSets=1; Pod/dex-78f4dcc67d-6mptx condition="ContainersReady" status="True" reason=""; Pod/dex-78f4dcc67d-6mptx condition="Initialized" status="True" reason=""; Pod/dex-78f4dcc67d-6mptx condition="PodReadyToStartContainers" status="True" reason=""; Pod/dex-78f4dcc67d-6mptx condition="PodScheduled" status="True" reason=""; Pod/dex-78f4dcc67d-6mptx condition="Ready" status="True" reason=""; Pod/dex-78f4dcc67d-6mptx container="dex" ready=true restarts=0; Pod/dex-78f4dcc67d-6mptx phase="Running" reason=""
{
msg: "console fixture namespace \"console-z2wqbfw\": configure Console ExternalAuth through RP: context deadline exceeded; Dex namespace=\"console-z2wqbfw\"; Deployment/dex condition=\"Available\" status=\"True\" reason=\"MinimumReplicasAvailable\"; Deployment/dex condition=\"Progressing\" status=\"True\" reason=\"NewReplicaSetAvailable\"; Deployment/dex generation=1 observed=1 replicas=1 ready=1 available=1; Dex Pods=1; Dex ReplicaSets=1; Pod/dex-78f4dcc67d-6mptx condition=\"ContainersReady\" status=\"True\" reason=\"\"; Pod/dex-78f4dcc67d-6mptx condition=\"Initialized\" status=\"True\" reason=\"\"; Pod/dex-78f4dcc67d-6mptx condition=\"PodReadyToStartContainers\" status=\"True\" reason=\"\"; Pod/dex-78f4dcc67d-6mptx condition=\"PodScheduled\" status=\"True\" reason=\"\"; Pod/dex-78f4dcc67d-6mptx condition=\"Ready\" status=\"True\" reason=\"\"; Pod/dex-78f4dcc67d-6mptx container=\"dex\" ready=true restarts=0; Pod/dex-78f4dcc67d-6mptx phase=\"Running\" reason=\"\"",
err: <*fmt.wrapError | 0x10354ed78880>{
msg: "configure Console ExternalAuth through RP: context deadline exceeded",
err: <context.deadlineExceededError>{},
},
}
occurredContributing tests (1)
Affected runs (1)
| |||||||||||||||||||||||||||||||||||||||||||||||||||||
timeout during GetAdminRESTConfigForHCPCluster while waiting for hcpcluster credsfull failure pattern: timeout during GetAdminRESTConfigForHCPCluster while waiting for hcpcluster creds Signal: IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | unknown | 1 | 3.45%1 of 29 job runs affected | IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | Show trend detailsOct 12: 0 · Oct 13: 0 · Oct 14: 0 · Oct 15: 0 · Oct 16: 0 · Oct 17: 0 · Oct 18: 0 · Oct 19: 0 · Oct 20: 0 · Oct 21: 0 · Oct 22: 0 · Oct 23: 0 · Oct 24: 0 · Oct 25: 1 | none | |||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/e2e/cluster_container_registry.go:657]: failed to get admin REST config
Unexpected error:
<*fmt.wrapErrors | 0xa204c386ab0>:
failed waiting for hcpCluster="acr-pull" in resourcegroup="acr-pull-zhz5sxs2r4gt" to finish getting creds, caused by: timeout '2.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster20261001 for cluster acr-pull in resource group acr-pull-zhz5sxs2r4gt, error: context deadline exceeded
{
msg: "failed waiting for hcpCluster=\"acr-pull\" in resourcegroup=\"acr-pull-zhz5sxs2r4gt\" to finish getting creds, caused by: timeout '2.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster20261001 for cluster acr-pull in resource group acr-pull-zhz5sxs2r4gt, error: context deadline exceeded",
errs: [
<*errors.errorString | 0xa204b336550>{
s: "timeout '2.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster20261001 for cluster acr-pull in resource group acr-pull-zhz5sxs2r4gt",
},
<context.deadlineExceededError>{},
],
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_container_registry.go:657]: failed to get admin REST config
Unexpected error:
<*fmt.wrapErrors | 0xa204c386ab0>:
failed waiting for hcpCluster="acr-pull" in resourcegroup="acr-pull-zhz5sxs2r4gt" to finish getting creds, caused by: timeout '2.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster20261001 for cluster acr-pull in resource group acr-pull-zhz5sxs2r4gt, error: context deadline exceeded
{
msg: "failed waiting for hcpCluster=\"acr-pull\" in resourcegroup=\"acr-pull-zhz5sxs2r4gt\" to finish getting creds, caused by: timeout '2.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster20261001 for cluster acr-pull in resource group acr-pull-zhz5sxs2r4gt, error: context deadline exceeded",
errs: [
<*errors.errorString | 0xa204b336550>{
s: "timeout '2.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster20261001 for cluster acr-pull in resource group acr-pull-zhz5sxs2r4gt",
},
<context.deadlineExceededError>{},
],
}
occurredContributing tests (1)
Affected runs (1)
| |||||||||||||||||||||||||||||||||||||||||||||||||||||
Environment: STG
Daily Run Outcomes
No tests below 95.00% in this window with at least 10 runs.
Up to 50 failure patterns are loaded (minimum 1.00% impact), with 10 shown by default. Click column headers to re-sort.
Jump to the full Failure Patterns view for this window
| Failure patternThe canonical description of a recurring CI failure, extracted and normalized from raw logs. | Failed atThe stage of the job run where this failure occurred: 'provision' (environment setup, DEV only), 'e2e' (test suite execution), or 'other' (CI infrastructure issues that did not produce a failure pattern). | Number of distinct job runs where this failure pattern was detected. | Percentage of all job runs in this environment affected by this failure pattern during the selected window. | Classification of this failure pattern: Regression (likely caused by a specific PR), Flake (intermittent failure spread across days), Noise (low-quality or generic pattern), or Indeterminate. | TrendShows daily activity for this failure pattern in a trailing window anchored to the selected end date. The sparkline covers at least 7 days and at most 14 days, depending on the current window size. | Also inOther environments where the same failure pattern was also detected during the selected window. | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
ERROR CODE: InvalidRequestContent; detail message Invalid value: "4.21": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when c...full failure pattern: ERROR CODE: InvalidRequestContent; detail message Invalid value: "4.21": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is "nightly"; provider Microsoft.RedHatOpenShift Signal: FlakeSignal: Flake — present in 4 prior week(s); spread across 6 days | e2e | 6 | 22.22%6 of 27 job runs affected | FlakeSignal: Flake — present in 4 prior week(s); spread across 6 days | Show trend detailsOct 12: 4 · Oct 13: 5 · Oct 14: 5 · Oct 15: 5 · Oct 16: 5 · Oct 17: 5 · Oct 18: 0 · Oct 19: 0 · Oct 20: 0 · Oct 21: 0 · Oct 22: 0 · Oct 23: 0 · Oct 24: 0 · Oct 25: 0 | PROD | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (3)fail [github.com/Azure/ARO-HCP/test/e2e/idms_lifecycle.go:175]: failed to update cluster with second ImageDigestMirror set
Unexpected error:
<*exported.ResponseError | 0x4fc6ce77170>:
PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/idms-7qq88b66sc2l/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/idms-e2e-hcp-cluster
--------------------------------------------------------------------------------
RESPONSE 400: 400 Bad Request
ERROR CODE: InvalidRequestContent
--------------------------------------------------------------------------------
{
"error": {
"code": "InvalidRequestContent",
"message": "Invalid value: \"4.21\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"",
"target": "properties.version.id"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/idms-7qq88b66sc2l/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/idms-e2e-hcp-cluster
correlation ID: 09054a39-11cf-4b09-a485-31e6d3a5caaf
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/idms_lifecycle.go:175]: failed to update cluster with second ImageDigestMirror set
Unexpected error:
<*exported.ResponseError | 0x4fc6ce77170>:
PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/idms-7qq88b66sc2l/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/idms-e2e-hcp-cluster
--------------------------------------------------------------------------------
RESPONSE 400: 400 Bad Request
ERROR CODE: InvalidRequestContent
--------------------------------------------------------------------------------
{
"error": {
"code": "InvalidRequestContent",
"message": "Invalid value: \"4.21\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"",
"target": "properties.version.id"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/idms-7qq88b66sc2l/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/idms-e2e-hcp-cluster
correlation ID: 09054a39-11cf-4b09-a485-31e6d3a5caaf
client request ID: unavailable
occurredfail [github.com/Azure/ARO-HCP/test/e2e/cluster_autoscaling.go:173]: failed to update cluster "autoscaling-hcp-cluster" autoscaling configuration
Unexpected error:
<*exported.ResponseError | 0x2195bd58cde0>:
PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/autoscaling-cluster-58j2ssgd9lfs/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/autoscaling-hcp-cluster
--------------------------------------------------------------------------------
RESPONSE 400: 400 Bad Request
ERROR CODE: InvalidRequestContent
--------------------------------------------------------------------------------
{
"error": {
"code": "InvalidRequestContent",
"message": "Invalid value: \"4.21\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"",
"target": "properties.version.id"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/autoscaling-cluster-58j2ssgd9lfs/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/autoscaling-hcp-cluster
correlation ID: 4e9f2a97-920c-4952-a752-7a5aa59c6506
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_autoscaling.go:173]: failed to update cluster "autoscaling-hcp-cluster" autoscaling configuration
Unexpected error:
<*exported.ResponseError | 0x2195bd58cde0>:
PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/autoscaling-cluster-58j2ssgd9lfs/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/autoscaling-hcp-cluster
--------------------------------------------------------------------------------
RESPONSE 400: 400 Bad Request
ERROR CODE: InvalidRequestContent
--------------------------------------------------------------------------------
{
"error": {
"code": "InvalidRequestContent",
"message": "Invalid value: \"4.21\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"",
"target": "properties.version.id"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/autoscaling-cluster-58j2ssgd9lfs/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/autoscaling-hcp-cluster
correlation ID: 4e9f2a97-920c-4952-a752-7a5aa59c6506
client request ID: unavailable
occurredfail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:408]: failed to begin updating cluster "cidr-connectivity-test" authorized CIDRs
Unexpected error:
<*exported.ResponseError | 0x8e1dd31d290>:
PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/e2e-cidr-connectivity-mmllxt6jr4gv/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cidr-connectivity-test
--------------------------------------------------------------------------------
RESPONSE 400: 400 Bad Request
ERROR CODE: InvalidRequestContent
--------------------------------------------------------------------------------
{
"error": {
"code": "InvalidRequestContent",
"message": "Invalid value: \"4.21\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"",
"target": "properties.version.id"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/e2e-cidr-connectivity-mmllxt6jr4gv/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cidr-connectivity-test
correlation ID: bf0c2965-cddd-42e6-9d62-d3767563c18a
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:408]: failed to begin updating cluster "cidr-connectivity-test" authorized CIDRs
Unexpected error:
<*exported.ResponseError | 0x8e1dd31d290>:
PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/e2e-cidr-connectivity-mmllxt6jr4gv/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cidr-connectivity-test
--------------------------------------------------------------------------------
RESPONSE 400: 400 Bad Request
ERROR CODE: InvalidRequestContent
--------------------------------------------------------------------------------
{
"error": {
"code": "InvalidRequestContent",
"message": "Invalid value: \"4.21\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"",
"target": "properties.version.id"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/e2e-cidr-connectivity-mmllxt6jr4gv/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cidr-connectivity-test
correlation ID: bf0c2965-cddd-42e6-9d62-d3767563c18a
client request ID: unavailable
occurredContributing tests (5)
Affected runs (6)
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; provider Microsoft...full failure pattern: ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; provider Microsoft.RedHatOpenShift Signal: FlakeSignal: Flake — present in 2 prior week(s); spread across 3 days | e2e | 3 | 11.11%3 of 27 job runs affected | FlakeSignal: Flake — present in 2 prior week(s); spread across 3 days | Show trend detailsOct 12: 5 · Oct 13: 1 · Oct 14: 3 · Oct 15: 0 · Oct 16: 0 · Oct 17: 0 · Oct 18: 0 · Oct 19: 0 · Oct 20: 0 · Oct 21: 0 · Oct 22: 0 · Oct 23: 0 · Oct 24: 0 · Oct 25: 0 | PROD | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (3)fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:133]: failed to create HCP cluster "cp-ystream-upgrade-5-0-snb95l" at install version 4.22.0-0.nightly-multi-2026-10-14-193612
Unexpected error:
<*fmt.wrapError | 0x1f800f0eba0>:
failed to create HCP cluster cp-ystream-upgrade-5-0-snb95l: failed waiting for cluster="cp-ystream-upgrade-5-0-snb95l" in resourcegroup="rg-cp-ystream-upgrade-5-0-9pm7qw6t4kxg" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/5b5b73d5-ad43-4248-a379-3589ba793716
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/5b5b73d5-ad43-4248-a379-3589ba793716",
"name": "5b5b73d5-ad43-4248-a379-3589ba793716",
"status": "Failed",
"startTime": "2026-10-14T23:28:48.207254546Z",
"endTime": "2026-10-14T23:57:58.000803929Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/5b5b73d5-ad43-4248-a379-3589ba793716
correlation ID: 8af90513-a16b-4576-aebf-be0202178446
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:133]: failed to create HCP cluster "cp-ystream-upgrade-5-0-snb95l" at install version 4.22.0-0.nightly-multi-2026-10-14-193612
Unexpected error:
<*fmt.wrapError | 0x1f800f0eba0>:
failed to create HCP cluster cp-ystream-upgrade-5-0-snb95l: failed waiting for cluster="cp-ystream-upgrade-5-0-snb95l" in resourcegroup="rg-cp-ystream-upgrade-5-0-9pm7qw6t4kxg" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/5b5b73d5-ad43-4248-a379-3589ba793716
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/5b5b73d5-ad43-4248-a379-3589ba793716",
"name": "5b5b73d5-ad43-4248-a379-3589ba793716",
"status": "Failed",
"startTime": "2026-10-14T23:28:48.207254546Z",
"endTime": "2026-10-14T23:57:58.000803929Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/5b5b73d5-ad43-4248-a379-3589ba793716
correlation ID: 8af90513-a16b-4576-aebf-be0202178446
client request ID: unavailable
occurredfail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-nightly-4-22-4bfb22rwmqzl/cluster-nightly-4-22-8zxbz4 should provision
Unexpected error:
<*fmt.wrapError | 0x363444c03360>:
failed to create HCP cluster cluster-nightly-4-22-8zxbz4: failed waiting for cluster="cluster-nightly-4-22-8zxbz4" in resourcegroup="rg-nightly-4-22-4bfb22rwmqzl" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/4e7e5bf8-d998-4cf9-b7a2-a75ca9ab02f3
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/4e7e5bf8-d998-4cf9-b7a2-a75ca9ab02f3",
"name": "4e7e5bf8-d998-4cf9-b7a2-a75ca9ab02f3",
"status": "Failed",
"startTime": "2026-10-14T23:25:15.190269818Z",
"endTime": "2026-10-14T23:54:17.885160466Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/4e7e5bf8-d998-4cf9-b7a2-a75ca9ab02f3
correlation ID: eee8c816-0fc0-4cfa-8c09-a91c4755ee62
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-nightly-4-22-4bfb22rwmqzl/cluster-nightly-4-22-8zxbz4 should provision
Unexpected error:
<*fmt.wrapError | 0x363444c03360>:
failed to create HCP cluster cluster-nightly-4-22-8zxbz4: failed waiting for cluster="cluster-nightly-4-22-8zxbz4" in resourcegroup="rg-nightly-4-22-4bfb22rwmqzl" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/4e7e5bf8-d998-4cf9-b7a2-a75ca9ab02f3
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/4e7e5bf8-d998-4cf9-b7a2-a75ca9ab02f3",
"name": "4e7e5bf8-d998-4cf9-b7a2-a75ca9ab02f3",
"status": "Failed",
"startTime": "2026-10-14T23:25:15.190269818Z",
"endTime": "2026-10-14T23:54:17.885160466Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/4e7e5bf8-d998-4cf9-b7a2-a75ca9ab02f3
correlation ID: eee8c816-0fc0-4cfa-8c09-a91c4755ee62
client request ID: unavailable
occurredfail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-nightly-4-21-7c4xdncsltnp/cluster-nightly-4-21-7xdrt5 should provision
Unexpected error:
<*fmt.wrapError | 0xc38fe936560>:
failed to create HCP cluster cluster-nightly-4-21-7xdrt5: failed waiting for cluster="cluster-nightly-4-21-7xdrt5" in resourcegroup="rg-nightly-4-21-7c4xdncsltnp" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/bc06cf13-34dc-4e88-8b6c-774417a9e32b
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/bc06cf13-34dc-4e88-8b6c-774417a9e32b",
"name": "bc06cf13-34dc-4e88-8b6c-774417a9e32b",
"status": "Failed",
"startTime": "2026-10-14T23:25:05.12965498Z",
"endTime": "2026-10-14T23:54:07.890200924Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/bc06cf13-34dc-4e88-8b6c-774417a9e32b
correlation ID: 0f756d77-a36e-4289-82ab-c5ef8b73feb4
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-nightly-4-21-7c4xdncsltnp/cluster-nightly-4-21-7xdrt5 should provision
Unexpected error:
<*fmt.wrapError | 0xc38fe936560>:
failed to create HCP cluster cluster-nightly-4-21-7xdrt5: failed waiting for cluster="cluster-nightly-4-21-7xdrt5" in resourcegroup="rg-nightly-4-21-7c4xdncsltnp" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/bc06cf13-34dc-4e88-8b6c-774417a9e32b
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/bc06cf13-34dc-4e88-8b6c-774417a9e32b",
"name": "bc06cf13-34dc-4e88-8b6c-774417a9e32b",
"status": "Failed",
"startTime": "2026-10-14T23:25:05.12965498Z",
"endTime": "2026-10-14T23:54:07.890200924Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/bc06cf13-34dc-4e88-8b6c-774417a9e32b
correlation ID: 0f756d77-a36e-4289-82ab-c5ef8b73feb4
client request ID: unavailable
occurredContributing tests (7)
Affected runs (3)
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
VerifyDefaultIngressServingCertificateSecretWiring timed out after 10m0s: IngressController/default references &LocalObjectReference{Name:cluster-ingress-cert,}, expected secret "...full failure pattern: VerifyDefaultIngressServingCertificateSecretWiring timed out after 10m0s: IngressController/default references &LocalObjectReference{Name:cluster-ingress-cert,}, expected secret "default-ingress-cert"Signal: NoiseSignal: Noise — struct/object fragment — New failure pattern — no prior history | e2e | 2 | 7.41%2 of 27 job runs affected | NoiseSignal: Noise — struct/object fragment — New failure pattern — no prior history | Show trend detailsOct 12: 0 · Oct 13: 0 · Oct 14: 0 · Oct 15: 0 · Oct 16: 0 · Oct 17: 0 · Oct 18: 0 · Oct 19: 0 · Oct 20: 0 · Oct 21: 0 · Oct 22: 2 · Oct 23: 0 · Oct 24: 1 · Oct 25: 0 | DEV, PROD | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (3)fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:220]: default-ingress-cert must exist and be referenced by IngressController/default for cluster "cluster-candidate-4-20-h45rtr"
Expected success, but got an error:
<*fmt.wrapError | 0x31f100c0b6e0>:
VerifyDefaultIngressServingCertificateSecretWiring timed out after 10m0s: IngressController/default references &LocalObjectReference{Name:cluster-ingress-cert,}, expected secret "default-ingress-cert"
{
msg: "VerifyDefaultIngressServingCertificateSecretWiring timed out after 10m0s: IngressController/default references &LocalObjectReference{Name:cluster-ingress-cert,}, expected secret \"default-ingress-cert\"",
err: <*verifiers.pollObservationError | 0x31f100c0b6c0>{
observation: <*errors.errorString | 0x31f1027391b0>{
s: "IngressController/default references &LocalObjectReference{Name:cluster-ingress-cert,}, expected secret \"default-ingress-cert\"",
},
interruption: <*fmt.wrapError | 0x31f100c0b6a0>{
msg: "getting default ingress serving certificate secret openshift-ingress/default-ingress-cert: context deadline exceeded",
err: <context.deadlineExceededError>{},
},
},
}
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:220]: default-ingress-cert must exist and be referenced by IngressController/default for cluster "cluster-candidate-4-20-h45rtr"
Expected success, but got an error:
<*fmt.wrapError | 0x31f100c0b6e0>:
VerifyDefaultIngressServingCertificateSecretWiring timed out after 10m0s: IngressController/default references &LocalObjectReference{Name:cluster-ingress-cert,}, expected secret "default-ingress-cert"
{
msg: "VerifyDefaultIngressServingCertificateSecretWiring timed out after 10m0s: IngressController/default references &LocalObjectReference{Name:cluster-ingress-cert,}, expected secret \"default-ingress-cert\"",
err: <*verifiers.pollObservationError | 0x31f100c0b6c0>{
observation: <*errors.errorString | 0x31f1027391b0>{
s: "IngressController/default references &LocalObjectReference{Name:cluster-ingress-cert,}, expected secret \"default-ingress-cert\"",
},
interruption: <*fmt.wrapError | 0x31f100c0b6a0>{
msg: "getting default ingress serving certificate secret openshift-ingress/default-ingress-cert: context deadline exceeded",
err: <context.deadlineExceededError>{},
},
},
}fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:216]: default-ingress-cert must exist and be referenced by IngressController/default for cluster "cluster-nightly-4-21-w7wgsb"
Expected success, but got an error:
<*fmt.wrapError | 0x80bfcd600c0>:
VerifyDefaultIngressServingCertificateSecretWiring timed out after 10m0s: IngressController/default references &LocalObjectReference{Name:cluster-ingress-cert,}, expected secret "default-ingress-cert"
{
msg: "VerifyDefaultIngressServingCertificateSecretWiring timed out after 10m0s: IngressController/default references &LocalObjectReference{Name:cluster-ingress-cert,}, expected secret \"default-ingress-cert\"",
err: <*verifiers.pollObservationError | 0x80bfcd60080>{
observation: <*errors.errorString | 0x80bfe902e80>{
s: "IngressController/default references &LocalObjectReference{Name:cluster-ingress-cert,}, expected secret \"default-ingress-cert\"",
},
interruption: <*fmt.wrapError | 0x80bfcd60060>{
msg: "getting default ingress serving certificate secret openshift-ingress/default-ingress-cert: context deadline exceeded",
err: <context.deadlineExceededError>{},
},
},
}
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:216]: default-ingress-cert must exist and be referenced by IngressController/default for cluster "cluster-nightly-4-21-w7wgsb"
Expected success, but got an error:
<*fmt.wrapError | 0x80bfcd600c0>:
VerifyDefaultIngressServingCertificateSecretWiring timed out after 10m0s: IngressController/default references &LocalObjectReference{Name:cluster-ingress-cert,}, expected secret "default-ingress-cert"
{
msg: "VerifyDefaultIngressServingCertificateSecretWiring timed out after 10m0s: IngressController/default references &LocalObjectReference{Name:cluster-ingress-cert,}, expected secret \"default-ingress-cert\"",
err: <*verifiers.pollObservationError | 0x80bfcd60080>{
observation: <*errors.errorString | 0x80bfe902e80>{
s: "IngressController/default references &LocalObjectReference{Name:cluster-ingress-cert,}, expected secret \"default-ingress-cert\"",
},
interruption: <*fmt.wrapError | 0x80bfcd60060>{
msg: "getting default ingress serving certificate secret openshift-ingress/default-ingress-cert: context deadline exceeded",
err: <context.deadlineExceededError>{},
},
},
}fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:216]: default-ingress-cert must exist and be referenced by IngressController/default for cluster "cluster-nightly-4-22-45dcr7"
Expected success, but got an error:
<*fmt.wrapError | 0x3387fb166020>:
VerifyDefaultIngressServingCertificateSecretWiring timed out after 10m0s: IngressController/default references &LocalObjectReference{Name:cluster-ingress-cert,}, expected secret "default-ingress-cert"
{
msg: "VerifyDefaultIngressServingCertificateSecretWiring timed out after 10m0s: IngressController/default references &LocalObjectReference{Name:cluster-ingress-cert,}, expected secret \"default-ingress-cert\"",
err: <*errors.errorString | 0x3387fcff27f0>{
s: "IngressController/default references &LocalObjectReference{Name:cluster-ingress-cert,}, expected secret \"default-ingress-cert\"",
},
}
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:216]: default-ingress-cert must exist and be referenced by IngressController/default for cluster "cluster-nightly-4-22-45dcr7"
Expected success, but got an error:
<*fmt.wrapError | 0x3387fb166020>:
VerifyDefaultIngressServingCertificateSecretWiring timed out after 10m0s: IngressController/default references &LocalObjectReference{Name:cluster-ingress-cert,}, expected secret "default-ingress-cert"
{
msg: "VerifyDefaultIngressServingCertificateSecretWiring timed out after 10m0s: IngressController/default references &LocalObjectReference{Name:cluster-ingress-cert,}, expected secret \"default-ingress-cert\"",
err: <*errors.errorString | 0x3387fcff27f0>{
s: "IngressController/default references &LocalObjectReference{Name:cluster-ingress-cert,}, expected secret \"default-ingress-cert\"",
},
}Contributing tests (3)
Affected runs (2)
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
ACM ingress secret openshift-ingress/cluster-ingress-cert exists; expected ACM certificate delivery to be suppressedfull failure pattern: ACM ingress secret openshift-ingress/cluster-ingress-cert exists; expected ACM certificate delivery to be suppressed Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 6 day(s) | e2e | 8 | 29.63%8 of 27 job runs affected | IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 6 day(s) | Show trend detailsOct 12: 0 · Oct 13: 0 · Oct 14: 0 · Oct 15: 0 · Oct 16: 1 · Oct 17: 2 · Oct 18: 1 · Oct 19: 0 · Oct 20: 0 · Oct 21: 0 · Oct 22: 5 · Oct 23: 9 · Oct 24: 5 · Oct 25: 0 | DEV, PROD | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (3)fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:511]: ACM must not create its ingress certificate secret
Expected success, but got an error:
<*errors.errorString | 0x3fdcccb98ca0>:
ACM ingress secret openshift-ingress/cluster-ingress-cert exists; expected ACM certificate delivery to be suppressed
{
s: "ACM ingress secret openshift-ingress/cluster-ingress-cert exists; expected ACM certificate delivery to be suppressed",
}
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:511]: ACM must not create its ingress certificate secret
Expected success, but got an error:
<*errors.errorString | 0x3fdcccb98ca0>:
ACM ingress secret openshift-ingress/cluster-ingress-cert exists; expected ACM certificate delivery to be suppressed
{
s: "ACM ingress secret openshift-ingress/cluster-ingress-cert exists; expected ACM certificate delivery to be suppressed",
}fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:511]: ACM must not create its ingress certificate secret
Expected success, but got an error:
<*errors.errorString | 0x3b9be2369340>:
ACM ingress secret openshift-ingress/cluster-ingress-cert exists; expected ACM certificate delivery to be suppressed
{
s: "ACM ingress secret openshift-ingress/cluster-ingress-cert exists; expected ACM certificate delivery to be suppressed",
}
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:511]: ACM must not create its ingress certificate secret
Expected success, but got an error:
<*errors.errorString | 0x3b9be2369340>:
ACM ingress secret openshift-ingress/cluster-ingress-cert exists; expected ACM certificate delivery to be suppressed
{
s: "ACM ingress secret openshift-ingress/cluster-ingress-cert exists; expected ACM certificate delivery to be suppressed",
}fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:511]: ACM must not create its ingress certificate secret
Expected success, but got an error:
<*errors.errorString | 0x300e1e583d00>:
ACM ingress secret openshift-ingress/cluster-ingress-cert exists; expected ACM certificate delivery to be suppressed
{
s: "ACM ingress secret openshift-ingress/cluster-ingress-cert exists; expected ACM certificate delivery to be suppressed",
}
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:511]: ACM must not create its ingress certificate secret
Expected success, but got an error:
<*errors.errorString | 0x300e1e583d00>:
ACM ingress secret openshift-ingress/cluster-ingress-cert exists; expected ACM certificate delivery to be suppressed
{
s: "ACM ingress secret openshift-ingress/cluster-ingress-cert exists; expected ACM certificate delivery to be suppressed",
}Contributing tests (12)
Affected runs (8)
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
VerifyDefaultIngressServingCertificateSecretWiring timed out after 10m0full failure pattern: VerifyDefaultIngressServingCertificateSecretWiring timed out after 10m0 Signal: IndeterminateSignal: Indeterminate — no prior history; active 3 day(s) — New failure pattern — no prior history | e2e | 4 | 14.81%4 of 27 job runs affected | IndeterminateSignal: Indeterminate — no prior history; active 3 day(s) — New failure pattern — no prior history | Show trend detailsOct 12: 0 · Oct 13: 0 · Oct 14: 0 · Oct 15: 0 · Oct 16: 0 · Oct 17: 0 · Oct 18: 0 · Oct 19: 0 · Oct 20: 0 · Oct 21: 0 · Oct 22: 3 · Oct 23: 10 · Oct 24: 4 · Oct 25: 0 | DEV, PROD | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (3)fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:220]: default-ingress-cert must exist and be referenced by IngressController/default for cluster "cluster-candidate-5-0-nbp8zs"
Expected success, but got an error:
<*fmt.wrapError | 0x12807917cda0>:
VerifyDefaultIngressServingCertificateSecretWiring timed out after 10m0.001s: IngressController/default references &LocalObjectReference{Name:cluster-ingress-cert,}, expected secret "default-ingress-cert"
{
msg: "VerifyDefaultIngressServingCertificateSecretWiring timed out after 10m0.001s: IngressController/default references &LocalObjectReference{Name:cluster-ingress-cert,}, expected secret \"default-ingress-cert\"",
err: <*verifiers.pollObservationError | 0x12807917cd80>{
observation: <*errors.errorString | 0x128078c4fdb0>{
s: "IngressController/default references &LocalObjectReference{Name:cluster-ingress-cert,}, expected secret \"default-ingress-cert\"",
},
interruption: <*fmt.wrapError | 0x12807917cd40>{
msg: "getting default ingress serving certificate secret openshift-ingress/default-ingress-cert: context deadline exceeded",
err: <context.deadlineExceededError>{},
},
},
}
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:220]: default-ingress-cert must exist and be referenced by IngressController/default for cluster "cluster-candidate-5-0-nbp8zs"
Expected success, but got an error:
<*fmt.wrapError | 0x12807917cda0>:
VerifyDefaultIngressServingCertificateSecretWiring timed out after 10m0.001s: IngressController/default references &LocalObjectReference{Name:cluster-ingress-cert,}, expected secret "default-ingress-cert"
{
msg: "VerifyDefaultIngressServingCertificateSecretWiring timed out after 10m0.001s: IngressController/default references &LocalObjectReference{Name:cluster-ingress-cert,}, expected secret \"default-ingress-cert\"",
err: <*verifiers.pollObservationError | 0x12807917cd80>{
observation: <*errors.errorString | 0x128078c4fdb0>{
s: "IngressController/default references &LocalObjectReference{Name:cluster-ingress-cert,}, expected secret \"default-ingress-cert\"",
},
interruption: <*fmt.wrapError | 0x12807917cd40>{
msg: "getting default ingress serving certificate secret openshift-ingress/default-ingress-cert: context deadline exceeded",
err: <context.deadlineExceededError>{},
},
},
}fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:220]: default-ingress-cert must exist and be referenced by IngressController/default for cluster "cluster-candidate-4-21-tw5j7s"
Expected success, but got an error:
<*fmt.wrapError | 0x23a418cdb040>:
VerifyDefaultIngressServingCertificateSecretWiring timed out after 10m0.001s: IngressController/default references &LocalObjectReference{Name:cluster-ingress-cert,}, expected secret "default-ingress-cert"
{
msg: "VerifyDefaultIngressServingCertificateSecretWiring timed out after 10m0.001s: IngressController/default references &LocalObjectReference{Name:cluster-ingress-cert,}, expected secret \"default-ingress-cert\"",
err: <*errors.errorString | 0x23a41899b9c0>{
s: "IngressController/default references &LocalObjectReference{Name:cluster-ingress-cert,}, expected secret \"default-ingress-cert\"",
},
}
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:220]: default-ingress-cert must exist and be referenced by IngressController/default for cluster "cluster-candidate-4-21-tw5j7s"
Expected success, but got an error:
<*fmt.wrapError | 0x23a418cdb040>:
VerifyDefaultIngressServingCertificateSecretWiring timed out after 10m0.001s: IngressController/default references &LocalObjectReference{Name:cluster-ingress-cert,}, expected secret "default-ingress-cert"
{
msg: "VerifyDefaultIngressServingCertificateSecretWiring timed out after 10m0.001s: IngressController/default references &LocalObjectReference{Name:cluster-ingress-cert,}, expected secret \"default-ingress-cert\"",
err: <*errors.errorString | 0x23a41899b9c0>{
s: "IngressController/default references &LocalObjectReference{Name:cluster-ingress-cert,}, expected secret \"default-ingress-cert\"",
},
}fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:220]: default-ingress-cert must exist and be referenced by IngressController/default for cluster "cluster-candidate-4-22-kd6grh"
Expected success, but got an error:
<*fmt.wrapError | 0x3f6c5b4b0120>:
VerifyDefaultIngressServingCertificateSecretWiring timed out after 10m0.003s: IngressController/default references &LocalObjectReference{Name:cluster-ingress-cert,}, expected secret "default-ingress-cert"
{
msg: "VerifyDefaultIngressServingCertificateSecretWiring timed out after 10m0.003s: IngressController/default references &LocalObjectReference{Name:cluster-ingress-cert,}, expected secret \"default-ingress-cert\"",
err: <*verifiers.pollObservationError | 0x3f6c5b4b0100>{
observation: <*errors.errorString | 0x3f6c5c3c97b0>{
s: "IngressController/default references &LocalObjectReference{Name:cluster-ingress-cert,}, expected secret \"default-ingress-cert\"",
},
interruption: <*fmt.wrapError | 0x3f6c5b4b00a0>{
msg: "getting default ingress serving certificate secret openshift-ingress/default-ingress-cert: context deadline exceeded",
err: <context.deadlineExceededError>{},
},
},
}
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:220]: default-ingress-cert must exist and be referenced by IngressController/default for cluster "cluster-candidate-4-22-kd6grh"
Expected success, but got an error:
<*fmt.wrapError | 0x3f6c5b4b0120>:
VerifyDefaultIngressServingCertificateSecretWiring timed out after 10m0.003s: IngressController/default references &LocalObjectReference{Name:cluster-ingress-cert,}, expected secret "default-ingress-cert"
{
msg: "VerifyDefaultIngressServingCertificateSecretWiring timed out after 10m0.003s: IngressController/default references &LocalObjectReference{Name:cluster-ingress-cert,}, expected secret \"default-ingress-cert\"",
err: <*verifiers.pollObservationError | 0x3f6c5b4b0100>{
observation: <*errors.errorString | 0x3f6c5c3c97b0>{
s: "IngressController/default references &LocalObjectReference{Name:cluster-ingress-cert,}, expected secret \"default-ingress-cert\"",
},
interruption: <*fmt.wrapError | 0x3f6c5b4b00a0>{
msg: "getting default ingress serving certificate secret openshift-ingress/default-ingress-cert: context deadline exceeded",
err: <context.deadlineExceededError>{},
},
},
}Contributing tests (10)
Affected runs (4)
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
VerifyHyperShiftIngressCertificate timed out after 10m0full failure pattern: VerifyHyperShiftIngressCertificate timed out after 10m0 Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 3 day(s) | e2e | 4 | 14.81%4 of 27 job runs affected | IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 3 day(s) | Show trend detailsOct 12: 0 · Oct 13: 0 · Oct 14: 0 · Oct 15: 0 · Oct 16: 1 · Oct 17: 2 · Oct 18: 1 · Oct 19: 0 · Oct 20: 0 · Oct 21: 0 · Oct 22: 0 · Oct 23: 0 · Oct 24: 0 · Oct 25: 0 | DEV, PROD | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (3)fail [github.com/Azure/ARO-HCP/test/e2e/default_cluster_ingress_cert.go:165]: HyperShift's ingress certificate secret must exist and be referenced by IngressController/default
Expected success, but got an error:
<*fmt.wrapError | 0x1526b2d700a0>:
VerifyHyperShiftIngressCertificate timed out after 10m0.005s: getting HyperShift ingress secret openshift-ingress/default-ingress-cert: client rate limiter Wait returned an error: context deadline exceeded
{
msg: "VerifyHyperShiftIngressCertificate timed out after 10m0.005s: getting HyperShift ingress secret openshift-ingress/default-ingress-cert: client rate limiter Wait returned an error: context deadline exceeded",
err: <*fmt.wrapError | 0x1526b2d70080>{
msg: "getting HyperShift ingress secret openshift-ingress/default-ingress-cert: client rate limiter Wait returned an error: context deadline exceeded",
err: <*fmt.wrapError | 0x1526b2d70060>{
msg: "client rate limiter Wait returned an error: context deadline exceeded",
err: <context.deadlineExceededError>{},
},
},
}
fail [github.com/Azure/ARO-HCP/test/e2e/default_cluster_ingress_cert.go:165]: HyperShift's ingress certificate secret must exist and be referenced by IngressController/default
Expected success, but got an error:
<*fmt.wrapError | 0x1526b2d700a0>:
VerifyHyperShiftIngressCertificate timed out after 10m0.005s: getting HyperShift ingress secret openshift-ingress/default-ingress-cert: client rate limiter Wait returned an error: context deadline exceeded
{
msg: "VerifyHyperShiftIngressCertificate timed out after 10m0.005s: getting HyperShift ingress secret openshift-ingress/default-ingress-cert: client rate limiter Wait returned an error: context deadline exceeded",
err: <*fmt.wrapError | 0x1526b2d70080>{
msg: "getting HyperShift ingress secret openshift-ingress/default-ingress-cert: client rate limiter Wait returned an error: context deadline exceeded",
err: <*fmt.wrapError | 0x1526b2d70060>{
msg: "client rate limiter Wait returned an error: context deadline exceeded",
err: <context.deadlineExceededError>{},
},
},
}fail [github.com/Azure/ARO-HCP/test/e2e/default_cluster_ingress_cert.go:165]: HyperShift's ingress certificate secret must exist and be referenced by IngressController/default
Expected success, but got an error:
<*fmt.wrapError | 0x21f4804320>:
VerifyHyperShiftIngressCertificate timed out after 10m0.006s: getting HyperShift ingress secret openshift-ingress/default-ingress-cert: client rate limiter Wait returned an error: context deadline exceeded
{
msg: "VerifyHyperShiftIngressCertificate timed out after 10m0.006s: getting HyperShift ingress secret openshift-ingress/default-ingress-cert: client rate limiter Wait returned an error: context deadline exceeded",
err: <*fmt.wrapError | 0x21f48042e0>{
msg: "getting HyperShift ingress secret openshift-ingress/default-ingress-cert: client rate limiter Wait returned an error: context deadline exceeded",
err: <*fmt.wrapError | 0x21f4804280>{
msg: "client rate limiter Wait returned an error: context deadline exceeded",
err: <context.deadlineExceededError>{},
},
},
}
fail [github.com/Azure/ARO-HCP/test/e2e/default_cluster_ingress_cert.go:165]: HyperShift's ingress certificate secret must exist and be referenced by IngressController/default
Expected success, but got an error:
<*fmt.wrapError | 0x21f4804320>:
VerifyHyperShiftIngressCertificate timed out after 10m0.006s: getting HyperShift ingress secret openshift-ingress/default-ingress-cert: client rate limiter Wait returned an error: context deadline exceeded
{
msg: "VerifyHyperShiftIngressCertificate timed out after 10m0.006s: getting HyperShift ingress secret openshift-ingress/default-ingress-cert: client rate limiter Wait returned an error: context deadline exceeded",
err: <*fmt.wrapError | 0x21f48042e0>{
msg: "getting HyperShift ingress secret openshift-ingress/default-ingress-cert: client rate limiter Wait returned an error: context deadline exceeded",
err: <*fmt.wrapError | 0x21f4804280>{
msg: "client rate limiter Wait returned an error: context deadline exceeded",
err: <context.deadlineExceededError>{},
},
},
}fail [github.com/Azure/ARO-HCP/test/e2e/default_cluster_ingress_cert.go:165]: HyperShift's ingress certificate secret must exist and be referenced by IngressController/default
Expected success, but got an error:
<*fmt.wrapError | 0x27807b514a20>:
VerifyHyperShiftIngressCertificate timed out after 10m0.003s: getting HyperShift ingress secret openshift-ingress/default-ingress-cert: client rate limiter Wait returned an error: context deadline exceeded
{
msg: "VerifyHyperShiftIngressCertificate timed out after 10m0.003s: getting HyperShift ingress secret openshift-ingress/default-ingress-cert: client rate limiter Wait returned an error: context deadline exceeded",
err: <*fmt.wrapError | 0x27807b5149e0>{
msg: "getting HyperShift ingress secret openshift-ingress/default-ingress-cert: client rate limiter Wait returned an error: context deadline exceeded",
err: <*fmt.wrapError | 0x27807b514980>{
msg: "client rate limiter Wait returned an error: context deadline exceeded",
err: <context.deadlineExceededError>{},
},
},
}
fail [github.com/Azure/ARO-HCP/test/e2e/default_cluster_ingress_cert.go:165]: HyperShift's ingress certificate secret must exist and be referenced by IngressController/default
Expected success, but got an error:
<*fmt.wrapError | 0x27807b514a20>:
VerifyHyperShiftIngressCertificate timed out after 10m0.003s: getting HyperShift ingress secret openshift-ingress/default-ingress-cert: client rate limiter Wait returned an error: context deadline exceeded
{
msg: "VerifyHyperShiftIngressCertificate timed out after 10m0.003s: getting HyperShift ingress secret openshift-ingress/default-ingress-cert: client rate limiter Wait returned an error: context deadline exceeded",
err: <*fmt.wrapError | 0x27807b5149e0>{
msg: "getting HyperShift ingress secret openshift-ingress/default-ingress-cert: client rate limiter Wait returned an error: context deadline exceeded",
err: <*fmt.wrapError | 0x27807b514980>{
msg: "client rate limiter Wait returned an error: context deadline exceeded",
err: <context.deadlineExceededError>{},
},
},
}Contributing tests (2)
Affected runs (4)
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
external auth <external-auth>.Status was nilfull failure pattern: external auth <external-auth>.Status was nil Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 3 day(s) | e2e | 4 | 14.81%4 of 27 job runs affected | IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 3 day(s) | Show trend detailsOct 12: 0 · Oct 13: 0 · Oct 14: 0 · Oct 15: 0 · Oct 16: 1 · Oct 17: 2 · Oct 18: 1 · Oct 19: 0 · Oct 20: 0 · Oct 21: 0 · Oct 22: 0 · Oct 23: 0 · Oct 24: 0 · Oct 25: 0 | PROD | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (3)fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:343]: Timed out after 300.000s.
Degraded condition did not converge to False
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:323 with:
external auth Properties.Status was nil
Expected
<*armredhatopenshifthcp.ResourceStatus | 0x0>: nil
not to be nil
fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:343]: Timed out after 300.000s.
Degraded condition did not converge to False
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:323 with:
external auth Properties.Status was nil
Expected
<*armredhatopenshifthcp.ResourceStatus | 0x0>: nil
not to be nilfail [github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:343]: Timed out after 300.604s.
Degraded condition did not converge to False
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:323 with:
external auth Properties.Status was nil
Expected
<*armredhatopenshifthcp.ResourceStatus | 0x0>: nil
not to be nil
fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:343]: Timed out after 300.604s.
Degraded condition did not converge to False
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:323 with:
external auth Properties.Status was nil
Expected
<*armredhatopenshifthcp.ResourceStatus | 0x0>: nil
not to be nilfail [github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:343]: Timed out after 300.002s.
Degraded condition did not converge to False
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:323 with:
external auth Properties.Status was nil
Expected
<*armredhatopenshifthcp.ResourceStatus | 0x0>: nil
not to be nil
fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:343]: Timed out after 300.002s.
Degraded condition did not converge to False
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:323 with:
external auth Properties.Status was nil
Expected
<*armredhatopenshifthcp.ResourceStatus | 0x0>: nil
not to be nilContributing tests (2)
Affected runs (4)
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedC...full failure pattern: ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable; hosted cluster degraded: UnavailableReplicas: multiple components; provider Microsoft.RedHatOpenShift Signal: IndeterminateSignal: Indeterminate — seen in 2 prior week(s); active 2 day(s) | e2e | 2 | 7.41%2 of 27 job runs affected | IndeterminateSignal: Indeterminate — seen in 2 prior week(s); active 2 day(s) | Show trend detailsOct 12: 1 · Oct 13: 0 · Oct 14: 1 · Oct 15: 0 · Oct 16: 0 · Oct 17: 0 · Oct 18: 0 · Oct 19: 0 · Oct 20: 0 · Oct 21: 0 · Oct 22: 0 · Oct 23: 0 · Oct 24: 0 · Oct 25: 0 | DEV, PROD | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (2)fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:133]: failed to create HCP cluster "cp-ystream-upgrade-5-1-qfmkcj" at install version 5.0.0-0.nightly-multi-2026-10-14-152123
Unexpected error:
<*fmt.wrapError | 0x33329457b0a0>:
failed to create HCP cluster cp-ystream-upgrade-5-1-qfmkcj: failed waiting for cluster="cp-ystream-upgrade-5-1-qfmkcj" in resourcegroup="rg-cp-ystream-upgrade-5-1-667r4hptzmjc" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/b88eb811-d667-4408-bb78-6c86786b44f0
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/b88eb811-d667-4408-bb78-6c86786b44f0",
"name": "b88eb811-d667-4408-bb78-6c86786b44f0",
"status": "Failed",
"startTime": "2026-10-14T23:29:34.061824976Z",
"endTime": "2026-10-14T23:58:37.90403656Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: dns-operator, ingress-operator, openshift-controller-manager, cluster-network-operator, cluster-image-registry-operator, catalog-operator, openshift-route-controller-manager, packageserver, cluster-version-operator, cluster-storage-operator, cluster-node-tuning-operator, csi-snapshot-controller-operator, hosted-cluster-config-operator, olm-operator; hosted cluster degraded: UnavailableReplicas: [catalog-operator deployment has 1 unavailable replicas, cluster-image-registry-operator deployment has 1 unavailable replicas, cluster-network-operator deployment has 1 unavailable replicas, cluster-node-tuning-operator deployment has 1 unavailable replicas, cluster-policy-controller deployment has 1 unavailable replicas, cluster-storage-operator deployment has 1 unavailable replicas, cluster-version-operator deployment has 1 unavailable replicas, csi-snapshot-controller-operator deployment has 1 unavailable replicas, dns-operator deployment has 1 unavailable replicas, hosted-cluster-config-operator deployment has 1 unavailable replicas, ingress-operator deployment has 1 unavailable replicas, olm-operator deployment has 1 unavailable replicas, openshift-controller-manager deployment has 1 unavailable replicas, openshift-route-controller-manager deployment has 1 unavailable replicas, packageserver deployment has 3 unavailable replicas]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/b88eb811-d667-4408-bb78-6c86786b44f0
correlation ID: 8de1d04b-8264-4a06-8468-c991e9773a81
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:133]: failed to create HCP cluster "cp-ystream-upgrade-5-1-qfmkcj" at install version 5.0.0-0.nightly-multi-2026-10-14-152123
Unexpected error:
<*fmt.wrapError | 0x33329457b0a0>:
failed to create HCP cluster cp-ystream-upgrade-5-1-qfmkcj: failed waiting for cluster="cp-ystream-upgrade-5-1-qfmkcj" in resourcegroup="rg-cp-ystream-upgrade-5-1-667r4hptzmjc" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/b88eb811-d667-4408-bb78-6c86786b44f0
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/b88eb811-d667-4408-bb78-6c86786b44f0",
"name": "b88eb811-d667-4408-bb78-6c86786b44f0",
"status": "Failed",
"startTime": "2026-10-14T23:29:34.061824976Z",
"endTime": "2026-10-14T23:58:37.90403656Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: dns-operator, ingress-operator, openshift-controller-manager, cluster-network-operator, cluster-image-registry-operator, catalog-operator, openshift-route-controller-manager, packageserver, cluster-version-operator, cluster-storage-operator, cluster-node-tuning-operator, csi-snapshot-controller-operator, hosted-cluster-config-operator, olm-operator; hosted cluster degraded: UnavailableReplicas: [catalog-operator deployment has 1 unavailable replicas, cluster-image-registry-operator deployment has 1 unavailable replicas, cluster-network-operator deployment has 1 unavailable replicas, cluster-node-tuning-operator deployment has 1 unavailable replicas, cluster-policy-controller deployment has 1 unavailable replicas, cluster-storage-operator deployment has 1 unavailable replicas, cluster-version-operator deployment has 1 unavailable replicas, csi-snapshot-controller-operator deployment has 1 unavailable replicas, dns-operator deployment has 1 unavailable replicas, hosted-cluster-config-operator deployment has 1 unavailable replicas, ingress-operator deployment has 1 unavailable replicas, olm-operator deployment has 1 unavailable replicas, openshift-controller-manager deployment has 1 unavailable replicas, openshift-route-controller-manager deployment has 1 unavailable replicas, packageserver deployment has 3 unavailable replicas]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/b88eb811-d667-4408-bb78-6c86786b44f0
correlation ID: 8de1d04b-8264-4a06-8468-c991e9773a81
client request ID: unavailable
occurredfail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-nightly-4-23-jglxt7lcshss/cluster-nightly-4-23-r26z2q should provision
Unexpected error:
<*fmt.wrapError | 0x2d73d9d10e00>:
failed to create HCP cluster cluster-nightly-4-23-r26z2q: failed waiting for cluster="cluster-nightly-4-23-r26z2q" in resourcegroup="rg-nightly-4-23-jglxt7lcshss" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/41bf70ba-7344-4c62-9364-b6165697473d
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/41bf70ba-7344-4c62-9364-b6165697473d",
"name": "41bf70ba-7344-4c62-9364-b6165697473d",
"status": "Failed",
"startTime": "2026-10-12T23:27:02.793421826Z",
"endTime": "2026-10-12T23:56:08.063137348Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: catalog-operator, cluster-node-tuning-operator, dns-operator, machine-approver, cluster-storage-operator, kube-storage-version-migrator, cluster-policy-controller, hosted-cluster-config-operator, csi-snapshot-controller-operator, cluster-image-registry-operator, packageserver, cluster-version-operator, ingress-operator, control-plane-pki-operator, olm-operator, cluster-network-operator, konnectivity-agent; hosted cluster degraded: UnavailableReplicas: [catalog-operator deployment has 1 unavailable replicas, cluster-image-registry-operator deployment has 1 unavailable replicas, cluster-network-operator deployment has 1 unavailable replicas, cluster-node-tuning-operator deployment has 1 unavailable replicas, cluster-policy-controller deployment has 1 unavailable replicas, cluster-storage-operator deployment has 1 unavailable replicas, cluster-version-operator deployment has 1 unavailable replicas, control-plane-pki-operator deployment has 1 unavailable replicas, csi-snapshot-controller-operator deployment has 1 unavailable replicas, dns-operator deployment has 1 unavailable replicas, hosted-cluster-config-operator deployment has 1 unavailable replicas, ingress-operator deployment has 1 unavailable replicas, konnectivity-agent deployment has 1 unavailable replicas, kube-storage-version-migrator deployment has 1 unavailable replicas, machine-approver deployment has 1 unavailable replicas, olm-operator deployment has 1 unavailable replicas, packageserver deployment has 3 unavailable replicas]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/41bf70ba-7344-4c62-9364-b6165697473d
correlation ID: bbc14ae0-9ae0-4116-9b30-00ddeee9cda8
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-nightly-4-23-jglxt7lcshss/cluster-nightly-4-23-r26z2q should provision
Unexpected error:
<*fmt.wrapError | 0x2d73d9d10e00>:
failed to create HCP cluster cluster-nightly-4-23-r26z2q: failed waiting for cluster="cluster-nightly-4-23-r26z2q" in resourcegroup="rg-nightly-4-23-jglxt7lcshss" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/41bf70ba-7344-4c62-9364-b6165697473d
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/41bf70ba-7344-4c62-9364-b6165697473d",
"name": "41bf70ba-7344-4c62-9364-b6165697473d",
"status": "Failed",
"startTime": "2026-10-12T23:27:02.793421826Z",
"endTime": "2026-10-12T23:56:08.063137348Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: catalog-operator, cluster-node-tuning-operator, dns-operator, machine-approver, cluster-storage-operator, kube-storage-version-migrator, cluster-policy-controller, hosted-cluster-config-operator, csi-snapshot-controller-operator, cluster-image-registry-operator, packageserver, cluster-version-operator, ingress-operator, control-plane-pki-operator, olm-operator, cluster-network-operator, konnectivity-agent; hosted cluster degraded: UnavailableReplicas: [catalog-operator deployment has 1 unavailable replicas, cluster-image-registry-operator deployment has 1 unavailable replicas, cluster-network-operator deployment has 1 unavailable replicas, cluster-node-tuning-operator deployment has 1 unavailable replicas, cluster-policy-controller deployment has 1 unavailable replicas, cluster-storage-operator deployment has 1 unavailable replicas, cluster-version-operator deployment has 1 unavailable replicas, control-plane-pki-operator deployment has 1 unavailable replicas, csi-snapshot-controller-operator deployment has 1 unavailable replicas, dns-operator deployment has 1 unavailable replicas, hosted-cluster-config-operator deployment has 1 unavailable replicas, ingress-operator deployment has 1 unavailable replicas, konnectivity-agent deployment has 1 unavailable replicas, kube-storage-version-migrator deployment has 1 unavailable replicas, machine-approver deployment has 1 unavailable replicas, olm-operator deployment has 1 unavailable replicas, packageserver deployment has 3 unavailable replicas]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/41bf70ba-7344-4c62-9364-b6165697473d
correlation ID: bbc14ae0-9ae0-4116-9b30-00ddeee9cda8
client request ID: unavailable
occurredContributing tests (2)
Affected runs (2)
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
ERROR CODE: DeadlineExceeded; detail message node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hype...full failure pattern: ERROR CODE: DeadlineExceeded; detail message node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is <count>, want <count>; node pool condition AllNodesHealthy is False: <count> of <count> machines are not healthy Machine <node>: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.Compute/virtualMachines/<node> to exist; node pool condition AllMachinesReady is False: <count> of <count> machines are not ready Machine <node>: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.Compute/virtualMachines/<node> to exist; provider Microsoft.RedHatOpenShift Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 2 day(s) | e2e | 2 | 7.41%2 of 27 job runs affected | IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 2 day(s) | Show trend detailsOct 12: 1 · Oct 13: 0 · Oct 14: 1 · Oct 15: 0 · Oct 16: 0 · Oct 17: 0 · Oct 18: 0 · Oct 19: 0 · Oct 20: 0 · Oct 21: 0 · Oct 22: 0 · Oct 23: 0 · Oct 24: 0 · Oct 25: 0 | DEV, PROD | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (2)fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_cordon.go:117]: failed to create node pool cordon-np
Unexpected error:
<*fmt.wrapError | 0x2e2df3fce3a0>:
failed to create NodePool cordon-np: failed waiting for nodepool="cordon-np" for cluster "cordon-hcp-cluster" in resourcegroup="rg-cordon-test-cxh9xv97rj9f" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/0195b61d-2798-45a3-a546-b2e316a8e4be
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/0195b61d-2798-45a3-a546-b2e316a8e4be",
"name": "0195b61d-2798-45a3-a546-b2e316a8e4be",
"status": "Failed",
"startTime": "2026-10-14T03:10:09.167253269Z",
"endTime": "2026-10-14T03:39:12.28722327Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine v9u7s9d4e2s3d9f-cordon-np-j7dj6-snwxb: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cordon-test-cxh9xv97rj9f--managed/providers/Microsoft.Compute/virtualMachines/v9u7s9d4e2s3d9f-cordon-np-j7dj6-snwxb to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine v9u7s9d4e2s3d9f-cordon-np-j7dj6-snwxb: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cordon-test-cxh9xv97rj9f--managed/providers/Microsoft.Compute/virtualMachines/v9u7s9d4e2s3d9f-cordon-np-j7dj6-snwxb to exist\n"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/0195b61d-2798-45a3-a546-b2e316a8e4be
correlation ID: 6295da04-1f90-4641-a016-0502957f2d44
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_cordon.go:117]: failed to create node pool cordon-np
Unexpected error:
<*fmt.wrapError | 0x2e2df3fce3a0>:
failed to create NodePool cordon-np: failed waiting for nodepool="cordon-np" for cluster "cordon-hcp-cluster" in resourcegroup="rg-cordon-test-cxh9xv97rj9f" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/0195b61d-2798-45a3-a546-b2e316a8e4be
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/0195b61d-2798-45a3-a546-b2e316a8e4be",
"name": "0195b61d-2798-45a3-a546-b2e316a8e4be",
"status": "Failed",
"startTime": "2026-10-14T03:10:09.167253269Z",
"endTime": "2026-10-14T03:39:12.28722327Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine v9u7s9d4e2s3d9f-cordon-np-j7dj6-snwxb: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cordon-test-cxh9xv97rj9f--managed/providers/Microsoft.Compute/virtualMachines/v9u7s9d4e2s3d9f-cordon-np-j7dj6-snwxb to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine v9u7s9d4e2s3d9f-cordon-np-j7dj6-snwxb: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cordon-test-cxh9xv97rj9f--managed/providers/Microsoft.Compute/virtualMachines/v9u7s9d4e2s3d9f-cordon-np-j7dj6-snwxb to exist\n"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/0195b61d-2798-45a3-a546-b2e316a8e4be
correlation ID: 6295da04-1f90-4641-a016-0502957f2d44
client request ID: unavailable
occurredfail [github.com/Azure/ARO-HCP/test/e2e/nodepool_cordon.go:117]: failed to create node pool cordon-np
Unexpected error:
<*fmt.wrapError | 0x371cbc791620>:
failed to create NodePool cordon-np: failed waiting for nodepool="cordon-np" for cluster "cordon-hcp-cluster" in resourcegroup="rg-cordon-test-n9bmt8cwvq6g" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/7da88189-d1ef-4299-b869-95f7299be069
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/7da88189-d1ef-4299-b869-95f7299be069",
"name": "7da88189-d1ef-4299-b869-95f7299be069",
"status": "Failed",
"startTime": "2026-10-12T23:47:30.718672913Z",
"endTime": "2026-10-13T00:16:35.706367535Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine c1m4z2y4h9h6m1o-cordon-np-qfzbb-vrnqt: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cordon-test-n9bmt8cwvq6g--managed/providers/Microsoft.Compute/virtualMachines/c1m4z2y4h9h6m1o-cordon-np-qfzbb-vrnqt to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine c1m4z2y4h9h6m1o-cordon-np-qfzbb-vrnqt: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cordon-test-n9bmt8cwvq6g--managed/providers/Microsoft.Compute/virtualMachines/c1m4z2y4h9h6m1o-cordon-np-qfzbb-vrnqt to exist\n"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/7da88189-d1ef-4299-b869-95f7299be069
correlation ID: f89db514-fba8-4f47-b43b-0ea8cd9a0b5e
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_cordon.go:117]: failed to create node pool cordon-np
Unexpected error:
<*fmt.wrapError | 0x371cbc791620>:
failed to create NodePool cordon-np: failed waiting for nodepool="cordon-np" for cluster "cordon-hcp-cluster" in resourcegroup="rg-cordon-test-n9bmt8cwvq6g" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/7da88189-d1ef-4299-b869-95f7299be069
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/7da88189-d1ef-4299-b869-95f7299be069",
"name": "7da88189-d1ef-4299-b869-95f7299be069",
"status": "Failed",
"startTime": "2026-10-12T23:47:30.718672913Z",
"endTime": "2026-10-13T00:16:35.706367535Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine c1m4z2y4h9h6m1o-cordon-np-qfzbb-vrnqt: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cordon-test-n9bmt8cwvq6g--managed/providers/Microsoft.Compute/virtualMachines/c1m4z2y4h9h6m1o-cordon-np-qfzbb-vrnqt to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine c1m4z2y4h9h6m1o-cordon-np-qfzbb-vrnqt: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cordon-test-n9bmt8cwvq6g--managed/providers/Microsoft.Compute/virtualMachines/c1m4z2y4h9h6m1o-cordon-np-qfzbb-vrnqt to exist\n"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/7da88189-d1ef-4299-b869-95f7299be069
correlation ID: f89db514-fba8-4f47-b43b-0ea8cd9a0b5e
client request ID: unavailable
occurredContributing tests (2)
Affected runs (2)
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
ERROR CODE: InternalServerError; detail message [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not ...full failure pattern: ERROR CODE: InternalServerError; detail message [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated; provider Microsoft.RedHatOpenShift Signal: IndeterminateSignal: Indeterminate — seen in 2 prior week(s); active 1 day(s) | e2e | 1 | 3.70%1 of 27 job runs affected | IndeterminateSignal: Indeterminate — seen in 2 prior week(s); active 1 day(s) | Show trend detailsOct 12: 0 · Oct 13: 11 · Oct 14: 0 · Oct 15: 0 · Oct 16: 0 · Oct 17: 0 · Oct 18: 0 · Oct 19: 0 · Oct 20: 0 · Oct 21: 0 · Oct 22: 0 · Oct 23: 0 · Oct 24: 0 · Oct 25: 0 | none | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (3)fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-candidate-4-21-hkwrn8b75t7c/cluster-candidate-4-21-8m4xcg should provision
Unexpected error:
<*fmt.wrapError | 0x15168f06cae0>:
failed to create HCP cluster cluster-candidate-4-21-8m4xcg: failed waiting for cluster="cluster-candidate-4-21-8m4xcg" in resourcegroup="rg-candidate-4-21-hkwrn8b75t7c" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/8479abf5-39da-412b-bb04-87374bf3b757
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: InternalServerError
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/8479abf5-39da-412b-bb04-87374bf3b757",
"name": "8479abf5-39da-412b-bb04-87374bf3b757",
"status": "Failed",
"startTime": "2026-10-13T00:45:19.685007768Z",
"endTime": "2026-10-13T01:14:27.351105107Z",
"error": {
"code": "InternalServerError",
"message": "[cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/8479abf5-39da-412b-bb04-87374bf3b757
correlation ID: 46a9a6c6-2ede-454b-9940-a881ca6f8b60
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-candidate-4-21-hkwrn8b75t7c/cluster-candidate-4-21-8m4xcg should provision
Unexpected error:
<*fmt.wrapError | 0x15168f06cae0>:
failed to create HCP cluster cluster-candidate-4-21-8m4xcg: failed waiting for cluster="cluster-candidate-4-21-8m4xcg" in resourcegroup="rg-candidate-4-21-hkwrn8b75t7c" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/8479abf5-39da-412b-bb04-87374bf3b757
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: InternalServerError
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/8479abf5-39da-412b-bb04-87374bf3b757",
"name": "8479abf5-39da-412b-bb04-87374bf3b757",
"status": "Failed",
"startTime": "2026-10-13T00:45:19.685007768Z",
"endTime": "2026-10-13T01:14:27.351105107Z",
"error": {
"code": "InternalServerError",
"message": "[cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/8479abf5-39da-412b-bb04-87374bf3b757
correlation ID: 46a9a6c6-2ede-454b-9940-a881ca6f8b60
client request ID: unavailable
occurredfail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kas.go:101]: failed to create HCP cluster "private-kas" with private KAS
Unexpected error:
<*fmt.wrapError | 0x3152b3c1a500>:
failed to create HCP cluster private-kas: failed waiting for cluster="private-kas" in resourcegroup="private-kas-rtr2wnffp4rh" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/db0a48c2-f9bb-4a38-94f5-afccbc7d779c
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: InternalServerError
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/db0a48c2-f9bb-4a38-94f5-afccbc7d779c",
"name": "db0a48c2-f9bb-4a38-94f5-afccbc7d779c",
"status": "Failed",
"startTime": "2026-10-13T00:45:30.036006045Z",
"endTime": "2026-10-13T01:14:37.44566298Z",
"error": {
"code": "InternalServerError",
"message": "[cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/db0a48c2-f9bb-4a38-94f5-afccbc7d779c
correlation ID: 6b6c3060-a1bb-4638-afa6-7fb92caae9a8
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kas.go:101]: failed to create HCP cluster "private-kas" with private KAS
Unexpected error:
<*fmt.wrapError | 0x3152b3c1a500>:
failed to create HCP cluster private-kas: failed waiting for cluster="private-kas" in resourcegroup="private-kas-rtr2wnffp4rh" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/db0a48c2-f9bb-4a38-94f5-afccbc7d779c
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: InternalServerError
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/db0a48c2-f9bb-4a38-94f5-afccbc7d779c",
"name": "db0a48c2-f9bb-4a38-94f5-afccbc7d779c",
"status": "Failed",
"startTime": "2026-10-13T00:45:30.036006045Z",
"endTime": "2026-10-13T01:14:37.44566298Z",
"error": {
"code": "InternalServerError",
"message": "[cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/db0a48c2-f9bb-4a38-94f5-afccbc7d779c
correlation ID: 6b6c3060-a1bb-4638-afa6-7fb92caae9a8
client request ID: unavailable
occurredfail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:208]: failed to create HCP cluster "agg-cluster" with aggregated settings
Unexpected error:
<*fmt.wrapError | 0x15a1e46902c0>:
failed waiting for cluster="agg-cluster" in resourcegroup="feature-aggregation-xgppw4wmq2g9" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/8f006a7a-53bf-4566-93d0-8202c11da346
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: InternalServerError
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/8f006a7a-53bf-4566-93d0-8202c11da346",
"name": "8f006a7a-53bf-4566-93d0-8202c11da346",
"status": "Failed",
"startTime": "2026-10-13T00:52:36.6753625Z",
"endTime": "2026-10-13T01:21:37.350837076Z",
"error": {
"code": "InternalServerError",
"message": "[cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/8f006a7a-53bf-4566-93d0-8202c11da346
correlation ID: 8ed813fc-cdf9-4562-abe6-9eb495caf777
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:208]: failed to create HCP cluster "agg-cluster" with aggregated settings
Unexpected error:
<*fmt.wrapError | 0x15a1e46902c0>:
failed waiting for cluster="agg-cluster" in resourcegroup="feature-aggregation-xgppw4wmq2g9" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/8f006a7a-53bf-4566-93d0-8202c11da346
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: InternalServerError
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/8f006a7a-53bf-4566-93d0-8202c11da346",
"name": "8f006a7a-53bf-4566-93d0-8202c11da346",
"status": "Failed",
"startTime": "2026-10-13T00:52:36.6753625Z",
"endTime": "2026-10-13T01:21:37.350837076Z",
"error": {
"code": "InternalServerError",
"message": "[cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/8f006a7a-53bf-4566-93d0-8202c11da346
correlation ID: 8ed813fc-cdf9-4562-abe6-9eb495caf777
client request ID: unavailable
occurredContributing tests (11)
Affected runs (1)
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
ERROR CODE: InvalidRequestContent; detail message Invalid value: "aro-hcp.experimental.cluster.max-update-duration": unrecognized experimental tag; provider Microsoft.RedHatOpenSh...full failure pattern: ERROR CODE: InvalidRequestContent; detail message Invalid value: "aro-hcp.experimental.cluster.max-update-duration": unrecognized experimental tag; provider Microsoft.RedHatOpenShift Signal: IndeterminateSignal: Indeterminate — seen in 2 prior week(s); active 1 day(s) | e2e | 1 | 3.70%1 of 27 job runs affected | IndeterminateSignal: Indeterminate — seen in 2 prior week(s); active 1 day(s) | Show trend detailsOct 12: 0 · Oct 13: 10 · Oct 14: 0 · Oct 15: 0 · Oct 16: 0 · Oct 17: 0 · Oct 18: 0 · Oct 19: 0 · Oct 20: 0 · Oct 21: 0 · Oct 22: 0 · Oct 23: 0 · Oct 24: 0 · Oct 25: 0 | none | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (3)fail [github.com/Azure/ARO-HCP/test/e2e/idms_lifecycle.go:175]: failed to update cluster with second ImageDigestMirror set
Unexpected error:
<*exported.ResponseError | 0x1854e4898c30>:
PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/idms-v622rq7cjdv8/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/idms-e2e-hcp-cluster
--------------------------------------------------------------------------------
RESPONSE 400: 400 Bad Request
ERROR CODE: InvalidRequestContent
--------------------------------------------------------------------------------
{
"error": {
"code": "InvalidRequestContent",
"message": "Invalid value: \"aro-hcp.experimental.cluster.max-update-duration\": unrecognized experimental tag",
"target": "tags[aro-hcp.experimental.cluster.max-update-duration]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/idms-v622rq7cjdv8/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/idms-e2e-hcp-cluster
correlation ID: e25d2783-3afe-4763-a826-691a13ceb4bb
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/idms_lifecycle.go:175]: failed to update cluster with second ImageDigestMirror set
Unexpected error:
<*exported.ResponseError | 0x1854e4898c30>:
PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/idms-v622rq7cjdv8/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/idms-e2e-hcp-cluster
--------------------------------------------------------------------------------
RESPONSE 400: 400 Bad Request
ERROR CODE: InvalidRequestContent
--------------------------------------------------------------------------------
{
"error": {
"code": "InvalidRequestContent",
"message": "Invalid value: \"aro-hcp.experimental.cluster.max-update-duration\": unrecognized experimental tag",
"target": "tags[aro-hcp.experimental.cluster.max-update-duration]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/idms-v622rq7cjdv8/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/idms-e2e-hcp-cluster
correlation ID: e25d2783-3afe-4763-a826-691a13ceb4bb
client request ID: unavailable
occurredfail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:167]: failed to trigger y-stream upgrade of cluster "cp-ystream-upgrade-5-1-m6j7kp" to 5.1
Unexpected error:
<*exported.ResponseError | 0x13a151ae2450>:
PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cp-ystream-upgrade-5-1-lnhr67n47jrg/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cp-ystream-upgrade-5-1-m6j7kp
--------------------------------------------------------------------------------
RESPONSE 400: 400 Bad Request
ERROR CODE: InvalidRequestContent
--------------------------------------------------------------------------------
{
"error": {
"code": "InvalidRequestContent",
"message": "Invalid value: \"aro-hcp.experimental.cluster.max-update-duration\": unrecognized experimental tag",
"target": "tags[aro-hcp.experimental.cluster.max-update-duration]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cp-ystream-upgrade-5-1-lnhr67n47jrg/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cp-ystream-upgrade-5-1-m6j7kp
correlation ID: b7196ccc-8430-45a5-9628-149904d927cb
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:167]: failed to trigger y-stream upgrade of cluster "cp-ystream-upgrade-5-1-m6j7kp" to 5.1
Unexpected error:
<*exported.ResponseError | 0x13a151ae2450>:
PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cp-ystream-upgrade-5-1-lnhr67n47jrg/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cp-ystream-upgrade-5-1-m6j7kp
--------------------------------------------------------------------------------
RESPONSE 400: 400 Bad Request
ERROR CODE: InvalidRequestContent
--------------------------------------------------------------------------------
{
"error": {
"code": "InvalidRequestContent",
"message": "Invalid value: \"aro-hcp.experimental.cluster.max-update-duration\": unrecognized experimental tag",
"target": "tags[aro-hcp.experimental.cluster.max-update-duration]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cp-ystream-upgrade-5-1-lnhr67n47jrg/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cp-ystream-upgrade-5-1-m6j7kp
correlation ID: b7196ccc-8430-45a5-9628-149904d927cb
client request ID: unavailable
occurredfail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:193]: failed to update HCP cluster tags via PATCH
Unexpected error:
<*exported.ResponseError | 0x1d06db38a6c0>:
PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/patch-tags-kbw2fdfhjc7f/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/patch-tags-cluster
--------------------------------------------------------------------------------
RESPONSE 400: 400 Bad Request
ERROR CODE: InvalidRequestContent
--------------------------------------------------------------------------------
{
"error": {
"code": "InvalidRequestContent",
"message": "Invalid value: \"aro-hcp.experimental.cluster.max-update-duration\": unrecognized experimental tag",
"target": "tags[aro-hcp.experimental.cluster.max-update-duration]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/patch-tags-kbw2fdfhjc7f/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/patch-tags-cluster
correlation ID: 581e9f52-3486-4da8-af1a-3a90318bf15f
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:193]: failed to update HCP cluster tags via PATCH
Unexpected error:
<*exported.ResponseError | 0x1d06db38a6c0>:
PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/patch-tags-kbw2fdfhjc7f/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/patch-tags-cluster
--------------------------------------------------------------------------------
RESPONSE 400: 400 Bad Request
ERROR CODE: InvalidRequestContent
--------------------------------------------------------------------------------
{
"error": {
"code": "InvalidRequestContent",
"message": "Invalid value: \"aro-hcp.experimental.cluster.max-update-duration\": unrecognized experimental tag",
"target": "tags[aro-hcp.experimental.cluster.max-update-duration]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/patch-tags-kbw2fdfhjc7f/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/patch-tags-cluster
correlation ID: 581e9f52-3486-4da8-af1a-3a90318bf15f
client request ID: unavailable
occurredContributing tests (10)
Affected runs (1)
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
ERROR CODE: InvalidRequestContent; detail message Invalid value: "aro-hcp.experimental.nodepool.max-update-duration": unrecognized experimental tag; provider Microsoft.RedHatOpenS...full failure pattern: ERROR CODE: InvalidRequestContent; detail message Invalid value: "aro-hcp.experimental.nodepool.max-update-duration": unrecognized experimental tag; provider Microsoft.RedHatOpenShift Signal: IndeterminateSignal: Indeterminate — seen in 2 prior week(s); active 1 day(s) | e2e | 1 | 3.70%1 of 27 job runs affected | IndeterminateSignal: Indeterminate — seen in 2 prior week(s); active 1 day(s) | Show trend detailsOct 12: 0 · Oct 13: 9 · Oct 14: 0 · Oct 15: 0 · Oct 16: 0 · Oct 17: 0 · Oct 18: 0 · Oct 19: 0 · Oct 20: 0 · Oct 21: 0 · Oct 22: 0 · Oct 23: 0 · Oct 24: 0 · Oct 25: 0 | none | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (3)fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:195]: failed to upgrade node pool npupgrade-4-20 to version 4.20.42
Unexpected error:
<*fmt.wrapError | 0x31faef804060>:
failed to start nodepool "npupgrade-4-20" update in cluster "np-version-upgrade-cluster-w5jlcn" resourcegroup="rg-np-version-upgrade-w5jlcn-chd7h2jfcvzd": PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-version-upgrade-w5jlcn-chd7h2jfcvzd/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-version-upgrade-cluster-w5jlcn/nodePools/npupgrade-4-20
--------------------------------------------------------------------------------
RESPONSE 400: 400 Bad Request
ERROR CODE: InvalidRequestContent
--------------------------------------------------------------------------------
{
"error": {
"code": "InvalidRequestContent",
"message": "Invalid value: \"aro-hcp.experimental.nodepool.max-update-duration\": unrecognized experimental tag",
"target": "tags[aro-hcp.experimental.nodepool.max-update-duration]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-version-upgrade-w5jlcn-chd7h2jfcvzd/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-version-upgrade-cluster-w5jlcn/nodePools/npupgrade-4-20
correlation ID: 6037c313-80d6-4b5c-af77-583498b149c9
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:195]: failed to upgrade node pool npupgrade-4-20 to version 4.20.42
Unexpected error:
<*fmt.wrapError | 0x31faef804060>:
failed to start nodepool "npupgrade-4-20" update in cluster "np-version-upgrade-cluster-w5jlcn" resourcegroup="rg-np-version-upgrade-w5jlcn-chd7h2jfcvzd": PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-version-upgrade-w5jlcn-chd7h2jfcvzd/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-version-upgrade-cluster-w5jlcn/nodePools/npupgrade-4-20
--------------------------------------------------------------------------------
RESPONSE 400: 400 Bad Request
ERROR CODE: InvalidRequestContent
--------------------------------------------------------------------------------
{
"error": {
"code": "InvalidRequestContent",
"message": "Invalid value: \"aro-hcp.experimental.nodepool.max-update-duration\": unrecognized experimental tag",
"target": "tags[aro-hcp.experimental.nodepool.max-update-duration]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-version-upgrade-w5jlcn-chd7h2jfcvzd/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-version-upgrade-cluster-w5jlcn/nodePools/npupgrade-4-20
correlation ID: 6037c313-80d6-4b5c-af77-583498b149c9
client request ID: unavailable
occurredfail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:195]: failed to upgrade node pool npupgrade-4-21 to version 4.21.36
Unexpected error:
<*fmt.wrapError | 0x258acfd8b940>:
failed to start nodepool "npupgrade-4-21" update in cluster "np-version-upgrade-cluster-zfs5gc" resourcegroup="rg-np-version-upgrade-zfs5gc-ksr8l4d4q4gw": PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-version-upgrade-zfs5gc-ksr8l4d4q4gw/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-version-upgrade-cluster-zfs5gc/nodePools/npupgrade-4-21
--------------------------------------------------------------------------------
RESPONSE 400: 400 Bad Request
ERROR CODE: InvalidRequestContent
--------------------------------------------------------------------------------
{
"error": {
"code": "InvalidRequestContent",
"message": "Invalid value: \"aro-hcp.experimental.nodepool.max-update-duration\": unrecognized experimental tag",
"target": "tags[aro-hcp.experimental.nodepool.max-update-duration]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-version-upgrade-zfs5gc-ksr8l4d4q4gw/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-version-upgrade-cluster-zfs5gc/nodePools/npupgrade-4-21
correlation ID: 0b2d3160-3474-40bc-a9f5-2792d7568074
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:195]: failed to upgrade node pool npupgrade-4-21 to version 4.21.36
Unexpected error:
<*fmt.wrapError | 0x258acfd8b940>:
failed to start nodepool "npupgrade-4-21" update in cluster "np-version-upgrade-cluster-zfs5gc" resourcegroup="rg-np-version-upgrade-zfs5gc-ksr8l4d4q4gw": PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-version-upgrade-zfs5gc-ksr8l4d4q4gw/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-version-upgrade-cluster-zfs5gc/nodePools/npupgrade-4-21
--------------------------------------------------------------------------------
RESPONSE 400: 400 Bad Request
ERROR CODE: InvalidRequestContent
--------------------------------------------------------------------------------
{
"error": {
"code": "InvalidRequestContent",
"message": "Invalid value: \"aro-hcp.experimental.nodepool.max-update-duration\": unrecognized experimental tag",
"target": "tags[aro-hcp.experimental.nodepool.max-update-duration]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-version-upgrade-zfs5gc-ksr8l4d4q4gw/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-version-upgrade-cluster-zfs5gc/nodePools/npupgrade-4-21
correlation ID: 0b2d3160-3474-40bc-a9f5-2792d7568074
client request ID: unavailable
occurredfail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:768]: failed to update nodepool npdg-422-420 to downgrade target 4.20.42
Unexpected error:
<*fmt.wrapError | 0x1b553dfaa40>:
failed to start nodepool "npdg-422-420" update in cluster "np-dg-minor-v69sgj" resourcegroup="rg-np-dg-minor-v69sgj-tl6w2zktdhh2": PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-dg-minor-v69sgj-tl6w2zktdhh2/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-dg-minor-v69sgj/nodePools/npdg-422-420
--------------------------------------------------------------------------------
RESPONSE 400: 400 Bad Request
ERROR CODE: InvalidRequestContent
--------------------------------------------------------------------------------
{
"error": {
"code": "InvalidRequestContent",
"message": "Invalid value: \"aro-hcp.experimental.nodepool.max-update-duration\": unrecognized experimental tag",
"target": "tags[aro-hcp.experimental.nodepool.max-update-duration]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-dg-minor-v69sgj-tl6w2zktdhh2/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-dg-minor-v69sgj/nodePools/npdg-422-420
correlation ID: dfeeac19-d930-4394-8440-1041d3f8556f
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:768]: failed to update nodepool npdg-422-420 to downgrade target 4.20.42
Unexpected error:
<*fmt.wrapError | 0x1b553dfaa40>:
failed to start nodepool "npdg-422-420" update in cluster "np-dg-minor-v69sgj" resourcegroup="rg-np-dg-minor-v69sgj-tl6w2zktdhh2": PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-dg-minor-v69sgj-tl6w2zktdhh2/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-dg-minor-v69sgj/nodePools/npdg-422-420
--------------------------------------------------------------------------------
RESPONSE 400: 400 Bad Request
ERROR CODE: InvalidRequestContent
--------------------------------------------------------------------------------
{
"error": {
"code": "InvalidRequestContent",
"message": "Invalid value: \"aro-hcp.experimental.nodepool.max-update-duration\": unrecognized experimental tag",
"target": "tags[aro-hcp.experimental.nodepool.max-update-duration]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-dg-minor-v69sgj-tl6w2zktdhh2/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-dg-minor-v69sgj/nodePools/npdg-422-420
correlation ID: dfeeac19-d930-4394-8440-1041d3f8556f
client request ID: unavailable
occurredContributing tests (9)
Affected runs (1)
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
ERROR CODE: InvalidRequestContent; detail message Invalid value: "aro-hcp.experimental.cluster.z-stream-update-policy": unrecognized experimental tag; provider Microsoft.RedHatOpe...full failure pattern: ERROR CODE: InvalidRequestContent; detail message Invalid value: "aro-hcp.experimental.cluster.z-stream-update-policy": unrecognized experimental tag; provider Microsoft.RedHatOpenShift Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s) | e2e | 1 | 3.70%1 of 27 job runs affected | IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s) | Show trend detailsOct 12: 0 · Oct 13: 3 · Oct 14: 0 · Oct 15: 0 · Oct 16: 0 · Oct 17: 0 · Oct 18: 0 · Oct 19: 0 · Oct 20: 0 · Oct 21: 0 · Oct 22: 0 · Oct 23: 0 · Oct 24: 0 · Oct 25: 0 | none | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (3)fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:157]: failed to enable immediate z-stream updates for cluster "cluster-zstream-4-23-xw6srx" on minor version 4.23
Unexpected error:
<*exported.ResponseError | 0x2299302636e0>:
PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-23-5tqz59mk2x4j/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-23-xw6srx
--------------------------------------------------------------------------------
RESPONSE 400: 400 Bad Request
ERROR CODE: InvalidRequestContent
--------------------------------------------------------------------------------
{
"error": {
"code": "InvalidRequestContent",
"message": "Invalid value: \"aro-hcp.experimental.cluster.z-stream-update-policy\": unrecognized experimental tag",
"target": "tags[aro-hcp.experimental.cluster.z-stream-update-policy]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-23-5tqz59mk2x4j/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-23-xw6srx
correlation ID: d5115cbd-66f0-411a-9ed8-23fc462b21fe
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:157]: failed to enable immediate z-stream updates for cluster "cluster-zstream-4-23-xw6srx" on minor version 4.23
Unexpected error:
<*exported.ResponseError | 0x2299302636e0>:
PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-23-5tqz59mk2x4j/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-23-xw6srx
--------------------------------------------------------------------------------
RESPONSE 400: 400 Bad Request
ERROR CODE: InvalidRequestContent
--------------------------------------------------------------------------------
{
"error": {
"code": "InvalidRequestContent",
"message": "Invalid value: \"aro-hcp.experimental.cluster.z-stream-update-policy\": unrecognized experimental tag",
"target": "tags[aro-hcp.experimental.cluster.z-stream-update-policy]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-23-5tqz59mk2x4j/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-23-xw6srx
correlation ID: d5115cbd-66f0-411a-9ed8-23fc462b21fe
client request ID: unavailable
occurredfail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:157]: failed to enable immediate z-stream updates for cluster "cluster-zstream-5-0-xd29vz" on minor version 5.0
Unexpected error:
<*exported.ResponseError | 0x33ccd41339b0>:
PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-5-0-p7gnkhhshzxc/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-5-0-xd29vz
--------------------------------------------------------------------------------
RESPONSE 400: 400 Bad Request
ERROR CODE: InvalidRequestContent
--------------------------------------------------------------------------------
{
"error": {
"code": "InvalidRequestContent",
"message": "Invalid value: \"aro-hcp.experimental.cluster.z-stream-update-policy\": unrecognized experimental tag",
"target": "tags[aro-hcp.experimental.cluster.z-stream-update-policy]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-5-0-p7gnkhhshzxc/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-5-0-xd29vz
correlation ID: 1a4d52b7-b1bd-42a6-8e35-cd6a15284e2b
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:157]: failed to enable immediate z-stream updates for cluster "cluster-zstream-5-0-xd29vz" on minor version 5.0
Unexpected error:
<*exported.ResponseError | 0x33ccd41339b0>:
PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-5-0-p7gnkhhshzxc/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-5-0-xd29vz
--------------------------------------------------------------------------------
RESPONSE 400: 400 Bad Request
ERROR CODE: InvalidRequestContent
--------------------------------------------------------------------------------
{
"error": {
"code": "InvalidRequestContent",
"message": "Invalid value: \"aro-hcp.experimental.cluster.z-stream-update-policy\": unrecognized experimental tag",
"target": "tags[aro-hcp.experimental.cluster.z-stream-update-policy]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-5-0-p7gnkhhshzxc/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-5-0-xd29vz
correlation ID: 1a4d52b7-b1bd-42a6-8e35-cd6a15284e2b
client request ID: unavailable
occurredfail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:157]: failed to enable immediate z-stream updates for cluster "cluster-zstream-4-21-gk5dl2" on minor version 4.21
Unexpected error:
<*exported.ResponseError | 0x1206c2abaf90>:
PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-21-qct9sm5zfdhm/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-21-gk5dl2
--------------------------------------------------------------------------------
RESPONSE 400: 400 Bad Request
ERROR CODE: InvalidRequestContent
--------------------------------------------------------------------------------
{
"error": {
"code": "InvalidRequestContent",
"message": "Invalid value: \"aro-hcp.experimental.cluster.z-stream-update-policy\": unrecognized experimental tag",
"target": "tags[aro-hcp.experimental.cluster.z-stream-update-policy]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-21-qct9sm5zfdhm/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-21-gk5dl2
correlation ID: 72fd87ea-2d1e-422a-b1c8-66816eaede96
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:157]: failed to enable immediate z-stream updates for cluster "cluster-zstream-4-21-gk5dl2" on minor version 4.21
Unexpected error:
<*exported.ResponseError | 0x1206c2abaf90>:
PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-21-qct9sm5zfdhm/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-21-gk5dl2
--------------------------------------------------------------------------------
RESPONSE 400: 400 Bad Request
ERROR CODE: InvalidRequestContent
--------------------------------------------------------------------------------
{
"error": {
"code": "InvalidRequestContent",
"message": "Invalid value: \"aro-hcp.experimental.cluster.z-stream-update-policy\": unrecognized experimental tag",
"target": "tags[aro-hcp.experimental.cluster.z-stream-update-policy]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-21-qct9sm5zfdhm/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-21-gk5dl2
correlation ID: 72fd87ea-2d1e-422a-b1c8-66816eaede96
client request ID: unavailable
occurredContributing tests (3)
Affected runs (1)
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
CreateOrUpdateExternalAuthAndWait timed out after <minutes> minutes; context deadline exceededfull failure pattern: CreateOrUpdateExternalAuthAndWait timed out after <minutes> minutes; context deadline exceeded Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s) | e2e | 1 | 3.70%1 of 27 job runs affected | IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s) | Show trend detailsOct 12: 0 · Oct 13: 0 · Oct 14: 1 · Oct 15: 0 · Oct 16: 0 · Oct 17: 0 · Oct 18: 0 · Oct 19: 0 · Oct 20: 0 · Oct 21: 0 · Oct 22: 0 · Oct 23: 0 · Oct 24: 0 · Oct 25: 0 | INT, PROD | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:262]: failed to apply external auth config with prefix policy None
Unexpected error:
<*fmt.wrapErrors | 0x95f8488fc80>:
failed waiting for external auth "external-auth" in resourcegroup="external-auth-cluster-n6c4c4fd2ssk" for cluster="ea-cluster" to finish creating or updating, caused by: timeout '15.000000' minutes exceeded during CreateOrUpdateExternalAuthAndWait for external auth external-auth in cluster ea-cluster in resource group external-auth-cluster-n6c4c4fd2ssk, error: context deadline exceeded
{
msg: "failed waiting for external auth \"external-auth\" in resourcegroup=\"external-auth-cluster-n6c4c4fd2ssk\" for cluster=\"ea-cluster\" to finish creating or updating, caused by: timeout '15.000000' minutes exceeded during CreateOrUpdateExternalAuthAndWait for external auth external-auth in cluster ea-cluster in resource group external-auth-cluster-n6c4c4fd2ssk, error: context deadline exceeded",
errs: [
<*errors.errorString | 0x95f84458c30>{
s: "timeout '15.000000' minutes exceeded during CreateOrUpdateExternalAuthAndWait for external auth external-auth in cluster ea-cluster in resource group external-auth-cluster-n6c4c4fd2ssk",
},
<context.deadlineExceededError>{},
],
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:262]: failed to apply external auth config with prefix policy None
Unexpected error:
<*fmt.wrapErrors | 0x95f8488fc80>:
failed waiting for external auth "external-auth" in resourcegroup="external-auth-cluster-n6c4c4fd2ssk" for cluster="ea-cluster" to finish creating or updating, caused by: timeout '15.000000' minutes exceeded during CreateOrUpdateExternalAuthAndWait for external auth external-auth in cluster ea-cluster in resource group external-auth-cluster-n6c4c4fd2ssk, error: context deadline exceeded
{
msg: "failed waiting for external auth \"external-auth\" in resourcegroup=\"external-auth-cluster-n6c4c4fd2ssk\" for cluster=\"ea-cluster\" to finish creating or updating, caused by: timeout '15.000000' minutes exceeded during CreateOrUpdateExternalAuthAndWait for external auth external-auth in cluster ea-cluster in resource group external-auth-cluster-n6c4c4fd2ssk, error: context deadline exceeded",
errs: [
<*errors.errorString | 0x95f84458c30>{
s: "timeout '15.000000' minutes exceeded during CreateOrUpdateExternalAuthAndWait for external auth external-auth in cluster ea-cluster in resource group external-auth-cluster-n6c4c4fd2ssk",
},
<context.deadlineExceededError>{},
],
}
occurredContributing tests (1)
Affected runs (1)
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedC...full failure pattern: ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable; hosted cluster degraded: UnavailableReplicas: [azure-cloud-controller-manager, catalog-operator, hosted-cluster-config-operator, konnectivity-agent, machine-approver, olm-operator, openshift-apiserver, packageserver]; provider Microsoft.RedHatOpenShift Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s) | e2e | 1 | 3.70%1 of 27 job runs affected | IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s) | Show trend detailsOct 12: 1 · Oct 13: 0 · Oct 14: 0 · Oct 15: 0 · Oct 16: 0 · Oct 17: 0 · Oct 18: 0 · Oct 19: 0 · Oct 20: 0 · Oct 21: 0 · Oct 22: 0 · Oct 23: 0 · Oct 24: 0 · Oct 25: 0 | none | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-nightly-5-0-4c2bdskg9txb/cluster-nightly-5-0-44gd6d should provision
Unexpected error:
<*fmt.wrapError | 0xc4ca110a7e0>:
failed to create HCP cluster cluster-nightly-5-0-44gd6d: failed waiting for cluster="cluster-nightly-5-0-44gd6d" in resourcegroup="rg-nightly-5-0-4c2bdskg9txb" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/bf9bc1de-04dc-49de-bcdd-655adcca76f1
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/bf9bc1de-04dc-49de-bcdd-655adcca76f1",
"name": "bf9bc1de-04dc-49de-bcdd-655adcca76f1",
"status": "Failed",
"startTime": "2026-10-12T23:27:12.849511271Z",
"endTime": "2026-10-12T23:56:15.713757928Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: csi-snapshot-controller-operator, catalog-operator, certified-operators-catalog, openshift-route-controller-manager, cluster-policy-controller, dns-operator, cluster-image-registry-operator, packageserver, hosted-cluster-config-operator, azure-cloud-controller-manager, cluster-network-operator, ingress-operator, konnectivity-agent, cluster-version-operator, cluster-storage-operator, community-operators-catalog, olm-operator, redhat-operators-catalog, machine-approver, openshift-controller-manager, cluster-node-tuning-operator; hosted cluster degraded: UnavailableReplicas: [azure-cloud-controller-manager deployment has 1 unavailable replicas, catalog-operator deployment has 1 unavailable replicas, hosted-cluster-config-operator deployment has 1 unavailable replicas, konnectivity-agent deployment has 1 unavailable replicas, machine-approver deployment has 1 unavailable replicas, olm-operator deployment has 1 unavailable replicas, openshift-apiserver deployment has 1 unavailable replicas, packageserver deployment has 3 unavailable replicas]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/bf9bc1de-04dc-49de-bcdd-655adcca76f1
correlation ID: 37b4d83d-eab7-42e6-8d94-8b862a090192
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-nightly-5-0-4c2bdskg9txb/cluster-nightly-5-0-44gd6d should provision
Unexpected error:
<*fmt.wrapError | 0xc4ca110a7e0>:
failed to create HCP cluster cluster-nightly-5-0-44gd6d: failed waiting for cluster="cluster-nightly-5-0-44gd6d" in resourcegroup="rg-nightly-5-0-4c2bdskg9txb" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/bf9bc1de-04dc-49de-bcdd-655adcca76f1
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/bf9bc1de-04dc-49de-bcdd-655adcca76f1",
"name": "bf9bc1de-04dc-49de-bcdd-655adcca76f1",
"status": "Failed",
"startTime": "2026-10-12T23:27:12.849511271Z",
"endTime": "2026-10-12T23:56:15.713757928Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: csi-snapshot-controller-operator, catalog-operator, certified-operators-catalog, openshift-route-controller-manager, cluster-policy-controller, dns-operator, cluster-image-registry-operator, packageserver, hosted-cluster-config-operator, azure-cloud-controller-manager, cluster-network-operator, ingress-operator, konnectivity-agent, cluster-version-operator, cluster-storage-operator, community-operators-catalog, olm-operator, redhat-operators-catalog, machine-approver, openshift-controller-manager, cluster-node-tuning-operator; hosted cluster degraded: UnavailableReplicas: [azure-cloud-controller-manager deployment has 1 unavailable replicas, catalog-operator deployment has 1 unavailable replicas, hosted-cluster-config-operator deployment has 1 unavailable replicas, konnectivity-agent deployment has 1 unavailable replicas, machine-approver deployment has 1 unavailable replicas, olm-operator deployment has 1 unavailable replicas, openshift-apiserver deployment has 1 unavailable replicas, packageserver deployment has 3 unavailable replicas]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/bf9bc1de-04dc-49de-bcdd-655adcca76f1
correlation ID: 37b4d83d-eab7-42e6-8d94-8b862a090192
client request ID: unavailable
occurredContributing tests (1)
Affected runs (1)
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedC...full failure pattern: ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable; hosted cluster degraded: UnavailableReplicas: [catalog-operator, hosted-cluster-config-operator, olm-operator, openshift-apiserver, packageserver]; provider Microsoft.RedHatOpenShift Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s) | e2e | 1 | 3.70%1 of 27 job runs affected | IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s) | Show trend detailsOct 12: 0 · Oct 13: 0 · Oct 14: 1 · Oct 15: 0 · Oct 16: 0 · Oct 17: 0 · Oct 18: 0 · Oct 19: 0 · Oct 20: 0 · Oct 21: 0 · Oct 22: 0 · Oct 23: 0 · Oct 24: 0 · Oct 25: 0 | DEV, PROD | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-nightly-5-1-zxqcgtfv7nf5/cluster-nightly-5-1-2dz545 should provision
Unexpected error:
<*fmt.wrapError | 0x3ec63f648620>:
failed to create HCP cluster cluster-nightly-5-1-2dz545: failed waiting for cluster="cluster-nightly-5-1-2dz545" in resourcegroup="rg-nightly-5-1-zxqcgtfv7nf5" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/c1d8733b-0494-46c8-9bd3-0a4108ec0b5c
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/c1d8733b-0494-46c8-9bd3-0a4108ec0b5c",
"name": "c1d8733b-0494-46c8-9bd3-0a4108ec0b5c",
"status": "Failed",
"startTime": "2026-10-14T23:26:17.571795133Z",
"endTime": "2026-10-14T23:55:18.071165518Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: cluster-node-tuning-operator, ingress-operator, cluster-image-registry-operator, hosted-cluster-config-operator, cluster-storage-operator, olm-operator, redhat-operators-catalog, community-operators-catalog, certified-operators-catalog, cluster-policy-controller, catalog-operator, packageserver, openshift-controller-manager, dns-operator, openshift-route-controller-manager, cluster-version-operator, cluster-network-operator, csi-snapshot-controller-operator; hosted cluster degraded: UnavailableReplicas: [catalog-operator deployment has 1 unavailable replicas, hosted-cluster-config-operator deployment has 1 unavailable replicas, olm-operator deployment has 1 unavailable replicas, openshift-apiserver deployment has 1 unavailable replicas, packageserver deployment has 3 unavailable replicas]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/c1d8733b-0494-46c8-9bd3-0a4108ec0b5c
correlation ID: b45fa2ae-f53a-4948-b0fb-190aed3be0b8
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-nightly-5-1-zxqcgtfv7nf5/cluster-nightly-5-1-2dz545 should provision
Unexpected error:
<*fmt.wrapError | 0x3ec63f648620>:
failed to create HCP cluster cluster-nightly-5-1-2dz545: failed waiting for cluster="cluster-nightly-5-1-2dz545" in resourcegroup="rg-nightly-5-1-zxqcgtfv7nf5" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/c1d8733b-0494-46c8-9bd3-0a4108ec0b5c
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/c1d8733b-0494-46c8-9bd3-0a4108ec0b5c",
"name": "c1d8733b-0494-46c8-9bd3-0a4108ec0b5c",
"status": "Failed",
"startTime": "2026-10-14T23:26:17.571795133Z",
"endTime": "2026-10-14T23:55:18.071165518Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: cluster-node-tuning-operator, ingress-operator, cluster-image-registry-operator, hosted-cluster-config-operator, cluster-storage-operator, olm-operator, redhat-operators-catalog, community-operators-catalog, certified-operators-catalog, cluster-policy-controller, catalog-operator, packageserver, openshift-controller-manager, dns-operator, openshift-route-controller-manager, cluster-version-operator, cluster-network-operator, csi-snapshot-controller-operator; hosted cluster degraded: UnavailableReplicas: [catalog-operator deployment has 1 unavailable replicas, hosted-cluster-config-operator deployment has 1 unavailable replicas, olm-operator deployment has 1 unavailable replicas, openshift-apiserver deployment has 1 unavailable replicas, packageserver deployment has 3 unavailable replicas]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/c1d8733b-0494-46c8-9bd3-0a4108ec0b5c
correlation ID: b45fa2ae-f53a-4948-b0fb-190aed3be0b8
client request ID: unavailable
occurredContributing tests (1)
Affected runs (1)
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedC...full failure pattern: ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider]; provider Microsoft.RedHatOpenShift Signal: IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | e2e | 1 | 3.70%1 of 27 job runs affected | IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | Show trend detailsOct 12: 0 · Oct 13: 0 · Oct 14: 0 · Oct 15: 0 · Oct 16: 0 · Oct 17: 0 · Oct 18: 0 · Oct 19: 0 · Oct 20: 0 · Oct 21: 0 · Oct 22: 1 · Oct 23: 0 · Oct 24: 0 · Oct 25: 0 | DEV, INT, PROD | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/e2e/cluster_autoscaling.go:108]: failed to create HCP cluster "autoscaling-hcp-cluster" with custom autoscaling
Unexpected error:
<*fmt.wrapError | 0x3cf97cd86ae0>:
failed to create HCP cluster autoscaling-hcp-cluster: failed waiting for cluster="autoscaling-hcp-cluster" in resourcegroup="autoscaling-cluster-f4qstxjd2dn6" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/1ac5a43d-188f-41a4-873c-672951930233
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/1ac5a43d-188f-41a4-873c-672951930233",
"name": "1ac5a43d-188f-41a4-873c-672951930233",
"status": "Failed",
"startTime": "2026-10-22T02:02:55.710362817Z",
"endTime": "2026-10-22T02:32:02.510983362Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/1ac5a43d-188f-41a4-873c-672951930233
correlation ID: 9e15eb20-b5a4-4836-b8fc-ad2c0fd7a951
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_autoscaling.go:108]: failed to create HCP cluster "autoscaling-hcp-cluster" with custom autoscaling
Unexpected error:
<*fmt.wrapError | 0x3cf97cd86ae0>:
failed to create HCP cluster autoscaling-hcp-cluster: failed waiting for cluster="autoscaling-hcp-cluster" in resourcegroup="autoscaling-cluster-f4qstxjd2dn6" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/1ac5a43d-188f-41a4-873c-672951930233
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/1ac5a43d-188f-41a4-873c-672951930233",
"name": "1ac5a43d-188f-41a4-873c-672951930233",
"status": "Failed",
"startTime": "2026-10-22T02:02:55.710362817Z",
"endTime": "2026-10-22T02:32:02.510983362Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/1ac5a43d-188f-41a4-873c-672951930233
correlation ID: 9e15eb20-b5a4-4836-b8fc-ad2c0fd7a951
client request ID: unavailable
occurredContributing tests (1)
Affected runs (1)
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet complete...full failure pattern: ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version <version> is Partial (want Completed), started <duration> ago; hosted cluster degraded: UnavailableReplicas: [ignition-server-proxy]; provider Microsoft.RedHatOpenShift Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s) | e2e | 1 | 3.70%1 of 27 job runs affected | IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s) | Show trend detailsOct 12: 0 · Oct 13: 0 · Oct 14: 1 · Oct 15: 0 · Oct 16: 0 · Oct 17: 0 · Oct 18: 0 · Oct 19: 0 · Oct 20: 0 · Oct 21: 0 · Oct 22: 0 · Oct 23: 0 · Oct 24: 0 · Oct 25: 0 | DEV, PROD | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-nightly-5-0-8qt7kcrvhtlb/cluster-nightly-5-0-6wsq4v should provision
Unexpected error:
<*fmt.wrapError | 0x30f3685a760>:
failed to create HCP cluster cluster-nightly-5-0-6wsq4v: failed waiting for cluster="cluster-nightly-5-0-6wsq4v" in resourcegroup="rg-nightly-5-0-8qt7kcrvhtlb" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/70ee1e74-bffb-4dec-89fb-282756ca2270
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/70ee1e74-bffb-4dec-89fb-282756ca2270",
"name": "70ee1e74-bffb-4dec-89fb-282756ca2270",
"status": "Failed",
"startTime": "2026-10-14T23:26:07.782117382Z",
"endTime": "2026-10-14T23:55:07.903819031Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 5.0.0-0.nightly-multi-2026-10-14-152123 is Partial (want Completed), started 20m19s ago; hosted cluster degraded: UnavailableReplicas: ignition-server-proxy deployment has 1 unavailable replicas"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/70ee1e74-bffb-4dec-89fb-282756ca2270
correlation ID: e153794f-fb7c-4e3b-9441-a5dadf7b0225
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-nightly-5-0-8qt7kcrvhtlb/cluster-nightly-5-0-6wsq4v should provision
Unexpected error:
<*fmt.wrapError | 0x30f3685a760>:
failed to create HCP cluster cluster-nightly-5-0-6wsq4v: failed waiting for cluster="cluster-nightly-5-0-6wsq4v" in resourcegroup="rg-nightly-5-0-8qt7kcrvhtlb" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/70ee1e74-bffb-4dec-89fb-282756ca2270
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/70ee1e74-bffb-4dec-89fb-282756ca2270",
"name": "70ee1e74-bffb-4dec-89fb-282756ca2270",
"status": "Failed",
"startTime": "2026-10-14T23:26:07.782117382Z",
"endTime": "2026-10-14T23:55:07.903819031Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 5.0.0-0.nightly-multi-2026-10-14-152123 is Partial (want Completed), started 20m19s ago; hosted cluster degraded: UnavailableReplicas: ignition-server-proxy deployment has 1 unavailable replicas"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/70ee1e74-bffb-4dec-89fb-282756ca2270
correlation ID: e153794f-fb7c-4e3b-9441-a5dadf7b0225
client request ID: unavailable
occurredContributing tests (1)
Affected runs (1)
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
ERROR CODE: InternalServerError; detail message cluster deletion did not complete before the deadline; [resourceTeardown] tearing down pod network instances, pod networks, namespa...full failure pattern: ERROR CODE: InternalServerError; detail message cluster deletion did not complete before the deadline; [resourceTeardown] tearing down pod network instances, pod networks, namespaces; [descendantResources] remaining resources: applydesires, serviceProviderClusters; provider Microsoft.RedHatOpenShift Signal: IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | e2e | 1 | 3.70%1 of 27 job runs affected | IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | Show trend detailsOct 12: 0 · Oct 13: 0 · Oct 14: 0 · Oct 15: 0 · Oct 16: 0 · Oct 17: 0 · Oct 18: 0 · Oct 19: 0 · Oct 20: 0 · Oct 21: 0 · Oct 22: 0 · Oct 23: 0 · Oct 24: 1 · Oct 25: 0 | DEV, PROD | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:310]: Unexpected error:
<*errors.joinError | 0x1ea319f5c1c8>:
resource group "rg-nightly-5-0-zf68j445hrrz": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cluster-nightly-5-0-md89qh" in resourcegroup="rg-nightly-5-0-zf68j445hrrz" to finish deleting: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/1895187e-5536-47a1-a44d-a9b6da901fa1
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: InternalServerError
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/1895187e-5536-47a1-a44d-a9b6da901fa1",
"name": "1895187e-5536-47a1-a44d-a9b6da901fa1",
"status": "Failed",
"startTime": "2026-10-25T00:01:49.583919819Z",
"endTime": "2026-10-25T00:25:49.807200592Z",
"error": {
"code": "InternalServerError",
"message": "cluster deletion did not complete before the deadline; [descendantResources] remaining resources: 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/serviceProviderClusters, 3 microsoft.redhatopenshift/hcpopenshiftclusters/applydesires; [resourceTeardown] tearing down pod network instances (1), pod networks (1), namespaces (2)"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/1895187e-5536-47a1-a44d-a9b6da901fa1
correlation ID: 78c885d0-28f3-4f61-bdbb-0c2134f660b2
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:310]: Unexpected error:
<*errors.joinError | 0x1ea319f5c1c8>:
resource group "rg-nightly-5-0-zf68j445hrrz": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cluster-nightly-5-0-md89qh" in resourcegroup="rg-nightly-5-0-zf68j445hrrz" to finish deleting: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/1895187e-5536-47a1-a44d-a9b6da901fa1
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: InternalServerError
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/1895187e-5536-47a1-a44d-a9b6da901fa1",
"name": "1895187e-5536-47a1-a44d-a9b6da901fa1",
"status": "Failed",
"startTime": "2026-10-25T00:01:49.583919819Z",
"endTime": "2026-10-25T00:25:49.807200592Z",
"error": {
"code": "InternalServerError",
"message": "cluster deletion did not complete before the deadline; [descendantResources] remaining resources: 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/serviceProviderClusters, 3 microsoft.redhatopenshift/hcpopenshiftclusters/applydesires; [resourceTeardown] tearing down pod network instances (1), pod networks (1), namespaces (2)"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/1895187e-5536-47a1-a44d-a9b6da901fa1
correlation ID: 78c885d0-28f3-4f61-bdbb-0c2134f660b2
client request ID: unavailable
occurredContributing tests (1)
Affected runs (1)
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Unauthorizedfull failure pattern: VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Unauthorized Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s) | e2e | 1 | 3.70%1 of 27 job runs affected | IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s) | Show trend detailsOct 12: 1 · Oct 13: 0 · Oct 14: 0 · Oct 15: 0 · Oct 16: 0 · Oct 17: 0 · Oct 18: 0 · Oct 19: 0 · Oct 20: 0 · Oct 21: 0 · Oct 22: 0 · Oct 23: 0 · Oct 24: 0 · Oct 25: 0 | PROD | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/e2e/simple_negative_cases.go:124]: cluster negative-tests-cluster is not viable
Unexpected error:
<*errors.joinError | 0x3d800eb17200>:
VerifyBasicAccess failed: failed to list services: Unauthorized
VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Unauthorized
{
errs: [
<*fmt.wrapError | 0x3d800f297260>{
msg: "VerifyBasicAccess failed: failed to list services: Unauthorized",
err: <*fmt.wrapError | 0x3d800f297240>{
msg: "failed to list services: Unauthorized",
err: <*errors.StatusError | 0x3d800e191e00>{
ErrStatus: {
TypeMeta: {Kind: "", APIVersion: ""},
ListMeta: {
SelfLink: "",
ResourceVersion: "",
Continue: "",
RemainingItemCount: nil,
ShardInfo: nil,
},
Status: "Failure",
Message: "Unauthorized",
Reason: "Unauthorized",
Details: nil,
Code: 401,
},
},
},
},
<*fmt.wrapError | 0x3d800e204000>{
msg: "VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Unauthorized",
err: <*fmt.wrapError | 0x3d800eb05ee0>{
msg: "failed to list all APIServices: Unauthorized",
err: <*errors.StatusError | 0x3d800e0fab40>{
ErrStatus: {
TypeMeta: {Kind: "", APIVersion: ""},
ListMeta: {
SelfLink: "",
ResourceVersion: "",
Continue: "",
RemainingItemCount: nil,
ShardInfo: nil,
},
Status: "Failure",
Message: "Unauthorized",
Reason: "Unauthorized",
Details: nil,
Code: 401,
},
},
},
},
],
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/simple_negative_cases.go:124]: cluster negative-tests-cluster is not viable
Unexpected error:
<*errors.joinError | 0x3d800eb17200>:
VerifyBasicAccess failed: failed to list services: Unauthorized
VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Unauthorized
{
errs: [
<*fmt.wrapError | 0x3d800f297260>{
msg: "VerifyBasicAccess failed: failed to list services: Unauthorized",
err: <*fmt.wrapError | 0x3d800f297240>{
msg: "failed to list services: Unauthorized",
err: <*errors.StatusError | 0x3d800e191e00>{
ErrStatus: {
TypeMeta: {Kind: "", APIVersion: ""},
ListMeta: {
SelfLink: "",
ResourceVersion: "",
Continue: "",
RemainingItemCount: nil,
ShardInfo: nil,
},
Status: "Failure",
Message: "Unauthorized",
Reason: "Unauthorized",
Details: nil,
Code: 401,
},
},
},
},
<*fmt.wrapError | 0x3d800e204000>{
msg: "VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Unauthorized",
err: <*fmt.wrapError | 0x3d800eb05ee0>{
msg: "failed to list all APIServices: Unauthorized",
err: <*errors.StatusError | 0x3d800e0fab40>{
ErrStatus: {
TypeMeta: {Kind: "", APIVersion: ""},
ListMeta: {
SelfLink: "",
ResourceVersion: "",
Continue: "",
RemainingItemCount: nil,
ShardInfo: nil,
},
Status: "Failure",
Message: "Unauthorized",
Reason: "Unauthorized",
Details: nil,
Code: 401,
},
},
},
},
],
}
occurredContributing tests (1)
Affected runs (1)
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
VerifyCustomIngressCertificate timed out after 10m0full failure pattern: VerifyCustomIngressCertificate timed out after 10m0 Signal: IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | e2e | 1 | 3.70%1 of 27 job runs affected | IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | Show trend detailsOct 12: 0 · Oct 13: 0 · Oct 14: 0 · Oct 15: 0 · Oct 16: 0 · Oct 17: 0 · Oct 18: 0 · Oct 19: 0 · Oct 20: 0 · Oct 21: 0 · Oct 22: 0 · Oct 23: 0 · Oct 24: 0 · Oct 25: 1 | none | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:525]: sample application must serve the exact customer-specified certificate with a valid Let's Encrypt chain
Expected success, but got an error:
<*fmt.wrapError | 0x36a1186625c0>:
VerifyCustomIngressCertificate timed out after 10m0.003s: ingress route must serve HTTPS with the expected certificate: Get "https://agnhost-e2e-sample-app-l49xr.apps.aro.ingress-cert-le.0o3m.uksouth.aroapp-hcp-stg.azure.com": tls: failed to verify certificate: x509: certificate signed by unknown authority
{
msg: "VerifyCustomIngressCertificate timed out after 10m0.003s: ingress route must serve HTTPS with the expected certificate: Get \"https://agnhost-e2e-sample-app-l49xr.apps.aro.ingress-cert-le.0o3m.uksouth.aroapp-hcp-stg.azure.com\": tls: failed to verify certificate: x509: certificate signed by unknown authority",
err: <*fmt.wrapError | 0x36a1186623e0>{
msg: "ingress route must serve HTTPS with the expected certificate: Get \"https://agnhost-e2e-sample-app-l49xr.apps.aro.ingress-cert-le.0o3m.uksouth.aroapp-hcp-stg.azure.com\": tls: failed to verify certificate: x509: certificate signed by unknown authority",
err: <*url.Error | 0x36a118d74840>{
Op: "Get",
URL: "https://agnhost-e2e-sample-app-l49xr.apps.aro.ingress-cert-le.0o3m.uksouth.aroapp-hcp-stg.azure.com",
Err: <*tls.CertificateVerificationError | 0x36a118d74810>{
UnverifiedCertificates: [
{
Raw: [48, 130, 9, 19, 48, 130, 6, 251, 160, 3, 2, 1, 2, 2, 19, 73, 1, 74, 89, 119, 169, 186, 32, 59, 56, 235, 138, 87, 0, 0, 1, 74, 89, 119, 48, 13, 6, 9, 42, 134, 72, 134, 247, 13, 1, 1, 12, 5, 0, 48, 87, 49, 11, 48, 9, 6, 3, 85, 4, 6, 19, 2, 85, 83, 49, 30, 48, 28, 6, 3, 85, 4, 10, 19, 21, 77, 105, 99, 114, 111, 115, 111, 102, 116, 32, 67, 111, 114, 112, 111, 114, 97, 116, 105, 111, 110, 49, 40, 48, 38, 6, 3, 85, 4, 3, 19, 31, 77, 105, 99, 114, 111, 115, 111, 102, 116, 32, 84, 76, 83, 32, 71, 50, 32, 82, 83, 65, 32, 67, 65, 32, 79, 67, 83, 80, 32, 49, 48, 48, 30, 23, 13, 50, 54, 49, 48, 50, 53, 48, 49, 53, 54, 49, 48, 90, 23, 13, 50, 55, 48, 50, 48, 50, 48, 48, 53, 54, 49, 48, 90, 48, 120, 49, 11, 48, 9, 6, 3, 85, 4, 6, 19, 2, 85, 83, 49, 11, 48, 9, 6, 3, 85, 4, 8, 19, 2, 87, 65, 49, 16, 48, 14, 6, 3, 85, 4, 7, 19, 7, 82, 101, 100, 109, 111, 110, 100, 49, 30, 48, 28, 6, 3, 85, 4, 10, 19, 21, 77, 105, 99, 114, 111, 115, 111, 102, 116, 32, 67, 111, 114, 112, 111, 114, 97, 116, 105, 111, 110, 49, 42, 48, 40, 6, 3, 85, 4, 3, 19, 33, 114, 101, 115, 101, 114, 118, 101, 100, 46, 97, 114, 111, 97, 112, 112, 45, 104, 99, 112, 45, 115, 116, 103, 46, 97, 122, 117, 114, 101, 46, 99, 111, 109, 48, 130, 1, 34, 48, 13, 6, 9, 42, 134, 72, 134, 247, 13, 1, 1, 1, 5, 0, 3, 130, 1, 15, 0, 48, 130, 1, 10, 2, 130, 1, 1, 0, 187, 121, 183, 117, 223, 172, 129, 21, 158, 33, 40, 228, 200, 112, 100, 153, 170, 116, 250, 158, 245, 201, 135, 0, 237, 144, 31, 12, 163, 250, 168, 50, 92, 5, 220, 113, 137, 143, 3, 193, 167, 220, 193, 78, 195, 131, 29, 107, 67, 52, 86, 236, 30, 39, 30, 107, 206, 213, 234, 58, 181, 33, 44, 227, 147, 157, 23, 219, 20, 127, 20, 235, 246, 77, 135, 5, 240, 135, 231, 58, 154, 101, 19, 42, 160, 128, 73, 94, 79, 155, 62, 172, 213, 53, 57, 2, 149, 60, 188, 169, 233, 44, 216, 105, 66, 159, 148, 70, 178, 113, 177, 128, 123, 151, 10, 7, 254, 171, 205, 105, 34, 145, 93, 19, 218, 220, 219, 47, 129, 42, 85, 247, 229, 7, 118, 48, 182, 125, 84, 45, 187, 16, 138, 130, 74, 176, 14, 70, 229, 120, 108, 212, 172, 177, 174, 141, 161, 231, 167, 5, 108, 191, 229, 39, 27, 122, 242, 198, 32, 78, 125, 8, 168, 234, 200, 195, 10, 209, 101, 183, 248, 26, 10, 82, 78, 47, 196, 44, 55, 199, 207, 209, 35, 65, 164, 44, 162, 97, 81, 36, 122, 188, 105, 82, 214, 39, 252, 78, 115, 249, 159, 36, 212, 167, 6, 72, 63, 56, 220, 88, 50, 162, 50, 174, 185, 196, 246, 208, 84, 90, 212, 218, 52, 118, 184, 101, 201, 132, 48, 29, 45, 113, 80, 81, 157, 255, 235, 218, 49, 139, 40, 124, 111, 147, 143, 201, 2, 3, 1, 0, 1, 163, 130, 4, 181, 48, 130, 4, 177, 48, 130, 1, 4, 6, 10, 43, 6, 1, 4, 1, 214, 121, 2, 4, 2, 4, 129, 245, 4, 129, 242, 0, 240, 0, 118, 0, 68, 194, 189, 12, 233, 20, 14, 100, 165, 201, 74, 1, 147, 10, 90, 161, 187, 53, 151, 14, 0, 238, 17, 22, 137, 104, 42, 28, 68, 215, 181, 102, 0, 0, 1, 161, 112, 206, 239, 163, 0, 0, 4, 3, 0, 71, 48, 69, 2, 33, 0, 172, 23, 89, 180, 174, 255, 10, 75, 120, 126, 2, 114, 141, 52, 53, 9, 237, 84, 209, 98, 89, 54, 179, 9, 54, 118, 245, 178, 107, 207, 5, 69, 2, 32, 89, 15, 78, 239, 171, 216, 123, 169, 212, 117, 5, 12, 137, 15, 63, 83, 41, 96, 223, 14, 136, 244, 40, 31, 142, 25, 69, 157, 200, 47, 205, 167, 0, 118, 0, 0, 26, 93, 26, 28, 45, 147, 117, 182, 72, 85, 120, 248, 47, 113, 161, 174, 110, 239, 57, 125, 41, 124, 138, 227, 21, 123, 202, 222, 225, 160, 30, 0, 0, 1, 161, 112, 206, 240, 23, 0, 0, 4, 3, 0, 71, 48, 69, 2, 33, 0, 209, 5, 94, 127, 36, 105, 13, 219, 166, 20, 49, 161, 208, 24, 148, 233, 208, 121, 79, 240, 195, 85, 152, 81, 108, 234, 202, 116, 68, 189, 194, 57, 2, 32, 65, 141, 139, 86, 66, 203, 70, 108, 26, 159, 237, 96, 32, 237, 241, 53, 230, 112, 171, 119, 90, 74, 10, 14, 99, 1, 225, 78, 245, 158, 165, 135, 48, 27, 6, 9, 43, 6, 1, 4, 1, 130, 55, 21, 10, 4, 14, 48, 12, 48, 10, 6, 8, 43, 6, 1, 5, 5, 7, 3, 1, 48, 60, 6, 9, 43, 6, 1, 4, 1, 130, 55, 21, 7, 4, 47, 48, 45, 6, 37, 43, 6, 1, 4, 1, 130, 55, 21, 8, 135, 189, 215, 27, 129, 231, 235, 70, 130, 129, 157, 46, 142, 208, 12, 135, 240, 218, 29, 93, 133, 246, 239, 42, 129, 232, 226, 53, 2, 1, 100, 2, 1, 39, 48, 130, 1, 11, 6, 8, 43, 6, 1, 5, 5, 7, 1, 1, 4, 129, 254, 48, 129, 251, 48, 97, 6, 8, 43, 6, 1, 5, 5, 7, 48, 2, 134, 85, 104, 116, 116, 112, 58, 47, 47, 119, 119, 119, 46, 109, 105, 99, 114, 111, 115, 111, 102, 116, 46, 99, 111, 109, 47, 112, 107, 105, 111, 112, 115, 47, 99, 101, 114, 116, 115, 47, 77, 105, 99, 114, 111, 115, 111, 102, 116, 37, 50, 48, 84, 76, 83, 37, 50, 48, 71, 50, 37, 50, 48, 82, 83, 65, 37, 50, 48, 67, 65, 37, 50, 48, 79, 67, 83, 80, 37, 50, 48, 49, 48, 46, 99, 114, 116, 48, 103, 6, 8, 43, 6, 1, 5, 5, 7, 48, 2, 134, 91, 104, 116, 116, 112, 58, 47, 47, 99, 97, 105, 115, 115, 117, 101, 114, 115, 46, 109, 105, 99, 114, 111, 115, 111, 102, 116, 46, 99, 111, 109, 47, 112, 107, 105, 111, 112, 115, 47, 99, 101, 114, 116, 115, 47, 77, 105, 99, 114, 111, 115, 111, 102, 116, 37, 50, 48, 84, 76, 83, 37, 50, 48, 71, 50, 37, 50, 48, 82, 83, 65, 37, 50, 48, 67, 65, 37, 50, 48, 79, 67, 83, 80, 37, 50, 48, 49, 48, 46, 99, 114, 116, 48, 45, 6, 8, 43, 6, 1, 5, 5, 7, 48, 1, 134, 33, 104, 116, 116, 112, 58, 47, 47, 111, 110, 101, 111, 99, 115, 112, 46, 109, 105, 99, 114, 111, 115, 111, 102, 116, 46, 99, 111, 109, 47, 111, 99, 115, 112, 48, 29, 6, 3, 85, 29, 14, 4, 22, 4, 20, 138, 66, 241, 22, 211, 0, 191, 156, 21, 196, 84, 38, 230, 58, 211, 190, 115, 187, 163, 122, 48, 14, 6, 3, 85, 29, 15, 1, 1, 255, 4, 4, 3, 2, 5, 160, 48, 110, 6, 3, 85, 29, 17, 4, 103, 48, 101, 130, 64, 42, 46, 97, 112, 112, 115, 46, 97, 114, 111, 46, 105, 110, 103, 114, 101, 115, 115, 45, 99, 101, 114, 116, 45, 108, 101, 46, 48, 111, 51, 109, 46, 117, 107, 115, 111, 117, 116, 104, 46, 97, 114, 111, 97, 112, 112, 45, 104, 99, 112, 45, 115, 116, 103, 46, 97, 122, 117, 114, 101, 46, 99, 111, 109, 130, 33, 114, 101, 115, 101, 114, 118, 101, 100, 46, 97, 114, 111, 97, 112, 112, 45, 104, 99, 112, 45, 115, 116, 103, 46, 97, 122, 117, 114, 101, 46, 99, 111, 109, 48, 12, 6, 3, 85, 29, 19, 1, 1, 255, 4, 2, 48, 0, 48, 129, 241, 6, 3, 85, 29, 31, 4, 129, 233, 48, 129, 230, 48, 129, 227, 160, 129, 224, 160, 129, 221, 134, 108, 104, 116, 116, 112, 58, 47, 47, 119, 119, 119, 46, 109, 105, 99, 114, 111, 115, 111, 102, 116, 46, 99, 111, 109, 47, 112, 107, 105, 111, 112, 115, 47, 99, 114, 108, 47, 112, 97, 114, 116, 105, 116, 105, 111, 110, 47, 77, 105, 99, 114, 111, 115, 111, 102, 116, 37, 50, 48, 84, 76, 83, 37, 50, 48, 71, 50, 37, 50, 48, 82, 83, 65, 37, 50, 48, 67, 65, 37, 50, 48, 79, 67, 83, 80, 37, 50, 48, 49, 48, 95, 80, 97, 114, 116, 105, 116, 105, 111, 110, 48, 48, 48, 57, 57, 46, 99, 114, 108, 134, 109, 104, 116, 116, 112, 58, 47, 47, 99, 114, 108, 50, 46, 109, 105, 99, 114, 111, 115, 111, 102, 116, 46, 99, 111, 109, 47, 112, 107, 105, 111, 112, 115, 47, 99, 114, 108, 47, 112, 97, 114, 116, 105, 116, 105, 111, 110, 47, 77, 105, 99, 114, 111, 115, 111, 102, 116, 37, 50, 48, 84, 76, 83, 37, 50, 48, 71, 50, 37, 50, 48, 82, 83, 65, 37, 50, 48, 67, 65, 37, 50, 48, 79, 67, 83, 80, 37, 50, 48, 49, 48, 95, 80, 97, 114, 116, 105, 116, 105, 111, 110, 48, 48, 48, 57, 57, 46, 99, 114, 108, 48, 102, 6, 3, 85, 29, 32, 4, 95, 48, 93, 48, 8, 6, 6, 103, 129, 12, 1, 2, 2, 48, 81, 6, 12, 43, 6, 1, 4, 1, 130, 55, 76, 131, 125, 1, 1, 48, 65, 48, 63, 6, 8, 43, 6, 1, 5, 5, 7, 2, 1, 22, 51, 104, 116, 116, 112, 58, 47, 47, 119, 119, 119, 46, 109, 105, 99, 114, 111, 115, 111, 102, 116, 46, 99, 111, 109, 47, 112, 107, 105, 111, 112, 115, 47, 68, 111, 99, 115, 47, 82, 101, 112, 111, 115, 105, 116, 111, 114, 121, 46, 104, 116, 109, 48, 31, 6, 3, 85, 29, 35, 4, 24, 48, 22, 128, 20, 208, 76, 131, 209, 142, 113, 46, 61, 52, 177, 255, 155, 121, 44, 142, 116, 212, 226, 120, 6, 48, 19, 6, 3, 85, 29, 37, 4, 12, 48, 10, 6, 8, 43, 6, 1, 5, 5, 7, 3, 1, 48, 13, 6, 9, 42, 134, 72, 134, 247, 13, 1, 1, 12, 5, 0, 3, 130, 2, 1, 0, 127, 14, 76, 119, 16, 30, 113, 75, 136, 158, 234, 180, 48, 26, 140, 223, 208, 88, 116, 11, 139, 240, 141, 40, 46, 35, 73, 190, 186, 171, 118, 120, 99, 118, 88, 110, 181, 203, 42, 120, 241, 109, 206, 58, 1, 63, 144, 160, 72, 2, 211, 217, 25, 137, 246, 187, 186, 49, 138, 232, 122, 82, 3, 136, 22, 157, 87, 191, 86, 125, 221, 231, 158, 0, 204, 174, 39, 65, 179, 160, 12, 205, 190, 69, 207, 65, 139, 215, 99, 239, 19, 51, 250, 33, 66, 49, 250, 3, 217, 217, 178, 148, 58, 186, 139, 102, 86, 205, 171, 36, 144, 223, 188, 149, 248, 1, 103, 232, 163, 162, 100, 23, 166, 66, 215, 67, 190, 124, 120, 217, 106, 101, 246, 252, 8, 144, 121, 98, 136, 193, 124, 213, 80, 149, 73, 94, 215, 24, 197, 138, 7, 143, 140, 229, 172, 93, 155, 162, 211, 4, 222, 43, 210, 38, 230, 222, 23, 202, 214, 208, 103, 147, 145, 86, 53, 202, 100, 48, 50, 41, 5, 116, 80, 254, 160, 164, 22, 192, 242, 55, 78, 107, 236, 232, 202, 76, 47, 127, 5, 241, 21, 79, 8, 239, 16, 12, 106, 11, 200, 233, 173, 167, 160, 195, 61, 159, 42, 183, 223, 149, 235, 138, 251, 54, 65, 95, 191, 206, 35, 254, 15, 224, 194, 106, 185, 99, 23, 156, 154, 222, 84, 145, 1, 212, 126, 42, 74, 254, 71, 27, 110, 6, 83, 252, 190, 60, 29, 24, 98, 214, 226, 88, 108, 117, 86, 233, 3, 27, 205, 3, 203, 36, 3, 175, 73, 206, 16, 65, 117, 125, 162, 73, 22, 207, 82, 232, 237, 197, 40, 139, 79, 103, 210, 165, 182, 151, 205, 15, 253, 195, 116, 121, 178, 103, 118, 187, 165, 173, 124, 138, 96, 201, 232, 236, 212, 18, 179, 244, 160, 183, 40, 12, 30, 207, 237, 36, 120, 26, 57, 114, 158, 34, 152, 93, 132, 52, 124, 112, 222, 90, 37, 164, 188, 88, 53, 52, 110, 127, 166, 176, 30, 184, 129, 236, 40, 151, 237, 246, 44, 182, 131, 71, 47, 162, 122, 195, 81, 3, 7, 197, 200, 200, 48, 29, 56, 138, 158, 6, 33, 41, 210, 36, 197, 209, 154, 188, 46, 232, 167, 27, 177, 139, 150, 154, 50, 110, 13, 89, 233, 181, 191, 72, 106, 53, 174, 105, 231, 108, 61, 64, 154, 240, 69, 47, 220, 37, 46, 170, 187, 63, 76, 114, 220, 25, 31, 218, 164, 20, 35, 139, 252, 139, 174, 8, 42, 39, 150, 249, 226, 25, 42, 115, 161, 172, 59, 218, 77, 27, 104, 120, 74, 137, 124, 201, 198, 130, 184, 80, 219, 176, 188, 97, 76, 79, 77, 137, 160, 247, 211, 155, 231, 107, 56, 112, 147, 133, 216, 183, 201, 229, 98, 78, 26, 232, 122, 160, 50, 149, 73, 205, 59, 201, 173, 34, 22, 180, 220, 49, 209, 56, 199, 177, 177, 107, 162, 228, 222, 80, 57, 112, 238, 205],
RawTBSCertificate: [48, 130, 6, 251, 160, 3, 2, 1, 2, 2, 19, 73, 1, 74, 89, 119, 169, 186, 32, 59, 56, 235, 138, 87, 0, 0, 1, 74, 89, 119, 48, 13, 6, 9, 42, 134, 72, 134, 247, 13, 1, 1, 12, 5, 0, 48, 87, 49, 11, 48, 9, 6, 3, 85, 4, 6, 19, 2, 85, 83, 49, 30, 48, 28, 6, 3, 85, 4, 10, 19, 21, 77, 105, 99, 114, 111, 115, 111, 102, 116, 32, 67, 111, 114, 112, 111, 114, 97, 116, 105, 111, 110, 49, 40, 48, 38, 6, 3, 85, 4, 3, 19, 31, 77, 105, 99, 114, 111, 115, 111, 102, 116, 32, 84, 76, 83, 32, 71, 50, 32, 82, 83, 65, 32, 67, 65, 32, 79, 67, 83, 80, 32, 49, 48, 48, 30, 23, 13, 50, 54, 49, 48, 50, 53, 48, 49, 53, 54, 49, 48, 90, 23, 13, 50, 55, 48, 50, 48, 50, 48, 48, 53, 54, 49, 48, 90, 48, 120, 49, 11, 48, 9, 6, 3, 85, 4, 6, 19, 2, 85, 83, 49, 11, 48, 9, 6, 3, 85, 4, 8, 19, 2, 87, 65, 49, 16, 48, 14, 6, 3, 85, 4, 7, 19, 7, 82, 101, 100, 109, 111, 110, 100, 49, 30, 48, 28, 6, 3, 85, 4, 10, 19, 21, 77, 105, 99, 114, 111, 115, 111, 102, 116, 32, 67, 111, 114, 112, 111, 114, 97, 116, 105, 111, 110, 49, 42, 48, 40, 6, 3, 85, 4, 3, 19, 33, 114, 101, 115, 101, 114, 118, 101, 100, 46, 97, 114, 111, 97, 112, 112, 45, 104, 99, 112, 45, 115, 116, 103, 46, 97, 122, 117, 114, 101, 46, 99, 111, 109, 48, 130, 1, 34, 48, 13, 6, 9, 42, 134, 72, 134, 247, 13, 1, 1, 1, 5, 0, 3, 130, 1, 15, 0, 48, 130, 1, 10, 2, 130, 1, 1, 0, 187, 121, 183, 117, 223, 172, 129, 21, 158, 33, 40, 228, 200, 112, 100, 153, 170, 116, 250, 158, 245, 201, 135, 0, 237, 144, 31, 12, 163, 250, 168, 50, 92, 5, 220, 113, 137, 143, 3, 193, 167, 220, 193, 78, 195, 131, 29, 107, 67, 52, 86, 236, 30, 39, 30, 107, 206, 213, 234, 58, 181, 33, 44, 227, 147, 157, 23, 219, 20, 127, 20, 235, 246, 77, 135, 5, 240, 135, 231, 58, 154, 101, 19, 42, 160, 128, 73, 94, 79, 155, 62, 172, 213, 53, 57, 2, 149, 60, 188, 169, 233, 44, 216, 105, 66, 159, 148, 70, 178, 113, 177, 128, 123, 151, 10, 7, 254, 171, 205, 105, 34, 145, 93, 19, 218, 220, 219, 47, 129, 42, 85, 247, 229, 7, 118, 48, 182, 125, 84, 45, 187, 16, 138, 130, 74, 176, 14, 70, 229, 120, 108, 212, 172, 177, 174, 141, 161, 231, 167, 5, 108, 191, 229, 39, 27, 122, 242, 198, 32, 78, 125, 8, 168, 234, 200, 195, 10, 209, 101, 183, 248, 26, 10, 82, 78, 47, 196, 44, 55, 199, 207, 209, 35, 65, 164, 44, 162, 97, 81, 36, 122, 188, 105, 82, 214, 39, 252, 78, 115, 249, 159, 36, 212, 167, 6, 72, 63, 56, 220, 88, 50, 162, 50, 174, 185, 196, 246, 208, 84, 90, 212, 218, 52, 118, 184, 101, 201, 132, 48, 29, 45, 113, 80, 81, 157, 255, 235, 218, 49, 139, 40, 124, 111, 147, 143, 201, 2, 3, 1, 0, 1, 163, 130, 4, 181, 48, 130, 4, 177, 48, 130, 1, 4, 6, 10, 43, 6, 1, 4, 1, 214, 121, 2, 4, 2, 4, 129, 245, 4, 129, 242, 0, 240, 0, 118, 0, 68, 194, 189, 12, 233, 20, 14, 100, 165, 201, 74, 1, 147, 10, 90, 161, 187, 53, 151, 14, 0, 238, 17, 22, 137, 104, 42, 28, 68, 215, 181, 102, 0, 0, 1, 161, 112, 206, 239, 163, 0, 0, 4, 3, 0, 71, 48, 69, 2, 33, 0, 172, 23, 89, 180, 174, 255, 10, 75, 120, 126, 2, 114, 141, 52, 53, 9, 237, 84, 209, 98, 89, 54, 179, 9, 54, 118, 245, 178, 107, 207, 5, 69, 2, 32, 89, 15, 78, 239, 171, 216, 123, 169, 212, 117, 5, 12, 137, 15, 63, 83, 41, 96, 223, 14, 136, 244, 40, 31, 142, 25, 69, 157, 200, 47, 205, 167, 0, 118, 0, 0, 26, 93, 26, 28, 45, 147, 117, 182, 72, 85, 120, 248, 47, 113, 161, 174, 110, 239, 57, 125, 41, 124, 138, 227, 21, 123, 202, 222, 225, 160, 30, 0, 0, 1, 161, 112, 206, 240, 23, 0, 0, 4, 3, 0, 71, 48, 69, 2, 33, 0, 209, 5, 94, 127, 36, 105, 13, 219, 166, 20, 49, 161, 208, 24, 148, 233, 208, 121, 79, 240, 195, 85, 152, 81, 108, 234, 202, 116, 68, 189, 194, 57, 2, 32, 65, 141, 139, 86, 66, 203, 70, 108, 26, 159, 237, 96, 32, 237, 241, 53, 230, 112, 171, 119, 90, 74, 10, 14, 99, 1, 225, 78, 245, 158, 165, 135, 48, 27, 6, 9, 43, 6, 1, 4, 1, 130, 55, 21, 10, 4, 14, 48, 12, 48, 10, 6, 8, 43, 6, 1, 5, 5, 7, 3, 1, 48, 60, 6, 9, 43, 6, 1, 4, 1, 130, 55, 21, 7, 4, 47, 48, 45, 6, 37, 43, 6, 1, 4, 1, 130, 55, 21, 8, 135, 189, 215, 27, 129, 231, 235, 70, 130, 129, 157, 46, 142, 208, 12, 135, 240, 218, 29, 93, 133, 246, 239, 42, 129, 232, 226, 53, 2, 1, 100, 2, 1, 39, 48, 130, 1, 11, 6, 8, 43, 6, 1, 5, 5, 7, 1, 1, 4, 129, 254, 48, 129, 251, 48, 97, 6, 8, 43, 6, 1, 5, 5, 7, 48, 2, 134, 85, 104, 116, 116, 112, 58, 47, 47, 119, 119, 119, 46, 109, 105, 99, 114, 111, 115, 111, 102, 116, 46, 99, 111, 109, 47, 112, 107, 105, 111, 112, 115, 47, 99, 101, 114, 116, 115, 47, 77, 105, 99, 114, 111, 115, 111, 102, 116, 37, 50, 48, 84, 76, 83, 37, 50, 48, 71, 50, 37, 50, 48, 82, 83, 65, 37, 50, 48, 67, 65, 37, 50, 48, 79, 67, 83, 80, 37, 50, 48, 49, 48, 46, 99, 114, 116, 48, 103, 6, 8, 43, 6, 1, 5, 5, 7, 48, 2, 134, 91, 104, 116, 116, 112, 58, 47, 47, 99, 97, 105, 115, 115, 117, 101, 114, 115, 46, 109, 105, 99, 114, 111, 115, 111, 102, 116, 46, 99, 111, 109, 47, 112, 107, 105, 111, 112, 115, 47, 99, 101, 114, 116, 115, 47, 77, 105, 99, 114, 111, 115, 111, 102, 116, 37, 50, 48, 84, 76, 83, 37, 50, 48, 71, 50, 37, 50, 48, 82, 83, 65, 37, 50, 48, 67, 65, 37, 50, 48, 79, 67, 83, 80, 37, 50, 48, 49, 48, 46, 99, 114, 116, 48, 45, 6, 8, 43, 6, 1, 5, 5, 7, 48, 1, 134, 33, 104, 116, 116, 112, 58, 47, 47, 111, 110, 101, 111, 99, 115, 112, 46, 109, 105, 99, 114, 111, 115, 111, 102, 116, 46, 99, 111, 109, 47, 111, 99, 115, 112, 48, 29, 6, 3, 85, 29, 14, 4, 22, 4, 20, 138, 66, 241, 22, 211, 0, 191, 156, 21, 196, 84, 38, 230, 58, 211, 190, 115, 187, 163, 122, 48, 14, 6, 3, 85, 29, 15, 1, 1, 255, 4, 4, 3, 2, 5, 160, 48, 110, 6, 3, 85, 29, 17, 4, 103, 48, 101, 130, 64, 42, 46, 97, 112, 112, 115, 46, 97, 114, 111, 46, 105, 110, 103, 114, 101, 115, 115, 45, 99, 101, 114, 116, 45, 108, 101, 46, 48, 111, 51, 109, 46, 117, 107, 115, 111, 117, 116, 104, 46, 97, 114, 111, 97, 112, 112, 45, 104, 99, 112, 45, 115, 116, 103, 46, 97, 122, 117, 114, 101, 46, 99, 111, 109, 130, 33, 114, 101, 115, 101, 114, 118, 101, 100, 46, 97, 114, 111, 97, 112, 112, 45, 104, 99, 112, 45, 115, 116, 103, 46, 97, 122, 117, 114, 101, 46, 99, 111, 109, 48, 12, 6, 3, 85, 29, 19, 1, 1, 255, 4, 2, 48, 0, 48, 129, 241, 6, 3, 85, 29, 31, 4, 129, 233, 48, 129, 230, 48, 129, 227, 160, 129, 224, 160, 129, 221, 134, 108, 104, 116, 116, 112, 58, 47, 47, 119, 119, 119, 46, 109, 105, 99, 114, 111, 115, 111, 102, 116, 46, 99, 111, 109, 47, 112, 107, 105, 111, 112, 115, 47, 99, 114, 108, 47, 112, 97, 114, 116, 105, 116, 105, 111, 110, 47, 77, 105, 99, 114, 111, 115, 111, 102, 116, 37, 50, 48, 84, 76, 83, 37, 50, 48, 71, 50, 37, 50, 48, 82, 83, 65, 37, 50, 48, 67, 65, 37, 50, 48, 79, 67, 83, 80, 37, 50, 48, 49, 48, 95, 80, 97, 114, 116, 105, 116, 105, 111, 110, 48, 48, 48, 57, 57, 46, 99, 114, 108, 134, 109, 104, 116, 116, 112, 58, 47, 47, 99, 114, 108, 50, 46, 109, 105, 99, 114, 111, 115, 111, 102, 116, 46, 99, 111, 109, 47, 112, 107, 105, 111, 112, 115, 47, 99, 114, 108, 47, 112, 97, 114, 116, 105, 116, 105, 111, 110, 47, 77, 105, 99, 114, 111, 115, 111, 102, 116, 37, 50, 48, 84, 76, 83, 37, 50, 48, 71, 50, 37, 50, 48, 82, 83, 65, 37, 50, 48, 67, 65, 37, 50, 48, 79, 67, 83, 80, 37, 50, 48, 49, 48, 95, 80, 97, 114, 116, 105, 116, 105, 111, 110, 48, 48, 48, 57, 57, 46, 99, 114, 108, 48, 102, 6, 3, 85, 29, 32, 4, 95, 48, 93, 48, 8, 6, 6, 103, 129, 12, 1, 2, 2, 48, 81, 6, 12, 43, 6, 1, 4, 1, 130, 55, 76, 131, 125, 1, 1, 48, 65, 48, 63, 6, 8, 43, 6, 1, 5, 5, 7, 2, 1, 22, 51, 104, 116, 116, 112, 58, 47, 47, 119, 119, 119, 46, 109, 105, 99, 114, 111, 115, 111, 102, 116, 46, 99, 111, 109, 47, 112, 107, 105, 111, 112, 115, 47, 68, 111, 99, 115, 47, 82, 101, 112, 111, 115, 105, 116, 111, 114, 121, 46, 104, 116, 109, 48, 31, 6, 3, 85, 29, 35, 4, 24, 48, 22, 128, 20, 208, 76, 131, 209, 142, 113, 46, 61, 52, 177, 255, 155, 121, 44, 142, 116, 212, 226, 120, 6, 48, 19, 6, 3, 85, 29, 37, 4, 12, 48, 10, 6, 8, 43, 6, 1, 5, 5, 7, 3, 1],
RawSubjectPublicKeyInfo: [48, 130, 1, 34, 48, 13, 6, 9, 42, 134, 72, 134, 247, 13, 1, 1, 1, 5, 0, 3, 130, 1, 15, 0, 48, 130, 1, 10, 2, 130, 1, 1, 0, 187, 121, 183, 117, 223, 172, 129, 21, 158, 33, 40, 228, 200, 112, 100, 153, 170, 116, 250, 158, 245, 201, 135, 0, 237, 144, 31, 12, 163, 250, 168, 50, 92, 5, 220, 113, 137, 143, 3, 193, 167, 220, 193, 78, 195, 131, 29, 107, 67, 52, 86, 236, 30, 39, 30, 107, 206, 213, 234, 58, 181, 33, 44, 227, 147, 157, 23, 219, 20, 127, 20, 235, 246, 77, 135, 5, 240, 135, 231, 58, 154, 101, 19, 42, 160, 128, 73, 94, 79, 155, 62, 172, 213, 53, 57, 2, 149, 60, 188, 169, 233, 44, 216, 105, 66, 159, 148, 70, 178, 113, 177, 128, 123, 151, 10, 7, 254, 171, 205, 105, 34, 145, 93, 19, 218, 220, 219, 47, 129, 42, 85, 247, 229, 7, 118, 48, 182, 125, 84, 45, 187, 16, 138, 130, 74, 176, 14, 70, 229, 120, 108, 212, 172, 177, 174, 141, 161, 231, 167, 5, 108, 191, 229, 39, 27, 122, 242, 198, 32, 78, 125, 8, 168, 234, 200, 195, 10, 209, 101, 183, 248, 26, 10, 82, 78, 47, 196, 44, 55, 199, 207, 209, 35, 65, 164, 44, 162, 97, 81, 36, 122, 188, 105, 82, 214, 39, 252, 78, 115, 249, 159, 36, 212, 167, 6, 72, 63, 56, 220, 88, 50, 162, 50, 174, 185, 196, 246, 208, 84, 90, 212, 218, 52, 118, 184, 101, 201, 132, 48, 29, 45, 113, 80, 81, 157, 255, 235, 218, 49, 139, 40, 124, 111, 147, 143, 201, 2, 3, 1, 0, 1],
RawSubject: [48, 120, 49, 11, 48, 9, 6, 3, 85, 4, 6, 19, 2, 85, 83, 49, 11, 48, 9, 6, 3, 85, 4, 8, 19, 2, 87, 65, 49, 16, 48, 14, 6, 3, 85, 4, 7, 19, 7, 82, 101, 100, 109, 111, 110, 100, 49, 30, 48, 28, 6, 3, 85, 4, 10, 19, 21, 77, 105, 99, 114, 111, 115, 111, 102, 116, 32, 67, 111, 114, 112, 111, 114, 97, 116, 105, 111, 110, 49, 42, 48, 40, 6, 3, 85, 4, 3, 19, 33, 114, 101, 115, 101, 114, 118, 101, 100, 46, 97, 114, 111, 97, 112, 112, 45, 104, 99, 112, 45, 115, 116, 103, 46, 97, 122, 117, 114, 101, 46, 99, 111, 109],
RawIssuer: [48, 87, 49, 11, 48, 9, 6, 3, 85, 4, 6, 19, 2, 85, 83, 49, 30, 48, 28, 6, 3, 85, 4, 10, 19, 21, 77, 105, 99, 114, 111, 115, 111, 102, 116, 32, 67, 111, 114, 112, 111, 114, 97, 116, 105, 111, 110, 49, 40, 48, 38, 6, 3, 85, 4, 3, 19, 31, 77, 105, 99, 114, 111, 115, 111, 102, 116, 32, 84, 76, 83, 32, 71, 50, 32, 82, 83, 65, 32, 67, 65, 32, 79, 67, 83, 80, 32, 49, 48],
Signature: [127, 14, 76, 119, 16, 30, 113, 75, 136, 158, 234, 180, 48, 26, 140, 223, 208, 88, 116, 11, 139, 240, 141, 40, 46, 35, 73, 190, 186, 171, 118, 120, 99, 118, 88, 110, 181, 203, 42, 120, 241, 109, 206, 58, 1, 63, 144, 160, 72, 2, 211, 217, 25, 137, 246, 187, 186, 49, 138, 232, 122, 82, 3, 136, 22, 157, 87, 191, 86, 125, 221, 231, 158, 0, 204, 174, 39, 65, 179, 160, 12, 205, 190, 69, 207, 65, 139, 215, 99, 239, 19, 51, 250, 33, 66, 49, 250, 3, 217, 217, 178, 148, 58, 186, 139, 102, 86, 205, 171, 36, 144, 223, 188, 149, 248, 1, 103, 232, 163, 162, 100, 23, 166, 66, 215, 67, 190, 124, 120, 217, 106, 101, 246, 252, 8, 144, 121, 98, 136, 193, 124, 213, 80, 149, 73, 94, 215, 24, 197, 138, 7, 143, 140, 229, 172, 93, 155, 162, 211, 4, 222, 43, 210, 38, 230, 222, 23, 202, 214, 208, 103, 147, 145, 86, 53, 202, 100, 48, 50, 41, 5, 116, 80, 254, 160, 164, 22, 192, 242, 55, 78, 107, 236, 232, 202, 76, 47, 127, 5, 241, 21, 79, 8, 239, 16, 12, 106, 11, 200, 233, 173, 167, 160, 195, 61, 159, 42, 183, 223, 149, 235, 138, 251, 54, 65, 95, 191, 206, 35, 254, 15, 224, 194, 106, 185, 99, 23, 156, 154, 222, 84, 145, 1, 212, 126, 42, 74, 254, 71, 27, 110, 6, 83, 252, 190, 60, 29, 24, 98, 214, 226, 88, 108, 117, 86, 233, 3, 27, 205, 3, 203, 36, 3, 175, 73, 206, 16, 65, 117, 125, 162, 73, 22, 207, 82, 232, 237, 197, 40, 139, 79, 103, 210, 165, 182, 151, 205, 15, 253, 195, 116, 121, 178, 103, 118, 187, 165, 173, 124, 138, 96, 201, 232, 236, 212, 18, 179, 244, 160, 183, 40, 12, 30, 207, 237, 36, 120, 26, 57, 114, 158, 34, 152, 93, 132, 52, 124, 112, 222, 90, 37, 164, 188, 88, 53, 52, 110, 127, 166, 176, 30, 184, 129, 236, 40, 151, 237, 246, 44, 182, 131, 71, 47, 162, 122, 195, 81, 3, 7, 197, 200, 200, 48, 29, 56, 138, 158, 6, 33, 41, 210, 36, 197, 209, 154, 188, 46, 232, 167, 27, 177, 139, 150, 154, 50, 110, 13, 89, 233, 181, 191, 72, 106, 53, 174, 105, 231, 108, 61, 64, 154, 240, 69, 47, 220, 37, 46, 170, 187, 63, 76, 114, 220, 25, 31, 218, 164, 20, 35, 139, 252, 139, 174, 8, 42, 39, 150, 249, 226, 25, 42, 115, 161, 172, 59, 218, 77, 27, 104, 120, 74, 137, 124, 201, 198, 130, 184, 80, 219, 176, 188, 97, 76, 79, 77, 137, 160, 247, 211, 155, 231, 107, 56, 112, 147, 133, 216, 183, 201, 229, 98, 78, 26, 232, 122, 160, 50, 149, 73, 205, 59, 201, 173, 34, 22, 180, 220, 49, 209, 56, 199, 177, 177, 107, 162, 228, 222, 80, 57, 112, 238, 205],
SignatureAlgorithm: 5,
PublicKeyAlgorithm: 1,
PublicKey: <*rsa.PublicKey | 0x36a119f65b90>{
N: {
neg: false,
abs: [3569991644534247369, 3274486715548429274, 3780411684536201245, 13386095367080826074, 4555633294449980078, 8356885564098020936, 8844059572749990990, 2540492176423473444, 5633937353057226705, 779516051327683154, 2327935533544294595, 7836233832775480006, 12443919168731326213, 5381817252368116948, 13149759042380401282, 9307346103102305840, 2490874408770263855, 8905597817602166121, 4800718759162655104, 10753677448201427049, 5736247462418790658, 11125319528290994526, 17747990166117017402, 10636684124628129003, 14904076079720377571, 4842591071448276587, 12095755243242069355, 6630948406570189761, 17118216322786437170, 12282717644381587200, 11394433495304529049, 13509030274166194453],
},
E: 65537,
},
Version: 3,
SerialNumber: {
neg: false,
abs: [9968436300229532023, 6446808008473917675, 4784458],
},
Issuer: {
Country: ["US"],
Organization: [
"Microsoft Corporation",
],
OrganizationalUnit: nil,
Locality: nil,
Province: nil,
StreetAddress: nil,
PostalCode: nil,
SerialNumber: "",
CommonName: "Microsoft TLS G2 RSA CA OCSP 10",
Names: [
{
Type: [..., ..., ..., ...],
Value: <string>"US",
},
{
Type: [..., ..., ..., ...],
Value: <string>"Microsoft Corporation",
},
{
Type: [..., ..., ..., ...],
Value: <string>"Microsoft TLS G2 RSA CA OCSP 10",
},
],
ExtraNames: nil,
},
Subject: {
Country: ["US"],
Organization: [
"Microsoft Corporation",
],
OrganizationalUnit: nil,
Locality: ["Redmond"],
Province: ["WA"],
StreetAddress: nil,
PostalCode: nil,
SerialNumber: "",
CommonName: "reserved.aroapp-hcp-stg.azure.com",
Names: [
{
Type: [..., ..., ..., ...],
Value: <string>"US",
},
{
Type: [..., ..., ..., ...],
Value: <string>"WA",
},
{
Type: [..., ..., ..., ...],
Value: <string>"Redmond",
},
{
Type: [..., ..., ..., ...],
Value: <string>"Microsoft Corporation",
},
{
Type: [..., ..., ..., ...],
Value: <string>"reserved.aroapp-hcp-stg.azure.com",
},
],
ExtraNames: nil,
},
NotBefore: {wall: 0, ext: 63928490170, loc: nil},
NotAfter: {wall: 0, ext: 63937126570, loc: nil},
KeyUsage: 5,
Extensions: [
{
Id: [1, 3, 6, 1, 4, 1, 11129, 2, 4, 2],
Critical: false,
Value: [4, 129, 242, 0, 240, 0, 118, 0, 68, 194, 189, 12, 233, 20, 14, 100, 165, 201, 74, 1, 147, 10, 90, 161, 187, 53, 151, 14, 0, 238, 17, 22, 137, 104, 42, 28, 68, 215, 181, 102, 0, 0, 1, 161, 112, 206, 239, 163, 0, 0, 4, 3, 0, 71, 48, 69, 2, 33, 0, 172, 23, 89, 180, 174, 255, 10, 75, 120, 126, 2, 114, 141, 52, 53, 9, 237, 84, 209, 98, 89, 54, 179, 9, 54, 118, 245, 178, 107, 207, 5, 69, 2, 32, 89, 15, 78, 239, 171, 216, 123, 169, 212, 117, 5, 12, 137, 15, 63, 83, 41, 96, 223, 14, 136, 244, 40, 31, 142, 25, 69, 157, 200, 47, 205, 167, 0, 118, 0, 0, 26, 93, 26, 28, 45, 147, 117, 182, 72, 85, 120, 248, 47, 113, 161, 174, 110, 239, 57, 125, 41, 124, 138, 227, 21, 123, 202, 222, 225, 160, 30, 0, 0, 1, 161, 112, 206, 240, 23, 0, 0, 4, 3, 0, 71, 48, 69, 2, 33, 0, 209, 5, 94, 127, 36, 105, 13, 219, 166, 20, 49, 161, 208, 24, 148, 233, 208, 121, 79, 240, 195, 85, 152, 81, 108, 234, 202, 116, 68, 189, 194, 57, 2, 32, 65, 141, 139, 86, 66, 203, 70, 108, 26, 159, 237, 96, 32, 237, 241, 53, 230, 112, 171, 119, 90, 74, 10, 14, 99, 1, 225, 78, 245, 158, 165, 135],
},
{
Id: [1, 3, 6, 1, 4, 1, 311, 21, 10],
Critical: false,
Value: [48, 12, 48, 10, 6, 8, 43, 6, 1, 5, 5, 7, 3, 1],
},
{
Id: [1, 3, 6, 1, 4, 1, 311, 21, 7],
Critical: false,
Value: [48, 45, 6, 37, 43, 6, 1, 4, 1, 130, 55, 21, 8, 135, 189, 215, 27, 129, 231, 235, 70, 130, 129, 157, 46, 142, 208, 12, 135, 240, 218, 29, 93, 133, 246, 239, 42, 129, 232, 226, 53, 2, 1, 100, 2, 1, 39],
},
{
Id: [1, 3, 6, 1, 5, 5, 7, 1, 1],
Critical: false,
Value: [48, 129, 251, 48, 97, 6, 8, 43, 6, 1, 5, 5, 7, 48, 2, 134, 85, 104, 116, 116, 112, 58, 47, 47, 119, 119, 119, 46, 109, 105, 99, 114, 111, 115, 111, 102, 116, 46, 99, 111, 109, 47, 112, 107, 105, 111, 112, 115, 47, 99, 101, 114, 116, 115, 47, 77, 105, 99, 114, 111, 115, 111, 102, 116, 37, 50, 48, 84, 76, 83, 37, 50, 48, 71, 50, 37, 50, 48, 82, 83, 65, 37, 50, 48, 67, 65, 37, 50, 48, 79, 67, 83, 80, 37, 50, 48, 49, 48, 46, 99, 114, 116, 48, 103, 6, 8, 43, 6, 1, 5, 5, 7, 48, 2, 134, 91, 104, 116, 116, 112, 58, 47, 47, 99, 97, 105, 115, 115, 117, 101, 114, 115, 46, 109, 105, 99, 114, 111, 115, 111, 102, 116, 46, 99, 111, 109, 47, 112, 107, 105, 111, 112, 115, 47, 99, 101, 114, 116, 115, 47, 77, 105, 99, 114, 111, 115, 111, 102, 116, 37, 50, 48, 84, 76, 83, 37, 50, 48, 71, 50, 37, 50, 48, 82, 83, 65, 37, 50, 48, 67, 65, 37, 50, 48, 79, 67, 83, 80, 37, 50, 48, 49, 48, 46, 99, 114, 116, 48, 45, 6, 8, 43, 6, 1, 5, 5, 7, 48, 1, 134, 33, 104, 116, 116, 112, 58, 47, 47, 111, 110, 101, 111, 99, 115, 112, 46, 109, 105, 99, 114, 111, 115, 111, 102, 116, 46, 99, 111, 109, 47, 111, 99, 115, 112],
},
{
Id: [2, 5, 29, 14],
Critical: false,
Value: [4, 20, 138, 66, 241, 22, 211, 0, 191, 156, 21, 196, 84, 38, 230, 58, 211, 190, 115, 187, 163, 122],
},
{
Id: [2, 5, 29, 15],
Critical: true,
Value: [3, 2, 5, 160],
},
{
Id: [2, 5, 29, 17],
Critical: false,
Value: "0e\x82@*.apps.aro.ingress-cert-le.0o3m.uksouth.aroapp-hcp-stg.azure.com\x82!reserved.aroapp-hcp-stg.azure.com",
},
{Id: [2, 5, 29, 19], Critical: true, Value: [48, 0]},
{
Id: [2, 5, 29, 31],
Critical: false,
Value: "0\x81\xe60\x81㠁ࠁ݆lhttp://www.microsoft.com/pkiops/crl/partition/Microsoft%20TLS%20G2%20RSA%20CA%20OCSP%2010_Partition00099.crl\x86mhttp://crl2.microsoft.com/pkiops/crl/partition/Microsoft%20TLS%20G2%20RSA%20CA%20OCSP%2010_Partition00099.crl",
},
{
Id: [2, 5, 29, 32],
Critical: false,
Value: [48, 93, 48, 8, 6, 6, 103, 129, 12, 1, 2, 2, 48, 81, 6, 12, 43, 6, 1, 4, 1, 130, 55, 76, 131, 125, 1, 1, 48, 65, 48, 63, 6, 8, 43, 6, 1, 5, 5, 7, 2, 1, 22, 51, 104, 116, 116, 112, 58, 47, 47, 119, 119, 119, 46, 109, 105, 99, 114, 111, 115, 111, 102, 116, 46, 99, 111, 109, 47, 112, 107, 105, 111, 112, 115, 47, 68, 111, 99, 115, 47, 82, 101, 112, 111, 115, 105, 116, 111, 114, 121, 46, 104, 116, 109],
},
{
Id: [2, 5, 29, 35],
Critical: false,
Value: [48, 22, 128, 20, 208, 76, 131, 209, 142, 113, 46, 61, 52, 177, 255, 155, 121, 44, 142, 116, 212, 226, 120, 6],
},
{
Id: [2, 5, 29, 37],
Critical: false,
Value: [48, 10, 6, 8, 43, 6, 1, 5, 5, 7, 3, 1],
},
],
ExtraExtensions: nil,
UnhandledCriticalExtensions: nil,
ExtKeyUsage: [1],
UnknownExtKeyUsage: nil,
BasicConstraintsValid: true,
IsCA: false,
MaxPathLen: -1,
MaxPathLenZero: false,
SubjectKeyId: [138, 66, 241, 22, 211, 0, 191, 156, 21, 196, 84, 38, 230, 58, 211, 190, 115, 187, 163, 122],
AuthorityKeyId: [208, 76, 131, 209, 142, 113, 46, 61, 52, 177, 255, 155, 121, 44, 142, 116, 212, 226, 120, 6],
OCSPServer: [
"http://oneocsp.microsoft.com/ocsp",
],
IssuingCertificateURL: [
"http://www.microsoft.com/pkiops/certs/Microsoft%20TLS%20G2%20RSA%20CA%20OCSP%2010.crt",
"http://caissuers.microsoft.com/pkiops/certs/Microsoft%20TLS%20G2%20RSA%20CA%20OCSP%2010.crt",
],
DNSNames: [
"*.apps.aro.ingress-cert-le.0o3m.uksouth.aroapp-hcp-stg.azure.com",
"reserved.aroapp-hcp-stg.azure.com",
],
EmailAddresses: nil,
IPAddresses: nil,
URIs: nil,
PermittedDNSDomainsCritical: false,
PermittedDNSDomains: nil,
ExcludedDNSDomains: nil,
PermittedIPRanges: nil,
ExcludedIPRanges: nil,
PermittedEmailAddresses: nil,
ExcludedEmailAddresses: nil,
PermittedURIDomains: nil,
ExcludedURIDomains: nil,
CRLDistributionPoints: [
"http://www.microsoft.com/pkiops/crl/partition/Microsoft%20TLS%20G2%20RSA%20CA%20OCSP%2010_Partition00099.crl",
"http://crl2.microsoft.com/pkiops/crl/partition/Microsoft%20TLS%20G2%20RSA%20CA%20OCSP%2010_Partition00099.crl",
],
PolicyIdentifiers: [
[2, 23, 140, 1, 2, 2],
[1, 3, 6, 1, 4, 1, 311, 76, 509, 1, 1],
],
Policies: [
{
der: [103, 129, 12, 1, 2, 2],
},
{
der: [43, 6, 1, 4, 1, 130, 55, 76, 131, 125, 1, 1],
},
],
InhibitAnyPolicy: 0,
InhibitAnyPolicyZero: false,
InhibitPolicyMapping: 0,
InhibitPolicyMappingZero: false,
RequireExplicitPolicy: 0,
RequireExplicitPolicyZero: false,
PolicyMappings: nil,
},
{
Raw: [48, 130, 7, 184, 48, 130, 5, 160, 160, 3, 2, 1, 2, 2, 19, 51, 0, 0, 0, 15, 51, 32, 101, 55, 238, 66, 174, 79, 0, 0, 0, 0, 0, 15, 48, 13, 6, 9, 42, 134, 72, 134, 247, 13, 1, 1, 12, 5, 0, 48, 81, 49, 11, 48, 9, 6, 3, 85, 4, 6, 19, 2, 85, 83, 49, 30, 48, 28, 6, 3, 85, 4, 10, 19, 21, 77, 105, 99, 114, 111, 115, 111, 102, 116, 32, 67, 111, 114, 112, 111, 114, 97, 116, 105, 111, 110, 49, 34, 48, 32, 6, 3, 85, 4, 3, 19, 25, 77, 105, 99, 114, 111, 115, 111, 102, 116, 32, 84, 76, 83, 32, 82, 83, 65, 32, 82, 111, 111, 116, 32, 71, 50, 48, 30, 23, 13, 50, 53, 48, 56, 48, 49, 50, 48, 48, 51, 48, 51, 90, 23, 13, 50, 57, 48, 54, 48, 51, 50, 48, 48, 51, 48, 51, 90, 48, 87, 49, 11, 48, 9, 6, 3, 85, 4, 6, 19, 2, 85, 83, 49, 30, 48, 28, 6, 3, 85, 4, 10, 19, 21, 77, 105, 99, 114, 111, 115, 111, 102, 116, 32, 67, 111, 114, 112, 111, 114, 97, 116, 105, 111, 110, 49, 40, 48, 38, 6, 3, 85, 4, 3, 19, 31, 77, 105, 99, 114, 111, 115, 111, 102, 116, 32, 84, 76, 83, 32, 71, 50, 32, 82, 83, 65, 32, 67, 65, 32, 79, 67, 83, 80, 32, 49, 48, 48, 130, 2, 34, 48, 13, 6, 9, 42, 134, 72, 134, 247, 13, 1, 1, 1, 5, 0, 3, 130, 2, 15, 0, 48, 130, 2, 10, 2, 130, 2, 1, 0, 240, 12, 79, 91, 95, 252, 66, 28, 180, 206, 63, 245, 7, 43, 166, 131, 126, 201, 157, 121, 207, 246, 233, 172, 165, 119, 195, 55, 169, 250, 247, 7, 245, 45, 152, 248, 17, 102, 235, 66, 42, 14, 210, 209, 6, 25, 246, 119, 19, 95, 132, 236, 214, 81, 147, 108, 33, 227, 14, 133, 166, 255, 210, 236, 253, 36, 130, 189, 4, 249, 239, 141, 227, 225, 197, 221, 117, 208, 4, 154, 78, 20, 54, 25, 251, 45, 18, 163, 21, 192, 243, 111, 106, 68, 213, 113, 108, 241, 75, 152, 27, 121, 48, 247, 156, 23, 191, 125, 50, 9, 57, 91, 54, 140, 160, 8, 187, 64, 44, 150, 41, 238, 124, 189, 14, 248, 236, 109, 206, 193, 78, 118, 193, 88, 46, 78, 50, 154, 199, 124, 203, 199, 191, 183, 93, 247, 53, 84, 120, 204, 177, 135, 186, 5, 197, 53, 1, 112, 4, 195, 232, 93, 248, 177, 180, 223, 71, 205, 76, 102, 154, 232, 103, 212, 158, 30, 160, 33, 31, 51, 71, 5, 59, 68, 8, 207, 107, 136, 82, 89, 50, 120, 245, 65, 142, 37, 46, 208, 2, 13, 1, 207, 118, 247, 45, 52, 228, 103, 157, 104, 177, 128, 182, 77, 82, 179, 32, 151, 43, 165, 83, 140, 119, 149, 181, 115, 128, 205, 136, 255, 218, 151, 54, 227, 231, 111, 50, 160, 117, 241, 187, 191, 250, 60, 101, 171, 54, 17, 191, 118, 253, 71, 255, 108, 135, 96, 190, 12, 229, 234, 150, 245, 11, 162, 55, 39, 0, 209, 231, 2, 48, 1, 213, 128, 1, 194, 144, 206, 239, 179, 235, 211, 120, 96, 190, 10, 63, 251, 108, 198, 194, 209, 171, 39, 23, 23, 9, 132, 121, 145, 66, 142, 118, 104, 55, 17, 155, 206, 199, 118, 41, 191, 213, 92, 226, 124, 13, 95, 103, 82, 253, 109, 57, 250, 81, 45, 223, 178, 136, 3, 241, 97, 32, 170, 248, 180, 178, 159, 211, 250, 40, 176, 181, 167, 227, 131, 186, 250, 68, 59, 119, 220, 162, 44, 43, 232, 226, 83, 136, 154, 251, 12, 32, 222, 162, 123, 78, 128, 144, 113, 27, 124, 151, 212, 219, 252, 61, 154, 193, 161, 222, 234, 231, 174, 76, 229, 174, 175, 118, 172, 135, 205, 100, 188, 162, 52, 26, 211, 102, 195, 107, 97, 124, 155, 245, 230, 89, 121, 251, 128, 14, 22, 107, 192, 241, 41, 244, 237, 157, 151, 142, 58, 196, 41, 147, 7, 52, 255, 226, 191, 173, 154, 89, 37, 155, 134, 29, 67, 126, 203, 110, 149, 245, 142, 224, 169, 139, 87, 128, 78, 11, 230, 96, 125, 192, 73, 213, 4, 188, 32, 202, 4, 210, 92, 234, 63, 140, 73, 127, 29, 185, 186, 96, 109, 215, 180, 100, 179, 92, 52, 234, 56, 243, 249, 193, 248, 169, 11, 126, 186, 245, 209, 252, 122, 27, 127, 219, 8, 101, 125, 195, 151, 5, 236, 175, 47, 199, 222, 12, 250, 178, 63, 2, 3, 1, 0, 1, 163, 130, 2, 129, 48, 130, 2, 125, 48, 14, 6, 3, 85, 29, 15, 1, 1, 255, 4, 4, 3, 2, 1, 134, 48, 16, 6, 9, 43, 6, 1, 4, 1, 130, 55, 21, 1, 4, 3, 2, 1, 0, 48, 29, 6, 3, 85, 29, 14, 4, 22, 4, 20, 208, 76, 131, 209, 142, 113, 46, 61, 52, 177, 255, 155, 121, 44, 142, 116, 212, 226, 120, 6, 48, 19, 6, 3, 85, 29, 32, 4, 12, 48, 10, 48, 8, 6, 6, 103, 129, 12, 1, 2, 2, 48, 19, 6, 3, 85, 29, 37, 4, 12, 48, 10, 6, 8, 43, 6, 1, 5, 5, 7, 3, 1, 48, 25, 6, 9, 43, 6, 1, 4, 1, 130, 55, 20, 2, 4, 12, 30, 10, 0, 83, 0, 117, 0, 98, 0, 67, 0, 65, 48, 18, 6, 3, 85, 29, 19, 1, 1, 255, 4, 8, 48, 6, 1, 1, 255, 2, 1, 0, 48, 31, 6, 3, 85, 29, 35, 4, 24, 48, 22, 128, 20, 222, 145, 134, 72, 183, 161, 49, 89, 49, 241, 75, 95, 7, 169, 220, 136, 121, 218, 168, 118, 48, 129, 171, 6, 3, 85, 29, 31, 4, 129, 163, 48, 129, 160, 48, 129, 157, 160, 129, 154, 160, 129, 151, 134, 73, 104, 116, 116, 112, 58, 47, 47, 119, 119, 119, 46, 109, 105, 99, 114, 111, 115, 111, 102, 116, 46, 99, 111, 109, 47, 112, 107, 105, 111, 112, 115, 47, 99, 114, 108, 47, 77, 105, 99, 114, 111, 115, 111, 102, 116, 37, 50, 48, 84, 76, 83, 37, 50, 48, 82, 83, 65, 37, 50, 48, 82, 111, 111, 116, 37, 50, 48, 71, 50, 46, 99, 114, 108, 134, 74, 104, 116, 116, 112, 58, 47, 47, 99, 114, 108, 50, 46, 109, 105, 99, 114, 111, 115, 111, 102, 116, 46, 99, 111, 109, 47, 112, 107, 105, 111, 112, 115, 47, 99, 114, 108, 47, 77, 105, 99, 114, 111, 115, 111, 102, 116, 37, 50, 48, 84, 76, 83, 37, 50, 48, 82, 83, 65, 37, 50, 48, 82, 111, 111, 116, 37, 50, 48, 71, 50, 46, 99, 114, 108, 48, 130, 1, 16, 6, 8, 43, 6, 1, 5, 5, 7, 1, 1, 4, 130, 1, 2, 48, 129, 255, 48, 99, 6, 8, 43, 6, 1, 5, 5, 7, 48, 2, 134, 87, 104, 116, 116, 112, 58, 47, 47, 119, 119, 119, 46, 109, 105, 99, 114, 111, 115, 111, 102, 116, 46, 99, 111, 109, 47, 112, 107, 105, 111, 112, 115, 47, 99, 101, 114, 116, 115, 47, 77, 105, 99, 114, 111, 115, 111, 102, 116, 37, 50, 48, 84, 76, 83, 37, 50, 48, 82, 83, 65, 37, 50, 48, 82, 111, 111, 116, 37, 50, 48, 71, 50, 37, 50, 48, 45, 37, 50, 48, 120, 115, 105, 103, 110, 46, 99, 114, 116, 48, 105, 6, 8, 43, 6, 1, 5, 5, 7, 48, 2, 134, 93, 104, 116, 116, 112, 58, 47, 47, 99, 97, 105, 115, 115, 117, 101, 114, 115, 46, 109, 105, 99, 114, 111, 115, 111, 102, 116, 46, 99, 111, 109, 47, 112, 107, 105, 111, 112, 115, 47, 99, 101, 114, 116, 115, 47, 77, 105, 99, 114, 111, 115, 111, 102, 116, 37, 50, 48, 84, 76, 83, 37, 50, 48, 82, 83, 65, 37, 50, 48, 82, 111, 111, 116, 37, 50, 48, 71, 50, 37, 50, 48, 45, 37, 50, 48, 120, 115, 105, 103, 110, 46, 99, 114, 116, 48, 45, 6, 8, 43, 6, 1, 5, 5, 7, 48, 1, 134, 33, 104, 116, 116, 112, 58, 47, 47, 111, 110, 101, 111, 99, 115, 112, 46, 109, 105, 99, 114, 111, 115, 111, 102, 116, 46, 99, 111, 109, 47, 111, 99, 115, 112, 48, 13, 6, 9, 42, 134, 72, 134, 247, 13, 1, 1, 12, 5, 0, 3, 130, 2, 1, 0, 53, 25, 202, 233, 168, 117, 225, 216, 202, 253, 37, 193, 59, 82, 225, 231, 158, 43, 148, 226, 244, 58, 210, 79, 218, 6, 34, 82, 26, 1, 24, 227, 162, 186, 183, 183, 28, 36, 50, 117, 13, 103, 68, 125, 157, 74, 130, 75, 250, 227, 68, 233, 70, 246, 0, 69, 9, 194, 82, 233, 103, 37, 151, 199, 183, 139, 90, 119, 124, 64, 30, 148, 227, 188, 140, 109, 56, 41, 188, 39, 125, 18, 18, 61, 203, 203, 228, 196, 70, 226, 165, 1, 192, 16, 209, 32, 250, 14, 3, 228, 67, 219, 38, 135, 37, 172, 200, 160, 179, 4, 106, 94, 240, 67, 63, 15, 123, 245, 206, 35, 195, 183, 201, 190, 251, 222, 155, 125, 147, 179, 135, 62, 223, 198, 219, 61, 201, 244, 172, 113, 0, 210, 164, 101, 129, 54, 89, 159, 28, 54, 229, 137, 209, 122, 153, 74, 123, 85, 75, 194, 213, 140, 208, 121, 1, 121, 149, 15, 57, 67, 139, 179, 10, 123, 146, 213, 90, 16, 24, 118, 233, 29, 13, 111, 50, 49, 244, 189, 209, 22, 241, 168, 163, 103, 242, 14, 154, 112, 12, 227, 150, 175, 3, 244, 109, 189, 20, 121, 106, 54, 195, 241, 182, 115, 187, 198, 11, 213, 38, 70, 136, 84, 69, 112, 170, 204, 31, 61, 139, 111, 227, 228, 23, 166, 129, 0, 251, 125, 131, 88, 128, 226, 242, 60, 59, 163, 24, 98, 19, 54, 103, 255, 79, 226, 229, 111, 106, 103, 230, 232, 90, 100, 58, 160, 69, 157, 63, 23, 12, 102, 135, 240, 129, 39, 52, 32, 176, 53, 36, 100, 40, 118, 67, 122, 116, 152, 226, 43, 128, 9, 10, 188, 176, 185, 7, 71, 41, 40, 46, 125, 224, 119, 65, 164, 140, 54, 120, 186, 51, 29, 86, 142, 217, 16, 53, 137, 89, 131, 0, 105, 206, 183, 120, 189, 237, 175, 130, 63, 206, 200, 137, 50, 152, 89, 8, 93, 225, 1, 150, 88, 196, 101, 25, 125, 38, 133, 157, 16, 150, 229, 149, 197, 17, 206, 130, 111, 23, 6, 113, 21, 15, 44, 191, 55, 50, 82, 21, 104, 180, 149, 142, 83, 93, 151, 237, 227, 219, 19, 62, 193, 49, 177, 121, 147, 115, 125, 114, 59, 27, 76, 79, 6, 73, 21, 132, 115, 243, 29, 90, 81, 44, 164, 212, 88, 224, 25, 191, 53, 72, 41, 96, 153, 171, 234, 123, 66, 3, 190, 156, 61, 25, 247, 71, 91, 11, 172, 130, 87, 170, 13, 189, 255, 34, 96, 48, 223, 44, 79, 87, 211, 37, 213, 218, 253, 142, 6, 81, 78, 5, 53, 120, 152, 56, 163, 39, 49, 111, 49, 154, 169, 170, 77, 172, 58, 68, 104, 96, 112, 104, 244, 83, 107, 205, 178, 45, 174, 38, 24, 135, 223, 140, 87, 181, 248, 82, 193, 86, 231, 192, 177, 77, 95, 10, 85, 107, 20, 43, 84, 251, 61, 169, 198, 189, 96, 7, 248, 176, 244, 75, 227, 238, 171],
RawTBSCertificate: [48, 130, 5, 160, 160, 3, 2, 1, 2, 2, 19, 51, 0, 0, 0, 15, 51, 32, 101, 55, 238, 66, 174, 79, 0, 0, 0, 0, 0, 15, 48, 13, 6, 9, 42, 134, 72, 134, 247, 13, 1, 1, 12, 5, 0, 48, 81, 49, 11, 48, 9, 6, 3, 85, 4, 6, 19, 2, 85, 83, 49, 30, 48, 28, 6, 3, 85, 4, 10, 19, 21, 77, 105, 99, 114, 111, 115, 111, 102, 116, 32, 67, 111, 114, 112, 111, 114, 97, 116, 105, 111, 110, 49, 34, 48, 32, 6, 3, 85, 4, 3, 19, 25, 77, 105, 99, 114, 111, 115, 111, 102, 116, 32, 84, 76, 83, 32, 82, 83, 65, 32, 82, 111, 111, 116, 32, 71, 50, 48, 30, 23, 13, 50, 53, 48, 56, 48, 49, 50, 48, 48, 51, 48, 51, 90, 23, 13, 50, 57, 48, 54, 48, 51, 50, 48, 48, 51, 48, 51, 90, 48, 87, 49, 11, 48, 9, 6, 3, 85, 4, 6, 19, 2, 85, 83, 49, 30, 48, 28, 6, 3, 85, 4, 10, 19, 21, 77, 105, 99, 114, 111, 115, 111, 102, 116, 32, 67, 111, 114, 112, 111, 114, 97, 116, 105, 111, 110, 49, 40, 48, 38, 6, 3, 85, 4, 3, 19, 31, 77, 105, 99, 114, 111, 115, 111, 102, 116, 32, 84, 76, 83, 32, 71, 50, 32, 82, 83, 65, 32, 67, 65, 32, 79, 67, 83, 80, 32, 49, 48, 48, 130, 2, 34, 48, 13, 6, 9, 42, 134, 72, 134, 247, 13, 1, 1, 1, 5, 0, 3, 130, 2, 15, 0, 48, 130, 2, 10, 2, 130, 2, 1, 0, 240, 12, 79, 91, 95, 252, 66, 28, 180, 206, 63, 245, 7, 43, 166, 131, 126, 201, 157, 121, 207, 246, 233, 172, 165, 119, 195, 55, 169, 250, 247, 7, 245, 45, 152, 248, 17, 102, 235, 66, 42, 14, 210, 209, 6, 25, 246, 119, 19, 95, 132, 236, 214, 81, 147, 108, 33, 227, 14, 133, 166, 255, 210, 236, 253, 36, 130, 189, 4, 249, 239, 141, 227, 225, 197, 221, 117, 208, 4, 154, 78, 20, 54, 25, 251, 45, 18, 163, 21, 192, 243, 111, 106, 68, 213, 113, 108, 241, 75, 152, 27, 121, 48, 247, 156, 23, 191, 125, 50, 9, 57, 91, 54, 140, 160, 8, 187, 64, 44, 150, 41, 238, 124, 189, 14, 248, 236, 109, 206, 193, 78, 118, 193, 88, 46, 78, 50, 154, 199, 124, 203, 199, 191, 183, 93, 247, 53, 84, 120, 204, 177, 135, 186, 5, 197, 53, 1, 112, 4, 195, 232, 93, 248, 177, 180, 223, 71, 205, 76, 102, 154, 232, 103, 212, 158, 30, 160, 33, 31, 51, 71, 5, 59, 68, 8, 207, 107, 136, 82, 89, 50, 120, 245, 65, 142, 37, 46, 208, 2, 13, 1, 207, 118, 247, 45, 52, 228, 103, 157, 104, 177, 128, 182, 77, 82, 179, 32, 151, 43, 165, 83, 140, 119, 149, 181, 115, 128, 205, 136, 255, 218, 151, 54, 227, 231, 111, 50, 160, 117, 241, 187, 191, 250, 60, 101, 171, 54, 17, 191, 118, 253, 71, 255, 108, 135, 96, 190, 12, 229, 234, 150, 245, 11, 162, 55, 39, 0, 209, 231, 2, 48, 1, 213, 128, 1, 194, 144, 206, 239, 179, 235, 211, 120, 96, 190, 10, 63, 251, 108, 198, 194, 209, 171, 39, 23, 23, 9, 132, 121, 145, 66, 142, 118, 104, 55, 17, 155, 206, 199, 118, 41, 191, 213, 92, 226, 124, 13, 95, 103, 82, 253, 109, 57, 250, 81, 45, 223, 178, 136, 3, 241, 97, 32, 170, 248, 180, 178, 159, 211, 250, 40, 176, 181, 167, 227, 131, 186, 250, 68, 59, 119, 220, 162, 44, 43, 232, 226, 83, 136, 154, 251, 12, 32, 222, 162, 123, 78, 128, 144, 113, 27, 124, 151, 212, 219, 252, 61, 154, 193, 161, 222, 234, 231, 174, 76, 229, 174, 175, 118, 172, 135, 205, 100, 188, 162, 52, 26, 211, 102, 195, 107, 97, 124, 155, 245, 230, 89, 121, 251, 128, 14, 22, 107, 192, 241, 41, 244, 237, 157, 151, 142, 58, 196, 41, 147, 7, 52, 255, 226, 191, 173, 154, 89, 37, 155, 134, 29, 67, 126, 203, 110, 149, 245, 142, 224, 169, 139, 87, 128, 78, 11, 230, 96, 125, 192, 73, 213, 4, 188, 32, 202, 4, 210, 92, 234, 63, 140, 73, 127, 29, 185, 186, 96, 109, 215, 180, 100, 179, 92, 52, 234, 56, 243, 249, 193, 248, 169, 11, 126, 186, 245, 209, 252, 122, 27, 127, 219, 8, 101, 125, 195, 151, 5, 236, 175, 47, 199, 222, 12, 250, 178, 63, 2, 3, 1, 0, 1, 163, 130, 2, 129, 48, 130, 2, 125, 48, 14, 6, 3, 85, 29, 15, 1, 1, 255, 4, 4, 3, 2, 1, 134, 48, 16, 6, 9, 43, 6, 1, 4, 1, 130, 55, 21, 1, 4, 3, 2, 1, 0, 48, 29, 6, 3, 85, 29, 14, 4, 22, 4, 20, 208, 76, 131, 209, 142, 113, 46, 61, 52, 177, 255, 155, 121, 44, 142, 116, 212, 226, 120, 6, 48, 19, 6, 3, 85, 29, 32, 4, 12, 48, 10, 48, 8, 6, 6, 103, 129, 12, 1, 2, 2, 48, 19, 6, 3, 85, 29, 37, 4, 12, 48, 10, 6, 8, 43, 6, 1, 5, 5, 7, 3, 1, 48, 25, 6, 9, 43, 6, 1, 4, 1, 130, 55, 20, 2, 4, 12, 30, 10, 0, 83, 0, 117, 0, 98, 0, 67, 0, 65, 48, 18, 6, 3, 85, 29, 19, 1, 1, 255, 4, 8, 48, 6, 1, 1, 255, 2, 1, 0, 48, 31, 6, 3, 85, 29, 35, 4, 24, 48, 22, 128, 20, 222, 145, 134, 72, 183, 161, 49, 89, 49, 241, 75, 95, 7, 169, 220, 136, 121, 218, 168, 118, 48, 129, 171, 6, 3, 85, 29, 31, 4, 129, 163, 48, 129, 160, 48, 129, 157, 160, 129, 154, 160, 129, 151, 134, 73, 104, 116, 116, 112, 58, 47, 47, 119, 119, 119, 46, 109, 105, 99, 114, 111, 115, 111, 102, 116, 46, 99, 111, 109, 47, 112, 107, 105, 111, 112, 115, 47, 99, 114, 108, 47, 77, 105, 99, 114, 111, 115, 111, 102, 116, 37, 50, 48, 84, 76, 83, 37, 50, 48, 82, 83, 65, 37, 50, 48, 82, 111, 111, 116, 37, 50, 48, 71, 50, 46, 99, 114, 108, 134, 74, 104, 116, 116, 112, 58, 47, 47, 99, 114, 108, 50, 46, 109, 105, 99, 114, 111, 115, 111, 102, 116, 46, 99, 111, 109, 47, 112, 107, 105, 111, 112, 115, 47, 99, 114, 108, 47, 77, 105, 99, 114, 111, 115, 111, 102, 116, 37, 50, 48, 84, 76, 83, 37, 50, 48, 82, 83, 65, 37, 50, 48, 82, 111, 111, 116, 37, 50, 48, 71, 50, 46, 99, 114, 108, 48, 130, 1, 16, 6, 8, 43, 6, 1, 5, 5, 7, 1, 1, 4, 130, 1, 2, 48, 129, 255, 48, 99, 6, 8, 43, 6, 1, 5, 5, 7, 48, 2, 134, 87, 104, 116, 116, 112, 58, 47, 47, 119, 119, 119, 46, 109, 105, 99, 114, 111, 115, 111, 102, 116, 46, 99, 111, 109, 47, 112, 107, 105, 111, 112, 115, 47, 99, 101, 114, 116, 115, 47, 77, 105, 99, 114, 111, 115, 111, 102, 116, 37, 50, 48, 84, 76, 83, 37, 50, 48, 82, 83, 65, 37, 50, 48, 82, 111, 111, 116, 37, 50, 48, 71, 50, 37, 50, 48, 45, 37, 50, 48, 120, 115, 105, 103, 110, 46, 99, 114, 116, 48, 105, 6, 8, 43, 6, 1, 5, 5, 7, 48, 2, 134, 93, 104, 116, 116, 112, 58, 47, 47, 99, 97, 105, 115, 115, 117, 101, 114, 115, 46, 109, 105, 99, 114, 111, 115, 111, 102, 116, 46, 99, 111, 109, 47, 112, 107, 105, 111, 112, 115, 47, 99, 101, 114, 116, 115, 47, 77, 105, 99, 114, 111, 115, 111, 102, 116, 37, 50, 48, 84, 76, 83, 37, 50, 48, 82, 83, 65, 37, 50, 48, 82, 111, 111, 116, 37, 50, 48, 71, 50, 37, 50, 48, 45, 37, 50, 48, 120, 115, 105, 103, 110, 46, 99, 114, 116, 48, 45, 6, 8, 43, 6, 1, 5, 5, 7, 48, 1, 134, 33, 104, 116, 116, 112, 58, 47, 47, 111, 110, 101, 111, 99, 115, 112, 46, 109, 105, 99, 114, 111, 115, 111, 102, 116, 46, 99, 111, 109, 47, 111, 99, 115, 112],
RawSubjectPublicKeyInfo: [48, 130, 2, 34, 48, 13, 6, 9, 42, 134, 72, 134, 247, 13, 1, 1, 1, 5, 0, 3, 130, 2, 15, 0, 48, 130, 2, 10, 2, 130, 2, 1, 0, 240, 12, 79, 91, 95, 252, 66, 28, 180, 206, 63, 245, 7, 43, 166, 131, 126, 201, 157, 121, 207, 246, 233, 172, 165, 119, 195, 55, 169, 250, 247, 7, 245, 45, 152, 248, 17, 102, 235, 66, 42, 14, 210, 209, 6, 25, 246, 119, 19, 95, 132, 236, 214, 81, 147, 108, 33, 227, 14, 133, 166, 255, 210, 236, 253, 36, 130, 189, 4, 249, 239, 141, 227, 225, 197, 221, 117, 208, 4, 154, 78, 20, 54, 25, 251, 45, 18, 163, 21, 192, 243, 111, 106, 68, 213, 113, 108, 241, 75, 152, 27, 121, 48, 247, 156, 23, 191, 125, 50, 9, 57, 91, 54, 140, 160, 8, 187, 64, 44, 150, 41, 238, 124, 189, 14, 248, 236, 109, 206, 193, 78, 118, 193, 88, 46, 78, 50, 154, 199, 124, 203, 199, 191, 183, 93, 247, 53, 84, 120, 204, 177, 135, 186, 5, 197, 53, 1, 112, 4, 195, 232, 93, 248, 177, 180, 223, 71, 205, 76, 102, 154, 232, 103, 212, 158, 30, 160, 33, 31, 51, 71, 5, 59, 68, 8, 207, 107, 136, 82, 89, 50, 120, 245, 65, 142, 37, 46, 208, 2, 13, 1, 207, 118, 247, 45, 52, 228, 103, 157, 104, 177, 128, 182, 77, 82, 179, 32, 151, 43, 165, 83, 140, 119, 149, 181, 115, 128, 205, 136, 255, 218, 151, 54, 227, 231, 111, 50, 160, 117, 241, 187, 191, 250, 60, 101, 171, 54, 17, 191, 118, 253, 71, 255, 108, 135, 96, 190, 12, 229, 234, 150, 245, 11, 162, 55, 39, 0, 209, 231, 2, 48, 1, 213, 128, 1, 194, 144, 206, 239, 179, 235, 211, 120, 96, 190, 10, 63, 251, 108, 198, 194, 209, 171, 39, 23, 23, 9, 132, 121, 145, 66, 142, 118, 104, 55, 17, 155, 206, 199, 118, 41, 191, 213, 92, 226, 124, 13, 95, 103, 82, 253, 109, 57, 250, 81, 45, 223, 178, 136, 3, 241, 97, 32, 170, 248, 180, 178, 159, 211, 250, 40, 176, 181, 167, 227, 131, 186, 250, 68, 59, 119, 220, 162, 44, 43, 232, 226, 83, 136, 154, 251, 12, 32, 222, 162, 123, 78, 128, 144, 113, 27, 124, 151, 212, 219, 252, 61, 154, 193, 161, 222, 234, 231, 174, 76, 229, 174, 175, 118, 172, 135, 205, 100, 188, 162, 52, 26, 211, 102, 195, 107, 97, 124, 155, 245, 230, 89, 121, 251, 128, 14, 22, 107, 192, 241, 41, 244, 237, 157, 151, 142, 58, 196, 41, 147, 7, 52, 255, 226, 191, 173, 154, 89, 37, 155, 134, 29, 67, 126, 203, 110, 149, 245, 142, 224, 169, 139, 87, 128, 78, 11, 230, 96, 125, 192, 73, 213, 4, 188, 32, 202, 4, 210, 92, 234, 63, 140, 73, 127, 29, 185, 186, 96, 109, 215, 180, 100, 179, 92, 52, 234, 56, 243, 249, 193, 248, 169, 11, 126, 186, 245, 209, 252, 122, 27, 127, 219, 8, 101, 125, 195, 151, 5, 236, 175, 47, 199, 222, 12, 250, 178, 63, 2, 3, 1, 0, 1],
RawSubject: [48, 87, 49, 11, 48, 9, 6, 3, 85, 4, 6, 19, 2, 85, 83, 49, 30, 48, 28, 6, 3, 85, 4, 10, 19, 21, 77, 105, 99, 114, 111, 115, 111, 102, 116, 32, 67, 111, 114, 112, 111, 114, 97, 116, 105, 111, 110, 49, 40, 48, 38, 6, 3, 85, 4, 3, 19, 31, 77, 105, 99, 114, 111, 115, 111, 102, 116, 32, 84, 76, 83, 32, 71, 50, 32, 82, 83, 65, 32, 67, 65, 32, 79, 67, 83, 80, 32, 49, 48],
RawIssuer: [48, 81, 49, 11, 48, 9, 6, 3, 85, 4, 6, 19, 2, 85, 83, 49, 30, 48, 28, 6, 3, 85, 4, 10, 19, 21, 77, 105, 99, 114, 111, 115, 111, 102, 116, 32, 67, 111, 114, 112, 111, 114, 97, 116, 105, 111, 110, 49, 34, 48, 32, 6, 3, 85, 4, 3, 19, 25, 77, 105, 99, 114, 111, 115, 111, 102, 116, 32, 84, 76, 83, 32, 82, 83, 65, 32, 82, 111, 111, 116, 32, 71, 50],
Signature: [53, 25, 202, 233, 168, 117, 225, 216, 202, 253, 37, 193, 59, 82, 225, 231, 158, 43, 148, 226, 244, 58, 210, 79, 218, 6, 34, 82, 26, 1, 24, 227, 162, 186, 183, 183, 28, 36, 50, 117, 13, 103, 68, 125, 157, 74, 130, 75, 250, 227, 68, 233, 70, 246, 0, 69, 9, 194, 82, 233, 103, 37, 151, 199, 183, 139, 90, 119, 124, 64, 30, 148, 227, 188, 140, 109, 56, 41, 188, 39, 125, 18, 18, 61, 203, 203, 228, 196, 70, 226, 165, 1, 192, 16, 209, 32, 250, 14, 3, 228, 67, 219, 38, 135, 37, 172, 200, 160, 179, 4, 106, 94, 240, 67, 63, 15, 123, 245, 206, 35, 195, 183, 201, 190, 251, 222, 155, 125, 147, 179, 135, 62, 223, 198, 219, 61, 201, 244, 172, 113, 0, 210, 164, 101, 129, 54, 89, 159, 28, 54, 229, 137, 209, 122, 153, 74, 123, 85, 75, 194, 213, 140, 208, 121, 1, 121, 149, 15, 57, 67, 139, 179, 10, 123, 146, 213, 90, 16, 24, 118, 233, 29, 13, 111, 50, 49, 244, 189, 209, 22, 241, 168, 163, 103, 242, 14, 154, 112, 12, 227, 150, 175, 3, 244, 109, 189, 20, 121, 106, 54, 195, 241, 182, 115, 187, 198, 11, 213, 38, 70, 136, 84, 69, 112, 170, 204, 31, 61, 139, 111, 227, 228, 23, 166, 129, 0, 251, 125, 131, 88, 128, 226, 242, 60, 59, 163, 24, 98, 19, 54, 103, 255, 79, 226, 229, 111, 106, 103, 230, 232, 90, 100, 58, 160, 69, 157, 63, 23, 12, 102, 135, 240, 129, 39, 52, 32, 176, 53, 36, 100, 40, 118, 67, 122, 116, 152, 226, 43, 128, 9, 10, 188, 176, 185, 7, 71, 41, 40, 46, 125, 224, 119, 65, 164, 140, 54, 120, 186, 51, 29, 86, 142, 217, 16, 53, 137, 89, 131, 0, 105, 206, 183, 120, 189, 237, 175, 130, 63, 206, 200, 137, 50, 152, 89, 8, 93, 225, 1, 150, 88, 196, 101, 25, 125, 38, 133, 157, 16, 150, 229, 149, 197, 17, 206, 130, 111, 23, 6, 113, 21, 15, 44, 191, 55, 50, 82, 21, 104, 180, 149, 142, 83, 93, 151, 237, 227, 219, 19, 62, 193, 49, 177, 121, 147, 115, 125, 114, 59, 27, 76, 79, 6, 73, 21, 132, 115, 243, 29, 90, 81, 44, 164, 212, 88, 224, 25, 191, 53, 72, 41, 96, 153, 171, 234, 123, 66, 3, 190, 156, 61, 25, 247, 71, 91, 11, 172, 130, 87, 170, 13, 189, 255, 34, 96, 48, 223, 44, 79, 87, 211, 37, 213, 218, 253, 142, 6, 81, 78, 5, 53, 120, 152, 56, 163, 39, 49, 111, 49, 154, 169, 170, 77, 172, 58, 68, 104, 96, 112, 104, 244, 83, 107, 205, 178, 45, 174, 38, 24, 135, 223, 140, 87, 181, 248, 82, 193, 86, 231, 192, 177, 77, 95, 10, 85, 107, 20, 43, 84, 251, 61, 169, 198, 189, 96, 7, 248, 176, 244, 75, 227, 238, 171],
SignatureAlgorithm: 5,
PublicKeyAlgorithm: 1,
PublicKey: <*rsa.PublicKey | 0x36a118ac3670>{
N: {
neg: false,
abs: [12623528037057212991, 15782976484946675180, 9131881476679932799, 16877507857597638923, 6948446969886104628, 16879364175701064122, 15349600278999781980, 9245340169127444553, 7968545109705329495, 6423711443832766155, 10594494922037898650, 17648935788986680361, 18122500389998752041, 7737602644086511993, 7258855044086458051, 5541026985182791629, 4439073283700877230, 10408130435767131132, 18089869943821258368, 11685763412349126810, 16394152652608075740, 12871239331426121127, 9800942613550528692, 18261315811323797426, 15374412250364208978, 3968124059102095807, 685806708061009512, 7838166407260346135, 16993058175899549691, 15384298262262378419, 3974146097701597185, 13694573362644585378, 13796492994165901152, 13528806942954960401, 3955259362538190321, 13074935763441801879, 2348393719768315797, 11342510827905045171, 130453718204474471, 17672562703128855053, 634844305723830904, 11538538025395239748, 5505257917472087582, 16743812431760803789, 13404336697386206403, 6770939201695035783, 3646446187118575543, 14898275314275200590, 3021489551454039149, 3930692534149328022, 11247669138812385627, 7850138742151524599, 1567520330174748017, 5626181319708709539, 16420623271324550298, 18240848139101269901, 2441811390160163564, 1395980562250961772, 3030591394459285111, 17666945104455592770, 11923213182329091847, 9136006465603430828, 13028421093649262211, 17297287522694218268],
},
E: 65537,
},
Version: 3,
SerialNumber: {
neg: false,
abs: [12560257885759602703, 4278338880794178, 3342336],
},
Issuer: {
Country: ["US"],
Organization: [
"Microsoft Corporation",
],
OrganizationalUnit: nil,
Locality: nil,
Province: nil,
StreetAddress: nil,
PostalCode: nil,
SerialNumber: "",
CommonName: "Microsoft TLS RSA Root G2",
Names: [
{
Type: [..., ..., ..., ...],
Value: <string>"US",
},
{
Type: [..., ..., ..., ...],
Value: <string>"Microsoft Corporation",
},
{
Type: [..., ..., ..., ...],
Value: <string>"Microsoft TLS RSA Root G2",
},
],
ExtraNames: nil,
},
Subject: {
Country: ["US"],
Organization: [
"Microsoft Corporation",
],
OrganizationalUnit: nil,
Locality: nil,
Province: nil,
StreetAddress: nil,
PostalCode: nil,
SerialNumber: "",
CommonName: "Microsoft TLS G2 RSA CA OCSP 10",
Names: [
{
Type: [..., ..., ..., ...],
Value: <string>"US",
},
{
Type: [..., ..., ..., ...],
Value: <string>"Microsoft Corporation",
},
{
Type: [..., ..., ..., ...],
Value: <string>"Microsoft TLS G2 RSA CA OCSP 10",
},
],
ExtraNames: nil,
},
NotBefore: {wall: 0, ext: 63889675383, loc: nil},
NotAfter: {wall: 0, ext: 64010808183, loc: nil},
KeyUsage: 97,
Extensions: [
{
Id: [2, 5, 29, 15],
Critical: true,
Value: [3, 2, 1, 134],
},
{
Id: [1, 3, 6, 1, 4, 1, 311, 21, 1],
Critical: false,
Value: [2, 1, 0],
},
{
Id: [2, 5, 29, 14],
Critical: false,
Value: [4, 20, 208, 76, 131, 209, 142, 113, 46, 61, 52, 177, 255, 155, 121, 44, 142, 116, 212, 226, 120, 6],
},
{
Id: [2, 5, 29, 32],
Critical: false,
Value: [48, 10, 48, 8, 6, 6, 103, 129, 12, 1, 2, 2],
},
{
Id: [2, 5, 29, 37],
Critical: false,
Value: [48, 10, 6, 8, 43, 6, 1, 5, 5, 7, 3, 1],
},
{
Id: [1, 3, 6, 1, 4, 1, 311, 20, 2],
Critical: false,
Value: [30, 10, 0, 83, 0, 117, 0, 98, 0, 67, 0, 65],
},
{
Id: [2, 5, 29, 19],
Critical: true,
Value: [48, 6, 1, 1, 255, 2, 1, 0],
},
{
Id: [2, 5, 29, 35],
Critical: false,
Value: [48, 22, 128, 20, 222, 145, 134, 72, 183, 161, 49, 89, 49, 241, 75, 95, 7, 169, 220, 136, 121, 218, 168, 118],
},
{
Id: [2, 5, 29, 31],
Critical: false,
Value: "0\x81\xa00\x81\x9d\xa0\x81\x9a\xa0\x81\x97\x86Ihttp://www.microsoft.com/pkiops/crl/Microsoft%20TLS%20RSA%20Root%20G2.crl\x86Jhttp://crl2.microsoft.com/pkiops/crl/Microsoft%20TLS%20RSA%20Root%20G2.crl",
},
{
Id: [1, 3, 6, 1, 5, 5, 7, 1, 1],
Critical: false,
Value: [48, 129, 255, 48, 99, 6, 8, 43, 6, 1, 5, 5, 7, 48, 2, 134, 87, 104, 116, 116, 112, 58, 47, 47, 119, 119, 119, 46, 109, 105, 99, 114, 111, 115, 111, 102, 116, 46, 99, 111, 109, 47, 112, 107, 105, 111, 112, 115, 47, 99, 101, 114, 116, 115, 47, 77, 105, 99, 114, 111, 115, 111, 102, 116, 37, 50, 48, 84, 76, 83, 37, 50, 48, 82, 83, 65, 37, 50, 48, 82, 111, 111, 116, 37, 50, 48, 71, 50, 37, 50, 48, 45, 37, 50, 48, 120, 115, 105, 103, 110, 46, 99, 114, 116, 48, 105, 6, 8, 43, 6, 1, 5, 5, 7, 48, 2, 134, 93, 104, 116, 116, 112, 58, 47, 47, 99, 97, 105, 115, 115, 117, 101, 114, 115, 46, 109, 105, 99, 114, 111, 115, 111, 102, 116, 46, 99, 111, 109, 47, 112, 107, 105, 111, 112, 115, 47, 99, 101, 114, 116, 115, 47, 77, 105, 99, 114, 111, 115, 111, 102, 116, 37, 50, 48, 84, 76, 83, 37, 50, 48, 82, 83, 65, 37, 50, 48, 82, 111, 111, 116, 37, 50, 48, 71, 50, 37, 50, 48, 45, 37, 50, 48, 120, 115, 105, 103, 110, 46, 99, 114, 116, 48, 45, 6, 8, 43, 6, 1, 5, 5, 7, 48, 1, 134, 33, 104, 116, 116, 112, 58, 47, 47, 111, 110, 101, 111, 99, 115, 112, 46, 109, 105, 99, 114, 111, 115, 111, 102, 116, 46, 99, 111, 109, 47, 111, 99, 115, 112],
},
],
ExtraExtensions: nil,
UnhandledCriticalExtensions: nil,
ExtKeyUsage: [1],
UnknownExtKeyUsage: nil,
BasicConstraintsValid: true,
IsCA: true,
MaxPathLen: 0,
MaxPathLenZero: true,
SubjectKeyId: [208, 76, 131, 209, 142, 113, 46, 61, 52, 177, 255, 155, 121, 44, 142, 116, 212, 226, 120, 6],
AuthorityKeyId: [222, 145, 134, 72, 183, 161, 49, 89, 49, 241, 75, 95, 7, 169, 220, 136, 121, 218, 168, 118],
OCSPServer: [
"http://oneocsp.microsoft.com/ocsp",
],
IssuingCertificateURL: [
"http://www.microsoft.com/pkiops/certs/Microsoft%20TLS%20RSA%20Root%20G2%20-%20xsign.crt",
"http://caissuers.microsoft.com/pkiops/certs/Microsoft%20TLS%20RSA%20Root%20G2%20-%20xsign.crt",
],
DNSNames: nil,
EmailAddresses: nil,
IPAddresses: nil,
URIs: nil,
PermittedDNSDomainsCritical: false,
PermittedDNSDomains: nil,
ExcludedDNSDomains: nil,
PermittedIPRanges: nil,
ExcludedIPRanges: nil,
PermittedEmailAddresses: nil,
ExcludedEmailAddresses: nil,
PermittedURIDomains: nil,
ExcludedURIDomains: nil,
CRLDistributionPoints: [
"http://www.microsoft.com/pkiops/crl/Microsoft%20TLS%20RSA%20Root%20G2.crl",
"http://crl2.microsoft.com/pkiops/crl/Microsoft%20TLS%20RSA%20Root%20G2.crl",
],
PolicyIdentifiers: [
[2, 23, 140, 1, 2, 2],
],
Policies: [
{
der: [103, 129, 12, 1, 2, 2],
},
],
InhibitAnyPolicy: 0,
InhibitAnyPolicyZero: false,
InhibitPolicyMapping: 0,
InhibitPolicyMappingZero: false,
RequireExplicitPolicy: 0,
RequireExplicitPolicyZero: false,
PolicyMappings: nil,
},
{
Raw: [48, 130, 5, 137, 48, 130, 4, 113, 160, 3, 2, 1, 2, 2, 16, 11, 12, 107, 44, 70, 105, 23, 176, 71, 115, 198, 71, 212, 175, 192, 200, 48, 13, 6, 9, 42, 134, 72, 134, 247, 13, 1, 1, 12, 5, 0, 48, 97, 49, 11, 48, 9, 6, 3, 85, 4, 6, 19, 2, 85, 83, 49, 21, 48, 19, 6, 3, 85, 4, 10, 19, 12, 68, 105, 103, 105, 67, 101, 114, 116, 32, 73, 110, 99, 49, 25, 48, 23, 6, 3, 85, 4, 11, 19, 16, 119, 119, 119, 46, 100, 105, 103, 105, 99, 101, 114, 116, 46, 99, 111, 109, 49, 32, 48, 30, 6, 3, 85, 4, 3, 19, 23, 68, 105, 103, 105, 67, 101, 114, 116, 32, 71, 108, 111, 98, 97, 108, 32, 82, 111, 111, 116, 32, 71, 50, 48, 30, 23, 13, 50, 53, 48, 53, 50, 49, 48, 48, 48, 48, 48, 48, 90, 23, 13, 50, 57, 48, 54, 49, 57, 50, 51, 53, 57, 53, 57, 90, 48, 81, 49, 11, 48, 9, 6, 3, 85, 4, 6, 19, 2, 85, 83, 49, 30, 48, 28, 6, 3, 85, 4, 10, 19, 21, 77, 105, 99, 114, 111, 115, 111, 102, 116, 32, 67, 111, 114, 112, 111, 114, 97, 116, 105, 111, 110, 49, 34, 48, 32, 6, 3, 85, 4, 3, 19, 25, 77, 105, 99, 114, 111, 115, 111, 102, 116, 32, 84, 76, 83, 32, 82, 83, 65, 32, 82, 111, 111, 116, 32, 71, 50, 48, 130, 2, 34, 48, 13, 6, 9, 42, 134, 72, 134, 247, 13, 1, 1, 1, 5, 0, 3, 130, 2, 15, 0, 48, 130, 2, 10, 2, 130, 2, 1, 0, 223, 234, 139, 159, 71, 225, 40, 16, 113, 175, 65, 214, 25, 219, 146, 87, 222, 107, 171, 230, 46, 222, 173, 58, 96, 131, 183, 19, 152, 172, 91, 185, 50, 11, 123, 235, 159, 114, 39, 107, 93, 64, 171, 190, 24, 6, 147, 178, 13, 210, 18, 237, 184, 116, 245, 127, 47, 234, 168, 23, 148, 246, 51, 22, 87, 138, 222, 32, 124, 33, 221, 183, 218, 172, 222, 78, 100, 176, 90, 205, 75, 117, 243, 239, 2, 100, 77, 237, 82, 218, 163, 143, 4, 232, 145, 248, 215, 251, 63, 25, 253, 78, 69, 57, 94, 232, 41, 195, 214, 242, 103, 175, 179, 0, 148, 193, 141, 225, 240, 74, 24, 152, 57, 250, 192, 9, 132, 51, 65, 5, 3, 5, 171, 244, 117, 225, 83, 165, 33, 146, 92, 68, 76, 230, 96, 87, 164, 144, 2, 151, 199, 213, 216, 147, 4, 50, 74, 237, 133, 81, 237, 214, 197, 186, 160, 63, 54, 138, 47, 111, 76, 57, 168, 4, 105, 34, 132, 59, 117, 20, 247, 217, 62, 180, 169, 208, 7, 65, 56, 129, 126, 70, 72, 29, 232, 25, 230, 112, 184, 160, 170, 24, 221, 114, 191, 126, 125, 181, 108, 78, 25, 112, 17, 132, 4, 19, 129, 212, 98, 253, 156, 107, 241, 28, 221, 255, 236, 167, 224, 230, 199, 141, 49, 75, 191, 242, 107, 51, 204, 209, 140, 182, 254, 177, 27, 163, 74, 168, 95, 14, 127, 49, 118, 56, 63, 7, 71, 108, 1, 168, 172, 165, 20, 218, 130, 176, 95, 31, 108, 188, 137, 26, 192, 18, 42, 203, 123, 151, 31, 40, 46, 49, 22, 138, 83, 76, 117, 102, 249, 125, 1, 214, 194, 96, 170, 101, 143, 49, 158, 114, 203, 204, 155, 45, 229, 68, 25, 147, 87, 171, 110, 142, 194, 217, 46, 101, 176, 114, 45, 231, 161, 195, 161, 126, 73, 29, 89, 103, 214, 9, 212, 24, 18, 109, 192, 149, 69, 120, 201, 241, 96, 106, 228, 246, 238, 6, 36, 59, 115, 74, 90, 17, 194, 62, 151, 97, 192, 120, 238, 239, 19, 167, 48, 226, 74, 56, 45, 238, 88, 23, 19, 32, 183, 54, 42, 223, 11, 50, 180, 194, 56, 177, 145, 221, 18, 176, 69, 182, 124, 192, 177, 29, 98, 161, 157, 221, 113, 18, 21, 150, 1, 157, 10, 9, 245, 177, 221, 179, 217, 224, 56, 66, 52, 246, 105, 182, 195, 175, 22, 115, 254, 212, 67, 248, 67, 228, 88, 249, 79, 179, 243, 236, 214, 61, 148, 58, 64, 56, 21, 171, 12, 134, 208, 100, 206, 26, 99, 232, 132, 148, 85, 9, 90, 182, 4, 182, 209, 62, 154, 21, 199, 100, 17, 128, 208, 27, 130, 14, 77, 255, 69, 158, 178, 101, 7, 37, 31, 219, 154, 200, 198, 236, 214, 91, 133, 80, 4, 159, 28, 203, 93, 32, 130, 103, 158, 121, 226, 223, 232, 242, 232, 111, 196, 32, 114, 48, 134, 229, 157, 117, 39, 53, 2, 3, 1, 0, 1, 163, 130, 1, 75, 48, 130, 1, 71, 48, 15, 6, 3, 85, 29, 19, 1, 1, 255, 4, 5, 48, 3, 1, 1, 255, 48, 29, 6, 3, 85, 29, 14, 4, 22, 4, 20, 222, 145, 134, 72, 183, 161, 49, 89, 49, 241, 75, 95, 7, 169, 220, 136, 121, 218, 168, 118, 48, 31, 6, 3, 85, 29, 35, 4, 24, 48, 22, 128, 20, 78, 34, 84, 32, 24, 149, 230, 227, 110, 230, 15, 250, 250, 185, 18, 237, 6, 23, 143, 57, 48, 14, 6, 3, 85, 29, 15, 1, 1, 255, 4, 4, 3, 2, 1, 134, 48, 19, 6, 3, 85, 29, 37, 4, 12, 48, 10, 6, 8, 43, 6, 1, 5, 5, 7, 3, 1, 48, 118, 6, 8, 43, 6, 1, 5, 5, 7, 1, 1, 4, 106, 48, 104, 48, 36, 6, 8, 43, 6, 1, 5, 5, 7, 48, 1, 134, 24, 104, 116, 116, 112, 58, 47, 47, 111, 99, 115, 112, 46, 100, 105, 103, 105, 99, 101, 114, 116, 46, 99, 111, 109, 48, 64, 6, 8, 43, 6, 1, 5, 5, 7, 48, 2, 134, 52, 104, 116, 116, 112, 58, 47, 47, 99, 97, 99, 101, 114, 116, 115, 46, 100, 105, 103, 105, 99, 101, 114, 116, 46, 99, 111, 109, 47, 68, 105, 103, 105, 67, 101, 114, 116, 71, 108, 111, 98, 97, 108, 82, 111, 111, 116, 71, 50, 46, 99, 114, 116, 48, 66, 6, 3, 85, 29, 31, 4, 59, 48, 57, 48, 55, 160, 53, 160, 51, 134, 49, 104, 116, 116, 112, 58, 47, 47, 99, 114, 108, 51, 46, 100, 105, 103, 105, 99, 101, 114, 116, 46, 99, 111, 109, 47, 68, 105, 103, 105, 67, 101, 114, 116, 71, 108, 111, 98, 97, 108, 82, 111, 111, 116, 71, 50, 46, 99, 114, 108, 48, 19, 6, 3, 85, 29, 32, 4, 12, 48, 10, 48, 8, 6, 6, 103, 129, 12, 1, 2, 2, 48, 13, 6, 9, 42, 134, 72, 134, 247, 13, 1, 1, 12, 5, 0, 3, 130, 1, 1, 0, 11, 188, 180, 43, 55, 116, 197, 75, 167, 54, 2, 54, 198, 175, 225, 19, 204, 138, 154, 151, 27, 252, 227, 68, 140, 238, 84, 0, 229, 228, 65, 26, 70, 240, 11, 118, 52, 255, 153, 85, 98, 53, 170, 173, 138, 156, 12, 90, 98, 204, 67, 192, 203, 234, 74, 29, 189, 107, 29, 175, 177, 25, 37, 102, 156, 247, 121, 110, 91, 37, 128, 36, 129, 133, 199, 121, 45, 53, 95, 66, 9, 120, 237, 159, 226, 101, 213, 196, 32, 80, 130, 16, 143, 106, 58, 237, 24, 142, 72, 137, 171, 166, 9, 248, 225, 128, 252, 6, 117, 22, 39, 37, 90, 143, 184, 58, 112, 108, 24, 71, 240, 99, 244, 83, 14, 217, 21, 53, 32, 243, 145, 61, 19, 1, 160, 226, 250, 36, 8, 69, 86, 119, 209, 103, 169, 49, 66, 208, 87, 255, 26, 38, 190, 197, 67, 216, 87, 169, 99, 255, 175, 243, 58, 216, 246, 123, 75, 16, 162, 151, 161, 207, 102, 190, 103, 103, 9, 249, 181, 76, 231, 171, 25, 43, 191, 34, 61, 125, 222, 198, 247, 239, 99, 56, 44, 221, 86, 196, 249, 9, 82, 218, 223, 227, 136, 111, 213, 109, 107, 247, 8, 65, 47, 142, 62, 134, 49, 159, 241, 124, 74, 245, 214, 97, 164, 70, 70, 40, 239, 87, 186, 176, 111, 146, 216, 210, 119, 37, 237, 123, 166, 41, 206, 59, 114, 121, 17, 212, 224, 167, 84, 109, 53, 43, 164, 122],
RawTBSCertificate: [48, 130, 4, 113, 160, 3, 2, 1, 2, 2, 16, 11, 12, 107, 44, 70, 105, 23, 176, 71, 115, 198, 71, 212, 175, 192, 200, 48, 13, 6, 9, 42, 134, 72, 134, 247, 13, 1, 1, 12, 5, 0, 48, 97, 49, 11, 48, 9, 6, 3, 85, 4, 6, 19, 2, 85, 83, 49, 21, 48, 19, 6, 3, 85, 4, 10, 19, 12, 68, 105, 103, 105, 67, 101, 114, 116, 32, 73, 110, 99, 49, 25, 48, 23, 6, 3, 85, 4, 11, 19, 16, 119, 119, 119, 46, 100, 105, 103, 105, 99, 101, 114, 116, 46, 99, 111, 109, 49, 32, 48, 30, 6, 3, 85, 4, 3, 19, 23, 68, 105, 103, 105, 67, 101, 114, 116, 32, 71, 108, 111, 98, 97, 108, 32, 82, 111, 111, 116, 32, 71, 50, 48, 30, 23, 13, 50, 53, 48, 53, 50, 49, 48, 48, 48, 48, 48, 48, 90, 23, 13, 50, 57, 48, 54, 49, 57, 50, 51, 53, 57, 53, 57, 90, 48, 81, 49, 11, 48, 9, 6, 3, 85, 4, 6, 19, 2, 85, 83, 49, 30, 48, 28, 6, 3, 85, 4, 10, 19, 21, 77, 105, 99, 114, 111, 115, 111, 102, 116, 32, 67, 111, 114, 112, 111, 114, 97, 116, 105, 111, 110, 49, 34, 48, 32, 6, 3, 85, 4, 3, 19, 25, 77, 105, 99, 114, 111, 115, 111, 102, 116, 32, 84, 76, 83, 32, 82, 83, 65, 32, 82, 111, 111, 116, 32, 71, 50, 48, 130, 2, 34, 48, 13, 6, 9, 42, 134, 72, 134, 247, 13, 1, 1, 1, 5, 0, 3, 130, 2, 15, 0, 48, 130, 2, 10, 2, 130, 2, 1, 0, 223, 234, 139, 159, 71, 225, 40, 16, 113, 175, 65, 214, 25, 219, 146, 87, 222, 107, 171, 230, 46, 222, 173, 58, 96, 131, 183, 19, 152, 172, 91, 185, 50, 11, 123, 235, 159, 114, 39, 107, 93, 64, 171, 190, 24, 6, 147, 178, 13, 210, 18, 237, 184, 116, 245, 127, 47, 234, 168, 23, 148, 246, 51, 22, 87, 138, 222, 32, 124, 33, 221, 183, 218, 172, 222, 78, 100, 176, 90, 205, 75, 117, 243, 239, 2, 100, 77, 237, 82, 218, 163, 143, 4, 232, 145, 248, 215, 251, 63, 25, 253, 78, 69, 57, 94, 232, 41, 195, 214, 242, 103, 175, 179, 0, 148, 193, 141, 225, 240, 74, 24, 152, 57, 250, 192, 9, 132, 51, 65, 5, 3, 5, 171, 244, 117, 225, 83, 165, 33, 146, 92, 68, 76, 230, 96, 87, 164, 144, 2, 151, 199, 213, 216, 147, 4, 50, 74, 237, 133, 81, 237, 214, 197, 186, 160, 63, 54, 138, 47, 111, 76, 57, 168, 4, 105, 34, 132, 59, 117, 20, 247, 217, 62, 180, 169, 208, 7, 65, 56, 129, 126, 70, 72, 29, 232, 25, 230, 112, 184, 160, 170, 24, 221, 114, 191, 126, 125, 181, 108, 78, 25, 112, 17, 132, 4, 19, 129, 212, 98, 253, 156, 107, 241, 28, 221, 255, 236, 167, 224, 230, 199, 141, 49, 75, 191, 242, 107, 51, 204, 209, 140, 182, 254, 177, 27, 163, 74, 168, 95, 14, 127, 49, 118, 56, 63, 7, 71, 108, 1, 168, 172, 165, 20, 218, 130, 176, 95, 31, 108, 188, 137, 26, 192, 18, 42, 203, 123, 151, 31, 40, 46, 49, 22, 138, 83, 76, 117, 102, 249, 125, 1, 214, 194, 96, 170, 101, 143, 49, 158, 114, 203, 204, 155, 45, 229, 68, 25, 147, 87, 171, 110, 142, 194, 217, 46, 101, 176, 114, 45, 231, 161, 195, 161, 126, 73, 29, 89, 103, 214, 9, 212, 24, 18, 109, 192, 149, 69, 120, 201, 241, 96, 106, 228, 246, 238, 6, 36, 59, 115, 74, 90, 17, 194, 62, 151, 97, 192, 120, 238, 239, 19, 167, 48, 226, 74, 56, 45, 238, 88, 23, 19, 32, 183, 54, 42, 223, 11, 50, 180, 194, 56, 177, 145, 221, 18, 176, 69, 182, 124, 192, 177, 29, 98, 161, 157, 221, 113, 18, 21, 150, 1, 157, 10, 9, 245, 177, 221, 179, 217, 224, 56, 66, 52, 246, 105, 182, 195, 175, 22, 115, 254, 212, 67, 248, 67, 228, 88, 249, 79, 179, 243, 236, 214, 61, 148, 58, 64, 56, 21, 171, 12, 134, 208, 100, 206, 26, 99, 232, 132, 148, 85, 9, 90, 182, 4, 182, 209, 62, 154, 21, 199, 100, 17, 128, 208, 27, 130, 14, 77, 255, 69, 158, 178, 101, 7, 37, 31, 219, 154, 200, 198, 236, 214, 91, 133, 80, 4, 159, 28, 203, 93, 32, 130, 103, 158, 121, 226, 223, 232, 242, 232, 111, 196, 32, 114, 48, 134, 229, 157, 117, 39, 53, 2, 3, 1, 0, 1, 163, 130, 1, 75, 48, 130, 1, 71, 48, 15, 6, 3, 85, 29, 19, 1, 1, 255, 4, 5, 48, 3, 1, 1, 255, 48, 29, 6, 3, 85, 29, 14, 4, 22, 4, 20, 222, 145, 134, 72, 183, 161, 49, 89, 49, 241, 75, 95, 7, 169, 220, 136, 121, 218, 168, 118, 48, 31, 6, 3, 85, 29, 35, 4, 24, 48, 22, 128, 20, 78, 34, 84, 32, 24, 149, 230, 227, 110, 230, 15, 250, 250, 185, 18, 237, 6, 23, 143, 57, 48, 14, 6, 3, 85, 29, 15, 1, 1, 255, 4, 4, 3, 2, 1, 134, 48, 19, 6, 3, 85, 29, 37, 4, 12, 48, 10, 6, 8, 43, 6, 1, 5, 5, 7, 3, 1, 48, 118, 6, 8, 43, 6, 1, 5, 5, 7, 1, 1, 4, 106, 48, 104, 48, 36, 6, 8, 43, 6, 1, 5, 5, 7, 48, 1, 134, 24, 104, 116, 116, 112, 58, 47, 47, 111, 99, 115, 112, 46, 100, 105, 103, 105, 99, 101, 114, 116, 46, 99, 111, 109, 48, 64, 6, 8, 43, 6, 1, 5, 5, 7, 48, 2, 134, 52, 104, 116, 116, 112, 58, 47, 47, 99, 97, 99, 101, 114, 116, 115, 46, 100, 105, 103, 105, 99, 101, 114, 116, 46, 99, 111, 109, 47, 68, 105, 103, 105, 67, 101, 114, 116, 71, 108, 111, 98, 97, 108, 82, 111, 111, 116, 71, 50, 46, 99, 114, 116, 48, 66, 6, 3, 85, 29, 31, 4, 59, 48, 57, 48, 55, 160, 53, 160, 51, 134, 49, 104, 116, 116, 112, 58, 47, 47, 99, 114, 108, 51, 46, 100, 105, 103, 105, 99, 101, 114, 116, 46, 99, 111, 109, 47, 68, 105, 103, 105, 67, 101, 114, 116, 71, 108, 111, 98, 97, 108, 82, 111, 111, 116, 71, 50, 46, 99, 114, 108, 48, 19, 6, 3, 85, 29, 32, 4, 12, 48, 10, 48, 8, 6, 6, 103, 129, 12, 1, 2, 2],
RawSubjectPublicKeyInfo: [48, 130, 2, 34, 48, 13, 6, 9, 42, 134, 72, 134, 247, 13, 1, 1, 1, 5, 0, 3, 130, 2, 15, 0, 48, 130, 2, 10, 2, 130, 2, 1, 0, 223, 234, 139, 159, 71, 225, 40, 16, 113, 175, 65, 214, 25, 219, 146, 87, 222, 107, 171, 230, 46, 222, 173, 58, 96, 131, 183, 19, 152, 172, 91, 185, 50, 11, 123, 235, 159, 114, 39, 107, 93, 64, 171, 190, 24, 6, 147, 178, 13, 210, 18, 237, 184, 116, 245, 127, 47, 234, 168, 23, 148, 246, 51, 22, 87, 138, 222, 32, 124, 33, 221, 183, 218, 172, 222, 78, 100, 176, 90, 205, 75, 117, 243, 239, 2, 100, 77, 237, 82, 218, 163, 143, 4, 232, 145, 248, 215, 251, 63, 25, 253, 78, 69, 57, 94, 232, 41, 195, 214, 242, 103, 175, 179, 0, 148, 193, 141, 225, 240, 74, 24, 152, 57, 250, 192, 9, 132, 51, 65, 5, 3, 5, 171, 244, 117, 225, 83, 165, 33, 146, 92, 68, 76, 230, 96, 87, 164, 144, 2, 151, 199, 213, 216, 147, 4, 50, 74, 237, 133, 81, 237, 214, 197, 186, 160, 63, 54, 138, 47, 111, 76, 57, 168, 4, 105, 34, 132, 59, 117, 20, 247, 217, 62, 180, 169, 208, 7, 65, 56, 129, 126, 70, 72, 29, 232, 25, 230, 112, 184, 160, 170, 24, 221, 114, 191, 126, 125, 181, 108, 78, 25, 112, 17, 132, 4, 19, 129, 212, 98, 253, 156, 107, 241, 28, 221, 255, 236, 167, 224, 230, 199, 141, 49, 75, 191, 242, 107, 51, 204, 209, 140, 182, 254, 177, 27, 163, 74, 168, 95, 14, 127, 49, 118, 56, 63, 7, 71, 108, 1, 168, 172, 165, 20, 218, 130, 176, 95, 31, 108, 188, 137, 26, 192, 18, 42, 203, 123, 151, 31, 40, 46, 49, 22, 138, 83, 76, 117, 102, 249, 125, 1, 214, 194, 96, 170, 101, 143, 49, 158, 114, 203, 204, 155, 45, 229, 68, 25, 147, 87, 171, 110, 142, 194, 217, 46, 101, 176, 114, 45, 231, 161, 195, 161, 126, 73, 29, 89, 103, 214, 9, 212, 24, 18, 109, 192, 149, 69, 120, 201, 241, 96, 106, 228, 246, 238, 6, 36, 59, 115, 74, 90, 17, 194, 62, 151, 97, 192, 120, 238, 239, 19, 167, 48, 226, 74, 56, 45, 238, 88, 23, 19, 32, 183, 54, 42, 223, 11, 50, 180, 194, 56, 177, 145, 221, 18, 176, 69, 182, 124, 192, 177, 29, 98, 161, 157, 221, 113, 18, 21, 150, 1, 157, 10, 9, 245, 177, 221, 179, 217, 224, 56, 66, 52, 246, 105, 182, 195, 175, 22, 115, 254, 212, 67, 248, 67, 228, 88, 249, 79, 179, 243, 236, 214, 61, 148, 58, 64, 56, 21, 171, 12, 134, 208, 100, 206, 26, 99, 232, 132, 148, 85, 9, 90, 182, 4, 182, 209, 62, 154, 21, 199, 100, 17, 128, 208, 27, 130, 14, 77, 255, 69, 158, 178, 101, 7, 37, 31, 219, 154, 200, 198, 236, 214, 91, 133, 80, 4, 159, 28, 203, 93, 32, 130, 103, 158, 121, 226, 223, 232, 242, 232, 111, 196, 32, 114, 48, 134, 229, 157, 117, 39, 53, 2, 3, 1, 0, 1],
RawSubject: [48, 81, 49, 11, 48, 9, 6, 3, 85, 4, 6, 19, 2, 85, 83, 49, 30, 48, 28, 6, 3, 85, 4, 10, 19, 21, 77, 105, 99, 114, 111, 115, 111, 102, 116, 32, 67, 111, 114, 112, 111, 114, 97, 116, 105, 111, 110, 49, 34, 48, 32, 6, 3, 85, 4, 3, 19, 25, 77, 105, 99, 114, 111, 115, 111, 102, 116, 32, 84, 76, 83, 32, 82, 83, 65, 32, 82, 111, 111, 116, 32, 71, 50],
RawIssuer: [48, 97, 49, 11, 48, 9, 6, 3, 85, 4, 6, 19, 2, 85, 83, 49, 21, 48, 19, 6, 3, 85, 4, 10, 19, 12, 68, 105, 103, 105, 67, 101, 114, 116, 32, 73, 110, 99, 49, 25, 48, 23, 6, 3, 85, 4, 11, 19, 16, 119, 119, 119, 46, 100, 105, 103, 105, 99, 101, 114, 116, 46, 99, 111, 109, 49, 32, 48, 30, 6, 3, 85, 4, 3, 19, 23, 68, 105, 103, 105, 67, 101, 114, 116, 32, 71, 108, 111, 98, 97, 108, 32, 82, 111, 111, 116, 32, 71, 50],
Signature: [11, 188, 180, 43, 55, 116, 197, 75, 167, 54, 2, 54, 198, 175, 225, 19, 204, 138, 154, 151, 27, 252, 227, 68, 140, 238, 84, 0, 229, 228, 65, 26, 70, 240, 11, 118, 52, 255, 153, 85, 98, 53, 170, 173, 138, 156, 12, 90, 98, 204, 67, 192, 203, 234, 74, 29, 189, 107, 29, 175, 177, 25, 37, 102, 156, 247, 121, 110, 91, 37, 128, 36, 129, 133, 199, 121, 45, 53, 95, 66, 9, 120, 237, 159, 226, 101, 213, 196, 32, 80, 130, 16, 143, 106, 58, 237, 24, 142, 72, 137, 171, 166, 9, 248, 225, 128, 252, 6, 117, 22, 39, 37, 90, 143, 184, 58, 112, 108, 24, 71, 240, 99, 244, 83, 14, 217, 21, 53, 32, 243, 145, 61, 19, 1, 160, 226, 250, 36, 8, 69, 86, 119, 209, 103, 169, 49, 66, 208, 87, 255, 26, 38, 190, 197, 67, 216, 87, 169, 99, 255, 175, 243, 58, 216, 246, 123, 75, 16, 162, 151, 161, 207, 102, 190, 103, 103, 9, 249, 181, 76, 231, 171, 25, 43, 191, 34, 61, 125, 222, 198, 247, 239, 99, 56, 44, 221, 86, 196, 249, 9, 82, 218, 223, 227, 136, 111, 213, 109, 107, 247, 8, 65, 47, 142, 62, 134, 49, 159, 241, 124, 74, 245, 214, 97, 164, 70, 70, 40, 239, 87, 186, 176, 111, 146, 216, 210, 119, 37, 237, 123, 166, 41, 206, 59, 114, 121, 17, 212, 224, 167, 84, 109, 53, 43, 164, 122],
SignatureAlgorithm: 5,
PublicKeyAlgorithm: 1,
PublicKey: <*rsa.PublicKey | 0x36a118ac37b0>{
N: {
neg: false,
abs: [8228224839953229621, 16348041302357558304, 2074854447542410873, 14334067402580427935, 12854688567428618952, 14995722382209926558, 15077658020813214080, 9553354208224609462, 902637906707506152, 15437658075602359723, 4892132923038495724, 14100513645772686328, 15699610158021110198, 116260203274558899, 7107135261377697174, 1346652938663145757, 806906029520556509, 6347563231035665119, 1416154356297248238, 13996791138010853103, 17151436068643035665, 5005973024364225782, 15423091598760460437, 11656337824329587047, 14040304327591079399, 16520357453723561614, 10318202437165161261, 17977527309428632165, 3328466382962324838, 13840171658303184680, 9417131409404627226, 5146490298140267738, 6849551933690625799, 10139571647697603240, 3552143779013315793, 15996764607220205453, 9355211167345930524, 7804203173400544275, 12256789870797553077, 5196564742993393824, 12236288164267916870, 9528338169550421684, 3418034452713204002, 17138102837348087434, 15605821747820856657, 6942198289040656341, 6027260588802198758, 4685154310243513825, 1772230202496746547, 12898472891819094090, 6838761955263670191, 15563102318165378361, 5970264090691015160, 5437520332982603245, 15757213624540617421, 6308098459188714935, 3452756883569783574, 995879279813522815, 6719559476972000178, 3606112178562803563, 6954603544383871929, 16027092704045215034, 8191838635022848599, 16134862131230222352],
},
E: 65537,
},
Version: 3,
SerialNumber: {
neg: false,
abs: [5148676810827874504, 796129072041760688],
},
Issuer: {
Country: ["US"],
Organization: ["DigiCert Inc"],
OrganizationalUnit: ["www.digicert.com"],
Locality: nil,
Province: nil,
StreetAddress: nil,
PostalCode: nil,
SerialNumber: "",
CommonName: "DigiCert Global Root G2",
Names: [
{
Type: [..., ..., ..., ...],
Value: <string>"US",
},
{
Type: [..., ..., ..., ...],
Value: <string>"DigiCert Inc",
},
{
Type: [..., ..., ..., ...],
Value: <string>"www.digicert.com",
},
{
Type: [..., ..., ..., ...],
Value: <string>"DigiCert Global Root G2",
},
],
ExtraNames: nil,
},
Subject: {
Country: ["US"],
Organization: [
"Microsoft Corporation",
],
OrganizationalUnit: nil,
Locality: nil,
Province: nil,
StreetAddress: nil,
PostalCode: nil,
SerialNumber: "",
CommonName: "Microsoft TLS RSA Root G2",
Names: [
{
Type: [..., ..., ..., ...],
Value: <string>"US",
},
{
Type: [..., ..., ..., ...],
Value: <string>"Microsoft Corporation",
},
{
Type: [..., ..., ..., ...],
Value: <string>"Microsoft TLS RSA Root G2",
},
],
ExtraNames: nil,
},
NotBefore: {wall: 0, ext: 63883382400, loc: nil},
NotAfter: {wall: 0, ext: 64012204799, loc: nil},
KeyUsage: 97,
Extensions: [
{
Id: [2, 5, 29, 19],
Critical: true,
Value: [48, 3, 1, 1, 255],
},
{
Id: [2, 5, 29, 14],
Critical: false,
Value: [4, 20, 222, 145, 134, 72, 183, 161, 49, 89, 49, 241, 75, 95, 7, 169, 220, 136, 121, 218, 168, 118],
},
{
Id: [2, 5, 29, 35],
Critical: false,
Value: [48, 22, 128, 20, 78, 34, 84, 32, 24, 149, 230, 227, 110, 230, 15, 250, 250, 185, 18, 237, 6, 23, 143, 57],
},
{
Id: [2, 5, 29, 15],
Critical: true,
Value: [3, 2, 1, 134],
},
{
Id: [2, 5, 29, 37],
Critical: false,
Value: [48, 10, 6, 8, 43, 6, 1, 5, 5, 7, 3, 1],
},
{
Id: [1, 3, 6, 1, 5, 5, 7, 1, 1],
Critical: false,
Value: [48, 104, 48, 36, 6, 8, 43, 6, 1, 5, 5, 7, 48, 1, 134, 24, 104, 116, 116, 112, 58, 47, 47, 111, 99, 115, 112, 46, 100, 105, 103, 105, 99, 101, 114, 116, 46, 99, 111, 109, 48, 64, 6, 8, 43, 6, 1, 5, 5, 7, 48, 2, 134, 52, 104, 116, 116, 112, 58, 47, 47, 99, 97, 99, 101, 114, 116, 115, 46, 100, 105, 103, 105, 99, 101, 114, 116, 46, 99, 111, 109, 47, 68, 105, 103, 105, 67, 101, 114, 116, 71, 108, 111, 98, 97, 108, 82, 111, 111, 116, 71, 50, 46, 99, 114, 116],
},
{
Id: [2, 5, 29, 31],
Critical: false,
Value: "0907\xa05\xa03\x861http://crl3.digicert.com/DigiCertGlobalRootG2.crl",
},
{
Id: [2, 5, 29, 32],
Critical: false,
Value: [48, 10, 48, 8, 6, 6, 103, 129, 12, 1, 2, 2],
},
],
ExtraExtensions: nil,
UnhandledCriticalExtensions: nil,
ExtKeyUsage: [1],
UnknownExtKeyUsage: nil,
BasicConstraintsValid: true,
IsCA: true,
MaxPathLen: -1,
MaxPathLenZero: false,
SubjectKeyId: [222, 145, 134, 72, 183, 161, 49, 89, 49, 241, 75, 95, 7, 169, 220, 136, 121, 218, 168, 118],
AuthorityKeyId: [78, 34, 84, 32, 24, 149, 230, 227, 110, 230, 15, 250, 250, 185, 18, 237, 6, 23, 143, 57],
OCSPServer: [
"http://ocsp.digicert.com",
],
IssuingCertificateURL: [
"http://cacerts.digicert.com/DigiCertGlobalRootG2.crt",
],
DNSNames: nil,
EmailAddresses: nil,
IPAddresses: nil,
URIs: nil,
PermittedDNSDomainsCritical: false,
PermittedDNSDomains: nil,
ExcludedDNSDomains: nil,
PermittedIPRanges: nil,
ExcludedIPRanges: nil,
PermittedEmailAddresses: nil,
ExcludedEmailAddresses: nil,
PermittedURIDomains: nil,
ExcludedURIDomains: nil,
CRLDistributionPoints: [
"http://crl3.digicert.com/DigiCertGlobalRootG2.crl",
],
PolicyIdentifiers: [
[2, 23, 140, 1, 2, 2],
],
Policies: [
{
der: [103, 129, 12, 1, 2, 2],
},
],
InhibitAnyPolicy: 0,
InhibitAnyPolicyZero: false,
InhibitPolicyMapping: 0,
InhibitPolicyMappingZero: false,
RequireExplicitPolicy: 0,
RequireExplicitPolicyZero: false,
PolicyMappings: nil,
},
{
Raw: [48, 130, 3, 142, 48, 130, 2, 118, 160, 3, 2, 1, 2, 2, 16, 3, 58, 241, 230, 167, 17, 169, 160, 187, 40, 100, 177, 29, 9, 250, 229, 48, 13, 6, 9, 42, 134, 72, 134, 247, 13, 1, 1, 11, 5, 0, 48, 97, 49, 11, 48, 9, 6, 3, 85, 4, 6, 19, 2, 85, 83, 49, 21, 48, 19, 6, 3, 85, 4, 10, 19, 12, 68, 105, 103, 105, 67, 101, 114, 116, 32, 73, 110, 99, 49, 25, 48, 23, 6, 3, 85, 4, 11, 19, 16, 119, 119, 119, 46, 100, 105, 103, 105, 99, 101, 114, 116, 46, 99, 111, 109, 49, 32, 48, 30, 6, 3, 85, 4, 3, 19, 23, 68, 105, 103, 105, 67, 101, 114, 116, 32, 71, 108, 111, 98, 97, 108, 32, 82, 111, 111, 116, 32, 71, 50, 48, 30, 23, 13, 49, 51, 48, 56, 48, 49, 49, 50, 48, 48, 48, 48, 90, 23, 13, 51, 56, 48, 49, 49, 53, 49, 50, 48, 48, 48, 48, 90, 48, 97, 49, 11, 48, 9, 6, 3, 85, 4, 6, 19, 2, 85, 83, 49, 21, 48, 19, 6, 3, 85, 4, 10, 19, 12, 68, 105, 103, 105, 67, 101, 114, 116, 32, 73, 110, 99, 49, 25, 48, 23, 6, 3, 85, 4, 11, 19, 16, 119, 119, 119, 46, 100, 105, 103, 105, 99, 101, 114, 116, 46, 99, 111, 109, 49, 32, 48, 30, 6, 3, 85, 4, 3, 19, 23, 68, 105, 103, 105, 67, 101, 114, 116, 32, 71, 108, 111, 98, 97, 108, 32, 82, 111, 111, 116, 32, 71, 50, 48, 130, 1, 34, 48, 13, 6, 9, 42, 134, 72, 134, 247, 13, 1, 1, 1, 5, 0, 3, 130, 1, 15, 0, 48, 130, 1, 10, 2, 130, 1, 1, 0, 187, 55, 205, 52, 220, 123, 107, 201, 178, 104, 144, 173, 74, 117, 255, 70, 186, 33, 10, 8, 141, 245, 25, 84, 201, 251, 136, 219, 243, 174, 242, 58, 137, 145, 60, 122, 230, 171, 6, 26, 107, 207, 172, 45, 232, 94, 9, 36, 68, 186, 98, 154, 126, 214, 163, 168, 126, 224, 84, 117, 32, 5, 172, 80, 183, 156, 99, 26, 108, 48, 220, 218, 31, 25, 177, 215, 30, 222, 253, 215, 224, 203, 148, 131, 55, 174, 236, 31, 67, 78, 221, 123, 44, 210, 189, 46, 165, 47, 228, 169, 184, 173, 58, 212, 153, 164, 182, 37, 233, 155, 107, 0, 96, 146, 96, 255, 79, 33, 73, 24, 247, 103, 144, 171, 97, 6, 156, 143, 242, 186, 233, 180, 233, 146, 50, 107, 181, 243, 87, 232, 93, 27, 205, 140, 29, 171, 149, 4, 149, 73, 243, 53, 45, 150, 227, 73, 109, 221, 119, 227, 251, 73, 75, 180, 172, 85, 7, 169, 143, 149, 179, 180, 35, 187, 76, 109, 69, 240, 246, 169, 178, 149, 48, 180, 253, 76, 85, 140, 39, 74, 87, 20, 124, 130, 157, 205, 115, 146, 211, 22, 74, 6, 12, 140, 80, 209, 143, 30, 9, 190, 23, 161, 230, 33, 202, 253, 131, 229, 16, 188, 131, 165, 10, 196, 103, 40, 246, 115, 20, 20, 61, 70, 118, 195, 135, 20, 137, 33, 52, 77, 175, 15, 69, 12, 166, 73, 161, 186, 187, 156, 197, 177, 51, 131, 41, 133, 2, 3, 1, 0, 1, 163, 66, 48, 64, 48, 15, 6, 3, 85, 29, 19, 1, 1, 255, 4, 5, 48, 3, 1, 1, 255, 48, 14, 6, 3, 85, 29, 15, 1, 1, 255, 4, 4, 3, 2, 1, 134, 48, 29, 6, 3, 85, 29, 14, 4, 22, 4, 20, 78, 34, 84, 32, 24, 149, 230, 227, 110, 230, 15, 250, 250, 185, 18, 237, 6, 23, 143, 57, 48, 13, 6, 9, 42, 134, 72, 134, 247, 13, 1, 1, 11, 5, 0, 3, 130, 1, 1, 0, 96, 103, 40, 148, 111, 14, 72, 99, 235, 49, 221, 234, 103, 24, 213, 137, 125, 60, 197, 139, 74, 127, 233, 190, 219, 43, 23, 223, 176, 95, 115, 119, 42, 50, 19, 57, 129, 103, 66, 132, 35, 242, 69, 103, 53, 236, 136, 191, 248, 143, 176, 97, 12, 52, 164, 174, 32, 76, 132, 198, 219, 248, 53, 225, 118, 217, 223, 166, 66, 187, 199, 68, 8, 134, 127, 54, 116, 36, 90, 218, 108, 13, 20, 89, 53, 189, 242, 73, 221, 182, 31, 201, 179, 13, 71, 42, 61, 153, 47, 187, 92, 187, 181, 212, 32, 225, 153, 95, 83, 70, 21, 219, 104, 155, 240, 243, 48, 213, 62, 49, 226, 141, 132, 158, 227, 138, 218, 218, 150, 62, 53, 19, 165, 95, 240, 249, 112, 80, 112, 71, 65, 17, 87, 25, 78, 192, 143, 174, 6, 196, 149, 19, 23, 47, 27, 37, 159, 117, 242, 177, 142, 153, 161, 111, 19, 177, 65, 113, 254, 136, 42, 200, 79, 16, 32, 85, 215, 243, 20, 69, 229, 224, 68, 244, 234, 135, 149, 50, 147, 14, 254, 83, 70, 250, 44, 157, 255, 139, 34, 185, 75, 217, 9, 69, 164, 222, 164, 184, 154, 88, 221, 27, 125, 82, 159, 142, 89, 67, 136, 129, 164, 158, 38, 213, 111, 173, 221, 13, 198, 55, 125, 237, 3, 146, 27, 229, 119, 95, 118, 238, 60, 141, 196, 93, 86, 91, 162, 217, 102, 110, 179, 53, 55, 229, 50, 182],
RawTBSCertificate: [48, 130, 2, 118, 160, 3, 2, 1, 2, 2, 16, 3, 58, 241, 230, 167, 17, 169, 160, 187, 40, 100, 177, 29, 9, 250, 229, 48, 13, 6, 9, 42, 134, 72, 134, 247, 13, 1, 1, 11, 5, 0, 48, 97, 49, 11, 48, 9, 6, 3, 85, 4, 6, 19, 2, 85, 83, 49, 21, 48, 19, 6, 3, 85, 4, 10, 19, 12, 68, 105, 103, 105, 67, 101, 114, 116, 32, 73, 110, 99, 49, 25, 48, 23, 6, 3, 85, 4, 11, 19, 16, 119, 119, 119, 46, 100, 105, 103, 105, 99, 101, 114, 116, 46, 99, 111, 109, 49, 32, 48, 30, 6, 3, 85, 4, 3, 19, 23, 68, 105, 103, 105, 67, 101, 114, 116, 32, 71, 108, 111, 98, 97, 108, 32, 82, 111, 111, 116, 32, 71, 50, 48, 30, 23, 13, 49, 51, 48, 56, 48, 49, 49, 50, 48, 48, 48, 48, 90, 23, 13, 51, 56, 48, 49, 49, 53, 49, 50, 48, 48, 48, 48, 90, 48, 97, 49, 11, 48, 9, 6, 3, 85, 4, 6, 19, 2, 85, 83, 49, 21, 48, 19, 6, 3, 85, 4, 10, 19, 12, 68, 105, 103, 105, 67, 101, 114, 116, 32, 73, 110, 99, 49, 25, 48, 23, 6, 3, 85, 4, 11, 19, 16, 119, 119, 119, 46, 100, 105, 103, 105, 99, 101, 114, 116, 46, 99, 111, 109, 49, 32, 48, 30, 6, 3, 85, 4, 3, 19, 23, 68, 105, 103, 105, 67, 101, 114, 116, 32, 71, 108, 111, 98, 97, 108, 32, 82, 111, 111, 116, 32, 71, 50, 48, 130, 1, 34, 48, 13, 6, 9, 42, 134, 72, 134, 247, 13, 1, 1, 1, 5, 0, 3, 130, 1, 15, 0, 48, 130, 1, 10, 2, 130, 1, 1, 0, 187, 55, 205, 52, 220, 123, 107, 201, 178, 104, 144, 173, 74, 117, 255, 70, 186, 33, 10, 8, 141, 245, 25, 84, 201, 251, 136, 219, 243, 174, 242, 58, 137, 145, 60, 122, 230, 171, 6, 26, 107, 207, 172, 45, 232, 94, 9, 36, 68, 186, 98, 154, 126, 214, 163, 168, 126, 224, 84, 117, 32, 5, 172, 80, 183, 156, 99, 26, 108, 48, 220, 218, 31, 25, 177, 215, 30, 222, 253, 215, 224, 203, 148, 131, 55, 174, 236, 31, 67, 78, 221, 123, 44, 210, 189, 46, 165, 47, 228, 169, 184, 173, 58, 212, 153, 164, 182, 37, 233, 155, 107, 0, 96, 146, 96, 255, 79, 33, 73, 24, 247, 103, 144, 171, 97, 6, 156, 143, 242, 186, 233, 180, 233, 146, 50, 107, 181, 243, 87, 232, 93, 27, 205, 140, 29, 171, 149, 4, 149, 73, 243, 53, 45, 150, 227, 73, 109, 221, 119, 227, 251, 73, 75, 180, 172, 85, 7, 169, 143, 149, 179, 180, 35, 187, 76, 109, 69, 240, 246, 169, 178, 149, 48, 180, 253, 76, 85, 140, 39, 74, 87, 20, 124, 130, 157, 205, 115, 146, 211, 22, 74, 6, 12, 140, 80, 209, 143, 30, 9, 190, 23, 161, 230, 33, 202, 253, 131, 229, 16, 188, 131, 165, 10, 196, 103, 40, 246, 115, 20, 20, 61, 70, 118, 195, 135, 20, 137, 33, 52, 77, 175, 15, 69, 12, 166, 73, 161, 186, 187, 156, 197, 177, 51, 131, 41, 133, 2, 3, 1, 0, 1, 163, 66, 48, 64, 48, 15, 6, 3, 85, 29, 19, 1, 1, 255, 4, 5, 48, 3, 1, 1, 255, 48, 14, 6, 3, 85, 29, 15, 1, 1, 255, 4, 4, 3, 2, 1, 134, 48, 29, 6, 3, 85, 29, 14, 4, 22, 4, 20, 78, 34, 84, 32, 24, 149, 230, 227, 110, 230, 15, 250, 250, 185, 18, 237, 6, 23, 143, 57],
RawSubjectPublicKeyInfo: [48, 130, 1, 34, 48, 13, 6, 9, 42, 134, 72, 134, 247, 13, 1, 1, 1, 5, 0, 3, 130, 1, 15, 0, 48, 130, 1, 10, 2, 130, 1, 1, 0, 187, 55, 205, 52, 220, 123, 107, 201, 178, 104, 144, 173, 74, 117, 255, 70, 186, 33, 10, 8, 141, 245, 25, 84, 201, 251, 136, 219, 243, 174, 242, 58, 137, 145, 60, 122, 230, 171, 6, 26, 107, 207, 172, 45, 232, 94, 9, 36, 68, 186, 98, 154, 126, 214, 163, 168, 126, 224, 84, 117, 32, 5, 172, 80, 183, 156, 99, 26, 108, 48, 220, 218, 31, 25, 177, 215, 30, 222, 253, 215, 224, 203, 148, 131, 55, 174, 236, 31, 67, 78, 221, 123, 44, 210, 189, 46, 165, 47, 228, 169, 184, 173, 58, 212, 153, 164, 182, 37, 233, 155, 107, 0, 96, 146, 96, 255, 79, 33, 73, 24, 247, 103, 144, 171, 97, 6, 156, 143, 242, 186, 233, 180, 233, 146, 50, 107, 181, 243, 87, 232, 93, 27, 205, 140, 29, 171, 149, 4, 149, 73, 243, 53, 45, 150, 227, 73, 109, 221, 119, 227, 251, 73, 75, 180, 172, 85, 7, 169, 143, 149, 179, 180, 35, 187, 76, 109, 69, 240, 246, 169, 178, 149, 48, 180, 253, 76, 85, 140, 39, 74, 87, 20, 124, 130, 157, 205, 115, 146, 211, 22, 74, 6, 12, 140, 80, 209, 143, 30, 9, 190, 23, 161, 230, 33, 202, 253, 131, 229, 16, 188, 131, 165, 10, 196, 103, 40, 246, 115, 20, 20, 61, 70, 118, 195, 135, 20, 137, 33, 52, 77, 175, 15, 69, 12, 166, 73, 161, 186, 187, 156, 197, 177, 51, 131, 41, 133, 2, 3, 1, 0, 1],
RawSubject: [48, 97, 49, 11, 48, 9, 6, 3, 85, 4, 6, 19, 2, 85, 83, 49, 21, 48, 19, 6, 3, 85, 4, 10, 19, 12, 68, 105, 103, 105, 67, 101, 114, 116, 32, 73, 110, 99, 49, 25, 48, 23, 6, 3, 85, 4, 11, 19, 16, 119, 119, 119, 46, 100, 105, 103, 105, 99, 101, 114, 116, 46, 99, 111, 109, 49, 32, 48, 30, 6, 3, 85, 4, 3, 19, 23, 68, 105, 103, 105, 67, 101, 114, 116, 32, 71, 108, 111, 98, 97, 108, 32, 82, 111, 111, 116, 32, 71, 50],
RawIssuer: [48, 97, 49, 11, 48, 9, 6, 3, 85, 4, 6, 19, 2, 85, 83, 49, 21, 48, 19, 6, 3, 85, 4, 10, 19, 12, 68, 105, 103, 105, 67, 101, 114, 116, 32, 73, 110, 99, 49, 25, 48, 23, 6, 3, 85, 4, 11, 19, 16, 119, 119, 119, 46, 100, 105, 103, 105, 99, 101, 114, 116, 46, 99, 111, 109, 49, 32, 48, 30, 6, 3, 85, 4, 3, 19, 23, 68, 105, 103, 105, 67, 101, 114, 116, 32, 71, 108, 111, 98, 97, 108, 32, 82, 111, 111, 116, 32, 71, 50],
Signature: [96, 103, 40, 148, 111, 14, 72, 99, 235, 49, 221, 234, 103, 24, 213, 137, 125, 60, 197, 139, 74, 127, 233, 190, 219, 43, 23, 223, 176, 95, 115, 119, 42, 50, 19, 57, 129, 103, 66, 132, 35, 242, 69, 103, 53, 236, 136, 191, 248, 143, 176, 97, 12, 52, 164, 174, 32, 76, 132, 198, 219, 248, 53, 225, 118, 217, 223, 166, 66, 187, 199, 68, 8, 134, 127, 54, 116, 36, 90, 218, 108, 13, 20, 89, 53, 189, 242, 73, 221, 182, 31, 201, 179, 13, 71, 42, 61, 153, 47, 187, 92, 187, 181, 212, 32, 225, 153, 95, 83, 70, 21, 219, 104, 155, 240, 243, 48, 213, 62, 49, 226, 141, 132, 158, 227, 138, 218, 218, 150, 62, 53, 19, 165, 95, 240, 249, 112, 80, 112, 71, 65, 17, 87, 25, 78, 192, 143, 174, 6, 196, 149, 19, 23, 47, 27, 37, 159, 117, 242, 177, 142, 153, 161, 111, 19, 177, 65, 113, 254, 136, 42, 200, 79, 16, 32, 85, 215, 243, 20, 69, 229, 224, 68, 244, 234, 135, 149, 50, 147, 14, 254, 83, 70, 250, 44, 157, 255, 139, 34, 185, 75, 217, 9, 69, 164, 222, 164, 184, 154, 88, 221, 27, 125, 82, 159, 142, 89, 67, 136, 129, 164, 158, 38, 213, 111, 173, 221, 13, 198, 55, 125, 237, 3, 146, 27, 229, 119, 95, 118, 238, 60, 141, 196, 93, 86, 91, 162, 217, 102, 110, 179, 53, 55, 229, 50, 182],
SignatureAlgorithm: 4,
PublicKeyAlgorithm: 1,
PublicKey: <*rsa.PublicKey | 0x36a11782c840>{
N: {
neg: false,
abs: [13518897546323503493, 12614377001919816122, 8557832239203824717, 7433462159093546310, 9504020990483827396, 702024576034523901, 5333964605457796894, 8971906913408307990, 18252057450268743444, 5039799191733612724, 10346373305228545133, 18107087016177239977, 3285062882728703971, 2137966295034491701, 13110919595711778188, 17490549066068800107, 17827376715711028367, 6958731023901346072, 11071174057932843776, 11902983757762476756, 4850057369827327278, 16198203776063958047, 2241017827087547863, 13230558770443115738, 9142400105587256400, 4952379155939763112, 7768617195409377572, 9912770753372816922, 14554377099045433914, 13412012197143714132, 12855684210282069830, 13490476835732679625],
},
E: 65537,
},
Version: 3,
SerialNumber: {
neg: false,
abs: [13486139796020132581, 232764303710726560],
},
Issuer: {
Country: ["US"],
Organization: ["DigiCert Inc"],
OrganizationalUnit: ["www.digicert.com"],
Locality: nil,
Province: nil,
StreetAddress: nil,
PostalCode: nil,
SerialNumber: "",
CommonName: "DigiCert Global Root G2",
Names: [
{
Type: [..., ..., ..., ...],
Value: <string>"US",
},
{
Type: [..., ..., ..., ...],
Value: <string>"DigiCert Inc",
},
{
Type: [..., ..., ..., ...],
Value: <string>"www.digicert.com",
},
{
Type: [..., ..., ..., ...],
Value: <string>"DigiCert Global Root G2",
},
],
ExtraNames: nil,
},
Subject: {
Country: ["US"],
Organization: ["DigiCert Inc"],
OrganizationalUnit: ["www.digicert.com"],
Locality: nil,
Province: nil,
StreetAddress: nil,
PostalCode: nil,
SerialNumber: "",
CommonName: "DigiCert Global Root G2",
Names: [
{
Type: [..., ..., ..., ...],
Value: <string>"US",
},
{
Type: [..., ..., ..., ...],
Value: <string>"DigiCert Inc",
},
{
Type: [..., ..., ..., ...],
Value: <string>"www.digicert.com",
},
{
Type: [..., ..., ..., ...],
Value: <string>"DigiCert Global Root G2",
},
],
ExtraNames: nil,
},
NotBefore: {wall: 0, ext: 63510955200, loc: nil},
NotAfter: {wall: 0, ext: 64282766400, loc: nil},
KeyUsage: 97,
Extensions: [
{
Id: [2, 5, 29, 19],
Critical: true,
Value: [48, 3, 1, 1, 255],
},
{
Id: [2, 5, 29, 15],
Critical: true,
Value: [3, 2, 1, 134],
},
{
Id: [2, 5, 29, 14],
Critical: false,
Value: [4, 20, 78, 34, 84, 32, 24, 149, 230, 227, 110, 230, 15, 250, 250, 185, 18, 237, 6, 23, 143, 57],
},
],
ExtraExtensions: nil,
UnhandledCriticalExtensions: nil,
ExtKeyUsage: nil,
UnknownExtKeyUsage: nil,
BasicConstraintsValid: true,
IsCA: true,
MaxPathLen: -1,
MaxPathLenZero: false,
SubjectKeyId: [78, 34, 84, 32, 24, 149, 230, 227, 110, 230, 15, 250, 250, 185, 18, 237, 6, 23, 143, 57],
AuthorityKeyId: nil,
OCSPServer: nil,
IssuingCertificateURL: nil,
DNSNames: nil,
EmailAddresses: nil,
IPAddresses: nil,
URIs: nil,
PermittedDNSDomainsCritical: false,
PermittedDNSDomains: nil,
ExcludedDNSDomains: nil,
PermittedIPRanges: nil,
ExcludedIPRanges: nil,
PermittedEmailAddresses: nil,
ExcludedEmailAddresses: nil,
PermittedURIDomains: nil,
ExcludedURIDomains: nil,
CRLDistributionPoints: nil,
PolicyIdentifiers: nil,
Policies: nil,
InhibitAnyPolicy: 0,
InhibitAnyPolicyZero: false,
InhibitPolicyMapping: 0,
InhibitPolicyMappingZero: false,
RequireExplicitPolicy: 0,
RequireExplicitPolicyZero: false,
PolicyMappings: nil,
},
],
Err: <x509.UnknownAuthorityError>{
Cert: 0x36a1182f5208 (cyclic reference),
hintErr: nil,
hintCert: nil,
},
},
},
},
}
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:525]: sample application must serve the exact customer-specified certificate with a valid Let's Encrypt chain
Expected success, but got an error:
<*fmt.wrapError | 0x36a1186625c0>:
VerifyCustomIngressCertificate timed out after 10m0.003s: ingress route must serve HTTPS with the expected certificate: Get "https://agnhost-e2e-sample-app-l49xr.apps.aro.ingress-cert-le.0o3m.uksouth.aroapp-hcp-stg.azure.com": tls: failed to verify certificate: x509: certificate signed by unknown authority
{
msg: "VerifyCustomIngressCertificate timed out after 10m0.003s: ingress route must serve HTTPS with the expected certificate: Get \"https://agnhost-e2e-sample-app-l49xr.apps.aro.ingress-cert-le.0o3m.uksouth.aroapp-hcp-stg.azure.com\": tls: failed to verify certificate: x509: certificate signed by unknown authority",
err: <*fmt.wrapError | 0x36a1186623e0>{
msg: "ingress route must serve HTTPS with the expected certificate: Get \"https://agnhost-e2e-sample-app-l49xr.apps.aro.ingress-cert-le.0o3m.uksouth.aroapp-hcp-stg.azure.com\": tls: failed to verify certificate: x509: certificate signed by unknown authority",
err: <*url.Error | 0x36a118d74840>{
Op: "Get",
URL: "https://agnhost-e2e-sample-app-l49xr.apps.aro.ingress-cert-le.0o3m.uksouth.aroapp-hcp-stg.azure.com",
Err: <*tls.CertificateVerificationError | 0x36a118d74810>{
UnverifiedCertificates: [
{
Raw: [48, 130, 9, 19, 48, 130, 6, 251, 160, 3, 2, 1, 2, 2, 19, 73, 1, 74, 89, 119, 169, 186, 32, 59, 56, 235, 138, 87, 0, 0, 1, 74, 89, 119, 48, 13, 6, 9, 42, 134, 72, 134, 247, 13, 1, 1, 12, 5, 0, 48, 87, 49, 11, 48, 9, 6, 3, 85, 4, 6, 19, 2, 85, 83, 49, 30, 48, 28, 6, 3, 85, 4, 10, 19, 21, 77, 105, 99, 114, 111, 115, 111, 102, 116, 32, 67, 111, 114, 112, 111, 114, 97, 116, 105, 111, 110, 49, 40, 48, 38, 6, 3, 85, 4, 3, 19, 31, 77, 105, 99, 114, 111, 115, 111, 102, 116, 32, 84, 76, 83, 32, 71, 50, 32, 82, 83, 65, 32, 67, 65, 32, 79, 67, 83, 80, 32, 49, 48, 48, 30, 23, 13, 50, 54, 49, 48, 50, 53, 48, 49, 53, 54, 49, 48, 90, 23, 13, 50, 55, 48, 50, 48, 50, 48, 48, 53, 54, 49, 48, 90, 48, 120, 49, 11, 48, 9, 6, 3, 85, 4, 6, 19, 2, 85, 83, 49, 11, 48, 9, 6, 3, 85, 4, 8, 19, 2, 87, 65, 49, 16, 48, 14, 6, 3, 85, 4, 7, 19, 7, 82, 101, 100, 109, 111, 110, 100, 49, 30, 48, 28, 6, 3, 85, 4, 10, 19, 21, 77, 105, 99, 114, 111, 115, 111, 102, 116, 32, 67, 111, 114, 112, 111, 114, 97, 116, 105, 111, 110, 49, 42, 48, 40, 6, 3, 85, 4, 3, 19, 33, 114, 101, 115, 101, 114, 118, 101, 100, 46, 97, 114, 111, 97, 112, 112, 45, 104, 99, 112, 45, 115, 116, 103, 46, 97, 122, 117, 114, 101, 46, 99, 111, 109, 48, 130, 1, 34, 48, 13, 6, 9, 42, 134, 72, 134, 247, 13, 1, 1, 1, 5, 0, 3, 130, 1, 15, 0, 48, 130, 1, 10, 2, 130, 1, 1, 0, 187, 121, 183, 117, 223, 172, 129, 21, 158, 33, 40, 228, 200, 112, 100, 153, 170, 116, 250, 158, 245, 201, 135, 0, 237, 144, 31, 12, 163, 250, 168, 50, 92, 5, 220, 113, 137, 143, 3, 193, 167, 220, 193, 78, 195, 131, 29, 107, 67, 52, 86, 236, 30, 39, 30, 107, 206, 213, 234, 58, 181, 33, 44, 227, 147, 157, 23, 219, 20, 127, 20, 235, 246, 77, 135, 5, 240, 135, 231, 58, 154, 101, 19, 42, 160, 128, 73, 94, 79, 155, 62, 172, 213, 53, 57, 2, 149, 60, 188, 169, 233, 44, 216, 105, 66, 159, 148, 70, 178, 113, 177, 128, 123, 151, 10, 7, 254, 171, 205, 105, 34, 145, 93, 19, 218, 220, 219, 47, 129, 42, 85, 247, 229, 7, 118, 48, 182, 125, 84, 45, 187, 16, 138, 130, 74, 176, 14, 70, 229, 120, 108, 212, 172, 177, 174, 141, 161, 231, 167, 5, 108, 191, 229, 39, 27, 122, 242, 198, 32, 78, 125, 8, 168, 234, 200, 195, 10, 209, 101, 183, 248, 26, 10, 82, 78, 47, 196, 44, 55, 199, 207, 209, 35, 65, 164, 44, 162, 97, 81, 36, 122, 188, 105, 82, 214, 39, 252, 78, 115, 249, 159, 36, 212, 167, 6, 72, 63, 56, 220, 88, 50, 162, 50, 174, 185, 196, 246, 208, 84, 90, 212, 218, 52, 118, 184, 101, 201, 132, 48, 29, 45, 113, 80, 81, 157, 255, 235, 218, 49, 139, 40, 124, 111, 147, 143, 201, 2, 3, 1, 0, 1, 163, 130, 4, 181, 48, 130, 4, 177, 48, 130, 1, 4, 6, 10, 43, 6, 1, 4, 1, 214, 121, 2, 4, 2, 4, 129, 245, 4, 129, 242, 0, 240, 0, 118, 0, 68, 194, 189, 12, 233, 20, 14, 100, 165, 201, 74, 1, 147, 10, 90, 161, 187, 53, 151, 14, 0, 238, 17, 22, 137, 104, 42, 28, 68, 215, 181, 102, 0, 0, 1, 161, 112, 206, 239, 163, 0, 0, 4, 3, 0, 71, 48, 69, 2, 33, 0, 172, 23, 89, 180, 174, 255, 10, 75, 120, 126, 2, 114, 141, 52, 53, 9, 237, 84, 209, 98, 89, 54, 179, 9, 54, 118, 245, 178, 107, 207, 5, 69, 2, 32, 89, 15, 78, 239, 171, 216, 123, 169, 212, 117, 5, 12, 137, 15, 63, 83, 41, 96, 223, 14, 136, 244, 40, 31, 142, 25, 69, 157, 200, 47, 205, 167, 0, 118, 0, 0, 26, 93, 26, 28, 45, 147, 117, 182, 72, 85, 120, 248, 47, 113, 161, 174, 110, 239, 57, 125, 41, 124, 138, 227, 21, 123, 202, 222, 225, 160, 30, 0, 0, 1, 161, 112, 206, 240, 23, 0, 0, 4, 3, 0, 71, 48, 69, 2, 33, 0, 209, 5, 94, 127, 36, 105, 13, 219, 166, 20, 49, 161, 208, 24, 148, 233, 208, 121, 79, 240, 195, 85, 152, 81, 108, 234, 202, 116, 68, 189, 194, 57, 2, 32, 65, 141, 139, 86, 66, 203, 70, 108, 26, 159, 237, 96, 32, 237, 241, 53, 230, 112, 171, 119, 90, 74, 10, 14, 99, 1, 225, 78, 245, 158, 165, 135, 48, 27, 6, 9, 43, 6, 1, 4, 1, 130, 55, 21, 10, 4, 14, 48, 12, 48, 10, 6, 8, 43, 6, 1, 5, 5, 7, 3, 1, 48, 60, 6, 9, 43, 6, 1, 4, 1, 130, 55, 21, 7, 4, 47, 48, 45, 6, 37, 43, 6, 1, 4, 1, 130, 55, 21, 8, 135, 189, 215, 27, 129, 231, 235, 70, 130, 129, 157, 46, 142, 208, 12, 135, 240, 218, 29, 93, 133, 246, 239, 42, 129, 232, 226, 53, 2, 1, 100, 2, 1, 39, 48, 130, 1, 11, 6, 8, 43, 6, 1, 5, 5, 7, 1, 1, 4, 129, 254, 48, 129, 251, 48, 97, 6, 8, 43, 6, 1, 5, 5, 7, 48, 2, 134, 85, 104, 116, 116, 112, 58, 47, 47, 119, 119, 119, 46, 109, 105, 99, 114, 111, 115, 111, 102, 116, 46, 99, 111, 109, 47, 112, 107, 105, 111, 112, 115, 47, 99, 101, 114, 116, 115, 47, 77, 105, 99, 114, 111, 115, 111, 102, 116, 37, 50, 48, 84, 76, 83, 37, 50, 48, 71, 50, 37, 50, 48, 82, 83, 65, 37, 50, 48, 67, 65, 37, 50, 48, 79, 67, 83, 80, 37, 50, 48, 49, 48, 46, 99, 114, 116, 48, 103, 6, 8, 43, 6, 1, 5, 5, 7, 48, 2, 134, 91, 104, 116, 116, 112, 58, 47, 47, 99, 97, 105, 115, 115, 117, 101, 114, 115, 46, 109, 105, 99, 114, 111, 115, 111, 102, 116, 46, 99, 111, 109, 47, 112, 107, 105, 111, 112, 115, 47, 99, 101, 114, 116, 115, 47, 77, 105, 99, 114, 111, 115, 111, 102, 116, 37, 50, 48, 84, 76, 83, 37, 50, 48, 71, 50, 37, 50, 48, 82, 83, 65, 37, 50, 48, 67, 65, 37, 50, 48, 79, 67, 83, 80, 37, 50, 48, 49, 48, 46, 99, 114, 116, 48, 45, 6, 8, 43, 6, 1, 5, 5, 7, 48, 1, 134, 33, 104, 116, 116, 112, 58, 47, 47, 111, 110, 101, 111, 99, 115, 112, 46, 109, 105, 99, 114, 111, 115, 111, 102, 116, 46, 99, 111, 109, 47, 111, 99, 115, 112, 48, 29, 6, 3, 85, 29, 14, 4, 22, 4, 20, 138, 66, 241, 22, 211, 0, 191, 156, 21, 196, 84, 38, 230, 58, 211, 190, 115, 187, 163, 122, 48, 14, 6, 3, 85, 29, 15, 1, 1, 255, 4, 4, 3, 2, 5, 160, 48, 110, 6, 3, 85, 29, 17, 4, 103, 48, 101, 130, 64, 42, 46, 97, 112, 112, 115, 46, 97, 114, 111, 46, 105, 110, 103, 114, 101, 115, 115, 45, 99, 101, 114, 116, 45, 108, 101, 46, 48, 111, 51, 109, 46, 117, 107, 115, 111, 117, 116, 104, 46, 97, 114, 111, 97, 112, 112, 45, 104, 99, 112, 45, 115, 116, 103, 46, 97, 122, 117, 114, 101, 46, 99, 111, 109, 130, 33, 114, 101, 115, 101, 114, 118, 101, 100, 46, 97, 114, 111, 97, 112, 112, 45, 104, 99, 112, 45, 115, 116, 103, 46, 97, 122, 117, 114, 101, 46, 99, 111, 109, 48, 12, 6, 3, 85, 29, 19, 1, 1, 255, 4, 2, 48, 0, 48, 129, 241, 6, 3, 85, 29, 31, 4, 129, 233, 48, 129, 230, 48, 129, 227, 160, 129, 224, 160, 129, 221, 134, 108, 104, 116, 116, 112, 58, 47, 47, 119, 119, 119, 46, 109, 105, 99, 114, 111, 115, 111, 102, 116, 46, 99, 111, 109, 47, 112, 107, 105, 111, 112, 115, 47, 99, 114, 108, 47, 112, 97, 114, 116, 105, 116, 105, 111, 110, 47, 77, 105, 99, 114, 111, 115, 111, 102, 116, 37, 50, 48, 84, 76, 83, 37, 50, 48, 71, 50, 37, 50, 48, 82, 83, 65, 37, 50, 48, 67, 65, 37, 50, 48, 79, 67, 83, 80, 37, 50, 48, 49, 48, 95, 80, 97, 114, 116, 105, 116, 105, 111, 110, 48, 48, 48, 57, 57, 46, 99, 114, 108, 134, 109, 104, 116, 116, 112, 58, 47, 47, 99, 114, 108, 50, 46, 109, 105, 99, 114, 111, 115, 111, 102, 116, 46, 99, 111, 109, 47, 112, 107, 105, 111, 112, 115, 47, 99, 114, 108, 47, 112, 97, 114, 116, 105, 116, 105, 111, 110, 47, 77, 105, 99, 114, 111, 115, 111, 102, 116, 37, 50, 48, 84, 76, 83, 37, 50, 48, 71, 50, 37, 50, 48, 82, 83, 65, 37, 50, 48, 67, 65, 37, 50, 48, 79, 67, 83, 80, 37, 50, 48, 49, 48, 95, 80, 97, 114, 116, 105, 116, 105, 111, 110, 48, 48, 48, 57, 57, 46, 99, 114, 108, 48, 102, 6, 3, 85, 29, 32, 4, 95, 48, 93, 48, 8, 6, 6, 103, 129, 12, 1, 2, 2, 48, 81, 6, 12, 43, 6, 1, 4, 1, 130, 55, 76, 131, 125, 1, 1, 48, 65, 48, 63, 6, 8, 43, 6, 1, 5, 5, 7, 2, 1, 22, 51, 104, 116, 116, 112, 58, 47, 47, 119, 119, 119, 46, 109, 105, 99, 114, 111, 115, 111, 102, 116, 46, 99, 111, 109, 47, 112, 107, 105, 111, 112, 115, 47, 68, 111, 99, 115, 47, 82, 101, 112, 111, 115, 105, 116, 111, 114, 121, 46, 104, 116, 109, 48, 31, 6, 3, 85, 29, 35, 4, 24, 48, 22, 128, 20, 208, 76, 131, 209, 142, 113, 46, 61, 52, 177, 255, 155, 121, 44, 142, 116, 212, 226, 120, 6, 48, 19, 6, 3, 85, 29, 37, 4, 12, 48, 10, 6, 8, 43, 6, 1, 5, 5, 7, 3, 1, 48, 13, 6, 9, 42, 134, 72, 134, 247, 13, 1, 1, 12, 5, 0, 3, 130, 2, 1, 0, 127, 14, 76, 119, 16, 30, 113, 75, 136, 158, 234, 180, 48, 26, 140, 223, 208, 88, 116, 11, 139, 240, 141, 40, 46, 35, 73, 190, 186, 171, 118, 120, 99, 118, 88, 110, 181, 203, 42, 120, 241, 109, 206, 58, 1, 63, 144, 160, 72, 2, 211, 217, 25, 137, 246, 187, 186, 49, 138, 232, 122, 82, 3, 136, 22, 157, 87, 191, 86, 125, 221, 231, 158, 0, 204, 174, 39, 65, 179, 160, 12, 205, 190, 69, 207, 65, 139, 215, 99, 239, 19, 51, 250, 33, 66, 49, 250, 3, 217, 217, 178, 148, 58, 186, 139, 102, 86, 205, 171, 36, 144, 223, 188, 149, 248, 1, 103, 232, 163, 162, 100, 23, 166, 66, 215, 67, 190, 124, 120, 217, 106, 101, 246, 252, 8, 144, 121, 98, 136, 193, 124, 213, 80, 149, 73, 94, 215, 24, 197, 138, 7, 143, 140, 229, 172, 93, 155, 162, 211, 4, 222, 43, 210, 38, 230, 222, 23, 202, 214, 208, 103, 147, 145, 86, 53, 202, 100, 48, 50, 41, 5, 116, 80, 254, 160, 164, 22, 192, 242, 55, 78, 107, 236, 232, 202, 76, 47, 127, 5, 241, 21, 79, 8, 239, 16, 12, 106, 11, 200, 233, 173, 167, 160, 195, 61, 159, 42, 183, 223, 149, 235, 138, 251, 54, 65, 95, 191, 206, 35, 254, 15, 224, 194, 106, 185, 99, 23, 156, 154, 222, 84, 145, 1, 212, 126, 42, 74, 254, 71, 27, 110, 6, 83, 252, 190, 60, 29, 24, 98, 214, 226, 88, 108, 117, 86, 233, 3, 27, 205, 3, 203, 36, 3, 175, 73, 206, 16, 65, 117, 125, 162, 73, 22, 207, 82, 232, 237, 197, 40, 139, 79, 103, 210, 165, 182, 151, 205, 15, 253, 195, 116, 121, 178, 103, 118, 187, 165, 173, 124, 138, 96, 201, 232, 236, 212, 18, 179, 244, 160, 183, 40, 12, 30, 207, 237, 36, 120, 26, 57, 114, 158, 34, 152, 93, 132, 52, 124, 112, 222, 90, 37, 164, 188, 88, 53, 52, 110, 127, 166, 176, 30, 184, 129, 236, 40, 151, 237, 246, 44, 182, 131, 71, 47, 162, 122, 195, 81, 3, 7, 197, 200, 200, 48, 29, 56, 138, 158, 6, 33, 41, 210, 36, 197, 209, 154, 188, 46, 232, 167, 27, 177, 139, 150, 154, 50, 110, 13, 89, 233, 181, 191, 72, 106, 53, 174, 105, 231, 108, 61, 64, 154, 240, 69, 47, 220, 37, 46, 170, 187, 63, 76, 114, 220, 25, 31, 218, 164, 20, 35, 139, 252, 139, 174, 8, 42, 39, 150, 249, 226, 25, 42, 115, 161, 172, 59, 218, 77, 27, 104, 120, 74, 137, 124, 201, 198, 130, 184, 80, 219, 176, 188, 97, 76, 79, 77, 137, 160, 247, 211, 155, 231, 107, 56, 112, 147, 133, 216, 183, 201, 229, 98, 78, 26, 232, 122, 160, 50, 149, 73, 205, 59, 201, 173, 34, 22, 180, 220, 49, 209, 56, 199, 177, 177, 107, 162, 228, 222, 80, 57, 112, 238, 205],
RawTBSCertificate: [48, 130, 6, 251, 160, 3, 2, 1, 2, 2, 19, 73, 1, 74, 89, 119, 169, 186, 32, 59, 56, 235, 138, 87, 0, 0, 1, 74, 89, 119, 48, 13, 6, 9, 42, 134, 72, 134, 247, 13, 1, 1, 12, 5, 0, 48, 87, 49, 11, 48, 9, 6, 3, 85, 4, 6, 19, 2, 85, 83, 49, 30, 48, 28, 6, 3, 85, 4, 10, 19, 21, 77, 105, 99, 114, 111, 115, 111, 102, 116, 32, 67, 111, 114, 112, 111, 114, 97, 116, 105, 111, 110, 49, 40, 48, 38, 6, 3, 85, 4, 3, 19, 31, 77, 105, 99, 114, 111, 115, 111, 102, 116, 32, 84, 76, 83, 32, 71, 50, 32, 82, 83, 65, 32, 67, 65, 32, 79, 67, 83, 80, 32, 49, 48, 48, 30, 23, 13, 50, 54, 49, 48, 50, 53, 48, 49, 53, 54, 49, 48, 90, 23, 13, 50, 55, 48, 50, 48, 50, 48, 48, 53, 54, 49, 48, 90, 48, 120, 49, 11, 48, 9, 6, 3, 85, 4, 6, 19, 2, 85, 83, 49, 11, 48, 9, 6, 3, 85, 4, 8, 19, 2, 87, 65, 49, 16, 48, 14, 6, 3, 85, 4, 7, 19, 7, 82, 101, 100, 109, 111, 110, 100, 49, 30, 48, 28, 6, 3, 85, 4, 10, 19, 21, 77, 105, 99, 114, 111, 115, 111, 102, 116, 32, 67, 111, 114, 112, 111, 114, 97, 116, 105, 111, 110, 49, 42, 48, 40, 6, 3, 85, 4, 3, 19, 33, 114, 101, 115, 101, 114, 118, 101, 100, 46, 97, 114, 111, 97, 112, 112, 45, 104, 99, 112, 45, 115, 116, 103, 46, 97, 122, 117, 114, 101, 46, 99, 111, 109, 48, 130, 1, 34, 48, 13, 6, 9, 42, 134, 72, 134, 247, 13, 1, 1, 1, 5, 0, 3, 130, 1, 15, 0, 48, 130, 1, 10, 2, 130, 1, 1, 0, 187, 121, 183, 117, 223, 172, 129, 21, 158, 33, 40, 228, 200, 112, 100, 153, 170, 116, 250, 158, 245, 201, 135, 0, 237, 144, 31, 12, 163, 250, 168, 50, 92, 5, 220, 113, 137, 143, 3, 193, 167, 220, 193, 78, 195, 131, 29, 107, 67, 52, 86, 236, 30, 39, 30, 107, 206, 213, 234, 58, 181, 33, 44, 227, 147, 157, 23, 219, 20, 127, 20, 235, 246, 77, 135, 5, 240, 135, 231, 58, 154, 101, 19, 42, 160, 128, 73, 94, 79, 155, 62, 172, 213, 53, 57, 2, 149, 60, 188, 169, 233, 44, 216, 105, 66, 159, 148, 70, 178, 113, 177, 128, 123, 151, 10, 7, 254, 171, 205, 105, 34, 145, 93, 19, 218, 220, 219, 47, 129, 42, 85, 247, 229, 7, 118, 48, 182, 125, 84, 45, 187, 16, 138, 130, 74, 176, 14, 70, 229, 120, 108, 212, 172, 177, 174, 141, 161, 231, 167, 5, 108, 191, 229, 39, 27, 122, 242, 198, 32, 78, 125, 8, 168, 234, 200, 195, 10, 209, 101, 183, 248, 26, 10, 82, 78, 47, 196, 44, 55, 199, 207, 209, 35, 65, 164, 44, 162, 97, 81, 36, 122, 188, 105, 82, 214, 39, 252, 78, 115, 249, 159, 36, 212, 167, 6, 72, 63, 56, 220, 88, 50, 162, 50, 174, 185, 196, 246, 208, 84, 90, 212, 218, 52, 118, 184, 101, 201, 132, 48, 29, 45, 113, 80, 81, 157, 255, 235, 218, 49, 139, 40, 124, 111, 147, 143, 201, 2, 3, 1, 0, 1, 163, 130, 4, 181, 48, 130, 4, 177, 48, 130, 1, 4, 6, 10, 43, 6, 1, 4, 1, 214, 121, 2, 4, 2, 4, 129, 245, 4, 129, 242, 0, 240, 0, 118, 0, 68, 194, 189, 12, 233, 20, 14, 100, 165, 201, 74, 1, 147, 10, 90, 161, 187, 53, 151, 14, 0, 238, 17, 22, 137, 104, 42, 28, 68, 215, 181, 102, 0, 0, 1, 161, 112, 206, 239, 163, 0, 0, 4, 3, 0, 71, 48, 69, 2, 33, 0, 172, 23, 89, 180, 174, 255, 10, 75, 120, 126, 2, 114, 141, 52, 53, 9, 237, 84, 209, 98, 89, 54, 179, 9, 54, 118, 245, 178, 107, 207, 5, 69, 2, 32, 89, 15, 78, 239, 171, 216, 123, 169, 212, 117, 5, 12, 137, 15, 63, 83, 41, 96, 223, 14, 136, 244, 40, 31, 142, 25, 69, 157, 200, 47, 205, 167, 0, 118, 0, 0, 26, 93, 26, 28, 45, 147, 117, 182, 72, 85, 120, 248, 47, 113, 161, 174, 110, 239, 57, 125, 41, 124, 138, 227, 21, 123, 202, 222, 225, 160, 30, 0, 0, 1, 161, 112, 206, 240, 23, 0, 0, 4, 3, 0, 71, 48, 69, 2, 33, 0, 209, 5, 94, 127, 36, 105, 13, 219, 166, 20, 49, 161, 208, 24, 148, 233, 208, 121, 79, 240, 195, 85, 152, 81, 108, 234, 202, 116, 68, 189, 194, 57, 2, 32, 65, 141, 139, 86, 66, 203, 70, 108, 26, 159, 237, 96, 32, 237, 241, 53, 230, 112, 171, 119, 90, 74, 10, 14, 99, 1, 225, 78, 245, 158, 165, 135, 48, 27, 6, 9, 43, 6, 1, 4, 1, 130, 55, 21, 10, 4, 14, 48, 12, 48, 10, 6, 8, 43, 6, 1, 5, 5, 7, 3, 1, 48, 60, 6, 9, 43, 6, 1, 4, 1, 130, 55, 21, 7, 4, 47, 48, 45, 6, 37, 43, 6, 1, 4, 1, 130, 55, 21, 8, 135, 189, 215, 27, 129, 231, 235, 70, 130, 129, 157, 46, 142, 208, 12, 135, 240, 218, 29, 93, 133, 246, 239, 42, 129, 232, 226, 53, 2, 1, 100, 2, 1, 39, 48, 130, 1, 11, 6, 8, 43, 6, 1, 5, 5, 7, 1, 1, 4, 129, 254, 48, 129, 251, 48, 97, 6, 8, 43, 6, 1, 5, 5, 7, 48, 2, 134, 85, 104, 116, 116, 112, 58, 47, 47, 119, 119, 119, 46, 109, 105, 99, 114, 111, 115, 111, 102, 116, 46, 99, 111, 109, 47, 112, 107, 105, 111, 112, 115, 47, 99, 101, 114, 116, 115, 47, 77, 105, 99, 114, 111, 115, 111, 102, 116, 37, 50, 48, 84, 76, 83, 37, 50, 48, 71, 50, 37, 50, 48, 82, 83, 65, 37, 50, 48, 67, 65, 37, 50, 48, 79, 67, 83, 80, 37, 50, 48, 49, 48, 46, 99, 114, 116, 48, 103, 6, 8, 43, 6, 1, 5, 5, 7, 48, 2, 134, 91, 104, 116, 116, 112, 58, 47, 47, 99, 97, 105, 115, 115, 117, 101, 114, 115, 46, 109, 105, 99, 114, 111, 115, 111, 102, 116, 46, 99, 111, 109, 47, 112, 107, 105, 111, 112, 115, 47, 99, 101, 114, 116, 115, 47, 77, 105, 99, 114, 111, 115, 111, 102, 116, 37, 50, 48, 84, 76, 83, 37, 50, 48, 71, 50, 37, 50, 48, 82, 83, 65, 37, 50, 48, 67, 65, 37, 50, 48, 79, 67, 83, 80, 37, 50, 48, 49, 48, 46, 99, 114, 116, 48, 45, 6, 8, 43, 6, 1, 5, 5, 7, 48, 1, 134, 33, 104, 116, 116, 112, 58, 47, 47, 111, 110, 101, 111, 99, 115, 112, 46, 109, 105, 99, 114, 111, 115, 111, 102, 116, 46, 99, 111, 109, 47, 111, 99, 115, 112, 48, 29, 6, 3, 85, 29, 14, 4, 22, 4, 20, 138, 66, 241, 22, 211, 0, 191, 156, 21, 196, 84, 38, 230, 58, 211, 190, 115, 187, 163, 122, 48, 14, 6, 3, 85, 29, 15, 1, 1, 255, 4, 4, 3, 2, 5, 160, 48, 110, 6, 3, 85, 29, 17, 4, 103, 48, 101, 130, 64, 42, 46, 97, 112, 112, 115, 46, 97, 114, 111, 46, 105, 110, 103, 114, 101, 115, 115, 45, 99, 101, 114, 116, 45, 108, 101, 46, 48, 111, 51, 109, 46, 117, 107, 115, 111, 117, 116, 104, 46, 97, 114, 111, 97, 112, 112, 45, 104, 99, 112, 45, 115, 116, 103, 46, 97, 122, 117, 114, 101, 46, 99, 111, 109, 130, 33, 114, 101, 115, 101, 114, 118, 101, 100, 46, 97, 114, 111, 97, 112, 112, 45, 104, 99, 112, 45, 115, 116, 103, 46, 97, 122, 117, 114, 101, 46, 99, 111, 109, 48, 12, 6, 3, 85, 29, 19, 1, 1, 255, 4, 2, 48, 0, 48, 129, 241, 6, 3, 85, 29, 31, 4, 129, 233, 48, 129, 230, 48, 129, 227, 160, 129, 224, 160, 129, 221, 134, 108, 104, 116, 116, 112, 58, 47, 47, 119, 119, 119, 46, 109, 105, 99, 114, 111, 115, 111, 102, 116, 46, 99, 111, 109, 47, 112, 107, 105, 111, 112, 115, 47, 99, 114, 108, 47, 112, 97, 114, 116, 105, 116, 105, 111, 110, 47, 77, 105, 99, 114, 111, 115, 111, 102, 116, 37, 50, 48, 84, 76, 83, 37, 50, 48, 71, 50, 37, 50, 48, 82, 83, 65, 37, 50, 48, 67, 65, 37, 50, 48, 79, 67, 83, 80, 37, 50, 48, 49, 48, 95, 80, 97, 114, 116, 105, 116, 105, 111, 110, 48, 48, 48, 57, 57, 46, 99, 114, 108, 134, 109, 104, 116, 116, 112, 58, 47, 47, 99, 114, 108, 50, 46, 109, 105, 99, 114, 111, 115, 111, 102, 116, 46, 99, 111, 109, 47, 112, 107, 105, 111, 112, 115, 47, 99, 114, 108, 47, 112, 97, 114, 116, 105, 116, 105, 111, 110, 47, 77, 105, 99, 114, 111, 115, 111, 102, 116, 37, 50, 48, 84, 76, 83, 37, 50, 48, 71, 50, 37, 50, 48, 82, 83, 65, 37, 50, 48, 67, 65, 37, 50, 48, 79, 67, 83, 80, 37, 50, 48, 49, 48, 95, 80, 97, 114, 116, 105, 116, 105, 111, 110, 48, 48, 48, 57, 57, 46, 99, 114, 108, 48, 102, 6, 3, 85, 29, 32, 4, 95, 48, 93, 48, 8, 6, 6, 103, 129, 12, 1, 2, 2, 48, 81, 6, 12, 43, 6, 1, 4, 1, 130, 55, 76, 131, 125, 1, 1, 48, 65, 48, 63, 6, 8, 43, 6, 1, 5, 5, 7, 2, 1, 22, 51, 104, 116, 116, 112, 58, 47, 47, 119, 119, 119, 46, 109, 105, 99, 114, 111, 115, 111, 102, 116, 46, 99, 111, 109, 47, 112, 107, 105, 111, 112, 115, 47, 68, 111, 99, 115, 47, 82, 101, 112, 111, 115, 105, 116, 111, 114, 121, 46, 104, 116, 109, 48, 31, 6, 3, 85, 29, 35, 4, 24, 48, 22, 128, 20, 208, 76, 131, 209, 142, 113, 46, 61, 52, 177, 255, 155, 121, 44, 142, 116, 212, 226, 120, 6, 48, 19, 6, 3, 85, 29, 37, 4, 12, 48, 10, 6, 8, 43, 6, 1, 5, 5, 7, 3, 1],
RawSubjectPublicKeyInfo: [48, 130, 1, 34, 48, 13, 6, 9, 42, 134, 72, 134, 247, 13, 1, 1, 1, 5, 0, 3, 130, 1, 15, 0, 48, 130, 1, 10, 2, 130, 1, 1, 0, 187, 121, 183, 117, 223, 172, 129, 21, 158, 33, 40, 228, 200, 112, 100, 153, 170, 116, 250, 158, 245, 201, 135, 0, 237, 144, 31, 12, 163, 250, 168, 50, 92, 5, 220, 113, 137, 143, 3, 193, 167, 220, 193, 78, 195, 131, 29, 107, 67, 52, 86, 236, 30, 39, 30, 107, 206, 213, 234, 58, 181, 33, 44, 227, 147, 157, 23, 219, 20, 127, 20, 235, 246, 77, 135, 5, 240, 135, 231, 58, 154, 101, 19, 42, 160, 128, 73, 94, 79, 155, 62, 172, 213, 53, 57, 2, 149, 60, 188, 169, 233, 44, 216, 105, 66, 159, 148, 70, 178, 113, 177, 128, 123, 151, 10, 7, 254, 171, 205, 105, 34, 145, 93, 19, 218, 220, 219, 47, 129, 42, 85, 247, 229, 7, 118, 48, 182, 125, 84, 45, 187, 16, 138, 130, 74, 176, 14, 70, 229, 120, 108, 212, 172, 177, 174, 141, 161, 231, 167, 5, 108, 191, 229, 39, 27, 122, 242, 198, 32, 78, 125, 8, 168, 234, 200, 195, 10, 209, 101, 183, 248, 26, 10, 82, 78, 47, 196, 44, 55, 199, 207, 209, 35, 65, 164, 44, 162, 97, 81, 36, 122, 188, 105, 82, 214, 39, 252, 78, 115, 249, 159, 36, 212, 167, 6, 72, 63, 56, 220, 88, 50, 162, 50, 174, 185, 196, 246, 208, 84, 90, 212, 218, 52, 118, 184, 101, 201, 132, 48, 29, 45, 113, 80, 81, 157, 255, 235, 218, 49, 139, 40, 124, 111, 147, 143, 201, 2, 3, 1, 0, 1],
RawSubject: [48, 120, 49, 11, 48, 9, 6, 3, 85, 4, 6, 19, 2, 85, 83, 49, 11, 48, 9, 6, 3, 85, 4, 8, 19, 2, 87, 65, 49, 16, 48, 14, 6, 3, 85, 4, 7, 19, 7, 82, 101, 100, 109, 111, 110, 100, 49, 30, 48, 28, 6, 3, 85, 4, 10, 19, 21, 77, 105, 99, 114, 111, 115, 111, 102, 116, 32, 67, 111, 114, 112, 111, 114, 97, 116, 105, 111, 110, 49, 42, 48, 40, 6, 3, 85, 4, 3, 19, 33, 114, 101, 115, 101, 114, 118, 101, 100, 46, 97, 114, 111, 97, 112, 112, 45, 104, 99, 112, 45, 115, 116, 103, 46, 97, 122, 117, 114, 101, 46, 99, 111, 109],
RawIssuer: [48, 87, 49, 11, 48, 9, 6, 3, 85, 4, 6, 19, 2, 85, 83, 49, 30, 48, 28, 6, 3, 85, 4, 10, 19, 21, 77, 105, 99, 114, 111, 115, 111, 102, 116, 32, 67, 111, 114, 112, 111, 114, 97, 116, 105, 111, 110, 49, 40, 48, 38, 6, 3, 85, 4, 3, 19, 31, 77, 105, 99, 114, 111, 115, 111, 102, 116, 32, 84, 76, 83, 32, 71, 50, 32, 82, 83, 65, 32, 67, 65, 32, 79, 67, 83, 80, 32, 49, 48],
Signature: [127, 14, 76, 119, 16, 30, 113, 75, 136, 158, 234, 180, 48, 26, 140, 223, 208, 88, 116, 11, 139, 240, 141, 40, 46, 35, 73, 190, 186, 171, 118, 120, 99, 118, 88, 110, 181, 203, 42, 120, 241, 109, 206, 58, 1, 63, 144, 160, 72, 2, 211, 217, 25, 137, 246, 187, 186, 49, 138, 232, 122, 82, 3, 136, 22, 157, 87, 191, 86, 125, 221, 231, 158, 0, 204, 174, 39, 65, 179, 160, 12, 205, 190, 69, 207, 65, 139, 215, 99, 239, 19, 51, 250, 33, 66, 49, 250, 3, 217, 217, 178, 148, 58, 186, 139, 102, 86, 205, 171, 36, 144, 223, 188, 149, 248, 1, 103, 232, 163, 162, 100, 23, 166, 66, 215, 67, 190, 124, 120, 217, 106, 101, 246, 252, 8, 144, 121, 98, 136, 193, 124, 213, 80, 149, 73, 94, 215, 24, 197, 138, 7, 143, 140, 229, 172, 93, 155, 162, 211, 4, 222, 43, 210, 38, 230, 222, 23, 202, 214, 208, 103, 147, 145, 86, 53, 202, 100, 48, 50, 41, 5, 116, 80, 254, 160, 164, 22, 192, 242, 55, 78, 107, 236, 232, 202, 76, 47, 127, 5, 241, 21, 79, 8, 239, 16, 12, 106, 11, 200, 233, 173, 167, 160, 195, 61, 159, 42, 183, 223, 149, 235, 138, 251, 54, 65, 95, 191, 206, 35, 254, 15, 224, 194, 106, 185, 99, 23, 156, 154, 222, 84, 145, 1, 212, 126, 42, 74, 254, 71, 27, 110, 6, 83, 252, 190, 60, 29, 24, 98, 214, 226, 88, 108, 117, 86, 233, 3, 27, 205, 3, 203, 36, 3, 175, 73, 206, 16, 65, 117, 125, 162, 73, 22, 207, 82, 232, 237, 197, 40, 139, 79, 103, 210, 165, 182, 151, 205, 15, 253, 195, 116, 121, 178, 103, 118, 187, 165, 173, 124, 138, 96, 201, 232, 236, 212, 18, 179, 244, 160, 183, 40, 12, 30, 207, 237, 36, 120, 26, 57, 114, 158, 34, 152, 93, 132, 52, 124, 112, 222, 90, 37, 164, 188, 88, 53, 52, 110, 127, 166, 176, 30, 184, 129, 236, 40, 151, 237, 246, 44, 182, 131, 71, 47, 162, 122, 195, 81, 3, 7, 197, 200, 200, 48, 29, 56, 138, 158, 6, 33, 41, 210, 36, 197, 209, 154, 188, 46, 232, 167, 27, 177, 139, 150, 154, 50, 110, 13, 89, 233, 181, 191, 72, 106, 53, 174, 105, 231, 108, 61, 64, 154, 240, 69, 47, 220, 37, 46, 170, 187, 63, 76, 114, 220, 25, 31, 218, 164, 20, 35, 139, 252, 139, 174, 8, 42, 39, 150, 249, 226, 25, 42, 115, 161, 172, 59, 218, 77, 27, 104, 120, 74, 137, 124, 201, 198, 130, 184, 80, 219, 176, 188, 97, 76, 79, 77, 137, 160, 247, 211, 155, 231, 107, 56, 112, 147, 133, 216, 183, 201, 229, 98, 78, 26, 232, 122, 160, 50, 149, 73, 205, 59, 201, 173, 34, 22, 180, 220, 49, 209, 56, 199, 177, 177, 107, 162, 228, 222, 80, 57, 112, 238, 205],
SignatureAlgorithm: 5,
PublicKeyAlgorithm: 1,
PublicKey: <*rsa.PublicKey | 0x36a119f65b90>{
N: {
neg: false,
abs: [3569991644534247369, 3274486715548429274, 3780411684536201245, 13386095367080826074, 4555633294449980078, 8356885564098020936, 8844059572749990990, 2540492176423473444, 5633937353057226705, 779516051327683154, 2327935533544294595, 7836233832775480006, 12443919168731326213, 5381817252368116948, 13149759042380401282, 9307346103102305840, 2490874408770263855, 8905597817602166121, 4800718759162655104, 10753677448201427049, 5736247462418790658, 11125319528290994526, 17747990166117017402, 10636684124628129003, 14904076079720377571, 4842591071448276587, 12095755243242069355, 6630948406570189761, 17118216322786437170, 12282717644381587200, 11394433495304529049, 13509030274166194453],
},
E: 65537,
},
Version: 3,
SerialNumber: {
neg: false,
abs: [9968436300229532023, 6446808008473917675, 4784458],
},
Issuer: {
Country: ["US"],
Organization: [
"Microsoft Corporation",
],
OrganizationalUnit: nil,
Locality: nil,
Province: nil,
StreetAddress: nil,
PostalCode: nil,
SerialNumber: "",
CommonName: "Microsoft TLS G2 RSA CA OCSP 10",
Names: [
{
Type: [..., ..., ..., ...],
Value: <string>"US",
},
{
Type: [..., ..., ..., ...],
Value: <string>"Microsoft Corporation",
},
{
Type: [..., ..., ..., ...],
Value: <string>"Microsoft TLS G2 RSA CA OCSP 10",
},
],
ExtraNames: nil,
},
Subject: {
Country: ["US"],
Organization: [
"Microsoft Corporation",
],
OrganizationalUnit: nil,
Locality: ["Redmond"],
Province: ["WA"],
StreetAddress: nil,
PostalCode: nil,
SerialNumber: "",
CommonName: "reserved.aroapp-hcp-stg.azure.com",
Names: [
{
Type: [..., ..., ..., ...],
Value: <string>"US",
},
{
Type: [..., ..., ..., ...],
Value: <string>"WA",
},
{
Type: [..., ..., ..., ...],
Value: <string>"Redmond",
},
{
Type: [..., ..., ..., ...],
Value: <string>"Microsoft Corporation",
},
{
Type: [..., ..., ..., ...],
Value: <string>"reserved.aroapp-hcp-stg.azure.com",
},
],
ExtraNames: nil,
},
NotBefore: {wall: 0, ext: 63928490170, loc: nil},
NotAfter: {wall: 0, ext: 63937126570, loc: nil},
KeyUsage: 5,
Extensions: [
{
Id: [1, 3, 6, 1, 4, 1, 11129, 2, 4, 2],
Critical: false,
Value: [4, 129, 242, 0, 240, 0, 118, 0, 68, 194, 189, 12, 233, 20, 14, 100, 165, 201, 74, 1, 147, 10, 90, 161, 187, 53, 151, 14, 0, 238, 17, 22, 137, 104, 42, 28, 68, 215, 181, 102, 0, 0, 1, 161, 112, 206, 239, 163, 0, 0, 4, 3, 0, 71, 48, 69, 2, 33, 0, 172, 23, 89, 180, 174, 255, 10, 75, 120, 126, 2, 114, 141, 52, 53, 9, 237, 84, 209, 98, 89, 54, 179, 9, 54, 118, 245, 178, 107, 207, 5, 69, 2, 32, 89, 15, 78, 239, 171, 216, 123, 169, 212, 117, 5, 12, 137, 15, 63, 83, 41, 96, 223, 14, 136, 244, 40, 31, 142, 25, 69, 157, 200, 47, 205, 167, 0, 118, 0, 0, 26, 93, 26, 28, 45, 147, 117, 182, 72, 85, 120, 248, 47, 113, 161, 174, 110, 239, 57, 125, 41, 124, 138, 227, 21, 123, 202, 222, 225, 160, 30, 0, 0, 1, 161, 112, 206, 240, 23, 0, 0, 4, 3, 0, 71, 48, 69, 2, 33, 0, 209, 5, 94, 127, 36, 105, 13, 219, 166, 20, 49, 161, 208, 24, 148, 233, 208, 121, 79, 240, 195, 85, 152, 81, 108, 234, 202, 116, 68, 189, 194, 57, 2, 32, 65, 141, 139, 86, 66, 203, 70, 108, 26, 159, 237, 96, 32, 237, 241, 53, 230, 112, 171, 119, 90, 74, 10, 14, 99, 1, 225, 78, 245, 158, 165, 135],
},
{
Id: [1, 3, 6, 1, 4, 1, 311, 21, 10],
Critical: false,
Value: [48, 12, 48, 10, 6, 8, 43, 6, 1, 5, 5, 7, 3, 1],
},
{
Id: [1, 3, 6, 1, 4, 1, 311, 21, 7],
Critical: false,
Value: [48, 45, 6, 37, 43, 6, 1, 4, 1, 130, 55, 21, 8, 135, 189, 215, 27, 129, 231, 235, 70, 130, 129, 157, 46, 142, 208, 12, 135, 240, 218, 29, 93, 133, 246, 239, 42, 129, 232, 226, 53, 2, 1, 100, 2, 1, 39],
},
{
Id: [1, 3, 6, 1, 5, 5, 7, 1, 1],
Critical: false,
Value: [48, 129, 251, 48, 97, 6, 8, 43, 6, 1, 5, 5, 7, 48, 2, 134, 85, 104, 116, 116, 112, 58, 47, 47, 119, 119, 119, 46, 109, 105, 99, 114, 111, 115, 111, 102, 116, 46, 99, 111, 109, 47, 112, 107, 105, 111, 112, 115, 47, 99, 101, 114, 116, 115, 47, 77, 105, 99, 114, 111, 115, 111, 102, 116, 37, 50, 48, 84, 76, 83, 37, 50, 48, 71, 50, 37, 50, 48, 82, 83, 65, 37, 50, 48, 67, 65, 37, 50, 48, 79, 67, 83, 80, 37, 50, 48, 49, 48, 46, 99, 114, 116, 48, 103, 6, 8, 43, 6, 1, 5, 5, 7, 48, 2, 134, 91, 104, 116, 116, 112, 58, 47, 47, 99, 97, 105, 115, 115, 117, 101, 114, 115, 46, 109, 105, 99, 114, 111, 115, 111, 102, 116, 46, 99, 111, 109, 47, 112, 107, 105, 111, 112, 115, 47, 99, 101, 114, 116, 115, 47, 77, 105, 99, 114, 111, 115, 111, 102, 116, 37, 50, 48, 84, 76, 83, 37, 50, 48, 71, 50, 37, 50, 48, 82, 83, 65, 37, 50, 48, 67, 65, 37, 50, 48, 79, 67, 83, 80, 37, 50, 48, 49, 48, 46, 99, 114, 116, 48, 45, 6, 8, 43, 6, 1, 5, 5, 7, 48, 1, 134, 33, 104, 116, 116, 112, 58, 47, 47, 111, 110, 101, 111, 99, 115, 112, 46, 109, 105, 99, 114, 111, 115, 111, 102, 116, 46, 99, 111, 109, 47, 111, 99, 115, 112],
},
{
Id: [2, 5, 29, 14],
Critical: false,
Value: [4, 20, 138, 66, 241, 22, 211, 0, 191, 156, 21, 196, 84, 38, 230, 58, 211, 190, 115, 187, 163, 122],
},
{
Id: [2, 5, 29, 15],
Critical: true,
Value: [3, 2, 5, 160],
},
{
Id: [2, 5, 29, 17],
Critical: false,
Value: "0e\x82@*.apps.aro.ingress-cert-le.0o3m.uksouth.aroapp-hcp-stg.azure.com\x82!reserved.aroapp-hcp-stg.azure.com",
},
{Id: [2, 5, 29, 19], Critical: true, Value: [48, 0]},
{
Id: [2, 5, 29, 31],
Critical: false,
Value: "0\x81\xe60\x81㠁ࠁ݆lhttp://www.microsoft.com/pkiops/crl/partition/Microsoft%20TLS%20G2%20RSA%20CA%20OCSP%2010_Partition00099.crl\x86mhttp://crl2.microsoft.com/pkiops/crl/partition/Microsoft%20TLS%20G2%20RSA%20CA%20OCSP%2010_Partition00099.crl",
},
{
Id: [2, 5, 29, 32],
Critical: false,
Value: [48, 93, 48, 8, 6, 6, 103, 129, 12, 1, 2, 2, 48, 81, 6, 12, 43, 6, 1, 4, 1, 130, 55, 76, 131, 125, 1, 1, 48, 65, 48, 63, 6, 8, 43, 6, 1, 5, 5, 7, 2, 1, 22, 51, 104, 116, 116, 112, 58, 47, 47, 119, 119, 119, 46, 109, 105, 99, 114, 111, 115, 111, 102, 116, 46, 99, 111, 109, 47, 112, 107, 105, 111, 112, 115, 47, 68, 111, 99, 115, 47, 82, 101, 112, 111, 115, 105, 116, 111, 114, 121, 46, 104, 116, 109],
},
{
Id: [2, 5, 29, 35],
Critical: false,
Value: [48, 22, 128, 20, 208, 76, 131, 209, 142, 113, 46, 61, 52, 177, 255, 155, 121, 44, 142, 116, 212, 226, 120, 6],
},
{
Id: [2, 5, 29, 37],
Critical: false,
Value: [48, 10, 6, 8, 43, 6, 1, 5, 5, 7, 3, 1],
},
],
ExtraExtensions: nil,
UnhandledCriticalExtensions: nil,
ExtKeyUsage: [1],
UnknownExtKeyUsage: nil,
BasicConstraintsValid: true,
IsCA: false,
MaxPathLen: -1,
MaxPathLenZero: false,
SubjectKeyId: [138, 66, 241, 22, 211, 0, 191, 156, 21, 196, 84, 38, 230, 58, 211, 190, 115, 187, 163, 122],
AuthorityKeyId: [208, 76, 131, 209, 142, 113, 46, 61, 52, 177, 255, 155, 121, 44, 142, 116, 212, 226, 120, 6],
OCSPServer: [
"http://oneocsp.microsoft.com/ocsp",
],
IssuingCertificateURL: [
"http://www.microsoft.com/pkiops/certs/Microsoft%20TLS%20G2%20RSA%20CA%20OCSP%2010.crt",
"http://caissuers.microsoft.com/pkiops/certs/Microsoft%20TLS%20G2%20RSA%20CA%20OCSP%2010.crt",
],
DNSNames: [
"*.apps.aro.ingress-cert-le.0o3m.uksouth.aroapp-hcp-stg.azure.com",
"reserved.aroapp-hcp-stg.azure.com",
],
EmailAddresses: nil,
IPAddresses: nil,
URIs: nil,
PermittedDNSDomainsCritical: false,
PermittedDNSDomains: nil,
ExcludedDNSDomains: nil,
PermittedIPRanges: nil,
ExcludedIPRanges: nil,
PermittedEmailAddresses: nil,
ExcludedEmailAddresses: nil,
PermittedURIDomains: nil,
ExcludedURIDomains: nil,
CRLDistributionPoints: [
"http://www.microsoft.com/pkiops/crl/partition/Microsoft%20TLS%20G2%20RSA%20CA%20OCSP%2010_Partition00099.crl",
"http://crl2.microsoft.com/pkiops/crl/partition/Microsoft%20TLS%20G2%20RSA%20CA%20OCSP%2010_Partition00099.crl",
],
PolicyIdentifiers: [
[2, 23, 140, 1, 2, 2],
[1, 3, 6, 1, 4, 1, 311, 76, 509, 1, 1],
],
Policies: [
{
der: [103, 129, 12, 1, 2, 2],
},
{
der: [43, 6, 1, 4, 1, 130, 55, 76, 131, 125, 1, 1],
},
],
InhibitAnyPolicy: 0,
InhibitAnyPolicyZero: false,
InhibitPolicyMapping: 0,
InhibitPolicyMappingZero: false,
RequireExplicitPolicy: 0,
RequireExplicitPolicyZero: false,
PolicyMappings: nil,
},
{
Raw: [48, 130, 7, 184, 48, 130, 5, 160, 160, 3, 2, 1, 2, 2, 19, 51, 0, 0, 0, 15, 51, 32, 101, 55, 238, 66, 174, 79, 0, 0, 0, 0, 0, 15, 48, 13, 6, 9, 42, 134, 72, 134, 247, 13, 1, 1, 12, 5, 0, 48, 81, 49, 11, 48, 9, 6, 3, 85, 4, 6, 19, 2, 85, 83, 49, 30, 48, 28, 6, 3, 85, 4, 10, 19, 21, 77, 105, 99, 114, 111, 115, 111, 102, 116, 32, 67, 111, 114, 112, 111, 114, 97, 116, 105, 111, 110, 49, 34, 48, 32, 6, 3, 85, 4, 3, 19, 25, 77, 105, 99, 114, 111, 115, 111, 102, 116, 32, 84, 76, 83, 32, 82, 83, 65, 32, 82, 111, 111, 116, 32, 71, 50, 48, 30, 23, 13, 50, 53, 48, 56, 48, 49, 50, 48, 48, 51, 48, 51, 90, 23, 13, 50, 57, 48, 54, 48, 51, 50, 48, 48, 51, 48, 51, 90, 48, 87, 49, 11, 48, 9, 6, 3, 85, 4, 6, 19, 2, 85, 83, 49, 30, 48, 28, 6, 3, 85, 4, 10, 19, 21, 77, 105, 99, 114, 111, 115, 111, 102, 116, 32, 67, 111, 114, 112, 111, 114, 97, 116, 105, 111, 110, 49, 40, 48, 38, 6, 3, 85, 4, 3, 19, 31, 77, 105, 99, 114, 111, 115, 111, 102, 116, 32, 84, 76, 83, 32, 71, 50, 32, 82, 83, 65, 32, 67, 65, 32, 79, 67, 83, 80, 32, 49, 48, 48, 130, 2, 34, 48, 13, 6, 9, 42, 134, 72, 134, 247, 13, 1, 1, 1, 5, 0, 3, 130, 2, 15, 0, 48, 130, 2, 10, 2, 130, 2, 1, 0, 240, 12, 79, 91, 95, 252, 66, 28, 180, 206, 63, 245, 7, 43, 166, 131, 126, 201, 157, 121, 207, 246, 233, 172, 165, 119, 195, 55, 169, 250, 247, 7, 245, 45, 152, 248, 17, 102, 235, 66, 42, 14, 210, 209, 6, 25, 246, 119, 19, 95, 132, 236, 214, 81, 147, 108, 33, 227, 14, 133, 166, 255, 210, 236, 253, 36, 130, 189, 4, 249, 239, 141, 227, 225, 197, 221, 117, 208, 4, 154, 78, 20, 54, 25, 251, 45, 18, 163, 21, 192, 243, 111, 106, 68, 213, 113, 108, 241, 75, 152, 27, 121, 48, 247, 156, 23, 191, 125, 50, 9, 57, 91, 54, 140, 160, 8, 187, 64, 44, 150, 41, 238, 124, 189, 14, 248, 236, 109, 206, 193, 78, 118, 193, 88, 46, 78, 50, 154, 199, 124, 203, 199, 191, 183, 93, 247, 53, 84, 120, 204, 177, 135, 186, 5, 197, 53, 1, 112, 4, 195, 232, 93, 248, 177, 180, 223, 71, 205, 76, 102, 154, 232, 103, 212, 158, 30, 160, 33, 31, 51, 71, 5, 59, 68, 8, 207, 107, 136, 82, 89, 50, 120, 245, 65, 142, 37, 46, 208, 2, 13, 1, 207, 118, 247, 45, 52, 228, 103, 157, 104, 177, 128, 182, 77, 82, 179, 32, 151, 43, 165, 83, 140, 119, 149, 181, 115, 128, 205, 136, 255, 218, 151, 54, 227, 231, 111, 50, 160, 117, 241, 187, 191, 250, 60, 101, 171, 54, 17, 191, 118, 253, 71, 255, 108, 135, 96, 190, 12, 229, 234, 150, 245, 11, 162, 55, 39, 0, 209, 231, 2, 48, 1, 213, 128, 1, 194, 144, 206, 239, 179, 235, 211, 120, 96, 190, 10, 63, 251, 108, 198, 194, 209, 171, 39, 23, 23, 9, 132, 121, 145, 66, 142, 118, 104, 55, 17, 155, 206, 199, 118, 41, 191, 213, 92, 226, 124, 13, 95, 103, 82, 253, 109, 57, 250, 81, 45, 223, 178, 136, 3, 241, 97, 32, 170, 248, 180, 178, 159, 211, 250, 40, 176, 181, 167, 227, 131, 186, 250, 68, 59, 119, 220, 162, 44, 43, 232, 226, 83, 136, 154, 251, 12, 32, 222, 162, 123, 78, 128, 144, 113, 27, 124, 151, 212, 219, 252, 61, 154, 193, 161, 222, 234, 231, 174, 76, 229, 174, 175, 118, 172, 135, 205, 100, 188, 162, 52, 26, 211, 102, 195, 107, 97, 124, 155, 245, 230, 89, 121, 251, 128, 14, 22, 107, 192, 241, 41, 244, 237, 157, 151, 142, 58, 196, 41, 147, 7, 52, 255, 226, 191, 173, 154, 89, 37, 155, 134, 29, 67, 126, 203, 110, 149, 245, 142, 224, 169, 139, 87, 128, 78, 11, 230, 96, 125, 192, 73, 213, 4, 188, 32, 202, 4, 210, 92, 234, 63, 140, 73, 127, 29, 185, 186, 96, 109, 215, 180, 100, 179, 92, 52, 234, 56, 243, 249, 193, 248, 169, 11, 126, 186, 245, 209, 252, 122, 27, 127, 219, 8, 101, 125, 195, 151, 5, 236, 175, 47, 199, 222, 12, 250, 178, 63, 2, 3, 1, 0, 1, 163, 130, 2, 129, 48, 130, 2, 125, 48, 14, 6, 3, 85, 29, 15, 1, 1, 255, 4, 4, 3, 2, 1, 134, 48, 16, 6, 9, 43, 6, 1, 4, 1, 130, 55, 21, 1, 4, 3, 2, 1, 0, 48, 29, 6, 3, 85, 29, 14, 4, 22, 4, 20, 208, 76, 131, 209, 142, 113, 46, 61, 52, 177, 255, 155, 121, 44, 142, 116, 212, 226, 120, 6, 48, 19, 6, 3, 85, 29, 32, 4, 12, 48, 10, 48, 8, 6, 6, 103, 129, 12, 1, 2, 2, 48, 19, 6, 3, 85, 29, 37, 4, 12, 48, 10, 6, 8, 43, 6, 1, 5, 5, 7, 3, 1, 48, 25, 6, 9, 43, 6, 1, 4, 1, 130, 55, 20, 2, 4, 12, 30, 10, 0, 83, 0, 117, 0, 98, 0, 67, 0, 65, 48, 18, 6, 3, 85, 29, 19, 1, 1, 255, 4, 8, 48, 6, 1, 1, 255, 2, 1, 0, 48, 31, 6, 3, 85, 29, 35, 4, 24, 48, 22, 128, 20, 222, 145, 134, 72, 183, 161, 49, 89, 49, 241, 75, 95, 7, 169, 220, 136, 121, 218, 168, 118, 48, 129, 171, 6, 3, 85, 29, 31, 4, 129, 163, 48, 129, 160, 48, 129, 157, 160, 129, 154, 160, 129, 151, 134, 73, 104, 116, 116, 112, 58, 47, 47, 119, 119, 119, 46, 109, 105, 99, 114, 111, 115, 111, 102, 116, 46, 99, 111, 109, 47, 112, 107, 105, 111, 112, 115, 47, 99, 114, 108, 47, 77, 105, 99, 114, 111, 115, 111, 102, 116, 37, 50, 48, 84, 76, 83, 37, 50, 48, 82, 83, 65, 37, 50, 48, 82, 111, 111, 116, 37, 50, 48, 71, 50, 46, 99, 114, 108, 134, 74, 104, 116, 116, 112, 58, 47, 47, 99, 114, 108, 50, 46, 109, 105, 99, 114, 111, 115, 111, 102, 116, 46, 99, 111, 109, 47, 112, 107, 105, 111, 112, 115, 47, 99, 114, 108, 47, 77, 105, 99, 114, 111, 115, 111, 102, 116, 37, 50, 48, 84, 76, 83, 37, 50, 48, 82, 83, 65, 37, 50, 48, 82, 111, 111, 116, 37, 50, 48, 71, 50, 46, 99, 114, 108, 48, 130, 1, 16, 6, 8, 43, 6, 1, 5, 5, 7, 1, 1, 4, 130, 1, 2, 48, 129, 255, 48, 99, 6, 8, 43, 6, 1, 5, 5, 7, 48, 2, 134, 87, 104, 116, 116, 112, 58, 47, 47, 119, 119, 119, 46, 109, 105, 99, 114, 111, 115, 111, 102, 116, 46, 99, 111, 109, 47, 112, 107, 105, 111, 112, 115, 47, 99, 101, 114, 116, 115, 47, 77, 105, 99, 114, 111, 115, 111, 102, 116, 37, 50, 48, 84, 76, 83, 37, 50, 48, 82, 83, 65, 37, 50, 48, 82, 111, 111, 116, 37, 50, 48, 71, 50, 37, 50, 48, 45, 37, 50, 48, 120, 115, 105, 103, 110, 46, 99, 114, 116, 48, 105, 6, 8, 43, 6, 1, 5, 5, 7, 48, 2, 134, 93, 104, 116, 116, 112, 58, 47, 47, 99, 97, 105, 115, 115, 117, 101, 114, 115, 46, 109, 105, 99, 114, 111, 115, 111, 102, 116, 46, 99, 111, 109, 47, 112, 107, 105, 111, 112, 115, 47, 99, 101, 114, 116, 115, 47, 77, 105, 99, 114, 111, 115, 111, 102, 116, 37, 50, 48, 84, 76, 83, 37, 50, 48, 82, 83, 65, 37, 50, 48, 82, 111, 111, 116, 37, 50, 48, 71, 50, 37, 50, 48, 45, 37, 50, 48, 120, 115, 105, 103, 110, 46, 99, 114, 116, 48, 45, 6, 8, 43, 6, 1, 5, 5, 7, 48, 1, 134, 33, 104, 116, 116, 112, 58, 47, 47, 111, 110, 101, 111, 99, 115, 112, 46, 109, 105, 99, 114, 111, 115, 111, 102, 116, 46, 99, 111, 109, 47, 111, 99, 115, 112, 48, 13, 6, 9, 42, 134, 72, 134, 247, 13, 1, 1, 12, 5, 0, 3, 130, 2, 1, 0, 53, 25, 202, 233, 168, 117, 225, 216, 202, 253, 37, 193, 59, 82, 225, 231, 158, 43, 148, 226, 244, 58, 210, 79, 218, 6, 34, 82, 26, 1, 24, 227, 162, 186, 183, 183, 28, 36, 50, 117, 13, 103, 68, 125, 157, 74, 130, 75, 250, 227, 68, 233, 70, 246, 0, 69, 9, 194, 82, 233, 103, 37, 151, 199, 183, 139, 90, 119, 124, 64, 30, 148, 227, 188, 140, 109, 56, 41, 188, 39, 125, 18, 18, 61, 203, 203, 228, 196, 70, 226, 165, 1, 192, 16, 209, 32, 250, 14, 3, 228, 67, 219, 38, 135, 37, 172, 200, 160, 179, 4, 106, 94, 240, 67, 63, 15, 123, 245, 206, 35, 195, 183, 201, 190, 251, 222, 155, 125, 147, 179, 135, 62, 223, 198, 219, 61, 201, 244, 172, 113, 0, 210, 164, 101, 129, 54, 89, 159, 28, 54, 229, 137, 209, 122, 153, 74, 123, 85, 75, 194, 213, 140, 208, 121, 1, 121, 149, 15, 57, 67, 139, 179, 10, 123, 146, 213, 90, 16, 24, 118, 233, 29, 13, 111, 50, 49, 244, 189, 209, 22, 241, 168, 163, 103, 242, 14, 154, 112, 12, 227, 150, 175, 3, 244, 109, 189, 20, 121, 106, 54, 195, 241, 182, 115, 187, 198, 11, 213, 38, 70, 136, 84, 69, 112, 170, 204, 31, 61, 139, 111, 227, 228, 23, 166, 129, 0, 251, 125, 131, 88, 128, 226, 242, 60, 59, 163, 24, 98, 19, 54, 103, 255, 79, 226, 229, 111, 106, 103, 230, 232, 90, 100, 58, 160, 69, 157, 63, 23, 12, 102, 135, 240, 129, 39, 52, 32, 176, 53, 36, 100, 40, 118, 67, 122, 116, 152, 226, 43, 128, 9, 10, 188, 176, 185, 7, 71, 41, 40, 46, 125, 224, 119, 65, 164, 140, 54, 120, 186, 51, 29, 86, 142, 217, 16, 53, 137, 89, 131, 0, 105, 206, 183, 120, 189, 237, 175, 130, 63, 206, 200, 137, 50, 152, 89, 8, 93, 225, 1, 150, 88, 196, 101, 25, 125, 38, 133, 157, 16, 150, 229, 149, 197, 17, 206, 130, 111, 23, 6, 113, 21, 15, 44, 191, 55, 50, 82, 21, 104, 180, 149, 142, 83, 93, 151, 237, 227, 219, 19, 62, 193, 49, 177, 121, 147, 115, 125, 114, 59, 27, 76, 79, 6, 73, 21, 132, 115, 243, 29, 90, 81, 44, 164, 212, 88, 224, 25, 191, 53, 72, 41, 96, 153, 171, 234, 123, 66, 3, 190, 156, 61, 25, 247, 71, 91, 11, 172, 130, 87, 170, 13, 189, 255, 34, 96, 48, 223, 44, 79, 87, 211, 37, 213, 218, 253, 142, 6, 81, 78, 5, 53, 120, 152, 56, 163, 39, 49, 111, 49, 154, 169, 170, 77, 172, 58, 68, 104, 96, 112, 104, 244, 83, 107, 205, 178, 45, 174, 38, 24, 135, 223, 140, 87, 181, 248, 82, 193, 86, 231, 192, 177, 77, 95, 10, 85, 107, 20, 43, 84, 251, 61, 169, 198, 189, 96, 7, 248, 176, 244, 75, 227, 238, 171],
RawTBSCertificate: [48, 130, 5, 160, 160, 3, 2, 1, 2, 2, 19, 51, 0, 0, 0, 15, 51, 32, 101, 55, 238, 66, 174, 79, 0, 0, 0, 0, 0, 15, 48, 13, 6, 9, 42, 134, 72, 134, 247, 13, 1, 1, 12, 5, 0, 48, 81, 49, 11, 48, 9, 6, 3, 85, 4, 6, 19, 2, 85, 83, 49, 30, 48, 28, 6, 3, 85, 4, 10, 19, 21, 77, 105, 99, 114, 111, 115, 111, 102, 116, 32, 67, 111, 114, 112, 111, 114, 97, 116, 105, 111, 110, 49, 34, 48, 32, 6, 3, 85, 4, 3, 19, 25, 77, 105, 99, 114, 111, 115, 111, 102, 116, 32, 84, 76, 83, 32, 82, 83, 65, 32, 82, 111, 111, 116, 32, 71, 50, 48, 30, 23, 13, 50, 53, 48, 56, 48, 49, 50, 48, 48, 51, 48, 51, 90, 23, 13, 50, 57, 48, 54, 48, 51, 50, 48, 48, 51, 48, 51, 90, 48, 87, 49, 11, 48, 9, 6, 3, 85, 4, 6, 19, 2, 85, 83, 49, 30, 48, 28, 6, 3, 85, 4, 10, 19, 21, 77, 105, 99, 114, 111, 115, 111, 102, 116, 32, 67, 111, 114, 112, 111, 114, 97, 116, 105, 111, 110, 49, 40, 48, 38, 6, 3, 85, 4, 3, 19, 31, 77, 105, 99, 114, 111, 115, 111, 102, 116, 32, 84, 76, 83, 32, 71, 50, 32, 82, 83, 65, 32, 67, 65, 32, 79, 67, 83, 80, 32, 49, 48, 48, 130, 2, 34, 48, 13, 6, 9, 42, 134, 72, 134, 247, 13, 1, 1, 1, 5, 0, 3, 130, 2, 15, 0, 48, 130, 2, 10, 2, 130, 2, 1, 0, 240, 12, 79, 91, 95, 252, 66, 28, 180, 206, 63, 245, 7, 43, 166, 131, 126, 201, 157, 121, 207, 246, 233, 172, 165, 119, 195, 55, 169, 250, 247, 7, 245, 45, 152, 248, 17, 102, 235, 66, 42, 14, 210, 209, 6, 25, 246, 119, 19, 95, 132, 236, 214, 81, 147, 108, 33, 227, 14, 133, 166, 255, 210, 236, 253, 36, 130, 189, 4, 249, 239, 141, 227, 225, 197, 221, 117, 208, 4, 154, 78, 20, 54, 25, 251, 45, 18, 163, 21, 192, 243, 111, 106, 68, 213, 113, 108, 241, 75, 152, 27, 121, 48, 247, 156, 23, 191, 125, 50, 9, 57, 91, 54, 140, 160, 8, 187, 64, 44, 150, 41, 238, 124, 189, 14, 248, 236, 109, 206, 193, 78, 118, 193, 88, 46, 78, 50, 154, 199, 124, 203, 199, 191, 183, 93, 247, 53, 84, 120, 204, 177, 135, 186, 5, 197, 53, 1, 112, 4, 195, 232, 93, 248, 177, 180, 223, 71, 205, 76, 102, 154, 232, 103, 212, 158, 30, 160, 33, 31, 51, 71, 5, 59, 68, 8, 207, 107, 136, 82, 89, 50, 120, 245, 65, 142, 37, 46, 208, 2, 13, 1, 207, 118, 247, 45, 52, 228, 103, 157, 104, 177, 128, 182, 77, 82, 179, 32, 151, 43, 165, 83, 140, 119, 149, 181, 115, 128, 205, 136, 255, 218, 151, 54, 227, 231, 111, 50, 160, 117, 241, 187, 191, 250, 60, 101, 171, 54, 17, 191, 118, 253, 71, 255, 108, 135, 96, 190, 12, 229, 234, 150, 245, 11, 162, 55, 39, 0, 209, 231, 2, 48, 1, 213, 128, 1, 194, 144, 206, 239, 179, 235, 211, 120, 96, 190, 10, 63, 251, 108, 198, 194, 209, 171, 39, 23, 23, 9, 132, 121, 145, 66, 142, 118, 104, 55, 17, 155, 206, 199, 118, 41, 191, 213, 92, 226, 124, 13, 95, 103, 82, 253, 109, 57, 250, 81, 45, 223, 178, 136, 3, 241, 97, 32, 170, 248, 180, 178, 159, 211, 250, 40, 176, 181, 167, 227, 131, 186, 250, 68, 59, 119, 220, 162, 44, 43, 232, 226, 83, 136, 154, 251, 12, 32, 222, 162, 123, 78, 128, 144, 113, 27, 124, 151, 212, 219, 252, 61, 154, 193, 161, 222, 234, 231, 174, 76, 229, 174, 175, 118, 172, 135, 205, 100, 188, 162, 52, 26, 211, 102, 195, 107, 97, 124, 155, 245, 230, 89, 121, 251, 128, 14, 22, 107, 192, 241, 41, 244, 237, 157, 151, 142, 58, 196, 41, 147, 7, 52, 255, 226, 191, 173, 154, 89, 37, 155, 134, 29, 67, 126, 203, 110, 149, 245, 142, 224, 169, 139, 87, 128, 78, 11, 230, 96, 125, 192, 73, 213, 4, 188, 32, 202, 4, 210, 92, 234, 63, 140, 73, 127, 29, 185, 186, 96, 109, 215, 180, 100, 179, 92, 52, 234, 56, 243, 249, 193, 248, 169, 11, 126, 186, 245, 209, 252, 122, 27, 127, 219, 8, 101, 125, 195, 151, 5, 236, 175, 47, 199, 222, 12, 250, 178, 63, 2, 3, 1, 0, 1, 163, 130, 2, 129, 48, 130, 2, 125, 48, 14, 6, 3, 85, 29, 15, 1, 1, 255, 4, 4, 3, 2, 1, 134, 48, 16, 6, 9, 43, 6, 1, 4, 1, 130, 55, 21, 1, 4, 3, 2, 1, 0, 48, 29, 6, 3, 85, 29, 14, 4, 22, 4, 20, 208, 76, 131, 209, 142, 113, 46, 61, 52, 177, 255, 155, 121, 44, 142, 116, 212, 226, 120, 6, 48, 19, 6, 3, 85, 29, 32, 4, 12, 48, 10, 48, 8, 6, 6, 103, 129, 12, 1, 2, 2, 48, 19, 6, 3, 85, 29, 37, 4, 12, 48, 10, 6, 8, 43, 6, 1, 5, 5, 7, 3, 1, 48, 25, 6, 9, 43, 6, 1, 4, 1, 130, 55, 20, 2, 4, 12, 30, 10, 0, 83, 0, 117, 0, 98, 0, 67, 0, 65, 48, 18, 6, 3, 85, 29, 19, 1, 1, 255, 4, 8, 48, 6, 1, 1, 255, 2, 1, 0, 48, 31, 6, 3, 85, 29, 35, 4, 24, 48, 22, 128, 20, 222, 145, 134, 72, 183, 161, 49, 89, 49, 241, 75, 95, 7, 169, 220, 136, 121, 218, 168, 118, 48, 129, 171, 6, 3, 85, 29, 31, 4, 129, 163, 48, 129, 160, 48, 129, 157, 160, 129, 154, 160, 129, 151, 134, 73, 104, 116, 116, 112, 58, 47, 47, 119, 119, 119, 46, 109, 105, 99, 114, 111, 115, 111, 102, 116, 46, 99, 111, 109, 47, 112, 107, 105, 111, 112, 115, 47, 99, 114, 108, 47, 77, 105, 99, 114, 111, 115, 111, 102, 116, 37, 50, 48, 84, 76, 83, 37, 50, 48, 82, 83, 65, 37, 50, 48, 82, 111, 111, 116, 37, 50, 48, 71, 50, 46, 99, 114, 108, 134, 74, 104, 116, 116, 112, 58, 47, 47, 99, 114, 108, 50, 46, 109, 105, 99, 114, 111, 115, 111, 102, 116, 46, 99, 111, 109, 47, 112, 107, 105, 111, 112, 115, 47, 99, 114, 108, 47, 77, 105, 99, 114, 111, 115, 111, 102, 116, 37, 50, 48, 84, 76, 83, 37, 50, 48, 82, 83, 65, 37, 50, 48, 82, 111, 111, 116, 37, 50, 48, 71, 50, 46, 99, 114, 108, 48, 130, 1, 16, 6, 8, 43, 6, 1, 5, 5, 7, 1, 1, 4, 130, 1, 2, 48, 129, 255, 48, 99, 6, 8, 43, 6, 1, 5, 5, 7, 48, 2, 134, 87, 104, 116, 116, 112, 58, 47, 47, 119, 119, 119, 46, 109, 105, 99, 114, 111, 115, 111, 102, 116, 46, 99, 111, 109, 47, 112, 107, 105, 111, 112, 115, 47, 99, 101, 114, 116, 115, 47, 77, 105, 99, 114, 111, 115, 111, 102, 116, 37, 50, 48, 84, 76, 83, 37, 50, 48, 82, 83, 65, 37, 50, 48, 82, 111, 111, 116, 37, 50, 48, 71, 50, 37, 50, 48, 45, 37, 50, 48, 120, 115, 105, 103, 110, 46, 99, 114, 116, 48, 105, 6, 8, 43, 6, 1, 5, 5, 7, 48, 2, 134, 93, 104, 116, 116, 112, 58, 47, 47, 99, 97, 105, 115, 115, 117, 101, 114, 115, 46, 109, 105, 99, 114, 111, 115, 111, 102, 116, 46, 99, 111, 109, 47, 112, 107, 105, 111, 112, 115, 47, 99, 101, 114, 116, 115, 47, 77, 105, 99, 114, 111, 115, 111, 102, 116, 37, 50, 48, 84, 76, 83, 37, 50, 48, 82, 83, 65, 37, 50, 48, 82, 111, 111, 116, 37, 50, 48, 71, 50, 37, 50, 48, 45, 37, 50, 48, 120, 115, 105, 103, 110, 46, 99, 114, 116, 48, 45, 6, 8, 43, 6, 1, 5, 5, 7, 48, 1, 134, 33, 104, 116, 116, 112, 58, 47, 47, 111, 110, 101, 111, 99, 115, 112, 46, 109, 105, 99, 114, 111, 115, 111, 102, 116, 46, 99, 111, 109, 47, 111, 99, 115, 112],
RawSubjectPublicKeyInfo: [48, 130, 2, 34, 48, 13, 6, 9, 42, 134, 72, 134, 247, 13, 1, 1, 1, 5, 0, 3, 130, 2, 15, 0, 48, 130, 2, 10, 2, 130, 2, 1, 0, 240, 12, 79, 91, 95, 252, 66, 28, 180, 206, 63, 245, 7, 43, 166, 131, 126, 201, 157, 121, 207, 246, 233, 172, 165, 119, 195, 55, 169, 250, 247, 7, 245, 45, 152, 248, 17, 102, 235, 66, 42, 14, 210, 209, 6, 25, 246, 119, 19, 95, 132, 236, 214, 81, 147, 108, 33, 227, 14, 133, 166, 255, 210, 236, 253, 36, 130, 189, 4, 249, 239, 141, 227, 225, 197, 221, 117, 208, 4, 154, 78, 20, 54, 25, 251, 45, 18, 163, 21, 192, 243, 111, 106, 68, 213, 113, 108, 241, 75, 152, 27, 121, 48, 247, 156, 23, 191, 125, 50, 9, 57, 91, 54, 140, 160, 8, 187, 64, 44, 150, 41, 238, 124, 189, 14, 248, 236, 109, 206, 193, 78, 118, 193, 88, 46, 78, 50, 154, 199, 124, 203, 199, 191, 183, 93, 247, 53, 84, 120, 204, 177, 135, 186, 5, 197, 53, 1, 112, 4, 195, 232, 93, 248, 177, 180, 223, 71, 205, 76, 102, 154, 232, 103, 212, 158, 30, 160, 33, 31, 51, 71, 5, 59, 68, 8, 207, 107, 136, 82, 89, 50, 120, 245, 65, 142, 37, 46, 208, 2, 13, 1, 207, 118, 247, 45, 52, 228, 103, 157, 104, 177, 128, 182, 77, 82, 179, 32, 151, 43, 165, 83, 140, 119, 149, 181, 115, 128, 205, 136, 255, 218, 151, 54, 227, 231, 111, 50, 160, 117, 241, 187, 191, 250, 60, 101, 171, 54, 17, 191, 118, 253, 71, 255, 108, 135, 96, 190, 12, 229, 234, 150, 245, 11, 162, 55, 39, 0, 209, 231, 2, 48, 1, 213, 128, 1, 194, 144, 206, 239, 179, 235, 211, 120, 96, 190, 10, 63, 251, 108, 198, 194, 209, 171, 39, 23, 23, 9, 132, 121, 145, 66, 142, 118, 104, 55, 17, 155, 206, 199, 118, 41, 191, 213, 92, 226, 124, 13, 95, 103, 82, 253, 109, 57, 250, 81, 45, 223, 178, 136, 3, 241, 97, 32, 170, 248, 180, 178, 159, 211, 250, 40, 176, 181, 167, 227, 131, 186, 250, 68, 59, 119, 220, 162, 44, 43, 232, 226, 83, 136, 154, 251, 12, 32, 222, 162, 123, 78, 128, 144, 113, 27, 124, 151, 212, 219, 252, 61, 154, 193, 161, 222, 234, 231, 174, 76, 229, 174, 175, 118, 172, 135, 205, 100, 188, 162, 52, 26, 211, 102, 195, 107, 97, 124, 155, 245, 230, 89, 121, 251, 128, 14, 22, 107, 192, 241, 41, 244, 237, 157, 151, 142, 58, 196, 41, 147, 7, 52, 255, 226, 191, 173, 154, 89, 37, 155, 134, 29, 67, 126, 203, 110, 149, 245, 142, 224, 169, 139, 87, 128, 78, 11, 230, 96, 125, 192, 73, 213, 4, 188, 32, 202, 4, 210, 92, 234, 63, 140, 73, 127, 29, 185, 186, 96, 109, 215, 180, 100, 179, 92, 52, 234, 56, 243, 249, 193, 248, 169, 11, 126, 186, 245, 209, 252, 122, 27, 127, 219, 8, 101, 125, 195, 151, 5, 236, 175, 47, 199, 222, 12, 250, 178, 63, 2, 3, 1, 0, 1],
RawSubject: [48, 87, 49, 11, 48, 9, 6, 3, 85, 4, 6, 19, 2, 85, 83, 49, 30, 48, 28, 6, 3, 85, 4, 10, 19, 21, 77, 105, 99, 114, 111, 115, 111, 102, 116, 32, 67, 111, 114, 112, 111, 114, 97, 116, 105, 111, 110, 49, 40, 48, 38, 6, 3, 85, 4, 3, 19, 31, 77, 105, 99, 114, 111, 115, 111, 102, 116, 32, 84, 76, 83, 32, 71, 50, 32, 82, 83, 65, 32, 67, 65, 32, 79, 67, 83, 80, 32, 49, 48],
RawIssuer: [48, 81, 49, 11, 48, 9, 6, 3, 85, 4, 6, 19, 2, 85, 83, 49, 30, 48, 28, 6, 3, 85, 4, 10, 19, 21, 77, 105, 99, 114, 111, 115, 111, 102, 116, 32, 67, 111, 114, 112, 111, 114, 97, 116, 105, 111, 110, 49, 34, 48, 32, 6, 3, 85, 4, 3, 19, 25, 77, 105, 99, 114, 111, 115, 111, 102, 116, 32, 84, 76, 83, 32, 82, 83, 65, 32, 82, 111, 111, 116, 32, 71, 50],
Signature: [53, 25, 202, 233, 168, 117, 225, 216, 202, 253, 37, 193, 59, 82, 225, 231, 158, 43, 148, 226, 244, 58, 210, 79, 218, 6, 34, 82, 26, 1, 24, 227, 162, 186, 183, 183, 28, 36, 50, 117, 13, 103, 68, 125, 157, 74, 130, 75, 250, 227, 68, 233, 70, 246, 0, 69, 9, 194, 82, 233, 103, 37, 151, 199, 183, 139, 90, 119, 124, 64, 30, 148, 227, 188, 140, 109, 56, 41, 188, 39, 125, 18, 18, 61, 203, 203, 228, 196, 70, 226, 165, 1, 192, 16, 209, 32, 250, 14, 3, 228, 67, 219, 38, 135, 37, 172, 200, 160, 179, 4, 106, 94, 240, 67, 63, 15, 123, 245, 206, 35, 195, 183, 201, 190, 251, 222, 155, 125, 147, 179, 135, 62, 223, 198, 219, 61, 201, 244, 172, 113, 0, 210, 164, 101, 129, 54, 89, 159, 28, 54, 229, 137, 209, 122, 153, 74, 123, 85, 75, 194, 213, 140, 208, 121, 1, 121, 149, 15, 57, 67, 139, 179, 10, 123, 146, 213, 90, 16, 24, 118, 233, 29, 13, 111, 50, 49, 244, 189, 209, 22, 241, 168, 163, 103, 242, 14, 154, 112, 12, 227, 150, 175, 3, 244, 109, 189, 20, 121, 106, 54, 195, 241, 182, 115, 187, 198, 11, 213, 38, 70, 136, 84, 69, 112, 170, 204, 31, 61, 139, 111, 227, 228, 23, 166, 129, 0, 251, 125, 131, 88, 128, 226, 242, 60, 59, 163, 24, 98, 19, 54, 103, 255, 79, 226, 229, 111, 106, 103, 230, 232, 90, 100, 58, 160, 69, 157, 63, 23, 12, 102, 135, 240, 129, 39, 52, 32, 176, 53, 36, 100, 40, 118, 67, 122, 116, 152, 226, 43, 128, 9, 10, 188, 176, 185, 7, 71, 41, 40, 46, 125, 224, 119, 65, 164, 140, 54, 120, 186, 51, 29, 86, 142, 217, 16, 53, 137, 89, 131, 0, 105, 206, 183, 120, 189, 237, 175, 130, 63, 206, 200, 137, 50, 152, 89, 8, 93, 225, 1, 150, 88, 196, 101, 25, 125, 38, 133, 157, 16, 150, 229, 149, 197, 17, 206, 130, 111, 23, 6, 113, 21, 15, 44, 191, 55, 50, 82, 21, 104, 180, 149, 142, 83, 93, 151, 237, 227, 219, 19, 62, 193, 49, 177, 121, 147, 115, 125, 114, 59, 27, 76, 79, 6, 73, 21, 132, 115, 243, 29, 90, 81, 44, 164, 212, 88, 224, 25, 191, 53, 72, 41, 96, 153, 171, 234, 123, 66, 3, 190, 156, 61, 25, 247, 71, 91, 11, 172, 130, 87, 170, 13, 189, 255, 34, 96, 48, 223, 44, 79, 87, 211, 37, 213, 218, 253, 142, 6, 81, 78, 5, 53, 120, 152, 56, 163, 39, 49, 111, 49, 154, 169, 170, 77, 172, 58, 68, 104, 96, 112, 104, 244, 83, 107, 205, 178, 45, 174, 38, 24, 135, 223, 140, 87, 181, 248, 82, 193, 86, 231, 192, 177, 77, 95, 10, 85, 107, 20, 43, 84, 251, 61, 169, 198, 189, 96, 7, 248, 176, 244, 75, 227, 238, 171],
SignatureAlgorithm: 5,
PublicKeyAlgorithm: 1,
PublicKey: <*rsa.PublicKey | 0x36a118ac3670>{
N: {
neg: false,
abs: [12623528037057212991, 15782976484946675180, 9131881476679932799, 16877507857597638923, 6948446969886104628, 16879364175701064122, 15349600278999781980, 9245340169127444553, 7968545109705329495, 6423711443832766155, 10594494922037898650, 17648935788986680361, 18122500389998752041, 7737602644086511993, 7258855044086458051, 5541026985182791629, 4439073283700877230, 10408130435767131132, 18089869943821258368, 11685763412349126810, 16394152652608075740, 12871239331426121127, 9800942613550528692, 18261315811323797426, 15374412250364208978, 3968124059102095807, 685806708061009512, 7838166407260346135, 16993058175899549691, 15384298262262378419, 3974146097701597185, 13694573362644585378, 13796492994165901152, 13528806942954960401, 3955259362538190321, 13074935763441801879, 2348393719768315797, 11342510827905045171, 130453718204474471, 17672562703128855053, 634844305723830904, 11538538025395239748, 5505257917472087582, 16743812431760803789, 13404336697386206403, 6770939201695035783, 3646446187118575543, 14898275314275200590, 3021489551454039149, 3930692534149328022, 11247669138812385627, 7850138742151524599, 1567520330174748017, 5626181319708709539, 16420623271324550298, 18240848139101269901, 2441811390160163564, 1395980562250961772, 3030591394459285111, 17666945104455592770, 11923213182329091847, 9136006465603430828, 13028421093649262211, 17297287522694218268],
},
E: 65537,
},
Version: 3,
SerialNumber: {
neg: false,
abs: [12560257885759602703, 4278338880794178, 3342336],
},
Issuer: {
Country: ["US"],
Organization: [
"Microsoft Corporation",
],
OrganizationalUnit: nil,
Locality: nil,
Province: nil,
StreetAddress: nil,
PostalCode: nil,
SerialNumber: "",
CommonName: "Microsoft TLS RSA Root G2",
Names: [
{
Type: [..., ..., ..., ...],
Value: <string>"US",
},
{
Type: [..., ..., ..., ...],
Value: <string>"Microsoft Corporation",
},
{
Type: [..., ..., ..., ...],
Value: <string>"Microsoft TLS RSA Root G2",
},
],
ExtraNames: nil,
},
Subject: {
Country: ["US"],
Organization: [
"Microsoft Corporation",
],
OrganizationalUnit: nil,
Locality: nil,
Province: nil,
StreetAddress: nil,
PostalCode: nil,
SerialNumber: "",
CommonName: "Microsoft TLS G2 RSA CA OCSP 10",
Names: [
{
Type: [..., ..., ..., ...],
Value: <string>"US",
},
{
Type: [..., ..., ..., ...],
Value: <string>"Microsoft Corporation",
},
{
Type: [..., ..., ..., ...],
Value: <string>"Microsoft TLS G2 RSA CA OCSP 10",
},
],
ExtraNames: nil,
},
NotBefore: {wall: 0, ext: 63889675383, loc: nil},
NotAfter: {wall: 0, ext: 64010808183, loc: nil},
KeyUsage: 97,
Extensions: [
{
Id: [2, 5, 29, 15],
Critical: true,
Value: [3, 2, 1, 134],
},
{
Id: [1, 3, 6, 1, 4, 1, 311, 21, 1],
Critical: false,
Value: [2, 1, 0],
},
{
Id: [2, 5, 29, 14],
Critical: false,
Value: [4, 20, 208, 76, 131, 209, 142, 113, 46, 61, 52, 177, 255, 155, 121, 44, 142, 116, 212, 226, 120, 6],
},
{
Id: [2, 5, 29, 32],
Critical: false,
Value: [48, 10, 48, 8, 6, 6, 103, 129, 12, 1, 2, 2],
},
{
Id: [2, 5, 29, 37],
Critical: false,
Value: [48, 10, 6, 8, 43, 6, 1, 5, 5, 7, 3, 1],
},
{
Id: [1, 3, 6, 1, 4, 1, 311, 20, 2],
Critical: false,
Value: [30, 10, 0, 83, 0, 117, 0, 98, 0, 67, 0, 65],
},
{
Id: [2, 5, 29, 19],
Critical: true,
Value: [48, 6, 1, 1, 255, 2, 1, 0],
},
{
Id: [2, 5, 29, 35],
Critical: false,
Value: [48, 22, 128, 20, 222, 145, 134, 72, 183, 161, 49, 89, 49, 241, 75, 95, 7, 169, 220, 136, 121, 218, 168, 118],
},
{
Id: [2, 5, 29, 31],
Critical: false,
Value: "0\x81\xa00\x81\x9d\xa0\x81\x9a\xa0\x81\x97\x86Ihttp://www.microsoft.com/pkiops/crl/Microsoft%20TLS%20RSA%20Root%20G2.crl\x86Jhttp://crl2.microsoft.com/pkiops/crl/Microsoft%20TLS%20RSA%20Root%20G2.crl",
},
{
Id: [1, 3, 6, 1, 5, 5, 7, 1, 1],
Critical: false,
Value: [48, 129, 255, 48, 99, 6, 8, 43, 6, 1, 5, 5, 7, 48, 2, 134, 87, 104, 116, 116, 112, 58, 47, 47, 119, 119, 119, 46, 109, 105, 99, 114, 111, 115, 111, 102, 116, 46, 99, 111, 109, 47, 112, 107, 105, 111, 112, 115, 47, 99, 101, 114, 116, 115, 47, 77, 105, 99, 114, 111, 115, 111, 102, 116, 37, 50, 48, 84, 76, 83, 37, 50, 48, 82, 83, 65, 37, 50, 48, 82, 111, 111, 116, 37, 50, 48, 71, 50, 37, 50, 48, 45, 37, 50, 48, 120, 115, 105, 103, 110, 46, 99, 114, 116, 48, 105, 6, 8, 43, 6, 1, 5, 5, 7, 48, 2, 134, 93, 104, 116, 116, 112, 58, 47, 47, 99, 97, 105, 115, 115, 117, 101, 114, 115, 46, 109, 105, 99, 114, 111, 115, 111, 102, 116, 46, 99, 111, 109, 47, 112, 107, 105, 111, 112, 115, 47, 99, 101, 114, 116, 115, 47, 77, 105, 99, 114, 111, 115, 111, 102, 116, 37, 50, 48, 84, 76, 83, 37, 50, 48, 82, 83, 65, 37, 50, 48, 82, 111, 111, 116, 37, 50, 48, 71, 50, 37, 50, 48, 45, 37, 50, 48, 120, 115, 105, 103, 110, 46, 99, 114, 116, 48, 45, 6, 8, 43, 6, 1, 5, 5, 7, 48, 1, 134, 33, 104, 116, 116, 112, 58, 47, 47, 111, 110, 101, 111, 99, 115, 112, 46, 109, 105, 99, 114, 111, 115, 111, 102, 116, 46, 99, 111, 109, 47, 111, 99, 115, 112],
},
],
ExtraExtensions: nil,
UnhandledCriticalExtensions: nil,
ExtKeyUsage: [1],
UnknownExtKeyUsage: nil,
BasicConstraintsValid: true,
IsCA: true,
MaxPathLen: 0,
MaxPathLenZero: true,
SubjectKeyId: [208, 76, 131, 209, 142, 113, 46, 61, 52, 177, 255, 155, 121, 44, 142, 116, 212, 226, 120, 6],
AuthorityKeyId: [222, 145, 134, 72, 183, 161, 49, 89, 49, 241, 75, 95, 7, 169, 220, 136, 121, 218, 168, 118],
OCSPServer: [
"http://oneocsp.microsoft.com/ocsp",
],
IssuingCertificateURL: [
"http://www.microsoft.com/pkiops/certs/Microsoft%20TLS%20RSA%20Root%20G2%20-%20xsign.crt",
"http://caissuers.microsoft.com/pkiops/certs/Microsoft%20TLS%20RSA%20Root%20G2%20-%20xsign.crt",
],
DNSNames: nil,
EmailAddresses: nil,
IPAddresses: nil,
URIs: nil,
PermittedDNSDomainsCritical: false,
PermittedDNSDomains: nil,
ExcludedDNSDomains: nil,
PermittedIPRanges: nil,
ExcludedIPRanges: nil,
PermittedEmailAddresses: nil,
ExcludedEmailAddresses: nil,
PermittedURIDomains: nil,
ExcludedURIDomains: nil,
CRLDistributionPoints: [
"http://www.microsoft.com/pkiops/crl/Microsoft%20TLS%20RSA%20Root%20G2.crl",
"http://crl2.microsoft.com/pkiops/crl/Microsoft%20TLS%20RSA%20Root%20G2.crl",
],
PolicyIdentifiers: [
[2, 23, 140, 1, 2, 2],
],
Policies: [
{
der: [103, 129, 12, 1, 2, 2],
},
],
InhibitAnyPolicy: 0,
InhibitAnyPolicyZero: false,
InhibitPolicyMapping: 0,
InhibitPolicyMappingZero: false,
RequireExplicitPolicy: 0,
RequireExplicitPolicyZero: false,
PolicyMappings: nil,
},
{
Raw: [48, 130, 5, 137, 48, 130, 4, 113, 160, 3, 2, 1, 2, 2, 16, 11, 12, 107, 44, 70, 105, 23, 176, 71, 115, 198, 71, 212, 175, 192, 200, 48, 13, 6, 9, 42, 134, 72, 134, 247, 13, 1, 1, 12, 5, 0, 48, 97, 49, 11, 48, 9, 6, 3, 85, 4, 6, 19, 2, 85, 83, 49, 21, 48, 19, 6, 3, 85, 4, 10, 19, 12, 68, 105, 103, 105, 67, 101, 114, 116, 32, 73, 110, 99, 49, 25, 48, 23, 6, 3, 85, 4, 11, 19, 16, 119, 119, 119, 46, 100, 105, 103, 105, 99, 101, 114, 116, 46, 99, 111, 109, 49, 32, 48, 30, 6, 3, 85, 4, 3, 19, 23, 68, 105, 103, 105, 67, 101, 114, 116, 32, 71, 108, 111, 98, 97, 108, 32, 82, 111, 111, 116, 32, 71, 50, 48, 30, 23, 13, 50, 53, 48, 53, 50, 49, 48, 48, 48, 48, 48, 48, 90, 23, 13, 50, 57, 48, 54, 49, 57, 50, 51, 53, 57, 53, 57, 90, 48, 81, 49, 11, 48, 9, 6, 3, 85, 4, 6, 19, 2, 85, 83, 49, 30, 48, 28, 6, 3, 85, 4, 10, 19, 21, 77, 105, 99, 114, 111, 115, 111, 102, 116, 32, 67, 111, 114, 112, 111, 114, 97, 116, 105, 111, 110, 49, 34, 48, 32, 6, 3, 85, 4, 3, 19, 25, 77, 105, 99, 114, 111, 115, 111, 102, 116, 32, 84, 76, 83, 32, 82, 83, 65, 32, 82, 111, 111, 116, 32, 71, 50, 48, 130, 2, 34, 48, 13, 6, 9, 42, 134, 72, 134, 247, 13, 1, 1, 1, 5, 0, 3, 130, 2, 15, 0, 48, 130, 2, 10, 2, 130, 2, 1, 0, 223, 234, 139, 159, 71, 225, 40, 16, 113, 175, 65, 214, 25, 219, 146, 87, 222, 107, 171, 230, 46, 222, 173, 58, 96, 131, 183, 19, 152, 172, 91, 185, 50, 11, 123, 235, 159, 114, 39, 107, 93, 64, 171, 190, 24, 6, 147, 178, 13, 210, 18, 237, 184, 116, 245, 127, 47, 234, 168, 23, 148, 246, 51, 22, 87, 138, 222, 32, 124, 33, 221, 183, 218, 172, 222, 78, 100, 176, 90, 205, 75, 117, 243, 239, 2, 100, 77, 237, 82, 218, 163, 143, 4, 232, 145, 248, 215, 251, 63, 25, 253, 78, 69, 57, 94, 232, 41, 195, 214, 242, 103, 175, 179, 0, 148, 193, 141, 225, 240, 74, 24, 152, 57, 250, 192, 9, 132, 51, 65, 5, 3, 5, 171, 244, 117, 225, 83, 165, 33, 146, 92, 68, 76, 230, 96, 87, 164, 144, 2, 151, 199, 213, 216, 147, 4, 50, 74, 237, 133, 81, 237, 214, 197, 186, 160, 63, 54, 138, 47, 111, 76, 57, 168, 4, 105, 34, 132, 59, 117, 20, 247, 217, 62, 180, 169, 208, 7, 65, 56, 129, 126, 70, 72, 29, 232, 25, 230, 112, 184, 160, 170, 24, 221, 114, 191, 126, 125, 181, 108, 78, 25, 112, 17, 132, 4, 19, 129, 212, 98, 253, 156, 107, 241, 28, 221, 255, 236, 167, 224, 230, 199, 141, 49, 75, 191, 242, 107, 51, 204, 209, 140, 182, 254, 177, 27, 163, 74, 168, 95, 14, 127, 49, 118, 56, 63, 7, 71, 108, 1, 168, 172, 165, 20, 218, 130, 176, 95, 31, 108, 188, 137, 26, 192, 18, 42, 203, 123, 151, 31, 40, 46, 49, 22, 138, 83, 76, 117, 102, 249, 125, 1, 214, 194, 96, 170, 101, 143, 49, 158, 114, 203, 204, 155, 45, 229, 68, 25, 147, 87, 171, 110, 142, 194, 217, 46, 101, 176, 114, 45, 231, 161, 195, 161, 126, 73, 29, 89, 103, 214, 9, 212, 24, 18, 109, 192, 149, 69, 120, 201, 241, 96, 106, 228, 246, 238, 6, 36, 59, 115, 74, 90, 17, 194, 62, 151, 97, 192, 120, 238, 239, 19, 167, 48, 226, 74, 56, 45, 238, 88, 23, 19, 32, 183, 54, 42, 223, 11, 50, 180, 194, 56, 177, 145, 221, 18, 176, 69, 182, 124, 192, 177, 29, 98, 161, 157, 221, 113, 18, 21, 150, 1, 157, 10, 9, 245, 177, 221, 179, 217, 224, 56, 66, 52, 246, 105, 182, 195, 175, 22, 115, 254, 212, 67, 248, 67, 228, 88, 249, 79, 179, 243, 236, 214, 61, 148, 58, 64, 56, 21, 171, 12, 134, 208, 100, 206, 26, 99, 232, 132, 148, 85, 9, 90, 182, 4, 182, 209, 62, 154, 21, 199, 100, 17, 128, 208, 27, 130, 14, 77, 255, 69, 158, 178, 101, 7, 37, 31, 219, 154, 200, 198, 236, 214, 91, 133, 80, 4, 159, 28, 203, 93, 32, 130, 103, 158, 121, 226, 223, 232, 242, 232, 111, 196, 32, 114, 48, 134, 229, 157, 117, 39, 53, 2, 3, 1, 0, 1, 163, 130, 1, 75, 48, 130, 1, 71, 48, 15, 6, 3, 85, 29, 19, 1, 1, 255, 4, 5, 48, 3, 1, 1, 255, 48, 29, 6, 3, 85, 29, 14, 4, 22, 4, 20, 222, 145, 134, 72, 183, 161, 49, 89, 49, 241, 75, 95, 7, 169, 220, 136, 121, 218, 168, 118, 48, 31, 6, 3, 85, 29, 35, 4, 24, 48, 22, 128, 20, 78, 34, 84, 32, 24, 149, 230, 227, 110, 230, 15, 250, 250, 185, 18, 237, 6, 23, 143, 57, 48, 14, 6, 3, 85, 29, 15, 1, 1, 255, 4, 4, 3, 2, 1, 134, 48, 19, 6, 3, 85, 29, 37, 4, 12, 48, 10, 6, 8, 43, 6, 1, 5, 5, 7, 3, 1, 48, 118, 6, 8, 43, 6, 1, 5, 5, 7, 1, 1, 4, 106, 48, 104, 48, 36, 6, 8, 43, 6, 1, 5, 5, 7, 48, 1, 134, 24, 104, 116, 116, 112, 58, 47, 47, 111, 99, 115, 112, 46, 100, 105, 103, 105, 99, 101, 114, 116, 46, 99, 111, 109, 48, 64, 6, 8, 43, 6, 1, 5, 5, 7, 48, 2, 134, 52, 104, 116, 116, 112, 58, 47, 47, 99, 97, 99, 101, 114, 116, 115, 46, 100, 105, 103, 105, 99, 101, 114, 116, 46, 99, 111, 109, 47, 68, 105, 103, 105, 67, 101, 114, 116, 71, 108, 111, 98, 97, 108, 82, 111, 111, 116, 71, 50, 46, 99, 114, 116, 48, 66, 6, 3, 85, 29, 31, 4, 59, 48, 57, 48, 55, 160, 53, 160, 51, 134, 49, 104, 116, 116, 112, 58, 47, 47, 99, 114, 108, 51, 46, 100, 105, 103, 105, 99, 101, 114, 116, 46, 99, 111, 109, 47, 68, 105, 103, 105, 67, 101, 114, 116, 71, 108, 111, 98, 97, 108, 82, 111, 111, 116, 71, 50, 46, 99, 114, 108, 48, 19, 6, 3, 85, 29, 32, 4, 12, 48, 10, 48, 8, 6, 6, 103, 129, 12, 1, 2, 2, 48, 13, 6, 9, 42, 134, 72, 134, 247, 13, 1, 1, 12, 5, 0, 3, 130, 1, 1, 0, 11, 188, 180, 43, 55, 116, 197, 75, 167, 54, 2, 54, 198, 175, 225, 19, 204, 138, 154, 151, 27, 252, 227, 68, 140, 238, 84, 0, 229, 228, 65, 26, 70, 240, 11, 118, 52, 255, 153, 85, 98, 53, 170, 173, 138, 156, 12, 90, 98, 204, 67, 192, 203, 234, 74, 29, 189, 107, 29, 175, 177, 25, 37, 102, 156, 247, 121, 110, 91, 37, 128, 36, 129, 133, 199, 121, 45, 53, 95, 66, 9, 120, 237, 159, 226, 101, 213, 196, 32, 80, 130, 16, 143, 106, 58, 237, 24, 142, 72, 137, 171, 166, 9, 248, 225, 128, 252, 6, 117, 22, 39, 37, 90, 143, 184, 58, 112, 108, 24, 71, 240, 99, 244, 83, 14, 217, 21, 53, 32, 243, 145, 61, 19, 1, 160, 226, 250, 36, 8, 69, 86, 119, 209, 103, 169, 49, 66, 208, 87, 255, 26, 38, 190, 197, 67, 216, 87, 169, 99, 255, 175, 243, 58, 216, 246, 123, 75, 16, 162, 151, 161, 207, 102, 190, 103, 103, 9, 249, 181, 76, 231, 171, 25, 43, 191, 34, 61, 125, 222, 198, 247, 239, 99, 56, 44, 221, 86, 196, 249, 9, 82, 218, 223, 227, 136, 111, 213, 109, 107, 247, 8, 65, 47, 142, 62, 134, 49, 159, 241, 124, 74, 245, 214, 97, 164, 70, 70, 40, 239, 87, 186, 176, 111, 146, 216, 210, 119, 37, 237, 123, 166, 41, 206, 59, 114, 121, 17, 212, 224, 167, 84, 109, 53, 43, 164, 122],
RawTBSCertificate: [48, 130, 4, 113, 160, 3, 2, 1, 2, 2, 16, 11, 12, 107, 44, 70, 105, 23, 176, 71, 115, 198, 71, 212, 175, 192, 200, 48, 13, 6, 9, 42, 134, 72, 134, 247, 13, 1, 1, 12, 5, 0, 48, 97, 49, 11, 48, 9, 6, 3, 85, 4, 6, 19, 2, 85, 83, 49, 21, 48, 19, 6, 3, 85, 4, 10, 19, 12, 68, 105, 103, 105, 67, 101, 114, 116, 32, 73, 110, 99, 49, 25, 48, 23, 6, 3, 85, 4, 11, 19, 16, 119, 119, 119, 46, 100, 105, 103, 105, 99, 101, 114, 116, 46, 99, 111, 109, 49, 32, 48, 30, 6, 3, 85, 4, 3, 19, 23, 68, 105, 103, 105, 67, 101, 114, 116, 32, 71, 108, 111, 98, 97, 108, 32, 82, 111, 111, 116, 32, 71, 50, 48, 30, 23, 13, 50, 53, 48, 53, 50, 49, 48, 48, 48, 48, 48, 48, 90, 23, 13, 50, 57, 48, 54, 49, 57, 50, 51, 53, 57, 53, 57, 90, 48, 81, 49, 11, 48, 9, 6, 3, 85, 4, 6, 19, 2, 85, 83, 49, 30, 48, 28, 6, 3, 85, 4, 10, 19, 21, 77, 105, 99, 114, 111, 115, 111, 102, 116, 32, 67, 111, 114, 112, 111, 114, 97, 116, 105, 111, 110, 49, 34, 48, 32, 6, 3, 85, 4, 3, 19, 25, 77, 105, 99, 114, 111, 115, 111, 102, 116, 32, 84, 76, 83, 32, 82, 83, 65, 32, 82, 111, 111, 116, 32, 71, 50, 48, 130, 2, 34, 48, 13, 6, 9, 42, 134, 72, 134, 247, 13, 1, 1, 1, 5, 0, 3, 130, 2, 15, 0, 48, 130, 2, 10, 2, 130, 2, 1, 0, 223, 234, 139, 159, 71, 225, 40, 16, 113, 175, 65, 214, 25, 219, 146, 87, 222, 107, 171, 230, 46, 222, 173, 58, 96, 131, 183, 19, 152, 172, 91, 185, 50, 11, 123, 235, 159, 114, 39, 107, 93, 64, 171, 190, 24, 6, 147, 178, 13, 210, 18, 237, 184, 116, 245, 127, 47, 234, 168, 23, 148, 246, 51, 22, 87, 138, 222, 32, 124, 33, 221, 183, 218, 172, 222, 78, 100, 176, 90, 205, 75, 117, 243, 239, 2, 100, 77, 237, 82, 218, 163, 143, 4, 232, 145, 248, 215, 251, 63, 25, 253, 78, 69, 57, 94, 232, 41, 195, 214, 242, 103, 175, 179, 0, 148, 193, 141, 225, 240, 74, 24, 152, 57, 250, 192, 9, 132, 51, 65, 5, 3, 5, 171, 244, 117, 225, 83, 165, 33, 146, 92, 68, 76, 230, 96, 87, 164, 144, 2, 151, 199, 213, 216, 147, 4, 50, 74, 237, 133, 81, 237, 214, 197, 186, 160, 63, 54, 138, 47, 111, 76, 57, 168, 4, 105, 34, 132, 59, 117, 20, 247, 217, 62, 180, 169, 208, 7, 65, 56, 129, 126, 70, 72, 29, 232, 25, 230, 112, 184, 160, 170, 24, 221, 114, 191, 126, 125, 181, 108, 78, 25, 112, 17, 132, 4, 19, 129, 212, 98, 253, 156, 107, 241, 28, 221, 255, 236, 167, 224, 230, 199, 141, 49, 75, 191, 242, 107, 51, 204, 209, 140, 182, 254, 177, 27, 163, 74, 168, 95, 14, 127, 49, 118, 56, 63, 7, 71, 108, 1, 168, 172, 165, 20, 218, 130, 176, 95, 31, 108, 188, 137, 26, 192, 18, 42, 203, 123, 151, 31, 40, 46, 49, 22, 138, 83, 76, 117, 102, 249, 125, 1, 214, 194, 96, 170, 101, 143, 49, 158, 114, 203, 204, 155, 45, 229, 68, 25, 147, 87, 171, 110, 142, 194, 217, 46, 101, 176, 114, 45, 231, 161, 195, 161, 126, 73, 29, 89, 103, 214, 9, 212, 24, 18, 109, 192, 149, 69, 120, 201, 241, 96, 106, 228, 246, 238, 6, 36, 59, 115, 74, 90, 17, 194, 62, 151, 97, 192, 120, 238, 239, 19, 167, 48, 226, 74, 56, 45, 238, 88, 23, 19, 32, 183, 54, 42, 223, 11, 50, 180, 194, 56, 177, 145, 221, 18, 176, 69, 182, 124, 192, 177, 29, 98, 161, 157, 221, 113, 18, 21, 150, 1, 157, 10, 9, 245, 177, 221, 179, 217, 224, 56, 66, 52, 246, 105, 182, 195, 175, 22, 115, 254, 212, 67, 248, 67, 228, 88, 249, 79, 179, 243, 236, 214, 61, 148, 58, 64, 56, 21, 171, 12, 134, 208, 100, 206, 26, 99, 232, 132, 148, 85, 9, 90, 182, 4, 182, 209, 62, 154, 21, 199, 100, 17, 128, 208, 27, 130, 14, 77, 255, 69, 158, 178, 101, 7, 37, 31, 219, 154, 200, 198, 236, 214, 91, 133, 80, 4, 159, 28, 203, 93, 32, 130, 103, 158, 121, 226, 223, 232, 242, 232, 111, 196, 32, 114, 48, 134, 229, 157, 117, 39, 53, 2, 3, 1, 0, 1, 163, 130, 1, 75, 48, 130, 1, 71, 48, 15, 6, 3, 85, 29, 19, 1, 1, 255, 4, 5, 48, 3, 1, 1, 255, 48, 29, 6, 3, 85, 29, 14, 4, 22, 4, 20, 222, 145, 134, 72, 183, 161, 49, 89, 49, 241, 75, 95, 7, 169, 220, 136, 121, 218, 168, 118, 48, 31, 6, 3, 85, 29, 35, 4, 24, 48, 22, 128, 20, 78, 34, 84, 32, 24, 149, 230, 227, 110, 230, 15, 250, 250, 185, 18, 237, 6, 23, 143, 57, 48, 14, 6, 3, 85, 29, 15, 1, 1, 255, 4, 4, 3, 2, 1, 134, 48, 19, 6, 3, 85, 29, 37, 4, 12, 48, 10, 6, 8, 43, 6, 1, 5, 5, 7, 3, 1, 48, 118, 6, 8, 43, 6, 1, 5, 5, 7, 1, 1, 4, 106, 48, 104, 48, 36, 6, 8, 43, 6, 1, 5, 5, 7, 48, 1, 134, 24, 104, 116, 116, 112, 58, 47, 47, 111, 99, 115, 112, 46, 100, 105, 103, 105, 99, 101, 114, 116, 46, 99, 111, 109, 48, 64, 6, 8, 43, 6, 1, 5, 5, 7, 48, 2, 134, 52, 104, 116, 116, 112, 58, 47, 47, 99, 97, 99, 101, 114, 116, 115, 46, 100, 105, 103, 105, 99, 101, 114, 116, 46, 99, 111, 109, 47, 68, 105, 103, 105, 67, 101, 114, 116, 71, 108, 111, 98, 97, 108, 82, 111, 111, 116, 71, 50, 46, 99, 114, 116, 48, 66, 6, 3, 85, 29, 31, 4, 59, 48, 57, 48, 55, 160, 53, 160, 51, 134, 49, 104, 116, 116, 112, 58, 47, 47, 99, 114, 108, 51, 46, 100, 105, 103, 105, 99, 101, 114, 116, 46, 99, 111, 109, 47, 68, 105, 103, 105, 67, 101, 114, 116, 71, 108, 111, 98, 97, 108, 82, 111, 111, 116, 71, 50, 46, 99, 114, 108, 48, 19, 6, 3, 85, 29, 32, 4, 12, 48, 10, 48, 8, 6, 6, 103, 129, 12, 1, 2, 2],
RawSubjectPublicKeyInfo: [48, 130, 2, 34, 48, 13, 6, 9, 42, 134, 72, 134, 247, 13, 1, 1, 1, 5, 0, 3, 130, 2, 15, 0, 48, 130, 2, 10, 2, 130, 2, 1, 0, 223, 234, 139, 159, 71, 225, 40, 16, 113, 175, 65, 214, 25, 219, 146, 87, 222, 107, 171, 230, 46, 222, 173, 58, 96, 131, 183, 19, 152, 172, 91, 185, 50, 11, 123, 235, 159, 114, 39, 107, 93, 64, 171, 190, 24, 6, 147, 178, 13, 210, 18, 237, 184, 116, 245, 127, 47, 234, 168, 23, 148, 246, 51, 22, 87, 138, 222, 32, 124, 33, 221, 183, 218, 172, 222, 78, 100, 176, 90, 205, 75, 117, 243, 239, 2, 100, 77, 237, 82, 218, 163, 143, 4, 232, 145, 248, 215, 251, 63, 25, 253, 78, 69, 57, 94, 232, 41, 195, 214, 242, 103, 175, 179, 0, 148, 193, 141, 225, 240, 74, 24, 152, 57, 250, 192, 9, 132, 51, 65, 5, 3, 5, 171, 244, 117, 225, 83, 165, 33, 146, 92, 68, 76, 230, 96, 87, 164, 144, 2, 151, 199, 213, 216, 147, 4, 50, 74, 237, 133, 81, 237, 214, 197, 186, 160, 63, 54, 138, 47, 111, 76, 57, 168, 4, 105, 34, 132, 59, 117, 20, 247, 217, 62, 180, 169, 208, 7, 65, 56, 129, 126, 70, 72, 29, 232, 25, 230, 112, 184, 160, 170, 24, 221, 114, 191, 126, 125, 181, 108, 78, 25, 112, 17, 132, 4, 19, 129, 212, 98, 253, 156, 107, 241, 28, 221, 255, 236, 167, 224, 230, 199, 141, 49, 75, 191, 242, 107, 51, 204, 209, 140, 182, 254, 177, 27, 163, 74, 168, 95, 14, 127, 49, 118, 56, 63, 7, 71, 108, 1, 168, 172, 165, 20, 218, 130, 176, 95, 31, 108, 188, 137, 26, 192, 18, 42, 203, 123, 151, 31, 40, 46, 49, 22, 138, 83, 76, 117, 102, 249, 125, 1, 214, 194, 96, 170, 101, 143, 49, 158, 114, 203, 204, 155, 45, 229, 68, 25, 147, 87, 171, 110, 142, 194, 217, 46, 101, 176, 114, 45, 231, 161, 195, 161, 126, 73, 29, 89, 103, 214, 9, 212, 24, 18, 109, 192, 149, 69, 120, 201, 241, 96, 106, 228, 246, 238, 6, 36, 59, 115, 74, 90, 17, 194, 62, 151, 97, 192, 120, 238, 239, 19, 167, 48, 226, 74, 56, 45, 238, 88, 23, 19, 32, 183, 54, 42, 223, 11, 50, 180, 194, 56, 177, 145, 221, 18, 176, 69, 182, 124, 192, 177, 29, 98, 161, 157, 221, 113, 18, 21, 150, 1, 157, 10, 9, 245, 177, 221, 179, 217, 224, 56, 66, 52, 246, 105, 182, 195, 175, 22, 115, 254, 212, 67, 248, 67, 228, 88, 249, 79, 179, 243, 236, 214, 61, 148, 58, 64, 56, 21, 171, 12, 134, 208, 100, 206, 26, 99, 232, 132, 148, 85, 9, 90, 182, 4, 182, 209, 62, 154, 21, 199, 100, 17, 128, 208, 27, 130, 14, 77, 255, 69, 158, 178, 101, 7, 37, 31, 219, 154, 200, 198, 236, 214, 91, 133, 80, 4, 159, 28, 203, 93, 32, 130, 103, 158, 121, 226, 223, 232, 242, 232, 111, 196, 32, 114, 48, 134, 229, 157, 117, 39, 53, 2, 3, 1, 0, 1],
RawSubject: [48, 81, 49, 11, 48, 9, 6, 3, 85, 4, 6, 19, 2, 85, 83, 49, 30, 48, 28, 6, 3, 85, 4, 10, 19, 21, 77, 105, 99, 114, 111, 115, 111, 102, 116, 32, 67, 111, 114, 112, 111, 114, 97, 116, 105, 111, 110, 49, 34, 48, 32, 6, 3, 85, 4, 3, 19, 25, 77, 105, 99, 114, 111, 115, 111, 102, 116, 32, 84, 76, 83, 32, 82, 83, 65, 32, 82, 111, 111, 116, 32, 71, 50],
RawIssuer: [48, 97, 49, 11, 48, 9, 6, 3, 85, 4, 6, 19, 2, 85, 83, 49, 21, 48, 19, 6, 3, 85, 4, 10, 19, 12, 68, 105, 103, 105, 67, 101, 114, 116, 32, 73, 110, 99, 49, 25, 48, 23, 6, 3, 85, 4, 11, 19, 16, 119, 119, 119, 46, 100, 105, 103, 105, 99, 101, 114, 116, 46, 99, 111, 109, 49, 32, 48, 30, 6, 3, 85, 4, 3, 19, 23, 68, 105, 103, 105, 67, 101, 114, 116, 32, 71, 108, 111, 98, 97, 108, 32, 82, 111, 111, 116, 32, 71, 50],
Signature: [11, 188, 180, 43, 55, 116, 197, 75, 167, 54, 2, 54, 198, 175, 225, 19, 204, 138, 154, 151, 27, 252, 227, 68, 140, 238, 84, 0, 229, 228, 65, 26, 70, 240, 11, 118, 52, 255, 153, 85, 98, 53, 170, 173, 138, 156, 12, 90, 98, 204, 67, 192, 203, 234, 74, 29, 189, 107, 29, 175, 177, 25, 37, 102, 156, 247, 121, 110, 91, 37, 128, 36, 129, 133, 199, 121, 45, 53, 95, 66, 9, 120, 237, 159, 226, 101, 213, 196, 32, 80, 130, 16, 143, 106, 58, 237, 24, 142, 72, 137, 171, 166, 9, 248, 225, 128, 252, 6, 117, 22, 39, 37, 90, 143, 184, 58, 112, 108, 24, 71, 240, 99, 244, 83, 14, 217, 21, 53, 32, 243, 145, 61, 19, 1, 160, 226, 250, 36, 8, 69, 86, 119, 209, 103, 169, 49, 66, 208, 87, 255, 26, 38, 190, 197, 67, 216, 87, 169, 99, 255, 175, 243, 58, 216, 246, 123, 75, 16, 162, 151, 161, 207, 102, 190, 103, 103, 9, 249, 181, 76, 231, 171, 25, 43, 191, 34, 61, 125, 222, 198, 247, 239, 99, 56, 44, 221, 86, 196, 249, 9, 82, 218, 223, 227, 136, 111, 213, 109, 107, 247, 8, 65, 47, 142, 62, 134, 49, 159, 241, 124, 74, 245, 214, 97, 164, 70, 70, 40, 239, 87, 186, 176, 111, 146, 216, 210, 119, 37, 237, 123, 166, 41, 206, 59, 114, 121, 17, 212, 224, 167, 84, 109, 53, 43, 164, 122],
SignatureAlgorithm: 5,
PublicKeyAlgorithm: 1,
PublicKey: <*rsa.PublicKey | 0x36a118ac37b0>{
N: {
neg: false,
abs: [8228224839953229621, 16348041302357558304, 2074854447542410873, 14334067402580427935, 12854688567428618952, 14995722382209926558, 15077658020813214080, 9553354208224609462, 902637906707506152, 15437658075602359723, 4892132923038495724, 14100513645772686328, 15699610158021110198, 116260203274558899, 7107135261377697174, 1346652938663145757, 806906029520556509, 6347563231035665119, 1416154356297248238, 13996791138010853103, 17151436068643035665, 5005973024364225782, 15423091598760460437, 11656337824329587047, 14040304327591079399, 16520357453723561614, 10318202437165161261, 17977527309428632165, 3328466382962324838, 13840171658303184680, 9417131409404627226, 5146490298140267738, 6849551933690625799, 10139571647697603240, 3552143779013315793, 15996764607220205453, 9355211167345930524, 7804203173400544275, 12256789870797553077, 5196564742993393824, 12236288164267916870, 9528338169550421684, 3418034452713204002, 17138102837348087434, 15605821747820856657, 6942198289040656341, 6027260588802198758, 4685154310243513825, 1772230202496746547, 12898472891819094090, 6838761955263670191, 15563102318165378361, 5970264090691015160, 5437520332982603245, 15757213624540617421, 6308098459188714935, 3452756883569783574, 995879279813522815, 6719559476972000178, 3606112178562803563, 6954603544383871929, 16027092704045215034, 8191838635022848599, 16134862131230222352],
},
E: 65537,
},
Version: 3,
SerialNumber: {
neg: false,
abs: [5148676810827874504, 796129072041760688],
},
Issuer: {
Country: ["US"],
Organization: ["DigiCert Inc"],
OrganizationalUnit: ["www.digicert.com"],
Locality: nil,
Province: nil,
StreetAddress: nil,
PostalCode: nil,
SerialNumber: "",
CommonName: "DigiCert Global Root G2",
Names: [
{
Type: [..., ..., ..., ...],
Value: <string>"US",
},
{
Type: [..., ..., ..., ...],
Value: <string>"DigiCert Inc",
},
{
Type: [..., ..., ..., ...],
Value: <string>"www.digicert.com",
},
{
Type: [..., ..., ..., ...],
Value: <string>"DigiCert Global Root G2",
},
],
ExtraNames: nil,
},
Subject: {
Country: ["US"],
Organization: [
"Microsoft Corporation",
],
OrganizationalUnit: nil,
Locality: nil,
Province: nil,
StreetAddress: nil,
PostalCode: nil,
SerialNumber: "",
CommonName: "Microsoft TLS RSA Root G2",
Names: [
{
Type: [..., ..., ..., ...],
Value: <string>"US",
},
{
Type: [..., ..., ..., ...],
Value: <string>"Microsoft Corporation",
},
{
Type: [..., ..., ..., ...],
Value: <string>"Microsoft TLS RSA Root G2",
},
],
ExtraNames: nil,
},
NotBefore: {wall: 0, ext: 63883382400, loc: nil},
NotAfter: {wall: 0, ext: 64012204799, loc: nil},
KeyUsage: 97,
Extensions: [
{
Id: [2, 5, 29, 19],
Critical: true,
Value: [48, 3, 1, 1, 255],
},
{
Id: [2, 5, 29, 14],
Critical: false,
Value: [4, 20, 222, 145, 134, 72, 183, 161, 49, 89, 49, 241, 75, 95, 7, 169, 220, 136, 121, 218, 168, 118],
},
{
Id: [2, 5, 29, 35],
Critical: false,
Value: [48, 22, 128, 20, 78, 34, 84, 32, 24, 149, 230, 227, 110, 230, 15, 250, 250, 185, 18, 237, 6, 23, 143, 57],
},
{
Id: [2, 5, 29, 15],
Critical: true,
Value: [3, 2, 1, 134],
},
{
Id: [2, 5, 29, 37],
Critical: false,
Value: [48, 10, 6, 8, 43, 6, 1, 5, 5, 7, 3, 1],
},
{
Id: [1, 3, 6, 1, 5, 5, 7, 1, 1],
Critical: false,
Value: [48, 104, 48, 36, 6, 8, 43, 6, 1, 5, 5, 7, 48, 1, 134, 24, 104, 116, 116, 112, 58, 47, 47, 111, 99, 115, 112, 46, 100, 105, 103, 105, 99, 101, 114, 116, 46, 99, 111, 109, 48, 64, 6, 8, 43, 6, 1, 5, 5, 7, 48, 2, 134, 52, 104, 116, 116, 112, 58, 47, 47, 99, 97, 99, 101, 114, 116, 115, 46, 100, 105, 103, 105, 99, 101, 114, 116, 46, 99, 111, 109, 47, 68, 105, 103, 105, 67, 101, 114, 116, 71, 108, 111, 98, 97, 108, 82, 111, 111, 116, 71, 50, 46, 99, 114, 116],
},
{
Id: [2, 5, 29, 31],
Critical: false,
Value: "0907\xa05\xa03\x861http://crl3.digicert.com/DigiCertGlobalRootG2.crl",
},
{
Id: [2, 5, 29, 32],
Critical: false,
Value: [48, 10, 48, 8, 6, 6, 103, 129, 12, 1, 2, 2],
},
],
ExtraExtensions: nil,
UnhandledCriticalExtensions: nil,
ExtKeyUsage: [1],
UnknownExtKeyUsage: nil,
BasicConstraintsValid: true,
IsCA: true,
MaxPathLen: -1,
MaxPathLenZero: false,
SubjectKeyId: [222, 145, 134, 72, 183, 161, 49, 89, 49, 241, 75, 95, 7, 169, 220, 136, 121, 218, 168, 118],
AuthorityKeyId: [78, 34, 84, 32, 24, 149, 230, 227, 110, 230, 15, 250, 250, 185, 18, 237, 6, 23, 143, 57],
OCSPServer: [
"http://ocsp.digicert.com",
],
IssuingCertificateURL: [
"http://cacerts.digicert.com/DigiCertGlobalRootG2.crt",
],
DNSNames: nil,
EmailAddresses: nil,
IPAddresses: nil,
URIs: nil,
PermittedDNSDomainsCritical: false,
PermittedDNSDomains: nil,
ExcludedDNSDomains: nil,
PermittedIPRanges: nil,
ExcludedIPRanges: nil,
PermittedEmailAddresses: nil,
ExcludedEmailAddresses: nil,
PermittedURIDomains: nil,
ExcludedURIDomains: nil,
CRLDistributionPoints: [
"http://crl3.digicert.com/DigiCertGlobalRootG2.crl",
],
PolicyIdentifiers: [
[2, 23, 140, 1, 2, 2],
],
Policies: [
{
der: [103, 129, 12, 1, 2, 2],
},
],
InhibitAnyPolicy: 0,
InhibitAnyPolicyZero: false,
InhibitPolicyMapping: 0,
InhibitPolicyMappingZero: false,
RequireExplicitPolicy: 0,
RequireExplicitPolicyZero: false,
PolicyMappings: nil,
},
{
Raw: [48, 130, 3, 142, 48, 130, 2, 118, 160, 3, 2, 1, 2, 2, 16, 3, 58, 241, 230, 167, 17, 169, 160, 187, 40, 100, 177, 29, 9, 250, 229, 48, 13, 6, 9, 42, 134, 72, 134, 247, 13, 1, 1, 11, 5, 0, 48, 97, 49, 11, 48, 9, 6, 3, 85, 4, 6, 19, 2, 85, 83, 49, 21, 48, 19, 6, 3, 85, 4, 10, 19, 12, 68, 105, 103, 105, 67, 101, 114, 116, 32, 73, 110, 99, 49, 25, 48, 23, 6, 3, 85, 4, 11, 19, 16, 119, 119, 119, 46, 100, 105, 103, 105, 99, 101, 114, 116, 46, 99, 111, 109, 49, 32, 48, 30, 6, 3, 85, 4, 3, 19, 23, 68, 105, 103, 105, 67, 101, 114, 116, 32, 71, 108, 111, 98, 97, 108, 32, 82, 111, 111, 116, 32, 71, 50, 48, 30, 23, 13, 49, 51, 48, 56, 48, 49, 49, 50, 48, 48, 48, 48, 90, 23, 13, 51, 56, 48, 49, 49, 53, 49, 50, 48, 48, 48, 48, 90, 48, 97, 49, 11, 48, 9, 6, 3, 85, 4, 6, 19, 2, 85, 83, 49, 21, 48, 19, 6, 3, 85, 4, 10, 19, 12, 68, 105, 103, 105, 67, 101, 114, 116, 32, 73, 110, 99, 49, 25, 48, 23, 6, 3, 85, 4, 11, 19, 16, 119, 119, 119, 46, 100, 105, 103, 105, 99, 101, 114, 116, 46, 99, 111, 109, 49, 32, 48, 30, 6, 3, 85, 4, 3, 19, 23, 68, 105, 103, 105, 67, 101, 114, 116, 32, 71, 108, 111, 98, 97, 108, 32, 82, 111, 111, 116, 32, 71, 50, 48, 130, 1, 34, 48, 13, 6, 9, 42, 134, 72, 134, 247, 13, 1, 1, 1, 5, 0, 3, 130, 1, 15, 0, 48, 130, 1, 10, 2, 130, 1, 1, 0, 187, 55, 205, 52, 220, 123, 107, 201, 178, 104, 144, 173, 74, 117, 255, 70, 186, 33, 10, 8, 141, 245, 25, 84, 201, 251, 136, 219, 243, 174, 242, 58, 137, 145, 60, 122, 230, 171, 6, 26, 107, 207, 172, 45, 232, 94, 9, 36, 68, 186, 98, 154, 126, 214, 163, 168, 126, 224, 84, 117, 32, 5, 172, 80, 183, 156, 99, 26, 108, 48, 220, 218, 31, 25, 177, 215, 30, 222, 253, 215, 224, 203, 148, 131, 55, 174, 236, 31, 67, 78, 221, 123, 44, 210, 189, 46, 165, 47, 228, 169, 184, 173, 58, 212, 153, 164, 182, 37, 233, 155, 107, 0, 96, 146, 96, 255, 79, 33, 73, 24, 247, 103, 144, 171, 97, 6, 156, 143, 242, 186, 233, 180, 233, 146, 50, 107, 181, 243, 87, 232, 93, 27, 205, 140, 29, 171, 149, 4, 149, 73, 243, 53, 45, 150, 227, 73, 109, 221, 119, 227, 251, 73, 75, 180, 172, 85, 7, 169, 143, 149, 179, 180, 35, 187, 76, 109, 69, 240, 246, 169, 178, 149, 48, 180, 253, 76, 85, 140, 39, 74, 87, 20, 124, 130, 157, 205, 115, 146, 211, 22, 74, 6, 12, 140, 80, 209, 143, 30, 9, 190, 23, 161, 230, 33, 202, 253, 131, 229, 16, 188, 131, 165, 10, 196, 103, 40, 246, 115, 20, 20, 61, 70, 118, 195, 135, 20, 137, 33, 52, 77, 175, 15, 69, 12, 166, 73, 161, 186, 187, 156, 197, 177, 51, 131, 41, 133, 2, 3, 1, 0, 1, 163, 66, 48, 64, 48, 15, 6, 3, 85, 29, 19, 1, 1, 255, 4, 5, 48, 3, 1, 1, 255, 48, 14, 6, 3, 85, 29, 15, 1, 1, 255, 4, 4, 3, 2, 1, 134, 48, 29, 6, 3, 85, 29, 14, 4, 22, 4, 20, 78, 34, 84, 32, 24, 149, 230, 227, 110, 230, 15, 250, 250, 185, 18, 237, 6, 23, 143, 57, 48, 13, 6, 9, 42, 134, 72, 134, 247, 13, 1, 1, 11, 5, 0, 3, 130, 1, 1, 0, 96, 103, 40, 148, 111, 14, 72, 99, 235, 49, 221, 234, 103, 24, 213, 137, 125, 60, 197, 139, 74, 127, 233, 190, 219, 43, 23, 223, 176, 95, 115, 119, 42, 50, 19, 57, 129, 103, 66, 132, 35, 242, 69, 103, 53, 236, 136, 191, 248, 143, 176, 97, 12, 52, 164, 174, 32, 76, 132, 198, 219, 248, 53, 225, 118, 217, 223, 166, 66, 187, 199, 68, 8, 134, 127, 54, 116, 36, 90, 218, 108, 13, 20, 89, 53, 189, 242, 73, 221, 182, 31, 201, 179, 13, 71, 42, 61, 153, 47, 187, 92, 187, 181, 212, 32, 225, 153, 95, 83, 70, 21, 219, 104, 155, 240, 243, 48, 213, 62, 49, 226, 141, 132, 158, 227, 138, 218, 218, 150, 62, 53, 19, 165, 95, 240, 249, 112, 80, 112, 71, 65, 17, 87, 25, 78, 192, 143, 174, 6, 196, 149, 19, 23, 47, 27, 37, 159, 117, 242, 177, 142, 153, 161, 111, 19, 177, 65, 113, 254, 136, 42, 200, 79, 16, 32, 85, 215, 243, 20, 69, 229, 224, 68, 244, 234, 135, 149, 50, 147, 14, 254, 83, 70, 250, 44, 157, 255, 139, 34, 185, 75, 217, 9, 69, 164, 222, 164, 184, 154, 88, 221, 27, 125, 82, 159, 142, 89, 67, 136, 129, 164, 158, 38, 213, 111, 173, 221, 13, 198, 55, 125, 237, 3, 146, 27, 229, 119, 95, 118, 238, 60, 141, 196, 93, 86, 91, 162, 217, 102, 110, 179, 53, 55, 229, 50, 182],
RawTBSCertificate: [48, 130, 2, 118, 160, 3, 2, 1, 2, 2, 16, 3, 58, 241, 230, 167, 17, 169, 160, 187, 40, 100, 177, 29, 9, 250, 229, 48, 13, 6, 9, 42, 134, 72, 134, 247, 13, 1, 1, 11, 5, 0, 48, 97, 49, 11, 48, 9, 6, 3, 85, 4, 6, 19, 2, 85, 83, 49, 21, 48, 19, 6, 3, 85, 4, 10, 19, 12, 68, 105, 103, 105, 67, 101, 114, 116, 32, 73, 110, 99, 49, 25, 48, 23, 6, 3, 85, 4, 11, 19, 16, 119, 119, 119, 46, 100, 105, 103, 105, 99, 101, 114, 116, 46, 99, 111, 109, 49, 32, 48, 30, 6, 3, 85, 4, 3, 19, 23, 68, 105, 103, 105, 67, 101, 114, 116, 32, 71, 108, 111, 98, 97, 108, 32, 82, 111, 111, 116, 32, 71, 50, 48, 30, 23, 13, 49, 51, 48, 56, 48, 49, 49, 50, 48, 48, 48, 48, 90, 23, 13, 51, 56, 48, 49, 49, 53, 49, 50, 48, 48, 48, 48, 90, 48, 97, 49, 11, 48, 9, 6, 3, 85, 4, 6, 19, 2, 85, 83, 49, 21, 48, 19, 6, 3, 85, 4, 10, 19, 12, 68, 105, 103, 105, 67, 101, 114, 116, 32, 73, 110, 99, 49, 25, 48, 23, 6, 3, 85, 4, 11, 19, 16, 119, 119, 119, 46, 100, 105, 103, 105, 99, 101, 114, 116, 46, 99, 111, 109, 49, 32, 48, 30, 6, 3, 85, 4, 3, 19, 23, 68, 105, 103, 105, 67, 101, 114, 116, 32, 71, 108, 111, 98, 97, 108, 32, 82, 111, 111, 116, 32, 71, 50, 48, 130, 1, 34, 48, 13, 6, 9, 42, 134, 72, 134, 247, 13, 1, 1, 1, 5, 0, 3, 130, 1, 15, 0, 48, 130, 1, 10, 2, 130, 1, 1, 0, 187, 55, 205, 52, 220, 123, 107, 201, 178, 104, 144, 173, 74, 117, 255, 70, 186, 33, 10, 8, 141, 245, 25, 84, 201, 251, 136, 219, 243, 174, 242, 58, 137, 145, 60, 122, 230, 171, 6, 26, 107, 207, 172, 45, 232, 94, 9, 36, 68, 186, 98, 154, 126, 214, 163, 168, 126, 224, 84, 117, 32, 5, 172, 80, 183, 156, 99, 26, 108, 48, 220, 218, 31, 25, 177, 215, 30, 222, 253, 215, 224, 203, 148, 131, 55, 174, 236, 31, 67, 78, 221, 123, 44, 210, 189, 46, 165, 47, 228, 169, 184, 173, 58, 212, 153, 164, 182, 37, 233, 155, 107, 0, 96, 146, 96, 255, 79, 33, 73, 24, 247, 103, 144, 171, 97, 6, 156, 143, 242, 186, 233, 180, 233, 146, 50, 107, 181, 243, 87, 232, 93, 27, 205, 140, 29, 171, 149, 4, 149, 73, 243, 53, 45, 150, 227, 73, 109, 221, 119, 227, 251, 73, 75, 180, 172, 85, 7, 169, 143, 149, 179, 180, 35, 187, 76, 109, 69, 240, 246, 169, 178, 149, 48, 180, 253, 76, 85, 140, 39, 74, 87, 20, 124, 130, 157, 205, 115, 146, 211, 22, 74, 6, 12, 140, 80, 209, 143, 30, 9, 190, 23, 161, 230, 33, 202, 253, 131, 229, 16, 188, 131, 165, 10, 196, 103, 40, 246, 115, 20, 20, 61, 70, 118, 195, 135, 20, 137, 33, 52, 77, 175, 15, 69, 12, 166, 73, 161, 186, 187, 156, 197, 177, 51, 131, 41, 133, 2, 3, 1, 0, 1, 163, 66, 48, 64, 48, 15, 6, 3, 85, 29, 19, 1, 1, 255, 4, 5, 48, 3, 1, 1, 255, 48, 14, 6, 3, 85, 29, 15, 1, 1, 255, 4, 4, 3, 2, 1, 134, 48, 29, 6, 3, 85, 29, 14, 4, 22, 4, 20, 78, 34, 84, 32, 24, 149, 230, 227, 110, 230, 15, 250, 250, 185, 18, 237, 6, 23, 143, 57],
RawSubjectPublicKeyInfo: [48, 130, 1, 34, 48, 13, 6, 9, 42, 134, 72, 134, 247, 13, 1, 1, 1, 5, 0, 3, 130, 1, 15, 0, 48, 130, 1, 10, 2, 130, 1, 1, 0, 187, 55, 205, 52, 220, 123, 107, 201, 178, 104, 144, 173, 74, 117, 255, 70, 186, 33, 10, 8, 141, 245, 25, 84, 201, 251, 136, 219, 243, 174, 242, 58, 137, 145, 60, 122, 230, 171, 6, 26, 107, 207, 172, 45, 232, 94, 9, 36, 68, 186, 98, 154, 126, 214, 163, 168, 126, 224, 84, 117, 32, 5, 172, 80, 183, 156, 99, 26, 108, 48, 220, 218, 31, 25, 177, 215, 30, 222, 253, 215, 224, 203, 148, 131, 55, 174, 236, 31, 67, 78, 221, 123, 44, 210, 189, 46, 165, 47, 228, 169, 184, 173, 58, 212, 153, 164, 182, 37, 233, 155, 107, 0, 96, 146, 96, 255, 79, 33, 73, 24, 247, 103, 144, 171, 97, 6, 156, 143, 242, 186, 233, 180, 233, 146, 50, 107, 181, 243, 87, 232, 93, 27, 205, 140, 29, 171, 149, 4, 149, 73, 243, 53, 45, 150, 227, 73, 109, 221, 119, 227, 251, 73, 75, 180, 172, 85, 7, 169, 143, 149, 179, 180, 35, 187, 76, 109, 69, 240, 246, 169, 178, 149, 48, 180, 253, 76, 85, 140, 39, 74, 87, 20, 124, 130, 157, 205, 115, 146, 211, 22, 74, 6, 12, 140, 80, 209, 143, 30, 9, 190, 23, 161, 230, 33, 202, 253, 131, 229, 16, 188, 131, 165, 10, 196, 103, 40, 246, 115, 20, 20, 61, 70, 118, 195, 135, 20, 137, 33, 52, 77, 175, 15, 69, 12, 166, 73, 161, 186, 187, 156, 197, 177, 51, 131, 41, 133, 2, 3, 1, 0, 1],
RawSubject: [48, 97, 49, 11, 48, 9, 6, 3, 85, 4, 6, 19, 2, 85, 83, 49, 21, 48, 19, 6, 3, 85, 4, 10, 19, 12, 68, 105, 103, 105, 67, 101, 114, 116, 32, 73, 110, 99, 49, 25, 48, 23, 6, 3, 85, 4, 11, 19, 16, 119, 119, 119, 46, 100, 105, 103, 105, 99, 101, 114, 116, 46, 99, 111, 109, 49, 32, 48, 30, 6, 3, 85, 4, 3, 19, 23, 68, 105, 103, 105, 67, 101, 114, 116, 32, 71, 108, 111, 98, 97, 108, 32, 82, 111, 111, 116, 32, 71, 50],
RawIssuer: [48, 97, 49, 11, 48, 9, 6, 3, 85, 4, 6, 19, 2, 85, 83, 49, 21, 48, 19, 6, 3, 85, 4, 10, 19, 12, 68, 105, 103, 105, 67, 101, 114, 116, 32, 73, 110, 99, 49, 25, 48, 23, 6, 3, 85, 4, 11, 19, 16, 119, 119, 119, 46, 100, 105, 103, 105, 99, 101, 114, 116, 46, 99, 111, 109, 49, 32, 48, 30, 6, 3, 85, 4, 3, 19, 23, 68, 105, 103, 105, 67, 101, 114, 116, 32, 71, 108, 111, 98, 97, 108, 32, 82, 111, 111, 116, 32, 71, 50],
Signature: [96, 103, 40, 148, 111, 14, 72, 99, 235, 49, 221, 234, 103, 24, 213, 137, 125, 60, 197, 139, 74, 127, 233, 190, 219, 43, 23, 223, 176, 95, 115, 119, 42, 50, 19, 57, 129, 103, 66, 132, 35, 242, 69, 103, 53, 236, 136, 191, 248, 143, 176, 97, 12, 52, 164, 174, 32, 76, 132, 198, 219, 248, 53, 225, 118, 217, 223, 166, 66, 187, 199, 68, 8, 134, 127, 54, 116, 36, 90, 218, 108, 13, 20, 89, 53, 189, 242, 73, 221, 182, 31, 201, 179, 13, 71, 42, 61, 153, 47, 187, 92, 187, 181, 212, 32, 225, 153, 95, 83, 70, 21, 219, 104, 155, 240, 243, 48, 213, 62, 49, 226, 141, 132, 158, 227, 138, 218, 218, 150, 62, 53, 19, 165, 95, 240, 249, 112, 80, 112, 71, 65, 17, 87, 25, 78, 192, 143, 174, 6, 196, 149, 19, 23, 47, 27, 37, 159, 117, 242, 177, 142, 153, 161, 111, 19, 177, 65, 113, 254, 136, 42, 200, 79, 16, 32, 85, 215, 243, 20, 69, 229, 224, 68, 244, 234, 135, 149, 50, 147, 14, 254, 83, 70, 250, 44, 157, 255, 139, 34, 185, 75, 217, 9, 69, 164, 222, 164, 184, 154, 88, 221, 27, 125, 82, 159, 142, 89, 67, 136, 129, 164, 158, 38, 213, 111, 173, 221, 13, 198, 55, 125, 237, 3, 146, 27, 229, 119, 95, 118, 238, 60, 141, 196, 93, 86, 91, 162, 217, 102, 110, 179, 53, 55, 229, 50, 182],
SignatureAlgorithm: 4,
PublicKeyAlgorithm: 1,
PublicKey: <*rsa.PublicKey | 0x36a11782c840>{
N: {
neg: false,
abs: [13518897546323503493, 12614377001919816122, 8557832239203824717, 7433462159093546310, 9504020990483827396, 702024576034523901, 5333964605457796894, 8971906913408307990, 18252057450268743444, 5039799191733612724, 10346373305228545133, 18107087016177239977, 3285062882728703971, 2137966295034491701, 13110919595711778188, 17490549066068800107, 17827376715711028367, 6958731023901346072, 11071174057932843776, 11902983757762476756, 4850057369827327278, 16198203776063958047, 2241017827087547863, 13230558770443115738, 9142400105587256400, 4952379155939763112, 7768617195409377572, 9912770753372816922, 14554377099045433914, 13412012197143714132, 12855684210282069830, 13490476835732679625],
},
E: 65537,
},
Version: 3,
SerialNumber: {
neg: false,
abs: [13486139796020132581, 232764303710726560],
},
Issuer: {
Country: ["US"],
Organization: ["DigiCert Inc"],
OrganizationalUnit: ["www.digicert.com"],
Locality: nil,
Province: nil,
StreetAddress: nil,
PostalCode: nil,
SerialNumber: "",
CommonName: "DigiCert Global Root G2",
Names: [
{
Type: [..., ..., ..., ...],
Value: <string>"US",
},
{
Type: [..., ..., ..., ...],
Value: <string>"DigiCert Inc",
},
{
Type: [..., ..., ..., ...],
Value: <string>"www.digicert.com",
},
{
Type: [..., ..., ..., ...],
Value: <string>"DigiCert Global Root G2",
},
],
ExtraNames: nil,
},
Subject: {
Country: ["US"],
Organization: ["DigiCert Inc"],
OrganizationalUnit: ["www.digicert.com"],
Locality: nil,
Province: nil,
StreetAddress: nil,
PostalCode: nil,
SerialNumber: "",
CommonName: "DigiCert Global Root G2",
Names: [
{
Type: [..., ..., ..., ...],
Value: <string>"US",
},
{
Type: [..., ..., ..., ...],
Value: <string>"DigiCert Inc",
},
{
Type: [..., ..., ..., ...],
Value: <string>"www.digicert.com",
},
{
Type: [..., ..., ..., ...],
Value: <string>"DigiCert Global Root G2",
},
],
ExtraNames: nil,
},
NotBefore: {wall: 0, ext: 63510955200, loc: nil},
NotAfter: {wall: 0, ext: 64282766400, loc: nil},
KeyUsage: 97,
Extensions: [
{
Id: [2, 5, 29, 19],
Critical: true,
Value: [48, 3, 1, 1, 255],
},
{
Id: [2, 5, 29, 15],
Critical: true,
Value: [3, 2, 1, 134],
},
{
Id: [2, 5, 29, 14],
Critical: false,
Value: [4, 20, 78, 34, 84, 32, 24, 149, 230, 227, 110, 230, 15, 250, 250, 185, 18, 237, 6, 23, 143, 57],
},
],
ExtraExtensions: nil,
UnhandledCriticalExtensions: nil,
ExtKeyUsage: nil,
UnknownExtKeyUsage: nil,
BasicConstraintsValid: true,
IsCA: true,
MaxPathLen: -1,
MaxPathLenZero: false,
SubjectKeyId: [78, 34, 84, 32, 24, 149, 230, 227, 110, 230, 15, 250, 250, 185, 18, 237, 6, 23, 143, 57],
AuthorityKeyId: nil,
OCSPServer: nil,
IssuingCertificateURL: nil,
DNSNames: nil,
EmailAddresses: nil,
IPAddresses: nil,
URIs: nil,
PermittedDNSDomainsCritical: false,
PermittedDNSDomains: nil,
ExcludedDNSDomains: nil,
PermittedIPRanges: nil,
ExcludedIPRanges: nil,
PermittedEmailAddresses: nil,
ExcludedEmailAddresses: nil,
PermittedURIDomains: nil,
ExcludedURIDomains: nil,
CRLDistributionPoints: nil,
PolicyIdentifiers: nil,
Policies: nil,
InhibitAnyPolicy: 0,
InhibitAnyPolicyZero: false,
InhibitPolicyMapping: 0,
InhibitPolicyMappingZero: false,
RequireExplicitPolicy: 0,
RequireExplicitPolicyZero: false,
PolicyMappings: nil,
},
],
Err: <x509.UnknownAuthorityError>{
Cert: 0x36a1182f5208 (cyclic reference),
hintErr: nil,
hintCert: nil,
},
},
},
},
}Contributing tests (1)
Affected runs (1)
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
VerifyImagePulled(registry.redhat.io/ubi-minimal) timed out after 3m0.006s: failed to list pods in namespace pullsecret-image-test: context deadline exceededfull failure pattern: VerifyImagePulled(registry.redhat.io/ubi-minimal) timed out after 3m0.006s: failed to list pods in namespace pullsecret-image-test: context deadline exceeded Signal: IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | e2e | 1 | 3.70%1 of 27 job runs affected | IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | Show trend detailsOct 12: 0 · Oct 13: 0 · Oct 14: 0 · Oct 15: 0 · Oct 16: 0 · Oct 17: 0 · Oct 18: 0 · Oct 19: 0 · Oct 20: 0 · Oct 21: 1 · Oct 22: 0 · Oct 23: 0 · Oct 24: 0 · Oct 25: 0 | none | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/e2e/cluster_pullsecret.go:297]: failed to wait for image from registry.redhat.io to be pulled successfully with the added pull secret
Unexpected error:
<*fmt.wrapError | 0x2528dfc02700>:
VerifyImagePulled(registry.redhat.io/ubi-minimal) timed out after 3m0.006s: failed to list pods in namespace pullsecret-image-test: context deadline exceeded
{
msg: "VerifyImagePulled(registry.redhat.io/ubi-minimal) timed out after 3m0.006s: failed to list pods in namespace pullsecret-image-test: context deadline exceeded",
err: <*fmt.wrapError | 0x2528dfc026e0>{
msg: "failed to list pods in namespace pullsecret-image-test: context deadline exceeded",
err: <context.deadlineExceededError>{},
},
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_pullsecret.go:297]: failed to wait for image from registry.redhat.io to be pulled successfully with the added pull secret
Unexpected error:
<*fmt.wrapError | 0x2528dfc02700>:
VerifyImagePulled(registry.redhat.io/ubi-minimal) timed out after 3m0.006s: failed to list pods in namespace pullsecret-image-test: context deadline exceeded
{
msg: "VerifyImagePulled(registry.redhat.io/ubi-minimal) timed out after 3m0.006s: failed to list pods in namespace pullsecret-image-test: context deadline exceeded",
err: <*fmt.wrapError | 0x2528dfc026e0>{
msg: "failed to list pods in namespace pullsecret-image-test: context deadline exceeded",
err: <context.deadlineExceededError>{},
},
}
occurredContributing tests (1)
Affected runs (1)
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
certificate not ready: reason=DoesNotExist message=Issuing certificate as Secret does not existfull failure pattern: certificate not ready: reason=DoesNotExist message=Issuing certificate as Secret does not exist Signal: IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | e2e | 1 | 3.70%1 of 27 job runs affected | IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | Show trend detailsOct 12: 0 · Oct 13: 0 · Oct 14: 0 · Oct 15: 0 · Oct 16: 0 · Oct 17: 0 · Oct 18: 0 · Oct 19: 0 · Oct 20: 0 · Oct 21: 0 · Oct 22: 0 · Oct 23: 1 · Oct 24: 0 · Oct 25: 0 | DEV | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:493]: Timed out after 600.001s.
Certificate should be issued by Let's Encrypt staging via ACME DNS-01
Expected success, but got an error:
<*errors.errorString | 0x1a2d0a2b0a40>:
certificate not ready: reason=DoesNotExist message=Issuing certificate as Secret does not exist
{
s: "certificate not ready: reason=DoesNotExist message=Issuing certificate as Secret does not exist",
}
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:493]: Timed out after 600.001s.
Certificate should be issued by Let's Encrypt staging via ACME DNS-01
Expected success, but got an error:
<*errors.errorString | 0x1a2d0a2b0a40>:
certificate not ready: reason=DoesNotExist message=Issuing certificate as Secret does not exist
{
s: "certificate not ready: reason=DoesNotExist message=Issuing certificate as Secret does not exist",
}Contributing tests (1)
Affected runs (1)
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Environment: PROD
Daily Run Outcomes
| Pass rate | Runs | Date (UTC) | Suite | Test |
|---|---|---|---|---|
| 76.92% | 13 | prod/parallel | Authorized CIDRs Connectivity should allow API access only from authorized VM IP | |
| 84.62% | 13 | prod/parallel | Customer should be able to create a cluster and node pools with aggregated advanced features | |
| 84.62% | 13 | prod/parallel | Customer should be able to create an HCP cluster and manage ImageDigestMirrors | |
| 84.62% | 13 | prod/parallel | Customer should be able to create an HCP cluster with custom autoscaling and update the autoscaling configuration | |
| 84.62% | 13 | prod/parallel | Update HCPOpenShiftCluster Positive creates a cluster and updates tags with a PATCH request |
Up to 50 failure patterns are loaded (minimum 1.00% impact), with 10 shown by default. Click column headers to re-sort.
Jump to the full Failure Patterns view for this window
| Failure patternThe canonical description of a recurring CI failure, extracted and normalized from raw logs. | Failed atThe stage of the job run where this failure occurred: 'provision' (environment setup, DEV only), 'e2e' (test suite execution), or 'other' (CI infrastructure issues that did not produce a failure pattern). | Number of distinct job runs where this failure pattern was detected. | Percentage of all job runs in this environment affected by this failure pattern during the selected window. | Classification of this failure pattern: Regression (likely caused by a specific PR), Flake (intermittent failure spread across days), Noise (low-quality or generic pattern), or Indeterminate. | TrendShows daily activity for this failure pattern in a trailing window anchored to the selected end date. The sparkline covers at least 7 days and at most 14 days, depending on the current window size. | Also inOther environments where the same failure pattern was also detected during the selected window. | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
ERROR CODE: InvalidRequestContent; detail message Invalid value: "4.21": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when c...full failure pattern: ERROR CODE: InvalidRequestContent; detail message Invalid value: "4.21": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is "nightly"; provider Microsoft.RedHatOpenShift Signal: FlakeSignal: Flake — present in 4 prior week(s); spread across 9 days | e2e | 9 | 9.38%9 of 96 job runs affected | FlakeSignal: Flake — present in 4 prior week(s); spread across 9 days | Show trend detailsOct 12: 4 · Oct 13: 5 · Oct 14: 5 · Oct 15: 5 · Oct 16: 5 · Oct 17: 5 · Oct 18: 5 · Oct 19: 5 · Oct 20: 5 · Oct 21: 0 · Oct 22: 0 · Oct 23: 0 · Oct 24: 0 · Oct 25: 0 | STG | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (3)fail [github.com/Azure/ARO-HCP/test/e2e/idms_lifecycle.go:175]: failed to update cluster with second ImageDigestMirror set
Unexpected error:
<*exported.ResponseError | 0x16962b1f6f00>:
PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/idms-m6l655w65gt8/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/idms-e2e-hcp-cluster
--------------------------------------------------------------------------------
RESPONSE 400: 400 Bad Request
ERROR CODE: InvalidRequestContent
--------------------------------------------------------------------------------
{
"error": {
"code": "InvalidRequestContent",
"message": "Invalid value: \"4.21\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"",
"target": "properties.version.id"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/idms-m6l655w65gt8/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/idms-e2e-hcp-cluster
correlation ID: 9ef8b68a-bcc4-4e41-9cf0-46d6272fce74
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/idms_lifecycle.go:175]: failed to update cluster with second ImageDigestMirror set
Unexpected error:
<*exported.ResponseError | 0x16962b1f6f00>:
PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/idms-m6l655w65gt8/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/idms-e2e-hcp-cluster
--------------------------------------------------------------------------------
RESPONSE 400: 400 Bad Request
ERROR CODE: InvalidRequestContent
--------------------------------------------------------------------------------
{
"error": {
"code": "InvalidRequestContent",
"message": "Invalid value: \"4.21\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"",
"target": "properties.version.id"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/idms-m6l655w65gt8/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/idms-e2e-hcp-cluster
correlation ID: 9ef8b68a-bcc4-4e41-9cf0-46d6272fce74
client request ID: unavailable
occurredfail [github.com/Azure/ARO-HCP/test/e2e/cluster_autoscaling.go:173]: failed to update cluster "autoscaling-hcp-cluster" autoscaling configuration
Unexpected error:
<*exported.ResponseError | 0xaa2ec6a3830>:
PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/autoscaling-cluster-sm6hpjzmbhjv/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/autoscaling-hcp-cluster
--------------------------------------------------------------------------------
RESPONSE 400: 400 Bad Request
ERROR CODE: InvalidRequestContent
--------------------------------------------------------------------------------
{
"error": {
"code": "InvalidRequestContent",
"message": "Invalid value: \"4.21\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"",
"target": "properties.version.id"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/autoscaling-cluster-sm6hpjzmbhjv/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/autoscaling-hcp-cluster
correlation ID: 11208f71-8029-42d4-aa43-9f639823791f
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_autoscaling.go:173]: failed to update cluster "autoscaling-hcp-cluster" autoscaling configuration
Unexpected error:
<*exported.ResponseError | 0xaa2ec6a3830>:
PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/autoscaling-cluster-sm6hpjzmbhjv/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/autoscaling-hcp-cluster
--------------------------------------------------------------------------------
RESPONSE 400: 400 Bad Request
ERROR CODE: InvalidRequestContent
--------------------------------------------------------------------------------
{
"error": {
"code": "InvalidRequestContent",
"message": "Invalid value: \"4.21\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"",
"target": "properties.version.id"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/autoscaling-cluster-sm6hpjzmbhjv/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/autoscaling-hcp-cluster
correlation ID: 11208f71-8029-42d4-aa43-9f639823791f
client request ID: unavailable
occurredfail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:408]: failed to begin updating cluster "cidr-connectivity-test" authorized CIDRs
Unexpected error:
<*exported.ResponseError | 0x32eaa59cc210>:
PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/e2e-cidr-connectivity-r92b5qqc8544/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cidr-connectivity-test
--------------------------------------------------------------------------------
RESPONSE 400: 400 Bad Request
ERROR CODE: InvalidRequestContent
--------------------------------------------------------------------------------
{
"error": {
"code": "InvalidRequestContent",
"message": "Invalid value: \"4.21\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"",
"target": "properties.version.id"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/e2e-cidr-connectivity-r92b5qqc8544/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cidr-connectivity-test
correlation ID: 37d0ceb3-cb28-4457-98b9-e7ba4d280731
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:408]: failed to begin updating cluster "cidr-connectivity-test" authorized CIDRs
Unexpected error:
<*exported.ResponseError | 0x32eaa59cc210>:
PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/e2e-cidr-connectivity-r92b5qqc8544/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cidr-connectivity-test
--------------------------------------------------------------------------------
RESPONSE 400: 400 Bad Request
ERROR CODE: InvalidRequestContent
--------------------------------------------------------------------------------
{
"error": {
"code": "InvalidRequestContent",
"message": "Invalid value: \"4.21\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"",
"target": "properties.version.id"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/e2e-cidr-connectivity-r92b5qqc8544/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cidr-connectivity-test
correlation ID: 37d0ceb3-cb28-4457-98b9-e7ba4d280731
client request ID: unavailable
occurredContributing tests (5)
Affected runs (9)
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedC...full failure pattern: ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider]; provider Microsoft.RedHatOpenShift Signal: FlakeSignal: Flake — present in 2 prior week(s); spread across 4 days | e2e | 4 | 4.17%4 of 96 job runs affected | FlakeSignal: Flake — present in 2 prior week(s); spread across 4 days | Show trend detailsOct 12: 1 · Oct 13: 0 · Oct 14: 0 · Oct 15: 1 · Oct 16: 0 · Oct 17: 0 · Oct 18: 0 · Oct 19: 0 · Oct 20: 9 · Oct 21: 0 · Oct 22: 10 · Oct 23: 0 · Oct 24: 0 · Oct 25: 0 | DEV, INT, STG | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (3)fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_nodepool_osdisk.go:80]: failed to create HCP cluster "hcp-cluster-np-128"
Unexpected error:
<*fmt.wrapError | 0xd879a7a0b60>:
failed to create HCP cluster hcp-cluster-np-128: failed waiting for cluster="hcp-cluster-np-128" in resourcegroup="clusternp128-7ht4n2l8tbfm" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/94676dae-e02f-4af7-9503-8526a46b136a
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/94676dae-e02f-4af7-9503-8526a46b136a",
"name": "94676dae-e02f-4af7-9503-8526a46b136a",
"status": "Failed",
"startTime": "2026-10-22T01:33:54.675375561Z",
"endTime": "2026-10-22T02:02:56.694987866Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/94676dae-e02f-4af7-9503-8526a46b136a
correlation ID: eb262c16-35c1-4306-89d7-1ab7a4a6f8e5
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_nodepool_osdisk.go:80]: failed to create HCP cluster "hcp-cluster-np-128"
Unexpected error:
<*fmt.wrapError | 0xd879a7a0b60>:
failed to create HCP cluster hcp-cluster-np-128: failed waiting for cluster="hcp-cluster-np-128" in resourcegroup="clusternp128-7ht4n2l8tbfm" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/94676dae-e02f-4af7-9503-8526a46b136a
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/94676dae-e02f-4af7-9503-8526a46b136a",
"name": "94676dae-e02f-4af7-9503-8526a46b136a",
"status": "Failed",
"startTime": "2026-10-22T01:33:54.675375561Z",
"endTime": "2026-10-22T02:02:56.694987866Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/94676dae-e02f-4af7-9503-8526a46b136a
correlation ID: eb262c16-35c1-4306-89d7-1ab7a4a6f8e5
client request ID: unavailable
occurredfail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:443]: failed to create HCP cluster np-skip-minor-fvrddf at version 4.22.19
Unexpected error:
<*fmt.wrapError | 0x3b64a153a380>:
failed to create HCP cluster np-skip-minor-fvrddf: failed waiting for cluster="np-skip-minor-fvrddf" in resourcegroup="rg-np-skip-minor-fvrddf-fpw64csf8rm6" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/fcd2c383-b881-499c-8bed-cb24dce8c319
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/fcd2c383-b881-499c-8bed-cb24dce8c319",
"name": "fcd2c383-b881-499c-8bed-cb24dce8c319",
"status": "Failed",
"startTime": "2026-10-22T03:10:54.281108118Z",
"endTime": "2026-10-22T03:39:55.15624633Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/fcd2c383-b881-499c-8bed-cb24dce8c319
correlation ID: e6ae3233-540c-4141-81e8-0b055bb45703
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:443]: failed to create HCP cluster np-skip-minor-fvrddf at version 4.22.19
Unexpected error:
<*fmt.wrapError | 0x3b64a153a380>:
failed to create HCP cluster np-skip-minor-fvrddf: failed waiting for cluster="np-skip-minor-fvrddf" in resourcegroup="rg-np-skip-minor-fvrddf-fpw64csf8rm6" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/fcd2c383-b881-499c-8bed-cb24dce8c319
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/fcd2c383-b881-499c-8bed-cb24dce8c319",
"name": "fcd2c383-b881-499c-8bed-cb24dce8c319",
"status": "Failed",
"startTime": "2026-10-22T03:10:54.281108118Z",
"endTime": "2026-10-22T03:39:55.15624633Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/fcd2c383-b881-499c-8bed-cb24dce8c319
correlation ID: e6ae3233-540c-4141-81e8-0b055bb45703
client request ID: unavailable
occurredfail [github.com/Azure/ARO-HCP/test/e2e/external_auth_list_and_verify.go:87]: failed to create HCP cluster for external auth list test
Unexpected error:
<*fmt.wrapError | 0xbec85184960>:
failed to create HCP cluster ea-listr985hw: failed waiting for cluster="ea-listr985hw" in resourcegroup="ea-list-psnhlv4lhlmk" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/d538c387-3fea-488a-a164-ab083059af27
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/d538c387-3fea-488a-a164-ab083059af27",
"name": "d538c387-3fea-488a-a164-ab083059af27",
"status": "Failed",
"startTime": "2026-10-22T02:25:04.906647307Z",
"endTime": "2026-10-22T02:54:10.540852911Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/d538c387-3fea-488a-a164-ab083059af27
correlation ID: 924be768-33e9-47c5-9219-bd9273d15ea1
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_list_and_verify.go:87]: failed to create HCP cluster for external auth list test
Unexpected error:
<*fmt.wrapError | 0xbec85184960>:
failed to create HCP cluster ea-listr985hw: failed waiting for cluster="ea-listr985hw" in resourcegroup="ea-list-psnhlv4lhlmk" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/d538c387-3fea-488a-a164-ab083059af27
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/d538c387-3fea-488a-a164-ab083059af27",
"name": "d538c387-3fea-488a-a164-ab083059af27",
"status": "Failed",
"startTime": "2026-10-22T02:25:04.906647307Z",
"endTime": "2026-10-22T02:54:10.540852911Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/d538c387-3fea-488a-a164-ab083059af27
correlation ID: 924be768-33e9-47c5-9219-bd9273d15ea1
client request ID: unavailable
occurredContributing tests (18)
Affected runs (4)
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
VerifyBasicAccess failed: failed to list services: Unauthorizedfull failure pattern: VerifyBasicAccess failed: failed to list services: Unauthorized Signal: FlakeSignal: Flake — present in 2 prior week(s); spread across 4 days | e2e | 4 | 4.17%4 of 96 job runs affected | FlakeSignal: Flake — present in 2 prior week(s); spread across 4 days | Show trend detailsOct 12: 0 · Oct 13: 0 · Oct 14: 0 · Oct 15: 0 · Oct 16: 0 · Oct 17: 0 · Oct 18: 0 · Oct 19: 2 · Oct 20: 0 · Oct 21: 0 · Oct 22: 0 · Oct 23: 1 · Oct 24: 2 · Oct 25: 1 | none | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (3)fail [github.com/Azure/ARO-HCP/test/e2e/oidc_issuer_workload_identity.go:254]: cluster oidc-wi-cluster is not viable
Unexpected error:
<*errors.joinError | 0x5c10dd80030>:
VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Unauthorized
VerifyBasicAccess failed: failed to list services: Unauthorized
{
errs: [
<*fmt.wrapError | 0x5c10d35ddc0>{
msg: "VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Unauthorized",
err: <*fmt.wrapError | 0x5c10d35dda0>{
msg: "failed to list all APIServices: Unauthorized",
err: <*errors.StatusError | 0x5c10db35540>{
ErrStatus: {
TypeMeta: {Kind: "", APIVersion: ""},
ListMeta: {
SelfLink: "",
ResourceVersion: "",
Continue: "",
RemainingItemCount: nil,
ShardInfo: nil,
},
Status: "Failure",
Message: "Unauthorized",
Reason: "Unauthorized",
Details: nil,
Code: 401,
},
},
},
},
<*fmt.wrapError | 0x5c10dd88020>{
msg: "VerifyBasicAccess failed: failed to list services: Unauthorized",
err: <*fmt.wrapError | 0x5c10dd88000>{
msg: "failed to list services: Unauthorized",
err: <*errors.StatusError | 0x5c10dbae320>{
ErrStatus: {
TypeMeta: {Kind: "", APIVersion: ""},
ListMeta: {
SelfLink: "",
ResourceVersion: "",
Continue: "",
RemainingItemCount: nil,
ShardInfo: nil,
},
Status: "Failure",
Message: "Unauthorized",
Reason: "Unauthorized",
Details: nil,
Code: 401,
},
},
},
},
],
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/oidc_issuer_workload_identity.go:254]: cluster oidc-wi-cluster is not viable
Unexpected error:
<*errors.joinError | 0x5c10dd80030>:
VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Unauthorized
VerifyBasicAccess failed: failed to list services: Unauthorized
{
errs: [
<*fmt.wrapError | 0x5c10d35ddc0>{
msg: "VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Unauthorized",
err: <*fmt.wrapError | 0x5c10d35dda0>{
msg: "failed to list all APIServices: Unauthorized",
err: <*errors.StatusError | 0x5c10db35540>{
ErrStatus: {
TypeMeta: {Kind: "", APIVersion: ""},
ListMeta: {
SelfLink: "",
ResourceVersion: "",
Continue: "",
RemainingItemCount: nil,
ShardInfo: nil,
},
Status: "Failure",
Message: "Unauthorized",
Reason: "Unauthorized",
Details: nil,
Code: 401,
},
},
},
},
<*fmt.wrapError | 0x5c10dd88020>{
msg: "VerifyBasicAccess failed: failed to list services: Unauthorized",
err: <*fmt.wrapError | 0x5c10dd88000>{
msg: "failed to list services: Unauthorized",
err: <*errors.StatusError | 0x5c10dbae320>{
ErrStatus: {
TypeMeta: {Kind: "", APIVersion: ""},
ListMeta: {
SelfLink: "",
ResourceVersion: "",
Continue: "",
RemainingItemCount: nil,
ShardInfo: nil,
},
Status: "Failure",
Message: "Unauthorized",
Reason: "Unauthorized",
Details: nil,
Code: 401,
},
},
},
},
],
}
occurredfail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:140]: failed to verify HCP cluster "cluster-nightly-4-22-67pgbb" is viable
Unexpected error:
<*errors.joinError | 0x21cecb2413f8>:
VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Unauthorized
VerifyBasicAccess failed: failed to list services: Unauthorized
{
errs: [
<*fmt.wrapError | 0x21cecb85e9a0>{
msg: "VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Unauthorized",
err: <*fmt.wrapError | 0x21cecb85e980>{
msg: "failed to list all APIServices: Unauthorized",
err: <*errors.StatusError | 0x21cecae4e500>{
ErrStatus: {
TypeMeta: {Kind: "", APIVersion: ""},
ListMeta: {
SelfLink: "",
ResourceVersion: "",
Continue: "",
RemainingItemCount: nil,
ShardInfo: nil,
},
Status: "Failure",
Message: "Unauthorized",
Reason: "Unauthorized",
Details: nil,
Code: 401,
},
},
},
},
<*fmt.wrapError | 0x21cecb471000>{
msg: "VerifyBasicAccess failed: failed to list services: Unauthorized",
err: <*fmt.wrapError | 0x21cecb470fe0>{
msg: "failed to list services: Unauthorized",
err: <*errors.StatusError | 0x21cecb1806e0>{
ErrStatus: {
TypeMeta: {Kind: "", APIVersion: ""},
ListMeta: {
SelfLink: "",
ResourceVersion: "",
Continue: "",
RemainingItemCount: nil,
ShardInfo: nil,
},
Status: "Failure",
Message: "Unauthorized",
Reason: "Unauthorized",
Details: nil,
Code: 401,
},
},
},
},
],
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:140]: failed to verify HCP cluster "cluster-nightly-4-22-67pgbb" is viable
Unexpected error:
<*errors.joinError | 0x21cecb2413f8>:
VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Unauthorized
VerifyBasicAccess failed: failed to list services: Unauthorized
{
errs: [
<*fmt.wrapError | 0x21cecb85e9a0>{
msg: "VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Unauthorized",
err: <*fmt.wrapError | 0x21cecb85e980>{
msg: "failed to list all APIServices: Unauthorized",
err: <*errors.StatusError | 0x21cecae4e500>{
ErrStatus: {
TypeMeta: {Kind: "", APIVersion: ""},
ListMeta: {
SelfLink: "",
ResourceVersion: "",
Continue: "",
RemainingItemCount: nil,
ShardInfo: nil,
},
Status: "Failure",
Message: "Unauthorized",
Reason: "Unauthorized",
Details: nil,
Code: 401,
},
},
},
},
<*fmt.wrapError | 0x21cecb471000>{
msg: "VerifyBasicAccess failed: failed to list services: Unauthorized",
err: <*fmt.wrapError | 0x21cecb470fe0>{
msg: "failed to list services: Unauthorized",
err: <*errors.StatusError | 0x21cecb1806e0>{
ErrStatus: {
TypeMeta: {Kind: "", APIVersion: ""},
ListMeta: {
SelfLink: "",
ResourceVersion: "",
Continue: "",
RemainingItemCount: nil,
ShardInfo: nil,
},
Status: "Failure",
Message: "Unauthorized",
Reason: "Unauthorized",
Details: nil,
Code: 401,
},
},
},
},
],
}
occurredfail [github.com/Azure/ARO-HCP/test/e2e/simple_negative_cases.go:124]: cluster negative-tests-cluster is not viable
Unexpected error:
<*errors.joinError | 0x13bc429c0870>:
VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Unauthorized
VerifyBasicAccess failed: failed to list services: Unauthorized
{
errs: [
<*fmt.wrapError | 0x13bc42aa9920>{
msg: "VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Unauthorized",
err: <*fmt.wrapError | 0x13bc42aa9900>{
msg: "failed to list all APIServices: Unauthorized",
err: <*errors.StatusError | 0x13bc41a5d400>{
ErrStatus: {
TypeMeta: {Kind: "", APIVersion: ""},
ListMeta: {
SelfLink: "",
ResourceVersion: "",
Continue: "",
RemainingItemCount: nil,
ShardInfo: nil,
},
Status: "Failure",
Message: "Unauthorized",
Reason: "Unauthorized",
Details: nil,
Code: 401,
},
},
},
},
<*fmt.wrapError | 0x13bc425c16c0>{
msg: "VerifyBasicAccess failed: failed to list services: Unauthorized",
err: <*fmt.wrapError | 0x13bc425c16a0>{
msg: "failed to list services: Unauthorized",
err: <*errors.StatusError | 0x13bc416d8960>{
ErrStatus: {
TypeMeta: {Kind: "", APIVersion: ""},
ListMeta: {
SelfLink: "",
ResourceVersion: "",
Continue: "",
RemainingItemCount: nil,
ShardInfo: nil,
},
Status: "Failure",
Message: "Unauthorized",
Reason: "Unauthorized",
Details: nil,
Code: 401,
},
},
},
},
],
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/simple_negative_cases.go:124]: cluster negative-tests-cluster is not viable
Unexpected error:
<*errors.joinError | 0x13bc429c0870>:
VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Unauthorized
VerifyBasicAccess failed: failed to list services: Unauthorized
{
errs: [
<*fmt.wrapError | 0x13bc42aa9920>{
msg: "VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Unauthorized",
err: <*fmt.wrapError | 0x13bc42aa9900>{
msg: "failed to list all APIServices: Unauthorized",
err: <*errors.StatusError | 0x13bc41a5d400>{
ErrStatus: {
TypeMeta: {Kind: "", APIVersion: ""},
ListMeta: {
SelfLink: "",
ResourceVersion: "",
Continue: "",
RemainingItemCount: nil,
ShardInfo: nil,
},
Status: "Failure",
Message: "Unauthorized",
Reason: "Unauthorized",
Details: nil,
Code: 401,
},
},
},
},
<*fmt.wrapError | 0x13bc425c16c0>{
msg: "VerifyBasicAccess failed: failed to list services: Unauthorized",
err: <*fmt.wrapError | 0x13bc425c16a0>{
msg: "failed to list services: Unauthorized",
err: <*errors.StatusError | 0x13bc416d8960>{
ErrStatus: {
TypeMeta: {Kind: "", APIVersion: ""},
ListMeta: {
SelfLink: "",
ResourceVersion: "",
Continue: "",
RemainingItemCount: nil,
ShardInfo: nil,
},
Status: "Failure",
Message: "Unauthorized",
Reason: "Unauthorized",
Details: nil,
Code: 401,
},
},
},
},
],
}
occurredContributing tests (4)
Affected runs (4)
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Unauthorizedfull failure pattern: VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Unauthorized Signal: FlakeSignal: Flake — present in 3 prior week(s); spread across 3 days | e2e | 3 | 3.12%3 of 96 job runs affected | FlakeSignal: Flake — present in 3 prior week(s); spread across 3 days | Show trend detailsOct 12: 1 · Oct 13: 0 · Oct 14: 1 · Oct 15: 0 · Oct 16: 0 · Oct 17: 0 · Oct 18: 0 · Oct 19: 0 · Oct 20: 0 · Oct 21: 0 · Oct 22: 0 · Oct 23: 1 · Oct 24: 0 · Oct 25: 0 | STG | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (3)fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:140]: failed to verify HCP cluster "cluster-nightly-4-21-hwnzpg" is viable
Unexpected error:
<*errors.joinError | 0x2be9a2dba918>:
VerifyBasicAccess failed: failed to list services: Unauthorized
VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Unauthorized
{
errs: [
<*fmt.wrapError | 0x2be9a2235d00>{
msg: "VerifyBasicAccess failed: failed to list services: Unauthorized",
err: <*fmt.wrapError | 0x2be9a2235cc0>{
msg: "failed to list services: Unauthorized",
err: <*errors.StatusError | 0x2be9a29e6c80>{
ErrStatus: {
TypeMeta: {Kind: "", APIVersion: ""},
ListMeta: {
SelfLink: "",
ResourceVersion: "",
Continue: "",
RemainingItemCount: nil,
ShardInfo: nil,
},
Status: "Failure",
Message: "Unauthorized",
Reason: "Unauthorized",
Details: nil,
Code: 401,
},
},
},
},
<*fmt.wrapError | 0x2be9a21cc080>{
msg: "VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Unauthorized",
err: <*fmt.wrapError | 0x2be9a21cc020>{
msg: "failed to list all APIServices: Unauthorized",
err: <*errors.StatusError | 0x2be9a3120140>{
ErrStatus: {
TypeMeta: {Kind: "", APIVersion: ""},
ListMeta: {
SelfLink: "",
ResourceVersion: "",
Continue: "",
RemainingItemCount: nil,
ShardInfo: nil,
},
Status: "Failure",
Message: "Unauthorized",
Reason: "Unauthorized",
Details: nil,
Code: 401,
},
},
},
},
],
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:140]: failed to verify HCP cluster "cluster-nightly-4-21-hwnzpg" is viable
Unexpected error:
<*errors.joinError | 0x2be9a2dba918>:
VerifyBasicAccess failed: failed to list services: Unauthorized
VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Unauthorized
{
errs: [
<*fmt.wrapError | 0x2be9a2235d00>{
msg: "VerifyBasicAccess failed: failed to list services: Unauthorized",
err: <*fmt.wrapError | 0x2be9a2235cc0>{
msg: "failed to list services: Unauthorized",
err: <*errors.StatusError | 0x2be9a29e6c80>{
ErrStatus: {
TypeMeta: {Kind: "", APIVersion: ""},
ListMeta: {
SelfLink: "",
ResourceVersion: "",
Continue: "",
RemainingItemCount: nil,
ShardInfo: nil,
},
Status: "Failure",
Message: "Unauthorized",
Reason: "Unauthorized",
Details: nil,
Code: 401,
},
},
},
},
<*fmt.wrapError | 0x2be9a21cc080>{
msg: "VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Unauthorized",
err: <*fmt.wrapError | 0x2be9a21cc020>{
msg: "failed to list all APIServices: Unauthorized",
err: <*errors.StatusError | 0x2be9a3120140>{
ErrStatus: {
TypeMeta: {Kind: "", APIVersion: ""},
ListMeta: {
SelfLink: "",
ResourceVersion: "",
Continue: "",
RemainingItemCount: nil,
ShardInfo: nil,
},
Status: "Failure",
Message: "Unauthorized",
Reason: "Unauthorized",
Details: nil,
Code: 401,
},
},
},
},
],
}
occurredfail [github.com/Azure/ARO-HCP/test/e2e/cluster_delete_cx_rg.go:106]: failed to verify HCP cluster "cx-rg-hcp-cluster" is viable
Unexpected error:
<*errors.joinError | 0x9c519e26018>:
VerifyBasicAccess failed: failed to list services: Unauthorized
VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Unauthorized
{
errs: [
<*fmt.wrapError | 0x9c519869940>{
msg: "VerifyBasicAccess failed: failed to list services: Unauthorized",
err: <*fmt.wrapError | 0x9c519869920>{
msg: "failed to list services: Unauthorized",
err: <*errors.StatusError | 0x9c5196cebe0>{
ErrStatus: {
TypeMeta: {Kind: "", APIVersion: ""},
ListMeta: {
SelfLink: "",
ResourceVersion: "",
Continue: "",
RemainingItemCount: nil,
ShardInfo: nil,
},
Status: "Failure",
Message: "Unauthorized",
Reason: "Unauthorized",
Details: nil,
Code: 401,
},
},
},
},
<*fmt.wrapError | 0x9c51a1232c0>{
msg: "VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Unauthorized",
err: <*fmt.wrapError | 0x9c51a1232a0>{
msg: "failed to list all APIServices: Unauthorized",
err: <*errors.StatusError | 0x9c519493900>{
ErrStatus: {
TypeMeta: {Kind: "", APIVersion: ""},
ListMeta: {
SelfLink: "",
ResourceVersion: "",
Continue: "",
RemainingItemCount: nil,
ShardInfo: nil,
},
Status: "Failure",
Message: "Unauthorized",
Reason: "Unauthorized",
Details: nil,
Code: 401,
},
},
},
},
],
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delete_cx_rg.go:106]: failed to verify HCP cluster "cx-rg-hcp-cluster" is viable
Unexpected error:
<*errors.joinError | 0x9c519e26018>:
VerifyBasicAccess failed: failed to list services: Unauthorized
VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Unauthorized
{
errs: [
<*fmt.wrapError | 0x9c519869940>{
msg: "VerifyBasicAccess failed: failed to list services: Unauthorized",
err: <*fmt.wrapError | 0x9c519869920>{
msg: "failed to list services: Unauthorized",
err: <*errors.StatusError | 0x9c5196cebe0>{
ErrStatus: {
TypeMeta: {Kind: "", APIVersion: ""},
ListMeta: {
SelfLink: "",
ResourceVersion: "",
Continue: "",
RemainingItemCount: nil,
ShardInfo: nil,
},
Status: "Failure",
Message: "Unauthorized",
Reason: "Unauthorized",
Details: nil,
Code: 401,
},
},
},
},
<*fmt.wrapError | 0x9c51a1232c0>{
msg: "VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Unauthorized",
err: <*fmt.wrapError | 0x9c51a1232a0>{
msg: "failed to list all APIServices: Unauthorized",
err: <*errors.StatusError | 0x9c519493900>{
ErrStatus: {
TypeMeta: {Kind: "", APIVersion: ""},
ListMeta: {
SelfLink: "",
ResourceVersion: "",
Continue: "",
RemainingItemCount: nil,
ShardInfo: nil,
},
Status: "Failure",
Message: "Unauthorized",
Reason: "Unauthorized",
Details: nil,
Code: 401,
},
},
},
},
],
}
occurredfail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:175]: failed to verify HCP cluster viability for patch-tags test
Unexpected error:
<*errors.joinError | 0x2d289f100030>:
VerifyBasicAccess failed: failed to list services: Unauthorized
VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Unauthorized
{
errs: [
<*fmt.wrapError | 0x2d289e7b3500>{
msg: "VerifyBasicAccess failed: failed to list services: Unauthorized",
err: <*fmt.wrapError | 0x2d289e7b34e0>{
msg: "failed to list services: Unauthorized",
err: <*errors.StatusError | 0x2d289f0dae60>{
ErrStatus: {
TypeMeta: {Kind: "", APIVersion: ""},
ListMeta: {
SelfLink: "",
ResourceVersion: "",
Continue: "",
RemainingItemCount: nil,
ShardInfo: nil,
},
Status: "Failure",
Message: "Unauthorized",
Reason: "Unauthorized",
Details: nil,
Code: 401,
},
},
},
},
<*fmt.wrapError | 0x2d289e6a6040>{
msg: "VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Unauthorized",
err: <*fmt.wrapError | 0x2d289e6a6020>{
msg: "failed to list all APIServices: Unauthorized",
err: <*errors.StatusError | 0x2d289e168dc0>{
ErrStatus: {
TypeMeta: {Kind: "", APIVersion: ""},
ListMeta: {
SelfLink: "",
ResourceVersion: "",
Continue: "",
RemainingItemCount: nil,
ShardInfo: nil,
},
Status: "Failure",
Message: "Unauthorized",
Reason: "Unauthorized",
Details: nil,
Code: 401,
},
},
},
},
],
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:175]: failed to verify HCP cluster viability for patch-tags test
Unexpected error:
<*errors.joinError | 0x2d289f100030>:
VerifyBasicAccess failed: failed to list services: Unauthorized
VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Unauthorized
{
errs: [
<*fmt.wrapError | 0x2d289e7b3500>{
msg: "VerifyBasicAccess failed: failed to list services: Unauthorized",
err: <*fmt.wrapError | 0x2d289e7b34e0>{
msg: "failed to list services: Unauthorized",
err: <*errors.StatusError | 0x2d289f0dae60>{
ErrStatus: {
TypeMeta: {Kind: "", APIVersion: ""},
ListMeta: {
SelfLink: "",
ResourceVersion: "",
Continue: "",
RemainingItemCount: nil,
ShardInfo: nil,
},
Status: "Failure",
Message: "Unauthorized",
Reason: "Unauthorized",
Details: nil,
Code: 401,
},
},
},
},
<*fmt.wrapError | 0x2d289e6a6040>{
msg: "VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Unauthorized",
err: <*fmt.wrapError | 0x2d289e6a6020>{
msg: "failed to list all APIServices: Unauthorized",
err: <*errors.StatusError | 0x2d289e168dc0>{
ErrStatus: {
TypeMeta: {Kind: "", APIVersion: ""},
ListMeta: {
SelfLink: "",
ResourceVersion: "",
Continue: "",
RemainingItemCount: nil,
ShardInfo: nil,
},
Status: "Failure",
Message: "Unauthorized",
Reason: "Unauthorized",
Details: nil,
Code: 401,
},
},
},
},
],
}
occurredContributing tests (3)
Affected runs (3)
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
VerifyDefaultIngressServingCertificateSecretWiring timed out after 10m0s: IngressController/default references &LocalObjectReference{Name:cluster-ingress-cert,}, expected secret "...full failure pattern: VerifyDefaultIngressServingCertificateSecretWiring timed out after 10m0s: IngressController/default references &LocalObjectReference{Name:cluster-ingress-cert,}, expected secret "default-ingress-cert"Signal: NoiseSignal: Noise — struct/object fragment — New failure pattern — no prior history | e2e | 1 | 1.04%1 of 96 job runs affected | NoiseSignal: Noise — struct/object fragment — New failure pattern — no prior history | Show trend detailsOct 12: 0 · Oct 13: 0 · Oct 14: 0 · Oct 15: 0 · Oct 16: 0 · Oct 17: 0 · Oct 18: 0 · Oct 19: 0 · Oct 20: 0 · Oct 21: 0 · Oct 22: 0 · Oct 23: 1 · Oct 24: 0 · Oct 25: 0 | DEV, STG | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:220]: default-ingress-cert must exist and be referenced by IngressController/default for cluster "cluster-nightly-4-23-v65qkr"
Expected success, but got an error:
<*fmt.wrapError | 0xbd03a06ce0>:
VerifyDefaultIngressServingCertificateSecretWiring timed out after 10m0s: IngressController/default references &LocalObjectReference{Name:cluster-ingress-cert,}, expected secret "default-ingress-cert"
{
msg: "VerifyDefaultIngressServingCertificateSecretWiring timed out after 10m0s: IngressController/default references &LocalObjectReference{Name:cluster-ingress-cert,}, expected secret \"default-ingress-cert\"",
err: <*verifiers.pollObservationError | 0xbd03a06ca0>{
observation: <*errors.errorString | 0xbd05b0abd0>{
s: "IngressController/default references &LocalObjectReference{Name:cluster-ingress-cert,}, expected secret \"default-ingress-cert\"",
},
interruption: <*fmt.wrapError | 0xbd03a06c60>{
msg: "getting default ingress serving certificate secret openshift-ingress/default-ingress-cert: context deadline exceeded",
err: <context.deadlineExceededError>{},
},
},
}
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:220]: default-ingress-cert must exist and be referenced by IngressController/default for cluster "cluster-nightly-4-23-v65qkr"
Expected success, but got an error:
<*fmt.wrapError | 0xbd03a06ce0>:
VerifyDefaultIngressServingCertificateSecretWiring timed out after 10m0s: IngressController/default references &LocalObjectReference{Name:cluster-ingress-cert,}, expected secret "default-ingress-cert"
{
msg: "VerifyDefaultIngressServingCertificateSecretWiring timed out after 10m0s: IngressController/default references &LocalObjectReference{Name:cluster-ingress-cert,}, expected secret \"default-ingress-cert\"",
err: <*verifiers.pollObservationError | 0xbd03a06ca0>{
observation: <*errors.errorString | 0xbd05b0abd0>{
s: "IngressController/default references &LocalObjectReference{Name:cluster-ingress-cert,}, expected secret \"default-ingress-cert\"",
},
interruption: <*fmt.wrapError | 0xbd03a06c60>{
msg: "getting default ingress serving certificate secret openshift-ingress/default-ingress-cert: context deadline exceeded",
err: <context.deadlineExceededError>{},
},
},
}Contributing tests (1)
Affected runs (1)
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
ACM ingress secret openshift-ingress/cluster-ingress-cert exists; expected ACM certificate delivery to be suppressedfull failure pattern: ACM ingress secret openshift-ingress/cluster-ingress-cert exists; expected ACM certificate delivery to be suppressed Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 7 day(s) | e2e | 10 | 10.42%10 of 96 job runs affected | IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 7 day(s) | Show trend detailsOct 12: 0 · Oct 13: 0 · Oct 14: 0 · Oct 15: 0 · Oct 16: 1 · Oct 17: 2 · Oct 18: 2 · Oct 19: 1 · Oct 20: 0 · Oct 21: 0 · Oct 22: 5 · Oct 23: 10 · Oct 24: 5 · Oct 25: 0 | DEV, STG | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (3)fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:511]: ACM must not create its ingress certificate secret
Expected success, but got an error:
<*errors.errorString | 0x9f6ae185170>:
ACM ingress secret openshift-ingress/cluster-ingress-cert exists; expected ACM certificate delivery to be suppressed
{
s: "ACM ingress secret openshift-ingress/cluster-ingress-cert exists; expected ACM certificate delivery to be suppressed",
}
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:511]: ACM must not create its ingress certificate secret
Expected success, but got an error:
<*errors.errorString | 0x9f6ae185170>:
ACM ingress secret openshift-ingress/cluster-ingress-cert exists; expected ACM certificate delivery to be suppressed
{
s: "ACM ingress secret openshift-ingress/cluster-ingress-cert exists; expected ACM certificate delivery to be suppressed",
}fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:511]: ACM must not create its ingress certificate secret
Expected success, but got an error:
<*errors.errorString | 0x1e14e7538290>:
ACM ingress secret openshift-ingress/cluster-ingress-cert exists; expected ACM certificate delivery to be suppressed
{
s: "ACM ingress secret openshift-ingress/cluster-ingress-cert exists; expected ACM certificate delivery to be suppressed",
}
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:511]: ACM must not create its ingress certificate secret
Expected success, but got an error:
<*errors.errorString | 0x1e14e7538290>:
ACM ingress secret openshift-ingress/cluster-ingress-cert exists; expected ACM certificate delivery to be suppressed
{
s: "ACM ingress secret openshift-ingress/cluster-ingress-cert exists; expected ACM certificate delivery to be suppressed",
}fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:511]: ACM must not create its ingress certificate secret
Expected success, but got an error:
<*errors.errorString | 0x3184654d9920>:
ACM ingress secret openshift-ingress/cluster-ingress-cert exists; expected ACM certificate delivery to be suppressed
{
s: "ACM ingress secret openshift-ingress/cluster-ingress-cert exists; expected ACM certificate delivery to be suppressed",
}
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:511]: ACM must not create its ingress certificate secret
Expected success, but got an error:
<*errors.errorString | 0x3184654d9920>:
ACM ingress secret openshift-ingress/cluster-ingress-cert exists; expected ACM certificate delivery to be suppressed
{
s: "ACM ingress secret openshift-ingress/cluster-ingress-cert exists; expected ACM certificate delivery to be suppressed",
}Contributing tests (12)
Affected runs (10)
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
VerifyHyperShiftIngressCertificate timed out after 10m0full failure pattern: VerifyHyperShiftIngressCertificate timed out after 10m0 Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 4 day(s) | e2e | 6 | 6.25%6 of 96 job runs affected | IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 4 day(s) | Show trend detailsOct 12: 0 · Oct 13: 0 · Oct 14: 0 · Oct 15: 0 · Oct 16: 1 · Oct 17: 2 · Oct 18: 2 · Oct 19: 1 · Oct 20: 0 · Oct 21: 0 · Oct 22: 0 · Oct 23: 0 · Oct 24: 0 · Oct 25: 0 | DEV, STG | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (3)fail [github.com/Azure/ARO-HCP/test/e2e/default_cluster_ingress_cert.go:165]: HyperShift's ingress certificate secret must exist and be referenced by IngressController/default
Expected success, but got an error:
<*fmt.wrapError | 0x2064b1710780>:
VerifyHyperShiftIngressCertificate timed out after 10m0.009s: getting HyperShift ingress secret openshift-ingress/default-ingress-cert: client rate limiter Wait returned an error: context deadline exceeded
{
msg: "VerifyHyperShiftIngressCertificate timed out after 10m0.009s: getting HyperShift ingress secret openshift-ingress/default-ingress-cert: client rate limiter Wait returned an error: context deadline exceeded",
err: <*fmt.wrapError | 0x2064b1710700>{
msg: "getting HyperShift ingress secret openshift-ingress/default-ingress-cert: client rate limiter Wait returned an error: context deadline exceeded",
err: <*fmt.wrapError | 0x2064b17106a0>{
msg: "client rate limiter Wait returned an error: context deadline exceeded",
err: <context.deadlineExceededError>{},
},
},
}
fail [github.com/Azure/ARO-HCP/test/e2e/default_cluster_ingress_cert.go:165]: HyperShift's ingress certificate secret must exist and be referenced by IngressController/default
Expected success, but got an error:
<*fmt.wrapError | 0x2064b1710780>:
VerifyHyperShiftIngressCertificate timed out after 10m0.009s: getting HyperShift ingress secret openshift-ingress/default-ingress-cert: client rate limiter Wait returned an error: context deadline exceeded
{
msg: "VerifyHyperShiftIngressCertificate timed out after 10m0.009s: getting HyperShift ingress secret openshift-ingress/default-ingress-cert: client rate limiter Wait returned an error: context deadline exceeded",
err: <*fmt.wrapError | 0x2064b1710700>{
msg: "getting HyperShift ingress secret openshift-ingress/default-ingress-cert: client rate limiter Wait returned an error: context deadline exceeded",
err: <*fmt.wrapError | 0x2064b17106a0>{
msg: "client rate limiter Wait returned an error: context deadline exceeded",
err: <context.deadlineExceededError>{},
},
},
}fail [github.com/Azure/ARO-HCP/test/e2e/default_cluster_ingress_cert.go:165]: HyperShift's ingress certificate secret must exist and be referenced by IngressController/default
Expected success, but got an error:
<*fmt.wrapError | 0xa8f3937e880>:
VerifyHyperShiftIngressCertificate timed out after 10m0.006s: getting HyperShift ingress secret openshift-ingress/default-ingress-cert: client rate limiter Wait returned an error: context deadline exceeded
{
msg: "VerifyHyperShiftIngressCertificate timed out after 10m0.006s: getting HyperShift ingress secret openshift-ingress/default-ingress-cert: client rate limiter Wait returned an error: context deadline exceeded",
err: <*fmt.wrapError | 0xa8f3937e840>{
msg: "getting HyperShift ingress secret openshift-ingress/default-ingress-cert: client rate limiter Wait returned an error: context deadline exceeded",
err: <*fmt.wrapError | 0xa8f3937e7c0>{
msg: "client rate limiter Wait returned an error: context deadline exceeded",
err: <context.deadlineExceededError>{},
},
},
}
fail [github.com/Azure/ARO-HCP/test/e2e/default_cluster_ingress_cert.go:165]: HyperShift's ingress certificate secret must exist and be referenced by IngressController/default
Expected success, but got an error:
<*fmt.wrapError | 0xa8f3937e880>:
VerifyHyperShiftIngressCertificate timed out after 10m0.006s: getting HyperShift ingress secret openshift-ingress/default-ingress-cert: client rate limiter Wait returned an error: context deadline exceeded
{
msg: "VerifyHyperShiftIngressCertificate timed out after 10m0.006s: getting HyperShift ingress secret openshift-ingress/default-ingress-cert: client rate limiter Wait returned an error: context deadline exceeded",
err: <*fmt.wrapError | 0xa8f3937e840>{
msg: "getting HyperShift ingress secret openshift-ingress/default-ingress-cert: client rate limiter Wait returned an error: context deadline exceeded",
err: <*fmt.wrapError | 0xa8f3937e7c0>{
msg: "client rate limiter Wait returned an error: context deadline exceeded",
err: <context.deadlineExceededError>{},
},
},
}fail [github.com/Azure/ARO-HCP/test/e2e/default_cluster_ingress_cert.go:165]: HyperShift's ingress certificate secret must exist and be referenced by IngressController/default
Expected success, but got an error:
<*fmt.wrapError | 0x1025ee5cc400>:
VerifyHyperShiftIngressCertificate timed out after 10m0.002s: getting HyperShift ingress secret openshift-ingress/default-ingress-cert: client rate limiter Wait returned an error: context deadline exceeded
{
msg: "VerifyHyperShiftIngressCertificate timed out after 10m0.002s: getting HyperShift ingress secret openshift-ingress/default-ingress-cert: client rate limiter Wait returned an error: context deadline exceeded",
err: <*fmt.wrapError | 0x1025ee5cc3e0>{
msg: "getting HyperShift ingress secret openshift-ingress/default-ingress-cert: client rate limiter Wait returned an error: context deadline exceeded",
err: <*fmt.wrapError | 0x1025ee5cc3c0>{
msg: "client rate limiter Wait returned an error: context deadline exceeded",
err: <context.deadlineExceededError>{},
},
},
}
fail [github.com/Azure/ARO-HCP/test/e2e/default_cluster_ingress_cert.go:165]: HyperShift's ingress certificate secret must exist and be referenced by IngressController/default
Expected success, but got an error:
<*fmt.wrapError | 0x1025ee5cc400>:
VerifyHyperShiftIngressCertificate timed out after 10m0.002s: getting HyperShift ingress secret openshift-ingress/default-ingress-cert: client rate limiter Wait returned an error: context deadline exceeded
{
msg: "VerifyHyperShiftIngressCertificate timed out after 10m0.002s: getting HyperShift ingress secret openshift-ingress/default-ingress-cert: client rate limiter Wait returned an error: context deadline exceeded",
err: <*fmt.wrapError | 0x1025ee5cc3e0>{
msg: "getting HyperShift ingress secret openshift-ingress/default-ingress-cert: client rate limiter Wait returned an error: context deadline exceeded",
err: <*fmt.wrapError | 0x1025ee5cc3c0>{
msg: "client rate limiter Wait returned an error: context deadline exceeded",
err: <context.deadlineExceededError>{},
},
},
}Contributing tests (2)
Affected runs (6)
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
external auth <external-auth>.Status was nilfull failure pattern: external auth <external-auth>.Status was nil Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 4 day(s) | e2e | 6 | 6.25%6 of 96 job runs affected | IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 4 day(s) | Show trend detailsOct 12: 0 · Oct 13: 0 · Oct 14: 0 · Oct 15: 0 · Oct 16: 1 · Oct 17: 2 · Oct 18: 2 · Oct 19: 1 · Oct 20: 0 · Oct 21: 0 · Oct 22: 0 · Oct 23: 0 · Oct 24: 0 · Oct 25: 0 | STG | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (3)fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:343]: Timed out after 300.003s.
Degraded condition did not converge to False
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:323 with:
external auth Properties.Status was nil
Expected
<*armredhatopenshifthcp.ResourceStatus | 0x0>: nil
not to be nil
fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:343]: Timed out after 300.003s.
Degraded condition did not converge to False
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:323 with:
external auth Properties.Status was nil
Expected
<*armredhatopenshifthcp.ResourceStatus | 0x0>: nil
not to be nilfail [github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:343]: Timed out after 300.004s.
Degraded condition did not converge to False
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:323 with:
external auth Properties.Status was nil
Expected
<*armredhatopenshifthcp.ResourceStatus | 0x0>: nil
not to be nil
fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:343]: Timed out after 300.004s.
Degraded condition did not converge to False
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:323 with:
external auth Properties.Status was nil
Expected
<*armredhatopenshifthcp.ResourceStatus | 0x0>: nil
not to be nilfail [github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:343]: Timed out after 300.001s.
Degraded condition did not converge to False
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:323 with:
external auth Properties.Status was nil
Expected
<*armredhatopenshifthcp.ResourceStatus | 0x0>: nil
not to be nil
fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:343]: Timed out after 300.001s.
Degraded condition did not converge to False
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:323 with:
external auth Properties.Status was nil
Expected
<*armredhatopenshifthcp.ResourceStatus | 0x0>: nil
not to be nilContributing tests (2)
Affected runs (6)
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
VerifyDefaultIngressServingCertificateSecretWiring timed out after 10m0full failure pattern: VerifyDefaultIngressServingCertificateSecretWiring timed out after 10m0 Signal: IndeterminateSignal: Indeterminate — no prior history; active 3 day(s) — New failure pattern — no prior history | e2e | 4 | 4.17%4 of 96 job runs affected | IndeterminateSignal: Indeterminate — no prior history; active 3 day(s) — New failure pattern — no prior history | Show trend detailsOct 12: 0 · Oct 13: 0 · Oct 14: 0 · Oct 15: 0 · Oct 16: 0 · Oct 17: 0 · Oct 18: 0 · Oct 19: 0 · Oct 20: 0 · Oct 21: 0 · Oct 22: 5 · Oct 23: 7 · Oct 24: 5 · Oct 25: 0 | DEV, STG | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (3)fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:220]: default-ingress-cert must exist and be referenced by IngressController/default for cluster "cluster-candidate-5-0-vsfrhd"
Expected success, but got an error:
<*fmt.wrapError | 0x1b75911f3b00>:
VerifyDefaultIngressServingCertificateSecretWiring timed out after 10m0.004s: IngressController/default references &LocalObjectReference{Name:cluster-ingress-cert,}, expected secret "default-ingress-cert"
{
msg: "VerifyDefaultIngressServingCertificateSecretWiring timed out after 10m0.004s: IngressController/default references &LocalObjectReference{Name:cluster-ingress-cert,}, expected secret \"default-ingress-cert\"",
err: <*verifiers.pollObservationError | 0x1b75911f3ae0>{
observation: <*errors.errorString | 0x1b7593506210>{
s: "IngressController/default references &LocalObjectReference{Name:cluster-ingress-cert,}, expected secret \"default-ingress-cert\"",
},
interruption: <*fmt.wrapError | 0x1b75911f3a60>{
msg: "getting default ingress serving certificate secret openshift-ingress/default-ingress-cert: context deadline exceeded",
err: <context.deadlineExceededError>{},
},
},
}
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:220]: default-ingress-cert must exist and be referenced by IngressController/default for cluster "cluster-candidate-5-0-vsfrhd"
Expected success, but got an error:
<*fmt.wrapError | 0x1b75911f3b00>:
VerifyDefaultIngressServingCertificateSecretWiring timed out after 10m0.004s: IngressController/default references &LocalObjectReference{Name:cluster-ingress-cert,}, expected secret "default-ingress-cert"
{
msg: "VerifyDefaultIngressServingCertificateSecretWiring timed out after 10m0.004s: IngressController/default references &LocalObjectReference{Name:cluster-ingress-cert,}, expected secret \"default-ingress-cert\"",
err: <*verifiers.pollObservationError | 0x1b75911f3ae0>{
observation: <*errors.errorString | 0x1b7593506210>{
s: "IngressController/default references &LocalObjectReference{Name:cluster-ingress-cert,}, expected secret \"default-ingress-cert\"",
},
interruption: <*fmt.wrapError | 0x1b75911f3a60>{
msg: "getting default ingress serving certificate secret openshift-ingress/default-ingress-cert: context deadline exceeded",
err: <context.deadlineExceededError>{},
},
},
}fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:220]: default-ingress-cert must exist and be referenced by IngressController/default for cluster "cluster-candidate-4-23-hvm2zb"
Expected success, but got an error:
<*fmt.wrapError | 0x3a2d820f0740>:
VerifyDefaultIngressServingCertificateSecretWiring timed out after 10m0.009s: getting default ingress serving certificate secret openshift-ingress/default-ingress-cert: client rate limiter Wait returned an error: rate: Wait(n=1) would exceed context deadline
{
msg: "VerifyDefaultIngressServingCertificateSecretWiring timed out after 10m0.009s: getting default ingress serving certificate secret openshift-ingress/default-ingress-cert: client rate limiter Wait returned an error: rate: Wait(n=1) would exceed context deadline",
err: <*fmt.wrapError | 0x3a2d820f0700>{
msg: "getting default ingress serving certificate secret openshift-ingress/default-ingress-cert: client rate limiter Wait returned an error: rate: Wait(n=1) would exceed context deadline",
err: <*fmt.wrapError | 0x3a2d820f06c0>{
msg: "client rate limiter Wait returned an error: rate: Wait(n=1) would exceed context deadline",
err: <*errors.errorString | 0x3a2d82526860>{
s: "rate: Wait(n=1) would exceed context deadline",
},
},
},
}
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:220]: default-ingress-cert must exist and be referenced by IngressController/default for cluster "cluster-candidate-4-23-hvm2zb"
Expected success, but got an error:
<*fmt.wrapError | 0x3a2d820f0740>:
VerifyDefaultIngressServingCertificateSecretWiring timed out after 10m0.009s: getting default ingress serving certificate secret openshift-ingress/default-ingress-cert: client rate limiter Wait returned an error: rate: Wait(n=1) would exceed context deadline
{
msg: "VerifyDefaultIngressServingCertificateSecretWiring timed out after 10m0.009s: getting default ingress serving certificate secret openshift-ingress/default-ingress-cert: client rate limiter Wait returned an error: rate: Wait(n=1) would exceed context deadline",
err: <*fmt.wrapError | 0x3a2d820f0700>{
msg: "getting default ingress serving certificate secret openshift-ingress/default-ingress-cert: client rate limiter Wait returned an error: rate: Wait(n=1) would exceed context deadline",
err: <*fmt.wrapError | 0x3a2d820f06c0>{
msg: "client rate limiter Wait returned an error: rate: Wait(n=1) would exceed context deadline",
err: <*errors.errorString | 0x3a2d82526860>{
s: "rate: Wait(n=1) would exceed context deadline",
},
},
},
}fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:220]: default-ingress-cert must exist and be referenced by IngressController/default for cluster "cluster-candidate-4-20-m5lclb"
Expected success, but got an error:
<*fmt.wrapError | 0x12d085988600>:
VerifyDefaultIngressServingCertificateSecretWiring timed out after 10m0.007s: IngressController/default references &LocalObjectReference{Name:cluster-ingress-cert,}, expected secret "default-ingress-cert"
{
msg: "VerifyDefaultIngressServingCertificateSecretWiring timed out after 10m0.007s: IngressController/default references &LocalObjectReference{Name:cluster-ingress-cert,}, expected secret \"default-ingress-cert\"",
err: <*errors.errorString | 0x12d085c685e0>{
s: "IngressController/default references &LocalObjectReference{Name:cluster-ingress-cert,}, expected secret \"default-ingress-cert\"",
},
}
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:220]: default-ingress-cert must exist and be referenced by IngressController/default for cluster "cluster-candidate-4-20-m5lclb"
Expected success, but got an error:
<*fmt.wrapError | 0x12d085988600>:
VerifyDefaultIngressServingCertificateSecretWiring timed out after 10m0.007s: IngressController/default references &LocalObjectReference{Name:cluster-ingress-cert,}, expected secret "default-ingress-cert"
{
msg: "VerifyDefaultIngressServingCertificateSecretWiring timed out after 10m0.007s: IngressController/default references &LocalObjectReference{Name:cluster-ingress-cert,}, expected secret \"default-ingress-cert\"",
err: <*errors.errorString | 0x12d085c685e0>{
s: "IngressController/default references &LocalObjectReference{Name:cluster-ingress-cert,}, expected secret \"default-ingress-cert\"",
},
}Contributing tests (10)
Affected runs (4)
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
ERROR CODE: DeadlineExceeded; detail message node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hype...full failure pattern: ERROR CODE: DeadlineExceeded; detail message node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is <count>, want <count>; node pool condition AllNodesHealthy is False: <count> of <count> machines are not healthy Machine <node>: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.Compute/virtualMachines/<node> to exist Machine <node>: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.Compute/virtualMachines/<node> to exist; node pool condition AllMachinesReady is False: <count> of <count> machines are not ready Machine <node>: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.Compute/virtualMachines/<node> to exist Machine <node>: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.Compute/virtualMachines/<node> to exist; provider Microsoft.RedHatOpenShift Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 3 day(s) | e2e | 4 | 4.17%4 of 96 job runs affected | IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 3 day(s) | Show trend detailsOct 12: 0 · Oct 13: 0 · Oct 14: 0 · Oct 15: 2 · Oct 16: 0 · Oct 17: 0 · Oct 18: 0 · Oct 19: 0 · Oct 20: 1 · Oct 21: 0 · Oct 22: 0 · Oct 23: 1 · Oct 24: 0 · Oct 25: 0 | DEV, INT | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (3)fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delete_cx_rg.go:141]: failed to create node pools in parallel for cluster "cx-rg-hcp-cluster"
Unexpected error:
<*fmt.wrapError | 0x9a8a07812a0>:
failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "cx-rg-hcp-cluster" in resourcegroup="cx-rg-cluster-npbbgrcnjvjf" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/switzerlandnorth/hcpOperationStatuses/5c43f0b5-5e0d-4c5c-89a9-5d1f04167fe7
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/switzerlandnorth/hcpOperationStatuses/5c43f0b5-5e0d-4c5c-89a9-5d1f04167fe7",
"name": "5c43f0b5-5e0d-4c5c-89a9-5d1f04167fe7",
"status": "Failed",
"startTime": "2026-10-23T14:54:47.282384682Z",
"endTime": "2026-10-23T15:13:53.224517483Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine d2h8j1q0z4a1f4u-np-1-ntqn4-2vwr5: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/cx-rg-cluster-npbbgrcnjvjf--managed/providers/Microsoft.Compute/virtualMachines/d2h8j1q0z4a1f4u-np-1-ntqn4-2vwr5 to exist\nMachine d2h8j1q0z4a1f4u-np-1-ntqn4-rkqlm: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/cx-rg-cluster-npbbgrcnjvjf--managed/providers/Microsoft.Compute/virtualMachines/d2h8j1q0z4a1f4u-np-1-ntqn4-rkqlm to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine d2h8j1q0z4a1f4u-np-1-ntqn4-2vwr5: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/cx-rg-cluster-npbbgrcnjvjf--managed/providers/Microsoft.Compute/virtualMachines/d2h8j1q0z4a1f4u-np-1-ntqn4-2vwr5 to exist\nMachine d2h8j1q0z4a1f4u-np-1-ntqn4-rkqlm: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/cx-rg-cluster-npbbgrcnjvjf--managed/providers/Microsoft.Compute/virtualMachines/d2h8j1q0z4a1f4u-np-1-ntqn4-rkqlm to exist\n"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/switzerlandnorth/hcpOperationStatuses/5c43f0b5-5e0d-4c5c-89a9-5d1f04167fe7
correlation ID: 90989f90-f502-41d8-955e-974a04d1e9d9
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delete_cx_rg.go:141]: failed to create node pools in parallel for cluster "cx-rg-hcp-cluster"
Unexpected error:
<*fmt.wrapError | 0x9a8a07812a0>:
failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "cx-rg-hcp-cluster" in resourcegroup="cx-rg-cluster-npbbgrcnjvjf" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/switzerlandnorth/hcpOperationStatuses/5c43f0b5-5e0d-4c5c-89a9-5d1f04167fe7
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/switzerlandnorth/hcpOperationStatuses/5c43f0b5-5e0d-4c5c-89a9-5d1f04167fe7",
"name": "5c43f0b5-5e0d-4c5c-89a9-5d1f04167fe7",
"status": "Failed",
"startTime": "2026-10-23T14:54:47.282384682Z",
"endTime": "2026-10-23T15:13:53.224517483Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine d2h8j1q0z4a1f4u-np-1-ntqn4-2vwr5: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/cx-rg-cluster-npbbgrcnjvjf--managed/providers/Microsoft.Compute/virtualMachines/d2h8j1q0z4a1f4u-np-1-ntqn4-2vwr5 to exist\nMachine d2h8j1q0z4a1f4u-np-1-ntqn4-rkqlm: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/cx-rg-cluster-npbbgrcnjvjf--managed/providers/Microsoft.Compute/virtualMachines/d2h8j1q0z4a1f4u-np-1-ntqn4-rkqlm to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine d2h8j1q0z4a1f4u-np-1-ntqn4-2vwr5: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/cx-rg-cluster-npbbgrcnjvjf--managed/providers/Microsoft.Compute/virtualMachines/d2h8j1q0z4a1f4u-np-1-ntqn4-2vwr5 to exist\nMachine d2h8j1q0z4a1f4u-np-1-ntqn4-rkqlm: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/cx-rg-cluster-npbbgrcnjvjf--managed/providers/Microsoft.Compute/virtualMachines/d2h8j1q0z4a1f4u-np-1-ntqn4-rkqlm to exist\n"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/switzerlandnorth/hcpOperationStatuses/5c43f0b5-5e0d-4c5c-89a9-5d1f04167fe7
correlation ID: 90989f90-f502-41d8-955e-974a04d1e9d9
client request ID: unavailable
occurredfail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:165]: failed to create node pool npupgrade-4-21 with version 4.21.38
Unexpected error:
<*fmt.wrapError | 0x3fd8b58f89c0>:
failed to create NodePool npupgrade-4-21: failed waiting for nodepool="npupgrade-4-21" for cluster "np-version-upgrade-cluster-r7w7ff" in resourcegroup="rg-np-version-upgrade-r7w7ff-gzcdhfsz9lf7" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/australiaeast/hcpOperationStatuses/392e4864-c483-4741-bf7f-aafe39a6a315
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/australiaeast/hcpOperationStatuses/392e4864-c483-4741-bf7f-aafe39a6a315",
"name": "392e4864-c483-4741-bf7f-aafe39a6a315",
"status": "Failed",
"startTime": "2026-10-20T03:14:53.109279817Z",
"endTime": "2026-10-20T03:33:59.378971686Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine p4u4z6x8s3x8m0b-npupgrade-4-21-wfvbp-42hc7: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-version-upgrade-r7w7ff-gzcdhfsz9lf7-np-upgrade-r7-b213f862/providers/Microsoft.Compute/virtualMachines/p4u4z6x8s3x8m0b-npupgrade-4-21-wfvbp-42hc7 to exist\nMachine p4u4z6x8s3x8m0b-npupgrade-4-21-wfvbp-pkjw5: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-version-upgrade-r7w7ff-gzcdhfsz9lf7-np-upgrade-r7-b213f862/providers/Microsoft.Compute/virtualMachines/p4u4z6x8s3x8m0b-npupgrade-4-21-wfvbp-pkjw5 to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine p4u4z6x8s3x8m0b-npupgrade-4-21-wfvbp-42hc7: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-version-upgrade-r7w7ff-gzcdhfsz9lf7-np-upgrade-r7-b213f862/providers/Microsoft.Compute/virtualMachines/p4u4z6x8s3x8m0b-npupgrade-4-21-wfvbp-42hc7 to exist\nMachine p4u4z6x8s3x8m0b-npupgrade-4-21-wfvbp-pkjw5: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-version-upgrade-r7w7ff-gzcdhfsz9lf7-np-upgrade-r7-b213f862/providers/Microsoft.Compute/virtualMachines/p4u4z6x8s3x8m0b-npupgrade-4-21-wfvbp-pkjw5 to exist\n"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/australiaeast/hcpOperationStatuses/392e4864-c483-4741-bf7f-aafe39a6a315
correlation ID: 93f4ee4e-2929-477b-b173-ef81cb3abee4
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:165]: failed to create node pool npupgrade-4-21 with version 4.21.38
Unexpected error:
<*fmt.wrapError | 0x3fd8b58f89c0>:
failed to create NodePool npupgrade-4-21: failed waiting for nodepool="npupgrade-4-21" for cluster "np-version-upgrade-cluster-r7w7ff" in resourcegroup="rg-np-version-upgrade-r7w7ff-gzcdhfsz9lf7" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/australiaeast/hcpOperationStatuses/392e4864-c483-4741-bf7f-aafe39a6a315
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/australiaeast/hcpOperationStatuses/392e4864-c483-4741-bf7f-aafe39a6a315",
"name": "392e4864-c483-4741-bf7f-aafe39a6a315",
"status": "Failed",
"startTime": "2026-10-20T03:14:53.109279817Z",
"endTime": "2026-10-20T03:33:59.378971686Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine p4u4z6x8s3x8m0b-npupgrade-4-21-wfvbp-42hc7: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-version-upgrade-r7w7ff-gzcdhfsz9lf7-np-upgrade-r7-b213f862/providers/Microsoft.Compute/virtualMachines/p4u4z6x8s3x8m0b-npupgrade-4-21-wfvbp-42hc7 to exist\nMachine p4u4z6x8s3x8m0b-npupgrade-4-21-wfvbp-pkjw5: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-version-upgrade-r7w7ff-gzcdhfsz9lf7-np-upgrade-r7-b213f862/providers/Microsoft.Compute/virtualMachines/p4u4z6x8s3x8m0b-npupgrade-4-21-wfvbp-pkjw5 to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine p4u4z6x8s3x8m0b-npupgrade-4-21-wfvbp-42hc7: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-version-upgrade-r7w7ff-gzcdhfsz9lf7-np-upgrade-r7-b213f862/providers/Microsoft.Compute/virtualMachines/p4u4z6x8s3x8m0b-npupgrade-4-21-wfvbp-42hc7 to exist\nMachine p4u4z6x8s3x8m0b-npupgrade-4-21-wfvbp-pkjw5: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-version-upgrade-r7w7ff-gzcdhfsz9lf7-np-upgrade-r7-b213f862/providers/Microsoft.Compute/virtualMachines/p4u4z6x8s3x8m0b-npupgrade-4-21-wfvbp-pkjw5 to exist\n"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/australiaeast/hcpOperationStatuses/392e4864-c483-4741-bf7f-aafe39a6a315
correlation ID: 93f4ee4e-2929-477b-b173-ef81cb3abee4
client request ID: unavailable
occurredfail [github.com/Azure/ARO-HCP/test/e2e/cluster_fips_mode.go:103]: failed to create node pool "np-1" for fips-enabled cluster "fips-enabled-cluster"
Unexpected error:
<*fmt.wrapError | 0x3d468a7c3d60>:
failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "fips-enabled-cluster" in resourcegroup="fips-enabled-sdw8m7lp2fnw" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/3fb5e0fc-3699-4b03-ac28-00c6e56abac9
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/3fb5e0fc-3699-4b03-ac28-00c6e56abac9",
"name": "3fb5e0fc-3699-4b03-ac28-00c6e56abac9",
"status": "Failed",
"startTime": "2026-10-15T23:18:55.172688796Z",
"endTime": "2026-10-15T23:47:59.507731058Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine t5q8c0p7j0p8q4v-np-1-t5hzk-lk85n: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/fips-enabled-sdw8m7lp2fnw--managed/providers/Microsoft.Compute/virtualMachines/t5q8c0p7j0p8q4v-np-1-t5hzk-lk85n to exist\nMachine t5q8c0p7j0p8q4v-np-1-t5hzk-rnrqf: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/fips-enabled-sdw8m7lp2fnw--managed/providers/Microsoft.Compute/virtualMachines/t5q8c0p7j0p8q4v-np-1-t5hzk-rnrqf to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine t5q8c0p7j0p8q4v-np-1-t5hzk-lk85n: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/fips-enabled-sdw8m7lp2fnw--managed/providers/Microsoft.Compute/virtualMachines/t5q8c0p7j0p8q4v-np-1-t5hzk-lk85n to exist\nMachine t5q8c0p7j0p8q4v-np-1-t5hzk-rnrqf: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/fips-enabled-sdw8m7lp2fnw--managed/providers/Microsoft.Compute/virtualMachines/t5q8c0p7j0p8q4v-np-1-t5hzk-rnrqf to exist\n"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/3fb5e0fc-3699-4b03-ac28-00c6e56abac9
correlation ID: df77bc05-17d5-4c4c-bad3-0bd2a5dbd8cc
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_fips_mode.go:103]: failed to create node pool "np-1" for fips-enabled cluster "fips-enabled-cluster"
Unexpected error:
<*fmt.wrapError | 0x3d468a7c3d60>:
failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "fips-enabled-cluster" in resourcegroup="fips-enabled-sdw8m7lp2fnw" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/3fb5e0fc-3699-4b03-ac28-00c6e56abac9
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/3fb5e0fc-3699-4b03-ac28-00c6e56abac9",
"name": "3fb5e0fc-3699-4b03-ac28-00c6e56abac9",
"status": "Failed",
"startTime": "2026-10-15T23:18:55.172688796Z",
"endTime": "2026-10-15T23:47:59.507731058Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine t5q8c0p7j0p8q4v-np-1-t5hzk-lk85n: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/fips-enabled-sdw8m7lp2fnw--managed/providers/Microsoft.Compute/virtualMachines/t5q8c0p7j0p8q4v-np-1-t5hzk-lk85n to exist\nMachine t5q8c0p7j0p8q4v-np-1-t5hzk-rnrqf: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/fips-enabled-sdw8m7lp2fnw--managed/providers/Microsoft.Compute/virtualMachines/t5q8c0p7j0p8q4v-np-1-t5hzk-rnrqf to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine t5q8c0p7j0p8q4v-np-1-t5hzk-lk85n: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/fips-enabled-sdw8m7lp2fnw--managed/providers/Microsoft.Compute/virtualMachines/t5q8c0p7j0p8q4v-np-1-t5hzk-lk85n to exist\nMachine t5q8c0p7j0p8q4v-np-1-t5hzk-rnrqf: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/fips-enabled-sdw8m7lp2fnw--managed/providers/Microsoft.Compute/virtualMachines/t5q8c0p7j0p8q4v-np-1-t5hzk-rnrqf to exist\n"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/3fb5e0fc-3699-4b03-ac28-00c6e56abac9
correlation ID: df77bc05-17d5-4c4c-bad3-0bd2a5dbd8cc
client request ID: unavailable
occurredContributing tests (4)
Affected runs (4)
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
ERROR CODE: DeadlineExceeded; detail message node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hype...full failure pattern: ERROR CODE: DeadlineExceeded; detail message node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is <count>, want <count>; node pool condition AllNodesHealthy is False: <count> of <count> machines are not healthy Machine <node>: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.Compute/virtualMachines/<node> to exist; node pool condition AllMachinesReady is False: <count> of <count> machines are not ready Machine <node>: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.Compute/virtualMachines/<node> to exist; provider Microsoft.RedHatOpenShift Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 3 day(s) | e2e | 4 | 4.17%4 of 96 job runs affected | IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 3 day(s) | Show trend detailsOct 12: 1 · Oct 13: 0 · Oct 14: 0 · Oct 15: 2 · Oct 16: 0 · Oct 17: 0 · Oct 18: 0 · Oct 19: 0 · Oct 20: 0 · Oct 21: 1 · Oct 22: 0 · Oct 23: 0 · Oct 24: 0 · Oct 25: 0 | DEV, STG | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (3)fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delete_cx_rg.go:141]: failed to create node pools in parallel for cluster "cx-rg-hcp-cluster"
Unexpected error:
<*fmt.wrapError | 0x29388dc21b20>:
failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "cx-rg-hcp-cluster" in resourcegroup="cx-rg-cluster-l6ddnj7vhmkq" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/8669d017-4213-4ec1-86fe-07c71e8067ec
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/8669d017-4213-4ec1-86fe-07c71e8067ec",
"name": "8669d017-4213-4ec1-86fe-07c71e8067ec",
"status": "Failed",
"startTime": "2026-10-21T16:34:55.668662338Z",
"endTime": "2026-10-21T16:53:59.759177283Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine s3e2z4z8o4i6e7y-np-1-7pccj-mpwmx: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/cx-rg-cluster-l6ddnj7vhmkq--managed/providers/Microsoft.Compute/virtualMachines/s3e2z4z8o4i6e7y-np-1-7pccj-mpwmx to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine s3e2z4z8o4i6e7y-np-1-7pccj-mpwmx: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/cx-rg-cluster-l6ddnj7vhmkq--managed/providers/Microsoft.Compute/virtualMachines/s3e2z4z8o4i6e7y-np-1-7pccj-mpwmx to exist\n"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/8669d017-4213-4ec1-86fe-07c71e8067ec
correlation ID: 7627942b-6e2a-4d1f-a2c9-7a7ff2104478
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delete_cx_rg.go:141]: failed to create node pools in parallel for cluster "cx-rg-hcp-cluster"
Unexpected error:
<*fmt.wrapError | 0x29388dc21b20>:
failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "cx-rg-hcp-cluster" in resourcegroup="cx-rg-cluster-l6ddnj7vhmkq" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/8669d017-4213-4ec1-86fe-07c71e8067ec
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/8669d017-4213-4ec1-86fe-07c71e8067ec",
"name": "8669d017-4213-4ec1-86fe-07c71e8067ec",
"status": "Failed",
"startTime": "2026-10-21T16:34:55.668662338Z",
"endTime": "2026-10-21T16:53:59.759177283Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine s3e2z4z8o4i6e7y-np-1-7pccj-mpwmx: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/cx-rg-cluster-l6ddnj7vhmkq--managed/providers/Microsoft.Compute/virtualMachines/s3e2z4z8o4i6e7y-np-1-7pccj-mpwmx to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine s3e2z4z8o4i6e7y-np-1-7pccj-mpwmx: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/cx-rg-cluster-l6ddnj7vhmkq--managed/providers/Microsoft.Compute/virtualMachines/s3e2z4z8o4i6e7y-np-1-7pccj-mpwmx to exist\n"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/8669d017-4213-4ec1-86fe-07c71e8067ec
correlation ID: 7627942b-6e2a-4d1f-a2c9-7a7ff2104478
client request ID: unavailable
occurredfail [github.com/Azure/ARO-HCP/test/e2e/cluster_fips_mode.go:103]: failed to create node pool "np-1" for fips-enabled cluster "fips-enabled-cluster"
Unexpected error:
<*fmt.wrapError | 0x25ccdbb84fe0>:
failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "fips-enabled-cluster" in resourcegroup="fips-enabled-fwqz99tj9vqg" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/australiaeast/hcpOperationStatuses/3b0e9b20-034d-4163-98db-b3ac4dede16f
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/australiaeast/hcpOperationStatuses/3b0e9b20-034d-4163-98db-b3ac4dede16f",
"name": "3b0e9b20-034d-4163-98db-b3ac4dede16f",
"status": "Failed",
"startTime": "2026-10-15T07:00:37.486714177Z",
"endTime": "2026-10-15T07:29:37.711665502Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine b4r7i6b1m2k8s5l-np-1-mpxck-gf8ht: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/fips-enabled-fwqz99tj9vqg--managed/providers/Microsoft.Compute/virtualMachines/b4r7i6b1m2k8s5l-np-1-mpxck-gf8ht to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine b4r7i6b1m2k8s5l-np-1-mpxck-gf8ht: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/fips-enabled-fwqz99tj9vqg--managed/providers/Microsoft.Compute/virtualMachines/b4r7i6b1m2k8s5l-np-1-mpxck-gf8ht to exist\n"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/australiaeast/hcpOperationStatuses/3b0e9b20-034d-4163-98db-b3ac4dede16f
correlation ID: 0ad357fe-5f92-4836-9361-6bb0422c841f
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_fips_mode.go:103]: failed to create node pool "np-1" for fips-enabled cluster "fips-enabled-cluster"
Unexpected error:
<*fmt.wrapError | 0x25ccdbb84fe0>:
failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "fips-enabled-cluster" in resourcegroup="fips-enabled-fwqz99tj9vqg" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/australiaeast/hcpOperationStatuses/3b0e9b20-034d-4163-98db-b3ac4dede16f
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/australiaeast/hcpOperationStatuses/3b0e9b20-034d-4163-98db-b3ac4dede16f",
"name": "3b0e9b20-034d-4163-98db-b3ac4dede16f",
"status": "Failed",
"startTime": "2026-10-15T07:00:37.486714177Z",
"endTime": "2026-10-15T07:29:37.711665502Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine b4r7i6b1m2k8s5l-np-1-mpxck-gf8ht: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/fips-enabled-fwqz99tj9vqg--managed/providers/Microsoft.Compute/virtualMachines/b4r7i6b1m2k8s5l-np-1-mpxck-gf8ht to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine b4r7i6b1m2k8s5l-np-1-mpxck-gf8ht: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/fips-enabled-fwqz99tj9vqg--managed/providers/Microsoft.Compute/virtualMachines/b4r7i6b1m2k8s5l-np-1-mpxck-gf8ht to exist\n"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/australiaeast/hcpOperationStatuses/3b0e9b20-034d-4163-98db-b3ac4dede16f
correlation ID: 0ad357fe-5f92-4836-9361-6bb0422c841f
client request ID: unavailable
occurredfail [github.com/Azure/ARO-HCP/test/e2e/nodepool_cordon.go:117]: failed to create node pool cordon-np
Unexpected error:
<*fmt.wrapError | 0x10ea921035c0>:
failed to create NodePool cordon-np: failed waiting for nodepool="cordon-np" for cluster "cordon-hcp-cluster" in resourcegroup="rg-cordon-test-4426rkkk84lt" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/cf6cd4fd-40ae-433a-b987-569919cfb0d4
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/cf6cd4fd-40ae-433a-b987-569919cfb0d4",
"name": "cf6cd4fd-40ae-433a-b987-569919cfb0d4",
"status": "Failed",
"startTime": "2026-10-15T04:26:27.342644977Z",
"endTime": "2026-10-15T04:55:33.710462341Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine a3g3l9y4g0x3y3s-cordon-np-rjbnx-7swmq: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cordon-test-4426rkkk84lt--managed/providers/Microsoft.Compute/virtualMachines/a3g3l9y4g0x3y3s-cordon-np-rjbnx-7swmq to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine a3g3l9y4g0x3y3s-cordon-np-rjbnx-7swmq: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cordon-test-4426rkkk84lt--managed/providers/Microsoft.Compute/virtualMachines/a3g3l9y4g0x3y3s-cordon-np-rjbnx-7swmq to exist\n"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/cf6cd4fd-40ae-433a-b987-569919cfb0d4
correlation ID: 442271c1-98e4-4b63-9254-ce9229dd3320
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_cordon.go:117]: failed to create node pool cordon-np
Unexpected error:
<*fmt.wrapError | 0x10ea921035c0>:
failed to create NodePool cordon-np: failed waiting for nodepool="cordon-np" for cluster "cordon-hcp-cluster" in resourcegroup="rg-cordon-test-4426rkkk84lt" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/cf6cd4fd-40ae-433a-b987-569919cfb0d4
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/cf6cd4fd-40ae-433a-b987-569919cfb0d4",
"name": "cf6cd4fd-40ae-433a-b987-569919cfb0d4",
"status": "Failed",
"startTime": "2026-10-15T04:26:27.342644977Z",
"endTime": "2026-10-15T04:55:33.710462341Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine a3g3l9y4g0x3y3s-cordon-np-rjbnx-7swmq: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cordon-test-4426rkkk84lt--managed/providers/Microsoft.Compute/virtualMachines/a3g3l9y4g0x3y3s-cordon-np-rjbnx-7swmq to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine a3g3l9y4g0x3y3s-cordon-np-rjbnx-7swmq: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cordon-test-4426rkkk84lt--managed/providers/Microsoft.Compute/virtualMachines/a3g3l9y4g0x3y3s-cordon-np-rjbnx-7swmq to exist\n"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/cf6cd4fd-40ae-433a-b987-569919cfb0d4
correlation ID: 442271c1-98e4-4b63-9254-ce9229dd3320
client request ID: unavailable
occurredContributing tests (3)
Affected runs (4)
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
DNS for route host <route-host> did not resolve: DNS for <route-host> did not resolve within <duration> (last error: lookup <host> on <dns-server>: dial udp <ip>: i/o timeout): co...full failure pattern: DNS for route host <route-host> did not resolve: DNS for <route-host> did not resolve within <duration> (last error: lookup <host> on <dns-server>: dial udp <ip>: i/o timeout): context deadline exceeded Signal: IndeterminateSignal: Indeterminate — no prior history; active 2 day(s) — New failure pattern — no prior history | e2e | 3 | 3.12%3 of 96 job runs affected | IndeterminateSignal: Indeterminate — no prior history; active 2 day(s) — New failure pattern — no prior history | Show trend detailsOct 12: 0 · Oct 13: 0 · Oct 14: 0 · Oct 15: 0 · Oct 16: 0 · Oct 17: 0 · Oct 18: 0 · Oct 19: 0 · Oct 20: 0 · Oct 21: 0 · Oct 22: 0 · Oct 23: 5 · Oct 24: 0 · Oct 25: 7 | DEV, INT | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (3)fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:190]: failed to verify simple web app runs on cluster "cluster-candidate-4-20-5gtk26"
Unexpected error:
<*fmt.wrapError | 0x187dce020660>:
DNS for route host agnhost-e2e-sample-app-vvdwh.apps.aro.l5t3q5b8q6k0t1r.850o.eastus2.aroapp-hcp.io did not resolve: DNS for agnhost-e2e-sample-app-vvdwh.apps.aro.l5t3q5b8q6k0t1r.850o.eastus2.aroapp-hcp.io did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-vvdwh.apps.aro.l5t3q5b8q6k0t1r.850o.eastus2.aroapp-hcp.io on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded
{
msg: "DNS for route host agnhost-e2e-sample-app-vvdwh.apps.aro.l5t3q5b8q6k0t1r.850o.eastus2.aroapp-hcp.io did not resolve: DNS for agnhost-e2e-sample-app-vvdwh.apps.aro.l5t3q5b8q6k0t1r.850o.eastus2.aroapp-hcp.io did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-vvdwh.apps.aro.l5t3q5b8q6k0t1r.850o.eastus2.aroapp-hcp.io on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
err: <*fmt.wrapError | 0x187dce1941c0>{
msg: "DNS for agnhost-e2e-sample-app-vvdwh.apps.aro.l5t3q5b8q6k0t1r.850o.eastus2.aroapp-hcp.io did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-vvdwh.apps.aro.l5t3q5b8q6k0t1r.850o.eastus2.aroapp-hcp.io on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
err: <context.deadlineExceededError>{},
},
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:190]: failed to verify simple web app runs on cluster "cluster-candidate-4-20-5gtk26"
Unexpected error:
<*fmt.wrapError | 0x187dce020660>:
DNS for route host agnhost-e2e-sample-app-vvdwh.apps.aro.l5t3q5b8q6k0t1r.850o.eastus2.aroapp-hcp.io did not resolve: DNS for agnhost-e2e-sample-app-vvdwh.apps.aro.l5t3q5b8q6k0t1r.850o.eastus2.aroapp-hcp.io did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-vvdwh.apps.aro.l5t3q5b8q6k0t1r.850o.eastus2.aroapp-hcp.io on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded
{
msg: "DNS for route host agnhost-e2e-sample-app-vvdwh.apps.aro.l5t3q5b8q6k0t1r.850o.eastus2.aroapp-hcp.io did not resolve: DNS for agnhost-e2e-sample-app-vvdwh.apps.aro.l5t3q5b8q6k0t1r.850o.eastus2.aroapp-hcp.io did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-vvdwh.apps.aro.l5t3q5b8q6k0t1r.850o.eastus2.aroapp-hcp.io on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
err: <*fmt.wrapError | 0x187dce1941c0>{
msg: "DNS for agnhost-e2e-sample-app-vvdwh.apps.aro.l5t3q5b8q6k0t1r.850o.eastus2.aroapp-hcp.io did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-vvdwh.apps.aro.l5t3q5b8q6k0t1r.850o.eastus2.aroapp-hcp.io on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
err: <context.deadlineExceededError>{},
},
}
occurredfail [github.com/Azure/ARO-HCP/test/e2e/cluster_nodepool_active_versions.go:149]: failed to verify simple web app runs on v20261001 cluster "v20261001"
Unexpected error:
<*fmt.wrapError | 0x33aec8dca6e0>:
DNS for route host agnhost-e2e-sample-app-bmd4l.apps.aro.v20261001.ez3w.eastus2.aroapp-hcp.io did not resolve: DNS for agnhost-e2e-sample-app-bmd4l.apps.aro.v20261001.ez3w.eastus2.aroapp-hcp.io did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-bmd4l.apps.aro.v20261001.ez3w.eastus2.aroapp-hcp.io on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded
{
msg: "DNS for route host agnhost-e2e-sample-app-bmd4l.apps.aro.v20261001.ez3w.eastus2.aroapp-hcp.io did not resolve: DNS for agnhost-e2e-sample-app-bmd4l.apps.aro.v20261001.ez3w.eastus2.aroapp-hcp.io did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-bmd4l.apps.aro.v20261001.ez3w.eastus2.aroapp-hcp.io on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
err: <*fmt.wrapError | 0x33aec9a24040>{
msg: "DNS for agnhost-e2e-sample-app-bmd4l.apps.aro.v20261001.ez3w.eastus2.aroapp-hcp.io did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-bmd4l.apps.aro.v20261001.ez3w.eastus2.aroapp-hcp.io on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
err: <context.deadlineExceededError>{},
},
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_nodepool_active_versions.go:149]: failed to verify simple web app runs on v20261001 cluster "v20261001"
Unexpected error:
<*fmt.wrapError | 0x33aec8dca6e0>:
DNS for route host agnhost-e2e-sample-app-bmd4l.apps.aro.v20261001.ez3w.eastus2.aroapp-hcp.io did not resolve: DNS for agnhost-e2e-sample-app-bmd4l.apps.aro.v20261001.ez3w.eastus2.aroapp-hcp.io did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-bmd4l.apps.aro.v20261001.ez3w.eastus2.aroapp-hcp.io on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded
{
msg: "DNS for route host agnhost-e2e-sample-app-bmd4l.apps.aro.v20261001.ez3w.eastus2.aroapp-hcp.io did not resolve: DNS for agnhost-e2e-sample-app-bmd4l.apps.aro.v20261001.ez3w.eastus2.aroapp-hcp.io did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-bmd4l.apps.aro.v20261001.ez3w.eastus2.aroapp-hcp.io on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
err: <*fmt.wrapError | 0x33aec9a24040>{
msg: "DNS for agnhost-e2e-sample-app-bmd4l.apps.aro.v20261001.ez3w.eastus2.aroapp-hcp.io did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-bmd4l.apps.aro.v20261001.ez3w.eastus2.aroapp-hcp.io on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
err: <context.deadlineExceededError>{},
},
}
occurredfail [github.com/Azure/ARO-HCP/test/e2e/cluster_version_backlevel.go:201]: failed to verify simple web app on back-level cluster version 4.20
Unexpected error:
<*fmt.wrapError | 0xaa72ff3eb40>:
DNS for route host agnhost-e2e-sample-app-7hxtd.apps.aro.v2i3p0q7g9o9o1k.cnfo.eastus2.aroapp-hcp.io did not resolve: DNS for agnhost-e2e-sample-app-7hxtd.apps.aro.v2i3p0q7g9o9o1k.cnfo.eastus2.aroapp-hcp.io did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-7hxtd.apps.aro.v2i3p0q7g9o9o1k.cnfo.eastus2.aroapp-hcp.io on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded
{
msg: "DNS for route host agnhost-e2e-sample-app-7hxtd.apps.aro.v2i3p0q7g9o9o1k.cnfo.eastus2.aroapp-hcp.io did not resolve: DNS for agnhost-e2e-sample-app-7hxtd.apps.aro.v2i3p0q7g9o9o1k.cnfo.eastus2.aroapp-hcp.io did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-7hxtd.apps.aro.v2i3p0q7g9o9o1k.cnfo.eastus2.aroapp-hcp.io on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
err: <*fmt.wrapError | 0xaa72fd7c9e0>{
msg: "DNS for agnhost-e2e-sample-app-7hxtd.apps.aro.v2i3p0q7g9o9o1k.cnfo.eastus2.aroapp-hcp.io did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-7hxtd.apps.aro.v2i3p0q7g9o9o1k.cnfo.eastus2.aroapp-hcp.io on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
err: <context.deadlineExceededError>{},
},
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_version_backlevel.go:201]: failed to verify simple web app on back-level cluster version 4.20
Unexpected error:
<*fmt.wrapError | 0xaa72ff3eb40>:
DNS for route host agnhost-e2e-sample-app-7hxtd.apps.aro.v2i3p0q7g9o9o1k.cnfo.eastus2.aroapp-hcp.io did not resolve: DNS for agnhost-e2e-sample-app-7hxtd.apps.aro.v2i3p0q7g9o9o1k.cnfo.eastus2.aroapp-hcp.io did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-7hxtd.apps.aro.v2i3p0q7g9o9o1k.cnfo.eastus2.aroapp-hcp.io on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded
{
msg: "DNS for route host agnhost-e2e-sample-app-7hxtd.apps.aro.v2i3p0q7g9o9o1k.cnfo.eastus2.aroapp-hcp.io did not resolve: DNS for agnhost-e2e-sample-app-7hxtd.apps.aro.v2i3p0q7g9o9o1k.cnfo.eastus2.aroapp-hcp.io did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-7hxtd.apps.aro.v2i3p0q7g9o9o1k.cnfo.eastus2.aroapp-hcp.io on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
err: <*fmt.wrapError | 0xaa72fd7c9e0>{
msg: "DNS for agnhost-e2e-sample-app-7hxtd.apps.aro.v2i3p0q7g9o9o1k.cnfo.eastus2.aroapp-hcp.io did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-7hxtd.apps.aro.v2i3p0q7g9o9o1k.cnfo.eastus2.aroapp-hcp.io on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
err: <context.deadlineExceededError>{},
},
}
occurredContributing tests (8)
Affected runs (3)
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedC...full failure pattern: ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable; hosted cluster degraded: UnavailableReplicas: multiple components; provider Microsoft.RedHatOpenShift Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 3 day(s) | e2e | 3 | 3.12%3 of 96 job runs affected | IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 3 day(s) | Show trend detailsOct 12: 1 · Oct 13: 0 · Oct 14: 0 · Oct 15: 0 · Oct 16: 0 · Oct 17: 0 · Oct 18: 0 · Oct 19: 0 · Oct 20: 2 · Oct 21: 0 · Oct 22: 9 · Oct 23: 0 · Oct 24: 0 · Oct 25: 0 | DEV, STG | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (3)fail [github.com/Azure/ARO-HCP/test/e2e/cluster_autoscaling.go:108]: failed to create HCP cluster "autoscaling-hcp-cluster" with custom autoscaling
Unexpected error:
<*fmt.wrapError | 0x19243183a940>:
failed to create HCP cluster autoscaling-hcp-cluster: failed waiting for cluster="autoscaling-hcp-cluster" in resourcegroup="autoscaling-cluster-df27xfnfzvrk" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/92eda184-8a06-461e-9d73-222caa436db8
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/92eda184-8a06-461e-9d73-222caa436db8",
"name": "92eda184-8a06-461e-9d73-222caa436db8",
"status": "Failed",
"startTime": "2026-10-22T01:34:06.303940712Z",
"endTime": "2026-10-22T02:03:06.782181855Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: ingress-operator, capi-provider, azure-cloud-controller-manager, csi-snapshot-controller-operator, catalog-operator, redhat-operators-catalog, certified-operators-catalog, cluster-image-registry-operator, control-plane-pki-operator, kube-controller-manager, machine-approver, cluster-network-operator, konnectivity-agent, redhat-marketplace-catalog, olm-operator, packageserver, cluster-policy-controller, kube-scheduler, cluster-version-operator, openshift-controller-manager, openshift-route-controller-manager, community-operators-catalog, hosted-cluster-config-operator, cluster-storage-operator, cluster-node-tuning-operator, dns-operator; hosted cluster degraded: UnavailableReplicas: [azure-cloud-controller-manager deployment has 1 unavailable replicas, capi-provider deployment has 2 unavailable replicas, catalog-operator deployment has 1 unavailable replicas, control-plane-pki-operator deployment has 1 unavailable replicas, hosted-cluster-config-operator deployment has 1 unavailable replicas, konnectivity-agent deployment has 1 unavailable replicas, kube-controller-manager deployment has 1 unavailable replicas, kube-scheduler deployment has 1 unavailable replicas, machine-approver deployment has 1 unavailable replicas, olm-operator deployment has 1 unavailable replicas, openshift-apiserver deployment has 1 unavailable replicas, packageserver deployment has 3 unavailable replicas]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/92eda184-8a06-461e-9d73-222caa436db8
correlation ID: 8174d018-ca0b-4942-ba1c-716f164824c9
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_autoscaling.go:108]: failed to create HCP cluster "autoscaling-hcp-cluster" with custom autoscaling
Unexpected error:
<*fmt.wrapError | 0x19243183a940>:
failed to create HCP cluster autoscaling-hcp-cluster: failed waiting for cluster="autoscaling-hcp-cluster" in resourcegroup="autoscaling-cluster-df27xfnfzvrk" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/92eda184-8a06-461e-9d73-222caa436db8
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/92eda184-8a06-461e-9d73-222caa436db8",
"name": "92eda184-8a06-461e-9d73-222caa436db8",
"status": "Failed",
"startTime": "2026-10-22T01:34:06.303940712Z",
"endTime": "2026-10-22T02:03:06.782181855Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: ingress-operator, capi-provider, azure-cloud-controller-manager, csi-snapshot-controller-operator, catalog-operator, redhat-operators-catalog, certified-operators-catalog, cluster-image-registry-operator, control-plane-pki-operator, kube-controller-manager, machine-approver, cluster-network-operator, konnectivity-agent, redhat-marketplace-catalog, olm-operator, packageserver, cluster-policy-controller, kube-scheduler, cluster-version-operator, openshift-controller-manager, openshift-route-controller-manager, community-operators-catalog, hosted-cluster-config-operator, cluster-storage-operator, cluster-node-tuning-operator, dns-operator; hosted cluster degraded: UnavailableReplicas: [azure-cloud-controller-manager deployment has 1 unavailable replicas, capi-provider deployment has 2 unavailable replicas, catalog-operator deployment has 1 unavailable replicas, control-plane-pki-operator deployment has 1 unavailable replicas, hosted-cluster-config-operator deployment has 1 unavailable replicas, konnectivity-agent deployment has 1 unavailable replicas, kube-controller-manager deployment has 1 unavailable replicas, kube-scheduler deployment has 1 unavailable replicas, machine-approver deployment has 1 unavailable replicas, olm-operator deployment has 1 unavailable replicas, openshift-apiserver deployment has 1 unavailable replicas, packageserver deployment has 3 unavailable replicas]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/92eda184-8a06-461e-9d73-222caa436db8
correlation ID: 8174d018-ca0b-4942-ba1c-716f164824c9
client request ID: unavailable
occurredfail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:163]: failed to create HCP cluster for patch-tags test
Unexpected error:
<*fmt.wrapError | 0x2d4d33b70620>:
failed to create HCP cluster patch-tags-cluster: failed waiting for cluster="patch-tags-cluster" in resourcegroup="patch-tags-b5r6x68bzp5t" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/a651d305-0bc4-4e50-8659-237a66cb9f85
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/a651d305-0bc4-4e50-8659-237a66cb9f85",
"name": "a651d305-0bc4-4e50-8659-237a66cb9f85",
"status": "Failed",
"startTime": "2026-10-22T02:20:28.664519088Z",
"endTime": "2026-10-22T02:49:30.367118005Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: openshift-route-controller-manager, control-plane-pki-operator, catalog-operator, olm-operator, openshift-controller-manager, hosted-cluster-config-operator, azure-cloud-controller-manager, cluster-network-operator, cluster-policy-controller, cluster-image-registry-operator, redhat-operators-catalog, redhat-marketplace-catalog, cluster-node-tuning-operator, machine-approver, dns-operator, ingress-operator, community-operators-catalog, kube-controller-manager, cluster-storage-operator, csi-snapshot-controller-operator, konnectivity-agent, certified-operators-catalog; hosted cluster degraded: UnavailableReplicas: [azure-cloud-controller-manager deployment has 1 unavailable replicas, catalog-operator deployment has 1 unavailable replicas, certified-operators-catalog deployment has 1 unavailable replicas, cluster-image-registry-operator deployment has 1 unavailable replicas, cluster-network-operator deployment has 1 unavailable replicas, cluster-node-tuning-operator deployment has 1 unavailable replicas, cluster-policy-controller deployment has 1 unavailable replicas, cluster-storage-operator deployment has 1 unavailable replicas, community-operators-catalog deployment has 1 unavailable replicas, control-plane-pki-operator deployment has 1 unavailable replicas, csi-snapshot-controller-operator deployment has 1 unavailable replicas, dns-operator deployment has 2 unavailable replicas, hosted-cluster-config-operator deployment has 1 unavailable replicas, ingress-operator deployment has 1 unavailable replicas, konnectivity-agent deployment has 1 unavailable replicas, kube-controller-manager deployment has 1 unavailable replicas, machine-approver deployment has 1 unavailable replicas, olm-operator deployment has 1 unavailable replicas, openshift-controller-manager deployment has 1 unavailable replicas, openshift-route-controller-manager deployment has 1 unavailable replicas, packageserver deployment has 1 unavailable replicas, redhat-marketplace-catalog deployment has 1 unavailable replicas, redhat-operators-catalog deployment has 1 unavailable replicas]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/a651d305-0bc4-4e50-8659-237a66cb9f85
correlation ID: 1caf4b05-a329-41fc-bcd5-7a64b3705d89
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:163]: failed to create HCP cluster for patch-tags test
Unexpected error:
<*fmt.wrapError | 0x2d4d33b70620>:
failed to create HCP cluster patch-tags-cluster: failed waiting for cluster="patch-tags-cluster" in resourcegroup="patch-tags-b5r6x68bzp5t" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/a651d305-0bc4-4e50-8659-237a66cb9f85
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/a651d305-0bc4-4e50-8659-237a66cb9f85",
"name": "a651d305-0bc4-4e50-8659-237a66cb9f85",
"status": "Failed",
"startTime": "2026-10-22T02:20:28.664519088Z",
"endTime": "2026-10-22T02:49:30.367118005Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: openshift-route-controller-manager, control-plane-pki-operator, catalog-operator, olm-operator, openshift-controller-manager, hosted-cluster-config-operator, azure-cloud-controller-manager, cluster-network-operator, cluster-policy-controller, cluster-image-registry-operator, redhat-operators-catalog, redhat-marketplace-catalog, cluster-node-tuning-operator, machine-approver, dns-operator, ingress-operator, community-operators-catalog, kube-controller-manager, cluster-storage-operator, csi-snapshot-controller-operator, konnectivity-agent, certified-operators-catalog; hosted cluster degraded: UnavailableReplicas: [azure-cloud-controller-manager deployment has 1 unavailable replicas, catalog-operator deployment has 1 unavailable replicas, certified-operators-catalog deployment has 1 unavailable replicas, cluster-image-registry-operator deployment has 1 unavailable replicas, cluster-network-operator deployment has 1 unavailable replicas, cluster-node-tuning-operator deployment has 1 unavailable replicas, cluster-policy-controller deployment has 1 unavailable replicas, cluster-storage-operator deployment has 1 unavailable replicas, community-operators-catalog deployment has 1 unavailable replicas, control-plane-pki-operator deployment has 1 unavailable replicas, csi-snapshot-controller-operator deployment has 1 unavailable replicas, dns-operator deployment has 2 unavailable replicas, hosted-cluster-config-operator deployment has 1 unavailable replicas, ingress-operator deployment has 1 unavailable replicas, konnectivity-agent deployment has 1 unavailable replicas, kube-controller-manager deployment has 1 unavailable replicas, machine-approver deployment has 1 unavailable replicas, olm-operator deployment has 1 unavailable replicas, openshift-controller-manager deployment has 1 unavailable replicas, openshift-route-controller-manager deployment has 1 unavailable replicas, packageserver deployment has 1 unavailable replicas, redhat-marketplace-catalog deployment has 1 unavailable replicas, redhat-operators-catalog deployment has 1 unavailable replicas]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/a651d305-0bc4-4e50-8659-237a66cb9f85
correlation ID: 1caf4b05-a329-41fc-bcd5-7a64b3705d89
client request ID: unavailable
occurredfail [github.com/Azure/ARO-HCP/test/e2e/nodepool_osdisk_encryption.go:127]: failed to create HCP cluster des-encrypt
Unexpected error:
<*fmt.wrapError | 0x119df757e3e0>:
failed to create HCP cluster des-encrypt: failed waiting for cluster="des-encrypt" in resourcegroup="des-encrypt-hzm47sjhc426" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/6a6c258e-71e6-4920-8857-17ab7dc1e45f
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/6a6c258e-71e6-4920-8857-17ab7dc1e45f",
"name": "6a6c258e-71e6-4920-8857-17ab7dc1e45f",
"status": "Failed",
"startTime": "2026-10-22T03:09:48.207242699Z",
"endTime": "2026-10-22T03:38:55.0934354Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: cluster-version-operator, dns-operator, hosted-cluster-config-operator, azure-cloud-controller-manager, openshift-controller-manager, cluster-node-tuning-operator, catalog-operator, cluster-storage-operator, cluster-network-operator, cluster-image-registry-operator, certified-operators-catalog, csi-snapshot-controller-operator, konnectivity-agent, community-operators-catalog, kube-controller-manager, kube-scheduler, olm-operator, redhat-marketplace-catalog, openshift-route-controller-manager, cluster-policy-controller, packageserver, redhat-operators-catalog, control-plane-pki-operator, ingress-operator, machine-approver; hosted cluster degraded: UnavailableReplicas: [azure-cloud-controller-manager deployment has 1 unavailable replicas, catalog-operator deployment has 1 unavailable replicas, control-plane-pki-operator deployment has 1 unavailable replicas, hosted-cluster-config-operator deployment has 1 unavailable replicas, konnectivity-agent deployment has 1 unavailable replicas, kube-controller-manager deployment has 1 unavailable replicas, kube-scheduler deployment has 1 unavailable replicas, machine-approver deployment has 1 unavailable replicas, olm-operator deployment has 1 unavailable replicas, openshift-apiserver deployment has 1 unavailable replicas, packageserver deployment has 3 unavailable replicas]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/6a6c258e-71e6-4920-8857-17ab7dc1e45f
correlation ID: 80334f53-9a3b-4203-a7f4-9d0844089824
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_osdisk_encryption.go:127]: failed to create HCP cluster des-encrypt
Unexpected error:
<*fmt.wrapError | 0x119df757e3e0>:
failed to create HCP cluster des-encrypt: failed waiting for cluster="des-encrypt" in resourcegroup="des-encrypt-hzm47sjhc426" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/6a6c258e-71e6-4920-8857-17ab7dc1e45f
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/6a6c258e-71e6-4920-8857-17ab7dc1e45f",
"name": "6a6c258e-71e6-4920-8857-17ab7dc1e45f",
"status": "Failed",
"startTime": "2026-10-22T03:09:48.207242699Z",
"endTime": "2026-10-22T03:38:55.0934354Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: cluster-version-operator, dns-operator, hosted-cluster-config-operator, azure-cloud-controller-manager, openshift-controller-manager, cluster-node-tuning-operator, catalog-operator, cluster-storage-operator, cluster-network-operator, cluster-image-registry-operator, certified-operators-catalog, csi-snapshot-controller-operator, konnectivity-agent, community-operators-catalog, kube-controller-manager, kube-scheduler, olm-operator, redhat-marketplace-catalog, openshift-route-controller-manager, cluster-policy-controller, packageserver, redhat-operators-catalog, control-plane-pki-operator, ingress-operator, machine-approver; hosted cluster degraded: UnavailableReplicas: [azure-cloud-controller-manager deployment has 1 unavailable replicas, catalog-operator deployment has 1 unavailable replicas, control-plane-pki-operator deployment has 1 unavailable replicas, hosted-cluster-config-operator deployment has 1 unavailable replicas, konnectivity-agent deployment has 1 unavailable replicas, kube-controller-manager deployment has 1 unavailable replicas, kube-scheduler deployment has 1 unavailable replicas, machine-approver deployment has 1 unavailable replicas, olm-operator deployment has 1 unavailable replicas, openshift-apiserver deployment has 1 unavailable replicas, packageserver deployment has 3 unavailable replicas]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/6a6c258e-71e6-4920-8857-17ab7dc1e45f
correlation ID: 80334f53-9a3b-4203-a7f4-9d0844089824
client request ID: unavailable
occurredContributing tests (11)
Affected runs (3)
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
CreateOrUpdateExternalAuthAndWait timed out after <minutes> minutes; context deadline exceededfull failure pattern: CreateOrUpdateExternalAuthAndWait timed out after <minutes> minutes; context deadline exceeded Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 2 day(s) | e2e | 3 | 3.12%3 of 96 job runs affected | IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 2 day(s) | Show trend detailsOct 12: 0 · Oct 13: 0 · Oct 14: 2 · Oct 15: 1 · Oct 16: 0 · Oct 17: 0 · Oct 18: 0 · Oct 19: 0 · Oct 20: 0 · Oct 21: 0 · Oct 22: 0 · Oct 23: 0 · Oct 24: 0 · Oct 25: 0 | INT, STG | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (3)fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:262]: failed to apply external auth config with prefix policy None
Unexpected error:
<*fmt.wrapErrors | 0x1d3b45812a50>:
failed waiting for external auth "external-auth" in resourcegroup="external-auth-cluster-5lx5864xtdrb" for cluster="ea-cluster" to finish creating or updating, caused by: timeout '15.000000' minutes exceeded during CreateOrUpdateExternalAuthAndWait for external auth external-auth in cluster ea-cluster in resource group external-auth-cluster-5lx5864xtdrb, error: context deadline exceeded
{
msg: "failed waiting for external auth \"external-auth\" in resourcegroup=\"external-auth-cluster-5lx5864xtdrb\" for cluster=\"ea-cluster\" to finish creating or updating, caused by: timeout '15.000000' minutes exceeded during CreateOrUpdateExternalAuthAndWait for external auth external-auth in cluster ea-cluster in resource group external-auth-cluster-5lx5864xtdrb, error: context deadline exceeded",
errs: [
<*errors.errorString | 0x1d3b4591ca00>{
s: "timeout '15.000000' minutes exceeded during CreateOrUpdateExternalAuthAndWait for external auth external-auth in cluster ea-cluster in resource group external-auth-cluster-5lx5864xtdrb",
},
<context.deadlineExceededError>{},
],
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:262]: failed to apply external auth config with prefix policy None
Unexpected error:
<*fmt.wrapErrors | 0x1d3b45812a50>:
failed waiting for external auth "external-auth" in resourcegroup="external-auth-cluster-5lx5864xtdrb" for cluster="ea-cluster" to finish creating or updating, caused by: timeout '15.000000' minutes exceeded during CreateOrUpdateExternalAuthAndWait for external auth external-auth in cluster ea-cluster in resource group external-auth-cluster-5lx5864xtdrb, error: context deadline exceeded
{
msg: "failed waiting for external auth \"external-auth\" in resourcegroup=\"external-auth-cluster-5lx5864xtdrb\" for cluster=\"ea-cluster\" to finish creating or updating, caused by: timeout '15.000000' minutes exceeded during CreateOrUpdateExternalAuthAndWait for external auth external-auth in cluster ea-cluster in resource group external-auth-cluster-5lx5864xtdrb, error: context deadline exceeded",
errs: [
<*errors.errorString | 0x1d3b4591ca00>{
s: "timeout '15.000000' minutes exceeded during CreateOrUpdateExternalAuthAndWait for external auth external-auth in cluster ea-cluster in resource group external-auth-cluster-5lx5864xtdrb",
},
<context.deadlineExceededError>{},
],
}
occurredfail [github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:262]: failed to apply external auth config with prefix policy None
Unexpected error:
<*fmt.wrapErrors | 0x2be35a9dc6c0>:
failed waiting for external auth "external-auth" in resourcegroup="external-auth-cluster-4swfh9bhdjlh" for cluster="ea-cluster" to finish creating or updating, caused by: timeout '15.000000' minutes exceeded during CreateOrUpdateExternalAuthAndWait for external auth external-auth in cluster ea-cluster in resource group external-auth-cluster-4swfh9bhdjlh, error: context deadline exceeded
{
msg: "failed waiting for external auth \"external-auth\" in resourcegroup=\"external-auth-cluster-4swfh9bhdjlh\" for cluster=\"ea-cluster\" to finish creating or updating, caused by: timeout '15.000000' minutes exceeded during CreateOrUpdateExternalAuthAndWait for external auth external-auth in cluster ea-cluster in resource group external-auth-cluster-4swfh9bhdjlh, error: context deadline exceeded",
errs: [
<*errors.errorString | 0x2be35bcc7190>{
s: "timeout '15.000000' minutes exceeded during CreateOrUpdateExternalAuthAndWait for external auth external-auth in cluster ea-cluster in resource group external-auth-cluster-4swfh9bhdjlh",
},
<context.deadlineExceededError>{},
],
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:262]: failed to apply external auth config with prefix policy None
Unexpected error:
<*fmt.wrapErrors | 0x2be35a9dc6c0>:
failed waiting for external auth "external-auth" in resourcegroup="external-auth-cluster-4swfh9bhdjlh" for cluster="ea-cluster" to finish creating or updating, caused by: timeout '15.000000' minutes exceeded during CreateOrUpdateExternalAuthAndWait for external auth external-auth in cluster ea-cluster in resource group external-auth-cluster-4swfh9bhdjlh, error: context deadline exceeded
{
msg: "failed waiting for external auth \"external-auth\" in resourcegroup=\"external-auth-cluster-4swfh9bhdjlh\" for cluster=\"ea-cluster\" to finish creating or updating, caused by: timeout '15.000000' minutes exceeded during CreateOrUpdateExternalAuthAndWait for external auth external-auth in cluster ea-cluster in resource group external-auth-cluster-4swfh9bhdjlh, error: context deadline exceeded",
errs: [
<*errors.errorString | 0x2be35bcc7190>{
s: "timeout '15.000000' minutes exceeded during CreateOrUpdateExternalAuthAndWait for external auth external-auth in cluster ea-cluster in resource group external-auth-cluster-4swfh9bhdjlh",
},
<context.deadlineExceededError>{},
],
}
occurredfail [github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:262]: failed to apply external auth config with prefix policy None
Unexpected error:
<*fmt.wrapErrors | 0x130faa8f320>:
failed waiting for external auth "external-auth" in resourcegroup="external-auth-cluster-bkm7lwv87ztr" for cluster="ea-cluster" to finish creating or updating, caused by: timeout '15.000000' minutes exceeded during CreateOrUpdateExternalAuthAndWait for external auth external-auth in cluster ea-cluster in resource group external-auth-cluster-bkm7lwv87ztr, error: context deadline exceeded
{
msg: "failed waiting for external auth \"external-auth\" in resourcegroup=\"external-auth-cluster-bkm7lwv87ztr\" for cluster=\"ea-cluster\" to finish creating or updating, caused by: timeout '15.000000' minutes exceeded during CreateOrUpdateExternalAuthAndWait for external auth external-auth in cluster ea-cluster in resource group external-auth-cluster-bkm7lwv87ztr, error: context deadline exceeded",
errs: [
<*errors.errorString | 0x130f942f8e0>{
s: "timeout '15.000000' minutes exceeded during CreateOrUpdateExternalAuthAndWait for external auth external-auth in cluster ea-cluster in resource group external-auth-cluster-bkm7lwv87ztr",
},
<context.deadlineExceededError>{},
],
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:262]: failed to apply external auth config with prefix policy None
Unexpected error:
<*fmt.wrapErrors | 0x130faa8f320>:
failed waiting for external auth "external-auth" in resourcegroup="external-auth-cluster-bkm7lwv87ztr" for cluster="ea-cluster" to finish creating or updating, caused by: timeout '15.000000' minutes exceeded during CreateOrUpdateExternalAuthAndWait for external auth external-auth in cluster ea-cluster in resource group external-auth-cluster-bkm7lwv87ztr, error: context deadline exceeded
{
msg: "failed waiting for external auth \"external-auth\" in resourcegroup=\"external-auth-cluster-bkm7lwv87ztr\" for cluster=\"ea-cluster\" to finish creating or updating, caused by: timeout '15.000000' minutes exceeded during CreateOrUpdateExternalAuthAndWait for external auth external-auth in cluster ea-cluster in resource group external-auth-cluster-bkm7lwv87ztr, error: context deadline exceeded",
errs: [
<*errors.errorString | 0x130f942f8e0>{
s: "timeout '15.000000' minutes exceeded during CreateOrUpdateExternalAuthAndWait for external auth external-auth in cluster ea-cluster in resource group external-auth-cluster-bkm7lwv87ztr",
},
<context.deadlineExceededError>{},
],
}
occurredContributing tests (2)
Affected runs (3)
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
DNS for route host agnhost-e2e-sample-app-<id>full failure pattern: DNS for route host agnhost-e2e-sample-app-<id> Signal: IndeterminateSignal: Indeterminate — no prior history; active 2 day(s) — New failure pattern — no prior history | e2e | 3 | 3.12%3 of 96 job runs affected | IndeterminateSignal: Indeterminate — no prior history; active 2 day(s) — New failure pattern — no prior history | Show trend detailsOct 12: 0 · Oct 13: 0 · Oct 14: 0 · Oct 15: 0 · Oct 16: 0 · Oct 17: 0 · Oct 18: 0 · Oct 19: 0 · Oct 20: 0 · Oct 21: 0 · Oct 22: 0 · Oct 23: 2 · Oct 24: 0 · Oct 25: 1 | DEV, INT | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (3)fail [github.com/Azure/ARO-HCP/test/e2e/default_cluster_ingress_cert.go:169]: sample application must serve the delivered default certificate
Expected success, but got an error:
<*fmt.wrapError | 0x1d8dba328160>:
DNS for route host agnhost-e2e-sample-app-52rjm.apps.aro.n5x4o1n9j0c5w9n.rqrz.eastus2.aroapp-hcp.io did not resolve: DNS for agnhost-e2e-sample-app-52rjm.apps.aro.n5x4o1n9j0c5w9n.rqrz.eastus2.aroapp-hcp.io did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-52rjm.apps.aro.n5x4o1n9j0c5w9n.rqrz.eastus2.aroapp-hcp.io on 172.30.0.10:53: no such host): context deadline exceeded
{
msg: "DNS for route host agnhost-e2e-sample-app-52rjm.apps.aro.n5x4o1n9j0c5w9n.rqrz.eastus2.aroapp-hcp.io did not resolve: DNS for agnhost-e2e-sample-app-52rjm.apps.aro.n5x4o1n9j0c5w9n.rqrz.eastus2.aroapp-hcp.io did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-52rjm.apps.aro.n5x4o1n9j0c5w9n.rqrz.eastus2.aroapp-hcp.io on 172.30.0.10:53: no such host): context deadline exceeded",
err: <*fmt.wrapError | 0x1d8dba3280e0>{
msg: "DNS for agnhost-e2e-sample-app-52rjm.apps.aro.n5x4o1n9j0c5w9n.rqrz.eastus2.aroapp-hcp.io did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-52rjm.apps.aro.n5x4o1n9j0c5w9n.rqrz.eastus2.aroapp-hcp.io on 172.30.0.10:53: no such host): context deadline exceeded",
err: <context.deadlineExceededError>{},
},
}
fail [github.com/Azure/ARO-HCP/test/e2e/default_cluster_ingress_cert.go:169]: sample application must serve the delivered default certificate
Expected success, but got an error:
<*fmt.wrapError | 0x1d8dba328160>:
DNS for route host agnhost-e2e-sample-app-52rjm.apps.aro.n5x4o1n9j0c5w9n.rqrz.eastus2.aroapp-hcp.io did not resolve: DNS for agnhost-e2e-sample-app-52rjm.apps.aro.n5x4o1n9j0c5w9n.rqrz.eastus2.aroapp-hcp.io did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-52rjm.apps.aro.n5x4o1n9j0c5w9n.rqrz.eastus2.aroapp-hcp.io on 172.30.0.10:53: no such host): context deadline exceeded
{
msg: "DNS for route host agnhost-e2e-sample-app-52rjm.apps.aro.n5x4o1n9j0c5w9n.rqrz.eastus2.aroapp-hcp.io did not resolve: DNS for agnhost-e2e-sample-app-52rjm.apps.aro.n5x4o1n9j0c5w9n.rqrz.eastus2.aroapp-hcp.io did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-52rjm.apps.aro.n5x4o1n9j0c5w9n.rqrz.eastus2.aroapp-hcp.io on 172.30.0.10:53: no such host): context deadline exceeded",
err: <*fmt.wrapError | 0x1d8dba3280e0>{
msg: "DNS for agnhost-e2e-sample-app-52rjm.apps.aro.n5x4o1n9j0c5w9n.rqrz.eastus2.aroapp-hcp.io did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-52rjm.apps.aro.n5x4o1n9j0c5w9n.rqrz.eastus2.aroapp-hcp.io on 172.30.0.10:53: no such host): context deadline exceeded",
err: <context.deadlineExceededError>{},
},
}fail [github.com/Azure/ARO-HCP/test/e2e/default_cluster_ingress_cert.go:169]: sample application must serve the delivered default certificate
Expected success, but got an error:
<*fmt.wrapError | 0x2aeb303488e0>:
DNS for route host agnhost-e2e-sample-app-g4jkv.apps.aro.s7l8k8t7p1f9x2x.w4l4.eastus2.aroapp-hcp.io did not resolve: DNS for agnhost-e2e-sample-app-g4jkv.apps.aro.s7l8k8t7p1f9x2x.w4l4.eastus2.aroapp-hcp.io did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-g4jkv.apps.aro.s7l8k8t7p1f9x2x.w4l4.eastus2.aroapp-hcp.io on 172.30.0.10:53: no such host): context deadline exceeded
{
msg: "DNS for route host agnhost-e2e-sample-app-g4jkv.apps.aro.s7l8k8t7p1f9x2x.w4l4.eastus2.aroapp-hcp.io did not resolve: DNS for agnhost-e2e-sample-app-g4jkv.apps.aro.s7l8k8t7p1f9x2x.w4l4.eastus2.aroapp-hcp.io did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-g4jkv.apps.aro.s7l8k8t7p1f9x2x.w4l4.eastus2.aroapp-hcp.io on 172.30.0.10:53: no such host): context deadline exceeded",
err: <*fmt.wrapError | 0x2aeb303488c0>{
msg: "DNS for agnhost-e2e-sample-app-g4jkv.apps.aro.s7l8k8t7p1f9x2x.w4l4.eastus2.aroapp-hcp.io did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-g4jkv.apps.aro.s7l8k8t7p1f9x2x.w4l4.eastus2.aroapp-hcp.io on 172.30.0.10:53: no such host): context deadline exceeded",
err: <context.deadlineExceededError>{},
},
}
fail [github.com/Azure/ARO-HCP/test/e2e/default_cluster_ingress_cert.go:169]: sample application must serve the delivered default certificate
Expected success, but got an error:
<*fmt.wrapError | 0x2aeb303488e0>:
DNS for route host agnhost-e2e-sample-app-g4jkv.apps.aro.s7l8k8t7p1f9x2x.w4l4.eastus2.aroapp-hcp.io did not resolve: DNS for agnhost-e2e-sample-app-g4jkv.apps.aro.s7l8k8t7p1f9x2x.w4l4.eastus2.aroapp-hcp.io did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-g4jkv.apps.aro.s7l8k8t7p1f9x2x.w4l4.eastus2.aroapp-hcp.io on 172.30.0.10:53: no such host): context deadline exceeded
{
msg: "DNS for route host agnhost-e2e-sample-app-g4jkv.apps.aro.s7l8k8t7p1f9x2x.w4l4.eastus2.aroapp-hcp.io did not resolve: DNS for agnhost-e2e-sample-app-g4jkv.apps.aro.s7l8k8t7p1f9x2x.w4l4.eastus2.aroapp-hcp.io did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-g4jkv.apps.aro.s7l8k8t7p1f9x2x.w4l4.eastus2.aroapp-hcp.io on 172.30.0.10:53: no such host): context deadline exceeded",
err: <*fmt.wrapError | 0x2aeb303488c0>{
msg: "DNS for agnhost-e2e-sample-app-g4jkv.apps.aro.s7l8k8t7p1f9x2x.w4l4.eastus2.aroapp-hcp.io did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-g4jkv.apps.aro.s7l8k8t7p1f9x2x.w4l4.eastus2.aroapp-hcp.io on 172.30.0.10:53: no such host): context deadline exceeded",
err: <context.deadlineExceededError>{},
},
}fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:526]: sample application must serve the exact customer-specified certificate with a valid Let's Encrypt chain
Expected success, but got an error:
<*fmt.wrapError | 0x33122d4ce1c0>:
DNS for route host agnhost-e2e-sample-app-2lr49.apps.aro.ingress-cert-le.xc5o.eastus2.aroapp-hcp.io did not resolve: DNS for agnhost-e2e-sample-app-2lr49.apps.aro.ingress-cert-le.xc5o.eastus2.aroapp-hcp.io did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-2lr49.apps.aro.ingress-cert-le.xc5o.eastus2.aroapp-hcp.io on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded
{
msg: "DNS for route host agnhost-e2e-sample-app-2lr49.apps.aro.ingress-cert-le.xc5o.eastus2.aroapp-hcp.io did not resolve: DNS for agnhost-e2e-sample-app-2lr49.apps.aro.ingress-cert-le.xc5o.eastus2.aroapp-hcp.io did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-2lr49.apps.aro.ingress-cert-le.xc5o.eastus2.aroapp-hcp.io on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
err: <*fmt.wrapError | 0x33122d4ce1a0>{
msg: "DNS for agnhost-e2e-sample-app-2lr49.apps.aro.ingress-cert-le.xc5o.eastus2.aroapp-hcp.io did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-2lr49.apps.aro.ingress-cert-le.xc5o.eastus2.aroapp-hcp.io on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
err: <context.deadlineExceededError>{},
},
}
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:526]: sample application must serve the exact customer-specified certificate with a valid Let's Encrypt chain
Expected success, but got an error:
<*fmt.wrapError | 0x33122d4ce1c0>:
DNS for route host agnhost-e2e-sample-app-2lr49.apps.aro.ingress-cert-le.xc5o.eastus2.aroapp-hcp.io did not resolve: DNS for agnhost-e2e-sample-app-2lr49.apps.aro.ingress-cert-le.xc5o.eastus2.aroapp-hcp.io did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-2lr49.apps.aro.ingress-cert-le.xc5o.eastus2.aroapp-hcp.io on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded
{
msg: "DNS for route host agnhost-e2e-sample-app-2lr49.apps.aro.ingress-cert-le.xc5o.eastus2.aroapp-hcp.io did not resolve: DNS for agnhost-e2e-sample-app-2lr49.apps.aro.ingress-cert-le.xc5o.eastus2.aroapp-hcp.io did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-2lr49.apps.aro.ingress-cert-le.xc5o.eastus2.aroapp-hcp.io on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
err: <*fmt.wrapError | 0x33122d4ce1a0>{
msg: "DNS for agnhost-e2e-sample-app-2lr49.apps.aro.ingress-cert-le.xc5o.eastus2.aroapp-hcp.io did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-2lr49.apps.aro.ingress-cert-le.xc5o.eastus2.aroapp-hcp.io on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
err: <context.deadlineExceededError>{},
},
}Contributing tests (2)
Affected runs (3)
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedC...full failure pattern: ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider, ignition-server]; provider Microsoft.RedHatOpenShift Signal: IndeterminateSignal: Indeterminate — no prior history; active 2 day(s) — New failure pattern — no prior history | e2e | 2 | 2.08%2 of 96 job runs affected | IndeterminateSignal: Indeterminate — no prior history; active 2 day(s) — New failure pattern — no prior history | Show trend detailsOct 12: 0 · Oct 13: 0 · Oct 14: 0 · Oct 15: 0 · Oct 16: 0 · Oct 17: 0 · Oct 18: 0 · Oct 19: 0 · Oct 20: 12 · Oct 21: 0 · Oct 22: 7 · Oct 23: 0 · Oct 24: 0 · Oct 25: 0 | none | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (3)fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:86]: failed to create HCP cluster for patch-name test
Unexpected error:
<*fmt.wrapError | 0x12e0378a8e0>:
failed to create HCP cluster patch-name-cluster: failed waiting for cluster="patch-name-cluster" in resourcegroup="patch-name-s6hb8822zv5x" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/b0fcfc1d-56f0-4a0e-be27-eb59d78d3391
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/b0fcfc1d-56f0-4a0e-be27-eb59d78d3391",
"name": "b0fcfc1d-56f0-4a0e-be27-eb59d78d3391",
"status": "Failed",
"startTime": "2026-10-22T01:34:12.606084954Z",
"endTime": "2026-10-22T02:03:16.732035531Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider deployment has 1 unavailable replicas, ignition-server deployment has 1 unavailable replicas]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/b0fcfc1d-56f0-4a0e-be27-eb59d78d3391
correlation ID: d29a315a-5b37-4a91-af7b-1415cfe4e5ec
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:86]: failed to create HCP cluster for patch-name test
Unexpected error:
<*fmt.wrapError | 0x12e0378a8e0>:
failed to create HCP cluster patch-name-cluster: failed waiting for cluster="patch-name-cluster" in resourcegroup="patch-name-s6hb8822zv5x" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/b0fcfc1d-56f0-4a0e-be27-eb59d78d3391
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/b0fcfc1d-56f0-4a0e-be27-eb59d78d3391",
"name": "b0fcfc1d-56f0-4a0e-be27-eb59d78d3391",
"status": "Failed",
"startTime": "2026-10-22T01:34:12.606084954Z",
"endTime": "2026-10-22T02:03:16.732035531Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider deployment has 1 unavailable replicas, ignition-server deployment has 1 unavailable replicas]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/b0fcfc1d-56f0-4a0e-be27-eb59d78d3391
correlation ID: d29a315a-5b37-4a91-af7b-1415cfe4e5ec
client request ID: unavailable
occurredfail [github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:98]: failed to create HCP cluster for external auth test
Unexpected error:
<*fmt.wrapError | 0x33ca21720880>:
failed to create HCP cluster ea-cluster: failed waiting for cluster="ea-cluster" in resourcegroup="external-auth-cluster-6zgdp7hhpk7k" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/c707ea6e-b74b-443a-9b32-fe7f80a738f2
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/c707ea6e-b74b-443a-9b32-fe7f80a738f2",
"name": "c707ea6e-b74b-443a-9b32-fe7f80a738f2",
"status": "Failed",
"startTime": "2026-10-22T02:25:00.37199898Z",
"endTime": "2026-10-22T02:54:00.386898896Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider deployment has 1 unavailable replicas, ignition-server deployment has 1 unavailable replicas]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/c707ea6e-b74b-443a-9b32-fe7f80a738f2
correlation ID: cd0ac0e6-4a75-4391-a2be-e65722141c43
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:98]: failed to create HCP cluster for external auth test
Unexpected error:
<*fmt.wrapError | 0x33ca21720880>:
failed to create HCP cluster ea-cluster: failed waiting for cluster="ea-cluster" in resourcegroup="external-auth-cluster-6zgdp7hhpk7k" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/c707ea6e-b74b-443a-9b32-fe7f80a738f2
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/c707ea6e-b74b-443a-9b32-fe7f80a738f2",
"name": "c707ea6e-b74b-443a-9b32-fe7f80a738f2",
"status": "Failed",
"startTime": "2026-10-22T02:25:00.37199898Z",
"endTime": "2026-10-22T02:54:00.386898896Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider deployment has 1 unavailable replicas, ignition-server deployment has 1 unavailable replicas]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/c707ea6e-b74b-443a-9b32-fe7f80a738f2
correlation ID: cd0ac0e6-4a75-4391-a2be-e65722141c43
client request ID: unavailable
occurredfail [github.com/Azure/ARO-HCP/test/e2e/simple_negative_cases.go:110]: failed to create HCP cluster negative-tests-cluster
Unexpected error:
<*fmt.wrapError | 0x363c02e8080>:
failed to create HCP cluster negative-tests-cluster: failed waiting for cluster="negative-tests-cluster" in resourcegroup="rg-negative-tests-jbrqpz7ctxtq" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/21e6bfc0-fe4d-4dac-b056-c840f710f982
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/21e6bfc0-fe4d-4dac-b056-c840f710f982",
"name": "21e6bfc0-fe4d-4dac-b056-c840f710f982",
"status": "Failed",
"startTime": "2026-10-22T03:11:29.420591363Z",
"endTime": "2026-10-22T03:40:35.058626079Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider deployment has 1 unavailable replicas, ignition-server deployment has 1 unavailable replicas]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/21e6bfc0-fe4d-4dac-b056-c840f710f982
correlation ID: c21da3fc-8f7d-45b1-bbc4-5d3a6ee72767
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/simple_negative_cases.go:110]: failed to create HCP cluster negative-tests-cluster
Unexpected error:
<*fmt.wrapError | 0x363c02e8080>:
failed to create HCP cluster negative-tests-cluster: failed waiting for cluster="negative-tests-cluster" in resourcegroup="rg-negative-tests-jbrqpz7ctxtq" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/21e6bfc0-fe4d-4dac-b056-c840f710f982
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/21e6bfc0-fe4d-4dac-b056-c840f710f982",
"name": "21e6bfc0-fe4d-4dac-b056-c840f710f982",
"status": "Failed",
"startTime": "2026-10-22T03:11:29.420591363Z",
"endTime": "2026-10-22T03:40:35.058626079Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider deployment has 1 unavailable replicas, ignition-server deployment has 1 unavailable replicas]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/21e6bfc0-fe4d-4dac-b056-c840f710f982
correlation ID: c21da3fc-8f7d-45b1-bbc4-5d3a6ee72767
client request ID: unavailable
occurredContributing tests (15)
Affected runs (2)
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedC...full failure pattern: ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable; hosted cluster degraded: UnavailableReplicas: [kube-apiserver]; provider Microsoft.RedHatOpenShift Signal: IndeterminateSignal: Indeterminate — no prior history; active 2 day(s) — New failure pattern — no prior history | e2e | 2 | 2.08%2 of 96 job runs affected | IndeterminateSignal: Indeterminate — no prior history; active 2 day(s) — New failure pattern — no prior history | Show trend detailsOct 12: 0 · Oct 13: 0 · Oct 14: 0 · Oct 15: 0 · Oct 16: 0 · Oct 17: 0 · Oct 18: 0 · Oct 19: 0 · Oct 20: 2 · Oct 21: 0 · Oct 22: 9 · Oct 23: 0 · Oct 24: 0 · Oct 25: 0 | none | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (3)fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kas.go:101]: failed to create HCP cluster "private-kas" with private KAS
Unexpected error:
<*fmt.wrapError | 0x2951a1afed40>:
failed to create HCP cluster private-kas: failed waiting for cluster="private-kas" in resourcegroup="private-kas-hfpdp8lmxh49" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/0119dd29-201c-4c22-9ebd-7d3a5b8701fc
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/0119dd29-201c-4c22-9ebd-7d3a5b8701fc",
"name": "0119dd29-201c-4c22-9ebd-7d3a5b8701fc",
"status": "Failed",
"startTime": "2026-10-22T01:34:06.130621039Z",
"endTime": "2026-10-22T02:03:06.668308317Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: olm-collect-profiles, openshift-apiserver, openshift-route-controller-manager, dns-operator, csi-snapshot-controller-operator, kube-scheduler, cluster-policy-controller, packageserver, certified-operators-catalog, hosted-cluster-config-operator, cluster-version-operator, openshift-controller-manager, olm-operator, community-operators-catalog, machine-approver, control-plane-pki-operator, cluster-network-operator, konnectivity-agent, kube-storage-version-migrator, catalog-operator, cluster-storage-operator, redhat-operators-catalog, cluster-autoscaler, kube-controller-manager, azure-cloud-controller-manager, cluster-node-tuning-operator, ingress-operator, cluster-image-registry-operator; hosted cluster degraded: UnavailableReplicas: kube-apiserver deployment has 1 unavailable replicas"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/0119dd29-201c-4c22-9ebd-7d3a5b8701fc
correlation ID: 909da2c2-f472-4c31-8509-d71d62f94fb1
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kas.go:101]: failed to create HCP cluster "private-kas" with private KAS
Unexpected error:
<*fmt.wrapError | 0x2951a1afed40>:
failed to create HCP cluster private-kas: failed waiting for cluster="private-kas" in resourcegroup="private-kas-hfpdp8lmxh49" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/0119dd29-201c-4c22-9ebd-7d3a5b8701fc
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/0119dd29-201c-4c22-9ebd-7d3a5b8701fc",
"name": "0119dd29-201c-4c22-9ebd-7d3a5b8701fc",
"status": "Failed",
"startTime": "2026-10-22T01:34:06.130621039Z",
"endTime": "2026-10-22T02:03:06.668308317Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: olm-collect-profiles, openshift-apiserver, openshift-route-controller-manager, dns-operator, csi-snapshot-controller-operator, kube-scheduler, cluster-policy-controller, packageserver, certified-operators-catalog, hosted-cluster-config-operator, cluster-version-operator, openshift-controller-manager, olm-operator, community-operators-catalog, machine-approver, control-plane-pki-operator, cluster-network-operator, konnectivity-agent, kube-storage-version-migrator, catalog-operator, cluster-storage-operator, redhat-operators-catalog, cluster-autoscaler, kube-controller-manager, azure-cloud-controller-manager, cluster-node-tuning-operator, ingress-operator, cluster-image-registry-operator; hosted cluster degraded: UnavailableReplicas: kube-apiserver deployment has 1 unavailable replicas"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/0119dd29-201c-4c22-9ebd-7d3a5b8701fc
correlation ID: 909da2c2-f472-4c31-8509-d71d62f94fb1
client request ID: unavailable
occurredfail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:124]: HCP cluster rg-candidate-5-1-2s5zlgwb6bct/cluster-candidate-5-1-84zdqd should provision
Unexpected error:
<*fmt.wrapError | 0x16cd8e8f9880>:
failed to create HCP cluster cluster-candidate-5-1-84zdqd: failed waiting for cluster="cluster-candidate-5-1-84zdqd" in resourcegroup="rg-candidate-5-1-2s5zlgwb6bct" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/9d897978-69ec-403d-ad9b-fd32c33747d4
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/9d897978-69ec-403d-ad9b-fd32c33747d4",
"name": "9d897978-69ec-403d-ad9b-fd32c33747d4",
"status": "Failed",
"startTime": "2026-10-22T02:21:55.892991844Z",
"endTime": "2026-10-22T02:51:00.378883542Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: machine-approver, csi-snapshot-controller-operator, cluster-image-registry-operator, catalog-operator, kube-scheduler, dns-operator, konnectivity-agent, kube-controller-manager, cluster-storage-operator, cluster-autoscaler, cluster-policy-controller, hosted-cluster-config-operator, olm-operator, olm-collect-profiles, community-operators-catalog, cluster-version-operator, kube-storage-version-migrator, packageserver, control-plane-pki-operator, openshift-controller-manager, azure-cloud-controller-manager, openshift-apiserver, openshift-route-controller-manager, redhat-operators-catalog, certified-operators-catalog, cluster-network-operator, cluster-node-tuning-operator, ingress-operator; hosted cluster degraded: UnavailableReplicas: kube-apiserver deployment has 1 unavailable replicas"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/9d897978-69ec-403d-ad9b-fd32c33747d4
correlation ID: bdf3de8a-4b44-40d3-b952-4e50b1bc65e3
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:124]: HCP cluster rg-candidate-5-1-2s5zlgwb6bct/cluster-candidate-5-1-84zdqd should provision
Unexpected error:
<*fmt.wrapError | 0x16cd8e8f9880>:
failed to create HCP cluster cluster-candidate-5-1-84zdqd: failed waiting for cluster="cluster-candidate-5-1-84zdqd" in resourcegroup="rg-candidate-5-1-2s5zlgwb6bct" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/9d897978-69ec-403d-ad9b-fd32c33747d4
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/9d897978-69ec-403d-ad9b-fd32c33747d4",
"name": "9d897978-69ec-403d-ad9b-fd32c33747d4",
"status": "Failed",
"startTime": "2026-10-22T02:21:55.892991844Z",
"endTime": "2026-10-22T02:51:00.378883542Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: machine-approver, csi-snapshot-controller-operator, cluster-image-registry-operator, catalog-operator, kube-scheduler, dns-operator, konnectivity-agent, kube-controller-manager, cluster-storage-operator, cluster-autoscaler, cluster-policy-controller, hosted-cluster-config-operator, olm-operator, olm-collect-profiles, community-operators-catalog, cluster-version-operator, kube-storage-version-migrator, packageserver, control-plane-pki-operator, openshift-controller-manager, azure-cloud-controller-manager, openshift-apiserver, openshift-route-controller-manager, redhat-operators-catalog, certified-operators-catalog, cluster-network-operator, cluster-node-tuning-operator, ingress-operator; hosted cluster degraded: UnavailableReplicas: kube-apiserver deployment has 1 unavailable replicas"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/9d897978-69ec-403d-ad9b-fd32c33747d4
correlation ID: bdf3de8a-4b44-40d3-b952-4e50b1bc65e3
client request ID: unavailable
occurredfail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:125]: failed to create HCP cluster "cluster-zstream-4-21-xzrpbt" with version 4.21.38 on candidate channel
Unexpected error:
<*fmt.wrapError | 0x913e9d26a60>:
failed to create HCP cluster cluster-zstream-4-21-xzrpbt: failed waiting for cluster="cluster-zstream-4-21-xzrpbt" in resourcegroup="rg-zstream-upgrade-4-21-xgpgfghfg7wk" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/d76aa69d-45fb-439f-889e-97ea3ff34f99
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/d76aa69d-45fb-439f-889e-97ea3ff34f99",
"name": "d76aa69d-45fb-439f-889e-97ea3ff34f99",
"status": "Failed",
"startTime": "2026-10-22T02:22:56.334901591Z",
"endTime": "2026-10-22T02:52:00.375914793Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: cluster-storage-operator, dns-operator, redhat-operators-catalog, certified-operators-catalog, control-plane-pki-operator, cluster-version-operator, openshift-route-controller-manager, azure-cloud-controller-manager, redhat-marketplace-catalog, cluster-autoscaler, hosted-cluster-config-operator, olm-operator, packageserver, ingress-operator, cluster-image-registry-operator, machine-approver, cluster-policy-controller, cluster-node-tuning-operator, konnectivity-agent, kube-controller-manager, openshift-controller-manager, cluster-network-operator, catalog-operator, community-operators-catalog, kube-scheduler, openshift-apiserver, csi-snapshot-controller-operator, olm-collect-profiles; hosted cluster degraded: UnavailableReplicas: kube-apiserver deployment has 1 unavailable replicas"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/d76aa69d-45fb-439f-889e-97ea3ff34f99
correlation ID: b1fc9a13-2fc1-4373-b2a1-ce2064832e2e
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:125]: failed to create HCP cluster "cluster-zstream-4-21-xzrpbt" with version 4.21.38 on candidate channel
Unexpected error:
<*fmt.wrapError | 0x913e9d26a60>:
failed to create HCP cluster cluster-zstream-4-21-xzrpbt: failed waiting for cluster="cluster-zstream-4-21-xzrpbt" in resourcegroup="rg-zstream-upgrade-4-21-xgpgfghfg7wk" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/d76aa69d-45fb-439f-889e-97ea3ff34f99
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/d76aa69d-45fb-439f-889e-97ea3ff34f99",
"name": "d76aa69d-45fb-439f-889e-97ea3ff34f99",
"status": "Failed",
"startTime": "2026-10-22T02:22:56.334901591Z",
"endTime": "2026-10-22T02:52:00.375914793Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: cluster-storage-operator, dns-operator, redhat-operators-catalog, certified-operators-catalog, control-plane-pki-operator, cluster-version-operator, openshift-route-controller-manager, azure-cloud-controller-manager, redhat-marketplace-catalog, cluster-autoscaler, hosted-cluster-config-operator, olm-operator, packageserver, ingress-operator, cluster-image-registry-operator, machine-approver, cluster-policy-controller, cluster-node-tuning-operator, konnectivity-agent, kube-controller-manager, openshift-controller-manager, cluster-network-operator, catalog-operator, community-operators-catalog, kube-scheduler, openshift-apiserver, csi-snapshot-controller-operator, olm-collect-profiles; hosted cluster degraded: UnavailableReplicas: kube-apiserver deployment has 1 unavailable replicas"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/d76aa69d-45fb-439f-889e-97ea3ff34f99
correlation ID: b1fc9a13-2fc1-4373-b2a1-ce2064832e2e
client request ID: unavailable
occurredContributing tests (11)
Affected runs (2)
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedC...full failure pattern: ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable; hosted cluster degraded: UnavailableReplicas: [capi-provider, kube-apiserver]; provider Microsoft.RedHatOpenShift Signal: IndeterminateSignal: Indeterminate — no prior history; active 2 day(s) — New failure pattern — no prior history | e2e | 2 | 2.08%2 of 96 job runs affected | IndeterminateSignal: Indeterminate — no prior history; active 2 day(s) — New failure pattern — no prior history | Show trend detailsOct 12: 0 · Oct 13: 0 · Oct 14: 0 · Oct 15: 0 · Oct 16: 0 · Oct 17: 0 · Oct 18: 0 · Oct 19: 0 · Oct 20: 6 · Oct 21: 0 · Oct 22: 2 · Oct 23: 0 · Oct 24: 0 · Oct 25: 0 | none | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (3)fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:125]: failed to create HCP cluster "cluster-zstream-4-22-dsklst" with version 4.22.18 on candidate channel
Unexpected error:
<*fmt.wrapError | 0x19639643c920>:
failed to create HCP cluster cluster-zstream-4-22-dsklst: failed waiting for cluster="cluster-zstream-4-22-dsklst" in resourcegroup="rg-zstream-upgrade-4-22-5t2tjvkj99ps" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/13f8c2f6-a8c6-4ec9-bef8-08b6e4a75a38
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/13f8c2f6-a8c6-4ec9-bef8-08b6e4a75a38",
"name": "13f8c2f6-a8c6-4ec9-bef8-08b6e4a75a38",
"status": "Failed",
"startTime": "2026-10-22T02:22:46.501521761Z",
"endTime": "2026-10-22T02:51:50.360464178Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: cluster-version-operator, ingress-operator, cluster-image-registry-operator, community-operators-catalog, machine-approver, azure-cloud-controller-manager, dns-operator, hosted-cluster-config-operator, csi-snapshot-controller-operator, packageserver, openshift-controller-manager, konnectivity-agent, certified-operators-catalog, kube-storage-version-migrator, catalog-operator, capi-provider, openshift-apiserver, openshift-route-controller-manager, cluster-storage-operator, cluster-network-operator, cluster-autoscaler, cluster-policy-controller, kube-scheduler, olm-collect-profiles, redhat-operators-catalog, control-plane-pki-operator, kube-controller-manager, cluster-node-tuning-operator, olm-operator; hosted cluster degraded: UnavailableReplicas: [capi-provider deployment has 2 unavailable replicas, kube-apiserver deployment has 1 unavailable replicas]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/13f8c2f6-a8c6-4ec9-bef8-08b6e4a75a38
correlation ID: 4f330ac9-7640-45b9-b580-8c05eae65daf
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:125]: failed to create HCP cluster "cluster-zstream-4-22-dsklst" with version 4.22.18 on candidate channel
Unexpected error:
<*fmt.wrapError | 0x19639643c920>:
failed to create HCP cluster cluster-zstream-4-22-dsklst: failed waiting for cluster="cluster-zstream-4-22-dsklst" in resourcegroup="rg-zstream-upgrade-4-22-5t2tjvkj99ps" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/13f8c2f6-a8c6-4ec9-bef8-08b6e4a75a38
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/13f8c2f6-a8c6-4ec9-bef8-08b6e4a75a38",
"name": "13f8c2f6-a8c6-4ec9-bef8-08b6e4a75a38",
"status": "Failed",
"startTime": "2026-10-22T02:22:46.501521761Z",
"endTime": "2026-10-22T02:51:50.360464178Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: cluster-version-operator, ingress-operator, cluster-image-registry-operator, community-operators-catalog, machine-approver, azure-cloud-controller-manager, dns-operator, hosted-cluster-config-operator, csi-snapshot-controller-operator, packageserver, openshift-controller-manager, konnectivity-agent, certified-operators-catalog, kube-storage-version-migrator, catalog-operator, capi-provider, openshift-apiserver, openshift-route-controller-manager, cluster-storage-operator, cluster-network-operator, cluster-autoscaler, cluster-policy-controller, kube-scheduler, olm-collect-profiles, redhat-operators-catalog, control-plane-pki-operator, kube-controller-manager, cluster-node-tuning-operator, olm-operator; hosted cluster degraded: UnavailableReplicas: [capi-provider deployment has 2 unavailable replicas, kube-apiserver deployment has 1 unavailable replicas]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/13f8c2f6-a8c6-4ec9-bef8-08b6e4a75a38
correlation ID: 4f330ac9-7640-45b9-b580-8c05eae65daf
client request ID: unavailable
occurredfail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:125]: failed to create HCP cluster "cluster-zstream-5-0-bzlw46" with version 5.0.0-rc.6 on candidate channel
Unexpected error:
<*fmt.wrapError | 0x202d14170440>:
failed to create HCP cluster cluster-zstream-5-0-bzlw46: failed waiting for cluster="cluster-zstream-5-0-bzlw46" in resourcegroup="rg-zstream-upgrade-5-0-l5ckf5pqgjzq" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/b119ec20-94fb-4593-816b-4d56620a6799
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/b119ec20-94fb-4593-816b-4d56620a6799",
"name": "b119ec20-94fb-4593-816b-4d56620a6799",
"status": "Failed",
"startTime": "2026-10-22T02:22:58.803577295Z",
"endTime": "2026-10-22T02:52:00.551152403Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: hosted-cluster-config-operator, cluster-node-tuning-operator, csi-snapshot-controller-operator, cluster-storage-operator, redhat-operators-catalog, capi-provider, cluster-network-operator, ingress-operator, olm-collect-profiles, openshift-controller-manager, azure-cloud-controller-manager, kube-storage-version-migrator, cluster-image-registry-operator, konnectivity-agent, machine-approver, kube-scheduler, cluster-version-operator, dns-operator, catalog-operator, cluster-autoscaler, control-plane-pki-operator, kube-controller-manager, openshift-route-controller-manager, cluster-policy-controller, olm-operator, packageserver, community-operators-catalog, certified-operators-catalog, openshift-apiserver; hosted cluster degraded: UnavailableReplicas: [capi-provider deployment has 2 unavailable replicas, kube-apiserver deployment has 1 unavailable replicas]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/b119ec20-94fb-4593-816b-4d56620a6799
correlation ID: cb65955f-e919-441d-963c-27a6a9495a9b
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:125]: failed to create HCP cluster "cluster-zstream-5-0-bzlw46" with version 5.0.0-rc.6 on candidate channel
Unexpected error:
<*fmt.wrapError | 0x202d14170440>:
failed to create HCP cluster cluster-zstream-5-0-bzlw46: failed waiting for cluster="cluster-zstream-5-0-bzlw46" in resourcegroup="rg-zstream-upgrade-5-0-l5ckf5pqgjzq" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/b119ec20-94fb-4593-816b-4d56620a6799
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/b119ec20-94fb-4593-816b-4d56620a6799",
"name": "b119ec20-94fb-4593-816b-4d56620a6799",
"status": "Failed",
"startTime": "2026-10-22T02:22:58.803577295Z",
"endTime": "2026-10-22T02:52:00.551152403Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: hosted-cluster-config-operator, cluster-node-tuning-operator, csi-snapshot-controller-operator, cluster-storage-operator, redhat-operators-catalog, capi-provider, cluster-network-operator, ingress-operator, olm-collect-profiles, openshift-controller-manager, azure-cloud-controller-manager, kube-storage-version-migrator, cluster-image-registry-operator, konnectivity-agent, machine-approver, kube-scheduler, cluster-version-operator, dns-operator, catalog-operator, cluster-autoscaler, control-plane-pki-operator, kube-controller-manager, openshift-route-controller-manager, cluster-policy-controller, olm-operator, packageserver, community-operators-catalog, certified-operators-catalog, openshift-apiserver; hosted cluster degraded: UnavailableReplicas: [capi-provider deployment has 2 unavailable replicas, kube-apiserver deployment has 1 unavailable replicas]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/b119ec20-94fb-4593-816b-4d56620a6799
correlation ID: cb65955f-e919-441d-963c-27a6a9495a9b
client request ID: unavailable
occurredfail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:146]: failed to create HCP cluster np-version-upgrade-cluster-zjk46l with version 4.21.39
Unexpected error:
<*fmt.wrapError | 0x1de0e4b15080>:
failed to create HCP cluster np-version-upgrade-cluster-zjk46l: failed waiting for cluster="np-version-upgrade-cluster-zjk46l" in resourcegroup="rg-np-version-upgrade-zjk46l-52b2swhcrs62" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/7572214c-58bb-42db-9c41-46711f0d39e6
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/7572214c-58bb-42db-9c41-46711f0d39e6",
"name": "7572214c-58bb-42db-9c41-46711f0d39e6",
"status": "Failed",
"startTime": "2026-10-20T06:21:20.599092026Z",
"endTime": "2026-10-20T06:50:20.225694377Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: cluster-policy-controller, azure-cloud-controller-manager, dns-operator, control-plane-pki-operator, kube-scheduler, cluster-storage-operator, csi-snapshot-controller-operator, redhat-operators-catalog, community-operators-catalog, cluster-network-operator, packageserver, kube-controller-manager, openshift-controller-manager, catalog-operator, olm-collect-profiles, capi-provider, olm-operator, hosted-cluster-config-operator, ingress-operator, konnectivity-agent, certified-operators-catalog, machine-approver, openshift-apiserver, cluster-version-operator, cluster-image-registry-operator, cluster-autoscaler, openshift-route-controller-manager, cluster-node-tuning-operator, redhat-marketplace-catalog; hosted cluster degraded: UnavailableReplicas: [capi-provider deployment has 2 unavailable replicas, kube-apiserver deployment has 1 unavailable replicas]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/7572214c-58bb-42db-9c41-46711f0d39e6
correlation ID: 2a2f8f2f-1289-48c9-97d5-1f5137565aa0
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:146]: failed to create HCP cluster np-version-upgrade-cluster-zjk46l with version 4.21.39
Unexpected error:
<*fmt.wrapError | 0x1de0e4b15080>:
failed to create HCP cluster np-version-upgrade-cluster-zjk46l: failed waiting for cluster="np-version-upgrade-cluster-zjk46l" in resourcegroup="rg-np-version-upgrade-zjk46l-52b2swhcrs62" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/7572214c-58bb-42db-9c41-46711f0d39e6
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/7572214c-58bb-42db-9c41-46711f0d39e6",
"name": "7572214c-58bb-42db-9c41-46711f0d39e6",
"status": "Failed",
"startTime": "2026-10-20T06:21:20.599092026Z",
"endTime": "2026-10-20T06:50:20.225694377Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: cluster-policy-controller, azure-cloud-controller-manager, dns-operator, control-plane-pki-operator, kube-scheduler, cluster-storage-operator, csi-snapshot-controller-operator, redhat-operators-catalog, community-operators-catalog, cluster-network-operator, packageserver, kube-controller-manager, openshift-controller-manager, catalog-operator, olm-collect-profiles, capi-provider, olm-operator, hosted-cluster-config-operator, ingress-operator, konnectivity-agent, certified-operators-catalog, machine-approver, openshift-apiserver, cluster-version-operator, cluster-image-registry-operator, cluster-autoscaler, openshift-route-controller-manager, cluster-node-tuning-operator, redhat-marketplace-catalog; hosted cluster degraded: UnavailableReplicas: [capi-provider deployment has 2 unavailable replicas, kube-apiserver deployment has 1 unavailable replicas]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/7572214c-58bb-42db-9c41-46711f0d39e6
correlation ID: 2a2f8f2f-1289-48c9-97d5-1f5137565aa0
client request ID: unavailable
occurredContributing tests (8)
Affected runs (2)
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedC...full failure pattern: ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable; hosted cluster degraded: UnavailableReplicas: [catalog-operator, hosted-cluster-config-operator, konnectivity-agent, kube-controller-manager, kube-scheduler, olm-operator, openshift-apiserver, packageserver]; provider Microsoft.RedHatOpenShift Signal: IndeterminateSignal: Indeterminate — no prior history; active 2 day(s) — New failure pattern — no prior history | e2e | 2 | 2.08%2 of 96 job runs affected | IndeterminateSignal: Indeterminate — no prior history; active 2 day(s) — New failure pattern — no prior history | Show trend detailsOct 12: 0 · Oct 13: 0 · Oct 14: 0 · Oct 15: 0 · Oct 16: 0 · Oct 17: 0 · Oct 18: 0 · Oct 19: 0 · Oct 20: 4 · Oct 21: 0 · Oct 22: 3 · Oct 23: 0 · Oct 24: 0 · Oct 25: 0 | none | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (3)fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:133]: failed to create HCP cluster "cp-ystream-upgrade-4-22-pp2v6j" at install version 4.21
Unexpected error:
<*fmt.wrapError | 0x9f443c1ebc0>:
failed to create HCP cluster cp-ystream-upgrade-4-22-pp2v6j: failed waiting for cluster="cp-ystream-upgrade-4-22-pp2v6j" in resourcegroup="rg-cp-ystream-upgrade-4-22-4brmx258wcvt" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/c465a539-e0e2-4a15-96b0-b991fee60416
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/c465a539-e0e2-4a15-96b0-b991fee60416",
"name": "c465a539-e0e2-4a15-96b0-b991fee60416",
"status": "Failed",
"startTime": "2026-10-22T02:23:40.34684753Z",
"endTime": "2026-10-22T02:52:40.362346793Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: csi-snapshot-controller-operator, cluster-storage-operator, cluster-policy-controller, packageserver, hosted-cluster-config-operator, cluster-image-registry-operator, kube-controller-manager, cluster-version-operator, community-operators-catalog, openshift-controller-manager, cluster-node-tuning-operator, redhat-operators-catalog, redhat-marketplace-catalog, certified-operators-catalog, openshift-route-controller-manager, kube-scheduler, cluster-network-operator, dns-operator, konnectivity-agent, olm-operator, ingress-operator, catalog-operator; hosted cluster degraded: UnavailableReplicas: [catalog-operator deployment has 1 unavailable replicas, hosted-cluster-config-operator deployment has 1 unavailable replicas, konnectivity-agent deployment has 1 unavailable replicas, kube-controller-manager deployment has 1 unavailable replicas, kube-scheduler deployment has 1 unavailable replicas, olm-operator deployment has 1 unavailable replicas, openshift-apiserver deployment has 1 unavailable replicas, packageserver deployment has 3 unavailable replicas]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/c465a539-e0e2-4a15-96b0-b991fee60416
correlation ID: 81f12348-0a31-441f-af5f-8f18eef28715
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:133]: failed to create HCP cluster "cp-ystream-upgrade-4-22-pp2v6j" at install version 4.21
Unexpected error:
<*fmt.wrapError | 0x9f443c1ebc0>:
failed to create HCP cluster cp-ystream-upgrade-4-22-pp2v6j: failed waiting for cluster="cp-ystream-upgrade-4-22-pp2v6j" in resourcegroup="rg-cp-ystream-upgrade-4-22-4brmx258wcvt" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/c465a539-e0e2-4a15-96b0-b991fee60416
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/c465a539-e0e2-4a15-96b0-b991fee60416",
"name": "c465a539-e0e2-4a15-96b0-b991fee60416",
"status": "Failed",
"startTime": "2026-10-22T02:23:40.34684753Z",
"endTime": "2026-10-22T02:52:40.362346793Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: csi-snapshot-controller-operator, cluster-storage-operator, cluster-policy-controller, packageserver, hosted-cluster-config-operator, cluster-image-registry-operator, kube-controller-manager, cluster-version-operator, community-operators-catalog, openshift-controller-manager, cluster-node-tuning-operator, redhat-operators-catalog, redhat-marketplace-catalog, certified-operators-catalog, openshift-route-controller-manager, kube-scheduler, cluster-network-operator, dns-operator, konnectivity-agent, olm-operator, ingress-operator, catalog-operator; hosted cluster degraded: UnavailableReplicas: [catalog-operator deployment has 1 unavailable replicas, hosted-cluster-config-operator deployment has 1 unavailable replicas, konnectivity-agent deployment has 1 unavailable replicas, kube-controller-manager deployment has 1 unavailable replicas, kube-scheduler deployment has 1 unavailable replicas, olm-operator deployment has 1 unavailable replicas, openshift-apiserver deployment has 1 unavailable replicas, packageserver deployment has 3 unavailable replicas]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/c465a539-e0e2-4a15-96b0-b991fee60416
correlation ID: 81f12348-0a31-441f-af5f-8f18eef28715
client request ID: unavailable
occurredfail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:146]: failed to create HCP cluster np-version-upgrade-cluster-vwsld8 with version 4.20.43
Unexpected error:
<*fmt.wrapError | 0xc02210a0420>:
failed to create HCP cluster np-version-upgrade-cluster-vwsld8: failed waiting for cluster="np-version-upgrade-cluster-vwsld8" in resourcegroup="rg-np-version-upgrade-vwsld8-b78v68n9cqj2" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/7061ec27-2552-401b-a8b5-5bcf7cc27448
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/7061ec27-2552-401b-a8b5-5bcf7cc27448",
"name": "7061ec27-2552-401b-a8b5-5bcf7cc27448",
"status": "Failed",
"startTime": "2026-10-22T03:10:01.822651936Z",
"endTime": "2026-10-22T03:39:05.079618983Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: cluster-version-operator, hosted-cluster-config-operator, cluster-image-registry-operator, catalog-operator, cluster-storage-operator, csi-snapshot-controller-operator, community-operators-catalog, cluster-node-tuning-operator, ingress-operator, redhat-marketplace-catalog, openshift-controller-manager, packageserver, cluster-policy-controller, dns-operator, olm-operator, cluster-network-operator, redhat-operators-catalog, certified-operators-catalog, openshift-route-controller-manager, konnectivity-agent, kube-controller-manager; hosted cluster degraded: UnavailableReplicas: [catalog-operator deployment has 1 unavailable replicas, hosted-cluster-config-operator deployment has 1 unavailable replicas, konnectivity-agent deployment has 1 unavailable replicas, kube-controller-manager deployment has 1 unavailable replicas, kube-scheduler deployment has 1 unavailable replicas, olm-operator deployment has 1 unavailable replicas, openshift-apiserver deployment has 1 unavailable replicas, packageserver deployment has 3 unavailable replicas]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/7061ec27-2552-401b-a8b5-5bcf7cc27448
correlation ID: 8d4a80f0-34d4-471d-b30b-2c4ee315636d
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:146]: failed to create HCP cluster np-version-upgrade-cluster-vwsld8 with version 4.20.43
Unexpected error:
<*fmt.wrapError | 0xc02210a0420>:
failed to create HCP cluster np-version-upgrade-cluster-vwsld8: failed waiting for cluster="np-version-upgrade-cluster-vwsld8" in resourcegroup="rg-np-version-upgrade-vwsld8-b78v68n9cqj2" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/7061ec27-2552-401b-a8b5-5bcf7cc27448
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/7061ec27-2552-401b-a8b5-5bcf7cc27448",
"name": "7061ec27-2552-401b-a8b5-5bcf7cc27448",
"status": "Failed",
"startTime": "2026-10-22T03:10:01.822651936Z",
"endTime": "2026-10-22T03:39:05.079618983Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: cluster-version-operator, hosted-cluster-config-operator, cluster-image-registry-operator, catalog-operator, cluster-storage-operator, csi-snapshot-controller-operator, community-operators-catalog, cluster-node-tuning-operator, ingress-operator, redhat-marketplace-catalog, openshift-controller-manager, packageserver, cluster-policy-controller, dns-operator, olm-operator, cluster-network-operator, redhat-operators-catalog, certified-operators-catalog, openshift-route-controller-manager, konnectivity-agent, kube-controller-manager; hosted cluster degraded: UnavailableReplicas: [catalog-operator deployment has 1 unavailable replicas, hosted-cluster-config-operator deployment has 1 unavailable replicas, konnectivity-agent deployment has 1 unavailable replicas, kube-controller-manager deployment has 1 unavailable replicas, kube-scheduler deployment has 1 unavailable replicas, olm-operator deployment has 1 unavailable replicas, openshift-apiserver deployment has 1 unavailable replicas, packageserver deployment has 3 unavailable replicas]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/7061ec27-2552-401b-a8b5-5bcf7cc27448
correlation ID: 8d4a80f0-34d4-471d-b30b-2c4ee315636d
client request ID: unavailable
occurredfail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:146]: failed to create HCP cluster np-version-upgrade-cluster-hlfqz4 with version 4.21.39
Unexpected error:
<*fmt.wrapError | 0x26fe4f09e7e0>:
failed to create HCP cluster np-version-upgrade-cluster-hlfqz4: failed waiting for cluster="np-version-upgrade-cluster-hlfqz4" in resourcegroup="rg-np-version-upgrade-hlfqz4-hxtt44szpfn8" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/9b1bdec9-cda2-4c03-8af1-a62744ae5bac
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/9b1bdec9-cda2-4c03-8af1-a62744ae5bac",
"name": "9b1bdec9-cda2-4c03-8af1-a62744ae5bac",
"status": "Failed",
"startTime": "2026-10-22T03:09:57.381547287Z",
"endTime": "2026-10-22T03:39:05.056394482Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: konnectivity-agent, redhat-operators-catalog, catalog-operator, cluster-version-operator, cluster-network-operator, dns-operator, csi-snapshot-controller-operator, redhat-marketplace-catalog, cluster-policy-controller, cluster-storage-operator, openshift-route-controller-manager, kube-controller-manager, ingress-operator, packageserver, olm-operator, kube-scheduler, hosted-cluster-config-operator, openshift-controller-manager, cluster-node-tuning-operator, cluster-image-registry-operator, community-operators-catalog, certified-operators-catalog; hosted cluster degraded: UnavailableReplicas: [catalog-operator deployment has 1 unavailable replicas, hosted-cluster-config-operator deployment has 1 unavailable replicas, konnectivity-agent deployment has 1 unavailable replicas, kube-controller-manager deployment has 1 unavailable replicas, kube-scheduler deployment has 1 unavailable replicas, olm-operator deployment has 1 unavailable replicas, openshift-apiserver deployment has 1 unavailable replicas, packageserver deployment has 3 unavailable replicas]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/9b1bdec9-cda2-4c03-8af1-a62744ae5bac
correlation ID: ae3669ea-d6b2-47bf-90e6-616390cf31a2
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:146]: failed to create HCP cluster np-version-upgrade-cluster-hlfqz4 with version 4.21.39
Unexpected error:
<*fmt.wrapError | 0x26fe4f09e7e0>:
failed to create HCP cluster np-version-upgrade-cluster-hlfqz4: failed waiting for cluster="np-version-upgrade-cluster-hlfqz4" in resourcegroup="rg-np-version-upgrade-hlfqz4-hxtt44szpfn8" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/9b1bdec9-cda2-4c03-8af1-a62744ae5bac
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/9b1bdec9-cda2-4c03-8af1-a62744ae5bac",
"name": "9b1bdec9-cda2-4c03-8af1-a62744ae5bac",
"status": "Failed",
"startTime": "2026-10-22T03:09:57.381547287Z",
"endTime": "2026-10-22T03:39:05.056394482Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: konnectivity-agent, redhat-operators-catalog, catalog-operator, cluster-version-operator, cluster-network-operator, dns-operator, csi-snapshot-controller-operator, redhat-marketplace-catalog, cluster-policy-controller, cluster-storage-operator, openshift-route-controller-manager, kube-controller-manager, ingress-operator, packageserver, olm-operator, kube-scheduler, hosted-cluster-config-operator, openshift-controller-manager, cluster-node-tuning-operator, cluster-image-registry-operator, community-operators-catalog, certified-operators-catalog; hosted cluster degraded: UnavailableReplicas: [catalog-operator deployment has 1 unavailable replicas, hosted-cluster-config-operator deployment has 1 unavailable replicas, konnectivity-agent deployment has 1 unavailable replicas, kube-controller-manager deployment has 1 unavailable replicas, kube-scheduler deployment has 1 unavailable replicas, olm-operator deployment has 1 unavailable replicas, openshift-apiserver deployment has 1 unavailable replicas, packageserver deployment has 3 unavailable replicas]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/9b1bdec9-cda2-4c03-8af1-a62744ae5bac
correlation ID: ae3669ea-d6b2-47bf-90e6-616390cf31a2
client request ID: unavailable
occurredContributing tests (6)
Affected runs (2)
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedC...full failure pattern: ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider, router]; provider Microsoft.RedHatOpenShift Signal: IndeterminateSignal: Indeterminate — no prior history; active 2 day(s) — New failure pattern — no prior history | e2e | 2 | 2.08%2 of 96 job runs affected | IndeterminateSignal: Indeterminate — no prior history; active 2 day(s) — New failure pattern — no prior history | Show trend detailsOct 12: 0 · Oct 13: 0 · Oct 14: 0 · Oct 15: 0 · Oct 16: 0 · Oct 17: 0 · Oct 18: 0 · Oct 19: 0 · Oct 20: 3 · Oct 21: 0 · Oct 22: 4 · Oct 23: 0 · Oct 24: 0 · Oct 25: 0 | DEV | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (3)fail [github.com/Azure/ARO-HCP/test/e2e/cluster_nodepool_active_versions.go:80]: failed to create HCP cluster "v20261001" via v20261001
Unexpected error:
<*fmt.wrapError | 0x3a75f4e12f60>:
failed to create HCP cluster v20261001: failed waiting for cluster="v20261001" in resourcegroup="v20261001-fx25mzlnzj9t" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/0f9269de-192b-41ce-bc41-692ff3dda38f
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/0f9269de-192b-41ce-bc41-692ff3dda38f",
"name": "0f9269de-192b-41ce-bc41-692ff3dda38f",
"status": "Failed",
"startTime": "2026-10-22T01:34:16.144784385Z",
"endTime": "2026-10-22T02:03:16.753815438Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider deployment has 1 unavailable replicas, router deployment has 1 unavailable replicas]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/0f9269de-192b-41ce-bc41-692ff3dda38f
correlation ID: 0c97842f-0279-4b59-9196-c2a2e8ab56cb
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_nodepool_active_versions.go:80]: failed to create HCP cluster "v20261001" via v20261001
Unexpected error:
<*fmt.wrapError | 0x3a75f4e12f60>:
failed to create HCP cluster v20261001: failed waiting for cluster="v20261001" in resourcegroup="v20261001-fx25mzlnzj9t" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/0f9269de-192b-41ce-bc41-692ff3dda38f
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/0f9269de-192b-41ce-bc41-692ff3dda38f",
"name": "0f9269de-192b-41ce-bc41-692ff3dda38f",
"status": "Failed",
"startTime": "2026-10-22T01:34:16.144784385Z",
"endTime": "2026-10-22T02:03:16.753815438Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider deployment has 1 unavailable replicas, router deployment has 1 unavailable replicas]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/0f9269de-192b-41ce-bc41-692ff3dda38f
correlation ID: 0c97842f-0279-4b59-9196-c2a2e8ab56cb
client request ID: unavailable
occurredfail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:208]: failed to create HCP cluster "agg-cluster" with aggregated settings
Unexpected error:
<*fmt.wrapError | 0x7938a410140>:
failed waiting for cluster="agg-cluster" in resourcegroup="feature-aggregation-ql2jtgg2jsjl" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/65c2874a-4e4d-4754-a6c4-c23ee1f34bb9
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/65c2874a-4e4d-4754-a6c4-c23ee1f34bb9",
"name": "65c2874a-4e4d-4754-a6c4-c23ee1f34bb9",
"status": "Failed",
"startTime": "2026-10-22T01:38:14.381676151Z",
"endTime": "2026-10-22T02:07:16.666379289Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider deployment has 1 unavailable replicas, router deployment has 1 unavailable replicas]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/65c2874a-4e4d-4754-a6c4-c23ee1f34bb9
correlation ID: 3892a8ef-862d-414b-aadd-333496c0245e
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:208]: failed to create HCP cluster "agg-cluster" with aggregated settings
Unexpected error:
<*fmt.wrapError | 0x7938a410140>:
failed waiting for cluster="agg-cluster" in resourcegroup="feature-aggregation-ql2jtgg2jsjl" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/65c2874a-4e4d-4754-a6c4-c23ee1f34bb9
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/65c2874a-4e4d-4754-a6c4-c23ee1f34bb9",
"name": "65c2874a-4e4d-4754-a6c4-c23ee1f34bb9",
"status": "Failed",
"startTime": "2026-10-22T01:38:14.381676151Z",
"endTime": "2026-10-22T02:07:16.666379289Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider deployment has 1 unavailable replicas, router deployment has 1 unavailable replicas]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/65c2874a-4e4d-4754-a6c4-c23ee1f34bb9
correlation ID: 3892a8ef-862d-414b-aadd-333496c0245e
client request ID: unavailable
occurredfail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kv.go:99]: failed to create HCP cluster "private-kv-cluster" with private keyvault
Unexpected error:
<*fmt.wrapError | 0x327c31abadc0>:
failed waiting for cluster="private-kv-cluster" in resourcegroup="private-keyvault-vzj54nlw5csp" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/f9fa1671-b5d8-42db-9ac9-d89b520cbbf9
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/f9fa1671-b5d8-42db-9ac9-d89b520cbbf9",
"name": "f9fa1671-b5d8-42db-9ac9-d89b520cbbf9",
"status": "Failed",
"startTime": "2026-10-22T01:35:36.727726764Z",
"endTime": "2026-10-22T02:04:36.824596885Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider deployment has 1 unavailable replicas, router deployment has 1 unavailable replicas]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/f9fa1671-b5d8-42db-9ac9-d89b520cbbf9
correlation ID: ba4affaa-364d-40fe-b039-d27e78c0a5e0
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kv.go:99]: failed to create HCP cluster "private-kv-cluster" with private keyvault
Unexpected error:
<*fmt.wrapError | 0x327c31abadc0>:
failed waiting for cluster="private-kv-cluster" in resourcegroup="private-keyvault-vzj54nlw5csp" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/f9fa1671-b5d8-42db-9ac9-d89b520cbbf9
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/f9fa1671-b5d8-42db-9ac9-d89b520cbbf9",
"name": "f9fa1671-b5d8-42db-9ac9-d89b520cbbf9",
"status": "Failed",
"startTime": "2026-10-22T01:35:36.727726764Z",
"endTime": "2026-10-22T02:04:36.824596885Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider deployment has 1 unavailable replicas, router deployment has 1 unavailable replicas]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/f9fa1671-b5d8-42db-9ac9-d89b520cbbf9
correlation ID: ba4affaa-364d-40fe-b039-d27e78c0a5e0
client request ID: unavailable
occurredContributing tests (7)
Affected runs (2)
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
DNS for route host <route-host> did not resolve: DNS for <route-host> did not resolve within <duration> (last error: lookup <host> on <dns-server>: no such host): context deadline...full failure pattern: DNS for route host <route-host> did not resolve: DNS for <route-host> did not resolve within <duration> (last error: lookup <host> on <dns-server>: no such host): context deadline exceeded Signal: IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | e2e | 2 | 2.08%2 of 96 job runs affected | IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | Show trend detailsOct 12: 0 · Oct 13: 0 · Oct 14: 0 · Oct 15: 0 · Oct 16: 0 · Oct 17: 0 · Oct 18: 0 · Oct 19: 0 · Oct 20: 0 · Oct 21: 0 · Oct 22: 0 · Oct 23: 6 · Oct 24: 0 · Oct 25: 0 | DEV, INT | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (3)fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:190]: failed to verify simple web app runs on cluster "cluster-candidate-5-0-wnz6kv"
Unexpected error:
<*fmt.wrapError | 0x1aca60620e60>:
DNS for route host agnhost-e2e-sample-app-5cwbl.apps.aro.c1s1r7e8q7f4j2u.vfmi.eastus2.aroapp-hcp.io did not resolve: DNS for agnhost-e2e-sample-app-5cwbl.apps.aro.c1s1r7e8q7f4j2u.vfmi.eastus2.aroapp-hcp.io did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-5cwbl.apps.aro.c1s1r7e8q7f4j2u.vfmi.eastus2.aroapp-hcp.io on 172.30.0.10:53: no such host): context deadline exceeded
{
msg: "DNS for route host agnhost-e2e-sample-app-5cwbl.apps.aro.c1s1r7e8q7f4j2u.vfmi.eastus2.aroapp-hcp.io did not resolve: DNS for agnhost-e2e-sample-app-5cwbl.apps.aro.c1s1r7e8q7f4j2u.vfmi.eastus2.aroapp-hcp.io did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-5cwbl.apps.aro.c1s1r7e8q7f4j2u.vfmi.eastus2.aroapp-hcp.io on 172.30.0.10:53: no such host): context deadline exceeded",
err: <*fmt.wrapError | 0x1aca604dcae0>{
msg: "DNS for agnhost-e2e-sample-app-5cwbl.apps.aro.c1s1r7e8q7f4j2u.vfmi.eastus2.aroapp-hcp.io did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-5cwbl.apps.aro.c1s1r7e8q7f4j2u.vfmi.eastus2.aroapp-hcp.io on 172.30.0.10:53: no such host): context deadline exceeded",
err: <context.deadlineExceededError>{},
},
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:190]: failed to verify simple web app runs on cluster "cluster-candidate-5-0-wnz6kv"
Unexpected error:
<*fmt.wrapError | 0x1aca60620e60>:
DNS for route host agnhost-e2e-sample-app-5cwbl.apps.aro.c1s1r7e8q7f4j2u.vfmi.eastus2.aroapp-hcp.io did not resolve: DNS for agnhost-e2e-sample-app-5cwbl.apps.aro.c1s1r7e8q7f4j2u.vfmi.eastus2.aroapp-hcp.io did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-5cwbl.apps.aro.c1s1r7e8q7f4j2u.vfmi.eastus2.aroapp-hcp.io on 172.30.0.10:53: no such host): context deadline exceeded
{
msg: "DNS for route host agnhost-e2e-sample-app-5cwbl.apps.aro.c1s1r7e8q7f4j2u.vfmi.eastus2.aroapp-hcp.io did not resolve: DNS for agnhost-e2e-sample-app-5cwbl.apps.aro.c1s1r7e8q7f4j2u.vfmi.eastus2.aroapp-hcp.io did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-5cwbl.apps.aro.c1s1r7e8q7f4j2u.vfmi.eastus2.aroapp-hcp.io on 172.30.0.10:53: no such host): context deadline exceeded",
err: <*fmt.wrapError | 0x1aca604dcae0>{
msg: "DNS for agnhost-e2e-sample-app-5cwbl.apps.aro.c1s1r7e8q7f4j2u.vfmi.eastus2.aroapp-hcp.io did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-5cwbl.apps.aro.c1s1r7e8q7f4j2u.vfmi.eastus2.aroapp-hcp.io on 172.30.0.10:53: no such host): context deadline exceeded",
err: <context.deadlineExceededError>{},
},
}
occurredfail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:190]: failed to verify simple web app runs on cluster "cluster-candidate-4-20-q5x22b"
Unexpected error:
<*fmt.wrapError | 0x21a3a7aea6e0>:
DNS for route host agnhost-e2e-sample-app-27qvf.apps.aro.l8d0x7x9p0e3b5l.tq9n.eastus2.aroapp-hcp.io did not resolve: DNS for agnhost-e2e-sample-app-27qvf.apps.aro.l8d0x7x9p0e3b5l.tq9n.eastus2.aroapp-hcp.io did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-27qvf.apps.aro.l8d0x7x9p0e3b5l.tq9n.eastus2.aroapp-hcp.io on 172.30.0.10:53: no such host): context deadline exceeded
{
msg: "DNS for route host agnhost-e2e-sample-app-27qvf.apps.aro.l8d0x7x9p0e3b5l.tq9n.eastus2.aroapp-hcp.io did not resolve: DNS for agnhost-e2e-sample-app-27qvf.apps.aro.l8d0x7x9p0e3b5l.tq9n.eastus2.aroapp-hcp.io did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-27qvf.apps.aro.l8d0x7x9p0e3b5l.tq9n.eastus2.aroapp-hcp.io on 172.30.0.10:53: no such host): context deadline exceeded",
err: <*fmt.wrapError | 0x21a3a7548980>{
msg: "DNS for agnhost-e2e-sample-app-27qvf.apps.aro.l8d0x7x9p0e3b5l.tq9n.eastus2.aroapp-hcp.io did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-27qvf.apps.aro.l8d0x7x9p0e3b5l.tq9n.eastus2.aroapp-hcp.io on 172.30.0.10:53: no such host): context deadline exceeded",
err: <context.deadlineExceededError>{},
},
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:190]: failed to verify simple web app runs on cluster "cluster-candidate-4-20-q5x22b"
Unexpected error:
<*fmt.wrapError | 0x21a3a7aea6e0>:
DNS for route host agnhost-e2e-sample-app-27qvf.apps.aro.l8d0x7x9p0e3b5l.tq9n.eastus2.aroapp-hcp.io did not resolve: DNS for agnhost-e2e-sample-app-27qvf.apps.aro.l8d0x7x9p0e3b5l.tq9n.eastus2.aroapp-hcp.io did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-27qvf.apps.aro.l8d0x7x9p0e3b5l.tq9n.eastus2.aroapp-hcp.io on 172.30.0.10:53: no such host): context deadline exceeded
{
msg: "DNS for route host agnhost-e2e-sample-app-27qvf.apps.aro.l8d0x7x9p0e3b5l.tq9n.eastus2.aroapp-hcp.io did not resolve: DNS for agnhost-e2e-sample-app-27qvf.apps.aro.l8d0x7x9p0e3b5l.tq9n.eastus2.aroapp-hcp.io did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-27qvf.apps.aro.l8d0x7x9p0e3b5l.tq9n.eastus2.aroapp-hcp.io on 172.30.0.10:53: no such host): context deadline exceeded",
err: <*fmt.wrapError | 0x21a3a7548980>{
msg: "DNS for agnhost-e2e-sample-app-27qvf.apps.aro.l8d0x7x9p0e3b5l.tq9n.eastus2.aroapp-hcp.io did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-27qvf.apps.aro.l8d0x7x9p0e3b5l.tq9n.eastus2.aroapp-hcp.io on 172.30.0.10:53: no such host): context deadline exceeded",
err: <context.deadlineExceededError>{},
},
}
occurredfail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:190]: failed to verify simple web app runs on cluster "cluster-candidate-4-21-5p9hrv"
Unexpected error:
<*fmt.wrapError | 0x359991ca6200>:
DNS for route host agnhost-e2e-sample-app-dbdwm.apps.aro.n0p5j9k4k3x6p4o.lcm9.eastus2.aroapp-hcp.io did not resolve: DNS for agnhost-e2e-sample-app-dbdwm.apps.aro.n0p5j9k4k3x6p4o.lcm9.eastus2.aroapp-hcp.io did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-dbdwm.apps.aro.n0p5j9k4k3x6p4o.lcm9.eastus2.aroapp-hcp.io on 172.30.0.10:53: no such host): context deadline exceeded
{
msg: "DNS for route host agnhost-e2e-sample-app-dbdwm.apps.aro.n0p5j9k4k3x6p4o.lcm9.eastus2.aroapp-hcp.io did not resolve: DNS for agnhost-e2e-sample-app-dbdwm.apps.aro.n0p5j9k4k3x6p4o.lcm9.eastus2.aroapp-hcp.io did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-dbdwm.apps.aro.n0p5j9k4k3x6p4o.lcm9.eastus2.aroapp-hcp.io on 172.30.0.10:53: no such host): context deadline exceeded",
err: <*fmt.wrapError | 0x359991cc83e0>{
msg: "DNS for agnhost-e2e-sample-app-dbdwm.apps.aro.n0p5j9k4k3x6p4o.lcm9.eastus2.aroapp-hcp.io did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-dbdwm.apps.aro.n0p5j9k4k3x6p4o.lcm9.eastus2.aroapp-hcp.io on 172.30.0.10:53: no such host): context deadline exceeded",
err: <context.deadlineExceededError>{},
},
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:190]: failed to verify simple web app runs on cluster "cluster-candidate-4-21-5p9hrv"
Unexpected error:
<*fmt.wrapError | 0x359991ca6200>:
DNS for route host agnhost-e2e-sample-app-dbdwm.apps.aro.n0p5j9k4k3x6p4o.lcm9.eastus2.aroapp-hcp.io did not resolve: DNS for agnhost-e2e-sample-app-dbdwm.apps.aro.n0p5j9k4k3x6p4o.lcm9.eastus2.aroapp-hcp.io did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-dbdwm.apps.aro.n0p5j9k4k3x6p4o.lcm9.eastus2.aroapp-hcp.io on 172.30.0.10:53: no such host): context deadline exceeded
{
msg: "DNS for route host agnhost-e2e-sample-app-dbdwm.apps.aro.n0p5j9k4k3x6p4o.lcm9.eastus2.aroapp-hcp.io did not resolve: DNS for agnhost-e2e-sample-app-dbdwm.apps.aro.n0p5j9k4k3x6p4o.lcm9.eastus2.aroapp-hcp.io did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-dbdwm.apps.aro.n0p5j9k4k3x6p4o.lcm9.eastus2.aroapp-hcp.io on 172.30.0.10:53: no such host): context deadline exceeded",
err: <*fmt.wrapError | 0x359991cc83e0>{
msg: "DNS for agnhost-e2e-sample-app-dbdwm.apps.aro.n0p5j9k4k3x6p4o.lcm9.eastus2.aroapp-hcp.io did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-dbdwm.apps.aro.n0p5j9k4k3x6p4o.lcm9.eastus2.aroapp-hcp.io on 172.30.0.10:53: no such host): context deadline exceeded",
err: <context.deadlineExceededError>{},
},
}
occurredContributing tests (5)
Affected runs (2)
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedC...full failure pattern: ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider, ignition-server-proxy]; provider Microsoft.RedHatOpenShift Signal: IndeterminateSignal: Indeterminate — no prior history; active 2 day(s) — New failure pattern — no prior history | e2e | 2 | 2.08%2 of 96 job runs affected | IndeterminateSignal: Indeterminate — no prior history; active 2 day(s) — New failure pattern — no prior history | Show trend detailsOct 12: 0 · Oct 13: 0 · Oct 14: 0 · Oct 15: 0 · Oct 16: 0 · Oct 17: 0 · Oct 18: 0 · Oct 19: 0 · Oct 20: 4 · Oct 21: 0 · Oct 22: 2 · Oct 23: 0 · Oct 24: 0 · Oct 25: 0 | none | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (3)fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:303]: failed to create HCP cluster np-noedge-nbxbs6
Unexpected error:
<*fmt.wrapError | 0x24bdb9a64a20>:
failed to create HCP cluster np-noedge-nbxbs6: failed waiting for cluster="np-noedge-nbxbs6" in resourcegroup="rg-np-noedge-nbxbs6-kqkfkllsq598" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/66bec196-bb1d-462f-9d3a-76228415a662
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/66bec196-bb1d-462f-9d3a-76228415a662",
"name": "66bec196-bb1d-462f-9d3a-76228415a662",
"status": "Failed",
"startTime": "2026-10-22T03:10:43.148572951Z",
"endTime": "2026-10-22T03:39:45.079565863Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider deployment has 1 unavailable replicas, ignition-server-proxy deployment has 1 unavailable replicas]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/66bec196-bb1d-462f-9d3a-76228415a662
correlation ID: 547b3712-e9f5-410b-87e2-dd2721dd720f
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:303]: failed to create HCP cluster np-noedge-nbxbs6
Unexpected error:
<*fmt.wrapError | 0x24bdb9a64a20>:
failed to create HCP cluster np-noedge-nbxbs6: failed waiting for cluster="np-noedge-nbxbs6" in resourcegroup="rg-np-noedge-nbxbs6-kqkfkllsq598" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/66bec196-bb1d-462f-9d3a-76228415a662
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/66bec196-bb1d-462f-9d3a-76228415a662",
"name": "66bec196-bb1d-462f-9d3a-76228415a662",
"status": "Failed",
"startTime": "2026-10-22T03:10:43.148572951Z",
"endTime": "2026-10-22T03:39:45.079565863Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider deployment has 1 unavailable replicas, ignition-server-proxy deployment has 1 unavailable replicas]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/66bec196-bb1d-462f-9d3a-76228415a662
correlation ID: 547b3712-e9f5-410b-87e2-dd2721dd720f
client request ID: unavailable
occurredfail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:722]: failed to create HCP cluster
Unexpected error:
<*fmt.wrapError | 0x1f5c59786c0>:
failed to create HCP cluster np-dg-minor-8ks925: failed waiting for cluster="np-dg-minor-8ks925" in resourcegroup="rg-np-dg-minor-8ks925-jgl47czbdtkx" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/3c0ead9e-e84e-4227-ab86-537912304e63
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/3c0ead9e-e84e-4227-ab86-537912304e63",
"name": "3c0ead9e-e84e-4227-ab86-537912304e63",
"status": "Failed",
"startTime": "2026-10-22T03:10:59.807393048Z",
"endTime": "2026-10-22T03:40:05.155981258Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider deployment has 1 unavailable replicas, ignition-server-proxy deployment has 1 unavailable replicas]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/3c0ead9e-e84e-4227-ab86-537912304e63
correlation ID: f292fc5b-a9dd-4382-9a58-48e706cd1073
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:722]: failed to create HCP cluster
Unexpected error:
<*fmt.wrapError | 0x1f5c59786c0>:
failed to create HCP cluster np-dg-minor-8ks925: failed waiting for cluster="np-dg-minor-8ks925" in resourcegroup="rg-np-dg-minor-8ks925-jgl47czbdtkx" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/3c0ead9e-e84e-4227-ab86-537912304e63
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/3c0ead9e-e84e-4227-ab86-537912304e63",
"name": "3c0ead9e-e84e-4227-ab86-537912304e63",
"status": "Failed",
"startTime": "2026-10-22T03:10:59.807393048Z",
"endTime": "2026-10-22T03:40:05.155981258Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider deployment has 1 unavailable replicas, ignition-server-proxy deployment has 1 unavailable replicas]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/3c0ead9e-e84e-4227-ab86-537912304e63
correlation ID: f292fc5b-a9dd-4382-9a58-48e706cd1073
client request ID: unavailable
occurredfail [github.com/Azure/ARO-HCP/test/e2e/oidc_issuer_workload_identity.go:227]: failed to create HCP cluster oidc-wi-cluster
Unexpected error:
<*fmt.wrapError | 0x253bb877e5a0>:
failed to create HCP cluster oidc-wi-cluster: failed waiting for cluster="oidc-wi-cluster" in resourcegroup="oidc-wi-nng4k2kf29bc" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/b46a45a4-dcba-413a-9b74-f904cf76e493
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/b46a45a4-dcba-413a-9b74-f904cf76e493",
"name": "b46a45a4-dcba-413a-9b74-f904cf76e493",
"status": "Failed",
"startTime": "2026-10-20T06:29:17.820195399Z",
"endTime": "2026-10-20T06:58:20.185763239Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider deployment has 1 unavailable replicas, ignition-server-proxy deployment has 1 unavailable replicas]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/b46a45a4-dcba-413a-9b74-f904cf76e493
correlation ID: a247ec97-7f49-437d-bb0b-6d5fc996e055
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/oidc_issuer_workload_identity.go:227]: failed to create HCP cluster oidc-wi-cluster
Unexpected error:
<*fmt.wrapError | 0x253bb877e5a0>:
failed to create HCP cluster oidc-wi-cluster: failed waiting for cluster="oidc-wi-cluster" in resourcegroup="oidc-wi-nng4k2kf29bc" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/b46a45a4-dcba-413a-9b74-f904cf76e493
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/b46a45a4-dcba-413a-9b74-f904cf76e493",
"name": "b46a45a4-dcba-413a-9b74-f904cf76e493",
"status": "Failed",
"startTime": "2026-10-20T06:29:17.820195399Z",
"endTime": "2026-10-20T06:58:20.185763239Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider deployment has 1 unavailable replicas, ignition-server-proxy deployment has 1 unavailable replicas]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/b46a45a4-dcba-413a-9b74-f904cf76e493
correlation ID: a247ec97-7f49-437d-bb0b-6d5fc996e055
client request ID: unavailable
occurredContributing tests (6)
Affected runs (2)
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedC...full failure pattern: ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster has no control plane endpoint host; provider Microsoft.RedHatOpenShift Signal: IndeterminateSignal: Indeterminate — no prior history; active 2 day(s) — New failure pattern — no prior history | e2e | 2 | 2.08%2 of 96 job runs affected | IndeterminateSignal: Indeterminate — no prior history; active 2 day(s) — New failure pattern — no prior history | Show trend detailsOct 12: 0 · Oct 13: 0 · Oct 14: 0 · Oct 15: 0 · Oct 16: 0 · Oct 17: 0 · Oct 18: 0 · Oct 19: 0 · Oct 20: 3 · Oct 21: 0 · Oct 22: 2 · Oct 23: 0 · Oct 24: 0 · Oct 25: 0 | DEV | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (3)fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:291]: failed to create HCP cluster node-limit-cluster with MaxNodesTotal limit
Unexpected error:
<*fmt.wrapError | 0x262004875180>:
failed to create HCP cluster node-limit-cluster: failed waiting for cluster="node-limit-cluster" in resourcegroup="node-limit-test-v4w4cbtgl6st" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/0ffe01b2-d2c4-4e16-a6d6-042cbbeb51ec
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/0ffe01b2-d2c4-4e16-a6d6-042cbbeb51ec",
"name": "0ffe01b2-d2c4-4e16-a6d6-042cbbeb51ec",
"status": "Failed",
"startTime": "2026-10-22T02:27:27.45827201Z",
"endTime": "2026-10-22T02:56:30.386782585Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster has no control plane endpoint host"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/0ffe01b2-d2c4-4e16-a6d6-042cbbeb51ec
correlation ID: a0e42f8f-bf63-4670-b77c-e6ba06f19444
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:291]: failed to create HCP cluster node-limit-cluster with MaxNodesTotal limit
Unexpected error:
<*fmt.wrapError | 0x262004875180>:
failed to create HCP cluster node-limit-cluster: failed waiting for cluster="node-limit-cluster" in resourcegroup="node-limit-test-v4w4cbtgl6st" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/0ffe01b2-d2c4-4e16-a6d6-042cbbeb51ec
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/0ffe01b2-d2c4-4e16-a6d6-042cbbeb51ec",
"name": "0ffe01b2-d2c4-4e16-a6d6-042cbbeb51ec",
"status": "Failed",
"startTime": "2026-10-22T02:27:27.45827201Z",
"endTime": "2026-10-22T02:56:30.386782585Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster has no control plane endpoint host"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/0ffe01b2-d2c4-4e16-a6d6-042cbbeb51ec
correlation ID: a0e42f8f-bf63-4670-b77c-e6ba06f19444
client request ID: unavailable
occurredfail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:133]: failed to create HCP cluster "cp-ystream-upgrade-4-21-q2kkkp" at install version 4.20
Unexpected error:
<*fmt.wrapError | 0x300bf5b7a840>:
failed to create HCP cluster cp-ystream-upgrade-4-21-q2kkkp: failed waiting for cluster="cp-ystream-upgrade-4-21-q2kkkp" in resourcegroup="rg-cp-ystream-upgrade-4-21-t4mn7sf29xrr" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/54a4385e-0b42-4987-acf3-1b0207b030a5
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/54a4385e-0b42-4987-acf3-1b0207b030a5",
"name": "54a4385e-0b42-4987-acf3-1b0207b030a5",
"status": "Failed",
"startTime": "2026-10-22T02:23:48.751941498Z",
"endTime": "2026-10-22T02:52:50.375994252Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster has no control plane endpoint host"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/54a4385e-0b42-4987-acf3-1b0207b030a5
correlation ID: f3ca853a-64dc-4e52-ab57-050647054f81
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:133]: failed to create HCP cluster "cp-ystream-upgrade-4-21-q2kkkp" at install version 4.20
Unexpected error:
<*fmt.wrapError | 0x300bf5b7a840>:
failed to create HCP cluster cp-ystream-upgrade-4-21-q2kkkp: failed waiting for cluster="cp-ystream-upgrade-4-21-q2kkkp" in resourcegroup="rg-cp-ystream-upgrade-4-21-t4mn7sf29xrr" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/54a4385e-0b42-4987-acf3-1b0207b030a5
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/54a4385e-0b42-4987-acf3-1b0207b030a5",
"name": "54a4385e-0b42-4987-acf3-1b0207b030a5",
"status": "Failed",
"startTime": "2026-10-22T02:23:48.751941498Z",
"endTime": "2026-10-22T02:52:50.375994252Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster has no control plane endpoint host"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/54a4385e-0b42-4987-acf3-1b0207b030a5
correlation ID: f3ca853a-64dc-4e52-ab57-050647054f81
client request ID: unavailable
occurredfail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:106]: failed to create HCP cluster "cilium-cluster" with no CNI and private etcd
Unexpected error:
<*fmt.wrapError | 0x106428050380>:
failed waiting for cluster="cilium-cluster" in resourcegroup="complex-cilium-kv-85x64ntjvwjg" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/65962c7a-48f0-4bdf-835e-094037d9bd68
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/65962c7a-48f0-4bdf-835e-094037d9bd68",
"name": "65962c7a-48f0-4bdf-835e-094037d9bd68",
"status": "Failed",
"startTime": "2026-10-20T04:48:06.594464666Z",
"endTime": "2026-10-20T05:17:15.387268555Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster has no control plane endpoint host"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/65962c7a-48f0-4bdf-835e-094037d9bd68
correlation ID: 64f4406b-d45b-4b7b-8f16-7904d655ca01
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:106]: failed to create HCP cluster "cilium-cluster" with no CNI and private etcd
Unexpected error:
<*fmt.wrapError | 0x106428050380>:
failed waiting for cluster="cilium-cluster" in resourcegroup="complex-cilium-kv-85x64ntjvwjg" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/65962c7a-48f0-4bdf-835e-094037d9bd68
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/65962c7a-48f0-4bdf-835e-094037d9bd68",
"name": "65962c7a-48f0-4bdf-835e-094037d9bd68",
"status": "Failed",
"startTime": "2026-10-20T04:48:06.594464666Z",
"endTime": "2026-10-20T05:17:15.387268555Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster has no control plane endpoint host"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/65962c7a-48f0-4bdf-835e-094037d9bd68
correlation ID: 64f4406b-d45b-4b7b-8f16-7904d655ca01
client request ID: unavailable
occurredContributing tests (5)
Affected runs (2)
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
VerifySimpleWebApp failed: DNS for route host <route-host> did not resolve: DNS for <route-host> did not resolve within <duration> (last error: lookup <host> on <dns-server>: dial...full failure pattern: VerifySimpleWebApp failed: DNS for route host <route-host> did not resolve: DNS for <route-host> did not resolve within <duration> (last error: lookup <host> on <dns-server>: dial udp <ip>: i/o timeout): context deadline exceeded Signal: IndeterminateSignal: Indeterminate — no prior history; active 2 day(s) — New failure pattern — no prior history | e2e | 2 | 2.08%2 of 96 job runs affected | IndeterminateSignal: Indeterminate — no prior history; active 2 day(s) — New failure pattern — no prior history | Show trend detailsOct 12: 0 · Oct 13: 0 · Oct 14: 0 · Oct 15: 0 · Oct 16: 0 · Oct 17: 0 · Oct 18: 0 · Oct 19: 0 · Oct 20: 0 · Oct 21: 0 · Oct 22: 0 · Oct 23: 3 · Oct 24: 0 · Oct 25: 2 | DEV, INT | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (3)fail [github.com/Azure/ARO-HCP/test/e2e/cluster_autoscaling.go:203]: failed to verify cluster workloads and pods after autoscaling update
Unexpected error:
<*errors.joinError | 0x33a1f1315158>:
VerifySimpleWebApp failed: DNS for route host agnhost-e2e-sample-app-59cgj.apps.aro.s4i2d1i2e0s3o3m.o0jt.eastus2.aroapp-hcp.io did not resolve: DNS for agnhost-e2e-sample-app-59cgj.apps.aro.s4i2d1i2e0s3o3m.o0jt.eastus2.aroapp-hcp.io did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-59cgj.apps.aro.s4i2d1i2e0s3o3m.o0jt.eastus2.aroapp-hcp.io on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded
{
errs: [
<*fmt.wrapError | 0x33a1f0cea780>{
msg: "VerifySimpleWebApp failed: DNS for route host agnhost-e2e-sample-app-59cgj.apps.aro.s4i2d1i2e0s3o3m.o0jt.eastus2.aroapp-hcp.io did not resolve: DNS for agnhost-e2e-sample-app-59cgj.apps.aro.s4i2d1i2e0s3o3m.o0jt.eastus2.aroapp-hcp.io did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-59cgj.apps.aro.s4i2d1i2e0s3o3m.o0jt.eastus2.aroapp-hcp.io on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
err: <*fmt.wrapError | 0x33a1f0bd2f00>{
msg: "DNS for route host agnhost-e2e-sample-app-59cgj.apps.aro.s4i2d1i2e0s3o3m.o0jt.eastus2.aroapp-hcp.io did not resolve: DNS for agnhost-e2e-sample-app-59cgj.apps.aro.s4i2d1i2e0s3o3m.o0jt.eastus2.aroapp-hcp.io did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-59cgj.apps.aro.s4i2d1i2e0s3o3m.o0jt.eastus2.aroapp-hcp.io on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
err: <*fmt.wrapError | 0x33a1f13e4360>{
msg: "DNS for agnhost-e2e-sample-app-59cgj.apps.aro.s4i2d1i2e0s3o3m.o0jt.eastus2.aroapp-hcp.io did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-59cgj.apps.aro.s4i2d1i2e0s3o3m.o0jt.eastus2.aroapp-hcp.io on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
err: <context.deadlineExceededError>{},
},
},
},
],
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_autoscaling.go:203]: failed to verify cluster workloads and pods after autoscaling update
Unexpected error:
<*errors.joinError | 0x33a1f1315158>:
VerifySimpleWebApp failed: DNS for route host agnhost-e2e-sample-app-59cgj.apps.aro.s4i2d1i2e0s3o3m.o0jt.eastus2.aroapp-hcp.io did not resolve: DNS for agnhost-e2e-sample-app-59cgj.apps.aro.s4i2d1i2e0s3o3m.o0jt.eastus2.aroapp-hcp.io did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-59cgj.apps.aro.s4i2d1i2e0s3o3m.o0jt.eastus2.aroapp-hcp.io on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded
{
errs: [
<*fmt.wrapError | 0x33a1f0cea780>{
msg: "VerifySimpleWebApp failed: DNS for route host agnhost-e2e-sample-app-59cgj.apps.aro.s4i2d1i2e0s3o3m.o0jt.eastus2.aroapp-hcp.io did not resolve: DNS for agnhost-e2e-sample-app-59cgj.apps.aro.s4i2d1i2e0s3o3m.o0jt.eastus2.aroapp-hcp.io did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-59cgj.apps.aro.s4i2d1i2e0s3o3m.o0jt.eastus2.aroapp-hcp.io on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
err: <*fmt.wrapError | 0x33a1f0bd2f00>{
msg: "DNS for route host agnhost-e2e-sample-app-59cgj.apps.aro.s4i2d1i2e0s3o3m.o0jt.eastus2.aroapp-hcp.io did not resolve: DNS for agnhost-e2e-sample-app-59cgj.apps.aro.s4i2d1i2e0s3o3m.o0jt.eastus2.aroapp-hcp.io did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-59cgj.apps.aro.s4i2d1i2e0s3o3m.o0jt.eastus2.aroapp-hcp.io on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
err: <*fmt.wrapError | 0x33a1f13e4360>{
msg: "DNS for agnhost-e2e-sample-app-59cgj.apps.aro.s4i2d1i2e0s3o3m.o0jt.eastus2.aroapp-hcp.io did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-59cgj.apps.aro.s4i2d1i2e0s3o3m.o0jt.eastus2.aroapp-hcp.io on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
err: <context.deadlineExceededError>{},
},
},
},
],
}
occurredfail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_cni_cilium.go:186]: failed to run simple web app and connectivity check app with cilium CNI
Unexpected error:
<*errors.joinError | 0xeeb7f5e0a68>:
VerifySimpleWebApp failed: DNS for route host agnhost-e2e-sample-app-4kcwc.apps.aro.cilium-cl.en3z.eastus2.aroapp-hcp.io did not resolve: DNS for agnhost-e2e-sample-app-4kcwc.apps.aro.cilium-cl.en3z.eastus2.aroapp-hcp.io did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-4kcwc.apps.aro.cilium-cl.en3z.eastus2.aroapp-hcp.io on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded
{
errs: [
<*fmt.wrapError | 0xeeb7f3b69e0>{
msg: "VerifySimpleWebApp failed: DNS for route host agnhost-e2e-sample-app-4kcwc.apps.aro.cilium-cl.en3z.eastus2.aroapp-hcp.io did not resolve: DNS for agnhost-e2e-sample-app-4kcwc.apps.aro.cilium-cl.en3z.eastus2.aroapp-hcp.io did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-4kcwc.apps.aro.cilium-cl.en3z.eastus2.aroapp-hcp.io on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
err: <*fmt.wrapError | 0xeeb7f5137a0>{
msg: "DNS for route host agnhost-e2e-sample-app-4kcwc.apps.aro.cilium-cl.en3z.eastus2.aroapp-hcp.io did not resolve: DNS for agnhost-e2e-sample-app-4kcwc.apps.aro.cilium-cl.en3z.eastus2.aroapp-hcp.io did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-4kcwc.apps.aro.cilium-cl.en3z.eastus2.aroapp-hcp.io on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
err: <*fmt.wrapError | 0xeeb7f512040>{
msg: "DNS for agnhost-e2e-sample-app-4kcwc.apps.aro.cilium-cl.en3z.eastus2.aroapp-hcp.io did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-4kcwc.apps.aro.cilium-cl.en3z.eastus2.aroapp-hcp.io on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
err: <context.deadlineExceededError>{},
},
},
},
],
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_cni_cilium.go:186]: failed to run simple web app and connectivity check app with cilium CNI
Unexpected error:
<*errors.joinError | 0xeeb7f5e0a68>:
VerifySimpleWebApp failed: DNS for route host agnhost-e2e-sample-app-4kcwc.apps.aro.cilium-cl.en3z.eastus2.aroapp-hcp.io did not resolve: DNS for agnhost-e2e-sample-app-4kcwc.apps.aro.cilium-cl.en3z.eastus2.aroapp-hcp.io did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-4kcwc.apps.aro.cilium-cl.en3z.eastus2.aroapp-hcp.io on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded
{
errs: [
<*fmt.wrapError | 0xeeb7f3b69e0>{
msg: "VerifySimpleWebApp failed: DNS for route host agnhost-e2e-sample-app-4kcwc.apps.aro.cilium-cl.en3z.eastus2.aroapp-hcp.io did not resolve: DNS for agnhost-e2e-sample-app-4kcwc.apps.aro.cilium-cl.en3z.eastus2.aroapp-hcp.io did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-4kcwc.apps.aro.cilium-cl.en3z.eastus2.aroapp-hcp.io on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
err: <*fmt.wrapError | 0xeeb7f5137a0>{
msg: "DNS for route host agnhost-e2e-sample-app-4kcwc.apps.aro.cilium-cl.en3z.eastus2.aroapp-hcp.io did not resolve: DNS for agnhost-e2e-sample-app-4kcwc.apps.aro.cilium-cl.en3z.eastus2.aroapp-hcp.io did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-4kcwc.apps.aro.cilium-cl.en3z.eastus2.aroapp-hcp.io on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
err: <*fmt.wrapError | 0xeeb7f512040>{
msg: "DNS for agnhost-e2e-sample-app-4kcwc.apps.aro.cilium-cl.en3z.eastus2.aroapp-hcp.io did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-4kcwc.apps.aro.cilium-cl.en3z.eastus2.aroapp-hcp.io on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
err: <context.deadlineExceededError>{},
},
},
},
],
}
occurredfail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:197]: failed to run simple web app and connectivity check with cilium CNI
Unexpected error:
<*errors.joinError | 0xdf1d3a48c78>:
VerifySimpleWebApp failed: DNS for route host agnhost-e2e-sample-app-rx2sd.apps.aro.cilium-cluster.k38l.eastus2.aroapp-hcp.io did not resolve: DNS for agnhost-e2e-sample-app-rx2sd.apps.aro.cilium-cluster.k38l.eastus2.aroapp-hcp.io did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-rx2sd.apps.aro.cilium-cluster.k38l.eastus2.aroapp-hcp.io on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded
{
errs: [
<*fmt.wrapError | 0xdf1d48b58c0>{
msg: "VerifySimpleWebApp failed: DNS for route host agnhost-e2e-sample-app-rx2sd.apps.aro.cilium-cluster.k38l.eastus2.aroapp-hcp.io did not resolve: DNS for agnhost-e2e-sample-app-rx2sd.apps.aro.cilium-cluster.k38l.eastus2.aroapp-hcp.io did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-rx2sd.apps.aro.cilium-cluster.k38l.eastus2.aroapp-hcp.io on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
err: <*fmt.wrapError | 0xdf1d48b46a0>{
msg: "DNS for route host agnhost-e2e-sample-app-rx2sd.apps.aro.cilium-cluster.k38l.eastus2.aroapp-hcp.io did not resolve: DNS for agnhost-e2e-sample-app-rx2sd.apps.aro.cilium-cluster.k38l.eastus2.aroapp-hcp.io did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-rx2sd.apps.aro.cilium-cluster.k38l.eastus2.aroapp-hcp.io on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
err: <*fmt.wrapError | 0xdf1d394a3a0>{
msg: "DNS for agnhost-e2e-sample-app-rx2sd.apps.aro.cilium-cluster.k38l.eastus2.aroapp-hcp.io did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-rx2sd.apps.aro.cilium-cluster.k38l.eastus2.aroapp-hcp.io on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
err: <context.deadlineExceededError>{},
},
},
},
],
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:197]: failed to run simple web app and connectivity check with cilium CNI
Unexpected error:
<*errors.joinError | 0xdf1d3a48c78>:
VerifySimpleWebApp failed: DNS for route host agnhost-e2e-sample-app-rx2sd.apps.aro.cilium-cluster.k38l.eastus2.aroapp-hcp.io did not resolve: DNS for agnhost-e2e-sample-app-rx2sd.apps.aro.cilium-cluster.k38l.eastus2.aroapp-hcp.io did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-rx2sd.apps.aro.cilium-cluster.k38l.eastus2.aroapp-hcp.io on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded
{
errs: [
<*fmt.wrapError | 0xdf1d48b58c0>{
msg: "VerifySimpleWebApp failed: DNS for route host agnhost-e2e-sample-app-rx2sd.apps.aro.cilium-cluster.k38l.eastus2.aroapp-hcp.io did not resolve: DNS for agnhost-e2e-sample-app-rx2sd.apps.aro.cilium-cluster.k38l.eastus2.aroapp-hcp.io did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-rx2sd.apps.aro.cilium-cluster.k38l.eastus2.aroapp-hcp.io on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
err: <*fmt.wrapError | 0xdf1d48b46a0>{
msg: "DNS for route host agnhost-e2e-sample-app-rx2sd.apps.aro.cilium-cluster.k38l.eastus2.aroapp-hcp.io did not resolve: DNS for agnhost-e2e-sample-app-rx2sd.apps.aro.cilium-cluster.k38l.eastus2.aroapp-hcp.io did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-rx2sd.apps.aro.cilium-cluster.k38l.eastus2.aroapp-hcp.io on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
err: <*fmt.wrapError | 0xdf1d394a3a0>{
msg: "DNS for agnhost-e2e-sample-app-rx2sd.apps.aro.cilium-cluster.k38l.eastus2.aroapp-hcp.io did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-rx2sd.apps.aro.cilium-cluster.k38l.eastus2.aroapp-hcp.io on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
err: <context.deadlineExceededError>{},
},
},
},
],
}
occurredContributing tests (4)
Affected runs (2)
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Customer should be able to test admin credentials before cluster ready, then full admin credential lifecycle: Cluster provisioning failedfull failure pattern: Customer should be able to test admin credentials before cluster ready, then full admin credential lifecycle: Cluster provisioning failed Signal: IndeterminateSignal: Indeterminate — seen in 2 prior week(s); active 2 day(s) | e2e | 2 | 2.08%2 of 96 job runs affected | IndeterminateSignal: Indeterminate — seen in 2 prior week(s); active 2 day(s) | Show trend detailsOct 12: 0 · Oct 13: 0 · Oct 14: 0 · Oct 15: 0 · Oct 16: 0 · Oct 17: 0 · Oct 18: 0 · Oct 19: 0 · Oct 20: 1 · Oct 21: 0 · Oct 22: 1 · Oct 23: 0 · Oct 24: 0 · Oct 25: 0 | DEV | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/e2e/admin_credential_lifecycle.go:158]: Cluster provisioning failed fail [github.com/Azure/ARO-HCP/test/e2e/admin_credential_lifecycle.go:158]: Cluster provisioning failed Contributing tests (1)
Affected runs (2)
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedC...full failure pattern: ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable; hosted cluster degraded: UnavailableReplicas: [ignition-server-proxy, kube-apiserver]; provider Microsoft.RedHatOpenShift Signal: IndeterminateSignal: Indeterminate — no prior history; active 2 day(s) — New failure pattern — no prior history | e2e | 2 | 2.08%2 of 96 job runs affected | IndeterminateSignal: Indeterminate — no prior history; active 2 day(s) — New failure pattern — no prior history | Show trend detailsOct 12: 0 · Oct 13: 0 · Oct 14: 0 · Oct 15: 0 · Oct 16: 0 · Oct 17: 0 · Oct 18: 0 · Oct 19: 0 · Oct 20: 1 · Oct 21: 0 · Oct 22: 1 · Oct 23: 0 · Oct 24: 0 · Oct 25: 0 | none | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (2)fail [github.com/Azure/ARO-HCP/test/e2e/shoebox_logs.go:272]: failed to create HCP cluster shoebox-hcp-cluster
Unexpected error:
<*fmt.wrapError | 0x283eb71121c0>:
failed to create HCP cluster shoebox-hcp-cluster: failed waiting for cluster="shoebox-hcp-cluster" in resourcegroup="shoebox-cluster-wqltj9p8d5tp" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/07e1caae-294a-4885-b57d-e584ce7036c1
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/07e1caae-294a-4885-b57d-e584ce7036c1",
"name": "07e1caae-294a-4885-b57d-e584ce7036c1",
"status": "Failed",
"startTime": "2026-10-22T03:11:14.830364623Z",
"endTime": "2026-10-22T03:40:15.05749374Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: openshift-route-controller-manager, kube-controller-manager, kube-scheduler, openshift-apiserver, cluster-storage-operator, ingress-operator, community-operators-catalog, cluster-version-operator, redhat-marketplace-catalog, certified-operators-catalog, azure-cloud-controller-manager, cluster-network-operator, cluster-image-registry-operator, konnectivity-agent, machine-approver, csi-snapshot-controller-operator, packageserver, cluster-policy-controller, dns-operator, catalog-operator, redhat-operators-catalog, cluster-autoscaler, olm-operator, control-plane-pki-operator, hosted-cluster-config-operator, openshift-controller-manager, cluster-node-tuning-operator, olm-collect-profiles; hosted cluster degraded: UnavailableReplicas: [ignition-server-proxy deployment has 1 unavailable replicas, kube-apiserver deployment has 1 unavailable replicas]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/07e1caae-294a-4885-b57d-e584ce7036c1
correlation ID: 042010a3-104c-478c-8958-4456058069f0
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/shoebox_logs.go:272]: failed to create HCP cluster shoebox-hcp-cluster
Unexpected error:
<*fmt.wrapError | 0x283eb71121c0>:
failed to create HCP cluster shoebox-hcp-cluster: failed waiting for cluster="shoebox-hcp-cluster" in resourcegroup="shoebox-cluster-wqltj9p8d5tp" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/07e1caae-294a-4885-b57d-e584ce7036c1
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/07e1caae-294a-4885-b57d-e584ce7036c1",
"name": "07e1caae-294a-4885-b57d-e584ce7036c1",
"status": "Failed",
"startTime": "2026-10-22T03:11:14.830364623Z",
"endTime": "2026-10-22T03:40:15.05749374Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: openshift-route-controller-manager, kube-controller-manager, kube-scheduler, openshift-apiserver, cluster-storage-operator, ingress-operator, community-operators-catalog, cluster-version-operator, redhat-marketplace-catalog, certified-operators-catalog, azure-cloud-controller-manager, cluster-network-operator, cluster-image-registry-operator, konnectivity-agent, machine-approver, csi-snapshot-controller-operator, packageserver, cluster-policy-controller, dns-operator, catalog-operator, redhat-operators-catalog, cluster-autoscaler, olm-operator, control-plane-pki-operator, hosted-cluster-config-operator, openshift-controller-manager, cluster-node-tuning-operator, olm-collect-profiles; hosted cluster degraded: UnavailableReplicas: [ignition-server-proxy deployment has 1 unavailable replicas, kube-apiserver deployment has 1 unavailable replicas]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/07e1caae-294a-4885-b57d-e584ce7036c1
correlation ID: 042010a3-104c-478c-8958-4456058069f0
client request ID: unavailable
occurredfail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:125]: failed to create HCP cluster "cluster-zstream-4-22-s5dcsw" with version 4.22.18 on candidate channel
Unexpected error:
<*fmt.wrapError | 0x15e376eb4ba0>:
failed to create HCP cluster cluster-zstream-4-22-s5dcsw: failed waiting for cluster="cluster-zstream-4-22-s5dcsw" in resourcegroup="rg-zstream-upgrade-4-22-t6t94hh7j44m" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/08e66858-550b-405f-b39d-e72d4d3457b0
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/08e66858-550b-405f-b39d-e72d4d3457b0",
"name": "08e66858-550b-405f-b39d-e72d4d3457b0",
"status": "Failed",
"startTime": "2026-10-20T05:35:01.18923569Z",
"endTime": "2026-10-20T06:04:10.578239766Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: cluster-version-operator, kube-storage-version-migrator, azure-cloud-controller-manager, packageserver, community-operators-catalog, cluster-autoscaler, machine-approver, control-plane-pki-operator, openshift-apiserver, hosted-cluster-config-operator, dns-operator, catalog-operator, cluster-storage-operator, ingress-operator, konnectivity-agent, openshift-controller-manager, olm-collect-profiles, kube-controller-manager, kube-scheduler, openshift-route-controller-manager, cluster-policy-controller, cluster-network-operator, cluster-node-tuning-operator, csi-snapshot-controller-operator, cluster-image-registry-operator, redhat-operators-catalog, olm-operator, certified-operators-catalog; hosted cluster degraded: UnavailableReplicas: [ignition-server-proxy deployment has 1 unavailable replicas, kube-apiserver deployment has 1 unavailable replicas]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/08e66858-550b-405f-b39d-e72d4d3457b0
correlation ID: feffceb9-72b0-45a3-8f5c-db4fbe3e6004
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:125]: failed to create HCP cluster "cluster-zstream-4-22-s5dcsw" with version 4.22.18 on candidate channel
Unexpected error:
<*fmt.wrapError | 0x15e376eb4ba0>:
failed to create HCP cluster cluster-zstream-4-22-s5dcsw: failed waiting for cluster="cluster-zstream-4-22-s5dcsw" in resourcegroup="rg-zstream-upgrade-4-22-t6t94hh7j44m" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/08e66858-550b-405f-b39d-e72d4d3457b0
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/08e66858-550b-405f-b39d-e72d4d3457b0",
"name": "08e66858-550b-405f-b39d-e72d4d3457b0",
"status": "Failed",
"startTime": "2026-10-20T05:35:01.18923569Z",
"endTime": "2026-10-20T06:04:10.578239766Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: cluster-version-operator, kube-storage-version-migrator, azure-cloud-controller-manager, packageserver, community-operators-catalog, cluster-autoscaler, machine-approver, control-plane-pki-operator, openshift-apiserver, hosted-cluster-config-operator, dns-operator, catalog-operator, cluster-storage-operator, ingress-operator, konnectivity-agent, openshift-controller-manager, olm-collect-profiles, kube-controller-manager, kube-scheduler, openshift-route-controller-manager, cluster-policy-controller, cluster-network-operator, cluster-node-tuning-operator, csi-snapshot-controller-operator, cluster-image-registry-operator, redhat-operators-catalog, olm-operator, certified-operators-catalog; hosted cluster degraded: UnavailableReplicas: [ignition-server-proxy deployment has 1 unavailable replicas, kube-apiserver deployment has 1 unavailable replicas]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/08e66858-550b-405f-b39d-e72d4d3457b0
correlation ID: feffceb9-72b0-45a3-8f5c-db4fbe3e6004
client request ID: unavailable
occurredContributing tests (2)
Affected runs (2)
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
ERROR CODE: InternalServerError; detail message cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster still exists; [clusterServic...full failure pattern: ERROR CODE: InternalServerError; detail message cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster still exists; [clusterServiceStatus] state is "uninstalling"; [descendantResources] remaining resources: nodePools, serviceProviderClusters, serviceProviderNodePools; provider Microsoft.RedHatOpenShift Signal: IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | e2e | 2 | 2.08%2 of 96 job runs affected | IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | Show trend detailsOct 12: 0 · Oct 13: 0 · Oct 14: 0 · Oct 15: 0 · Oct 16: 0 · Oct 17: 0 · Oct 18: 0 · Oct 19: 0 · Oct 20: 0 · Oct 21: 0 · Oct 22: 2 · Oct 23: 0 · Oct 24: 0 · Oct 25: 0 | DEV | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (2)fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:310]: Unexpected error:
<*errors.joinError | 0x3eb0083dc060>:
resource group "private-keyvault-dtwhsct926n6": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="private-kv-cluster" in resourcegroup="private-keyvault-dtwhsct926n6" to finish deleting: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westeurope/hcpOperationStatuses/baf15df3-25ce-45bb-9f9e-ce6a52637752
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: InternalServerError
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westeurope/hcpOperationStatuses/baf15df3-25ce-45bb-9f9e-ce6a52637752",
"name": "baf15df3-25ce-45bb-9f9e-ce6a52637752",
"status": "Failed",
"startTime": "2026-10-22T05:08:07.811493592Z",
"endTime": "2026-10-22T05:32:16.130563503Z",
"error": {
"code": "InternalServerError",
"message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2tjkddd25goskik0nf1lqaqp6mgu12qg still exists (deletion dispatched at 2026-10-22T05:08:08Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/nodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/nodepools/serviceProviderNodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westeurope/hcpOperationStatuses/baf15df3-25ce-45bb-9f9e-ce6a52637752
correlation ID: a461bf17-668e-42e4-a6f7-c14803c5d886
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:310]: Unexpected error:
<*errors.joinError | 0x3eb0083dc060>:
resource group "private-keyvault-dtwhsct926n6": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="private-kv-cluster" in resourcegroup="private-keyvault-dtwhsct926n6" to finish deleting: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westeurope/hcpOperationStatuses/baf15df3-25ce-45bb-9f9e-ce6a52637752
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: InternalServerError
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westeurope/hcpOperationStatuses/baf15df3-25ce-45bb-9f9e-ce6a52637752",
"name": "baf15df3-25ce-45bb-9f9e-ce6a52637752",
"status": "Failed",
"startTime": "2026-10-22T05:08:07.811493592Z",
"endTime": "2026-10-22T05:32:16.130563503Z",
"error": {
"code": "InternalServerError",
"message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2tjkddd25goskik0nf1lqaqp6mgu12qg still exists (deletion dispatched at 2026-10-22T05:08:08Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/nodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/nodepools/serviceProviderNodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westeurope/hcpOperationStatuses/baf15df3-25ce-45bb-9f9e-ce6a52637752
correlation ID: a461bf17-668e-42e4-a6f7-c14803c5d886
client request ID: unavailable
occurredfail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:310]: Unexpected error:
<*errors.joinError | 0x16e186f694b8>:
resource group "ephemeral-osdisk-k7jcdd2kqwvh": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="ephemeral-disk" in resourcegroup="ephemeral-osdisk-k7jcdd2kqwvh" to finish deleting: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/36a51701-9d7a-4ad4-860f-d0b45481d5d2
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: InternalServerError
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/36a51701-9d7a-4ad4-860f-d0b45481d5d2",
"name": "36a51701-9d7a-4ad4-860f-d0b45481d5d2",
"status": "Failed",
"startTime": "2026-10-22T03:23:14.821436082Z",
"endTime": "2026-10-22T03:47:22.914912373Z",
"error": {
"code": "InternalServerError",
"message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2tjj0cikkek0bc3m05t20hehmct394q5 still exists (deletion dispatched at 2026-10-22T03:23:15Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/nodePools, 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/serviceProviderClusters, 1 microsoft.redhatopenshift/hcpopenshiftclusters/nodepools/serviceProviderNodePools"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/36a51701-9d7a-4ad4-860f-d0b45481d5d2
correlation ID: b94afee3-9d6b-4695-a5b0-3f0b5bdf76c0
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:310]: Unexpected error:
<*errors.joinError | 0x16e186f694b8>:
resource group "ephemeral-osdisk-k7jcdd2kqwvh": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="ephemeral-disk" in resourcegroup="ephemeral-osdisk-k7jcdd2kqwvh" to finish deleting: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/36a51701-9d7a-4ad4-860f-d0b45481d5d2
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: InternalServerError
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/36a51701-9d7a-4ad4-860f-d0b45481d5d2",
"name": "36a51701-9d7a-4ad4-860f-d0b45481d5d2",
"status": "Failed",
"startTime": "2026-10-22T03:23:14.821436082Z",
"endTime": "2026-10-22T03:47:22.914912373Z",
"error": {
"code": "InternalServerError",
"message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2tjj0cikkek0bc3m05t20hehmct394q5 still exists (deletion dispatched at 2026-10-22T03:23:15Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/nodePools, 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/serviceProviderClusters, 1 microsoft.redhatopenshift/hcpopenshiftclusters/nodepools/serviceProviderNodePools"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/36a51701-9d7a-4ad4-860f-d0b45481d5d2
correlation ID: b94afee3-9d6b-4695-a5b0-3f0b5bdf76c0
client request ID: unavailable
occurredContributing tests (2)
Affected runs (2)
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
ERROR CODE: InternalServerError; detail message cluster deletion did not complete before the deadline; [resourceTeardown] tearing down pod network instances, pod networks, namespa...full failure pattern: ERROR CODE: InternalServerError; detail message cluster deletion did not complete before the deadline; [resourceTeardown] tearing down pod network instances, pod networks, namespaces; [descendantResources] remaining resources: applydesires, serviceProviderClusters; provider Microsoft.RedHatOpenShift Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 2 day(s) | e2e | 2 | 2.08%2 of 96 job runs affected | IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 2 day(s) | Show trend detailsOct 12: 0 · Oct 13: 0 · Oct 14: 0 · Oct 15: 0 · Oct 16: 0 · Oct 17: 1 · Oct 18: 0 · Oct 19: 0 · Oct 20: 0 · Oct 21: 0 · Oct 22: 0 · Oct 23: 0 · Oct 24: 1 · Oct 25: 0 | DEV, STG | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (2)fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:310]: Unexpected error:
<*errors.joinError | 0x2874318ca1e0>:
resource group "rg-nightly-4-21-kv7pzz4ch49v": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cluster-nightly-4-21-7x886k" in resourcegroup="rg-nightly-4-21-kv7pzz4ch49v" to finish deleting: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/35ebc587-6a8d-48f1-a4a4-b752ee43dd71
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: InternalServerError
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/35ebc587-6a8d-48f1-a4a4-b752ee43dd71",
"name": "35ebc587-6a8d-48f1-a4a4-b752ee43dd71",
"status": "Failed",
"startTime": "2026-10-24T23:51:18.597955269Z",
"endTime": "2026-10-25T00:15:22.050322008Z",
"error": {
"code": "InternalServerError",
"message": "cluster deletion did not complete before the deadline; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters, 3 microsoft.redhatopenshift/hcpopenshiftclusters/applydesires; [resourceTeardown] tearing down pod network instances (1), pod networks (1), namespaces (2)"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/35ebc587-6a8d-48f1-a4a4-b752ee43dd71
correlation ID: e4ebdf9b-312f-4c03-affc-b5c4cbd39c75
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:310]: Unexpected error:
<*errors.joinError | 0x2874318ca1e0>:
resource group "rg-nightly-4-21-kv7pzz4ch49v": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cluster-nightly-4-21-7x886k" in resourcegroup="rg-nightly-4-21-kv7pzz4ch49v" to finish deleting: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/35ebc587-6a8d-48f1-a4a4-b752ee43dd71
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: InternalServerError
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/35ebc587-6a8d-48f1-a4a4-b752ee43dd71",
"name": "35ebc587-6a8d-48f1-a4a4-b752ee43dd71",
"status": "Failed",
"startTime": "2026-10-24T23:51:18.597955269Z",
"endTime": "2026-10-25T00:15:22.050322008Z",
"error": {
"code": "InternalServerError",
"message": "cluster deletion did not complete before the deadline; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters, 3 microsoft.redhatopenshift/hcpopenshiftclusters/applydesires; [resourceTeardown] tearing down pod network instances (1), pod networks (1), namespaces (2)"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/35ebc587-6a8d-48f1-a4a4-b752ee43dd71
correlation ID: e4ebdf9b-312f-4c03-affc-b5c4cbd39c75
client request ID: unavailable
occurredfail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:309]: Unexpected error:
<*errors.joinError | 0x26a811ac2138>:
resource group "rg-nightly-4-20-t74cxvx2rdnz": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cluster-nightly-4-20-cb8gb2" in resourcegroup="rg-nightly-4-20-t74cxvx2rdnz" to finish deleting: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/51202163-2b8d-4c4b-abd9-e60c454750ef
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: InternalServerError
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/51202163-2b8d-4c4b-abd9-e60c454750ef",
"name": "51202163-2b8d-4c4b-abd9-e60c454750ef",
"status": "Failed",
"startTime": "2026-10-17T23:43:19.794851177Z",
"endTime": "2026-10-18T00:07:19.354701033Z",
"error": {
"code": "InternalServerError",
"message": "cluster deletion did not complete before the deadline; [descendantResources] remaining resources: 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/serviceProviderClusters, 3 microsoft.redhatopenshift/hcpopenshiftclusters/applydesires; [resourceTeardown] tearing down pod network instances (1), pod networks (1), namespaces (2)"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/51202163-2b8d-4c4b-abd9-e60c454750ef
correlation ID: d4f725b2-2286-488f-91cf-637285920d3b
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:309]: Unexpected error:
<*errors.joinError | 0x26a811ac2138>:
resource group "rg-nightly-4-20-t74cxvx2rdnz": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cluster-nightly-4-20-cb8gb2" in resourcegroup="rg-nightly-4-20-t74cxvx2rdnz" to finish deleting: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/51202163-2b8d-4c4b-abd9-e60c454750ef
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: InternalServerError
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/51202163-2b8d-4c4b-abd9-e60c454750ef",
"name": "51202163-2b8d-4c4b-abd9-e60c454750ef",
"status": "Failed",
"startTime": "2026-10-17T23:43:19.794851177Z",
"endTime": "2026-10-18T00:07:19.354701033Z",
"error": {
"code": "InternalServerError",
"message": "cluster deletion did not complete before the deadline; [descendantResources] remaining resources: 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/serviceProviderClusters, 3 microsoft.redhatopenshift/hcpopenshiftclusters/applydesires; [resourceTeardown] tearing down pod network instances (1), pod networks (1), namespaces (2)"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/51202163-2b8d-4c4b-abd9-e60c454750ef
correlation ID: d4f725b2-2286-488f-91cf-637285920d3b
client request ID: unavailable
occurredContributing tests (2)
Affected runs (2)
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
cluster operators not available: console (Degraded=True, Progressing=False, Available=False, Upgradeable=True, EvaluationConditionsDetected=Unknown), ingress (Available=False, Pro...full failure pattern: cluster operators not available: console (Degraded=True, Progressing=False, Available=False, Upgradeable=True, EvaluationConditionsDetected=Unknown), ingress (Available=False, Progressing=True, Degraded=True, Upgradeable=True Signal: IndeterminateSignal: Indeterminate — no prior history; active 2 day(s) — New failure pattern — no prior history | e2e | 2 | 2.08%2 of 96 job runs affected | IndeterminateSignal: Indeterminate — no prior history; active 2 day(s) — New failure pattern — no prior history | Show trend detailsOct 12: 0 · Oct 13: 0 · Oct 14: 0 · Oct 15: 0 · Oct 16: 0 · Oct 17: 0 · Oct 18: 0 · Oct 19: 0 · Oct 20: 0 · Oct 21: 0 · Oct 22: 0 · Oct 23: 1 · Oct 24: 0 · Oct 25: 1 | none | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (2)fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:255]: failed to verify cluster operators are available after external auth config creation
Unexpected error:
<*errors.joinError | 0x471790ed8f0>:
context deadline exceeded
cluster operators not available: console (Degraded=True, Progressing=False, Available=False, Upgradeable=True, EvaluationConditionsDetected=Unknown), ingress (Available=False, Progressing=True, Degraded=True, Upgradeable=True, EvaluationConditionsDetected=False)
{
errs: [
<context.deadlineExceededError>{},
<*errors.errorString | 0x4717a706200>{
s: "cluster operators not available: console (Degraded=True, Progressing=False, Available=False, Upgradeable=True, EvaluationConditionsDetected=Unknown), ingress (Available=False, Progressing=True, Degraded=True, Upgradeable=True, EvaluationConditionsDetected=False)",
},
],
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:255]: failed to verify cluster operators are available after external auth config creation
Unexpected error:
<*errors.joinError | 0x471790ed8f0>:
context deadline exceeded
cluster operators not available: console (Degraded=True, Progressing=False, Available=False, Upgradeable=True, EvaluationConditionsDetected=Unknown), ingress (Available=False, Progressing=True, Degraded=True, Upgradeable=True, EvaluationConditionsDetected=False)
{
errs: [
<context.deadlineExceededError>{},
<*errors.errorString | 0x4717a706200>{
s: "cluster operators not available: console (Degraded=True, Progressing=False, Available=False, Upgradeable=True, EvaluationConditionsDetected=Unknown), ingress (Available=False, Progressing=True, Degraded=True, Upgradeable=True, EvaluationConditionsDetected=False)",
},
],
}
occurredfail [github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:255]: failed to verify cluster operators are available after external auth config creation
Unexpected error:
<*errors.joinError | 0x397e13438ac8>:
context deadline exceeded
cluster operators not available: console (Degraded=True, Progressing=False, Available=False, Upgradeable=True, EvaluationConditionsDetected=Unknown), ingress (Available=False, Progressing=True, Degraded=True, Upgradeable=True, EvaluationConditionsDetected=False)
{
errs: [
<context.deadlineExceededError>{},
<*errors.errorString | 0x397e134eabf0>{
s: "cluster operators not available: console (Degraded=True, Progressing=False, Available=False, Upgradeable=True, EvaluationConditionsDetected=Unknown), ingress (Available=False, Progressing=True, Degraded=True, Upgradeable=True, EvaluationConditionsDetected=False)",
},
],
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:255]: failed to verify cluster operators are available after external auth config creation
Unexpected error:
<*errors.joinError | 0x397e13438ac8>:
context deadline exceeded
cluster operators not available: console (Degraded=True, Progressing=False, Available=False, Upgradeable=True, EvaluationConditionsDetected=Unknown), ingress (Available=False, Progressing=True, Degraded=True, Upgradeable=True, EvaluationConditionsDetected=False)
{
errs: [
<context.deadlineExceededError>{},
<*errors.errorString | 0x397e134eabf0>{
s: "cluster operators not available: console (Degraded=True, Progressing=False, Available=False, Upgradeable=True, EvaluationConditionsDetected=Unknown), ingress (Available=False, Progressing=True, Degraded=True, Upgradeable=True, EvaluationConditionsDetected=False)",
},
],
}
occurredContributing tests (1)
Affected runs (2)
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
public ingress (console) should be reachable from outside the VNet for private KAS cluster, but got error: Get "<url>": dial tcp: lookup <host> on <dns-server>: no such hostfull failure pattern: public ingress (console) should be reachable from outside the VNet for private KAS cluster, but got error: Get "<url>": dial tcp: lookup <host> on <dns-server>: no such host Signal: IndeterminateSignal: Indeterminate — no prior history; active 2 day(s) — New failure pattern — no prior history | e2e | 2 | 2.08%2 of 96 job runs affected | IndeterminateSignal: Indeterminate — no prior history; active 2 day(s) — New failure pattern — no prior history | Show trend detailsOct 12: 0 · Oct 13: 0 · Oct 14: 0 · Oct 15: 0 · Oct 16: 0 · Oct 17: 0 · Oct 18: 0 · Oct 19: 0 · Oct 20: 0 · Oct 21: 0 · Oct 22: 0 · Oct 23: 1 · Oct 24: 0 · Oct 25: 1 | none | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (2)fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kas.go:225]: Timed out after 600.000s.
public ingress should be reachable from outside the VNet despite private KAS
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kas.go:223 with:
public ingress (console) should be reachable from outside the VNet for private KAS cluster, but got error: Get "https://console-openshift-console.apps.aro.private-kas.zjlv.eastus2.aroapp-hcp.io": dial tcp: lookup console-openshift-console.apps.aro.private-kas.zjlv.eastus2.aroapp-hcp.io on 172.30.0.10:53: no such host
Unexpected error:
<*url.Error | 0x1bbed2a5bb90>:
Get "https://console-openshift-console.apps.aro.private-kas.zjlv.eastus2.aroapp-hcp.io": dial tcp: lookup console-openshift-console.apps.aro.private-kas.zjlv.eastus2.aroapp-hcp.io on 172.30.0.10:53: no such host
{
Op: "Get",
URL: "https://console-openshift-console.apps.aro.private-kas.zjlv.eastus2.aroapp-hcp.io",
Err: <*net.OpError | 0x1bbed27f1680>{
Op: "dial",
Net: "tcp",
Source: nil,
Addr: nil,
Err: <*net.DNSError | 0x1bbed27f1450>{
UnwrapErr: nil,
Err: "no such host",
Name: "console-openshift-console.apps.aro.private-kas.zjlv.eastus2.aroapp-hcp.io",
Server: "172.30.0.10:53",
IsTimeout: false,
IsTemporary: false,
IsNotFound: true,
},
},
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kas.go:225]: Timed out after 600.000s.
public ingress should be reachable from outside the VNet despite private KAS
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kas.go:223 with:
public ingress (console) should be reachable from outside the VNet for private KAS cluster, but got error: Get "https://console-openshift-console.apps.aro.private-kas.zjlv.eastus2.aroapp-hcp.io": dial tcp: lookup console-openshift-console.apps.aro.private-kas.zjlv.eastus2.aroapp-hcp.io on 172.30.0.10:53: no such host
Unexpected error:
<*url.Error | 0x1bbed2a5bb90>:
Get "https://console-openshift-console.apps.aro.private-kas.zjlv.eastus2.aroapp-hcp.io": dial tcp: lookup console-openshift-console.apps.aro.private-kas.zjlv.eastus2.aroapp-hcp.io on 172.30.0.10:53: no such host
{
Op: "Get",
URL: "https://console-openshift-console.apps.aro.private-kas.zjlv.eastus2.aroapp-hcp.io",
Err: <*net.OpError | 0x1bbed27f1680>{
Op: "dial",
Net: "tcp",
Source: nil,
Addr: nil,
Err: <*net.DNSError | 0x1bbed27f1450>{
UnwrapErr: nil,
Err: "no such host",
Name: "console-openshift-console.apps.aro.private-kas.zjlv.eastus2.aroapp-hcp.io",
Server: "172.30.0.10:53",
IsTimeout: false,
IsTemporary: false,
IsNotFound: true,
},
},
}
occurredfail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kas.go:225]: Timed out after 600.000s.
public ingress should be reachable from outside the VNet despite private KAS
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kas.go:223 with:
public ingress (console) should be reachable from outside the VNet for private KAS cluster, but got error: Get "https://console-openshift-console.apps.aro.private-kas.w1c2.eastus2.aroapp-hcp.io": dial tcp: lookup console-openshift-console.apps.aro.private-kas.w1c2.eastus2.aroapp-hcp.io on 172.30.0.10:53: no such host
Unexpected error:
<*url.Error | 0x20ec8dc8cd20>:
Get "https://console-openshift-console.apps.aro.private-kas.w1c2.eastus2.aroapp-hcp.io": dial tcp: lookup console-openshift-console.apps.aro.private-kas.w1c2.eastus2.aroapp-hcp.io on 172.30.0.10:53: no such host
{
Op: "Get",
URL: "https://console-openshift-console.apps.aro.private-kas.w1c2.eastus2.aroapp-hcp.io",
Err: <*net.OpError | 0x20ec8bc10690>{
Op: "dial",
Net: "tcp",
Source: nil,
Addr: nil,
Err: <*net.DNSError | 0x20ec8bc103c0>{
UnwrapErr: nil,
Err: "no such host",
Name: "console-openshift-console.apps.aro.private-kas.w1c2.eastus2.aroapp-hcp.io",
Server: "172.30.0.10:53",
IsTimeout: false,
IsTemporary: false,
IsNotFound: true,
},
},
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kas.go:225]: Timed out after 600.000s.
public ingress should be reachable from outside the VNet despite private KAS
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kas.go:223 with:
public ingress (console) should be reachable from outside the VNet for private KAS cluster, but got error: Get "https://console-openshift-console.apps.aro.private-kas.w1c2.eastus2.aroapp-hcp.io": dial tcp: lookup console-openshift-console.apps.aro.private-kas.w1c2.eastus2.aroapp-hcp.io on 172.30.0.10:53: no such host
Unexpected error:
<*url.Error | 0x20ec8dc8cd20>:
Get "https://console-openshift-console.apps.aro.private-kas.w1c2.eastus2.aroapp-hcp.io": dial tcp: lookup console-openshift-console.apps.aro.private-kas.w1c2.eastus2.aroapp-hcp.io on 172.30.0.10:53: no such host
{
Op: "Get",
URL: "https://console-openshift-console.apps.aro.private-kas.w1c2.eastus2.aroapp-hcp.io",
Err: <*net.OpError | 0x20ec8bc10690>{
Op: "dial",
Net: "tcp",
Source: nil,
Addr: nil,
Err: <*net.DNSError | 0x20ec8bc103c0>{
UnwrapErr: nil,
Err: "no such host",
Name: "console-openshift-console.apps.aro.private-kas.w1c2.eastus2.aroapp-hcp.io",
Server: "172.30.0.10:53",
IsTimeout: false,
IsTemporary: false,
IsNotFound: true,
},
},
}
occurredContributing tests (1)
Affected runs (2)
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
timeout during CreateHCPClusterAndWait; context deadline exceededfull failure pattern: timeout during CreateHCPClusterAndWait; context deadline exceeded Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 2 day(s) | e2e | 2 | 2.08%2 of 96 job runs affected | IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 2 day(s) | Show trend detailsOct 12: 0 · Oct 13: 0 · Oct 14: 0 · Oct 15: 0 · Oct 16: 0 · Oct 17: 0 · Oct 18: 0 · Oct 19: 0 · Oct 20: 1 · Oct 21: 0 · Oct 22: 1 · Oct 23: 0 · Oct 24: 0 · Oct 25: 0 | DEV | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (2)fail [github.com/Azure/ARO-HCP/test/e2e/cluster_version_backlevel.go:152]: failed to create HCP cluster version 4.20
Unexpected error:
<*fmt.wrapErrors | 0x146ba53e5a0>:
failed waiting for cluster="cluster-ver-4-20" in resourcegroup="rg-cluster-back-version-vgggt2t7qcsd" to finish creating, caused by: timeout '30.000000' minutes exceeded during CreateHCPClusterAndWait for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-vgggt2t7qcsd, error: context deadline exceeded
{
msg: "failed waiting for cluster=\"cluster-ver-4-20\" in resourcegroup=\"rg-cluster-back-version-vgggt2t7qcsd\" to finish creating, caused by: timeout '30.000000' minutes exceeded during CreateHCPClusterAndWait for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-vgggt2t7qcsd, error: context deadline exceeded",
errs: [
<*errors.errorString | 0x146ba6ff790>{
s: "timeout '30.000000' minutes exceeded during CreateHCPClusterAndWait for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-vgggt2t7qcsd",
},
<context.deadlineExceededError>{},
],
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_version_backlevel.go:152]: failed to create HCP cluster version 4.20
Unexpected error:
<*fmt.wrapErrors | 0x146ba53e5a0>:
failed waiting for cluster="cluster-ver-4-20" in resourcegroup="rg-cluster-back-version-vgggt2t7qcsd" to finish creating, caused by: timeout '30.000000' minutes exceeded during CreateHCPClusterAndWait for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-vgggt2t7qcsd, error: context deadline exceeded
{
msg: "failed waiting for cluster=\"cluster-ver-4-20\" in resourcegroup=\"rg-cluster-back-version-vgggt2t7qcsd\" to finish creating, caused by: timeout '30.000000' minutes exceeded during CreateHCPClusterAndWait for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-vgggt2t7qcsd, error: context deadline exceeded",
errs: [
<*errors.errorString | 0x146ba6ff790>{
s: "timeout '30.000000' minutes exceeded during CreateHCPClusterAndWait for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-vgggt2t7qcsd",
},
<context.deadlineExceededError>{},
],
}
occurredfail [github.com/Azure/ARO-HCP/test/e2e/cluster_version_backlevel.go:152]: failed to create HCP cluster version 4.20
Unexpected error:
<*fmt.wrapErrors | 0x3a0f21760840>:
failed waiting for cluster="cluster-ver-4-20" in resourcegroup="rg-cluster-back-version-m69f8px6srdw" to finish creating, caused by: timeout '30.000000' minutes exceeded during CreateHCPClusterAndWait for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-m69f8px6srdw, error: context deadline exceeded
{
msg: "failed waiting for cluster=\"cluster-ver-4-20\" in resourcegroup=\"rg-cluster-back-version-m69f8px6srdw\" to finish creating, caused by: timeout '30.000000' minutes exceeded during CreateHCPClusterAndWait for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-m69f8px6srdw, error: context deadline exceeded",
errs: [
<*errors.errorString | 0x3a0f2173b0c0>{
s: "timeout '30.000000' minutes exceeded during CreateHCPClusterAndWait for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-m69f8px6srdw",
},
<context.deadlineExceededError>{},
],
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_version_backlevel.go:152]: failed to create HCP cluster version 4.20
Unexpected error:
<*fmt.wrapErrors | 0x3a0f21760840>:
failed waiting for cluster="cluster-ver-4-20" in resourcegroup="rg-cluster-back-version-m69f8px6srdw" to finish creating, caused by: timeout '30.000000' minutes exceeded during CreateHCPClusterAndWait for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-m69f8px6srdw, error: context deadline exceeded
{
msg: "failed waiting for cluster=\"cluster-ver-4-20\" in resourcegroup=\"rg-cluster-back-version-m69f8px6srdw\" to finish creating, caused by: timeout '30.000000' minutes exceeded during CreateHCPClusterAndWait for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-m69f8px6srdw, error: context deadline exceeded",
errs: [
<*errors.errorString | 0x3a0f2173b0c0>{
s: "timeout '30.000000' minutes exceeded during CreateHCPClusterAndWait for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-m69f8px6srdw",
},
<context.deadlineExceededError>{},
],
}
occurredContributing tests (1)
Affected runs (2)
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
<Azure error internals omitted>full failure pattern: <Azure error internals omitted> Signal: IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | e2e | 1 | 1.04%1 of 96 job runs affected | IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | Show trend detailsOct 12: 0 · Oct 13: 0 · Oct 14: 0 · Oct 15: 0 · Oct 16: 0 · Oct 17: 0 · Oct 18: 0 · Oct 19: 0 · Oct 20: 0 · Oct 21: 0 · Oct 22: 7 · Oct 23: 0 · Oct 24: 0 · Oct 25: 0 | none | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (3)fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:310]: Unexpected error:
<*errors.joinError | 0x2a26cf54c078>:
resource group "np-autoscaling-ncn4v65w96fx": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="np-autoscale-cluster" in resourcegroup="np-autoscaling-ncn4v65w96fx" to finish deleting: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/d43b55c4-7331-4381-b4fd-71f64a776433
--------------------------------------------------------------------------------
RESPONSE 502: 502 Bad Gateway
ERROR CODE UNAVAILABLE
--------------------------------------------------------------------------------
Response contained no body
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/d43b55c4-7331-4381-b4fd-71f64a776433
correlation ID: 1d901265-7895-4b99-bf69-cdc848b0d0b5
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:310]: Unexpected error:
<*errors.joinError | 0x2a26cf54c078>:
resource group "np-autoscaling-ncn4v65w96fx": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="np-autoscale-cluster" in resourcegroup="np-autoscaling-ncn4v65w96fx" to finish deleting: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/d43b55c4-7331-4381-b4fd-71f64a776433
--------------------------------------------------------------------------------
RESPONSE 502: 502 Bad Gateway
ERROR CODE UNAVAILABLE
--------------------------------------------------------------------------------
Response contained no body
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/d43b55c4-7331-4381-b4fd-71f64a776433
correlation ID: 1d901265-7895-4b99-bf69-cdc848b0d0b5
client request ID: unavailable
occurredfail [github.com/Azure/ARO-HCP/test/e2e/shoebox_logs.go:272]: failed to create HCP cluster shoebox-hcp-cluster
Unexpected error:
<*fmt.wrapError | 0x1745c0262280>:
failed to create HCP cluster shoebox-hcp-cluster: failed waiting for cluster="shoebox-hcp-cluster" in resourcegroup="shoebox-cluster-ddrtzn7r9n6f" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/aa390de3-b57c-4004-ae00-8d256a07e3fb
--------------------------------------------------------------------------------
RESPONSE 502: 502 Bad Gateway
ERROR CODE UNAVAILABLE
--------------------------------------------------------------------------------
Response contained no body
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/aa390de3-b57c-4004-ae00-8d256a07e3fb
correlation ID: 23755ba0-9dc6-42ab-bdd9-3208cee1312c
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/shoebox_logs.go:272]: failed to create HCP cluster shoebox-hcp-cluster
Unexpected error:
<*fmt.wrapError | 0x1745c0262280>:
failed to create HCP cluster shoebox-hcp-cluster: failed waiting for cluster="shoebox-hcp-cluster" in resourcegroup="shoebox-cluster-ddrtzn7r9n6f" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/aa390de3-b57c-4004-ae00-8d256a07e3fb
--------------------------------------------------------------------------------
RESPONSE 502: 502 Bad Gateway
ERROR CODE UNAVAILABLE
--------------------------------------------------------------------------------
Response contained no body
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/aa390de3-b57c-4004-ae00-8d256a07e3fb
correlation ID: 23755ba0-9dc6-42ab-bdd9-3208cee1312c
client request ID: unavailable
occurredfail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:175]: failed to get admin REST config for cluster np-version-upgrade-cluster-6klfkt
Unexpected error:
<*fmt.wrapError | 0x31e04d7a70c0>:
failed waiting for hcpCluster="np-version-upgrade-cluster-6klfkt" in resourcegroup="rg-np-version-upgrade-6klfkt-x72h4v22zqrn" to finish getting creds: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/0b189d0f-2181-4aed-9e7f-51e3cb6c01ee
--------------------------------------------------------------------------------
RESPONSE 502: 502 Bad Gateway
ERROR CODE UNAVAILABLE
--------------------------------------------------------------------------------
Response contained no body
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/0b189d0f-2181-4aed-9e7f-51e3cb6c01ee
correlation ID: acc4317f-332d-41ad-92b6-3b3da8362c30
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:175]: failed to get admin REST config for cluster np-version-upgrade-cluster-6klfkt
Unexpected error:
<*fmt.wrapError | 0x31e04d7a70c0>:
failed waiting for hcpCluster="np-version-upgrade-cluster-6klfkt" in resourcegroup="rg-np-version-upgrade-6klfkt-x72h4v22zqrn" to finish getting creds: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/0b189d0f-2181-4aed-9e7f-51e3cb6c01ee
--------------------------------------------------------------------------------
RESPONSE 502: 502 Bad Gateway
ERROR CODE UNAVAILABLE
--------------------------------------------------------------------------------
Response contained no body
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/0b189d0f-2181-4aed-9e7f-51e3cb6c01ee
correlation ID: acc4317f-332d-41ad-92b6-3b3da8362c30
client request ID: unavailable
occurredContributing tests (7)
Affected runs (1)
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
failed waiting for nodepool="<nodepool>" for cluster "<cluster>" in resourcegroup="<resource-group>" to finish creating: GET <url>full failure pattern: failed waiting for nodepool="<nodepool>" for cluster "<cluster>" in resourcegroup="<resource-group>" to finish creating: GET <url> Signal: IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | e2e | 1 | 1.04%1 of 96 job runs affected | IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | Show trend detailsOct 12: 0 · Oct 13: 0 · Oct 14: 0 · Oct 15: 0 · Oct 16: 0 · Oct 17: 0 · Oct 18: 0 · Oct 19: 0 · Oct 20: 0 · Oct 21: 0 · Oct 22: 7 · Oct 23: 0 · Oct 24: 0 · Oct 25: 0 | none | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (3)fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:461]: failed to create nodepool nps-420-422 at version 4.20.43
Unexpected error:
<*fmt.wrapError | 0x378ab2d1b4e0>:
failed to create NodePool nps-420-422: failed waiting for nodepool="nps-420-422" for cluster "np-skip-minor-p9x242" in resourcegroup="rg-np-skip-minor-p9x242-tqcdq45mhp7v" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/773f88ac-95e8-4da7-bc09-57a818deb175
--------------------------------------------------------------------------------
RESPONSE 502: 502 Bad Gateway
ERROR CODE UNAVAILABLE
--------------------------------------------------------------------------------
Response contained no body
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/773f88ac-95e8-4da7-bc09-57a818deb175
correlation ID: 8aa44ab4-58c2-423f-9a40-2f66225c45a7
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:461]: failed to create nodepool nps-420-422 at version 4.20.43
Unexpected error:
<*fmt.wrapError | 0x378ab2d1b4e0>:
failed to create NodePool nps-420-422: failed waiting for nodepool="nps-420-422" for cluster "np-skip-minor-p9x242" in resourcegroup="rg-np-skip-minor-p9x242-tqcdq45mhp7v" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/773f88ac-95e8-4da7-bc09-57a818deb175
--------------------------------------------------------------------------------
RESPONSE 502: 502 Bad Gateway
ERROR CODE UNAVAILABLE
--------------------------------------------------------------------------------
Response contained no body
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/773f88ac-95e8-4da7-bc09-57a818deb175
correlation ID: 8aa44ab4-58c2-423f-9a40-2f66225c45a7
client request ID: unavailable
occurredfail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:321]: failed to create nodepool npnoedge-4-20 at version 4.20.42
Unexpected error:
<*fmt.wrapError | 0x8e881283980>:
failed to create NodePool npnoedge-4-20: failed waiting for nodepool="npnoedge-4-20" for cluster "np-noedge-8tlm65" in resourcegroup="rg-np-noedge-8tlm65-6rtrf6fxs8pp" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/657d92b1-0420-465c-a20f-051929954d68
--------------------------------------------------------------------------------
RESPONSE 502: 502 Bad Gateway
ERROR CODE UNAVAILABLE
--------------------------------------------------------------------------------
Response contained no body
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/657d92b1-0420-465c-a20f-051929954d68
correlation ID: da25ea79-7ee4-4aa5-a3cd-24d74907d5d3
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:321]: failed to create nodepool npnoedge-4-20 at version 4.20.42
Unexpected error:
<*fmt.wrapError | 0x8e881283980>:
failed to create NodePool npnoedge-4-20: failed waiting for nodepool="npnoedge-4-20" for cluster "np-noedge-8tlm65" in resourcegroup="rg-np-noedge-8tlm65-6rtrf6fxs8pp" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/657d92b1-0420-465c-a20f-051929954d68
--------------------------------------------------------------------------------
RESPONSE 502: 502 Bad Gateway
ERROR CODE UNAVAILABLE
--------------------------------------------------------------------------------
Response contained no body
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/657d92b1-0420-465c-a20f-051929954d68
correlation ID: da25ea79-7ee4-4aa5-a3cd-24d74907d5d3
client request ID: unavailable
occurredfail [github.com/Azure/ARO-HCP/test/e2e/oidc_issuer_workload_identity.go:270]: failed to create node pool oidc-wi-np
Unexpected error:
<*fmt.wrapError | 0x11adfdef64a0>:
failed to create NodePool oidc-wi-np: failed waiting for nodepool="oidc-wi-np" for cluster "oidc-wi-cluster" in resourcegroup="oidc-wi-nm2jdjsx4m9t" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/7ab31f32-e5a9-4966-9e78-a32d1786c989
--------------------------------------------------------------------------------
RESPONSE 502: 502 Bad Gateway
ERROR CODE UNAVAILABLE
--------------------------------------------------------------------------------
Response contained no body
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/7ab31f32-e5a9-4966-9e78-a32d1786c989
correlation ID: f811a401-7f9a-42bd-861f-59e24ffc103f
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/oidc_issuer_workload_identity.go:270]: failed to create node pool oidc-wi-np
Unexpected error:
<*fmt.wrapError | 0x11adfdef64a0>:
failed to create NodePool oidc-wi-np: failed waiting for nodepool="oidc-wi-np" for cluster "oidc-wi-cluster" in resourcegroup="oidc-wi-nm2jdjsx4m9t" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/7ab31f32-e5a9-4966-9e78-a32d1786c989
--------------------------------------------------------------------------------
RESPONSE 502: 502 Bad Gateway
ERROR CODE UNAVAILABLE
--------------------------------------------------------------------------------
Response contained no body
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/7ab31f32-e5a9-4966-9e78-a32d1786c989
correlation ID: f811a401-7f9a-42bd-861f-59e24ffc103f
client request ID: unavailable
occurredContributing tests (7)
Affected runs (1)
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedC...full failure pattern: ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable; hosted cluster degraded: UnavailableReplicas: [catalog-operator, hosted-cluster-config-operator, olm-operator, openshift-apiserver, packageserver]; provider Microsoft.RedHatOpenShift Signal: IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | e2e | 1 | 1.04%1 of 96 job runs affected | IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | Show trend detailsOct 12: 0 · Oct 13: 0 · Oct 14: 0 · Oct 15: 0 · Oct 16: 0 · Oct 17: 0 · Oct 18: 0 · Oct 19: 0 · Oct 20: 0 · Oct 21: 0 · Oct 22: 3 · Oct 23: 0 · Oct 24: 0 · Oct 25: 0 | DEV, STG | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (3)fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:125]: failed to create HCP cluster "cluster-zstream-4-23-tvqgcd" with version 4.23.0-ec.0 on candidate channel
Unexpected error:
<*fmt.wrapError | 0x3a1b74a0c6e0>:
failed to create HCP cluster cluster-zstream-4-23-tvqgcd: failed waiting for cluster="cluster-zstream-4-23-tvqgcd" in resourcegroup="rg-zstream-upgrade-4-23-z4qsqrdw854d" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/bdede344-c320-476c-93e8-c3141c6d343a
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/bdede344-c320-476c-93e8-c3141c6d343a",
"name": "bdede344-c320-476c-93e8-c3141c6d343a",
"status": "Failed",
"startTime": "2026-10-22T02:22:59.55049493Z",
"endTime": "2026-10-22T02:52:00.596740917Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: cluster-policy-controller, hosted-cluster-config-operator, cluster-node-tuning-operator, cluster-version-operator, openshift-controller-manager, cluster-network-operator, openshift-route-controller-manager, cluster-storage-operator, ingress-operator, csi-snapshot-controller-operator, redhat-operators-catalog, certified-operators-catalog, dns-operator, community-operators-catalog, olm-operator, cluster-image-registry-operator, catalog-operator, packageserver; hosted cluster degraded: UnavailableReplicas: [catalog-operator deployment has 1 unavailable replicas, hosted-cluster-config-operator deployment has 1 unavailable replicas, olm-operator deployment has 1 unavailable replicas, openshift-apiserver deployment has 1 unavailable replicas, packageserver deployment has 3 unavailable replicas]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/bdede344-c320-476c-93e8-c3141c6d343a
correlation ID: 946c8b3d-71c1-46d3-ad96-760f309b6eb8
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:125]: failed to create HCP cluster "cluster-zstream-4-23-tvqgcd" with version 4.23.0-ec.0 on candidate channel
Unexpected error:
<*fmt.wrapError | 0x3a1b74a0c6e0>:
failed to create HCP cluster cluster-zstream-4-23-tvqgcd: failed waiting for cluster="cluster-zstream-4-23-tvqgcd" in resourcegroup="rg-zstream-upgrade-4-23-z4qsqrdw854d" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/bdede344-c320-476c-93e8-c3141c6d343a
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/bdede344-c320-476c-93e8-c3141c6d343a",
"name": "bdede344-c320-476c-93e8-c3141c6d343a",
"status": "Failed",
"startTime": "2026-10-22T02:22:59.55049493Z",
"endTime": "2026-10-22T02:52:00.596740917Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: cluster-policy-controller, hosted-cluster-config-operator, cluster-node-tuning-operator, cluster-version-operator, openshift-controller-manager, cluster-network-operator, openshift-route-controller-manager, cluster-storage-operator, ingress-operator, csi-snapshot-controller-operator, redhat-operators-catalog, certified-operators-catalog, dns-operator, community-operators-catalog, olm-operator, cluster-image-registry-operator, catalog-operator, packageserver; hosted cluster degraded: UnavailableReplicas: [catalog-operator deployment has 1 unavailable replicas, hosted-cluster-config-operator deployment has 1 unavailable replicas, olm-operator deployment has 1 unavailable replicas, openshift-apiserver deployment has 1 unavailable replicas, packageserver deployment has 3 unavailable replicas]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/bdede344-c320-476c-93e8-c3141c6d343a
correlation ID: 946c8b3d-71c1-46d3-ad96-760f309b6eb8
client request ID: unavailable
occurredfail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_ingress.go:87]: failed to create HCP cluster "private-ingress" with private ingress
Unexpected error:
<*fmt.wrapError | 0x3e201b18a700>:
failed to create HCP cluster private-ingress: failed waiting for cluster="private-ingress" in resourcegroup="private-ingress-w4bh6t74lrm8" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/3ffa3393-14e6-485e-a06e-d34ad16e45af
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/3ffa3393-14e6-485e-a06e-d34ad16e45af",
"name": "3ffa3393-14e6-485e-a06e-d34ad16e45af",
"status": "Failed",
"startTime": "2026-10-22T01:33:57.248605332Z",
"endTime": "2026-10-22T02:03:06.666838162Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: cluster-version-operator, cluster-policy-controller, olm-operator, openshift-route-controller-manager, cluster-storage-operator, packageserver, redhat-marketplace-catalog, community-operators-catalog, cluster-network-operator, dns-operator, ingress-operator, catalog-operator, certified-operators-catalog, redhat-operators-catalog, hosted-cluster-config-operator, cluster-node-tuning-operator, csi-snapshot-controller-operator, openshift-controller-manager, cluster-image-registry-operator; hosted cluster degraded: UnavailableReplicas: [catalog-operator deployment has 1 unavailable replicas, hosted-cluster-config-operator deployment has 1 unavailable replicas, olm-operator deployment has 1 unavailable replicas, openshift-apiserver deployment has 1 unavailable replicas, packageserver deployment has 3 unavailable replicas]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/3ffa3393-14e6-485e-a06e-d34ad16e45af
correlation ID: d4851c32-d0b7-4ef1-9333-ecb958420c6c
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_ingress.go:87]: failed to create HCP cluster "private-ingress" with private ingress
Unexpected error:
<*fmt.wrapError | 0x3e201b18a700>:
failed to create HCP cluster private-ingress: failed waiting for cluster="private-ingress" in resourcegroup="private-ingress-w4bh6t74lrm8" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/3ffa3393-14e6-485e-a06e-d34ad16e45af
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/3ffa3393-14e6-485e-a06e-d34ad16e45af",
"name": "3ffa3393-14e6-485e-a06e-d34ad16e45af",
"status": "Failed",
"startTime": "2026-10-22T01:33:57.248605332Z",
"endTime": "2026-10-22T02:03:06.666838162Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: cluster-version-operator, cluster-policy-controller, olm-operator, openshift-route-controller-manager, cluster-storage-operator, packageserver, redhat-marketplace-catalog, community-operators-catalog, cluster-network-operator, dns-operator, ingress-operator, catalog-operator, certified-operators-catalog, redhat-operators-catalog, hosted-cluster-config-operator, cluster-node-tuning-operator, csi-snapshot-controller-operator, openshift-controller-manager, cluster-image-registry-operator; hosted cluster degraded: UnavailableReplicas: [catalog-operator deployment has 1 unavailable replicas, hosted-cluster-config-operator deployment has 1 unavailable replicas, olm-operator deployment has 1 unavailable replicas, openshift-apiserver deployment has 1 unavailable replicas, packageserver deployment has 3 unavailable replicas]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/3ffa3393-14e6-485e-a06e-d34ad16e45af
correlation ID: d4851c32-d0b7-4ef1-9333-ecb958420c6c
client request ID: unavailable
occurredfail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:125]: failed to create HCP cluster "cluster-zstream-4-20-zj4mcv" with version 4.20.42 on candidate channel
Unexpected error:
<*fmt.wrapError | 0x336a41a7c8a0>:
failed to create HCP cluster cluster-zstream-4-20-zj4mcv: failed waiting for cluster="cluster-zstream-4-20-zj4mcv" in resourcegroup="rg-zstream-upgrade-4-20-2z67w2qcbxs6" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/03c686a3-8b51-4273-9c18-ce6884c5a033
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/03c686a3-8b51-4273-9c18-ce6884c5a033",
"name": "03c686a3-8b51-4273-9c18-ce6884c5a033",
"status": "Failed",
"startTime": "2026-10-22T02:22:44.492283349Z",
"endTime": "2026-10-22T02:51:50.526771089Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: certified-operators-catalog, cluster-storage-operator, cluster-image-registry-operator, catalog-operator, openshift-controller-manager, cluster-policy-controller, cluster-network-operator, dns-operator, redhat-operators-catalog, cluster-version-operator, hosted-cluster-config-operator, cluster-node-tuning-operator, csi-snapshot-controller-operator, redhat-marketplace-catalog, openshift-route-controller-manager, ingress-operator, olm-operator, packageserver, community-operators-catalog; hosted cluster degraded: UnavailableReplicas: [catalog-operator deployment has 1 unavailable replicas, hosted-cluster-config-operator deployment has 1 unavailable replicas, olm-operator deployment has 1 unavailable replicas, openshift-apiserver deployment has 1 unavailable replicas, packageserver deployment has 3 unavailable replicas]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/03c686a3-8b51-4273-9c18-ce6884c5a033
correlation ID: 5cab7c5c-8f8d-4763-8268-cddbff9d06f5
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:125]: failed to create HCP cluster "cluster-zstream-4-20-zj4mcv" with version 4.20.42 on candidate channel
Unexpected error:
<*fmt.wrapError | 0x336a41a7c8a0>:
failed to create HCP cluster cluster-zstream-4-20-zj4mcv: failed waiting for cluster="cluster-zstream-4-20-zj4mcv" in resourcegroup="rg-zstream-upgrade-4-20-2z67w2qcbxs6" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/03c686a3-8b51-4273-9c18-ce6884c5a033
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/03c686a3-8b51-4273-9c18-ce6884c5a033",
"name": "03c686a3-8b51-4273-9c18-ce6884c5a033",
"status": "Failed",
"startTime": "2026-10-22T02:22:44.492283349Z",
"endTime": "2026-10-22T02:51:50.526771089Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: certified-operators-catalog, cluster-storage-operator, cluster-image-registry-operator, catalog-operator, openshift-controller-manager, cluster-policy-controller, cluster-network-operator, dns-operator, redhat-operators-catalog, cluster-version-operator, hosted-cluster-config-operator, cluster-node-tuning-operator, csi-snapshot-controller-operator, redhat-marketplace-catalog, openshift-route-controller-manager, ingress-operator, olm-operator, packageserver, community-operators-catalog; hosted cluster degraded: UnavailableReplicas: [catalog-operator deployment has 1 unavailable replicas, hosted-cluster-config-operator deployment has 1 unavailable replicas, olm-operator deployment has 1 unavailable replicas, openshift-apiserver deployment has 1 unavailable replicas, packageserver deployment has 3 unavailable replicas]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/03c686a3-8b51-4273-9c18-ce6884c5a033
correlation ID: 5cab7c5c-8f8d-4763-8268-cddbff9d06f5
client request ID: unavailable
occurredContributing tests (3)
Affected runs (1)
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedC...full failure pattern: ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable; hosted cluster degraded: UnavailableReplicas: [cluster-policy-controller, kube-controller-manager, openshift-controller-manager, openshift-route-controller-manager, packageserver]; provider Microsoft.RedHatOpenShift Signal: IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | e2e | 1 | 1.04%1 of 96 job runs affected | IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | Show trend detailsOct 12: 0 · Oct 13: 0 · Oct 14: 0 · Oct 15: 0 · Oct 16: 0 · Oct 17: 0 · Oct 18: 0 · Oct 19: 0 · Oct 20: 2 · Oct 21: 0 · Oct 22: 0 · Oct 23: 0 · Oct 24: 0 · Oct 25: 0 | none | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (2)fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_labels_taints.go:92]: failed to create HCP cluster cluster-np-labels-taints
Unexpected error:
<*fmt.wrapError | 0x87fabdfe6a0>:
failed to create HCP cluster cluster-np-labels-taints: failed waiting for cluster="cluster-np-labels-taints" in resourcegroup="rg-np-labels-taints-nb9lb62595s7" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/879b385a-2ae7-4f99-a707-d8f19a66cb12
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/879b385a-2ae7-4f99-a707-d8f19a66cb12",
"name": "879b385a-2ae7-4f99-a707-d8f19a66cb12",
"status": "Failed",
"startTime": "2026-10-20T06:20:18.481707716Z",
"endTime": "2026-10-20T06:49:20.225008169Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: openshift-controller-manager, cluster-policy-controller, openshift-route-controller-manager; hosted cluster degraded: UnavailableReplicas: [cluster-policy-controller deployment has 1 unavailable replicas, kube-controller-manager deployment has 1 unavailable replicas, openshift-controller-manager deployment has 1 unavailable replicas, openshift-route-controller-manager deployment has 1 unavailable replicas, packageserver deployment has 1 unavailable replicas]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/879b385a-2ae7-4f99-a707-d8f19a66cb12
correlation ID: cf264c90-6764-45bb-a904-1799b612e538
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_labels_taints.go:92]: failed to create HCP cluster cluster-np-labels-taints
Unexpected error:
<*fmt.wrapError | 0x87fabdfe6a0>:
failed to create HCP cluster cluster-np-labels-taints: failed waiting for cluster="cluster-np-labels-taints" in resourcegroup="rg-np-labels-taints-nb9lb62595s7" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/879b385a-2ae7-4f99-a707-d8f19a66cb12
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/879b385a-2ae7-4f99-a707-d8f19a66cb12",
"name": "879b385a-2ae7-4f99-a707-d8f19a66cb12",
"status": "Failed",
"startTime": "2026-10-20T06:20:18.481707716Z",
"endTime": "2026-10-20T06:49:20.225008169Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: openshift-controller-manager, cluster-policy-controller, openshift-route-controller-manager; hosted cluster degraded: UnavailableReplicas: [cluster-policy-controller deployment has 1 unavailable replicas, kube-controller-manager deployment has 1 unavailable replicas, openshift-controller-manager deployment has 1 unavailable replicas, openshift-route-controller-manager deployment has 1 unavailable replicas, packageserver deployment has 1 unavailable replicas]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/879b385a-2ae7-4f99-a707-d8f19a66cb12
correlation ID: cf264c90-6764-45bb-a904-1799b612e538
client request ID: unavailable
occurredfail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:147]: failed to create HCP cluster
Unexpected error:
<*fmt.wrapError | 0x2dc51d83d2c0>:
failed to create HCP cluster ingress-cert-le: failed waiting for cluster="ingress-cert-le" in resourcegroup="ingress-cert-le-dsxq6jxzmbtw" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/30bbed47-9a91-40fe-b99b-caa97b286f39
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/30bbed47-9a91-40fe-b99b-caa97b286f39",
"name": "30bbed47-9a91-40fe-b99b-caa97b286f39",
"status": "Failed",
"startTime": "2026-10-20T04:46:24.928973066Z",
"endTime": "2026-10-20T05:15:25.520014214Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: openshift-route-controller-manager, openshift-controller-manager, cluster-policy-controller; hosted cluster degraded: UnavailableReplicas: [cluster-policy-controller deployment has 1 unavailable replicas, kube-controller-manager deployment has 1 unavailable replicas, openshift-controller-manager deployment has 1 unavailable replicas, openshift-route-controller-manager deployment has 1 unavailable replicas, packageserver deployment has 1 unavailable replicas]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/30bbed47-9a91-40fe-b99b-caa97b286f39
correlation ID: d0e5fc28-aced-4621-9518-831e81cdc82c
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:147]: failed to create HCP cluster
Unexpected error:
<*fmt.wrapError | 0x2dc51d83d2c0>:
failed to create HCP cluster ingress-cert-le: failed waiting for cluster="ingress-cert-le" in resourcegroup="ingress-cert-le-dsxq6jxzmbtw" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/30bbed47-9a91-40fe-b99b-caa97b286f39
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/30bbed47-9a91-40fe-b99b-caa97b286f39",
"name": "30bbed47-9a91-40fe-b99b-caa97b286f39",
"status": "Failed",
"startTime": "2026-10-20T04:46:24.928973066Z",
"endTime": "2026-10-20T05:15:25.520014214Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: openshift-route-controller-manager, openshift-controller-manager, cluster-policy-controller; hosted cluster degraded: UnavailableReplicas: [cluster-policy-controller deployment has 1 unavailable replicas, kube-controller-manager deployment has 1 unavailable replicas, openshift-controller-manager deployment has 1 unavailable replicas, openshift-route-controller-manager deployment has 1 unavailable replicas, packageserver deployment has 1 unavailable replicas]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/30bbed47-9a91-40fe-b99b-caa97b286f39
correlation ID: d0e5fc28-aced-4621-9518-831e81cdc82c
client request ID: unavailable
occurredContributing tests (2)
Affected runs (1)
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedC...full failure pattern: ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable; hosted cluster degraded: UnavailableReplicas: [cluster-policy-controller]; provider Microsoft.RedHatOpenShift Signal: IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | e2e | 1 | 1.04%1 of 96 job runs affected | IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | Show trend detailsOct 12: 0 · Oct 13: 0 · Oct 14: 0 · Oct 15: 0 · Oct 16: 0 · Oct 17: 0 · Oct 18: 0 · Oct 19: 0 · Oct 20: 2 · Oct 21: 0 · Oct 22: 0 · Oct 23: 0 · Oct 24: 0 · Oct 25: 0 | none | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (2)fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_ephemeral_osdisk.go:84]: failed to create HCP cluster ephemeral-disk
Unexpected error:
<*fmt.wrapError | 0x1f03b61ccac0>:
failed to create HCP cluster ephemeral-disk: failed waiting for cluster="ephemeral-disk" in resourcegroup="ephemeral-osdisk-xkdl5ldkwkxf" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/60150ecc-c0c8-4ddc-abff-d4417c838152
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/60150ecc-c0c8-4ddc-abff-d4417c838152",
"name": "60150ecc-c0c8-4ddc-abff-d4417c838152",
"status": "Failed",
"startTime": "2026-10-20T06:17:57.832413824Z",
"endTime": "2026-10-20T06:47:00.220229717Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: cluster-policy-controller; hosted cluster degraded: UnavailableReplicas: cluster-policy-controller deployment has 1 unavailable replicas"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/60150ecc-c0c8-4ddc-abff-d4417c838152
correlation ID: 9cce0d52-7c81-4f97-a269-e15c1e2fd419
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_ephemeral_osdisk.go:84]: failed to create HCP cluster ephemeral-disk
Unexpected error:
<*fmt.wrapError | 0x1f03b61ccac0>:
failed to create HCP cluster ephemeral-disk: failed waiting for cluster="ephemeral-disk" in resourcegroup="ephemeral-osdisk-xkdl5ldkwkxf" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/60150ecc-c0c8-4ddc-abff-d4417c838152
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/60150ecc-c0c8-4ddc-abff-d4417c838152",
"name": "60150ecc-c0c8-4ddc-abff-d4417c838152",
"status": "Failed",
"startTime": "2026-10-20T06:17:57.832413824Z",
"endTime": "2026-10-20T06:47:00.220229717Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: cluster-policy-controller; hosted cluster degraded: UnavailableReplicas: cluster-policy-controller deployment has 1 unavailable replicas"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/60150ecc-c0c8-4ddc-abff-d4417c838152
correlation ID: 9cce0d52-7c81-4f97-a269-e15c1e2fd419
client request ID: unavailable
occurredfail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:163]: failed to create HCP cluster for patch-tags test
Unexpected error:
<*fmt.wrapError | 0x3bda3528c680>:
failed to create HCP cluster patch-tags-cluster: failed waiting for cluster="patch-tags-cluster" in resourcegroup="patch-tags-fxsg6grsgkll" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/2c866c88-017a-4ca1-a34c-aab419b47bd0
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/2c866c88-017a-4ca1-a34c-aab419b47bd0",
"name": "2c866c88-017a-4ca1-a34c-aab419b47bd0",
"status": "Failed",
"startTime": "2026-10-20T05:23:14.734977136Z",
"endTime": "2026-10-20T05:52:21.558247967Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: cluster-policy-controller; hosted cluster degraded: UnavailableReplicas: cluster-policy-controller deployment has 1 unavailable replicas"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/2c866c88-017a-4ca1-a34c-aab419b47bd0
correlation ID: 2af21d6c-711a-4dc5-b3d3-22e3f2257e3e
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:163]: failed to create HCP cluster for patch-tags test
Unexpected error:
<*fmt.wrapError | 0x3bda3528c680>:
failed to create HCP cluster patch-tags-cluster: failed waiting for cluster="patch-tags-cluster" in resourcegroup="patch-tags-fxsg6grsgkll" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/2c866c88-017a-4ca1-a34c-aab419b47bd0
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/2c866c88-017a-4ca1-a34c-aab419b47bd0",
"name": "2c866c88-017a-4ca1-a34c-aab419b47bd0",
"status": "Failed",
"startTime": "2026-10-20T05:23:14.734977136Z",
"endTime": "2026-10-20T05:52:21.558247967Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: cluster-policy-controller; hosted cluster degraded: UnavailableReplicas: cluster-policy-controller deployment has 1 unavailable replicas"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/2c866c88-017a-4ca1-a34c-aab419b47bd0
correlation ID: 2af21d6c-711a-4dc5-b3d3-22e3f2257e3e
client request ID: unavailable
occurredContributing tests (2)
Affected runs (1)
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedC...full failure pattern: ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider, cluster-api, ignition-server]; provider Microsoft.RedHatOpenShift Signal: IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | e2e | 1 | 1.04%1 of 96 job runs affected | IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | Show trend detailsOct 12: 0 · Oct 13: 0 · Oct 14: 0 · Oct 15: 0 · Oct 16: 0 · Oct 17: 0 · Oct 18: 0 · Oct 19: 0 · Oct 20: 0 · Oct 21: 0 · Oct 22: 2 · Oct 23: 0 · Oct 24: 0 · Oct 25: 0 | none | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (2)fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:133]: failed to create HCP cluster "cp-ystream-upgrade-4-23-bbjx5c" at install version 4.22
Unexpected error:
<*fmt.wrapError | 0x2052152a2720>:
failed to create HCP cluster cp-ystream-upgrade-4-23-bbjx5c: failed waiting for cluster="cp-ystream-upgrade-4-23-bbjx5c" in resourcegroup="rg-cp-ystream-upgrade-4-23-56g2lp75qrkf" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/02ef6437-1d48-43c3-ad2d-b3b8f18f1b53
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/02ef6437-1d48-43c3-ad2d-b3b8f18f1b53",
"name": "02ef6437-1d48-43c3-ad2d-b3b8f18f1b53",
"status": "Failed",
"startTime": "2026-10-22T02:23:39.532402776Z",
"endTime": "2026-10-22T02:52:40.523543888Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider deployment has 1 unavailable replicas, cluster-api deployment has 1 unavailable replicas, ignition-server deployment has 1 unavailable replicas]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/02ef6437-1d48-43c3-ad2d-b3b8f18f1b53
correlation ID: 43512322-23b0-4759-b738-201cec5eb6f3
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:133]: failed to create HCP cluster "cp-ystream-upgrade-4-23-bbjx5c" at install version 4.22
Unexpected error:
<*fmt.wrapError | 0x2052152a2720>:
failed to create HCP cluster cp-ystream-upgrade-4-23-bbjx5c: failed waiting for cluster="cp-ystream-upgrade-4-23-bbjx5c" in resourcegroup="rg-cp-ystream-upgrade-4-23-56g2lp75qrkf" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/02ef6437-1d48-43c3-ad2d-b3b8f18f1b53
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/02ef6437-1d48-43c3-ad2d-b3b8f18f1b53",
"name": "02ef6437-1d48-43c3-ad2d-b3b8f18f1b53",
"status": "Failed",
"startTime": "2026-10-22T02:23:39.532402776Z",
"endTime": "2026-10-22T02:52:40.523543888Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider deployment has 1 unavailable replicas, cluster-api deployment has 1 unavailable replicas, ignition-server deployment has 1 unavailable replicas]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/02ef6437-1d48-43c3-ad2d-b3b8f18f1b53
correlation ID: 43512322-23b0-4759-b738-201cec5eb6f3
client request ID: unavailable
occurredfail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:124]: HCP cluster rg-candidate-5-0-gmftj87c69s6/cluster-candidate-5-0-25snrj should provision
Unexpected error:
<*fmt.wrapError | 0x1ed3b41bc940>:
failed to create HCP cluster cluster-candidate-5-0-25snrj: failed waiting for cluster="cluster-candidate-5-0-25snrj" in resourcegroup="rg-candidate-5-0-gmftj87c69s6" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/27425e0a-5b12-47d6-9369-3a3a90aa3e86
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/27425e0a-5b12-47d6-9369-3a3a90aa3e86",
"name": "27425e0a-5b12-47d6-9369-3a3a90aa3e86",
"status": "Failed",
"startTime": "2026-10-22T02:21:48.683878735Z",
"endTime": "2026-10-22T02:50:50.395326627Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider deployment has 1 unavailable replicas, cluster-api deployment has 2 unavailable replicas, ignition-server deployment has 1 unavailable replicas]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/27425e0a-5b12-47d6-9369-3a3a90aa3e86
correlation ID: 52162704-0f0b-4a93-a71e-0f363b9222b6
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:124]: HCP cluster rg-candidate-5-0-gmftj87c69s6/cluster-candidate-5-0-25snrj should provision
Unexpected error:
<*fmt.wrapError | 0x1ed3b41bc940>:
failed to create HCP cluster cluster-candidate-5-0-25snrj: failed waiting for cluster="cluster-candidate-5-0-25snrj" in resourcegroup="rg-candidate-5-0-gmftj87c69s6" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/27425e0a-5b12-47d6-9369-3a3a90aa3e86
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/27425e0a-5b12-47d6-9369-3a3a90aa3e86",
"name": "27425e0a-5b12-47d6-9369-3a3a90aa3e86",
"status": "Failed",
"startTime": "2026-10-22T02:21:48.683878735Z",
"endTime": "2026-10-22T02:50:50.395326627Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider deployment has 1 unavailable replicas, cluster-api deployment has 2 unavailable replicas, ignition-server deployment has 1 unavailable replicas]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/27425e0a-5b12-47d6-9369-3a3a90aa3e86
correlation ID: 52162704-0f0b-4a93-a71e-0f363b9222b6
client request ID: unavailable
occurredContributing tests (2)
Affected runs (1)
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedC...full failure pattern: ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider, ignition-server-proxy, router]; provider Microsoft.RedHatOpenShift Signal: IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | e2e | 1 | 1.04%1 of 96 job runs affected | IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | Show trend detailsOct 12: 0 · Oct 13: 0 · Oct 14: 0 · Oct 15: 0 · Oct 16: 0 · Oct 17: 0 · Oct 18: 0 · Oct 19: 0 · Oct 20: 0 · Oct 21: 0 · Oct 22: 2 · Oct 23: 0 · Oct 24: 0 · Oct 25: 0 | none | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (2)fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:124]: HCP cluster rg-candidate-4-21-76q9tvzg8rn9/cluster-candidate-4-21-5lg29k should provision
Unexpected error:
<*fmt.wrapError | 0x143589e84400>:
failed to create HCP cluster cluster-candidate-4-21-5lg29k: failed waiting for cluster="cluster-candidate-4-21-5lg29k" in resourcegroup="rg-candidate-4-21-76q9tvzg8rn9" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/edb6a2fa-979a-4af5-a995-cd3897780e01
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/edb6a2fa-979a-4af5-a995-cd3897780e01",
"name": "edb6a2fa-979a-4af5-a995-cd3897780e01",
"status": "Failed",
"startTime": "2026-10-22T02:21:35.877670552Z",
"endTime": "2026-10-22T02:50:40.514192951Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider deployment has 1 unavailable replicas, ignition-server-proxy deployment has 1 unavailable replicas, router deployment has 1 unavailable replicas]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/edb6a2fa-979a-4af5-a995-cd3897780e01
correlation ID: 85af8528-9942-40bd-9e4c-05e55063f9c4
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:124]: HCP cluster rg-candidate-4-21-76q9tvzg8rn9/cluster-candidate-4-21-5lg29k should provision
Unexpected error:
<*fmt.wrapError | 0x143589e84400>:
failed to create HCP cluster cluster-candidate-4-21-5lg29k: failed waiting for cluster="cluster-candidate-4-21-5lg29k" in resourcegroup="rg-candidate-4-21-76q9tvzg8rn9" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/edb6a2fa-979a-4af5-a995-cd3897780e01
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/edb6a2fa-979a-4af5-a995-cd3897780e01",
"name": "edb6a2fa-979a-4af5-a995-cd3897780e01",
"status": "Failed",
"startTime": "2026-10-22T02:21:35.877670552Z",
"endTime": "2026-10-22T02:50:40.514192951Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider deployment has 1 unavailable replicas, ignition-server-proxy deployment has 1 unavailable replicas, router deployment has 1 unavailable replicas]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/edb6a2fa-979a-4af5-a995-cd3897780e01
correlation ID: 85af8528-9942-40bd-9e4c-05e55063f9c4
client request ID: unavailable
occurredfail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:124]: HCP cluster rg-candidate-4-22-gp6dgjrbtz82/cluster-candidate-4-22-qlh9hc should provision
Unexpected error:
<*fmt.wrapError | 0x1de55d1398a0>:
failed to create HCP cluster cluster-candidate-4-22-qlh9hc: failed waiting for cluster="cluster-candidate-4-22-qlh9hc" in resourcegroup="rg-candidate-4-22-gp6dgjrbtz82" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/8cc15b2a-8360-41a2-8a2a-f0798be9df0d
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/8cc15b2a-8360-41a2-8a2a-f0798be9df0d",
"name": "8cc15b2a-8360-41a2-8a2a-f0798be9df0d",
"status": "Failed",
"startTime": "2026-10-22T02:21:54.045486495Z",
"endTime": "2026-10-22T02:51:00.49750165Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider deployment has 1 unavailable replicas, ignition-server-proxy deployment has 1 unavailable replicas, router deployment has 1 unavailable replicas]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/8cc15b2a-8360-41a2-8a2a-f0798be9df0d
correlation ID: 1eeaec8e-068a-41c6-8cee-4c44e2977d7a
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:124]: HCP cluster rg-candidate-4-22-gp6dgjrbtz82/cluster-candidate-4-22-qlh9hc should provision
Unexpected error:
<*fmt.wrapError | 0x1de55d1398a0>:
failed to create HCP cluster cluster-candidate-4-22-qlh9hc: failed waiting for cluster="cluster-candidate-4-22-qlh9hc" in resourcegroup="rg-candidate-4-22-gp6dgjrbtz82" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/8cc15b2a-8360-41a2-8a2a-f0798be9df0d
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/8cc15b2a-8360-41a2-8a2a-f0798be9df0d",
"name": "8cc15b2a-8360-41a2-8a2a-f0798be9df0d",
"status": "Failed",
"startTime": "2026-10-22T02:21:54.045486495Z",
"endTime": "2026-10-22T02:51:00.49750165Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider deployment has 1 unavailable replicas, ignition-server-proxy deployment has 1 unavailable replicas, router deployment has 1 unavailable replicas]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/8cc15b2a-8360-41a2-8a2a-f0798be9df0d
correlation ID: 1eeaec8e-068a-41c6-8cee-4c44e2977d7a
client request ID: unavailable
occurredContributing tests (2)
Affected runs (1)
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet complete...full failure pattern: ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version <version> is Partial (want Completed), started <duration> ago; hosted cluster degraded: UnavailableReplicas: [ignition-server-proxy]; provider Microsoft.RedHatOpenShift Signal: IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | e2e | 1 | 1.04%1 of 96 job runs affected | IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | Show trend detailsOct 12: 0 · Oct 13: 0 · Oct 14: 0 · Oct 15: 0 · Oct 16: 0 · Oct 17: 0 · Oct 18: 0 · Oct 19: 0 · Oct 20: 2 · Oct 21: 0 · Oct 22: 0 · Oct 23: 0 · Oct 24: 0 · Oct 25: 0 | DEV, STG | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (2)fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:124]: HCP cluster rg-candidate-5-1-jn77rzqmfkjz/cluster-candidate-5-1-j65tbc should provision
Unexpected error:
<*fmt.wrapError | 0x332b74a98940>:
failed to create HCP cluster cluster-candidate-5-1-j65tbc: failed waiting for cluster="cluster-candidate-5-1-j65tbc" in resourcegroup="rg-candidate-5-1-jn77rzqmfkjz" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/be69220f-b32f-412c-9de4-3a1f140f5135
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/be69220f-b32f-412c-9de4-3a1f140f5135",
"name": "be69220f-b32f-412c-9de4-3a1f140f5135",
"status": "Failed",
"startTime": "2026-10-20T05:34:23.131093305Z",
"endTime": "2026-10-20T06:03:30.67205401Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 5.1.0-ec.1 is Partial (want Completed), started 9m40s ago; hosted cluster degraded: UnavailableReplicas: ignition-server-proxy deployment has 1 unavailable replicas"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/be69220f-b32f-412c-9de4-3a1f140f5135
correlation ID: 8aaf06ce-d046-4047-82a0-22fbcc3d8734
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:124]: HCP cluster rg-candidate-5-1-jn77rzqmfkjz/cluster-candidate-5-1-j65tbc should provision
Unexpected error:
<*fmt.wrapError | 0x332b74a98940>:
failed to create HCP cluster cluster-candidate-5-1-j65tbc: failed waiting for cluster="cluster-candidate-5-1-j65tbc" in resourcegroup="rg-candidate-5-1-jn77rzqmfkjz" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/be69220f-b32f-412c-9de4-3a1f140f5135
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/be69220f-b32f-412c-9de4-3a1f140f5135",
"name": "be69220f-b32f-412c-9de4-3a1f140f5135",
"status": "Failed",
"startTime": "2026-10-20T05:34:23.131093305Z",
"endTime": "2026-10-20T06:03:30.67205401Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 5.1.0-ec.1 is Partial (want Completed), started 9m40s ago; hosted cluster degraded: UnavailableReplicas: ignition-server-proxy deployment has 1 unavailable replicas"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/be69220f-b32f-412c-9de4-3a1f140f5135
correlation ID: 8aaf06ce-d046-4047-82a0-22fbcc3d8734
client request ID: unavailable
occurredfail [github.com/Azure/ARO-HCP/test/e2e/default_cluster_ingress_cert.go:103]: failed to create HCP cluster
Unexpected error:
<*fmt.wrapError | 0xae91b2b14a0>:
failed to create HCP cluster default-ingress-cert: failed waiting for cluster="default-ingress-cert" in resourcegroup="ingress-cert-le-5zjb4vwqmp6k" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/74351f12-7936-4804-bc6e-44e43566ba15
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/74351f12-7936-4804-bc6e-44e43566ba15",
"name": "74351f12-7936-4804-bc6e-44e43566ba15",
"status": "Failed",
"startTime": "2026-10-20T05:37:02.298920957Z",
"endTime": "2026-10-20T06:06:10.689160757Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 5.1.0-ec.1 is Partial (want Completed), started 13m18s ago; hosted cluster degraded: UnavailableReplicas: ignition-server-proxy deployment has 1 unavailable replicas"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/74351f12-7936-4804-bc6e-44e43566ba15
correlation ID: 139f95f2-0f80-424c-9b9b-7d135360bb2a
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/default_cluster_ingress_cert.go:103]: failed to create HCP cluster
Unexpected error:
<*fmt.wrapError | 0xae91b2b14a0>:
failed to create HCP cluster default-ingress-cert: failed waiting for cluster="default-ingress-cert" in resourcegroup="ingress-cert-le-5zjb4vwqmp6k" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/74351f12-7936-4804-bc6e-44e43566ba15
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/74351f12-7936-4804-bc6e-44e43566ba15",
"name": "74351f12-7936-4804-bc6e-44e43566ba15",
"status": "Failed",
"startTime": "2026-10-20T05:37:02.298920957Z",
"endTime": "2026-10-20T06:06:10.689160757Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 5.1.0-ec.1 is Partial (want Completed), started 13m18s ago; hosted cluster degraded: UnavailableReplicas: ignition-server-proxy deployment has 1 unavailable replicas"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/74351f12-7936-4804-bc6e-44e43566ba15
correlation ID: 139f95f2-0f80-424c-9b9b-7d135360bb2a
client request ID: unavailable
occurredContributing tests (2)
Affected runs (1)
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; provider Microsoft...full failure pattern: ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; provider Microsoft.RedHatOpenShift Signal: IndeterminateSignal: Indeterminate — seen in 2 prior week(s); active 1 day(s) | e2e | 1 | 1.04%1 of 96 job runs affected | IndeterminateSignal: Indeterminate — seen in 2 prior week(s); active 1 day(s) | Show trend detailsOct 12: 0 · Oct 13: 0 · Oct 14: 0 · Oct 15: 0 · Oct 16: 1 · Oct 17: 0 · Oct 18: 0 · Oct 19: 0 · Oct 20: 0 · Oct 21: 0 · Oct 22: 0 · Oct 23: 0 · Oct 24: 0 · Oct 25: 0 | STG | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kv.go:99]: failed to create HCP cluster "private-kv-cluster" with private keyvault
Unexpected error:
<*fmt.wrapError | 0x16bfc8866960>:
failed waiting for cluster="private-kv-cluster" in resourcegroup="private-keyvault-lrl2sfwlg6t6" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/bed95f6c-e54d-4fe5-bec3-0a0dd868334e
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/bed95f6c-e54d-4fe5-bec3-0a0dd868334e",
"name": "bed95f6c-e54d-4fe5-bec3-0a0dd868334e",
"status": "Failed",
"startTime": "2026-10-16T23:12:01.466039152Z",
"endTime": "2026-10-16T23:41:09.816691023Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/bed95f6c-e54d-4fe5-bec3-0a0dd868334e
correlation ID: 696a9369-e43c-4253-91d0-e6c9182748ca
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kv.go:99]: failed to create HCP cluster "private-kv-cluster" with private keyvault
Unexpected error:
<*fmt.wrapError | 0x16bfc8866960>:
failed waiting for cluster="private-kv-cluster" in resourcegroup="private-keyvault-lrl2sfwlg6t6" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/bed95f6c-e54d-4fe5-bec3-0a0dd868334e
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/bed95f6c-e54d-4fe5-bec3-0a0dd868334e",
"name": "bed95f6c-e54d-4fe5-bec3-0a0dd868334e",
"status": "Failed",
"startTime": "2026-10-16T23:12:01.466039152Z",
"endTime": "2026-10-16T23:41:09.816691023Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/bed95f6c-e54d-4fe5-bec3-0a0dd868334e
correlation ID: 696a9369-e43c-4253-91d0-e6c9182748ca
client request ID: unavailable
occurredContributing tests (1)
Affected runs (1)
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
"Couldn't get current server API group list" err="the server has asked for the client to provide credentials"full failure pattern: "Couldn't get current server API group list" err="the server has asked for the client to provide credentials" Signal: IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | e2e | 1 | 1.04%1 of 96 job runs affected | IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | Show trend detailsOct 12: 0 · Oct 13: 0 · Oct 14: 0 · Oct 15: 0 · Oct 16: 0 · Oct 17: 0 · Oct 18: 0 · Oct 19: 0 · Oct 20: 0 · Oct 21: 1 · Oct 22: 0 · Oct 23: 0 · Oct 24: 0 · Oct 25: 0 | none | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:349]: Timed out after 307.279s.
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:334 with:
kubectl should execute successfully from authorized VM with OIDC token
Unexpected error:
<*errors.errorString | 0x337cde092010>:
E1021 23:30:50.528002 2868 memcache.go:381] "Couldn't get current server API group list" err="the server has asked for the client to provide credentials"
{
s: "E1021 23:30:50.528002 2868 memcache.go:381] \"Couldn't get current server API group list\" err=\"the server has asked for the client to provide credentials\"",
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:349]: Timed out after 307.279s.
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:334 with:
kubectl should execute successfully from authorized VM with OIDC token
Unexpected error:
<*errors.errorString | 0x337cde092010>:
E1021 23:30:50.528002 2868 memcache.go:381] "Couldn't get current server API group list" err="the server has asked for the client to provide credentials"
{
s: "E1021 23:30:50.528002 2868 memcache.go:381] \"Couldn't get current server API group list\" err=\"the server has asked for the client to provide credentials\"",
}
occurredContributing tests (1)
Affected runs (1)
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
All ClusterOperators should report Available=True, but these are not available: [console (Available=False) ingress (Available=False)]full failure pattern: All ClusterOperators should report Available=True, but these are not available: [console (Available=False) ingress (Available=False)] Signal: IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | e2e | 1 | 1.04%1 of 96 job runs affected | IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | Show trend detailsOct 12: 0 · Oct 13: 0 · Oct 14: 0 · Oct 15: 0 · Oct 16: 0 · Oct 17: 0 · Oct 18: 0 · Oct 19: 0 · Oct 20: 0 · Oct 21: 0 · Oct 22: 0 · Oct 23: 1 · Oct 24: 0 · Oct 25: 0 | none | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:378]: Timed out after 634.146s.
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:377 with:
All ClusterOperators should report Available=True, but these are not available: [console (Available=False) ingress (Available=False)]
Expected
<[]string | len:2, cap:2>: [
"console (Available=False)",
"ingress (Available=False)",
]
to be empty
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:378]: Timed out after 634.146s.
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:377 with:
All ClusterOperators should report Available=True, but these are not available: [console (Available=False) ingress (Available=False)]
Expected
<[]string | len:2, cap:2>: [
"console (Available=False)",
"ingress (Available=False)",
]
to be emptyContributing tests (1)
Affected runs (1)
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
DNS for route host agnhost-e2e-sample-app-lpqxsfull failure pattern: DNS for route host agnhost-e2e-sample-app-lpqxs Signal: IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | e2e | 1 | 1.04%1 of 96 job runs affected | IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | Show trend detailsOct 12: 0 · Oct 13: 0 · Oct 14: 0 · Oct 15: 0 · Oct 16: 0 · Oct 17: 0 · Oct 18: 0 · Oct 19: 0 · Oct 20: 0 · Oct 21: 0 · Oct 22: 0 · Oct 23: 1 · Oct 24: 0 · Oct 25: 0 | none | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/e2e/default_cluster_ingress_cert.go:169]: sample application must serve the delivered default certificate
Expected success, but got an error:
<*fmt.wrapError | 0x398579d77aa0>:
DNS for route host agnhost-e2e-sample-app-lpqxs.apps.aro.d1o5s8c9e5q6t1e.xrjv.eastus2.aroapp-hcp.io did not resolve: DNS for agnhost-e2e-sample-app-lpqxs.apps.aro.d1o5s8c9e5q6t1e.xrjv.eastus2.aroapp-hcp.io did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-lpqxs.apps.aro.d1o5s8c9e5q6t1e.xrjv.eastus2.aroapp-hcp.io on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded
{
msg: "DNS for route host agnhost-e2e-sample-app-lpqxs.apps.aro.d1o5s8c9e5q6t1e.xrjv.eastus2.aroapp-hcp.io did not resolve: DNS for agnhost-e2e-sample-app-lpqxs.apps.aro.d1o5s8c9e5q6t1e.xrjv.eastus2.aroapp-hcp.io did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-lpqxs.apps.aro.d1o5s8c9e5q6t1e.xrjv.eastus2.aroapp-hcp.io on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
err: <*fmt.wrapError | 0x398579d77a60>{
msg: "DNS for agnhost-e2e-sample-app-lpqxs.apps.aro.d1o5s8c9e5q6t1e.xrjv.eastus2.aroapp-hcp.io did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-lpqxs.apps.aro.d1o5s8c9e5q6t1e.xrjv.eastus2.aroapp-hcp.io on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
err: <context.deadlineExceededError>{},
},
}
fail [github.com/Azure/ARO-HCP/test/e2e/default_cluster_ingress_cert.go:169]: sample application must serve the delivered default certificate
Expected success, but got an error:
<*fmt.wrapError | 0x398579d77aa0>:
DNS for route host agnhost-e2e-sample-app-lpqxs.apps.aro.d1o5s8c9e5q6t1e.xrjv.eastus2.aroapp-hcp.io did not resolve: DNS for agnhost-e2e-sample-app-lpqxs.apps.aro.d1o5s8c9e5q6t1e.xrjv.eastus2.aroapp-hcp.io did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-lpqxs.apps.aro.d1o5s8c9e5q6t1e.xrjv.eastus2.aroapp-hcp.io on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded
{
msg: "DNS for route host agnhost-e2e-sample-app-lpqxs.apps.aro.d1o5s8c9e5q6t1e.xrjv.eastus2.aroapp-hcp.io did not resolve: DNS for agnhost-e2e-sample-app-lpqxs.apps.aro.d1o5s8c9e5q6t1e.xrjv.eastus2.aroapp-hcp.io did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-lpqxs.apps.aro.d1o5s8c9e5q6t1e.xrjv.eastus2.aroapp-hcp.io on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
err: <*fmt.wrapError | 0x398579d77a60>{
msg: "DNS for agnhost-e2e-sample-app-lpqxs.apps.aro.d1o5s8c9e5q6t1e.xrjv.eastus2.aroapp-hcp.io did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-lpqxs.apps.aro.d1o5s8c9e5q6t1e.xrjv.eastus2.aroapp-hcp.io on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
err: <context.deadlineExceededError>{},
},
}Contributing tests (1)
Affected runs (1)
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedC...full failure pattern: ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable; hosted cluster degraded: UnavailableReplicas: [catalog-operator, hosted-cluster-config-operator, konnectivity-agent, olm-operator, openshift-apiserver, packageserver]; provider Microsoft.RedHatOpenShift Signal: IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | e2e | 1 | 1.04%1 of 96 job runs affected | IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | Show trend detailsOct 12: 0 · Oct 13: 0 · Oct 14: 0 · Oct 15: 0 · Oct 16: 0 · Oct 17: 0 · Oct 18: 0 · Oct 19: 0 · Oct 20: 1 · Oct 21: 0 · Oct 22: 0 · Oct 23: 0 · Oct 24: 0 · Oct 25: 0 | none | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kas.go:101]: failed to create HCP cluster "private-kas" with private KAS
Unexpected error:
<*fmt.wrapError | 0x9ffbe0a6720>:
failed to create HCP cluster private-kas: failed waiting for cluster="private-kas" in resourcegroup="private-kas-c2tg56ndlf7n" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/862cf1bb-7dd3-40c1-a414-17b3b0d583bd
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/862cf1bb-7dd3-40c1-a414-17b3b0d583bd",
"name": "862cf1bb-7dd3-40c1-a414-17b3b0d583bd",
"status": "Failed",
"startTime": "2026-10-20T04:46:25.770566501Z",
"endTime": "2026-10-20T05:15:25.516223632Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: cluster-node-tuning-operator, cluster-storage-operator, community-operators-catalog, certified-operators-catalog, cluster-network-operator, dns-operator, catalog-operator, cluster-version-operator, ingress-operator, olm-operator, openshift-route-controller-manager, packageserver, hosted-cluster-config-operator, csi-snapshot-controller-operator, konnectivity-agent, redhat-operators-catalog, cluster-image-registry-operator, openshift-controller-manager, cluster-policy-controller; hosted cluster degraded: UnavailableReplicas: [catalog-operator deployment has 1 unavailable replicas, hosted-cluster-config-operator deployment has 1 unavailable replicas, konnectivity-agent deployment has 1 unavailable replicas, olm-operator deployment has 1 unavailable replicas, openshift-apiserver deployment has 1 unavailable replicas, packageserver deployment has 3 unavailable replicas]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/862cf1bb-7dd3-40c1-a414-17b3b0d583bd
correlation ID: d433d7b9-c808-4842-80b1-be90eb0e3f4a
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kas.go:101]: failed to create HCP cluster "private-kas" with private KAS
Unexpected error:
<*fmt.wrapError | 0x9ffbe0a6720>:
failed to create HCP cluster private-kas: failed waiting for cluster="private-kas" in resourcegroup="private-kas-c2tg56ndlf7n" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/862cf1bb-7dd3-40c1-a414-17b3b0d583bd
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/862cf1bb-7dd3-40c1-a414-17b3b0d583bd",
"name": "862cf1bb-7dd3-40c1-a414-17b3b0d583bd",
"status": "Failed",
"startTime": "2026-10-20T04:46:25.770566501Z",
"endTime": "2026-10-20T05:15:25.516223632Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: cluster-node-tuning-operator, cluster-storage-operator, community-operators-catalog, certified-operators-catalog, cluster-network-operator, dns-operator, catalog-operator, cluster-version-operator, ingress-operator, olm-operator, openshift-route-controller-manager, packageserver, hosted-cluster-config-operator, csi-snapshot-controller-operator, konnectivity-agent, redhat-operators-catalog, cluster-image-registry-operator, openshift-controller-manager, cluster-policy-controller; hosted cluster degraded: UnavailableReplicas: [catalog-operator deployment has 1 unavailable replicas, hosted-cluster-config-operator deployment has 1 unavailable replicas, konnectivity-agent deployment has 1 unavailable replicas, olm-operator deployment has 1 unavailable replicas, openshift-apiserver deployment has 1 unavailable replicas, packageserver deployment has 3 unavailable replicas]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/862cf1bb-7dd3-40c1-a414-17b3b0d583bd
correlation ID: d433d7b9-c808-4842-80b1-be90eb0e3f4a
client request ID: unavailable
occurredContributing tests (1)
Affected runs (1)
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedC...full failure pattern: ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable; hosted cluster degraded: UnavailableReplicas: [cluster-policy-controller, ignition-server-proxy, openshift-route-controller-manager]; provider Microsoft.RedHatOpenShift Signal: IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | e2e | 1 | 1.04%1 of 96 job runs affected | IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | Show trend detailsOct 12: 0 · Oct 13: 0 · Oct 14: 0 · Oct 15: 0 · Oct 16: 0 · Oct 17: 0 · Oct 18: 0 · Oct 19: 0 · Oct 20: 1 · Oct 21: 0 · Oct 22: 0 · Oct 23: 0 · Oct 24: 0 · Oct 25: 0 | none | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_delete.go:81]: failed to create HCP cluster np-delete-hcp-cluster
Unexpected error:
<*fmt.wrapError | 0x3e05109c4140>:
failed to create HCP cluster np-delete-hcp-cluster: failed waiting for cluster="np-delete-hcp-cluster" in resourcegroup="nodepool-delete-7cbm7pwl25bd" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/c7aa03ac-1598-47ec-a13f-ec7f148e56dd
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/c7aa03ac-1598-47ec-a13f-ec7f148e56dd",
"name": "c7aa03ac-1598-47ec-a13f-ec7f148e56dd",
"status": "Failed",
"startTime": "2026-10-20T06:17:40.509470506Z",
"endTime": "2026-10-20T06:46:40.233866938Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: cluster-policy-controller, openshift-route-controller-manager; hosted cluster degraded: UnavailableReplicas: [cluster-policy-controller deployment has 1 unavailable replicas, ignition-server-proxy deployment has 1 unavailable replicas, openshift-route-controller-manager deployment has 1 unavailable replicas]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/c7aa03ac-1598-47ec-a13f-ec7f148e56dd
correlation ID: 3a5074fe-d20a-46db-b752-31f0ad8d374b
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_delete.go:81]: failed to create HCP cluster np-delete-hcp-cluster
Unexpected error:
<*fmt.wrapError | 0x3e05109c4140>:
failed to create HCP cluster np-delete-hcp-cluster: failed waiting for cluster="np-delete-hcp-cluster" in resourcegroup="nodepool-delete-7cbm7pwl25bd" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/c7aa03ac-1598-47ec-a13f-ec7f148e56dd
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/c7aa03ac-1598-47ec-a13f-ec7f148e56dd",
"name": "c7aa03ac-1598-47ec-a13f-ec7f148e56dd",
"status": "Failed",
"startTime": "2026-10-20T06:17:40.509470506Z",
"endTime": "2026-10-20T06:46:40.233866938Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: cluster-policy-controller, openshift-route-controller-manager; hosted cluster degraded: UnavailableReplicas: [cluster-policy-controller deployment has 1 unavailable replicas, ignition-server-proxy deployment has 1 unavailable replicas, openshift-route-controller-manager deployment has 1 unavailable replicas]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/c7aa03ac-1598-47ec-a13f-ec7f148e56dd
correlation ID: 3a5074fe-d20a-46db-b752-31f0ad8d374b
client request ID: unavailable
occurredContributing tests (1)
Affected runs (1)
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedC...full failure pattern: ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable; hosted cluster degraded: UnavailableReplicas: [cluster-policy-controller, konnectivity-agent, kube-controller-manager, openshift-controller-manager, openshift-route-controller-manager, packageserver]; provider Microsoft.RedHatOpenShift Signal: IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | e2e | 1 | 1.04%1 of 96 job runs affected | IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | Show trend detailsOct 12: 0 · Oct 13: 0 · Oct 14: 0 · Oct 15: 0 · Oct 16: 0 · Oct 17: 0 · Oct 18: 0 · Oct 19: 0 · Oct 20: 1 · Oct 21: 0 · Oct 22: 0 · Oct 23: 0 · Oct 24: 0 · Oct 25: 0 | none | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/e2e/cluster_tls_endpoints.go:96]: failed to create HCP cluster for TLS endpoint test
Unexpected error:
<*fmt.wrapError | 0x3e9e304bc800>:
failed to create HCP cluster tls-endpoint-hcp-cluster: failed waiting for cluster="tls-endpoint-hcp-cluster" in resourcegroup="tls-endpoint-cluster-hrm4cs7zffg2" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/a343792f-4e4d-47a0-a712-41d3ec54b93a
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/a343792f-4e4d-47a0-a712-41d3ec54b93a",
"name": "a343792f-4e4d-47a0-a712-41d3ec54b93a",
"status": "Failed",
"startTime": "2026-10-20T04:46:15.624065092Z",
"endTime": "2026-10-20T05:15:15.524291644Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: konnectivity-agent, openshift-controller-manager, openshift-route-controller-manager, cluster-policy-controller; hosted cluster degraded: UnavailableReplicas: [cluster-policy-controller deployment has 1 unavailable replicas, konnectivity-agent deployment has 1 unavailable replicas, kube-controller-manager deployment has 1 unavailable replicas, openshift-controller-manager deployment has 1 unavailable replicas, openshift-route-controller-manager deployment has 1 unavailable replicas, packageserver deployment has 1 unavailable replicas]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/a343792f-4e4d-47a0-a712-41d3ec54b93a
correlation ID: 8327c480-3421-4e24-a66c-3cea0477dfa6
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_tls_endpoints.go:96]: failed to create HCP cluster for TLS endpoint test
Unexpected error:
<*fmt.wrapError | 0x3e9e304bc800>:
failed to create HCP cluster tls-endpoint-hcp-cluster: failed waiting for cluster="tls-endpoint-hcp-cluster" in resourcegroup="tls-endpoint-cluster-hrm4cs7zffg2" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/a343792f-4e4d-47a0-a712-41d3ec54b93a
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/a343792f-4e4d-47a0-a712-41d3ec54b93a",
"name": "a343792f-4e4d-47a0-a712-41d3ec54b93a",
"status": "Failed",
"startTime": "2026-10-20T04:46:15.624065092Z",
"endTime": "2026-10-20T05:15:15.524291644Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: konnectivity-agent, openshift-controller-manager, openshift-route-controller-manager, cluster-policy-controller; hosted cluster degraded: UnavailableReplicas: [cluster-policy-controller deployment has 1 unavailable replicas, konnectivity-agent deployment has 1 unavailable replicas, kube-controller-manager deployment has 1 unavailable replicas, openshift-controller-manager deployment has 1 unavailable replicas, openshift-route-controller-manager deployment has 1 unavailable replicas, packageserver deployment has 1 unavailable replicas]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/a343792f-4e4d-47a0-a712-41d3ec54b93a
correlation ID: 8327c480-3421-4e24-a66c-3cea0477dfa6
client request ID: unavailable
occurredContributing tests (1)
Affected runs (1)
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedC...full failure pattern: ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable; hosted cluster degraded: UnavailableReplicas: [kube-apiserver, router]; provider Microsoft.RedHatOpenShift Signal: IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | e2e | 1 | 1.04%1 of 96 job runs affected | IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | Show trend detailsOct 12: 0 · Oct 13: 0 · Oct 14: 0 · Oct 15: 0 · Oct 16: 0 · Oct 17: 0 · Oct 18: 0 · Oct 19: 0 · Oct 20: 0 · Oct 21: 0 · Oct 22: 1 · Oct 23: 0 · Oct 24: 0 · Oct 25: 0 | DEV | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:124]: HCP cluster rg-candidate-4-23-d44bvbtv22rz/cluster-candidate-4-23-f2rpqw should provision
Unexpected error:
<*fmt.wrapError | 0x126c5cc0e60>:
failed to create HCP cluster cluster-candidate-4-23-f2rpqw: failed waiting for cluster="cluster-candidate-4-23-f2rpqw" in resourcegroup="rg-candidate-4-23-d44bvbtv22rz" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/efe747a0-4a74-4970-aae9-c5a7e67a1f5b
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/efe747a0-4a74-4970-aae9-c5a7e67a1f5b",
"name": "efe747a0-4a74-4970-aae9-c5a7e67a1f5b",
"status": "Failed",
"startTime": "2026-10-22T02:21:35.82450299Z",
"endTime": "2026-10-22T02:50:40.503091649Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: openshift-route-controller-manager, catalog-operator, redhat-operators-catalog, machine-approver, openshift-controller-manager, konnectivity-agent, olm-operator, certified-operators-catalog, kube-controller-manager, cluster-version-operator, azure-cloud-controller-manager, kube-storage-version-migrator, community-operators-catalog, openshift-apiserver, cluster-policy-controller, cluster-autoscaler, cluster-network-operator, dns-operator, control-plane-pki-operator, kube-scheduler, hosted-cluster-config-operator, cluster-storage-operator, cluster-node-tuning-operator, csi-snapshot-controller-operator, packageserver, ingress-operator, cluster-image-registry-operator, olm-collect-profiles; hosted cluster degraded: UnavailableReplicas: [kube-apiserver deployment has 1 unavailable replicas, router deployment has 1 unavailable replicas]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/efe747a0-4a74-4970-aae9-c5a7e67a1f5b
correlation ID: 08db8465-69b5-4f99-aabf-f55441aa0ace
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:124]: HCP cluster rg-candidate-4-23-d44bvbtv22rz/cluster-candidate-4-23-f2rpqw should provision
Unexpected error:
<*fmt.wrapError | 0x126c5cc0e60>:
failed to create HCP cluster cluster-candidate-4-23-f2rpqw: failed waiting for cluster="cluster-candidate-4-23-f2rpqw" in resourcegroup="rg-candidate-4-23-d44bvbtv22rz" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/efe747a0-4a74-4970-aae9-c5a7e67a1f5b
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/efe747a0-4a74-4970-aae9-c5a7e67a1f5b",
"name": "efe747a0-4a74-4970-aae9-c5a7e67a1f5b",
"status": "Failed",
"startTime": "2026-10-22T02:21:35.82450299Z",
"endTime": "2026-10-22T02:50:40.503091649Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: openshift-route-controller-manager, catalog-operator, redhat-operators-catalog, machine-approver, openshift-controller-manager, konnectivity-agent, olm-operator, certified-operators-catalog, kube-controller-manager, cluster-version-operator, azure-cloud-controller-manager, kube-storage-version-migrator, community-operators-catalog, openshift-apiserver, cluster-policy-controller, cluster-autoscaler, cluster-network-operator, dns-operator, control-plane-pki-operator, kube-scheduler, hosted-cluster-config-operator, cluster-storage-operator, cluster-node-tuning-operator, csi-snapshot-controller-operator, packageserver, ingress-operator, cluster-image-registry-operator, olm-collect-profiles; hosted cluster degraded: UnavailableReplicas: [kube-apiserver deployment has 1 unavailable replicas, router deployment has 1 unavailable replicas]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/efe747a0-4a74-4970-aae9-c5a7e67a1f5b
correlation ID: 08db8465-69b5-4f99-aabf-f55441aa0ace
client request ID: unavailable
occurredContributing tests (1)
Affected runs (1)
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedC...full failure pattern: ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable; hosted cluster degraded: UnavailableReplicas: [openshift-route-controller-manager]; provider Microsoft.RedHatOpenShift Signal: IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | e2e | 1 | 1.04%1 of 96 job runs affected | IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | Show trend detailsOct 12: 0 · Oct 13: 0 · Oct 14: 0 · Oct 15: 0 · Oct 16: 0 · Oct 17: 0 · Oct 18: 0 · Oct 19: 0 · Oct 20: 0 · Oct 21: 0 · Oct 22: 1 · Oct 23: 0 · Oct 24: 0 · Oct 25: 0 | none | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_delete.go:81]: failed to create HCP cluster np-delete-hcp-cluster
Unexpected error:
<*fmt.wrapError | 0x3ede8eb28c00>:
failed to create HCP cluster np-delete-hcp-cluster: failed waiting for cluster="np-delete-hcp-cluster" in resourcegroup="nodepool-delete-qzb5xz65mghw" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/43a33130-1505-4488-8d2c-b0f13de47cff
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/43a33130-1505-4488-8d2c-b0f13de47cff",
"name": "43a33130-1505-4488-8d2c-b0f13de47cff",
"status": "Failed",
"startTime": "2026-10-22T03:04:08.329961738Z",
"endTime": "2026-10-22T03:33:15.140670523Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: openshift-route-controller-manager; hosted cluster degraded: UnavailableReplicas: openshift-route-controller-manager deployment has 1 unavailable replicas"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/43a33130-1505-4488-8d2c-b0f13de47cff
correlation ID: 2c680bb1-7f2a-45c1-a484-bbb38ed78677
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_delete.go:81]: failed to create HCP cluster np-delete-hcp-cluster
Unexpected error:
<*fmt.wrapError | 0x3ede8eb28c00>:
failed to create HCP cluster np-delete-hcp-cluster: failed waiting for cluster="np-delete-hcp-cluster" in resourcegroup="nodepool-delete-qzb5xz65mghw" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/43a33130-1505-4488-8d2c-b0f13de47cff
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/43a33130-1505-4488-8d2c-b0f13de47cff",
"name": "43a33130-1505-4488-8d2c-b0f13de47cff",
"status": "Failed",
"startTime": "2026-10-22T03:04:08.329961738Z",
"endTime": "2026-10-22T03:33:15.140670523Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: openshift-route-controller-manager; hosted cluster degraded: UnavailableReplicas: openshift-route-controller-manager deployment has 1 unavailable replicas"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/43a33130-1505-4488-8d2c-b0f13de47cff
correlation ID: 2c680bb1-7f2a-45c1-a484-bbb38ed78677
client request ID: unavailable
occurredContributing tests (1)
Affected runs (1)
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedC...full failure pattern: ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KASLoadBalancerNotReachable: Get "<url>": EOF; hosted cluster degraded: UnavailableReplicas: [ignition-server, kube-apiserver]; provider Microsoft.RedHatOpenShift Signal: IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | e2e | 1 | 1.04%1 of 96 job runs affected | IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | Show trend detailsOct 12: 0 · Oct 13: 0 · Oct 14: 0 · Oct 15: 0 · Oct 16: 0 · Oct 17: 0 · Oct 18: 0 · Oct 19: 0 · Oct 20: 1 · Oct 21: 0 · Oct 22: 0 · Oct 23: 0 · Oct 24: 0 · Oct 25: 0 | none | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/e2e/simple_negative_cases.go:110]: failed to create HCP cluster negative-tests-cluster
Unexpected error:
<*fmt.wrapError | 0x28688821ac40>:
failed to create HCP cluster negative-tests-cluster: failed waiting for cluster="negative-tests-cluster" in resourcegroup="rg-negative-tests-69bxttgtnjdc" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/81fcc58f-c93e-4c14-8462-568e76b1f007
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/81fcc58f-c93e-4c14-8462-568e76b1f007",
"name": "81fcc58f-c93e-4c14-8462-568e76b1f007",
"status": "Failed",
"startTime": "2026-10-20T06:29:44.602488636Z",
"endTime": "2026-10-20T06:58:50.194412463Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KASLoadBalancerNotReachable: Get \"https://api-int.w4p3g9f2y8m5o4t.2kot.eastus2euap.aroapp-hcp.io:443/healthz\": EOF; hosted cluster degraded: UnavailableReplicas: [ignition-server deployment has 1 unavailable replicas, kube-apiserver deployment has 1 unavailable replicas]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/81fcc58f-c93e-4c14-8462-568e76b1f007
correlation ID: 57aab543-83b3-413d-b1ad-731d8e533b5f
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/simple_negative_cases.go:110]: failed to create HCP cluster negative-tests-cluster
Unexpected error:
<*fmt.wrapError | 0x28688821ac40>:
failed to create HCP cluster negative-tests-cluster: failed waiting for cluster="negative-tests-cluster" in resourcegroup="rg-negative-tests-69bxttgtnjdc" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/81fcc58f-c93e-4c14-8462-568e76b1f007
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/81fcc58f-c93e-4c14-8462-568e76b1f007",
"name": "81fcc58f-c93e-4c14-8462-568e76b1f007",
"status": "Failed",
"startTime": "2026-10-20T06:29:44.602488636Z",
"endTime": "2026-10-20T06:58:50.194412463Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KASLoadBalancerNotReachable: Get \"https://api-int.w4p3g9f2y8m5o4t.2kot.eastus2euap.aroapp-hcp.io:443/healthz\": EOF; hosted cluster degraded: UnavailableReplicas: [ignition-server deployment has 1 unavailable replicas, kube-apiserver deployment has 1 unavailable replicas]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/81fcc58f-c93e-4c14-8462-568e76b1f007
correlation ID: 57aab543-83b3-413d-b1ad-731d8e533b5f
client request ID: unavailable
occurredContributing tests (1)
Affected runs (1)
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet complete...full failure pattern: ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version <version> is Partial (want Completed), started <duration> ago; hosted cluster degraded: UnavailableReplicas: [cluster-policy-controller, ignition-server-proxy, kube-controller-manager, packageserver]; provider Microsoft.RedHatOpenShift Signal: IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | e2e | 1 | 1.04%1 of 96 job runs affected | IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | Show trend detailsOct 12: 0 · Oct 13: 0 · Oct 14: 0 · Oct 15: 0 · Oct 16: 0 · Oct 17: 0 · Oct 18: 0 · Oct 19: 0 · Oct 20: 1 · Oct 21: 0 · Oct 22: 0 · Oct 23: 0 · Oct 24: 0 · Oct 25: 0 | none | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_list_and_verify.go:87]: failed to create HCP cluster for external auth list test
Unexpected error:
<*fmt.wrapError | 0x2072efeace0>:
failed to create HCP cluster ea-listmlx87m: failed waiting for cluster="ea-listmlx87m" in resourcegroup="ea-list-nlprzq99rzmw" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/9cb322ee-80e1-4c68-9a22-2c73b38798de
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/9cb322ee-80e1-4c68-9a22-2c73b38798de",
"name": "9cb322ee-80e1-4c68-9a22-2c73b38798de",
"status": "Failed",
"startTime": "2026-10-20T05:37:03.327418936Z",
"endTime": "2026-10-20T06:06:10.675381294Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.39 is Partial (want Completed), started 14m23s ago; hosted cluster degraded: UnavailableReplicas: [cluster-policy-controller deployment has 1 unavailable replicas, ignition-server-proxy deployment has 1 unavailable replicas, kube-controller-manager deployment has 1 unavailable replicas, packageserver deployment has 1 unavailable replicas]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/9cb322ee-80e1-4c68-9a22-2c73b38798de
correlation ID: b09d3f5f-9c93-4d50-8097-63a061652137
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_list_and_verify.go:87]: failed to create HCP cluster for external auth list test
Unexpected error:
<*fmt.wrapError | 0x2072efeace0>:
failed to create HCP cluster ea-listmlx87m: failed waiting for cluster="ea-listmlx87m" in resourcegroup="ea-list-nlprzq99rzmw" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/9cb322ee-80e1-4c68-9a22-2c73b38798de
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/9cb322ee-80e1-4c68-9a22-2c73b38798de",
"name": "9cb322ee-80e1-4c68-9a22-2c73b38798de",
"status": "Failed",
"startTime": "2026-10-20T05:37:03.327418936Z",
"endTime": "2026-10-20T06:06:10.675381294Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.39 is Partial (want Completed), started 14m23s ago; hosted cluster degraded: UnavailableReplicas: [cluster-policy-controller deployment has 1 unavailable replicas, ignition-server-proxy deployment has 1 unavailable replicas, kube-controller-manager deployment has 1 unavailable replicas, packageserver deployment has 1 unavailable replicas]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/9cb322ee-80e1-4c68-9a22-2c73b38798de
correlation ID: b09d3f5f-9c93-4d50-8097-63a061652137
client request ID: unavailable
occurredContributing tests (1)
Affected runs (1)
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||