Executive Status
EnvEnvironment partition: dev, int, stg, or prod. | GoalConfigured success target and run scope for this environment. INT/STG/PROD use all E2E job runs. DEV uses post-good runs plus Tide batches: runs on the final commit of a PR that later merged, or batch retests of PRs that passed their individual e2e-parallel checks. | RunsNumber of job runs in the selected goal definition for this environment. | SuccessSuccess rate on the selected goal definition: (runs - failed runs) / runs * 100. | Gap vs targetDifference in percentage points between current success and the configured target rate. | Change vs prevHow much the success rate changed compared with the immediately preceding equal-length window, using the same goal definition as this row. | Provision successProvisioning here means infrastructure setup before E2E tests start. A run counts as provision-successful if it got past provisioning, even if it later failed during E2E. Runs that failed for 'Other' CI-infrastructure reasons are excluded because those runs never reached the provision step. DEV rows use post-good runs plus Tide batches. INT/STG/PROD show n/a because provisioning is not part of those environments. | Provision change vs prevChange in provision success, in percentage points, compared with the previous equal-length window. Uses the same provision-success definition as the Provision success column. DEV only; INT/STG/PROD show n/a. | E2E successAmong runs that reached E2E execution, this shows how many completed successfully. Provision failures and 'Other' CI-infrastructure failures are excluded because those runs never reached the E2E step. DEV rows use only runs after the final push to a PR that later merged. INT/STG/PROD use all E2E job runs. | E2E success vs prevChange in E2E success, in percentage points, compared with the previous equal-length window. Uses the same E2E-success definition as the E2E success column. |
|---|---|---|---|---|---|---|---|---|---|
| DEV | 95% - Post-good runs + Tide batches | 101 | 46.53% (off track) | -48.47pp | -0.78pp | 91.67% (88/96) | -1.48pp | 53.41% (47/88) | -0.58pp |
| INT | 95% - All E2E job runs | 15 | 73.33% (off track) | -21.67pp | +59.54pp | n/a | n/a | 100.00% (11/11) | +0.00pp |
| STG | 95% - All E2E job runs | 14 | 42.86% (off track) | -52.14pp | +16.19pp | n/a | n/a | 60.00% (6/10) | +26.67pp |
| PROD | 95% - All E2E job runs | 46 | 69.57% (off track) | -25.43pp | +11.94pp | n/a | n/a | 76.19% (32/42) | +15.48pp |
Environment: DEV
Daily Run Outcomes
Daily Run Outcomes for DEV Goal Basis (post-good + batches)
No tests below 95.00% in this window with at least 10 runs.
Up to 50 failure patterns are loaded (minimum 1.00% impact), with 10 shown by default. Click column headers to re-sort.
Jump to the full Failure Patterns view for this window
| Failure patternThe canonical description of a recurring CI failure, extracted and normalized from raw logs. | Failed atThe stage of the job run where this failure occurred: 'provision' (environment setup, DEV only), 'e2e' (test suite execution), or 'other' (CI infrastructure issues that did not produce a failure pattern). | Number of distinct job runs where this failure pattern was detected. | Percentage of all job runs in this environment affected by this failure pattern during the selected window. | Classification of this failure pattern: Regression (likely caused by a specific PR), Flake (intermittent failure spread across days), Noise (low-quality or generic pattern), or Indeterminate. | TrendShows daily activity for this failure pattern in a trailing window anchored to the selected end date. The sparkline covers at least 7 days and at most 14 days, depending on the current window size. | Also inOther environments where the same failure pattern was also detected during the selected window. | ||||||||||||||||||||||||||||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
alert [hcp] KubePodNotReady firedfull failure pattern: alert [hcp] KubePodNotReady fired Signal: FlakeSignal: Flake — present in 4 prior week(s); spread across 4 days | alert | 10 | 4.26%10 of 235 job runs affected | FlakeSignal: Flake — present in 4 prior week(s); spread across 4 days | Show trend detailsOct 25: 1 · Oct 26: 4 · Oct 27: 5 · Oct 28: 1 · Oct 29: 0 · Oct 30: 0 · Oct 31: 0 · Nov 1: 0 | none | ||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (3)alert fired 1 time(s) Firing 1: State: Resolved Started: 2026-10-28T20:00:50Z Ended: 2026-10-28T20:06:48Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j5001984-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tnvsms9l35e1dde1cr2bnekd5epm5j9-ingress-cert-le", pod="openshift-apiserver-7cb4d98fc-rzx96", region="westus3", severity="warning" Description: Pod ocm-arohcpci01-2tnvsms9l35e1dde1cr2bnekd5epm5j9-ingress-cert-le/openshift-apiserver-7cb4d98fc-rzx96 has been in a non-ready state for longer than 5 minutes. alert fired 2 time(s) (1 unknown, 1 known) Firing 1: State: Resolved Started: 2026-10-27T23:40:09Z Ended: 2026-10-27T23:57:09Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j1933056-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tneda066vj6kdgdjqh3ub42gu0lckg9-ephemeral-disk", pod="etcd-0", region="centralus", severity="warning" Description: Pod ocm-arohcpci01-2tneda066vj6kdgdjqh3ub42gu0lckg9-ephemeral-disk/etcd-0 has been in a non-ready state for longer than 5 minutes. alert fired 13 time(s) (12 unknown, 1 known) Firing 1: State: Fired (not resolved) Started: 2026-10-27T15:44:12Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j1476864-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tn75avtj8u4futjt2om8gigffgddu35-ingress-cert-le", pod="openshift-apiserver-5fbf969589-rwzww", region="westus3", severity="warning" Description: Pod ocm-arohcpci01-2tn75avtj8u4futjt2om8gigffgddu35-ingress-cert-le/openshift-apiserver-5fbf969589-rwzww has been in a non-ready state for longer than 5 minutes. Firing 2: State: Fired (not resolved) Started: 2026-10-27T15:44:12Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j1476864-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tn74o7qhu6go8i6sdevboooi4cvm5o8-o9s7l1c0u6a0c2b", pod="openshift-apiserver-7c75f5b8c-869mc", region="westus3", severity="warning" Description: Pod ocm-arohcpci01-2tn74o7qhu6go8i6sdevboooi4cvm5o8-o9s7l1c0u6a0c2b/openshift-apiserver-7c75f5b8c-869mc has been in a non-ready state for longer than 5 minutes. Firing 3: State: Fired (not resolved) Started: 2026-10-27T15:45:13Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j1476864-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tn75q9b8j8s3323r8hbovnl3fa3ah93-a8h6t7r0k6i7a1w", pod="openshift-apiserver-6cf589c77c-9cn7v", region="westus3", severity="warning" Description: Pod ocm-arohcpci01-2tn75q9b8j8s3323r8hbovnl3fa3ah93-a8h6t7r0k6i7a1w/openshift-apiserver-6cf589c77c-9cn7v has been in a non-ready state for longer than 5 minutes. Firing 4: State: Fired (not resolved) Started: 2026-10-27T15:45:13Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j1476864-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tn74rifog60es9rvt2t2mfrpvs36h9e-t2y6g6x3g9p9s5t", pod="openshift-apiserver-9c9cfdf89-62q4b", region="westus3", severity="warning" Description: Pod ocm-arohcpci01-2tn74rifog60es9rvt2t2mfrpvs36h9e-t2y6g6x3g9p9s5t/openshift-apiserver-9c9cfdf89-62q4b has been in a non-ready state for longer than 5 minutes. Firing 5: State: Fired (not resolved) Started: 2026-10-27T15:45:13Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j1476864-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tn757i0j3orr665p0dcqg487jcvp9a9-q1n4k4z8u2y9x7u", pod="openshift-apiserver-7bbccfbcd8-nj4zh", region="westus3", severity="warning" Description: Pod ocm-arohcpci01-2tn757i0j3orr665p0dcqg487jcvp9a9-q1n4k4z8u2y9x7u/openshift-apiserver-7bbccfbcd8-nj4zh has been in a non-ready state for longer than 5 minutes. Firing 6: State: Fired (not resolved) Started: 2026-10-27T15:45:13Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j1476864-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tn75637s7ka00dgh3b0o5nl473kscrl-h2n8s4a0l1b6e9x", pod="openshift-apiserver-7b79cf8d6d-klztg", region="westus3", severity="warning" Description: Pod ocm-arohcpci01-2tn75637s7ka00dgh3b0o5nl473kscrl-h2n8s4a0l1b6e9x/openshift-apiserver-7b79cf8d6d-klztg has been in a non-ready state for longer than 5 minutes. Firing 7: State: Fired (not resolved) Started: 2026-10-27T15:45:13Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j1476864-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tn7522f0rjmsl54mp9vum53bfeis22t-x1e5u3y9j2l5a2u", pod="openshift-apiserver-7d75b894c6-n4cmx", region="westus3", severity="warning" Description: Pod ocm-arohcpci01-2tn7522f0rjmsl54mp9vum53bfeis22t-x1e5u3y9j2l5a2u/openshift-apiserver-7d75b894c6-n4cmx has been in a non-ready state for longer than 5 minutes. Firing 8: State: Fired (not resolved) Started: 2026-10-27T15:46:12Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j1476864-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tn75ndq1oh2entq0cv9ulrug5jog853-ingress-cert-le", pod="openshift-apiserver-764b69ffcb-7ngbg", region="westus3", severity="warning" Description: Pod ocm-arohcpci01-2tn75ndq1oh2entq0cv9ulrug5jog853-ingress-cert-le/openshift-apiserver-764b69ffcb-7ngbg has been in a non-ready state for longer than 5 minutes. Firing 9: State: Fired (not resolved) Started: 2026-10-27T15:46:12Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j1476864-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tn7cebrepv6hitmognobm7c9cl2clnd-o5c0v9b6h0c5z0a", pod="cluster-policy-controller-698bfd9c65-v6xdj", region="westus3", severity="warning" Description: Pod ocm-arohcpci01-2tn7cebrepv6hitmognobm7c9cl2clnd-o5c0v9b6h0c5z0a/cluster-policy-controller-698bfd9c65-v6xdj has been in a non-ready state for longer than 5 minutes. Firing 10: State: Fired (not resolved) Started: 2026-10-27T15:46:12Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j1476864-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tn75h85al80vo23t362mjtc3qgd04dp-c1b6v6c2x7c8p8f", pod="openshift-apiserver-57bf4dc46f-h7mj5", region="westus3", severity="warning" Description: Pod ocm-arohcpci01-2tn75h85al80vo23t362mjtc3qgd04dp-c1b6v6c2x7c8p8f/openshift-apiserver-57bf4dc46f-h7mj5 has been in a non-ready state for longer than 5 minutes. Firing 11: State: Fired (not resolved) Started: 2026-10-27T15:46:12Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j1476864-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tn759i4ns4omt60u9uecngkrn5mj618-u8q4i0h7g0i1r6k", pod="openshift-apiserver-59b86f6c67-76jwk", region="westus3", severity="warning" Description: Pod ocm-arohcpci01-2tn759i4ns4omt60u9uecngkrn5mj618-u8q4i0h7g0i1r6k/openshift-apiserver-59b86f6c67-76jwk has been in a non-ready state for longer than 5 minutes. Firing 12: State: Fired (not resolved) Started: 2026-10-27T15:47:14Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j1476864-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tn7615u8n39pf897sksqb5f7beb9d2u-b0o3o9t7e5r0u9g", pod="openshift-apiserver-77c77b89b8-kr9pl", region="westus3", severity="warning" Description: Pod ocm-arohcpci01-2tn7615u8n39pf897sksqb5f7beb9d2u-b0o3o9t7e5r0u9g/openshift-apiserver-77c77b89b8-kr9pl has been in a non-ready state for longer than 5 minutes. Contributing tests (1)
Affected runs (10) | ||||||||||||||||||||||||||||||||||||||||||||||||||
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedC...full failure pattern: ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider]; provider Microsoft.RedHatOpenShift Signal: FlakeSignal: Flake — present in 4 prior week(s); spread across 4 days | e2e | 10 | 4.26%10 of 235 job runs affected | FlakeSignal: Flake — present in 4 prior week(s); spread across 4 days | Show trend detailsOct 25: 1 · Oct 26: 2 · Oct 27: 3 · Oct 28: 5 · Oct 29: 0 · Oct 30: 0 · Oct 31: 0 · Nov 1: 0 | INT, PROD | ||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (3)fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kas.go:101]: failed to create HCP cluster "private-kas" with private KAS
Unexpected error:
<*fmt.wrapError | 0x3a2a28c8a260>:
failed to create HCP cluster private-kas: failed waiting for cluster="private-kas" in resourcegroup="private-kas-h8dst6v865z6" to finish creating: GET https://rp.j1157504.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/455ab4ea-ab03-4a58-91c6-5ad5406405ff
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/455ab4ea-ab03-4a58-91c6-5ad5406405ff",
"name": "455ab4ea-ab03-4a58-91c6-5ad5406405ff",
"status": "Failed",
"startTime": "2026-10-28T20:16:02.73250898Z",
"endTime": "2026-10-28T20:45:04.251585998Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://rp.j1157504.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/455ab4ea-ab03-4a58-91c6-5ad5406405ff
correlation ID: 582a61df-2172-42e6-962b-b544533d3da6
client request ID: a46122fb-f2eb-46f8-9be1-73dd5a442cce
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kas.go:101]: failed to create HCP cluster "private-kas" with private KAS
Unexpected error:
<*fmt.wrapError | 0x3a2a28c8a260>:
failed to create HCP cluster private-kas: failed waiting for cluster="private-kas" in resourcegroup="private-kas-h8dst6v865z6" to finish creating: GET https://rp.j1157504.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/455ab4ea-ab03-4a58-91c6-5ad5406405ff
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/455ab4ea-ab03-4a58-91c6-5ad5406405ff",
"name": "455ab4ea-ab03-4a58-91c6-5ad5406405ff",
"status": "Failed",
"startTime": "2026-10-28T20:16:02.73250898Z",
"endTime": "2026-10-28T20:45:04.251585998Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://rp.j1157504.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/455ab4ea-ab03-4a58-91c6-5ad5406405ff
correlation ID: 582a61df-2172-42e6-962b-b544533d3da6
client request ID: a46122fb-f2eb-46f8-9be1-73dd5a442cce
occurredfail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:111]: failed to create HCP cluster "cidr-connectivity-test" with authorized CIDRs
Unexpected error:
<*fmt.wrapError | 0x370d5101eb20>:
failed to create HCP cluster cidr-connectivity-test: failed waiting for cluster="cidr-connectivity-test" in resourcegroup="e2e-cidr-connectivity-tgjxrh8mcrw6" to finish creating: GET https://rp.j8717568.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/cce1648d-4b31-48bb-9884-f170bb5391eb
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/cce1648d-4b31-48bb-9884-f170bb5391eb",
"name": "cce1648d-4b31-48bb-9884-f170bb5391eb",
"status": "Failed",
"startTime": "2026-10-28T19:28:55.409225524Z",
"endTime": "2026-10-28T19:58:03.222551854Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://rp.j8717568.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/cce1648d-4b31-48bb-9884-f170bb5391eb
correlation ID: d6e5235d-8c3c-4404-82e9-f3e3b3165e1e
client request ID: 4d040942-ab41-4fbc-8d6e-d69a84105ef1
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:111]: failed to create HCP cluster "cidr-connectivity-test" with authorized CIDRs
Unexpected error:
<*fmt.wrapError | 0x370d5101eb20>:
failed to create HCP cluster cidr-connectivity-test: failed waiting for cluster="cidr-connectivity-test" in resourcegroup="e2e-cidr-connectivity-tgjxrh8mcrw6" to finish creating: GET https://rp.j8717568.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/cce1648d-4b31-48bb-9884-f170bb5391eb
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/cce1648d-4b31-48bb-9884-f170bb5391eb",
"name": "cce1648d-4b31-48bb-9884-f170bb5391eb",
"status": "Failed",
"startTime": "2026-10-28T19:28:55.409225524Z",
"endTime": "2026-10-28T19:58:03.222551854Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://rp.j8717568.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/cce1648d-4b31-48bb-9884-f170bb5391eb
correlation ID: d6e5235d-8c3c-4404-82e9-f3e3b3165e1e
client request ID: 4d040942-ab41-4fbc-8d6e-d69a84105ef1
occurredfail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:146]: failed to create HCP cluster np-version-upgrade-cluster-jl2sj8 with version 4.21.40
Unexpected error:
<*fmt.wrapError | 0x2ee0c1834d20>:
failed to create HCP cluster np-version-upgrade-cluster-jl2sj8: failed waiting for cluster="np-version-upgrade-cluster-jl2sj8" in resourcegroup="rg-np-version-upgrade-jl2sj8-22vp9qvqkmcf" to finish creating: GET https://rp.j9822336.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/267d3daf-f468-46eb-af3a-f1b72d573583
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/267d3daf-f468-46eb-af3a-f1b72d573583",
"name": "267d3daf-f468-46eb-af3a-f1b72d573583",
"status": "Failed",
"startTime": "2026-10-28T19:42:27.649991944Z",
"endTime": "2026-10-28T20:11:28.996573337Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://rp.j9822336.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/267d3daf-f468-46eb-af3a-f1b72d573583
correlation ID: c45a1c96-f9ee-44e0-8639-c952cdb59a23
client request ID: fe15252c-edf3-4b0e-8aaa-b5cd06393883
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:146]: failed to create HCP cluster np-version-upgrade-cluster-jl2sj8 with version 4.21.40
Unexpected error:
<*fmt.wrapError | 0x2ee0c1834d20>:
failed to create HCP cluster np-version-upgrade-cluster-jl2sj8: failed waiting for cluster="np-version-upgrade-cluster-jl2sj8" in resourcegroup="rg-np-version-upgrade-jl2sj8-22vp9qvqkmcf" to finish creating: GET https://rp.j9822336.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/267d3daf-f468-46eb-af3a-f1b72d573583
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/267d3daf-f468-46eb-af3a-f1b72d573583",
"name": "267d3daf-f468-46eb-af3a-f1b72d573583",
"status": "Failed",
"startTime": "2026-10-28T19:42:27.649991944Z",
"endTime": "2026-10-28T20:11:28.996573337Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://rp.j9822336.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/267d3daf-f468-46eb-af3a-f1b72d573583
correlation ID: c45a1c96-f9ee-44e0-8639-c952cdb59a23
client request ID: fe15252c-edf3-4b0e-8aaa-b5cd06393883
occurredContributing tests (10)
Affected runs (10) | ||||||||||||||||||||||||||||||||||||||||||||||||||
alert [hcp] KubePodCrashLooping firedfull failure pattern: alert [hcp] KubePodCrashLooping fired Signal: FlakeSignal: Flake — present in 4 prior week(s); spread across 4 days | alert | 9 | 3.83%9 of 235 job runs affected | FlakeSignal: Flake — present in 4 prior week(s); spread across 4 days | Show trend detailsOct 25: 2 · Oct 26: 3 · Oct 27: 2 · Oct 28: 4 · Oct 29: 0 · Oct 30: 0 · Oct 31: 0 · Nov 1: 0 | none | ||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (3)alert fired 2 time(s) Firing 1: State: Resolved Started: 2026-10-28T20:39:20Z Ended: 2026-10-28T21:05:19Z Severity: Sev3 Labels: alertname="KubePodCrashLooping", cluster="ci01-j1157504-mgmt-2", component="kubernetes-infrastructure", container="etcd", endpoint="http", environment="ci01", instance="10.128.64.172:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j1157504/providers/microsoft.monitor/accounts/hcps-j1157504", microsoft.metrics_include_label="service", namespace="ocm-arohcpci01-2to0k0skbecjdmuid7733ieqr7h7trvg-private-kas", pod="etcd-0", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", reason="CrashLoopBackOff", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="777a1913-78dc-4e11-9844-4f50d6675327" Description: Pod ocm-arohcpci01-2to0k0skbecjdmuid7733ieqr7h7trvg-private-kas/etcd-0 (etcd) is in waiting state (reason: "CrashLoopBackOff"). Firing 2: State: Resolved Started: 2026-10-28T20:40:18Z Ended: 2026-10-28T21:06:20Z Severity: Sev3 Labels: alertname="KubePodCrashLooping", cluster="ci01-j1157504-mgmt-2", component="kubernetes-infrastructure", container="etcd", endpoint="http", environment="ci01", instance="10.128.64.172:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j1157504/providers/microsoft.monitor/accounts/hcps-j1157504", microsoft.metrics_include_label="service", namespace="ocm-arohcpci01-2to0k0skbecjdmuid7733ieqr7h7trvg-private-kas", pod="etcd-0", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", reason="CrashLoopBackOff", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="777a1913-78dc-4e11-9844-4f50d6675327" Description: Pod ocm-arohcpci01-2to0k0skbecjdmuid7733ieqr7h7trvg-private-kas/etcd-0 (etcd) is in waiting state (reason: "CrashLoopBackOff"). alert fired 2 time(s) Firing 1: State: Resolved Started: 2026-10-28T19:50:44Z Ended: 2026-10-28T20:18:45Z Severity: Sev3 Labels: alertname="KubePodCrashLooping", cluster="ci01-j8717568-mgmt-2", component="kubernetes-infrastructure", container="etcd", endpoint="http", environment="ci01", instance="10.128.64.254:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j8717568/providers/microsoft.monitor/accounts/hcps-j8717568", microsoft.metrics_include_label="service", namespace="ocm-arohcpci01-2tnvtu6q95ufl5426ec9nsjbnsqfa0aa-a8y2r7f6q8h7z0t", pod="etcd-0", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", reason="CrashLoopBackOff", region="centralus", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="aad698ce-6b47-41d1-8ddd-e6e4cb15b5d7" Description: Pod ocm-arohcpci01-2tnvtu6q95ufl5426ec9nsjbnsqfa0aa-a8y2r7f6q8h7z0t/etcd-0 (etcd) is in waiting state (reason: "CrashLoopBackOff"). Firing 2: State: Resolved Started: 2026-10-28T19:50:44Z Ended: 2026-10-28T20:19:45Z Severity: Sev3 Labels: alertname="KubePodCrashLooping", cluster="ci01-j8717568-mgmt-2", component="kubernetes-infrastructure", container="etcd", endpoint="http", environment="ci01", instance="10.128.64.254:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j8717568/providers/microsoft.monitor/accounts/hcps-j8717568", microsoft.metrics_include_label="service", namespace="ocm-arohcpci01-2tnvtu6q95ufl5426ec9nsjbnsqfa0aa-a8y2r7f6q8h7z0t", pod="etcd-0", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", reason="CrashLoopBackOff", region="centralus", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="aad698ce-6b47-41d1-8ddd-e6e4cb15b5d7" Description: Pod ocm-arohcpci01-2tnvtu6q95ufl5426ec9nsjbnsqfa0aa-a8y2r7f6q8h7z0t/etcd-0 (etcd) is in waiting state (reason: "CrashLoopBackOff"). alert fired 2 time(s) Firing 1: State: Resolved Started: 2026-10-28T12:59:31Z Ended: 2026-10-28T13:28:30Z Severity: Sev3 Labels: alertname="KubePodCrashLooping", cluster="ci01-j6945280-mgmt-1", component="kubernetes-infrastructure", container="etcd", endpoint="http", environment="ci01", instance="10.128.64.108:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j6945280/providers/microsoft.monitor/accounts/hcps-j6945280", microsoft.metrics_include_label="service", namespace="ocm-arohcpci01-2tnptam9khdl3svuvm4seuq5gr0ieit6-v5j4g5d3s9p2z6k", pod="etcd-0", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", reason="CrashLoopBackOff", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="5496ebfd-2ab4-43b8-9f9a-e0483581f4bd" Description: Pod ocm-arohcpci01-2tnptam9khdl3svuvm4seuq5gr0ieit6-v5j4g5d3s9p2z6k/etcd-0 (etcd) is in waiting state (reason: "CrashLoopBackOff"). Firing 2: State: Resolved Started: 2026-10-28T13:00:33Z Ended: 2026-10-28T13:27:30Z Severity: Sev3 Labels: alertname="KubePodCrashLooping", cluster="ci01-j6945280-mgmt-1", component="kubernetes-infrastructure", container="etcd", endpoint="http", environment="ci01", instance="10.128.64.108:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j6945280/providers/microsoft.monitor/accounts/hcps-j6945280", microsoft.metrics_include_label="service", namespace="ocm-arohcpci01-2tnptam9khdl3svuvm4seuq5gr0ieit6-v5j4g5d3s9p2z6k", pod="etcd-0", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", reason="CrashLoopBackOff", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="5496ebfd-2ab4-43b8-9f9a-e0483581f4bd" Description: Pod ocm-arohcpci01-2tnptam9khdl3svuvm4seuq5gr0ieit6-v5j4g5d3s9p2z6k/etcd-0 (etcd) is in waiting state (reason: "CrashLoopBackOff"). Contributing tests (1)
| ||||||||||||||||||||||||||||||||||||||||||||||||||
alert [hcp] KubeDeploymentReplicasMismatch firedfull failure pattern: alert [hcp] KubeDeploymentReplicasMismatch fired Signal: FlakeSignal: Flake — present in 4 prior week(s); spread across 4 days | alert | 8 | 3.40%8 of 235 job runs affected | FlakeSignal: Flake — present in 4 prior week(s); spread across 4 days | Show trend detailsOct 25: 1 · Oct 26: 2 · Oct 27: 2 · Oct 28: 4 · Oct 29: 0 · Oct 30: 0 · Oct 31: 0 · Nov 1: 0 | none | ||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (3)alert fired 2 time(s) Firing 1: State: Resolved Started: 2026-10-28T20:52:20Z Ended: 2026-10-28T21:03:23Z Severity: Sev3 Labels: alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j1157504-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="capi-provider", endpoint="http", environment="ci01", instance="10.128.64.172:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j1157504/providers/microsoft.monitor/accounts/hcps-j1157504", microsoft.metrics_include_label="service", namespace="ocm-arohcpci01-2to0k0skbecjdmuid7733ieqr7h7trvg-private-kas", pod="arohcp-monitor-kube-state-metrics-85d48f46ff-p48cd", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning" Description: Deployment ocm-arohcpci01-2to0k0skbecjdmuid7733ieqr7h7trvg-private-kas/capi-provider has not matched the expected number of replicas for longer than 30 minutes. Firing 2: State: Resolved Started: 2026-10-28T20:52:20Z Ended: 2026-10-28T21:03:23Z Severity: Sev3 Labels: alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j1157504-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="capi-provider", endpoint="http", environment="ci01", instance="10.128.64.172:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j1157504/providers/microsoft.monitor/accounts/hcps-j1157504", microsoft.metrics_include_label="service", namespace="ocm-arohcpci01-2to0k0skbecjdmuid7733ieqr7h7trvg-private-kas", pod="arohcp-monitor-kube-state-metrics-85d48f46ff-p48cd", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning" Description: Deployment ocm-arohcpci01-2to0k0skbecjdmuid7733ieqr7h7trvg-private-kas/capi-provider has not matched the expected number of replicas for longer than 30 minutes. alert fired 2 time(s) Firing 1: State: Resolved Started: 2026-10-28T20:03:50Z Ended: 2026-10-28T20:15:46Z Severity: Sev3 Labels: alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j8717568-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="capi-provider", endpoint="http", environment="ci01", instance="10.128.64.254:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j8717568/providers/microsoft.monitor/accounts/hcps-j8717568", microsoft.metrics_include_label="service", namespace="ocm-arohcpci01-2tnvtu6q95ufl5426ec9nsjbnsqfa0aa-a8y2r7f6q8h7z0t", pod="arohcp-monitor-kube-state-metrics-85d48f46ff-59dhx", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="centralus", service="arohcp-monitor-kube-state-metrics", severity="warning" Description: Deployment ocm-arohcpci01-2tnvtu6q95ufl5426ec9nsjbnsqfa0aa-a8y2r7f6q8h7z0t/capi-provider has not matched the expected number of replicas for longer than 30 minutes. Firing 2: State: Resolved Started: 2026-10-28T20:03:50Z Ended: 2026-10-28T20:15:46Z Severity: Sev3 Labels: alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j8717568-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="capi-provider", endpoint="http", environment="ci01", instance="10.128.64.254:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j8717568/providers/microsoft.monitor/accounts/hcps-j8717568", microsoft.metrics_include_label="service", namespace="ocm-arohcpci01-2tnvtu6q95ufl5426ec9nsjbnsqfa0aa-a8y2r7f6q8h7z0t", pod="arohcp-monitor-kube-state-metrics-85d48f46ff-59dhx", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="centralus", service="arohcp-monitor-kube-state-metrics", severity="warning" Description: Deployment ocm-arohcpci01-2tnvtu6q95ufl5426ec9nsjbnsqfa0aa-a8y2r7f6q8h7z0t/capi-provider has not matched the expected number of replicas for longer than 30 minutes. alert fired 1 time(s) Firing 1: State: Resolved Started: 2026-10-28T13:12:32Z Ended: 2026-10-28T13:28:32Z Severity: Sev3 Labels: alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j6945280-mgmt-1", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="capi-provider", endpoint="http", environment="ci01", instance="10.128.64.108:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j6945280/providers/microsoft.monitor/accounts/hcps-j6945280", microsoft.metrics_include_label="service", namespace="ocm-arohcpci01-2tnptam9khdl3svuvm4seuq5gr0ieit6-v5j4g5d3s9p2z6k", pod="arohcp-monitor-kube-state-metrics-85d48f46ff-fct9q", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning" Description: Deployment ocm-arohcpci01-2tnptam9khdl3svuvm4seuq5gr0ieit6-v5j4g5d3s9p2z6k/capi-provider has not matched the expected number of replicas for longer than 30 minutes. Contributing tests (1)
| ||||||||||||||||||||||||||||||||||||||||||||||||||
alert [hcp] KubeStatefulSetReplicasMismatch firedfull failure pattern: alert [hcp] KubeStatefulSetReplicasMismatch fired Signal: FlakeSignal: Flake — present in 4 prior week(s); spread across 4 days | alert | 8 | 3.40%8 of 235 job runs affected | FlakeSignal: Flake — present in 4 prior week(s); spread across 4 days | Show trend detailsOct 25: 1 · Oct 26: 2 · Oct 27: 2 · Oct 28: 4 · Oct 29: 0 · Oct 30: 0 · Oct 31: 0 · Nov 1: 0 | none | ||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (3)alert fired 2 time(s) Firing 1: State: Resolved Started: 2026-10-28T20:37:24Z Ended: 2026-10-28T21:04:22Z Severity: Sev3 Labels: alertname="KubeStatefulSetReplicasMismatch", cluster="ci01-j1157504-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", endpoint="http", environment="ci01", instance="10.128.64.172:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j1157504/providers/microsoft.monitor/accounts/hcps-j1157504", microsoft.metrics_include_label="service", namespace="ocm-arohcpci01-2to0k0skbecjdmuid7733ieqr7h7trvg-private-kas", pod="arohcp-monitor-kube-state-metrics-85d48f46ff-p48cd", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", statefulset="etcd" Description: StatefulSet ocm-arohcpci01-2to0k0skbecjdmuid7733ieqr7h7trvg-private-kas/etcd has not matched the expected number of replicas for longer than 15 minutes. Firing 2: State: Resolved Started: 2026-10-28T20:37:24Z Ended: 2026-10-28T21:01:24Z Severity: Sev3 Labels: alertname="KubeStatefulSetReplicasMismatch", cluster="ci01-j1157504-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", endpoint="http", environment="ci01", instance="10.128.64.172:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j1157504/providers/microsoft.monitor/accounts/hcps-j1157504", microsoft.metrics_include_label="service", namespace="ocm-arohcpci01-2to0k0skbecjdmuid7733ieqr7h7trvg-private-kas", pod="arohcp-monitor-kube-state-metrics-85d48f46ff-p48cd", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", statefulset="etcd" Description: StatefulSet ocm-arohcpci01-2to0k0skbecjdmuid7733ieqr7h7trvg-private-kas/etcd has not matched the expected number of replicas for longer than 15 minutes. alert fired 2 time(s) Firing 1: State: Resolved Started: 2026-10-28T19:48:54Z Ended: 2026-10-28T20:14:47Z Severity: Sev3 Labels: alertname="KubeStatefulSetReplicasMismatch", cluster="ci01-j8717568-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", endpoint="http", environment="ci01", instance="10.128.64.254:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j8717568/providers/microsoft.monitor/accounts/hcps-j8717568", microsoft.metrics_include_label="service", namespace="ocm-arohcpci01-2tnvtu6q95ufl5426ec9nsjbnsqfa0aa-a8y2r7f6q8h7z0t", pod="arohcp-monitor-kube-state-metrics-85d48f46ff-59dhx", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="centralus", service="arohcp-monitor-kube-state-metrics", severity="warning", statefulset="etcd" Description: StatefulSet ocm-arohcpci01-2tnvtu6q95ufl5426ec9nsjbnsqfa0aa-a8y2r7f6q8h7z0t/etcd has not matched the expected number of replicas for longer than 15 minutes. Firing 2: State: Resolved Started: 2026-10-28T19:48:54Z Ended: 2026-10-28T20:14:47Z Severity: Sev3 Labels: alertname="KubeStatefulSetReplicasMismatch", cluster="ci01-j8717568-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", endpoint="http", environment="ci01", instance="10.128.64.254:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j8717568/providers/microsoft.monitor/accounts/hcps-j8717568", microsoft.metrics_include_label="service", namespace="ocm-arohcpci01-2tnvtu6q95ufl5426ec9nsjbnsqfa0aa-a8y2r7f6q8h7z0t", pod="arohcp-monitor-kube-state-metrics-85d48f46ff-59dhx", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="centralus", service="arohcp-monitor-kube-state-metrics", severity="warning", statefulset="etcd" Description: StatefulSet ocm-arohcpci01-2tnvtu6q95ufl5426ec9nsjbnsqfa0aa-a8y2r7f6q8h7z0t/etcd has not matched the expected number of replicas for longer than 15 minutes. alert fired 2 time(s) Firing 1: State: Resolved Started: 2026-10-28T12:57:34Z Ended: 2026-10-28T13:23:33Z Severity: Sev3 Labels: alertname="KubeStatefulSetReplicasMismatch", cluster="ci01-j6945280-mgmt-1", component="kubernetes-infrastructure", container="kube-state-metrics", endpoint="http", environment="ci01", instance="10.128.64.108:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j6945280/providers/microsoft.monitor/accounts/hcps-j6945280", microsoft.metrics_include_label="service", namespace="ocm-arohcpci01-2tnptam9khdl3svuvm4seuq5gr0ieit6-v5j4g5d3s9p2z6k", pod="arohcp-monitor-kube-state-metrics-85d48f46ff-fct9q", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning", statefulset="etcd" Description: StatefulSet ocm-arohcpci01-2tnptam9khdl3svuvm4seuq5gr0ieit6-v5j4g5d3s9p2z6k/etcd has not matched the expected number of replicas for longer than 15 minutes. Firing 2: State: Resolved Started: 2026-10-28T12:57:34Z Ended: 2026-10-28T13:23:33Z Severity: Sev3 Labels: alertname="KubeStatefulSetReplicasMismatch", cluster="ci01-j6945280-mgmt-1", component="kubernetes-infrastructure", container="kube-state-metrics", endpoint="http", environment="ci01", instance="10.128.64.108:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j6945280/providers/microsoft.monitor/accounts/hcps-j6945280", microsoft.metrics_include_label="service", namespace="ocm-arohcpci01-2tnptam9khdl3svuvm4seuq5gr0ieit6-v5j4g5d3s9p2z6k", pod="arohcp-monitor-kube-state-metrics-85d48f46ff-fct9q", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning", statefulset="etcd" Description: StatefulSet ocm-arohcpci01-2tnptam9khdl3svuvm4seuq5gr0ieit6-v5j4g5d3s9p2z6k/etcd has not matched the expected number of replicas for longer than 15 minutes. Contributing tests (1)
| ||||||||||||||||||||||||||||||||||||||||||||||||||
failed to create nodepool <nodepool>: failed waiting for nodepool="<nodepool>" for cluster "<cluster>" in resourcegroup="<resource-group>" to finish creating: GET <url>full failure pattern: failed to create nodepool <nodepool>: failed waiting for nodepool="<nodepool>" for cluster "<cluster>" in resourcegroup="<resource-group>" to finish creating: GET <url> Signal: FlakeSignal: Flake — present in 4 prior week(s); spread across 3 days | e2e | 5 | 2.13%5 of 235 job runs affected | FlakeSignal: Flake — present in 4 prior week(s); spread across 3 days | Show trend detailsOct 25: 0 · Oct 26: 4 · Oct 27: 3 · Oct 28: 1 · Oct 29: 0 · Oct 30: 0 · Oct 31: 0 · Nov 1: 0 | STG | ||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (3)fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_delete.go:128]: failed to create node pools
Expected success, but got an error:
<*fmt.wrapError | 0x2db4fc89320>:
failed to create NodePool np-delete-main: failed waiting for nodepool="np-delete-main" for cluster "np-delete-hcp-cluster" in resourcegroup="nodepool-delete-zbnm2gmmrc7k" to finish creating: GET https://rp.j8443648.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/13b873cd-2efa-4559-ab15-ff37ab2d031b
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/13b873cd-2efa-4559-ab15-ff37ab2d031b",
"name": "13b873cd-2efa-4559-ab15-ff37ab2d031b",
"status": "Failed",
"startTime": "2026-10-28T16:52:57.688694522Z",
"endTime": "2026-10-28T17:12:06.752450543Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine j8c0h2z2t3o8e9b-np-delete-main-69xnv-msqbr: NodeNotHealthy: * Node.Ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine j8c0h2z2t3o8e9b-np-delete-main-69xnv-msqbr: NotReady: * NodeHealthy:\n * Node.Ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?\n"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://rp.j8443648.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/13b873cd-2efa-4559-ab15-ff37ab2d031b
correlation ID: 596c64f1-beb7-4366-a3b9-24ac7fe089dc
client request ID: 7b2dfc27-0f04-4e6d-a8a1-787b24c6e549
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_delete.go:128]: failed to create node pools
Expected success, but got an error:
<*fmt.wrapError | 0x2db4fc89320>:
failed to create NodePool np-delete-main: failed waiting for nodepool="np-delete-main" for cluster "np-delete-hcp-cluster" in resourcegroup="nodepool-delete-zbnm2gmmrc7k" to finish creating: GET https://rp.j8443648.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/13b873cd-2efa-4559-ab15-ff37ab2d031b
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/13b873cd-2efa-4559-ab15-ff37ab2d031b",
"name": "13b873cd-2efa-4559-ab15-ff37ab2d031b",
"status": "Failed",
"startTime": "2026-10-28T16:52:57.688694522Z",
"endTime": "2026-10-28T17:12:06.752450543Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine j8c0h2z2t3o8e9b-np-delete-main-69xnv-msqbr: NodeNotHealthy: * Node.Ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine j8c0h2z2t3o8e9b-np-delete-main-69xnv-msqbr: NotReady: * NodeHealthy:\n * Node.Ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?\n"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://rp.j8443648.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/13b873cd-2efa-4559-ab15-ff37ab2d031b
correlation ID: 596c64f1-beb7-4366-a3b9-24ac7fe089dc
client request ID: 7b2dfc27-0f04-4e6d-a8a1-787b24c6e549fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:166]: failed to create node pool
Expected success, but got an error:
<*fmt.wrapError | 0xb71d4fca080>:
failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "ingress-cert-le" in resourcegroup="ingress-cert-le-bqbx2znvzb54" to finish creating: GET https://rp.j1933056.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/0ed006bd-b8aa-4770-9062-3fd090d4fea9
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/0ed006bd-b8aa-4770-9062-3fd090d4fea9",
"name": "0ed006bd-b8aa-4770-9062-3fd090d4fea9",
"status": "Failed",
"startTime": "2026-10-27T23:25:38.076776565Z",
"endTime": "2026-10-27T23:44:40.197052822Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine ingress-cert-le-np-1-xjlrw-mjpj4: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/ingress-cert-le-bqbx2znvzb54--managed/providers/Microsoft.Compute/virtualMachines/ingress-cert-le-np-1-xjlrw-mjpj4 to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine ingress-cert-le-np-1-xjlrw-7jnb9: Failed: interfaces failed to create or update. err: failed to get existing resource ingress-cert-le-bqbx2znvzb54--managed/ingress-cert-le-np-1-xjlrw-7jnb9-nic (service: interfaces): GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/ingress-cert-le-bqbx2znvzb54--managed/providers/Microsoft.Network/networkInterfaces/ingress-cert-le-np-1-xjlrw-7jnb9-nic\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"SubscriptionRequestsThrottled\",\n \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor '8d1ff6d0-aadd-4331-9171-9443ac5f4337' exceeded. Please try again after '10' seconds after additional tokens are available. Refer ... message truncated\n"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://rp.j1933056.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/0ed006bd-b8aa-4770-9062-3fd090d4fea9
correlation ID: d0096963-5f3a-4ca8-942f-d516adbeb2fc
client request ID: 336f75bd-67db-4bb4-b035-81a9154977bb
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:166]: failed to create node pool
Expected success, but got an error:
<*fmt.wrapError | 0xb71d4fca080>:
failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "ingress-cert-le" in resourcegroup="ingress-cert-le-bqbx2znvzb54" to finish creating: GET https://rp.j1933056.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/0ed006bd-b8aa-4770-9062-3fd090d4fea9
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/0ed006bd-b8aa-4770-9062-3fd090d4fea9",
"name": "0ed006bd-b8aa-4770-9062-3fd090d4fea9",
"status": "Failed",
"startTime": "2026-10-27T23:25:38.076776565Z",
"endTime": "2026-10-27T23:44:40.197052822Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine ingress-cert-le-np-1-xjlrw-mjpj4: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/ingress-cert-le-bqbx2znvzb54--managed/providers/Microsoft.Compute/virtualMachines/ingress-cert-le-np-1-xjlrw-mjpj4 to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine ingress-cert-le-np-1-xjlrw-7jnb9: Failed: interfaces failed to create or update. err: failed to get existing resource ingress-cert-le-bqbx2znvzb54--managed/ingress-cert-le-np-1-xjlrw-7jnb9-nic (service: interfaces): GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/ingress-cert-le-bqbx2znvzb54--managed/providers/Microsoft.Network/networkInterfaces/ingress-cert-le-np-1-xjlrw-7jnb9-nic\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"SubscriptionRequestsThrottled\",\n \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor '8d1ff6d0-aadd-4331-9171-9443ac5f4337' exceeded. Please try again after '10' seconds after additional tokens are available. Refer ... message truncated\n"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://rp.j1933056.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/0ed006bd-b8aa-4770-9062-3fd090d4fea9
correlation ID: d0096963-5f3a-4ca8-942f-d516adbeb2fc
client request ID: 336f75bd-67db-4bb4-b035-81a9154977bbfail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:166]: failed to create node pool
Expected success, but got an error:
<*fmt.wrapError | 0x3591356d60>:
failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "ingress-cert-le" in resourcegroup="ingress-cert-le-6b928vvvc9lv" to finish creating: GET https://rp.j1933056.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/e68fe219-929c-499f-9d94-52a89aa54331
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/e68fe219-929c-499f-9d94-52a89aa54331",
"name": "e68fe219-929c-499f-9d94-52a89aa54331",
"status": "Failed",
"startTime": "2026-10-27T23:27:39.482004964Z",
"endTime": "2026-10-27T23:46:40.23395218Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine ingress-cert-le-np-1-qtp5t-vlfjh: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/ingress-cert-le-6b928vvvc9lv--managed/providers/Microsoft.Compute/virtualMachines/ingress-cert-le-np-1-qtp5t-vlfjh to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine ingress-cert-le-np-1-qtp5t-vlfjh: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/ingress-cert-le-6b928vvvc9lv--managed/providers/Microsoft.Compute/virtualMachines/ingress-cert-le-np-1-qtp5t-vlfjh to exist\n"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://rp.j1933056.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/e68fe219-929c-499f-9d94-52a89aa54331
correlation ID: 0fae4e0c-a402-45c6-8599-10c6e41f505d
client request ID: c1da63b5-9f53-4e68-8989-fc0f3ef1f3cb
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:166]: failed to create node pool
Expected success, but got an error:
<*fmt.wrapError | 0x3591356d60>:
failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "ingress-cert-le" in resourcegroup="ingress-cert-le-6b928vvvc9lv" to finish creating: GET https://rp.j1933056.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/e68fe219-929c-499f-9d94-52a89aa54331
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/e68fe219-929c-499f-9d94-52a89aa54331",
"name": "e68fe219-929c-499f-9d94-52a89aa54331",
"status": "Failed",
"startTime": "2026-10-27T23:27:39.482004964Z",
"endTime": "2026-10-27T23:46:40.23395218Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine ingress-cert-le-np-1-qtp5t-vlfjh: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/ingress-cert-le-6b928vvvc9lv--managed/providers/Microsoft.Compute/virtualMachines/ingress-cert-le-np-1-qtp5t-vlfjh to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine ingress-cert-le-np-1-qtp5t-vlfjh: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/ingress-cert-le-6b928vvvc9lv--managed/providers/Microsoft.Compute/virtualMachines/ingress-cert-le-np-1-qtp5t-vlfjh to exist\n"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://rp.j1933056.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/e68fe219-929c-499f-9d94-52a89aa54331
correlation ID: 0fae4e0c-a402-45c6-8599-10c6e41f505d
client request ID: c1da63b5-9f53-4e68-8989-fc0f3ef1f3cbContributing tests (3)
Affected runs (5) | ||||||||||||||||||||||||||||||||||||||||||||||||||
ERROR CODE: DeadlineExceeded; detail message node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hype...full failure pattern: ERROR CODE: DeadlineExceeded; detail message node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift node pool not cached yet; provider Microsoft.RedHatOpenShift Signal: FlakeSignal: Flake — present in 3 prior week(s); spread across 3 days | e2e | 3 | 1.28%3 of 235 job runs affected | FlakeSignal: Flake — present in 3 prior week(s); spread across 3 days | Show trend detailsOct 25: 1 · Oct 26: 2 · Oct 27: 1 · Oct 28: 0 · Oct 29: 0 · Oct 30: 0 · Oct 31: 0 · Nov 1: 0 | PROD | ||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (3)fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kas.go:159]: failed to create node pool "np-1" for private KAS cluster "private-kas"
Unexpected error:
<*fmt.wrapError | 0x19f0aa4bea20>:
failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "private-kas" in resourcegroup="private-kas-wj8x9hp7bqx5" to finish creating: GET https://rp.j8023424.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/6a73d0d2-8fd5-4af1-adbb-f385ed243d20
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/6a73d0d2-8fd5-4af1-adbb-f385ed243d20",
"name": "6a73d0d2-8fd5-4af1-adbb-f385ed243d20",
"status": "Failed",
"startTime": "2026-10-27T08:09:38.679552409Z",
"endTime": "2026-10-27T08:28:40.674204457Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift node pool not cached yet"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://rp.j8023424.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/6a73d0d2-8fd5-4af1-adbb-f385ed243d20
correlation ID: 51afa422-8d59-4334-91a3-bc90ade1f885
client request ID: ebb6ee5a-68e9-481e-8fab-49a3ebe79568
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kas.go:159]: failed to create node pool "np-1" for private KAS cluster "private-kas"
Unexpected error:
<*fmt.wrapError | 0x19f0aa4bea20>:
failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "private-kas" in resourcegroup="private-kas-wj8x9hp7bqx5" to finish creating: GET https://rp.j8023424.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/6a73d0d2-8fd5-4af1-adbb-f385ed243d20
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/6a73d0d2-8fd5-4af1-adbb-f385ed243d20",
"name": "6a73d0d2-8fd5-4af1-adbb-f385ed243d20",
"status": "Failed",
"startTime": "2026-10-27T08:09:38.679552409Z",
"endTime": "2026-10-27T08:28:40.674204457Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift node pool not cached yet"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://rp.j8023424.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/6a73d0d2-8fd5-4af1-adbb-f385ed243d20
correlation ID: 51afa422-8d59-4334-91a3-bc90ade1f885
client request ID: ebb6ee5a-68e9-481e-8fab-49a3ebe79568
occurredfail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:404]: failed to verify cilium is running and nodes are Ready for cluster "agg-cluster"
Unexpected error:
<*errors.joinError | 0x2d970cc94090>:
VerifyNodesReady failed: no nodes found in the cluster
VerifyCiliumOperational failed: not all pods in kube-system namespace are running: failed to list pods in kube-system namespace: context deadline exceeded
failed to create NodePool agg-np-a: failed waiting for nodepool="agg-np-a" for cluster "agg-cluster" in resourcegroup="feature-aggregation-rx9rnfvmj655" to finish creating: GET https://rp.j9849088.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/c8761660-38f6-4612-a201-a6ddec925f84
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/c8761660-38f6-4612-a201-a6ddec925f84",
"name": "c8761660-38f6-4612-a201-a6ddec925f84",
"status": "Failed",
"startTime": "2026-10-26T09:38:13.907922394Z",
"endTime": "2026-10-26T09:57:20.458727782Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift node pool not cached yet"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://rp.j9849088.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/c8761660-38f6-4612-a201-a6ddec925f84
correlation ID: 4c993536-8323-4301-a24e-f6918a189628
client request ID: e13d57f0-4312-4271-8e8e-02240879d813
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:404]: failed to verify cilium is running and nodes are Ready for cluster "agg-cluster"
Unexpected error:
<*errors.joinError | 0x2d970cc94090>:
VerifyNodesReady failed: no nodes found in the cluster
VerifyCiliumOperational failed: not all pods in kube-system namespace are running: failed to list pods in kube-system namespace: context deadline exceeded
failed to create NodePool agg-np-a: failed waiting for nodepool="agg-np-a" for cluster "agg-cluster" in resourcegroup="feature-aggregation-rx9rnfvmj655" to finish creating: GET https://rp.j9849088.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/c8761660-38f6-4612-a201-a6ddec925f84
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/c8761660-38f6-4612-a201-a6ddec925f84",
"name": "c8761660-38f6-4612-a201-a6ddec925f84",
"status": "Failed",
"startTime": "2026-10-26T09:38:13.907922394Z",
"endTime": "2026-10-26T09:57:20.458727782Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift node pool not cached yet"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://rp.j9849088.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/c8761660-38f6-4612-a201-a6ddec925f84
correlation ID: 4c993536-8323-4301-a24e-f6918a189628
client request ID: e13d57f0-4312-4271-8e8e-02240879d813
occurredfail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:154]: failed to create AZ nodepool autoscale-az with autoscaling
Unexpected error:
<*fmt.wrapError | 0x13f2532896c0>:
failed to create NodePool autoscale-az: failed waiting for nodepool="autoscale-az" for cluster "np-autoscale-cluster" in resourcegroup="np-autoscaling-n22957ncbgjt" to finish creating: GET https://rp.j9769984.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/ecac4fcd-9bd7-4a22-9510-51d365a0dee8
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/ecac4fcd-9bd7-4a22-9510-51d365a0dee8",
"name": "ecac4fcd-9bd7-4a22-9510-51d365a0dee8",
"status": "Failed",
"startTime": "2026-10-26T03:58:11.830961438Z",
"endTime": "2026-10-26T04:17:16.025648143Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift node pool not cached yet"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://rp.j9769984.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/ecac4fcd-9bd7-4a22-9510-51d365a0dee8
correlation ID: 5b299ccf-a19f-4517-91e6-0d46d8d87839
client request ID: 050413c1-cc10-4b65-a905-bd6edbbaa97f
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:154]: failed to create AZ nodepool autoscale-az with autoscaling
Unexpected error:
<*fmt.wrapError | 0x13f2532896c0>:
failed to create NodePool autoscale-az: failed waiting for nodepool="autoscale-az" for cluster "np-autoscale-cluster" in resourcegroup="np-autoscaling-n22957ncbgjt" to finish creating: GET https://rp.j9769984.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/ecac4fcd-9bd7-4a22-9510-51d365a0dee8
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/ecac4fcd-9bd7-4a22-9510-51d365a0dee8",
"name": "ecac4fcd-9bd7-4a22-9510-51d365a0dee8",
"status": "Failed",
"startTime": "2026-10-26T03:58:11.830961438Z",
"endTime": "2026-10-26T04:17:16.025648143Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift node pool not cached yet"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://rp.j9769984.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/ecac4fcd-9bd7-4a22-9510-51d365a0dee8
correlation ID: 5b299ccf-a19f-4517-91e6-0d46d8d87839
client request ID: 050413c1-cc10-4b65-a905-bd6edbbaa97f
occurredContributing tests (3)
Affected runs (3)
| ||||||||||||||||||||||||||||||||||||||||||||||||||
ERROR CODE: DeadlineExceeded; detail message node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hype...full failure pattern: ERROR CODE: DeadlineExceeded; detail message node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is <count>, want >= <count> (autoscaling min); node pool condition AllNodesHealthy is False: <count> of <count> machines are not healthy Machine <node>: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.Compute/virtualMachines/<node> to exist; node pool condition AllMachinesReady is False: <count> of <count> machines are not ready Machine <node>: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.Compute/virtualMachines/<node> to exist; provider Microsoft.RedHatOpenShift Signal: FlakeSignal: Flake — present in 2 prior week(s); spread across 3 days | e2e | 3 | 1.28%3 of 235 job runs affected | FlakeSignal: Flake — present in 2 prior week(s); spread across 3 days | Show trend detailsOct 25: 0 · Oct 26: 1 · Oct 27: 1 · Oct 28: 1 · Oct 29: 0 · Oct 30: 0 · Oct 31: 0 · Nov 1: 0 | none | ||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (3)fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:220]: failed to create no-AZ nodepool autoscale-noaz with autoscaling
Unexpected error:
<*fmt.wrapError | 0x1c827a968340>:
failed to create NodePool autoscale-noaz: failed waiting for nodepool="autoscale-noaz" for cluster "np-autoscale-cluster" in resourcegroup="np-autoscaling-9k6kks4sqmhh" to finish creating: GET https://rp.j5392000.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/e7e6b86c-a0ff-4c35-94e7-3aa8b285e19e
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/e7e6b86c-a0ff-4c35-94e7-3aa8b285e19e",
"name": "e7e6b86c-a0ff-4c35-94e7-3aa8b285e19e",
"status": "Failed",
"startTime": "2026-10-28T17:17:13.271950373Z",
"endTime": "2026-10-28T17:36:19.733049446Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 2, want \u003e= 3 (autoscaling min); node pool condition AllNodesHealthy is False: 1 of 3 machines are not healthy\nMachine x5b5y8h0f8t5k2t-autoscale-noaz-b47sx-574cm: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/np-autoscaling-9k6kks4sqmhh--managed/providers/Microsoft.Compute/virtualMachines/x5b5y8h0f8t5k2t-autoscale-noaz-b47sx-574cm to exist\n; node pool condition AllMachinesReady is False: 1 of 3 machines are not ready\nMachine x5b5y8h0f8t5k2t-autoscale-noaz-b47sx-574cm: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/np-autoscaling-9k6kks4sqmhh--managed/providers/Microsoft.Compute/virtualMachines/x5b5y8h0f8t5k2t-autoscale-noaz-b47sx-574cm to exist\n"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://rp.j5392000.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/e7e6b86c-a0ff-4c35-94e7-3aa8b285e19e
correlation ID: e5b7a290-af02-4ff7-9f0a-851a1b652878
client request ID: a046022e-1fb3-43ea-9318-d95c8087e3da
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:220]: failed to create no-AZ nodepool autoscale-noaz with autoscaling
Unexpected error:
<*fmt.wrapError | 0x1c827a968340>:
failed to create NodePool autoscale-noaz: failed waiting for nodepool="autoscale-noaz" for cluster "np-autoscale-cluster" in resourcegroup="np-autoscaling-9k6kks4sqmhh" to finish creating: GET https://rp.j5392000.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/e7e6b86c-a0ff-4c35-94e7-3aa8b285e19e
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/e7e6b86c-a0ff-4c35-94e7-3aa8b285e19e",
"name": "e7e6b86c-a0ff-4c35-94e7-3aa8b285e19e",
"status": "Failed",
"startTime": "2026-10-28T17:17:13.271950373Z",
"endTime": "2026-10-28T17:36:19.733049446Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 2, want \u003e= 3 (autoscaling min); node pool condition AllNodesHealthy is False: 1 of 3 machines are not healthy\nMachine x5b5y8h0f8t5k2t-autoscale-noaz-b47sx-574cm: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/np-autoscaling-9k6kks4sqmhh--managed/providers/Microsoft.Compute/virtualMachines/x5b5y8h0f8t5k2t-autoscale-noaz-b47sx-574cm to exist\n; node pool condition AllMachinesReady is False: 1 of 3 machines are not ready\nMachine x5b5y8h0f8t5k2t-autoscale-noaz-b47sx-574cm: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/np-autoscaling-9k6kks4sqmhh--managed/providers/Microsoft.Compute/virtualMachines/x5b5y8h0f8t5k2t-autoscale-noaz-b47sx-574cm to exist\n"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://rp.j5392000.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/e7e6b86c-a0ff-4c35-94e7-3aa8b285e19e
correlation ID: e5b7a290-af02-4ff7-9f0a-851a1b652878
client request ID: a046022e-1fb3-43ea-9318-d95c8087e3da
occurredfail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:220]: failed to create no-AZ nodepool autoscale-noaz with autoscaling
Unexpected error:
<*fmt.wrapError | 0x2c3a9a99f660>:
failed to create NodePool autoscale-noaz: failed waiting for nodepool="autoscale-noaz" for cluster "np-autoscale-cluster" in resourcegroup="np-autoscaling-hqzrkb5qdbkx" to finish creating: GET https://rp.j1666560.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/ff22fd81-847d-4bcb-b22f-4250d9071095
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/ff22fd81-847d-4bcb-b22f-4250d9071095",
"name": "ff22fd81-847d-4bcb-b22f-4250d9071095",
"status": "Failed",
"startTime": "2026-10-27T20:11:38.810985949Z",
"endTime": "2026-10-27T20:30:42.147810175Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 2, want \u003e= 3 (autoscaling min); node pool condition AllNodesHealthy is False: 1 of 3 machines are not healthy\nMachine m7z0h7l7n3n4n3s-autoscale-noaz-hfvwg-htn8f: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/np-autoscaling-hqzrkb5qdbkx--managed/providers/Microsoft.Compute/virtualMachines/m7z0h7l7n3n4n3s-autoscale-noaz-hfvwg-htn8f to exist\n; node pool condition AllMachinesReady is False: 1 of 3 machines are not ready\nMachine m7z0h7l7n3n4n3s-autoscale-noaz-hfvwg-htn8f: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/np-autoscaling-hqzrkb5qdbkx--managed/providers/Microsoft.Compute/virtualMachines/m7z0h7l7n3n4n3s-autoscale-noaz-hfvwg-htn8f to exist\n"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://rp.j1666560.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/ff22fd81-847d-4bcb-b22f-4250d9071095
correlation ID: 20b2a10f-dd2b-4a62-8cff-21fa59649da0
client request ID: a877eec2-04c5-44bd-9434-61383daeb4f5
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:220]: failed to create no-AZ nodepool autoscale-noaz with autoscaling
Unexpected error:
<*fmt.wrapError | 0x2c3a9a99f660>:
failed to create NodePool autoscale-noaz: failed waiting for nodepool="autoscale-noaz" for cluster "np-autoscale-cluster" in resourcegroup="np-autoscaling-hqzrkb5qdbkx" to finish creating: GET https://rp.j1666560.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/ff22fd81-847d-4bcb-b22f-4250d9071095
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/ff22fd81-847d-4bcb-b22f-4250d9071095",
"name": "ff22fd81-847d-4bcb-b22f-4250d9071095",
"status": "Failed",
"startTime": "2026-10-27T20:11:38.810985949Z",
"endTime": "2026-10-27T20:30:42.147810175Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 2, want \u003e= 3 (autoscaling min); node pool condition AllNodesHealthy is False: 1 of 3 machines are not healthy\nMachine m7z0h7l7n3n4n3s-autoscale-noaz-hfvwg-htn8f: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/np-autoscaling-hqzrkb5qdbkx--managed/providers/Microsoft.Compute/virtualMachines/m7z0h7l7n3n4n3s-autoscale-noaz-hfvwg-htn8f to exist\n; node pool condition AllMachinesReady is False: 1 of 3 machines are not ready\nMachine m7z0h7l7n3n4n3s-autoscale-noaz-hfvwg-htn8f: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/np-autoscaling-hqzrkb5qdbkx--managed/providers/Microsoft.Compute/virtualMachines/m7z0h7l7n3n4n3s-autoscale-noaz-hfvwg-htn8f to exist\n"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://rp.j1666560.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/ff22fd81-847d-4bcb-b22f-4250d9071095
correlation ID: 20b2a10f-dd2b-4a62-8cff-21fa59649da0
client request ID: a877eec2-04c5-44bd-9434-61383daeb4f5
occurredfail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:220]: failed to create no-AZ nodepool autoscale-noaz with autoscaling
Unexpected error:
<*fmt.wrapError | 0xa75b49b000>:
failed to create NodePool autoscale-noaz: failed waiting for nodepool="autoscale-noaz" for cluster "np-autoscale-cluster" in resourcegroup="np-autoscaling-c7rz5j8lqwzr" to finish creating: GET https://rp.j8788992.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/116d7ad9-9cc2-4c9e-8347-2623016e8d01
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/116d7ad9-9cc2-4c9e-8347-2623016e8d01",
"name": "116d7ad9-9cc2-4c9e-8347-2623016e8d01",
"status": "Failed",
"startTime": "2026-10-26T16:04:30.855869275Z",
"endTime": "2026-10-26T16:23:34.187561349Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 2, want \u003e= 3 (autoscaling min); node pool condition AllNodesHealthy is False: 1 of 3 machines are not healthy\nMachine o9a0o9o2f5k6s4q-autoscale-noaz-rb8z8-ffnt2: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/np-autoscaling-c7rz5j8lqwzr--managed/providers/Microsoft.Compute/virtualMachines/o9a0o9o2f5k6s4q-autoscale-noaz-rb8z8-ffnt2 to exist\n; node pool condition AllMachinesReady is False: 1 of 3 machines are not ready\nMachine o9a0o9o2f5k6s4q-autoscale-noaz-rb8z8-ffnt2: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/np-autoscaling-c7rz5j8lqwzr--managed/providers/Microsoft.Compute/virtualMachines/o9a0o9o2f5k6s4q-autoscale-noaz-rb8z8-ffnt2 to exist\n"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://rp.j8788992.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/116d7ad9-9cc2-4c9e-8347-2623016e8d01
correlation ID: 0413d2ef-0809-4a88-a400-52222573fc14
client request ID: 2d020f95-fbad-4853-8497-2a3822d5b948
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:220]: failed to create no-AZ nodepool autoscale-noaz with autoscaling
Unexpected error:
<*fmt.wrapError | 0xa75b49b000>:
failed to create NodePool autoscale-noaz: failed waiting for nodepool="autoscale-noaz" for cluster "np-autoscale-cluster" in resourcegroup="np-autoscaling-c7rz5j8lqwzr" to finish creating: GET https://rp.j8788992.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/116d7ad9-9cc2-4c9e-8347-2623016e8d01
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/116d7ad9-9cc2-4c9e-8347-2623016e8d01",
"name": "116d7ad9-9cc2-4c9e-8347-2623016e8d01",
"status": "Failed",
"startTime": "2026-10-26T16:04:30.855869275Z",
"endTime": "2026-10-26T16:23:34.187561349Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 2, want \u003e= 3 (autoscaling min); node pool condition AllNodesHealthy is False: 1 of 3 machines are not healthy\nMachine o9a0o9o2f5k6s4q-autoscale-noaz-rb8z8-ffnt2: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/np-autoscaling-c7rz5j8lqwzr--managed/providers/Microsoft.Compute/virtualMachines/o9a0o9o2f5k6s4q-autoscale-noaz-rb8z8-ffnt2 to exist\n; node pool condition AllMachinesReady is False: 1 of 3 machines are not ready\nMachine o9a0o9o2f5k6s4q-autoscale-noaz-rb8z8-ffnt2: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/np-autoscaling-c7rz5j8lqwzr--managed/providers/Microsoft.Compute/virtualMachines/o9a0o9o2f5k6s4q-autoscale-noaz-rb8z8-ffnt2 to exist\n"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://rp.j8788992.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/116d7ad9-9cc2-4c9e-8347-2623016e8d01
correlation ID: 0413d2ef-0809-4a88-a400-52222573fc14
client request ID: 2d020f95-fbad-4853-8497-2a3822d5b948
occurredContributing tests (1)
Affected runs (3)
| ||||||||||||||||||||||||||||||||||||||||||||||||||
at least one schedule should have a LastBackupTimefull failure pattern: at least one schedule should have a LastBackupTime Signal: IndeterminateSignal: Indeterminate — no prior history; active 2 day(s) — New failure pattern — no prior history | e2e | 23 | 9.79%23 of 235 job runs affected | IndeterminateSignal: Indeterminate — no prior history; active 2 day(s) — New failure pattern — no prior history | Show trend detailsOct 25: 0 · Oct 26: 1 · Oct 27: 22 · Oct 28: 0 · Oct 29: 0 · Oct 30: 0 · Oct 31: 0 · Nov 1: 0 | none | ||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (3)fail [github.com/Azure/ARO-HCP/test/e2e/hcp_backups.go:166]: Timed out after 660.004s.
at least one schedule should have a LastBackupTime
Expected
<bool>: false
to be true
fail [github.com/Azure/ARO-HCP/test/e2e/hcp_backups.go:166]: Timed out after 660.004s.
at least one schedule should have a LastBackupTime
Expected
<bool>: false
to be truefail [github.com/Azure/ARO-HCP/test/e2e/hcp_backups.go:166]: Timed out after 660.008s.
at least one schedule should have a LastBackupTime
Expected
<bool>: false
to be true
fail [github.com/Azure/ARO-HCP/test/e2e/hcp_backups.go:166]: Timed out after 660.008s.
at least one schedule should have a LastBackupTime
Expected
<bool>: false
to be truefail [github.com/Azure/ARO-HCP/test/e2e/hcp_backups.go:166]: Timed out after 660.005s.
at least one schedule should have a LastBackupTime
Expected
<bool>: false
to be true
fail [github.com/Azure/ARO-HCP/test/e2e/hcp_backups.go:166]: Timed out after 660.005s.
at least one schedule should have a LastBackupTime
Expected
<bool>: false
to be trueContributing tests (1)
Affected runs (23) | ||||||||||||||||||||||||||||||||||||||||||||||||||
alert [svc] KubePodCrashLooping firedfull failure pattern: alert [svc] KubePodCrashLooping fired Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 4 day(s) | alert | 10 | 4.26%10 of 235 job runs affected | IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 4 day(s) | Show trend detailsOct 25: 4 · Oct 26: 4 · Oct 27: 2 · Oct 28: 4 · Oct 29: 0 · Oct 30: 0 · Oct 31: 0 · Nov 1: 0 | none | ||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (3)alert fired 4 time(s) Firing 1: State: Fired (not resolved) Started: 2026-10-28T13:27:25Z Severity: Sev3 Labels: alertname="KubePodCrashLooping", cluster="ci01-j0229248-mgmt-1", component="kubernetes-infrastructure", container="hypershift-addon-agent", endpoint="http", environment="ci01", instance="10.128.64.125:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0229248/providers/microsoft.monitor/accounts/services-j0229248", microsoft.metrics_include_label="service", namespace="open-cluster-management-agent-addon", pod="hypershift-addon-agent-7fc5986cf8-phdkm", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", reason="CrashLoopBackOff", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="5471b0ac-6a29-48cb-ab86-2893cd772fbd" Description: Pod open-cluster-management-agent-addon/hypershift-addon-agent-7fc5986cf8-phdkm (hypershift-addon-agent) is in waiting state (reason: "CrashLoopBackOff"). Firing 2: State: Fired (not resolved) Started: 2026-10-28T13:29:30Z Severity: Sev3 Labels: alertname="KubePodCrashLooping", cluster="ci01-j0229248-mgmt-2", component="kubernetes-infrastructure", container="hypershift-addon-agent", endpoint="http", environment="ci01", instance="10.128.64.195:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0229248/providers/microsoft.monitor/accounts/services-j0229248", microsoft.metrics_include_label="service", namespace="open-cluster-management-agent-addon", pod="hypershift-addon-agent-7fc5986cf8-84g47", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", reason="CrashLoopBackOff", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="9462daaa-47d3-4b5e-b73b-01a9f857a137" Description: Pod open-cluster-management-agent-addon/hypershift-addon-agent-7fc5986cf8-84g47 (hypershift-addon-agent) is in waiting state (reason: "CrashLoopBackOff"). Firing 3: State: Fired (not resolved) Started: 2026-10-28T13:29:30Z Severity: Sev3 Labels: alertname="KubePodCrashLooping", cluster="ci01-j0229248-mgmt-2", component="kubernetes-infrastructure", container="hypershift-addon-agent", endpoint="http", environment="ci01", instance="10.128.64.195:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0229248/providers/microsoft.monitor/accounts/services-j0229248", microsoft.metrics_include_label="service", namespace="open-cluster-management-agent-addon", pod="hypershift-addon-agent-7fc5986cf8-84g47", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", reason="CrashLoopBackOff", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="9462daaa-47d3-4b5e-b73b-01a9f857a137" Description: Pod open-cluster-management-agent-addon/hypershift-addon-agent-7fc5986cf8-84g47 (hypershift-addon-agent) is in waiting state (reason: "CrashLoopBackOff"). Firing 4: State: Fired (not resolved) Started: 2026-10-28T13:32:24Z Severity: Sev3 Labels: alertname="KubePodCrashLooping", cluster="ci01-j0229248-mgmt-1", component="kubernetes-infrastructure", container="hypershift-addon-agent", endpoint="http", environment="ci01", instance="10.128.64.125:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0229248/providers/microsoft.monitor/accounts/services-j0229248", microsoft.metrics_include_label="service", namespace="open-cluster-management-agent-addon", pod="hypershift-addon-agent-7fc5986cf8-phdkm", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", reason="CrashLoopBackOff", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="5471b0ac-6a29-48cb-ab86-2893cd772fbd" Description: Pod open-cluster-management-agent-addon/hypershift-addon-agent-7fc5986cf8-phdkm (hypershift-addon-agent) is in waiting state (reason: "CrashLoopBackOff"). alert fired 4 time(s) Firing 1: State: Fired (not resolved) Started: 2026-10-28T10:42:37Z Severity: Sev3 Labels: alertname="KubePodCrashLooping", cluster="ci01-j1445504-mgmt-2", component="kubernetes-infrastructure", container="hypershift-addon-agent", endpoint="http", environment="ci01", instance="10.128.65.26:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j1445504/providers/microsoft.monitor/accounts/services-j1445504", microsoft.metrics_include_label="service", namespace="open-cluster-management-agent-addon", pod="hypershift-addon-agent-7fc5986cf8-5dmcb", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", reason="CrashLoopBackOff", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="6520e185-94fe-45c4-99a9-813a1698d907" Description: Pod open-cluster-management-agent-addon/hypershift-addon-agent-7fc5986cf8-5dmcb (hypershift-addon-agent) is in waiting state (reason: "CrashLoopBackOff"). Firing 2: State: Fired (not resolved) Started: 2026-10-28T10:43:33Z Severity: Sev3 Labels: alertname="KubePodCrashLooping", cluster="ci01-j1445504-mgmt-1", component="kubernetes-infrastructure", container="hypershift-addon-agent", endpoint="http", environment="ci01", instance="10.128.64.97:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j1445504/providers/microsoft.monitor/accounts/services-j1445504", microsoft.metrics_include_label="service", namespace="open-cluster-management-agent-addon", pod="hypershift-addon-agent-7fc5986cf8-ts4n7", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", reason="CrashLoopBackOff", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="0a28f799-e0bd-4698-b612-8c6756c4b517" Description: Pod open-cluster-management-agent-addon/hypershift-addon-agent-7fc5986cf8-ts4n7 (hypershift-addon-agent) is in waiting state (reason: "CrashLoopBackOff"). Firing 3: State: Fired (not resolved) Started: 2026-10-28T10:43:33Z Severity: Sev3 Labels: alertname="KubePodCrashLooping", cluster="ci01-j1445504-mgmt-1", component="kubernetes-infrastructure", container="hypershift-addon-agent", endpoint="http", environment="ci01", instance="10.128.64.97:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j1445504/providers/microsoft.monitor/accounts/services-j1445504", microsoft.metrics_include_label="service", namespace="open-cluster-management-agent-addon", pod="hypershift-addon-agent-7fc5986cf8-ts4n7", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", reason="CrashLoopBackOff", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="0a28f799-e0bd-4698-b612-8c6756c4b517" Description: Pod open-cluster-management-agent-addon/hypershift-addon-agent-7fc5986cf8-ts4n7 (hypershift-addon-agent) is in waiting state (reason: "CrashLoopBackOff"). Firing 4: State: Fired (not resolved) Started: 2026-10-28T10:47:32Z Severity: Sev3 Labels: alertname="KubePodCrashLooping", cluster="ci01-j1445504-mgmt-2", component="kubernetes-infrastructure", container="hypershift-addon-agent", endpoint="http", environment="ci01", instance="10.128.65.26:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j1445504/providers/microsoft.monitor/accounts/services-j1445504", microsoft.metrics_include_label="service", namespace="open-cluster-management-agent-addon", pod="hypershift-addon-agent-7fc5986cf8-5dmcb", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", reason="CrashLoopBackOff", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="6520e185-94fe-45c4-99a9-813a1698d907" Description: Pod open-cluster-management-agent-addon/hypershift-addon-agent-7fc5986cf8-5dmcb (hypershift-addon-agent) is in waiting state (reason: "CrashLoopBackOff"). alert fired 4 time(s) Firing 1: State: Fired (not resolved) Started: 2026-10-28T08:13:15Z Severity: Sev3 Labels: alertname="KubePodCrashLooping", cluster="ci01-j3004160-mgmt-1", component="kubernetes-infrastructure", container="hypershift-addon-agent", endpoint="http", environment="ci01", instance="10.128.64.217:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j3004160/providers/microsoft.monitor/accounts/services-j3004160", microsoft.metrics_include_label="service", namespace="open-cluster-management-agent-addon", pod="hypershift-addon-agent-7fc5986cf8-krfd7", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", reason="CrashLoopBackOff", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="2298911c-0832-493f-a9ef-f65728646e11" Description: Pod open-cluster-management-agent-addon/hypershift-addon-agent-7fc5986cf8-krfd7 (hypershift-addon-agent) is in waiting state (reason: "CrashLoopBackOff"). Firing 2: State: Fired (not resolved) Started: 2026-10-28T08:13:16Z Severity: Sev3 Labels: alertname="KubePodCrashLooping", cluster="ci01-j3004160-mgmt-1", component="kubernetes-infrastructure", container="hypershift-addon-agent", endpoint="http", environment="ci01", instance="10.128.64.217:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j3004160/providers/microsoft.monitor/accounts/services-j3004160", microsoft.metrics_include_label="service", namespace="open-cluster-management-agent-addon", pod="hypershift-addon-agent-7fc5986cf8-krfd7", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", reason="CrashLoopBackOff", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="2298911c-0832-493f-a9ef-f65728646e11" Description: Pod open-cluster-management-agent-addon/hypershift-addon-agent-7fc5986cf8-krfd7 (hypershift-addon-agent) is in waiting state (reason: "CrashLoopBackOff"). Firing 3: State: Fired (not resolved) Started: 2026-10-28T08:15:13Z Severity: Sev3 Labels: alertname="KubePodCrashLooping", cluster="ci01-j3004160-mgmt-2", component="kubernetes-infrastructure", container="hypershift-addon-agent", endpoint="http", environment="ci01", instance="10.128.64.208:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j3004160/providers/microsoft.monitor/accounts/services-j3004160", microsoft.metrics_include_label="service", namespace="open-cluster-management-agent-addon", pod="hypershift-addon-agent-7fc5986cf8-fgmvz", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", reason="CrashLoopBackOff", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="e461328c-3664-4b8e-907f-f8f42456ba46" Description: Pod open-cluster-management-agent-addon/hypershift-addon-agent-7fc5986cf8-fgmvz (hypershift-addon-agent) is in waiting state (reason: "CrashLoopBackOff"). Firing 4: State: Fired (not resolved) Started: 2026-10-28T08:15:13Z Severity: Sev3 Labels: alertname="KubePodCrashLooping", cluster="ci01-j3004160-mgmt-2", component="kubernetes-infrastructure", container="hypershift-addon-agent", endpoint="http", environment="ci01", instance="10.128.64.208:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j3004160/providers/microsoft.monitor/accounts/services-j3004160", microsoft.metrics_include_label="service", namespace="open-cluster-management-agent-addon", pod="hypershift-addon-agent-7fc5986cf8-fgmvz", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", reason="CrashLoopBackOff", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="e461328c-3664-4b8e-907f-f8f42456ba46" Description: Pod open-cluster-management-agent-addon/hypershift-addon-agent-7fc5986cf8-fgmvz (hypershift-addon-agent) is in waiting state (reason: "CrashLoopBackOff"). Contributing tests (1)
Affected runs (10) | ||||||||||||||||||||||||||||||||||||||||||||||||||
ERROR CODE: DeploymentFailed; detail code AzureKeyVaultKmsValidateIdentityPermissionCustomerError; detail message The identity does not have keys encrypt/decrypt permission on key...full failure pattern: ERROR CODE: DeploymentFailed; detail code AzureKeyVaultKmsValidateIdentityPermissionCustomerError; detail message The identity does not have keys encrypt/decrypt permission on key vault "<url>"; provider Microsoft.ContainerService Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 3 day(s) | provision | 8 | 3.40%8 of 235 job runs affected | IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 3 day(s) | Show trend detailsOct 25: 0 · Oct 26: 2 · Oct 27: 4 · Oct 28: 2 · Oct 29: 0 · Oct 30: 0 · Oct 31: 0 · Nov 1: 0 | none | ||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (3)time=2026-10-28T16:43:28.511Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster description="Step cluster\n Kind: ARM\n Template: templates/svc-cluster.bicep\n Parameters: configurations/svc-cluster.tmpl.bicepparam"
time=2026-10-28T16:43:31.273Z level=DEBUG+2 msg="Starting ARM deployment" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster
time=2026-10-28T16:43:32.448Z level=DEBUG+3 msg="Deployment started" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster deployment=f309630496c74d69129833a43742f2e9eec7ceb9d4411ea07b979bc73b89fd87 portal=https://ms.portal.azure.com/#view/Microsoft_Azure_Resources/DeploymentDetails.MenuView/~/overview/id/%2Fsubscriptions%2FXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX%2FresourceGroups%2Fhcp-underlay-ci01-j2862080-svc%2Fproviders%2FMicrosoft.Resources%2Fdeployments%2Ff309630496c74d69129833a43742f2e9eec7ceb9d4411ea07b979bc73b89fd87
time=2026-10-28T16:45:32.811Z level=INFO msg="Retrying step" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster step=cluster error="failed to run ARM step: failed to poll deployment: failed to wait for deployment completion: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j2862080-svc/providers/Microsoft.Resources/deployments/f309630496c74d69129833a43742f2e9eec7ceb9d4411ea07b979bc73b89fd87/operationStatuses/08584084010736719502\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n \"status\": \"Failed\",\n \"error\": {\n \"code\": \"DeploymentFailed\",\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n \"details\": [\n {\n \"code\": \"Conflict\",\n \"message\": \"{\\r\\n \\\"status\\\": \\\"Failed\\\",\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j2862080-svc/providers/Microsoft.Resources/deployments/cluster-rybldmoz73lrm\\\",\\r\\n \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j2862080-svc/providers/Microsoft.ContainerService/managedClusters/ci01-j2862080-svc\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"AzureKeyVaultKmsValidateIdentityPermissionCustomerError\\\",\\r\\n \\\"message\\\": \\\"The identity does not have keys encrypt/decrypt permission on key vault \\\\\\\"https://ah-ci01-se-j2862080-1.vault.azure.net\\\\\\\"\\\"\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n }\\r\\n}\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n"
time=2026-10-28T16:47:32.817Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster description="Step cluster\n Kind: ARM\n Template: templates/svc-cluster.bicep\n Parameters: configurations/svc-cluster.tmpl.bicepparam"
time=2026-10-28T16:47:34.143Z level=DEBUG+2 msg="Starting ARM deployment" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster
time=2026-10-28T16:47:35.068Z level=DEBUG+3 msg="Deployment started" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster deployment=8fa925f61f96a8233f154e3da5c65d168518bb692c424a9f42720dad4bf60006 portal=https://ms.portal.azure.com/#view/Microsoft_Azure_Resources/DeploymentDetails.MenuView/~/overview/id/%2Fsubscriptions%2FXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX%2FresourceGroups%2Fhcp-underlay-ci01-j2862080-svc%2Fproviders%2FMicrosoft.Resources%2Fdeployments%2F8fa925f61f96a8233f154e3da5c65d168518bb692c424a9f42720dad4bf60006
time=2026-10-28T16:49:05.326Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster err="failed to run ARM step: failed to poll deployment: failed to wait for deployment completion: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j2862080-svc/providers/Microsoft.Resources/deployments/8fa925f61f96a8233f154e3da5c65d168518bb692c424a9f42720dad4bf60006/operationStatuses/08584084008309075646\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n \"status\": \"Failed\",\n \"error\": {\n \"code\": \"DeploymentFailed\",\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n \"details\": [\n {\n \"code\": \"Conflict\",\n \"message\": \"{\\r\\n \\\"status\\\": \\\"Failed\\\",\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"message\\\": \\\"Failed to obtain the resource body, because the resource is currently in a 'Failed' provisioning state. If this resource has not been modified by a prior step in this deployment, you may need to address this before retrying the deployment.\\\"\\r\\n }\\r\\n}\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n"time=2026-10-28T15:32:48.082Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster description="Step cluster\n Kind: ARM\n Template: templates/svc-cluster.bicep\n Parameters: configurations/svc-cluster.tmpl.bicepparam"
time=2026-10-28T15:32:50.394Z level=DEBUG+2 msg="Starting ARM deployment" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster
time=2026-10-28T15:32:51.656Z level=DEBUG+3 msg="Deployment started" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster deployment=51bb59b51713025e67da1bbb1f81509d9f0e687a1b531b2b697347d81ab333d6 portal=https://ms.portal.azure.com/#view/Microsoft_Azure_Resources/DeploymentDetails.MenuView/~/overview/id/%2Fsubscriptions%2FXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX%2FresourceGroups%2Fhcp-underlay-ci01-j1862528-svc%2Fproviders%2FMicrosoft.Resources%2Fdeployments%2F51bb59b51713025e67da1bbb1f81509d9f0e687a1b531b2b697347d81ab333d6
time=2026-10-28T15:35:22.185Z level=INFO msg="Retrying step" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster step=cluster error="failed to run ARM step: failed to poll deployment: failed to wait for deployment completion: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j1862528-svc/providers/Microsoft.Resources/deployments/51bb59b51713025e67da1bbb1f81509d9f0e687a1b531b2b697347d81ab333d6/operationStatuses/08584084053144753487\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n \"status\": \"Failed\",\n \"error\": {\n \"code\": \"DeploymentFailed\",\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n \"details\": [\n {\n \"code\": \"Conflict\",\n \"message\": \"{\\r\\n \\\"status\\\": \\\"Failed\\\",\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j1862528-svc/providers/Microsoft.Resources/deployments/cluster-5xxgmkpcjnpl2\\\",\\r\\n \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j1862528-svc/providers/Microsoft.ContainerService/managedClusters/ci01-j1862528-svc\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"AzureKeyVaultKmsValidateIdentityPermissionCustomerError\\\",\\r\\n \\\"message\\\": \\\"The identity does not have keys encrypt/decrypt permission on key vault \\\\\\\"https://ah-ci01-se-j1862528-1.vault.azure.net\\\\\\\"\\\"\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n }\\r\\n}\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n"
time=2026-10-28T15:37:22.187Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster description="Step cluster\n Kind: ARM\n Template: templates/svc-cluster.bicep\n Parameters: configurations/svc-cluster.tmpl.bicepparam"
time=2026-10-28T15:37:23.185Z level=DEBUG+2 msg="Starting ARM deployment" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster
time=2026-10-28T15:37:24.435Z level=DEBUG+3 msg="Deployment started" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster deployment=f330c62a337aea146d3097645390c7352ebc193053cfec3eadc02b161a2aba5f portal=https://ms.portal.azure.com/#view/Microsoft_Azure_Resources/DeploymentDetails.MenuView/~/overview/id/%2Fsubscriptions%2FXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX%2FresourceGroups%2Fhcp-underlay-ci01-j1862528-svc%2Fproviders%2FMicrosoft.Resources%2Fdeployments%2Ff330c62a337aea146d3097645390c7352ebc193053cfec3eadc02b161a2aba5f
time=2026-10-28T15:38:54.741Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster err="failed to run ARM step: failed to poll deployment: failed to wait for deployment completion: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j1862528-svc/providers/Microsoft.Resources/deployments/f330c62a337aea146d3097645390c7352ebc193053cfec3eadc02b161a2aba5f/operationStatuses/08584084050417029665\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n \"status\": \"Failed\",\n \"error\": {\n \"code\": \"DeploymentFailed\",\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n \"details\": [\n {\n \"code\": \"Conflict\",\n \"message\": \"{\\r\\n \\\"status\\\": \\\"Failed\\\",\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"message\\\": \\\"Failed to obtain the resource body, because the resource is currently in a 'Failed' provisioning state. If this resource has not been modified by a prior step in this deployment, you may need to address this before retrying the deployment.\\\"\\r\\n }\\r\\n}\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n"time=2026-10-27T21:46:01.135Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster description="Step cluster\n Kind: ARM\n Template: templates/svc-cluster.bicep\n Parameters: configurations/svc-cluster.tmpl.bicepparam"
time=2026-10-27T21:46:03.401Z level=DEBUG+2 msg="Starting ARM deployment" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster
time=2026-10-27T21:46:05.062Z level=DEBUG+3 msg="Deployment started" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster deployment=ec85056f79198d17f9a4a06a9d62ec213e8ff2f13643b300963fd031a60589b0 portal=https://ms.portal.azure.com/#view/Microsoft_Azure_Resources/DeploymentDetails.MenuView/~/overview/id/%2Fsubscriptions%2FXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX%2FresourceGroups%2Fhcp-underlay-ci01-j0909440-svc%2Fproviders%2FMicrosoft.Resources%2Fdeployments%2Fec85056f79198d17f9a4a06a9d62ec213e8ff2f13643b300963fd031a60589b0
time=2026-10-27T21:48:05.587Z level=INFO msg="Retrying step" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster step=cluster error="failed to run ARM step: failed to poll deployment: failed to wait for deployment completion: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0909440-svc/providers/Microsoft.Resources/deployments/ec85056f79198d17f9a4a06a9d62ec213e8ff2f13643b300963fd031a60589b0/operationStatuses/08584084693213550197\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n \"status\": \"Failed\",\n \"error\": {\n \"code\": \"DeploymentFailed\",\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n \"details\": [\n {\n \"code\": \"Conflict\",\n \"message\": \"{\\r\\n \\\"status\\\": \\\"Failed\\\",\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j0909440-svc/providers/Microsoft.Resources/deployments/cluster-r5scxlmdedogk\\\",\\r\\n \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j0909440-svc/providers/Microsoft.ContainerService/managedClusters/ci01-j0909440-svc\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"AzureKeyVaultKmsValidateIdentityPermissionCustomerError\\\",\\r\\n \\\"message\\\": \\\"The identity does not have keys encrypt/decrypt permission on key vault \\\\\\\"https://ah-ci01-se-j0909440-1.vault.azure.net\\\\\\\"\\\"\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n }\\r\\n}\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n"
time=2026-10-27T21:50:05.589Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster description="Step cluster\n Kind: ARM\n Template: templates/svc-cluster.bicep\n Parameters: configurations/svc-cluster.tmpl.bicepparam"
time=2026-10-27T21:50:06.576Z level=DEBUG+2 msg="Starting ARM deployment" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster
time=2026-10-27T21:50:08.087Z level=DEBUG+3 msg="Deployment started" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster deployment=cd575f368230e7d22208e5f91e10168040367ee73eee75ae5c003a42111d906b portal=https://ms.portal.azure.com/#view/Microsoft_Azure_Resources/DeploymentDetails.MenuView/~/overview/id/%2Fsubscriptions%2FXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX%2FresourceGroups%2Fhcp-underlay-ci01-j0909440-svc%2Fproviders%2FMicrosoft.Resources%2Fdeployments%2Fcd575f368230e7d22208e5f91e10168040367ee73eee75ae5c003a42111d906b
time=2026-10-27T21:51:38.615Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster err="failed to run ARM step: failed to poll deployment: failed to wait for deployment completion: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0909440-svc/providers/Microsoft.Resources/deployments/cd575f368230e7d22208e5f91e10168040367ee73eee75ae5c003a42111d906b/operationStatuses/08584084690782406556\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n \"status\": \"Failed\",\n \"error\": {\n \"code\": \"DeploymentFailed\",\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n \"details\": [\n {\n \"code\": \"Conflict\",\n \"message\": \"{\\r\\n \\\"status\\\": \\\"Failed\\\",\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"message\\\": \\\"Failed to obtain the resource body, because the resource is currently in a 'Failed' provisioning state. If this resource has not been modified by a prior step in this deployment, you may need to address this before retrying the deployment.\\\"\\r\\n }\\r\\n}\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n"Contributing tests (1)
| ||||||||||||||||||||||||||||||||||||||||||||||||||
alert [svc] BackendControllerQueueDepthHigh firedfull failure pattern: alert [svc] BackendControllerQueueDepthHigh fired Signal: IndeterminateSignal: Indeterminate — seen in 4 prior week(s); active 2 day(s) | alert | 7 | 2.98%7 of 235 job runs affected | IndeterminateSignal: Indeterminate — seen in 4 prior week(s); active 2 day(s) | Show trend detailsOct 25: 0 · Oct 26: 0 · Oct 27: 5 · Oct 28: 2 · Oct 29: 0 · Oct 30: 0 · Oct 31: 0 · Nov 1: 0 | none | ||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (3)alert fired 1 time(s) Firing 1: State: Resolved Started: 2026-10-28T21:09:13Z Ended: 2026-10-28T21:26:12Z Severity: Sev3 Labels: alertname="BackendControllerQueueDepthHigh", cluster="ci01-j8906368-svc", component="backend", name="clusterresources", region="westus3", severity="warning" Description: Backend controller workqueue clusterresources has had a depth > 10 for more than 15 minutes, indicating work is accumulating faster than it can be processed. alert fired 1 time(s) Firing 1: State: Resolved Started: 2026-10-28T19:42:52Z Ended: 2026-10-28T20:02:53Z Severity: Sev3 Labels: alertname="BackendControllerQueueDepthHigh", cluster="ci01-j3991936-svc", component="backend", name="csstatedump", region="westus3", severity="warning" Description: Backend controller workqueue csstatedump has had a depth > 10 for more than 15 minutes, indicating work is accumulating faster than it can be processed. alert fired 1 time(s) Firing 1: State: Resolved Started: 2026-10-27T20:20:10Z Ended: 2026-10-27T20:29:09Z Severity: Sev3 Labels: alertname="BackendControllerQueueDepthHigh", cluster="ci01-j7981312-svc", component="backend", name="csstatedump", region="westus3", severity="warning" Description: Backend controller workqueue csstatedump has had a depth > 10 for more than 15 minutes, indicating work is accumulating faster than it can be processed. Contributing tests (1)
Affected runs (7) | ||||||||||||||||||||||||||||||||||||||||||||||||||
alert [svc] KubePodNotReady firedfull failure pattern: alert [svc] KubePodNotReady fired Signal: IndeterminateSignal: Indeterminate — seen in 4 prior week(s); active 2 day(s) | alert | 7 | 2.98%7 of 235 job runs affected | IndeterminateSignal: Indeterminate — seen in 4 prior week(s); active 2 day(s) | Show trend detailsOct 25: 0 · Oct 26: 0 · Oct 27: 6 · Oct 28: 1 · Oct 29: 0 · Oct 30: 0 · Oct 31: 0 · Nov 1: 0 | none | ||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (3)alert fired 4 time(s) Firing 1: State: Fired (not resolved) Started: 2026-10-28T17:07:36Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j3195648-svc", component="kubernetes-infrastructure", namespace="clusters-service", pod="clusters-service-64b56768b9-r7v9d", region="canadacentral", severity="warning" Description: Pod clusters-service/clusters-service-64b56768b9-r7v9d has been in a non-ready state for longer than 5 minutes. Firing 2: State: Fired (not resolved) Started: 2026-10-28T17:07:36Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j3195648-svc", component="kubernetes-infrastructure", namespace="fleet", pod="fleet-controller-7b9596cccb-g9thf", region="canadacentral", severity="warning" Description: Pod fleet/fleet-controller-7b9596cccb-g9thf has been in a non-ready state for longer than 5 minutes. Firing 3: State: Fired (not resolved) Started: 2026-10-28T17:07:36Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j3195648-svc", component="kubernetes-infrastructure", namespace="aro-hcp-admin-api", pod="admin-api-6d56468ffd-t9snj", region="canadacentral", severity="warning" Description: Pod aro-hcp-admin-api/admin-api-6d56468ffd-t9snj has been in a non-ready state for longer than 5 minutes. Firing 4: State: Fired (not resolved) Started: 2026-10-28T17:07:36Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j3195648-svc", component="kubernetes-infrastructure", namespace="aro-hcp-admin-api", pod="admin-api-6d56468ffd-rm24l", region="canadacentral", severity="warning" Description: Pod aro-hcp-admin-api/admin-api-6d56468ffd-rm24l has been in a non-ready state for longer than 5 minutes. alert fired 21 time(s) Firing 1: State: Fired (not resolved) Started: 2026-10-28T00:28:19Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j8270720-svc", component="kubernetes-infrastructure", namespace="aks-istio-system", pod="azure-service-mesh-istio-cni-addon-node-ck87t", region="westus3", severity="warning" Description: Pod aks-istio-system/azure-service-mesh-istio-cni-addon-node-ck87t has been in a non-ready state for longer than 5 minutes. Firing 2: State: Fired (not resolved) Started: 2026-10-28T00:28:20Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j8270720-svc", component="kubernetes-infrastructure", namespace="kube-system", pod="aks-secrets-store-csi-driver-n68qf", region="westus3", severity="warning" Description: Pod kube-system/aks-secrets-store-csi-driver-n68qf has been in a non-ready state for longer than 5 minutes. Firing 3: State: Fired (not resolved) Started: 2026-10-28T00:28:20Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j8270720-svc", component="kubernetes-infrastructure", namespace="kube-system", pod="ama-metrics-node-bx6p7", region="westus3", severity="warning" Description: Pod kube-system/ama-metrics-node-bx6p7 has been in a non-ready state for longer than 5 minutes. Firing 4: State: Fired (not resolved) Started: 2026-10-28T00:28:20Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j8270720-svc", component="kubernetes-infrastructure", namespace="kube-system", pod="ama-metrics-node-6fggw", region="westus3", severity="warning" Description: Pod kube-system/ama-metrics-node-6fggw has been in a non-ready state for longer than 5 minutes. Firing 5: State: Fired (not resolved) Started: 2026-10-28T00:28:20Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j8270720-svc", component="kubernetes-infrastructure", namespace="kube-system", pod="eraser-aks-system-42508117-vmss000002-249kw", region="westus3", severity="warning" Description: Pod kube-system/eraser-aks-system-42508117-vmss000002-249kw has been in a non-ready state for longer than 5 minutes. Firing 6: State: Fired (not resolved) Started: 2026-10-28T00:28:20Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j8270720-svc", component="kubernetes-infrastructure", namespace="aks-istio-system", pod="azure-service-mesh-istio-cni-addon-node-f2bs8", region="westus3", severity="warning" Description: Pod aks-istio-system/azure-service-mesh-istio-cni-addon-node-f2bs8 has been in a non-ready state for longer than 5 minutes. Firing 7: State: Fired (not resolved) Started: 2026-10-28T00:28:20Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j8270720-svc", component="kubernetes-infrastructure", namespace="kube-system", pod="aks-secrets-store-csi-driver-pwgk9", region="westus3", severity="warning" Description: Pod kube-system/aks-secrets-store-csi-driver-pwgk9 has been in a non-ready state for longer than 5 minutes. Firing 8: State: Fired (not resolved) Started: 2026-10-28T00:28:20Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j8270720-svc", component="kubernetes-infrastructure", namespace="kube-system", pod="ama-metrics-node-sb9gv", region="westus3", severity="warning" Description: Pod kube-system/ama-metrics-node-sb9gv has been in a non-ready state for longer than 5 minutes. Firing 9: State: Fired (not resolved) Started: 2026-10-28T00:28:20Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j8270720-svc", component="kubernetes-infrastructure", namespace="kube-system", pod="eraser-aks-system-42508117-vmss000001-z2292", region="westus3", severity="warning" Description: Pod kube-system/eraser-aks-system-42508117-vmss000001-z2292 has been in a non-ready state for longer than 5 minutes. Firing 10: State: Fired (not resolved) Started: 2026-10-28T00:28:20Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j8270720-svc", component="kubernetes-infrastructure", namespace="kube-system", pod="aks-secrets-store-csi-driver-sgh48", region="westus3", severity="warning" Description: Pod kube-system/aks-secrets-store-csi-driver-sgh48 has been in a non-ready state for longer than 5 minutes. Firing 11: State: Fired (not resolved) Started: 2026-10-28T00:28:21Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j8270720-svc", component="kubernetes-infrastructure", namespace="aks-istio-system", pod="azure-service-mesh-istio-cni-addon-node-jkt84", region="westus3", severity="warning" Description: Pod aks-istio-system/azure-service-mesh-istio-cni-addon-node-jkt84 has been in a non-ready state for longer than 5 minutes. Firing 12: State: Fired (not resolved) Started: 2026-10-28T00:28:21Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j8270720-svc", component="kubernetes-infrastructure", namespace="kube-system", pod="eraser-aks-system-42508117-vmss000000-b6bzq", region="westus3", severity="warning" Description: Pod kube-system/eraser-aks-system-42508117-vmss000000-b6bzq has been in a non-ready state for longer than 5 minutes. Firing 13: State: Fired (not resolved) Started: 2026-10-28T00:28:21Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j8270720-svc", component="kubernetes-infrastructure", namespace="kube-system", pod="hubble-relay-58594b5dd7-p7rcj", region="westus3", severity="warning" Description: Pod kube-system/hubble-relay-58594b5dd7-p7rcj has been in a non-ready state for longer than 5 minutes. Firing 14: State: Fired (not resolved) Started: 2026-10-28T00:30:21Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j8270720-svc", component="kubernetes-infrastructure", namespace="sessiongate", pod="sessiongate-8696d46cbc-gcq7v", region="westus3", severity="warning" Description: Pod sessiongate/sessiongate-8696d46cbc-gcq7v has been in a non-ready state for longer than 5 minutes. Firing 15: State: Fired (not resolved) Started: 2026-10-28T00:30:21Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j8270720-svc", component="kubernetes-infrastructure", namespace="sessiongate", pod="sessiongate-8696d46cbc-7tl64", region="westus3", severity="warning" Description: Pod sessiongate/sessiongate-8696d46cbc-7tl64 has been in a non-ready state for longer than 5 minutes. Firing 16: State: Fired (not resolved) Started: 2026-10-28T00:30:21Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j8270720-svc", component="kubernetes-infrastructure", namespace="maestro", pod="maestro-6bc6657c6b-nc5dx", region="westus3", severity="warning" Description: Pod maestro/maestro-6bc6657c6b-nc5dx has been in a non-ready state for longer than 5 minutes. Firing 17: State: Fired (not resolved) Started: 2026-10-28T00:30:21Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j8270720-svc", component="kubernetes-infrastructure", namespace="clusters-service", pod="clusters-service-567b46758d-6wzh8", region="westus3", severity="warning" Description: Pod clusters-service/clusters-service-567b46758d-6wzh8 has been in a non-ready state for longer than 5 minutes. Firing 18: State: Fired (not resolved) Started: 2026-10-28T00:30:21Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j8270720-svc", component="kubernetes-infrastructure", namespace="clusters-service", pod="clusters-service-567b46758d-rsmsf", region="westus3", severity="warning" Description: Pod clusters-service/clusters-service-567b46758d-rsmsf has been in a non-ready state for longer than 5 minutes. Firing 19: State: Fired (not resolved) Started: 2026-10-28T00:30:21Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j8270720-svc", component="kubernetes-infrastructure", namespace="clusters-service", pod="ocm-cs-db-7594d5ccfc-vd2b7", region="westus3", severity="warning" Description: Pod clusters-service/ocm-cs-db-7594d5ccfc-vd2b7 has been in a non-ready state for longer than 5 minutes. Firing 20: State: Fired (not resolved) Started: 2026-10-28T00:30:21Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j8270720-svc", component="kubernetes-infrastructure", namespace="clusters-service", pod="clusters-service-567b46758d-hzqfx", region="westus3", severity="warning" Description: Pod clusters-service/clusters-service-567b46758d-hzqfx has been in a non-ready state for longer than 5 minutes. Firing 21: State: Fired (not resolved) Started: 2026-10-28T00:30:21Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j8270720-svc", component="kubernetes-infrastructure", namespace="maestro", pod="maestro-6bc6657c6b-stp2q", region="westus3", severity="warning" Description: Pod maestro/maestro-6bc6657c6b-stp2q has been in a non-ready state for longer than 5 minutes. alert fired 8 time(s) Firing 1: State: Fired (not resolved) Started: 2026-10-27T15:12:43Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j9877504-mgmt-2", component="kubernetes-infrastructure", namespace="multicluster-engine", pod="ocm-controller-859484fc49-qhntv", region="westus3", severity="warning" Description: Pod multicluster-engine/ocm-controller-859484fc49-qhntv has been in a non-ready state for longer than 5 minutes. Firing 2: State: Fired (not resolved) Started: 2026-10-27T15:12:43Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j9877504-mgmt-2", component="kubernetes-infrastructure", namespace="multicluster-engine", pod="cluster-manager-c5d6c587f-fd8rf", region="westus3", severity="warning" Description: Pod multicluster-engine/cluster-manager-c5d6c587f-fd8rf has been in a non-ready state for longer than 5 minutes. Firing 3: State: Fired (not resolved) Started: 2026-10-27T15:12:43Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j9877504-mgmt-2", component="kubernetes-infrastructure", namespace="multicluster-engine", pod="multicluster-engine-operator-79c768d748-lg8vd", region="westus3", severity="warning" Description: Pod multicluster-engine/multicluster-engine-operator-79c768d748-lg8vd has been in a non-ready state for longer than 5 minutes. Firing 4: State: Fired (not resolved) Started: 2026-10-27T15:12:43Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j9877504-mgmt-2", component="kubernetes-infrastructure", namespace="kube-applier", pod="kube-applier-8475cd7c99-dfzv8", region="westus3", severity="warning" Description: Pod kube-applier/kube-applier-8475cd7c99-dfzv8 has been in a non-ready state for longer than 5 minutes. Firing 5: State: Fired (not resolved) Started: 2026-10-27T15:12:43Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j9877504-mgmt-2", component="kubernetes-infrastructure", namespace="multicluster-engine", pod="ocm-webhook-68676899fd-mklv5", region="westus3", severity="warning" Description: Pod multicluster-engine/ocm-webhook-68676899fd-mklv5 has been in a non-ready state for longer than 5 minutes. Firing 6: State: Fired (not resolved) Started: 2026-10-27T15:12:43Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j9877504-mgmt-2", component="kubernetes-infrastructure", namespace="mgmt-agent", pod="mgmt-agent-6dc8498b6f-ntd7x", region="westus3", severity="warning" Description: Pod mgmt-agent/mgmt-agent-6dc8498b6f-ntd7x has been in a non-ready state for longer than 5 minutes. Firing 7: State: Fired (not resolved) Started: 2026-10-27T15:12:43Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j9877504-mgmt-2", component="kubernetes-infrastructure", namespace="multicluster-engine", pod="managedcluster-import-controller-v2-6cb46794f7-bzb25", region="westus3", severity="warning" Description: Pod multicluster-engine/managedcluster-import-controller-v2-6cb46794f7-bzb25 has been in a non-ready state for longer than 5 minutes. Firing 8: State: Fired (not resolved) Started: 2026-10-27T15:12:43Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j9877504-mgmt-2", component="kubernetes-infrastructure", namespace="secret-sync-controller", pod="secrets-store-sync-controller-manager-7f4b9bc5df-9hg5c", region="westus3", severity="warning" Description: Pod secret-sync-controller/secrets-store-sync-controller-manager-7f4b9bc5df-9hg5c has been in a non-ready state for longer than 5 minutes. Contributing tests (1)
Affected runs (7) | ||||||||||||||||||||||||||||||||||||||||||||||||||
DNS for route host <route-host> did not resolve: DNS for <route-host> did not resolve within <duration> (last error: lookup <host> on <dns-server>: dial udp <ip>: i/o timeout): co...full failure pattern: DNS for route host <route-host> did not resolve: DNS for <route-host> did not resolve within <duration> (last error: lookup <host> on <dns-server>: dial udp <ip>: i/o timeout): context deadline exceeded Signal: IndeterminateSignal: Indeterminate — seen in 4 prior week(s); active 2 day(s) | e2e | 5 | 2.13%5 of 235 job runs affected | IndeterminateSignal: Indeterminate — seen in 4 prior week(s); active 2 day(s) | Show trend detailsOct 25: 0 · Oct 26: 0 · Oct 27: 4 · Oct 28: 1 · Oct 29: 0 · Oct 30: 0 · Oct 31: 0 · Nov 1: 0 | none | ||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (3)fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:241]: failed to verify simple web app runs on cluster "cluster-candidate-5-1-678df7"
Unexpected error:
<*fmt.wrapError | 0x2d18a6e97e00>:
DNS for route host agnhost-e2e-sample-app-vqnk2.apps.aro.c7e9m0p1o3l4d4j.d5kj.j8717568.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-vqnk2.apps.aro.c7e9m0p1o3l4d4j.d5kj.j8717568.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-vqnk2.apps.aro.c7e9m0p1o3l4d4j.d5kj.j8717568.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded
{
msg: "DNS for route host agnhost-e2e-sample-app-vqnk2.apps.aro.c7e9m0p1o3l4d4j.d5kj.j8717568.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-vqnk2.apps.aro.c7e9m0p1o3l4d4j.d5kj.j8717568.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-vqnk2.apps.aro.c7e9m0p1o3l4d4j.d5kj.j8717568.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
err: <*fmt.wrapError | 0x2d18a6c74540>{
msg: "DNS for agnhost-e2e-sample-app-vqnk2.apps.aro.c7e9m0p1o3l4d4j.d5kj.j8717568.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-vqnk2.apps.aro.c7e9m0p1o3l4d4j.d5kj.j8717568.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
err: <context.deadlineExceededError>{},
},
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:241]: failed to verify simple web app runs on cluster "cluster-candidate-5-1-678df7"
Unexpected error:
<*fmt.wrapError | 0x2d18a6e97e00>:
DNS for route host agnhost-e2e-sample-app-vqnk2.apps.aro.c7e9m0p1o3l4d4j.d5kj.j8717568.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-vqnk2.apps.aro.c7e9m0p1o3l4d4j.d5kj.j8717568.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-vqnk2.apps.aro.c7e9m0p1o3l4d4j.d5kj.j8717568.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded
{
msg: "DNS for route host agnhost-e2e-sample-app-vqnk2.apps.aro.c7e9m0p1o3l4d4j.d5kj.j8717568.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-vqnk2.apps.aro.c7e9m0p1o3l4d4j.d5kj.j8717568.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-vqnk2.apps.aro.c7e9m0p1o3l4d4j.d5kj.j8717568.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
err: <*fmt.wrapError | 0x2d18a6c74540>{
msg: "DNS for agnhost-e2e-sample-app-vqnk2.apps.aro.c7e9m0p1o3l4d4j.d5kj.j8717568.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-vqnk2.apps.aro.c7e9m0p1o3l4d4j.d5kj.j8717568.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
err: <context.deadlineExceededError>{},
},
}
occurredfail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:241]: failed to verify simple web app runs on cluster "cluster-candidate-4-20-rvmpld"
Unexpected error:
<*fmt.wrapError | 0x1aa0726d2620>:
DNS for route host agnhost-e2e-sample-app-ffznf.apps.aro.j5p0x2q6g6t1g2c.gabn.j8373120.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-ffznf.apps.aro.j5p0x2q6g6t1g2c.gabn.j8373120.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-ffznf.apps.aro.j5p0x2q6g6t1g2c.gabn.j8373120.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded
{
msg: "DNS for route host agnhost-e2e-sample-app-ffznf.apps.aro.j5p0x2q6g6t1g2c.gabn.j8373120.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-ffznf.apps.aro.j5p0x2q6g6t1g2c.gabn.j8373120.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-ffznf.apps.aro.j5p0x2q6g6t1g2c.gabn.j8373120.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
err: <*fmt.wrapError | 0x1aa071c17780>{
msg: "DNS for agnhost-e2e-sample-app-ffznf.apps.aro.j5p0x2q6g6t1g2c.gabn.j8373120.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-ffznf.apps.aro.j5p0x2q6g6t1g2c.gabn.j8373120.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
err: <context.deadlineExceededError>{},
},
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:241]: failed to verify simple web app runs on cluster "cluster-candidate-4-20-rvmpld"
Unexpected error:
<*fmt.wrapError | 0x1aa0726d2620>:
DNS for route host agnhost-e2e-sample-app-ffznf.apps.aro.j5p0x2q6g6t1g2c.gabn.j8373120.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-ffznf.apps.aro.j5p0x2q6g6t1g2c.gabn.j8373120.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-ffznf.apps.aro.j5p0x2q6g6t1g2c.gabn.j8373120.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded
{
msg: "DNS for route host agnhost-e2e-sample-app-ffznf.apps.aro.j5p0x2q6g6t1g2c.gabn.j8373120.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-ffznf.apps.aro.j5p0x2q6g6t1g2c.gabn.j8373120.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-ffznf.apps.aro.j5p0x2q6g6t1g2c.gabn.j8373120.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
err: <*fmt.wrapError | 0x1aa071c17780>{
msg: "DNS for agnhost-e2e-sample-app-ffznf.apps.aro.j5p0x2q6g6t1g2c.gabn.j8373120.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-ffznf.apps.aro.j5p0x2q6g6t1g2c.gabn.j8373120.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
err: <context.deadlineExceededError>{},
},
}
occurredfail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_v20260901preview.go:121]: failed to verify simple web app runs on v20260901preview cluster "v20260901"
Unexpected error:
<*fmt.wrapError | 0x1cc8b3a2b3a0>:
DNS for route host agnhost-e2e-sample-app-czc2x.apps.aro.v20260901.yl4k.j4071168.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-czc2x.apps.aro.v20260901.yl4k.j4071168.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-czc2x.apps.aro.v20260901.yl4k.j4071168.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded
{
msg: "DNS for route host agnhost-e2e-sample-app-czc2x.apps.aro.v20260901.yl4k.j4071168.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-czc2x.apps.aro.v20260901.yl4k.j4071168.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-czc2x.apps.aro.v20260901.yl4k.j4071168.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
err: <*fmt.wrapError | 0x1cc8b3559460>{
msg: "DNS for agnhost-e2e-sample-app-czc2x.apps.aro.v20260901.yl4k.j4071168.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-czc2x.apps.aro.v20260901.yl4k.j4071168.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
err: <context.deadlineExceededError>{},
},
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_v20260901preview.go:121]: failed to verify simple web app runs on v20260901preview cluster "v20260901"
Unexpected error:
<*fmt.wrapError | 0x1cc8b3a2b3a0>:
DNS for route host agnhost-e2e-sample-app-czc2x.apps.aro.v20260901.yl4k.j4071168.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-czc2x.apps.aro.v20260901.yl4k.j4071168.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-czc2x.apps.aro.v20260901.yl4k.j4071168.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded
{
msg: "DNS for route host agnhost-e2e-sample-app-czc2x.apps.aro.v20260901.yl4k.j4071168.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-czc2x.apps.aro.v20260901.yl4k.j4071168.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-czc2x.apps.aro.v20260901.yl4k.j4071168.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
err: <*fmt.wrapError | 0x1cc8b3559460>{
msg: "DNS for agnhost-e2e-sample-app-czc2x.apps.aro.v20260901.yl4k.j4071168.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-czc2x.apps.aro.v20260901.yl4k.j4071168.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
err: <context.deadlineExceededError>{},
},
}
occurredContributing tests (5)
Affected runs (5) | ||||||||||||||||||||||||||||||||||||||||||||||||||
ERROR CODE: DeadlineExceeded; detail code SubscriptionRequestsThrottled; detail message Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor '<u...full failure pattern: ERROR CODE: DeadlineExceeded; detail code SubscriptionRequestsThrottled; detail message Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor '<uuid>' exceeded. Please try again after '<seconds>' seconds after additional tokens are available.; provider Microsoft.RedHatOpenShift Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 2 day(s) | e2e | 5 | 2.13%5 of 235 job runs affected | IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 2 day(s) | Show trend detailsOct 25: 0 · Oct 26: 0 · Oct 27: 2 · Oct 28: 3 · Oct 29: 0 · Oct 30: 0 · Oct 31: 0 · Nov 1: 0 | none | ||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (3)fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_cni_cilium.go:182]: failed to verify cilium is running and nodes are Ready for cluster "cilium-cl"
Unexpected error:
<*errors.joinError | 0x740ee223cf8>:
failed to create NodePool cilium-np: failed waiting for nodepool="cilium-np" for cluster "cilium-cl" in resourcegroup="cni-cilium-dmmfjzfldpm8" to finish creating: GET https://rp.j8717568.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/e51bb1e4-4996-454f-a81d-9354613a75ee
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/e51bb1e4-4996-454f-a81d-9354613a75ee",
"name": "e51bb1e4-4996-454f-a81d-9354613a75ee",
"status": "Failed",
"startTime": "2026-10-28T19:36:46.639467207Z",
"endTime": "2026-10-28T19:55:53.152282953Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine cilium-cl-cilium-np-8z842-x96hm: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine cilium-cl-cilium-np-8z842-tlsx7: Failed: interfaces failed to create or update. err: failed to get existing resource cni-cilium-dmmfjzfldpm8--managed/cilium-cl-cilium-np-8z842-tlsx7-nic (service: interfaces): GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/cni-cilium-dmmfjzfldpm8--managed/providers/Microsoft.Network/networkInterfaces/cilium-cl-cilium-np-8z842-tlsx7-nic\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"SubscriptionRequestsThrottled\",\n \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor '1d2f4929-515e-4e8d-8580-836e819f8d50' exceeded. Please try again after '1' seconds after additional tokens are available. Refer to https://aka... message truncated\n"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://rp.j8717568.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/e51bb1e4-4996-454f-a81d-9354613a75ee
correlation ID: 4963ae88-5109-4671-873f-f28a9deccdac
client request ID: 38a19ead-d2a2-4130-9e04-1364166950d2
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_cni_cilium.go:182]: failed to verify cilium is running and nodes are Ready for cluster "cilium-cl"
Unexpected error:
<*errors.joinError | 0x740ee223cf8>:
failed to create NodePool cilium-np: failed waiting for nodepool="cilium-np" for cluster "cilium-cl" in resourcegroup="cni-cilium-dmmfjzfldpm8" to finish creating: GET https://rp.j8717568.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/e51bb1e4-4996-454f-a81d-9354613a75ee
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/e51bb1e4-4996-454f-a81d-9354613a75ee",
"name": "e51bb1e4-4996-454f-a81d-9354613a75ee",
"status": "Failed",
"startTime": "2026-10-28T19:36:46.639467207Z",
"endTime": "2026-10-28T19:55:53.152282953Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine cilium-cl-cilium-np-8z842-x96hm: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine cilium-cl-cilium-np-8z842-tlsx7: Failed: interfaces failed to create or update. err: failed to get existing resource cni-cilium-dmmfjzfldpm8--managed/cilium-cl-cilium-np-8z842-tlsx7-nic (service: interfaces): GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/cni-cilium-dmmfjzfldpm8--managed/providers/Microsoft.Network/networkInterfaces/cilium-cl-cilium-np-8z842-tlsx7-nic\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"SubscriptionRequestsThrottled\",\n \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor '1d2f4929-515e-4e8d-8580-836e819f8d50' exceeded. Please try again after '1' seconds after additional tokens are available. Refer to https://aka... message truncated\n"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://rp.j8717568.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/e51bb1e4-4996-454f-a81d-9354613a75ee
correlation ID: 4963ae88-5109-4671-873f-f28a9deccdac
client request ID: 38a19ead-d2a2-4130-9e04-1364166950d2
occurredfail [github.com/Azure/ARO-HCP/test/e2e/cluster_fips_mode.go:103]: failed to create node pool "np-1" for fips-enabled cluster "fips-enabled-cluster"
Unexpected error:
<*fmt.wrapError | 0x335ced2f1b40>:
failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "fips-enabled-cluster" in resourcegroup="fips-enabled-kfwkzrbn6vbm" to finish creating: GET https://rp.j4206336.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/77628145-9ee8-4d12-9dd2-5dced29b5ecb
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/77628145-9ee8-4d12-9dd2-5dced29b5ecb",
"name": "77628145-9ee8-4d12-9dd2-5dced29b5ecb",
"status": "Failed",
"startTime": "2026-10-28T16:28:12.935779531Z",
"endTime": "2026-10-28T16:47:20.605052936Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine j6k4t1t6z4j2k5z-np-1-5nm58-fsnz7: Failed: virtualmachine failed to create or update. err: failed to get existing resource fips-enabled-kfwkzrbn6vbm--managed/j6k4t1t6z4j2k5z-np-1-5nm58-fsnz7 (service: virtualmachine): GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/fips-enabled-kfwkzrbn6vbm--managed/providers/Microsoft.Compute/virtualMachines/j6k4t1t6z4j2k5z-np-1-5nm58-fsnz7\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"SubscriptionRequestsThrottled\",\n \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'b117cf0f-0276-4486-9691-4f00f5a90e1b' exceeded. Please try again after '1' seconds after additional tokens are available. Refer to https:... message truncated\n"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://rp.j4206336.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/77628145-9ee8-4d12-9dd2-5dced29b5ecb
correlation ID: c83d59d4-8297-41d1-87c1-fbe0352a6163
client request ID: 4dcc2ce6-6521-4d7f-b86f-f0309a80ecee
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_fips_mode.go:103]: failed to create node pool "np-1" for fips-enabled cluster "fips-enabled-cluster"
Unexpected error:
<*fmt.wrapError | 0x335ced2f1b40>:
failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "fips-enabled-cluster" in resourcegroup="fips-enabled-kfwkzrbn6vbm" to finish creating: GET https://rp.j4206336.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/77628145-9ee8-4d12-9dd2-5dced29b5ecb
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/77628145-9ee8-4d12-9dd2-5dced29b5ecb",
"name": "77628145-9ee8-4d12-9dd2-5dced29b5ecb",
"status": "Failed",
"startTime": "2026-10-28T16:28:12.935779531Z",
"endTime": "2026-10-28T16:47:20.605052936Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine j6k4t1t6z4j2k5z-np-1-5nm58-fsnz7: Failed: virtualmachine failed to create or update. err: failed to get existing resource fips-enabled-kfwkzrbn6vbm--managed/j6k4t1t6z4j2k5z-np-1-5nm58-fsnz7 (service: virtualmachine): GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/fips-enabled-kfwkzrbn6vbm--managed/providers/Microsoft.Compute/virtualMachines/j6k4t1t6z4j2k5z-np-1-5nm58-fsnz7\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"SubscriptionRequestsThrottled\",\n \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'b117cf0f-0276-4486-9691-4f00f5a90e1b' exceeded. Please try again after '1' seconds after additional tokens are available. Refer to https:... message truncated\n"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://rp.j4206336.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/77628145-9ee8-4d12-9dd2-5dced29b5ecb
correlation ID: c83d59d4-8297-41d1-87c1-fbe0352a6163
client request ID: 4dcc2ce6-6521-4d7f-b86f-f0309a80ecee
occurredfail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_v20260901preview.go:102]: failed to create node pool "np-1" for v20260901preview cluster "v20260901"
Unexpected error:
<*fmt.wrapError | 0x1ecbca3c5be0>:
failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "v20260901" in resourcegroup="v20260901preview-p9l8lv6w4lzt" to finish creating: GET https://rp.j5142016.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/0d4205c1-5db7-479b-92dc-43f88aa7a88d
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/0d4205c1-5db7-479b-92dc-43f88aa7a88d",
"name": "0d4205c1-5db7-479b-92dc-43f88aa7a88d",
"status": "Failed",
"startTime": "2026-10-28T06:54:59.774467587Z",
"endTime": "2026-10-28T07:14:00.6647054Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine v20260901-np-1-mxkn8-rr6bs: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/v20260901preview-p9l8lv6w4lzt--managed/providers/Microsoft.Compute/virtualMachines/v20260901-np-1-mxkn8-rr6bs to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine v20260901-np-1-mxkn8-c8z72: Failed: interfaces failed to create or update. err: failed to get existing resource v20260901preview-p9l8lv6w4lzt--managed/v20260901-np-1-mxkn8-c8z72-nic (service: interfaces): GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/v20260901preview-p9l8lv6w4lzt--managed/providers/Microsoft.Network/networkInterfaces/v20260901-np-1-mxkn8-c8z72-nic\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"SubscriptionRequestsThrottled\",\n \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'db27175c-5bd0-48b4-929a-41de9a53ffbf' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.m... message truncated\n"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://rp.j5142016.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/0d4205c1-5db7-479b-92dc-43f88aa7a88d
correlation ID: 5492e0f1-946f-49bb-8474-ef7f078e1f6f
client request ID: 12bebe6d-b861-48ec-942f-b913321fe64f
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_v20260901preview.go:102]: failed to create node pool "np-1" for v20260901preview cluster "v20260901"
Unexpected error:
<*fmt.wrapError | 0x1ecbca3c5be0>:
failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "v20260901" in resourcegroup="v20260901preview-p9l8lv6w4lzt" to finish creating: GET https://rp.j5142016.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/0d4205c1-5db7-479b-92dc-43f88aa7a88d
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/0d4205c1-5db7-479b-92dc-43f88aa7a88d",
"name": "0d4205c1-5db7-479b-92dc-43f88aa7a88d",
"status": "Failed",
"startTime": "2026-10-28T06:54:59.774467587Z",
"endTime": "2026-10-28T07:14:00.6647054Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine v20260901-np-1-mxkn8-rr6bs: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/v20260901preview-p9l8lv6w4lzt--managed/providers/Microsoft.Compute/virtualMachines/v20260901-np-1-mxkn8-rr6bs to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine v20260901-np-1-mxkn8-c8z72: Failed: interfaces failed to create or update. err: failed to get existing resource v20260901preview-p9l8lv6w4lzt--managed/v20260901-np-1-mxkn8-c8z72-nic (service: interfaces): GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/v20260901preview-p9l8lv6w4lzt--managed/providers/Microsoft.Network/networkInterfaces/v20260901-np-1-mxkn8-c8z72-nic\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"SubscriptionRequestsThrottled\",\n \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'db27175c-5bd0-48b4-929a-41de9a53ffbf' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.m... message truncated\n"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://rp.j5142016.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/0d4205c1-5db7-479b-92dc-43f88aa7a88d
correlation ID: 5492e0f1-946f-49bb-8474-ef7f078e1f6f
client request ID: 12bebe6d-b861-48ec-942f-b913321fe64f
occurredContributing tests (3)
Affected runs (5) | ||||||||||||||||||||||||||||||||||||||||||||||||||
VerifyCloudNetworkConfigEgressIP(<ip>) failed: failed to label node <node> egress-assignable: failed to apply k8s.ovn.org/egress-assignable label to node "<node>": nodes "<node>" ...full failure pattern: VerifyCloudNetworkConfigEgressIP(<ip>) failed: failed to label node <node> egress-assignable: failed to apply k8s.ovn.org/egress-assignable label to node "<node>": nodes "<node>" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 3 day(s) | e2e | 5 | 2.13%5 of 235 job runs affected | IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 3 day(s) | Show trend detailsOct 25: 0 · Oct 26: 1 · Oct 27: 2 · Oct 28: 2 · Oct 29: 0 · Oct 30: 0 · Oct 31: 0 · Nov 1: 0 | INT | ||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (3)fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:289]: failed to verify cluster operators on cluster "cluster-candidate-5-1-nxx9xr"
Unexpected error:
<*errors.joinError | 0x138a2d1f8048>:
VerifyCloudNetworkConfigEgressIP(10.0.0.7) failed: failed to label node e3a5t5r1l8c7f8q-np-nxx9xr-qn466-6zltb egress-assignable: failed to apply k8s.ovn.org/egress-assignable label to node "e3a5t5r1l8c7f8q-np-nxx9xr-qn466-6zltb": nodes "e3a5t5r1l8c7f8q-np-nxx9xr-qn466-6zltb" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized
{
errs: [
<*fmt.wrapError | 0x138a2cdf5020>{
msg: "VerifyCloudNetworkConfigEgressIP(10.0.0.7) failed: failed to label node e3a5t5r1l8c7f8q-np-nxx9xr-qn466-6zltb egress-assignable: failed to apply k8s.ovn.org/egress-assignable label to node \"e3a5t5r1l8c7f8q-np-nxx9xr-qn466-6zltb\": nodes \"e3a5t5r1l8c7f8q-np-nxx9xr-qn466-6zltb\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
err: <*fmt.wrapError | 0x138a2cdf5000>{
msg: "failed to label node e3a5t5r1l8c7f8q-np-nxx9xr-qn466-6zltb egress-assignable: failed to apply k8s.ovn.org/egress-assignable label to node \"e3a5t5r1l8c7f8q-np-nxx9xr-qn466-6zltb\": nodes \"e3a5t5r1l8c7f8q-np-nxx9xr-qn466-6zltb\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
err: <*fmt.wrapError | 0x138a2cdf4fe0>{
msg: "failed to apply k8s.ovn.org/egress-assignable label to node \"e3a5t5r1l8c7f8q-np-nxx9xr-qn466-6zltb\": nodes \"e3a5t5r1l8c7f8q-np-nxx9xr-qn466-6zltb\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
err: <*errors.StatusError | 0x138a2cd0dcc0>{
ErrStatus: {
TypeMeta: {Kind: "", APIVersion: ""},
ListMeta: {
SelfLink: "",
ResourceVersion: "",
Continue: "",
RemainingItemCount: nil,
ShardInfo: nil,
},
Status: "Failure",
Message: "nodes \"e3a5t5r1l8c7f8q-np-nxx9xr-qn466-6zltb\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
Reason: "Forbidden",
Details: {
Name: "e3a5t5r1l8c7f8q-np-nxx9xr-qn466-6zltb",
Group: "",
Kind: "nodes",
UID: "",
Causes: nil,
RetryAfterSeconds: 0,
},
Code: 403,
},
},
},
},
},
],
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:289]: failed to verify cluster operators on cluster "cluster-candidate-5-1-nxx9xr"
Unexpected error:
<*errors.joinError | 0x138a2d1f8048>:
VerifyCloudNetworkConfigEgressIP(10.0.0.7) failed: failed to label node e3a5t5r1l8c7f8q-np-nxx9xr-qn466-6zltb egress-assignable: failed to apply k8s.ovn.org/egress-assignable label to node "e3a5t5r1l8c7f8q-np-nxx9xr-qn466-6zltb": nodes "e3a5t5r1l8c7f8q-np-nxx9xr-qn466-6zltb" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized
{
errs: [
<*fmt.wrapError | 0x138a2cdf5020>{
msg: "VerifyCloudNetworkConfigEgressIP(10.0.0.7) failed: failed to label node e3a5t5r1l8c7f8q-np-nxx9xr-qn466-6zltb egress-assignable: failed to apply k8s.ovn.org/egress-assignable label to node \"e3a5t5r1l8c7f8q-np-nxx9xr-qn466-6zltb\": nodes \"e3a5t5r1l8c7f8q-np-nxx9xr-qn466-6zltb\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
err: <*fmt.wrapError | 0x138a2cdf5000>{
msg: "failed to label node e3a5t5r1l8c7f8q-np-nxx9xr-qn466-6zltb egress-assignable: failed to apply k8s.ovn.org/egress-assignable label to node \"e3a5t5r1l8c7f8q-np-nxx9xr-qn466-6zltb\": nodes \"e3a5t5r1l8c7f8q-np-nxx9xr-qn466-6zltb\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
err: <*fmt.wrapError | 0x138a2cdf4fe0>{
msg: "failed to apply k8s.ovn.org/egress-assignable label to node \"e3a5t5r1l8c7f8q-np-nxx9xr-qn466-6zltb\": nodes \"e3a5t5r1l8c7f8q-np-nxx9xr-qn466-6zltb\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
err: <*errors.StatusError | 0x138a2cd0dcc0>{
ErrStatus: {
TypeMeta: {Kind: "", APIVersion: ""},
ListMeta: {
SelfLink: "",
ResourceVersion: "",
Continue: "",
RemainingItemCount: nil,
ShardInfo: nil,
},
Status: "Failure",
Message: "nodes \"e3a5t5r1l8c7f8q-np-nxx9xr-qn466-6zltb\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
Reason: "Forbidden",
Details: {
Name: "e3a5t5r1l8c7f8q-np-nxx9xr-qn466-6zltb",
Group: "",
Kind: "nodes",
UID: "",
Causes: nil,
RetryAfterSeconds: 0,
},
Code: 403,
},
},
},
},
},
],
}
occurredfail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:289]: failed to verify cluster operators on cluster "cluster-candidate-5-1-x4d58d"
Unexpected error:
<*errors.joinError | 0x2a036e0a6030>:
VerifyCloudNetworkConfigEgressIP(10.0.0.7) failed: failed to label node t5n3k9o3y9v8w3y-np-x4d58d-tdn4x-htxvc egress-assignable: failed to apply k8s.ovn.org/egress-assignable label to node "t5n3k9o3y9v8w3y-np-x4d58d-tdn4x-htxvc": nodes "t5n3k9o3y9v8w3y-np-x4d58d-tdn4x-htxvc" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized
{
errs: [
<*fmt.wrapError | 0x2a036cc0d0a0>{
msg: "VerifyCloudNetworkConfigEgressIP(10.0.0.7) failed: failed to label node t5n3k9o3y9v8w3y-np-x4d58d-tdn4x-htxvc egress-assignable: failed to apply k8s.ovn.org/egress-assignable label to node \"t5n3k9o3y9v8w3y-np-x4d58d-tdn4x-htxvc\": nodes \"t5n3k9o3y9v8w3y-np-x4d58d-tdn4x-htxvc\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
err: <*fmt.wrapError | 0x2a036cc0d080>{
msg: "failed to label node t5n3k9o3y9v8w3y-np-x4d58d-tdn4x-htxvc egress-assignable: failed to apply k8s.ovn.org/egress-assignable label to node \"t5n3k9o3y9v8w3y-np-x4d58d-tdn4x-htxvc\": nodes \"t5n3k9o3y9v8w3y-np-x4d58d-tdn4x-htxvc\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
err: <*fmt.wrapError | 0x2a036cc0d060>{
msg: "failed to apply k8s.ovn.org/egress-assignable label to node \"t5n3k9o3y9v8w3y-np-x4d58d-tdn4x-htxvc\": nodes \"t5n3k9o3y9v8w3y-np-x4d58d-tdn4x-htxvc\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
err: <*errors.StatusError | 0x2a036cc3c780>{
ErrStatus: {
TypeMeta: {Kind: "", APIVersion: ""},
ListMeta: {
SelfLink: "",
ResourceVersion: "",
Continue: "",
RemainingItemCount: nil,
ShardInfo: nil,
},
Status: "Failure",
Message: "nodes \"t5n3k9o3y9v8w3y-np-x4d58d-tdn4x-htxvc\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
Reason: "Forbidden",
Details: {
Name: "t5n3k9o3y9v8w3y-np-x4d58d-tdn4x-htxvc",
Group: "",
Kind: "nodes",
UID: "",
Causes: nil,
RetryAfterSeconds: 0,
},
Code: 403,
},
},
},
},
},
],
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:289]: failed to verify cluster operators on cluster "cluster-candidate-5-1-x4d58d"
Unexpected error:
<*errors.joinError | 0x2a036e0a6030>:
VerifyCloudNetworkConfigEgressIP(10.0.0.7) failed: failed to label node t5n3k9o3y9v8w3y-np-x4d58d-tdn4x-htxvc egress-assignable: failed to apply k8s.ovn.org/egress-assignable label to node "t5n3k9o3y9v8w3y-np-x4d58d-tdn4x-htxvc": nodes "t5n3k9o3y9v8w3y-np-x4d58d-tdn4x-htxvc" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized
{
errs: [
<*fmt.wrapError | 0x2a036cc0d0a0>{
msg: "VerifyCloudNetworkConfigEgressIP(10.0.0.7) failed: failed to label node t5n3k9o3y9v8w3y-np-x4d58d-tdn4x-htxvc egress-assignable: failed to apply k8s.ovn.org/egress-assignable label to node \"t5n3k9o3y9v8w3y-np-x4d58d-tdn4x-htxvc\": nodes \"t5n3k9o3y9v8w3y-np-x4d58d-tdn4x-htxvc\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
err: <*fmt.wrapError | 0x2a036cc0d080>{
msg: "failed to label node t5n3k9o3y9v8w3y-np-x4d58d-tdn4x-htxvc egress-assignable: failed to apply k8s.ovn.org/egress-assignable label to node \"t5n3k9o3y9v8w3y-np-x4d58d-tdn4x-htxvc\": nodes \"t5n3k9o3y9v8w3y-np-x4d58d-tdn4x-htxvc\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
err: <*fmt.wrapError | 0x2a036cc0d060>{
msg: "failed to apply k8s.ovn.org/egress-assignable label to node \"t5n3k9o3y9v8w3y-np-x4d58d-tdn4x-htxvc\": nodes \"t5n3k9o3y9v8w3y-np-x4d58d-tdn4x-htxvc\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
err: <*errors.StatusError | 0x2a036cc3c780>{
ErrStatus: {
TypeMeta: {Kind: "", APIVersion: ""},
ListMeta: {
SelfLink: "",
ResourceVersion: "",
Continue: "",
RemainingItemCount: nil,
ShardInfo: nil,
},
Status: "Failure",
Message: "nodes \"t5n3k9o3y9v8w3y-np-x4d58d-tdn4x-htxvc\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
Reason: "Forbidden",
Details: {
Name: "t5n3k9o3y9v8w3y-np-x4d58d-tdn4x-htxvc",
Group: "",
Kind: "nodes",
UID: "",
Causes: nil,
RetryAfterSeconds: 0,
},
Code: 403,
},
},
},
},
},
],
}
occurredfail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:289]: failed to verify cluster operators on cluster "cluster-candidate-5-0-j22hl8"
Unexpected error:
<*errors.joinError | 0x237cb4aa4138>:
VerifyCloudNetworkConfigEgressIP(10.0.0.7) failed: failed to label node s9t1y1s2c1s6e5d-np-j22hl8-cg7vv-84h99 egress-assignable: failed to apply k8s.ovn.org/egress-assignable label to node "s9t1y1s2c1s6e5d-np-j22hl8-cg7vv-84h99": nodes "s9t1y1s2c1s6e5d-np-j22hl8-cg7vv-84h99" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized
{
errs: [
<*fmt.wrapError | 0x237cb4952040>{
msg: "VerifyCloudNetworkConfigEgressIP(10.0.0.7) failed: failed to label node s9t1y1s2c1s6e5d-np-j22hl8-cg7vv-84h99 egress-assignable: failed to apply k8s.ovn.org/egress-assignable label to node \"s9t1y1s2c1s6e5d-np-j22hl8-cg7vv-84h99\": nodes \"s9t1y1s2c1s6e5d-np-j22hl8-cg7vv-84h99\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
err: <*fmt.wrapError | 0x237cb4952020>{
msg: "failed to label node s9t1y1s2c1s6e5d-np-j22hl8-cg7vv-84h99 egress-assignable: failed to apply k8s.ovn.org/egress-assignable label to node \"s9t1y1s2c1s6e5d-np-j22hl8-cg7vv-84h99\": nodes \"s9t1y1s2c1s6e5d-np-j22hl8-cg7vv-84h99\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
err: <*fmt.wrapError | 0x237cb4952000>{
msg: "failed to apply k8s.ovn.org/egress-assignable label to node \"s9t1y1s2c1s6e5d-np-j22hl8-cg7vv-84h99\": nodes \"s9t1y1s2c1s6e5d-np-j22hl8-cg7vv-84h99\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
err: <*errors.StatusError | 0x237cb49b60a0>{
ErrStatus: {
TypeMeta: {Kind: "", APIVersion: ""},
ListMeta: {
SelfLink: "",
ResourceVersion: "",
Continue: "",
RemainingItemCount: nil,
ShardInfo: nil,
},
Status: "Failure",
Message: "nodes \"s9t1y1s2c1s6e5d-np-j22hl8-cg7vv-84h99\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
Reason: "Forbidden",
Details: {
Name: "s9t1y1s2c1s6e5d-np-j22hl8-cg7vv-84h99",
Group: "",
Kind: "nodes",
UID: "",
Causes: nil,
RetryAfterSeconds: 0,
},
Code: 403,
},
},
},
},
},
],
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:289]: failed to verify cluster operators on cluster "cluster-candidate-5-0-j22hl8"
Unexpected error:
<*errors.joinError | 0x237cb4aa4138>:
VerifyCloudNetworkConfigEgressIP(10.0.0.7) failed: failed to label node s9t1y1s2c1s6e5d-np-j22hl8-cg7vv-84h99 egress-assignable: failed to apply k8s.ovn.org/egress-assignable label to node "s9t1y1s2c1s6e5d-np-j22hl8-cg7vv-84h99": nodes "s9t1y1s2c1s6e5d-np-j22hl8-cg7vv-84h99" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized
{
errs: [
<*fmt.wrapError | 0x237cb4952040>{
msg: "VerifyCloudNetworkConfigEgressIP(10.0.0.7) failed: failed to label node s9t1y1s2c1s6e5d-np-j22hl8-cg7vv-84h99 egress-assignable: failed to apply k8s.ovn.org/egress-assignable label to node \"s9t1y1s2c1s6e5d-np-j22hl8-cg7vv-84h99\": nodes \"s9t1y1s2c1s6e5d-np-j22hl8-cg7vv-84h99\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
err: <*fmt.wrapError | 0x237cb4952020>{
msg: "failed to label node s9t1y1s2c1s6e5d-np-j22hl8-cg7vv-84h99 egress-assignable: failed to apply k8s.ovn.org/egress-assignable label to node \"s9t1y1s2c1s6e5d-np-j22hl8-cg7vv-84h99\": nodes \"s9t1y1s2c1s6e5d-np-j22hl8-cg7vv-84h99\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
err: <*fmt.wrapError | 0x237cb4952000>{
msg: "failed to apply k8s.ovn.org/egress-assignable label to node \"s9t1y1s2c1s6e5d-np-j22hl8-cg7vv-84h99\": nodes \"s9t1y1s2c1s6e5d-np-j22hl8-cg7vv-84h99\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
err: <*errors.StatusError | 0x237cb49b60a0>{
ErrStatus: {
TypeMeta: {Kind: "", APIVersion: ""},
ListMeta: {
SelfLink: "",
ResourceVersion: "",
Continue: "",
RemainingItemCount: nil,
ShardInfo: nil,
},
Status: "Failure",
Message: "nodes \"s9t1y1s2c1s6e5d-np-j22hl8-cg7vv-84h99\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
Reason: "Forbidden",
Details: {
Name: "s9t1y1s2c1s6e5d-np-j22hl8-cg7vv-84h99",
Group: "",
Kind: "nodes",
UID: "",
Causes: nil,
RetryAfterSeconds: 0,
},
Code: 403,
},
},
},
},
},
],
}
occurredContributing tests (3)
Affected runs (5) | ||||||||||||||||||||||||||||||||||||||||||||||||||
alert [svc] ClusterCredentialMetricMissing firedfull failure pattern: alert [svc] ClusterCredentialMetricMissing fired Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 3 day(s) | alert | 3 | 1.28%3 of 235 job runs affected | IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 3 day(s) | Show trend detailsOct 25: 1 · Oct 26: 0 · Oct 27: 2 · Oct 28: 1 · Oct 29: 0 · Oct 30: 0 · Oct 31: 0 · Nov 1: 0 | none | ||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (3)alert fired 1 time(s) Firing 1: State: Fired (not resolved) Started: 2026-10-28T01:28:46Z Severity: Sev3 Labels: alertname="ClusterCredentialMetricMissing", cluster="ci01-j5915264-svc", severity="warning" Description: Credential refresher expiration metric is missing for service cluster ci01-j5915264-svc. alert fired 1 time(s) Firing 1: State: Fired (not resolved) Started: 2026-10-27T21:37:46Z Severity: Sev3 Labels: alertname="ClusterCredentialMetricMissing", cluster="ci01-j3842944-svc", severity="warning" Description: Credential refresher expiration metric is missing for service cluster ci01-j3842944-svc. alert fired 1 time(s) Firing 1: State: Fired (not resolved) Started: 2026-10-27T18:18:42Z Severity: Sev3 Labels: alertname="ClusterCredentialMetricMissing", cluster="ci01-j4499328-svc", severity="warning" Description: Credential refresher expiration metric is missing for service cluster ci01-j4499328-svc. Contributing tests (1)
Affected runs (3)
| ||||||||||||||||||||||||||||||||||||||||||||||||||
Environment: INT
Daily Run Outcomes
No tests below 95.00% in this window with at least 10 runs.
Up to 50 failure patterns are loaded (minimum 1.00% impact), with 10 shown by default. Click column headers to re-sort.
Jump to the full Failure Patterns view for this window
| Failure patternThe canonical description of a recurring CI failure, extracted and normalized from raw logs. | Failed atThe stage of the job run where this failure occurred: 'provision' (environment setup, DEV only), 'e2e' (test suite execution), or 'other' (CI infrastructure issues that did not produce a failure pattern). | Number of distinct job runs where this failure pattern was detected. | Percentage of all job runs in this environment affected by this failure pattern during the selected window. | Classification of this failure pattern: Regression (likely caused by a specific PR), Flake (intermittent failure spread across days), Noise (low-quality or generic pattern), or Indeterminate. | TrendShows daily activity for this failure pattern in a trailing window anchored to the selected end date. The sparkline covers at least 7 days and at most 14 days, depending on the current window size. | Also inOther environments where the same failure pattern was also detected during the selected window. | |||||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedC...full failure pattern: ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider]; provider Microsoft.RedHatOpenShift Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s) | unknown | 2 | 13.33%2 of 15 job runs affected | IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s) | Show trend detailsOct 25: 0 · Oct 26: 0 · Oct 27: 2 · Oct 28: 0 · Oct 29: 0 · Oct 30: 0 · Oct 31: 0 · Nov 1: 0 | DEV, PROD | |||||||||||||||||||||
Full failure examples (2)fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:150]: failed to create HCP cluster
Unexpected error:
<*fmt.wrapError | 0x24681019eca0>:
failed to create HCP cluster ingress-cert-le: failed waiting for cluster="ingress-cert-le" in resourcegroup="ingress-cert-le-m6dlx86v7zvt" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/4249c9a2-6a00-4a31-a13c-7b395eeda2a9
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/4249c9a2-6a00-4a31-a13c-7b395eeda2a9",
"name": "4249c9a2-6a00-4a31-a13c-7b395eeda2a9",
"status": "Failed",
"startTime": "2026-10-27T21:21:19.945219062Z",
"endTime": "2026-10-27T21:50:23.791391981Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/4249c9a2-6a00-4a31-a13c-7b395eeda2a9
correlation ID: 79f0a27c-81eb-45e3-ba1c-7ed22276e9b1
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:150]: failed to create HCP cluster
Unexpected error:
<*fmt.wrapError | 0x24681019eca0>:
failed to create HCP cluster ingress-cert-le: failed waiting for cluster="ingress-cert-le" in resourcegroup="ingress-cert-le-m6dlx86v7zvt" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/4249c9a2-6a00-4a31-a13c-7b395eeda2a9
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/4249c9a2-6a00-4a31-a13c-7b395eeda2a9",
"name": "4249c9a2-6a00-4a31-a13c-7b395eeda2a9",
"status": "Failed",
"startTime": "2026-10-27T21:21:19.945219062Z",
"endTime": "2026-10-27T21:50:23.791391981Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/4249c9a2-6a00-4a31-a13c-7b395eeda2a9
correlation ID: 79f0a27c-81eb-45e3-ba1c-7ed22276e9b1
client request ID: unavailable
occurredfail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:154]: HCP cluster rg-candidate-4-20-x79lxq7pcp2s/cluster-candidate-4-20-wz9qjx should provision
Unexpected error:
<*fmt.wrapError | 0x1ea33da4c3e0>:
failed to create HCP cluster cluster-candidate-4-20-wz9qjx: failed waiting for cluster="cluster-candidate-4-20-wz9qjx" in resourcegroup="rg-candidate-4-20-x79lxq7pcp2s" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/5ef510b2-7fbf-47d3-a252-f868567b1a44
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/5ef510b2-7fbf-47d3-a252-f868567b1a44",
"name": "5ef510b2-7fbf-47d3-a252-f868567b1a44",
"status": "Failed",
"startTime": "2026-10-27T16:50:03.68579052Z",
"endTime": "2026-10-27T17:19:03.174752422Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/5ef510b2-7fbf-47d3-a252-f868567b1a44
correlation ID: 0794d196-09b5-4818-a7ca-7f5fb13bca4d
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:154]: HCP cluster rg-candidate-4-20-x79lxq7pcp2s/cluster-candidate-4-20-wz9qjx should provision
Unexpected error:
<*fmt.wrapError | 0x1ea33da4c3e0>:
failed to create HCP cluster cluster-candidate-4-20-wz9qjx: failed waiting for cluster="cluster-candidate-4-20-wz9qjx" in resourcegroup="rg-candidate-4-20-x79lxq7pcp2s" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/5ef510b2-7fbf-47d3-a252-f868567b1a44
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/5ef510b2-7fbf-47d3-a252-f868567b1a44",
"name": "5ef510b2-7fbf-47d3-a252-f868567b1a44",
"status": "Failed",
"startTime": "2026-10-27T16:50:03.68579052Z",
"endTime": "2026-10-27T17:19:03.174752422Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/5ef510b2-7fbf-47d3-a252-f868567b1a44
correlation ID: 0794d196-09b5-4818-a7ca-7f5fb13bca4d
client request ID: unavailable
occurredContributing tests (2)
Affected runs (2)
| |||||||||||||||||||||||||||
timeout during GetAdminRESTConfigForHCPCluster while waiting for hcpcluster credsfull failure pattern: timeout during GetAdminRESTConfigForHCPCluster while waiting for hcpcluster creds Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 2 day(s) | unknown | 1 | 6.67%1 of 15 job runs affected | IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 2 day(s) | Show trend detailsOct 25: 1 · Oct 26: 0 · Oct 27: 1 · Oct 28: 0 · Oct 29: 0 · Oct 30: 0 · Oct 31: 0 · Nov 1: 0 | none | |||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/e2e/cluster_container_registry.go:657]: failed to get admin REST config
Unexpected error:
<*fmt.wrapErrors | 0x13a2362ee210>:
failed waiting for hcpCluster="acr-pull" in resourcegroup="acr-pull-5cjwmbnpxtmc" to finish getting creds, caused by: timeout '2.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster20261001 for cluster acr-pull in resource group acr-pull-5cjwmbnpxtmc, error: context deadline exceeded
{
msg: "failed waiting for hcpCluster=\"acr-pull\" in resourcegroup=\"acr-pull-5cjwmbnpxtmc\" to finish getting creds, caused by: timeout '2.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster20261001 for cluster acr-pull in resource group acr-pull-5cjwmbnpxtmc, error: context deadline exceeded",
errs: [
<*errors.errorString | 0x13a235e7d850>{
s: "timeout '2.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster20261001 for cluster acr-pull in resource group acr-pull-5cjwmbnpxtmc",
},
<context.deadlineExceededError>{},
],
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_container_registry.go:657]: failed to get admin REST config
Unexpected error:
<*fmt.wrapErrors | 0x13a2362ee210>:
failed waiting for hcpCluster="acr-pull" in resourcegroup="acr-pull-5cjwmbnpxtmc" to finish getting creds, caused by: timeout '2.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster20261001 for cluster acr-pull in resource group acr-pull-5cjwmbnpxtmc, error: context deadline exceeded
{
msg: "failed waiting for hcpCluster=\"acr-pull\" in resourcegroup=\"acr-pull-5cjwmbnpxtmc\" to finish getting creds, caused by: timeout '2.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster20261001 for cluster acr-pull in resource group acr-pull-5cjwmbnpxtmc, error: context deadline exceeded",
errs: [
<*errors.errorString | 0x13a235e7d850>{
s: "timeout '2.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster20261001 for cluster acr-pull in resource group acr-pull-5cjwmbnpxtmc",
},
<context.deadlineExceededError>{},
],
}
occurredContributing tests (1)
Affected runs (1)
| |||||||||||||||||||||||||||
ERROR CODE: NotFound; detail message The resource record '*.e2e-dns-fn44p6' does not exist in resource group '<resource-group>' of subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX...full failure pattern: ERROR CODE: NotFound; detail message The resource record '*.e2e-dns-fn44p6' does not exist in resource group '<resource-group>' of subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX'.; provider Microsoft.Network Signal: IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | unknown | 1 | 6.67%1 of 15 job runs affected | IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | Show trend detailsOct 25: 0 · Oct 26: 1 · Oct 27: 0 · Oct 28: 0 · Oct 29: 0 · Oct 30: 0 · Oct 31: 0 · Nov 1: 0 | none | |||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:248]: failed to verify cluster operators on cluster "cluster-candidate-4-21-rf46z8"
Unexpected error:
<*errors.joinError | 0x3ac741d4138>:
VerifyIngressDNSRecord(aro.l5z2t8a8n2n6x2i.7cr2.westus3.aroapp-hcp.azure-test.net) failed: VerifyIngressDNSRecord(aro.l5z2t8a8n2n6x2i.7cr2.westus3.aroapp-hcp.azure-test.net) timed out after 10m0.001s: DNS A record "*.e2e-dns-fn44p6" not yet present in zone rg-candidate-4-21-vgzkq7npmrnr-candidate-rf46z8--managed/aro.l5z2t8a8n2n6x2i.7cr2.westus3.aroapp-hcp.azure-test.net: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-21-vgzkq7npmrnr-candidate-rf46z8--managed/providers/Microsoft.Network/dnsZones/aro.l5z2t8a8n2n6x2i.7cr2.westus3.aroapp-hcp.azure-test.net/A/*.e2e-dns-fn44p6
--------------------------------------------------------------------------------
RESPONSE 404: 404 Not Found
ERROR CODE: NotFound
--------------------------------------------------------------------------------
{
"code": "NotFound",
"message": "The resource record '*.e2e-dns-fn44p6' does not exist in resource group 'rg-candidate-4-21-vgzkq7npmrnr-candidate-rf46z8--managed' of subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX'."
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-21-vgzkq7npmrnr-candidate-rf46z8--managed/providers/Microsoft.Network/dnsZones/aro.l5z2t8a8n2n6x2i.7cr2.westus3.aroapp-hcp.azure-test.net/A/*.e2e-dns-fn44p6
correlation ID: bf6f75cd-f285-4895-ad70-7d62eff4cba2
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:248]: failed to verify cluster operators on cluster "cluster-candidate-4-21-rf46z8"
Unexpected error:
<*errors.joinError | 0x3ac741d4138>:
VerifyIngressDNSRecord(aro.l5z2t8a8n2n6x2i.7cr2.westus3.aroapp-hcp.azure-test.net) failed: VerifyIngressDNSRecord(aro.l5z2t8a8n2n6x2i.7cr2.westus3.aroapp-hcp.azure-test.net) timed out after 10m0.001s: DNS A record "*.e2e-dns-fn44p6" not yet present in zone rg-candidate-4-21-vgzkq7npmrnr-candidate-rf46z8--managed/aro.l5z2t8a8n2n6x2i.7cr2.westus3.aroapp-hcp.azure-test.net: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-21-vgzkq7npmrnr-candidate-rf46z8--managed/providers/Microsoft.Network/dnsZones/aro.l5z2t8a8n2n6x2i.7cr2.westus3.aroapp-hcp.azure-test.net/A/*.e2e-dns-fn44p6
--------------------------------------------------------------------------------
RESPONSE 404: 404 Not Found
ERROR CODE: NotFound
--------------------------------------------------------------------------------
{
"code": "NotFound",
"message": "The resource record '*.e2e-dns-fn44p6' does not exist in resource group 'rg-candidate-4-21-vgzkq7npmrnr-candidate-rf46z8--managed' of subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX'."
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-21-vgzkq7npmrnr-candidate-rf46z8--managed/providers/Microsoft.Network/dnsZones/aro.l5z2t8a8n2n6x2i.7cr2.westus3.aroapp-hcp.azure-test.net/A/*.e2e-dns-fn44p6
correlation ID: bf6f75cd-f285-4895-ad70-7d62eff4cba2
client request ID: unavailable
occurredContributing tests (1)
Affected runs (1)
| |||||||||||||||||||||||||||
VerifyCloudNetworkConfigEgressIP(<ip>) failed: failed to label node <node> egress-assignable: failed to apply k8s.ovn.org/egress-assignable label to node "<node>": nodes "<node>" ...full failure pattern: VerifyCloudNetworkConfigEgressIP(<ip>) failed: failed to label node <node> egress-assignable: failed to apply k8s.ovn.org/egress-assignable label to node "<node>": nodes "<node>" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not Signal: IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | unknown | 1 | 6.67%1 of 15 job runs affected | IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | Show trend detailsOct 25: 0 · Oct 26: 0 · Oct 27: 1 · Oct 28: 0 · Oct 29: 0 · Oct 30: 0 · Oct 31: 0 · Nov 1: 0 | DEV | |||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:289]: failed to verify cluster operators on cluster "cluster-candidate-4-22-cghxqb"
Unexpected error:
<*errors.joinError | 0x315939ea25b8>:
VerifyCloudNetworkConfigEgressIP(10.0.0.7) failed: failed to label node p3m2y8l9j1y3v1d-np-cghxqb-7rwcj-2g4cv egress-assignable: failed to apply k8s.ovn.org/egress-assignable label to node "p3m2y8l9j1y3v1d-np-cghxqb-7rwcj-2g4cv": nodes "p3m2y8l9j1y3v1d-np-cghxqb-7rwcj-2g4cv" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized
{
errs: [
<*fmt.wrapError | 0x31593941d8c0>{
msg: "VerifyCloudNetworkConfigEgressIP(10.0.0.7) failed: failed to label node p3m2y8l9j1y3v1d-np-cghxqb-7rwcj-2g4cv egress-assignable: failed to apply k8s.ovn.org/egress-assignable label to node \"p3m2y8l9j1y3v1d-np-cghxqb-7rwcj-2g4cv\": nodes \"p3m2y8l9j1y3v1d-np-cghxqb-7rwcj-2g4cv\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
err: <*fmt.wrapError | 0x31593941d880>{
msg: "failed to label node p3m2y8l9j1y3v1d-np-cghxqb-7rwcj-2g4cv egress-assignable: failed to apply k8s.ovn.org/egress-assignable label to node \"p3m2y8l9j1y3v1d-np-cghxqb-7rwcj-2g4cv\": nodes \"p3m2y8l9j1y3v1d-np-cghxqb-7rwcj-2g4cv\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
err: <*fmt.wrapError | 0x31593941d840>{
msg: "failed to apply k8s.ovn.org/egress-assignable label to node \"p3m2y8l9j1y3v1d-np-cghxqb-7rwcj-2g4cv\": nodes \"p3m2y8l9j1y3v1d-np-cghxqb-7rwcj-2g4cv\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
err: <*errors.StatusError | 0x31593a26f680>{
ErrStatus: {
TypeMeta: {Kind: "", APIVersion: ""},
ListMeta: {
SelfLink: "",
ResourceVersion: "",
Continue: "",
RemainingItemCount: nil,
ShardInfo: nil,
},
Status: "Failure",
Message: "nodes \"p3m2y8l9j1y3v1d-np-cghxqb-7rwcj-2g4cv\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
Reason: "Forbidden",
Details: {
Name: "p3m2y8l9j1y3v1d-np-cghxqb-7rwcj-2g4cv",
Group: "",
Kind: "nodes",
UID: "",
Causes: nil,
RetryAfterSeconds: 0,
},
Code: 403,
},
},
},
},
},
],
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:289]: failed to verify cluster operators on cluster "cluster-candidate-4-22-cghxqb"
Unexpected error:
<*errors.joinError | 0x315939ea25b8>:
VerifyCloudNetworkConfigEgressIP(10.0.0.7) failed: failed to label node p3m2y8l9j1y3v1d-np-cghxqb-7rwcj-2g4cv egress-assignable: failed to apply k8s.ovn.org/egress-assignable label to node "p3m2y8l9j1y3v1d-np-cghxqb-7rwcj-2g4cv": nodes "p3m2y8l9j1y3v1d-np-cghxqb-7rwcj-2g4cv" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized
{
errs: [
<*fmt.wrapError | 0x31593941d8c0>{
msg: "VerifyCloudNetworkConfigEgressIP(10.0.0.7) failed: failed to label node p3m2y8l9j1y3v1d-np-cghxqb-7rwcj-2g4cv egress-assignable: failed to apply k8s.ovn.org/egress-assignable label to node \"p3m2y8l9j1y3v1d-np-cghxqb-7rwcj-2g4cv\": nodes \"p3m2y8l9j1y3v1d-np-cghxqb-7rwcj-2g4cv\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
err: <*fmt.wrapError | 0x31593941d880>{
msg: "failed to label node p3m2y8l9j1y3v1d-np-cghxqb-7rwcj-2g4cv egress-assignable: failed to apply k8s.ovn.org/egress-assignable label to node \"p3m2y8l9j1y3v1d-np-cghxqb-7rwcj-2g4cv\": nodes \"p3m2y8l9j1y3v1d-np-cghxqb-7rwcj-2g4cv\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
err: <*fmt.wrapError | 0x31593941d840>{
msg: "failed to apply k8s.ovn.org/egress-assignable label to node \"p3m2y8l9j1y3v1d-np-cghxqb-7rwcj-2g4cv\": nodes \"p3m2y8l9j1y3v1d-np-cghxqb-7rwcj-2g4cv\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
err: <*errors.StatusError | 0x31593a26f680>{
ErrStatus: {
TypeMeta: {Kind: "", APIVersion: ""},
ListMeta: {
SelfLink: "",
ResourceVersion: "",
Continue: "",
RemainingItemCount: nil,
ShardInfo: nil,
},
Status: "Failure",
Message: "nodes \"p3m2y8l9j1y3v1d-np-cghxqb-7rwcj-2g4cv\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
Reason: "Forbidden",
Details: {
Name: "p3m2y8l9j1y3v1d-np-cghxqb-7rwcj-2g4cv",
Group: "",
Kind: "nodes",
UID: "",
Causes: nil,
RetryAfterSeconds: 0,
},
Code: 403,
},
},
},
},
},
],
}
occurredContributing tests (1)
Affected runs (1)
| |||||||||||||||||||||||||||
Environment: STG
Daily Run Outcomes
| Pass rate | Runs | Date (UTC) | Suite | Test |
|---|---|---|---|---|
| 91.67% | 12 | stage/parallel | Customer should be able to create an HCP cluster and manage pull secrets | |
| 91.67% | 12 | stage/parallel | Customer should be able to create an HCP cluster with custom autoscaling and update the autoscaling configuration |
Up to 50 failure patterns are loaded (minimum 1.00% impact), with 10 shown by default. Click column headers to re-sort.
Jump to the full Failure Patterns view for this window
| Failure patternThe canonical description of a recurring CI failure, extracted and normalized from raw logs. | Failed atThe stage of the job run where this failure occurred: 'provision' (environment setup, DEV only), 'e2e' (test suite execution), or 'other' (CI infrastructure issues that did not produce a failure pattern). | Number of distinct job runs where this failure pattern was detected. | Percentage of all job runs in this environment affected by this failure pattern during the selected window. | Classification of this failure pattern: Regression (likely caused by a specific PR), Flake (intermittent failure spread across days), Noise (low-quality or generic pattern), or Indeterminate. | TrendShows daily activity for this failure pattern in a trailing window anchored to the selected end date. The sparkline covers at least 7 days and at most 14 days, depending on the current window size. | Also inOther environments where the same failure pattern was also detected during the selected window. | ||||||||||||||||||||||||||||||||||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
ACM ingress secret openshift-ingress/cluster-ingress-cert exists; expected ACM certificate delivery to be suppressedfull failure pattern: ACM ingress secret openshift-ingress/cluster-ingress-cert exists; expected ACM certificate delivery to be suppressed Signal: IndeterminateSignal: Indeterminate — seen in 2 prior week(s); active 2 day(s) | e2e | 2 | 14.29%2 of 14 job runs affected | IndeterminateSignal: Indeterminate — seen in 2 prior week(s); active 2 day(s) | Show trend detailsOct 25: 0 · Oct 26: 2 · Oct 27: 3 · Oct 28: 0 · Oct 29: 0 · Oct 30: 0 · Oct 31: 0 · Nov 1: 0 | PROD | ||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (3)fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:528]: ACM must not create its ingress certificate secret
Expected success, but got an error:
<*errors.errorString | 0x66df03e8b20>:
ACM ingress secret openshift-ingress/cluster-ingress-cert exists; expected ACM certificate delivery to be suppressed
{
s: "ACM ingress secret openshift-ingress/cluster-ingress-cert exists; expected ACM certificate delivery to be suppressed",
}
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:528]: ACM must not create its ingress certificate secret
Expected success, but got an error:
<*errors.errorString | 0x66df03e8b20>:
ACM ingress secret openshift-ingress/cluster-ingress-cert exists; expected ACM certificate delivery to be suppressed
{
s: "ACM ingress secret openshift-ingress/cluster-ingress-cert exists; expected ACM certificate delivery to be suppressed",
}fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:528]: ACM must not create its ingress certificate secret
Expected success, but got an error:
<*errors.errorString | 0x2de6fa3020f0>:
ACM ingress secret openshift-ingress/cluster-ingress-cert exists; expected ACM certificate delivery to be suppressed
{
s: "ACM ingress secret openshift-ingress/cluster-ingress-cert exists; expected ACM certificate delivery to be suppressed",
}
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:528]: ACM must not create its ingress certificate secret
Expected success, but got an error:
<*errors.errorString | 0x2de6fa3020f0>:
ACM ingress secret openshift-ingress/cluster-ingress-cert exists; expected ACM certificate delivery to be suppressed
{
s: "ACM ingress secret openshift-ingress/cluster-ingress-cert exists; expected ACM certificate delivery to be suppressed",
}fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:260]: ACM must not create its ingress certificate secret for cluster "cluster-candidate-4-23-wlxxtl"
Expected success, but got an error:
<*errors.errorString | 0x30c9514302d0>:
ACM ingress secret openshift-ingress/cluster-ingress-cert exists; expected ACM certificate delivery to be suppressed
{
s: "ACM ingress secret openshift-ingress/cluster-ingress-cert exists; expected ACM certificate delivery to be suppressed",
}
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:260]: ACM must not create its ingress certificate secret for cluster "cluster-candidate-4-23-wlxxtl"
Expected success, but got an error:
<*errors.errorString | 0x30c9514302d0>:
ACM ingress secret openshift-ingress/cluster-ingress-cert exists; expected ACM certificate delivery to be suppressed
{
s: "ACM ingress secret openshift-ingress/cluster-ingress-cert exists; expected ACM certificate delivery to be suppressed",
}Contributing tests (5)
Affected runs (2)
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||
VerifyDefaultIngressServingCertificateSecretWiring timed out after 10m0full failure pattern: VerifyDefaultIngressServingCertificateSecretWiring timed out after 10m0 Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 2 day(s) | e2e | 2 | 14.29%2 of 14 job runs affected | IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 2 day(s) | Show trend detailsOct 25: 0 · Oct 26: 2 · Oct 27: 1 · Oct 28: 0 · Oct 29: 0 · Oct 30: 0 · Oct 31: 0 · Nov 1: 0 | PROD | ||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (3)fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:256]: default-ingress-cert must exist and be referenced by IngressController/default for cluster "cluster-candidate-5-0-2nz4xc"
Expected success, but got an error:
<*fmt.wrapError | 0x203abb848000>:
VerifyDefaultIngressServingCertificateSecretWiring timed out after 10m0.001s: IngressController/default references &LocalObjectReference{Name:cluster-ingress-cert,}, expected secret "default-ingress-cert"
{
msg: "VerifyDefaultIngressServingCertificateSecretWiring timed out after 10m0.001s: IngressController/default references &LocalObjectReference{Name:cluster-ingress-cert,}, expected secret \"default-ingress-cert\"",
err: <*errors.errorString | 0x203ab927a940>{
s: "IngressController/default references &LocalObjectReference{Name:cluster-ingress-cert,}, expected secret \"default-ingress-cert\"",
},
}
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:256]: default-ingress-cert must exist and be referenced by IngressController/default for cluster "cluster-candidate-5-0-2nz4xc"
Expected success, but got an error:
<*fmt.wrapError | 0x203abb848000>:
VerifyDefaultIngressServingCertificateSecretWiring timed out after 10m0.001s: IngressController/default references &LocalObjectReference{Name:cluster-ingress-cert,}, expected secret "default-ingress-cert"
{
msg: "VerifyDefaultIngressServingCertificateSecretWiring timed out after 10m0.001s: IngressController/default references &LocalObjectReference{Name:cluster-ingress-cert,}, expected secret \"default-ingress-cert\"",
err: <*errors.errorString | 0x203ab927a940>{
s: "IngressController/default references &LocalObjectReference{Name:cluster-ingress-cert,}, expected secret \"default-ingress-cert\"",
},
}fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:256]: default-ingress-cert must exist and be referenced by IngressController/default for cluster "cluster-nightly-5-0-q6mvvg"
Expected success, but got an error:
<*fmt.wrapError | 0x1ec67b8dad40>:
VerifyDefaultIngressServingCertificateSecretWiring timed out after 10m0.007s: IngressController/default references &LocalObjectReference{Name:cluster-ingress-cert,}, expected secret "default-ingress-cert"
{
msg: "VerifyDefaultIngressServingCertificateSecretWiring timed out after 10m0.007s: IngressController/default references &LocalObjectReference{Name:cluster-ingress-cert,}, expected secret \"default-ingress-cert\"",
err: <*errors.errorString | 0x1ec67cfb92a0>{
s: "IngressController/default references &LocalObjectReference{Name:cluster-ingress-cert,}, expected secret \"default-ingress-cert\"",
},
}
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:256]: default-ingress-cert must exist and be referenced by IngressController/default for cluster "cluster-nightly-5-0-q6mvvg"
Expected success, but got an error:
<*fmt.wrapError | 0x1ec67b8dad40>:
VerifyDefaultIngressServingCertificateSecretWiring timed out after 10m0.007s: IngressController/default references &LocalObjectReference{Name:cluster-ingress-cert,}, expected secret "default-ingress-cert"
{
msg: "VerifyDefaultIngressServingCertificateSecretWiring timed out after 10m0.007s: IngressController/default references &LocalObjectReference{Name:cluster-ingress-cert,}, expected secret \"default-ingress-cert\"",
err: <*errors.errorString | 0x1ec67cfb92a0>{
s: "IngressController/default references &LocalObjectReference{Name:cluster-ingress-cert,}, expected secret \"default-ingress-cert\"",
},
}fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:256]: default-ingress-cert must exist and be referenced by IngressController/default for cluster "cluster-nightly-4-23-9qvdk7"
Expected success, but got an error:
<*fmt.wrapError | 0x38ab336f65a0>:
VerifyDefaultIngressServingCertificateSecretWiring timed out after 10m0.004s: IngressController/default references &LocalObjectReference{Name:cluster-ingress-cert,}, expected secret "default-ingress-cert"
{
msg: "VerifyDefaultIngressServingCertificateSecretWiring timed out after 10m0.004s: IngressController/default references &LocalObjectReference{Name:cluster-ingress-cert,}, expected secret \"default-ingress-cert\"",
err: <*errors.errorString | 0x38ab34e123c0>{
s: "IngressController/default references &LocalObjectReference{Name:cluster-ingress-cert,}, expected secret \"default-ingress-cert\"",
},
}
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:256]: default-ingress-cert must exist and be referenced by IngressController/default for cluster "cluster-nightly-4-23-9qvdk7"
Expected success, but got an error:
<*fmt.wrapError | 0x38ab336f65a0>:
VerifyDefaultIngressServingCertificateSecretWiring timed out after 10m0.004s: IngressController/default references &LocalObjectReference{Name:cluster-ingress-cert,}, expected secret "default-ingress-cert"
{
msg: "VerifyDefaultIngressServingCertificateSecretWiring timed out after 10m0.004s: IngressController/default references &LocalObjectReference{Name:cluster-ingress-cert,}, expected secret \"default-ingress-cert\"",
err: <*errors.errorString | 0x38ab34e123c0>{
s: "IngressController/default references &LocalObjectReference{Name:cluster-ingress-cert,}, expected secret \"default-ingress-cert\"",
},
}Contributing tests (3)
Affected runs (2)
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||
ERROR CODE: DeadlineExceeded; detail message node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hype...full failure pattern: ERROR CODE: DeadlineExceeded; detail message node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is <count>, want <count>; node pool condition AllNodesHealthy is False: <count> of <count> machines are not healthy Machine <node>: InspectionFailed: Waiting for AzureMachine to report spec.providerID; node pool condition AllMachinesReady is False: <count> of <count> machines are not ready Machine <node>: Creating: virtualmachine creating or updating; provider Microsoft.RedHatOpenShift Signal: IndeterminateSignal: Indeterminate — seen in 2 prior week(s); active 1 day(s) | unknown | 1 | 7.14%1 of 14 job runs affected | IndeterminateSignal: Indeterminate — seen in 2 prior week(s); active 1 day(s) | Show trend detailsOct 25: 0 · Oct 26: 0 · Oct 27: 10 · Oct 28: 0 · Oct 29: 0 · Oct 30: 0 · Oct 31: 0 · Nov 1: 0 | none | ||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (3)fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:181]: failed to create node pool "np-jxwc9m" for cluster "cluster-candidate-5-1-jxwc9m"
Unexpected error:
<*fmt.wrapError | 0x2089be556480>:
failed to create NodePool np-jxwc9m: failed waiting for nodepool="np-jxwc9m" for cluster "cluster-candidate-5-1-jxwc9m" in resourcegroup="rg-candidate-5-1-t5bnkh9ptx8p" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/aa13a852-544e-4ae1-9ee9-db477af0802a
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/aa13a852-544e-4ae1-9ee9-db477af0802a",
"name": "aa13a852-544e-4ae1-9ee9-db477af0802a",
"status": "Failed",
"startTime": "2026-10-27T08:36:20.928028726Z",
"endTime": "2026-10-27T08:55:28.505111283Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine f5e7a9i9o6b7x5n-np-jxwc9m-l25lf-nmkn7: InspectionFailed: Waiting for AzureMachine to report spec.providerID\nMachine f5e7a9i9o6b7x5n-np-jxwc9m-l25lf-rxvtq: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine f5e7a9i9o6b7x5n-np-jxwc9m-l25lf-nmkn7: Creating: virtualmachine creating or updating\nMachine f5e7a9i9o6b7x5n-np-jxwc9m-l25lf-rxvtq: Creating: virtualmachine creating or updating\n"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/aa13a852-544e-4ae1-9ee9-db477af0802a
correlation ID: 994d5b05-7775-48cd-93aa-f17d4c56d0ef
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:181]: failed to create node pool "np-jxwc9m" for cluster "cluster-candidate-5-1-jxwc9m"
Unexpected error:
<*fmt.wrapError | 0x2089be556480>:
failed to create NodePool np-jxwc9m: failed waiting for nodepool="np-jxwc9m" for cluster "cluster-candidate-5-1-jxwc9m" in resourcegroup="rg-candidate-5-1-t5bnkh9ptx8p" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/aa13a852-544e-4ae1-9ee9-db477af0802a
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/aa13a852-544e-4ae1-9ee9-db477af0802a",
"name": "aa13a852-544e-4ae1-9ee9-db477af0802a",
"status": "Failed",
"startTime": "2026-10-27T08:36:20.928028726Z",
"endTime": "2026-10-27T08:55:28.505111283Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine f5e7a9i9o6b7x5n-np-jxwc9m-l25lf-nmkn7: InspectionFailed: Waiting for AzureMachine to report spec.providerID\nMachine f5e7a9i9o6b7x5n-np-jxwc9m-l25lf-rxvtq: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine f5e7a9i9o6b7x5n-np-jxwc9m-l25lf-nmkn7: Creating: virtualmachine creating or updating\nMachine f5e7a9i9o6b7x5n-np-jxwc9m-l25lf-rxvtq: Creating: virtualmachine creating or updating\n"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/aa13a852-544e-4ae1-9ee9-db477af0802a
correlation ID: 994d5b05-7775-48cd-93aa-f17d4c56d0ef
client request ID: unavailable
occurredfail [github.com/Azure/ARO-HCP/test/e2e/cluster_nodepool_active_versions.go:96]: failed to create node pool "np-1" for v20261001 cluster "v20261001"
Unexpected error:
<*fmt.wrapError | 0x1520efb25aa0>:
failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "v20261001" in resourcegroup="v20261001-g5mddw2znmwk" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/a9c68b69-2f13-4771-88f2-779ee3c74191
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/a9c68b69-2f13-4771-88f2-779ee3c74191",
"name": "a9c68b69-2f13-4771-88f2-779ee3c74191",
"status": "Failed",
"startTime": "2026-10-27T08:30:17.391270547Z",
"endTime": "2026-10-27T08:49:18.18634688Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine v20261001-np-1-gzw7k-gkpzc: InspectionFailed: Waiting for AzureMachine to report spec.providerID\nMachine v20261001-np-1-gzw7k-sxqb6: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine v20261001-np-1-gzw7k-gkpzc: Creating: virtualmachine creating or updating\nMachine v20261001-np-1-gzw7k-sxqb6: Creating: virtualmachine creating or updating\n"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/a9c68b69-2f13-4771-88f2-779ee3c74191
correlation ID: c7660621-545a-4ad8-89c0-9dd90d63841c
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_nodepool_active_versions.go:96]: failed to create node pool "np-1" for v20261001 cluster "v20261001"
Unexpected error:
<*fmt.wrapError | 0x1520efb25aa0>:
failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "v20261001" in resourcegroup="v20261001-g5mddw2znmwk" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/a9c68b69-2f13-4771-88f2-779ee3c74191
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/a9c68b69-2f13-4771-88f2-779ee3c74191",
"name": "a9c68b69-2f13-4771-88f2-779ee3c74191",
"status": "Failed",
"startTime": "2026-10-27T08:30:17.391270547Z",
"endTime": "2026-10-27T08:49:18.18634688Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine v20261001-np-1-gzw7k-gkpzc: InspectionFailed: Waiting for AzureMachine to report spec.providerID\nMachine v20261001-np-1-gzw7k-sxqb6: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine v20261001-np-1-gzw7k-gkpzc: Creating: virtualmachine creating or updating\nMachine v20261001-np-1-gzw7k-sxqb6: Creating: virtualmachine creating or updating\n"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/a9c68b69-2f13-4771-88f2-779ee3c74191
correlation ID: c7660621-545a-4ad8-89c0-9dd90d63841c
client request ID: unavailable
occurredfail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:193]: failed to verify nodes are Ready with Cilium CNI for cluster "cilium-cluster"
Unexpected error:
<*errors.joinError | 0x3baee6990108>:
VerifyNodesReady failed: no nodes found in the cluster
VerifyCiliumOperational failed: not all pods in kube-system namespace are running: failed to list pods in kube-system namespace: context deadline exceeded
failed to create NodePool cilium-np: failed waiting for nodepool="cilium-np" for cluster "cilium-cluster" in resourcegroup="complex-cilium-kv-2gdwf4pbz22s" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/0e019fa7-e26d-44c3-b5c0-c1a3e7ecc4af
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/0e019fa7-e26d-44c3-b5c0-c1a3e7ecc4af",
"name": "0e019fa7-e26d-44c3-b5c0-c1a3e7ecc4af",
"status": "Failed",
"startTime": "2026-10-27T08:12:10.541346261Z",
"endTime": "2026-10-27T08:31:14.925265024Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine cilium-cluster-cilium-np-ffmd5-h94sn: InspectionFailed: Waiting for AzureMachine to report spec.providerID\nMachine cilium-cluster-cilium-np-ffmd5-hvcp5: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine cilium-cluster-cilium-np-ffmd5-h94sn: Creating: virtualmachine creating or updating\nMachine cilium-cluster-cilium-np-ffmd5-hvcp5: Creating: virtualmachine creating or updating\n"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/0e019fa7-e26d-44c3-b5c0-c1a3e7ecc4af
correlation ID: 6e36e413-6dec-4393-bcb3-4788f0d108b3
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:193]: failed to verify nodes are Ready with Cilium CNI for cluster "cilium-cluster"
Unexpected error:
<*errors.joinError | 0x3baee6990108>:
VerifyNodesReady failed: no nodes found in the cluster
VerifyCiliumOperational failed: not all pods in kube-system namespace are running: failed to list pods in kube-system namespace: context deadline exceeded
failed to create NodePool cilium-np: failed waiting for nodepool="cilium-np" for cluster "cilium-cluster" in resourcegroup="complex-cilium-kv-2gdwf4pbz22s" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/0e019fa7-e26d-44c3-b5c0-c1a3e7ecc4af
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/0e019fa7-e26d-44c3-b5c0-c1a3e7ecc4af",
"name": "0e019fa7-e26d-44c3-b5c0-c1a3e7ecc4af",
"status": "Failed",
"startTime": "2026-10-27T08:12:10.541346261Z",
"endTime": "2026-10-27T08:31:14.925265024Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine cilium-cluster-cilium-np-ffmd5-h94sn: InspectionFailed: Waiting for AzureMachine to report spec.providerID\nMachine cilium-cluster-cilium-np-ffmd5-hvcp5: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine cilium-cluster-cilium-np-ffmd5-h94sn: Creating: virtualmachine creating or updating\nMachine cilium-cluster-cilium-np-ffmd5-hvcp5: Creating: virtualmachine creating or updating\n"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/0e019fa7-e26d-44c3-b5c0-c1a3e7ecc4af
correlation ID: 6e36e413-6dec-4393-bcb3-4788f0d108b3
client request ID: unavailable
occurredContributing tests (10)
Affected runs (1)
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||
VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Unauthorizedfull failure pattern: VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Unauthorized Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s) | e2e | 1 | 7.14%1 of 14 job runs affected | IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s) | Show trend detailsOct 25: 0 · Oct 26: 0 · Oct 27: 1 · Oct 28: 0 · Oct 29: 0 · Oct 30: 0 · Oct 31: 0 · Nov 1: 0 | PROD | ||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:168]: failed to verify HCP cluster "cluster-nightly-4-21-htnxwk" is viable
Unexpected error:
<*errors.joinError | 0x28946bc08d20>:
VerifyBasicAccess failed: failed to list services: Unauthorized
VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Unauthorized
{
errs: [
<*fmt.wrapError | 0x28946b722320>{
msg: "VerifyBasicAccess failed: failed to list services: Unauthorized",
err: <*fmt.wrapError | 0x28946b722300>{
msg: "failed to list services: Unauthorized",
err: <*errors.StatusError | 0x28946b314d20>{
ErrStatus: {
TypeMeta: {Kind: "", APIVersion: ""},
ListMeta: {
SelfLink: "",
ResourceVersion: "",
Continue: "",
RemainingItemCount: nil,
ShardInfo: nil,
},
Status: "Failure",
Message: "Unauthorized",
Reason: "Unauthorized",
Details: nil,
Code: 401,
},
},
},
},
<*fmt.wrapError | 0x28946b60da40>{
msg: "VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Unauthorized",
err: <*fmt.wrapError | 0x28946b60da20>{
msg: "failed to list all APIServices: Unauthorized",
err: <*errors.StatusError | 0x28946b518dc0>{
ErrStatus: {
TypeMeta: {Kind: "", APIVersion: ""},
ListMeta: {
SelfLink: "",
ResourceVersion: "",
Continue: "",
RemainingItemCount: nil,
ShardInfo: nil,
},
Status: "Failure",
Message: "Unauthorized",
Reason: "Unauthorized",
Details: nil,
Code: 401,
},
},
},
},
],
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:168]: failed to verify HCP cluster "cluster-nightly-4-21-htnxwk" is viable
Unexpected error:
<*errors.joinError | 0x28946bc08d20>:
VerifyBasicAccess failed: failed to list services: Unauthorized
VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Unauthorized
{
errs: [
<*fmt.wrapError | 0x28946b722320>{
msg: "VerifyBasicAccess failed: failed to list services: Unauthorized",
err: <*fmt.wrapError | 0x28946b722300>{
msg: "failed to list services: Unauthorized",
err: <*errors.StatusError | 0x28946b314d20>{
ErrStatus: {
TypeMeta: {Kind: "", APIVersion: ""},
ListMeta: {
SelfLink: "",
ResourceVersion: "",
Continue: "",
RemainingItemCount: nil,
ShardInfo: nil,
},
Status: "Failure",
Message: "Unauthorized",
Reason: "Unauthorized",
Details: nil,
Code: 401,
},
},
},
},
<*fmt.wrapError | 0x28946b60da40>{
msg: "VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Unauthorized",
err: <*fmt.wrapError | 0x28946b60da20>{
msg: "failed to list all APIServices: Unauthorized",
err: <*errors.StatusError | 0x28946b518dc0>{
ErrStatus: {
TypeMeta: {Kind: "", APIVersion: ""},
ListMeta: {
SelfLink: "",
ResourceVersion: "",
Continue: "",
RemainingItemCount: nil,
ShardInfo: nil,
},
Status: "Failure",
Message: "Unauthorized",
Reason: "Unauthorized",
Details: nil,
Code: 401,
},
},
},
},
],
}
occurredContributing tests (1)
Affected runs (1)
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||
certificate not ready: reason=DoesNotExist message=Issuing certificate as Secret does not existfull failure pattern: certificate not ready: reason=DoesNotExist message=Issuing certificate as Secret does not exist Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s) | unknown | 1 | 7.14%1 of 14 job runs affected | IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s) | Show trend detailsOct 25: 0 · Oct 26: 0 · Oct 27: 0 · Oct 28: 1 · Oct 29: 0 · Oct 30: 0 · Oct 31: 0 · Nov 1: 0 | none | ||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:510]: Timed out after 600.004s.
Certificate should be issued by Let's Encrypt staging via ACME DNS-01
Expected success, but got an error:
<*errors.errorString | 0x3b09d8d2ba10>:
certificate not ready: reason=DoesNotExist message=Issuing certificate as Secret does not exist
{
s: "certificate not ready: reason=DoesNotExist message=Issuing certificate as Secret does not exist",
}
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:510]: Timed out after 600.004s.
Certificate should be issued by Let's Encrypt staging via ACME DNS-01
Expected success, but got an error:
<*errors.errorString | 0x3b09d8d2ba10>:
certificate not ready: reason=DoesNotExist message=Issuing certificate as Secret does not exist
{
s: "certificate not ready: reason=DoesNotExist message=Issuing certificate as Secret does not exist",
}Contributing tests (1)
Affected runs (1)
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||
ERROR CODE: DeadlineExceeded; detail message node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hype...full failure pattern: ERROR CODE: DeadlineExceeded; detail message node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is <count>, want >= <count> (autoscaling min); node pool condition AllNodesHealthy is False: <count> of <count> machines are not healthy Machine <node>: InspectionFailed: Waiting for AzureMachine to report spec.providerID; node pool condition AllMachinesReady is False: <count> of <count> machines are not ready Machine <node>: Creating: virtualmachine creating or updating; provider Microsoft.RedHatOpenShift Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s) | unknown | 1 | 7.14%1 of 14 job runs affected | IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s) | Show trend detailsOct 25: 0 · Oct 26: 0 · Oct 27: 1 · Oct 28: 0 · Oct 29: 0 · Oct 30: 0 · Oct 31: 0 · Nov 1: 0 | none | ||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:404]: failed to verify cilium is running and nodes are Ready for cluster "agg-cluster"
Unexpected error:
<*errors.joinError | 0xa0fb7d04138>:
VerifyNodesReady failed: no nodes found in the cluster
VerifyCiliumOperational failed: not all pods in kube-system namespace are running: failed to list pods in kube-system namespace: context deadline exceeded
failed to create NodePool agg-np-a: failed waiting for nodepool="agg-np-a" for cluster "agg-cluster" in resourcegroup="feature-aggregation-xf9jw4l5cdpf" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/0006ad13-87f3-4fc4-8b43-20306abddf58
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/0006ad13-87f3-4fc4-8b43-20306abddf58",
"name": "0006ad13-87f3-4fc4-8b43-20306abddf58",
"status": "Failed",
"startTime": "2026-10-27T08:14:54.971524126Z",
"endTime": "2026-10-27T08:33:58.162558969Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want \u003e= 2 (autoscaling min); node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine agg-cluster-agg-np-a-7cl7p-8fh5m: InspectionFailed: Waiting for AzureMachine to report spec.providerID\nMachine agg-cluster-agg-np-a-7cl7p-fd22m: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine agg-cluster-agg-np-a-7cl7p-8fh5m: Creating: virtualmachine creating or updating\nMachine agg-cluster-agg-np-a-7cl7p-fd22m: Creating: virtualmachine creating or updating\n"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/0006ad13-87f3-4fc4-8b43-20306abddf58
correlation ID: 7bfa7cc1-239c-4ae3-a9bc-578e94ecc5a3
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:404]: failed to verify cilium is running and nodes are Ready for cluster "agg-cluster"
Unexpected error:
<*errors.joinError | 0xa0fb7d04138>:
VerifyNodesReady failed: no nodes found in the cluster
VerifyCiliumOperational failed: not all pods in kube-system namespace are running: failed to list pods in kube-system namespace: context deadline exceeded
failed to create NodePool agg-np-a: failed waiting for nodepool="agg-np-a" for cluster "agg-cluster" in resourcegroup="feature-aggregation-xf9jw4l5cdpf" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/0006ad13-87f3-4fc4-8b43-20306abddf58
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/0006ad13-87f3-4fc4-8b43-20306abddf58",
"name": "0006ad13-87f3-4fc4-8b43-20306abddf58",
"status": "Failed",
"startTime": "2026-10-27T08:14:54.971524126Z",
"endTime": "2026-10-27T08:33:58.162558969Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want \u003e= 2 (autoscaling min); node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine agg-cluster-agg-np-a-7cl7p-8fh5m: InspectionFailed: Waiting for AzureMachine to report spec.providerID\nMachine agg-cluster-agg-np-a-7cl7p-fd22m: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine agg-cluster-agg-np-a-7cl7p-8fh5m: Creating: virtualmachine creating or updating\nMachine agg-cluster-agg-np-a-7cl7p-fd22m: Creating: virtualmachine creating or updating\n"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/0006ad13-87f3-4fc4-8b43-20306abddf58
correlation ID: 7bfa7cc1-239c-4ae3-a9bc-578e94ecc5a3
client request ID: unavailable
occurredContributing tests (1)
Affected runs (1)
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||
failed to create nodepool <nodepool>: failed waiting for nodepool="<nodepool>" for cluster "<cluster>" in resourcegroup="<resource-group>" to finish creating: GET <url>full failure pattern: failed to create nodepool <nodepool>: failed waiting for nodepool="<nodepool>" for cluster "<cluster>" in resourcegroup="<resource-group>" to finish creating: GET <url> Signal: IndeterminateSignal: Indeterminate — seen in 2 prior week(s); active 1 day(s) | e2e | 1 | 7.14%1 of 14 job runs affected | IndeterminateSignal: Indeterminate — seen in 2 prior week(s); active 1 day(s) | Show trend detailsOct 25: 0 · Oct 26: 0 · Oct 27: 0 · Oct 28: 1 · Oct 29: 0 · Oct 30: 0 · Oct 31: 0 · Nov 1: 0 | DEV | ||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_delete.go:128]: failed to create node pools
Expected success, but got an error:
<*fmt.wrapError | 0x8979cd42dc0>:
failed to create NodePool np-deletable: failed waiting for nodepool="np-deletable" for cluster "np-delete-hcp-cluster" in resourcegroup="nodepool-delete-d4wglkkdjhfj" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/4691f9f8-c66b-49b4-9a08-68ce0d3e72a6
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/4691f9f8-c66b-49b4-9a08-68ce0d3e72a6",
"name": "4691f9f8-c66b-49b4-9a08-68ce0d3e72a6",
"status": "Failed",
"startTime": "2026-10-28T03:01:21.21908488Z",
"endTime": "2026-10-28T03:20:21.275752007Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine p0i5l3o3i9z8d7r-np-deletable-95t5c-lvd52: NodeNotHealthy: * Node.Ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine p0i5l3o3i9z8d7r-np-deletable-95t5c-lvd52: NotReady: * NodeHealthy:\n * Node.Ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?\n"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/4691f9f8-c66b-49b4-9a08-68ce0d3e72a6
correlation ID: 48b19522-cae3-4702-8aae-982e9a4042a0
client request ID: unavailable
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_delete.go:128]: failed to create node pools
Expected success, but got an error:
<*fmt.wrapError | 0x8979cd42dc0>:
failed to create NodePool np-deletable: failed waiting for nodepool="np-deletable" for cluster "np-delete-hcp-cluster" in resourcegroup="nodepool-delete-d4wglkkdjhfj" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/4691f9f8-c66b-49b4-9a08-68ce0d3e72a6
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/4691f9f8-c66b-49b4-9a08-68ce0d3e72a6",
"name": "4691f9f8-c66b-49b4-9a08-68ce0d3e72a6",
"status": "Failed",
"startTime": "2026-10-28T03:01:21.21908488Z",
"endTime": "2026-10-28T03:20:21.275752007Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine p0i5l3o3i9z8d7r-np-deletable-95t5c-lvd52: NodeNotHealthy: * Node.Ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine p0i5l3o3i9z8d7r-np-deletable-95t5c-lvd52: NotReady: * NodeHealthy:\n * Node.Ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?\n"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/4691f9f8-c66b-49b4-9a08-68ce0d3e72a6
correlation ID: 48b19522-cae3-4702-8aae-982e9a4042a0
client request ID: unavailableContributing tests (1)
Affected runs (1)
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||
failed to create nodepool <nodepool>: failed waiting for nodepool="<nodepool>" for cluster "<cluster>" in resourcegroup="<resource-group>" to finish creating: GET <url>full failure pattern: failed to create nodepool <nodepool>: failed waiting for nodepool="<nodepool>" for cluster "<cluster>" in resourcegroup="<resource-group>" to finish creating: GET <url> Signal: IndeterminateSignal: Indeterminate — seen in 2 prior week(s); active 1 day(s) | unknown | 1 | 7.14%1 of 14 job runs affected | IndeterminateSignal: Indeterminate — seen in 2 prior week(s); active 1 day(s) | Show trend detailsOct 25: 0 · Oct 26: 0 · Oct 27: 1 · Oct 28: 0 · Oct 29: 0 · Oct 30: 0 · Oct 31: 0 · Nov 1: 0 | DEV | ||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_delete.go:128]: failed to create node pools
Expected success, but got an error:
<*fmt.wrapError | 0x3b12e2cba60>:
failed to create NodePool np-delete-main: failed waiting for nodepool="np-delete-main" for cluster "np-delete-hcp-cluster" in resourcegroup="nodepool-delete-4j5vtdb84h8v" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/083c3fe4-0c43-4b6a-9905-e8c1aa97e192
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/083c3fe4-0c43-4b6a-9905-e8c1aa97e192",
"name": "083c3fe4-0c43-4b6a-9905-e8c1aa97e192",
"status": "Failed",
"startTime": "2026-10-27T09:13:51.032005174Z",
"endTime": "2026-10-27T09:32:53.54866754Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine o3t1v0i5b5o7c5l-np-delete-main-bg4kp-nd4jp: InspectionFailed: Waiting for AzureMachine to report spec.providerID\nMachine o3t1v0i5b5o7c5l-np-delete-main-bg4kp-q624c: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine o3t1v0i5b5o7c5l-np-delete-main-bg4kp-nd4jp: Creating: virtualmachine creating or updating\nMachine o3t1v0i5b5o7c5l-np-delete-main-bg4kp-q624c: Creating: virtualmachine creating or updating\n"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/083c3fe4-0c43-4b6a-9905-e8c1aa97e192
correlation ID: 01ff1c80-ae7b-4cd9-aeb7-68197786fff9
client request ID: unavailable
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_delete.go:128]: failed to create node pools
Expected success, but got an error:
<*fmt.wrapError | 0x3b12e2cba60>:
failed to create NodePool np-delete-main: failed waiting for nodepool="np-delete-main" for cluster "np-delete-hcp-cluster" in resourcegroup="nodepool-delete-4j5vtdb84h8v" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/083c3fe4-0c43-4b6a-9905-e8c1aa97e192
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/083c3fe4-0c43-4b6a-9905-e8c1aa97e192",
"name": "083c3fe4-0c43-4b6a-9905-e8c1aa97e192",
"status": "Failed",
"startTime": "2026-10-27T09:13:51.032005174Z",
"endTime": "2026-10-27T09:32:53.54866754Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine o3t1v0i5b5o7c5l-np-delete-main-bg4kp-nd4jp: InspectionFailed: Waiting for AzureMachine to report spec.providerID\nMachine o3t1v0i5b5o7c5l-np-delete-main-bg4kp-q624c: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine o3t1v0i5b5o7c5l-np-delete-main-bg4kp-nd4jp: Creating: virtualmachine creating or updating\nMachine o3t1v0i5b5o7c5l-np-delete-main-bg4kp-q624c: Creating: virtualmachine creating or updating\n"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/083c3fe4-0c43-4b6a-9905-e8c1aa97e192
correlation ID: 01ff1c80-ae7b-4cd9-aeb7-68197786fff9
client request ID: unavailableContributing tests (1)
Affected runs (1)
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Environment: PROD
Daily Run Outcomes
| Pass rate | Runs | Date (UTC) | Suite | Test |
|---|---|---|---|---|
| 40.00% | 10 | prod/parallel | Customer should be able to issue the default ingress serving certificate from Let's Encrypt using cert-manager with ACME DNS-01 challenges for cluster version ... | |
| 40.00% | 10 | prod/parallel | Customer should be able to issue the default ingress serving certificate from Let's Encrypt using cert-manager with ACME DNS-01 challenges for cluster version ... | |
| 50.00% | 10 | prod/parallel | Customer should be able to issue the default ingress serving certificate from Let's Encrypt using cert-manager with ACME DNS-01 challenges for cluster version ... | |
| 50.00% | 10 | prod/parallel | Customer should be able to issue the default ingress serving certificate from Let's Encrypt using cert-manager with ACME DNS-01 challenges for cluster version ... | |
| 50.00% | 10 | prod/parallel | Customer should be able to issue the default ingress serving certificate from Let's Encrypt using cert-manager with ACME DNS-01 challenges for cluster version ... |
Up to 50 failure patterns are loaded (minimum 1.00% impact), with 10 shown by default. Click column headers to re-sort.
Jump to the full Failure Patterns view for this window
| Failure patternThe canonical description of a recurring CI failure, extracted and normalized from raw logs. | Failed atThe stage of the job run where this failure occurred: 'provision' (environment setup, DEV only), 'e2e' (test suite execution), or 'other' (CI infrastructure issues that did not produce a failure pattern). | Number of distinct job runs where this failure pattern was detected. | Percentage of all job runs in this environment affected by this failure pattern during the selected window. | Classification of this failure pattern: Regression (likely caused by a specific PR), Flake (intermittent failure spread across days), Noise (low-quality or generic pattern), or Indeterminate. | TrendShows daily activity for this failure pattern in a trailing window anchored to the selected end date. The sparkline covers at least 7 days and at most 14 days, depending on the current window size. | Also inOther environments where the same failure pattern was also detected during the selected window. | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
VerifyDefaultIngressServingCertificateSecretWiring timed out after 10m0s: IngressController/default references &LocalObjectReference{Name:cluster-ingress-cert,}, expected secret "...full failure pattern: VerifyDefaultIngressServingCertificateSecretWiring timed out after 10m0s: IngressController/default references &LocalObjectReference{Name:cluster-ingress-cert,}, expected secret "default-ingress-cert"Signal: NoiseSignal: Noise — struct/object fragment | e2e | 1 | 2.17%1 of 46 job runs affected | NoiseSignal: Noise — struct/object fragment | Show trend detailsOct 25: 0 · Oct 26: 1 · Oct 27: 0 · Oct 28: 0 · Oct 29: 0 · Oct 30: 0 · Oct 31: 0 · Nov 1: 0 | none | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:256]: default-ingress-cert must exist and be referenced by IngressController/default for cluster "cluster-nightly-4-23-m28flb"
Expected success, but got an error:
<*fmt.wrapError | 0x27a8bf192000>:
VerifyDefaultIngressServingCertificateSecretWiring timed out after 10m0s: IngressController/default references &LocalObjectReference{Name:cluster-ingress-cert,}, expected secret "default-ingress-cert"
{
msg: "VerifyDefaultIngressServingCertificateSecretWiring timed out after 10m0s: IngressController/default references &LocalObjectReference{Name:cluster-ingress-cert,}, expected secret \"default-ingress-cert\"",
err: <*errors.errorString | 0x27a8c0a02f80>{
s: "IngressController/default references &LocalObjectReference{Name:cluster-ingress-cert,}, expected secret \"default-ingress-cert\"",
},
}
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:256]: default-ingress-cert must exist and be referenced by IngressController/default for cluster "cluster-nightly-4-23-m28flb"
Expected success, but got an error:
<*fmt.wrapError | 0x27a8bf192000>:
VerifyDefaultIngressServingCertificateSecretWiring timed out after 10m0s: IngressController/default references &LocalObjectReference{Name:cluster-ingress-cert,}, expected secret "default-ingress-cert"
{
msg: "VerifyDefaultIngressServingCertificateSecretWiring timed out after 10m0s: IngressController/default references &LocalObjectReference{Name:cluster-ingress-cert,}, expected secret \"default-ingress-cert\"",
err: <*errors.errorString | 0x27a8c0a02f80>{
s: "IngressController/default references &LocalObjectReference{Name:cluster-ingress-cert,}, expected secret \"default-ingress-cert\"",
},
}Contributing tests (1)
Affected runs (1)
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedC...full failure pattern: ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider]; provider Microsoft.RedHatOpenShift Signal: IndeterminateSignal: Indeterminate — seen in 3 prior week(s); active 2 day(s) | e2e | 4 | 8.70%4 of 46 job runs affected | IndeterminateSignal: Indeterminate — seen in 3 prior week(s); active 2 day(s) | Show trend detailsOct 25: 0 · Oct 26: 0 · Oct 27: 2 · Oct 28: 19 · Oct 29: 0 · Oct 30: 0 · Oct 31: 0 · Nov 1: 0 | DEV, INT | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (3)fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:133]: failed to create HCP cluster "cp-ystream-upgrade-4-23-s7k2sd" at install version 4.22
Unexpected error:
<*fmt.wrapError | 0x297062e2e820>:
failed to create HCP cluster cp-ystream-upgrade-4-23-s7k2sd: failed waiting for cluster="cp-ystream-upgrade-4-23-s7k2sd" in resourcegroup="rg-cp-ystream-upgrade-4-23-np7xjvwc8xct" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/5d9c960a-8f4b-476d-bd5b-132c0ce76460
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/5d9c960a-8f4b-476d-bd5b-132c0ce76460",
"name": "5d9c960a-8f4b-476d-bd5b-132c0ce76460",
"status": "Failed",
"startTime": "2026-10-28T20:12:32.825742556Z",
"endTime": "2026-10-28T20:41:35.067951058Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/5d9c960a-8f4b-476d-bd5b-132c0ce76460
correlation ID: 00646159-dae9-4d36-b0f6-38a33eb4750d
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:133]: failed to create HCP cluster "cp-ystream-upgrade-4-23-s7k2sd" at install version 4.22
Unexpected error:
<*fmt.wrapError | 0x297062e2e820>:
failed to create HCP cluster cp-ystream-upgrade-4-23-s7k2sd: failed waiting for cluster="cp-ystream-upgrade-4-23-s7k2sd" in resourcegroup="rg-cp-ystream-upgrade-4-23-np7xjvwc8xct" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/5d9c960a-8f4b-476d-bd5b-132c0ce76460
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/5d9c960a-8f4b-476d-bd5b-132c0ce76460",
"name": "5d9c960a-8f4b-476d-bd5b-132c0ce76460",
"status": "Failed",
"startTime": "2026-10-28T20:12:32.825742556Z",
"endTime": "2026-10-28T20:41:35.067951058Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/5d9c960a-8f4b-476d-bd5b-132c0ce76460
correlation ID: 00646159-dae9-4d36-b0f6-38a33eb4750d
client request ID: unavailable
occurredfail [github.com/Azure/ARO-HCP/test/e2e/cluster_nsg_subnet_reuse.go:107]: failed to create seed HCP cluster basic-cluster
Unexpected error:
<*fmt.wrapError | 0x10164f31ca00>:
failed to create HCP cluster basic-cluster: failed waiting for cluster="basic-cluster" in resourcegroup="rg-cluster-nsg-subnet-reuse-p669mb2thnqj" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/f26dc7e7-f380-4b37-87cb-db720fa2be9b
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/f26dc7e7-f380-4b37-87cb-db720fa2be9b",
"name": "f26dc7e7-f380-4b37-87cb-db720fa2be9b",
"status": "Failed",
"startTime": "2026-10-28T19:27:58.648367476Z",
"endTime": "2026-10-28T19:57:03.816857754Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/f26dc7e7-f380-4b37-87cb-db720fa2be9b
correlation ID: b3870728-f2f7-48de-a0fa-7f5442586990
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_nsg_subnet_reuse.go:107]: failed to create seed HCP cluster basic-cluster
Unexpected error:
<*fmt.wrapError | 0x10164f31ca00>:
failed to create HCP cluster basic-cluster: failed waiting for cluster="basic-cluster" in resourcegroup="rg-cluster-nsg-subnet-reuse-p669mb2thnqj" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/f26dc7e7-f380-4b37-87cb-db720fa2be9b
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/f26dc7e7-f380-4b37-87cb-db720fa2be9b",
"name": "f26dc7e7-f380-4b37-87cb-db720fa2be9b",
"status": "Failed",
"startTime": "2026-10-28T19:27:58.648367476Z",
"endTime": "2026-10-28T19:57:03.816857754Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/f26dc7e7-f380-4b37-87cb-db720fa2be9b
correlation ID: b3870728-f2f7-48de-a0fa-7f5442586990
client request ID: unavailable
occurredfail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:133]: failed to create HCP cluster "cp-ystream-upgrade-4-21-lrsdfl" at install version 4.20
Unexpected error:
<*fmt.wrapError | 0x3c350f225e0>:
failed to create HCP cluster cp-ystream-upgrade-4-21-lrsdfl: failed waiting for cluster="cp-ystream-upgrade-4-21-lrsdfl" in resourcegroup="rg-cp-ystream-upgrade-4-21-vfbxfhgg5xwh" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/034eed87-a977-47cf-8ff5-2b8292aa4616
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/034eed87-a977-47cf-8ff5-2b8292aa4616",
"name": "034eed87-a977-47cf-8ff5-2b8292aa4616",
"status": "Failed",
"startTime": "2026-10-28T20:12:34.208698264Z",
"endTime": "2026-10-28T20:41:35.072616907Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/034eed87-a977-47cf-8ff5-2b8292aa4616
correlation ID: 880b9f59-ff57-4ef3-9ec1-2561f0663b62
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:133]: failed to create HCP cluster "cp-ystream-upgrade-4-21-lrsdfl" at install version 4.20
Unexpected error:
<*fmt.wrapError | 0x3c350f225e0>:
failed to create HCP cluster cp-ystream-upgrade-4-21-lrsdfl: failed waiting for cluster="cp-ystream-upgrade-4-21-lrsdfl" in resourcegroup="rg-cp-ystream-upgrade-4-21-vfbxfhgg5xwh" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/034eed87-a977-47cf-8ff5-2b8292aa4616
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/034eed87-a977-47cf-8ff5-2b8292aa4616",
"name": "034eed87-a977-47cf-8ff5-2b8292aa4616",
"status": "Failed",
"startTime": "2026-10-28T20:12:34.208698264Z",
"endTime": "2026-10-28T20:41:35.072616907Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/034eed87-a977-47cf-8ff5-2b8292aa4616
correlation ID: 880b9f59-ff57-4ef3-9ec1-2561f0663b62
client request ID: unavailable
occurredContributing tests (16)
Affected runs (4)
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedC...full failure pattern: ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider, ignition-server]; provider Microsoft.RedHatOpenShift Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s) | e2e | 2 | 4.35%2 of 46 job runs affected | IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s) | Show trend detailsOct 25: 0 · Oct 26: 0 · Oct 27: 0 · Oct 28: 16 · Oct 29: 0 · Oct 30: 0 · Oct 31: 0 · Nov 1: 0 | none | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (3)fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_cni_cilium.go:88]: failed to create HCP cluster "cilium-cl" without CNI
Unexpected error:
<*fmt.wrapError | 0x22868c3a2660>:
failed to create HCP cluster cilium-cl: failed waiting for cluster="cilium-cl" in resourcegroup="cni-cilium-s6kbbzpqdbx9" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/a285ee7f-e10b-45e8-90d8-9ff129ec2b53
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/a285ee7f-e10b-45e8-90d8-9ff129ec2b53",
"name": "a285ee7f-e10b-45e8-90d8-9ff129ec2b53",
"status": "Failed",
"startTime": "2026-10-28T19:24:09.588021132Z",
"endTime": "2026-10-28T19:53:13.758028769Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider deployment has 1 unavailable replicas, ignition-server deployment has 1 unavailable replicas]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/a285ee7f-e10b-45e8-90d8-9ff129ec2b53
correlation ID: 8f02eb4e-1e36-4dd4-b460-7588fb948e71
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_cni_cilium.go:88]: failed to create HCP cluster "cilium-cl" without CNI
Unexpected error:
<*fmt.wrapError | 0x22868c3a2660>:
failed to create HCP cluster cilium-cl: failed waiting for cluster="cilium-cl" in resourcegroup="cni-cilium-s6kbbzpqdbx9" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/a285ee7f-e10b-45e8-90d8-9ff129ec2b53
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/a285ee7f-e10b-45e8-90d8-9ff129ec2b53",
"name": "a285ee7f-e10b-45e8-90d8-9ff129ec2b53",
"status": "Failed",
"startTime": "2026-10-28T19:24:09.588021132Z",
"endTime": "2026-10-28T19:53:13.758028769Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider deployment has 1 unavailable replicas, ignition-server deployment has 1 unavailable replicas]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/a285ee7f-e10b-45e8-90d8-9ff129ec2b53
correlation ID: 8f02eb4e-1e36-4dd4-b460-7588fb948e71
client request ID: unavailable
occurredfail [github.com/Azure/ARO-HCP/test/e2e/idms_lifecycle.go:113]: failed to create HCP cluster with ImageDigestMirrors
Unexpected error:
<*fmt.wrapError | 0x263f7a1ecb40>:
failed to create HCP cluster idms-e2e-hcp-cluster: failed waiting for cluster="idms-e2e-hcp-cluster" in resourcegroup="idms-s854pt5lzxq5" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/bc13db95-dad4-4bc2-9929-827244e95d2e
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/bc13db95-dad4-4bc2-9929-827244e95d2e",
"name": "bc13db95-dad4-4bc2-9929-827244e95d2e",
"status": "Failed",
"startTime": "2026-10-28T20:14:34.589492538Z",
"endTime": "2026-10-28T20:43:34.88292258Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider deployment has 1 unavailable replicas, ignition-server deployment has 1 unavailable replicas]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/bc13db95-dad4-4bc2-9929-827244e95d2e
correlation ID: da1e3131-8ea4-44cd-87a3-857957002869
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/idms_lifecycle.go:113]: failed to create HCP cluster with ImageDigestMirrors
Unexpected error:
<*fmt.wrapError | 0x263f7a1ecb40>:
failed to create HCP cluster idms-e2e-hcp-cluster: failed waiting for cluster="idms-e2e-hcp-cluster" in resourcegroup="idms-s854pt5lzxq5" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/bc13db95-dad4-4bc2-9929-827244e95d2e
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/bc13db95-dad4-4bc2-9929-827244e95d2e",
"name": "bc13db95-dad4-4bc2-9929-827244e95d2e",
"status": "Failed",
"startTime": "2026-10-28T20:14:34.589492538Z",
"endTime": "2026-10-28T20:43:34.88292258Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider deployment has 1 unavailable replicas, ignition-server deployment has 1 unavailable replicas]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/bc13db95-dad4-4bc2-9929-827244e95d2e
correlation ID: da1e3131-8ea4-44cd-87a3-857957002869
client request ID: unavailable
occurredfail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:133]: failed to create HCP cluster "cp-ystream-upgrade-5-1-clmhlg" at install version 5.0
Unexpected error:
<*fmt.wrapError | 0x2d29037be7e0>:
failed to create HCP cluster cp-ystream-upgrade-5-1-clmhlg: failed waiting for cluster="cp-ystream-upgrade-5-1-clmhlg" in resourcegroup="rg-cp-ystream-upgrade-5-1-m7flj58khr4n" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/75f18780-46df-4ca4-b758-e34d2262ebd4
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/75f18780-46df-4ca4-b758-e34d2262ebd4",
"name": "75f18780-46df-4ca4-b758-e34d2262ebd4",
"status": "Failed",
"startTime": "2026-10-28T20:12:38.563887566Z",
"endTime": "2026-10-28T20:41:44.890635353Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider deployment has 1 unavailable replicas, ignition-server deployment has 1 unavailable replicas]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/75f18780-46df-4ca4-b758-e34d2262ebd4
correlation ID: ae6aa7e5-e845-4182-9588-f52bb0a88718
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:133]: failed to create HCP cluster "cp-ystream-upgrade-5-1-clmhlg" at install version 5.0
Unexpected error:
<*fmt.wrapError | 0x2d29037be7e0>:
failed to create HCP cluster cp-ystream-upgrade-5-1-clmhlg: failed waiting for cluster="cp-ystream-upgrade-5-1-clmhlg" in resourcegroup="rg-cp-ystream-upgrade-5-1-m7flj58khr4n" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/75f18780-46df-4ca4-b758-e34d2262ebd4
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/75f18780-46df-4ca4-b758-e34d2262ebd4",
"name": "75f18780-46df-4ca4-b758-e34d2262ebd4",
"status": "Failed",
"startTime": "2026-10-28T20:12:38.563887566Z",
"endTime": "2026-10-28T20:41:44.890635353Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider deployment has 1 unavailable replicas, ignition-server deployment has 1 unavailable replicas]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/75f18780-46df-4ca4-b758-e34d2262ebd4
correlation ID: ae6aa7e5-e845-4182-9588-f52bb0a88718
client request ID: unavailable
occurredContributing tests (13)
Affected runs (2)
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedC...full failure pattern: ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable; hosted cluster degraded: UnavailableReplicas: multiple components; provider Microsoft.RedHatOpenShift Signal: IndeterminateSignal: Indeterminate — seen in 2 prior week(s); active 1 day(s) | e2e | 2 | 4.35%2 of 46 job runs affected | IndeterminateSignal: Indeterminate — seen in 2 prior week(s); active 1 day(s) | Show trend detailsOct 25: 0 · Oct 26: 0 · Oct 27: 0 · Oct 28: 14 · Oct 29: 0 · Oct 30: 0 · Oct 31: 0 · Nov 1: 0 | none | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (3)fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:154]: HCP cluster rg-candidate-4-20-qtj68sbhkffm/cluster-candidate-4-20-hxsx6r should provision
Unexpected error:
<*fmt.wrapError | 0x15a8256fc4a0>:
failed to create HCP cluster cluster-candidate-4-20-hxsx6r: failed waiting for cluster="cluster-candidate-4-20-hxsx6r" in resourcegroup="rg-candidate-4-20-qtj68sbhkffm" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/93a53e80-e0f1-4f90-b3ef-4df9d072a238
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/93a53e80-e0f1-4f90-b3ef-4df9d072a238",
"name": "93a53e80-e0f1-4f90-b3ef-4df9d072a238",
"status": "Failed",
"startTime": "2026-10-28T20:04:15.708912895Z",
"endTime": "2026-10-28T20:33:16.216619801Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: kube-controller-manager, kube-scheduler, hosted-cluster-config-operator, machine-approver, control-plane-pki-operator, azure-cloud-controller-manager, konnectivity-agent, csi-snapshot-controller-operator, redhat-operators-catalog, certified-operators-catalog, openshift-controller-manager, olm-operator, cluster-storage-operator, redhat-marketplace-catalog, community-operators-catalog, cluster-network-operator, cluster-node-tuning-operator, dns-operator, ingress-operator, cluster-image-registry-operator, packageserver, catalog-operator, cluster-policy-controller, cluster-version-operator, openshift-route-controller-manager; hosted cluster degraded: UnavailableReplicas: [azure-cloud-controller-manager deployment has 1 unavailable replicas, catalog-operator deployment has 1 unavailable replicas, control-plane-pki-operator deployment has 1 unavailable replicas, hosted-cluster-config-operator deployment has 1 unavailable replicas, konnectivity-agent deployment has 1 unavailable replicas, kube-controller-manager deployment has 1 unavailable replicas, kube-scheduler deployment has 1 unavailable replicas, machine-approver deployment has 1 unavailable replicas, olm-operator deployment has 1 unavailable replicas, openshift-apiserver deployment has 1 unavailable replicas, packageserver deployment has 3 unavailable replicas]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/93a53e80-e0f1-4f90-b3ef-4df9d072a238
correlation ID: b596877d-09f6-4478-a504-ebb349eebdd1
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:154]: HCP cluster rg-candidate-4-20-qtj68sbhkffm/cluster-candidate-4-20-hxsx6r should provision
Unexpected error:
<*fmt.wrapError | 0x15a8256fc4a0>:
failed to create HCP cluster cluster-candidate-4-20-hxsx6r: failed waiting for cluster="cluster-candidate-4-20-hxsx6r" in resourcegroup="rg-candidate-4-20-qtj68sbhkffm" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/93a53e80-e0f1-4f90-b3ef-4df9d072a238
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/93a53e80-e0f1-4f90-b3ef-4df9d072a238",
"name": "93a53e80-e0f1-4f90-b3ef-4df9d072a238",
"status": "Failed",
"startTime": "2026-10-28T20:04:15.708912895Z",
"endTime": "2026-10-28T20:33:16.216619801Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: kube-controller-manager, kube-scheduler, hosted-cluster-config-operator, machine-approver, control-plane-pki-operator, azure-cloud-controller-manager, konnectivity-agent, csi-snapshot-controller-operator, redhat-operators-catalog, certified-operators-catalog, openshift-controller-manager, olm-operator, cluster-storage-operator, redhat-marketplace-catalog, community-operators-catalog, cluster-network-operator, cluster-node-tuning-operator, dns-operator, ingress-operator, cluster-image-registry-operator, packageserver, catalog-operator, cluster-policy-controller, cluster-version-operator, openshift-route-controller-manager; hosted cluster degraded: UnavailableReplicas: [azure-cloud-controller-manager deployment has 1 unavailable replicas, catalog-operator deployment has 1 unavailable replicas, control-plane-pki-operator deployment has 1 unavailable replicas, hosted-cluster-config-operator deployment has 1 unavailable replicas, konnectivity-agent deployment has 1 unavailable replicas, kube-controller-manager deployment has 1 unavailable replicas, kube-scheduler deployment has 1 unavailable replicas, machine-approver deployment has 1 unavailable replicas, olm-operator deployment has 1 unavailable replicas, openshift-apiserver deployment has 1 unavailable replicas, packageserver deployment has 3 unavailable replicas]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/93a53e80-e0f1-4f90-b3ef-4df9d072a238
correlation ID: b596877d-09f6-4478-a504-ebb349eebdd1
client request ID: unavailable
occurredfail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:150]: failed to create HCP cluster
Unexpected error:
<*fmt.wrapError | 0xd149dec0ae0>:
failed to create HCP cluster ingress-cert-le: failed waiting for cluster="ingress-cert-le" in resourcegroup="ingress-cert-le-6nnwct2hkjrk" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/89ea4fd7-6047-49a5-afc1-c0d8ef75994d
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/89ea4fd7-6047-49a5-afc1-c0d8ef75994d",
"name": "89ea4fd7-6047-49a5-afc1-c0d8ef75994d",
"status": "Failed",
"startTime": "2026-10-28T19:24:20.936404311Z",
"endTime": "2026-10-28T19:53:23.678300779Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: kube-scheduler, cluster-version-operator, openshift-controller-manager, kube-storage-version-migrator, catalog-operator, azure-cloud-controller-manager, dns-operator, redhat-operators-catalog, cluster-storage-operator, control-plane-pki-operator, openshift-route-controller-manager, hosted-cluster-config-operator, packageserver, konnectivity-agent, cluster-node-tuning-operator, cluster-policy-controller, csi-snapshot-controller-operator, cluster-image-registry-operator, olm-operator, certified-operators-catalog, machine-approver, cluster-network-operator, ingress-operator, community-operators-catalog; hosted cluster degraded: UnavailableReplicas: [azure-cloud-controller-manager deployment has 1 unavailable replicas, catalog-operator deployment has 1 unavailable replicas, control-plane-pki-operator deployment has 1 unavailable replicas, hosted-cluster-config-operator deployment has 1 unavailable replicas, konnectivity-agent deployment has 1 unavailable replicas, kube-scheduler deployment has 1 unavailable replicas, kube-storage-version-migrator deployment has 1 unavailable replicas, machine-approver deployment has 1 unavailable replicas, olm-operator deployment has 1 unavailable replicas, openshift-apiserver deployment has 1 unavailable replicas, packageserver deployment has 3 unavailable replicas]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/89ea4fd7-6047-49a5-afc1-c0d8ef75994d
correlation ID: c78028a7-799f-4fe4-8afb-462431828f6c
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:150]: failed to create HCP cluster
Unexpected error:
<*fmt.wrapError | 0xd149dec0ae0>:
failed to create HCP cluster ingress-cert-le: failed waiting for cluster="ingress-cert-le" in resourcegroup="ingress-cert-le-6nnwct2hkjrk" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/89ea4fd7-6047-49a5-afc1-c0d8ef75994d
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/89ea4fd7-6047-49a5-afc1-c0d8ef75994d",
"name": "89ea4fd7-6047-49a5-afc1-c0d8ef75994d",
"status": "Failed",
"startTime": "2026-10-28T19:24:20.936404311Z",
"endTime": "2026-10-28T19:53:23.678300779Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: kube-scheduler, cluster-version-operator, openshift-controller-manager, kube-storage-version-migrator, catalog-operator, azure-cloud-controller-manager, dns-operator, redhat-operators-catalog, cluster-storage-operator, control-plane-pki-operator, openshift-route-controller-manager, hosted-cluster-config-operator, packageserver, konnectivity-agent, cluster-node-tuning-operator, cluster-policy-controller, csi-snapshot-controller-operator, cluster-image-registry-operator, olm-operator, certified-operators-catalog, machine-approver, cluster-network-operator, ingress-operator, community-operators-catalog; hosted cluster degraded: UnavailableReplicas: [azure-cloud-controller-manager deployment has 1 unavailable replicas, catalog-operator deployment has 1 unavailable replicas, control-plane-pki-operator deployment has 1 unavailable replicas, hosted-cluster-config-operator deployment has 1 unavailable replicas, konnectivity-agent deployment has 1 unavailable replicas, kube-scheduler deployment has 1 unavailable replicas, kube-storage-version-migrator deployment has 1 unavailable replicas, machine-approver deployment has 1 unavailable replicas, olm-operator deployment has 1 unavailable replicas, openshift-apiserver deployment has 1 unavailable replicas, packageserver deployment has 3 unavailable replicas]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/89ea4fd7-6047-49a5-afc1-c0d8ef75994d
correlation ID: c78028a7-799f-4fe4-8afb-462431828f6c
client request ID: unavailable
occurredfail [github.com/Azure/ARO-HCP/test/e2e/clusters_sharing_resgroup.go:131]: failed to wait for first cluster "basic-hcp-cluster" to complete creation (timeout '30.000000' minutes)
Unexpected error:
<*exported.ResponseError | 0x3b3d00e688d0>:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/e764f65a-623b-45b4-b7fa-b7b1ef90a8a0
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/e764f65a-623b-45b4-b7fa-b7b1ef90a8a0",
"name": "e764f65a-623b-45b4-b7fa-b7b1ef90a8a0",
"status": "Failed",
"startTime": "2026-10-28T20:59:31.361013478Z",
"endTime": "2026-10-28T21:28:40.621642045Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: cluster-version-operator, openshift-controller-manager, hosted-cluster-config-operator, packageserver, dns-operator, ingress-operator, olm-operator, certified-operators-catalog, cluster-network-operator, cluster-node-tuning-operator, redhat-operators-catalog, azure-cloud-controller-manager, csi-snapshot-controller-operator, cluster-image-registry-operator, catalog-operator, cluster-policy-controller, control-plane-pki-operator, openshift-route-controller-manager, konnectivity-agent, redhat-marketplace-catalog, community-operators-catalog, cluster-storage-operator, machine-approver; hosted cluster degraded: UnavailableReplicas: [azure-cloud-controller-manager deployment has 1 unavailable replicas, catalog-operator deployment has 1 unavailable replicas, control-plane-pki-operator deployment has 1 unavailable replicas, hosted-cluster-config-operator deployment has 1 unavailable replicas, konnectivity-agent deployment has 1 unavailable replicas, kube-scheduler deployment has 1 unavailable replicas, machine-approver deployment has 1 unavailable replicas, olm-operator deployment has 1 unavailable replicas, openshift-apiserver deployment has 1 unavailable replicas, packageserver deployment has 3 unavailable replicas]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/e764f65a-623b-45b4-b7fa-b7b1ef90a8a0
correlation ID: caf44bbd-ef58-4e77-8074-5770ca1391e6
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/clusters_sharing_resgroup.go:131]: failed to wait for first cluster "basic-hcp-cluster" to complete creation (timeout '30.000000' minutes)
Unexpected error:
<*exported.ResponseError | 0x3b3d00e688d0>:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/e764f65a-623b-45b4-b7fa-b7b1ef90a8a0
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/e764f65a-623b-45b4-b7fa-b7b1ef90a8a0",
"name": "e764f65a-623b-45b4-b7fa-b7b1ef90a8a0",
"status": "Failed",
"startTime": "2026-10-28T20:59:31.361013478Z",
"endTime": "2026-10-28T21:28:40.621642045Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: cluster-version-operator, openshift-controller-manager, hosted-cluster-config-operator, packageserver, dns-operator, ingress-operator, olm-operator, certified-operators-catalog, cluster-network-operator, cluster-node-tuning-operator, redhat-operators-catalog, azure-cloud-controller-manager, csi-snapshot-controller-operator, cluster-image-registry-operator, catalog-operator, cluster-policy-controller, control-plane-pki-operator, openshift-route-controller-manager, konnectivity-agent, redhat-marketplace-catalog, community-operators-catalog, cluster-storage-operator, machine-approver; hosted cluster degraded: UnavailableReplicas: [azure-cloud-controller-manager deployment has 1 unavailable replicas, catalog-operator deployment has 1 unavailable replicas, control-plane-pki-operator deployment has 1 unavailable replicas, hosted-cluster-config-operator deployment has 1 unavailable replicas, konnectivity-agent deployment has 1 unavailable replicas, kube-scheduler deployment has 1 unavailable replicas, machine-approver deployment has 1 unavailable replicas, olm-operator deployment has 1 unavailable replicas, openshift-apiserver deployment has 1 unavailable replicas, packageserver deployment has 3 unavailable replicas]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/e764f65a-623b-45b4-b7fa-b7b1ef90a8a0
correlation ID: caf44bbd-ef58-4e77-8074-5770ca1391e6
client request ID: unavailable
occurredContributing tests (11)
Affected runs (2)
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedC...full failure pattern: ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider, router]; provider Microsoft.RedHatOpenShift Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s) | e2e | 2 | 4.35%2 of 46 job runs affected | IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s) | Show trend detailsOct 25: 0 · Oct 26: 0 · Oct 27: 0 · Oct 28: 12 · Oct 29: 0 · Oct 30: 0 · Oct 31: 0 · Nov 1: 0 | none | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (3)fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:154]: HCP cluster rg-candidate-4-23-jshv8zgq8wk9/cluster-candidate-4-23-c6gbbp should provision
Unexpected error:
<*fmt.wrapError | 0xba6b52aa80>:
failed to create HCP cluster cluster-candidate-4-23-c6gbbp: failed waiting for cluster="cluster-candidate-4-23-c6gbbp" in resourcegroup="rg-candidate-4-23-jshv8zgq8wk9" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/7a88bbad-1a89-47a4-a4f2-a1b36a2086dd
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/7a88bbad-1a89-47a4-a4f2-a1b36a2086dd",
"name": "7a88bbad-1a89-47a4-a4f2-a1b36a2086dd",
"status": "Failed",
"startTime": "2026-10-28T20:10:20.02531103Z",
"endTime": "2026-10-28T20:39:25.022579343Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider deployment has 1 unavailable replicas, router deployment has 1 unavailable replicas]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/7a88bbad-1a89-47a4-a4f2-a1b36a2086dd
correlation ID: 15324628-570b-42b9-80a7-dbd435939211
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:154]: HCP cluster rg-candidate-4-23-jshv8zgq8wk9/cluster-candidate-4-23-c6gbbp should provision
Unexpected error:
<*fmt.wrapError | 0xba6b52aa80>:
failed to create HCP cluster cluster-candidate-4-23-c6gbbp: failed waiting for cluster="cluster-candidate-4-23-c6gbbp" in resourcegroup="rg-candidate-4-23-jshv8zgq8wk9" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/7a88bbad-1a89-47a4-a4f2-a1b36a2086dd
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/7a88bbad-1a89-47a4-a4f2-a1b36a2086dd",
"name": "7a88bbad-1a89-47a4-a4f2-a1b36a2086dd",
"status": "Failed",
"startTime": "2026-10-28T20:10:20.02531103Z",
"endTime": "2026-10-28T20:39:25.022579343Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider deployment has 1 unavailable replicas, router deployment has 1 unavailable replicas]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/7a88bbad-1a89-47a4-a4f2-a1b36a2086dd
correlation ID: 15324628-570b-42b9-80a7-dbd435939211
client request ID: unavailable
occurredfail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:154]: HCP cluster rg-candidate-5-0-k2brb9cchnfb/cluster-candidate-5-0-c64hdj should provision
Unexpected error:
<*fmt.wrapError | 0x104ef3fa7e0>:
failed to create HCP cluster cluster-candidate-5-0-c64hdj: failed waiting for cluster="cluster-candidate-5-0-c64hdj" in resourcegroup="rg-candidate-5-0-k2brb9cchnfb" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/4e3df969-37c8-42e0-bc9c-a513f08b762b
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/4e3df969-37c8-42e0-bc9c-a513f08b762b",
"name": "4e3df969-37c8-42e0-bc9c-a513f08b762b",
"status": "Failed",
"startTime": "2026-10-28T20:10:48.049950576Z",
"endTime": "2026-10-28T20:39:55.034999436Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider deployment has 1 unavailable replicas, router deployment has 1 unavailable replicas]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/4e3df969-37c8-42e0-bc9c-a513f08b762b
correlation ID: b9bdaba2-ec65-4fd2-abc1-4305bb14fac0
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:154]: HCP cluster rg-candidate-5-0-k2brb9cchnfb/cluster-candidate-5-0-c64hdj should provision
Unexpected error:
<*fmt.wrapError | 0x104ef3fa7e0>:
failed to create HCP cluster cluster-candidate-5-0-c64hdj: failed waiting for cluster="cluster-candidate-5-0-c64hdj" in resourcegroup="rg-candidate-5-0-k2brb9cchnfb" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/4e3df969-37c8-42e0-bc9c-a513f08b762b
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/4e3df969-37c8-42e0-bc9c-a513f08b762b",
"name": "4e3df969-37c8-42e0-bc9c-a513f08b762b",
"status": "Failed",
"startTime": "2026-10-28T20:10:48.049950576Z",
"endTime": "2026-10-28T20:39:55.034999436Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider deployment has 1 unavailable replicas, router deployment has 1 unavailable replicas]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/4e3df969-37c8-42e0-bc9c-a513f08b762b
correlation ID: b9bdaba2-ec65-4fd2-abc1-4305bb14fac0
client request ID: unavailable
occurredfail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:106]: failed to create HCP cluster "cilium-cluster" with no CNI and private etcd
Unexpected error:
<*fmt.wrapError | 0x1da2d78427a0>:
failed waiting for cluster="cilium-cluster" in resourcegroup="complex-cilium-kv-r4m67m749clh" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/ae56e62c-ca10-46a7-88ef-7fb6101ba7f0
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/ae56e62c-ca10-46a7-88ef-7fb6101ba7f0",
"name": "ae56e62c-ca10-46a7-88ef-7fb6101ba7f0",
"status": "Failed",
"startTime": "2026-10-28T19:26:05.081696809Z",
"endTime": "2026-10-28T19:55:13.864538233Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider deployment has 1 unavailable replicas, router deployment has 1 unavailable replicas]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/ae56e62c-ca10-46a7-88ef-7fb6101ba7f0
correlation ID: 259a0ca4-d067-4227-b0b5-f5f22a7f7fe0
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:106]: failed to create HCP cluster "cilium-cluster" with no CNI and private etcd
Unexpected error:
<*fmt.wrapError | 0x1da2d78427a0>:
failed waiting for cluster="cilium-cluster" in resourcegroup="complex-cilium-kv-r4m67m749clh" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/ae56e62c-ca10-46a7-88ef-7fb6101ba7f0
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/ae56e62c-ca10-46a7-88ef-7fb6101ba7f0",
"name": "ae56e62c-ca10-46a7-88ef-7fb6101ba7f0",
"status": "Failed",
"startTime": "2026-10-28T19:26:05.081696809Z",
"endTime": "2026-10-28T19:55:13.864538233Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider deployment has 1 unavailable replicas, router deployment has 1 unavailable replicas]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/ae56e62c-ca10-46a7-88ef-7fb6101ba7f0
correlation ID: 259a0ca4-d067-4227-b0b5-f5f22a7f7fe0
client request ID: unavailable
occurredContributing tests (9)
Affected runs (2)
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
ACM ingress secret openshift-ingress/cluster-ingress-cert exists; expected ACM certificate delivery to be suppressedfull failure pattern: ACM ingress secret openshift-ingress/cluster-ingress-cert exists; expected ACM certificate delivery to be suppressed Signal: IndeterminateSignal: Indeterminate — seen in 2 prior week(s); active 2 day(s) | e2e | 2 | 4.35%2 of 46 job runs affected | IndeterminateSignal: Indeterminate — seen in 2 prior week(s); active 2 day(s) | Show trend detailsOct 25: 0 · Oct 26: 5 · Oct 27: 3 · Oct 28: 0 · Oct 29: 0 · Oct 30: 0 · Oct 31: 0 · Nov 1: 0 | STG | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (3)fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:260]: ACM must not create its ingress certificate secret for cluster "cluster-candidate-4-23-lkx8t5"
Expected success, but got an error:
<*errors.errorString | 0x3306da82b390>:
ACM ingress secret openshift-ingress/cluster-ingress-cert exists; expected ACM certificate delivery to be suppressed
{
s: "ACM ingress secret openshift-ingress/cluster-ingress-cert exists; expected ACM certificate delivery to be suppressed",
}
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:260]: ACM must not create its ingress certificate secret for cluster "cluster-candidate-4-23-lkx8t5"
Expected success, but got an error:
<*errors.errorString | 0x3306da82b390>:
ACM ingress secret openshift-ingress/cluster-ingress-cert exists; expected ACM certificate delivery to be suppressed
{
s: "ACM ingress secret openshift-ingress/cluster-ingress-cert exists; expected ACM certificate delivery to be suppressed",
}fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:528]: ACM must not create its ingress certificate secret
Expected success, but got an error:
<*errors.errorString | 0xdf906826d40>:
ACM ingress secret openshift-ingress/cluster-ingress-cert exists; expected ACM certificate delivery to be suppressed
{
s: "ACM ingress secret openshift-ingress/cluster-ingress-cert exists; expected ACM certificate delivery to be suppressed",
}
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:528]: ACM must not create its ingress certificate secret
Expected success, but got an error:
<*errors.errorString | 0xdf906826d40>:
ACM ingress secret openshift-ingress/cluster-ingress-cert exists; expected ACM certificate delivery to be suppressed
{
s: "ACM ingress secret openshift-ingress/cluster-ingress-cert exists; expected ACM certificate delivery to be suppressed",
}fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:528]: ACM must not create its ingress certificate secret
Expected success, but got an error:
<*errors.errorString | 0x3454a17e2bc0>:
ACM ingress secret openshift-ingress/cluster-ingress-cert exists; expected ACM certificate delivery to be suppressed
{
s: "ACM ingress secret openshift-ingress/cluster-ingress-cert exists; expected ACM certificate delivery to be suppressed",
}
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:528]: ACM must not create its ingress certificate secret
Expected success, but got an error:
<*errors.errorString | 0x3454a17e2bc0>:
ACM ingress secret openshift-ingress/cluster-ingress-cert exists; expected ACM certificate delivery to be suppressed
{
s: "ACM ingress secret openshift-ingress/cluster-ingress-cert exists; expected ACM certificate delivery to be suppressed",
}Contributing tests (8)
Affected runs (2)
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedC...full failure pattern: ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable; hosted cluster degraded: UnavailableReplicas: [kube-apiserver]; provider Microsoft.RedHatOpenShift Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s) | e2e | 2 | 4.35%2 of 46 job runs affected | IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s) | Show trend detailsOct 25: 0 · Oct 26: 0 · Oct 27: 0 · Oct 28: 7 · Oct 29: 0 · Oct 30: 0 · Oct 31: 0 · Nov 1: 0 | none | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (3)fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_list_and_verify.go:87]: failed to create HCP cluster for external auth list test
Unexpected error:
<*fmt.wrapError | 0x34cbd3d68900>:
failed to create HCP cluster ea-list8sj7pd: failed waiting for cluster="ea-list8sj7pd" in resourcegroup="ea-list-qk95gg7gpcnb" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/91a3c445-9466-4d25-8cd8-de90f532d7c6
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/91a3c445-9466-4d25-8cd8-de90f532d7c6",
"name": "91a3c445-9466-4d25-8cd8-de90f532d7c6",
"status": "Failed",
"startTime": "2026-10-28T20:14:32.731497539Z",
"endTime": "2026-10-28T20:43:34.905565303Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: openshift-route-controller-manager, hosted-cluster-config-operator, konnectivity-agent, machine-approver, olm-collect-profiles, community-operators-catalog, openshift-apiserver, cluster-version-operator, cluster-storage-operator, redhat-operators-catalog, redhat-marketplace-catalog, certified-operators-catalog, csi-snapshot-controller-operator, cluster-autoscaler, control-plane-pki-operator, kube-controller-manager, kube-scheduler, ingress-operator, cluster-image-registry-operator, packageserver, openshift-controller-manager, azure-cloud-controller-manager, cluster-network-operator, dns-operator, catalog-operator, cluster-policy-controller, cluster-node-tuning-operator, olm-operator; hosted cluster degraded: UnavailableReplicas: kube-apiserver deployment has 1 unavailable replicas"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/91a3c445-9466-4d25-8cd8-de90f532d7c6
correlation ID: 92e26e96-cd60-4139-8146-f029c76dbd9e
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_list_and_verify.go:87]: failed to create HCP cluster for external auth list test
Unexpected error:
<*fmt.wrapError | 0x34cbd3d68900>:
failed to create HCP cluster ea-list8sj7pd: failed waiting for cluster="ea-list8sj7pd" in resourcegroup="ea-list-qk95gg7gpcnb" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/91a3c445-9466-4d25-8cd8-de90f532d7c6
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/91a3c445-9466-4d25-8cd8-de90f532d7c6",
"name": "91a3c445-9466-4d25-8cd8-de90f532d7c6",
"status": "Failed",
"startTime": "2026-10-28T20:14:32.731497539Z",
"endTime": "2026-10-28T20:43:34.905565303Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: openshift-route-controller-manager, hosted-cluster-config-operator, konnectivity-agent, machine-approver, olm-collect-profiles, community-operators-catalog, openshift-apiserver, cluster-version-operator, cluster-storage-operator, redhat-operators-catalog, redhat-marketplace-catalog, certified-operators-catalog, csi-snapshot-controller-operator, cluster-autoscaler, control-plane-pki-operator, kube-controller-manager, kube-scheduler, ingress-operator, cluster-image-registry-operator, packageserver, openshift-controller-manager, azure-cloud-controller-manager, cluster-network-operator, dns-operator, catalog-operator, cluster-policy-controller, cluster-node-tuning-operator, olm-operator; hosted cluster degraded: UnavailableReplicas: kube-apiserver deployment has 1 unavailable replicas"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/91a3c445-9466-4d25-8cd8-de90f532d7c6
correlation ID: 92e26e96-cd60-4139-8146-f029c76dbd9e
client request ID: unavailable
occurredfail [github.com/Azure/ARO-HCP/test/e2e/shoebox_logs.go:272]: failed to create HCP cluster shoebox-hcp-cluster
Unexpected error:
<*fmt.wrapError | 0x118c82e68960>:
failed to create HCP cluster shoebox-hcp-cluster: failed waiting for cluster="shoebox-hcp-cluster" in resourcegroup="shoebox-cluster-5fcqfhg6xm85" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/241d3860-c6a5-496d-9a47-0ba2f3c27d4f
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/241d3860-c6a5-496d-9a47-0ba2f3c27d4f",
"name": "241d3860-c6a5-496d-9a47-0ba2f3c27d4f",
"status": "Failed",
"startTime": "2026-10-28T20:58:03.268943167Z",
"endTime": "2026-10-28T21:27:10.630246232Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: hosted-cluster-config-operator, catalog-operator, dns-operator, olm-operator, redhat-marketplace-catalog, csi-snapshot-controller-operator, cluster-image-registry-operator, konnectivity-agent, community-operators-catalog, control-plane-pki-operator, kube-controller-manager, cluster-policy-controller, packageserver, machine-approver, openshift-controller-manager, cluster-storage-operator, olm-collect-profiles, redhat-operators-catalog, cluster-autoscaler, cluster-version-operator, openshift-route-controller-manager, azure-cloud-controller-manager, kube-scheduler, cluster-network-operator, cluster-node-tuning-operator, ingress-operator, certified-operators-catalog, openshift-apiserver; hosted cluster degraded: UnavailableReplicas: kube-apiserver deployment has 1 unavailable replicas"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/241d3860-c6a5-496d-9a47-0ba2f3c27d4f
correlation ID: 3553f468-7cef-45de-9639-6e7d0ba9a86e
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/shoebox_logs.go:272]: failed to create HCP cluster shoebox-hcp-cluster
Unexpected error:
<*fmt.wrapError | 0x118c82e68960>:
failed to create HCP cluster shoebox-hcp-cluster: failed waiting for cluster="shoebox-hcp-cluster" in resourcegroup="shoebox-cluster-5fcqfhg6xm85" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/241d3860-c6a5-496d-9a47-0ba2f3c27d4f
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/241d3860-c6a5-496d-9a47-0ba2f3c27d4f",
"name": "241d3860-c6a5-496d-9a47-0ba2f3c27d4f",
"status": "Failed",
"startTime": "2026-10-28T20:58:03.268943167Z",
"endTime": "2026-10-28T21:27:10.630246232Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: hosted-cluster-config-operator, catalog-operator, dns-operator, olm-operator, redhat-marketplace-catalog, csi-snapshot-controller-operator, cluster-image-registry-operator, konnectivity-agent, community-operators-catalog, control-plane-pki-operator, kube-controller-manager, cluster-policy-controller, packageserver, machine-approver, openshift-controller-manager, cluster-storage-operator, olm-collect-profiles, redhat-operators-catalog, cluster-autoscaler, cluster-version-operator, openshift-route-controller-manager, azure-cloud-controller-manager, kube-scheduler, cluster-network-operator, cluster-node-tuning-operator, ingress-operator, certified-operators-catalog, openshift-apiserver; hosted cluster degraded: UnavailableReplicas: kube-apiserver deployment has 1 unavailable replicas"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/241d3860-c6a5-496d-9a47-0ba2f3c27d4f
correlation ID: 3553f468-7cef-45de-9639-6e7d0ba9a86e
client request ID: unavailable
occurredfail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:154]: HCP cluster rg-candidate-4-21-mjsj484l4cg7/cluster-candidate-4-21-zqjdxv should provision
Unexpected error:
<*fmt.wrapError | 0x3e94b832e820>:
failed to create HCP cluster cluster-candidate-4-21-zqjdxv: failed waiting for cluster="cluster-candidate-4-21-zqjdxv" in resourcegroup="rg-candidate-4-21-mjsj484l4cg7" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/e7fb8359-1343-4d42-8975-4e247aae3bc5
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/e7fb8359-1343-4d42-8975-4e247aae3bc5",
"name": "e7fb8359-1343-4d42-8975-4e247aae3bc5",
"status": "Failed",
"startTime": "2026-10-28T20:06:06.856819198Z",
"endTime": "2026-10-28T20:35:15.000751357Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: openshift-controller-manager, cluster-node-tuning-operator, cluster-autoscaler, machine-approver, control-plane-pki-operator, openshift-apiserver, cluster-storage-operator, community-operators-catalog, ingress-operator, olm-operator, cluster-network-operator, packageserver, redhat-marketplace-catalog, cluster-version-operator, catalog-operator, certified-operators-catalog, cluster-policy-controller, dns-operator, konnectivity-agent, redhat-operators-catalog, kube-controller-manager, kube-scheduler, azure-cloud-controller-manager, cluster-image-registry-operator, olm-collect-profiles, openshift-route-controller-manager, hosted-cluster-config-operator, csi-snapshot-controller-operator; hosted cluster degraded: UnavailableReplicas: kube-apiserver deployment has 1 unavailable replicas"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/e7fb8359-1343-4d42-8975-4e247aae3bc5
correlation ID: 0c3580d1-f96a-47b1-aaec-1d49dda7d990
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:154]: HCP cluster rg-candidate-4-21-mjsj484l4cg7/cluster-candidate-4-21-zqjdxv should provision
Unexpected error:
<*fmt.wrapError | 0x3e94b832e820>:
failed to create HCP cluster cluster-candidate-4-21-zqjdxv: failed waiting for cluster="cluster-candidate-4-21-zqjdxv" in resourcegroup="rg-candidate-4-21-mjsj484l4cg7" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/e7fb8359-1343-4d42-8975-4e247aae3bc5
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/e7fb8359-1343-4d42-8975-4e247aae3bc5",
"name": "e7fb8359-1343-4d42-8975-4e247aae3bc5",
"status": "Failed",
"startTime": "2026-10-28T20:06:06.856819198Z",
"endTime": "2026-10-28T20:35:15.000751357Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: openshift-controller-manager, cluster-node-tuning-operator, cluster-autoscaler, machine-approver, control-plane-pki-operator, openshift-apiserver, cluster-storage-operator, community-operators-catalog, ingress-operator, olm-operator, cluster-network-operator, packageserver, redhat-marketplace-catalog, cluster-version-operator, catalog-operator, certified-operators-catalog, cluster-policy-controller, dns-operator, konnectivity-agent, redhat-operators-catalog, kube-controller-manager, kube-scheduler, azure-cloud-controller-manager, cluster-image-registry-operator, olm-collect-profiles, openshift-route-controller-manager, hosted-cluster-config-operator, csi-snapshot-controller-operator; hosted cluster degraded: UnavailableReplicas: kube-apiserver deployment has 1 unavailable replicas"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/e7fb8359-1343-4d42-8975-4e247aae3bc5
correlation ID: 0c3580d1-f96a-47b1-aaec-1d49dda7d990
client request ID: unavailable
occurredContributing tests (6)
Affected runs (2)
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedC...full failure pattern: ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable; hosted cluster degraded: UnavailableReplicas: [capi-provider, kube-apiserver]; provider Microsoft.RedHatOpenShift Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s) | e2e | 2 | 4.35%2 of 46 job runs affected | IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s) | Show trend detailsOct 25: 0 · Oct 26: 0 · Oct 27: 0 · Oct 28: 7 · Oct 29: 0 · Oct 30: 0 · Oct 31: 0 · Nov 1: 0 | none | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (3)fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_ingress.go:87]: failed to create HCP cluster "private-ingress" with private ingress
Unexpected error:
<*fmt.wrapError | 0x1d67385a4720>:
failed to create HCP cluster private-ingress: failed waiting for cluster="private-ingress" in resourcegroup="private-ingress-292fs2ltfn6d" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/de6ffede-1c17-4a98-a65e-6e4b517b05bf
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/de6ffede-1c17-4a98-a65e-6e4b517b05bf",
"name": "de6ffede-1c17-4a98-a65e-6e4b517b05bf",
"status": "Failed",
"startTime": "2026-10-28T19:24:18.787179838Z",
"endTime": "2026-10-28T19:53:23.770490111Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: control-plane-pki-operator, kube-controller-manager, catalog-operator, cluster-policy-controller, cluster-node-tuning-operator, ingress-operator, redhat-operators-catalog, redhat-marketplace-catalog, capi-provider, cluster-network-operator, csi-snapshot-controller-operator, konnectivity-agent, community-operators-catalog, cluster-storage-operator, olm-collect-profiles, certified-operators-catalog, hosted-cluster-config-operator, cluster-image-registry-operator, kube-scheduler, openshift-apiserver, cluster-version-operator, openshift-route-controller-manager, azure-cloud-controller-manager, dns-operator, olm-operator, openshift-controller-manager, packageserver, cluster-autoscaler, machine-approver; hosted cluster degraded: UnavailableReplicas: [capi-provider deployment has 2 unavailable replicas, kube-apiserver deployment has 1 unavailable replicas]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/de6ffede-1c17-4a98-a65e-6e4b517b05bf
correlation ID: 5f8d39c8-29ad-4bdc-8281-5d6497fadbe8
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_ingress.go:87]: failed to create HCP cluster "private-ingress" with private ingress
Unexpected error:
<*fmt.wrapError | 0x1d67385a4720>:
failed to create HCP cluster private-ingress: failed waiting for cluster="private-ingress" in resourcegroup="private-ingress-292fs2ltfn6d" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/de6ffede-1c17-4a98-a65e-6e4b517b05bf
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/de6ffede-1c17-4a98-a65e-6e4b517b05bf",
"name": "de6ffede-1c17-4a98-a65e-6e4b517b05bf",
"status": "Failed",
"startTime": "2026-10-28T19:24:18.787179838Z",
"endTime": "2026-10-28T19:53:23.770490111Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: control-plane-pki-operator, kube-controller-manager, catalog-operator, cluster-policy-controller, cluster-node-tuning-operator, ingress-operator, redhat-operators-catalog, redhat-marketplace-catalog, capi-provider, cluster-network-operator, csi-snapshot-controller-operator, konnectivity-agent, community-operators-catalog, cluster-storage-operator, olm-collect-profiles, certified-operators-catalog, hosted-cluster-config-operator, cluster-image-registry-operator, kube-scheduler, openshift-apiserver, cluster-version-operator, openshift-route-controller-manager, azure-cloud-controller-manager, dns-operator, olm-operator, openshift-controller-manager, packageserver, cluster-autoscaler, machine-approver; hosted cluster degraded: UnavailableReplicas: [capi-provider deployment has 2 unavailable replicas, kube-apiserver deployment has 1 unavailable replicas]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/de6ffede-1c17-4a98-a65e-6e4b517b05bf
correlation ID: 5f8d39c8-29ad-4bdc-8281-5d6497fadbe8
client request ID: unavailable
occurredfail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_v20260901preview.go:86]: failed to create HCP cluster "v20260901" via v20260901preview
Unexpected error:
<*fmt.wrapError | 0x23a60cd6e680>:
failed to create HCP cluster v20260901: failed waiting for cluster="v20260901" in resourcegroup="v20260901preview-ggtvxjcb6gls" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/6b3c6bdf-43a8-4a44-8c7a-90f60ddcdcbc
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/6b3c6bdf-43a8-4a44-8c7a-90f60ddcdcbc",
"name": "6b3c6bdf-43a8-4a44-8c7a-90f60ddcdcbc",
"status": "Failed",
"startTime": "2026-10-28T19:24:25.094565507Z",
"endTime": "2026-10-28T19:53:33.692195544Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: catalog-operator, cluster-autoscaler, olm-collect-profiles, redhat-operators-catalog, kube-scheduler, ingress-operator, cluster-policy-controller, cluster-network-operator, cluster-image-registry-operator, olm-operator, capi-provider, openshift-apiserver, csi-snapshot-controller-operator, redhat-marketplace-catalog, certified-operators-catalog, cluster-version-operator, azure-cloud-controller-manager, dns-operator, kube-controller-manager, openshift-controller-manager, hosted-cluster-config-operator, packageserver, openshift-route-controller-manager, cluster-storage-operator, cluster-node-tuning-operator, community-operators-catalog, machine-approver, control-plane-pki-operator, konnectivity-agent; hosted cluster degraded: UnavailableReplicas: [capi-provider deployment has 2 unavailable replicas, kube-apiserver deployment has 1 unavailable replicas]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/6b3c6bdf-43a8-4a44-8c7a-90f60ddcdcbc
correlation ID: fb16cf1e-d070-4dc2-b0ad-1f7ce0d9da44
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_v20260901preview.go:86]: failed to create HCP cluster "v20260901" via v20260901preview
Unexpected error:
<*fmt.wrapError | 0x23a60cd6e680>:
failed to create HCP cluster v20260901: failed waiting for cluster="v20260901" in resourcegroup="v20260901preview-ggtvxjcb6gls" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/6b3c6bdf-43a8-4a44-8c7a-90f60ddcdcbc
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/6b3c6bdf-43a8-4a44-8c7a-90f60ddcdcbc",
"name": "6b3c6bdf-43a8-4a44-8c7a-90f60ddcdcbc",
"status": "Failed",
"startTime": "2026-10-28T19:24:25.094565507Z",
"endTime": "2026-10-28T19:53:33.692195544Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: catalog-operator, cluster-autoscaler, olm-collect-profiles, redhat-operators-catalog, kube-scheduler, ingress-operator, cluster-policy-controller, cluster-network-operator, cluster-image-registry-operator, olm-operator, capi-provider, openshift-apiserver, csi-snapshot-controller-operator, redhat-marketplace-catalog, certified-operators-catalog, cluster-version-operator, azure-cloud-controller-manager, dns-operator, kube-controller-manager, openshift-controller-manager, hosted-cluster-config-operator, packageserver, openshift-route-controller-manager, cluster-storage-operator, cluster-node-tuning-operator, community-operators-catalog, machine-approver, control-plane-pki-operator, konnectivity-agent; hosted cluster degraded: UnavailableReplicas: [capi-provider deployment has 2 unavailable replicas, kube-apiserver deployment has 1 unavailable replicas]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/6b3c6bdf-43a8-4a44-8c7a-90f60ddcdcbc
correlation ID: fb16cf1e-d070-4dc2-b0ad-1f7ce0d9da44
client request ID: unavailable
occurredfail [github.com/Azure/ARO-HCP/test/e2e/cluster_delete_missing_identities.go:114]: failed to create HCP cluster "missing-mi-hcp-cluster"
Unexpected error:
<*fmt.wrapError | 0x34142edd6a40>:
failed to create HCP cluster missing-mi-hcp-cluster: failed waiting for cluster="missing-mi-hcp-cluster" in resourcegroup="delete-mi-rg-7s64tm5p2pnw" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/f063aff6-fa6b-4397-9b14-f08322d800e8
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/f063aff6-fa6b-4397-9b14-f08322d800e8",
"name": "f063aff6-fa6b-4397-9b14-f08322d800e8",
"status": "Failed",
"startTime": "2026-10-28T19:24:14.723008935Z",
"endTime": "2026-10-28T19:53:23.809094912Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: catalog-operator, capi-provider, cluster-version-operator, cluster-storage-operator, cluster-network-operator, cluster-image-registry-operator, olm-collect-profiles, openshift-controller-manager, cluster-node-tuning-operator, dns-operator, packageserver, cluster-autoscaler, azure-cloud-controller-manager, olm-operator, redhat-marketplace-catalog, kube-controller-manager, kube-scheduler, openshift-apiserver, openshift-route-controller-manager, cluster-policy-controller, community-operators-catalog, redhat-operators-catalog, control-plane-pki-operator, konnectivity-agent, certified-operators-catalog, hosted-cluster-config-operator, ingress-operator, csi-snapshot-controller-operator, machine-approver; hosted cluster degraded: UnavailableReplicas: [capi-provider deployment has 2 unavailable replicas, kube-apiserver deployment has 1 unavailable replicas]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/f063aff6-fa6b-4397-9b14-f08322d800e8
correlation ID: 9da90882-319b-4147-94e3-9897b59ff90d
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delete_missing_identities.go:114]: failed to create HCP cluster "missing-mi-hcp-cluster"
Unexpected error:
<*fmt.wrapError | 0x34142edd6a40>:
failed to create HCP cluster missing-mi-hcp-cluster: failed waiting for cluster="missing-mi-hcp-cluster" in resourcegroup="delete-mi-rg-7s64tm5p2pnw" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/f063aff6-fa6b-4397-9b14-f08322d800e8
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/f063aff6-fa6b-4397-9b14-f08322d800e8",
"name": "f063aff6-fa6b-4397-9b14-f08322d800e8",
"status": "Failed",
"startTime": "2026-10-28T19:24:14.723008935Z",
"endTime": "2026-10-28T19:53:23.809094912Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: catalog-operator, capi-provider, cluster-version-operator, cluster-storage-operator, cluster-network-operator, cluster-image-registry-operator, olm-collect-profiles, openshift-controller-manager, cluster-node-tuning-operator, dns-operator, packageserver, cluster-autoscaler, azure-cloud-controller-manager, olm-operator, redhat-marketplace-catalog, kube-controller-manager, kube-scheduler, openshift-apiserver, openshift-route-controller-manager, cluster-policy-controller, community-operators-catalog, redhat-operators-catalog, control-plane-pki-operator, konnectivity-agent, certified-operators-catalog, hosted-cluster-config-operator, ingress-operator, csi-snapshot-controller-operator, machine-approver; hosted cluster degraded: UnavailableReplicas: [capi-provider deployment has 2 unavailable replicas, kube-apiserver deployment has 1 unavailable replicas]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/f063aff6-fa6b-4397-9b14-f08322d800e8
correlation ID: 9da90882-319b-4147-94e3-9897b59ff90d
client request ID: unavailable
occurredContributing tests (7)
Affected runs (2)
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedC...full failure pattern: ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable; hosted cluster degraded: UnavailableReplicas: [catalog-operator, hosted-cluster-config-operator, olm-operator, openshift-apiserver, packageserver]; provider Microsoft.RedHatOpenShift Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s) | e2e | 2 | 4.35%2 of 46 job runs affected | IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s) | Show trend detailsOct 25: 0 · Oct 26: 0 · Oct 27: 0 · Oct 28: 7 · Oct 29: 0 · Oct 30: 0 · Oct 31: 0 · Nov 1: 0 | none | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (3)fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:102]: failed to create HCP cluster np-autoscale-cluster
Unexpected error:
<*fmt.wrapError | 0x2a9a9c302780>:
failed to create HCP cluster np-autoscale-cluster: failed waiting for cluster="np-autoscale-cluster" in resourcegroup="np-autoscaling-dq6rz78qbzwk" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/36e9fb0c-0ea6-49be-917a-e6ad54933e8b
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/36e9fb0c-0ea6-49be-917a-e6ad54933e8b",
"name": "36e9fb0c-0ea6-49be-917a-e6ad54933e8b",
"status": "Failed",
"startTime": "2026-10-28T20:14:46.976794807Z",
"endTime": "2026-10-28T20:43:55.053369978Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: olm-operator, cluster-network-operator, ingress-operator, community-operators-catalog, dns-operator, packageserver, openshift-controller-manager, cluster-storage-operator, certified-operators-catalog, cluster-version-operator, hosted-cluster-config-operator, cluster-node-tuning-operator, redhat-operators-catalog, csi-snapshot-controller-operator, catalog-operator, redhat-marketplace-catalog, cluster-policy-controller, openshift-route-controller-manager, cluster-image-registry-operator; hosted cluster degraded: UnavailableReplicas: [catalog-operator deployment has 1 unavailable replicas, hosted-cluster-config-operator deployment has 1 unavailable replicas, olm-operator deployment has 1 unavailable replicas, openshift-apiserver deployment has 1 unavailable replicas, packageserver deployment has 3 unavailable replicas]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/36e9fb0c-0ea6-49be-917a-e6ad54933e8b
correlation ID: 0302f0c3-0eb3-48dd-9510-05b3665bc29f
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:102]: failed to create HCP cluster np-autoscale-cluster
Unexpected error:
<*fmt.wrapError | 0x2a9a9c302780>:
failed to create HCP cluster np-autoscale-cluster: failed waiting for cluster="np-autoscale-cluster" in resourcegroup="np-autoscaling-dq6rz78qbzwk" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/36e9fb0c-0ea6-49be-917a-e6ad54933e8b
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/36e9fb0c-0ea6-49be-917a-e6ad54933e8b",
"name": "36e9fb0c-0ea6-49be-917a-e6ad54933e8b",
"status": "Failed",
"startTime": "2026-10-28T20:14:46.976794807Z",
"endTime": "2026-10-28T20:43:55.053369978Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: olm-operator, cluster-network-operator, ingress-operator, community-operators-catalog, dns-operator, packageserver, openshift-controller-manager, cluster-storage-operator, certified-operators-catalog, cluster-version-operator, hosted-cluster-config-operator, cluster-node-tuning-operator, redhat-operators-catalog, csi-snapshot-controller-operator, catalog-operator, redhat-marketplace-catalog, cluster-policy-controller, openshift-route-controller-manager, cluster-image-registry-operator; hosted cluster degraded: UnavailableReplicas: [catalog-operator deployment has 1 unavailable replicas, hosted-cluster-config-operator deployment has 1 unavailable replicas, olm-operator deployment has 1 unavailable replicas, openshift-apiserver deployment has 1 unavailable replicas, packageserver deployment has 3 unavailable replicas]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/36e9fb0c-0ea6-49be-917a-e6ad54933e8b
correlation ID: 0302f0c3-0eb3-48dd-9510-05b3665bc29f
client request ID: unavailable
occurredfail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:125]: failed to create HCP cluster "cluster-zstream-4-22-p65ngr" with version 4.22.19 on candidate channel
Unexpected error:
<*fmt.wrapError | 0x3f72bfdcc60>:
failed to create HCP cluster cluster-zstream-4-22-p65ngr: failed waiting for cluster="cluster-zstream-4-22-p65ngr" in resourcegroup="rg-zstream-upgrade-4-22-fk4qxqhpjj29" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/8ec9fe1f-0e5b-4c8f-94c5-92a6c0e3d366
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/8ec9fe1f-0e5b-4c8f-94c5-92a6c0e3d366",
"name": "8ec9fe1f-0e5b-4c8f-94c5-92a6c0e3d366",
"status": "Failed",
"startTime": "2026-10-28T14:39:56.97734875Z",
"endTime": "2026-10-28T15:08:58.578863234Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: openshift-route-controller-manager, cluster-node-tuning-operator, ingress-operator, csi-snapshot-controller-operator, community-operators-catalog, dns-operator, cluster-image-registry-operator, openshift-controller-manager, cluster-policy-controller, cluster-version-operator, cluster-storage-operator, cluster-network-operator, catalog-operator, hosted-cluster-config-operator, olm-operator, packageserver, redhat-operators-catalog, certified-operators-catalog; hosted cluster degraded: UnavailableReplicas: [catalog-operator deployment has 1 unavailable replicas, hosted-cluster-config-operator deployment has 1 unavailable replicas, olm-operator deployment has 1 unavailable replicas, openshift-apiserver deployment has 1 unavailable replicas, packageserver deployment has 3 unavailable replicas]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/8ec9fe1f-0e5b-4c8f-94c5-92a6c0e3d366
correlation ID: 094c53a6-4603-4b58-b443-8cca1dbc9dbb
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:125]: failed to create HCP cluster "cluster-zstream-4-22-p65ngr" with version 4.22.19 on candidate channel
Unexpected error:
<*fmt.wrapError | 0x3f72bfdcc60>:
failed to create HCP cluster cluster-zstream-4-22-p65ngr: failed waiting for cluster="cluster-zstream-4-22-p65ngr" in resourcegroup="rg-zstream-upgrade-4-22-fk4qxqhpjj29" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/8ec9fe1f-0e5b-4c8f-94c5-92a6c0e3d366
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/8ec9fe1f-0e5b-4c8f-94c5-92a6c0e3d366",
"name": "8ec9fe1f-0e5b-4c8f-94c5-92a6c0e3d366",
"status": "Failed",
"startTime": "2026-10-28T14:39:56.97734875Z",
"endTime": "2026-10-28T15:08:58.578863234Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: openshift-route-controller-manager, cluster-node-tuning-operator, ingress-operator, csi-snapshot-controller-operator, community-operators-catalog, dns-operator, cluster-image-registry-operator, openshift-controller-manager, cluster-policy-controller, cluster-version-operator, cluster-storage-operator, cluster-network-operator, catalog-operator, hosted-cluster-config-operator, olm-operator, packageserver, redhat-operators-catalog, certified-operators-catalog; hosted cluster degraded: UnavailableReplicas: [catalog-operator deployment has 1 unavailable replicas, hosted-cluster-config-operator deployment has 1 unavailable replicas, olm-operator deployment has 1 unavailable replicas, openshift-apiserver deployment has 1 unavailable replicas, packageserver deployment has 3 unavailable replicas]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/8ec9fe1f-0e5b-4c8f-94c5-92a6c0e3d366
correlation ID: 094c53a6-4603-4b58-b443-8cca1dbc9dbb
client request ID: unavailable
occurredfail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:168]: failed to create HCP cluster for patch-tags test
Unexpected error:
<*fmt.wrapError | 0x32cee6f94b40>:
failed to create HCP cluster patch-tags-cluster: failed waiting for cluster="patch-tags-cluster" in resourcegroup="patch-tags-gkhf8l89gfpt" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/e910a1ec-64c8-4b3f-89a4-590c87d733cf
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/e910a1ec-64c8-4b3f-89a4-590c87d733cf",
"name": "e910a1ec-64c8-4b3f-89a4-590c87d733cf",
"status": "Failed",
"startTime": "2026-10-28T14:39:19.663998842Z",
"endTime": "2026-10-28T15:08:28.451578554Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: csi-snapshot-controller-operator, cluster-image-registry-operator, catalog-operator, cluster-policy-controller, redhat-operators-catalog, cluster-network-operator, openshift-route-controller-manager, dns-operator, packageserver, hosted-cluster-config-operator, cluster-storage-operator, cluster-node-tuning-operator, ingress-operator, redhat-marketplace-catalog, certified-operators-catalog, community-operators-catalog, olm-operator, cluster-version-operator, openshift-controller-manager; hosted cluster degraded: UnavailableReplicas: [catalog-operator deployment has 1 unavailable replicas, hosted-cluster-config-operator deployment has 1 unavailable replicas, olm-operator deployment has 1 unavailable replicas, openshift-apiserver deployment has 1 unavailable replicas, packageserver deployment has 3 unavailable replicas]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/e910a1ec-64c8-4b3f-89a4-590c87d733cf
correlation ID: 9764c911-c5b8-4802-ae2b-3bbd27636314
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:168]: failed to create HCP cluster for patch-tags test
Unexpected error:
<*fmt.wrapError | 0x32cee6f94b40>:
failed to create HCP cluster patch-tags-cluster: failed waiting for cluster="patch-tags-cluster" in resourcegroup="patch-tags-gkhf8l89gfpt" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/e910a1ec-64c8-4b3f-89a4-590c87d733cf
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/e910a1ec-64c8-4b3f-89a4-590c87d733cf",
"name": "e910a1ec-64c8-4b3f-89a4-590c87d733cf",
"status": "Failed",
"startTime": "2026-10-28T14:39:19.663998842Z",
"endTime": "2026-10-28T15:08:28.451578554Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: csi-snapshot-controller-operator, cluster-image-registry-operator, catalog-operator, cluster-policy-controller, redhat-operators-catalog, cluster-network-operator, openshift-route-controller-manager, dns-operator, packageserver, hosted-cluster-config-operator, cluster-storage-operator, cluster-node-tuning-operator, ingress-operator, redhat-marketplace-catalog, certified-operators-catalog, community-operators-catalog, olm-operator, cluster-version-operator, openshift-controller-manager; hosted cluster degraded: UnavailableReplicas: [catalog-operator deployment has 1 unavailable replicas, hosted-cluster-config-operator deployment has 1 unavailable replicas, olm-operator deployment has 1 unavailable replicas, openshift-apiserver deployment has 1 unavailable replicas, packageserver deployment has 3 unavailable replicas]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/e910a1ec-64c8-4b3f-89a4-590c87d733cf
correlation ID: 9764c911-c5b8-4802-ae2b-3bbd27636314
client request ID: unavailable
occurredContributing tests (7)
Affected runs (2)
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet complete...full failure pattern: ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version <version> is Partial (want Completed), started <duration> ago; hosted cluster degraded: UnavailableReplicas: [ignition-server-proxy, kube-controller-manager]; provider Microsoft.RedHatOpenShift Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s) | e2e | 2 | 4.35%2 of 46 job runs affected | IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s) | Show trend detailsOct 25: 0 · Oct 26: 0 · Oct 27: 0 · Oct 28: 6 · Oct 29: 0 · Oct 30: 0 · Oct 31: 0 · Nov 1: 0 | none | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (3)fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:291]: failed to create HCP cluster node-limit-cluster with MaxNodesTotal limit
Unexpected error:
<*fmt.wrapError | 0xe23c97da540>:
failed to create HCP cluster node-limit-cluster: failed waiting for cluster="node-limit-cluster" in resourcegroup="node-limit-test-bsr8ks7wrjr6" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/3e534289-6bed-48ab-bc7c-f5aee1b8cdc3
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/3e534289-6bed-48ab-bc7c-f5aee1b8cdc3",
"name": "3e534289-6bed-48ab-bc7c-f5aee1b8cdc3",
"status": "Failed",
"startTime": "2026-10-28T20:28:51.316103778Z",
"endTime": "2026-10-28T20:57:55.004654509Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.40 is Partial (want Completed), started 25m35s ago; hosted cluster degraded: UnavailableReplicas: [ignition-server-proxy deployment has 1 unavailable replicas, kube-controller-manager deployment has 1 unavailable replicas]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/3e534289-6bed-48ab-bc7c-f5aee1b8cdc3
correlation ID: 5b00d586-2a8f-4b57-a9e1-84466b469fca
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:291]: failed to create HCP cluster node-limit-cluster with MaxNodesTotal limit
Unexpected error:
<*fmt.wrapError | 0xe23c97da540>:
failed to create HCP cluster node-limit-cluster: failed waiting for cluster="node-limit-cluster" in resourcegroup="node-limit-test-bsr8ks7wrjr6" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/3e534289-6bed-48ab-bc7c-f5aee1b8cdc3
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/3e534289-6bed-48ab-bc7c-f5aee1b8cdc3",
"name": "3e534289-6bed-48ab-bc7c-f5aee1b8cdc3",
"status": "Failed",
"startTime": "2026-10-28T20:28:51.316103778Z",
"endTime": "2026-10-28T20:57:55.004654509Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.40 is Partial (want Completed), started 25m35s ago; hosted cluster degraded: UnavailableReplicas: [ignition-server-proxy deployment has 1 unavailable replicas, kube-controller-manager deployment has 1 unavailable replicas]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/3e534289-6bed-48ab-bc7c-f5aee1b8cdc3
correlation ID: 5b00d586-2a8f-4b57-a9e1-84466b469fca
client request ID: unavailable
occurredfail [github.com/Azure/ARO-HCP/test/e2e/nodepool_delete.go:81]: failed to create HCP cluster np-delete-hcp-cluster
Unexpected error:
<*fmt.wrapError | 0x38a796b1a740>:
failed to create HCP cluster np-delete-hcp-cluster: failed waiting for cluster="np-delete-hcp-cluster" in resourcegroup="nodepool-delete-lvjvqhcdtkrf" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/499291aa-b279-429d-80ba-7786fc6cfa93
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/499291aa-b279-429d-80ba-7786fc6cfa93",
"name": "499291aa-b279-429d-80ba-7786fc6cfa93",
"status": "Failed",
"startTime": "2026-10-28T20:38:17.873619905Z",
"endTime": "2026-10-28T21:07:24.99192406Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.40 is Partial (want Completed), started 24m7s ago; hosted cluster degraded: UnavailableReplicas: [ignition-server-proxy deployment has 1 unavailable replicas, kube-controller-manager deployment has 1 unavailable replicas]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/499291aa-b279-429d-80ba-7786fc6cfa93
correlation ID: 7577cb59-123a-459f-a8f3-f4a03306f15d
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_delete.go:81]: failed to create HCP cluster np-delete-hcp-cluster
Unexpected error:
<*fmt.wrapError | 0x38a796b1a740>:
failed to create HCP cluster np-delete-hcp-cluster: failed waiting for cluster="np-delete-hcp-cluster" in resourcegroup="nodepool-delete-lvjvqhcdtkrf" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/499291aa-b279-429d-80ba-7786fc6cfa93
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/499291aa-b279-429d-80ba-7786fc6cfa93",
"name": "499291aa-b279-429d-80ba-7786fc6cfa93",
"status": "Failed",
"startTime": "2026-10-28T20:38:17.873619905Z",
"endTime": "2026-10-28T21:07:24.99192406Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.40 is Partial (want Completed), started 24m7s ago; hosted cluster degraded: UnavailableReplicas: [ignition-server-proxy deployment has 1 unavailable replicas, kube-controller-manager deployment has 1 unavailable replicas]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/499291aa-b279-429d-80ba-7786fc6cfa93
correlation ID: 7577cb59-123a-459f-a8f3-f4a03306f15d
client request ID: unavailable
occurredfail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:146]: failed to create HCP cluster np-version-upgrade-cluster-cz8ptt with version 4.20.44
Unexpected error:
<*fmt.wrapError | 0x1c7b6c0a8440>:
failed to create HCP cluster np-version-upgrade-cluster-cz8ptt: failed waiting for cluster="np-version-upgrade-cluster-cz8ptt" in resourcegroup="rg-np-version-upgrade-cz8ptt-hnxbd7dxv6dv" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/6a1a42fe-bb72-488b-adbf-55a400b476cf
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/6a1a42fe-bb72-488b-adbf-55a400b476cf",
"name": "6a1a42fe-bb72-488b-adbf-55a400b476cf",
"status": "Failed",
"startTime": "2026-10-28T20:56:27.61784978Z",
"endTime": "2026-10-28T21:25:30.606619249Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.20.44 is Partial (want Completed), started 25m13s ago; hosted cluster degraded: UnavailableReplicas: [ignition-server-proxy deployment has 1 unavailable replicas, kube-controller-manager deployment has 1 unavailable replicas]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/6a1a42fe-bb72-488b-adbf-55a400b476cf
correlation ID: 4dbfaa07-bbd8-4943-933b-1b427a38701f
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:146]: failed to create HCP cluster np-version-upgrade-cluster-cz8ptt with version 4.20.44
Unexpected error:
<*fmt.wrapError | 0x1c7b6c0a8440>:
failed to create HCP cluster np-version-upgrade-cluster-cz8ptt: failed waiting for cluster="np-version-upgrade-cluster-cz8ptt" in resourcegroup="rg-np-version-upgrade-cz8ptt-hnxbd7dxv6dv" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/6a1a42fe-bb72-488b-adbf-55a400b476cf
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/6a1a42fe-bb72-488b-adbf-55a400b476cf",
"name": "6a1a42fe-bb72-488b-adbf-55a400b476cf",
"status": "Failed",
"startTime": "2026-10-28T20:56:27.61784978Z",
"endTime": "2026-10-28T21:25:30.606619249Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.20.44 is Partial (want Completed), started 25m13s ago; hosted cluster degraded: UnavailableReplicas: [ignition-server-proxy deployment has 1 unavailable replicas, kube-controller-manager deployment has 1 unavailable replicas]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/6a1a42fe-bb72-488b-adbf-55a400b476cf
correlation ID: 4dbfaa07-bbd8-4943-933b-1b427a38701f
client request ID: unavailable
occurredContributing tests (6)
Affected runs (2)
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
VerifyDefaultIngressServingCertificateSecretWiring timed out after 10m0full failure pattern: VerifyDefaultIngressServingCertificateSecretWiring timed out after 10m0 Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 2 day(s) | e2e | 2 | 4.35%2 of 46 job runs affected | IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 2 day(s) | Show trend detailsOct 25: 0 · Oct 26: 4 · Oct 27: 1 · Oct 28: 0 · Oct 29: 0 · Oct 30: 0 · Oct 31: 0 · Nov 1: 0 | STG | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (3)fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:256]: default-ingress-cert must exist and be referenced by IngressController/default for cluster "cluster-candidate-5-0-lbfkz7"
Expected success, but got an error:
<*fmt.wrapError | 0x625ce9962c0>:
VerifyDefaultIngressServingCertificateSecretWiring timed out after 10m0.001s: IngressController/default references &LocalObjectReference{Name:cluster-ingress-cert,}, expected secret "default-ingress-cert"
{
msg: "VerifyDefaultIngressServingCertificateSecretWiring timed out after 10m0.001s: IngressController/default references &LocalObjectReference{Name:cluster-ingress-cert,}, expected secret \"default-ingress-cert\"",
err: <*errors.errorString | 0x625d05dc8e0>{
s: "IngressController/default references &LocalObjectReference{Name:cluster-ingress-cert,}, expected secret \"default-ingress-cert\"",
},
}
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:256]: default-ingress-cert must exist and be referenced by IngressController/default for cluster "cluster-candidate-5-0-lbfkz7"
Expected success, but got an error:
<*fmt.wrapError | 0x625ce9962c0>:
VerifyDefaultIngressServingCertificateSecretWiring timed out after 10m0.001s: IngressController/default references &LocalObjectReference{Name:cluster-ingress-cert,}, expected secret "default-ingress-cert"
{
msg: "VerifyDefaultIngressServingCertificateSecretWiring timed out after 10m0.001s: IngressController/default references &LocalObjectReference{Name:cluster-ingress-cert,}, expected secret \"default-ingress-cert\"",
err: <*errors.errorString | 0x625d05dc8e0>{
s: "IngressController/default references &LocalObjectReference{Name:cluster-ingress-cert,}, expected secret \"default-ingress-cert\"",
},
}fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:256]: default-ingress-cert must exist and be referenced by IngressController/default for cluster "cluster-nightly-5-0-6gm2q7"
Expected success, but got an error:
<*fmt.wrapError | 0xb6fec4b44e0>:
VerifyDefaultIngressServingCertificateSecretWiring timed out after 10m0.005s: IngressController/default references &LocalObjectReference{Name:cluster-ingress-cert,}, expected secret "default-ingress-cert"
{
msg: "VerifyDefaultIngressServingCertificateSecretWiring timed out after 10m0.005s: IngressController/default references &LocalObjectReference{Name:cluster-ingress-cert,}, expected secret \"default-ingress-cert\"",
err: <*verifiers.pollObservationError | 0xb6fec4b44c0>{
observation: <*errors.errorString | 0xb6fee7deaa0>{
s: "IngressController/default references &LocalObjectReference{Name:cluster-ingress-cert,}, expected secret \"default-ingress-cert\"",
},
interruption: <*fmt.wrapError | 0xb6fec4b44a0>{
msg: "getting default ingress serving certificate secret openshift-ingress/default-ingress-cert: context deadline exceeded",
err: <context.deadlineExceededError>{},
},
},
}
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:256]: default-ingress-cert must exist and be referenced by IngressController/default for cluster "cluster-nightly-5-0-6gm2q7"
Expected success, but got an error:
<*fmt.wrapError | 0xb6fec4b44e0>:
VerifyDefaultIngressServingCertificateSecretWiring timed out after 10m0.005s: IngressController/default references &LocalObjectReference{Name:cluster-ingress-cert,}, expected secret "default-ingress-cert"
{
msg: "VerifyDefaultIngressServingCertificateSecretWiring timed out after 10m0.005s: IngressController/default references &LocalObjectReference{Name:cluster-ingress-cert,}, expected secret \"default-ingress-cert\"",
err: <*verifiers.pollObservationError | 0xb6fec4b44c0>{
observation: <*errors.errorString | 0xb6fee7deaa0>{
s: "IngressController/default references &LocalObjectReference{Name:cluster-ingress-cert,}, expected secret \"default-ingress-cert\"",
},
interruption: <*fmt.wrapError | 0xb6fec4b44a0>{
msg: "getting default ingress serving certificate secret openshift-ingress/default-ingress-cert: context deadline exceeded",
err: <context.deadlineExceededError>{},
},
},
}fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:256]: default-ingress-cert must exist and be referenced by IngressController/default for cluster "cluster-nightly-4-22-l72l2j"
Expected success, but got an error:
<*fmt.wrapError | 0x8108240e540>:
VerifyDefaultIngressServingCertificateSecretWiring timed out after 10m0.006s: IngressController/default references &LocalObjectReference{Name:cluster-ingress-cert,}, expected secret "default-ingress-cert"
{
msg: "VerifyDefaultIngressServingCertificateSecretWiring timed out after 10m0.006s: IngressController/default references &LocalObjectReference{Name:cluster-ingress-cert,}, expected secret \"default-ingress-cert\"",
err: <*verifiers.pollObservationError | 0x8108240e520>{
observation: <*errors.errorString | 0x81084000ba0>{
s: "IngressController/default references &LocalObjectReference{Name:cluster-ingress-cert,}, expected secret \"default-ingress-cert\"",
},
interruption: <*fmt.wrapError | 0x8108240e500>{
msg: "getting default ingress serving certificate secret openshift-ingress/default-ingress-cert: context deadline exceeded",
err: <context.deadlineExceededError>{},
},
},
}
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:256]: default-ingress-cert must exist and be referenced by IngressController/default for cluster "cluster-nightly-4-22-l72l2j"
Expected success, but got an error:
<*fmt.wrapError | 0x8108240e540>:
VerifyDefaultIngressServingCertificateSecretWiring timed out after 10m0.006s: IngressController/default references &LocalObjectReference{Name:cluster-ingress-cert,}, expected secret "default-ingress-cert"
{
msg: "VerifyDefaultIngressServingCertificateSecretWiring timed out after 10m0.006s: IngressController/default references &LocalObjectReference{Name:cluster-ingress-cert,}, expected secret \"default-ingress-cert\"",
err: <*verifiers.pollObservationError | 0x8108240e520>{
observation: <*errors.errorString | 0x81084000ba0>{
s: "IngressController/default references &LocalObjectReference{Name:cluster-ingress-cert,}, expected secret \"default-ingress-cert\"",
},
interruption: <*fmt.wrapError | 0x8108240e500>{
msg: "getting default ingress serving certificate secret openshift-ingress/default-ingress-cert: context deadline exceeded",
err: <context.deadlineExceededError>{},
},
},
}Contributing tests (5)
Affected runs (2)
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedC...full failure pattern: ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider, ignition-server-proxy]; provider Microsoft.RedHatOpenShift Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s) | e2e | 2 | 4.35%2 of 46 job runs affected | IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s) | Show trend detailsOct 25: 0 · Oct 26: 0 · Oct 27: 0 · Oct 28: 4 · Oct 29: 0 · Oct 30: 0 · Oct 31: 0 · Nov 1: 0 | none | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (3)fail [github.com/Azure/ARO-HCP/test/e2e/image_registry_cluster_create.go:89]: failed to create HCP cluster with image registry disabled
Unexpected error:
<*fmt.wrapError | 0x2c03b21d6c20>:
failed to create HCP cluster disabled-image-registry-hcp-cluster: failed waiting for cluster="disabled-image-registry-hcp-cluster" in resourcegroup="disabled-image-registry-7fv9wkcmzdx9" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/7f5aec2c-eaa8-4eef-ae1b-398990595f26
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/7f5aec2c-eaa8-4eef-ae1b-398990595f26",
"name": "7f5aec2c-eaa8-4eef-ae1b-398990595f26",
"status": "Failed",
"startTime": "2026-10-28T20:14:37.658468995Z",
"endTime": "2026-10-28T20:43:44.89685741Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider deployment has 1 unavailable replicas, ignition-server-proxy deployment has 1 unavailable replicas]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/7f5aec2c-eaa8-4eef-ae1b-398990595f26
correlation ID: 1b096272-5f83-459d-a9b3-80242ffba49d
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/image_registry_cluster_create.go:89]: failed to create HCP cluster with image registry disabled
Unexpected error:
<*fmt.wrapError | 0x2c03b21d6c20>:
failed to create HCP cluster disabled-image-registry-hcp-cluster: failed waiting for cluster="disabled-image-registry-hcp-cluster" in resourcegroup="disabled-image-registry-7fv9wkcmzdx9" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/7f5aec2c-eaa8-4eef-ae1b-398990595f26
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/7f5aec2c-eaa8-4eef-ae1b-398990595f26",
"name": "7f5aec2c-eaa8-4eef-ae1b-398990595f26",
"status": "Failed",
"startTime": "2026-10-28T20:14:37.658468995Z",
"endTime": "2026-10-28T20:43:44.89685741Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider deployment has 1 unavailable replicas, ignition-server-proxy deployment has 1 unavailable replicas]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/7f5aec2c-eaa8-4eef-ae1b-398990595f26
correlation ID: 1b096272-5f83-459d-a9b3-80242ffba49d
client request ID: unavailable
occurredfail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:146]: failed to create HCP cluster np-version-upgrade-cluster-t67p9t with version 4.21.40
Unexpected error:
<*fmt.wrapError | 0xbe0fc68500>:
failed to create HCP cluster np-version-upgrade-cluster-t67p9t: failed waiting for cluster="np-version-upgrade-cluster-t67p9t" in resourcegroup="rg-np-version-upgrade-t67p9t-kpsvm25jmqsm" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/9e35dee8-3994-4ba7-9234-74a03d742751
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/9e35dee8-3994-4ba7-9234-74a03d742751",
"name": "9e35dee8-3994-4ba7-9234-74a03d742751",
"status": "Failed",
"startTime": "2026-10-28T15:21:15.512939467Z",
"endTime": "2026-10-28T15:50:17.639783703Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider deployment has 1 unavailable replicas, ignition-server-proxy deployment has 1 unavailable replicas]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/9e35dee8-3994-4ba7-9234-74a03d742751
correlation ID: 8d785f53-f7f0-40a1-aa91-751064d7e050
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:146]: failed to create HCP cluster np-version-upgrade-cluster-t67p9t with version 4.21.40
Unexpected error:
<*fmt.wrapError | 0xbe0fc68500>:
failed to create HCP cluster np-version-upgrade-cluster-t67p9t: failed waiting for cluster="np-version-upgrade-cluster-t67p9t" in resourcegroup="rg-np-version-upgrade-t67p9t-kpsvm25jmqsm" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/9e35dee8-3994-4ba7-9234-74a03d742751
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/9e35dee8-3994-4ba7-9234-74a03d742751",
"name": "9e35dee8-3994-4ba7-9234-74a03d742751",
"status": "Failed",
"startTime": "2026-10-28T15:21:15.512939467Z",
"endTime": "2026-10-28T15:50:17.639783703Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider deployment has 1 unavailable replicas, ignition-server-proxy deployment has 1 unavailable replicas]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/9e35dee8-3994-4ba7-9234-74a03d742751
correlation ID: 8d785f53-f7f0-40a1-aa91-751064d7e050
client request ID: unavailable
occurredfail [github.com/Azure/ARO-HCP/test/e2e/cluster_autoscaling.go:108]: failed to create HCP cluster "autoscaling-hcp-cluster" with custom autoscaling
Unexpected error:
<*fmt.wrapError | 0x24f1c7fcefa0>:
failed to create HCP cluster autoscaling-hcp-cluster: failed waiting for cluster="autoscaling-hcp-cluster" in resourcegroup="autoscaling-cluster-bhc75ljrxzvb" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/3ee91092-37d4-4687-8899-086b94094407
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/3ee91092-37d4-4687-8899-086b94094407",
"name": "3ee91092-37d4-4687-8899-086b94094407",
"status": "Failed",
"startTime": "2026-10-28T13:42:47.010012275Z",
"endTime": "2026-10-28T14:11:50.916418144Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider deployment has 1 unavailable replicas, ignition-server-proxy deployment has 1 unavailable replicas]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/3ee91092-37d4-4687-8899-086b94094407
correlation ID: 881a23bb-07d5-4161-bd1d-8f0790ff0f30
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_autoscaling.go:108]: failed to create HCP cluster "autoscaling-hcp-cluster" with custom autoscaling
Unexpected error:
<*fmt.wrapError | 0x24f1c7fcefa0>:
failed to create HCP cluster autoscaling-hcp-cluster: failed waiting for cluster="autoscaling-hcp-cluster" in resourcegroup="autoscaling-cluster-bhc75ljrxzvb" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/3ee91092-37d4-4687-8899-086b94094407
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/3ee91092-37d4-4687-8899-086b94094407",
"name": "3ee91092-37d4-4687-8899-086b94094407",
"status": "Failed",
"startTime": "2026-10-28T13:42:47.010012275Z",
"endTime": "2026-10-28T14:11:50.916418144Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider deployment has 1 unavailable replicas, ignition-server-proxy deployment has 1 unavailable replicas]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/3ee91092-37d4-4687-8899-086b94094407
correlation ID: 881a23bb-07d5-4161-bd1d-8f0790ff0f30
client request ID: unavailable
occurredContributing tests (4)
Affected runs (2)
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedC...full failure pattern: ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster has no control plane endpoint host; provider Microsoft.RedHatOpenShift Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s) | e2e | 2 | 4.35%2 of 46 job runs affected | IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s) | Show trend detailsOct 25: 0 · Oct 26: 0 · Oct 27: 0 · Oct 28: 4 · Oct 29: 0 · Oct 30: 0 · Oct 31: 0 · Nov 1: 0 | none | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (3)fail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:111]: failed to create HCP cluster "cidr-connectivity-test" with authorized CIDRs
Unexpected error:
<*fmt.wrapError | 0x14a5e2008300>:
failed to create HCP cluster cidr-connectivity-test: failed waiting for cluster="cidr-connectivity-test" in resourcegroup="e2e-cidr-connectivity-k46j9j85xq5z" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/a961a70a-5b5a-4175-bb58-82fb36beb466
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/a961a70a-5b5a-4175-bb58-82fb36beb466",
"name": "a961a70a-5b5a-4175-bb58-82fb36beb466",
"status": "Failed",
"startTime": "2026-10-28T19:26:12.743400806Z",
"endTime": "2026-10-28T19:55:13.7057976Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster has no control plane endpoint host"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/a961a70a-5b5a-4175-bb58-82fb36beb466
correlation ID: 5835c446-bdab-409a-8175-cbf5109c1943
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:111]: failed to create HCP cluster "cidr-connectivity-test" with authorized CIDRs
Unexpected error:
<*fmt.wrapError | 0x14a5e2008300>:
failed to create HCP cluster cidr-connectivity-test: failed waiting for cluster="cidr-connectivity-test" in resourcegroup="e2e-cidr-connectivity-k46j9j85xq5z" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/a961a70a-5b5a-4175-bb58-82fb36beb466
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/a961a70a-5b5a-4175-bb58-82fb36beb466",
"name": "a961a70a-5b5a-4175-bb58-82fb36beb466",
"status": "Failed",
"startTime": "2026-10-28T19:26:12.743400806Z",
"endTime": "2026-10-28T19:55:13.7057976Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster has no control plane endpoint host"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/a961a70a-5b5a-4175-bb58-82fb36beb466
correlation ID: 5835c446-bdab-409a-8175-cbf5109c1943
client request ID: unavailable
occurredfail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:208]: failed to create HCP cluster "agg-cluster" with aggregated settings
Unexpected error:
<*fmt.wrapError | 0x1683f5097100>:
failed waiting for cluster="agg-cluster" in resourcegroup="feature-aggregation-mhvkjzh45hwc" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/e9a36944-a3a3-4eaf-9fdc-98e8262c979b
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/e9a36944-a3a3-4eaf-9fdc-98e8262c979b",
"name": "e9a36944-a3a3-4eaf-9fdc-98e8262c979b",
"status": "Failed",
"startTime": "2026-10-28T19:28:18.023653922Z",
"endTime": "2026-10-28T19:57:23.841118908Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster has no control plane endpoint host"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/e9a36944-a3a3-4eaf-9fdc-98e8262c979b
correlation ID: 7d36723c-bee1-4adc-86af-798e4647be9f
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:208]: failed to create HCP cluster "agg-cluster" with aggregated settings
Unexpected error:
<*fmt.wrapError | 0x1683f5097100>:
failed waiting for cluster="agg-cluster" in resourcegroup="feature-aggregation-mhvkjzh45hwc" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/e9a36944-a3a3-4eaf-9fdc-98e8262c979b
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/e9a36944-a3a3-4eaf-9fdc-98e8262c979b",
"name": "e9a36944-a3a3-4eaf-9fdc-98e8262c979b",
"status": "Failed",
"startTime": "2026-10-28T19:28:18.023653922Z",
"endTime": "2026-10-28T19:57:23.841118908Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster has no control plane endpoint host"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/e9a36944-a3a3-4eaf-9fdc-98e8262c979b
correlation ID: 7d36723c-bee1-4adc-86af-798e4647be9f
client request ID: unavailable
occurredfail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:133]: failed to create HCP cluster "cp-ystream-upgrade-5-0-jqd7cs" at install version 4.22
Unexpected error:
<*fmt.wrapError | 0x13016dc5e5e0>:
failed to create HCP cluster cp-ystream-upgrade-5-0-jqd7cs: failed waiting for cluster="cp-ystream-upgrade-5-0-jqd7cs" in resourcegroup="rg-cp-ystream-upgrade-5-0-ft5czs88jtdf" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/1dc97088-f5bb-4932-bd4c-14b5c83e689d
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/1dc97088-f5bb-4932-bd4c-14b5c83e689d",
"name": "1dc97088-f5bb-4932-bd4c-14b5c83e689d",
"status": "Failed",
"startTime": "2026-10-28T20:12:34.851456324Z",
"endTime": "2026-10-28T20:41:34.873934441Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster has no control plane endpoint host"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/1dc97088-f5bb-4932-bd4c-14b5c83e689d
correlation ID: 03cba8d0-c85c-4970-a348-de9b3f376ff4
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:133]: failed to create HCP cluster "cp-ystream-upgrade-5-0-jqd7cs" at install version 4.22
Unexpected error:
<*fmt.wrapError | 0x13016dc5e5e0>:
failed to create HCP cluster cp-ystream-upgrade-5-0-jqd7cs: failed waiting for cluster="cp-ystream-upgrade-5-0-jqd7cs" in resourcegroup="rg-cp-ystream-upgrade-5-0-ft5czs88jtdf" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/1dc97088-f5bb-4932-bd4c-14b5c83e689d
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/1dc97088-f5bb-4932-bd4c-14b5c83e689d",
"name": "1dc97088-f5bb-4932-bd4c-14b5c83e689d",
"status": "Failed",
"startTime": "2026-10-28T20:12:34.851456324Z",
"endTime": "2026-10-28T20:41:34.873934441Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster has no control plane endpoint host"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/1dc97088-f5bb-4932-bd4c-14b5c83e689d
correlation ID: 03cba8d0-c85c-4970-a348-de9b3f376ff4
client request ID: unavailable
occurredContributing tests (3)
Affected runs (2)
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
timeout during CreateHCPClusterAndWait; context deadline exceededfull failure pattern: timeout during CreateHCPClusterAndWait; context deadline exceeded Signal: IndeterminateSignal: Indeterminate — seen in 2 prior week(s); active 1 day(s) | e2e | 2 | 4.35%2 of 46 job runs affected | IndeterminateSignal: Indeterminate — seen in 2 prior week(s); active 1 day(s) | Show trend detailsOct 25: 0 · Oct 26: 0 · Oct 27: 0 · Oct 28: 3 · Oct 29: 0 · Oct 30: 0 · Oct 31: 0 · Nov 1: 0 | none | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (3)fail [github.com/Azure/ARO-HCP/test/e2e/cluster_version_backlevel.go:152]: failed to create HCP cluster version 4.20
Unexpected error:
<*fmt.wrapErrors | 0xfa629f24ea0>:
failed waiting for cluster="cluster-ver-4-20" in resourcegroup="rg-cluster-back-version-zkznqc7xhzd4" to finish creating, caused by: timeout '30.000000' minutes exceeded during CreateHCPClusterAndWait for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-zkznqc7xhzd4, error: context deadline exceeded
{
msg: "failed waiting for cluster=\"cluster-ver-4-20\" in resourcegroup=\"rg-cluster-back-version-zkznqc7xhzd4\" to finish creating, caused by: timeout '30.000000' minutes exceeded during CreateHCPClusterAndWait for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-zkznqc7xhzd4, error: context deadline exceeded",
errs: [
<*errors.errorString | 0xfa629c70d20>{
s: "timeout '30.000000' minutes exceeded during CreateHCPClusterAndWait for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-zkznqc7xhzd4",
},
<context.deadlineExceededError>{},
],
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_version_backlevel.go:152]: failed to create HCP cluster version 4.20
Unexpected error:
<*fmt.wrapErrors | 0xfa629f24ea0>:
failed waiting for cluster="cluster-ver-4-20" in resourcegroup="rg-cluster-back-version-zkznqc7xhzd4" to finish creating, caused by: timeout '30.000000' minutes exceeded during CreateHCPClusterAndWait for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-zkznqc7xhzd4, error: context deadline exceeded
{
msg: "failed waiting for cluster=\"cluster-ver-4-20\" in resourcegroup=\"rg-cluster-back-version-zkznqc7xhzd4\" to finish creating, caused by: timeout '30.000000' minutes exceeded during CreateHCPClusterAndWait for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-zkznqc7xhzd4, error: context deadline exceeded",
errs: [
<*errors.errorString | 0xfa629c70d20>{
s: "timeout '30.000000' minutes exceeded during CreateHCPClusterAndWait for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-zkznqc7xhzd4",
},
<context.deadlineExceededError>{},
],
}
occurredfail [github.com/Azure/ARO-HCP/test/e2e/idms_lifecycle.go:113]: failed to create HCP cluster with ImageDigestMirrors
Unexpected error:
<*fmt.wrapErrors | 0x1f0baa544120>:
failed to create HCP cluster idms-e2e-hcp-cluster, caused by: timeout '30.000000' minutes exceeded during CreateHCPCluster20251223FromParam for cluster idms-e2e-hcp-cluster in resource group idms-n4hrqndpjj88, error: failed waiting for cluster="idms-e2e-hcp-cluster" in resourcegroup="idms-n4hrqndpjj88" to finish creating, caused by: timeout '30.000000' minutes exceeded during CreateHCPCluster20251223FromParam for cluster idms-e2e-hcp-cluster in resource group idms-n4hrqndpjj88, error: context deadline exceeded
{
msg: "failed to create HCP cluster idms-e2e-hcp-cluster, caused by: timeout '30.000000' minutes exceeded during CreateHCPCluster20251223FromParam for cluster idms-e2e-hcp-cluster in resource group idms-n4hrqndpjj88, error: failed waiting for cluster=\"idms-e2e-hcp-cluster\" in resourcegroup=\"idms-n4hrqndpjj88\" to finish creating, caused by: timeout '30.000000' minutes exceeded during CreateHCPCluster20251223FromParam for cluster idms-e2e-hcp-cluster in resource group idms-n4hrqndpjj88, error: context deadline exceeded",
errs: [
<*errors.errorString | 0x1f0baa40b610>{
s: "timeout '30.000000' minutes exceeded during CreateHCPCluster20251223FromParam for cluster idms-e2e-hcp-cluster in resource group idms-n4hrqndpjj88",
},
<*fmt.wrapErrors | 0x1f0baa5440c0>{
msg: "failed waiting for cluster=\"idms-e2e-hcp-cluster\" in resourcegroup=\"idms-n4hrqndpjj88\" to finish creating, caused by: timeout '30.000000' minutes exceeded during CreateHCPCluster20251223FromParam for cluster idms-e2e-hcp-cluster in resource group idms-n4hrqndpjj88, error: context deadline exceeded",
errs: [
<*errors.errorString | 0x1f0baa40b610>0x1f0baa40b610 (cyclic reference),
<context.deadlineExceededError>{},
],
},
],
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/idms_lifecycle.go:113]: failed to create HCP cluster with ImageDigestMirrors
Unexpected error:
<*fmt.wrapErrors | 0x1f0baa544120>:
failed to create HCP cluster idms-e2e-hcp-cluster, caused by: timeout '30.000000' minutes exceeded during CreateHCPCluster20251223FromParam for cluster idms-e2e-hcp-cluster in resource group idms-n4hrqndpjj88, error: failed waiting for cluster="idms-e2e-hcp-cluster" in resourcegroup="idms-n4hrqndpjj88" to finish creating, caused by: timeout '30.000000' minutes exceeded during CreateHCPCluster20251223FromParam for cluster idms-e2e-hcp-cluster in resource group idms-n4hrqndpjj88, error: context deadline exceeded
{
msg: "failed to create HCP cluster idms-e2e-hcp-cluster, caused by: timeout '30.000000' minutes exceeded during CreateHCPCluster20251223FromParam for cluster idms-e2e-hcp-cluster in resource group idms-n4hrqndpjj88, error: failed waiting for cluster=\"idms-e2e-hcp-cluster\" in resourcegroup=\"idms-n4hrqndpjj88\" to finish creating, caused by: timeout '30.000000' minutes exceeded during CreateHCPCluster20251223FromParam for cluster idms-e2e-hcp-cluster in resource group idms-n4hrqndpjj88, error: context deadline exceeded",
errs: [
<*errors.errorString | 0x1f0baa40b610>{
s: "timeout '30.000000' minutes exceeded during CreateHCPCluster20251223FromParam for cluster idms-e2e-hcp-cluster in resource group idms-n4hrqndpjj88",
},
<*fmt.wrapErrors | 0x1f0baa5440c0>{
msg: "failed waiting for cluster=\"idms-e2e-hcp-cluster\" in resourcegroup=\"idms-n4hrqndpjj88\" to finish creating, caused by: timeout '30.000000' minutes exceeded during CreateHCPCluster20251223FromParam for cluster idms-e2e-hcp-cluster in resource group idms-n4hrqndpjj88, error: context deadline exceeded",
errs: [
<*errors.errorString | 0x1f0baa40b610>0x1f0baa40b610 (cyclic reference),
<context.deadlineExceededError>{},
],
},
],
}
occurredfail [github.com/Azure/ARO-HCP/test/e2e/cluster_version_backlevel.go:152]: failed to create HCP cluster version 4.20
Unexpected error:
<*fmt.wrapErrors | 0x22e10818450>:
failed waiting for cluster="cluster-ver-4-20" in resourcegroup="rg-cluster-back-version-992c2j7mq2x8" to finish creating, caused by: timeout '30.000000' minutes exceeded during CreateHCPClusterAndWait for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-992c2j7mq2x8, error: context deadline exceeded
{
msg: "failed waiting for cluster=\"cluster-ver-4-20\" in resourcegroup=\"rg-cluster-back-version-992c2j7mq2x8\" to finish creating, caused by: timeout '30.000000' minutes exceeded during CreateHCPClusterAndWait for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-992c2j7mq2x8, error: context deadline exceeded",
errs: [
<*errors.errorString | 0x22e1036b8e0>{
s: "timeout '30.000000' minutes exceeded during CreateHCPClusterAndWait for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-992c2j7mq2x8",
},
<context.deadlineExceededError>{},
],
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_version_backlevel.go:152]: failed to create HCP cluster version 4.20
Unexpected error:
<*fmt.wrapErrors | 0x22e10818450>:
failed waiting for cluster="cluster-ver-4-20" in resourcegroup="rg-cluster-back-version-992c2j7mq2x8" to finish creating, caused by: timeout '30.000000' minutes exceeded during CreateHCPClusterAndWait for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-992c2j7mq2x8, error: context deadline exceeded
{
msg: "failed waiting for cluster=\"cluster-ver-4-20\" in resourcegroup=\"rg-cluster-back-version-992c2j7mq2x8\" to finish creating, caused by: timeout '30.000000' minutes exceeded during CreateHCPClusterAndWait for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-992c2j7mq2x8, error: context deadline exceeded",
errs: [
<*errors.errorString | 0x22e1036b8e0>{
s: "timeout '30.000000' minutes exceeded during CreateHCPClusterAndWait for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-992c2j7mq2x8",
},
<context.deadlineExceededError>{},
],
}
occurredContributing tests (2)
Affected runs (2)
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Customer should be able to test admin credentials before cluster ready, then full admin credential lifecycle: Cluster provisioning failedfull failure pattern: Customer should be able to test admin credentials before cluster ready, then full admin credential lifecycle: Cluster provisioning failed Signal: IndeterminateSignal: Indeterminate — seen in 2 prior week(s); active 1 day(s) | e2e | 2 | 4.35%2 of 46 job runs affected | IndeterminateSignal: Indeterminate — seen in 2 prior week(s); active 1 day(s) | Show trend detailsOct 25: 0 · Oct 26: 0 · Oct 27: 0 · Oct 28: 2 · Oct 29: 0 · Oct 30: 0 · Oct 31: 0 · Nov 1: 0 | none | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/e2e/admin_credential_lifecycle.go:158]: Cluster provisioning failed fail [github.com/Azure/ARO-HCP/test/e2e/admin_credential_lifecycle.go:158]: Cluster provisioning failed Contributing tests (1)
Affected runs (2)
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
ERROR CODE: DeadlineExceeded; detail message node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hype...full failure pattern: ERROR CODE: DeadlineExceeded; detail message node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift node pool not cached yet; provider Microsoft.RedHatOpenShift Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 2 day(s) | e2e | 2 | 4.35%2 of 46 job runs affected | IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 2 day(s) | Show trend detailsOct 25: 1 · Oct 26: 0 · Oct 27: 2 · Oct 28: 0 · Oct 29: 0 · Oct 30: 0 · Oct 31: 0 · Nov 1: 0 | DEV | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (2)fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:154]: failed to create AZ nodepool autoscale-az with autoscaling
Unexpected error:
<*fmt.wrapError | 0x341fcf187480>:
failed to create NodePool autoscale-az: failed waiting for nodepool="autoscale-az" for cluster "np-autoscale-cluster" in resourcegroup="np-autoscaling-6d2jd2rhkhpr" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2/hcpOperationStatuses/354ac5e7-358b-4ba7-b33a-800a398e3d90
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2/hcpOperationStatuses/354ac5e7-358b-4ba7-b33a-800a398e3d90",
"name": "354ac5e7-358b-4ba7-b33a-800a398e3d90",
"status": "Failed",
"startTime": "2026-10-27T21:18:08.702878641Z",
"endTime": "2026-10-27T21:37:14.197553955Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift node pool not cached yet"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2/hcpOperationStatuses/354ac5e7-358b-4ba7-b33a-800a398e3d90
correlation ID: 2d032843-08dc-4bd5-a784-b1e25c32c35a
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:154]: failed to create AZ nodepool autoscale-az with autoscaling
Unexpected error:
<*fmt.wrapError | 0x341fcf187480>:
failed to create NodePool autoscale-az: failed waiting for nodepool="autoscale-az" for cluster "np-autoscale-cluster" in resourcegroup="np-autoscaling-6d2jd2rhkhpr" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2/hcpOperationStatuses/354ac5e7-358b-4ba7-b33a-800a398e3d90
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2/hcpOperationStatuses/354ac5e7-358b-4ba7-b33a-800a398e3d90",
"name": "354ac5e7-358b-4ba7-b33a-800a398e3d90",
"status": "Failed",
"startTime": "2026-10-27T21:18:08.702878641Z",
"endTime": "2026-10-27T21:37:14.197553955Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift node pool not cached yet"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2/hcpOperationStatuses/354ac5e7-358b-4ba7-b33a-800a398e3d90
correlation ID: 2d032843-08dc-4bd5-a784-b1e25c32c35a
client request ID: unavailable
occurredfail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:154]: failed to create AZ nodepool autoscale-az with autoscaling
Unexpected error:
<*fmt.wrapError | 0x2462c6c0b540>:
failed to create NodePool autoscale-az: failed waiting for nodepool="autoscale-az" for cluster "np-autoscale-cluster" in resourcegroup="np-autoscaling-bl9lknth6jps" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2/hcpOperationStatuses/97c439db-9ae6-4e86-85bd-71d21a1fe1e7
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2/hcpOperationStatuses/97c439db-9ae6-4e86-85bd-71d21a1fe1e7",
"name": "97c439db-9ae6-4e86-85bd-71d21a1fe1e7",
"status": "Failed",
"startTime": "2026-10-27T18:48:30.242794404Z",
"endTime": "2026-10-27T19:07:38.465503369Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift node pool not cached yet"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2/hcpOperationStatuses/97c439db-9ae6-4e86-85bd-71d21a1fe1e7
correlation ID: f85455a5-fb5a-4cdd-8e2d-d003c7fb917e
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:154]: failed to create AZ nodepool autoscale-az with autoscaling
Unexpected error:
<*fmt.wrapError | 0x2462c6c0b540>:
failed to create NodePool autoscale-az: failed waiting for nodepool="autoscale-az" for cluster "np-autoscale-cluster" in resourcegroup="np-autoscaling-bl9lknth6jps" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2/hcpOperationStatuses/97c439db-9ae6-4e86-85bd-71d21a1fe1e7
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2/hcpOperationStatuses/97c439db-9ae6-4e86-85bd-71d21a1fe1e7",
"name": "97c439db-9ae6-4e86-85bd-71d21a1fe1e7",
"status": "Failed",
"startTime": "2026-10-27T18:48:30.242794404Z",
"endTime": "2026-10-27T19:07:38.465503369Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift node pool not cached yet"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2/hcpOperationStatuses/97c439db-9ae6-4e86-85bd-71d21a1fe1e7
correlation ID: f85455a5-fb5a-4cdd-8e2d-d003c7fb917e
client request ID: unavailable
occurredContributing tests (1)
Affected runs (2)
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no ...full failure pattern: ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated; [tlsCertificates] TLS certificates not yet confirmed; [clusterServiceClusterStatus] cluster service is pending; [defaultIngressServingCertificate] default ingress certificate not yet observed; provider Microsoft.RedHatOpenShift Signal: IndeterminateSignal: Indeterminate — no prior history; active 2 day(s) — New failure pattern — no prior history | e2e | 2 | 4.35%2 of 46 job runs affected | IndeterminateSignal: Indeterminate — no prior history; active 2 day(s) — New failure pattern — no prior history | Show trend detailsOct 25: 0 · Oct 26: 1 · Oct 27: 0 · Oct 28: 1 · Oct 29: 0 · Oct 30: 0 · Oct 31: 0 · Nov 1: 0 | none | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (2)fail [github.com/Azure/ARO-HCP/test/e2e/cluster_autoscaling.go:108]: failed to create HCP cluster "autoscaling-hcp-cluster" with custom autoscaling
Unexpected error:
<*fmt.wrapError | 0x12961691c220>:
failed to create HCP cluster autoscaling-hcp-cluster: failed waiting for cluster="autoscaling-hcp-cluster" in resourcegroup="autoscaling-cluster-zjbhtbfkbk4z" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/australiaeast/hcpOperationStatuses/b9e8b50e-a67a-443d-9059-6dace5a0e1b3
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/australiaeast/hcpOperationStatuses/b9e8b50e-a67a-443d-9059-6dace5a0e1b3",
"name": "b9e8b50e-a67a-443d-9059-6dace5a0e1b3",
"status": "Failed",
"startTime": "2026-10-28T02:05:53.148166937Z",
"endTime": "2026-10-28T02:34:59.674611282Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated; [tlsCertificates] TLS certificates not yet confirmed; [clusterServiceClusterStatus] cluster service is pending; [defaultIngressServingCertificate] default ingress certificate not yet observed"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/australiaeast/hcpOperationStatuses/b9e8b50e-a67a-443d-9059-6dace5a0e1b3
correlation ID: 5a9d9ef9-c6e1-45c7-a60d-afc7b67a11c1
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_autoscaling.go:108]: failed to create HCP cluster "autoscaling-hcp-cluster" with custom autoscaling
Unexpected error:
<*fmt.wrapError | 0x12961691c220>:
failed to create HCP cluster autoscaling-hcp-cluster: failed waiting for cluster="autoscaling-hcp-cluster" in resourcegroup="autoscaling-cluster-zjbhtbfkbk4z" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/australiaeast/hcpOperationStatuses/b9e8b50e-a67a-443d-9059-6dace5a0e1b3
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/australiaeast/hcpOperationStatuses/b9e8b50e-a67a-443d-9059-6dace5a0e1b3",
"name": "b9e8b50e-a67a-443d-9059-6dace5a0e1b3",
"status": "Failed",
"startTime": "2026-10-28T02:05:53.148166937Z",
"endTime": "2026-10-28T02:34:59.674611282Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated; [tlsCertificates] TLS certificates not yet confirmed; [clusterServiceClusterStatus] cluster service is pending; [defaultIngressServingCertificate] default ingress certificate not yet observed"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/australiaeast/hcpOperationStatuses/b9e8b50e-a67a-443d-9059-6dace5a0e1b3
correlation ID: 5a9d9ef9-c6e1-45c7-a60d-afc7b67a11c1
client request ID: unavailable
occurredfail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:722]: failed to create HCP cluster
Unexpected error:
<*fmt.wrapError | 0x1138be74e660>:
failed to create HCP cluster np-dg-minor-gsd8rp: failed waiting for cluster="np-dg-minor-gsd8rp" in resourcegroup="rg-np-dg-minor-gsd8rp-mv6pwhz6p5qt" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/australiaeast/hcpOperationStatuses/b44a2fb9-01ac-4953-9693-2e6baa16d65c
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/australiaeast/hcpOperationStatuses/b44a2fb9-01ac-4953-9693-2e6baa16d65c",
"name": "b44a2fb9-01ac-4953-9693-2e6baa16d65c",
"status": "Failed",
"startTime": "2026-10-26T17:26:50.386134218Z",
"endTime": "2026-10-26T17:55:54.373104457Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated; [tlsCertificates] TLS certificates not yet confirmed; [clusterServiceClusterStatus] cluster service is pending; [defaultIngressServingCertificate] default ingress certificate not yet observed"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/australiaeast/hcpOperationStatuses/b44a2fb9-01ac-4953-9693-2e6baa16d65c
correlation ID: f252e869-38a8-48ec-8d30-9db8cb2d096a
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:722]: failed to create HCP cluster
Unexpected error:
<*fmt.wrapError | 0x1138be74e660>:
failed to create HCP cluster np-dg-minor-gsd8rp: failed waiting for cluster="np-dg-minor-gsd8rp" in resourcegroup="rg-np-dg-minor-gsd8rp-mv6pwhz6p5qt" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/australiaeast/hcpOperationStatuses/b44a2fb9-01ac-4953-9693-2e6baa16d65c
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/australiaeast/hcpOperationStatuses/b44a2fb9-01ac-4953-9693-2e6baa16d65c",
"name": "b44a2fb9-01ac-4953-9693-2e6baa16d65c",
"status": "Failed",
"startTime": "2026-10-26T17:26:50.386134218Z",
"endTime": "2026-10-26T17:55:54.373104457Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated; [tlsCertificates] TLS certificates not yet confirmed; [clusterServiceClusterStatus] cluster service is pending; [defaultIngressServingCertificate] default ingress certificate not yet observed"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/australiaeast/hcpOperationStatuses/b44a2fb9-01ac-4953-9693-2e6baa16d65c
correlation ID: f252e869-38a8-48ec-8d30-9db8cb2d096a
client request ID: unavailable
occurredContributing tests (2)
Affected runs (2)
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedC...full failure pattern: ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable; hosted cluster degraded: UnavailableReplicas: [catalog-operator, hosted-cluster-config-operator, konnectivity-agent, kube-controller-manager, kube-scheduler, olm-operator, openshift-apiserver, packageserver]; provider Microsoft.RedHatOpenShift Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s) | e2e | 1 | 2.17%1 of 46 job runs affected | IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s) | Show trend detailsOct 25: 0 · Oct 26: 0 · Oct 27: 0 · Oct 28: 6 · Oct 29: 0 · Oct 30: 0 · Oct 31: 0 · Nov 1: 0 | none | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (3)fail [github.com/Azure/ARO-HCP/test/e2e/cluster_autoscaling.go:108]: failed to create HCP cluster "autoscaling-hcp-cluster" with custom autoscaling
Unexpected error:
<*fmt.wrapError | 0xc983e72b20>:
failed to create HCP cluster autoscaling-hcp-cluster: failed waiting for cluster="autoscaling-hcp-cluster" in resourcegroup="autoscaling-cluster-h8vnhnh28g7n" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/338180fd-fd37-4118-8ce5-da19d007d2d1
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/338180fd-fd37-4118-8ce5-da19d007d2d1",
"name": "338180fd-fd37-4118-8ce5-da19d007d2d1",
"status": "Failed",
"startTime": "2026-10-28T19:24:20.13733405Z",
"endTime": "2026-10-28T19:53:23.880336461Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: kube-scheduler, redhat-operators-catalog, redhat-marketplace-catalog, certified-operators-catalog, community-operators-catalog, cluster-node-tuning-operator, cluster-image-registry-operator, cluster-version-operator, kube-controller-manager, catalog-operator, openshift-controller-manager, dns-operator, olm-operator, cluster-network-operator, openshift-route-controller-manager, hosted-cluster-config-operator, cluster-storage-operator, ingress-operator, csi-snapshot-controller-operator, konnectivity-agent, packageserver, cluster-policy-controller; hosted cluster degraded: UnavailableReplicas: [catalog-operator deployment has 1 unavailable replicas, hosted-cluster-config-operator deployment has 1 unavailable replicas, konnectivity-agent deployment has 1 unavailable replicas, kube-controller-manager deployment has 1 unavailable replicas, kube-scheduler deployment has 1 unavailable replicas, olm-operator deployment has 1 unavailable replicas, openshift-apiserver deployment has 1 unavailable replicas, packageserver deployment has 3 unavailable replicas]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/338180fd-fd37-4118-8ce5-da19d007d2d1
correlation ID: 92c6fcaf-99bb-49ef-8360-6f963a51cd80
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_autoscaling.go:108]: failed to create HCP cluster "autoscaling-hcp-cluster" with custom autoscaling
Unexpected error:
<*fmt.wrapError | 0xc983e72b20>:
failed to create HCP cluster autoscaling-hcp-cluster: failed waiting for cluster="autoscaling-hcp-cluster" in resourcegroup="autoscaling-cluster-h8vnhnh28g7n" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/338180fd-fd37-4118-8ce5-da19d007d2d1
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/338180fd-fd37-4118-8ce5-da19d007d2d1",
"name": "338180fd-fd37-4118-8ce5-da19d007d2d1",
"status": "Failed",
"startTime": "2026-10-28T19:24:20.13733405Z",
"endTime": "2026-10-28T19:53:23.880336461Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: kube-scheduler, redhat-operators-catalog, redhat-marketplace-catalog, certified-operators-catalog, community-operators-catalog, cluster-node-tuning-operator, cluster-image-registry-operator, cluster-version-operator, kube-controller-manager, catalog-operator, openshift-controller-manager, dns-operator, olm-operator, cluster-network-operator, openshift-route-controller-manager, hosted-cluster-config-operator, cluster-storage-operator, ingress-operator, csi-snapshot-controller-operator, konnectivity-agent, packageserver, cluster-policy-controller; hosted cluster degraded: UnavailableReplicas: [catalog-operator deployment has 1 unavailable replicas, hosted-cluster-config-operator deployment has 1 unavailable replicas, konnectivity-agent deployment has 1 unavailable replicas, kube-controller-manager deployment has 1 unavailable replicas, kube-scheduler deployment has 1 unavailable replicas, olm-operator deployment has 1 unavailable replicas, openshift-apiserver deployment has 1 unavailable replicas, packageserver deployment has 3 unavailable replicas]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/338180fd-fd37-4118-8ce5-da19d007d2d1
correlation ID: 92c6fcaf-99bb-49ef-8360-6f963a51cd80
client request ID: unavailable
occurredfail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:125]: failed to create HCP cluster "cluster-zstream-4-22-swblz4" with version 4.22.19 on candidate channel
Unexpected error:
<*fmt.wrapError | 0x3981bcd4a5c0>:
failed to create HCP cluster cluster-zstream-4-22-swblz4: failed waiting for cluster="cluster-zstream-4-22-swblz4" in resourcegroup="rg-zstream-upgrade-4-22-nkrsx2q2bfk4" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/fa758444-9da4-4261-b65a-fb9af0416aae
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/fa758444-9da4-4261-b65a-fb9af0416aae",
"name": "fa758444-9da4-4261-b65a-fb9af0416aae",
"status": "Failed",
"startTime": "2026-10-28T20:11:05.048147318Z",
"endTime": "2026-10-28T20:40:15.036753709Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: catalog-operator, cluster-policy-controller, kube-controller-manager, cluster-version-operator, community-operators-catalog, dns-operator, csi-snapshot-controller-operator, konnectivity-agent, olm-operator, kube-scheduler, hosted-cluster-config-operator, cluster-image-registry-operator, redhat-operators-catalog, certified-operators-catalog, openshift-controller-manager, ingress-operator, packageserver, openshift-route-controller-manager, cluster-node-tuning-operator, cluster-storage-operator, cluster-network-operator; hosted cluster degraded: UnavailableReplicas: [catalog-operator deployment has 1 unavailable replicas, hosted-cluster-config-operator deployment has 1 unavailable replicas, konnectivity-agent deployment has 1 unavailable replicas, kube-controller-manager deployment has 1 unavailable replicas, kube-scheduler deployment has 1 unavailable replicas, olm-operator deployment has 1 unavailable replicas, openshift-apiserver deployment has 1 unavailable replicas, packageserver deployment has 3 unavailable replicas]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/fa758444-9da4-4261-b65a-fb9af0416aae
correlation ID: 8fa02e62-e5c9-4192-8dc2-5346dc0423f9
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:125]: failed to create HCP cluster "cluster-zstream-4-22-swblz4" with version 4.22.19 on candidate channel
Unexpected error:
<*fmt.wrapError | 0x3981bcd4a5c0>:
failed to create HCP cluster cluster-zstream-4-22-swblz4: failed waiting for cluster="cluster-zstream-4-22-swblz4" in resourcegroup="rg-zstream-upgrade-4-22-nkrsx2q2bfk4" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/fa758444-9da4-4261-b65a-fb9af0416aae
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/fa758444-9da4-4261-b65a-fb9af0416aae",
"name": "fa758444-9da4-4261-b65a-fb9af0416aae",
"status": "Failed",
"startTime": "2026-10-28T20:11:05.048147318Z",
"endTime": "2026-10-28T20:40:15.036753709Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: catalog-operator, cluster-policy-controller, kube-controller-manager, cluster-version-operator, community-operators-catalog, dns-operator, csi-snapshot-controller-operator, konnectivity-agent, olm-operator, kube-scheduler, hosted-cluster-config-operator, cluster-image-registry-operator, redhat-operators-catalog, certified-operators-catalog, openshift-controller-manager, ingress-operator, packageserver, openshift-route-controller-manager, cluster-node-tuning-operator, cluster-storage-operator, cluster-network-operator; hosted cluster degraded: UnavailableReplicas: [catalog-operator deployment has 1 unavailable replicas, hosted-cluster-config-operator deployment has 1 unavailable replicas, konnectivity-agent deployment has 1 unavailable replicas, kube-controller-manager deployment has 1 unavailable replicas, kube-scheduler deployment has 1 unavailable replicas, olm-operator deployment has 1 unavailable replicas, openshift-apiserver deployment has 1 unavailable replicas, packageserver deployment has 3 unavailable replicas]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/fa758444-9da4-4261-b65a-fb9af0416aae
correlation ID: 8fa02e62-e5c9-4192-8dc2-5346dc0423f9
client request ID: unavailable
occurredfail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:146]: failed to create HCP cluster np-version-upgrade-cluster-dn5ff6 with version 4.21.40
Unexpected error:
<*fmt.wrapError | 0x9790da22600>:
failed to create HCP cluster np-version-upgrade-cluster-dn5ff6: failed waiting for cluster="np-version-upgrade-cluster-dn5ff6" in resourcegroup="rg-np-version-upgrade-dn5ff6-gc44wf2l4jv6" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/e487a831-bc1a-4554-9930-0ef5e5dd164e
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/e487a831-bc1a-4554-9930-0ef5e5dd164e",
"name": "e487a831-bc1a-4554-9930-0ef5e5dd164e",
"status": "Failed",
"startTime": "2026-10-28T20:55:25.009646036Z",
"endTime": "2026-10-28T21:24:30.625209621Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: redhat-operators-catalog, hosted-cluster-config-operator, dns-operator, ingress-operator, catalog-operator, kube-controller-manager, cluster-policy-controller, cluster-node-tuning-operator, cluster-image-registry-operator, kube-scheduler, openshift-controller-manager, openshift-route-controller-manager, cluster-network-operator, csi-snapshot-controller-operator, konnectivity-agent, olm-operator, redhat-marketplace-catalog, community-operators-catalog, certified-operators-catalog, cluster-version-operator, cluster-storage-operator, packageserver; hosted cluster degraded: UnavailableReplicas: [catalog-operator deployment has 1 unavailable replicas, hosted-cluster-config-operator deployment has 1 unavailable replicas, konnectivity-agent deployment has 1 unavailable replicas, kube-controller-manager deployment has 1 unavailable replicas, kube-scheduler deployment has 1 unavailable replicas, olm-operator deployment has 1 unavailable replicas, openshift-apiserver deployment has 1 unavailable replicas, packageserver deployment has 3 unavailable replicas]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/e487a831-bc1a-4554-9930-0ef5e5dd164e
correlation ID: 9f77dae1-d4df-43a9-aae8-bcfb3bc64031
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:146]: failed to create HCP cluster np-version-upgrade-cluster-dn5ff6 with version 4.21.40
Unexpected error:
<*fmt.wrapError | 0x9790da22600>:
failed to create HCP cluster np-version-upgrade-cluster-dn5ff6: failed waiting for cluster="np-version-upgrade-cluster-dn5ff6" in resourcegroup="rg-np-version-upgrade-dn5ff6-gc44wf2l4jv6" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/e487a831-bc1a-4554-9930-0ef5e5dd164e
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/e487a831-bc1a-4554-9930-0ef5e5dd164e",
"name": "e487a831-bc1a-4554-9930-0ef5e5dd164e",
"status": "Failed",
"startTime": "2026-10-28T20:55:25.009646036Z",
"endTime": "2026-10-28T21:24:30.625209621Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: redhat-operators-catalog, hosted-cluster-config-operator, dns-operator, ingress-operator, catalog-operator, kube-controller-manager, cluster-policy-controller, cluster-node-tuning-operator, cluster-image-registry-operator, kube-scheduler, openshift-controller-manager, openshift-route-controller-manager, cluster-network-operator, csi-snapshot-controller-operator, konnectivity-agent, olm-operator, redhat-marketplace-catalog, community-operators-catalog, certified-operators-catalog, cluster-version-operator, cluster-storage-operator, packageserver; hosted cluster degraded: UnavailableReplicas: [catalog-operator deployment has 1 unavailable replicas, hosted-cluster-config-operator deployment has 1 unavailable replicas, konnectivity-agent deployment has 1 unavailable replicas, kube-controller-manager deployment has 1 unavailable replicas, kube-scheduler deployment has 1 unavailable replicas, olm-operator deployment has 1 unavailable replicas, openshift-apiserver deployment has 1 unavailable replicas, packageserver deployment has 3 unavailable replicas]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/e487a831-bc1a-4554-9930-0ef5e5dd164e
correlation ID: 9f77dae1-d4df-43a9-aae8-bcfb3bc64031
client request ID: unavailable
occurredContributing tests (6)
Affected runs (1)
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet complete...full failure pattern: ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version <version> is Partial (want Completed), started <duration> ago; hosted cluster degraded: UnavailableReplicas: [kube-controller-manager]; provider Microsoft.RedHatOpenShift Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s) | e2e | 1 | 2.17%1 of 46 job runs affected | IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s) | Show trend detailsOct 25: 0 · Oct 26: 0 · Oct 27: 0 · Oct 28: 3 · Oct 29: 0 · Oct 30: 0 · Oct 31: 0 · Nov 1: 0 | none | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (3)fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:150]: failed to create HCP cluster
Unexpected error:
<*fmt.wrapError | 0x16967f5def20>:
failed to create HCP cluster ingress-cert-le: failed waiting for cluster="ingress-cert-le" in resourcegroup="ingress-cert-le-z7wppdvdd55v" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/54478daa-47e5-4530-91c2-342cc29a1749
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/54478daa-47e5-4530-91c2-342cc29a1749",
"name": "54478daa-47e5-4530-91c2-342cc29a1749",
"status": "Failed",
"startTime": "2026-10-28T19:24:21.628677591Z",
"endTime": "2026-10-28T19:53:23.838328082Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.22.20 is Partial (want Completed), started 26m9s ago; hosted cluster degraded: UnavailableReplicas: kube-controller-manager deployment has 1 unavailable replicas"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/54478daa-47e5-4530-91c2-342cc29a1749
correlation ID: 7c6978d3-1a22-47a9-861d-b27f5d6a3f88
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:150]: failed to create HCP cluster
Unexpected error:
<*fmt.wrapError | 0x16967f5def20>:
failed to create HCP cluster ingress-cert-le: failed waiting for cluster="ingress-cert-le" in resourcegroup="ingress-cert-le-z7wppdvdd55v" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/54478daa-47e5-4530-91c2-342cc29a1749
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/54478daa-47e5-4530-91c2-342cc29a1749",
"name": "54478daa-47e5-4530-91c2-342cc29a1749",
"status": "Failed",
"startTime": "2026-10-28T19:24:21.628677591Z",
"endTime": "2026-10-28T19:53:23.838328082Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.22.20 is Partial (want Completed), started 26m9s ago; hosted cluster degraded: UnavailableReplicas: kube-controller-manager deployment has 1 unavailable replicas"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/54478daa-47e5-4530-91c2-342cc29a1749
correlation ID: 7c6978d3-1a22-47a9-861d-b27f5d6a3f88
client request ID: unavailable
occurredfail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:150]: failed to create HCP cluster
Unexpected error:
<*fmt.wrapError | 0x3d00e53ba4e0>:
failed to create HCP cluster ingress-cert-le: failed waiting for cluster="ingress-cert-le" in resourcegroup="ingress-cert-le-mpx2klxjj7k9" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/3d9941f3-6ece-4bcd-980b-9e776d1e3227
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/3d9941f3-6ece-4bcd-980b-9e776d1e3227",
"name": "3d9941f3-6ece-4bcd-980b-9e776d1e3227",
"status": "Failed",
"startTime": "2026-10-28T19:24:17.131948207Z",
"endTime": "2026-10-28T19:53:23.815769706Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.20.44 is Partial (want Completed), started 24m27s ago; hosted cluster degraded: UnavailableReplicas: kube-controller-manager deployment has 1 unavailable replicas"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/3d9941f3-6ece-4bcd-980b-9e776d1e3227
correlation ID: c1be684a-2dcb-4ac2-af67-f7bd12ce1f70
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:150]: failed to create HCP cluster
Unexpected error:
<*fmt.wrapError | 0x3d00e53ba4e0>:
failed to create HCP cluster ingress-cert-le: failed waiting for cluster="ingress-cert-le" in resourcegroup="ingress-cert-le-mpx2klxjj7k9" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/3d9941f3-6ece-4bcd-980b-9e776d1e3227
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/3d9941f3-6ece-4bcd-980b-9e776d1e3227",
"name": "3d9941f3-6ece-4bcd-980b-9e776d1e3227",
"status": "Failed",
"startTime": "2026-10-28T19:24:17.131948207Z",
"endTime": "2026-10-28T19:53:23.815769706Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.20.44 is Partial (want Completed), started 24m27s ago; hosted cluster degraded: UnavailableReplicas: kube-controller-manager deployment has 1 unavailable replicas"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/3d9941f3-6ece-4bcd-980b-9e776d1e3227
correlation ID: c1be684a-2dcb-4ac2-af67-f7bd12ce1f70
client request ID: unavailable
occurredfail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:146]: failed to create HCP cluster np-version-upgrade-cluster-95f5bw with version 4.21.40
Unexpected error:
<*fmt.wrapError | 0x1c6ab1a4a4c0>:
failed to create HCP cluster np-version-upgrade-cluster-95f5bw: failed waiting for cluster="np-version-upgrade-cluster-95f5bw" in resourcegroup="rg-np-version-upgrade-95f5bw-mppp9bg5hxs9" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/13f26b26-2324-48a4-82df-d8b324b13996
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/13f26b26-2324-48a4-82df-d8b324b13996",
"name": "13f26b26-2324-48a4-82df-d8b324b13996",
"status": "Failed",
"startTime": "2026-10-28T20:52:21.783931077Z",
"endTime": "2026-10-28T21:21:30.723052246Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.40 is Partial (want Completed), started 26m5s ago; hosted cluster degraded: UnavailableReplicas: kube-controller-manager deployment has 1 unavailable replicas"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/13f26b26-2324-48a4-82df-d8b324b13996
correlation ID: 9f0bfbd0-f67d-4e88-9467-75878a8fe604
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:146]: failed to create HCP cluster np-version-upgrade-cluster-95f5bw with version 4.21.40
Unexpected error:
<*fmt.wrapError | 0x1c6ab1a4a4c0>:
failed to create HCP cluster np-version-upgrade-cluster-95f5bw: failed waiting for cluster="np-version-upgrade-cluster-95f5bw" in resourcegroup="rg-np-version-upgrade-95f5bw-mppp9bg5hxs9" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/13f26b26-2324-48a4-82df-d8b324b13996
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/13f26b26-2324-48a4-82df-d8b324b13996",
"name": "13f26b26-2324-48a4-82df-d8b324b13996",
"status": "Failed",
"startTime": "2026-10-28T20:52:21.783931077Z",
"endTime": "2026-10-28T21:21:30.723052246Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.40 is Partial (want Completed), started 26m5s ago; hosted cluster degraded: UnavailableReplicas: kube-controller-manager deployment has 1 unavailable replicas"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/13f26b26-2324-48a4-82df-d8b324b13996
correlation ID: 9f0bfbd0-f67d-4e88-9467-75878a8fe604
client request ID: unavailable
occurredContributing tests (3)
Affected runs (1)
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Unauthorizedfull failure pattern: VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Unauthorized Signal: IndeterminateSignal: Indeterminate — seen in 4 prior week(s); active 1 day(s) | e2e | 1 | 2.17%1 of 46 job runs affected | IndeterminateSignal: Indeterminate — seen in 4 prior week(s); active 1 day(s) | Show trend detailsOct 25: 0 · Oct 26: 1 · Oct 27: 0 · Oct 28: 0 · Oct 29: 0 · Oct 30: 0 · Oct 31: 0 · Nov 1: 0 | STG | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/e2e/simple_negative_cases.go:124]: cluster negative-tests-cluster is not viable
Unexpected error:
<*errors.joinError | 0x2f7afefabce0>:
VerifyBasicAccess failed: failed to list services: Unauthorized
VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Unauthorized
{
errs: [
<*fmt.wrapError | 0x2f7afec3aa20>{
msg: "VerifyBasicAccess failed: failed to list services: Unauthorized",
err: <*fmt.wrapError | 0x2f7afec3aa00>{
msg: "failed to list services: Unauthorized",
err: <*errors.StatusError | 0x2f7aff732b40>{
ErrStatus: {
TypeMeta: {Kind: "", APIVersion: ""},
ListMeta: {
SelfLink: "",
ResourceVersion: "",
Continue: "",
RemainingItemCount: nil,
ShardInfo: nil,
},
Status: "Failure",
Message: "Unauthorized",
Reason: "Unauthorized",
Details: nil,
Code: 401,
},
},
},
},
<*fmt.wrapError | 0x2f7afee8d220>{
msg: "VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Unauthorized",
err: <*fmt.wrapError | 0x2f7afee8d200>{
msg: "failed to list all APIServices: Unauthorized",
err: <*errors.StatusError | 0x2f7afee1fae0>{
ErrStatus: {
TypeMeta: {Kind: "", APIVersion: ""},
ListMeta: {
SelfLink: "",
ResourceVersion: "",
Continue: "",
RemainingItemCount: nil,
ShardInfo: nil,
},
Status: "Failure",
Message: "Unauthorized",
Reason: "Unauthorized",
Details: nil,
Code: 401,
},
},
},
},
],
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/simple_negative_cases.go:124]: cluster negative-tests-cluster is not viable
Unexpected error:
<*errors.joinError | 0x2f7afefabce0>:
VerifyBasicAccess failed: failed to list services: Unauthorized
VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Unauthorized
{
errs: [
<*fmt.wrapError | 0x2f7afec3aa20>{
msg: "VerifyBasicAccess failed: failed to list services: Unauthorized",
err: <*fmt.wrapError | 0x2f7afec3aa00>{
msg: "failed to list services: Unauthorized",
err: <*errors.StatusError | 0x2f7aff732b40>{
ErrStatus: {
TypeMeta: {Kind: "", APIVersion: ""},
ListMeta: {
SelfLink: "",
ResourceVersion: "",
Continue: "",
RemainingItemCount: nil,
ShardInfo: nil,
},
Status: "Failure",
Message: "Unauthorized",
Reason: "Unauthorized",
Details: nil,
Code: 401,
},
},
},
},
<*fmt.wrapError | 0x2f7afee8d220>{
msg: "VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Unauthorized",
err: <*fmt.wrapError | 0x2f7afee8d200>{
msg: "failed to list all APIServices: Unauthorized",
err: <*errors.StatusError | 0x2f7afee1fae0>{
ErrStatus: {
TypeMeta: {Kind: "", APIVersion: ""},
ListMeta: {
SelfLink: "",
ResourceVersion: "",
Continue: "",
RemainingItemCount: nil,
ShardInfo: nil,
},
Status: "Failure",
Message: "Unauthorized",
Reason: "Unauthorized",
Details: nil,
Code: 401,
},
},
},
},
],
}
occurredContributing tests (1)
Affected runs (1)
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
ERROR CODE: DeadlineExceeded; detail message node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hype...full failure pattern: ERROR CODE: DeadlineExceeded; detail message node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is <count>, want <count>; node pool condition AllNodesHealthy is False: <count> of <count> machines are not healthy Machine <node>: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.Compute/virtualMachines/<node> to exist; node pool condition AllMachinesReady is False: <count> of <count> machines are not ready Machine <node>: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.Compute/virtualMachines/<node> to exist; provider Microsoft.RedHatOpenShift Signal: IndeterminateSignal: Indeterminate — seen in 2 prior week(s); active 1 day(s) | e2e | 1 | 2.17%1 of 46 job runs affected | IndeterminateSignal: Indeterminate — seen in 2 prior week(s); active 1 day(s) | Show trend detailsOct 25: 0 · Oct 26: 0 · Oct 27: 1 · Oct 28: 0 · Oct 29: 0 · Oct 30: 0 · Oct 31: 0 · Nov 1: 0 | DEV | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/e2e/arm64_nodepool.go:109]: failed to create ARM64 nodepool "arm64-vm-np-1" with VM size "Standard_D2plds_v6"
Unexpected error:
<*fmt.wrapError | 0x161cee87c740>:
failed to create NodePool arm64-vm-np-1: failed waiting for nodepool="arm64-vm-np-1" for cluster "arm64-vm-hcp-cluster" in resourcegroup="arm64-vm-cluster-9tq8tw8pkh5c" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/aa729d08-abec-481d-a5d5-3a517e315df9
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/aa729d08-abec-481d-a5d5-3a517e315df9",
"name": "aa729d08-abec-481d-a5d5-3a517e315df9",
"status": "Failed",
"startTime": "2026-10-27T18:45:35.525302675Z",
"endTime": "2026-10-27T19:04:37.944268372Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine q5i4q4e4c0d2d0d-arm64-vm-np-1-qfjp4-9wwh2: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/arm64-vm-cluster-9tq8tw8pkh5c--managed/providers/Microsoft.Compute/virtualMachines/q5i4q4e4c0d2d0d-arm64-vm-np-1-qfjp4-9wwh2 to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine q5i4q4e4c0d2d0d-arm64-vm-np-1-qfjp4-9wwh2: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/arm64-vm-cluster-9tq8tw8pkh5c--managed/providers/Microsoft.Compute/virtualMachines/q5i4q4e4c0d2d0d-arm64-vm-np-1-qfjp4-9wwh2 to exist\n"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/aa729d08-abec-481d-a5d5-3a517e315df9
correlation ID: bbfa9866-a276-4fbe-b94b-bc8519188ea5
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/arm64_nodepool.go:109]: failed to create ARM64 nodepool "arm64-vm-np-1" with VM size "Standard_D2plds_v6"
Unexpected error:
<*fmt.wrapError | 0x161cee87c740>:
failed to create NodePool arm64-vm-np-1: failed waiting for nodepool="arm64-vm-np-1" for cluster "arm64-vm-hcp-cluster" in resourcegroup="arm64-vm-cluster-9tq8tw8pkh5c" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/aa729d08-abec-481d-a5d5-3a517e315df9
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/aa729d08-abec-481d-a5d5-3a517e315df9",
"name": "aa729d08-abec-481d-a5d5-3a517e315df9",
"status": "Failed",
"startTime": "2026-10-27T18:45:35.525302675Z",
"endTime": "2026-10-27T19:04:37.944268372Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine q5i4q4e4c0d2d0d-arm64-vm-np-1-qfjp4-9wwh2: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/arm64-vm-cluster-9tq8tw8pkh5c--managed/providers/Microsoft.Compute/virtualMachines/q5i4q4e4c0d2d0d-arm64-vm-np-1-qfjp4-9wwh2 to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine q5i4q4e4c0d2d0d-arm64-vm-np-1-qfjp4-9wwh2: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/arm64-vm-cluster-9tq8tw8pkh5c--managed/providers/Microsoft.Compute/virtualMachines/q5i4q4e4c0d2d0d-arm64-vm-np-1-qfjp4-9wwh2 to exist\n"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/aa729d08-abec-481d-a5d5-3a517e315df9
correlation ID: bbfa9866-a276-4fbe-b94b-bc8519188ea5
client request ID: unavailable
occurredContributing tests (1)
Affected runs (1)
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedC...full failure pattern: ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable; hosted cluster degraded: UnavailableReplicas: [ignition-server-proxy, kube-apiserver]; provider Microsoft.RedHatOpenShift Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s) | e2e | 1 | 2.17%1 of 46 job runs affected | IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s) | Show trend detailsOct 25: 0 · Oct 26: 0 · Oct 27: 0 · Oct 28: 1 · Oct 29: 0 · Oct 30: 0 · Oct 31: 0 · Nov 1: 0 | none | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_labels_taints.go:92]: failed to create HCP cluster cluster-np-labels-taints
Unexpected error:
<*fmt.wrapError | 0xd2fa34b2580>:
failed to create HCP cluster cluster-np-labels-taints: failed waiting for cluster="cluster-np-labels-taints" in resourcegroup="rg-np-labels-taints-8rnw5kdb6p2l" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/e56720ec-9cdb-4896-bffe-b107388330fa
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/e56720ec-9cdb-4896-bffe-b107388330fa",
"name": "e56720ec-9cdb-4896-bffe-b107388330fa",
"status": "Failed",
"startTime": "2026-10-28T15:21:26.535304826Z",
"endTime": "2026-10-28T15:50:27.739063015Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: openshift-apiserver, cluster-policy-controller, cluster-image-registry-operator, cluster-autoscaler, hosted-cluster-config-operator, cluster-network-operator, dns-operator, catalog-operator, community-operators-catalog, control-plane-pki-operator, azure-cloud-controller-manager, ingress-operator, kube-controller-manager, cluster-storage-operator, konnectivity-agent, packageserver, kube-scheduler, cluster-version-operator, openshift-controller-manager, openshift-route-controller-manager, cluster-node-tuning-operator, olm-collect-profiles, csi-snapshot-controller-operator, redhat-operators-catalog, olm-operator, redhat-marketplace-catalog, machine-approver, certified-operators-catalog; hosted cluster degraded: UnavailableReplicas: [ignition-server-proxy deployment has 1 unavailable replicas, kube-apiserver deployment has 1 unavailable replicas]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/e56720ec-9cdb-4896-bffe-b107388330fa
correlation ID: 87508886-196b-41dd-b14a-e423d6c69bea
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_labels_taints.go:92]: failed to create HCP cluster cluster-np-labels-taints
Unexpected error:
<*fmt.wrapError | 0xd2fa34b2580>:
failed to create HCP cluster cluster-np-labels-taints: failed waiting for cluster="cluster-np-labels-taints" in resourcegroup="rg-np-labels-taints-8rnw5kdb6p2l" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/e56720ec-9cdb-4896-bffe-b107388330fa
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/e56720ec-9cdb-4896-bffe-b107388330fa",
"name": "e56720ec-9cdb-4896-bffe-b107388330fa",
"status": "Failed",
"startTime": "2026-10-28T15:21:26.535304826Z",
"endTime": "2026-10-28T15:50:27.739063015Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: openshift-apiserver, cluster-policy-controller, cluster-image-registry-operator, cluster-autoscaler, hosted-cluster-config-operator, cluster-network-operator, dns-operator, catalog-operator, community-operators-catalog, control-plane-pki-operator, azure-cloud-controller-manager, ingress-operator, kube-controller-manager, cluster-storage-operator, konnectivity-agent, packageserver, kube-scheduler, cluster-version-operator, openshift-controller-manager, openshift-route-controller-manager, cluster-node-tuning-operator, olm-collect-profiles, csi-snapshot-controller-operator, redhat-operators-catalog, olm-operator, redhat-marketplace-catalog, machine-approver, certified-operators-catalog; hosted cluster degraded: UnavailableReplicas: [ignition-server-proxy deployment has 1 unavailable replicas, kube-apiserver deployment has 1 unavailable replicas]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/e56720ec-9cdb-4896-bffe-b107388330fa
correlation ID: 87508886-196b-41dd-b14a-e423d6c69bea
client request ID: unavailable
occurredContributing tests (1)
Affected runs (1)
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedC...full failure pattern: ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider, cluster-api, ignition-server]; provider Microsoft.RedHatOpenShift Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s) | e2e | 1 | 2.17%1 of 46 job runs affected | IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s) | Show trend detailsOct 25: 0 · Oct 26: 0 · Oct 27: 0 · Oct 28: 1 · Oct 29: 0 · Oct 30: 0 · Oct 31: 0 · Nov 1: 0 | none | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delete_cx_rg.go:92]: failed to create HCP cluster "cx-rg-hcp-cluster"
Unexpected error:
<*fmt.wrapError | 0x320c1cc145a0>:
failed to create HCP cluster cx-rg-hcp-cluster: failed waiting for cluster="cx-rg-hcp-cluster" in resourcegroup="cx-rg-cluster-z8bmdl4f2djz" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/6f2afbda-3f8a-4abe-93da-0b42da0b54b6
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/6f2afbda-3f8a-4abe-93da-0b42da0b54b6",
"name": "6f2afbda-3f8a-4abe-93da-0b42da0b54b6",
"status": "Failed",
"startTime": "2026-10-28T13:43:08.173393078Z",
"endTime": "2026-10-28T14:12:10.809270873Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider deployment has 1 unavailable replicas, cluster-api deployment has 2 unavailable replicas, ignition-server deployment has 1 unavailable replicas]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/6f2afbda-3f8a-4abe-93da-0b42da0b54b6
correlation ID: 446aa419-d2ee-4b05-829a-2ef1a0102b8b
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delete_cx_rg.go:92]: failed to create HCP cluster "cx-rg-hcp-cluster"
Unexpected error:
<*fmt.wrapError | 0x320c1cc145a0>:
failed to create HCP cluster cx-rg-hcp-cluster: failed waiting for cluster="cx-rg-hcp-cluster" in resourcegroup="cx-rg-cluster-z8bmdl4f2djz" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/6f2afbda-3f8a-4abe-93da-0b42da0b54b6
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/6f2afbda-3f8a-4abe-93da-0b42da0b54b6",
"name": "6f2afbda-3f8a-4abe-93da-0b42da0b54b6",
"status": "Failed",
"startTime": "2026-10-28T13:43:08.173393078Z",
"endTime": "2026-10-28T14:12:10.809270873Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider deployment has 1 unavailable replicas, cluster-api deployment has 2 unavailable replicas, ignition-server deployment has 1 unavailable replicas]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/6f2afbda-3f8a-4abe-93da-0b42da0b54b6
correlation ID: 446aa419-d2ee-4b05-829a-2ef1a0102b8b
client request ID: unavailable
occurredContributing tests (1)
Affected runs (1)
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet complete...full failure pattern: ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version <version> is Partial (want Completed), started <duration> ago; hosted cluster degraded: UnavailableReplicas: [ignition-server-proxy]; provider Microsoft.RedHatOpenShift Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s) | e2e | 1 | 2.17%1 of 46 job runs affected | IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s) | Show trend detailsOct 25: 0 · Oct 26: 0 · Oct 27: 0 · Oct 28: 1 · Oct 29: 0 · Oct 30: 0 · Oct 31: 0 · Nov 1: 0 | none | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/e2e/cluster_tls_endpoints.go:96]: failed to create HCP cluster for TLS endpoint test
Unexpected error:
<*fmt.wrapError | 0x2df91d94ebc0>:
failed to create HCP cluster tls-endpoint-hcp-cluster: failed waiting for cluster="tls-endpoint-hcp-cluster" in resourcegroup="tls-endpoint-cluster-c9fcp5td85qq" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/57f0f2bf-9284-42db-8d6e-b13db18de25a
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/57f0f2bf-9284-42db-8d6e-b13db18de25a",
"name": "57f0f2bf-9284-42db-8d6e-b13db18de25a",
"status": "Failed",
"startTime": "2026-10-28T19:52:14.429975606Z",
"endTime": "2026-10-28T20:21:16.149275515Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.40 is Partial (want Completed), started 25m21s ago; hosted cluster degraded: UnavailableReplicas: ignition-server-proxy deployment has 1 unavailable replicas"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/57f0f2bf-9284-42db-8d6e-b13db18de25a
correlation ID: 7836a1e8-ab2a-417c-9954-360298a60ae2
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_tls_endpoints.go:96]: failed to create HCP cluster for TLS endpoint test
Unexpected error:
<*fmt.wrapError | 0x2df91d94ebc0>:
failed to create HCP cluster tls-endpoint-hcp-cluster: failed waiting for cluster="tls-endpoint-hcp-cluster" in resourcegroup="tls-endpoint-cluster-c9fcp5td85qq" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/57f0f2bf-9284-42db-8d6e-b13db18de25a
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/57f0f2bf-9284-42db-8d6e-b13db18de25a",
"name": "57f0f2bf-9284-42db-8d6e-b13db18de25a",
"status": "Failed",
"startTime": "2026-10-28T19:52:14.429975606Z",
"endTime": "2026-10-28T20:21:16.149275515Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.40 is Partial (want Completed), started 25m21s ago; hosted cluster degraded: UnavailableReplicas: ignition-server-proxy deployment has 1 unavailable replicas"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/57f0f2bf-9284-42db-8d6e-b13db18de25a
correlation ID: 7836a1e8-ab2a-417c-9954-360298a60ae2
client request ID: unavailable
occurredContributing tests (1)
Affected runs (1)
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
ERROR CODE: InternalServerError; detail message cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster still exists; [clusterServic...full failure pattern: ERROR CODE: InternalServerError; detail message cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster still exists; [clusterServiceStatus] state is "uninstalling"; [descendantResources] remaining resources: nodePools, serviceProviderClusters, serviceProviderNodePools; provider Microsoft.RedHatOpenShift Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s) | e2e | 1 | 2.17%1 of 46 job runs affected | IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s) | Show trend detailsOct 25: 0 · Oct 26: 0 · Oct 27: 0 · Oct 28: 1 · Oct 29: 0 · Oct 30: 0 · Oct 31: 0 · Nov 1: 0 | none | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:310]: Unexpected error:
<*errors.joinError | 0x144b1f0220a8>:
resource group "ingress-cert-le-pqlv4rswg88h": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="ingress-cert-le" in resourcegroup="ingress-cert-le-pqlv4rswg88h" to finish deleting: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/f2c1c705-5b8d-4bbe-98a3-39f7905fcad2
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: InternalServerError
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/f2c1c705-5b8d-4bbe-98a3-39f7905fcad2",
"name": "f2c1c705-5b8d-4bbe-98a3-39f7905fcad2",
"status": "Failed",
"startTime": "2026-10-28T14:05:17.896484451Z",
"endTime": "2026-10-28T14:29:22.39155595Z",
"error": {
"code": "InternalServerError",
"message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2tnqroht6vs8i90m1kekfo1uicveo895 still exists (deletion dispatched at 2026-10-28T14:05:18Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/nodePools, 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/serviceProviderClusters, 1 microsoft.redhatopenshift/hcpopenshiftclusters/nodepools/serviceProviderNodePools"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/f2c1c705-5b8d-4bbe-98a3-39f7905fcad2
correlation ID: 3972a876-9ac5-4cc1-a07c-3501cf7d75bd
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:310]: Unexpected error:
<*errors.joinError | 0x144b1f0220a8>:
resource group "ingress-cert-le-pqlv4rswg88h": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="ingress-cert-le" in resourcegroup="ingress-cert-le-pqlv4rswg88h" to finish deleting: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/f2c1c705-5b8d-4bbe-98a3-39f7905fcad2
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: InternalServerError
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/f2c1c705-5b8d-4bbe-98a3-39f7905fcad2",
"name": "f2c1c705-5b8d-4bbe-98a3-39f7905fcad2",
"status": "Failed",
"startTime": "2026-10-28T14:05:17.896484451Z",
"endTime": "2026-10-28T14:29:22.39155595Z",
"error": {
"code": "InternalServerError",
"message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2tnqroht6vs8i90m1kekfo1uicveo895 still exists (deletion dispatched at 2026-10-28T14:05:18Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/nodePools, 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/serviceProviderClusters, 1 microsoft.redhatopenshift/hcpopenshiftclusters/nodepools/serviceProviderNodePools"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/f2c1c705-5b8d-4bbe-98a3-39f7905fcad2
correlation ID: 3972a876-9ac5-4cc1-a07c-3501cf7d75bd
client request ID: unavailable
occurredContributing tests (1)
Affected runs (1)
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedC...full failure pattern: ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable; hosted cluster degraded: UnavailableReplicas: [kube-apiserver, router]; provider Microsoft.RedHatOpenShift Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s) | e2e | 1 | 2.17%1 of 46 job runs affected | IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s) | Show trend detailsOct 25: 0 · Oct 26: 0 · Oct 27: 0 · Oct 28: 1 · Oct 29: 0 · Oct 30: 0 · Oct 31: 0 · Nov 1: 0 | none | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_delete.go:81]: failed to create HCP cluster np-delete-hcp-cluster
Unexpected error:
<*fmt.wrapError | 0x33d30568cba0>:
failed to create HCP cluster np-delete-hcp-cluster: failed waiting for cluster="np-delete-hcp-cluster" in resourcegroup="nodepool-delete-rn9mbdwr669k" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/30b1eeed-8a58-491f-a679-903582bd2428
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/30b1eeed-8a58-491f-a679-903582bd2428",
"name": "30b1eeed-8a58-491f-a679-903582bd2428",
"status": "Failed",
"startTime": "2026-10-28T15:21:06.381277659Z",
"endTime": "2026-10-28T15:50:07.62996819Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: openshift-route-controller-manager, catalog-operator, cluster-version-operator, cluster-storage-operator, csi-snapshot-controller-operator, cluster-policy-controller, olm-operator, kube-scheduler, dns-operator, certified-operators-catalog, openshift-controller-manager, cluster-image-registry-operator, olm-collect-profiles, redhat-operators-catalog, openshift-apiserver, hosted-cluster-config-operator, cluster-network-operator, konnectivity-agent, machine-approver, kube-controller-manager, azure-cloud-controller-manager, cluster-node-tuning-operator, ingress-operator, packageserver, community-operators-catalog, cluster-autoscaler, control-plane-pki-operator, redhat-marketplace-catalog; hosted cluster degraded: UnavailableReplicas: [kube-apiserver deployment has 1 unavailable replicas, router deployment has 1 unavailable replicas]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/30b1eeed-8a58-491f-a679-903582bd2428
correlation ID: 375547a6-70a5-4d64-8ca8-71de42a27606
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_delete.go:81]: failed to create HCP cluster np-delete-hcp-cluster
Unexpected error:
<*fmt.wrapError | 0x33d30568cba0>:
failed to create HCP cluster np-delete-hcp-cluster: failed waiting for cluster="np-delete-hcp-cluster" in resourcegroup="nodepool-delete-rn9mbdwr669k" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/30b1eeed-8a58-491f-a679-903582bd2428
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/30b1eeed-8a58-491f-a679-903582bd2428",
"name": "30b1eeed-8a58-491f-a679-903582bd2428",
"status": "Failed",
"startTime": "2026-10-28T15:21:06.381277659Z",
"endTime": "2026-10-28T15:50:07.62996819Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: openshift-route-controller-manager, catalog-operator, cluster-version-operator, cluster-storage-operator, csi-snapshot-controller-operator, cluster-policy-controller, olm-operator, kube-scheduler, dns-operator, certified-operators-catalog, openshift-controller-manager, cluster-image-registry-operator, olm-collect-profiles, redhat-operators-catalog, openshift-apiserver, hosted-cluster-config-operator, cluster-network-operator, konnectivity-agent, machine-approver, kube-controller-manager, azure-cloud-controller-manager, cluster-node-tuning-operator, ingress-operator, packageserver, community-operators-catalog, cluster-autoscaler, control-plane-pki-operator, redhat-marketplace-catalog; hosted cluster degraded: UnavailableReplicas: [kube-apiserver deployment has 1 unavailable replicas, router deployment has 1 unavailable replicas]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/30b1eeed-8a58-491f-a679-903582bd2428
correlation ID: 375547a6-70a5-4d64-8ca8-71de42a27606
client request ID: unavailable
occurredContributing tests (1)
Affected runs (1)
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedC...full failure pattern: ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable; hosted cluster degraded: UnavailableReplicas: [capi-provider, ignition-server-proxy, kube-apiserver]; provider Microsoft.RedHatOpenShift Signal: IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | e2e | 1 | 2.17%1 of 46 job runs affected | IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | Show trend detailsOct 25: 0 · Oct 26: 0 · Oct 27: 0 · Oct 28: 1 · Oct 29: 0 · Oct 30: 0 · Oct 31: 0 · Nov 1: 0 | none | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:125]: failed to create HCP cluster "cluster-zstream-5-0-s9k5d8" with version 5.0.0-rc.6 on candidate channel
Unexpected error:
<*fmt.wrapError | 0x219ad233aca0>:
failed to create HCP cluster cluster-zstream-5-0-s9k5d8: failed waiting for cluster="cluster-zstream-5-0-s9k5d8" in resourcegroup="rg-zstream-upgrade-5-0-n6xzg4dztfw8" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/e768a2f1-bf75-4afe-9acf-741914b73091
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/e768a2f1-bf75-4afe-9acf-741914b73091",
"name": "e768a2f1-bf75-4afe-9acf-741914b73091",
"status": "Failed",
"startTime": "2026-10-28T20:11:31.970768227Z",
"endTime": "2026-10-28T20:40:35.054714998Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: control-plane-pki-operator, dns-operator, konnectivity-agent, redhat-operators-catalog, certified-operators-catalog, azure-cloud-controller-manager, csi-snapshot-controller-operator, kube-controller-manager, cluster-version-operator, cluster-policy-controller, hosted-cluster-config-operator, openshift-route-controller-manager, cluster-storage-operator, cluster-network-operator, cluster-autoscaler, machine-approver, kube-scheduler, openshift-apiserver, openshift-controller-manager, cluster-image-registry-operator, catalog-operator, capi-provider, kube-storage-version-migrator, olm-operator, packageserver, olm-collect-profiles, community-operators-catalog, cluster-node-tuning-operator, ingress-operator; hosted cluster degraded: UnavailableReplicas: [capi-provider deployment has 2 unavailable replicas, ignition-server-proxy deployment has 1 unavailable replicas, kube-apiserver deployment has 1 unavailable replicas]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/e768a2f1-bf75-4afe-9acf-741914b73091
correlation ID: 12e6213e-1673-4bde-bd5f-964565fbe8d0
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:125]: failed to create HCP cluster "cluster-zstream-5-0-s9k5d8" with version 5.0.0-rc.6 on candidate channel
Unexpected error:
<*fmt.wrapError | 0x219ad233aca0>:
failed to create HCP cluster cluster-zstream-5-0-s9k5d8: failed waiting for cluster="cluster-zstream-5-0-s9k5d8" in resourcegroup="rg-zstream-upgrade-5-0-n6xzg4dztfw8" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/e768a2f1-bf75-4afe-9acf-741914b73091
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/e768a2f1-bf75-4afe-9acf-741914b73091",
"name": "e768a2f1-bf75-4afe-9acf-741914b73091",
"status": "Failed",
"startTime": "2026-10-28T20:11:31.970768227Z",
"endTime": "2026-10-28T20:40:35.054714998Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: control-plane-pki-operator, dns-operator, konnectivity-agent, redhat-operators-catalog, certified-operators-catalog, azure-cloud-controller-manager, csi-snapshot-controller-operator, kube-controller-manager, cluster-version-operator, cluster-policy-controller, hosted-cluster-config-operator, openshift-route-controller-manager, cluster-storage-operator, cluster-network-operator, cluster-autoscaler, machine-approver, kube-scheduler, openshift-apiserver, openshift-controller-manager, cluster-image-registry-operator, catalog-operator, capi-provider, kube-storage-version-migrator, olm-operator, packageserver, olm-collect-profiles, community-operators-catalog, cluster-node-tuning-operator, ingress-operator; hosted cluster degraded: UnavailableReplicas: [capi-provider deployment has 2 unavailable replicas, ignition-server-proxy deployment has 1 unavailable replicas, kube-apiserver deployment has 1 unavailable replicas]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/e768a2f1-bf75-4afe-9acf-741914b73091
correlation ID: 12e6213e-1673-4bde-bd5f-964565fbe8d0
client request ID: unavailable
occurredContributing tests (1)
Affected runs (1)
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedC...full failure pattern: ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable; hosted cluster degraded: UnavailableReplicas: [catalog-operator, hosted-cluster-config-operator, ignition-server-proxy, kube-scheduler, olm-operator, openshift-apiserver, packageserver]; provider Microsoft.RedHatOpenShift Signal: IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | e2e | 1 | 2.17%1 of 46 job runs affected | IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | Show trend detailsOct 25: 0 · Oct 26: 0 · Oct 27: 0 · Oct 28: 1 · Oct 29: 0 · Oct 30: 0 · Oct 31: 0 · Nov 1: 0 | none | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:582]: failed to create HCP cluster
Unexpected error:
<*fmt.wrapError | 0x1c9f11d305c0>:
failed to create HCP cluster np-downgrade-pgblzc: failed waiting for cluster="np-downgrade-pgblzc" in resourcegroup="rg-np-downgrade-pgblzc-w296czt7c4h5" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/c3f52426-50e3-477e-b6d2-730dd7ef4644
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/c3f52426-50e3-477e-b6d2-730dd7ef4644",
"name": "c3f52426-50e3-477e-b6d2-730dd7ef4644",
"status": "Failed",
"startTime": "2026-10-28T20:57:10.002650956Z",
"endTime": "2026-10-28T21:26:10.63929862Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: packageserver, redhat-operators-catalog, cluster-version-operator, certified-operators-catalog, hosted-cluster-config-operator, cluster-storage-operator, ingress-operator, kube-scheduler, cluster-policy-controller, cluster-network-operator, cluster-node-tuning-operator, redhat-marketplace-catalog, community-operators-catalog, csi-snapshot-controller-operator, openshift-controller-manager, openshift-route-controller-manager, dns-operator, cluster-image-registry-operator, olm-operator, catalog-operator; hosted cluster degraded: UnavailableReplicas: [catalog-operator deployment has 1 unavailable replicas, hosted-cluster-config-operator deployment has 1 unavailable replicas, ignition-server-proxy deployment has 1 unavailable replicas, kube-scheduler deployment has 1 unavailable replicas, olm-operator deployment has 1 unavailable replicas, openshift-apiserver deployment has 1 unavailable replicas, packageserver deployment has 3 unavailable replicas]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/c3f52426-50e3-477e-b6d2-730dd7ef4644
correlation ID: 3652d2cb-5ad7-44b4-9342-18162d59e0da
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:582]: failed to create HCP cluster
Unexpected error:
<*fmt.wrapError | 0x1c9f11d305c0>:
failed to create HCP cluster np-downgrade-pgblzc: failed waiting for cluster="np-downgrade-pgblzc" in resourcegroup="rg-np-downgrade-pgblzc-w296czt7c4h5" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/c3f52426-50e3-477e-b6d2-730dd7ef4644
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/c3f52426-50e3-477e-b6d2-730dd7ef4644",
"name": "c3f52426-50e3-477e-b6d2-730dd7ef4644",
"status": "Failed",
"startTime": "2026-10-28T20:57:10.002650956Z",
"endTime": "2026-10-28T21:26:10.63929862Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: packageserver, redhat-operators-catalog, cluster-version-operator, certified-operators-catalog, hosted-cluster-config-operator, cluster-storage-operator, ingress-operator, kube-scheduler, cluster-policy-controller, cluster-network-operator, cluster-node-tuning-operator, redhat-marketplace-catalog, community-operators-catalog, csi-snapshot-controller-operator, openshift-controller-manager, openshift-route-controller-manager, dns-operator, cluster-image-registry-operator, olm-operator, catalog-operator; hosted cluster degraded: UnavailableReplicas: [catalog-operator deployment has 1 unavailable replicas, hosted-cluster-config-operator deployment has 1 unavailable replicas, ignition-server-proxy deployment has 1 unavailable replicas, kube-scheduler deployment has 1 unavailable replicas, olm-operator deployment has 1 unavailable replicas, openshift-apiserver deployment has 1 unavailable replicas, packageserver deployment has 3 unavailable replicas]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/c3f52426-50e3-477e-b6d2-730dd7ef4644
correlation ID: 3652d2cb-5ad7-44b4-9342-18162d59e0da
client request ID: unavailable
occurredContributing tests (1)
Affected runs (1)
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedC...full failure pattern: ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable; hosted cluster degraded: UnavailableReplicas: [catalog-operator, hosted-cluster-config-operator, kube-controller-manager, olm-operator, openshift-apiserver, packageserver]; provider Microsoft.RedHatOpenShift Signal: IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | e2e | 1 | 2.17%1 of 46 job runs affected | IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | Show trend detailsOct 25: 0 · Oct 26: 0 · Oct 27: 0 · Oct 28: 1 · Oct 29: 0 · Oct 30: 0 · Oct 31: 0 · Nov 1: 0 | none | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:98]: failed to create HCP cluster for external auth test
Unexpected error:
<*fmt.wrapError | 0x3da3f19209c0>:
failed to create HCP cluster ea-cluster: failed waiting for cluster="ea-cluster" in resourcegroup="external-auth-cluster-p9cvh6b6lj55" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/9c5a9e43-a50b-4146-b8d2-de67c8eeeb10
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/9c5a9e43-a50b-4146-b8d2-de67c8eeeb10",
"name": "9c5a9e43-a50b-4146-b8d2-de67c8eeeb10",
"status": "Failed",
"startTime": "2026-10-28T20:14:36.875741431Z",
"endTime": "2026-10-28T20:43:44.887394978Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: cluster-node-tuning-operator, dns-operator, catalog-operator, certified-operators-catalog, csi-snapshot-controller-operator, redhat-marketplace-catalog, olm-operator, cluster-policy-controller, hosted-cluster-config-operator, cluster-network-operator, packageserver, ingress-operator, cluster-image-registry-operator, kube-controller-manager, cluster-version-operator, redhat-operators-catalog, openshift-controller-manager, openshift-route-controller-manager, cluster-storage-operator, community-operators-catalog; hosted cluster degraded: UnavailableReplicas: [catalog-operator deployment has 1 unavailable replicas, hosted-cluster-config-operator deployment has 1 unavailable replicas, kube-controller-manager deployment has 1 unavailable replicas, olm-operator deployment has 1 unavailable replicas, openshift-apiserver deployment has 1 unavailable replicas, packageserver deployment has 3 unavailable replicas]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/9c5a9e43-a50b-4146-b8d2-de67c8eeeb10
correlation ID: 84abeaa9-4051-418b-b066-32246401955a
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:98]: failed to create HCP cluster for external auth test
Unexpected error:
<*fmt.wrapError | 0x3da3f19209c0>:
failed to create HCP cluster ea-cluster: failed waiting for cluster="ea-cluster" in resourcegroup="external-auth-cluster-p9cvh6b6lj55" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/9c5a9e43-a50b-4146-b8d2-de67c8eeeb10
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/9c5a9e43-a50b-4146-b8d2-de67c8eeeb10",
"name": "9c5a9e43-a50b-4146-b8d2-de67c8eeeb10",
"status": "Failed",
"startTime": "2026-10-28T20:14:36.875741431Z",
"endTime": "2026-10-28T20:43:44.887394978Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: cluster-node-tuning-operator, dns-operator, catalog-operator, certified-operators-catalog, csi-snapshot-controller-operator, redhat-marketplace-catalog, olm-operator, cluster-policy-controller, hosted-cluster-config-operator, cluster-network-operator, packageserver, ingress-operator, cluster-image-registry-operator, kube-controller-manager, cluster-version-operator, redhat-operators-catalog, openshift-controller-manager, openshift-route-controller-manager, cluster-storage-operator, community-operators-catalog; hosted cluster degraded: UnavailableReplicas: [catalog-operator deployment has 1 unavailable replicas, hosted-cluster-config-operator deployment has 1 unavailable replicas, kube-controller-manager deployment has 1 unavailable replicas, olm-operator deployment has 1 unavailable replicas, openshift-apiserver deployment has 1 unavailable replicas, packageserver deployment has 3 unavailable replicas]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/9c5a9e43-a50b-4146-b8d2-de67c8eeeb10
correlation ID: 84abeaa9-4051-418b-b066-32246401955a
client request ID: unavailable
occurredContributing tests (1)
Affected runs (1)
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedC...full failure pattern: ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable; hosted cluster degraded: UnavailableReplicas: [catalog-operator, hosted-cluster-config-operator, kube-scheduler, olm-operator, openshift-apiserver, packageserver]; provider Microsoft.RedHatOpenShift Signal: IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | e2e | 1 | 2.17%1 of 46 job runs affected | IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | Show trend detailsOct 25: 0 · Oct 26: 0 · Oct 27: 0 · Oct 28: 1 · Oct 29: 0 · Oct 30: 0 · Oct 31: 0 · Nov 1: 0 | none | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:303]: failed to create HCP cluster np-noedge-2tf7pf
Unexpected error:
<*fmt.wrapError | 0x257516f84460>:
failed to create HCP cluster np-noedge-2tf7pf: failed waiting for cluster="np-noedge-2tf7pf" in resourcegroup="rg-np-noedge-2tf7pf-xb8pwtjzh82j" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/92128bbf-b05b-4a42-b590-261b4b996ffc
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/92128bbf-b05b-4a42-b590-261b4b996ffc",
"name": "92128bbf-b05b-4a42-b590-261b4b996ffc",
"status": "Failed",
"startTime": "2026-10-28T15:21:31.746407548Z",
"endTime": "2026-10-28T15:50:37.764425706Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: catalog-operator, openshift-controller-manager, openshift-route-controller-manager, redhat-operators-catalog, community-operators-catalog, ingress-operator, csi-snapshot-controller-operator, packageserver, cluster-image-registry-operator, dns-operator, cluster-policy-controller, hosted-cluster-config-operator, cluster-node-tuning-operator, cluster-version-operator, cluster-storage-operator, redhat-marketplace-catalog, certified-operators-catalog, cluster-network-operator, olm-operator; hosted cluster degraded: UnavailableReplicas: [catalog-operator deployment has 1 unavailable replicas, hosted-cluster-config-operator deployment has 1 unavailable replicas, kube-scheduler deployment has 1 unavailable replicas, olm-operator deployment has 1 unavailable replicas, openshift-apiserver deployment has 1 unavailable replicas, packageserver deployment has 3 unavailable replicas]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/92128bbf-b05b-4a42-b590-261b4b996ffc
correlation ID: b2029c0e-c3d1-46d0-9bac-f3b40a92dc09
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:303]: failed to create HCP cluster np-noedge-2tf7pf
Unexpected error:
<*fmt.wrapError | 0x257516f84460>:
failed to create HCP cluster np-noedge-2tf7pf: failed waiting for cluster="np-noedge-2tf7pf" in resourcegroup="rg-np-noedge-2tf7pf-xb8pwtjzh82j" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/92128bbf-b05b-4a42-b590-261b4b996ffc
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/92128bbf-b05b-4a42-b590-261b4b996ffc",
"name": "92128bbf-b05b-4a42-b590-261b4b996ffc",
"status": "Failed",
"startTime": "2026-10-28T15:21:31.746407548Z",
"endTime": "2026-10-28T15:50:37.764425706Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: catalog-operator, openshift-controller-manager, openshift-route-controller-manager, redhat-operators-catalog, community-operators-catalog, ingress-operator, csi-snapshot-controller-operator, packageserver, cluster-image-registry-operator, dns-operator, cluster-policy-controller, hosted-cluster-config-operator, cluster-node-tuning-operator, cluster-version-operator, cluster-storage-operator, redhat-marketplace-catalog, certified-operators-catalog, cluster-network-operator, olm-operator; hosted cluster degraded: UnavailableReplicas: [catalog-operator deployment has 1 unavailable replicas, hosted-cluster-config-operator deployment has 1 unavailable replicas, kube-scheduler deployment has 1 unavailable replicas, olm-operator deployment has 1 unavailable replicas, openshift-apiserver deployment has 1 unavailable replicas, packageserver deployment has 3 unavailable replicas]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/92128bbf-b05b-4a42-b590-261b4b996ffc
correlation ID: b2029c0e-c3d1-46d0-9bac-f3b40a92dc09
client request ID: unavailable
occurredContributing tests (1)
Affected runs (1)
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedC...full failure pattern: ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable; hosted cluster degraded: UnavailableReplicas: [cluster-policy-controller, konnectivity-agent, kube-scheduler, openshift-controller-manager, openshift-route-controller-manager]; provider Microsoft.RedHatOpenShift Signal: IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | e2e | 1 | 2.17%1 of 46 job runs affected | IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | Show trend detailsOct 25: 0 · Oct 26: 0 · Oct 27: 0 · Oct 28: 1 · Oct 29: 0 · Oct 30: 0 · Oct 31: 0 · Nov 1: 0 | none | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_ephemeral_osdisk.go:84]: failed to create HCP cluster ephemeral-disk
Unexpected error:
<*fmt.wrapError | 0x2840a3490800>:
failed to create HCP cluster ephemeral-disk: failed waiting for cluster="ephemeral-disk" in resourcegroup="ephemeral-osdisk-9tjwn77pblt6" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/f493db72-5a08-4ef3-8c73-4d055bd2007e
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/f493db72-5a08-4ef3-8c73-4d055bd2007e",
"name": "f493db72-5a08-4ef3-8c73-4d055bd2007e",
"status": "Failed",
"startTime": "2026-10-28T20:48:29.562853635Z",
"endTime": "2026-10-28T21:17:30.601011159Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: kube-scheduler, openshift-route-controller-manager, cluster-policy-controller, konnectivity-agent, openshift-controller-manager; hosted cluster degraded: UnavailableReplicas: [cluster-policy-controller deployment has 1 unavailable replicas, konnectivity-agent deployment has 1 unavailable replicas, kube-scheduler deployment has 1 unavailable replicas, openshift-controller-manager deployment has 1 unavailable replicas, openshift-route-controller-manager deployment has 1 unavailable replicas]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/f493db72-5a08-4ef3-8c73-4d055bd2007e
correlation ID: 12f04724-ca79-49ae-ac94-baef82ad2b08
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_ephemeral_osdisk.go:84]: failed to create HCP cluster ephemeral-disk
Unexpected error:
<*fmt.wrapError | 0x2840a3490800>:
failed to create HCP cluster ephemeral-disk: failed waiting for cluster="ephemeral-disk" in resourcegroup="ephemeral-osdisk-9tjwn77pblt6" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/f493db72-5a08-4ef3-8c73-4d055bd2007e
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/f493db72-5a08-4ef3-8c73-4d055bd2007e",
"name": "f493db72-5a08-4ef3-8c73-4d055bd2007e",
"status": "Failed",
"startTime": "2026-10-28T20:48:29.562853635Z",
"endTime": "2026-10-28T21:17:30.601011159Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: kube-scheduler, openshift-route-controller-manager, cluster-policy-controller, konnectivity-agent, openshift-controller-manager; hosted cluster degraded: UnavailableReplicas: [cluster-policy-controller deployment has 1 unavailable replicas, konnectivity-agent deployment has 1 unavailable replicas, kube-scheduler deployment has 1 unavailable replicas, openshift-controller-manager deployment has 1 unavailable replicas, openshift-route-controller-manager deployment has 1 unavailable replicas]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/f493db72-5a08-4ef3-8c73-4d055bd2007e
correlation ID: 12f04724-ca79-49ae-ac94-baef82ad2b08
client request ID: unavailable
occurredContributing tests (1)
Affected runs (1)
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedC...full failure pattern: ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable; hosted cluster degraded: UnavailableReplicas: [cluster-policy-controller, konnectivity-agent, kube-scheduler, openshift-controller-manager]; provider Microsoft.RedHatOpenShift Signal: IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | e2e | 1 | 2.17%1 of 46 job runs affected | IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | Show trend detailsOct 25: 0 · Oct 26: 0 · Oct 27: 0 · Oct 28: 1 · Oct 29: 0 · Oct 30: 0 · Oct 31: 0 · Nov 1: 0 | none | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/e2e/oidc_issuer_workload_identity.go:227]: failed to create HCP cluster oidc-wi-cluster
Unexpected error:
<*fmt.wrapError | 0x3290e061a680>:
failed to create HCP cluster oidc-wi-cluster: failed waiting for cluster="oidc-wi-cluster" in resourcegroup="oidc-wi-qdsgxxj79bfq" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/5e4ef4af-8f03-4c8b-bbbb-d82bdd2ed0e9
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/5e4ef4af-8f03-4c8b-bbbb-d82bdd2ed0e9",
"name": "5e4ef4af-8f03-4c8b-bbbb-d82bdd2ed0e9",
"status": "Failed",
"startTime": "2026-10-28T20:57:27.436526224Z",
"endTime": "2026-10-28T21:26:30.622804028Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: konnectivity-agent, cluster-policy-controller, openshift-controller-manager, kube-scheduler; hosted cluster degraded: UnavailableReplicas: [cluster-policy-controller deployment has 1 unavailable replicas, konnectivity-agent deployment has 1 unavailable replicas, kube-scheduler deployment has 1 unavailable replicas, openshift-controller-manager deployment has 1 unavailable replicas]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/5e4ef4af-8f03-4c8b-bbbb-d82bdd2ed0e9
correlation ID: 2fd33249-a131-4a13-a1a2-03c755186865
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/oidc_issuer_workload_identity.go:227]: failed to create HCP cluster oidc-wi-cluster
Unexpected error:
<*fmt.wrapError | 0x3290e061a680>:
failed to create HCP cluster oidc-wi-cluster: failed waiting for cluster="oidc-wi-cluster" in resourcegroup="oidc-wi-qdsgxxj79bfq" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/5e4ef4af-8f03-4c8b-bbbb-d82bdd2ed0e9
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/5e4ef4af-8f03-4c8b-bbbb-d82bdd2ed0e9",
"name": "5e4ef4af-8f03-4c8b-bbbb-d82bdd2ed0e9",
"status": "Failed",
"startTime": "2026-10-28T20:57:27.436526224Z",
"endTime": "2026-10-28T21:26:30.622804028Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: konnectivity-agent, cluster-policy-controller, openshift-controller-manager, kube-scheduler; hosted cluster degraded: UnavailableReplicas: [cluster-policy-controller deployment has 1 unavailable replicas, konnectivity-agent deployment has 1 unavailable replicas, kube-scheduler deployment has 1 unavailable replicas, openshift-controller-manager deployment has 1 unavailable replicas]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/5e4ef4af-8f03-4c8b-bbbb-d82bdd2ed0e9
correlation ID: 2fd33249-a131-4a13-a1a2-03c755186865
client request ID: unavailable
occurredContributing tests (1)
Affected runs (1)
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedC...full failure pattern: ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable; hosted cluster degraded: UnavailableReplicas: [cluster-policy-controller, konnectivity-agent, openshift-controller-manager, openshift-route-controller-manager, packageserver]; provider Microsoft.RedHatOpenShift Signal: IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | e2e | 1 | 2.17%1 of 46 job runs affected | IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | Show trend detailsOct 25: 0 · Oct 26: 0 · Oct 27: 0 · Oct 28: 1 · Oct 29: 0 · Oct 30: 0 · Oct 31: 0 · Nov 1: 0 | none | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:168]: failed to create HCP cluster for patch-tags test
Unexpected error:
<*fmt.wrapError | 0x1d947782ea20>:
failed to create HCP cluster patch-tags-cluster: failed waiting for cluster="patch-tags-cluster" in resourcegroup="patch-tags-w5rx25h5j8dw" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/0a9f3022-f355-43e2-9259-81dfd53aa74e
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/0a9f3022-f355-43e2-9259-81dfd53aa74e",
"name": "0a9f3022-f355-43e2-9259-81dfd53aa74e",
"status": "Failed",
"startTime": "2026-10-28T20:01:08.393167298Z",
"endTime": "2026-10-28T20:30:16.146752142Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: openshift-route-controller-manager, openshift-controller-manager, cluster-policy-controller, konnectivity-agent; hosted cluster degraded: UnavailableReplicas: [cluster-policy-controller deployment has 1 unavailable replicas, konnectivity-agent deployment has 1 unavailable replicas, openshift-controller-manager deployment has 1 unavailable replicas, openshift-route-controller-manager deployment has 1 unavailable replicas, packageserver deployment has 1 unavailable replicas]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/0a9f3022-f355-43e2-9259-81dfd53aa74e
correlation ID: 1eb4abcf-932d-4da4-828c-22172d8177f5
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:168]: failed to create HCP cluster for patch-tags test
Unexpected error:
<*fmt.wrapError | 0x1d947782ea20>:
failed to create HCP cluster patch-tags-cluster: failed waiting for cluster="patch-tags-cluster" in resourcegroup="patch-tags-w5rx25h5j8dw" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/0a9f3022-f355-43e2-9259-81dfd53aa74e
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/0a9f3022-f355-43e2-9259-81dfd53aa74e",
"name": "0a9f3022-f355-43e2-9259-81dfd53aa74e",
"status": "Failed",
"startTime": "2026-10-28T20:01:08.393167298Z",
"endTime": "2026-10-28T20:30:16.146752142Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: openshift-route-controller-manager, openshift-controller-manager, cluster-policy-controller, konnectivity-agent; hosted cluster degraded: UnavailableReplicas: [cluster-policy-controller deployment has 1 unavailable replicas, konnectivity-agent deployment has 1 unavailable replicas, openshift-controller-manager deployment has 1 unavailable replicas, openshift-route-controller-manager deployment has 1 unavailable replicas, packageserver deployment has 1 unavailable replicas]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/0a9f3022-f355-43e2-9259-81dfd53aa74e
correlation ID: 1eb4abcf-932d-4da4-828c-22172d8177f5
client request ID: unavailable
occurredContributing tests (1)
Affected runs (1)
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedC...full failure pattern: ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable; hosted cluster degraded: UnavailableReplicas: [cluster-policy-controller, kube-controller-manager, kube-scheduler, openshift-controller-manager, openshift-route-controller-manager]; provider Microsoft.RedHatOpenShift Signal: IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | e2e | 1 | 2.17%1 of 46 job runs affected | IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | Show trend detailsOct 25: 0 · Oct 26: 0 · Oct 27: 0 · Oct 28: 1 · Oct 29: 0 · Oct 30: 0 · Oct 31: 0 · Nov 1: 0 | none | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:146]: failed to create HCP cluster np-version-upgrade-cluster-xgw4k6 with version 4.20.44
Unexpected error:
<*fmt.wrapError | 0x2f2f6cd22960>:
failed to create HCP cluster np-version-upgrade-cluster-xgw4k6: failed waiting for cluster="np-version-upgrade-cluster-xgw4k6" in resourcegroup="rg-np-version-upgrade-xgw4k6-fb2pqq2pp9pn" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/67af219e-4e40-4a49-ae77-1d482039a909
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/67af219e-4e40-4a49-ae77-1d482039a909",
"name": "67af219e-4e40-4a49-ae77-1d482039a909",
"status": "Failed",
"startTime": "2026-10-28T15:21:37.478622401Z",
"endTime": "2026-10-28T15:50:37.723667825Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: openshift-controller-manager, kube-scheduler, cluster-policy-controller, openshift-route-controller-manager; hosted cluster degraded: UnavailableReplicas: [cluster-policy-controller deployment has 1 unavailable replicas, kube-controller-manager deployment has 1 unavailable replicas, kube-scheduler deployment has 1 unavailable replicas, openshift-controller-manager deployment has 1 unavailable replicas, openshift-route-controller-manager deployment has 1 unavailable replicas]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/67af219e-4e40-4a49-ae77-1d482039a909
correlation ID: b05be546-d520-46a0-8713-ceb437dfacc8
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:146]: failed to create HCP cluster np-version-upgrade-cluster-xgw4k6 with version 4.20.44
Unexpected error:
<*fmt.wrapError | 0x2f2f6cd22960>:
failed to create HCP cluster np-version-upgrade-cluster-xgw4k6: failed waiting for cluster="np-version-upgrade-cluster-xgw4k6" in resourcegroup="rg-np-version-upgrade-xgw4k6-fb2pqq2pp9pn" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/67af219e-4e40-4a49-ae77-1d482039a909
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/67af219e-4e40-4a49-ae77-1d482039a909",
"name": "67af219e-4e40-4a49-ae77-1d482039a909",
"status": "Failed",
"startTime": "2026-10-28T15:21:37.478622401Z",
"endTime": "2026-10-28T15:50:37.723667825Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: openshift-controller-manager, kube-scheduler, cluster-policy-controller, openshift-route-controller-manager; hosted cluster degraded: UnavailableReplicas: [cluster-policy-controller deployment has 1 unavailable replicas, kube-controller-manager deployment has 1 unavailable replicas, kube-scheduler deployment has 1 unavailable replicas, openshift-controller-manager deployment has 1 unavailable replicas, openshift-route-controller-manager deployment has 1 unavailable replicas]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/67af219e-4e40-4a49-ae77-1d482039a909
correlation ID: b05be546-d520-46a0-8713-ceb437dfacc8
client request ID: unavailable
occurredContributing tests (1)
Affected runs (1)
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedC...full failure pattern: ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable; hosted cluster degraded: UnavailableReplicas: [cluster-policy-controller, kube-controller-manager]; provider Microsoft.RedHatOpenShift Signal: IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | e2e | 1 | 2.17%1 of 46 job runs affected | IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | Show trend detailsOct 25: 0 · Oct 26: 0 · Oct 27: 0 · Oct 28: 1 · Oct 29: 0 · Oct 30: 0 · Oct 31: 0 · Nov 1: 0 | none | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:91]: failed to create HCP cluster for patch-name test
Unexpected error:
<*fmt.wrapError | 0x28ad82ebac20>:
failed to create HCP cluster patch-name-cluster: failed waiting for cluster="patch-name-cluster" in resourcegroup="patch-name-8dptc8vk8gtv" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/d2708466-dcce-4112-be7d-6e0e6a3a5117
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/d2708466-dcce-4112-be7d-6e0e6a3a5117",
"name": "d2708466-dcce-4112-be7d-6e0e6a3a5117",
"status": "Failed",
"startTime": "2026-10-28T20:01:11.67931549Z",
"endTime": "2026-10-28T20:30:16.178438369Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: cluster-policy-controller; hosted cluster degraded: UnavailableReplicas: [cluster-policy-controller deployment has 1 unavailable replicas, kube-controller-manager deployment has 1 unavailable replicas]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/d2708466-dcce-4112-be7d-6e0e6a3a5117
correlation ID: 66a835d6-6997-4516-8fbc-7fd19b5ac311
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:91]: failed to create HCP cluster for patch-name test
Unexpected error:
<*fmt.wrapError | 0x28ad82ebac20>:
failed to create HCP cluster patch-name-cluster: failed waiting for cluster="patch-name-cluster" in resourcegroup="patch-name-8dptc8vk8gtv" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/d2708466-dcce-4112-be7d-6e0e6a3a5117
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/d2708466-dcce-4112-be7d-6e0e6a3a5117",
"name": "d2708466-dcce-4112-be7d-6e0e6a3a5117",
"status": "Failed",
"startTime": "2026-10-28T20:01:11.67931549Z",
"endTime": "2026-10-28T20:30:16.178438369Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: cluster-policy-controller; hosted cluster degraded: UnavailableReplicas: [cluster-policy-controller deployment has 1 unavailable replicas, kube-controller-manager deployment has 1 unavailable replicas]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/d2708466-dcce-4112-be7d-6e0e6a3a5117
correlation ID: 66a835d6-6997-4516-8fbc-7fd19b5ac311
client request ID: unavailable
occurredContributing tests (1)
Affected runs (1)
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedC...full failure pattern: ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable; hosted cluster degraded: UnavailableReplicas: [cluster-policy-controller, openshift-controller-manager, openshift-route-controller-manager]; provider Microsoft.RedHatOpenShift Signal: IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | e2e | 1 | 2.17%1 of 46 job runs affected | IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | Show trend detailsOct 25: 0 · Oct 26: 0 · Oct 27: 0 · Oct 28: 1 · Oct 29: 0 · Oct 30: 0 · Oct 31: 0 · Nov 1: 0 | none | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:91]: failed to create HCP cluster for patch-name test
Unexpected error:
<*fmt.wrapError | 0x3d893d926940>:
failed to create HCP cluster patch-name-cluster: failed waiting for cluster="patch-name-cluster" in resourcegroup="patch-name-9zwtss6f6vt9" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/7bc6a3bc-3918-4c91-853b-58acb06d5c9a
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/7bc6a3bc-3918-4c91-853b-58acb06d5c9a",
"name": "7bc6a3bc-3918-4c91-853b-58acb06d5c9a",
"status": "Failed",
"startTime": "2026-10-28T14:39:20.447678223Z",
"endTime": "2026-10-28T15:08:28.688541915Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: cluster-policy-controller, openshift-route-controller-manager, openshift-controller-manager; hosted cluster degraded: UnavailableReplicas: [cluster-policy-controller deployment has 1 unavailable replicas, openshift-controller-manager deployment has 1 unavailable replicas, openshift-route-controller-manager deployment has 1 unavailable replicas]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/7bc6a3bc-3918-4c91-853b-58acb06d5c9a
correlation ID: 2b84f196-091a-41b2-98d7-60b50644dab4
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:91]: failed to create HCP cluster for patch-name test
Unexpected error:
<*fmt.wrapError | 0x3d893d926940>:
failed to create HCP cluster patch-name-cluster: failed waiting for cluster="patch-name-cluster" in resourcegroup="patch-name-9zwtss6f6vt9" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/7bc6a3bc-3918-4c91-853b-58acb06d5c9a
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/7bc6a3bc-3918-4c91-853b-58acb06d5c9a",
"name": "7bc6a3bc-3918-4c91-853b-58acb06d5c9a",
"status": "Failed",
"startTime": "2026-10-28T14:39:20.447678223Z",
"endTime": "2026-10-28T15:08:28.688541915Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: cluster-policy-controller, openshift-route-controller-manager, openshift-controller-manager; hosted cluster degraded: UnavailableReplicas: [cluster-policy-controller deployment has 1 unavailable replicas, openshift-controller-manager deployment has 1 unavailable replicas, openshift-route-controller-manager deployment has 1 unavailable replicas]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/7bc6a3bc-3918-4c91-853b-58acb06d5c9a
correlation ID: 2b84f196-091a-41b2-98d7-60b50644dab4
client request ID: unavailable
occurredContributing tests (1)
Affected runs (1)
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedC...full failure pattern: ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable; hosted cluster degraded: UnavailableReplicas: [konnectivity-agent, kube-controller-manager, kube-scheduler]; provider Microsoft.RedHatOpenShift Signal: IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | e2e | 1 | 2.17%1 of 46 job runs affected | IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | Show trend detailsOct 25: 0 · Oct 26: 0 · Oct 27: 0 · Oct 28: 1 · Oct 29: 0 · Oct 30: 0 · Oct 31: 0 · Nov 1: 0 | none | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:150]: failed to create HCP cluster
Unexpected error:
<*fmt.wrapError | 0x186bf96476a0>:
failed to create HCP cluster ingress-cert-le: failed waiting for cluster="ingress-cert-le" in resourcegroup="ingress-cert-le-f9vz9brpqmbx" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/07dce0e9-a762-440b-9ccd-61b15fa63bf8
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/07dce0e9-a762-440b-9ccd-61b15fa63bf8",
"name": "07dce0e9-a762-440b-9ccd-61b15fa63bf8",
"status": "Failed",
"startTime": "2026-10-28T13:42:56.996297525Z",
"endTime": "2026-10-28T14:12:00.914226401Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: kube-scheduler, konnectivity-agent; hosted cluster degraded: UnavailableReplicas: [konnectivity-agent deployment has 1 unavailable replicas, kube-controller-manager deployment has 1 unavailable replicas, kube-scheduler deployment has 1 unavailable replicas]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/07dce0e9-a762-440b-9ccd-61b15fa63bf8
correlation ID: b4a6856e-5332-498a-8060-89b9ebb7175d
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:150]: failed to create HCP cluster
Unexpected error:
<*fmt.wrapError | 0x186bf96476a0>:
failed to create HCP cluster ingress-cert-le: failed waiting for cluster="ingress-cert-le" in resourcegroup="ingress-cert-le-f9vz9brpqmbx" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/07dce0e9-a762-440b-9ccd-61b15fa63bf8
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/07dce0e9-a762-440b-9ccd-61b15fa63bf8",
"name": "07dce0e9-a762-440b-9ccd-61b15fa63bf8",
"status": "Failed",
"startTime": "2026-10-28T13:42:56.996297525Z",
"endTime": "2026-10-28T14:12:00.914226401Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: kube-scheduler, konnectivity-agent; hosted cluster degraded: UnavailableReplicas: [konnectivity-agent deployment has 1 unavailable replicas, kube-controller-manager deployment has 1 unavailable replicas, kube-scheduler deployment has 1 unavailable replicas]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/07dce0e9-a762-440b-9ccd-61b15fa63bf8
correlation ID: b4a6856e-5332-498a-8060-89b9ebb7175d
client request ID: unavailable
occurredContributing tests (1)
Affected runs (1)
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedC...full failure pattern: ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KASLoadBalancerNotReachable: health check to APIServer endpoint api-int.e0m1j3e6v6o1i5i.b0k7.eastus2euap.aroapp-hcp.io failed: Get "<url>": EOF; hosted cluster degraded: UnavailableReplicas: [ignition-server, kube-apiserver]; provider Microsoft.RedHatOpenShift Signal: IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | e2e | 1 | 2.17%1 of 46 job runs affected | IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | Show trend detailsOct 25: 0 · Oct 26: 0 · Oct 27: 0 · Oct 28: 1 · Oct 29: 0 · Oct 30: 0 · Oct 31: 0 · Nov 1: 0 | none | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/e2e/simple_negative_cases.go:110]: failed to create HCP cluster negative-tests-cluster
Unexpected error:
<*fmt.wrapError | 0x1308b97a65a0>:
failed to create HCP cluster negative-tests-cluster: failed waiting for cluster="negative-tests-cluster" in resourcegroup="rg-negative-tests-79mst7jcpg45" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/94a5e742-d4dc-4cd5-bf1f-345949f27c96
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/94a5e742-d4dc-4cd5-bf1f-345949f27c96",
"name": "94a5e742-d4dc-4cd5-bf1f-345949f27c96",
"status": "Failed",
"startTime": "2026-10-28T20:58:09.127282806Z",
"endTime": "2026-10-28T21:27:10.682711777Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KASLoadBalancerNotReachable: health check to APIServer endpoint api-int.e0m1j3e6v6o1i5i.b0k7.eastus2euap.aroapp-hcp.io failed: Get \"https://api-int.e0m1j3e6v6o1i5i.b0k7.eastus2euap.aroapp-hcp.io:443/healthz?verbose\": EOF; hosted cluster degraded: UnavailableReplicas: [ignition-server deployment has 1 unavailable replicas, kube-apiserver deployment has 1 unavailable replicas]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/94a5e742-d4dc-4cd5-bf1f-345949f27c96
correlation ID: 49b88924-064f-4f52-9480-109cf607a390
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/simple_negative_cases.go:110]: failed to create HCP cluster negative-tests-cluster
Unexpected error:
<*fmt.wrapError | 0x1308b97a65a0>:
failed to create HCP cluster negative-tests-cluster: failed waiting for cluster="negative-tests-cluster" in resourcegroup="rg-negative-tests-79mst7jcpg45" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/94a5e742-d4dc-4cd5-bf1f-345949f27c96
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/94a5e742-d4dc-4cd5-bf1f-345949f27c96",
"name": "94a5e742-d4dc-4cd5-bf1f-345949f27c96",
"status": "Failed",
"startTime": "2026-10-28T20:58:09.127282806Z",
"endTime": "2026-10-28T21:27:10.682711777Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KASLoadBalancerNotReachable: health check to APIServer endpoint api-int.e0m1j3e6v6o1i5i.b0k7.eastus2euap.aroapp-hcp.io failed: Get \"https://api-int.e0m1j3e6v6o1i5i.b0k7.eastus2euap.aroapp-hcp.io:443/healthz?verbose\": EOF; hosted cluster degraded: UnavailableReplicas: [ignition-server deployment has 1 unavailable replicas, kube-apiserver deployment has 1 unavailable replicas]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/94a5e742-d4dc-4cd5-bf1f-345949f27c96
correlation ID: 49b88924-064f-4f52-9480-109cf607a390
client request ID: unavailable
occurredContributing tests (1)
Affected runs (1)
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedC...full failure pattern: ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KASLoadBalancerNotReachable: health check to APIServer endpoint api-int.p8k0e2b5k4j2e4n.nvz7.eastus2euap.aroapp-hcp.io failed: Get "<url>": EOF; hosted cluster degraded: UnavailableReplicas: [ignition-server, kube-apiserver]; provider Microsoft.RedHatOpenShift Signal: IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | e2e | 1 | 2.17%1 of 46 job runs affected | IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | Show trend detailsOct 25: 0 · Oct 26: 0 · Oct 27: 0 · Oct 28: 1 · Oct 29: 0 · Oct 30: 0 · Oct 31: 0 · Nov 1: 0 | none | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:102]: failed to create HCP cluster np-autoscale-cluster
Unexpected error:
<*fmt.wrapError | 0x7c22833c460>:
failed to create HCP cluster np-autoscale-cluster: failed waiting for cluster="np-autoscale-cluster" in resourcegroup="np-autoscaling-kn67v2kn8ztj" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/d1ad5413-0977-4c3c-abd1-937a8f55bfc2
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/d1ad5413-0977-4c3c-abd1-937a8f55bfc2",
"name": "d1ad5413-0977-4c3c-abd1-937a8f55bfc2",
"status": "Failed",
"startTime": "2026-10-28T15:21:06.28860999Z",
"endTime": "2026-10-28T15:50:07.63308505Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KASLoadBalancerNotReachable: health check to APIServer endpoint api-int.p8k0e2b5k4j2e4n.nvz7.eastus2euap.aroapp-hcp.io failed: Get \"https://api-int.p8k0e2b5k4j2e4n.nvz7.eastus2euap.aroapp-hcp.io:443/healthz?verbose\": EOF; hosted cluster degraded: UnavailableReplicas: [ignition-server deployment has 1 unavailable replicas, kube-apiserver deployment has 1 unavailable replicas]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/d1ad5413-0977-4c3c-abd1-937a8f55bfc2
correlation ID: de1e5d6e-772f-44e9-aec7-7b1dbc7151a7
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:102]: failed to create HCP cluster np-autoscale-cluster
Unexpected error:
<*fmt.wrapError | 0x7c22833c460>:
failed to create HCP cluster np-autoscale-cluster: failed waiting for cluster="np-autoscale-cluster" in resourcegroup="np-autoscaling-kn67v2kn8ztj" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/d1ad5413-0977-4c3c-abd1-937a8f55bfc2
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/d1ad5413-0977-4c3c-abd1-937a8f55bfc2",
"name": "d1ad5413-0977-4c3c-abd1-937a8f55bfc2",
"status": "Failed",
"startTime": "2026-10-28T15:21:06.28860999Z",
"endTime": "2026-10-28T15:50:07.63308505Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KASLoadBalancerNotReachable: health check to APIServer endpoint api-int.p8k0e2b5k4j2e4n.nvz7.eastus2euap.aroapp-hcp.io failed: Get \"https://api-int.p8k0e2b5k4j2e4n.nvz7.eastus2euap.aroapp-hcp.io:443/healthz?verbose\": EOF; hosted cluster degraded: UnavailableReplicas: [ignition-server deployment has 1 unavailable replicas, kube-apiserver deployment has 1 unavailable replicas]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/d1ad5413-0977-4c3c-abd1-937a8f55bfc2
correlation ID: de1e5d6e-772f-44e9-aec7-7b1dbc7151a7
client request ID: unavailable
occurredContributing tests (1)
Affected runs (1)
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet complete...full failure pattern: ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version <version> is Partial (want Completed), started <duration> ago; provider Microsoft.RedHatOpenShift Signal: IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | e2e | 1 | 2.17%1 of 46 job runs affected | IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | Show trend detailsOct 25: 0 · Oct 26: 0 · Oct 27: 0 · Oct 28: 1 · Oct 29: 0 · Oct 30: 0 · Oct 31: 0 · Nov 1: 0 | none | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:303]: failed to create HCP cluster np-noedge-g5tsjc
Unexpected error:
<*fmt.wrapError | 0xd7eab0d0620>:
failed to create HCP cluster np-noedge-g5tsjc: failed waiting for cluster="np-noedge-g5tsjc" in resourcegroup="rg-np-noedge-g5tsjc-pf5lpmzmdnbx" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/36728157-067b-4375-a959-9c9ce43e534b
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/36728157-067b-4375-a959-9c9ce43e534b",
"name": "36728157-067b-4375-a959-9c9ce43e534b",
"status": "Failed",
"startTime": "2026-10-28T20:56:26.269135944Z",
"endTime": "2026-10-28T21:25:30.687402107Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.20.44 is Partial (want Completed), started 24m57s ago"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/36728157-067b-4375-a959-9c9ce43e534b
correlation ID: ff24c2ee-7aff-4c46-91cc-8fde998760f5
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:303]: failed to create HCP cluster np-noedge-g5tsjc
Unexpected error:
<*fmt.wrapError | 0xd7eab0d0620>:
failed to create HCP cluster np-noedge-g5tsjc: failed waiting for cluster="np-noedge-g5tsjc" in resourcegroup="rg-np-noedge-g5tsjc-pf5lpmzmdnbx" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/36728157-067b-4375-a959-9c9ce43e534b
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/36728157-067b-4375-a959-9c9ce43e534b",
"name": "36728157-067b-4375-a959-9c9ce43e534b",
"status": "Failed",
"startTime": "2026-10-28T20:56:26.269135944Z",
"endTime": "2026-10-28T21:25:30.687402107Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.20.44 is Partial (want Completed), started 24m57s ago"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/36728157-067b-4375-a959-9c9ce43e534b
correlation ID: ff24c2ee-7aff-4c46-91cc-8fde998760f5
client request ID: unavailable
occurredContributing tests (1)
Affected runs (1)
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
ERROR CODE: DeadlineExceeded; detail message node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hype...full failure pattern: ERROR CODE: DeadlineExceeded; detail message node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is <count>, want <count>; node pool condition AllNodesHealthy is False: <count> of <count> machines are not healthy Machine <node>: InspectionFailed: Waiting for AzureMachine to report spec.providerID; node pool condition AllMachinesReady is False: <count> of <count> machines are not ready Machine <node>: Creating: interfaces creating or updating; provider Microsoft.RedHatOpenShift Signal: IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | e2e | 1 | 2.17%1 of 46 job runs affected | IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | Show trend detailsOct 25: 0 · Oct 26: 0 · Oct 27: 1 · Oct 28: 0 · Oct 29: 0 · Oct 30: 0 · Oct 31: 0 · Nov 1: 0 | none | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_ephemeral_osdisk.go:108]: failed to create nodepool ephemeral-np with ephemeral OS disk
Unexpected error:
<*fmt.wrapError | 0x27665d95f6c0>:
failed to create NodePool ephemeral-np: failed waiting for nodepool="ephemeral-np" for cluster "ephemeral-disk" in resourcegroup="ephemeral-osdisk-dhqxbzp84k6j" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/australiaeast/hcpOperationStatuses/7b5c9bdf-eb40-4dfb-a637-a90c825e7658
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/australiaeast/hcpOperationStatuses/7b5c9bdf-eb40-4dfb-a637-a90c825e7658",
"name": "7b5c9bdf-eb40-4dfb-a637-a90c825e7658",
"status": "Failed",
"startTime": "2026-10-27T03:14:20.369455888Z",
"endTime": "2026-10-27T03:33:20.394901036Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine ephemeral-disk-ephemeral-np-gqrvp-269v2: InspectionFailed: Waiting for AzureMachine to report spec.providerID\nMachine ephemeral-disk-ephemeral-np-gqrvp-q67fk: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine ephemeral-disk-ephemeral-np-gqrvp-269v2: Creating: interfaces creating or updating\nMachine ephemeral-disk-ephemeral-np-gqrvp-q67fk: Creating: interfaces creating or updating\n"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/australiaeast/hcpOperationStatuses/7b5c9bdf-eb40-4dfb-a637-a90c825e7658
correlation ID: a243a81d-f70c-497e-9a6c-d0f5669ed0d7
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_ephemeral_osdisk.go:108]: failed to create nodepool ephemeral-np with ephemeral OS disk
Unexpected error:
<*fmt.wrapError | 0x27665d95f6c0>:
failed to create NodePool ephemeral-np: failed waiting for nodepool="ephemeral-np" for cluster "ephemeral-disk" in resourcegroup="ephemeral-osdisk-dhqxbzp84k6j" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/australiaeast/hcpOperationStatuses/7b5c9bdf-eb40-4dfb-a637-a90c825e7658
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/australiaeast/hcpOperationStatuses/7b5c9bdf-eb40-4dfb-a637-a90c825e7658",
"name": "7b5c9bdf-eb40-4dfb-a637-a90c825e7658",
"status": "Failed",
"startTime": "2026-10-27T03:14:20.369455888Z",
"endTime": "2026-10-27T03:33:20.394901036Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine ephemeral-disk-ephemeral-np-gqrvp-269v2: InspectionFailed: Waiting for AzureMachine to report spec.providerID\nMachine ephemeral-disk-ephemeral-np-gqrvp-q67fk: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine ephemeral-disk-ephemeral-np-gqrvp-269v2: Creating: interfaces creating or updating\nMachine ephemeral-disk-ephemeral-np-gqrvp-q67fk: Creating: interfaces creating or updating\n"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/australiaeast/hcpOperationStatuses/7b5c9bdf-eb40-4dfb-a637-a90c825e7658
correlation ID: a243a81d-f70c-497e-9a6c-d0f5669ed0d7
client request ID: unavailable
occurredContributing tests (1)
Affected runs (1)
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||