Executive Status
EnvEnvironment partition: dev, int, stg, or prod. | GoalConfigured success target and run scope for this environment. INT/STG/PROD use all E2E job runs. DEV uses post-good runs plus Tide batches: runs on the final commit of a PR that later merged, or batch retests of PRs that passed their individual e2e-parallel checks. | RunsNumber of job runs in the selected goal definition for this environment. | SuccessSuccess rate on the selected goal definition: (runs - failed runs) / runs * 100. | Gap vs targetDifference in percentage points between current success and the configured target rate. | Change vs prevHow much the success rate changed compared with the immediately preceding equal-length window, using the same goal definition as this row. | Provision successProvisioning here means infrastructure setup before E2E tests start. A run counts as provision-successful if it got past provisioning, even if it later failed during E2E. Runs that failed for 'Other' CI-infrastructure reasons are excluded because those runs never reached the provision step. DEV rows use post-good runs plus Tide batches. INT/STG/PROD show n/a because provisioning is not part of those environments. | Provision change vs prevChange in provision success, in percentage points, compared with the previous equal-length window. Uses the same provision-success definition as the Provision success column. DEV only; INT/STG/PROD show n/a. | E2E successAmong runs that reached E2E execution, this shows how many completed successfully. Provision failures and 'Other' CI-infrastructure failures are excluded because those runs never reached the E2E step. DEV rows use only runs after the final push to a PR that later merged. INT/STG/PROD use all E2E job runs. | E2E success vs prevChange in E2E success, in percentage points, compared with the previous equal-length window. Uses the same E2E-success definition as the E2E success column. |
|---|---|---|---|---|---|---|---|---|---|
| DEV | 95% - Post-good runs + Tide batches | 186 | 47.31% (off track) | -47.69pp | +4.87pp | 90.17% (156/173) | -0.74pp | 56.41% (88/156) | +8.08pp |
| INT | 95% - All E2E job runs | 43 | 34.88% (off track) | -60.12pp | +34.88pp | n/a | n/a | 100.00% (15/15) | n/a |
| STG | 95% - All E2E job runs | 23 | 34.78% (off track) | -60.22pp | +9.78pp | n/a | n/a | 42.11% (8/19) | +8.77pp |
| PROD | 95% - All E2E job runs | 75 | 60.00% (off track) | -35.00pp | +0.00pp | n/a | n/a | 66.18% (45/68) | +6.18pp |
Environment: DEV
Daily Run Outcomes
Daily Run Outcomes for DEV Goal Basis (post-good + batches)
No tests below 95.00% in this window with at least 10 runs.
Up to 50 failure patterns are loaded (minimum 1.00% impact), with 10 shown by default. Click column headers to re-sort.
Jump to the full Failure Patterns view for this window
| Failure patternThe canonical description of a recurring CI failure, extracted and normalized from raw logs. | Failed atThe stage of the job run where this failure occurred: 'provision' (environment setup, DEV only), 'e2e' (test suite execution), or 'other' (CI infrastructure issues that did not produce a failure pattern). | Number of distinct job runs where this failure pattern was detected. | Percentage of all job runs in this environment affected by this failure pattern during the selected window. | Classification of this failure pattern: Regression (likely caused by a specific PR), Flake (intermittent failure spread across days), Noise (low-quality or generic pattern), or Indeterminate. | TrendShows daily activity for this failure pattern in a trailing window anchored to the selected end date. The sparkline covers at least 7 days and at most 14 days, depending on the current window size. | Also inOther environments where the same failure pattern was also detected during the selected window. | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
alert [hcp] KubePodNotReady firedfull failure pattern: alert [hcp] KubePodNotReady fired Signal: FlakeSignal: Flake — present in 4 prior week(s); spread across 7 days | alert | 18 | 3.88%18 of 464 job runs affected | FlakeSignal: Flake — present in 4 prior week(s); spread across 7 days | Show trend detailsOct 21: 11 · Oct 22: 4 · Oct 23: 3 · Oct 24: 0 · Oct 25: 1 · Oct 26: 4 · Oct 27: 5 · Oct 28: 1 | none | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (3)alert fired 1 time(s) Firing 1: State: Resolved Started: 2026-10-28T20:00:50Z Ended: 2026-10-28T20:06:48Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j5001984-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tnvsms9l35e1dde1cr2bnekd5epm5j9-ingress-cert-le", pod="openshift-apiserver-7cb4d98fc-rzx96", region="westus3", severity="warning" Description: Pod ocm-arohcpci01-2tnvsms9l35e1dde1cr2bnekd5epm5j9-ingress-cert-le/openshift-apiserver-7cb4d98fc-rzx96 has been in a non-ready state for longer than 5 minutes. alert fired 2 time(s) (1 unknown, 1 known) Firing 1: State: Resolved Started: 2026-10-27T23:40:09Z Ended: 2026-10-27T23:57:09Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j1933056-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tneda066vj6kdgdjqh3ub42gu0lckg9-ephemeral-disk", pod="etcd-0", region="centralus", severity="warning" Description: Pod ocm-arohcpci01-2tneda066vj6kdgdjqh3ub42gu0lckg9-ephemeral-disk/etcd-0 has been in a non-ready state for longer than 5 minutes. alert fired 13 time(s) (12 unknown, 1 known) Firing 1: State: Fired (not resolved) Started: 2026-10-27T15:44:12Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j1476864-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tn75avtj8u4futjt2om8gigffgddu35-ingress-cert-le", pod="openshift-apiserver-5fbf969589-rwzww", region="westus3", severity="warning" Description: Pod ocm-arohcpci01-2tn75avtj8u4futjt2om8gigffgddu35-ingress-cert-le/openshift-apiserver-5fbf969589-rwzww has been in a non-ready state for longer than 5 minutes. Firing 2: State: Fired (not resolved) Started: 2026-10-27T15:44:12Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j1476864-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tn74o7qhu6go8i6sdevboooi4cvm5o8-o9s7l1c0u6a0c2b", pod="openshift-apiserver-7c75f5b8c-869mc", region="westus3", severity="warning" Description: Pod ocm-arohcpci01-2tn74o7qhu6go8i6sdevboooi4cvm5o8-o9s7l1c0u6a0c2b/openshift-apiserver-7c75f5b8c-869mc has been in a non-ready state for longer than 5 minutes. Firing 3: State: Fired (not resolved) Started: 2026-10-27T15:45:13Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j1476864-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tn75q9b8j8s3323r8hbovnl3fa3ah93-a8h6t7r0k6i7a1w", pod="openshift-apiserver-6cf589c77c-9cn7v", region="westus3", severity="warning" Description: Pod ocm-arohcpci01-2tn75q9b8j8s3323r8hbovnl3fa3ah93-a8h6t7r0k6i7a1w/openshift-apiserver-6cf589c77c-9cn7v has been in a non-ready state for longer than 5 minutes. Firing 4: State: Fired (not resolved) Started: 2026-10-27T15:45:13Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j1476864-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tn74rifog60es9rvt2t2mfrpvs36h9e-t2y6g6x3g9p9s5t", pod="openshift-apiserver-9c9cfdf89-62q4b", region="westus3", severity="warning" Description: Pod ocm-arohcpci01-2tn74rifog60es9rvt2t2mfrpvs36h9e-t2y6g6x3g9p9s5t/openshift-apiserver-9c9cfdf89-62q4b has been in a non-ready state for longer than 5 minutes. Firing 5: State: Fired (not resolved) Started: 2026-10-27T15:45:13Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j1476864-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tn757i0j3orr665p0dcqg487jcvp9a9-q1n4k4z8u2y9x7u", pod="openshift-apiserver-7bbccfbcd8-nj4zh", region="westus3", severity="warning" Description: Pod ocm-arohcpci01-2tn757i0j3orr665p0dcqg487jcvp9a9-q1n4k4z8u2y9x7u/openshift-apiserver-7bbccfbcd8-nj4zh has been in a non-ready state for longer than 5 minutes. Firing 6: State: Fired (not resolved) Started: 2026-10-27T15:45:13Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j1476864-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tn75637s7ka00dgh3b0o5nl473kscrl-h2n8s4a0l1b6e9x", pod="openshift-apiserver-7b79cf8d6d-klztg", region="westus3", severity="warning" Description: Pod ocm-arohcpci01-2tn75637s7ka00dgh3b0o5nl473kscrl-h2n8s4a0l1b6e9x/openshift-apiserver-7b79cf8d6d-klztg has been in a non-ready state for longer than 5 minutes. Firing 7: State: Fired (not resolved) Started: 2026-10-27T15:45:13Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j1476864-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tn7522f0rjmsl54mp9vum53bfeis22t-x1e5u3y9j2l5a2u", pod="openshift-apiserver-7d75b894c6-n4cmx", region="westus3", severity="warning" Description: Pod ocm-arohcpci01-2tn7522f0rjmsl54mp9vum53bfeis22t-x1e5u3y9j2l5a2u/openshift-apiserver-7d75b894c6-n4cmx has been in a non-ready state for longer than 5 minutes. Firing 8: State: Fired (not resolved) Started: 2026-10-27T15:46:12Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j1476864-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tn75ndq1oh2entq0cv9ulrug5jog853-ingress-cert-le", pod="openshift-apiserver-764b69ffcb-7ngbg", region="westus3", severity="warning" Description: Pod ocm-arohcpci01-2tn75ndq1oh2entq0cv9ulrug5jog853-ingress-cert-le/openshift-apiserver-764b69ffcb-7ngbg has been in a non-ready state for longer than 5 minutes. Firing 9: State: Fired (not resolved) Started: 2026-10-27T15:46:12Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j1476864-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tn7cebrepv6hitmognobm7c9cl2clnd-o5c0v9b6h0c5z0a", pod="cluster-policy-controller-698bfd9c65-v6xdj", region="westus3", severity="warning" Description: Pod ocm-arohcpci01-2tn7cebrepv6hitmognobm7c9cl2clnd-o5c0v9b6h0c5z0a/cluster-policy-controller-698bfd9c65-v6xdj has been in a non-ready state for longer than 5 minutes. Firing 10: State: Fired (not resolved) Started: 2026-10-27T15:46:12Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j1476864-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tn75h85al80vo23t362mjtc3qgd04dp-c1b6v6c2x7c8p8f", pod="openshift-apiserver-57bf4dc46f-h7mj5", region="westus3", severity="warning" Description: Pod ocm-arohcpci01-2tn75h85al80vo23t362mjtc3qgd04dp-c1b6v6c2x7c8p8f/openshift-apiserver-57bf4dc46f-h7mj5 has been in a non-ready state for longer than 5 minutes. Firing 11: State: Fired (not resolved) Started: 2026-10-27T15:46:12Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j1476864-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tn759i4ns4omt60u9uecngkrn5mj618-u8q4i0h7g0i1r6k", pod="openshift-apiserver-59b86f6c67-76jwk", region="westus3", severity="warning" Description: Pod ocm-arohcpci01-2tn759i4ns4omt60u9uecngkrn5mj618-u8q4i0h7g0i1r6k/openshift-apiserver-59b86f6c67-76jwk has been in a non-ready state for longer than 5 minutes. Firing 12: State: Fired (not resolved) Started: 2026-10-27T15:47:14Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j1476864-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tn7615u8n39pf897sksqb5f7beb9d2u-b0o3o9t7e5r0u9g", pod="openshift-apiserver-77c77b89b8-kr9pl", region="westus3", severity="warning" Description: Pod ocm-arohcpci01-2tn7615u8n39pf897sksqb5f7beb9d2u-b0o3o9t7e5r0u9g/openshift-apiserver-77c77b89b8-kr9pl has been in a non-ready state for longer than 5 minutes. Contributing tests (1)
Affected runs (18) | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedC...full failure pattern: ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider]; provider Microsoft.RedHatOpenShift Signal: FlakeSignal: Flake — present in 4 prior week(s); spread across 6 days | e2e | 15 | 3.23%15 of 464 job runs affected | FlakeSignal: Flake — present in 4 prior week(s); spread across 6 days | Show trend detailsOct 21: 0 · Oct 22: 3 · Oct 23: 2 · Oct 24: 0 · Oct 25: 1 · Oct 26: 2 · Oct 27: 3 · Oct 28: 4 | INT, PROD, STG | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (3)fail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:111]: failed to create HCP cluster "cidr-connectivity-test" with authorized CIDRs
Unexpected error:
<*fmt.wrapError | 0x370d5101eb20>:
failed to create HCP cluster cidr-connectivity-test: failed waiting for cluster="cidr-connectivity-test" in resourcegroup="e2e-cidr-connectivity-tgjxrh8mcrw6" to finish creating: GET https://rp.j8717568.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/cce1648d-4b31-48bb-9884-f170bb5391eb
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/cce1648d-4b31-48bb-9884-f170bb5391eb",
"name": "cce1648d-4b31-48bb-9884-f170bb5391eb",
"status": "Failed",
"startTime": "2026-10-28T19:28:55.409225524Z",
"endTime": "2026-10-28T19:58:03.222551854Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://rp.j8717568.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/cce1648d-4b31-48bb-9884-f170bb5391eb
correlation ID: d6e5235d-8c3c-4404-82e9-f3e3b3165e1e
client request ID: 4d040942-ab41-4fbc-8d6e-d69a84105ef1
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:111]: failed to create HCP cluster "cidr-connectivity-test" with authorized CIDRs
Unexpected error:
<*fmt.wrapError | 0x370d5101eb20>:
failed to create HCP cluster cidr-connectivity-test: failed waiting for cluster="cidr-connectivity-test" in resourcegroup="e2e-cidr-connectivity-tgjxrh8mcrw6" to finish creating: GET https://rp.j8717568.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/cce1648d-4b31-48bb-9884-f170bb5391eb
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/cce1648d-4b31-48bb-9884-f170bb5391eb",
"name": "cce1648d-4b31-48bb-9884-f170bb5391eb",
"status": "Failed",
"startTime": "2026-10-28T19:28:55.409225524Z",
"endTime": "2026-10-28T19:58:03.222551854Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://rp.j8717568.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/cce1648d-4b31-48bb-9884-f170bb5391eb
correlation ID: d6e5235d-8c3c-4404-82e9-f3e3b3165e1e
client request ID: 4d040942-ab41-4fbc-8d6e-d69a84105ef1
occurredfail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:146]: failed to create HCP cluster np-version-upgrade-cluster-jl2sj8 with version 4.21.40
Unexpected error:
<*fmt.wrapError | 0x2ee0c1834d20>:
failed to create HCP cluster np-version-upgrade-cluster-jl2sj8: failed waiting for cluster="np-version-upgrade-cluster-jl2sj8" in resourcegroup="rg-np-version-upgrade-jl2sj8-22vp9qvqkmcf" to finish creating: GET https://rp.j9822336.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/267d3daf-f468-46eb-af3a-f1b72d573583
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/267d3daf-f468-46eb-af3a-f1b72d573583",
"name": "267d3daf-f468-46eb-af3a-f1b72d573583",
"status": "Failed",
"startTime": "2026-10-28T19:42:27.649991944Z",
"endTime": "2026-10-28T20:11:28.996573337Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://rp.j9822336.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/267d3daf-f468-46eb-af3a-f1b72d573583
correlation ID: c45a1c96-f9ee-44e0-8639-c952cdb59a23
client request ID: fe15252c-edf3-4b0e-8aaa-b5cd06393883
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:146]: failed to create HCP cluster np-version-upgrade-cluster-jl2sj8 with version 4.21.40
Unexpected error:
<*fmt.wrapError | 0x2ee0c1834d20>:
failed to create HCP cluster np-version-upgrade-cluster-jl2sj8: failed waiting for cluster="np-version-upgrade-cluster-jl2sj8" in resourcegroup="rg-np-version-upgrade-jl2sj8-22vp9qvqkmcf" to finish creating: GET https://rp.j9822336.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/267d3daf-f468-46eb-af3a-f1b72d573583
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/267d3daf-f468-46eb-af3a-f1b72d573583",
"name": "267d3daf-f468-46eb-af3a-f1b72d573583",
"status": "Failed",
"startTime": "2026-10-28T19:42:27.649991944Z",
"endTime": "2026-10-28T20:11:28.996573337Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://rp.j9822336.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/267d3daf-f468-46eb-af3a-f1b72d573583
correlation ID: c45a1c96-f9ee-44e0-8639-c952cdb59a23
client request ID: fe15252c-edf3-4b0e-8aaa-b5cd06393883
occurredfail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:133]: failed to create HCP cluster "cp-ystream-upgrade-4-22-llxcnv" at install version 4.21
Unexpected error:
<*fmt.wrapError | 0x1b596ab10140>:
failed to create HCP cluster cp-ystream-upgrade-4-22-llxcnv: failed waiting for cluster="cp-ystream-upgrade-4-22-llxcnv" in resourcegroup="rg-cp-ystream-upgrade-4-22-468dv2nskdkq" to finish creating: GET https://rp.j6945280.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/e79580c0-76a4-4e28-820e-080632c7ada2
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/e79580c0-76a4-4e28-820e-080632c7ada2",
"name": "e79580c0-76a4-4e28-820e-080632c7ada2",
"status": "Failed",
"startTime": "2026-10-28T12:38:01.23060152Z",
"endTime": "2026-10-28T13:07:03.13964352Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://rp.j6945280.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/e79580c0-76a4-4e28-820e-080632c7ada2
correlation ID: 8869047d-1b08-4ff0-8212-00001a9147bd
client request ID: 0bcb310d-278e-4828-86a5-41d1c5537311
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:133]: failed to create HCP cluster "cp-ystream-upgrade-4-22-llxcnv" at install version 4.21
Unexpected error:
<*fmt.wrapError | 0x1b596ab10140>:
failed to create HCP cluster cp-ystream-upgrade-4-22-llxcnv: failed waiting for cluster="cp-ystream-upgrade-4-22-llxcnv" in resourcegroup="rg-cp-ystream-upgrade-4-22-468dv2nskdkq" to finish creating: GET https://rp.j6945280.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/e79580c0-76a4-4e28-820e-080632c7ada2
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/e79580c0-76a4-4e28-820e-080632c7ada2",
"name": "e79580c0-76a4-4e28-820e-080632c7ada2",
"status": "Failed",
"startTime": "2026-10-28T12:38:01.23060152Z",
"endTime": "2026-10-28T13:07:03.13964352Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://rp.j6945280.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/e79580c0-76a4-4e28-820e-080632c7ada2
correlation ID: 8869047d-1b08-4ff0-8212-00001a9147bd
client request ID: 0bcb310d-278e-4828-86a5-41d1c5537311
occurredContributing tests (15)
Affected runs (15) | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
alert [hcp] KubePodCrashLooping firedfull failure pattern: alert [hcp] KubePodCrashLooping fired Signal: FlakeSignal: Flake — present in 4 prior week(s); spread across 6 days | alert | 15 | 3.23%15 of 464 job runs affected | FlakeSignal: Flake — present in 4 prior week(s); spread across 6 days | Show trend detailsOct 21: 0 · Oct 22: 2 · Oct 23: 3 · Oct 24: 0 · Oct 25: 2 · Oct 26: 3 · Oct 27: 2 · Oct 28: 3 | none | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (3)alert fired 2 time(s) Firing 1: State: Resolved Started: 2026-10-28T19:50:44Z Ended: 2026-10-28T20:18:45Z Severity: Sev3 Labels: alertname="KubePodCrashLooping", cluster="ci01-j8717568-mgmt-2", component="kubernetes-infrastructure", container="etcd", endpoint="http", environment="ci01", instance="10.128.64.254:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j8717568/providers/microsoft.monitor/accounts/hcps-j8717568", microsoft.metrics_include_label="service", namespace="ocm-arohcpci01-2tnvtu6q95ufl5426ec9nsjbnsqfa0aa-a8y2r7f6q8h7z0t", pod="etcd-0", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", reason="CrashLoopBackOff", region="centralus", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="aad698ce-6b47-41d1-8ddd-e6e4cb15b5d7" Description: Pod ocm-arohcpci01-2tnvtu6q95ufl5426ec9nsjbnsqfa0aa-a8y2r7f6q8h7z0t/etcd-0 (etcd) is in waiting state (reason: "CrashLoopBackOff"). Firing 2: State: Resolved Started: 2026-10-28T19:50:44Z Ended: 2026-10-28T20:19:45Z Severity: Sev3 Labels: alertname="KubePodCrashLooping", cluster="ci01-j8717568-mgmt-2", component="kubernetes-infrastructure", container="etcd", endpoint="http", environment="ci01", instance="10.128.64.254:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j8717568/providers/microsoft.monitor/accounts/hcps-j8717568", microsoft.metrics_include_label="service", namespace="ocm-arohcpci01-2tnvtu6q95ufl5426ec9nsjbnsqfa0aa-a8y2r7f6q8h7z0t", pod="etcd-0", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", reason="CrashLoopBackOff", region="centralus", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="aad698ce-6b47-41d1-8ddd-e6e4cb15b5d7" Description: Pod ocm-arohcpci01-2tnvtu6q95ufl5426ec9nsjbnsqfa0aa-a8y2r7f6q8h7z0t/etcd-0 (etcd) is in waiting state (reason: "CrashLoopBackOff"). alert fired 2 time(s) Firing 1: State: Resolved Started: 2026-10-28T12:59:31Z Ended: 2026-10-28T13:28:30Z Severity: Sev3 Labels: alertname="KubePodCrashLooping", cluster="ci01-j6945280-mgmt-1", component="kubernetes-infrastructure", container="etcd", endpoint="http", environment="ci01", instance="10.128.64.108:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j6945280/providers/microsoft.monitor/accounts/hcps-j6945280", microsoft.metrics_include_label="service", namespace="ocm-arohcpci01-2tnptam9khdl3svuvm4seuq5gr0ieit6-v5j4g5d3s9p2z6k", pod="etcd-0", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", reason="CrashLoopBackOff", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="5496ebfd-2ab4-43b8-9f9a-e0483581f4bd" Description: Pod ocm-arohcpci01-2tnptam9khdl3svuvm4seuq5gr0ieit6-v5j4g5d3s9p2z6k/etcd-0 (etcd) is in waiting state (reason: "CrashLoopBackOff"). Firing 2: State: Resolved Started: 2026-10-28T13:00:33Z Ended: 2026-10-28T13:27:30Z Severity: Sev3 Labels: alertname="KubePodCrashLooping", cluster="ci01-j6945280-mgmt-1", component="kubernetes-infrastructure", container="etcd", endpoint="http", environment="ci01", instance="10.128.64.108:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j6945280/providers/microsoft.monitor/accounts/hcps-j6945280", microsoft.metrics_include_label="service", namespace="ocm-arohcpci01-2tnptam9khdl3svuvm4seuq5gr0ieit6-v5j4g5d3s9p2z6k", pod="etcd-0", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", reason="CrashLoopBackOff", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="5496ebfd-2ab4-43b8-9f9a-e0483581f4bd" Description: Pod ocm-arohcpci01-2tnptam9khdl3svuvm4seuq5gr0ieit6-v5j4g5d3s9p2z6k/etcd-0 (etcd) is in waiting state (reason: "CrashLoopBackOff"). alert fired 2 time(s) Firing 1: State: Resolved Started: 2026-10-28T08:20:17Z Ended: 2026-10-28T08:44:17Z Severity: Sev3 Labels: alertname="KubePodCrashLooping", cluster="ci01-j3004160-mgmt-1", component="kubernetes-infrastructure", container="etcd", endpoint="http", environment="ci01", instance="10.128.64.217:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j3004160/providers/microsoft.monitor/accounts/hcps-j3004160", microsoft.metrics_include_label="service", namespace="ocm-arohcpci01-2tnloambpnud3aq6p4sg9mdteg909a3n-z3x1x9z7m5g8p6s", pod="etcd-0", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", reason="CrashLoopBackOff", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="a66d88b1-9c74-40e9-87a6-eb699288069f" Description: Pod ocm-arohcpci01-2tnloambpnud3aq6p4sg9mdteg909a3n-z3x1x9z7m5g8p6s/etcd-0 (etcd) is in waiting state (reason: "CrashLoopBackOff"). Firing 2: State: Resolved Started: 2026-10-28T08:20:17Z Ended: 2026-10-28T08:44:17Z Severity: Sev3 Labels: alertname="KubePodCrashLooping", cluster="ci01-j3004160-mgmt-1", component="kubernetes-infrastructure", container="etcd", endpoint="http", environment="ci01", instance="10.128.64.217:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j3004160/providers/microsoft.monitor/accounts/hcps-j3004160", microsoft.metrics_include_label="service", namespace="ocm-arohcpci01-2tnloambpnud3aq6p4sg9mdteg909a3n-z3x1x9z7m5g8p6s", pod="etcd-0", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", reason="CrashLoopBackOff", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="a66d88b1-9c74-40e9-87a6-eb699288069f" Description: Pod ocm-arohcpci01-2tnloambpnud3aq6p4sg9mdteg909a3n-z3x1x9z7m5g8p6s/etcd-0 (etcd) is in waiting state (reason: "CrashLoopBackOff"). Contributing tests (1)
Affected runs (15) | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
failed to create nodepool <nodepool>: failed waiting for nodepool="<nodepool>" for cluster "<cluster>" in resourcegroup="<resource-group>" to finish creating: GET <url>full failure pattern: failed to create nodepool <nodepool>: failed waiting for nodepool="<nodepool>" for cluster "<cluster>" in resourcegroup="<resource-group>" to finish creating: GET <url> Signal: FlakeSignal: Flake — present in 4 prior week(s); spread across 6 days | e2e | 14 | 3.02%14 of 464 job runs affected | FlakeSignal: Flake — present in 4 prior week(s); spread across 6 days | Show trend detailsOct 21: 0 · Oct 22: 2 · Oct 23: 6 · Oct 24: 1 · Oct 25: 0 · Oct 26: 4 · Oct 27: 3 · Oct 28: 1 | STG | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (3)fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_delete.go:128]: failed to create node pools
Expected success, but got an error:
<*fmt.wrapError | 0x2db4fc89320>:
failed to create NodePool np-delete-main: failed waiting for nodepool="np-delete-main" for cluster "np-delete-hcp-cluster" in resourcegroup="nodepool-delete-zbnm2gmmrc7k" to finish creating: GET https://rp.j8443648.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/13b873cd-2efa-4559-ab15-ff37ab2d031b
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/13b873cd-2efa-4559-ab15-ff37ab2d031b",
"name": "13b873cd-2efa-4559-ab15-ff37ab2d031b",
"status": "Failed",
"startTime": "2026-10-28T16:52:57.688694522Z",
"endTime": "2026-10-28T17:12:06.752450543Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine j8c0h2z2t3o8e9b-np-delete-main-69xnv-msqbr: NodeNotHealthy: * Node.Ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine j8c0h2z2t3o8e9b-np-delete-main-69xnv-msqbr: NotReady: * NodeHealthy:\n * Node.Ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?\n"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://rp.j8443648.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/13b873cd-2efa-4559-ab15-ff37ab2d031b
correlation ID: 596c64f1-beb7-4366-a3b9-24ac7fe089dc
client request ID: 7b2dfc27-0f04-4e6d-a8a1-787b24c6e549
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_delete.go:128]: failed to create node pools
Expected success, but got an error:
<*fmt.wrapError | 0x2db4fc89320>:
failed to create NodePool np-delete-main: failed waiting for nodepool="np-delete-main" for cluster "np-delete-hcp-cluster" in resourcegroup="nodepool-delete-zbnm2gmmrc7k" to finish creating: GET https://rp.j8443648.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/13b873cd-2efa-4559-ab15-ff37ab2d031b
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/13b873cd-2efa-4559-ab15-ff37ab2d031b",
"name": "13b873cd-2efa-4559-ab15-ff37ab2d031b",
"status": "Failed",
"startTime": "2026-10-28T16:52:57.688694522Z",
"endTime": "2026-10-28T17:12:06.752450543Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine j8c0h2z2t3o8e9b-np-delete-main-69xnv-msqbr: NodeNotHealthy: * Node.Ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine j8c0h2z2t3o8e9b-np-delete-main-69xnv-msqbr: NotReady: * NodeHealthy:\n * Node.Ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?\n"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://rp.j8443648.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/13b873cd-2efa-4559-ab15-ff37ab2d031b
correlation ID: 596c64f1-beb7-4366-a3b9-24ac7fe089dc
client request ID: 7b2dfc27-0f04-4e6d-a8a1-787b24c6e549fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:166]: failed to create node pool
Expected success, but got an error:
<*fmt.wrapError | 0xb71d4fca080>:
failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "ingress-cert-le" in resourcegroup="ingress-cert-le-bqbx2znvzb54" to finish creating: GET https://rp.j1933056.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/0ed006bd-b8aa-4770-9062-3fd090d4fea9
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/0ed006bd-b8aa-4770-9062-3fd090d4fea9",
"name": "0ed006bd-b8aa-4770-9062-3fd090d4fea9",
"status": "Failed",
"startTime": "2026-10-27T23:25:38.076776565Z",
"endTime": "2026-10-27T23:44:40.197052822Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine ingress-cert-le-np-1-xjlrw-mjpj4: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/ingress-cert-le-bqbx2znvzb54--managed/providers/Microsoft.Compute/virtualMachines/ingress-cert-le-np-1-xjlrw-mjpj4 to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine ingress-cert-le-np-1-xjlrw-7jnb9: Failed: interfaces failed to create or update. err: failed to get existing resource ingress-cert-le-bqbx2znvzb54--managed/ingress-cert-le-np-1-xjlrw-7jnb9-nic (service: interfaces): GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/ingress-cert-le-bqbx2znvzb54--managed/providers/Microsoft.Network/networkInterfaces/ingress-cert-le-np-1-xjlrw-7jnb9-nic\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"SubscriptionRequestsThrottled\",\n \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor '8d1ff6d0-aadd-4331-9171-9443ac5f4337' exceeded. Please try again after '10' seconds after additional tokens are available. Refer ... message truncated\n"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://rp.j1933056.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/0ed006bd-b8aa-4770-9062-3fd090d4fea9
correlation ID: d0096963-5f3a-4ca8-942f-d516adbeb2fc
client request ID: 336f75bd-67db-4bb4-b035-81a9154977bb
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:166]: failed to create node pool
Expected success, but got an error:
<*fmt.wrapError | 0xb71d4fca080>:
failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "ingress-cert-le" in resourcegroup="ingress-cert-le-bqbx2znvzb54" to finish creating: GET https://rp.j1933056.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/0ed006bd-b8aa-4770-9062-3fd090d4fea9
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/0ed006bd-b8aa-4770-9062-3fd090d4fea9",
"name": "0ed006bd-b8aa-4770-9062-3fd090d4fea9",
"status": "Failed",
"startTime": "2026-10-27T23:25:38.076776565Z",
"endTime": "2026-10-27T23:44:40.197052822Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine ingress-cert-le-np-1-xjlrw-mjpj4: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/ingress-cert-le-bqbx2znvzb54--managed/providers/Microsoft.Compute/virtualMachines/ingress-cert-le-np-1-xjlrw-mjpj4 to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine ingress-cert-le-np-1-xjlrw-7jnb9: Failed: interfaces failed to create or update. err: failed to get existing resource ingress-cert-le-bqbx2znvzb54--managed/ingress-cert-le-np-1-xjlrw-7jnb9-nic (service: interfaces): GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/ingress-cert-le-bqbx2znvzb54--managed/providers/Microsoft.Network/networkInterfaces/ingress-cert-le-np-1-xjlrw-7jnb9-nic\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"SubscriptionRequestsThrottled\",\n \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor '8d1ff6d0-aadd-4331-9171-9443ac5f4337' exceeded. Please try again after '10' seconds after additional tokens are available. Refer ... message truncated\n"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://rp.j1933056.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/0ed006bd-b8aa-4770-9062-3fd090d4fea9
correlation ID: d0096963-5f3a-4ca8-942f-d516adbeb2fc
client request ID: 336f75bd-67db-4bb4-b035-81a9154977bbfail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:166]: failed to create node pool
Expected success, but got an error:
<*fmt.wrapError | 0x3591356d60>:
failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "ingress-cert-le" in resourcegroup="ingress-cert-le-6b928vvvc9lv" to finish creating: GET https://rp.j1933056.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/e68fe219-929c-499f-9d94-52a89aa54331
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/e68fe219-929c-499f-9d94-52a89aa54331",
"name": "e68fe219-929c-499f-9d94-52a89aa54331",
"status": "Failed",
"startTime": "2026-10-27T23:27:39.482004964Z",
"endTime": "2026-10-27T23:46:40.23395218Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine ingress-cert-le-np-1-qtp5t-vlfjh: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/ingress-cert-le-6b928vvvc9lv--managed/providers/Microsoft.Compute/virtualMachines/ingress-cert-le-np-1-qtp5t-vlfjh to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine ingress-cert-le-np-1-qtp5t-vlfjh: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/ingress-cert-le-6b928vvvc9lv--managed/providers/Microsoft.Compute/virtualMachines/ingress-cert-le-np-1-qtp5t-vlfjh to exist\n"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://rp.j1933056.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/e68fe219-929c-499f-9d94-52a89aa54331
correlation ID: 0fae4e0c-a402-45c6-8599-10c6e41f505d
client request ID: c1da63b5-9f53-4e68-8989-fc0f3ef1f3cb
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:166]: failed to create node pool
Expected success, but got an error:
<*fmt.wrapError | 0x3591356d60>:
failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "ingress-cert-le" in resourcegroup="ingress-cert-le-6b928vvvc9lv" to finish creating: GET https://rp.j1933056.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/e68fe219-929c-499f-9d94-52a89aa54331
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/e68fe219-929c-499f-9d94-52a89aa54331",
"name": "e68fe219-929c-499f-9d94-52a89aa54331",
"status": "Failed",
"startTime": "2026-10-27T23:27:39.482004964Z",
"endTime": "2026-10-27T23:46:40.23395218Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine ingress-cert-le-np-1-qtp5t-vlfjh: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/ingress-cert-le-6b928vvvc9lv--managed/providers/Microsoft.Compute/virtualMachines/ingress-cert-le-np-1-qtp5t-vlfjh to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine ingress-cert-le-np-1-qtp5t-vlfjh: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/ingress-cert-le-6b928vvvc9lv--managed/providers/Microsoft.Compute/virtualMachines/ingress-cert-le-np-1-qtp5t-vlfjh to exist\n"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://rp.j1933056.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/e68fe219-929c-499f-9d94-52a89aa54331
correlation ID: 0fae4e0c-a402-45c6-8599-10c6e41f505d
client request ID: c1da63b5-9f53-4e68-8989-fc0f3ef1f3cbContributing tests (6)
Affected runs (14) | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
alert [hcp] KubeDeploymentReplicasMismatch firedfull failure pattern: alert [hcp] KubeDeploymentReplicasMismatch fired Signal: FlakeSignal: Flake — present in 4 prior week(s); spread across 7 days | alert | 14 | 3.02%14 of 464 job runs affected | FlakeSignal: Flake — present in 4 prior week(s); spread across 7 days | Show trend detailsOct 21: 1 · Oct 22: 3 · Oct 23: 3 · Oct 24: 0 · Oct 25: 1 · Oct 26: 2 · Oct 27: 2 · Oct 28: 3 | none | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (3)alert fired 2 time(s) Firing 1: State: Resolved Started: 2026-10-28T20:03:50Z Ended: 2026-10-28T20:15:46Z Severity: Sev3 Labels: alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j8717568-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="capi-provider", endpoint="http", environment="ci01", instance="10.128.64.254:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j8717568/providers/microsoft.monitor/accounts/hcps-j8717568", microsoft.metrics_include_label="service", namespace="ocm-arohcpci01-2tnvtu6q95ufl5426ec9nsjbnsqfa0aa-a8y2r7f6q8h7z0t", pod="arohcp-monitor-kube-state-metrics-85d48f46ff-59dhx", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="centralus", service="arohcp-monitor-kube-state-metrics", severity="warning" Description: Deployment ocm-arohcpci01-2tnvtu6q95ufl5426ec9nsjbnsqfa0aa-a8y2r7f6q8h7z0t/capi-provider has not matched the expected number of replicas for longer than 30 minutes. Firing 2: State: Resolved Started: 2026-10-28T20:03:50Z Ended: 2026-10-28T20:15:46Z Severity: Sev3 Labels: alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j8717568-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="capi-provider", endpoint="http", environment="ci01", instance="10.128.64.254:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j8717568/providers/microsoft.monitor/accounts/hcps-j8717568", microsoft.metrics_include_label="service", namespace="ocm-arohcpci01-2tnvtu6q95ufl5426ec9nsjbnsqfa0aa-a8y2r7f6q8h7z0t", pod="arohcp-monitor-kube-state-metrics-85d48f46ff-59dhx", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="centralus", service="arohcp-monitor-kube-state-metrics", severity="warning" Description: Deployment ocm-arohcpci01-2tnvtu6q95ufl5426ec9nsjbnsqfa0aa-a8y2r7f6q8h7z0t/capi-provider has not matched the expected number of replicas for longer than 30 minutes. alert fired 1 time(s) Firing 1: State: Resolved Started: 2026-10-28T13:12:32Z Ended: 2026-10-28T13:28:32Z Severity: Sev3 Labels: alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j6945280-mgmt-1", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="capi-provider", endpoint="http", environment="ci01", instance="10.128.64.108:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j6945280/providers/microsoft.monitor/accounts/hcps-j6945280", microsoft.metrics_include_label="service", namespace="ocm-arohcpci01-2tnptam9khdl3svuvm4seuq5gr0ieit6-v5j4g5d3s9p2z6k", pod="arohcp-monitor-kube-state-metrics-85d48f46ff-fct9q", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning" Description: Deployment ocm-arohcpci01-2tnptam9khdl3svuvm4seuq5gr0ieit6-v5j4g5d3s9p2z6k/capi-provider has not matched the expected number of replicas for longer than 30 minutes. alert fired 2 time(s) Firing 1: State: Resolved Started: 2026-10-28T08:32:18Z Ended: 2026-10-28T08:41:20Z Severity: Sev3 Labels: alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j3004160-mgmt-1", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="capi-provider", endpoint="http", environment="ci01", instance="10.128.64.217:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j3004160/providers/microsoft.monitor/accounts/hcps-j3004160", microsoft.metrics_include_label="service", namespace="ocm-arohcpci01-2tnloambpnud3aq6p4sg9mdteg909a3n-z3x1x9z7m5g8p6s", pod="arohcp-monitor-kube-state-metrics-68ff96c687-7dfbl", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning" Description: Deployment ocm-arohcpci01-2tnloambpnud3aq6p4sg9mdteg909a3n-z3x1x9z7m5g8p6s/capi-provider has not matched the expected number of replicas for longer than 30 minutes. Firing 2: State: Resolved Started: 2026-10-28T08:32:18Z Ended: 2026-10-28T08:42:23Z Severity: Sev3 Labels: alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j3004160-mgmt-1", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="capi-provider", endpoint="http", environment="ci01", instance="10.128.64.217:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j3004160/providers/microsoft.monitor/accounts/hcps-j3004160", microsoft.metrics_include_label="service", namespace="ocm-arohcpci01-2tnloambpnud3aq6p4sg9mdteg909a3n-z3x1x9z7m5g8p6s", pod="arohcp-monitor-kube-state-metrics-68ff96c687-7dfbl", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning" Description: Deployment ocm-arohcpci01-2tnloambpnud3aq6p4sg9mdteg909a3n-z3x1x9z7m5g8p6s/capi-provider has not matched the expected number of replicas for longer than 30 minutes. Contributing tests (1)
Affected runs (14) | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
alert [svc] KubePodNotReady firedfull failure pattern: alert [svc] KubePodNotReady fired Signal: FlakeSignal: Flake — present in 4 prior week(s); spread across 5 days | alert | 14 | 3.02%14 of 464 job runs affected | FlakeSignal: Flake — present in 4 prior week(s); spread across 5 days | Show trend detailsOct 21: 0 · Oct 22: 4 · Oct 23: 1 · Oct 24: 2 · Oct 25: 0 · Oct 26: 0 · Oct 27: 6 · Oct 28: 1 | none | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (3)alert fired 4 time(s) Firing 1: State: Fired (not resolved) Started: 2026-10-28T17:07:36Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j3195648-svc", component="kubernetes-infrastructure", namespace="clusters-service", pod="clusters-service-64b56768b9-r7v9d", region="canadacentral", severity="warning" Description: Pod clusters-service/clusters-service-64b56768b9-r7v9d has been in a non-ready state for longer than 5 minutes. Firing 2: State: Fired (not resolved) Started: 2026-10-28T17:07:36Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j3195648-svc", component="kubernetes-infrastructure", namespace="fleet", pod="fleet-controller-7b9596cccb-g9thf", region="canadacentral", severity="warning" Description: Pod fleet/fleet-controller-7b9596cccb-g9thf has been in a non-ready state for longer than 5 minutes. Firing 3: State: Fired (not resolved) Started: 2026-10-28T17:07:36Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j3195648-svc", component="kubernetes-infrastructure", namespace="aro-hcp-admin-api", pod="admin-api-6d56468ffd-t9snj", region="canadacentral", severity="warning" Description: Pod aro-hcp-admin-api/admin-api-6d56468ffd-t9snj has been in a non-ready state for longer than 5 minutes. Firing 4: State: Fired (not resolved) Started: 2026-10-28T17:07:36Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j3195648-svc", component="kubernetes-infrastructure", namespace="aro-hcp-admin-api", pod="admin-api-6d56468ffd-rm24l", region="canadacentral", severity="warning" Description: Pod aro-hcp-admin-api/admin-api-6d56468ffd-rm24l has been in a non-ready state for longer than 5 minutes. alert fired 21 time(s) Firing 1: State: Fired (not resolved) Started: 2026-10-28T00:28:19Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j8270720-svc", component="kubernetes-infrastructure", namespace="aks-istio-system", pod="azure-service-mesh-istio-cni-addon-node-ck87t", region="westus3", severity="warning" Description: Pod aks-istio-system/azure-service-mesh-istio-cni-addon-node-ck87t has been in a non-ready state for longer than 5 minutes. Firing 2: State: Fired (not resolved) Started: 2026-10-28T00:28:20Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j8270720-svc", component="kubernetes-infrastructure", namespace="kube-system", pod="aks-secrets-store-csi-driver-n68qf", region="westus3", severity="warning" Description: Pod kube-system/aks-secrets-store-csi-driver-n68qf has been in a non-ready state for longer than 5 minutes. Firing 3: State: Fired (not resolved) Started: 2026-10-28T00:28:20Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j8270720-svc", component="kubernetes-infrastructure", namespace="kube-system", pod="ama-metrics-node-bx6p7", region="westus3", severity="warning" Description: Pod kube-system/ama-metrics-node-bx6p7 has been in a non-ready state for longer than 5 minutes. Firing 4: State: Fired (not resolved) Started: 2026-10-28T00:28:20Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j8270720-svc", component="kubernetes-infrastructure", namespace="kube-system", pod="ama-metrics-node-6fggw", region="westus3", severity="warning" Description: Pod kube-system/ama-metrics-node-6fggw has been in a non-ready state for longer than 5 minutes. Firing 5: State: Fired (not resolved) Started: 2026-10-28T00:28:20Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j8270720-svc", component="kubernetes-infrastructure", namespace="kube-system", pod="eraser-aks-system-42508117-vmss000002-249kw", region="westus3", severity="warning" Description: Pod kube-system/eraser-aks-system-42508117-vmss000002-249kw has been in a non-ready state for longer than 5 minutes. Firing 6: State: Fired (not resolved) Started: 2026-10-28T00:28:20Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j8270720-svc", component="kubernetes-infrastructure", namespace="aks-istio-system", pod="azure-service-mesh-istio-cni-addon-node-f2bs8", region="westus3", severity="warning" Description: Pod aks-istio-system/azure-service-mesh-istio-cni-addon-node-f2bs8 has been in a non-ready state for longer than 5 minutes. Firing 7: State: Fired (not resolved) Started: 2026-10-28T00:28:20Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j8270720-svc", component="kubernetes-infrastructure", namespace="kube-system", pod="aks-secrets-store-csi-driver-pwgk9", region="westus3", severity="warning" Description: Pod kube-system/aks-secrets-store-csi-driver-pwgk9 has been in a non-ready state for longer than 5 minutes. Firing 8: State: Fired (not resolved) Started: 2026-10-28T00:28:20Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j8270720-svc", component="kubernetes-infrastructure", namespace="kube-system", pod="ama-metrics-node-sb9gv", region="westus3", severity="warning" Description: Pod kube-system/ama-metrics-node-sb9gv has been in a non-ready state for longer than 5 minutes. Firing 9: State: Fired (not resolved) Started: 2026-10-28T00:28:20Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j8270720-svc", component="kubernetes-infrastructure", namespace="kube-system", pod="eraser-aks-system-42508117-vmss000001-z2292", region="westus3", severity="warning" Description: Pod kube-system/eraser-aks-system-42508117-vmss000001-z2292 has been in a non-ready state for longer than 5 minutes. Firing 10: State: Fired (not resolved) Started: 2026-10-28T00:28:20Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j8270720-svc", component="kubernetes-infrastructure", namespace="kube-system", pod="aks-secrets-store-csi-driver-sgh48", region="westus3", severity="warning" Description: Pod kube-system/aks-secrets-store-csi-driver-sgh48 has been in a non-ready state for longer than 5 minutes. Firing 11: State: Fired (not resolved) Started: 2026-10-28T00:28:21Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j8270720-svc", component="kubernetes-infrastructure", namespace="aks-istio-system", pod="azure-service-mesh-istio-cni-addon-node-jkt84", region="westus3", severity="warning" Description: Pod aks-istio-system/azure-service-mesh-istio-cni-addon-node-jkt84 has been in a non-ready state for longer than 5 minutes. Firing 12: State: Fired (not resolved) Started: 2026-10-28T00:28:21Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j8270720-svc", component="kubernetes-infrastructure", namespace="kube-system", pod="eraser-aks-system-42508117-vmss000000-b6bzq", region="westus3", severity="warning" Description: Pod kube-system/eraser-aks-system-42508117-vmss000000-b6bzq has been in a non-ready state for longer than 5 minutes. Firing 13: State: Fired (not resolved) Started: 2026-10-28T00:28:21Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j8270720-svc", component="kubernetes-infrastructure", namespace="kube-system", pod="hubble-relay-58594b5dd7-p7rcj", region="westus3", severity="warning" Description: Pod kube-system/hubble-relay-58594b5dd7-p7rcj has been in a non-ready state for longer than 5 minutes. Firing 14: State: Fired (not resolved) Started: 2026-10-28T00:30:21Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j8270720-svc", component="kubernetes-infrastructure", namespace="sessiongate", pod="sessiongate-8696d46cbc-gcq7v", region="westus3", severity="warning" Description: Pod sessiongate/sessiongate-8696d46cbc-gcq7v has been in a non-ready state for longer than 5 minutes. Firing 15: State: Fired (not resolved) Started: 2026-10-28T00:30:21Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j8270720-svc", component="kubernetes-infrastructure", namespace="sessiongate", pod="sessiongate-8696d46cbc-7tl64", region="westus3", severity="warning" Description: Pod sessiongate/sessiongate-8696d46cbc-7tl64 has been in a non-ready state for longer than 5 minutes. Firing 16: State: Fired (not resolved) Started: 2026-10-28T00:30:21Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j8270720-svc", component="kubernetes-infrastructure", namespace="maestro", pod="maestro-6bc6657c6b-nc5dx", region="westus3", severity="warning" Description: Pod maestro/maestro-6bc6657c6b-nc5dx has been in a non-ready state for longer than 5 minutes. Firing 17: State: Fired (not resolved) Started: 2026-10-28T00:30:21Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j8270720-svc", component="kubernetes-infrastructure", namespace="clusters-service", pod="clusters-service-567b46758d-6wzh8", region="westus3", severity="warning" Description: Pod clusters-service/clusters-service-567b46758d-6wzh8 has been in a non-ready state for longer than 5 minutes. Firing 18: State: Fired (not resolved) Started: 2026-10-28T00:30:21Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j8270720-svc", component="kubernetes-infrastructure", namespace="clusters-service", pod="clusters-service-567b46758d-rsmsf", region="westus3", severity="warning" Description: Pod clusters-service/clusters-service-567b46758d-rsmsf has been in a non-ready state for longer than 5 minutes. Firing 19: State: Fired (not resolved) Started: 2026-10-28T00:30:21Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j8270720-svc", component="kubernetes-infrastructure", namespace="clusters-service", pod="ocm-cs-db-7594d5ccfc-vd2b7", region="westus3", severity="warning" Description: Pod clusters-service/ocm-cs-db-7594d5ccfc-vd2b7 has been in a non-ready state for longer than 5 minutes. Firing 20: State: Fired (not resolved) Started: 2026-10-28T00:30:21Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j8270720-svc", component="kubernetes-infrastructure", namespace="clusters-service", pod="clusters-service-567b46758d-hzqfx", region="westus3", severity="warning" Description: Pod clusters-service/clusters-service-567b46758d-hzqfx has been in a non-ready state for longer than 5 minutes. Firing 21: State: Fired (not resolved) Started: 2026-10-28T00:30:21Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j8270720-svc", component="kubernetes-infrastructure", namespace="maestro", pod="maestro-6bc6657c6b-stp2q", region="westus3", severity="warning" Description: Pod maestro/maestro-6bc6657c6b-stp2q has been in a non-ready state for longer than 5 minutes. alert fired 8 time(s) Firing 1: State: Fired (not resolved) Started: 2026-10-27T15:12:43Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j9877504-mgmt-2", component="kubernetes-infrastructure", namespace="multicluster-engine", pod="ocm-controller-859484fc49-qhntv", region="westus3", severity="warning" Description: Pod multicluster-engine/ocm-controller-859484fc49-qhntv has been in a non-ready state for longer than 5 minutes. Firing 2: State: Fired (not resolved) Started: 2026-10-27T15:12:43Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j9877504-mgmt-2", component="kubernetes-infrastructure", namespace="multicluster-engine", pod="cluster-manager-c5d6c587f-fd8rf", region="westus3", severity="warning" Description: Pod multicluster-engine/cluster-manager-c5d6c587f-fd8rf has been in a non-ready state for longer than 5 minutes. Firing 3: State: Fired (not resolved) Started: 2026-10-27T15:12:43Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j9877504-mgmt-2", component="kubernetes-infrastructure", namespace="multicluster-engine", pod="multicluster-engine-operator-79c768d748-lg8vd", region="westus3", severity="warning" Description: Pod multicluster-engine/multicluster-engine-operator-79c768d748-lg8vd has been in a non-ready state for longer than 5 minutes. Firing 4: State: Fired (not resolved) Started: 2026-10-27T15:12:43Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j9877504-mgmt-2", component="kubernetes-infrastructure", namespace="kube-applier", pod="kube-applier-8475cd7c99-dfzv8", region="westus3", severity="warning" Description: Pod kube-applier/kube-applier-8475cd7c99-dfzv8 has been in a non-ready state for longer than 5 minutes. Firing 5: State: Fired (not resolved) Started: 2026-10-27T15:12:43Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j9877504-mgmt-2", component="kubernetes-infrastructure", namespace="multicluster-engine", pod="ocm-webhook-68676899fd-mklv5", region="westus3", severity="warning" Description: Pod multicluster-engine/ocm-webhook-68676899fd-mklv5 has been in a non-ready state for longer than 5 minutes. Firing 6: State: Fired (not resolved) Started: 2026-10-27T15:12:43Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j9877504-mgmt-2", component="kubernetes-infrastructure", namespace="mgmt-agent", pod="mgmt-agent-6dc8498b6f-ntd7x", region="westus3", severity="warning" Description: Pod mgmt-agent/mgmt-agent-6dc8498b6f-ntd7x has been in a non-ready state for longer than 5 minutes. Firing 7: State: Fired (not resolved) Started: 2026-10-27T15:12:43Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j9877504-mgmt-2", component="kubernetes-infrastructure", namespace="multicluster-engine", pod="managedcluster-import-controller-v2-6cb46794f7-bzb25", region="westus3", severity="warning" Description: Pod multicluster-engine/managedcluster-import-controller-v2-6cb46794f7-bzb25 has been in a non-ready state for longer than 5 minutes. Firing 8: State: Fired (not resolved) Started: 2026-10-27T15:12:43Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j9877504-mgmt-2", component="kubernetes-infrastructure", namespace="secret-sync-controller", pod="secrets-store-sync-controller-manager-7f4b9bc5df-9hg5c", region="westus3", severity="warning" Description: Pod secret-sync-controller/secrets-store-sync-controller-manager-7f4b9bc5df-9hg5c has been in a non-ready state for longer than 5 minutes. Contributing tests (1)
Affected runs (14) | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
alert [hcp] KubeStatefulSetReplicasMismatch firedfull failure pattern: alert [hcp] KubeStatefulSetReplicasMismatch fired Signal: FlakeSignal: Flake — present in 4 prior week(s); spread across 6 days | alert | 12 | 2.59%12 of 464 job runs affected | FlakeSignal: Flake — present in 4 prior week(s); spread across 6 days | Show trend detailsOct 21: 0 · Oct 22: 2 · Oct 23: 2 · Oct 24: 0 · Oct 25: 1 · Oct 26: 2 · Oct 27: 2 · Oct 28: 3 | none | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (3)alert fired 2 time(s) Firing 1: State: Resolved Started: 2026-10-28T19:48:54Z Ended: 2026-10-28T20:14:47Z Severity: Sev3 Labels: alertname="KubeStatefulSetReplicasMismatch", cluster="ci01-j8717568-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", endpoint="http", environment="ci01", instance="10.128.64.254:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j8717568/providers/microsoft.monitor/accounts/hcps-j8717568", microsoft.metrics_include_label="service", namespace="ocm-arohcpci01-2tnvtu6q95ufl5426ec9nsjbnsqfa0aa-a8y2r7f6q8h7z0t", pod="arohcp-monitor-kube-state-metrics-85d48f46ff-59dhx", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="centralus", service="arohcp-monitor-kube-state-metrics", severity="warning", statefulset="etcd" Description: StatefulSet ocm-arohcpci01-2tnvtu6q95ufl5426ec9nsjbnsqfa0aa-a8y2r7f6q8h7z0t/etcd has not matched the expected number of replicas for longer than 15 minutes. Firing 2: State: Resolved Started: 2026-10-28T19:48:54Z Ended: 2026-10-28T20:14:47Z Severity: Sev3 Labels: alertname="KubeStatefulSetReplicasMismatch", cluster="ci01-j8717568-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", endpoint="http", environment="ci01", instance="10.128.64.254:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j8717568/providers/microsoft.monitor/accounts/hcps-j8717568", microsoft.metrics_include_label="service", namespace="ocm-arohcpci01-2tnvtu6q95ufl5426ec9nsjbnsqfa0aa-a8y2r7f6q8h7z0t", pod="arohcp-monitor-kube-state-metrics-85d48f46ff-59dhx", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="centralus", service="arohcp-monitor-kube-state-metrics", severity="warning", statefulset="etcd" Description: StatefulSet ocm-arohcpci01-2tnvtu6q95ufl5426ec9nsjbnsqfa0aa-a8y2r7f6q8h7z0t/etcd has not matched the expected number of replicas for longer than 15 minutes. alert fired 2 time(s) Firing 1: State: Resolved Started: 2026-10-28T12:57:34Z Ended: 2026-10-28T13:23:33Z Severity: Sev3 Labels: alertname="KubeStatefulSetReplicasMismatch", cluster="ci01-j6945280-mgmt-1", component="kubernetes-infrastructure", container="kube-state-metrics", endpoint="http", environment="ci01", instance="10.128.64.108:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j6945280/providers/microsoft.monitor/accounts/hcps-j6945280", microsoft.metrics_include_label="service", namespace="ocm-arohcpci01-2tnptam9khdl3svuvm4seuq5gr0ieit6-v5j4g5d3s9p2z6k", pod="arohcp-monitor-kube-state-metrics-85d48f46ff-fct9q", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning", statefulset="etcd" Description: StatefulSet ocm-arohcpci01-2tnptam9khdl3svuvm4seuq5gr0ieit6-v5j4g5d3s9p2z6k/etcd has not matched the expected number of replicas for longer than 15 minutes. Firing 2: State: Resolved Started: 2026-10-28T12:57:34Z Ended: 2026-10-28T13:23:33Z Severity: Sev3 Labels: alertname="KubeStatefulSetReplicasMismatch", cluster="ci01-j6945280-mgmt-1", component="kubernetes-infrastructure", container="kube-state-metrics", endpoint="http", environment="ci01", instance="10.128.64.108:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j6945280/providers/microsoft.monitor/accounts/hcps-j6945280", microsoft.metrics_include_label="service", namespace="ocm-arohcpci01-2tnptam9khdl3svuvm4seuq5gr0ieit6-v5j4g5d3s9p2z6k", pod="arohcp-monitor-kube-state-metrics-85d48f46ff-fct9q", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning", statefulset="etcd" Description: StatefulSet ocm-arohcpci01-2tnptam9khdl3svuvm4seuq5gr0ieit6-v5j4g5d3s9p2z6k/etcd has not matched the expected number of replicas for longer than 15 minutes. alert fired 2 time(s) Firing 1: State: Resolved Started: 2026-10-28T08:17:22Z Ended: 2026-10-28T08:40:22Z Severity: Sev3 Labels: alertname="KubeStatefulSetReplicasMismatch", cluster="ci01-j3004160-mgmt-1", component="kubernetes-infrastructure", container="kube-state-metrics", endpoint="http", environment="ci01", instance="10.128.64.217:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j3004160/providers/microsoft.monitor/accounts/hcps-j3004160", microsoft.metrics_include_label="service", namespace="ocm-arohcpci01-2tnloambpnud3aq6p4sg9mdteg909a3n-z3x1x9z7m5g8p6s", pod="arohcp-monitor-kube-state-metrics-68ff96c687-7dfbl", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", statefulset="etcd" Description: StatefulSet ocm-arohcpci01-2tnloambpnud3aq6p4sg9mdteg909a3n-z3x1x9z7m5g8p6s/etcd has not matched the expected number of replicas for longer than 15 minutes. Firing 2: State: Resolved Started: 2026-10-28T08:17:22Z Ended: 2026-10-28T08:40:22Z Severity: Sev3 Labels: alertname="KubeStatefulSetReplicasMismatch", cluster="ci01-j3004160-mgmt-1", component="kubernetes-infrastructure", container="kube-state-metrics", endpoint="http", environment="ci01", instance="10.128.64.217:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j3004160/providers/microsoft.monitor/accounts/hcps-j3004160", microsoft.metrics_include_label="service", namespace="ocm-arohcpci01-2tnloambpnud3aq6p4sg9mdteg909a3n-z3x1x9z7m5g8p6s", pod="arohcp-monitor-kube-state-metrics-68ff96c687-7dfbl", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", statefulset="etcd" Description: StatefulSet ocm-arohcpci01-2tnloambpnud3aq6p4sg9mdteg909a3n-z3x1x9z7m5g8p6s/etcd has not matched the expected number of replicas for longer than 15 minutes. Contributing tests (1)
Affected runs (12) | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
ERROR CODE: DeadlineExceeded; detail message node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hype...full failure pattern: ERROR CODE: DeadlineExceeded; detail message node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is <count>, want <count>; node pool condition AllNodesHealthy is False: <count> of <count> machines are not healthy Machine <node>: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.Compute/virtualMachines/<node> to exist; node pool condition AllMachinesReady is False: <count> of <count> machines are not ready Machine <node>: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.Compute/virtualMachines/<node> to exist; provider Microsoft.RedHatOpenShift Signal: FlakeSignal: Flake — present in 2 prior week(s); spread across 6 days | e2e | 11 | 2.37%11 of 464 job runs affected | FlakeSignal: Flake — present in 2 prior week(s); spread across 6 days | Show trend detailsOct 21: 3 · Oct 22: 5 · Oct 23: 2 · Oct 24: 2 · Oct 25: 0 · Oct 26: 1 · Oct 27: 1 · Oct 28: 0 | PROD | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (3)fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_nodepool_osdisk.go:96]: failed to create node pool "nodepool-128GiB" with 128GiB osDisk
Unexpected error:
<*fmt.wrapError | 0x3d4aea46a060>:
failed to create NodePool nodepool-128GiB: failed waiting for nodepool="nodepool-128GiB" for cluster "hcp-cluster-np-128" in resourcegroup="clusternp128-85bttqbvwzdg" to finish creating: GET https://rp.j7635968.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/7c4e2323-4cc6-4875-b033-1cc27f1dc9c9
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/7c4e2323-4cc6-4875-b033-1cc27f1dc9c9",
"name": "7c4e2323-4cc6-4875-b033-1cc27f1dc9c9",
"status": "Failed",
"startTime": "2026-10-27T17:47:01.433391882Z",
"endTime": "2026-10-27T18:06:02.631484435Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine b3p4u5e0h1k8y7f-nodepool-128gib-6n7h6-rw99j: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/clusternp128-85bttqbvwzdg--managed/providers/Microsoft.Compute/virtualMachines/b3p4u5e0h1k8y7f-nodepool-128gib-6n7h6-rw99j to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine b3p4u5e0h1k8y7f-nodepool-128gib-6n7h6-rw99j: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/clusternp128-85bttqbvwzdg--managed/providers/Microsoft.Compute/virtualMachines/b3p4u5e0h1k8y7f-nodepool-128gib-6n7h6-rw99j to exist\n"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://rp.j7635968.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/7c4e2323-4cc6-4875-b033-1cc27f1dc9c9
correlation ID: b503cd11-01c7-4b49-bd8f-b11bdfa64e3b
client request ID: 8e981bdf-718f-4374-bc9d-9fe858083611
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_nodepool_osdisk.go:96]: failed to create node pool "nodepool-128GiB" with 128GiB osDisk
Unexpected error:
<*fmt.wrapError | 0x3d4aea46a060>:
failed to create NodePool nodepool-128GiB: failed waiting for nodepool="nodepool-128GiB" for cluster "hcp-cluster-np-128" in resourcegroup="clusternp128-85bttqbvwzdg" to finish creating: GET https://rp.j7635968.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/7c4e2323-4cc6-4875-b033-1cc27f1dc9c9
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/7c4e2323-4cc6-4875-b033-1cc27f1dc9c9",
"name": "7c4e2323-4cc6-4875-b033-1cc27f1dc9c9",
"status": "Failed",
"startTime": "2026-10-27T17:47:01.433391882Z",
"endTime": "2026-10-27T18:06:02.631484435Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine b3p4u5e0h1k8y7f-nodepool-128gib-6n7h6-rw99j: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/clusternp128-85bttqbvwzdg--managed/providers/Microsoft.Compute/virtualMachines/b3p4u5e0h1k8y7f-nodepool-128gib-6n7h6-rw99j to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine b3p4u5e0h1k8y7f-nodepool-128gib-6n7h6-rw99j: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/clusternp128-85bttqbvwzdg--managed/providers/Microsoft.Compute/virtualMachines/b3p4u5e0h1k8y7f-nodepool-128gib-6n7h6-rw99j to exist\n"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://rp.j7635968.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/7c4e2323-4cc6-4875-b033-1cc27f1dc9c9
correlation ID: b503cd11-01c7-4b49-bd8f-b11bdfa64e3b
client request ID: 8e981bdf-718f-4374-bc9d-9fe858083611
occurredfail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:165]: failed to create node pool npupgrade-4-20 with version 4.20.44
Unexpected error:
<*fmt.wrapError | 0x75e04e9f9a0>:
failed to create NodePool npupgrade-4-20: failed waiting for nodepool="npupgrade-4-20" for cluster "np-version-upgrade-cluster-lmxnc7" in resourcegroup="rg-np-version-upgrade-lmxnc7-q9mf48fq597k" to finish creating: GET https://rp.j8018432.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/d40573b8-7e51-417d-bd40-a08b0b1a57bd
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/d40573b8-7e51-417d-bd40-a08b0b1a57bd",
"name": "d40573b8-7e51-417d-bd40-a08b0b1a57bd",
"status": "Failed",
"startTime": "2026-10-26T16:25:47.925761101Z",
"endTime": "2026-10-26T16:44:52.193515957Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine s7q2t8r2n3l4t7i-npupgrade-4-20-29l2x-hnzm5: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-version-upgrade-lmxnc7-q9mf48fq597k-np-upgrade-lm-08336457/providers/Microsoft.Compute/virtualMachines/s7q2t8r2n3l4t7i-npupgrade-4-20-29l2x-hnzm5 to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine s7q2t8r2n3l4t7i-npupgrade-4-20-29l2x-hnzm5: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-version-upgrade-lmxnc7-q9mf48fq597k-np-upgrade-lm-08336457/providers/Microsoft.Compute/virtualMachines/s7q2t8r2n3l4t7i-npupgrade-4-20-29l2x-hnzm5 to exist\n"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://rp.j8018432.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/d40573b8-7e51-417d-bd40-a08b0b1a57bd
correlation ID: f7b1fc07-103a-435d-9acb-75f17e0334fb
client request ID: 9e0fce66-a04c-49ec-96a9-4c1b9377e626
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:165]: failed to create node pool npupgrade-4-20 with version 4.20.44
Unexpected error:
<*fmt.wrapError | 0x75e04e9f9a0>:
failed to create NodePool npupgrade-4-20: failed waiting for nodepool="npupgrade-4-20" for cluster "np-version-upgrade-cluster-lmxnc7" in resourcegroup="rg-np-version-upgrade-lmxnc7-q9mf48fq597k" to finish creating: GET https://rp.j8018432.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/d40573b8-7e51-417d-bd40-a08b0b1a57bd
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/d40573b8-7e51-417d-bd40-a08b0b1a57bd",
"name": "d40573b8-7e51-417d-bd40-a08b0b1a57bd",
"status": "Failed",
"startTime": "2026-10-26T16:25:47.925761101Z",
"endTime": "2026-10-26T16:44:52.193515957Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine s7q2t8r2n3l4t7i-npupgrade-4-20-29l2x-hnzm5: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-version-upgrade-lmxnc7-q9mf48fq597k-np-upgrade-lm-08336457/providers/Microsoft.Compute/virtualMachines/s7q2t8r2n3l4t7i-npupgrade-4-20-29l2x-hnzm5 to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine s7q2t8r2n3l4t7i-npupgrade-4-20-29l2x-hnzm5: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-version-upgrade-lmxnc7-q9mf48fq597k-np-upgrade-lm-08336457/providers/Microsoft.Compute/virtualMachines/s7q2t8r2n3l4t7i-npupgrade-4-20-29l2x-hnzm5 to exist\n"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://rp.j8018432.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/d40573b8-7e51-417d-bd40-a08b0b1a57bd
correlation ID: f7b1fc07-103a-435d-9acb-75f17e0334fb
client request ID: 9e0fce66-a04c-49ec-96a9-4c1b9377e626
occurredfail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:165]: failed to create node pool npupgrade-4-20 with version 4.20.43
Unexpected error:
<*fmt.wrapError | 0x1f6cce913900>:
failed to create NodePool npupgrade-4-20: failed waiting for nodepool="npupgrade-4-20" for cluster "np-version-upgrade-cluster-b559ft" in resourcegroup="rg-np-version-upgrade-b559ft-j4qdwfs6szxx" to finish creating: GET https://rp.j3761152.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/489dec27-e4f4-49eb-a330-6052b204ad8e
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/489dec27-e4f4-49eb-a330-6052b204ad8e",
"name": "489dec27-e4f4-49eb-a330-6052b204ad8e",
"status": "Failed",
"startTime": "2026-10-24T05:53:26.952860579Z",
"endTime": "2026-10-24T06:12:28.455904126Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine p8m7i5l8y6g1n1v-npupgrade-4-20-kdgcn-qg9p4: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-version-upgrade-b559ft-j4qdwfs6szxx-np-upgrade-b5-5fe002ad/providers/Microsoft.Compute/virtualMachines/p8m7i5l8y6g1n1v-npupgrade-4-20-kdgcn-qg9p4 to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine p8m7i5l8y6g1n1v-npupgrade-4-20-kdgcn-qg9p4: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-version-upgrade-b559ft-j4qdwfs6szxx-np-upgrade-b5-5fe002ad/providers/Microsoft.Compute/virtualMachines/p8m7i5l8y6g1n1v-npupgrade-4-20-kdgcn-qg9p4 to exist\n"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://rp.j3761152.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/489dec27-e4f4-49eb-a330-6052b204ad8e
correlation ID: d5af14b9-8e48-4974-af98-abb8c215a6d4
client request ID: 1283d9b2-c265-4f0a-97ea-52c82d8cd670
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:165]: failed to create node pool npupgrade-4-20 with version 4.20.43
Unexpected error:
<*fmt.wrapError | 0x1f6cce913900>:
failed to create NodePool npupgrade-4-20: failed waiting for nodepool="npupgrade-4-20" for cluster "np-version-upgrade-cluster-b559ft" in resourcegroup="rg-np-version-upgrade-b559ft-j4qdwfs6szxx" to finish creating: GET https://rp.j3761152.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/489dec27-e4f4-49eb-a330-6052b204ad8e
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/489dec27-e4f4-49eb-a330-6052b204ad8e",
"name": "489dec27-e4f4-49eb-a330-6052b204ad8e",
"status": "Failed",
"startTime": "2026-10-24T05:53:26.952860579Z",
"endTime": "2026-10-24T06:12:28.455904126Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine p8m7i5l8y6g1n1v-npupgrade-4-20-kdgcn-qg9p4: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-version-upgrade-b559ft-j4qdwfs6szxx-np-upgrade-b5-5fe002ad/providers/Microsoft.Compute/virtualMachines/p8m7i5l8y6g1n1v-npupgrade-4-20-kdgcn-qg9p4 to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine p8m7i5l8y6g1n1v-npupgrade-4-20-kdgcn-qg9p4: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-version-upgrade-b559ft-j4qdwfs6szxx-np-upgrade-b5-5fe002ad/providers/Microsoft.Compute/virtualMachines/p8m7i5l8y6g1n1v-npupgrade-4-20-kdgcn-qg9p4 to exist\n"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://rp.j3761152.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/489dec27-e4f4-49eb-a330-6052b204ad8e
correlation ID: d5af14b9-8e48-4974-af98-abb8c215a6d4
client request ID: 1283d9b2-c265-4f0a-97ea-52c82d8cd670
occurredContributing tests (7)
Affected runs (11) | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
DNS for route host <route-host> did not resolve: DNS for <route-host> did not resolve within <duration> (last error: lookup <host> on <dns-server>: dial udp <ip>: i/o timeout): co...full failure pattern: DNS for route host <route-host> did not resolve: DNS for <route-host> did not resolve within <duration> (last error: lookup <host> on <dns-server>: dial udp <ip>: i/o timeout): context deadline exceeded Signal: FlakeSignal: Flake — present in 4 prior week(s); spread across 6 days | e2e | 11 | 2.37%11 of 464 job runs affected | FlakeSignal: Flake — present in 4 prior week(s); spread across 6 days | Show trend detailsOct 21: 0 · Oct 22: 2 · Oct 23: 1 · Oct 24: 1 · Oct 25: 2 · Oct 26: 0 · Oct 27: 4 · Oct 28: 1 | INT, PROD | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (3)fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:241]: failed to verify simple web app runs on cluster "cluster-candidate-5-1-678df7"
Unexpected error:
<*fmt.wrapError | 0x2d18a6e97e00>:
DNS for route host agnhost-e2e-sample-app-vqnk2.apps.aro.c7e9m0p1o3l4d4j.d5kj.j8717568.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-vqnk2.apps.aro.c7e9m0p1o3l4d4j.d5kj.j8717568.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-vqnk2.apps.aro.c7e9m0p1o3l4d4j.d5kj.j8717568.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded
{
msg: "DNS for route host agnhost-e2e-sample-app-vqnk2.apps.aro.c7e9m0p1o3l4d4j.d5kj.j8717568.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-vqnk2.apps.aro.c7e9m0p1o3l4d4j.d5kj.j8717568.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-vqnk2.apps.aro.c7e9m0p1o3l4d4j.d5kj.j8717568.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
err: <*fmt.wrapError | 0x2d18a6c74540>{
msg: "DNS for agnhost-e2e-sample-app-vqnk2.apps.aro.c7e9m0p1o3l4d4j.d5kj.j8717568.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-vqnk2.apps.aro.c7e9m0p1o3l4d4j.d5kj.j8717568.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
err: <context.deadlineExceededError>{},
},
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:241]: failed to verify simple web app runs on cluster "cluster-candidate-5-1-678df7"
Unexpected error:
<*fmt.wrapError | 0x2d18a6e97e00>:
DNS for route host agnhost-e2e-sample-app-vqnk2.apps.aro.c7e9m0p1o3l4d4j.d5kj.j8717568.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-vqnk2.apps.aro.c7e9m0p1o3l4d4j.d5kj.j8717568.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-vqnk2.apps.aro.c7e9m0p1o3l4d4j.d5kj.j8717568.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded
{
msg: "DNS for route host agnhost-e2e-sample-app-vqnk2.apps.aro.c7e9m0p1o3l4d4j.d5kj.j8717568.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-vqnk2.apps.aro.c7e9m0p1o3l4d4j.d5kj.j8717568.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-vqnk2.apps.aro.c7e9m0p1o3l4d4j.d5kj.j8717568.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
err: <*fmt.wrapError | 0x2d18a6c74540>{
msg: "DNS for agnhost-e2e-sample-app-vqnk2.apps.aro.c7e9m0p1o3l4d4j.d5kj.j8717568.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-vqnk2.apps.aro.c7e9m0p1o3l4d4j.d5kj.j8717568.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
err: <context.deadlineExceededError>{},
},
}
occurredfail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:241]: failed to verify simple web app runs on cluster "cluster-candidate-4-20-rvmpld"
Unexpected error:
<*fmt.wrapError | 0x1aa0726d2620>:
DNS for route host agnhost-e2e-sample-app-ffznf.apps.aro.j5p0x2q6g6t1g2c.gabn.j8373120.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-ffznf.apps.aro.j5p0x2q6g6t1g2c.gabn.j8373120.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-ffznf.apps.aro.j5p0x2q6g6t1g2c.gabn.j8373120.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded
{
msg: "DNS for route host agnhost-e2e-sample-app-ffznf.apps.aro.j5p0x2q6g6t1g2c.gabn.j8373120.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-ffznf.apps.aro.j5p0x2q6g6t1g2c.gabn.j8373120.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-ffznf.apps.aro.j5p0x2q6g6t1g2c.gabn.j8373120.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
err: <*fmt.wrapError | 0x1aa071c17780>{
msg: "DNS for agnhost-e2e-sample-app-ffznf.apps.aro.j5p0x2q6g6t1g2c.gabn.j8373120.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-ffznf.apps.aro.j5p0x2q6g6t1g2c.gabn.j8373120.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
err: <context.deadlineExceededError>{},
},
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:241]: failed to verify simple web app runs on cluster "cluster-candidate-4-20-rvmpld"
Unexpected error:
<*fmt.wrapError | 0x1aa0726d2620>:
DNS for route host agnhost-e2e-sample-app-ffznf.apps.aro.j5p0x2q6g6t1g2c.gabn.j8373120.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-ffznf.apps.aro.j5p0x2q6g6t1g2c.gabn.j8373120.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-ffznf.apps.aro.j5p0x2q6g6t1g2c.gabn.j8373120.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded
{
msg: "DNS for route host agnhost-e2e-sample-app-ffznf.apps.aro.j5p0x2q6g6t1g2c.gabn.j8373120.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-ffznf.apps.aro.j5p0x2q6g6t1g2c.gabn.j8373120.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-ffznf.apps.aro.j5p0x2q6g6t1g2c.gabn.j8373120.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
err: <*fmt.wrapError | 0x1aa071c17780>{
msg: "DNS for agnhost-e2e-sample-app-ffznf.apps.aro.j5p0x2q6g6t1g2c.gabn.j8373120.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-ffznf.apps.aro.j5p0x2q6g6t1g2c.gabn.j8373120.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
err: <context.deadlineExceededError>{},
},
}
occurredfail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_v20260901preview.go:121]: failed to verify simple web app runs on v20260901preview cluster "v20260901"
Unexpected error:
<*fmt.wrapError | 0x1cc8b3a2b3a0>:
DNS for route host agnhost-e2e-sample-app-czc2x.apps.aro.v20260901.yl4k.j4071168.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-czc2x.apps.aro.v20260901.yl4k.j4071168.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-czc2x.apps.aro.v20260901.yl4k.j4071168.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded
{
msg: "DNS for route host agnhost-e2e-sample-app-czc2x.apps.aro.v20260901.yl4k.j4071168.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-czc2x.apps.aro.v20260901.yl4k.j4071168.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-czc2x.apps.aro.v20260901.yl4k.j4071168.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
err: <*fmt.wrapError | 0x1cc8b3559460>{
msg: "DNS for agnhost-e2e-sample-app-czc2x.apps.aro.v20260901.yl4k.j4071168.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-czc2x.apps.aro.v20260901.yl4k.j4071168.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
err: <context.deadlineExceededError>{},
},
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_v20260901preview.go:121]: failed to verify simple web app runs on v20260901preview cluster "v20260901"
Unexpected error:
<*fmt.wrapError | 0x1cc8b3a2b3a0>:
DNS for route host agnhost-e2e-sample-app-czc2x.apps.aro.v20260901.yl4k.j4071168.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-czc2x.apps.aro.v20260901.yl4k.j4071168.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-czc2x.apps.aro.v20260901.yl4k.j4071168.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded
{
msg: "DNS for route host agnhost-e2e-sample-app-czc2x.apps.aro.v20260901.yl4k.j4071168.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-czc2x.apps.aro.v20260901.yl4k.j4071168.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-czc2x.apps.aro.v20260901.yl4k.j4071168.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
err: <*fmt.wrapError | 0x1cc8b3559460>{
msg: "DNS for agnhost-e2e-sample-app-czc2x.apps.aro.v20260901.yl4k.j4071168.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-czc2x.apps.aro.v20260901.yl4k.j4071168.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
err: <context.deadlineExceededError>{},
},
}
occurredContributing tests (10)
Affected runs (11) | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
alert [svc] BackendControllerQueueDepthHigh firedfull failure pattern: alert [svc] BackendControllerQueueDepthHigh fired Signal: FlakeSignal: Flake — present in 4 prior week(s); spread across 6 days | alert | 10 | 2.16%10 of 464 job runs affected | FlakeSignal: Flake — present in 4 prior week(s); spread across 6 days | Show trend detailsOct 21: 8 · Oct 22: 2 · Oct 23: 1 · Oct 24: 1 · Oct 25: 0 · Oct 26: 0 · Oct 27: 5 · Oct 28: 1 | none | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (3)alert fired 1 time(s) Firing 1: State: Resolved Started: 2026-10-28T19:42:52Z Ended: 2026-10-28T20:02:53Z Severity: Sev3 Labels: alertname="BackendControllerQueueDepthHigh", cluster="ci01-j3991936-svc", component="backend", name="csstatedump", region="westus3", severity="warning" Description: Backend controller workqueue csstatedump has had a depth > 10 for more than 15 minutes, indicating work is accumulating faster than it can be processed. alert fired 1 time(s) Firing 1: State: Resolved Started: 2026-10-27T20:20:10Z Ended: 2026-10-27T20:29:09Z Severity: Sev3 Labels: alertname="BackendControllerQueueDepthHigh", cluster="ci01-j7981312-svc", component="backend", name="csstatedump", region="westus3", severity="warning" Description: Backend controller workqueue csstatedump has had a depth > 10 for more than 15 minutes, indicating work is accumulating faster than it can be processed. alert fired 2 time(s) Firing 1: State: Resolved Started: 2026-10-27T15:28:06Z Ended: 2026-10-27T15:45:04Z Severity: Sev3 Labels: alertname="BackendControllerQueueDepthHigh", cluster="ci01-j4184192-svc", component="backend", name="csstatedump", region="centralus", severity="warning" Description: Backend controller workqueue csstatedump has had a depth > 10 for more than 15 minutes, indicating work is accumulating faster than it can be processed. Firing 2: State: Resolved Started: 2026-10-27T15:28:06Z Ended: 2026-10-27T15:45:04Z Severity: Sev3 Labels: alertname="BackendControllerQueueDepthHigh", cluster="ci01-j4184192-svc", component="backend", name="clusterresources", region="centralus", severity="warning" Description: Backend controller workqueue clusterresources has had a depth > 10 for more than 15 minutes, indicating work is accumulating faster than it can be processed. Contributing tests (1)
Affected runs (10) | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
alert [svc] KubeDeploymentReplicasMismatch firedfull failure pattern: alert [svc] KubeDeploymentReplicasMismatch fired Signal: FlakeSignal: Flake — present in 4 prior week(s); spread across 6 days | alert | 10 | 2.16%10 of 464 job runs affected | FlakeSignal: Flake — present in 4 prior week(s); spread across 6 days | Show trend detailsOct 21: 1 · Oct 22: 6 · Oct 23: 1 · Oct 24: 1 · Oct 25: 1 · Oct 26: 0 · Oct 27: 1 · Oct 28: 0 | none | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (3)alert fired 2 time(s) Firing 1: State: Fired (not resolved) Started: 2026-10-27T14:32:19Z Severity: Sev3 Labels: alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j8622848-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="klusterlet-addon-workmgr", endpoint="http", environment="ci01", instance="10.128.64.158:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j8622848/providers/microsoft.monitor/accounts/services-j8622848", microsoft.metrics_include_label="service", namespace="open-cluster-management-agent-addon", pod="arohcp-monitor-kube-state-metrics-85d48f46ff-xxzkn", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning" Description: Deployment open-cluster-management-agent-addon/klusterlet-addon-workmgr has not matched the expected number of replicas for longer than 30 minutes. Firing 2: State: Fired (not resolved) Started: 2026-10-27T14:33:18Z Severity: Sev3 Labels: alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j8622848-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="klusterlet-addon-workmgr", endpoint="http", environment="ci01", instance="10.128.64.158:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j8622848/providers/microsoft.monitor/accounts/services-j8622848", microsoft.metrics_include_label="service", namespace="open-cluster-management-agent-addon", pod="arohcp-monitor-kube-state-metrics-85d48f46ff-xxzkn", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning" Description: Deployment open-cluster-management-agent-addon/klusterlet-addon-workmgr has not matched the expected number of replicas for longer than 30 minutes. alert fired 3 time(s) Firing 1: State: Resolved Started: 2026-10-25T19:03:55Z Ended: 2026-10-25T19:10:56Z Severity: Sev3 Labels: alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j3671936-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="config-policy-controller", endpoint="http", environment="ci01", instance="10.128.64.162:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j3671936/providers/microsoft.monitor/accounts/services-j3671936", microsoft.metrics_include_label="service", namespace="klusterlet-2tlvp310lq0lqg3o9bn0eeqgoijvib5c", pod="arohcp-monitor-kube-state-metrics-85d48f46ff-7vdl7", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning" Description: Deployment klusterlet-2tlvp310lq0lqg3o9bn0eeqgoijvib5c/config-policy-controller has not matched the expected number of replicas for longer than 30 minutes. Firing 2: State: Resolved Started: 2026-10-25T19:03:55Z Ended: 2026-10-25T19:09:56Z Severity: Sev3 Labels: alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j3671936-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="klusterlet-addon-workmgr", endpoint="http", environment="ci01", instance="10.128.64.162:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j3671936/providers/microsoft.monitor/accounts/services-j3671936", microsoft.metrics_include_label="service", namespace="klusterlet-2tlvp310lq0lqg3o9bn0eeqgoijvib5c", pod="arohcp-monitor-kube-state-metrics-85d48f46ff-7vdl7", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning" Description: Deployment klusterlet-2tlvp310lq0lqg3o9bn0eeqgoijvib5c/klusterlet-addon-workmgr has not matched the expected number of replicas for longer than 30 minutes. Firing 3: State: Resolved Started: 2026-10-25T19:03:55Z Ended: 2026-10-25T19:10:56Z Severity: Sev3 Labels: alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j3671936-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="governance-policy-framework", endpoint="http", environment="ci01", instance="10.128.64.162:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j3671936/providers/microsoft.monitor/accounts/services-j3671936", microsoft.metrics_include_label="service", namespace="klusterlet-2tlvp310lq0lqg3o9bn0eeqgoijvib5c", pod="arohcp-monitor-kube-state-metrics-85d48f46ff-7vdl7", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning" Description: Deployment klusterlet-2tlvp310lq0lqg3o9bn0eeqgoijvib5c/governance-policy-framework has not matched the expected number of replicas for longer than 30 minutes. alert fired 6 time(s) Firing 1: State: Fired (not resolved) Started: 2026-10-24T10:00:39Z Severity: Sev3 Labels: alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j4869376-mgmt-1", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="klusterlet-addon-workmgr", endpoint="http", environment="ci01", instance="10.128.65.11:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j4869376/providers/microsoft.monitor/accounts/services-j4869376", microsoft.metrics_include_label="service", namespace="open-cluster-management-agent-addon", pod="arohcp-monitor-kube-state-metrics-85d48f46ff-kc252", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="centralus", service="arohcp-monitor-kube-state-metrics", severity="warning" Description: Deployment open-cluster-management-agent-addon/klusterlet-addon-workmgr has not matched the expected number of replicas for longer than 30 minutes. Firing 2: State: Fired (not resolved) Started: 2026-10-24T10:00:39Z Severity: Sev3 Labels: alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j4869376-mgmt-1", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="klusterlet-addon-workmgr", endpoint="http", environment="ci01", instance="10.128.65.11:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j4869376/providers/microsoft.monitor/accounts/services-j4869376", microsoft.metrics_include_label="service", namespace="open-cluster-management-agent-addon", pod="arohcp-monitor-kube-state-metrics-85d48f46ff-kc252", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="centralus", service="arohcp-monitor-kube-state-metrics", severity="warning" Description: Deployment open-cluster-management-agent-addon/klusterlet-addon-workmgr has not matched the expected number of replicas for longer than 30 minutes. Firing 3: State: Resolved Started: 2026-10-24T10:09:36Z Ended: 2026-10-24T10:18:36Z Severity: Sev3 Labels: alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j4869376-mgmt-1", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="klusterlet-addon-workmgr", endpoint="http", environment="ci01", instance="10.128.65.11:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j4869376/providers/microsoft.monitor/accounts/services-j4869376", microsoft.metrics_include_label="service", namespace="klusterlet-2tl2qrt5450mmm2tpbrvv2e0lvko6ltl", pod="arohcp-monitor-kube-state-metrics-85d48f46ff-kc252", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="centralus", service="arohcp-monitor-kube-state-metrics", severity="warning" Description: Deployment klusterlet-2tl2qrt5450mmm2tpbrvv2e0lvko6ltl/klusterlet-addon-workmgr has not matched the expected number of replicas for longer than 30 minutes. Firing 4: State: Resolved Started: 2026-10-24T10:09:36Z Ended: 2026-10-24T10:18:36Z Severity: Sev3 Labels: alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j4869376-mgmt-1", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="klusterlet-addon-workmgr", endpoint="http", environment="ci01", instance="10.128.65.11:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j4869376/providers/microsoft.monitor/accounts/services-j4869376", microsoft.metrics_include_label="service", namespace="klusterlet-2tl2qrt5450mmm2tpbrvv2e0lvko6ltl", pod="arohcp-monitor-kube-state-metrics-85d48f46ff-kc252", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="centralus", service="arohcp-monitor-kube-state-metrics", severity="warning" Description: Deployment klusterlet-2tl2qrt5450mmm2tpbrvv2e0lvko6ltl/klusterlet-addon-workmgr has not matched the expected number of replicas for longer than 30 minutes. Firing 5: State: Resolved Started: 2026-10-24T10:10:38Z Ended: 2026-10-24T10:21:36Z Severity: Sev3 Labels: alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j4869376-mgmt-1", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="klusterlet-addon-workmgr", endpoint="http", environment="ci01", instance="10.128.65.11:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j4869376/providers/microsoft.monitor/accounts/services-j4869376", microsoft.metrics_include_label="service", namespace="klusterlet-2tl2qq6lifhtq61o83ao3dcajsdckdi5", pod="arohcp-monitor-kube-state-metrics-85d48f46ff-kc252", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="centralus", service="arohcp-monitor-kube-state-metrics", severity="warning" Description: Deployment klusterlet-2tl2qq6lifhtq61o83ao3dcajsdckdi5/klusterlet-addon-workmgr has not matched the expected number of replicas for longer than 30 minutes. Firing 6: State: Resolved Started: 2026-10-24T10:10:38Z Ended: 2026-10-24T10:21:36Z Severity: Sev3 Labels: alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j4869376-mgmt-1", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="klusterlet-addon-workmgr", endpoint="http", environment="ci01", instance="10.128.65.11:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j4869376/providers/microsoft.monitor/accounts/services-j4869376", microsoft.metrics_include_label="service", namespace="klusterlet-2tl2qq6lifhtq61o83ao3dcajsdckdi5", pod="arohcp-monitor-kube-state-metrics-85d48f46ff-kc252", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="centralus", service="arohcp-monitor-kube-state-metrics", severity="warning" Description: Deployment klusterlet-2tl2qq6lifhtq61o83ao3dcajsdckdi5/klusterlet-addon-workmgr has not matched the expected number of replicas for longer than 30 minutes. Contributing tests (1)
Affected runs (10) | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
ERROR CODE: DeadlineExceeded; detail message node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hype...full failure pattern: ERROR CODE: DeadlineExceeded; detail message node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift node pool not cached yet; provider Microsoft.RedHatOpenShift Signal: FlakeSignal: Flake — present in 3 prior week(s); spread across 5 days | e2e | 7 | 1.51%7 of 464 job runs affected | FlakeSignal: Flake — present in 3 prior week(s); spread across 5 days | Show trend detailsOct 21: 2 · Oct 22: 0 · Oct 23: 5 · Oct 24: 0 · Oct 25: 1 · Oct 26: 2 · Oct 27: 1 · Oct 28: 0 | PROD | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (3)fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kas.go:159]: failed to create node pool "np-1" for private KAS cluster "private-kas"
Unexpected error:
<*fmt.wrapError | 0x19f0aa4bea20>:
failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "private-kas" in resourcegroup="private-kas-wj8x9hp7bqx5" to finish creating: GET https://rp.j8023424.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/6a73d0d2-8fd5-4af1-adbb-f385ed243d20
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/6a73d0d2-8fd5-4af1-adbb-f385ed243d20",
"name": "6a73d0d2-8fd5-4af1-adbb-f385ed243d20",
"status": "Failed",
"startTime": "2026-10-27T08:09:38.679552409Z",
"endTime": "2026-10-27T08:28:40.674204457Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift node pool not cached yet"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://rp.j8023424.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/6a73d0d2-8fd5-4af1-adbb-f385ed243d20
correlation ID: 51afa422-8d59-4334-91a3-bc90ade1f885
client request ID: ebb6ee5a-68e9-481e-8fab-49a3ebe79568
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kas.go:159]: failed to create node pool "np-1" for private KAS cluster "private-kas"
Unexpected error:
<*fmt.wrapError | 0x19f0aa4bea20>:
failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "private-kas" in resourcegroup="private-kas-wj8x9hp7bqx5" to finish creating: GET https://rp.j8023424.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/6a73d0d2-8fd5-4af1-adbb-f385ed243d20
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/6a73d0d2-8fd5-4af1-adbb-f385ed243d20",
"name": "6a73d0d2-8fd5-4af1-adbb-f385ed243d20",
"status": "Failed",
"startTime": "2026-10-27T08:09:38.679552409Z",
"endTime": "2026-10-27T08:28:40.674204457Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift node pool not cached yet"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://rp.j8023424.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/6a73d0d2-8fd5-4af1-adbb-f385ed243d20
correlation ID: 51afa422-8d59-4334-91a3-bc90ade1f885
client request ID: ebb6ee5a-68e9-481e-8fab-49a3ebe79568
occurredfail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:404]: failed to verify cilium is running and nodes are Ready for cluster "agg-cluster"
Unexpected error:
<*errors.joinError | 0x2d970cc94090>:
VerifyNodesReady failed: no nodes found in the cluster
VerifyCiliumOperational failed: not all pods in kube-system namespace are running: failed to list pods in kube-system namespace: context deadline exceeded
failed to create NodePool agg-np-a: failed waiting for nodepool="agg-np-a" for cluster "agg-cluster" in resourcegroup="feature-aggregation-rx9rnfvmj655" to finish creating: GET https://rp.j9849088.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/c8761660-38f6-4612-a201-a6ddec925f84
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/c8761660-38f6-4612-a201-a6ddec925f84",
"name": "c8761660-38f6-4612-a201-a6ddec925f84",
"status": "Failed",
"startTime": "2026-10-26T09:38:13.907922394Z",
"endTime": "2026-10-26T09:57:20.458727782Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift node pool not cached yet"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://rp.j9849088.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/c8761660-38f6-4612-a201-a6ddec925f84
correlation ID: 4c993536-8323-4301-a24e-f6918a189628
client request ID: e13d57f0-4312-4271-8e8e-02240879d813
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:404]: failed to verify cilium is running and nodes are Ready for cluster "agg-cluster"
Unexpected error:
<*errors.joinError | 0x2d970cc94090>:
VerifyNodesReady failed: no nodes found in the cluster
VerifyCiliumOperational failed: not all pods in kube-system namespace are running: failed to list pods in kube-system namespace: context deadline exceeded
failed to create NodePool agg-np-a: failed waiting for nodepool="agg-np-a" for cluster "agg-cluster" in resourcegroup="feature-aggregation-rx9rnfvmj655" to finish creating: GET https://rp.j9849088.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/c8761660-38f6-4612-a201-a6ddec925f84
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/c8761660-38f6-4612-a201-a6ddec925f84",
"name": "c8761660-38f6-4612-a201-a6ddec925f84",
"status": "Failed",
"startTime": "2026-10-26T09:38:13.907922394Z",
"endTime": "2026-10-26T09:57:20.458727782Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift node pool not cached yet"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://rp.j9849088.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/c8761660-38f6-4612-a201-a6ddec925f84
correlation ID: 4c993536-8323-4301-a24e-f6918a189628
client request ID: e13d57f0-4312-4271-8e8e-02240879d813
occurredfail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:154]: failed to create AZ nodepool autoscale-az with autoscaling
Unexpected error:
<*fmt.wrapError | 0x13f2532896c0>:
failed to create NodePool autoscale-az: failed waiting for nodepool="autoscale-az" for cluster "np-autoscale-cluster" in resourcegroup="np-autoscaling-n22957ncbgjt" to finish creating: GET https://rp.j9769984.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/ecac4fcd-9bd7-4a22-9510-51d365a0dee8
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/ecac4fcd-9bd7-4a22-9510-51d365a0dee8",
"name": "ecac4fcd-9bd7-4a22-9510-51d365a0dee8",
"status": "Failed",
"startTime": "2026-10-26T03:58:11.830961438Z",
"endTime": "2026-10-26T04:17:16.025648143Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift node pool not cached yet"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://rp.j9769984.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/ecac4fcd-9bd7-4a22-9510-51d365a0dee8
correlation ID: 5b299ccf-a19f-4517-91e6-0d46d8d87839
client request ID: 050413c1-cc10-4b65-a905-bd6edbbaa97f
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:154]: failed to create AZ nodepool autoscale-az with autoscaling
Unexpected error:
<*fmt.wrapError | 0x13f2532896c0>:
failed to create NodePool autoscale-az: failed waiting for nodepool="autoscale-az" for cluster "np-autoscale-cluster" in resourcegroup="np-autoscaling-n22957ncbgjt" to finish creating: GET https://rp.j9769984.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/ecac4fcd-9bd7-4a22-9510-51d365a0dee8
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/ecac4fcd-9bd7-4a22-9510-51d365a0dee8",
"name": "ecac4fcd-9bd7-4a22-9510-51d365a0dee8",
"status": "Failed",
"startTime": "2026-10-26T03:58:11.830961438Z",
"endTime": "2026-10-26T04:17:16.025648143Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift node pool not cached yet"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://rp.j9769984.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/ecac4fcd-9bd7-4a22-9510-51d365a0dee8
correlation ID: 5b299ccf-a19f-4517-91e6-0d46d8d87839
client request ID: 050413c1-cc10-4b65-a905-bd6edbbaa97f
occurredContributing tests (9)
Affected runs (7) | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
failed to run istio upgrade step: failed to start canary: start-canary: ARM operation timed out or failed: context deadline exceededfull failure pattern: failed to run istio upgrade step: failed to start canary: start-canary: ARM operation timed out or failed: context deadline exceeded Signal: FlakeSignal: Flake — present in 2 prior week(s); spread across 4 days | provision | 7 | 1.51%7 of 464 job runs affected | FlakeSignal: Flake — present in 2 prior week(s); spread across 4 days | Show trend detailsOct 21: 2 · Oct 22: 3 · Oct 23: 2 · Oct 24: 2 · Oct 25: 0 · Oct 26: 0 · Oct 27: 0 · Oct 28: 0 | none | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (3)time=2026-10-24T16:56:57.345Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=istio-upgrade description="Step istio-upgrade\n Kind: IstioUpgrade\n AKSCluster: {{ svc.aks.name }}\n"
time=2026-10-24T16:56:58.695Z level=INFO msg="Istio upgrade -- cluster state" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=istio-upgrade cluster=ci01-j5468416-svc versions=asm-1-30 logger=istio-upgrade k8sVersion=1.35 provisioningState=Succeeded installedRevisions=[asm-1-29] availableUpgrades=[asm-1-30] upgradeInProgress=false
time=2026-10-24T16:56:58.695Z level=INFO msg="Upgrading to svc.istio.versions target" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=istio-upgrade cluster=ci01-j5468416-svc versions=asm-1-30 logger=istio-upgrade from=asm-1-29 to=asm-1-30 k8sVersion=1.35
time=2026-10-24T16:56:59.317Z level=INFO msg="Istio upgrade -- mesh state" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=istio-upgrade cluster=ci01-j5468416-svc versions=asm-1-30 logger=istio-upgrade namespaces=0 controlPlanes=asm-1-29(2/2) ingressGateways=1 webhooks=1 configMaps=2
time=2026-10-24T16:56:59.317Z level=INFO msg="Starting canary -- installing target alongside current" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=istio-upgrade cluster=ci01-j5468416-svc versions=asm-1-30 logger=istio-upgrade
time=2026-10-24T16:56:59.317Z level=INFO msg="Starting ARM operation" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=istio-upgrade stepID=Microsoft.Azure.ARO.HCP.Service.Infra/service/istio-upgrade operation=start-canary cluster=ci01-j5468416-svc
time=2026-10-24T17:12:01.792Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=istio-upgrade err="failed to run istio upgrade step: failed to start canary: start-canary: ARM operation timed out or failed: context deadline exceeded"time=2026-10-24T12:42:08.055Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=istio-upgrade description="Step istio-upgrade\n Kind: IstioUpgrade\n AKSCluster: {{ svc.aks.name }}\n"
time=2026-10-24T12:42:09.448Z level=INFO msg="Istio upgrade -- cluster state" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=istio-upgrade cluster=ci01-j1222656-svc versions=asm-1-30 logger=istio-upgrade k8sVersion=1.35 provisioningState=Succeeded installedRevisions=[asm-1-29] availableUpgrades=[asm-1-30] upgradeInProgress=false
time=2026-10-24T12:42:09.448Z level=INFO msg="Upgrading to svc.istio.versions target" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=istio-upgrade cluster=ci01-j1222656-svc versions=asm-1-30 logger=istio-upgrade from=asm-1-29 to=asm-1-30 k8sVersion=1.35
time=2026-10-24T12:42:10.162Z level=INFO msg="Istio upgrade -- mesh state" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=istio-upgrade cluster=ci01-j1222656-svc versions=asm-1-30 logger=istio-upgrade namespaces=0 controlPlanes=asm-1-29(2/2) ingressGateways=1 webhooks=1 configMaps=2
time=2026-10-24T12:42:10.162Z level=INFO msg="Starting canary -- installing target alongside current" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=istio-upgrade cluster=ci01-j1222656-svc versions=asm-1-30 logger=istio-upgrade
time=2026-10-24T12:42:10.162Z level=INFO msg="Starting ARM operation" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=istio-upgrade stepID=Microsoft.Azure.ARO.HCP.Service.Infra/service/istio-upgrade operation=start-canary cluster=ci01-j1222656-svc
time=2026-10-24T12:57:12.942Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=istio-upgrade err="failed to run istio upgrade step: failed to start canary: start-canary: ARM operation timed out or failed: context deadline exceeded"time=2026-10-23T10:04:28.865Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=istio-upgrade description="Step istio-upgrade\n Kind: IstioUpgrade\n AKSCluster: {{ svc.aks.name }}\n"
time=2026-10-23T10:04:30.857Z level=INFO msg="Istio upgrade -- cluster state" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=istio-upgrade cluster=ci01-j4942976-svc versions=asm-1-30 logger=istio-upgrade k8sVersion=1.35 provisioningState=Succeeded installedRevisions=[asm-1-29] availableUpgrades=[asm-1-30] upgradeInProgress=false
time=2026-10-23T10:04:30.857Z level=INFO msg="Upgrading to svc.istio.versions target" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=istio-upgrade cluster=ci01-j4942976-svc versions=asm-1-30 logger=istio-upgrade from=asm-1-29 to=asm-1-30 k8sVersion=1.35
time=2026-10-23T10:04:32.118Z level=INFO msg="Istio upgrade -- mesh state" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=istio-upgrade cluster=ci01-j4942976-svc versions=asm-1-30 logger=istio-upgrade namespaces=0 controlPlanes=asm-1-29(2/2) ingressGateways=1 webhooks=1 configMaps=2
time=2026-10-23T10:04:32.118Z level=INFO msg="Starting canary -- installing target alongside current" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=istio-upgrade cluster=ci01-j4942976-svc versions=asm-1-30 logger=istio-upgrade
time=2026-10-23T10:04:32.118Z level=INFO msg="Starting ARM operation" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=istio-upgrade stepID=Microsoft.Azure.ARO.HCP.Service.Infra/service/istio-upgrade operation=start-canary cluster=ci01-j4942976-svc
time=2026-10-23T10:19:36.095Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=istio-upgrade err="failed to run istio upgrade step: failed to start canary: start-canary: ARM operation timed out or failed: context deadline exceeded"Contributing tests (1)
Affected runs (7) | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
alert [svc] KubeDeploymentRolloutStuck firedfull failure pattern: alert [svc] KubeDeploymentRolloutStuck fired Signal: FlakeSignal: Flake — present in 4 prior week(s); spread across 5 days | alert | 5 | 1.08%5 of 464 job runs affected | FlakeSignal: Flake — present in 4 prior week(s); spread across 5 days | Show trend detailsOct 21: 1 · Oct 22: 2 · Oct 23: 1 · Oct 24: 1 · Oct 25: 0 · Oct 26: 0 · Oct 27: 1 · Oct 28: 0 | none | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (3)alert fired 2 time(s) Firing 1: State: Fired (not resolved) Started: 2026-10-27T14:42:17Z Severity: Sev3 Labels: alertname="KubeDeploymentRolloutStuck", cluster="ci01-j8622848-mgmt-2", component="kubernetes-infrastructure", condition="Progressing", container="kube-state-metrics", deployment="klusterlet-addon-workmgr", endpoint="http", environment="ci01", instance="10.128.64.158:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j8622848/providers/microsoft.monitor/accounts/services-j8622848", microsoft.metrics_include_label="service", namespace="open-cluster-management-agent-addon", pod="arohcp-monitor-kube-state-metrics-85d48f46ff-xxzkn", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", reason="progressdeadlineexceeded", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning", status="false" Description: Rollout of deployment open-cluster-management-agent-addon/klusterlet-addon-workmgr is not progressing for longer than 30 minutes. Firing 2: State: Fired (not resolved) Started: 2026-10-27T14:43:21Z Severity: Sev3 Labels: alertname="KubeDeploymentRolloutStuck", cluster="ci01-j8622848-mgmt-2", component="kubernetes-infrastructure", condition="Progressing", container="kube-state-metrics", deployment="klusterlet-addon-workmgr", endpoint="http", environment="ci01", instance="10.128.64.158:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j8622848/providers/microsoft.monitor/accounts/services-j8622848", microsoft.metrics_include_label="service", namespace="open-cluster-management-agent-addon", pod="arohcp-monitor-kube-state-metrics-85d48f46ff-xxzkn", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", reason="progressdeadlineexceeded", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning", status="false" Description: Rollout of deployment open-cluster-management-agent-addon/klusterlet-addon-workmgr is not progressing for longer than 30 minutes. alert fired 2 time(s) Firing 1: State: Fired (not resolved) Started: 2026-10-24T10:10:38Z Severity: Sev3 Labels: alertname="KubeDeploymentRolloutStuck", cluster="ci01-j4869376-mgmt-1", component="kubernetes-infrastructure", condition="Progressing", container="kube-state-metrics", deployment="klusterlet-addon-workmgr", endpoint="http", environment="ci01", instance="10.128.65.11:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j4869376/providers/microsoft.monitor/accounts/services-j4869376", microsoft.metrics_include_label="service", namespace="open-cluster-management-agent-addon", pod="arohcp-monitor-kube-state-metrics-85d48f46ff-kc252", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", reason="progressdeadlineexceeded", region="centralus", service="arohcp-monitor-kube-state-metrics", severity="warning", status="false" Description: Rollout of deployment open-cluster-management-agent-addon/klusterlet-addon-workmgr is not progressing for longer than 30 minutes. Firing 2: State: Fired (not resolved) Started: 2026-10-24T10:10:38Z Severity: Sev3 Labels: alertname="KubeDeploymentRolloutStuck", cluster="ci01-j4869376-mgmt-1", component="kubernetes-infrastructure", condition="Progressing", container="kube-state-metrics", deployment="klusterlet-addon-workmgr", endpoint="http", environment="ci01", instance="10.128.65.11:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j4869376/providers/microsoft.monitor/accounts/services-j4869376", microsoft.metrics_include_label="service", namespace="open-cluster-management-agent-addon", pod="arohcp-monitor-kube-state-metrics-85d48f46ff-kc252", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", reason="progressdeadlineexceeded", region="centralus", service="arohcp-monitor-kube-state-metrics", severity="warning", status="false" Description: Rollout of deployment open-cluster-management-agent-addon/klusterlet-addon-workmgr is not progressing for longer than 30 minutes. alert fired 2 time(s) Firing 1: State: Fired (not resolved) Started: 2026-10-23T12:56:11Z Severity: Sev3 Labels: alertname="KubeDeploymentRolloutStuck", cluster="ci01-j0830592-mgmt-2", component="kubernetes-infrastructure", condition="Progressing", container="kube-state-metrics", deployment="klusterlet-addon-workmgr", endpoint="http", environment="ci01", instance="10.128.64.252:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0830592/providers/microsoft.monitor/accounts/services-j0830592", microsoft.metrics_include_label="service", namespace="open-cluster-management-agent-addon", pod="arohcp-monitor-kube-state-metrics-85d48f46ff-g9k4t", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", reason="progressdeadlineexceeded", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning", status="false" Description: Rollout of deployment open-cluster-management-agent-addon/klusterlet-addon-workmgr is not progressing for longer than 30 minutes. Firing 2: State: Fired (not resolved) Started: 2026-10-23T12:56:11Z Severity: Sev3 Labels: alertname="KubeDeploymentRolloutStuck", cluster="ci01-j0830592-mgmt-2", component="kubernetes-infrastructure", condition="Progressing", container="kube-state-metrics", deployment="klusterlet-addon-workmgr", endpoint="http", environment="ci01", instance="10.128.64.252:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0830592/providers/microsoft.monitor/accounts/services-j0830592", microsoft.metrics_include_label="service", namespace="open-cluster-management-agent-addon", pod="arohcp-monitor-kube-state-metrics-85d48f46ff-g9k4t", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", reason="progressdeadlineexceeded", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning", status="false" Description: Rollout of deployment open-cluster-management-agent-addon/klusterlet-addon-workmgr is not progressing for longer than 30 minutes. Contributing tests (1)
Affected runs (5) | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
alert [svc] KubePodCrashLooping firedfull failure pattern: alert [svc] KubePodCrashLooping fired Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 8 day(s) | alert | 25 | 5.39%25 of 464 job runs affected | IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 8 day(s) | Show trend detailsOct 21: 2 · Oct 22: 6 · Oct 23: 3 · Oct 24: 2 · Oct 25: 4 · Oct 26: 4 · Oct 27: 2 · Oct 28: 4 | none | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (3)alert fired 4 time(s) Firing 1: State: Fired (not resolved) Started: 2026-10-28T13:27:25Z Severity: Sev3 Labels: alertname="KubePodCrashLooping", cluster="ci01-j0229248-mgmt-1", component="kubernetes-infrastructure", container="hypershift-addon-agent", endpoint="http", environment="ci01", instance="10.128.64.125:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0229248/providers/microsoft.monitor/accounts/services-j0229248", microsoft.metrics_include_label="service", namespace="open-cluster-management-agent-addon", pod="hypershift-addon-agent-7fc5986cf8-phdkm", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", reason="CrashLoopBackOff", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="5471b0ac-6a29-48cb-ab86-2893cd772fbd" Description: Pod open-cluster-management-agent-addon/hypershift-addon-agent-7fc5986cf8-phdkm (hypershift-addon-agent) is in waiting state (reason: "CrashLoopBackOff"). Firing 2: State: Fired (not resolved) Started: 2026-10-28T13:29:30Z Severity: Sev3 Labels: alertname="KubePodCrashLooping", cluster="ci01-j0229248-mgmt-2", component="kubernetes-infrastructure", container="hypershift-addon-agent", endpoint="http", environment="ci01", instance="10.128.64.195:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0229248/providers/microsoft.monitor/accounts/services-j0229248", microsoft.metrics_include_label="service", namespace="open-cluster-management-agent-addon", pod="hypershift-addon-agent-7fc5986cf8-84g47", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", reason="CrashLoopBackOff", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="9462daaa-47d3-4b5e-b73b-01a9f857a137" Description: Pod open-cluster-management-agent-addon/hypershift-addon-agent-7fc5986cf8-84g47 (hypershift-addon-agent) is in waiting state (reason: "CrashLoopBackOff"). Firing 3: State: Fired (not resolved) Started: 2026-10-28T13:29:30Z Severity: Sev3 Labels: alertname="KubePodCrashLooping", cluster="ci01-j0229248-mgmt-2", component="kubernetes-infrastructure", container="hypershift-addon-agent", endpoint="http", environment="ci01", instance="10.128.64.195:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0229248/providers/microsoft.monitor/accounts/services-j0229248", microsoft.metrics_include_label="service", namespace="open-cluster-management-agent-addon", pod="hypershift-addon-agent-7fc5986cf8-84g47", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", reason="CrashLoopBackOff", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="9462daaa-47d3-4b5e-b73b-01a9f857a137" Description: Pod open-cluster-management-agent-addon/hypershift-addon-agent-7fc5986cf8-84g47 (hypershift-addon-agent) is in waiting state (reason: "CrashLoopBackOff"). Firing 4: State: Fired (not resolved) Started: 2026-10-28T13:32:24Z Severity: Sev3 Labels: alertname="KubePodCrashLooping", cluster="ci01-j0229248-mgmt-1", component="kubernetes-infrastructure", container="hypershift-addon-agent", endpoint="http", environment="ci01", instance="10.128.64.125:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0229248/providers/microsoft.monitor/accounts/services-j0229248", microsoft.metrics_include_label="service", namespace="open-cluster-management-agent-addon", pod="hypershift-addon-agent-7fc5986cf8-phdkm", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", reason="CrashLoopBackOff", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="5471b0ac-6a29-48cb-ab86-2893cd772fbd" Description: Pod open-cluster-management-agent-addon/hypershift-addon-agent-7fc5986cf8-phdkm (hypershift-addon-agent) is in waiting state (reason: "CrashLoopBackOff"). alert fired 4 time(s) Firing 1: State: Fired (not resolved) Started: 2026-10-28T10:42:37Z Severity: Sev3 Labels: alertname="KubePodCrashLooping", cluster="ci01-j1445504-mgmt-2", component="kubernetes-infrastructure", container="hypershift-addon-agent", endpoint="http", environment="ci01", instance="10.128.65.26:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j1445504/providers/microsoft.monitor/accounts/services-j1445504", microsoft.metrics_include_label="service", namespace="open-cluster-management-agent-addon", pod="hypershift-addon-agent-7fc5986cf8-5dmcb", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", reason="CrashLoopBackOff", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="6520e185-94fe-45c4-99a9-813a1698d907" Description: Pod open-cluster-management-agent-addon/hypershift-addon-agent-7fc5986cf8-5dmcb (hypershift-addon-agent) is in waiting state (reason: "CrashLoopBackOff"). Firing 2: State: Fired (not resolved) Started: 2026-10-28T10:43:33Z Severity: Sev3 Labels: alertname="KubePodCrashLooping", cluster="ci01-j1445504-mgmt-1", component="kubernetes-infrastructure", container="hypershift-addon-agent", endpoint="http", environment="ci01", instance="10.128.64.97:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j1445504/providers/microsoft.monitor/accounts/services-j1445504", microsoft.metrics_include_label="service", namespace="open-cluster-management-agent-addon", pod="hypershift-addon-agent-7fc5986cf8-ts4n7", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", reason="CrashLoopBackOff", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="0a28f799-e0bd-4698-b612-8c6756c4b517" Description: Pod open-cluster-management-agent-addon/hypershift-addon-agent-7fc5986cf8-ts4n7 (hypershift-addon-agent) is in waiting state (reason: "CrashLoopBackOff"). Firing 3: State: Fired (not resolved) Started: 2026-10-28T10:43:33Z Severity: Sev3 Labels: alertname="KubePodCrashLooping", cluster="ci01-j1445504-mgmt-1", component="kubernetes-infrastructure", container="hypershift-addon-agent", endpoint="http", environment="ci01", instance="10.128.64.97:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j1445504/providers/microsoft.monitor/accounts/services-j1445504", microsoft.metrics_include_label="service", namespace="open-cluster-management-agent-addon", pod="hypershift-addon-agent-7fc5986cf8-ts4n7", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", reason="CrashLoopBackOff", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="0a28f799-e0bd-4698-b612-8c6756c4b517" Description: Pod open-cluster-management-agent-addon/hypershift-addon-agent-7fc5986cf8-ts4n7 (hypershift-addon-agent) is in waiting state (reason: "CrashLoopBackOff"). Firing 4: State: Fired (not resolved) Started: 2026-10-28T10:47:32Z Severity: Sev3 Labels: alertname="KubePodCrashLooping", cluster="ci01-j1445504-mgmt-2", component="kubernetes-infrastructure", container="hypershift-addon-agent", endpoint="http", environment="ci01", instance="10.128.65.26:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j1445504/providers/microsoft.monitor/accounts/services-j1445504", microsoft.metrics_include_label="service", namespace="open-cluster-management-agent-addon", pod="hypershift-addon-agent-7fc5986cf8-5dmcb", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", reason="CrashLoopBackOff", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="6520e185-94fe-45c4-99a9-813a1698d907" Description: Pod open-cluster-management-agent-addon/hypershift-addon-agent-7fc5986cf8-5dmcb (hypershift-addon-agent) is in waiting state (reason: "CrashLoopBackOff"). alert fired 4 time(s) Firing 1: State: Fired (not resolved) Started: 2026-10-28T08:13:15Z Severity: Sev3 Labels: alertname="KubePodCrashLooping", cluster="ci01-j3004160-mgmt-1", component="kubernetes-infrastructure", container="hypershift-addon-agent", endpoint="http", environment="ci01", instance="10.128.64.217:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j3004160/providers/microsoft.monitor/accounts/services-j3004160", microsoft.metrics_include_label="service", namespace="open-cluster-management-agent-addon", pod="hypershift-addon-agent-7fc5986cf8-krfd7", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", reason="CrashLoopBackOff", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="2298911c-0832-493f-a9ef-f65728646e11" Description: Pod open-cluster-management-agent-addon/hypershift-addon-agent-7fc5986cf8-krfd7 (hypershift-addon-agent) is in waiting state (reason: "CrashLoopBackOff"). Firing 2: State: Fired (not resolved) Started: 2026-10-28T08:13:16Z Severity: Sev3 Labels: alertname="KubePodCrashLooping", cluster="ci01-j3004160-mgmt-1", component="kubernetes-infrastructure", container="hypershift-addon-agent", endpoint="http", environment="ci01", instance="10.128.64.217:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j3004160/providers/microsoft.monitor/accounts/services-j3004160", microsoft.metrics_include_label="service", namespace="open-cluster-management-agent-addon", pod="hypershift-addon-agent-7fc5986cf8-krfd7", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", reason="CrashLoopBackOff", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="2298911c-0832-493f-a9ef-f65728646e11" Description: Pod open-cluster-management-agent-addon/hypershift-addon-agent-7fc5986cf8-krfd7 (hypershift-addon-agent) is in waiting state (reason: "CrashLoopBackOff"). Firing 3: State: Fired (not resolved) Started: 2026-10-28T08:15:13Z Severity: Sev3 Labels: alertname="KubePodCrashLooping", cluster="ci01-j3004160-mgmt-2", component="kubernetes-infrastructure", container="hypershift-addon-agent", endpoint="http", environment="ci01", instance="10.128.64.208:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j3004160/providers/microsoft.monitor/accounts/services-j3004160", microsoft.metrics_include_label="service", namespace="open-cluster-management-agent-addon", pod="hypershift-addon-agent-7fc5986cf8-fgmvz", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", reason="CrashLoopBackOff", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="e461328c-3664-4b8e-907f-f8f42456ba46" Description: Pod open-cluster-management-agent-addon/hypershift-addon-agent-7fc5986cf8-fgmvz (hypershift-addon-agent) is in waiting state (reason: "CrashLoopBackOff"). Firing 4: State: Fired (not resolved) Started: 2026-10-28T08:15:13Z Severity: Sev3 Labels: alertname="KubePodCrashLooping", cluster="ci01-j3004160-mgmt-2", component="kubernetes-infrastructure", container="hypershift-addon-agent", endpoint="http", environment="ci01", instance="10.128.64.208:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j3004160/providers/microsoft.monitor/accounts/services-j3004160", microsoft.metrics_include_label="service", namespace="open-cluster-management-agent-addon", pod="hypershift-addon-agent-7fc5986cf8-fgmvz", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", reason="CrashLoopBackOff", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="e461328c-3664-4b8e-907f-f8f42456ba46" Description: Pod open-cluster-management-agent-addon/hypershift-addon-agent-7fc5986cf8-fgmvz (hypershift-addon-agent) is in waiting state (reason: "CrashLoopBackOff"). Contributing tests (1)
Affected runs (25) | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
at least one schedule should have a LastBackupTimefull failure pattern: at least one schedule should have a LastBackupTime Signal: IndeterminateSignal: Indeterminate — no prior history; active 2 day(s) — New failure pattern — no prior history | e2e | 23 | 4.96%23 of 464 job runs affected | IndeterminateSignal: Indeterminate — no prior history; active 2 day(s) — New failure pattern — no prior history | Show trend detailsOct 21: 0 · Oct 22: 0 · Oct 23: 0 · Oct 24: 0 · Oct 25: 0 · Oct 26: 1 · Oct 27: 22 · Oct 28: 0 | none | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (3)fail [github.com/Azure/ARO-HCP/test/e2e/hcp_backups.go:166]: Timed out after 660.004s.
at least one schedule should have a LastBackupTime
Expected
<bool>: false
to be true
fail [github.com/Azure/ARO-HCP/test/e2e/hcp_backups.go:166]: Timed out after 660.004s.
at least one schedule should have a LastBackupTime
Expected
<bool>: false
to be truefail [github.com/Azure/ARO-HCP/test/e2e/hcp_backups.go:166]: Timed out after 660.008s.
at least one schedule should have a LastBackupTime
Expected
<bool>: false
to be true
fail [github.com/Azure/ARO-HCP/test/e2e/hcp_backups.go:166]: Timed out after 660.008s.
at least one schedule should have a LastBackupTime
Expected
<bool>: false
to be truefail [github.com/Azure/ARO-HCP/test/e2e/hcp_backups.go:166]: Timed out after 660.005s.
at least one schedule should have a LastBackupTime
Expected
<bool>: false
to be true
fail [github.com/Azure/ARO-HCP/test/e2e/hcp_backups.go:166]: Timed out after 660.005s.
at least one schedule should have a LastBackupTime
Expected
<bool>: false
to be trueContributing tests (1)
Affected runs (23) | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
failed to roll out Helm release: resource Deployment/secret-sync-controller/secrets-store-sync-controller-manager not ready. status: InProgress, message: Available: 0/1 context de...full failure pattern: failed to roll out Helm release: resource Deployment/secret-sync-controller/secrets-store-sync-controller-manager not ready. status: InProgress, message: Available: 0/1 context deadline exceeded Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 4 day(s) | provision | 14 | 3.02%14 of 464 job runs affected | IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 4 day(s) | Show trend detailsOct 21: 0 · Oct 22: 3 · Oct 23: 6 · Oct 24: 0 · Oct 25: 4 · Oct 26: 0 · Oct 27: 1 · Oct 28: 0 | none | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (3)time=2026-10-27T15:06:00.351Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.SecretSyncController resourceGroup=management step=deploy stamp=2 description="Step deploy\n Kind: Helm\n" time=2026-10-27T15:06:00.927Z level=INFO msg="Resolved input values." serviceGroup=Microsoft.Azure.ARO.HCP.SecretSyncController resourceGroup=management step=deploy stamp=2 values="map[controllerName:secrets-store-sync-controller-manager image:arohcpsvcdev.azurecr.io/secrets-store-sync/controller@sha256:72abeb657cfbdd6a5891fc102a16429875d70c5932ad5994524d0246db914013 imagePullPolicy:IfNotPresent metricsPort:8085 providerContainer:[map[args:[--endpoint=unix:///provider/azure.sock --construct-pem-chain=true --healthz-port=8989 --healthz-path=/healthz --healthz-timeout=5s] image:mcr.microsoft.com/oss/v2/azure/secrets-store/provider-azure@sha256:55a0ad895a8f10193bb070883194449f5252c057353887af1ba0767760a0d2f0 imagePullPolicy:IfNotPresent livenessProbe:map[failureThreshold:3 httpGet:map[path:/healthz port:8989] initialDelaySeconds:5 periodSeconds:30 timeoutSeconds:10] name:provider-azure-installer ports:[map[containerPort:8898 name:metrics protocol:TCP]] resources:map[limits:map[memory:unlimited] requests:map[cpu:50m memory:100Mi]] securityContext:map[allowPrivilegeEscalation:false capabilities:map[drop:[ALL]] readOnlyRootFilesystem:true runAsUser:0] volumeMounts:[map[mountPath:/provider name:providervol]]]] tokenRequestAudience:[map[audience:<nil>]] validatingAdmissionPolicies:map[allowedSecretTypes:[Opaque kubernetes.io/basic-auth bootstrap.kubernetes.io/token kubernetes.io/dockerconfigjson kubernetes.io/dockercfg kubernetes.io/ssh-auth kubernetes.io/tls] applyPolicies:true deniedSecretTypes:[kubernetes.io/service-account-token]]]" time=2026-10-27T15:06:00.927Z level=INFO msg="Applying namespaces." serviceGroup=Microsoft.Azure.ARO.HCP.SecretSyncController resourceGroup=management step=deploy stamp=2 time=2026-10-27T15:06:01.327Z level=INFO msg="Namespace applied successfully." serviceGroup=Microsoft.Azure.ARO.HCP.SecretSyncController resourceGroup=management step=deploy stamp=2 namespace=secret-sync-controller time=2026-10-27T15:06:01.636Z level=INFO msg="Rolling out Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.SecretSyncController resourceGroup=management step=deploy stamp=2 dryRun=false time=2026-10-27T15:06:01.636Z level=INFO msg="Searching for release history..." serviceGroup=Microsoft.Azure.ARO.HCP.SecretSyncController resourceGroup=management step=deploy stamp=2 time=2026-10-27T15:06:01.765Z level=INFO msg="No release history found, running the first release..." serviceGroup=Microsoft.Azure.ARO.HCP.SecretSyncController resourceGroup=management step=deploy stamp=2 time=2026-10-27T15:11:07.858Z level=ERROR msg="Failed to roll out the Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.SecretSyncController resourceGroup=management step=deploy stamp=2 err="resource Deployment/secret-sync-controller/secrets-store-sync-controller-manager not ready. status: InProgress, message: Available: 0/1\ncontext deadline exceeded" time=2026-10-27T15:11:07.859Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.SecretSyncController resourceGroup=management step=deploy stamp=2 err="stamp 2: error running Helm release deployment Step, failed to deploy helm release: failed to roll out Helm release: resource Deployment/secret-sync-controller/secrets-store-sync-controller-manager not ready. status: InProgress, message: Available: 0/1\ncontext deadline exceeded" time=2026-10-25T10:27:22.014Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.SecretSyncController resourceGroup=management step=deploy stamp=2 description="Step deploy\n Kind: Helm\n" time=2026-10-25T10:27:22.644Z level=INFO msg="Resolved input values." serviceGroup=Microsoft.Azure.ARO.HCP.SecretSyncController resourceGroup=management step=deploy stamp=2 values="map[controllerName:secrets-store-sync-controller-manager image:arohcpsvcdev.azurecr.io/secrets-store-sync/controller@sha256:b98f9f1c3c603f0e3ee70cd3d70746d8265b8d425df5b71f23c4b0dcb3eb979c imagePullPolicy:IfNotPresent metricsPort:8085 providerContainer:[map[args:[--endpoint=unix:///provider/azure.sock --construct-pem-chain=true --healthz-port=8989 --healthz-path=/healthz --healthz-timeout=5s] image:mcr.microsoft.com/oss/v2/azure/secrets-store/provider-azure@sha256:b451542dd17a9b1b1b400fa1ca768a492720b87b696f373f0494217d855aee8c imagePullPolicy:IfNotPresent livenessProbe:map[failureThreshold:3 httpGet:map[path:/healthz port:8989] initialDelaySeconds:5 periodSeconds:30 timeoutSeconds:10] name:provider-azure-installer ports:[map[containerPort:8898 name:metrics protocol:TCP]] resources:map[limits:map[memory:unlimited] requests:map[cpu:50m memory:100Mi]] securityContext:map[allowPrivilegeEscalation:false capabilities:map[drop:[ALL]] readOnlyRootFilesystem:true runAsUser:0] volumeMounts:[map[mountPath:/provider name:providervol]]]] tokenRequestAudience:[map[audience:<nil>]] validatingAdmissionPolicies:map[allowedSecretTypes:[Opaque kubernetes.io/basic-auth bootstrap.kubernetes.io/token kubernetes.io/dockerconfigjson kubernetes.io/dockercfg kubernetes.io/ssh-auth kubernetes.io/tls] applyPolicies:true deniedSecretTypes:[kubernetes.io/service-account-token]]]" time=2026-10-25T10:27:22.644Z level=INFO msg="Applying namespaces." serviceGroup=Microsoft.Azure.ARO.HCP.SecretSyncController resourceGroup=management step=deploy stamp=2 time=2026-10-25T10:27:22.918Z level=INFO msg="Namespace applied successfully." serviceGroup=Microsoft.Azure.ARO.HCP.SecretSyncController resourceGroup=management step=deploy stamp=2 namespace=secret-sync-controller time=2026-10-25T10:27:23.279Z level=INFO msg="Rolling out Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.SecretSyncController resourceGroup=management step=deploy stamp=2 dryRun=false time=2026-10-25T10:27:23.279Z level=INFO msg="Searching for release history..." serviceGroup=Microsoft.Azure.ARO.HCP.SecretSyncController resourceGroup=management step=deploy stamp=2 time=2026-10-25T10:27:23.445Z level=INFO msg="No release history found, running the first release..." serviceGroup=Microsoft.Azure.ARO.HCP.SecretSyncController resourceGroup=management step=deploy stamp=2 time=2026-10-25T10:32:28.984Z level=ERROR msg="Failed to roll out the Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.SecretSyncController resourceGroup=management step=deploy stamp=2 err="resource Deployment/secret-sync-controller/secrets-store-sync-controller-manager not ready. status: InProgress, message: Available: 0/1\ncontext deadline exceeded" time=2026-10-25T10:32:28.985Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.SecretSyncController resourceGroup=management step=deploy stamp=2 err="stamp 2: error running Helm release deployment Step, failed to deploy helm release: failed to roll out Helm release: resource Deployment/secret-sync-controller/secrets-store-sync-controller-manager not ready. status: InProgress, message: Available: 0/1\ncontext deadline exceeded" time=2026-10-25T08:14:44.299Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.SecretSyncController resourceGroup=management step=deploy stamp=1 description="Step deploy\n Kind: Helm\n" time=2026-10-25T08:14:45.051Z level=INFO msg="Resolved input values." serviceGroup=Microsoft.Azure.ARO.HCP.SecretSyncController resourceGroup=management step=deploy stamp=1 values="map[controllerName:secrets-store-sync-controller-manager image:arohcpsvcdev.azurecr.io/secrets-store-sync/controller@sha256:b98f9f1c3c603f0e3ee70cd3d70746d8265b8d425df5b71f23c4b0dcb3eb979c imagePullPolicy:IfNotPresent metricsPort:8085 providerContainer:[map[args:[--endpoint=unix:///provider/azure.sock --construct-pem-chain=true --healthz-port=8989 --healthz-path=/healthz --healthz-timeout=5s] image:mcr.microsoft.com/oss/v2/azure/secrets-store/provider-azure@sha256:b451542dd17a9b1b1b400fa1ca768a492720b87b696f373f0494217d855aee8c imagePullPolicy:IfNotPresent livenessProbe:map[failureThreshold:3 httpGet:map[path:/healthz port:8989] initialDelaySeconds:5 periodSeconds:30 timeoutSeconds:10] name:provider-azure-installer ports:[map[containerPort:8898 name:metrics protocol:TCP]] resources:map[limits:map[memory:unlimited] requests:map[cpu:50m memory:100Mi]] securityContext:map[allowPrivilegeEscalation:false capabilities:map[drop:[ALL]] readOnlyRootFilesystem:true runAsUser:0] volumeMounts:[map[mountPath:/provider name:providervol]]]] tokenRequestAudience:[map[audience:<nil>]] validatingAdmissionPolicies:map[allowedSecretTypes:[Opaque kubernetes.io/basic-auth bootstrap.kubernetes.io/token kubernetes.io/dockerconfigjson kubernetes.io/dockercfg kubernetes.io/ssh-auth kubernetes.io/tls] applyPolicies:true deniedSecretTypes:[kubernetes.io/service-account-token]]]" time=2026-10-25T08:14:45.051Z level=INFO msg="Applying namespaces." serviceGroup=Microsoft.Azure.ARO.HCP.SecretSyncController resourceGroup=management step=deploy stamp=1 time=2026-10-25T08:14:45.565Z level=INFO msg="Namespace applied successfully." serviceGroup=Microsoft.Azure.ARO.HCP.SecretSyncController resourceGroup=management step=deploy stamp=1 namespace=secret-sync-controller time=2026-10-25T08:14:45.984Z level=INFO msg="Rolling out Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.SecretSyncController resourceGroup=management step=deploy stamp=1 dryRun=false time=2026-10-25T08:14:45.984Z level=INFO msg="Searching for release history..." serviceGroup=Microsoft.Azure.ARO.HCP.SecretSyncController resourceGroup=management step=deploy stamp=1 time=2026-10-25T08:14:46.156Z level=INFO msg="No release history found, running the first release..." serviceGroup=Microsoft.Azure.ARO.HCP.SecretSyncController resourceGroup=management step=deploy stamp=1 time=2026-10-25T08:19:54.254Z level=ERROR msg="Failed to roll out the Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.SecretSyncController resourceGroup=management step=deploy stamp=1 err="resource Deployment/secret-sync-controller/secrets-store-sync-controller-manager not ready. status: InProgress, message: Available: 0/1\ncontext deadline exceeded" time=2026-10-25T08:19:54.254Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.SecretSyncController resourceGroup=management step=deploy stamp=1 err="stamp 1: error running Helm release deployment Step, failed to deploy helm release: failed to roll out Helm release: resource Deployment/secret-sync-controller/secrets-store-sync-controller-manager not ready. status: InProgress, message: Available: 0/1\ncontext deadline exceeded" Contributing tests (2)
Affected runs (14) | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
VerifyCloudNetworkConfigEgressIP(<ip>) failed: failed to label node <node> egress-assignable: failed to apply k8s.ovn.org/egress-assignable label to node "<node>": nodes "<node>" ...full failure pattern: VerifyCloudNetworkConfigEgressIP(<ip>) failed: failed to label node <node> egress-assignable: failed to apply k8s.ovn.org/egress-assignable label to node "<node>": nodes "<node>" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 5 day(s) | e2e | 10 | 2.16%10 of 464 job runs affected | IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 5 day(s) | Show trend detailsOct 21: 0 · Oct 22: 0 · Oct 23: 2 · Oct 24: 3 · Oct 25: 0 · Oct 26: 1 · Oct 27: 2 · Oct 28: 2 | INT | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (3)fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:289]: failed to verify cluster operators on cluster "cluster-candidate-5-1-nxx9xr"
Unexpected error:
<*errors.joinError | 0x138a2d1f8048>:
VerifyCloudNetworkConfigEgressIP(10.0.0.7) failed: failed to label node e3a5t5r1l8c7f8q-np-nxx9xr-qn466-6zltb egress-assignable: failed to apply k8s.ovn.org/egress-assignable label to node "e3a5t5r1l8c7f8q-np-nxx9xr-qn466-6zltb": nodes "e3a5t5r1l8c7f8q-np-nxx9xr-qn466-6zltb" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized
{
errs: [
<*fmt.wrapError | 0x138a2cdf5020>{
msg: "VerifyCloudNetworkConfigEgressIP(10.0.0.7) failed: failed to label node e3a5t5r1l8c7f8q-np-nxx9xr-qn466-6zltb egress-assignable: failed to apply k8s.ovn.org/egress-assignable label to node \"e3a5t5r1l8c7f8q-np-nxx9xr-qn466-6zltb\": nodes \"e3a5t5r1l8c7f8q-np-nxx9xr-qn466-6zltb\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
err: <*fmt.wrapError | 0x138a2cdf5000>{
msg: "failed to label node e3a5t5r1l8c7f8q-np-nxx9xr-qn466-6zltb egress-assignable: failed to apply k8s.ovn.org/egress-assignable label to node \"e3a5t5r1l8c7f8q-np-nxx9xr-qn466-6zltb\": nodes \"e3a5t5r1l8c7f8q-np-nxx9xr-qn466-6zltb\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
err: <*fmt.wrapError | 0x138a2cdf4fe0>{
msg: "failed to apply k8s.ovn.org/egress-assignable label to node \"e3a5t5r1l8c7f8q-np-nxx9xr-qn466-6zltb\": nodes \"e3a5t5r1l8c7f8q-np-nxx9xr-qn466-6zltb\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
err: <*errors.StatusError | 0x138a2cd0dcc0>{
ErrStatus: {
TypeMeta: {Kind: "", APIVersion: ""},
ListMeta: {
SelfLink: "",
ResourceVersion: "",
Continue: "",
RemainingItemCount: nil,
ShardInfo: nil,
},
Status: "Failure",
Message: "nodes \"e3a5t5r1l8c7f8q-np-nxx9xr-qn466-6zltb\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
Reason: "Forbidden",
Details: {
Name: "e3a5t5r1l8c7f8q-np-nxx9xr-qn466-6zltb",
Group: "",
Kind: "nodes",
UID: "",
Causes: nil,
RetryAfterSeconds: 0,
},
Code: 403,
},
},
},
},
},
],
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:289]: failed to verify cluster operators on cluster "cluster-candidate-5-1-nxx9xr"
Unexpected error:
<*errors.joinError | 0x138a2d1f8048>:
VerifyCloudNetworkConfigEgressIP(10.0.0.7) failed: failed to label node e3a5t5r1l8c7f8q-np-nxx9xr-qn466-6zltb egress-assignable: failed to apply k8s.ovn.org/egress-assignable label to node "e3a5t5r1l8c7f8q-np-nxx9xr-qn466-6zltb": nodes "e3a5t5r1l8c7f8q-np-nxx9xr-qn466-6zltb" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized
{
errs: [
<*fmt.wrapError | 0x138a2cdf5020>{
msg: "VerifyCloudNetworkConfigEgressIP(10.0.0.7) failed: failed to label node e3a5t5r1l8c7f8q-np-nxx9xr-qn466-6zltb egress-assignable: failed to apply k8s.ovn.org/egress-assignable label to node \"e3a5t5r1l8c7f8q-np-nxx9xr-qn466-6zltb\": nodes \"e3a5t5r1l8c7f8q-np-nxx9xr-qn466-6zltb\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
err: <*fmt.wrapError | 0x138a2cdf5000>{
msg: "failed to label node e3a5t5r1l8c7f8q-np-nxx9xr-qn466-6zltb egress-assignable: failed to apply k8s.ovn.org/egress-assignable label to node \"e3a5t5r1l8c7f8q-np-nxx9xr-qn466-6zltb\": nodes \"e3a5t5r1l8c7f8q-np-nxx9xr-qn466-6zltb\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
err: <*fmt.wrapError | 0x138a2cdf4fe0>{
msg: "failed to apply k8s.ovn.org/egress-assignable label to node \"e3a5t5r1l8c7f8q-np-nxx9xr-qn466-6zltb\": nodes \"e3a5t5r1l8c7f8q-np-nxx9xr-qn466-6zltb\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
err: <*errors.StatusError | 0x138a2cd0dcc0>{
ErrStatus: {
TypeMeta: {Kind: "", APIVersion: ""},
ListMeta: {
SelfLink: "",
ResourceVersion: "",
Continue: "",
RemainingItemCount: nil,
ShardInfo: nil,
},
Status: "Failure",
Message: "nodes \"e3a5t5r1l8c7f8q-np-nxx9xr-qn466-6zltb\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
Reason: "Forbidden",
Details: {
Name: "e3a5t5r1l8c7f8q-np-nxx9xr-qn466-6zltb",
Group: "",
Kind: "nodes",
UID: "",
Causes: nil,
RetryAfterSeconds: 0,
},
Code: 403,
},
},
},
},
},
],
}
occurredfail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:289]: failed to verify cluster operators on cluster "cluster-candidate-5-1-x4d58d"
Unexpected error:
<*errors.joinError | 0x2a036e0a6030>:
VerifyCloudNetworkConfigEgressIP(10.0.0.7) failed: failed to label node t5n3k9o3y9v8w3y-np-x4d58d-tdn4x-htxvc egress-assignable: failed to apply k8s.ovn.org/egress-assignable label to node "t5n3k9o3y9v8w3y-np-x4d58d-tdn4x-htxvc": nodes "t5n3k9o3y9v8w3y-np-x4d58d-tdn4x-htxvc" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized
{
errs: [
<*fmt.wrapError | 0x2a036cc0d0a0>{
msg: "VerifyCloudNetworkConfigEgressIP(10.0.0.7) failed: failed to label node t5n3k9o3y9v8w3y-np-x4d58d-tdn4x-htxvc egress-assignable: failed to apply k8s.ovn.org/egress-assignable label to node \"t5n3k9o3y9v8w3y-np-x4d58d-tdn4x-htxvc\": nodes \"t5n3k9o3y9v8w3y-np-x4d58d-tdn4x-htxvc\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
err: <*fmt.wrapError | 0x2a036cc0d080>{
msg: "failed to label node t5n3k9o3y9v8w3y-np-x4d58d-tdn4x-htxvc egress-assignable: failed to apply k8s.ovn.org/egress-assignable label to node \"t5n3k9o3y9v8w3y-np-x4d58d-tdn4x-htxvc\": nodes \"t5n3k9o3y9v8w3y-np-x4d58d-tdn4x-htxvc\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
err: <*fmt.wrapError | 0x2a036cc0d060>{
msg: "failed to apply k8s.ovn.org/egress-assignable label to node \"t5n3k9o3y9v8w3y-np-x4d58d-tdn4x-htxvc\": nodes \"t5n3k9o3y9v8w3y-np-x4d58d-tdn4x-htxvc\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
err: <*errors.StatusError | 0x2a036cc3c780>{
ErrStatus: {
TypeMeta: {Kind: "", APIVersion: ""},
ListMeta: {
SelfLink: "",
ResourceVersion: "",
Continue: "",
RemainingItemCount: nil,
ShardInfo: nil,
},
Status: "Failure",
Message: "nodes \"t5n3k9o3y9v8w3y-np-x4d58d-tdn4x-htxvc\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
Reason: "Forbidden",
Details: {
Name: "t5n3k9o3y9v8w3y-np-x4d58d-tdn4x-htxvc",
Group: "",
Kind: "nodes",
UID: "",
Causes: nil,
RetryAfterSeconds: 0,
},
Code: 403,
},
},
},
},
},
],
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:289]: failed to verify cluster operators on cluster "cluster-candidate-5-1-x4d58d"
Unexpected error:
<*errors.joinError | 0x2a036e0a6030>:
VerifyCloudNetworkConfigEgressIP(10.0.0.7) failed: failed to label node t5n3k9o3y9v8w3y-np-x4d58d-tdn4x-htxvc egress-assignable: failed to apply k8s.ovn.org/egress-assignable label to node "t5n3k9o3y9v8w3y-np-x4d58d-tdn4x-htxvc": nodes "t5n3k9o3y9v8w3y-np-x4d58d-tdn4x-htxvc" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized
{
errs: [
<*fmt.wrapError | 0x2a036cc0d0a0>{
msg: "VerifyCloudNetworkConfigEgressIP(10.0.0.7) failed: failed to label node t5n3k9o3y9v8w3y-np-x4d58d-tdn4x-htxvc egress-assignable: failed to apply k8s.ovn.org/egress-assignable label to node \"t5n3k9o3y9v8w3y-np-x4d58d-tdn4x-htxvc\": nodes \"t5n3k9o3y9v8w3y-np-x4d58d-tdn4x-htxvc\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
err: <*fmt.wrapError | 0x2a036cc0d080>{
msg: "failed to label node t5n3k9o3y9v8w3y-np-x4d58d-tdn4x-htxvc egress-assignable: failed to apply k8s.ovn.org/egress-assignable label to node \"t5n3k9o3y9v8w3y-np-x4d58d-tdn4x-htxvc\": nodes \"t5n3k9o3y9v8w3y-np-x4d58d-tdn4x-htxvc\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
err: <*fmt.wrapError | 0x2a036cc0d060>{
msg: "failed to apply k8s.ovn.org/egress-assignable label to node \"t5n3k9o3y9v8w3y-np-x4d58d-tdn4x-htxvc\": nodes \"t5n3k9o3y9v8w3y-np-x4d58d-tdn4x-htxvc\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
err: <*errors.StatusError | 0x2a036cc3c780>{
ErrStatus: {
TypeMeta: {Kind: "", APIVersion: ""},
ListMeta: {
SelfLink: "",
ResourceVersion: "",
Continue: "",
RemainingItemCount: nil,
ShardInfo: nil,
},
Status: "Failure",
Message: "nodes \"t5n3k9o3y9v8w3y-np-x4d58d-tdn4x-htxvc\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
Reason: "Forbidden",
Details: {
Name: "t5n3k9o3y9v8w3y-np-x4d58d-tdn4x-htxvc",
Group: "",
Kind: "nodes",
UID: "",
Causes: nil,
RetryAfterSeconds: 0,
},
Code: 403,
},
},
},
},
},
],
}
occurredfail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:289]: failed to verify cluster operators on cluster "cluster-candidate-5-0-j22hl8"
Unexpected error:
<*errors.joinError | 0x237cb4aa4138>:
VerifyCloudNetworkConfigEgressIP(10.0.0.7) failed: failed to label node s9t1y1s2c1s6e5d-np-j22hl8-cg7vv-84h99 egress-assignable: failed to apply k8s.ovn.org/egress-assignable label to node "s9t1y1s2c1s6e5d-np-j22hl8-cg7vv-84h99": nodes "s9t1y1s2c1s6e5d-np-j22hl8-cg7vv-84h99" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized
{
errs: [
<*fmt.wrapError | 0x237cb4952040>{
msg: "VerifyCloudNetworkConfigEgressIP(10.0.0.7) failed: failed to label node s9t1y1s2c1s6e5d-np-j22hl8-cg7vv-84h99 egress-assignable: failed to apply k8s.ovn.org/egress-assignable label to node \"s9t1y1s2c1s6e5d-np-j22hl8-cg7vv-84h99\": nodes \"s9t1y1s2c1s6e5d-np-j22hl8-cg7vv-84h99\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
err: <*fmt.wrapError | 0x237cb4952020>{
msg: "failed to label node s9t1y1s2c1s6e5d-np-j22hl8-cg7vv-84h99 egress-assignable: failed to apply k8s.ovn.org/egress-assignable label to node \"s9t1y1s2c1s6e5d-np-j22hl8-cg7vv-84h99\": nodes \"s9t1y1s2c1s6e5d-np-j22hl8-cg7vv-84h99\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
err: <*fmt.wrapError | 0x237cb4952000>{
msg: "failed to apply k8s.ovn.org/egress-assignable label to node \"s9t1y1s2c1s6e5d-np-j22hl8-cg7vv-84h99\": nodes \"s9t1y1s2c1s6e5d-np-j22hl8-cg7vv-84h99\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
err: <*errors.StatusError | 0x237cb49b60a0>{
ErrStatus: {
TypeMeta: {Kind: "", APIVersion: ""},
ListMeta: {
SelfLink: "",
ResourceVersion: "",
Continue: "",
RemainingItemCount: nil,
ShardInfo: nil,
},
Status: "Failure",
Message: "nodes \"s9t1y1s2c1s6e5d-np-j22hl8-cg7vv-84h99\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
Reason: "Forbidden",
Details: {
Name: "s9t1y1s2c1s6e5d-np-j22hl8-cg7vv-84h99",
Group: "",
Kind: "nodes",
UID: "",
Causes: nil,
RetryAfterSeconds: 0,
},
Code: 403,
},
},
},
},
},
],
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:289]: failed to verify cluster operators on cluster "cluster-candidate-5-0-j22hl8"
Unexpected error:
<*errors.joinError | 0x237cb4aa4138>:
VerifyCloudNetworkConfigEgressIP(10.0.0.7) failed: failed to label node s9t1y1s2c1s6e5d-np-j22hl8-cg7vv-84h99 egress-assignable: failed to apply k8s.ovn.org/egress-assignable label to node "s9t1y1s2c1s6e5d-np-j22hl8-cg7vv-84h99": nodes "s9t1y1s2c1s6e5d-np-j22hl8-cg7vv-84h99" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized
{
errs: [
<*fmt.wrapError | 0x237cb4952040>{
msg: "VerifyCloudNetworkConfigEgressIP(10.0.0.7) failed: failed to label node s9t1y1s2c1s6e5d-np-j22hl8-cg7vv-84h99 egress-assignable: failed to apply k8s.ovn.org/egress-assignable label to node \"s9t1y1s2c1s6e5d-np-j22hl8-cg7vv-84h99\": nodes \"s9t1y1s2c1s6e5d-np-j22hl8-cg7vv-84h99\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
err: <*fmt.wrapError | 0x237cb4952020>{
msg: "failed to label node s9t1y1s2c1s6e5d-np-j22hl8-cg7vv-84h99 egress-assignable: failed to apply k8s.ovn.org/egress-assignable label to node \"s9t1y1s2c1s6e5d-np-j22hl8-cg7vv-84h99\": nodes \"s9t1y1s2c1s6e5d-np-j22hl8-cg7vv-84h99\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
err: <*fmt.wrapError | 0x237cb4952000>{
msg: "failed to apply k8s.ovn.org/egress-assignable label to node \"s9t1y1s2c1s6e5d-np-j22hl8-cg7vv-84h99\": nodes \"s9t1y1s2c1s6e5d-np-j22hl8-cg7vv-84h99\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
err: <*errors.StatusError | 0x237cb49b60a0>{
ErrStatus: {
TypeMeta: {Kind: "", APIVersion: ""},
ListMeta: {
SelfLink: "",
ResourceVersion: "",
Continue: "",
RemainingItemCount: nil,
ShardInfo: nil,
},
Status: "Failure",
Message: "nodes \"s9t1y1s2c1s6e5d-np-j22hl8-cg7vv-84h99\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
Reason: "Forbidden",
Details: {
Name: "s9t1y1s2c1s6e5d-np-j22hl8-cg7vv-84h99",
Group: "",
Kind: "nodes",
UID: "",
Causes: nil,
RetryAfterSeconds: 0,
},
Code: 403,
},
},
},
},
},
],
}
occurredContributing tests (3)
Affected runs (10) | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
ERROR CODE: DeploymentFailed; detail code AzureKeyVaultKmsValidateIdentityPermissionCustomerError; detail message The identity does not have keys encrypt/decrypt permission on key...full failure pattern: ERROR CODE: DeploymentFailed; detail code AzureKeyVaultKmsValidateIdentityPermissionCustomerError; detail message The identity does not have keys encrypt/decrypt permission on key vault "<url>"; provider Microsoft.ContainerService Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 3 day(s) | provision | 8 | 1.72%8 of 464 job runs affected | IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 3 day(s) | Show trend detailsOct 21: 0 · Oct 22: 0 · Oct 23: 0 · Oct 24: 0 · Oct 25: 0 · Oct 26: 2 · Oct 27: 4 · Oct 28: 2 | none | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (3)time=2026-10-28T16:43:28.511Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster description="Step cluster\n Kind: ARM\n Template: templates/svc-cluster.bicep\n Parameters: configurations/svc-cluster.tmpl.bicepparam"
time=2026-10-28T16:43:31.273Z level=DEBUG+2 msg="Starting ARM deployment" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster
time=2026-10-28T16:43:32.448Z level=DEBUG+3 msg="Deployment started" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster deployment=f309630496c74d69129833a43742f2e9eec7ceb9d4411ea07b979bc73b89fd87 portal=https://ms.portal.azure.com/#view/Microsoft_Azure_Resources/DeploymentDetails.MenuView/~/overview/id/%2Fsubscriptions%2FXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX%2FresourceGroups%2Fhcp-underlay-ci01-j2862080-svc%2Fproviders%2FMicrosoft.Resources%2Fdeployments%2Ff309630496c74d69129833a43742f2e9eec7ceb9d4411ea07b979bc73b89fd87
time=2026-10-28T16:45:32.811Z level=INFO msg="Retrying step" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster step=cluster error="failed to run ARM step: failed to poll deployment: failed to wait for deployment completion: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j2862080-svc/providers/Microsoft.Resources/deployments/f309630496c74d69129833a43742f2e9eec7ceb9d4411ea07b979bc73b89fd87/operationStatuses/08584084010736719502\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n \"status\": \"Failed\",\n \"error\": {\n \"code\": \"DeploymentFailed\",\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n \"details\": [\n {\n \"code\": \"Conflict\",\n \"message\": \"{\\r\\n \\\"status\\\": \\\"Failed\\\",\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j2862080-svc/providers/Microsoft.Resources/deployments/cluster-rybldmoz73lrm\\\",\\r\\n \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j2862080-svc/providers/Microsoft.ContainerService/managedClusters/ci01-j2862080-svc\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"AzureKeyVaultKmsValidateIdentityPermissionCustomerError\\\",\\r\\n \\\"message\\\": \\\"The identity does not have keys encrypt/decrypt permission on key vault \\\\\\\"https://ah-ci01-se-j2862080-1.vault.azure.net\\\\\\\"\\\"\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n }\\r\\n}\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n"
time=2026-10-28T16:47:32.817Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster description="Step cluster\n Kind: ARM\n Template: templates/svc-cluster.bicep\n Parameters: configurations/svc-cluster.tmpl.bicepparam"
time=2026-10-28T16:47:34.143Z level=DEBUG+2 msg="Starting ARM deployment" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster
time=2026-10-28T16:47:35.068Z level=DEBUG+3 msg="Deployment started" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster deployment=8fa925f61f96a8233f154e3da5c65d168518bb692c424a9f42720dad4bf60006 portal=https://ms.portal.azure.com/#view/Microsoft_Azure_Resources/DeploymentDetails.MenuView/~/overview/id/%2Fsubscriptions%2FXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX%2FresourceGroups%2Fhcp-underlay-ci01-j2862080-svc%2Fproviders%2FMicrosoft.Resources%2Fdeployments%2F8fa925f61f96a8233f154e3da5c65d168518bb692c424a9f42720dad4bf60006
time=2026-10-28T16:49:05.326Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster err="failed to run ARM step: failed to poll deployment: failed to wait for deployment completion: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j2862080-svc/providers/Microsoft.Resources/deployments/8fa925f61f96a8233f154e3da5c65d168518bb692c424a9f42720dad4bf60006/operationStatuses/08584084008309075646\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n \"status\": \"Failed\",\n \"error\": {\n \"code\": \"DeploymentFailed\",\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n \"details\": [\n {\n \"code\": \"Conflict\",\n \"message\": \"{\\r\\n \\\"status\\\": \\\"Failed\\\",\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"message\\\": \\\"Failed to obtain the resource body, because the resource is currently in a 'Failed' provisioning state. If this resource has not been modified by a prior step in this deployment, you may need to address this before retrying the deployment.\\\"\\r\\n }\\r\\n}\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n"time=2026-10-28T15:32:48.082Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster description="Step cluster\n Kind: ARM\n Template: templates/svc-cluster.bicep\n Parameters: configurations/svc-cluster.tmpl.bicepparam"
time=2026-10-28T15:32:50.394Z level=DEBUG+2 msg="Starting ARM deployment" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster
time=2026-10-28T15:32:51.656Z level=DEBUG+3 msg="Deployment started" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster deployment=51bb59b51713025e67da1bbb1f81509d9f0e687a1b531b2b697347d81ab333d6 portal=https://ms.portal.azure.com/#view/Microsoft_Azure_Resources/DeploymentDetails.MenuView/~/overview/id/%2Fsubscriptions%2FXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX%2FresourceGroups%2Fhcp-underlay-ci01-j1862528-svc%2Fproviders%2FMicrosoft.Resources%2Fdeployments%2F51bb59b51713025e67da1bbb1f81509d9f0e687a1b531b2b697347d81ab333d6
time=2026-10-28T15:35:22.185Z level=INFO msg="Retrying step" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster step=cluster error="failed to run ARM step: failed to poll deployment: failed to wait for deployment completion: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j1862528-svc/providers/Microsoft.Resources/deployments/51bb59b51713025e67da1bbb1f81509d9f0e687a1b531b2b697347d81ab333d6/operationStatuses/08584084053144753487\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n \"status\": \"Failed\",\n \"error\": {\n \"code\": \"DeploymentFailed\",\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n \"details\": [\n {\n \"code\": \"Conflict\",\n \"message\": \"{\\r\\n \\\"status\\\": \\\"Failed\\\",\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j1862528-svc/providers/Microsoft.Resources/deployments/cluster-5xxgmkpcjnpl2\\\",\\r\\n \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j1862528-svc/providers/Microsoft.ContainerService/managedClusters/ci01-j1862528-svc\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"AzureKeyVaultKmsValidateIdentityPermissionCustomerError\\\",\\r\\n \\\"message\\\": \\\"The identity does not have keys encrypt/decrypt permission on key vault \\\\\\\"https://ah-ci01-se-j1862528-1.vault.azure.net\\\\\\\"\\\"\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n }\\r\\n}\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n"
time=2026-10-28T15:37:22.187Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster description="Step cluster\n Kind: ARM\n Template: templates/svc-cluster.bicep\n Parameters: configurations/svc-cluster.tmpl.bicepparam"
time=2026-10-28T15:37:23.185Z level=DEBUG+2 msg="Starting ARM deployment" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster
time=2026-10-28T15:37:24.435Z level=DEBUG+3 msg="Deployment started" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster deployment=f330c62a337aea146d3097645390c7352ebc193053cfec3eadc02b161a2aba5f portal=https://ms.portal.azure.com/#view/Microsoft_Azure_Resources/DeploymentDetails.MenuView/~/overview/id/%2Fsubscriptions%2FXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX%2FresourceGroups%2Fhcp-underlay-ci01-j1862528-svc%2Fproviders%2FMicrosoft.Resources%2Fdeployments%2Ff330c62a337aea146d3097645390c7352ebc193053cfec3eadc02b161a2aba5f
time=2026-10-28T15:38:54.741Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster err="failed to run ARM step: failed to poll deployment: failed to wait for deployment completion: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j1862528-svc/providers/Microsoft.Resources/deployments/f330c62a337aea146d3097645390c7352ebc193053cfec3eadc02b161a2aba5f/operationStatuses/08584084050417029665\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n \"status\": \"Failed\",\n \"error\": {\n \"code\": \"DeploymentFailed\",\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n \"details\": [\n {\n \"code\": \"Conflict\",\n \"message\": \"{\\r\\n \\\"status\\\": \\\"Failed\\\",\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"message\\\": \\\"Failed to obtain the resource body, because the resource is currently in a 'Failed' provisioning state. If this resource has not been modified by a prior step in this deployment, you may need to address this before retrying the deployment.\\\"\\r\\n }\\r\\n}\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n"time=2026-10-27T21:46:01.135Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster description="Step cluster\n Kind: ARM\n Template: templates/svc-cluster.bicep\n Parameters: configurations/svc-cluster.tmpl.bicepparam"
time=2026-10-27T21:46:03.401Z level=DEBUG+2 msg="Starting ARM deployment" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster
time=2026-10-27T21:46:05.062Z level=DEBUG+3 msg="Deployment started" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster deployment=ec85056f79198d17f9a4a06a9d62ec213e8ff2f13643b300963fd031a60589b0 portal=https://ms.portal.azure.com/#view/Microsoft_Azure_Resources/DeploymentDetails.MenuView/~/overview/id/%2Fsubscriptions%2FXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX%2FresourceGroups%2Fhcp-underlay-ci01-j0909440-svc%2Fproviders%2FMicrosoft.Resources%2Fdeployments%2Fec85056f79198d17f9a4a06a9d62ec213e8ff2f13643b300963fd031a60589b0
time=2026-10-27T21:48:05.587Z level=INFO msg="Retrying step" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster step=cluster error="failed to run ARM step: failed to poll deployment: failed to wait for deployment completion: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0909440-svc/providers/Microsoft.Resources/deployments/ec85056f79198d17f9a4a06a9d62ec213e8ff2f13643b300963fd031a60589b0/operationStatuses/08584084693213550197\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n \"status\": \"Failed\",\n \"error\": {\n \"code\": \"DeploymentFailed\",\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n \"details\": [\n {\n \"code\": \"Conflict\",\n \"message\": \"{\\r\\n \\\"status\\\": \\\"Failed\\\",\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j0909440-svc/providers/Microsoft.Resources/deployments/cluster-r5scxlmdedogk\\\",\\r\\n \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j0909440-svc/providers/Microsoft.ContainerService/managedClusters/ci01-j0909440-svc\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"AzureKeyVaultKmsValidateIdentityPermissionCustomerError\\\",\\r\\n \\\"message\\\": \\\"The identity does not have keys encrypt/decrypt permission on key vault \\\\\\\"https://ah-ci01-se-j0909440-1.vault.azure.net\\\\\\\"\\\"\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n }\\r\\n}\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n"
time=2026-10-27T21:50:05.589Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster description="Step cluster\n Kind: ARM\n Template: templates/svc-cluster.bicep\n Parameters: configurations/svc-cluster.tmpl.bicepparam"
time=2026-10-27T21:50:06.576Z level=DEBUG+2 msg="Starting ARM deployment" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster
time=2026-10-27T21:50:08.087Z level=DEBUG+3 msg="Deployment started" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster deployment=cd575f368230e7d22208e5f91e10168040367ee73eee75ae5c003a42111d906b portal=https://ms.portal.azure.com/#view/Microsoft_Azure_Resources/DeploymentDetails.MenuView/~/overview/id/%2Fsubscriptions%2FXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX%2FresourceGroups%2Fhcp-underlay-ci01-j0909440-svc%2Fproviders%2FMicrosoft.Resources%2Fdeployments%2Fcd575f368230e7d22208e5f91e10168040367ee73eee75ae5c003a42111d906b
time=2026-10-27T21:51:38.615Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster err="failed to run ARM step: failed to poll deployment: failed to wait for deployment completion: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0909440-svc/providers/Microsoft.Resources/deployments/cd575f368230e7d22208e5f91e10168040367ee73eee75ae5c003a42111d906b/operationStatuses/08584084690782406556\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n \"status\": \"Failed\",\n \"error\": {\n \"code\": \"DeploymentFailed\",\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n \"details\": [\n {\n \"code\": \"Conflict\",\n \"message\": \"{\\r\\n \\\"status\\\": \\\"Failed\\\",\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"message\\\": \\\"Failed to obtain the resource body, because the resource is currently in a 'Failed' provisioning state. If this resource has not been modified by a prior step in this deployment, you may need to address this before retrying the deployment.\\\"\\r\\n }\\r\\n}\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n"Contributing tests (1)
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
failed to run istio upgrade step: failed to complete canary: complete-canary: ARM operation timed out or failed: context deadline exceededfull failure pattern: failed to run istio upgrade step: failed to complete canary: complete-canary: ARM operation timed out or failed: context deadline exceeded Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 4 day(s) | provision | 7 | 1.51%7 of 464 job runs affected | IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 4 day(s) | Show trend detailsOct 21: 0 · Oct 22: 1 · Oct 23: 4 · Oct 24: 1 · Oct 25: 1 · Oct 26: 0 · Oct 27: 0 · Oct 28: 0 | none | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (3)time=2026-10-25T11:53:02.786Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=istio-upgrade description="Step istio-upgrade\n Kind: IstioUpgrade\n AKSCluster: {{ svc.aks.name }}\n"
time=2026-10-25T11:53:04.190Z level=INFO msg="Istio upgrade -- cluster state" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=istio-upgrade cluster=ci01-j8850688-svc versions=asm-1-30 logger=istio-upgrade k8sVersion=1.35 provisioningState=Succeeded installedRevisions=[asm-1-29] availableUpgrades=[asm-1-30] upgradeInProgress=false
time=2026-10-25T11:53:04.190Z level=INFO msg="Upgrading to svc.istio.versions target" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=istio-upgrade cluster=ci01-j8850688-svc versions=asm-1-30 logger=istio-upgrade from=asm-1-29 to=asm-1-30 k8sVersion=1.35
time=2026-10-25T11:53:05.198Z level=INFO msg="Istio upgrade -- mesh state" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=istio-upgrade cluster=ci01-j8850688-svc versions=asm-1-30 logger=istio-upgrade namespaces=0 controlPlanes=asm-1-29(2/2) ingressGateways=1 webhooks=1 configMaps=2
time=2026-10-25T11:53:05.198Z level=INFO msg="Starting canary -- installing target alongside current" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=istio-upgrade cluster=ci01-j8850688-svc versions=asm-1-30 logger=istio-upgrade
time=2026-10-25T11:53:05.198Z level=INFO msg="Starting ARM operation" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=istio-upgrade stepID=Microsoft.Azure.ARO.HCP.Service.Infra/service/istio-upgrade operation=start-canary cluster=ci01-j8850688-svc
time=2026-10-25T11:56:09.852Z level=INFO msg="Cluster provisioning state" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=istio-upgrade stepID=Microsoft.Azure.ARO.HCP.Service.Infra/service/istio-upgrade state=Succeeded
time=2026-10-25T11:56:09.852Z level=INFO msg="ARM operation complete" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=istio-upgrade stepID=Microsoft.Azure.ARO.HCP.Service.Infra/service/istio-upgrade operation=start-canary elapsed=3m5s
time=2026-10-25T11:56:09.852Z level=INFO msg="Step 1/9: Ensuring MISE ext-authz ConfigMap" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=istio-upgrade cluster=ci01-j8850688-svc versions=asm-1-30 logger=istio-upgrade
time=2026-10-25T11:56:10.416Z level=INFO msg="ConfigMap created" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=istio-upgrade cluster=ci01-j8850688-svc versions=asm-1-30 logger=istio-upgrade/revision-configmap name=istio-shared-configmap-asm-1-30
time=2026-10-25T11:56:10.416Z level=INFO msg="Step 2/9: Checking tag-based namespace safety" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=istio-upgrade cluster=ci01-j8850688-svc versions=asm-1-30 logger=istio-upgrade
time=2026-10-25T11:56:10.416Z level=INFO msg="Step 3/9: Verifying control plane health and flipping tag webhook" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=istio-upgrade cluster=ci01-j8850688-svc versions=asm-1-30 logger=istio-upgrade
time=2026-10-25T11:56:11.207Z level=INFO msg="Created revision tag webhook" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=istio-upgrade cluster=ci01-j8850688-svc versions=asm-1-30 logger=istio-upgrade/revision-tag webhook=istio-revision-tag-prod-stable-aks-istio-system revision=asm-1-30
time=2026-10-25T11:56:11.207Z level=INFO msg="Step 4/9: Ensuring ingress gateway annotations" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=istio-upgrade cluster=ci01-j8850688-svc versions=asm-1-30 logger=istio-upgrade
time=2026-10-25T11:56:11.237Z level=INFO msg="Applying annotations" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=istio-upgrade cluster=ci01-j8850688-svc versions=asm-1-30 logger=istio-upgrade/ingress-annotations service=aks-istio-ingressgateway-external
time=2026-10-25T11:56:11.450Z level=INFO msg="Step 5/9: Migrating workloads to target revision" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=istio-upgrade cluster=ci01-j8850688-svc versions=asm-1-30 logger=istio-upgrade
time=2026-10-25T11:56:11.510Z level=INFO msg="Step 6/9: Running health check" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=istio-upgrade cluster=ci01-j8850688-svc versions=asm-1-30 logger=istio-upgrade
time=2026-10-25T11:56:11.971Z level=INFO msg="Health check passed -- checking for orphaned workloads before completing canary" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=istio-upgrade cluster=ci01-j8850688-svc versions=asm-1-30 logger=istio-upgrade
time=2026-10-25T11:56:11.971Z level=INFO msg="Step 7/9: Running orphan guard" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=istio-upgrade cluster=ci01-j8850688-svc versions=asm-1-30 logger=istio-upgrade
time=2026-10-25T11:56:11.971Z level=INFO msg="No orphaned workloads -- completing canary" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=istio-upgrade cluster=ci01-j8850688-svc versions=asm-1-30 logger=istio-upgrade
time=2026-10-25T11:56:11.971Z level=INFO msg="Step 8/9: Completing canary via ARM (point of no return)" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=istio-upgrade cluster=ci01-j8850688-svc versions=asm-1-30 logger=istio-upgrade
time=2026-10-25T11:56:11.971Z level=INFO msg="Starting ARM operation" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=istio-upgrade stepID=Microsoft.Azure.ARO.HCP.Service.Infra/service/istio-upgrade operation=complete-canary cluster=ci01-j8850688-svc
time=2026-10-25T12:11:14.972Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=istio-upgrade err="failed to run istio upgrade step: failed to complete canary: complete-canary: ARM operation timed out or failed: context deadline exceeded"time=2026-10-24T03:19:41.940Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=istio-upgrade description="Step istio-upgrade\n Kind: IstioUpgrade\n AKSCluster: {{ svc.aks.name }}\n"
time=2026-10-24T03:19:44.124Z level=INFO msg="Istio upgrade -- cluster state" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=istio-upgrade cluster=ci01-j0729472-svc versions=asm-1-30 logger=istio-upgrade k8sVersion=1.35 provisioningState=Succeeded installedRevisions=[asm-1-29] availableUpgrades=[asm-1-30] upgradeInProgress=false
time=2026-10-24T03:19:44.124Z level=INFO msg="Upgrading to svc.istio.versions target" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=istio-upgrade cluster=ci01-j0729472-svc versions=asm-1-30 logger=istio-upgrade from=asm-1-29 to=asm-1-30 k8sVersion=1.35
time=2026-10-24T03:19:45.250Z level=INFO msg="Istio upgrade -- mesh state" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=istio-upgrade cluster=ci01-j0729472-svc versions=asm-1-30 logger=istio-upgrade namespaces=0 controlPlanes=asm-1-29(2/2) ingressGateways=1 webhooks=1 configMaps=2
time=2026-10-24T03:19:45.250Z level=INFO msg="Starting canary -- installing target alongside current" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=istio-upgrade cluster=ci01-j0729472-svc versions=asm-1-30 logger=istio-upgrade
time=2026-10-24T03:19:45.250Z level=INFO msg="Starting ARM operation" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=istio-upgrade stepID=Microsoft.Azure.ARO.HCP.Service.Infra/service/istio-upgrade operation=start-canary cluster=ci01-j0729472-svc
time=2026-10-24T03:21:51.429Z level=INFO msg="Cluster provisioning state" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=istio-upgrade stepID=Microsoft.Azure.ARO.HCP.Service.Infra/service/istio-upgrade state=Succeeded
time=2026-10-24T03:21:51.429Z level=INFO msg="ARM operation complete" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=istio-upgrade stepID=Microsoft.Azure.ARO.HCP.Service.Infra/service/istio-upgrade operation=start-canary elapsed=2m6s
time=2026-10-24T03:21:51.429Z level=INFO msg="Step 1/9: Ensuring MISE ext-authz ConfigMap" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=istio-upgrade cluster=ci01-j0729472-svc versions=asm-1-30 logger=istio-upgrade
time=2026-10-24T03:21:51.773Z level=INFO msg="ConfigMap created" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=istio-upgrade cluster=ci01-j0729472-svc versions=asm-1-30 logger=istio-upgrade/revision-configmap name=istio-shared-configmap-asm-1-30
time=2026-10-24T03:21:51.773Z level=INFO msg="Step 2/9: Checking tag-based namespace safety" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=istio-upgrade cluster=ci01-j0729472-svc versions=asm-1-30 logger=istio-upgrade
time=2026-10-24T03:21:51.773Z level=INFO msg="Step 3/9: Verifying control plane health and flipping tag webhook" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=istio-upgrade cluster=ci01-j0729472-svc versions=asm-1-30 logger=istio-upgrade
time=2026-10-24T03:21:52.375Z level=INFO msg="Created revision tag webhook" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=istio-upgrade cluster=ci01-j0729472-svc versions=asm-1-30 logger=istio-upgrade/revision-tag webhook=istio-revision-tag-prod-stable-aks-istio-system revision=asm-1-30
time=2026-10-24T03:21:52.375Z level=INFO msg="Step 4/9: Ensuring ingress gateway annotations" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=istio-upgrade cluster=ci01-j0729472-svc versions=asm-1-30 logger=istio-upgrade
time=2026-10-24T03:21:52.438Z level=INFO msg="Applying annotations" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=istio-upgrade cluster=ci01-j0729472-svc versions=asm-1-30 logger=istio-upgrade/ingress-annotations service=aks-istio-ingressgateway-external
time=2026-10-24T03:21:52.730Z level=INFO msg="Step 5/9: Migrating workloads to target revision" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=istio-upgrade cluster=ci01-j0729472-svc versions=asm-1-30 logger=istio-upgrade
time=2026-10-24T03:21:52.857Z level=INFO msg="Step 6/9: Running health check" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=istio-upgrade cluster=ci01-j0729472-svc versions=asm-1-30 logger=istio-upgrade
time=2026-10-24T03:21:53.744Z level=INFO msg="Health check passed -- checking for orphaned workloads before completing canary" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=istio-upgrade cluster=ci01-j0729472-svc versions=asm-1-30 logger=istio-upgrade
time=2026-10-24T03:21:53.744Z level=INFO msg="Step 7/9: Running orphan guard" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=istio-upgrade cluster=ci01-j0729472-svc versions=asm-1-30 logger=istio-upgrade
time=2026-10-24T03:21:53.744Z level=INFO msg="No orphaned workloads -- completing canary" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=istio-upgrade cluster=ci01-j0729472-svc versions=asm-1-30 logger=istio-upgrade
time=2026-10-24T03:21:53.744Z level=INFO msg="Step 8/9: Completing canary via ARM (point of no return)" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=istio-upgrade cluster=ci01-j0729472-svc versions=asm-1-30 logger=istio-upgrade
time=2026-10-24T03:21:53.744Z level=INFO msg="Starting ARM operation" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=istio-upgrade stepID=Microsoft.Azure.ARO.HCP.Service.Infra/service/istio-upgrade operation=complete-canary cluster=ci01-j0729472-svc
time=2026-10-24T03:36:56.654Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=istio-upgrade err="failed to run istio upgrade step: failed to complete canary: complete-canary: ARM operation timed out or failed: context deadline exceeded"time=2026-10-24T00:11:13.169Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=istio-upgrade description="Step istio-upgrade\n Kind: IstioUpgrade\n AKSCluster: {{ svc.aks.name }}\n"
time=2026-10-24T00:11:14.722Z level=INFO msg="Istio upgrade -- cluster state" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=istio-upgrade cluster=ci01-j2212352-svc versions=asm-1-30 logger=istio-upgrade k8sVersion=1.35 provisioningState=Succeeded installedRevisions=[asm-1-29] availableUpgrades=[asm-1-30] upgradeInProgress=false
time=2026-10-24T00:11:14.722Z level=INFO msg="Upgrading to svc.istio.versions target" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=istio-upgrade cluster=ci01-j2212352-svc versions=asm-1-30 logger=istio-upgrade from=asm-1-29 to=asm-1-30 k8sVersion=1.35
time=2026-10-24T00:11:15.608Z level=INFO msg="Istio upgrade -- mesh state" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=istio-upgrade cluster=ci01-j2212352-svc versions=asm-1-30 logger=istio-upgrade namespaces=0 controlPlanes=asm-1-29(2/2) ingressGateways=1 webhooks=1 configMaps=2
time=2026-10-24T00:11:15.608Z level=INFO msg="Starting canary -- installing target alongside current" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=istio-upgrade cluster=ci01-j2212352-svc versions=asm-1-30 logger=istio-upgrade
time=2026-10-24T00:11:15.609Z level=INFO msg="Starting ARM operation" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=istio-upgrade stepID=Microsoft.Azure.ARO.HCP.Service.Infra/service/istio-upgrade operation=start-canary cluster=ci01-j2212352-svc
time=2026-10-24T00:13:20.091Z level=INFO msg="Cluster provisioning state" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=istio-upgrade stepID=Microsoft.Azure.ARO.HCP.Service.Infra/service/istio-upgrade state=Succeeded
time=2026-10-24T00:13:20.091Z level=INFO msg="ARM operation complete" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=istio-upgrade stepID=Microsoft.Azure.ARO.HCP.Service.Infra/service/istio-upgrade operation=start-canary elapsed=2m4s
time=2026-10-24T00:13:20.091Z level=INFO msg="Step 1/9: Ensuring MISE ext-authz ConfigMap" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=istio-upgrade cluster=ci01-j2212352-svc versions=asm-1-30 logger=istio-upgrade
time=2026-10-24T00:13:20.407Z level=INFO msg="ConfigMap created" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=istio-upgrade cluster=ci01-j2212352-svc versions=asm-1-30 logger=istio-upgrade/revision-configmap name=istio-shared-configmap-asm-1-30
time=2026-10-24T00:13:20.407Z level=INFO msg="Step 2/9: Checking tag-based namespace safety" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=istio-upgrade cluster=ci01-j2212352-svc versions=asm-1-30 logger=istio-upgrade
time=2026-10-24T00:13:20.407Z level=INFO msg="Step 3/9: Verifying control plane health and flipping tag webhook" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=istio-upgrade cluster=ci01-j2212352-svc versions=asm-1-30 logger=istio-upgrade
time=2026-10-24T00:13:20.862Z level=INFO msg="Created revision tag webhook" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=istio-upgrade cluster=ci01-j2212352-svc versions=asm-1-30 logger=istio-upgrade/revision-tag webhook=istio-revision-tag-prod-stable-aks-istio-system revision=asm-1-30
time=2026-10-24T00:13:20.862Z level=INFO msg="Step 4/9: Ensuring ingress gateway annotations" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=istio-upgrade cluster=ci01-j2212352-svc versions=asm-1-30 logger=istio-upgrade
time=2026-10-24T00:13:20.894Z level=INFO msg="Applying annotations" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=istio-upgrade cluster=ci01-j2212352-svc versions=asm-1-30 logger=istio-upgrade/ingress-annotations service=aks-istio-ingressgateway-external
time=2026-10-24T00:13:21.097Z level=INFO msg="Step 5/9: Migrating workloads to target revision" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=istio-upgrade cluster=ci01-j2212352-svc versions=asm-1-30 logger=istio-upgrade
time=2026-10-24T00:13:21.168Z level=INFO msg="Step 6/9: Running health check" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=istio-upgrade cluster=ci01-j2212352-svc versions=asm-1-30 logger=istio-upgrade
time=2026-10-24T00:13:21.740Z level=INFO msg="Health check passed -- checking for orphaned workloads before completing canary" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=istio-upgrade cluster=ci01-j2212352-svc versions=asm-1-30 logger=istio-upgrade
time=2026-10-24T00:13:21.740Z level=INFO msg="Step 7/9: Running orphan guard" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=istio-upgrade cluster=ci01-j2212352-svc versions=asm-1-30 logger=istio-upgrade
time=2026-10-24T00:13:21.740Z level=INFO msg="No orphaned workloads -- completing canary" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=istio-upgrade cluster=ci01-j2212352-svc versions=asm-1-30 logger=istio-upgrade
time=2026-10-24T00:13:21.740Z level=INFO msg="Step 8/9: Completing canary via ARM (point of no return)" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=istio-upgrade cluster=ci01-j2212352-svc versions=asm-1-30 logger=istio-upgrade
time=2026-10-24T00:13:21.740Z level=INFO msg="Starting ARM operation" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=istio-upgrade stepID=Microsoft.Azure.ARO.HCP.Service.Infra/service/istio-upgrade operation=complete-canary cluster=ci01-j2212352-svc
time=2026-10-24T00:28:24.267Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=istio-upgrade err="failed to run istio upgrade step: failed to complete canary: complete-canary: ARM operation timed out or failed: context deadline exceeded"Contributing tests (1)
Affected runs (7) | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
ERROR CODE: DeadlineExceeded; detail code SubscriptionRequestsThrottled; detail message Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor '<u...full failure pattern: ERROR CODE: DeadlineExceeded; detail code SubscriptionRequestsThrottled; detail message Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor '<uuid>' exceeded. Please try again after '<seconds>' seconds after additional tokens are available.; provider Microsoft.RedHatOpenShift Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 4 day(s) | e2e | 7 | 1.51%7 of 464 job runs affected | IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 4 day(s) | Show trend detailsOct 21: 1 · Oct 22: 0 · Oct 23: 2 · Oct 24: 0 · Oct 25: 0 · Oct 26: 0 · Oct 27: 2 · Oct 28: 3 | INT | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (3)fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_cni_cilium.go:182]: failed to verify cilium is running and nodes are Ready for cluster "cilium-cl"
Unexpected error:
<*errors.joinError | 0x740ee223cf8>:
failed to create NodePool cilium-np: failed waiting for nodepool="cilium-np" for cluster "cilium-cl" in resourcegroup="cni-cilium-dmmfjzfldpm8" to finish creating: GET https://rp.j8717568.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/e51bb1e4-4996-454f-a81d-9354613a75ee
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/e51bb1e4-4996-454f-a81d-9354613a75ee",
"name": "e51bb1e4-4996-454f-a81d-9354613a75ee",
"status": "Failed",
"startTime": "2026-10-28T19:36:46.639467207Z",
"endTime": "2026-10-28T19:55:53.152282953Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine cilium-cl-cilium-np-8z842-x96hm: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine cilium-cl-cilium-np-8z842-tlsx7: Failed: interfaces failed to create or update. err: failed to get existing resource cni-cilium-dmmfjzfldpm8--managed/cilium-cl-cilium-np-8z842-tlsx7-nic (service: interfaces): GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/cni-cilium-dmmfjzfldpm8--managed/providers/Microsoft.Network/networkInterfaces/cilium-cl-cilium-np-8z842-tlsx7-nic\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"SubscriptionRequestsThrottled\",\n \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor '1d2f4929-515e-4e8d-8580-836e819f8d50' exceeded. Please try again after '1' seconds after additional tokens are available. Refer to https://aka... message truncated\n"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://rp.j8717568.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/e51bb1e4-4996-454f-a81d-9354613a75ee
correlation ID: 4963ae88-5109-4671-873f-f28a9deccdac
client request ID: 38a19ead-d2a2-4130-9e04-1364166950d2
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_cni_cilium.go:182]: failed to verify cilium is running and nodes are Ready for cluster "cilium-cl"
Unexpected error:
<*errors.joinError | 0x740ee223cf8>:
failed to create NodePool cilium-np: failed waiting for nodepool="cilium-np" for cluster "cilium-cl" in resourcegroup="cni-cilium-dmmfjzfldpm8" to finish creating: GET https://rp.j8717568.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/e51bb1e4-4996-454f-a81d-9354613a75ee
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/e51bb1e4-4996-454f-a81d-9354613a75ee",
"name": "e51bb1e4-4996-454f-a81d-9354613a75ee",
"status": "Failed",
"startTime": "2026-10-28T19:36:46.639467207Z",
"endTime": "2026-10-28T19:55:53.152282953Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine cilium-cl-cilium-np-8z842-x96hm: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine cilium-cl-cilium-np-8z842-tlsx7: Failed: interfaces failed to create or update. err: failed to get existing resource cni-cilium-dmmfjzfldpm8--managed/cilium-cl-cilium-np-8z842-tlsx7-nic (service: interfaces): GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/cni-cilium-dmmfjzfldpm8--managed/providers/Microsoft.Network/networkInterfaces/cilium-cl-cilium-np-8z842-tlsx7-nic\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"SubscriptionRequestsThrottled\",\n \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor '1d2f4929-515e-4e8d-8580-836e819f8d50' exceeded. Please try again after '1' seconds after additional tokens are available. Refer to https://aka... message truncated\n"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://rp.j8717568.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/e51bb1e4-4996-454f-a81d-9354613a75ee
correlation ID: 4963ae88-5109-4671-873f-f28a9deccdac
client request ID: 38a19ead-d2a2-4130-9e04-1364166950d2
occurredfail [github.com/Azure/ARO-HCP/test/e2e/cluster_fips_mode.go:103]: failed to create node pool "np-1" for fips-enabled cluster "fips-enabled-cluster"
Unexpected error:
<*fmt.wrapError | 0x335ced2f1b40>:
failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "fips-enabled-cluster" in resourcegroup="fips-enabled-kfwkzrbn6vbm" to finish creating: GET https://rp.j4206336.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/77628145-9ee8-4d12-9dd2-5dced29b5ecb
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/77628145-9ee8-4d12-9dd2-5dced29b5ecb",
"name": "77628145-9ee8-4d12-9dd2-5dced29b5ecb",
"status": "Failed",
"startTime": "2026-10-28T16:28:12.935779531Z",
"endTime": "2026-10-28T16:47:20.605052936Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine j6k4t1t6z4j2k5z-np-1-5nm58-fsnz7: Failed: virtualmachine failed to create or update. err: failed to get existing resource fips-enabled-kfwkzrbn6vbm--managed/j6k4t1t6z4j2k5z-np-1-5nm58-fsnz7 (service: virtualmachine): GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/fips-enabled-kfwkzrbn6vbm--managed/providers/Microsoft.Compute/virtualMachines/j6k4t1t6z4j2k5z-np-1-5nm58-fsnz7\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"SubscriptionRequestsThrottled\",\n \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'b117cf0f-0276-4486-9691-4f00f5a90e1b' exceeded. Please try again after '1' seconds after additional tokens are available. Refer to https:... message truncated\n"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://rp.j4206336.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/77628145-9ee8-4d12-9dd2-5dced29b5ecb
correlation ID: c83d59d4-8297-41d1-87c1-fbe0352a6163
client request ID: 4dcc2ce6-6521-4d7f-b86f-f0309a80ecee
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_fips_mode.go:103]: failed to create node pool "np-1" for fips-enabled cluster "fips-enabled-cluster"
Unexpected error:
<*fmt.wrapError | 0x335ced2f1b40>:
failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "fips-enabled-cluster" in resourcegroup="fips-enabled-kfwkzrbn6vbm" to finish creating: GET https://rp.j4206336.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/77628145-9ee8-4d12-9dd2-5dced29b5ecb
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/77628145-9ee8-4d12-9dd2-5dced29b5ecb",
"name": "77628145-9ee8-4d12-9dd2-5dced29b5ecb",
"status": "Failed",
"startTime": "2026-10-28T16:28:12.935779531Z",
"endTime": "2026-10-28T16:47:20.605052936Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine j6k4t1t6z4j2k5z-np-1-5nm58-fsnz7: Failed: virtualmachine failed to create or update. err: failed to get existing resource fips-enabled-kfwkzrbn6vbm--managed/j6k4t1t6z4j2k5z-np-1-5nm58-fsnz7 (service: virtualmachine): GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/fips-enabled-kfwkzrbn6vbm--managed/providers/Microsoft.Compute/virtualMachines/j6k4t1t6z4j2k5z-np-1-5nm58-fsnz7\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"SubscriptionRequestsThrottled\",\n \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'b117cf0f-0276-4486-9691-4f00f5a90e1b' exceeded. Please try again after '1' seconds after additional tokens are available. Refer to https:... message truncated\n"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://rp.j4206336.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/77628145-9ee8-4d12-9dd2-5dced29b5ecb
correlation ID: c83d59d4-8297-41d1-87c1-fbe0352a6163
client request ID: 4dcc2ce6-6521-4d7f-b86f-f0309a80ecee
occurredfail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_v20260901preview.go:102]: failed to create node pool "np-1" for v20260901preview cluster "v20260901"
Unexpected error:
<*fmt.wrapError | 0x1ecbca3c5be0>:
failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "v20260901" in resourcegroup="v20260901preview-p9l8lv6w4lzt" to finish creating: GET https://rp.j5142016.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/0d4205c1-5db7-479b-92dc-43f88aa7a88d
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/0d4205c1-5db7-479b-92dc-43f88aa7a88d",
"name": "0d4205c1-5db7-479b-92dc-43f88aa7a88d",
"status": "Failed",
"startTime": "2026-10-28T06:54:59.774467587Z",
"endTime": "2026-10-28T07:14:00.6647054Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine v20260901-np-1-mxkn8-rr6bs: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/v20260901preview-p9l8lv6w4lzt--managed/providers/Microsoft.Compute/virtualMachines/v20260901-np-1-mxkn8-rr6bs to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine v20260901-np-1-mxkn8-c8z72: Failed: interfaces failed to create or update. err: failed to get existing resource v20260901preview-p9l8lv6w4lzt--managed/v20260901-np-1-mxkn8-c8z72-nic (service: interfaces): GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/v20260901preview-p9l8lv6w4lzt--managed/providers/Microsoft.Network/networkInterfaces/v20260901-np-1-mxkn8-c8z72-nic\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"SubscriptionRequestsThrottled\",\n \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'db27175c-5bd0-48b4-929a-41de9a53ffbf' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.m... message truncated\n"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://rp.j5142016.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/0d4205c1-5db7-479b-92dc-43f88aa7a88d
correlation ID: 5492e0f1-946f-49bb-8474-ef7f078e1f6f
client request ID: 12bebe6d-b861-48ec-942f-b913321fe64f
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_v20260901preview.go:102]: failed to create node pool "np-1" for v20260901preview cluster "v20260901"
Unexpected error:
<*fmt.wrapError | 0x1ecbca3c5be0>:
failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "v20260901" in resourcegroup="v20260901preview-p9l8lv6w4lzt" to finish creating: GET https://rp.j5142016.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/0d4205c1-5db7-479b-92dc-43f88aa7a88d
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/0d4205c1-5db7-479b-92dc-43f88aa7a88d",
"name": "0d4205c1-5db7-479b-92dc-43f88aa7a88d",
"status": "Failed",
"startTime": "2026-10-28T06:54:59.774467587Z",
"endTime": "2026-10-28T07:14:00.6647054Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine v20260901-np-1-mxkn8-rr6bs: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/v20260901preview-p9l8lv6w4lzt--managed/providers/Microsoft.Compute/virtualMachines/v20260901-np-1-mxkn8-rr6bs to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine v20260901-np-1-mxkn8-c8z72: Failed: interfaces failed to create or update. err: failed to get existing resource v20260901preview-p9l8lv6w4lzt--managed/v20260901-np-1-mxkn8-c8z72-nic (service: interfaces): GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/v20260901preview-p9l8lv6w4lzt--managed/providers/Microsoft.Network/networkInterfaces/v20260901-np-1-mxkn8-c8z72-nic\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"SubscriptionRequestsThrottled\",\n \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'db27175c-5bd0-48b4-929a-41de9a53ffbf' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.m... message truncated\n"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://rp.j5142016.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/0d4205c1-5db7-479b-92dc-43f88aa7a88d
correlation ID: 5492e0f1-946f-49bb-8474-ef7f078e1f6f
client request ID: 12bebe6d-b861-48ec-942f-b913321fe64f
occurredContributing tests (3)
Affected runs (7) | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
DNS for route host <route-host> did not resolve: DNS for <route-host> did not resolve within <duration> (last error: lookup <host> on <dns-server>: no such host): context deadline...full failure pattern: DNS for route host <route-host> did not resolve: DNS for <route-host> did not resolve within <duration> (last error: lookup <host> on <dns-server>: no such host): context deadline exceeded Signal: IndeterminateSignal: Indeterminate — seen in 4 prior week(s); active 2 day(s) | e2e | 5 | 1.08%5 of 464 job runs affected | IndeterminateSignal: Indeterminate — seen in 4 prior week(s); active 2 day(s) | Show trend detailsOct 21: 0 · Oct 22: 3 · Oct 23: 0 · Oct 24: 0 · Oct 25: 0 · Oct 26: 0 · Oct 27: 0 · Oct 28: 2 | INT, PROD | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (3)fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_hypershift_presubmit.go:116]: failed to verify simple web app runs on cluster "cluster-hs-pre-krfsmn"
Unexpected error:
<*fmt.wrapError | 0x34ab4bc30760>:
DNS for route host agnhost-e2e-sample-app-hlljf.apps.aro.c5q3m7v0m2r2b6f.va75.j8974848.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-hlljf.apps.aro.c5q3m7v0m2r2b6f.va75.j8974848.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-hlljf.apps.aro.c5q3m7v0m2r2b6f.va75.j8974848.hcp.osadev.cloud on 172.30.0.10:53: no such host): context deadline exceeded
{
msg: "DNS for route host agnhost-e2e-sample-app-hlljf.apps.aro.c5q3m7v0m2r2b6f.va75.j8974848.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-hlljf.apps.aro.c5q3m7v0m2r2b6f.va75.j8974848.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-hlljf.apps.aro.c5q3m7v0m2r2b6f.va75.j8974848.hcp.osadev.cloud on 172.30.0.10:53: no such host): context deadline exceeded",
err: <*fmt.wrapError | 0x34ab4c426760>{
msg: "DNS for agnhost-e2e-sample-app-hlljf.apps.aro.c5q3m7v0m2r2b6f.va75.j8974848.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-hlljf.apps.aro.c5q3m7v0m2r2b6f.va75.j8974848.hcp.osadev.cloud on 172.30.0.10:53: no such host): context deadline exceeded",
err: <context.deadlineExceededError>{},
},
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_hypershift_presubmit.go:116]: failed to verify simple web app runs on cluster "cluster-hs-pre-krfsmn"
Unexpected error:
<*fmt.wrapError | 0x34ab4bc30760>:
DNS for route host agnhost-e2e-sample-app-hlljf.apps.aro.c5q3m7v0m2r2b6f.va75.j8974848.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-hlljf.apps.aro.c5q3m7v0m2r2b6f.va75.j8974848.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-hlljf.apps.aro.c5q3m7v0m2r2b6f.va75.j8974848.hcp.osadev.cloud on 172.30.0.10:53: no such host): context deadline exceeded
{
msg: "DNS for route host agnhost-e2e-sample-app-hlljf.apps.aro.c5q3m7v0m2r2b6f.va75.j8974848.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-hlljf.apps.aro.c5q3m7v0m2r2b6f.va75.j8974848.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-hlljf.apps.aro.c5q3m7v0m2r2b6f.va75.j8974848.hcp.osadev.cloud on 172.30.0.10:53: no such host): context deadline exceeded",
err: <*fmt.wrapError | 0x34ab4c426760>{
msg: "DNS for agnhost-e2e-sample-app-hlljf.apps.aro.c5q3m7v0m2r2b6f.va75.j8974848.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-hlljf.apps.aro.c5q3m7v0m2r2b6f.va75.j8974848.hcp.osadev.cloud on 172.30.0.10:53: no such host): context deadline exceeded",
err: <context.deadlineExceededError>{},
},
}
occurredfail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_hypershift_presubmit.go:116]: failed to verify simple web app runs on cluster "cluster-hs-pre-5pj7h6"
Unexpected error:
<*fmt.wrapError | 0x2590ea2143e0>:
DNS for route host agnhost-e2e-sample-app-dsbx7.apps.aro.m8c7u6n4b6i4x7c.8inl.j9099392.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-dsbx7.apps.aro.m8c7u6n4b6i4x7c.8inl.j9099392.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-dsbx7.apps.aro.m8c7u6n4b6i4x7c.8inl.j9099392.hcp.osadev.cloud on 172.30.0.10:53: no such host): context deadline exceeded
{
msg: "DNS for route host agnhost-e2e-sample-app-dsbx7.apps.aro.m8c7u6n4b6i4x7c.8inl.j9099392.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-dsbx7.apps.aro.m8c7u6n4b6i4x7c.8inl.j9099392.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-dsbx7.apps.aro.m8c7u6n4b6i4x7c.8inl.j9099392.hcp.osadev.cloud on 172.30.0.10:53: no such host): context deadline exceeded",
err: <*fmt.wrapError | 0x2590e99482e0>{
msg: "DNS for agnhost-e2e-sample-app-dsbx7.apps.aro.m8c7u6n4b6i4x7c.8inl.j9099392.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-dsbx7.apps.aro.m8c7u6n4b6i4x7c.8inl.j9099392.hcp.osadev.cloud on 172.30.0.10:53: no such host): context deadline exceeded",
err: <context.deadlineExceededError>{},
},
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_hypershift_presubmit.go:116]: failed to verify simple web app runs on cluster "cluster-hs-pre-5pj7h6"
Unexpected error:
<*fmt.wrapError | 0x2590ea2143e0>:
DNS for route host agnhost-e2e-sample-app-dsbx7.apps.aro.m8c7u6n4b6i4x7c.8inl.j9099392.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-dsbx7.apps.aro.m8c7u6n4b6i4x7c.8inl.j9099392.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-dsbx7.apps.aro.m8c7u6n4b6i4x7c.8inl.j9099392.hcp.osadev.cloud on 172.30.0.10:53: no such host): context deadline exceeded
{
msg: "DNS for route host agnhost-e2e-sample-app-dsbx7.apps.aro.m8c7u6n4b6i4x7c.8inl.j9099392.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-dsbx7.apps.aro.m8c7u6n4b6i4x7c.8inl.j9099392.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-dsbx7.apps.aro.m8c7u6n4b6i4x7c.8inl.j9099392.hcp.osadev.cloud on 172.30.0.10:53: no such host): context deadline exceeded",
err: <*fmt.wrapError | 0x2590e99482e0>{
msg: "DNS for agnhost-e2e-sample-app-dsbx7.apps.aro.m8c7u6n4b6i4x7c.8inl.j9099392.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-dsbx7.apps.aro.m8c7u6n4b6i4x7c.8inl.j9099392.hcp.osadev.cloud on 172.30.0.10:53: no such host): context deadline exceeded",
err: <context.deadlineExceededError>{},
},
}
occurredfail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:193]: failed to verify simple web app runs on cluster "cluster-candidate-4-22-8s2f2l"
Unexpected error:
<*fmt.wrapError | 0x30cafd6d0a20>:
DNS for route host agnhost-e2e-sample-app-56lgx.apps.aro.y9c6b2i7y5a3u5n.l4il.j7335040.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-56lgx.apps.aro.y9c6b2i7y5a3u5n.l4il.j7335040.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-56lgx.apps.aro.y9c6b2i7y5a3u5n.l4il.j7335040.hcp.osadev.cloud on 172.30.0.10:53: no such host): context deadline exceeded
{
msg: "DNS for route host agnhost-e2e-sample-app-56lgx.apps.aro.y9c6b2i7y5a3u5n.l4il.j7335040.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-56lgx.apps.aro.y9c6b2i7y5a3u5n.l4il.j7335040.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-56lgx.apps.aro.y9c6b2i7y5a3u5n.l4il.j7335040.hcp.osadev.cloud on 172.30.0.10:53: no such host): context deadline exceeded",
err: <*fmt.wrapError | 0x30cafd44e600>{
msg: "DNS for agnhost-e2e-sample-app-56lgx.apps.aro.y9c6b2i7y5a3u5n.l4il.j7335040.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-56lgx.apps.aro.y9c6b2i7y5a3u5n.l4il.j7335040.hcp.osadev.cloud on 172.30.0.10:53: no such host): context deadline exceeded",
err: <context.deadlineExceededError>{},
},
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:193]: failed to verify simple web app runs on cluster "cluster-candidate-4-22-8s2f2l"
Unexpected error:
<*fmt.wrapError | 0x30cafd6d0a20>:
DNS for route host agnhost-e2e-sample-app-56lgx.apps.aro.y9c6b2i7y5a3u5n.l4il.j7335040.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-56lgx.apps.aro.y9c6b2i7y5a3u5n.l4il.j7335040.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-56lgx.apps.aro.y9c6b2i7y5a3u5n.l4il.j7335040.hcp.osadev.cloud on 172.30.0.10:53: no such host): context deadline exceeded
{
msg: "DNS for route host agnhost-e2e-sample-app-56lgx.apps.aro.y9c6b2i7y5a3u5n.l4il.j7335040.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-56lgx.apps.aro.y9c6b2i7y5a3u5n.l4il.j7335040.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-56lgx.apps.aro.y9c6b2i7y5a3u5n.l4il.j7335040.hcp.osadev.cloud on 172.30.0.10:53: no such host): context deadline exceeded",
err: <*fmt.wrapError | 0x30cafd44e600>{
msg: "DNS for agnhost-e2e-sample-app-56lgx.apps.aro.y9c6b2i7y5a3u5n.l4il.j7335040.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-56lgx.apps.aro.y9c6b2i7y5a3u5n.l4il.j7335040.hcp.osadev.cloud on 172.30.0.10:53: no such host): context deadline exceeded",
err: <context.deadlineExceededError>{},
},
}
occurredContributing tests (3)
Affected runs (5) | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
alert [svc] ClusterCredentialMetricMissing firedfull failure pattern: alert [svc] ClusterCredentialMetricMissing fired Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 4 day(s) | alert | 5 | 1.08%5 of 464 job runs affected | IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 4 day(s) | Show trend detailsOct 21: 0 · Oct 22: 0 · Oct 23: 0 · Oct 24: 1 · Oct 25: 1 · Oct 26: 0 · Oct 27: 2 · Oct 28: 1 | none | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (3)alert fired 1 time(s) Firing 1: State: Fired (not resolved) Started: 2026-10-28T01:28:46Z Severity: Sev3 Labels: alertname="ClusterCredentialMetricMissing", cluster="ci01-j5915264-svc", severity="warning" Description: Credential refresher expiration metric is missing for service cluster ci01-j5915264-svc. alert fired 1 time(s) Firing 1: State: Fired (not resolved) Started: 2026-10-27T21:37:46Z Severity: Sev3 Labels: alertname="ClusterCredentialMetricMissing", cluster="ci01-j3842944-svc", severity="warning" Description: Credential refresher expiration metric is missing for service cluster ci01-j3842944-svc. alert fired 1 time(s) Firing 1: State: Fired (not resolved) Started: 2026-10-27T18:18:42Z Severity: Sev3 Labels: alertname="ClusterCredentialMetricMissing", cluster="ci01-j4499328-svc", severity="warning" Description: Credential refresher expiration metric is missing for service cluster ci01-j4499328-svc. Contributing tests (1)
Affected runs (5) | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Environment: INT
Daily Run Outcomes
No tests below 95.00% in this window with at least 10 runs.
Up to 50 failure patterns are loaded (minimum 1.00% impact), with 10 shown by default. Click column headers to re-sort.
Jump to the full Failure Patterns view for this window
| Failure patternThe canonical description of a recurring CI failure, extracted and normalized from raw logs. | Failed atThe stage of the job run where this failure occurred: 'provision' (environment setup, DEV only), 'e2e' (test suite execution), or 'other' (CI infrastructure issues that did not produce a failure pattern). | Number of distinct job runs where this failure pattern was detected. | Percentage of all job runs in this environment affected by this failure pattern during the selected window. | Classification of this failure pattern: Regression (likely caused by a specific PR), Flake (intermittent failure spread across days), Noise (low-quality or generic pattern), or Indeterminate. | TrendShows daily activity for this failure pattern in a trailing window anchored to the selected end date. The sparkline covers at least 7 days and at most 14 days, depending on the current window size. | Also inOther environments where the same failure pattern was also detected during the selected window. | |||||||||||||||||||||||||||||||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
tls: failed to verify certificate: x509: certificate signed by unknown authorityfull failure pattern: tls: failed to verify certificate: x509: certificate signed by unknown authority Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 2 day(s) | unknown | 12 | 27.91%12 of 43 job runs affected | IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 2 day(s) | Show trend detailsOct 21: 0 · Oct 22: 0 · Oct 23: 5 · Oct 24: 7 · Oct 25: 0 · Oct 26: 0 · Oct 27: 0 · Oct 28: 0 | none | |||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (3)fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:194]: failed to verify simple web app runs on cluster "cluster-candidate-4-23-l96rfk"
Unexpected error:
<*fmt.wrapError | 0x11cfecaf8760>:
[strict TLS verification] route was never reachable: Get "https://agnhost-e2e-sample-app-4vg4t.apps.aro.o6e4m4u5u4t4c4u.sbp8.westus3.aroapp-hcp.azure-test.net": tls: failed to verify certificate: x509: certificate signed by unknown authority
{
msg: "[strict TLS verification] route was never reachable: Get \"https://agnhost-e2e-sample-app-4vg4t.apps.aro.o6e4m4u5u4t4c4u.sbp8.westus3.aroapp-hcp.azure-test.net\": tls: failed to verify certificate: x509: certificate signed by unknown authority",
err: <*url.Error | 0x11cfee5e70e0>{
Op: "Get",
URL: "https://agnhost-e2e-sample-app-4vg4t.apps.aro.o6e4m4u5u4t4c4u.sbp8.westus3.aroapp-hcp.azure-test.net",
Err: <*tls.CertificateVerificationError | 0x11cfee5e70b0>{
UnverifiedCertificates: [
{
Raw: [48, 130, 4, 1, 48, 130, 2, 233, 160, 3, 2, 1, 2, 2, 8, 122, 194, 216, 171, 111, 107, 55, 112, 48, 13, 6, 9, 42, 134, 72, 134, 247, 13, 1, 1, 11, 5, 0, 48, 38, 49, 18, 48, 16, 6, 3, 85, 4, 11, 19, 9, 111, 112, 101, 110, 115, 104, 105, 102, 116, 49, 16, 48, 14, 6, 3, 85, 4, 3, 19, 7, 114, 111, 111, 116, 45, 99, 97, 48, 30, 23, 13, 50, 54, 49, 48, 50, 52, 50, 50, 50, 56, 48, 49, 90, 23, 13, 50, 55, 49, 48, 50, 52, 50, 50, 50, 56, 48, 49, 90, 48, 48, 49, 18, 48, 16, 6, 3, 85, 4, 10, 19, 9, 111, 112, 101, 110, 115, 104, 105, 102, 116, 49, 26, 48, 24, 6, 3, 85, 4, 3, 19, 17, 111, 112, 101, 110, 115, 104, 105, 102, 116, 45, 105, 110, 103, 114, 101, 115, 115, 48, 130, 1, 34, 48, 13, 6, 9, 42, 134, 72, 134, 247, 13, 1, 1, 1, 5, 0, 3, 130, 1, 15, 0, 48, 130, 1, 10, 2, 130, 1, 1, 0, 178, 55, 149, 119, 219, 63, 47, 5, 249, 2, 142, 64, 226, 181, 25, 19, 163, 170, 217, 26, 124, 161, 29, 250, 162, 212, 197, 31, 10, 179, 100, 47, 54, 215, 80, 7, 78, 121, 50, 101, 254, 19, 69, 126, 231, 240, 34, 4, 205, 235, 198, 28, 58, 139, 209, 113, 230, 105, 161, 163, 221, 192, 72, 27, 82, 64, 156, 231, 230, 222, 216, 170, 66, 81, 109, 14, 218, 138, 180, 42, 46, 37, 12, 20, 42, 46, 254, 75, 14, 239, 200, 146, 1, 44, 249, 109, 167, 6, 174, 142, 209, 55, 87, 205, 253, 244, 165, 78, 181, 174, 211, 88, 111, 195, 22, 44, 169, 29, 62, 197, 215, 209, 73, 64, 144, 182, 37, 230, 141, 3, 209, 58, 41, 137, 165, 79, 15, 206, 195, 232, 245, 252, 140, 232, 188, 179, 203, 150, 101, 191, 186, 47, 212, 214, 37, 240, 83, 70, 206, 45, 54, 33, 211, 136, 55, 189, 43, 255, 208, 60, 224, 146, 245, 105, 10, 7, 134, 154, 110, 155, 81, 215, 26, 170, 120, 141, 123, 32, 172, 156, 94, 207, 62, 225, 195, 61, 155, 50, 127, 235, 92, 32, 207, 220, 201, 84, 27, 1, 117, 152, 204, 160, 80, 33, 0, 199, 209, 234, 168, 90, 184, 171, 178, 224, 8, 13, 41, 49, 94, 136, 52, 225, 159, 183, 109, 30, 220, 54, 37, 247, 46, 253, 174, 26, 197, 35, 142, 122, 171, 24, 218, 246, 148, 31, 58, 235, 2, 3, 1, 0, 1, 163, 130, 1, 39, 48, 130, 1, 35, 48, 14, 6, 3, 85, 29, 15, 1, 1, 255, 4, 4, 3, 2, 5, 160, 48, 29, 6, 3, 85, 29, 37, 4, 22, 48, 20, 6, 8, 43, 6, 1, 5, 5, 7, 3, 2, 6, 8, 43, 6, 1, 5, 5, 7, 3, 1, 48, 12, 6, 3, 85, 29, 19, 1, 1, 255, 4, 2, 48, 0, 48, 73, 6, 3, 85, 29, 14, 4, 66, 4, 64, 197, 98, 86, 250, 137, 217, 107, 201, 120, 11, 250, 28, 8, 159, 50, 58, 147, 2, 127, 27, 39, 46, 176, 228, 89, 113, 119, 156, 18, 254, 186, 225, 239, 236, 201, 198, 127, 141, 38, 73, 33, 245, 173, 2, 193, 175, 61, 245, 132, 223, 97, 52, 79, 55, 120, 201, 63, 76, 209, 109, 45, 189, 34, 122, 48, 75, 6, 3, 85, 29, 35, 4, 68, 48, 66, 128, 64, 248, 50, 11, 119, 242, 227, 194, 167, 149, 215, 190, 179, 176, 143, 74, 56, 221, 252, 111, 212, 224, 237, 204, 6, 0, 197, 97, 180, 119, 93, 170, 15, 222, 152, 153, 8, 168, 38, 140, 186, 14, 89, 130, 247, 23, 168, 143, 191, 156, 56, 163, 177, 216, 127, 102, 115, 15, 90, 73, 84, 30, 110, 35, 172, 48, 76, 6, 3, 85, 29, 17, 4, 69, 48, 67, 130, 65, 42, 46, 97, 112, 112, 115, 46, 97, 114, 111, 46, 111, 54, 101, 52, 109, 52, 117, 53, 117, 52, 116, 52, 99, 52, 117, 46, 115, 98, 112, 56, 46, 119, 101, 115, 116, 117, 115, 51, 46, 97, 114, 111, 97, 112, 112, 45, 104, 99, 112, 46, 97, 122, 117, 114, 101, 45, 116, 101, 115, 116, 46, 110, 101, 116, 48, 13, 6, 9, 42, 134, 72, 134, 247, 13, 1, 1, 11, 5, 0, 3, 130, 1, 1, 0, 50, 25, 231, 199, 177, 69, 58, 16, 51, 91, 243, 219, 210, 199, 138, 155, 53, 245, 245, 106, 127, 215, 113, 116, 53, 212, 199, 147, 106, 103, 247, 189, 33, 171, 252, 21, 143, 150, 56, 31, 85, 0, 41, 147, 65, 24, 201, 140, 203, 150, 3, 31, 110, 137, 35, 75, 171, 175, 235, 209, 21, 26, 164, 179, 71, 67, 21, 133, 105, 14, 253, 71, 51, 45, 32, 43, 100, 94, 84, 47, 64, 244, 56, 58, 136, 157, 46, 87, 211, 40, 31, 180, 2, 165, 196, 210, 222, 204, 152, 251, 130, 232, 9, 182, 120, 239, 235, 22, 89, 58, 198, 78, 82, 139, 120, 162, 101, 8, 223, 107, 59, 223, 193, 151, 252, 30, 79, 0, 15, 40, 112, 104, 59, 157, 68, 154, 197, 178, 218, 209, 164, 56, 94, 93, 22, 22, 98, 191, 126, 106, 4, 194, 233, 190, 68, 193, 123, 232, 86, 187, 65, 26, 135, 65, 91, 129, 105, 54, 92, 78, 247, 245, 172, 13, 38, 206, 186, 5, 136, 153, 211, 200, 140, 153, 205, 176, 93, 247, 111, 230, 227, 13, 56, 73, 251, 143, 236, 121, 113, 183, 124, 61, 169, 206, 40, 146, 37, 166, 252, 132, 4, 153, 168, 84, 39, 23, 232, 218, 136, 80, 210, 143, 97, 106, 58, 193, 56, 253, 19, 202, 243, 199, 49, 235, 83, 136, 105, 217, 57, 123, 238, 192, 105, 226, 211, 224, 255, 115, 130, 221, 177, 40, 19, 51, 31, 249],
RawTBSCertificate: [48, 130, 2, 233, 160, 3, 2, 1, 2, 2, 8, 122, 194, 216, 171, 111, 107, 55, 112, 48, 13, 6, 9, 42, 134, 72, 134, 247, 13, 1, 1, 11, 5, 0, 48, 38, 49, 18, 48, 16, 6, 3, 85, 4, 11, 19, 9, 111, 112, 101, 110, 115, 104, 105, 102, 116, 49, 16, 48, 14, 6, 3, 85, 4, 3, 19, 7, 114, 111, 111, 116, 45, 99, 97, 48, 30, 23, 13, 50, 54, 49, 48, 50, 52, 50, 50, 50, 56, 48, 49, 90, 23, 13, 50, 55, 49, 48, 50, 52, 50, 50, 50, 56, 48, 49, 90, 48, 48, 49, 18, 48, 16, 6, 3, 85, 4, 10, 19, 9, 111, 112, 101, 110, 115, 104, 105, 102, 116, 49, 26, 48, 24, 6, 3, 85, 4, 3, 19, 17, 111, 112, 101, 110, 115, 104, 105, 102, 116, 45, 105, 110, 103, 114, 101, 115, 115, 48, 130, 1, 34, 48, 13, 6, 9, 42, 134, 72, 134, 247, 13, 1, 1, 1, 5, 0, 3, 130, 1, 15, 0, 48, 130, 1, 10, 2, 130, 1, 1, 0, 178, 55, 149, 119, 219, 63, 47, 5, 249, 2, 142, 64, 226, 181, 25, 19, 163, 170, 217, 26, 124, 161, 29, 250, 162, 212, 197, 31, 10, 179, 100, 47, 54, 215, 80, 7, 78, 121, 50, 101, 254, 19, 69, 126, 231, 240, 34, 4, 205, 235, 198, 28, 58, 139, 209, 113, 230, 105, 161, 163, 221, 192, 72, 27, 82, 64, 156, 231, 230, 222, 216, 170, 66, 81, 109, 14, 218, 138, 180, 42, 46, 37, 12, 20, 42, 46, 254, 75, 14, 239, 200, 146, 1, 44, 249, 109, 167, 6, 174, 142, 209, 55, 87, 205, 253, 244, 165, 78, 181, 174, 211, 88, 111, 195, 22, 44, 169, 29, 62, 197, 215, 209, 73, 64, 144, 182, 37, 230, 141, 3, 209, 58, 41, 137, 165, 79, 15, 206, 195, 232, 245, 252, 140, 232, 188, 179, 203, 150, 101, 191, 186, 47, 212, 214, 37, 240, 83, 70, 206, 45, 54, 33, 211, 136, 55, 189, 43, 255, 208, 60, 224, 146, 245, 105, 10, 7, 134, 154, 110, 155, 81, 215, 26, 170, 120, 141, 123, 32, 172, 156, 94, 207, 62, 225, 195, 61, 155, 50, 127, 235, 92, 32, 207, 220, 201, 84, 27, 1, 117, 152, 204, 160, 80, 33, 0, 199, 209, 234, 168, 90, 184, 171, 178, 224, 8, 13, 41, 49, 94, 136, 52, 225, 159, 183, 109, 30, 220, 54, 37, 247, 46, 253, 174, 26, 197, 35, 142, 122, 171, 24, 218, 246, 148, 31, 58, 235, 2, 3, 1, 0, 1, 163, 130, 1, 39, 48, 130, 1, 35, 48, 14, 6, 3, 85, 29, 15, 1, 1, 255, 4, 4, 3, 2, 5, 160, 48, 29, 6, 3, 85, 29, 37, 4, 22, 48, 20, 6, 8, 43, 6, 1, 5, 5, 7, 3, 2, 6, 8, 43, 6, 1, 5, 5, 7, 3, 1, 48, 12, 6, 3, 85, 29, 19, 1, 1, 255, 4, 2, 48, 0, 48, 73, 6, 3, 85, 29, 14, 4, 66, 4, 64, 197, 98, 86, 250, 137, 217, 107, 201, 120, 11, 250, 28, 8, 159, 50, 58, 147, 2, 127, 27, 39, 46, 176, 228, 89, 113, 119, 156, 18, 254, 186, 225, 239, 236, 201, 198, 127, 141, 38, 73, 33, 245, 173, 2, 193, 175, 61, 245, 132, 223, 97, 52, 79, 55, 120, 201, 63, 76, 209, 109, 45, 189, 34, 122, 48, 75, 6, 3, 85, 29, 35, 4, 68, 48, 66, 128, 64, 248, 50, 11, 119, 242, 227, 194, 167, 149, 215, 190, 179, 176, 143, 74, 56, 221, 252, 111, 212, 224, 237, 204, 6, 0, 197, 97, 180, 119, 93, 170, 15, 222, 152, 153, 8, 168, 38, 140, 186, 14, 89, 130, 247, 23, 168, 143, 191, 156, 56, 163, 177, 216, 127, 102, 115, 15, 90, 73, 84, 30, 110, 35, 172, 48, 76, 6, 3, 85, 29, 17, 4, 69, 48, 67, 130, 65, 42, 46, 97, 112, 112, 115, 46, 97, 114, 111, 46, 111, 54, 101, 52, 109, 52, 117, 53, 117, 52, 116, 52, 99, 52, 117, 46, 115, 98, 112, 56, 46, 119, 101, 115, 116, 117, 115, 51, 46, 97, 114, 111, 97, 112, 112, 45, 104, 99, 112, 46, 97, 122, 117, 114, 101, 45, 116, 101, 115, 116, 46, 110, 101, 116],
RawSubjectPublicKeyInfo: [48, 130, 1, 34, 48, 13, 6, 9, 42, 134, 72, 134, 247, 13, 1, 1, 1, 5, 0, 3, 130, 1, 15, 0, 48, 130, 1, 10, 2, 130, 1, 1, 0, 178, 55, 149, 119, 219, 63, 47, 5, 249, 2, 142, 64, 226, 181, 25, 19, 163, 170, 217, 26, 124, 161, 29, 250, 162, 212, 197, 31, 10, 179, 100, 47, 54, 215, 80, 7, 78, 121, 50, 101, 254, 19, 69, 126, 231, 240, 34, 4, 205, 235, 198, 28, 58, 139, 209, 113, 230, 105, 161, 163, 221, 192, 72, 27, 82, 64, 156, 231, 230, 222, 216, 170, 66, 81, 109, 14, 218, 138, 180, 42, 46, 37, 12, 20, 42, 46, 254, 75, 14, 239, 200, 146, 1, 44, 249, 109, 167, 6, 174, 142, 209, 55, 87, 205, 253, 244, 165, 78, 181, 174, 211, 88, 111, 195, 22, 44, 169, 29, 62, 197, 215, 209, 73, 64, 144, 182, 37, 230, 141, 3, 209, 58, 41, 137, 165, 79, 15, 206, 195, 232, 245, 252, 140, 232, 188, 179, 203, 150, 101, 191, 186, 47, 212, 214, 37, 240, 83, 70, 206, 45, 54, 33, 211, 136, 55, 189, 43, 255, 208, 60, 224, 146, 245, 105, 10, 7, 134, 154, 110, 155, 81, 215, 26, 170, 120, 141, 123, 32, 172, 156, 94, 207, 62, 225, 195, 61, 155, 50, 127, 235, 92, 32, 207, 220, 201, 84, 27, 1, 117, 152, 204, 160, 80, 33, 0, 199, 209, 234, 168, 90, 184, 171, 178, 224, 8, 13, 41, 49, 94, 136, 52, 225, 159, 183, 109, 30, 220, 54, 37, 247, 46, 253, 174, 26, 197, 35, 142, 122, 171, 24, 218, 246, 148, 31, 58, 235, 2, 3, 1, 0, 1],
RawSubject: [48, 48, 49, 18, 48, 16, 6, 3, 85, 4, 10, 19, 9, 111, 112, 101, 110, 115, 104, 105, 102, 116, 49, 26, 48, 24, 6, 3, 85, 4, 3, 19, 17, 111, 112, 101, 110, 115, 104, 105, 102, 116, 45, 105, 110, 103, 114, 101, 115, 115],
RawIssuer: [48, 38, 49, 18, 48, 16, 6, 3, 85, 4, 11, 19, 9, 111, 112, 101, 110, 115, 104, 105, 102, 116, 49, 16, 48, 14, 6, 3, 85, 4, 3, 19, 7, 114, 111, 111, 116, 45, 99, 97],
Signature: [50, 25, 231, 199, 177, 69, 58, 16, 51, 91, 243, 219, 210, 199, 138, 155, 53, 245, 245, 106, 127, 215, 113, 116, 53, 212, 199, 147, 106, 103, 247, 189, 33, 171, 252, 21, 143, 150, 56, 31, 85, 0, 41, 147, 65, 24, 201, 140, 203, 150, 3, 31, 110, 137, 35, 75, 171, 175, 235, 209, 21, 26, 164, 179, 71, 67, 21, 133, 105, 14, 253, 71, 51, 45, 32, 43, 100, 94, 84, 47, 64, 244, 56, 58, 136, 157, 46, 87, 211, 40, 31, 180, 2, 165, 196, 210, 222, 204, 152, 251, 130, 232, 9, 182, 120, 239, 235, 22, 89, 58, 198, 78, 82, 139, 120, 162, 101, 8, 223, 107, 59, 223, 193, 151, 252, 30, 79, 0, 15, 40, 112, 104, 59, 157, 68, 154, 197, 178, 218, 209, 164, 56, 94, 93, 22, 22, 98, 191, 126, 106, 4, 194, 233, 190, 68, 193, 123, 232, 86, 187, 65, 26, 135, 65, 91, 129, 105, 54, 92, 78, 247, 245, 172, 13, 38, 206, 186, 5, 136, 153, 211, 200, 140, 153, 205, 176, 93, 247, 111, 230, 227, 13, 56, 73, 251, 143, 236, 121, 113, 183, 124, 61, 169, 206, 40, 146, 37, 166, 252, 132, 4, 153, 168, 84, 39, 23, 232, 218, 136, 80, 210, 143, 97, 106, 58, 193, 56, 253, 19, 202, 243, 199, 49, 235, 83, 136, 105, 217, 57, 123, 238, 192, 105, 226, 211, 224, 255, 115, 130, 221, 177, 40, 19, 51, 31, 249],
SignatureAlgorithm: 4,
PublicKeyAlgorithm: 1,
PublicKey: <*rsa.PublicKey | 0x11cfee823170>{
N: {
neg: false,
abs: [12328844732508617451, 3386053924852305530, 11508787352079574519, 580165219016783073, 15126087406076015328, 8473747687808762055, 6638534397921598209, 4531117369475629035, 8686734636532653775, 9699186360867822250, 15005114980278798855, 3900631334283717631, 15336487295176330797, 13597435546774256175, 1139063161098112232, 10161195232013165903, 15551291529941624294, 8053304939740217029, 18299432943387333464, 12035498982612883405, 1076299365375342957, 3325077180629581387, 4778720590180299818, 5926909629444315306, 16602979226450151451, 14838171221845856625, 18308053321550668292, 3951715190352458341, 11733219666321171503, 11793477282006375930, 17943060274730047763, 12841897204482256645],
},
E: 65537,
},
Version: 3,
SerialNumber: {
neg: false,
abs: [8845870848929380208],
},
Issuer: {
Country: nil,
Organization: nil,
OrganizationalUnit: ["openshift"],
Locality: nil,
Province: nil,
StreetAddress: nil,
PostalCode: nil,
SerialNumber: "",
CommonName: "root-ca",
Names: [
{
Type: [2, 5, 4, 11],
Value: <string>"openshift",
},
{
Type: [2, 5, 4, 3],
Value: <string>"root-ca",
},
],
ExtraNames: nil,
},
Subject: {
Country: nil,
Organization: ["openshift"],
OrganizationalUnit: nil,
Locality: nil,
Province: nil,
StreetAddress: nil,
PostalCode: nil,
SerialNumber: "",
CommonName: "openshift-ingress",
Names: [
{
Type: [2, 5, 4, 10],
Value: <string>"openshift",
},
{
Type: [2, 5, 4, 3],
Value: <string>"openshift-ingress",
},
],
ExtraNames: nil,
},
NotBefore: {wall: 0, ext: 63928477681, loc: nil},
NotAfter: {wall: 0, ext: 63960013681, loc: nil},
KeyUsage: 5,
Extensions: [
{
Id: [2, 5, 29, 15],
Critical: true,
Value: [3, 2, 5, 160],
},
{
Id: [2, 5, 29, 37],
Critical: false,
Value: [48, 20, 6, 8, 43, 6, 1, 5, 5, 7, 3, 2, 6, 8, 43, 6, 1, 5, 5, 7, 3, 1],
},
{Id: [2, 5, 29, 19], Critical: true, Value: [48, 0]},
{
Id: [2, 5, 29, 14],
Critical: false,
Value: [4, 64, 197, 98, 86, 250, 137, 217, 107, 201, 120, 11, 250, 28, 8, 159, 50, 58, 147, 2, 127, 27, 39, 46, 176, 228, 89, 113, 119, 156, 18, 254, 186, 225, 239, 236, 201, 198, 127, 141, 38, 73, 33, 245, 173, 2, 193, 175, 61, 245, 132, 223, 97, 52, 79, 55, 120, 201, 63, 76, 209, 109, 45, 189, 34, 122],
},
{
Id: [2, 5, 29, 35],
Critical: false,
Value: [48, 66, 128, 64, 248, 50, 11, 119, 242, 227, 194, 167, 149, 215, 190, 179, 176, 143, 74, 56, 221, 252, 111, 212, 224, 237, 204, 6, 0, 197, 97, 180, 119, 93, 170, 15, 222, 152, 153, 8, 168, 38, 140, 186, 14, 89, 130, 247, 23, 168, 143, 191, 156, 56, 163, 177, 216, 127, 102, 115, 15, 90, 73, 84, 30, 110, 35, 172],
},
{
Id: [2, 5, 29, 17],
Critical: false,
Value: "0C\x82A*.apps.aro.o6e4m4u5u4t4c4u.sbp8.westus3.aroapp-hcp.azure-test.net",
},
],
ExtraExtensions: nil,
UnhandledCriticalExtensions: nil,
ExtKeyUsage: [2, 1],
UnknownExtKeyUsage: nil,
BasicConstraintsValid: true,
IsCA: false,
MaxPathLen: -1,
MaxPathLenZero: false,
SubjectKeyId: [197, 98, 86, 250, 137, 217, 107, 201, 120, 11, 250, 28, 8, 159, 50, 58, 147, 2, 127, 27, 39, 46, 176, 228, 89, 113, 119, 156, 18, 254, 186, 225, 239, 236, 201, 198, 127, 141, 38, 73, 33, 245, 173, 2, 193, 175, 61, 245, 132, 223, 97, 52, 79, 55, 120, 201, 63, 76, 209, 109, 45, 189, 34, 122],
AuthorityKeyId: [248, 50, 11, 119, 242, 227, 194, 167, 149, 215, 190, 179, 176, 143, 74, 56, 221, 252, 111, 212, 224, 237, 204, 6, 0, 197, 97, 180, 119, 93, 170, 15, 222, 152, 153, 8, 168, 38, 140, 186, 14, 89, 130, 247, 23, 168, 143, 191, 156, 56, 163, 177, 216, 127, 102, 115, 15, 90, 73, 84, 30, 110, 35, 172],
OCSPServer: nil,
IssuingCertificateURL: nil,
DNSNames: [
"*.apps.aro.o6e4m4u5u4t4c4u.sbp8.westus3.aroapp-hcp.azure-test.net",
],
EmailAddresses: nil,
IPAddresses: nil,
URIs: nil,
PermittedDNSDomainsCritical: false,
PermittedDNSDomains: nil,
ExcludedDNSDomains: nil,
PermittedIPRanges: nil,
ExcludedIPRanges: nil,
PermittedEmailAddresses: nil,
ExcludedEmailAddresses: nil,
PermittedURIDomains: nil,
ExcludedURIDomains: nil,
CRLDistributionPoints: nil,
PolicyIdentifiers: nil,
Policies: nil,
InhibitAnyPolicy: 0,
InhibitAnyPolicyZero: false,
InhibitPolicyMapping: 0,
InhibitPolicyMappingZero: false,
RequireExplicitPolicy: 0,
RequireExplicitPolicyZero: false,
PolicyMappings: nil,
},
],
Err: <x509.UnknownAuthorityError>{
Cert: 0x11cfed040008 (cyclic reference),
hintErr: nil,
hintCert: nil,
},
},
},
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:194]: failed to verify simple web app runs on cluster "cluster-candidate-4-23-l96rfk"
Unexpected error:
<*fmt.wrapError | 0x11cfecaf8760>:
[strict TLS verification] route was never reachable: Get "https://agnhost-e2e-sample-app-4vg4t.apps.aro.o6e4m4u5u4t4c4u.sbp8.westus3.aroapp-hcp.azure-test.net": tls: failed to verify certificate: x509: certificate signed by unknown authority
{
msg: "[strict TLS verification] route was never reachable: Get \"https://agnhost-e2e-sample-app-4vg4t.apps.aro.o6e4m4u5u4t4c4u.sbp8.westus3.aroapp-hcp.azure-test.net\": tls: failed to verify certificate: x509: certificate signed by unknown authority",
err: <*url.Error | 0x11cfee5e70e0>{
Op: "Get",
URL: "https://agnhost-e2e-sample-app-4vg4t.apps.aro.o6e4m4u5u4t4c4u.sbp8.westus3.aroapp-hcp.azure-test.net",
Err: <*tls.CertificateVerificationError | 0x11cfee5e70b0>{
UnverifiedCertificates: [
{
Raw: [48, 130, 4, 1, 48, 130, 2, 233, 160, 3, 2, 1, 2, 2, 8, 122, 194, 216, 171, 111, 107, 55, 112, 48, 13, 6, 9, 42, 134, 72, 134, 247, 13, 1, 1, 11, 5, 0, 48, 38, 49, 18, 48, 16, 6, 3, 85, 4, 11, 19, 9, 111, 112, 101, 110, 115, 104, 105, 102, 116, 49, 16, 48, 14, 6, 3, 85, 4, 3, 19, 7, 114, 111, 111, 116, 45, 99, 97, 48, 30, 23, 13, 50, 54, 49, 48, 50, 52, 50, 50, 50, 56, 48, 49, 90, 23, 13, 50, 55, 49, 48, 50, 52, 50, 50, 50, 56, 48, 49, 90, 48, 48, 49, 18, 48, 16, 6, 3, 85, 4, 10, 19, 9, 111, 112, 101, 110, 115, 104, 105, 102, 116, 49, 26, 48, 24, 6, 3, 85, 4, 3, 19, 17, 111, 112, 101, 110, 115, 104, 105, 102, 116, 45, 105, 110, 103, 114, 101, 115, 115, 48, 130, 1, 34, 48, 13, 6, 9, 42, 134, 72, 134, 247, 13, 1, 1, 1, 5, 0, 3, 130, 1, 15, 0, 48, 130, 1, 10, 2, 130, 1, 1, 0, 178, 55, 149, 119, 219, 63, 47, 5, 249, 2, 142, 64, 226, 181, 25, 19, 163, 170, 217, 26, 124, 161, 29, 250, 162, 212, 197, 31, 10, 179, 100, 47, 54, 215, 80, 7, 78, 121, 50, 101, 254, 19, 69, 126, 231, 240, 34, 4, 205, 235, 198, 28, 58, 139, 209, 113, 230, 105, 161, 163, 221, 192, 72, 27, 82, 64, 156, 231, 230, 222, 216, 170, 66, 81, 109, 14, 218, 138, 180, 42, 46, 37, 12, 20, 42, 46, 254, 75, 14, 239, 200, 146, 1, 44, 249, 109, 167, 6, 174, 142, 209, 55, 87, 205, 253, 244, 165, 78, 181, 174, 211, 88, 111, 195, 22, 44, 169, 29, 62, 197, 215, 209, 73, 64, 144, 182, 37, 230, 141, 3, 209, 58, 41, 137, 165, 79, 15, 206, 195, 232, 245, 252, 140, 232, 188, 179, 203, 150, 101, 191, 186, 47, 212, 214, 37, 240, 83, 70, 206, 45, 54, 33, 211, 136, 55, 189, 43, 255, 208, 60, 224, 146, 245, 105, 10, 7, 134, 154, 110, 155, 81, 215, 26, 170, 120, 141, 123, 32, 172, 156, 94, 207, 62, 225, 195, 61, 155, 50, 127, 235, 92, 32, 207, 220, 201, 84, 27, 1, 117, 152, 204, 160, 80, 33, 0, 199, 209, 234, 168, 90, 184, 171, 178, 224, 8, 13, 41, 49, 94, 136, 52, 225, 159, 183, 109, 30, 220, 54, 37, 247, 46, 253, 174, 26, 197, 35, 142, 122, 171, 24, 218, 246, 148, 31, 58, 235, 2, 3, 1, 0, 1, 163, 130, 1, 39, 48, 130, 1, 35, 48, 14, 6, 3, 85, 29, 15, 1, 1, 255, 4, 4, 3, 2, 5, 160, 48, 29, 6, 3, 85, 29, 37, 4, 22, 48, 20, 6, 8, 43, 6, 1, 5, 5, 7, 3, 2, 6, 8, 43, 6, 1, 5, 5, 7, 3, 1, 48, 12, 6, 3, 85, 29, 19, 1, 1, 255, 4, 2, 48, 0, 48, 73, 6, 3, 85, 29, 14, 4, 66, 4, 64, 197, 98, 86, 250, 137, 217, 107, 201, 120, 11, 250, 28, 8, 159, 50, 58, 147, 2, 127, 27, 39, 46, 176, 228, 89, 113, 119, 156, 18, 254, 186, 225, 239, 236, 201, 198, 127, 141, 38, 73, 33, 245, 173, 2, 193, 175, 61, 245, 132, 223, 97, 52, 79, 55, 120, 201, 63, 76, 209, 109, 45, 189, 34, 122, 48, 75, 6, 3, 85, 29, 35, 4, 68, 48, 66, 128, 64, 248, 50, 11, 119, 242, 227, 194, 167, 149, 215, 190, 179, 176, 143, 74, 56, 221, 252, 111, 212, 224, 237, 204, 6, 0, 197, 97, 180, 119, 93, 170, 15, 222, 152, 153, 8, 168, 38, 140, 186, 14, 89, 130, 247, 23, 168, 143, 191, 156, 56, 163, 177, 216, 127, 102, 115, 15, 90, 73, 84, 30, 110, 35, 172, 48, 76, 6, 3, 85, 29, 17, 4, 69, 48, 67, 130, 65, 42, 46, 97, 112, 112, 115, 46, 97, 114, 111, 46, 111, 54, 101, 52, 109, 52, 117, 53, 117, 52, 116, 52, 99, 52, 117, 46, 115, 98, 112, 56, 46, 119, 101, 115, 116, 117, 115, 51, 46, 97, 114, 111, 97, 112, 112, 45, 104, 99, 112, 46, 97, 122, 117, 114, 101, 45, 116, 101, 115, 116, 46, 110, 101, 116, 48, 13, 6, 9, 42, 134, 72, 134, 247, 13, 1, 1, 11, 5, 0, 3, 130, 1, 1, 0, 50, 25, 231, 199, 177, 69, 58, 16, 51, 91, 243, 219, 210, 199, 138, 155, 53, 245, 245, 106, 127, 215, 113, 116, 53, 212, 199, 147, 106, 103, 247, 189, 33, 171, 252, 21, 143, 150, 56, 31, 85, 0, 41, 147, 65, 24, 201, 140, 203, 150, 3, 31, 110, 137, 35, 75, 171, 175, 235, 209, 21, 26, 164, 179, 71, 67, 21, 133, 105, 14, 253, 71, 51, 45, 32, 43, 100, 94, 84, 47, 64, 244, 56, 58, 136, 157, 46, 87, 211, 40, 31, 180, 2, 165, 196, 210, 222, 204, 152, 251, 130, 232, 9, 182, 120, 239, 235, 22, 89, 58, 198, 78, 82, 139, 120, 162, 101, 8, 223, 107, 59, 223, 193, 151, 252, 30, 79, 0, 15, 40, 112, 104, 59, 157, 68, 154, 197, 178, 218, 209, 164, 56, 94, 93, 22, 22, 98, 191, 126, 106, 4, 194, 233, 190, 68, 193, 123, 232, 86, 187, 65, 26, 135, 65, 91, 129, 105, 54, 92, 78, 247, 245, 172, 13, 38, 206, 186, 5, 136, 153, 211, 200, 140, 153, 205, 176, 93, 247, 111, 230, 227, 13, 56, 73, 251, 143, 236, 121, 113, 183, 124, 61, 169, 206, 40, 146, 37, 166, 252, 132, 4, 153, 168, 84, 39, 23, 232, 218, 136, 80, 210, 143, 97, 106, 58, 193, 56, 253, 19, 202, 243, 199, 49, 235, 83, 136, 105, 217, 57, 123, 238, 192, 105, 226, 211, 224, 255, 115, 130, 221, 177, 40, 19, 51, 31, 249],
RawTBSCertificate: [48, 130, 2, 233, 160, 3, 2, 1, 2, 2, 8, 122, 194, 216, 171, 111, 107, 55, 112, 48, 13, 6, 9, 42, 134, 72, 134, 247, 13, 1, 1, 11, 5, 0, 48, 38, 49, 18, 48, 16, 6, 3, 85, 4, 11, 19, 9, 111, 112, 101, 110, 115, 104, 105, 102, 116, 49, 16, 48, 14, 6, 3, 85, 4, 3, 19, 7, 114, 111, 111, 116, 45, 99, 97, 48, 30, 23, 13, 50, 54, 49, 48, 50, 52, 50, 50, 50, 56, 48, 49, 90, 23, 13, 50, 55, 49, 48, 50, 52, 50, 50, 50, 56, 48, 49, 90, 48, 48, 49, 18, 48, 16, 6, 3, 85, 4, 10, 19, 9, 111, 112, 101, 110, 115, 104, 105, 102, 116, 49, 26, 48, 24, 6, 3, 85, 4, 3, 19, 17, 111, 112, 101, 110, 115, 104, 105, 102, 116, 45, 105, 110, 103, 114, 101, 115, 115, 48, 130, 1, 34, 48, 13, 6, 9, 42, 134, 72, 134, 247, 13, 1, 1, 1, 5, 0, 3, 130, 1, 15, 0, 48, 130, 1, 10, 2, 130, 1, 1, 0, 178, 55, 149, 119, 219, 63, 47, 5, 249, 2, 142, 64, 226, 181, 25, 19, 163, 170, 217, 26, 124, 161, 29, 250, 162, 212, 197, 31, 10, 179, 100, 47, 54, 215, 80, 7, 78, 121, 50, 101, 254, 19, 69, 126, 231, 240, 34, 4, 205, 235, 198, 28, 58, 139, 209, 113, 230, 105, 161, 163, 221, 192, 72, 27, 82, 64, 156, 231, 230, 222, 216, 170, 66, 81, 109, 14, 218, 138, 180, 42, 46, 37, 12, 20, 42, 46, 254, 75, 14, 239, 200, 146, 1, 44, 249, 109, 167, 6, 174, 142, 209, 55, 87, 205, 253, 244, 165, 78, 181, 174, 211, 88, 111, 195, 22, 44, 169, 29, 62, 197, 215, 209, 73, 64, 144, 182, 37, 230, 141, 3, 209, 58, 41, 137, 165, 79, 15, 206, 195, 232, 245, 252, 140, 232, 188, 179, 203, 150, 101, 191, 186, 47, 212, 214, 37, 240, 83, 70, 206, 45, 54, 33, 211, 136, 55, 189, 43, 255, 208, 60, 224, 146, 245, 105, 10, 7, 134, 154, 110, 155, 81, 215, 26, 170, 120, 141, 123, 32, 172, 156, 94, 207, 62, 225, 195, 61, 155, 50, 127, 235, 92, 32, 207, 220, 201, 84, 27, 1, 117, 152, 204, 160, 80, 33, 0, 199, 209, 234, 168, 90, 184, 171, 178, 224, 8, 13, 41, 49, 94, 136, 52, 225, 159, 183, 109, 30, 220, 54, 37, 247, 46, 253, 174, 26, 197, 35, 142, 122, 171, 24, 218, 246, 148, 31, 58, 235, 2, 3, 1, 0, 1, 163, 130, 1, 39, 48, 130, 1, 35, 48, 14, 6, 3, 85, 29, 15, 1, 1, 255, 4, 4, 3, 2, 5, 160, 48, 29, 6, 3, 85, 29, 37, 4, 22, 48, 20, 6, 8, 43, 6, 1, 5, 5, 7, 3, 2, 6, 8, 43, 6, 1, 5, 5, 7, 3, 1, 48, 12, 6, 3, 85, 29, 19, 1, 1, 255, 4, 2, 48, 0, 48, 73, 6, 3, 85, 29, 14, 4, 66, 4, 64, 197, 98, 86, 250, 137, 217, 107, 201, 120, 11, 250, 28, 8, 159, 50, 58, 147, 2, 127, 27, 39, 46, 176, 228, 89, 113, 119, 156, 18, 254, 186, 225, 239, 236, 201, 198, 127, 141, 38, 73, 33, 245, 173, 2, 193, 175, 61, 245, 132, 223, 97, 52, 79, 55, 120, 201, 63, 76, 209, 109, 45, 189, 34, 122, 48, 75, 6, 3, 85, 29, 35, 4, 68, 48, 66, 128, 64, 248, 50, 11, 119, 242, 227, 194, 167, 149, 215, 190, 179, 176, 143, 74, 56, 221, 252, 111, 212, 224, 237, 204, 6, 0, 197, 97, 180, 119, 93, 170, 15, 222, 152, 153, 8, 168, 38, 140, 186, 14, 89, 130, 247, 23, 168, 143, 191, 156, 56, 163, 177, 216, 127, 102, 115, 15, 90, 73, 84, 30, 110, 35, 172, 48, 76, 6, 3, 85, 29, 17, 4, 69, 48, 67, 130, 65, 42, 46, 97, 112, 112, 115, 46, 97, 114, 111, 46, 111, 54, 101, 52, 109, 52, 117, 53, 117, 52, 116, 52, 99, 52, 117, 46, 115, 98, 112, 56, 46, 119, 101, 115, 116, 117, 115, 51, 46, 97, 114, 111, 97, 112, 112, 45, 104, 99, 112, 46, 97, 122, 117, 114, 101, 45, 116, 101, 115, 116, 46, 110, 101, 116],
RawSubjectPublicKeyInfo: [48, 130, 1, 34, 48, 13, 6, 9, 42, 134, 72, 134, 247, 13, 1, 1, 1, 5, 0, 3, 130, 1, 15, 0, 48, 130, 1, 10, 2, 130, 1, 1, 0, 178, 55, 149, 119, 219, 63, 47, 5, 249, 2, 142, 64, 226, 181, 25, 19, 163, 170, 217, 26, 124, 161, 29, 250, 162, 212, 197, 31, 10, 179, 100, 47, 54, 215, 80, 7, 78, 121, 50, 101, 254, 19, 69, 126, 231, 240, 34, 4, 205, 235, 198, 28, 58, 139, 209, 113, 230, 105, 161, 163, 221, 192, 72, 27, 82, 64, 156, 231, 230, 222, 216, 170, 66, 81, 109, 14, 218, 138, 180, 42, 46, 37, 12, 20, 42, 46, 254, 75, 14, 239, 200, 146, 1, 44, 249, 109, 167, 6, 174, 142, 209, 55, 87, 205, 253, 244, 165, 78, 181, 174, 211, 88, 111, 195, 22, 44, 169, 29, 62, 197, 215, 209, 73, 64, 144, 182, 37, 230, 141, 3, 209, 58, 41, 137, 165, 79, 15, 206, 195, 232, 245, 252, 140, 232, 188, 179, 203, 150, 101, 191, 186, 47, 212, 214, 37, 240, 83, 70, 206, 45, 54, 33, 211, 136, 55, 189, 43, 255, 208, 60, 224, 146, 245, 105, 10, 7, 134, 154, 110, 155, 81, 215, 26, 170, 120, 141, 123, 32, 172, 156, 94, 207, 62, 225, 195, 61, 155, 50, 127, 235, 92, 32, 207, 220, 201, 84, 27, 1, 117, 152, 204, 160, 80, 33, 0, 199, 209, 234, 168, 90, 184, 171, 178, 224, 8, 13, 41, 49, 94, 136, 52, 225, 159, 183, 109, 30, 220, 54, 37, 247, 46, 253, 174, 26, 197, 35, 142, 122, 171, 24, 218, 246, 148, 31, 58, 235, 2, 3, 1, 0, 1],
RawSubject: [48, 48, 49, 18, 48, 16, 6, 3, 85, 4, 10, 19, 9, 111, 112, 101, 110, 115, 104, 105, 102, 116, 49, 26, 48, 24, 6, 3, 85, 4, 3, 19, 17, 111, 112, 101, 110, 115, 104, 105, 102, 116, 45, 105, 110, 103, 114, 101, 115, 115],
RawIssuer: [48, 38, 49, 18, 48, 16, 6, 3, 85, 4, 11, 19, 9, 111, 112, 101, 110, 115, 104, 105, 102, 116, 49, 16, 48, 14, 6, 3, 85, 4, 3, 19, 7, 114, 111, 111, 116, 45, 99, 97],
Signature: [50, 25, 231, 199, 177, 69, 58, 16, 51, 91, 243, 219, 210, 199, 138, 155, 53, 245, 245, 106, 127, 215, 113, 116, 53, 212, 199, 147, 106, 103, 247, 189, 33, 171, 252, 21, 143, 150, 56, 31, 85, 0, 41, 147, 65, 24, 201, 140, 203, 150, 3, 31, 110, 137, 35, 75, 171, 175, 235, 209, 21, 26, 164, 179, 71, 67, 21, 133, 105, 14, 253, 71, 51, 45, 32, 43, 100, 94, 84, 47, 64, 244, 56, 58, 136, 157, 46, 87, 211, 40, 31, 180, 2, 165, 196, 210, 222, 204, 152, 251, 130, 232, 9, 182, 120, 239, 235, 22, 89, 58, 198, 78, 82, 139, 120, 162, 101, 8, 223, 107, 59, 223, 193, 151, 252, 30, 79, 0, 15, 40, 112, 104, 59, 157, 68, 154, 197, 178, 218, 209, 164, 56, 94, 93, 22, 22, 98, 191, 126, 106, 4, 194, 233, 190, 68, 193, 123, 232, 86, 187, 65, 26, 135, 65, 91, 129, 105, 54, 92, 78, 247, 245, 172, 13, 38, 206, 186, 5, 136, 153, 211, 200, 140, 153, 205, 176, 93, 247, 111, 230, 227, 13, 56, 73, 251, 143, 236, 121, 113, 183, 124, 61, 169, 206, 40, 146, 37, 166, 252, 132, 4, 153, 168, 84, 39, 23, 232, 218, 136, 80, 210, 143, 97, 106, 58, 193, 56, 253, 19, 202, 243, 199, 49, 235, 83, 136, 105, 217, 57, 123, 238, 192, 105, 226, 211, 224, 255, 115, 130, 221, 177, 40, 19, 51, 31, 249],
SignatureAlgorithm: 4,
PublicKeyAlgorithm: 1,
PublicKey: <*rsa.PublicKey | 0x11cfee823170>{
N: {
neg: false,
abs: [12328844732508617451, 3386053924852305530, 11508787352079574519, 580165219016783073, 15126087406076015328, 8473747687808762055, 6638534397921598209, 4531117369475629035, 8686734636532653775, 9699186360867822250, 15005114980278798855, 3900631334283717631, 15336487295176330797, 13597435546774256175, 1139063161098112232, 10161195232013165903, 15551291529941624294, 8053304939740217029, 18299432943387333464, 12035498982612883405, 1076299365375342957, 3325077180629581387, 4778720590180299818, 5926909629444315306, 16602979226450151451, 14838171221845856625, 18308053321550668292, 3951715190352458341, 11733219666321171503, 11793477282006375930, 17943060274730047763, 12841897204482256645],
},
E: 65537,
},
Version: 3,
SerialNumber: {
neg: false,
abs: [8845870848929380208],
},
Issuer: {
Country: nil,
Organization: nil,
OrganizationalUnit: ["openshift"],
Locality: nil,
Province: nil,
StreetAddress: nil,
PostalCode: nil,
SerialNumber: "",
CommonName: "root-ca",
Names: [
{
Type: [2, 5, 4, 11],
Value: <string>"openshift",
},
{
Type: [2, 5, 4, 3],
Value: <string>"root-ca",
},
],
ExtraNames: nil,
},
Subject: {
Country: nil,
Organization: ["openshift"],
OrganizationalUnit: nil,
Locality: nil,
Province: nil,
StreetAddress: nil,
PostalCode: nil,
SerialNumber: "",
CommonName: "openshift-ingress",
Names: [
{
Type: [2, 5, 4, 10],
Value: <string>"openshift",
},
{
Type: [2, 5, 4, 3],
Value: <string>"openshift-ingress",
},
],
ExtraNames: nil,
},
NotBefore: {wall: 0, ext: 63928477681, loc: nil},
NotAfter: {wall: 0, ext: 63960013681, loc: nil},
KeyUsage: 5,
Extensions: [
{
Id: [2, 5, 29, 15],
Critical: true,
Value: [3, 2, 5, 160],
},
{
Id: [2, 5, 29, 37],
Critical: false,
Value: [48, 20, 6, 8, 43, 6, 1, 5, 5, 7, 3, 2, 6, 8, 43, 6, 1, 5, 5, 7, 3, 1],
},
{Id: [2, 5, 29, 19], Critical: true, Value: [48, 0]},
{
Id: [2, 5, 29, 14],
Critical: false,
Value: [4, 64, 197, 98, 86, 250, 137, 217, 107, 201, 120, 11, 250, 28, 8, 159, 50, 58, 147, 2, 127, 27, 39, 46, 176, 228, 89, 113, 119, 156, 18, 254, 186, 225, 239, 236, 201, 198, 127, 141, 38, 73, 33, 245, 173, 2, 193, 175, 61, 245, 132, 223, 97, 52, 79, 55, 120, 201, 63, 76, 209, 109, 45, 189, 34, 122],
},
{
Id: [2, 5, 29, 35],
Critical: false,
Value: [48, 66, 128, 64, 248, 50, 11, 119, 242, 227, 194, 167, 149, 215, 190, 179, 176, 143, 74, 56, 221, 252, 111, 212, 224, 237, 204, 6, 0, 197, 97, 180, 119, 93, 170, 15, 222, 152, 153, 8, 168, 38, 140, 186, 14, 89, 130, 247, 23, 168, 143, 191, 156, 56, 163, 177, 216, 127, 102, 115, 15, 90, 73, 84, 30, 110, 35, 172],
},
{
Id: [2, 5, 29, 17],
Critical: false,
Value: "0C\x82A*.apps.aro.o6e4m4u5u4t4c4u.sbp8.westus3.aroapp-hcp.azure-test.net",
},
],
ExtraExtensions: nil,
UnhandledCriticalExtensions: nil,
ExtKeyUsage: [2, 1],
UnknownExtKeyUsage: nil,
BasicConstraintsValid: true,
IsCA: false,
MaxPathLen: -1,
MaxPathLenZero: false,
SubjectKeyId: [197, 98, 86, 250, 137, 217, 107, 201, 120, 11, 250, 28, 8, 159, 50, 58, 147, 2, 127, 27, 39, 46, 176, 228, 89, 113, 119, 156, 18, 254, 186, 225, 239, 236, 201, 198, 127, 141, 38, 73, 33, 245, 173, 2, 193, 175, 61, 245, 132, 223, 97, 52, 79, 55, 120, 201, 63, 76, 209, 109, 45, 189, 34, 122],
AuthorityKeyId: [248, 50, 11, 119, 242, 227, 194, 167, 149, 215, 190, 179, 176, 143, 74, 56, 221, 252, 111, 212, 224, 237, 204, 6, 0, 197, 97, 180, 119, 93, 170, 15, 222, 152, 153, 8, 168, 38, 140, 186, 14, 89, 130, 247, 23, 168, 143, 191, 156, 56, 163, 177, 216, 127, 102, 115, 15, 90, 73, 84, 30, 110, 35, 172],
OCSPServer: nil,
IssuingCertificateURL: nil,
DNSNames: [
"*.apps.aro.o6e4m4u5u4t4c4u.sbp8.westus3.aroapp-hcp.azure-test.net",
],
EmailAddresses: nil,
IPAddresses: nil,
URIs: nil,
PermittedDNSDomainsCritical: false,
PermittedDNSDomains: nil,
ExcludedDNSDomains: nil,
PermittedIPRanges: nil,
ExcludedIPRanges: nil,
PermittedEmailAddresses: nil,
ExcludedEmailAddresses: nil,
PermittedURIDomains: nil,
ExcludedURIDomains: nil,
CRLDistributionPoints: nil,
PolicyIdentifiers: nil,
Policies: nil,
InhibitAnyPolicy: 0,
InhibitAnyPolicyZero: false,
InhibitPolicyMapping: 0,
InhibitPolicyMappingZero: false,
RequireExplicitPolicy: 0,
RequireExplicitPolicyZero: false,
PolicyMappings: nil,
},
],
Err: <x509.UnknownAuthorityError>{
Cert: 0x11cfed040008 (cyclic reference),
hintErr: nil,
hintCert: nil,
},
},
},
}
occurredfail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:194]: failed to verify simple web app runs on cluster "cluster-candidate-4-23-2rcw86"
Unexpected error:
<*fmt.wrapError | 0x2b2e578984a0>:
[strict TLS verification] route was never reachable: Get "https://agnhost-e2e-sample-app-w86zx.apps.aro.a2s9b8m2e4p7p0g.0mu8.westus3.aroapp-hcp.azure-test.net": tls: failed to verify certificate: x509: certificate signed by unknown authority
{
msg: "[strict TLS verification] route was never reachable: Get \"https://agnhost-e2e-sample-app-w86zx.apps.aro.a2s9b8m2e4p7p0g.0mu8.westus3.aroapp-hcp.azure-test.net\": tls: failed to verify certificate: x509: certificate signed by unknown authority",
err: <*url.Error | 0x2b2e57370780>{
Op: "Get",
URL: "https://agnhost-e2e-sample-app-w86zx.apps.aro.a2s9b8m2e4p7p0g.0mu8.westus3.aroapp-hcp.azure-test.net",
Err: <*tls.CertificateVerificationError | 0x2b2e57370750>{
UnverifiedCertificates: [
{
Raw: [48, 130, 4, 1, 48, 130, 2, 233, 160, 3, 2, 1, 2, 2, 8, 124, 87, 70, 122, 130, 6, 100, 82, 48, 13, 6, 9, 42, 134, 72, 134, 247, 13, 1, 1, 11, 5, 0, 48, 38, 49, 18, 48, 16, 6, 3, 85, 4, 11, 19, 9, 111, 112, 101, 110, 115, 104, 105, 102, 116, 49, 16, 48, 14, 6, 3, 85, 4, 3, 19, 7, 114, 111, 111, 116, 45, 99, 97, 48, 30, 23, 13, 50, 54, 49, 48, 50, 52, 50, 49, 48, 56, 48, 56, 90, 23, 13, 50, 55, 49, 48, 50, 52, 50, 49, 48, 56, 48, 56, 90, 48, 48, 49, 18, 48, 16, 6, 3, 85, 4, 10, 19, 9, 111, 112, 101, 110, 115, 104, 105, 102, 116, 49, 26, 48, 24, 6, 3, 85, 4, 3, 19, 17, 111, 112, 101, 110, 115, 104, 105, 102, 116, 45, 105, 110, 103, 114, 101, 115, 115, 48, 130, 1, 34, 48, 13, 6, 9, 42, 134, 72, 134, 247, 13, 1, 1, 1, 5, 0, 3, 130, 1, 15, 0, 48, 130, 1, 10, 2, 130, 1, 1, 0, 184, 6, 158, 64, 94, 24, 96, 40, 160, 154, 141, 196, 128, 87, 218, 59, 39, 71, 199, 234, 79, 240, 146, 24, 254, 106, 245, 45, 201, 136, 214, 64, 254, 110, 56, 158, 251, 103, 159, 125, 70, 9, 9, 192, 157, 185, 88, 45, 237, 209, 14, 238, 22, 11, 156, 42, 61, 172, 248, 175, 205, 29, 87, 44, 249, 69, 148, 194, 201, 88, 31, 36, 152, 175, 207, 120, 70, 231, 234, 239, 27, 120, 11, 188, 47, 126, 21, 187, 83, 61, 218, 253, 69, 150, 57, 242, 31, 129, 90, 78, 122, 162, 115, 193, 66, 231, 239, 171, 129, 241, 209, 248, 79, 198, 197, 15, 5, 20, 97, 114, 135, 82, 183, 39, 26, 12, 204, 167, 245, 131, 126, 229, 14, 152, 249, 66, 23, 145, 179, 201, 129, 84, 180, 47, 247, 191, 198, 219, 235, 14, 55, 141, 41, 250, 184, 135, 128, 115, 246, 220, 161, 224, 164, 119, 161, 199, 4, 69, 138, 127, 197, 71, 128, 167, 90, 212, 150, 2, 13, 143, 216, 191, 58, 228, 209, 93, 22, 58, 9, 137, 127, 203, 163, 74, 40, 217, 253, 186, 51, 153, 40, 52, 238, 73, 60, 57, 57, 108, 128, 2, 48, 134, 115, 108, 142, 65, 158, 113, 75, 25, 249, 189, 39, 130, 154, 24, 112, 209, 120, 229, 129, 223, 190, 147, 169, 232, 115, 83, 39, 47, 230, 201, 87, 216, 98, 161, 27, 95, 39, 237, 157, 102, 109, 222, 170, 149, 2, 3, 1, 0, 1, 163, 130, 1, 39, 48, 130, 1, 35, 48, 14, 6, 3, 85, 29, 15, 1, 1, 255, 4, 4, 3, 2, 5, 160, 48, 29, 6, 3, 85, 29, 37, 4, 22, 48, 20, 6, 8, 43, 6, 1, 5, 5, 7, 3, 2, 6, 8, 43, 6, 1, 5, 5, 7, 3, 1, 48, 12, 6, 3, 85, 29, 19, 1, 1, 255, 4, 2, 48, 0, 48, 73, 6, 3, 85, 29, 14, 4, 66, 4, 64, 43, 61, 184, 58, 190, 246, 117, 145, 150, 149, 52, 148, 50, 70, 171, 24, 140, 56, 164, 203, 105, 63, 65, 146, 183, 66, 114, 129, 170, 239, 140, 79, 13, 223, 192, 116, 206, 195, 234, 70, 154, 102, 164, 7, 77, 72, 8, 99, 67, 120, 61, 224, 48, 163, 84, 4, 250, 254, 27, 88, 242, 13, 99, 21, 48, 75, 6, 3, 85, 29, 35, 4, 68, 48, 66, 128, 64, 145, 115, 72, 248, 147, 3, 210, 10, 72, 90, 51, 130, 36, 255, 101, 123, 45, 151, 78, 158, 148, 254, 229, 16, 121, 180, 193, 131, 183, 127, 36, 88, 142, 33, 22, 114, 228, 61, 202, 29, 164, 17, 211, 221, 107, 244, 245, 206, 36, 16, 177, 96, 20, 74, 22, 17, 184, 132, 100, 119, 214, 222, 88, 253, 48, 76, 6, 3, 85, 29, 17, 4, 69, 48, 67, 130, 65, 42, 46, 97, 112, 112, 115, 46, 97, 114, 111, 46, 97, 50, 115, 57, 98, 56, 109, 50, 101, 52, 112, 55, 112, 48, 103, 46, 48, 109, 117, 56, 46, 119, 101, 115, 116, 117, 115, 51, 46, 97, 114, 111, 97, 112, 112, 45, 104, 99, 112, 46, 97, 122, 117, 114, 101, 45, 116, 101, 115, 116, 46, 110, 101, 116, 48, 13, 6, 9, 42, 134, 72, 134, 247, 13, 1, 1, 11, 5, 0, 3, 130, 1, 1, 0, 144, 63, 1, 9, 63, 203, 104, 158, 72, 190, 218, 173, 218, 174, 171, 175, 249, 99, 198, 163, 154, 183, 80, 70, 18, 90, 143, 16, 110, 40, 74, 15, 152, 104, 155, 119, 3, 189, 92, 182, 181, 193, 112, 193, 179, 171, 177, 195, 218, 45, 246, 105, 209, 209, 4, 147, 233, 109, 252, 165, 52, 200, 179, 47, 64, 202, 66, 155, 170, 48, 184, 165, 51, 160, 106, 229, 210, 44, 72, 109, 204, 32, 9, 38, 222, 130, 253, 176, 6, 246, 50, 104, 240, 45, 124, 77, 233, 84, 164, 109, 154, 110, 171, 77, 164, 253, 134, 225, 58, 131, 116, 48, 78, 1, 85, 150, 81, 149, 67, 171, 102, 216, 126, 229, 90, 56, 26, 33, 171, 124, 161, 209, 140, 187, 61, 34, 212, 215, 17, 134, 62, 6, 13, 208, 221, 55, 5, 187, 22, 63, 93, 164, 76, 124, 166, 239, 104, 210, 84, 78, 146, 114, 146, 113, 215, 97, 33, 13, 186, 135, 110, 182, 194, 44, 64, 215, 36, 43, 178, 21, 170, 171, 251, 180, 100, 40, 239, 189, 24, 174, 192, 195, 98, 108, 222, 96, 135, 172, 56, 114, 81, 213, 239, 84, 199, 27, 118, 56, 193, 239, 243, 191, 178, 97, 106, 201, 66, 252, 31, 190, 254, 107, 226, 184, 19, 188, 113, 44, 99, 24, 245, 52, 154, 97, 249, 178, 221, 232, 203, 63, 51, 4, 109, 167, 92, 246, 143, 188, 32, 32, 87, 98, 186, 5, 69, 17],
RawTBSCertificate: [48, 130, 2, 233, 160, 3, 2, 1, 2, 2, 8, 124, 87, 70, 122, 130, 6, 100, 82, 48, 13, 6, 9, 42, 134, 72, 134, 247, 13, 1, 1, 11, 5, 0, 48, 38, 49, 18, 48, 16, 6, 3, 85, 4, 11, 19, 9, 111, 112, 101, 110, 115, 104, 105, 102, 116, 49, 16, 48, 14, 6, 3, 85, 4, 3, 19, 7, 114, 111, 111, 116, 45, 99, 97, 48, 30, 23, 13, 50, 54, 49, 48, 50, 52, 50, 49, 48, 56, 48, 56, 90, 23, 13, 50, 55, 49, 48, 50, 52, 50, 49, 48, 56, 48, 56, 90, 48, 48, 49, 18, 48, 16, 6, 3, 85, 4, 10, 19, 9, 111, 112, 101, 110, 115, 104, 105, 102, 116, 49, 26, 48, 24, 6, 3, 85, 4, 3, 19, 17, 111, 112, 101, 110, 115, 104, 105, 102, 116, 45, 105, 110, 103, 114, 101, 115, 115, 48, 130, 1, 34, 48, 13, 6, 9, 42, 134, 72, 134, 247, 13, 1, 1, 1, 5, 0, 3, 130, 1, 15, 0, 48, 130, 1, 10, 2, 130, 1, 1, 0, 184, 6, 158, 64, 94, 24, 96, 40, 160, 154, 141, 196, 128, 87, 218, 59, 39, 71, 199, 234, 79, 240, 146, 24, 254, 106, 245, 45, 201, 136, 214, 64, 254, 110, 56, 158, 251, 103, 159, 125, 70, 9, 9, 192, 157, 185, 88, 45, 237, 209, 14, 238, 22, 11, 156, 42, 61, 172, 248, 175, 205, 29, 87, 44, 249, 69, 148, 194, 201, 88, 31, 36, 152, 175, 207, 120, 70, 231, 234, 239, 27, 120, 11, 188, 47, 126, 21, 187, 83, 61, 218, 253, 69, 150, 57, 242, 31, 129, 90, 78, 122, 162, 115, 193, 66, 231, 239, 171, 129, 241, 209, 248, 79, 198, 197, 15, 5, 20, 97, 114, 135, 82, 183, 39, 26, 12, 204, 167, 245, 131, 126, 229, 14, 152, 249, 66, 23, 145, 179, 201, 129, 84, 180, 47, 247, 191, 198, 219, 235, 14, 55, 141, 41, 250, 184, 135, 128, 115, 246, 220, 161, 224, 164, 119, 161, 199, 4, 69, 138, 127, 197, 71, 128, 167, 90, 212, 150, 2, 13, 143, 216, 191, 58, 228, 209, 93, 22, 58, 9, 137, 127, 203, 163, 74, 40, 217, 253, 186, 51, 153, 40, 52, 238, 73, 60, 57, 57, 108, 128, 2, 48, 134, 115, 108, 142, 65, 158, 113, 75, 25, 249, 189, 39, 130, 154, 24, 112, 209, 120, 229, 129, 223, 190, 147, 169, 232, 115, 83, 39, 47, 230, 201, 87, 216, 98, 161, 27, 95, 39, 237, 157, 102, 109, 222, 170, 149, 2, 3, 1, 0, 1, 163, 130, 1, 39, 48, 130, 1, 35, 48, 14, 6, 3, 85, 29, 15, 1, 1, 255, 4, 4, 3, 2, 5, 160, 48, 29, 6, 3, 85, 29, 37, 4, 22, 48, 20, 6, 8, 43, 6, 1, 5, 5, 7, 3, 2, 6, 8, 43, 6, 1, 5, 5, 7, 3, 1, 48, 12, 6, 3, 85, 29, 19, 1, 1, 255, 4, 2, 48, 0, 48, 73, 6, 3, 85, 29, 14, 4, 66, 4, 64, 43, 61, 184, 58, 190, 246, 117, 145, 150, 149, 52, 148, 50, 70, 171, 24, 140, 56, 164, 203, 105, 63, 65, 146, 183, 66, 114, 129, 170, 239, 140, 79, 13, 223, 192, 116, 206, 195, 234, 70, 154, 102, 164, 7, 77, 72, 8, 99, 67, 120, 61, 224, 48, 163, 84, 4, 250, 254, 27, 88, 242, 13, 99, 21, 48, 75, 6, 3, 85, 29, 35, 4, 68, 48, 66, 128, 64, 145, 115, 72, 248, 147, 3, 210, 10, 72, 90, 51, 130, 36, 255, 101, 123, 45, 151, 78, 158, 148, 254, 229, 16, 121, 180, 193, 131, 183, 127, 36, 88, 142, 33, 22, 114, 228, 61, 202, 29, 164, 17, 211, 221, 107, 244, 245, 206, 36, 16, 177, 96, 20, 74, 22, 17, 184, 132, 100, 119, 214, 222, 88, 253, 48, 76, 6, 3, 85, 29, 17, 4, 69, 48, 67, 130, 65, 42, 46, 97, 112, 112, 115, 46, 97, 114, 111, 46, 97, 50, 115, 57, 98, 56, 109, 50, 101, 52, 112, 55, 112, 48, 103, 46, 48, 109, 117, 56, 46, 119, 101, 115, 116, 117, 115, 51, 46, 97, 114, 111, 97, 112, 112, 45, 104, 99, 112, 46, 97, 122, 117, 114, 101, 45, 116, 101, 115, 116, 46, 110, 101, 116],
RawSubjectPublicKeyInfo: [48, 130, 1, 34, 48, 13, 6, 9, 42, 134, 72, 134, 247, 13, 1, 1, 1, 5, 0, 3, 130, 1, 15, 0, 48, 130, 1, 10, 2, 130, 1, 1, 0, 184, 6, 158, 64, 94, 24, 96, 40, 160, 154, 141, 196, 128, 87, 218, 59, 39, 71, 199, 234, 79, 240, 146, 24, 254, 106, 245, 45, 201, 136, 214, 64, 254, 110, 56, 158, 251, 103, 159, 125, 70, 9, 9, 192, 157, 185, 88, 45, 237, 209, 14, 238, 22, 11, 156, 42, 61, 172, 248, 175, 205, 29, 87, 44, 249, 69, 148, 194, 201, 88, 31, 36, 152, 175, 207, 120, 70, 231, 234, 239, 27, 120, 11, 188, 47, 126, 21, 187, 83, 61, 218, 253, 69, 150, 57, 242, 31, 129, 90, 78, 122, 162, 115, 193, 66, 231, 239, 171, 129, 241, 209, 248, 79, 198, 197, 15, 5, 20, 97, 114, 135, 82, 183, 39, 26, 12, 204, 167, 245, 131, 126, 229, 14, 152, 249, 66, 23, 145, 179, 201, 129, 84, 180, 47, 247, 191, 198, 219, 235, 14, 55, 141, 41, 250, 184, 135, 128, 115, 246, 220, 161, 224, 164, 119, 161, 199, 4, 69, 138, 127, 197, 71, 128, 167, 90, 212, 150, 2, 13, 143, 216, 191, 58, 228, 209, 93, 22, 58, 9, 137, 127, 203, 163, 74, 40, 217, 253, 186, 51, 153, 40, 52, 238, 73, 60, 57, 57, 108, 128, 2, 48, 134, 115, 108, 142, 65, 158, 113, 75, 25, 249, 189, 39, 130, 154, 24, 112, 209, 120, 229, 129, 223, 190, 147, 169, 232, 115, 83, 39, 47, 230, 201, 87, 216, 98, 161, 27, 95, 39, 237, 157, 102, 109, 222, 170, 149, 2, 3, 1, 0, 1],
RawSubject: [48, 48, 49, 18, 48, 16, 6, 3, 85, 4, 10, 19, 9, 111, 112, 101, 110, 115, 104, 105, 102, 116, 49, 26, 48, 24, 6, 3, 85, 4, 3, 19, 17, 111, 112, 101, 110, 115, 104, 105, 102, 116, 45, 105, 110, 103, 114, 101, 115, 115],
RawIssuer: [48, 38, 49, 18, 48, 16, 6, 3, 85, 4, 11, 19, 9, 111, 112, 101, 110, 115, 104, 105, 102, 116, 49, 16, 48, 14, 6, 3, 85, 4, 3, 19, 7, 114, 111, 111, 116, 45, 99, 97],
Signature: [144, 63, 1, 9, 63, 203, 104, 158, 72, 190, 218, 173, 218, 174, 171, 175, 249, 99, 198, 163, 154, 183, 80, 70, 18, 90, 143, 16, 110, 40, 74, 15, 152, 104, 155, 119, 3, 189, 92, 182, 181, 193, 112, 193, 179, 171, 177, 195, 218, 45, 246, 105, 209, 209, 4, 147, 233, 109, 252, 165, 52, 200, 179, 47, 64, 202, 66, 155, 170, 48, 184, 165, 51, 160, 106, 229, 210, 44, 72, 109, 204, 32, 9, 38, 222, 130, 253, 176, 6, 246, 50, 104, 240, 45, 124, 77, 233, 84, 164, 109, 154, 110, 171, 77, 164, 253, 134, 225, 58, 131, 116, 48, 78, 1, 85, 150, 81, 149, 67, 171, 102, 216, 126, 229, 90, 56, 26, 33, 171, 124, 161, 209, 140, 187, 61, 34, 212, 215, 17, 134, 62, 6, 13, 208, 221, 55, 5, 187, 22, 63, 93, 164, 76, 124, 166, 239, 104, 210, 84, 78, 146, 114, 146, 113, 215, 97, 33, 13, 186, 135, 110, 182, 194, 44, 64, 215, 36, 43, 178, 21, 170, 171, 251, 180, 100, 40, 239, 189, 24, 174, 192, 195, 98, 108, 222, 96, 135, 172, 56, 114, 81, 213, 239, 84, 199, 27, 118, 56, 193, 239, 243, 191, 178, 97, 106, 201, 66, 252, 31, 190, 254, 107, 226, 184, 19, 188, 113, 44, 99, 24, 245, 52, 154, 97, 249, 178, 221, 232, 203, 63, 51, 4, 109, 167, 92, 246, 143, 188, 32, 32, 87, 98, 186, 5, 69, 17],
SignatureAlgorithm: 4,
PublicKeyAlgorithm: 1,
PublicKey: <*rsa.PublicKey | 0x2b2e57843d70>{
N: {
neg: false,
abs: [2877128800215149205, 16629919685921545055, 13732506504615110447, 11103748926260740575, 11416989105297106818, 9223988340828442177, 2897202458616478060, 11766261893188694937, 15086238765284949963, 10809216967110507236, 9979912210168109780, 11664503868611822661, 3024932991847888604, 17852206095763650445, 1698336212534998063, 17691123283498957122, 9751057521779199143, 5748498642685419890, 4821085446021960184, 2270194980282856385, 5998191060056422898, 1979344939107882427, 11002240530182040303, 17961926253159653156, 4444200366252054316, 17136494472856312874, 5046575580329891885, 18333653388545728381, 18332734810168546880, 2830450700003086872, 11572718067588258363, 13260460152132755496],
},
E: 65537,
},
Version: 3,
SerialNumber: {
neg: false,
abs: [8959707475658302546],
},
Issuer: {
Country: nil,
Organization: nil,
OrganizationalUnit: ["openshift"],
Locality: nil,
Province: nil,
StreetAddress: nil,
PostalCode: nil,
SerialNumber: "",
CommonName: "root-ca",
Names: [
{
Type: [2, 5, 4, 11],
Value: <string>"openshift",
},
{
Type: [2, 5, 4, 3],
Value: <string>"root-ca",
},
],
ExtraNames: nil,
},
Subject: {
Country: nil,
Organization: ["openshift"],
OrganizationalUnit: nil,
Locality: nil,
Province: nil,
StreetAddress: nil,
PostalCode: nil,
SerialNumber: "",
CommonName: "openshift-ingress",
Names: [
{
Type: [2, 5, 4, 10],
Value: <string>"openshift",
},
{
Type: [2, 5, 4, 3],
Value: <string>"openshift-ingress",
},
],
ExtraNames: nil,
},
NotBefore: {wall: 0, ext: 63928472888, loc: nil},
NotAfter: {wall: 0, ext: 63960008888, loc: nil},
KeyUsage: 5,
Extensions: [
{
Id: [2, 5, 29, 15],
Critical: true,
Value: [3, 2, 5, 160],
},
{
Id: [2, 5, 29, 37],
Critical: false,
Value: [48, 20, 6, 8, 43, 6, 1, 5, 5, 7, 3, 2, 6, 8, 43, 6, 1, 5, 5, 7, 3, 1],
},
{Id: [2, 5, 29, 19], Critical: true, Value: [48, 0]},
{
Id: [2, 5, 29, 14],
Critical: false,
Value: [4, 64, 43, 61, 184, 58, 190, 246, 117, 145, 150, 149, 52, 148, 50, 70, 171, 24, 140, 56, 164, 203, 105, 63, 65, 146, 183, 66, 114, 129, 170, 239, 140, 79, 13, 223, 192, 116, 206, 195, 234, 70, 154, 102, 164, 7, 77, 72, 8, 99, 67, 120, 61, 224, 48, 163, 84, 4, 250, 254, 27, 88, 242, 13, 99, 21],
},
{
Id: [2, 5, 29, 35],
Critical: false,
Value: [48, 66, 128, 64, 145, 115, 72, 248, 147, 3, 210, 10, 72, 90, 51, 130, 36, 255, 101, 123, 45, 151, 78, 158, 148, 254, 229, 16, 121, 180, 193, 131, 183, 127, 36, 88, 142, 33, 22, 114, 228, 61, 202, 29, 164, 17, 211, 221, 107, 244, 245, 206, 36, 16, 177, 96, 20, 74, 22, 17, 184, 132, 100, 119, 214, 222, 88, 253],
},
{
Id: [2, 5, 29, 17],
Critical: false,
Value: "0C\x82A*.apps.aro.a2s9b8m2e4p7p0g.0mu8.westus3.aroapp-hcp.azure-test.net",
},
],
ExtraExtensions: nil,
UnhandledCriticalExtensions: nil,
ExtKeyUsage: [2, 1],
UnknownExtKeyUsage: nil,
BasicConstraintsValid: true,
IsCA: false,
MaxPathLen: -1,
MaxPathLenZero: false,
SubjectKeyId: [43, 61, 184, 58, 190, 246, 117, 145, 150, 149, 52, 148, 50, 70, 171, 24, 140, 56, 164, 203, 105, 63, 65, 146, 183, 66, 114, 129, 170, 239, 140, 79, 13, 223, 192, 116, 206, 195, 234, 70, 154, 102, 164, 7, 77, 72, 8, 99, 67, 120, 61, 224, 48, 163, 84, 4, 250, 254, 27, 88, 242, 13, 99, 21],
AuthorityKeyId: [145, 115, 72, 248, 147, 3, 210, 10, 72, 90, 51, 130, 36, 255, 101, 123, 45, 151, 78, 158, 148, 254, 229, 16, 121, 180, 193, 131, 183, 127, 36, 88, 142, 33, 22, 114, 228, 61, 202, 29, 164, 17, 211, 221, 107, 244, 245, 206, 36, 16, 177, 96, 20, 74, 22, 17, 184, 132, 100, 119, 214, 222, 88, 253],
OCSPServer: nil,
IssuingCertificateURL: nil,
DNSNames: [
"*.apps.aro.a2s9b8m2e4p7p0g.0mu8.westus3.aroapp-hcp.azure-test.net",
],
EmailAddresses: nil,
IPAddresses: nil,
URIs: nil,
PermittedDNSDomainsCritical: false,
PermittedDNSDomains: nil,
ExcludedDNSDomains: nil,
PermittedIPRanges: nil,
ExcludedIPRanges: nil,
PermittedEmailAddresses: nil,
ExcludedEmailAddresses: nil,
PermittedURIDomains: nil,
ExcludedURIDomains: nil,
CRLDistributionPoints: nil,
PolicyIdentifiers: nil,
Policies: nil,
InhibitAnyPolicy: 0,
InhibitAnyPolicyZero: false,
InhibitPolicyMapping: 0,
InhibitPolicyMappingZero: false,
RequireExplicitPolicy: 0,
RequireExplicitPolicyZero: false,
PolicyMappings: nil,
},
],
Err: <x509.UnknownAuthorityError>{
Cert: 0x2b2e55ba2008 (cyclic reference),
hintErr: nil,
hintCert: nil,
},
},
},
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:194]: failed to verify simple web app runs on cluster "cluster-candidate-4-23-2rcw86"
Unexpected error:
<*fmt.wrapError | 0x2b2e578984a0>:
[strict TLS verification] route was never reachable: Get "https://agnhost-e2e-sample-app-w86zx.apps.aro.a2s9b8m2e4p7p0g.0mu8.westus3.aroapp-hcp.azure-test.net": tls: failed to verify certificate: x509: certificate signed by unknown authority
{
msg: "[strict TLS verification] route was never reachable: Get \"https://agnhost-e2e-sample-app-w86zx.apps.aro.a2s9b8m2e4p7p0g.0mu8.westus3.aroapp-hcp.azure-test.net\": tls: failed to verify certificate: x509: certificate signed by unknown authority",
err: <*url.Error | 0x2b2e57370780>{
Op: "Get",
URL: "https://agnhost-e2e-sample-app-w86zx.apps.aro.a2s9b8m2e4p7p0g.0mu8.westus3.aroapp-hcp.azure-test.net",
Err: <*tls.CertificateVerificationError | 0x2b2e57370750>{
UnverifiedCertificates: [
{
Raw: [48, 130, 4, 1, 48, 130, 2, 233, 160, 3, 2, 1, 2, 2, 8, 124, 87, 70, 122, 130, 6, 100, 82, 48, 13, 6, 9, 42, 134, 72, 134, 247, 13, 1, 1, 11, 5, 0, 48, 38, 49, 18, 48, 16, 6, 3, 85, 4, 11, 19, 9, 111, 112, 101, 110, 115, 104, 105, 102, 116, 49, 16, 48, 14, 6, 3, 85, 4, 3, 19, 7, 114, 111, 111, 116, 45, 99, 97, 48, 30, 23, 13, 50, 54, 49, 48, 50, 52, 50, 49, 48, 56, 48, 56, 90, 23, 13, 50, 55, 49, 48, 50, 52, 50, 49, 48, 56, 48, 56, 90, 48, 48, 49, 18, 48, 16, 6, 3, 85, 4, 10, 19, 9, 111, 112, 101, 110, 115, 104, 105, 102, 116, 49, 26, 48, 24, 6, 3, 85, 4, 3, 19, 17, 111, 112, 101, 110, 115, 104, 105, 102, 116, 45, 105, 110, 103, 114, 101, 115, 115, 48, 130, 1, 34, 48, 13, 6, 9, 42, 134, 72, 134, 247, 13, 1, 1, 1, 5, 0, 3, 130, 1, 15, 0, 48, 130, 1, 10, 2, 130, 1, 1, 0, 184, 6, 158, 64, 94, 24, 96, 40, 160, 154, 141, 196, 128, 87, 218, 59, 39, 71, 199, 234, 79, 240, 146, 24, 254, 106, 245, 45, 201, 136, 214, 64, 254, 110, 56, 158, 251, 103, 159, 125, 70, 9, 9, 192, 157, 185, 88, 45, 237, 209, 14, 238, 22, 11, 156, 42, 61, 172, 248, 175, 205, 29, 87, 44, 249, 69, 148, 194, 201, 88, 31, 36, 152, 175, 207, 120, 70, 231, 234, 239, 27, 120, 11, 188, 47, 126, 21, 187, 83, 61, 218, 253, 69, 150, 57, 242, 31, 129, 90, 78, 122, 162, 115, 193, 66, 231, 239, 171, 129, 241, 209, 248, 79, 198, 197, 15, 5, 20, 97, 114, 135, 82, 183, 39, 26, 12, 204, 167, 245, 131, 126, 229, 14, 152, 249, 66, 23, 145, 179, 201, 129, 84, 180, 47, 247, 191, 198, 219, 235, 14, 55, 141, 41, 250, 184, 135, 128, 115, 246, 220, 161, 224, 164, 119, 161, 199, 4, 69, 138, 127, 197, 71, 128, 167, 90, 212, 150, 2, 13, 143, 216, 191, 58, 228, 209, 93, 22, 58, 9, 137, 127, 203, 163, 74, 40, 217, 253, 186, 51, 153, 40, 52, 238, 73, 60, 57, 57, 108, 128, 2, 48, 134, 115, 108, 142, 65, 158, 113, 75, 25, 249, 189, 39, 130, 154, 24, 112, 209, 120, 229, 129, 223, 190, 147, 169, 232, 115, 83, 39, 47, 230, 201, 87, 216, 98, 161, 27, 95, 39, 237, 157, 102, 109, 222, 170, 149, 2, 3, 1, 0, 1, 163, 130, 1, 39, 48, 130, 1, 35, 48, 14, 6, 3, 85, 29, 15, 1, 1, 255, 4, 4, 3, 2, 5, 160, 48, 29, 6, 3, 85, 29, 37, 4, 22, 48, 20, 6, 8, 43, 6, 1, 5, 5, 7, 3, 2, 6, 8, 43, 6, 1, 5, 5, 7, 3, 1, 48, 12, 6, 3, 85, 29, 19, 1, 1, 255, 4, 2, 48, 0, 48, 73, 6, 3, 85, 29, 14, 4, 66, 4, 64, 43, 61, 184, 58, 190, 246, 117, 145, 150, 149, 52, 148, 50, 70, 171, 24, 140, 56, 164, 203, 105, 63, 65, 146, 183, 66, 114, 129, 170, 239, 140, 79, 13, 223, 192, 116, 206, 195, 234, 70, 154, 102, 164, 7, 77, 72, 8, 99, 67, 120, 61, 224, 48, 163, 84, 4, 250, 254, 27, 88, 242, 13, 99, 21, 48, 75, 6, 3, 85, 29, 35, 4, 68, 48, 66, 128, 64, 145, 115, 72, 248, 147, 3, 210, 10, 72, 90, 51, 130, 36, 255, 101, 123, 45, 151, 78, 158, 148, 254, 229, 16, 121, 180, 193, 131, 183, 127, 36, 88, 142, 33, 22, 114, 228, 61, 202, 29, 164, 17, 211, 221, 107, 244, 245, 206, 36, 16, 177, 96, 20, 74, 22, 17, 184, 132, 100, 119, 214, 222, 88, 253, 48, 76, 6, 3, 85, 29, 17, 4, 69, 48, 67, 130, 65, 42, 46, 97, 112, 112, 115, 46, 97, 114, 111, 46, 97, 50, 115, 57, 98, 56, 109, 50, 101, 52, 112, 55, 112, 48, 103, 46, 48, 109, 117, 56, 46, 119, 101, 115, 116, 117, 115, 51, 46, 97, 114, 111, 97, 112, 112, 45, 104, 99, 112, 46, 97, 122, 117, 114, 101, 45, 116, 101, 115, 116, 46, 110, 101, 116, 48, 13, 6, 9, 42, 134, 72, 134, 247, 13, 1, 1, 11, 5, 0, 3, 130, 1, 1, 0, 144, 63, 1, 9, 63, 203, 104, 158, 72, 190, 218, 173, 218, 174, 171, 175, 249, 99, 198, 163, 154, 183, 80, 70, 18, 90, 143, 16, 110, 40, 74, 15, 152, 104, 155, 119, 3, 189, 92, 182, 181, 193, 112, 193, 179, 171, 177, 195, 218, 45, 246, 105, 209, 209, 4, 147, 233, 109, 252, 165, 52, 200, 179, 47, 64, 202, 66, 155, 170, 48, 184, 165, 51, 160, 106, 229, 210, 44, 72, 109, 204, 32, 9, 38, 222, 130, 253, 176, 6, 246, 50, 104, 240, 45, 124, 77, 233, 84, 164, 109, 154, 110, 171, 77, 164, 253, 134, 225, 58, 131, 116, 48, 78, 1, 85, 150, 81, 149, 67, 171, 102, 216, 126, 229, 90, 56, 26, 33, 171, 124, 161, 209, 140, 187, 61, 34, 212, 215, 17, 134, 62, 6, 13, 208, 221, 55, 5, 187, 22, 63, 93, 164, 76, 124, 166, 239, 104, 210, 84, 78, 146, 114, 146, 113, 215, 97, 33, 13, 186, 135, 110, 182, 194, 44, 64, 215, 36, 43, 178, 21, 170, 171, 251, 180, 100, 40, 239, 189, 24, 174, 192, 195, 98, 108, 222, 96, 135, 172, 56, 114, 81, 213, 239, 84, 199, 27, 118, 56, 193, 239, 243, 191, 178, 97, 106, 201, 66, 252, 31, 190, 254, 107, 226, 184, 19, 188, 113, 44, 99, 24, 245, 52, 154, 97, 249, 178, 221, 232, 203, 63, 51, 4, 109, 167, 92, 246, 143, 188, 32, 32, 87, 98, 186, 5, 69, 17],
RawTBSCertificate: [48, 130, 2, 233, 160, 3, 2, 1, 2, 2, 8, 124, 87, 70, 122, 130, 6, 100, 82, 48, 13, 6, 9, 42, 134, 72, 134, 247, 13, 1, 1, 11, 5, 0, 48, 38, 49, 18, 48, 16, 6, 3, 85, 4, 11, 19, 9, 111, 112, 101, 110, 115, 104, 105, 102, 116, 49, 16, 48, 14, 6, 3, 85, 4, 3, 19, 7, 114, 111, 111, 116, 45, 99, 97, 48, 30, 23, 13, 50, 54, 49, 48, 50, 52, 50, 49, 48, 56, 48, 56, 90, 23, 13, 50, 55, 49, 48, 50, 52, 50, 49, 48, 56, 48, 56, 90, 48, 48, 49, 18, 48, 16, 6, 3, 85, 4, 10, 19, 9, 111, 112, 101, 110, 115, 104, 105, 102, 116, 49, 26, 48, 24, 6, 3, 85, 4, 3, 19, 17, 111, 112, 101, 110, 115, 104, 105, 102, 116, 45, 105, 110, 103, 114, 101, 115, 115, 48, 130, 1, 34, 48, 13, 6, 9, 42, 134, 72, 134, 247, 13, 1, 1, 1, 5, 0, 3, 130, 1, 15, 0, 48, 130, 1, 10, 2, 130, 1, 1, 0, 184, 6, 158, 64, 94, 24, 96, 40, 160, 154, 141, 196, 128, 87, 218, 59, 39, 71, 199, 234, 79, 240, 146, 24, 254, 106, 245, 45, 201, 136, 214, 64, 254, 110, 56, 158, 251, 103, 159, 125, 70, 9, 9, 192, 157, 185, 88, 45, 237, 209, 14, 238, 22, 11, 156, 42, 61, 172, 248, 175, 205, 29, 87, 44, 249, 69, 148, 194, 201, 88, 31, 36, 152, 175, 207, 120, 70, 231, 234, 239, 27, 120, 11, 188, 47, 126, 21, 187, 83, 61, 218, 253, 69, 150, 57, 242, 31, 129, 90, 78, 122, 162, 115, 193, 66, 231, 239, 171, 129, 241, 209, 248, 79, 198, 197, 15, 5, 20, 97, 114, 135, 82, 183, 39, 26, 12, 204, 167, 245, 131, 126, 229, 14, 152, 249, 66, 23, 145, 179, 201, 129, 84, 180, 47, 247, 191, 198, 219, 235, 14, 55, 141, 41, 250, 184, 135, 128, 115, 246, 220, 161, 224, 164, 119, 161, 199, 4, 69, 138, 127, 197, 71, 128, 167, 90, 212, 150, 2, 13, 143, 216, 191, 58, 228, 209, 93, 22, 58, 9, 137, 127, 203, 163, 74, 40, 217, 253, 186, 51, 153, 40, 52, 238, 73, 60, 57, 57, 108, 128, 2, 48, 134, 115, 108, 142, 65, 158, 113, 75, 25, 249, 189, 39, 130, 154, 24, 112, 209, 120, 229, 129, 223, 190, 147, 169, 232, 115, 83, 39, 47, 230, 201, 87, 216, 98, 161, 27, 95, 39, 237, 157, 102, 109, 222, 170, 149, 2, 3, 1, 0, 1, 163, 130, 1, 39, 48, 130, 1, 35, 48, 14, 6, 3, 85, 29, 15, 1, 1, 255, 4, 4, 3, 2, 5, 160, 48, 29, 6, 3, 85, 29, 37, 4, 22, 48, 20, 6, 8, 43, 6, 1, 5, 5, 7, 3, 2, 6, 8, 43, 6, 1, 5, 5, 7, 3, 1, 48, 12, 6, 3, 85, 29, 19, 1, 1, 255, 4, 2, 48, 0, 48, 73, 6, 3, 85, 29, 14, 4, 66, 4, 64, 43, 61, 184, 58, 190, 246, 117, 145, 150, 149, 52, 148, 50, 70, 171, 24, 140, 56, 164, 203, 105, 63, 65, 146, 183, 66, 114, 129, 170, 239, 140, 79, 13, 223, 192, 116, 206, 195, 234, 70, 154, 102, 164, 7, 77, 72, 8, 99, 67, 120, 61, 224, 48, 163, 84, 4, 250, 254, 27, 88, 242, 13, 99, 21, 48, 75, 6, 3, 85, 29, 35, 4, 68, 48, 66, 128, 64, 145, 115, 72, 248, 147, 3, 210, 10, 72, 90, 51, 130, 36, 255, 101, 123, 45, 151, 78, 158, 148, 254, 229, 16, 121, 180, 193, 131, 183, 127, 36, 88, 142, 33, 22, 114, 228, 61, 202, 29, 164, 17, 211, 221, 107, 244, 245, 206, 36, 16, 177, 96, 20, 74, 22, 17, 184, 132, 100, 119, 214, 222, 88, 253, 48, 76, 6, 3, 85, 29, 17, 4, 69, 48, 67, 130, 65, 42, 46, 97, 112, 112, 115, 46, 97, 114, 111, 46, 97, 50, 115, 57, 98, 56, 109, 50, 101, 52, 112, 55, 112, 48, 103, 46, 48, 109, 117, 56, 46, 119, 101, 115, 116, 117, 115, 51, 46, 97, 114, 111, 97, 112, 112, 45, 104, 99, 112, 46, 97, 122, 117, 114, 101, 45, 116, 101, 115, 116, 46, 110, 101, 116],
RawSubjectPublicKeyInfo: [48, 130, 1, 34, 48, 13, 6, 9, 42, 134, 72, 134, 247, 13, 1, 1, 1, 5, 0, 3, 130, 1, 15, 0, 48, 130, 1, 10, 2, 130, 1, 1, 0, 184, 6, 158, 64, 94, 24, 96, 40, 160, 154, 141, 196, 128, 87, 218, 59, 39, 71, 199, 234, 79, 240, 146, 24, 254, 106, 245, 45, 201, 136, 214, 64, 254, 110, 56, 158, 251, 103, 159, 125, 70, 9, 9, 192, 157, 185, 88, 45, 237, 209, 14, 238, 22, 11, 156, 42, 61, 172, 248, 175, 205, 29, 87, 44, 249, 69, 148, 194, 201, 88, 31, 36, 152, 175, 207, 120, 70, 231, 234, 239, 27, 120, 11, 188, 47, 126, 21, 187, 83, 61, 218, 253, 69, 150, 57, 242, 31, 129, 90, 78, 122, 162, 115, 193, 66, 231, 239, 171, 129, 241, 209, 248, 79, 198, 197, 15, 5, 20, 97, 114, 135, 82, 183, 39, 26, 12, 204, 167, 245, 131, 126, 229, 14, 152, 249, 66, 23, 145, 179, 201, 129, 84, 180, 47, 247, 191, 198, 219, 235, 14, 55, 141, 41, 250, 184, 135, 128, 115, 246, 220, 161, 224, 164, 119, 161, 199, 4, 69, 138, 127, 197, 71, 128, 167, 90, 212, 150, 2, 13, 143, 216, 191, 58, 228, 209, 93, 22, 58, 9, 137, 127, 203, 163, 74, 40, 217, 253, 186, 51, 153, 40, 52, 238, 73, 60, 57, 57, 108, 128, 2, 48, 134, 115, 108, 142, 65, 158, 113, 75, 25, 249, 189, 39, 130, 154, 24, 112, 209, 120, 229, 129, 223, 190, 147, 169, 232, 115, 83, 39, 47, 230, 201, 87, 216, 98, 161, 27, 95, 39, 237, 157, 102, 109, 222, 170, 149, 2, 3, 1, 0, 1],
RawSubject: [48, 48, 49, 18, 48, 16, 6, 3, 85, 4, 10, 19, 9, 111, 112, 101, 110, 115, 104, 105, 102, 116, 49, 26, 48, 24, 6, 3, 85, 4, 3, 19, 17, 111, 112, 101, 110, 115, 104, 105, 102, 116, 45, 105, 110, 103, 114, 101, 115, 115],
RawIssuer: [48, 38, 49, 18, 48, 16, 6, 3, 85, 4, 11, 19, 9, 111, 112, 101, 110, 115, 104, 105, 102, 116, 49, 16, 48, 14, 6, 3, 85, 4, 3, 19, 7, 114, 111, 111, 116, 45, 99, 97],
Signature: [144, 63, 1, 9, 63, 203, 104, 158, 72, 190, 218, 173, 218, 174, 171, 175, 249, 99, 198, 163, 154, 183, 80, 70, 18, 90, 143, 16, 110, 40, 74, 15, 152, 104, 155, 119, 3, 189, 92, 182, 181, 193, 112, 193, 179, 171, 177, 195, 218, 45, 246, 105, 209, 209, 4, 147, 233, 109, 252, 165, 52, 200, 179, 47, 64, 202, 66, 155, 170, 48, 184, 165, 51, 160, 106, 229, 210, 44, 72, 109, 204, 32, 9, 38, 222, 130, 253, 176, 6, 246, 50, 104, 240, 45, 124, 77, 233, 84, 164, 109, 154, 110, 171, 77, 164, 253, 134, 225, 58, 131, 116, 48, 78, 1, 85, 150, 81, 149, 67, 171, 102, 216, 126, 229, 90, 56, 26, 33, 171, 124, 161, 209, 140, 187, 61, 34, 212, 215, 17, 134, 62, 6, 13, 208, 221, 55, 5, 187, 22, 63, 93, 164, 76, 124, 166, 239, 104, 210, 84, 78, 146, 114, 146, 113, 215, 97, 33, 13, 186, 135, 110, 182, 194, 44, 64, 215, 36, 43, 178, 21, 170, 171, 251, 180, 100, 40, 239, 189, 24, 174, 192, 195, 98, 108, 222, 96, 135, 172, 56, 114, 81, 213, 239, 84, 199, 27, 118, 56, 193, 239, 243, 191, 178, 97, 106, 201, 66, 252, 31, 190, 254, 107, 226, 184, 19, 188, 113, 44, 99, 24, 245, 52, 154, 97, 249, 178, 221, 232, 203, 63, 51, 4, 109, 167, 92, 246, 143, 188, 32, 32, 87, 98, 186, 5, 69, 17],
SignatureAlgorithm: 4,
PublicKeyAlgorithm: 1,
PublicKey: <*rsa.PublicKey | 0x2b2e57843d70>{
N: {
neg: false,
abs: [2877128800215149205, 16629919685921545055, 13732506504615110447, 11103748926260740575, 11416989105297106818, 9223988340828442177, 2897202458616478060, 11766261893188694937, 15086238765284949963, 10809216967110507236, 9979912210168109780, 11664503868611822661, 3024932991847888604, 17852206095763650445, 1698336212534998063, 17691123283498957122, 9751057521779199143, 5748498642685419890, 4821085446021960184, 2270194980282856385, 5998191060056422898, 1979344939107882427, 11002240530182040303, 17961926253159653156, 4444200366252054316, 17136494472856312874, 5046575580329891885, 18333653388545728381, 18332734810168546880, 2830450700003086872, 11572718067588258363, 13260460152132755496],
},
E: 65537,
},
Version: 3,
SerialNumber: {
neg: false,
abs: [8959707475658302546],
},
Issuer: {
Country: nil,
Organization: nil,
OrganizationalUnit: ["openshift"],
Locality: nil,
Province: nil,
StreetAddress: nil,
PostalCode: nil,
SerialNumber: "",
CommonName: "root-ca",
Names: [
{
Type: [2, 5, 4, 11],
Value: <string>"openshift",
},
{
Type: [2, 5, 4, 3],
Value: <string>"root-ca",
},
],
ExtraNames: nil,
},
Subject: {
Country: nil,
Organization: ["openshift"],
OrganizationalUnit: nil,
Locality: nil,
Province: nil,
StreetAddress: nil,
PostalCode: nil,
SerialNumber: "",
CommonName: "openshift-ingress",
Names: [
{
Type: [2, 5, 4, 10],
Value: <string>"openshift",
},
{
Type: [2, 5, 4, 3],
Value: <string>"openshift-ingress",
},
],
ExtraNames: nil,
},
NotBefore: {wall: 0, ext: 63928472888, loc: nil},
NotAfter: {wall: 0, ext: 63960008888, loc: nil},
KeyUsage: 5,
Extensions: [
{
Id: [2, 5, 29, 15],
Critical: true,
Value: [3, 2, 5, 160],
},
{
Id: [2, 5, 29, 37],
Critical: false,
Value: [48, 20, 6, 8, 43, 6, 1, 5, 5, 7, 3, 2, 6, 8, 43, 6, 1, 5, 5, 7, 3, 1],
},
{Id: [2, 5, 29, 19], Critical: true, Value: [48, 0]},
{
Id: [2, 5, 29, 14],
Critical: false,
Value: [4, 64, 43, 61, 184, 58, 190, 246, 117, 145, 150, 149, 52, 148, 50, 70, 171, 24, 140, 56, 164, 203, 105, 63, 65, 146, 183, 66, 114, 129, 170, 239, 140, 79, 13, 223, 192, 116, 206, 195, 234, 70, 154, 102, 164, 7, 77, 72, 8, 99, 67, 120, 61, 224, 48, 163, 84, 4, 250, 254, 27, 88, 242, 13, 99, 21],
},
{
Id: [2, 5, 29, 35],
Critical: false,
Value: [48, 66, 128, 64, 145, 115, 72, 248, 147, 3, 210, 10, 72, 90, 51, 130, 36, 255, 101, 123, 45, 151, 78, 158, 148, 254, 229, 16, 121, 180, 193, 131, 183, 127, 36, 88, 142, 33, 22, 114, 228, 61, 202, 29, 164, 17, 211, 221, 107, 244, 245, 206, 36, 16, 177, 96, 20, 74, 22, 17, 184, 132, 100, 119, 214, 222, 88, 253],
},
{
Id: [2, 5, 29, 17],
Critical: false,
Value: "0C\x82A*.apps.aro.a2s9b8m2e4p7p0g.0mu8.westus3.aroapp-hcp.azure-test.net",
},
],
ExtraExtensions: nil,
UnhandledCriticalExtensions: nil,
ExtKeyUsage: [2, 1],
UnknownExtKeyUsage: nil,
BasicConstraintsValid: true,
IsCA: false,
MaxPathLen: -1,
MaxPathLenZero: false,
SubjectKeyId: [43, 61, 184, 58, 190, 246, 117, 145, 150, 149, 52, 148, 50, 70, 171, 24, 140, 56, 164, 203, 105, 63, 65, 146, 183, 66, 114, 129, 170, 239, 140, 79, 13, 223, 192, 116, 206, 195, 234, 70, 154, 102, 164, 7, 77, 72, 8, 99, 67, 120, 61, 224, 48, 163, 84, 4, 250, 254, 27, 88, 242, 13, 99, 21],
AuthorityKeyId: [145, 115, 72, 248, 147, 3, 210, 10, 72, 90, 51, 130, 36, 255, 101, 123, 45, 151, 78, 158, 148, 254, 229, 16, 121, 180, 193, 131, 183, 127, 36, 88, 142, 33, 22, 114, 228, 61, 202, 29, 164, 17, 211, 221, 107, 244, 245, 206, 36, 16, 177, 96, 20, 74, 22, 17, 184, 132, 100, 119, 214, 222, 88, 253],
OCSPServer: nil,
IssuingCertificateURL: nil,
DNSNames: [
"*.apps.aro.a2s9b8m2e4p7p0g.0mu8.westus3.aroapp-hcp.azure-test.net",
],
EmailAddresses: nil,
IPAddresses: nil,
URIs: nil,
PermittedDNSDomainsCritical: false,
PermittedDNSDomains: nil,
ExcludedDNSDomains: nil,
PermittedIPRanges: nil,
ExcludedIPRanges: nil,
PermittedEmailAddresses: nil,
ExcludedEmailAddresses: nil,
PermittedURIDomains: nil,
ExcludedURIDomains: nil,
CRLDistributionPoints: nil,
PolicyIdentifiers: nil,
Policies: nil,
InhibitAnyPolicy: 0,
InhibitAnyPolicyZero: false,
InhibitPolicyMapping: 0,
InhibitPolicyMappingZero: false,
RequireExplicitPolicy: 0,
RequireExplicitPolicyZero: false,
PolicyMappings: nil,
},
],
Err: <x509.UnknownAuthorityError>{
Cert: 0x2b2e55ba2008 (cyclic reference),
hintErr: nil,
hintCert: nil,
},
},
},
}
occurredfail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:194]: failed to verify simple web app runs on cluster "cluster-candidate-4-23-wk4gjj"
Unexpected error:
<*fmt.wrapError | 0x30fb7a9c6d80>:
[strict TLS verification] route was never reachable: Get "https://agnhost-e2e-sample-app-25lgf.apps.aro.c9p0v7j9q7j4a1q.afld.westus3.aroapp-hcp.azure-test.net": tls: failed to verify certificate: x509: certificate signed by unknown authority
{
msg: "[strict TLS verification] route was never reachable: Get \"https://agnhost-e2e-sample-app-25lgf.apps.aro.c9p0v7j9q7j4a1q.afld.westus3.aroapp-hcp.azure-test.net\": tls: failed to verify certificate: x509: certificate signed by unknown authority",
err: <*url.Error | 0x30fb7cb9b320>{
Op: "Get",
URL: "https://agnhost-e2e-sample-app-25lgf.apps.aro.c9p0v7j9q7j4a1q.afld.westus3.aroapp-hcp.azure-test.net",
Err: <*tls.CertificateVerificationError | 0x30fb7cb9b2f0>{
UnverifiedCertificates: [
{
Raw: [48, 130, 4, 1, 48, 130, 2, 233, 160, 3, 2, 1, 2, 2, 8, 110, 83, 80, 98, 163, 186, 156, 29, 48, 13, 6, 9, 42, 134, 72, 134, 247, 13, 1, 1, 11, 5, 0, 48, 38, 49, 18, 48, 16, 6, 3, 85, 4, 11, 19, 9, 111, 112, 101, 110, 115, 104, 105, 102, 116, 49, 16, 48, 14, 6, 3, 85, 4, 3, 19, 7, 114, 111, 111, 116, 45, 99, 97, 48, 30, 23, 13, 50, 54, 49, 48, 50, 52, 49, 55, 50, 48, 48, 48, 90, 23, 13, 50, 55, 49, 48, 50, 52, 49, 55, 50, 48, 48, 48, 90, 48, 48, 49, 18, 48, 16, 6, 3, 85, 4, 10, 19, 9, 111, 112, 101, 110, 115, 104, 105, 102, 116, 49, 26, 48, 24, 6, 3, 85, 4, 3, 19, 17, 111, 112, 101, 110, 115, 104, 105, 102, 116, 45, 105, 110, 103, 114, 101, 115, 115, 48, 130, 1, 34, 48, 13, 6, 9, 42, 134, 72, 134, 247, 13, 1, 1, 1, 5, 0, 3, 130, 1, 15, 0, 48, 130, 1, 10, 2, 130, 1, 1, 0, 175, 180, 66, 13, 148, 254, 185, 112, 159, 180, 48, 58, 217, 14, 123, 23, 229, 28, 224, 241, 179, 210, 115, 71, 123, 94, 152, 56, 200, 179, 120, 80, 246, 68, 78, 194, 165, 17, 161, 64, 168, 40, 31, 243, 155, 207, 171, 17, 238, 179, 159, 199, 90, 32, 61, 243, 133, 233, 153, 229, 126, 69, 220, 234, 188, 36, 185, 6, 114, 77, 215, 59, 104, 34, 80, 95, 66, 69, 104, 166, 84, 17, 159, 234, 101, 109, 164, 153, 172, 156, 240, 134, 16, 200, 61, 179, 79, 164, 99, 227, 229, 46, 174, 192, 70, 194, 161, 207, 5, 78, 18, 21, 247, 101, 35, 19, 102, 159, 29, 233, 182, 86, 199, 53, 244, 175, 208, 228, 116, 49, 45, 21, 118, 178, 72, 104, 19, 188, 1, 131, 254, 127, 37, 139, 95, 60, 129, 254, 7, 18, 184, 51, 199, 217, 71, 107, 203, 113, 33, 123, 83, 246, 30, 234, 39, 214, 99, 207, 167, 116, 247, 66, 241, 15, 133, 22, 9, 238, 99, 105, 166, 167, 72, 142, 182, 229, 22, 51, 127, 84, 32, 214, 8, 185, 111, 104, 100, 152, 181, 219, 81, 90, 133, 133, 127, 114, 2, 134, 80, 248, 53, 178, 96, 3, 247, 147, 225, 186, 125, 75, 198, 237, 177, 202, 247, 71, 49, 31, 123, 47, 222, 47, 12, 75, 219, 147, 5, 27, 92, 161, 76, 204, 58, 211, 26, 120, 156, 19, 51, 136, 52, 116, 18, 166, 216, 205, 2, 3, 1, 0, 1, 163, 130, 1, 39, 48, 130, 1, 35, 48, 14, 6, 3, 85, 29, 15, 1, 1, 255, 4, 4, 3, 2, 5, 160, 48, 29, 6, 3, 85, 29, 37, 4, 22, 48, 20, 6, 8, 43, 6, 1, 5, 5, 7, 3, 2, 6, 8, 43, 6, 1, 5, 5, 7, 3, 1, 48, 12, 6, 3, 85, 29, 19, 1, 1, 255, 4, 2, 48, 0, 48, 73, 6, 3, 85, 29, 14, 4, 66, 4, 64, 184, 27, 111, 100, 185, 3, 60, 74, 232, 82, 187, 169, 146, 147, 185, 95, 74, 1, 30, 137, 225, 41, 223, 249, 128, 141, 254, 150, 25, 225, 242, 3, 235, 207, 147, 83, 107, 148, 237, 0, 226, 240, 42, 175, 47, 113, 123, 148, 128, 190, 12, 146, 119, 216, 235, 223, 47, 29, 124, 171, 8, 133, 245, 128, 48, 75, 6, 3, 85, 29, 35, 4, 68, 48, 66, 128, 64, 107, 35, 93, 140, 120, 74, 68, 242, 166, 39, 191, 159, 156, 255, 152, 139, 116, 160, 21, 220, 61, 249, 246, 136, 64, 148, 57, 252, 107, 221, 102, 126, 167, 207, 36, 223, 46, 112, 92, 229, 162, 35, 131, 186, 43, 139, 91, 12, 221, 184, 234, 124, 29, 20, 52, 38, 121, 7, 144, 253, 45, 84, 166, 41, 48, 76, 6, 3, 85, 29, 17, 4, 69, 48, 67, 130, 65, 42, 46, 97, 112, 112, 115, 46, 97, 114, 111, 46, 99, 57, 112, 48, 118, 55, 106, 57, 113, 55, 106, 52, 97, 49, 113, 46, 97, 102, 108, 100, 46, 119, 101, 115, 116, 117, 115, 51, 46, 97, 114, 111, 97, 112, 112, 45, 104, 99, 112, 46, 97, 122, 117, 114, 101, 45, 116, 101, 115, 116, 46, 110, 101, 116, 48, 13, 6, 9, 42, 134, 72, 134, 247, 13, 1, 1, 11, 5, 0, 3, 130, 1, 1, 0, 62, 37, 28, 104, 244, 85, 128, 159, 33, 132, 56, 114, 149, 155, 76, 150, 56, 190, 182, 145, 23, 80, 230, 251, 69, 191, 95, 185, 206, 12, 203, 223, 159, 176, 108, 31, 228, 30, 29, 147, 188, 224, 159, 222, 209, 80, 26, 111, 48, 5, 218, 228, 214, 243, 4, 174, 170, 173, 102, 5, 47, 112, 16, 26, 249, 76, 134, 179, 187, 39, 158, 241, 110, 27, 4, 56, 253, 35, 167, 85, 21, 75, 203, 60, 52, 83, 13, 153, 94, 185, 69, 153, 196, 55, 251, 226, 149, 80, 157, 57, 244, 94, 195, 179, 137, 174, 166, 85, 12, 161, 202, 212, 188, 14, 213, 25, 59, 74, 197, 148, 98, 52, 206, 23, 41, 197, 32, 214, 165, 60, 202, 205, 67, 25, 42, 38, 206, 141, 201, 237, 101, 166, 209, 131, 101, 208, 225, 123, 244, 235, 82, 50, 143, 145, 70, 217, 191, 27, 3, 130, 194, 10, 227, 129, 5, 165, 150, 45, 143, 154, 59, 167, 145, 196, 78, 140, 76, 209, 96, 166, 23, 54, 227, 85, 65, 210, 246, 64, 194, 155, 249, 210, 255, 119, 224, 229, 103, 89, 68, 18, 164, 119, 9, 207, 73, 245, 219, 68, 214, 3, 155, 42, 94, 212, 192, 93, 148, 231, 125, 114, 77, 118, 234, 177, 58, 169, 209, 85, 145, 75, 243, 82, 234, 209, 51, 198, 131, 119, 63, 14, 40, 242, 219, 78, 215, 35, 237, 91, 234, 2, 142, 86, 121, 121, 249, 150],
RawTBSCertificate: [48, 130, 2, 233, 160, 3, 2, 1, 2, 2, 8, 110, 83, 80, 98, 163, 186, 156, 29, 48, 13, 6, 9, 42, 134, 72, 134, 247, 13, 1, 1, 11, 5, 0, 48, 38, 49, 18, 48, 16, 6, 3, 85, 4, 11, 19, 9, 111, 112, 101, 110, 115, 104, 105, 102, 116, 49, 16, 48, 14, 6, 3, 85, 4, 3, 19, 7, 114, 111, 111, 116, 45, 99, 97, 48, 30, 23, 13, 50, 54, 49, 48, 50, 52, 49, 55, 50, 48, 48, 48, 90, 23, 13, 50, 55, 49, 48, 50, 52, 49, 55, 50, 48, 48, 48, 90, 48, 48, 49, 18, 48, 16, 6, 3, 85, 4, 10, 19, 9, 111, 112, 101, 110, 115, 104, 105, 102, 116, 49, 26, 48, 24, 6, 3, 85, 4, 3, 19, 17, 111, 112, 101, 110, 115, 104, 105, 102, 116, 45, 105, 110, 103, 114, 101, 115, 115, 48, 130, 1, 34, 48, 13, 6, 9, 42, 134, 72, 134, 247, 13, 1, 1, 1, 5, 0, 3, 130, 1, 15, 0, 48, 130, 1, 10, 2, 130, 1, 1, 0, 175, 180, 66, 13, 148, 254, 185, 112, 159, 180, 48, 58, 217, 14, 123, 23, 229, 28, 224, 241, 179, 210, 115, 71, 123, 94, 152, 56, 200, 179, 120, 80, 246, 68, 78, 194, 165, 17, 161, 64, 168, 40, 31, 243, 155, 207, 171, 17, 238, 179, 159, 199, 90, 32, 61, 243, 133, 233, 153, 229, 126, 69, 220, 234, 188, 36, 185, 6, 114, 77, 215, 59, 104, 34, 80, 95, 66, 69, 104, 166, 84, 17, 159, 234, 101, 109, 164, 153, 172, 156, 240, 134, 16, 200, 61, 179, 79, 164, 99, 227, 229, 46, 174, 192, 70, 194, 161, 207, 5, 78, 18, 21, 247, 101, 35, 19, 102, 159, 29, 233, 182, 86, 199, 53, 244, 175, 208, 228, 116, 49, 45, 21, 118, 178, 72, 104, 19, 188, 1, 131, 254, 127, 37, 139, 95, 60, 129, 254, 7, 18, 184, 51, 199, 217, 71, 107, 203, 113, 33, 123, 83, 246, 30, 234, 39, 214, 99, 207, 167, 116, 247, 66, 241, 15, 133, 22, 9, 238, 99, 105, 166, 167, 72, 142, 182, 229, 22, 51, 127, 84, 32, 214, 8, 185, 111, 104, 100, 152, 181, 219, 81, 90, 133, 133, 127, 114, 2, 134, 80, 248, 53, 178, 96, 3, 247, 147, 225, 186, 125, 75, 198, 237, 177, 202, 247, 71, 49, 31, 123, 47, 222, 47, 12, 75, 219, 147, 5, 27, 92, 161, 76, 204, 58, 211, 26, 120, 156, 19, 51, 136, 52, 116, 18, 166, 216, 205, 2, 3, 1, 0, 1, 163, 130, 1, 39, 48, 130, 1, 35, 48, 14, 6, 3, 85, 29, 15, 1, 1, 255, 4, 4, 3, 2, 5, 160, 48, 29, 6, 3, 85, 29, 37, 4, 22, 48, 20, 6, 8, 43, 6, 1, 5, 5, 7, 3, 2, 6, 8, 43, 6, 1, 5, 5, 7, 3, 1, 48, 12, 6, 3, 85, 29, 19, 1, 1, 255, 4, 2, 48, 0, 48, 73, 6, 3, 85, 29, 14, 4, 66, 4, 64, 184, 27, 111, 100, 185, 3, 60, 74, 232, 82, 187, 169, 146, 147, 185, 95, 74, 1, 30, 137, 225, 41, 223, 249, 128, 141, 254, 150, 25, 225, 242, 3, 235, 207, 147, 83, 107, 148, 237, 0, 226, 240, 42, 175, 47, 113, 123, 148, 128, 190, 12, 146, 119, 216, 235, 223, 47, 29, 124, 171, 8, 133, 245, 128, 48, 75, 6, 3, 85, 29, 35, 4, 68, 48, 66, 128, 64, 107, 35, 93, 140, 120, 74, 68, 242, 166, 39, 191, 159, 156, 255, 152, 139, 116, 160, 21, 220, 61, 249, 246, 136, 64, 148, 57, 252, 107, 221, 102, 126, 167, 207, 36, 223, 46, 112, 92, 229, 162, 35, 131, 186, 43, 139, 91, 12, 221, 184, 234, 124, 29, 20, 52, 38, 121, 7, 144, 253, 45, 84, 166, 41, 48, 76, 6, 3, 85, 29, 17, 4, 69, 48, 67, 130, 65, 42, 46, 97, 112, 112, 115, 46, 97, 114, 111, 46, 99, 57, 112, 48, 118, 55, 106, 57, 113, 55, 106, 52, 97, 49, 113, 46, 97, 102, 108, 100, 46, 119, 101, 115, 116, 117, 115, 51, 46, 97, 114, 111, 97, 112, 112, 45, 104, 99, 112, 46, 97, 122, 117, 114, 101, 45, 116, 101, 115, 116, 46, 110, 101, 116],
RawSubjectPublicKeyInfo: [48, 130, 1, 34, 48, 13, 6, 9, 42, 134, 72, 134, 247, 13, 1, 1, 1, 5, 0, 3, 130, 1, 15, 0, 48, 130, 1, 10, 2, 130, 1, 1, 0, 175, 180, 66, 13, 148, 254, 185, 112, 159, 180, 48, 58, 217, 14, 123, 23, 229, 28, 224, 241, 179, 210, 115, 71, 123, 94, 152, 56, 200, 179, 120, 80, 246, 68, 78, 194, 165, 17, 161, 64, 168, 40, 31, 243, 155, 207, 171, 17, 238, 179, 159, 199, 90, 32, 61, 243, 133, 233, 153, 229, 126, 69, 220, 234, 188, 36, 185, 6, 114, 77, 215, 59, 104, 34, 80, 95, 66, 69, 104, 166, 84, 17, 159, 234, 101, 109, 164, 153, 172, 156, 240, 134, 16, 200, 61, 179, 79, 164, 99, 227, 229, 46, 174, 192, 70, 194, 161, 207, 5, 78, 18, 21, 247, 101, 35, 19, 102, 159, 29, 233, 182, 86, 199, 53, 244, 175, 208, 228, 116, 49, 45, 21, 118, 178, 72, 104, 19, 188, 1, 131, 254, 127, 37, 139, 95, 60, 129, 254, 7, 18, 184, 51, 199, 217, 71, 107, 203, 113, 33, 123, 83, 246, 30, 234, 39, 214, 99, 207, 167, 116, 247, 66, 241, 15, 133, 22, 9, 238, 99, 105, 166, 167, 72, 142, 182, 229, 22, 51, 127, 84, 32, 214, 8, 185, 111, 104, 100, 152, 181, 219, 81, 90, 133, 133, 127, 114, 2, 134, 80, 248, 53, 178, 96, 3, 247, 147, 225, 186, 125, 75, 198, 237, 177, 202, 247, 71, 49, 31, 123, 47, 222, 47, 12, 75, 219, 147, 5, 27, 92, 161, 76, 204, 58, 211, 26, 120, 156, 19, 51, 136, 52, 116, 18, 166, 216, 205, 2, 3, 1, 0, 1],
RawSubject: [48, 48, 49, 18, 48, 16, 6, 3, 85, 4, 10, 19, 9, 111, 112, 101, 110, 115, 104, 105, 102, 116, 49, 26, 48, 24, 6, 3, 85, 4, 3, 19, 17, 111, 112, 101, 110, 115, 104, 105, 102, 116, 45, 105, 110, 103, 114, 101, 115, 115],
RawIssuer: [48, 38, 49, 18, 48, 16, 6, 3, 85, 4, 11, 19, 9, 111, 112, 101, 110, 115, 104, 105, 102, 116, 49, 16, 48, 14, 6, 3, 85, 4, 3, 19, 7, 114, 111, 111, 116, 45, 99, 97],
Signature: [62, 37, 28, 104, 244, 85, 128, 159, 33, 132, 56, 114, 149, 155, 76, 150, 56, 190, 182, 145, 23, 80, 230, 251, 69, 191, 95, 185, 206, 12, 203, 223, 159, 176, 108, 31, 228, 30, 29, 147, 188, 224, 159, 222, 209, 80, 26, 111, 48, 5, 218, 228, 214, 243, 4, 174, 170, 173, 102, 5, 47, 112, 16, 26, 249, 76, 134, 179, 187, 39, 158, 241, 110, 27, 4, 56, 253, 35, 167, 85, 21, 75, 203, 60, 52, 83, 13, 153, 94, 185, 69, 153, 196, 55, 251, 226, 149, 80, 157, 57, 244, 94, 195, 179, 137, 174, 166, 85, 12, 161, 202, 212, 188, 14, 213, 25, 59, 74, 197, 148, 98, 52, 206, 23, 41, 197, 32, 214, 165, 60, 202, 205, 67, 25, 42, 38, 206, 141, 201, 237, 101, 166, 209, 131, 101, 208, 225, 123, 244, 235, 82, 50, 143, 145, 70, 217, 191, 27, 3, 130, 194, 10, 227, 129, 5, 165, 150, 45, 143, 154, 59, 167, 145, 196, 78, 140, 76, 209, 96, 166, 23, 54, 227, 85, 65, 210, 246, 64, 194, 155, 249, 210, 255, 119, 224, 229, 103, 89, 68, 18, 164, 119, 9, 207, 73, 245, 219, 68, 214, 3, 155, 42, 94, 212, 192, 93, 148, 231, 125, 114, 77, 118, 234, 177, 58, 169, 209, 85, 145, 75, 243, 82, 234, 209, 51, 198, 131, 119, 63, 14, 40, 242, 219, 78, 215, 35, 237, 91, 234, 2, 142, 86, 121, 121, 249, 150],
SignatureAlgorithm: 4,
PublicKeyAlgorithm: 1,
PublicKey: <*rsa.PublicKey | 0x30fb7a9895b0>{
N: {
neg: false,
abs: [3713275565900749005, 5533862720488119315, 886043176200789153, 17818264461995138607, 16265450768615453130, 5834472357489801107, 5862144673391444614, 628656117149906395, 13178964295004987606, 715618696217184398, 12066541168516629782, 6050057140458382287, 14400619811794133371, 6862502860246136883, 1422013248764323211, 8372522752467421288, 13138907997452357860, 17826693266249031145, 5098815638656127509, 5738821655630622400, 12438080729488309683, 6057798802862875801, 7503648299216627878, 13557164215630681915, 9649412887563066602, 17200267080417426931, 12116969928591911697, 17745395029646614848, 8889710084128733264, 16509317662742049607, 11507875977146301207, 12660817078547036528],
},
E: 65537,
},
Version: 3,
SerialNumber: {
neg: false,
abs: [7949786151822990365],
},
Issuer: {
Country: nil,
Organization: nil,
OrganizationalUnit: ["openshift"],
Locality: nil,
Province: nil,
StreetAddress: nil,
PostalCode: nil,
SerialNumber: "",
CommonName: "root-ca",
Names: [
{
Type: [2, 5, 4, 11],
Value: <string>"openshift",
},
{
Type: [2, 5, 4, 3],
Value: <string>"root-ca",
},
],
ExtraNames: nil,
},
Subject: {
Country: nil,
Organization: ["openshift"],
OrganizationalUnit: nil,
Locality: nil,
Province: nil,
StreetAddress: nil,
PostalCode: nil,
SerialNumber: "",
CommonName: "openshift-ingress",
Names: [
{
Type: [2, 5, 4, 10],
Value: <string>"openshift",
},
{
Type: [2, 5, 4, 3],
Value: <string>"openshift-ingress",
},
],
ExtraNames: nil,
},
NotBefore: {wall: 0, ext: 63928459200, loc: nil},
NotAfter: {wall: 0, ext: 63959995200, loc: nil},
KeyUsage: 5,
Extensions: [
{
Id: [2, 5, 29, 15],
Critical: true,
Value: [3, 2, 5, 160],
},
{
Id: [2, 5, 29, 37],
Critical: false,
Value: [48, 20, 6, 8, 43, 6, 1, 5, 5, 7, 3, 2, 6, 8, 43, 6, 1, 5, 5, 7, 3, 1],
},
{Id: [2, 5, 29, 19], Critical: true, Value: [48, 0]},
{
Id: [2, 5, 29, 14],
Critical: false,
Value: [4, 64, 184, 27, 111, 100, 185, 3, 60, 74, 232, 82, 187, 169, 146, 147, 185, 95, 74, 1, 30, 137, 225, 41, 223, 249, 128, 141, 254, 150, 25, 225, 242, 3, 235, 207, 147, 83, 107, 148, 237, 0, 226, 240, 42, 175, 47, 113, 123, 148, 128, 190, 12, 146, 119, 216, 235, 223, 47, 29, 124, 171, 8, 133, 245, 128],
},
{
Id: [2, 5, 29, 35],
Critical: false,
Value: [48, 66, 128, 64, 107, 35, 93, 140, 120, 74, 68, 242, 166, 39, 191, 159, 156, 255, 152, 139, 116, 160, 21, 220, 61, 249, 246, 136, 64, 148, 57, 252, 107, 221, 102, 126, 167, 207, 36, 223, 46, 112, 92, 229, 162, 35, 131, 186, 43, 139, 91, 12, 221, 184, 234, 124, 29, 20, 52, 38, 121, 7, 144, 253, 45, 84, 166, 41],
},
{
Id: [2, 5, 29, 17],
Critical: false,
Value: "0C\x82A*.apps.aro.c9p0v7j9q7j4a1q.afld.westus3.aroapp-hcp.azure-test.net",
},
],
ExtraExtensions: nil,
UnhandledCriticalExtensions: nil,
ExtKeyUsage: [2, 1],
UnknownExtKeyUsage: nil,
BasicConstraintsValid: true,
IsCA: false,
MaxPathLen: -1,
MaxPathLenZero: false,
SubjectKeyId: [184, 27, 111, 100, 185, 3, 60, 74, 232, 82, 187, 169, 146, 147, 185, 95, 74, 1, 30, 137, 225, 41, 223, 249, 128, 141, 254, 150, 25, 225, 242, 3, 235, 207, 147, 83, 107, 148, 237, 0, 226, 240, 42, 175, 47, 113, 123, 148, 128, 190, 12, 146, 119, 216, 235, 223, 47, 29, 124, 171, 8, 133, 245, 128],
AuthorityKeyId: [107, 35, 93, 140, 120, 74, 68, 242, 166, 39, 191, 159, 156, 255, 152, 139, 116, 160, 21, 220, 61, 249, 246, 136, 64, 148, 57, 252, 107, 221, 102, 126, 167, 207, 36, 223, 46, 112, 92, 229, 162, 35, 131, 186, 43, 139, 91, 12, 221, 184, 234, 124, 29, 20, 52, 38, 121, 7, 144, 253, 45, 84, 166, 41],
OCSPServer: nil,
IssuingCertificateURL: nil,
DNSNames: [
"*.apps.aro.c9p0v7j9q7j4a1q.afld.westus3.aroapp-hcp.azure-test.net",
],
EmailAddresses: nil,
IPAddresses: nil,
URIs: nil,
PermittedDNSDomainsCritical: false,
PermittedDNSDomains: nil,
ExcludedDNSDomains: nil,
PermittedIPRanges: nil,
ExcludedIPRanges: nil,
PermittedEmailAddresses: nil,
ExcludedEmailAddresses: nil,
PermittedURIDomains: nil,
ExcludedURIDomains: nil,
CRLDistributionPoints: nil,
PolicyIdentifiers: nil,
Policies: nil,
InhibitAnyPolicy: 0,
InhibitAnyPolicyZero: false,
InhibitPolicyMapping: 0,
InhibitPolicyMappingZero: false,
RequireExplicitPolicy: 0,
RequireExplicitPolicyZero: false,
PolicyMappings: nil,
},
],
Err: <x509.UnknownAuthorityError>{
Cert: 0x30fb7a8f2c08 (cyclic reference),
hintErr: nil,
hintCert: nil,
},
},
},
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:194]: failed to verify simple web app runs on cluster "cluster-candidate-4-23-wk4gjj"
Unexpected error:
<*fmt.wrapError | 0x30fb7a9c6d80>:
[strict TLS verification] route was never reachable: Get "https://agnhost-e2e-sample-app-25lgf.apps.aro.c9p0v7j9q7j4a1q.afld.westus3.aroapp-hcp.azure-test.net": tls: failed to verify certificate: x509: certificate signed by unknown authority
{
msg: "[strict TLS verification] route was never reachable: Get \"https://agnhost-e2e-sample-app-25lgf.apps.aro.c9p0v7j9q7j4a1q.afld.westus3.aroapp-hcp.azure-test.net\": tls: failed to verify certificate: x509: certificate signed by unknown authority",
err: <*url.Error | 0x30fb7cb9b320>{
Op: "Get",
URL: "https://agnhost-e2e-sample-app-25lgf.apps.aro.c9p0v7j9q7j4a1q.afld.westus3.aroapp-hcp.azure-test.net",
Err: <*tls.CertificateVerificationError | 0x30fb7cb9b2f0>{
UnverifiedCertificates: [
{
Raw: [48, 130, 4, 1, 48, 130, 2, 233, 160, 3, 2, 1, 2, 2, 8, 110, 83, 80, 98, 163, 186, 156, 29, 48, 13, 6, 9, 42, 134, 72, 134, 247, 13, 1, 1, 11, 5, 0, 48, 38, 49, 18, 48, 16, 6, 3, 85, 4, 11, 19, 9, 111, 112, 101, 110, 115, 104, 105, 102, 116, 49, 16, 48, 14, 6, 3, 85, 4, 3, 19, 7, 114, 111, 111, 116, 45, 99, 97, 48, 30, 23, 13, 50, 54, 49, 48, 50, 52, 49, 55, 50, 48, 48, 48, 90, 23, 13, 50, 55, 49, 48, 50, 52, 49, 55, 50, 48, 48, 48, 90, 48, 48, 49, 18, 48, 16, 6, 3, 85, 4, 10, 19, 9, 111, 112, 101, 110, 115, 104, 105, 102, 116, 49, 26, 48, 24, 6, 3, 85, 4, 3, 19, 17, 111, 112, 101, 110, 115, 104, 105, 102, 116, 45, 105, 110, 103, 114, 101, 115, 115, 48, 130, 1, 34, 48, 13, 6, 9, 42, 134, 72, 134, 247, 13, 1, 1, 1, 5, 0, 3, 130, 1, 15, 0, 48, 130, 1, 10, 2, 130, 1, 1, 0, 175, 180, 66, 13, 148, 254, 185, 112, 159, 180, 48, 58, 217, 14, 123, 23, 229, 28, 224, 241, 179, 210, 115, 71, 123, 94, 152, 56, 200, 179, 120, 80, 246, 68, 78, 194, 165, 17, 161, 64, 168, 40, 31, 243, 155, 207, 171, 17, 238, 179, 159, 199, 90, 32, 61, 243, 133, 233, 153, 229, 126, 69, 220, 234, 188, 36, 185, 6, 114, 77, 215, 59, 104, 34, 80, 95, 66, 69, 104, 166, 84, 17, 159, 234, 101, 109, 164, 153, 172, 156, 240, 134, 16, 200, 61, 179, 79, 164, 99, 227, 229, 46, 174, 192, 70, 194, 161, 207, 5, 78, 18, 21, 247, 101, 35, 19, 102, 159, 29, 233, 182, 86, 199, 53, 244, 175, 208, 228, 116, 49, 45, 21, 118, 178, 72, 104, 19, 188, 1, 131, 254, 127, 37, 139, 95, 60, 129, 254, 7, 18, 184, 51, 199, 217, 71, 107, 203, 113, 33, 123, 83, 246, 30, 234, 39, 214, 99, 207, 167, 116, 247, 66, 241, 15, 133, 22, 9, 238, 99, 105, 166, 167, 72, 142, 182, 229, 22, 51, 127, 84, 32, 214, 8, 185, 111, 104, 100, 152, 181, 219, 81, 90, 133, 133, 127, 114, 2, 134, 80, 248, 53, 178, 96, 3, 247, 147, 225, 186, 125, 75, 198, 237, 177, 202, 247, 71, 49, 31, 123, 47, 222, 47, 12, 75, 219, 147, 5, 27, 92, 161, 76, 204, 58, 211, 26, 120, 156, 19, 51, 136, 52, 116, 18, 166, 216, 205, 2, 3, 1, 0, 1, 163, 130, 1, 39, 48, 130, 1, 35, 48, 14, 6, 3, 85, 29, 15, 1, 1, 255, 4, 4, 3, 2, 5, 160, 48, 29, 6, 3, 85, 29, 37, 4, 22, 48, 20, 6, 8, 43, 6, 1, 5, 5, 7, 3, 2, 6, 8, 43, 6, 1, 5, 5, 7, 3, 1, 48, 12, 6, 3, 85, 29, 19, 1, 1, 255, 4, 2, 48, 0, 48, 73, 6, 3, 85, 29, 14, 4, 66, 4, 64, 184, 27, 111, 100, 185, 3, 60, 74, 232, 82, 187, 169, 146, 147, 185, 95, 74, 1, 30, 137, 225, 41, 223, 249, 128, 141, 254, 150, 25, 225, 242, 3, 235, 207, 147, 83, 107, 148, 237, 0, 226, 240, 42, 175, 47, 113, 123, 148, 128, 190, 12, 146, 119, 216, 235, 223, 47, 29, 124, 171, 8, 133, 245, 128, 48, 75, 6, 3, 85, 29, 35, 4, 68, 48, 66, 128, 64, 107, 35, 93, 140, 120, 74, 68, 242, 166, 39, 191, 159, 156, 255, 152, 139, 116, 160, 21, 220, 61, 249, 246, 136, 64, 148, 57, 252, 107, 221, 102, 126, 167, 207, 36, 223, 46, 112, 92, 229, 162, 35, 131, 186, 43, 139, 91, 12, 221, 184, 234, 124, 29, 20, 52, 38, 121, 7, 144, 253, 45, 84, 166, 41, 48, 76, 6, 3, 85, 29, 17, 4, 69, 48, 67, 130, 65, 42, 46, 97, 112, 112, 115, 46, 97, 114, 111, 46, 99, 57, 112, 48, 118, 55, 106, 57, 113, 55, 106, 52, 97, 49, 113, 46, 97, 102, 108, 100, 46, 119, 101, 115, 116, 117, 115, 51, 46, 97, 114, 111, 97, 112, 112, 45, 104, 99, 112, 46, 97, 122, 117, 114, 101, 45, 116, 101, 115, 116, 46, 110, 101, 116, 48, 13, 6, 9, 42, 134, 72, 134, 247, 13, 1, 1, 11, 5, 0, 3, 130, 1, 1, 0, 62, 37, 28, 104, 244, 85, 128, 159, 33, 132, 56, 114, 149, 155, 76, 150, 56, 190, 182, 145, 23, 80, 230, 251, 69, 191, 95, 185, 206, 12, 203, 223, 159, 176, 108, 31, 228, 30, 29, 147, 188, 224, 159, 222, 209, 80, 26, 111, 48, 5, 218, 228, 214, 243, 4, 174, 170, 173, 102, 5, 47, 112, 16, 26, 249, 76, 134, 179, 187, 39, 158, 241, 110, 27, 4, 56, 253, 35, 167, 85, 21, 75, 203, 60, 52, 83, 13, 153, 94, 185, 69, 153, 196, 55, 251, 226, 149, 80, 157, 57, 244, 94, 195, 179, 137, 174, 166, 85, 12, 161, 202, 212, 188, 14, 213, 25, 59, 74, 197, 148, 98, 52, 206, 23, 41, 197, 32, 214, 165, 60, 202, 205, 67, 25, 42, 38, 206, 141, 201, 237, 101, 166, 209, 131, 101, 208, 225, 123, 244, 235, 82, 50, 143, 145, 70, 217, 191, 27, 3, 130, 194, 10, 227, 129, 5, 165, 150, 45, 143, 154, 59, 167, 145, 196, 78, 140, 76, 209, 96, 166, 23, 54, 227, 85, 65, 210, 246, 64, 194, 155, 249, 210, 255, 119, 224, 229, 103, 89, 68, 18, 164, 119, 9, 207, 73, 245, 219, 68, 214, 3, 155, 42, 94, 212, 192, 93, 148, 231, 125, 114, 77, 118, 234, 177, 58, 169, 209, 85, 145, 75, 243, 82, 234, 209, 51, 198, 131, 119, 63, 14, 40, 242, 219, 78, 215, 35, 237, 91, 234, 2, 142, 86, 121, 121, 249, 150],
RawTBSCertificate: [48, 130, 2, 233, 160, 3, 2, 1, 2, 2, 8, 110, 83, 80, 98, 163, 186, 156, 29, 48, 13, 6, 9, 42, 134, 72, 134, 247, 13, 1, 1, 11, 5, 0, 48, 38, 49, 18, 48, 16, 6, 3, 85, 4, 11, 19, 9, 111, 112, 101, 110, 115, 104, 105, 102, 116, 49, 16, 48, 14, 6, 3, 85, 4, 3, 19, 7, 114, 111, 111, 116, 45, 99, 97, 48, 30, 23, 13, 50, 54, 49, 48, 50, 52, 49, 55, 50, 48, 48, 48, 90, 23, 13, 50, 55, 49, 48, 50, 52, 49, 55, 50, 48, 48, 48, 90, 48, 48, 49, 18, 48, 16, 6, 3, 85, 4, 10, 19, 9, 111, 112, 101, 110, 115, 104, 105, 102, 116, 49, 26, 48, 24, 6, 3, 85, 4, 3, 19, 17, 111, 112, 101, 110, 115, 104, 105, 102, 116, 45, 105, 110, 103, 114, 101, 115, 115, 48, 130, 1, 34, 48, 13, 6, 9, 42, 134, 72, 134, 247, 13, 1, 1, 1, 5, 0, 3, 130, 1, 15, 0, 48, 130, 1, 10, 2, 130, 1, 1, 0, 175, 180, 66, 13, 148, 254, 185, 112, 159, 180, 48, 58, 217, 14, 123, 23, 229, 28, 224, 241, 179, 210, 115, 71, 123, 94, 152, 56, 200, 179, 120, 80, 246, 68, 78, 194, 165, 17, 161, 64, 168, 40, 31, 243, 155, 207, 171, 17, 238, 179, 159, 199, 90, 32, 61, 243, 133, 233, 153, 229, 126, 69, 220, 234, 188, 36, 185, 6, 114, 77, 215, 59, 104, 34, 80, 95, 66, 69, 104, 166, 84, 17, 159, 234, 101, 109, 164, 153, 172, 156, 240, 134, 16, 200, 61, 179, 79, 164, 99, 227, 229, 46, 174, 192, 70, 194, 161, 207, 5, 78, 18, 21, 247, 101, 35, 19, 102, 159, 29, 233, 182, 86, 199, 53, 244, 175, 208, 228, 116, 49, 45, 21, 118, 178, 72, 104, 19, 188, 1, 131, 254, 127, 37, 139, 95, 60, 129, 254, 7, 18, 184, 51, 199, 217, 71, 107, 203, 113, 33, 123, 83, 246, 30, 234, 39, 214, 99, 207, 167, 116, 247, 66, 241, 15, 133, 22, 9, 238, 99, 105, 166, 167, 72, 142, 182, 229, 22, 51, 127, 84, 32, 214, 8, 185, 111, 104, 100, 152, 181, 219, 81, 90, 133, 133, 127, 114, 2, 134, 80, 248, 53, 178, 96, 3, 247, 147, 225, 186, 125, 75, 198, 237, 177, 202, 247, 71, 49, 31, 123, 47, 222, 47, 12, 75, 219, 147, 5, 27, 92, 161, 76, 204, 58, 211, 26, 120, 156, 19, 51, 136, 52, 116, 18, 166, 216, 205, 2, 3, 1, 0, 1, 163, 130, 1, 39, 48, 130, 1, 35, 48, 14, 6, 3, 85, 29, 15, 1, 1, 255, 4, 4, 3, 2, 5, 160, 48, 29, 6, 3, 85, 29, 37, 4, 22, 48, 20, 6, 8, 43, 6, 1, 5, 5, 7, 3, 2, 6, 8, 43, 6, 1, 5, 5, 7, 3, 1, 48, 12, 6, 3, 85, 29, 19, 1, 1, 255, 4, 2, 48, 0, 48, 73, 6, 3, 85, 29, 14, 4, 66, 4, 64, 184, 27, 111, 100, 185, 3, 60, 74, 232, 82, 187, 169, 146, 147, 185, 95, 74, 1, 30, 137, 225, 41, 223, 249, 128, 141, 254, 150, 25, 225, 242, 3, 235, 207, 147, 83, 107, 148, 237, 0, 226, 240, 42, 175, 47, 113, 123, 148, 128, 190, 12, 146, 119, 216, 235, 223, 47, 29, 124, 171, 8, 133, 245, 128, 48, 75, 6, 3, 85, 29, 35, 4, 68, 48, 66, 128, 64, 107, 35, 93, 140, 120, 74, 68, 242, 166, 39, 191, 159, 156, 255, 152, 139, 116, 160, 21, 220, 61, 249, 246, 136, 64, 148, 57, 252, 107, 221, 102, 126, 167, 207, 36, 223, 46, 112, 92, 229, 162, 35, 131, 186, 43, 139, 91, 12, 221, 184, 234, 124, 29, 20, 52, 38, 121, 7, 144, 253, 45, 84, 166, 41, 48, 76, 6, 3, 85, 29, 17, 4, 69, 48, 67, 130, 65, 42, 46, 97, 112, 112, 115, 46, 97, 114, 111, 46, 99, 57, 112, 48, 118, 55, 106, 57, 113, 55, 106, 52, 97, 49, 113, 46, 97, 102, 108, 100, 46, 119, 101, 115, 116, 117, 115, 51, 46, 97, 114, 111, 97, 112, 112, 45, 104, 99, 112, 46, 97, 122, 117, 114, 101, 45, 116, 101, 115, 116, 46, 110, 101, 116],
RawSubjectPublicKeyInfo: [48, 130, 1, 34, 48, 13, 6, 9, 42, 134, 72, 134, 247, 13, 1, 1, 1, 5, 0, 3, 130, 1, 15, 0, 48, 130, 1, 10, 2, 130, 1, 1, 0, 175, 180, 66, 13, 148, 254, 185, 112, 159, 180, 48, 58, 217, 14, 123, 23, 229, 28, 224, 241, 179, 210, 115, 71, 123, 94, 152, 56, 200, 179, 120, 80, 246, 68, 78, 194, 165, 17, 161, 64, 168, 40, 31, 243, 155, 207, 171, 17, 238, 179, 159, 199, 90, 32, 61, 243, 133, 233, 153, 229, 126, 69, 220, 234, 188, 36, 185, 6, 114, 77, 215, 59, 104, 34, 80, 95, 66, 69, 104, 166, 84, 17, 159, 234, 101, 109, 164, 153, 172, 156, 240, 134, 16, 200, 61, 179, 79, 164, 99, 227, 229, 46, 174, 192, 70, 194, 161, 207, 5, 78, 18, 21, 247, 101, 35, 19, 102, 159, 29, 233, 182, 86, 199, 53, 244, 175, 208, 228, 116, 49, 45, 21, 118, 178, 72, 104, 19, 188, 1, 131, 254, 127, 37, 139, 95, 60, 129, 254, 7, 18, 184, 51, 199, 217, 71, 107, 203, 113, 33, 123, 83, 246, 30, 234, 39, 214, 99, 207, 167, 116, 247, 66, 241, 15, 133, 22, 9, 238, 99, 105, 166, 167, 72, 142, 182, 229, 22, 51, 127, 84, 32, 214, 8, 185, 111, 104, 100, 152, 181, 219, 81, 90, 133, 133, 127, 114, 2, 134, 80, 248, 53, 178, 96, 3, 247, 147, 225, 186, 125, 75, 198, 237, 177, 202, 247, 71, 49, 31, 123, 47, 222, 47, 12, 75, 219, 147, 5, 27, 92, 161, 76, 204, 58, 211, 26, 120, 156, 19, 51, 136, 52, 116, 18, 166, 216, 205, 2, 3, 1, 0, 1],
RawSubject: [48, 48, 49, 18, 48, 16, 6, 3, 85, 4, 10, 19, 9, 111, 112, 101, 110, 115, 104, 105, 102, 116, 49, 26, 48, 24, 6, 3, 85, 4, 3, 19, 17, 111, 112, 101, 110, 115, 104, 105, 102, 116, 45, 105, 110, 103, 114, 101, 115, 115],
RawIssuer: [48, 38, 49, 18, 48, 16, 6, 3, 85, 4, 11, 19, 9, 111, 112, 101, 110, 115, 104, 105, 102, 116, 49, 16, 48, 14, 6, 3, 85, 4, 3, 19, 7, 114, 111, 111, 116, 45, 99, 97],
Signature: [62, 37, 28, 104, 244, 85, 128, 159, 33, 132, 56, 114, 149, 155, 76, 150, 56, 190, 182, 145, 23, 80, 230, 251, 69, 191, 95, 185, 206, 12, 203, 223, 159, 176, 108, 31, 228, 30, 29, 147, 188, 224, 159, 222, 209, 80, 26, 111, 48, 5, 218, 228, 214, 243, 4, 174, 170, 173, 102, 5, 47, 112, 16, 26, 249, 76, 134, 179, 187, 39, 158, 241, 110, 27, 4, 56, 253, 35, 167, 85, 21, 75, 203, 60, 52, 83, 13, 153, 94, 185, 69, 153, 196, 55, 251, 226, 149, 80, 157, 57, 244, 94, 195, 179, 137, 174, 166, 85, 12, 161, 202, 212, 188, 14, 213, 25, 59, 74, 197, 148, 98, 52, 206, 23, 41, 197, 32, 214, 165, 60, 202, 205, 67, 25, 42, 38, 206, 141, 201, 237, 101, 166, 209, 131, 101, 208, 225, 123, 244, 235, 82, 50, 143, 145, 70, 217, 191, 27, 3, 130, 194, 10, 227, 129, 5, 165, 150, 45, 143, 154, 59, 167, 145, 196, 78, 140, 76, 209, 96, 166, 23, 54, 227, 85, 65, 210, 246, 64, 194, 155, 249, 210, 255, 119, 224, 229, 103, 89, 68, 18, 164, 119, 9, 207, 73, 245, 219, 68, 214, 3, 155, 42, 94, 212, 192, 93, 148, 231, 125, 114, 77, 118, 234, 177, 58, 169, 209, 85, 145, 75, 243, 82, 234, 209, 51, 198, 131, 119, 63, 14, 40, 242, 219, 78, 215, 35, 237, 91, 234, 2, 142, 86, 121, 121, 249, 150],
SignatureAlgorithm: 4,
PublicKeyAlgorithm: 1,
PublicKey: <*rsa.PublicKey | 0x30fb7a9895b0>{
N: {
neg: false,
abs: [3713275565900749005, 5533862720488119315, 886043176200789153, 17818264461995138607, 16265450768615453130, 5834472357489801107, 5862144673391444614, 628656117149906395, 13178964295004987606, 715618696217184398, 12066541168516629782, 6050057140458382287, 14400619811794133371, 6862502860246136883, 1422013248764323211, 8372522752467421288, 13138907997452357860, 17826693266249031145, 5098815638656127509, 5738821655630622400, 12438080729488309683, 6057798802862875801, 7503648299216627878, 13557164215630681915, 9649412887563066602, 17200267080417426931, 12116969928591911697, 17745395029646614848, 8889710084128733264, 16509317662742049607, 11507875977146301207, 12660817078547036528],
},
E: 65537,
},
Version: 3,
SerialNumber: {
neg: false,
abs: [7949786151822990365],
},
Issuer: {
Country: nil,
Organization: nil,
OrganizationalUnit: ["openshift"],
Locality: nil,
Province: nil,
StreetAddress: nil,
PostalCode: nil,
SerialNumber: "",
CommonName: "root-ca",
Names: [
{
Type: [2, 5, 4, 11],
Value: <string>"openshift",
},
{
Type: [2, 5, 4, 3],
Value: <string>"root-ca",
},
],
ExtraNames: nil,
},
Subject: {
Country: nil,
Organization: ["openshift"],
OrganizationalUnit: nil,
Locality: nil,
Province: nil,
StreetAddress: nil,
PostalCode: nil,
SerialNumber: "",
CommonName: "openshift-ingress",
Names: [
{
Type: [2, 5, 4, 10],
Value: <string>"openshift",
},
{
Type: [2, 5, 4, 3],
Value: <string>"openshift-ingress",
},
],
ExtraNames: nil,
},
NotBefore: {wall: 0, ext: 63928459200, loc: nil},
NotAfter: {wall: 0, ext: 63959995200, loc: nil},
KeyUsage: 5,
Extensions: [
{
Id: [2, 5, 29, 15],
Critical: true,
Value: [3, 2, 5, 160],
},
{
Id: [2, 5, 29, 37],
Critical: false,
Value: [48, 20, 6, 8, 43, 6, 1, 5, 5, 7, 3, 2, 6, 8, 43, 6, 1, 5, 5, 7, 3, 1],
},
{Id: [2, 5, 29, 19], Critical: true, Value: [48, 0]},
{
Id: [2, 5, 29, 14],
Critical: false,
Value: [4, 64, 184, 27, 111, 100, 185, 3, 60, 74, 232, 82, 187, 169, 146, 147, 185, 95, 74, 1, 30, 137, 225, 41, 223, 249, 128, 141, 254, 150, 25, 225, 242, 3, 235, 207, 147, 83, 107, 148, 237, 0, 226, 240, 42, 175, 47, 113, 123, 148, 128, 190, 12, 146, 119, 216, 235, 223, 47, 29, 124, 171, 8, 133, 245, 128],
},
{
Id: [2, 5, 29, 35],
Critical: false,
Value: [48, 66, 128, 64, 107, 35, 93, 140, 120, 74, 68, 242, 166, 39, 191, 159, 156, 255, 152, 139, 116, 160, 21, 220, 61, 249, 246, 136, 64, 148, 57, 252, 107, 221, 102, 126, 167, 207, 36, 223, 46, 112, 92, 229, 162, 35, 131, 186, 43, 139, 91, 12, 221, 184, 234, 124, 29, 20, 52, 38, 121, 7, 144, 253, 45, 84, 166, 41],
},
{
Id: [2, 5, 29, 17],
Critical: false,
Value: "0C\x82A*.apps.aro.c9p0v7j9q7j4a1q.afld.westus3.aroapp-hcp.azure-test.net",
},
],
ExtraExtensions: nil,
UnhandledCriticalExtensions: nil,
ExtKeyUsage: [2, 1],
UnknownExtKeyUsage: nil,
BasicConstraintsValid: true,
IsCA: false,
MaxPathLen: -1,
MaxPathLenZero: false,
SubjectKeyId: [184, 27, 111, 100, 185, 3, 60, 74, 232, 82, 187, 169, 146, 147, 185, 95, 74, 1, 30, 137, 225, 41, 223, 249, 128, 141, 254, 150, 25, 225, 242, 3, 235, 207, 147, 83, 107, 148, 237, 0, 226, 240, 42, 175, 47, 113, 123, 148, 128, 190, 12, 146, 119, 216, 235, 223, 47, 29, 124, 171, 8, 133, 245, 128],
AuthorityKeyId: [107, 35, 93, 140, 120, 74, 68, 242, 166, 39, 191, 159, 156, 255, 152, 139, 116, 160, 21, 220, 61, 249, 246, 136, 64, 148, 57, 252, 107, 221, 102, 126, 167, 207, 36, 223, 46, 112, 92, 229, 162, 35, 131, 186, 43, 139, 91, 12, 221, 184, 234, 124, 29, 20, 52, 38, 121, 7, 144, 253, 45, 84, 166, 41],
OCSPServer: nil,
IssuingCertificateURL: nil,
DNSNames: [
"*.apps.aro.c9p0v7j9q7j4a1q.afld.westus3.aroapp-hcp.azure-test.net",
],
EmailAddresses: nil,
IPAddresses: nil,
URIs: nil,
PermittedDNSDomainsCritical: false,
PermittedDNSDomains: nil,
ExcludedDNSDomains: nil,
PermittedIPRanges: nil,
ExcludedIPRanges: nil,
PermittedEmailAddresses: nil,
ExcludedEmailAddresses: nil,
PermittedURIDomains: nil,
ExcludedURIDomains: nil,
CRLDistributionPoints: nil,
PolicyIdentifiers: nil,
Policies: nil,
InhibitAnyPolicy: 0,
InhibitAnyPolicyZero: false,
InhibitPolicyMapping: 0,
InhibitPolicyMappingZero: false,
RequireExplicitPolicy: 0,
RequireExplicitPolicyZero: false,
PolicyMappings: nil,
},
],
Err: <x509.UnknownAuthorityError>{
Cert: 0x30fb7a8f2c08 (cyclic reference),
hintErr: nil,
hintCert: nil,
},
},
},
}
occurredContributing tests (1)
Affected runs (12)
| |||||||||||||||||||||||||||||||||||||||||||||||||||||
CreateOrUpdateExternalAuthAndWait timed out after <minutes> minutes; context deadline exceededfull failure pattern: CreateOrUpdateExternalAuthAndWait timed out after <minutes> minutes; context deadline exceeded Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 2 day(s) | unknown | 3 | 6.98%3 of 43 job runs affected | IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 2 day(s) | Show trend detailsOct 21: 0 · Oct 22: 2 · Oct 23: 1 · Oct 24: 0 · Oct 25: 0 · Oct 26: 0 · Oct 27: 0 · Oct 28: 0 | none | |||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (3)fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:181]: failed to create external auth config external-auth
Unexpected error:
<*fmt.wrapErrors | 0x1977dbbad170>:
failed waiting for external auth "external-auth" in resourcegroup="external-auth-cluster-c7xfsrnzff2g" for cluster="ea-cluster" to finish creating or updating, caused by: timeout '15.000000' minutes exceeded during CreateOrUpdateExternalAuthAndWait for external auth external-auth in cluster ea-cluster in resource group external-auth-cluster-c7xfsrnzff2g, error: context deadline exceeded
{
msg: "failed waiting for external auth \"external-auth\" in resourcegroup=\"external-auth-cluster-c7xfsrnzff2g\" for cluster=\"ea-cluster\" to finish creating or updating, caused by: timeout '15.000000' minutes exceeded during CreateOrUpdateExternalAuthAndWait for external auth external-auth in cluster ea-cluster in resource group external-auth-cluster-c7xfsrnzff2g, error: context deadline exceeded",
errs: [
<*errors.errorString | 0x1977dcaa2220>{
s: "timeout '15.000000' minutes exceeded during CreateOrUpdateExternalAuthAndWait for external auth external-auth in cluster ea-cluster in resource group external-auth-cluster-c7xfsrnzff2g",
},
<context.deadlineExceededError>{},
],
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:181]: failed to create external auth config external-auth
Unexpected error:
<*fmt.wrapErrors | 0x1977dbbad170>:
failed waiting for external auth "external-auth" in resourcegroup="external-auth-cluster-c7xfsrnzff2g" for cluster="ea-cluster" to finish creating or updating, caused by: timeout '15.000000' minutes exceeded during CreateOrUpdateExternalAuthAndWait for external auth external-auth in cluster ea-cluster in resource group external-auth-cluster-c7xfsrnzff2g, error: context deadline exceeded
{
msg: "failed waiting for external auth \"external-auth\" in resourcegroup=\"external-auth-cluster-c7xfsrnzff2g\" for cluster=\"ea-cluster\" to finish creating or updating, caused by: timeout '15.000000' minutes exceeded during CreateOrUpdateExternalAuthAndWait for external auth external-auth in cluster ea-cluster in resource group external-auth-cluster-c7xfsrnzff2g, error: context deadline exceeded",
errs: [
<*errors.errorString | 0x1977dcaa2220>{
s: "timeout '15.000000' minutes exceeded during CreateOrUpdateExternalAuthAndWait for external auth external-auth in cluster ea-cluster in resource group external-auth-cluster-c7xfsrnzff2g",
},
<context.deadlineExceededError>{},
],
}
occurredfail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:276]: failed to create external auth config "ext-auth-cidr"
Unexpected error:
<*fmt.wrapErrors | 0x2cc075309380>:
failed waiting for external auth "ext-auth-cidr" in resourcegroup="e2e-cidr-connectivity-rhgbbsmxx2nr" for cluster="cidr-connectivity-test" to finish creating or updating, caused by: timeout '15.000000' minutes exceeded during CreateOrUpdateExternalAuthAndWait for external auth ext-auth-cidr in cluster cidr-connectivity-test in resource group e2e-cidr-connectivity-rhgbbsmxx2nr, error: context deadline exceeded
{
msg: "failed waiting for external auth \"ext-auth-cidr\" in resourcegroup=\"e2e-cidr-connectivity-rhgbbsmxx2nr\" for cluster=\"cidr-connectivity-test\" to finish creating or updating, caused by: timeout '15.000000' minutes exceeded during CreateOrUpdateExternalAuthAndWait for external auth ext-auth-cidr in cluster cidr-connectivity-test in resource group e2e-cidr-connectivity-rhgbbsmxx2nr, error: context deadline exceeded",
errs: [
<*errors.errorString | 0x2cc074e74090>{
s: "timeout '15.000000' minutes exceeded during CreateOrUpdateExternalAuthAndWait for external auth ext-auth-cidr in cluster cidr-connectivity-test in resource group e2e-cidr-connectivity-rhgbbsmxx2nr",
},
<context.deadlineExceededError>{},
],
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:276]: failed to create external auth config "ext-auth-cidr"
Unexpected error:
<*fmt.wrapErrors | 0x2cc075309380>:
failed waiting for external auth "ext-auth-cidr" in resourcegroup="e2e-cidr-connectivity-rhgbbsmxx2nr" for cluster="cidr-connectivity-test" to finish creating or updating, caused by: timeout '15.000000' minutes exceeded during CreateOrUpdateExternalAuthAndWait for external auth ext-auth-cidr in cluster cidr-connectivity-test in resource group e2e-cidr-connectivity-rhgbbsmxx2nr, error: context deadline exceeded
{
msg: "failed waiting for external auth \"ext-auth-cidr\" in resourcegroup=\"e2e-cidr-connectivity-rhgbbsmxx2nr\" for cluster=\"cidr-connectivity-test\" to finish creating or updating, caused by: timeout '15.000000' minutes exceeded during CreateOrUpdateExternalAuthAndWait for external auth ext-auth-cidr in cluster cidr-connectivity-test in resource group e2e-cidr-connectivity-rhgbbsmxx2nr, error: context deadline exceeded",
errs: [
<*errors.errorString | 0x2cc074e74090>{
s: "timeout '15.000000' minutes exceeded during CreateOrUpdateExternalAuthAndWait for external auth ext-auth-cidr in cluster cidr-connectivity-test in resource group e2e-cidr-connectivity-rhgbbsmxx2nr",
},
<context.deadlineExceededError>{},
],
}
occurredfail [github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:181]: failed to create external auth config external-auth
Unexpected error:
<*fmt.wrapErrors | 0x32cbbfa08480>:
failed waiting for external auth "external-auth" in resourcegroup="external-auth-cluster-s8rx2m7kc8lm" for cluster="ea-cluster" to finish creating or updating, caused by: timeout '15.000000' minutes exceeded during CreateOrUpdateExternalAuthAndWait for external auth external-auth in cluster ea-cluster in resource group external-auth-cluster-s8rx2m7kc8lm, error: context deadline exceeded
{
msg: "failed waiting for external auth \"external-auth\" in resourcegroup=\"external-auth-cluster-s8rx2m7kc8lm\" for cluster=\"ea-cluster\" to finish creating or updating, caused by: timeout '15.000000' minutes exceeded during CreateOrUpdateExternalAuthAndWait for external auth external-auth in cluster ea-cluster in resource group external-auth-cluster-s8rx2m7kc8lm, error: context deadline exceeded",
errs: [
<*errors.errorString | 0x32cbbf682f10>{
s: "timeout '15.000000' minutes exceeded during CreateOrUpdateExternalAuthAndWait for external auth external-auth in cluster ea-cluster in resource group external-auth-cluster-s8rx2m7kc8lm",
},
<context.deadlineExceededError>{},
],
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:181]: failed to create external auth config external-auth
Unexpected error:
<*fmt.wrapErrors | 0x32cbbfa08480>:
failed waiting for external auth "external-auth" in resourcegroup="external-auth-cluster-s8rx2m7kc8lm" for cluster="ea-cluster" to finish creating or updating, caused by: timeout '15.000000' minutes exceeded during CreateOrUpdateExternalAuthAndWait for external auth external-auth in cluster ea-cluster in resource group external-auth-cluster-s8rx2m7kc8lm, error: context deadline exceeded
{
msg: "failed waiting for external auth \"external-auth\" in resourcegroup=\"external-auth-cluster-s8rx2m7kc8lm\" for cluster=\"ea-cluster\" to finish creating or updating, caused by: timeout '15.000000' minutes exceeded during CreateOrUpdateExternalAuthAndWait for external auth external-auth in cluster ea-cluster in resource group external-auth-cluster-s8rx2m7kc8lm, error: context deadline exceeded",
errs: [
<*errors.errorString | 0x32cbbf682f10>{
s: "timeout '15.000000' minutes exceeded during CreateOrUpdateExternalAuthAndWait for external auth external-auth in cluster ea-cluster in resource group external-auth-cluster-s8rx2m7kc8lm",
},
<context.deadlineExceededError>{},
],
}
occurredContributing tests (2)
Affected runs (3)
| |||||||||||||||||||||||||||||||||||||||||||||||||||||
DNS for route host <route-host> did not resolve: DNS for <route-host> did not resolve within <duration> (last error: lookup <host> on <dns-server>: dial udp <ip>: i/o timeout): co...full failure pattern: DNS for route host <route-host> did not resolve: DNS for <route-host> did not resolve within <duration> (last error: lookup <host> on <dns-server>: dial udp <ip>: i/o timeout): context deadline exceeded Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 3 day(s) | unknown | 3 | 6.98%3 of 43 job runs affected | IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 3 day(s) | Show trend detailsOct 21: 0 · Oct 22: 1 · Oct 23: 1 · Oct 24: 1 · Oct 25: 0 · Oct 26: 0 · Oct 27: 0 · Oct 28: 0 | DEV, PROD | |||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (3)fail [github.com/Azure/ARO-HCP/test/e2e/cluster_nodepool_active_versions.go:149]: failed to verify simple web app runs on v20261001 cluster "v20261001"
Unexpected error:
<*fmt.wrapError | 0xdad4da81e80>:
DNS for route host agnhost-e2e-sample-app-4gnm9.apps.aro.v20261001.bhnx.westus3.aroapp-hcp.azure-test.net did not resolve: DNS for agnhost-e2e-sample-app-4gnm9.apps.aro.v20261001.bhnx.westus3.aroapp-hcp.azure-test.net did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-4gnm9.apps.aro.v20261001.bhnx.westus3.aroapp-hcp.azure-test.net on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded
{
msg: "DNS for route host agnhost-e2e-sample-app-4gnm9.apps.aro.v20261001.bhnx.westus3.aroapp-hcp.azure-test.net did not resolve: DNS for agnhost-e2e-sample-app-4gnm9.apps.aro.v20261001.bhnx.westus3.aroapp-hcp.azure-test.net did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-4gnm9.apps.aro.v20261001.bhnx.westus3.aroapp-hcp.azure-test.net on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
err: <*fmt.wrapError | 0xdad4d9b41a0>{
msg: "DNS for agnhost-e2e-sample-app-4gnm9.apps.aro.v20261001.bhnx.westus3.aroapp-hcp.azure-test.net did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-4gnm9.apps.aro.v20261001.bhnx.westus3.aroapp-hcp.azure-test.net on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
err: <context.deadlineExceededError>{},
},
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_nodepool_active_versions.go:149]: failed to verify simple web app runs on v20261001 cluster "v20261001"
Unexpected error:
<*fmt.wrapError | 0xdad4da81e80>:
DNS for route host agnhost-e2e-sample-app-4gnm9.apps.aro.v20261001.bhnx.westus3.aroapp-hcp.azure-test.net did not resolve: DNS for agnhost-e2e-sample-app-4gnm9.apps.aro.v20261001.bhnx.westus3.aroapp-hcp.azure-test.net did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-4gnm9.apps.aro.v20261001.bhnx.westus3.aroapp-hcp.azure-test.net on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded
{
msg: "DNS for route host agnhost-e2e-sample-app-4gnm9.apps.aro.v20261001.bhnx.westus3.aroapp-hcp.azure-test.net did not resolve: DNS for agnhost-e2e-sample-app-4gnm9.apps.aro.v20261001.bhnx.westus3.aroapp-hcp.azure-test.net did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-4gnm9.apps.aro.v20261001.bhnx.westus3.aroapp-hcp.azure-test.net on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
err: <*fmt.wrapError | 0xdad4d9b41a0>{
msg: "DNS for agnhost-e2e-sample-app-4gnm9.apps.aro.v20261001.bhnx.westus3.aroapp-hcp.azure-test.net did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-4gnm9.apps.aro.v20261001.bhnx.westus3.aroapp-hcp.azure-test.net on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
err: <context.deadlineExceededError>{},
},
}
occurredfail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:194]: failed to verify simple web app runs on cluster "cluster-candidate-5-1-5f9zxt"
Unexpected error:
<*fmt.wrapError | 0x3db0320ca900>:
DNS for route host agnhost-e2e-sample-app-8tqcz.apps.aro.x0a9b9z6x4r0r3x.a2p1.westus3.aroapp-hcp.azure-test.net did not resolve: DNS for agnhost-e2e-sample-app-8tqcz.apps.aro.x0a9b9z6x4r0r3x.a2p1.westus3.aroapp-hcp.azure-test.net did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-8tqcz.apps.aro.x0a9b9z6x4r0r3x.a2p1.westus3.aroapp-hcp.azure-test.net on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded
{
msg: "DNS for route host agnhost-e2e-sample-app-8tqcz.apps.aro.x0a9b9z6x4r0r3x.a2p1.westus3.aroapp-hcp.azure-test.net did not resolve: DNS for agnhost-e2e-sample-app-8tqcz.apps.aro.x0a9b9z6x4r0r3x.a2p1.westus3.aroapp-hcp.azure-test.net did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-8tqcz.apps.aro.x0a9b9z6x4r0r3x.a2p1.westus3.aroapp-hcp.azure-test.net on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
err: <*fmt.wrapError | 0x3db031a2bd20>{
msg: "DNS for agnhost-e2e-sample-app-8tqcz.apps.aro.x0a9b9z6x4r0r3x.a2p1.westus3.aroapp-hcp.azure-test.net did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-8tqcz.apps.aro.x0a9b9z6x4r0r3x.a2p1.westus3.aroapp-hcp.azure-test.net on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
err: <context.deadlineExceededError>{},
},
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:194]: failed to verify simple web app runs on cluster "cluster-candidate-5-1-5f9zxt"
Unexpected error:
<*fmt.wrapError | 0x3db0320ca900>:
DNS for route host agnhost-e2e-sample-app-8tqcz.apps.aro.x0a9b9z6x4r0r3x.a2p1.westus3.aroapp-hcp.azure-test.net did not resolve: DNS for agnhost-e2e-sample-app-8tqcz.apps.aro.x0a9b9z6x4r0r3x.a2p1.westus3.aroapp-hcp.azure-test.net did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-8tqcz.apps.aro.x0a9b9z6x4r0r3x.a2p1.westus3.aroapp-hcp.azure-test.net on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded
{
msg: "DNS for route host agnhost-e2e-sample-app-8tqcz.apps.aro.x0a9b9z6x4r0r3x.a2p1.westus3.aroapp-hcp.azure-test.net did not resolve: DNS for agnhost-e2e-sample-app-8tqcz.apps.aro.x0a9b9z6x4r0r3x.a2p1.westus3.aroapp-hcp.azure-test.net did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-8tqcz.apps.aro.x0a9b9z6x4r0r3x.a2p1.westus3.aroapp-hcp.azure-test.net on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
err: <*fmt.wrapError | 0x3db031a2bd20>{
msg: "DNS for agnhost-e2e-sample-app-8tqcz.apps.aro.x0a9b9z6x4r0r3x.a2p1.westus3.aroapp-hcp.azure-test.net did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-8tqcz.apps.aro.x0a9b9z6x4r0r3x.a2p1.westus3.aroapp-hcp.azure-test.net on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
err: <context.deadlineExceededError>{},
},
}
occurredfail [github.com/Azure/ARO-HCP/test/e2e/cluster_nodepool_active_versions.go:149]: failed to verify simple web app runs on v20261001 cluster "v20261001"
Unexpected error:
<*fmt.wrapError | 0x1342e25441a0>:
DNS for route host agnhost-e2e-sample-app-6g5k9.apps.aro.v20261001.1f89.westus3.aroapp-hcp.azure-test.net did not resolve: DNS for agnhost-e2e-sample-app-6g5k9.apps.aro.v20261001.1f89.westus3.aroapp-hcp.azure-test.net did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-6g5k9.apps.aro.v20261001.1f89.westus3.aroapp-hcp.azure-test.net on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded
{
msg: "DNS for route host agnhost-e2e-sample-app-6g5k9.apps.aro.v20261001.1f89.westus3.aroapp-hcp.azure-test.net did not resolve: DNS for agnhost-e2e-sample-app-6g5k9.apps.aro.v20261001.1f89.westus3.aroapp-hcp.azure-test.net did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-6g5k9.apps.aro.v20261001.1f89.westus3.aroapp-hcp.azure-test.net on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
err: <*fmt.wrapError | 0x1342e251c060>{
msg: "DNS for agnhost-e2e-sample-app-6g5k9.apps.aro.v20261001.1f89.westus3.aroapp-hcp.azure-test.net did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-6g5k9.apps.aro.v20261001.1f89.westus3.aroapp-hcp.azure-test.net on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
err: <context.deadlineExceededError>{},
},
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_nodepool_active_versions.go:149]: failed to verify simple web app runs on v20261001 cluster "v20261001"
Unexpected error:
<*fmt.wrapError | 0x1342e25441a0>:
DNS for route host agnhost-e2e-sample-app-6g5k9.apps.aro.v20261001.1f89.westus3.aroapp-hcp.azure-test.net did not resolve: DNS for agnhost-e2e-sample-app-6g5k9.apps.aro.v20261001.1f89.westus3.aroapp-hcp.azure-test.net did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-6g5k9.apps.aro.v20261001.1f89.westus3.aroapp-hcp.azure-test.net on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded
{
msg: "DNS for route host agnhost-e2e-sample-app-6g5k9.apps.aro.v20261001.1f89.westus3.aroapp-hcp.azure-test.net did not resolve: DNS for agnhost-e2e-sample-app-6g5k9.apps.aro.v20261001.1f89.westus3.aroapp-hcp.azure-test.net did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-6g5k9.apps.aro.v20261001.1f89.westus3.aroapp-hcp.azure-test.net on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
err: <*fmt.wrapError | 0x1342e251c060>{
msg: "DNS for agnhost-e2e-sample-app-6g5k9.apps.aro.v20261001.1f89.westus3.aroapp-hcp.azure-test.net did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-6g5k9.apps.aro.v20261001.1f89.westus3.aroapp-hcp.azure-test.net on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
err: <context.deadlineExceededError>{},
},
}
occurredContributing tests (2)
Affected runs (3)
| |||||||||||||||||||||||||||||||||||||||||||||||||||||
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedC...full failure pattern: ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider]; provider Microsoft.RedHatOpenShift Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 2 day(s) | unknown | 3 | 6.98%3 of 43 job runs affected | IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 2 day(s) | Show trend detailsOct 21: 0 · Oct 22: 0 · Oct 23: 0 · Oct 24: 1 · Oct 25: 0 · Oct 26: 0 · Oct 27: 2 · Oct 28: 0 | DEV, PROD, STG | |||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (3)fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:150]: failed to create HCP cluster
Unexpected error:
<*fmt.wrapError | 0x24681019eca0>:
failed to create HCP cluster ingress-cert-le: failed waiting for cluster="ingress-cert-le" in resourcegroup="ingress-cert-le-m6dlx86v7zvt" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/4249c9a2-6a00-4a31-a13c-7b395eeda2a9
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/4249c9a2-6a00-4a31-a13c-7b395eeda2a9",
"name": "4249c9a2-6a00-4a31-a13c-7b395eeda2a9",
"status": "Failed",
"startTime": "2026-10-27T21:21:19.945219062Z",
"endTime": "2026-10-27T21:50:23.791391981Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/4249c9a2-6a00-4a31-a13c-7b395eeda2a9
correlation ID: 79f0a27c-81eb-45e3-ba1c-7ed22276e9b1
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:150]: failed to create HCP cluster
Unexpected error:
<*fmt.wrapError | 0x24681019eca0>:
failed to create HCP cluster ingress-cert-le: failed waiting for cluster="ingress-cert-le" in resourcegroup="ingress-cert-le-m6dlx86v7zvt" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/4249c9a2-6a00-4a31-a13c-7b395eeda2a9
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/4249c9a2-6a00-4a31-a13c-7b395eeda2a9",
"name": "4249c9a2-6a00-4a31-a13c-7b395eeda2a9",
"status": "Failed",
"startTime": "2026-10-27T21:21:19.945219062Z",
"endTime": "2026-10-27T21:50:23.791391981Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/4249c9a2-6a00-4a31-a13c-7b395eeda2a9
correlation ID: 79f0a27c-81eb-45e3-ba1c-7ed22276e9b1
client request ID: unavailable
occurredfail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:154]: HCP cluster rg-candidate-4-20-x79lxq7pcp2s/cluster-candidate-4-20-wz9qjx should provision
Unexpected error:
<*fmt.wrapError | 0x1ea33da4c3e0>:
failed to create HCP cluster cluster-candidate-4-20-wz9qjx: failed waiting for cluster="cluster-candidate-4-20-wz9qjx" in resourcegroup="rg-candidate-4-20-x79lxq7pcp2s" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/5ef510b2-7fbf-47d3-a252-f868567b1a44
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/5ef510b2-7fbf-47d3-a252-f868567b1a44",
"name": "5ef510b2-7fbf-47d3-a252-f868567b1a44",
"status": "Failed",
"startTime": "2026-10-27T16:50:03.68579052Z",
"endTime": "2026-10-27T17:19:03.174752422Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/5ef510b2-7fbf-47d3-a252-f868567b1a44
correlation ID: 0794d196-09b5-4818-a7ca-7f5fb13bca4d
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:154]: HCP cluster rg-candidate-4-20-x79lxq7pcp2s/cluster-candidate-4-20-wz9qjx should provision
Unexpected error:
<*fmt.wrapError | 0x1ea33da4c3e0>:
failed to create HCP cluster cluster-candidate-4-20-wz9qjx: failed waiting for cluster="cluster-candidate-4-20-wz9qjx" in resourcegroup="rg-candidate-4-20-x79lxq7pcp2s" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/5ef510b2-7fbf-47d3-a252-f868567b1a44
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/5ef510b2-7fbf-47d3-a252-f868567b1a44",
"name": "5ef510b2-7fbf-47d3-a252-f868567b1a44",
"status": "Failed",
"startTime": "2026-10-27T16:50:03.68579052Z",
"endTime": "2026-10-27T17:19:03.174752422Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/5ef510b2-7fbf-47d3-a252-f868567b1a44
correlation ID: 0794d196-09b5-4818-a7ca-7f5fb13bca4d
client request ID: unavailable
occurredfail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:125]: failed to create HCP cluster "cluster-zstream-4-22-njchjb" with version 4.22.18 on candidate channel
Unexpected error:
<*fmt.wrapError | 0x2294f905a340>:
failed to create HCP cluster cluster-zstream-4-22-njchjb: failed waiting for cluster="cluster-zstream-4-22-njchjb" in resourcegroup="rg-zstream-upgrade-4-22-5bksdkff6tvq" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/b7f42e30-e443-47e9-982e-3fe8abc06d9a
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/b7f42e30-e443-47e9-982e-3fe8abc06d9a",
"name": "b7f42e30-e443-47e9-982e-3fe8abc06d9a",
"status": "Failed",
"startTime": "2026-10-24T22:19:20.094796575Z",
"endTime": "2026-10-24T22:48:29.265123983Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/b7f42e30-e443-47e9-982e-3fe8abc06d9a
correlation ID: c6c76fd1-9a00-45d4-a377-4b7ed8ffb192
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:125]: failed to create HCP cluster "cluster-zstream-4-22-njchjb" with version 4.22.18 on candidate channel
Unexpected error:
<*fmt.wrapError | 0x2294f905a340>:
failed to create HCP cluster cluster-zstream-4-22-njchjb: failed waiting for cluster="cluster-zstream-4-22-njchjb" in resourcegroup="rg-zstream-upgrade-4-22-5bksdkff6tvq" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/b7f42e30-e443-47e9-982e-3fe8abc06d9a
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/b7f42e30-e443-47e9-982e-3fe8abc06d9a",
"name": "b7f42e30-e443-47e9-982e-3fe8abc06d9a",
"status": "Failed",
"startTime": "2026-10-24T22:19:20.094796575Z",
"endTime": "2026-10-24T22:48:29.265123983Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/b7f42e30-e443-47e9-982e-3fe8abc06d9a
correlation ID: c6c76fd1-9a00-45d4-a377-4b7ed8ffb192
client request ID: unavailable
occurredContributing tests (3)
Affected runs (3)
| |||||||||||||||||||||||||||||||||||||||||||||||||||||
VerifySimpleWebApp failed: DNS for route host <route-host> did not resolve: DNS for <route-host> did not resolve within <duration> (last error: lookup <host> on <dns-server>: dial...full failure pattern: VerifySimpleWebApp failed: DNS for route host <route-host> did not resolve: DNS for <route-host> did not resolve within <duration> (last error: lookup <host> on <dns-server>: dial udp <ip>: i/o timeout): context deadline exceeded Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 3 day(s) | unknown | 3 | 6.98%3 of 43 job runs affected | IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 3 day(s) | Show trend detailsOct 21: 0 · Oct 22: 0 · Oct 23: 1 · Oct 24: 1 · Oct 25: 1 · Oct 26: 0 · Oct 27: 0 · Oct 28: 0 | PROD | |||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (3)fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_cni_cilium.go:186]: failed to run simple web app and connectivity check app with cilium CNI
Unexpected error:
<*errors.joinError | 0x248e56e12588>:
VerifySimpleWebApp failed: DNS for route host agnhost-e2e-sample-app-84lh2.apps.aro.cilium-cl.t19t.westus3.aroapp-hcp.azure-test.net did not resolve: DNS for agnhost-e2e-sample-app-84lh2.apps.aro.cilium-cl.t19t.westus3.aroapp-hcp.azure-test.net did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-84lh2.apps.aro.cilium-cl.t19t.westus3.aroapp-hcp.azure-test.net on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded
{
errs: [
<*fmt.wrapError | 0x248e56f14360>{
msg: "VerifySimpleWebApp failed: DNS for route host agnhost-e2e-sample-app-84lh2.apps.aro.cilium-cl.t19t.westus3.aroapp-hcp.azure-test.net did not resolve: DNS for agnhost-e2e-sample-app-84lh2.apps.aro.cilium-cl.t19t.westus3.aroapp-hcp.azure-test.net did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-84lh2.apps.aro.cilium-cl.t19t.westus3.aroapp-hcp.azure-test.net on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
err: <*fmt.wrapError | 0x248e5757aa00>{
msg: "DNS for route host agnhost-e2e-sample-app-84lh2.apps.aro.cilium-cl.t19t.westus3.aroapp-hcp.azure-test.net did not resolve: DNS for agnhost-e2e-sample-app-84lh2.apps.aro.cilium-cl.t19t.westus3.aroapp-hcp.azure-test.net did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-84lh2.apps.aro.cilium-cl.t19t.westus3.aroapp-hcp.azure-test.net on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
err: <*fmt.wrapError | 0x248e57018e60>{
msg: "DNS for agnhost-e2e-sample-app-84lh2.apps.aro.cilium-cl.t19t.westus3.aroapp-hcp.azure-test.net did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-84lh2.apps.aro.cilium-cl.t19t.westus3.aroapp-hcp.azure-test.net on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
err: <context.deadlineExceededError>{},
},
},
},
],
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_cni_cilium.go:186]: failed to run simple web app and connectivity check app with cilium CNI
Unexpected error:
<*errors.joinError | 0x248e56e12588>:
VerifySimpleWebApp failed: DNS for route host agnhost-e2e-sample-app-84lh2.apps.aro.cilium-cl.t19t.westus3.aroapp-hcp.azure-test.net did not resolve: DNS for agnhost-e2e-sample-app-84lh2.apps.aro.cilium-cl.t19t.westus3.aroapp-hcp.azure-test.net did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-84lh2.apps.aro.cilium-cl.t19t.westus3.aroapp-hcp.azure-test.net on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded
{
errs: [
<*fmt.wrapError | 0x248e56f14360>{
msg: "VerifySimpleWebApp failed: DNS for route host agnhost-e2e-sample-app-84lh2.apps.aro.cilium-cl.t19t.westus3.aroapp-hcp.azure-test.net did not resolve: DNS for agnhost-e2e-sample-app-84lh2.apps.aro.cilium-cl.t19t.westus3.aroapp-hcp.azure-test.net did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-84lh2.apps.aro.cilium-cl.t19t.westus3.aroapp-hcp.azure-test.net on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
err: <*fmt.wrapError | 0x248e5757aa00>{
msg: "DNS for route host agnhost-e2e-sample-app-84lh2.apps.aro.cilium-cl.t19t.westus3.aroapp-hcp.azure-test.net did not resolve: DNS for agnhost-e2e-sample-app-84lh2.apps.aro.cilium-cl.t19t.westus3.aroapp-hcp.azure-test.net did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-84lh2.apps.aro.cilium-cl.t19t.westus3.aroapp-hcp.azure-test.net on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
err: <*fmt.wrapError | 0x248e57018e60>{
msg: "DNS for agnhost-e2e-sample-app-84lh2.apps.aro.cilium-cl.t19t.westus3.aroapp-hcp.azure-test.net did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-84lh2.apps.aro.cilium-cl.t19t.westus3.aroapp-hcp.azure-test.net on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
err: <context.deadlineExceededError>{},
},
},
},
],
}
occurredfail [github.com/Azure/ARO-HCP/test/e2e/cluster_autoscaling.go:203]: failed to verify cluster workloads and pods after autoscaling update
Unexpected error:
<*errors.joinError | 0x22d4bdfbc060>:
VerifySimpleWebApp failed: DNS for route host agnhost-e2e-sample-app-mcq7p.apps.aro.m0y7f5l6s2i7t2m.fbfu.westus3.aroapp-hcp.azure-test.net did not resolve: DNS for agnhost-e2e-sample-app-mcq7p.apps.aro.m0y7f5l6s2i7t2m.fbfu.westus3.aroapp-hcp.azure-test.net did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-mcq7p.apps.aro.m0y7f5l6s2i7t2m.fbfu.westus3.aroapp-hcp.azure-test.net on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded
{
errs: [
<*fmt.wrapError | 0x22d4bdf9c1a0>{
msg: "VerifySimpleWebApp failed: DNS for route host agnhost-e2e-sample-app-mcq7p.apps.aro.m0y7f5l6s2i7t2m.fbfu.westus3.aroapp-hcp.azure-test.net did not resolve: DNS for agnhost-e2e-sample-app-mcq7p.apps.aro.m0y7f5l6s2i7t2m.fbfu.westus3.aroapp-hcp.azure-test.net did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-mcq7p.apps.aro.m0y7f5l6s2i7t2m.fbfu.westus3.aroapp-hcp.azure-test.net on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
err: <*fmt.wrapError | 0x22d4bdee0f00>{
msg: "DNS for route host agnhost-e2e-sample-app-mcq7p.apps.aro.m0y7f5l6s2i7t2m.fbfu.westus3.aroapp-hcp.azure-test.net did not resolve: DNS for agnhost-e2e-sample-app-mcq7p.apps.aro.m0y7f5l6s2i7t2m.fbfu.westus3.aroapp-hcp.azure-test.net did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-mcq7p.apps.aro.m0y7f5l6s2i7t2m.fbfu.westus3.aroapp-hcp.azure-test.net on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
err: <*fmt.wrapError | 0x22d4bdf9da20>{
msg: "DNS for agnhost-e2e-sample-app-mcq7p.apps.aro.m0y7f5l6s2i7t2m.fbfu.westus3.aroapp-hcp.azure-test.net did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-mcq7p.apps.aro.m0y7f5l6s2i7t2m.fbfu.westus3.aroapp-hcp.azure-test.net on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
err: <context.deadlineExceededError>{},
},
},
},
],
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_autoscaling.go:203]: failed to verify cluster workloads and pods after autoscaling update
Unexpected error:
<*errors.joinError | 0x22d4bdfbc060>:
VerifySimpleWebApp failed: DNS for route host agnhost-e2e-sample-app-mcq7p.apps.aro.m0y7f5l6s2i7t2m.fbfu.westus3.aroapp-hcp.azure-test.net did not resolve: DNS for agnhost-e2e-sample-app-mcq7p.apps.aro.m0y7f5l6s2i7t2m.fbfu.westus3.aroapp-hcp.azure-test.net did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-mcq7p.apps.aro.m0y7f5l6s2i7t2m.fbfu.westus3.aroapp-hcp.azure-test.net on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded
{
errs: [
<*fmt.wrapError | 0x22d4bdf9c1a0>{
msg: "VerifySimpleWebApp failed: DNS for route host agnhost-e2e-sample-app-mcq7p.apps.aro.m0y7f5l6s2i7t2m.fbfu.westus3.aroapp-hcp.azure-test.net did not resolve: DNS for agnhost-e2e-sample-app-mcq7p.apps.aro.m0y7f5l6s2i7t2m.fbfu.westus3.aroapp-hcp.azure-test.net did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-mcq7p.apps.aro.m0y7f5l6s2i7t2m.fbfu.westus3.aroapp-hcp.azure-test.net on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
err: <*fmt.wrapError | 0x22d4bdee0f00>{
msg: "DNS for route host agnhost-e2e-sample-app-mcq7p.apps.aro.m0y7f5l6s2i7t2m.fbfu.westus3.aroapp-hcp.azure-test.net did not resolve: DNS for agnhost-e2e-sample-app-mcq7p.apps.aro.m0y7f5l6s2i7t2m.fbfu.westus3.aroapp-hcp.azure-test.net did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-mcq7p.apps.aro.m0y7f5l6s2i7t2m.fbfu.westus3.aroapp-hcp.azure-test.net on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
err: <*fmt.wrapError | 0x22d4bdf9da20>{
msg: "DNS for agnhost-e2e-sample-app-mcq7p.apps.aro.m0y7f5l6s2i7t2m.fbfu.westus3.aroapp-hcp.azure-test.net did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-mcq7p.apps.aro.m0y7f5l6s2i7t2m.fbfu.westus3.aroapp-hcp.azure-test.net on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
err: <context.deadlineExceededError>{},
},
},
},
],
}
occurredfail [github.com/Azure/ARO-HCP/test/e2e/cluster_autoscaling.go:203]: failed to verify cluster workloads and pods after autoscaling update
Unexpected error:
<*errors.joinError | 0x1b9fbf4865b8>:
VerifySimpleWebApp failed: DNS for route host agnhost-e2e-sample-app-6gx6w.apps.aro.v0o5x2p3z8t0f8t.72eg.westus3.aroapp-hcp.azure-test.net did not resolve: DNS for agnhost-e2e-sample-app-6gx6w.apps.aro.v0o5x2p3z8t0f8t.72eg.westus3.aroapp-hcp.azure-test.net did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-6gx6w.apps.aro.v0o5x2p3z8t0f8t.72eg.westus3.aroapp-hcp.azure-test.net on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded
{
errs: [
<*fmt.wrapError | 0x1b9fbd955860>{
msg: "VerifySimpleWebApp failed: DNS for route host agnhost-e2e-sample-app-6gx6w.apps.aro.v0o5x2p3z8t0f8t.72eg.westus3.aroapp-hcp.azure-test.net did not resolve: DNS for agnhost-e2e-sample-app-6gx6w.apps.aro.v0o5x2p3z8t0f8t.72eg.westus3.aroapp-hcp.azure-test.net did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-6gx6w.apps.aro.v0o5x2p3z8t0f8t.72eg.westus3.aroapp-hcp.azure-test.net on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
err: <*fmt.wrapError | 0x1b9fbd298140>{
msg: "DNS for route host agnhost-e2e-sample-app-6gx6w.apps.aro.v0o5x2p3z8t0f8t.72eg.westus3.aroapp-hcp.azure-test.net did not resolve: DNS for agnhost-e2e-sample-app-6gx6w.apps.aro.v0o5x2p3z8t0f8t.72eg.westus3.aroapp-hcp.azure-test.net did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-6gx6w.apps.aro.v0o5x2p3z8t0f8t.72eg.westus3.aroapp-hcp.azure-test.net on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
err: <*fmt.wrapError | 0x1b9fbe030640>{
msg: "DNS for agnhost-e2e-sample-app-6gx6w.apps.aro.v0o5x2p3z8t0f8t.72eg.westus3.aroapp-hcp.azure-test.net did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-6gx6w.apps.aro.v0o5x2p3z8t0f8t.72eg.westus3.aroapp-hcp.azure-test.net on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
err: <context.deadlineExceededError>{},
},
},
},
],
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_autoscaling.go:203]: failed to verify cluster workloads and pods after autoscaling update
Unexpected error:
<*errors.joinError | 0x1b9fbf4865b8>:
VerifySimpleWebApp failed: DNS for route host agnhost-e2e-sample-app-6gx6w.apps.aro.v0o5x2p3z8t0f8t.72eg.westus3.aroapp-hcp.azure-test.net did not resolve: DNS for agnhost-e2e-sample-app-6gx6w.apps.aro.v0o5x2p3z8t0f8t.72eg.westus3.aroapp-hcp.azure-test.net did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-6gx6w.apps.aro.v0o5x2p3z8t0f8t.72eg.westus3.aroapp-hcp.azure-test.net on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded
{
errs: [
<*fmt.wrapError | 0x1b9fbd955860>{
msg: "VerifySimpleWebApp failed: DNS for route host agnhost-e2e-sample-app-6gx6w.apps.aro.v0o5x2p3z8t0f8t.72eg.westus3.aroapp-hcp.azure-test.net did not resolve: DNS for agnhost-e2e-sample-app-6gx6w.apps.aro.v0o5x2p3z8t0f8t.72eg.westus3.aroapp-hcp.azure-test.net did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-6gx6w.apps.aro.v0o5x2p3z8t0f8t.72eg.westus3.aroapp-hcp.azure-test.net on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
err: <*fmt.wrapError | 0x1b9fbd298140>{
msg: "DNS for route host agnhost-e2e-sample-app-6gx6w.apps.aro.v0o5x2p3z8t0f8t.72eg.westus3.aroapp-hcp.azure-test.net did not resolve: DNS for agnhost-e2e-sample-app-6gx6w.apps.aro.v0o5x2p3z8t0f8t.72eg.westus3.aroapp-hcp.azure-test.net did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-6gx6w.apps.aro.v0o5x2p3z8t0f8t.72eg.westus3.aroapp-hcp.azure-test.net on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
err: <*fmt.wrapError | 0x1b9fbe030640>{
msg: "DNS for agnhost-e2e-sample-app-6gx6w.apps.aro.v0o5x2p3z8t0f8t.72eg.westus3.aroapp-hcp.azure-test.net did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-6gx6w.apps.aro.v0o5x2p3z8t0f8t.72eg.westus3.aroapp-hcp.azure-test.net on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
err: <context.deadlineExceededError>{},
},
},
},
],
}
occurredContributing tests (2)
Affected runs (3)
| |||||||||||||||||||||||||||||||||||||||||||||||||||||
timeout during GetAdminRESTConfigForHCPCluster while waiting for hcpcluster credsfull failure pattern: timeout during GetAdminRESTConfigForHCPCluster while waiting for hcpcluster creds Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 2 day(s) | unknown | 2 | 4.65%2 of 43 job runs affected | IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 2 day(s) | Show trend detailsOct 21: 0 · Oct 22: 0 · Oct 23: 0 · Oct 24: 0 · Oct 25: 1 · Oct 26: 0 · Oct 27: 1 · Oct 28: 0 | none | |||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (2)fail [github.com/Azure/ARO-HCP/test/e2e/cluster_container_registry.go:657]: failed to get admin REST config
Unexpected error:
<*fmt.wrapErrors | 0x13a2362ee210>:
failed waiting for hcpCluster="acr-pull" in resourcegroup="acr-pull-5cjwmbnpxtmc" to finish getting creds, caused by: timeout '2.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster20261001 for cluster acr-pull in resource group acr-pull-5cjwmbnpxtmc, error: context deadline exceeded
{
msg: "failed waiting for hcpCluster=\"acr-pull\" in resourcegroup=\"acr-pull-5cjwmbnpxtmc\" to finish getting creds, caused by: timeout '2.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster20261001 for cluster acr-pull in resource group acr-pull-5cjwmbnpxtmc, error: context deadline exceeded",
errs: [
<*errors.errorString | 0x13a235e7d850>{
s: "timeout '2.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster20261001 for cluster acr-pull in resource group acr-pull-5cjwmbnpxtmc",
},
<context.deadlineExceededError>{},
],
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_container_registry.go:657]: failed to get admin REST config
Unexpected error:
<*fmt.wrapErrors | 0x13a2362ee210>:
failed waiting for hcpCluster="acr-pull" in resourcegroup="acr-pull-5cjwmbnpxtmc" to finish getting creds, caused by: timeout '2.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster20261001 for cluster acr-pull in resource group acr-pull-5cjwmbnpxtmc, error: context deadline exceeded
{
msg: "failed waiting for hcpCluster=\"acr-pull\" in resourcegroup=\"acr-pull-5cjwmbnpxtmc\" to finish getting creds, caused by: timeout '2.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster20261001 for cluster acr-pull in resource group acr-pull-5cjwmbnpxtmc, error: context deadline exceeded",
errs: [
<*errors.errorString | 0x13a235e7d850>{
s: "timeout '2.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster20261001 for cluster acr-pull in resource group acr-pull-5cjwmbnpxtmc",
},
<context.deadlineExceededError>{},
],
}
occurredfail [github.com/Azure/ARO-HCP/test/e2e/cluster_container_registry.go:657]: failed to get admin REST config
Unexpected error:
<*fmt.wrapErrors | 0xa204c386ab0>:
failed waiting for hcpCluster="acr-pull" in resourcegroup="acr-pull-zhz5sxs2r4gt" to finish getting creds, caused by: timeout '2.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster20261001 for cluster acr-pull in resource group acr-pull-zhz5sxs2r4gt, error: context deadline exceeded
{
msg: "failed waiting for hcpCluster=\"acr-pull\" in resourcegroup=\"acr-pull-zhz5sxs2r4gt\" to finish getting creds, caused by: timeout '2.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster20261001 for cluster acr-pull in resource group acr-pull-zhz5sxs2r4gt, error: context deadline exceeded",
errs: [
<*errors.errorString | 0xa204b336550>{
s: "timeout '2.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster20261001 for cluster acr-pull in resource group acr-pull-zhz5sxs2r4gt",
},
<context.deadlineExceededError>{},
],
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_container_registry.go:657]: failed to get admin REST config
Unexpected error:
<*fmt.wrapErrors | 0xa204c386ab0>:
failed waiting for hcpCluster="acr-pull" in resourcegroup="acr-pull-zhz5sxs2r4gt" to finish getting creds, caused by: timeout '2.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster20261001 for cluster acr-pull in resource group acr-pull-zhz5sxs2r4gt, error: context deadline exceeded
{
msg: "failed waiting for hcpCluster=\"acr-pull\" in resourcegroup=\"acr-pull-zhz5sxs2r4gt\" to finish getting creds, caused by: timeout '2.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster20261001 for cluster acr-pull in resource group acr-pull-zhz5sxs2r4gt, error: context deadline exceeded",
errs: [
<*errors.errorString | 0xa204b336550>{
s: "timeout '2.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster20261001 for cluster acr-pull in resource group acr-pull-zhz5sxs2r4gt",
},
<context.deadlineExceededError>{},
],
}
occurredContributing tests (1)
Affected runs (2)
| |||||||||||||||||||||||||||||||||||||||||||||||||||||
ERROR CODE: DeadlineExceeded; detail code SubscriptionRequestsThrottled; detail message Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor '<u...full failure pattern: ERROR CODE: DeadlineExceeded; detail code SubscriptionRequestsThrottled; detail message Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor '<uuid>' exceeded. Please try again after '<seconds>' seconds after additional tokens are available.; provider Microsoft.RedHatOpenShift Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 2 day(s) | unknown | 1 | 2.33%1 of 43 job runs affected | IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 2 day(s) | Show trend detailsOct 21: 2 · Oct 22: 0 · Oct 23: 0 · Oct 24: 0 · Oct 25: 1 · Oct 26: 0 · Oct 27: 0 · Oct 28: 0 | DEV | |||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/e2e/gpu_nodepools_create_delete.go:118]: failed to create default nodepool np-1
Unexpected error:
<*fmt.wrapError | 0x3cdcfbca5680>:
failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "cluster-gpu-np" in resourcegroup="rg-gpu-nodepool-8vtwfvfssx9k" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/907ac069-8701-4021-b410-cf10181345c0
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/907ac069-8701-4021-b410-cf10181345c0",
"name": "907ac069-8701-4021-b410-cf10181345c0",
"status": "Failed",
"startTime": "2026-10-25T05:14:31.336281353Z",
"endTime": "2026-10-25T05:33:33.578871475Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine cluster-gpu-np-np-1-8v7xk-22mlx: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-gpu-nodepool-8vtwfvfssx9k--managed/providers/Microsoft.Compute/virtualMachines/cluster-gpu-np-np-1-8v7xk-22mlx to exist\nMachine cluster-gpu-np-np-1-8v7xk-p7hpl: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-gpu-nodepool-8vtwfvfssx9k--managed/providers/Microsoft.Compute/virtualMachines/cluster-gpu-np-np-1-8v7xk-p7hpl to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine cluster-gpu-np-np-1-8v7xk-p7hpl: Failed: interfaces failed to create or update. err: failed to get existing resource rg-gpu-nodepool-8vtwfvfssx9k--managed/cluster-gpu-np-np-1-8v7xk-p7hpl-nic (service: interfaces): GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-gpu-nodepool-8vtwfvfssx9k--managed/providers/Microsoft.Network/networkInterfaces/cluster-gpu-np-np-1-8v7xk-p7hpl-nic\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"SubscriptionRequestsThrottled\",\n \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor '6096f642-63a0-4dd6-958d-715a87f33535' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to ... message truncated\nMachine cluster-gpu-np-np-1-8v7xk-22mlx: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-gpu-nodepool-8vtwfvfssx9k--managed/providers/Microsoft.Compute/virtualMachines/cluster-gpu-np-np-1-8v7xk-22mlx to exist\n"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/907ac069-8701-4021-b410-cf10181345c0
correlation ID: 1c529a86-2bdd-43d7-ab0e-4267cd216305
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/gpu_nodepools_create_delete.go:118]: failed to create default nodepool np-1
Unexpected error:
<*fmt.wrapError | 0x3cdcfbca5680>:
failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "cluster-gpu-np" in resourcegroup="rg-gpu-nodepool-8vtwfvfssx9k" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/907ac069-8701-4021-b410-cf10181345c0
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/907ac069-8701-4021-b410-cf10181345c0",
"name": "907ac069-8701-4021-b410-cf10181345c0",
"status": "Failed",
"startTime": "2026-10-25T05:14:31.336281353Z",
"endTime": "2026-10-25T05:33:33.578871475Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine cluster-gpu-np-np-1-8v7xk-22mlx: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-gpu-nodepool-8vtwfvfssx9k--managed/providers/Microsoft.Compute/virtualMachines/cluster-gpu-np-np-1-8v7xk-22mlx to exist\nMachine cluster-gpu-np-np-1-8v7xk-p7hpl: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-gpu-nodepool-8vtwfvfssx9k--managed/providers/Microsoft.Compute/virtualMachines/cluster-gpu-np-np-1-8v7xk-p7hpl to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine cluster-gpu-np-np-1-8v7xk-p7hpl: Failed: interfaces failed to create or update. err: failed to get existing resource rg-gpu-nodepool-8vtwfvfssx9k--managed/cluster-gpu-np-np-1-8v7xk-p7hpl-nic (service: interfaces): GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-gpu-nodepool-8vtwfvfssx9k--managed/providers/Microsoft.Network/networkInterfaces/cluster-gpu-np-np-1-8v7xk-p7hpl-nic\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"SubscriptionRequestsThrottled\",\n \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor '6096f642-63a0-4dd6-958d-715a87f33535' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to ... message truncated\nMachine cluster-gpu-np-np-1-8v7xk-22mlx: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-gpu-nodepool-8vtwfvfssx9k--managed/providers/Microsoft.Compute/virtualMachines/cluster-gpu-np-np-1-8v7xk-22mlx to exist\n"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/907ac069-8701-4021-b410-cf10181345c0
correlation ID: 1c529a86-2bdd-43d7-ab0e-4267cd216305
client request ID: unavailable
occurredContributing tests (1)
Affected runs (1)
| |||||||||||||||||||||||||||||||||||||||||||||||||||||
DNS for route host <route-host> did not resolve: DNS for <route-host> did not resolve within <duration> (last error: lookup <host> on <dns-server>: no such host): context deadline...full failure pattern: DNS for route host <route-host> did not resolve: DNS for <route-host> did not resolve within <duration> (last error: lookup <host> on <dns-server>: no such host): context deadline exceeded Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s) | unknown | 1 | 2.33%1 of 43 job runs affected | IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s) | Show trend detailsOct 21: 0 · Oct 22: 0 · Oct 23: 0 · Oct 24: 1 · Oct 25: 0 · Oct 26: 0 · Oct 27: 0 · Oct 28: 0 | DEV, PROD | |||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_v20260901preview.go:121]: failed to verify simple web app runs on v20260901preview cluster "v20260901"
Unexpected error:
<*fmt.wrapError | 0x3f57e0004820>:
DNS for route host agnhost-e2e-sample-app-g99mf.apps.aro.v20260901.9ggg.westus3.aroapp-hcp.azure-test.net did not resolve: DNS for agnhost-e2e-sample-app-g99mf.apps.aro.v20260901.9ggg.westus3.aroapp-hcp.azure-test.net did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-g99mf.apps.aro.v20260901.9ggg.westus3.aroapp-hcp.azure-test.net on 172.30.0.10:53: no such host): context deadline exceeded
{
msg: "DNS for route host agnhost-e2e-sample-app-g99mf.apps.aro.v20260901.9ggg.westus3.aroapp-hcp.azure-test.net did not resolve: DNS for agnhost-e2e-sample-app-g99mf.apps.aro.v20260901.9ggg.westus3.aroapp-hcp.azure-test.net did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-g99mf.apps.aro.v20260901.9ggg.westus3.aroapp-hcp.azure-test.net on 172.30.0.10:53: no such host): context deadline exceeded",
err: <*fmt.wrapError | 0x3f57dfc6e000>{
msg: "DNS for agnhost-e2e-sample-app-g99mf.apps.aro.v20260901.9ggg.westus3.aroapp-hcp.azure-test.net did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-g99mf.apps.aro.v20260901.9ggg.westus3.aroapp-hcp.azure-test.net on 172.30.0.10:53: no such host): context deadline exceeded",
err: <context.deadlineExceededError>{},
},
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_v20260901preview.go:121]: failed to verify simple web app runs on v20260901preview cluster "v20260901"
Unexpected error:
<*fmt.wrapError | 0x3f57e0004820>:
DNS for route host agnhost-e2e-sample-app-g99mf.apps.aro.v20260901.9ggg.westus3.aroapp-hcp.azure-test.net did not resolve: DNS for agnhost-e2e-sample-app-g99mf.apps.aro.v20260901.9ggg.westus3.aroapp-hcp.azure-test.net did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-g99mf.apps.aro.v20260901.9ggg.westus3.aroapp-hcp.azure-test.net on 172.30.0.10:53: no such host): context deadline exceeded
{
msg: "DNS for route host agnhost-e2e-sample-app-g99mf.apps.aro.v20260901.9ggg.westus3.aroapp-hcp.azure-test.net did not resolve: DNS for agnhost-e2e-sample-app-g99mf.apps.aro.v20260901.9ggg.westus3.aroapp-hcp.azure-test.net did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-g99mf.apps.aro.v20260901.9ggg.westus3.aroapp-hcp.azure-test.net on 172.30.0.10:53: no such host): context deadline exceeded",
err: <*fmt.wrapError | 0x3f57dfc6e000>{
msg: "DNS for agnhost-e2e-sample-app-g99mf.apps.aro.v20260901.9ggg.westus3.aroapp-hcp.azure-test.net did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-g99mf.apps.aro.v20260901.9ggg.westus3.aroapp-hcp.azure-test.net on 172.30.0.10:53: no such host): context deadline exceeded",
err: <context.deadlineExceededError>{},
},
}
occurredContributing tests (1)
Affected runs (1)
| |||||||||||||||||||||||||||||||||||||||||||||||||||||
DNS for route host agnhost-e2e-sample-app-<id>full failure pattern: DNS for route host agnhost-e2e-sample-app-<id> Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s) | unknown | 1 | 2.33%1 of 43 job runs affected | IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s) | Show trend detailsOct 21: 0 · Oct 22: 0 · Oct 23: 0 · Oct 24: 1 · Oct 25: 0 · Oct 26: 0 · Oct 27: 0 · Oct 28: 0 | DEV, PROD | |||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:528]: sample application must serve the exact customer-specified certificate with a valid Let's Encrypt chain
Expected success, but got an error:
<*fmt.wrapError | 0x7f30d39c620>:
DNS for route host agnhost-e2e-sample-app-gt2zw.apps.aro.ingress-cert-le.msqo.westus3.aroapp-hcp.azure-test.net did not resolve: DNS for agnhost-e2e-sample-app-gt2zw.apps.aro.ingress-cert-le.msqo.westus3.aroapp-hcp.azure-test.net did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-gt2zw.apps.aro.ingress-cert-le.msqo.westus3.aroapp-hcp.azure-test.net on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded
{
msg: "DNS for route host agnhost-e2e-sample-app-gt2zw.apps.aro.ingress-cert-le.msqo.westus3.aroapp-hcp.azure-test.net did not resolve: DNS for agnhost-e2e-sample-app-gt2zw.apps.aro.ingress-cert-le.msqo.westus3.aroapp-hcp.azure-test.net did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-gt2zw.apps.aro.ingress-cert-le.msqo.westus3.aroapp-hcp.azure-test.net on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
err: <*fmt.wrapError | 0x7f30d39c600>{
msg: "DNS for agnhost-e2e-sample-app-gt2zw.apps.aro.ingress-cert-le.msqo.westus3.aroapp-hcp.azure-test.net did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-gt2zw.apps.aro.ingress-cert-le.msqo.westus3.aroapp-hcp.azure-test.net on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
err: <context.deadlineExceededError>{},
},
}
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:528]: sample application must serve the exact customer-specified certificate with a valid Let's Encrypt chain
Expected success, but got an error:
<*fmt.wrapError | 0x7f30d39c620>:
DNS for route host agnhost-e2e-sample-app-gt2zw.apps.aro.ingress-cert-le.msqo.westus3.aroapp-hcp.azure-test.net did not resolve: DNS for agnhost-e2e-sample-app-gt2zw.apps.aro.ingress-cert-le.msqo.westus3.aroapp-hcp.azure-test.net did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-gt2zw.apps.aro.ingress-cert-le.msqo.westus3.aroapp-hcp.azure-test.net on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded
{
msg: "DNS for route host agnhost-e2e-sample-app-gt2zw.apps.aro.ingress-cert-le.msqo.westus3.aroapp-hcp.azure-test.net did not resolve: DNS for agnhost-e2e-sample-app-gt2zw.apps.aro.ingress-cert-le.msqo.westus3.aroapp-hcp.azure-test.net did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-gt2zw.apps.aro.ingress-cert-le.msqo.westus3.aroapp-hcp.azure-test.net on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
err: <*fmt.wrapError | 0x7f30d39c600>{
msg: "DNS for agnhost-e2e-sample-app-gt2zw.apps.aro.ingress-cert-le.msqo.westus3.aroapp-hcp.azure-test.net did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-gt2zw.apps.aro.ingress-cert-le.msqo.westus3.aroapp-hcp.azure-test.net on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
err: <context.deadlineExceededError>{},
},
}Contributing tests (1)
Affected runs (1)
| |||||||||||||||||||||||||||||||||||||||||||||||||||||
ERROR CODE: DeadlineExceeded; detail code SubscriptionRequestsThrottled; detail message Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor '<u...full failure pattern: ERROR CODE: DeadlineExceeded; detail code SubscriptionRequestsThrottled; detail message Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor '<uuid>' exceeded. Ple... message truncated; provider Microsoft.RedHatOpenShift Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s) | unknown | 1 | 2.33%1 of 43 job runs affected | IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s) | Show trend detailsOct 21: 0 · Oct 22: 0 · Oct 23: 0 · Oct 24: 1 · Oct 25: 0 · Oct 26: 0 · Oct 27: 0 · Oct 28: 0 | none | |||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade_blocked_pod_creation.go:201]: failed to create node pool "np-deny-pods" for cluster "cp-ystream-deny-pods-5-1-fjj5cf"
Unexpected error:
<*fmt.wrapError | 0x3b484fe8ea00>:
failed to create NodePool np-deny-pods: failed waiting for nodepool="np-deny-pods" for cluster "cp-ystream-deny-pods-5-1-fjj5cf" in resourcegroup="rg-cp-ystream-deny-pods-5-1-j5b76vjznvvh" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/cf1ae438-bf01-4f86-8ab5-784122b7cb5a
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/cf1ae438-bf01-4f86-8ab5-784122b7cb5a",
"name": "cf1ae438-bf01-4f86-8ab5-784122b7cb5a",
"status": "Failed",
"startTime": "2026-10-24T03:36:53.315230765Z",
"endTime": "2026-10-24T03:56:02.496643822Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [hypershiftNodePool] node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine z4z5b9p8d5g2j9j-np-deny-pods-8gxvs-vs65j: Failed: interfaces failed to create or update. err: failed to get existing resource rg-cp-ystream-deny-pods-5-1-j5b76vjznvvh-cp-ystream-den-2ba76655/z4z5b9p8d5g2j9j-np-deny-pods-8gxvs-vs65j-nic (service: interfaces): GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cp-ystream-deny-pods-5-1-j5b76vjznvvh-cp-ystream-den-2ba76655/providers/Microsoft.Network/networkInterfaces/z4z5b9p8d5g2j9j-np-deny-pods-8gxvs-vs65j-nic\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"SubscriptionRequestsThrottled\",\n \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor '6096f642-63a0-4dd6-958d-715a87f33535' exceeded. Ple... message truncated\n"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/cf1ae438-bf01-4f86-8ab5-784122b7cb5a
correlation ID: 3b16cd17-d78f-4b89-a83b-76ef2e36cf89
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade_blocked_pod_creation.go:201]: failed to create node pool "np-deny-pods" for cluster "cp-ystream-deny-pods-5-1-fjj5cf"
Unexpected error:
<*fmt.wrapError | 0x3b484fe8ea00>:
failed to create NodePool np-deny-pods: failed waiting for nodepool="np-deny-pods" for cluster "cp-ystream-deny-pods-5-1-fjj5cf" in resourcegroup="rg-cp-ystream-deny-pods-5-1-j5b76vjznvvh" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/cf1ae438-bf01-4f86-8ab5-784122b7cb5a
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/cf1ae438-bf01-4f86-8ab5-784122b7cb5a",
"name": "cf1ae438-bf01-4f86-8ab5-784122b7cb5a",
"status": "Failed",
"startTime": "2026-10-24T03:36:53.315230765Z",
"endTime": "2026-10-24T03:56:02.496643822Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [hypershiftNodePool] node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine z4z5b9p8d5g2j9j-np-deny-pods-8gxvs-vs65j: Failed: interfaces failed to create or update. err: failed to get existing resource rg-cp-ystream-deny-pods-5-1-j5b76vjznvvh-cp-ystream-den-2ba76655/z4z5b9p8d5g2j9j-np-deny-pods-8gxvs-vs65j-nic (service: interfaces): GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cp-ystream-deny-pods-5-1-j5b76vjznvvh-cp-ystream-den-2ba76655/providers/Microsoft.Network/networkInterfaces/z4z5b9p8d5g2j9j-np-deny-pods-8gxvs-vs65j-nic\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"SubscriptionRequestsThrottled\",\n \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor '6096f642-63a0-4dd6-958d-715a87f33535' exceeded. Ple... message truncated\n"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/cf1ae438-bf01-4f86-8ab5-784122b7cb5a
correlation ID: 3b16cd17-d78f-4b89-a83b-76ef2e36cf89
client request ID: unavailable
occurredContributing tests (1)
Affected runs (1)
| |||||||||||||||||||||||||||||||||||||||||||||||||||||
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet complete...full failure pattern: ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version <version> is Partial (want Completed), started <duration> ago; hosted cluster degraded: UnavailableReplicas: [router]; provider Microsoft.RedHatOpenShift Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s) | unknown | 1 | 2.33%1 of 43 job runs affected | IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s) | Show trend detailsOct 21: 0 · Oct 22: 0 · Oct 23: 0 · Oct 24: 1 · Oct 25: 0 · Oct 26: 0 · Oct 27: 0 · Oct 28: 0 | DEV | |||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:208]: failed to create HCP cluster "agg-cluster" with aggregated settings
Unexpected error:
<*fmt.wrapError | 0x9134b719420>:
failed waiting for cluster="agg-cluster" in resourcegroup="feature-aggregation-c2zb7x4tvbjz" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/2d4481c2-5c8d-4e2e-a526-9b051ca6139f
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/2d4481c2-5c8d-4e2e-a526-9b051ca6139f",
"name": "2d4481c2-5c8d-4e2e-a526-9b051ca6139f",
"status": "Failed",
"startTime": "2026-10-24T17:16:34.851278228Z",
"endTime": "2026-10-24T17:45:35.221810696Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.39 is Partial (want Completed), started 22m57s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/2d4481c2-5c8d-4e2e-a526-9b051ca6139f
correlation ID: 4e6d7a8e-ea3e-4773-abae-818d5133da86
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:208]: failed to create HCP cluster "agg-cluster" with aggregated settings
Unexpected error:
<*fmt.wrapError | 0x9134b719420>:
failed waiting for cluster="agg-cluster" in resourcegroup="feature-aggregation-c2zb7x4tvbjz" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/2d4481c2-5c8d-4e2e-a526-9b051ca6139f
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/2d4481c2-5c8d-4e2e-a526-9b051ca6139f",
"name": "2d4481c2-5c8d-4e2e-a526-9b051ca6139f",
"status": "Failed",
"startTime": "2026-10-24T17:16:34.851278228Z",
"endTime": "2026-10-24T17:45:35.221810696Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.39 is Partial (want Completed), started 22m57s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/2d4481c2-5c8d-4e2e-a526-9b051ca6139f
correlation ID: 4e6d7a8e-ea3e-4773-abae-818d5133da86
client request ID: unavailable
occurredContributing tests (1)
Affected runs (1)
| |||||||||||||||||||||||||||||||||||||||||||||||||||||
ERROR CODE: DeadlineExceeded; detail message node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hype...full failure pattern: ERROR CODE: DeadlineExceeded; detail message node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is <count>, want <count>; node pool condition AllNodesHealthy is False: <count> of <count> machines are not healthy Machine <node>: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.Compute/virtualMachines/<node> to exist Machine <node>: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.Compute/virtualMachines/<node> to exist; node pool condition AllMachinesReady is False: <count> of <count> machines are not ready Machine <node>: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.Compute/virtualMachines/<node> to exist Machine <node>: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.Compute/virtualMachines/<node> to exist; provider Microsoft.RedHatOpenShift Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s) | unknown | 1 | 2.33%1 of 43 job runs affected | IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s) | Show trend detailsOct 21: 0 · Oct 22: 1 · Oct 23: 0 · Oct 24: 0 · Oct 25: 0 · Oct 26: 0 · Oct 27: 0 · Oct 28: 0 | DEV, PROD | |||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/e2e/simple_negative_cases.go:159]: failed to create node pool np-1
Unexpected error:
<*fmt.wrapError | 0x1e6e8804d3c0>:
failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "negative-tests-cluster" in resourcegroup="rg-negative-tests-m5vrcvff2ctf" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/75b8b1f9-f718-45b2-b1ee-a3009f824cf8
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/75b8b1f9-f718-45b2-b1ee-a3009f824cf8",
"name": "75b8b1f9-f718-45b2-b1ee-a3009f824cf8",
"status": "Failed",
"startTime": "2026-10-22T14:53:42.149827787Z",
"endTime": "2026-10-22T15:12:50.02702561Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine c2g2u9m5x1z4p0m-np-1-bfdzc-5kdm2: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-negative-tests-m5vrcvff2ctf--managed/providers/Microsoft.Compute/virtualMachines/c2g2u9m5x1z4p0m-np-1-bfdzc-5kdm2 to exist\nMachine c2g2u9m5x1z4p0m-np-1-bfdzc-kbcws: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-negative-tests-m5vrcvff2ctf--managed/providers/Microsoft.Compute/virtualMachines/c2g2u9m5x1z4p0m-np-1-bfdzc-kbcws to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine c2g2u9m5x1z4p0m-np-1-bfdzc-5kdm2: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-negative-tests-m5vrcvff2ctf--managed/providers/Microsoft.Compute/virtualMachines/c2g2u9m5x1z4p0m-np-1-bfdzc-5kdm2 to exist\nMachine c2g2u9m5x1z4p0m-np-1-bfdzc-kbcws: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-negative-tests-m5vrcvff2ctf--managed/providers/Microsoft.Compute/virtualMachines/c2g2u9m5x1z4p0m-np-1-bfdzc-kbcws to exist\n"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/75b8b1f9-f718-45b2-b1ee-a3009f824cf8
correlation ID: e990e88e-110c-474c-9878-aa84f73234da
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/simple_negative_cases.go:159]: failed to create node pool np-1
Unexpected error:
<*fmt.wrapError | 0x1e6e8804d3c0>:
failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "negative-tests-cluster" in resourcegroup="rg-negative-tests-m5vrcvff2ctf" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/75b8b1f9-f718-45b2-b1ee-a3009f824cf8
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/75b8b1f9-f718-45b2-b1ee-a3009f824cf8",
"name": "75b8b1f9-f718-45b2-b1ee-a3009f824cf8",
"status": "Failed",
"startTime": "2026-10-22T14:53:42.149827787Z",
"endTime": "2026-10-22T15:12:50.02702561Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine c2g2u9m5x1z4p0m-np-1-bfdzc-5kdm2: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-negative-tests-m5vrcvff2ctf--managed/providers/Microsoft.Compute/virtualMachines/c2g2u9m5x1z4p0m-np-1-bfdzc-5kdm2 to exist\nMachine c2g2u9m5x1z4p0m-np-1-bfdzc-kbcws: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-negative-tests-m5vrcvff2ctf--managed/providers/Microsoft.Compute/virtualMachines/c2g2u9m5x1z4p0m-np-1-bfdzc-kbcws to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine c2g2u9m5x1z4p0m-np-1-bfdzc-5kdm2: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-negative-tests-m5vrcvff2ctf--managed/providers/Microsoft.Compute/virtualMachines/c2g2u9m5x1z4p0m-np-1-bfdzc-5kdm2 to exist\nMachine c2g2u9m5x1z4p0m-np-1-bfdzc-kbcws: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-negative-tests-m5vrcvff2ctf--managed/providers/Microsoft.Compute/virtualMachines/c2g2u9m5x1z4p0m-np-1-bfdzc-kbcws to exist\n"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/75b8b1f9-f718-45b2-b1ee-a3009f824cf8
correlation ID: e990e88e-110c-474c-9878-aa84f73234da
client request ID: unavailable
occurredContributing tests (1)
Affected runs (1)
| |||||||||||||||||||||||||||||||||||||||||||||||||||||
ERROR CODE: DeploymentFailed; detail code DeploymentScriptACIProvisioningTimeout; detail message The supporting resource: 'Azure Container Instance' did not provision during the e...full failure pattern: ERROR CODE: DeploymentFailed; detail code DeploymentScriptACIProvisioningTimeout; detail message The supporting resource: 'Azure Container Instance' did not provision during the expected time. Correlation Id: <uuid>. Please try again later, if the issue persists contact technical support for further investigation. Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s) | unknown | 1 | 2.33%1 of 43 job runs affected | IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s) | Show trend detailsOct 21: 0 · Oct 22: 1 · Oct 23: 0 · Oct 24: 0 · Oct 25: 0 · Oct 26: 0 · Oct 27: 0 · Oct 28: 0 | none | |||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/e2e/cluster_etcd_mhsm.go:106]: failed to deploy and activate Managed HSM
Unexpected error:
<*fmt.wrapError | 0x563a226e760>:
failed waiting for deployment "mhsm-deploy-etcd-mhsm" in resourcegroup="etcd-mhsm-blgzqwztp2gh" to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/etcd-mhsm-blgzqwztp2gh/providers/Microsoft.Resources/deployments/mhsm-deploy-etcd-mhsm/operationStatuses/08584089270308456271
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeploymentFailed
--------------------------------------------------------------------------------
{
"status": "Failed",
"error": {
"code": "DeploymentFailed",
"message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.",
"details": [
{
"code": "Conflict",
"message": "{\r\n \"status\": \"failed\",\r\n \"error\": {\r\n \"code\": \"ResourceDeploymentFailure\",\r\n \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'failed'.\",\r\n \"details\": [\r\n {\r\n \"code\": \"DeploymentScriptACIProvisioningTimeout\",\r\n \"message\": \"The supporting resource: 'Azure Container Instance' did not provision during the expected time. Correlation Id: c1661afd-ea85-487b-8e90-65a3f8664855. Please try again later, if the issue persists contact technical support for further investigation.\"\r\n }\r\n ]\r\n }\r\n}"
}
]
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/etcd-mhsm-blgzqwztp2gh/providers/Microsoft.Resources/deployments/mhsm-deploy-etcd-mhsm/operationStatuses/08584089270308456271
correlation ID: 989b1e9c-002b-494d-ae3b-825cc5f4adf2
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_etcd_mhsm.go:106]: failed to deploy and activate Managed HSM
Unexpected error:
<*fmt.wrapError | 0x563a226e760>:
failed waiting for deployment "mhsm-deploy-etcd-mhsm" in resourcegroup="etcd-mhsm-blgzqwztp2gh" to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/etcd-mhsm-blgzqwztp2gh/providers/Microsoft.Resources/deployments/mhsm-deploy-etcd-mhsm/operationStatuses/08584089270308456271
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeploymentFailed
--------------------------------------------------------------------------------
{
"status": "Failed",
"error": {
"code": "DeploymentFailed",
"message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.",
"details": [
{
"code": "Conflict",
"message": "{\r\n \"status\": \"failed\",\r\n \"error\": {\r\n \"code\": \"ResourceDeploymentFailure\",\r\n \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'failed'.\",\r\n \"details\": [\r\n {\r\n \"code\": \"DeploymentScriptACIProvisioningTimeout\",\r\n \"message\": \"The supporting resource: 'Azure Container Instance' did not provision during the expected time. Correlation Id: c1661afd-ea85-487b-8e90-65a3f8664855. Please try again later, if the issue persists contact technical support for further investigation.\"\r\n }\r\n ]\r\n }\r\n}"
}
]
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/etcd-mhsm-blgzqwztp2gh/providers/Microsoft.Resources/deployments/mhsm-deploy-etcd-mhsm/operationStatuses/08584089270308456271
correlation ID: 989b1e9c-002b-494d-ae3b-825cc5f4adf2
client request ID: unavailable
occurredContributing tests (1)
Affected runs (1)
| |||||||||||||||||||||||||||||||||||||||||||||||||||||
ERROR CODE: NotFound; detail message The resource record '*.e2e-dns-c2hldx' does not exist in resource group '<resource-group>' of subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX...full failure pattern: ERROR CODE: NotFound; detail message The resource record '*.e2e-dns-c2hldx' does not exist in resource group '<resource-group>' of subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX'.; provider Microsoft.Network Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s) | unknown | 1 | 2.33%1 of 43 job runs affected | IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s) | Show trend detailsOct 21: 0 · Oct 22: 0 · Oct 23: 0 · Oct 24: 0 · Oct 25: 1 · Oct 26: 0 · Oct 27: 0 · Oct 28: 0 | none | |||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:246]: failed to verify cluster operators on cluster "cluster-candidate-5-1-g8n765"
Unexpected error:
<*errors.joinError | 0x254aca410498>:
VerifyIngressDNSRecord(aro.u3d0c7m0f4m7m7j.sg1c.westus3.aroapp-hcp.azure-test.net) failed: VerifyIngressDNSRecord(aro.u3d0c7m0f4m7m7j.sg1c.westus3.aroapp-hcp.azure-test.net) timed out after 10m0.008s: DNS A record "*.e2e-dns-c2hldx" not yet present in zone rg-candidate-5-1-jqlpsk27qjbp-candidate-g8n765--managed/aro.u3d0c7m0f4m7m7j.sg1c.westus3.aroapp-hcp.azure-test.net: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-5-1-jqlpsk27qjbp-candidate-g8n765--managed/providers/Microsoft.Network/dnsZones/aro.u3d0c7m0f4m7m7j.sg1c.westus3.aroapp-hcp.azure-test.net/A/*.e2e-dns-c2hldx
--------------------------------------------------------------------------------
RESPONSE 404: 404 Not Found
ERROR CODE: NotFound
--------------------------------------------------------------------------------
{
"code": "NotFound",
"message": "The resource record '*.e2e-dns-c2hldx' does not exist in resource group 'rg-candidate-5-1-jqlpsk27qjbp-candidate-g8n765--managed' of subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX'."
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-5-1-jqlpsk27qjbp-candidate-g8n765--managed/providers/Microsoft.Network/dnsZones/aro.u3d0c7m0f4m7m7j.sg1c.westus3.aroapp-hcp.azure-test.net/A/*.e2e-dns-c2hldx
correlation ID: 192cc89e-15d5-4e23-8532-45b3cbb07059
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:246]: failed to verify cluster operators on cluster "cluster-candidate-5-1-g8n765"
Unexpected error:
<*errors.joinError | 0x254aca410498>:
VerifyIngressDNSRecord(aro.u3d0c7m0f4m7m7j.sg1c.westus3.aroapp-hcp.azure-test.net) failed: VerifyIngressDNSRecord(aro.u3d0c7m0f4m7m7j.sg1c.westus3.aroapp-hcp.azure-test.net) timed out after 10m0.008s: DNS A record "*.e2e-dns-c2hldx" not yet present in zone rg-candidate-5-1-jqlpsk27qjbp-candidate-g8n765--managed/aro.u3d0c7m0f4m7m7j.sg1c.westus3.aroapp-hcp.azure-test.net: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-5-1-jqlpsk27qjbp-candidate-g8n765--managed/providers/Microsoft.Network/dnsZones/aro.u3d0c7m0f4m7m7j.sg1c.westus3.aroapp-hcp.azure-test.net/A/*.e2e-dns-c2hldx
--------------------------------------------------------------------------------
RESPONSE 404: 404 Not Found
ERROR CODE: NotFound
--------------------------------------------------------------------------------
{
"code": "NotFound",
"message": "The resource record '*.e2e-dns-c2hldx' does not exist in resource group 'rg-candidate-5-1-jqlpsk27qjbp-candidate-g8n765--managed' of subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX'."
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-5-1-jqlpsk27qjbp-candidate-g8n765--managed/providers/Microsoft.Network/dnsZones/aro.u3d0c7m0f4m7m7j.sg1c.westus3.aroapp-hcp.azure-test.net/A/*.e2e-dns-c2hldx
correlation ID: 192cc89e-15d5-4e23-8532-45b3cbb07059
client request ID: unavailable
occurredContributing tests (1)
Affected runs (1)
| |||||||||||||||||||||||||||||||||||||||||||||||||||||
ERROR CODE: NotFound; detail message The resource record '*.e2e-dns-fn44p6' does not exist in resource group '<resource-group>' of subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX...full failure pattern: ERROR CODE: NotFound; detail message The resource record '*.e2e-dns-fn44p6' does not exist in resource group '<resource-group>' of subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX'.; provider Microsoft.Network Signal: IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | unknown | 1 | 2.33%1 of 43 job runs affected | IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | Show trend detailsOct 21: 0 · Oct 22: 0 · Oct 23: 0 · Oct 24: 0 · Oct 25: 0 · Oct 26: 1 · Oct 27: 0 · Oct 28: 0 | none | |||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:248]: failed to verify cluster operators on cluster "cluster-candidate-4-21-rf46z8"
Unexpected error:
<*errors.joinError | 0x3ac741d4138>:
VerifyIngressDNSRecord(aro.l5z2t8a8n2n6x2i.7cr2.westus3.aroapp-hcp.azure-test.net) failed: VerifyIngressDNSRecord(aro.l5z2t8a8n2n6x2i.7cr2.westus3.aroapp-hcp.azure-test.net) timed out after 10m0.001s: DNS A record "*.e2e-dns-fn44p6" not yet present in zone rg-candidate-4-21-vgzkq7npmrnr-candidate-rf46z8--managed/aro.l5z2t8a8n2n6x2i.7cr2.westus3.aroapp-hcp.azure-test.net: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-21-vgzkq7npmrnr-candidate-rf46z8--managed/providers/Microsoft.Network/dnsZones/aro.l5z2t8a8n2n6x2i.7cr2.westus3.aroapp-hcp.azure-test.net/A/*.e2e-dns-fn44p6
--------------------------------------------------------------------------------
RESPONSE 404: 404 Not Found
ERROR CODE: NotFound
--------------------------------------------------------------------------------
{
"code": "NotFound",
"message": "The resource record '*.e2e-dns-fn44p6' does not exist in resource group 'rg-candidate-4-21-vgzkq7npmrnr-candidate-rf46z8--managed' of subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX'."
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-21-vgzkq7npmrnr-candidate-rf46z8--managed/providers/Microsoft.Network/dnsZones/aro.l5z2t8a8n2n6x2i.7cr2.westus3.aroapp-hcp.azure-test.net/A/*.e2e-dns-fn44p6
correlation ID: bf6f75cd-f285-4895-ad70-7d62eff4cba2
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:248]: failed to verify cluster operators on cluster "cluster-candidate-4-21-rf46z8"
Unexpected error:
<*errors.joinError | 0x3ac741d4138>:
VerifyIngressDNSRecord(aro.l5z2t8a8n2n6x2i.7cr2.westus3.aroapp-hcp.azure-test.net) failed: VerifyIngressDNSRecord(aro.l5z2t8a8n2n6x2i.7cr2.westus3.aroapp-hcp.azure-test.net) timed out after 10m0.001s: DNS A record "*.e2e-dns-fn44p6" not yet present in zone rg-candidate-4-21-vgzkq7npmrnr-candidate-rf46z8--managed/aro.l5z2t8a8n2n6x2i.7cr2.westus3.aroapp-hcp.azure-test.net: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-21-vgzkq7npmrnr-candidate-rf46z8--managed/providers/Microsoft.Network/dnsZones/aro.l5z2t8a8n2n6x2i.7cr2.westus3.aroapp-hcp.azure-test.net/A/*.e2e-dns-fn44p6
--------------------------------------------------------------------------------
RESPONSE 404: 404 Not Found
ERROR CODE: NotFound
--------------------------------------------------------------------------------
{
"code": "NotFound",
"message": "The resource record '*.e2e-dns-fn44p6' does not exist in resource group 'rg-candidate-4-21-vgzkq7npmrnr-candidate-rf46z8--managed' of subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX'."
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-21-vgzkq7npmrnr-candidate-rf46z8--managed/providers/Microsoft.Network/dnsZones/aro.l5z2t8a8n2n6x2i.7cr2.westus3.aroapp-hcp.azure-test.net/A/*.e2e-dns-fn44p6
correlation ID: bf6f75cd-f285-4895-ad70-7d62eff4cba2
client request ID: unavailable
occurredContributing tests (1)
Affected runs (1)
| |||||||||||||||||||||||||||||||||||||||||||||||||||||
ERROR CODE: NotFound; detail message The resource record '*.e2e-dns-jgd8x5' does not exist in resource group '<resource-group>' of subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX...full failure pattern: ERROR CODE: NotFound; detail message The resource record '*.e2e-dns-jgd8x5' does not exist in resource group '<resource-group>' of subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX'.; provider Microsoft.Network Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s) | unknown | 1 | 2.33%1 of 43 job runs affected | IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s) | Show trend detailsOct 21: 0 · Oct 22: 0 · Oct 23: 0 · Oct 24: 0 · Oct 25: 1 · Oct 26: 0 · Oct 27: 0 · Oct 28: 0 | none | |||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:246]: failed to verify cluster operators on cluster "cluster-candidate-4-22-652kdm"
Unexpected error:
<*errors.joinError | 0x282c0f144570>:
VerifyIngressDNSRecord(aro.u8o6a8v1u0z8q8o.bssq.westus3.aroapp-hcp.azure-test.net) failed: VerifyIngressDNSRecord(aro.u8o6a8v1u0z8q8o.bssq.westus3.aroapp-hcp.azure-test.net) timed out after 10m0.006s: DNS A record "*.e2e-dns-jgd8x5" not yet present in zone rg-candidate-4-22-b89xsggl4sd5-candidate-652kdm--managed/aro.u8o6a8v1u0z8q8o.bssq.westus3.aroapp-hcp.azure-test.net: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-22-b89xsggl4sd5-candidate-652kdm--managed/providers/Microsoft.Network/dnsZones/aro.u8o6a8v1u0z8q8o.bssq.westus3.aroapp-hcp.azure-test.net/A/*.e2e-dns-jgd8x5
--------------------------------------------------------------------------------
RESPONSE 404: 404 Not Found
ERROR CODE: NotFound
--------------------------------------------------------------------------------
{
"code": "NotFound",
"message": "The resource record '*.e2e-dns-jgd8x5' does not exist in resource group 'rg-candidate-4-22-b89xsggl4sd5-candidate-652kdm--managed' of subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX'."
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-22-b89xsggl4sd5-candidate-652kdm--managed/providers/Microsoft.Network/dnsZones/aro.u8o6a8v1u0z8q8o.bssq.westus3.aroapp-hcp.azure-test.net/A/*.e2e-dns-jgd8x5
correlation ID: a77d314a-799e-4560-8840-6738610cf0af
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:246]: failed to verify cluster operators on cluster "cluster-candidate-4-22-652kdm"
Unexpected error:
<*errors.joinError | 0x282c0f144570>:
VerifyIngressDNSRecord(aro.u8o6a8v1u0z8q8o.bssq.westus3.aroapp-hcp.azure-test.net) failed: VerifyIngressDNSRecord(aro.u8o6a8v1u0z8q8o.bssq.westus3.aroapp-hcp.azure-test.net) timed out after 10m0.006s: DNS A record "*.e2e-dns-jgd8x5" not yet present in zone rg-candidate-4-22-b89xsggl4sd5-candidate-652kdm--managed/aro.u8o6a8v1u0z8q8o.bssq.westus3.aroapp-hcp.azure-test.net: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-22-b89xsggl4sd5-candidate-652kdm--managed/providers/Microsoft.Network/dnsZones/aro.u8o6a8v1u0z8q8o.bssq.westus3.aroapp-hcp.azure-test.net/A/*.e2e-dns-jgd8x5
--------------------------------------------------------------------------------
RESPONSE 404: 404 Not Found
ERROR CODE: NotFound
--------------------------------------------------------------------------------
{
"code": "NotFound",
"message": "The resource record '*.e2e-dns-jgd8x5' does not exist in resource group 'rg-candidate-4-22-b89xsggl4sd5-candidate-652kdm--managed' of subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX'."
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-22-b89xsggl4sd5-candidate-652kdm--managed/providers/Microsoft.Network/dnsZones/aro.u8o6a8v1u0z8q8o.bssq.westus3.aroapp-hcp.azure-test.net/A/*.e2e-dns-jgd8x5
correlation ID: a77d314a-799e-4560-8840-6738610cf0af
client request ID: unavailable
occurredContributing tests (1)
Affected runs (1)
| |||||||||||||||||||||||||||||||||||||||||||||||||||||
ERROR CODE: NotFound; detail message The resource record '*.e2e-dns-rg99mn' does not exist in resource group '<resource-group>' of subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX...full failure pattern: ERROR CODE: NotFound; detail message The resource record '*.e2e-dns-rg99mn' does not exist in resource group '<resource-group>' of subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX'.; provider Microsoft.Network Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s) | unknown | 1 | 2.33%1 of 43 job runs affected | IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s) | Show trend detailsOct 21: 0 · Oct 22: 0 · Oct 23: 0 · Oct 24: 0 · Oct 25: 1 · Oct 26: 0 · Oct 27: 0 · Oct 28: 0 | none | |||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:246]: failed to verify cluster operators on cluster "cluster-candidate-4-22-4fhvgp"
Unexpected error:
<*errors.joinError | 0x3188fb29e5a0>:
VerifyIngressDNSRecord(aro.d8c1z3j0o2p5g9d.3jyi.westus3.aroapp-hcp.azure-test.net) failed: VerifyIngressDNSRecord(aro.d8c1z3j0o2p5g9d.3jyi.westus3.aroapp-hcp.azure-test.net) timed out after 10m0.005s: DNS A record "*.e2e-dns-rg99mn" not yet present in zone rg-candidate-4-22-sp7fr6hfrw9x-candidate-4fhvgp--managed/aro.d8c1z3j0o2p5g9d.3jyi.westus3.aroapp-hcp.azure-test.net: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-22-sp7fr6hfrw9x-candidate-4fhvgp--managed/providers/Microsoft.Network/dnsZones/aro.d8c1z3j0o2p5g9d.3jyi.westus3.aroapp-hcp.azure-test.net/A/*.e2e-dns-rg99mn
--------------------------------------------------------------------------------
RESPONSE 404: 404 Not Found
ERROR CODE: NotFound
--------------------------------------------------------------------------------
{
"code": "NotFound",
"message": "The resource record '*.e2e-dns-rg99mn' does not exist in resource group 'rg-candidate-4-22-sp7fr6hfrw9x-candidate-4fhvgp--managed' of subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX'."
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-22-sp7fr6hfrw9x-candidate-4fhvgp--managed/providers/Microsoft.Network/dnsZones/aro.d8c1z3j0o2p5g9d.3jyi.westus3.aroapp-hcp.azure-test.net/A/*.e2e-dns-rg99mn
correlation ID: cfd4563d-f15e-486d-ad09-c0f336987b6a
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:246]: failed to verify cluster operators on cluster "cluster-candidate-4-22-4fhvgp"
Unexpected error:
<*errors.joinError | 0x3188fb29e5a0>:
VerifyIngressDNSRecord(aro.d8c1z3j0o2p5g9d.3jyi.westus3.aroapp-hcp.azure-test.net) failed: VerifyIngressDNSRecord(aro.d8c1z3j0o2p5g9d.3jyi.westus3.aroapp-hcp.azure-test.net) timed out after 10m0.005s: DNS A record "*.e2e-dns-rg99mn" not yet present in zone rg-candidate-4-22-sp7fr6hfrw9x-candidate-4fhvgp--managed/aro.d8c1z3j0o2p5g9d.3jyi.westus3.aroapp-hcp.azure-test.net: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-22-sp7fr6hfrw9x-candidate-4fhvgp--managed/providers/Microsoft.Network/dnsZones/aro.d8c1z3j0o2p5g9d.3jyi.westus3.aroapp-hcp.azure-test.net/A/*.e2e-dns-rg99mn
--------------------------------------------------------------------------------
RESPONSE 404: 404 Not Found
ERROR CODE: NotFound
--------------------------------------------------------------------------------
{
"code": "NotFound",
"message": "The resource record '*.e2e-dns-rg99mn' does not exist in resource group 'rg-candidate-4-22-sp7fr6hfrw9x-candidate-4fhvgp--managed' of subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX'."
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-22-sp7fr6hfrw9x-candidate-4fhvgp--managed/providers/Microsoft.Network/dnsZones/aro.d8c1z3j0o2p5g9d.3jyi.westus3.aroapp-hcp.azure-test.net/A/*.e2e-dns-rg99mn
correlation ID: cfd4563d-f15e-486d-ad09-c0f336987b6a
client request ID: unavailable
occurredContributing tests (1)
Affected runs (1)
| |||||||||||||||||||||||||||||||||||||||||||||||||||||
Internal error occurred: failed calling webhook "webhookfull failure pattern: Internal error occurred: failed calling webhook "webhook Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s) | unknown | 1 | 2.33%1 of 43 job runs affected | IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s) | Show trend detailsOct 21: 0 · Oct 22: 0 · Oct 23: 0 · Oct 24: 1 · Oct 25: 0 · Oct 26: 0 · Oct 27: 0 · Oct 28: 0 | DEV | |||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:372]: Timed out after 120.003s.
ClusterIssuer CRD should be installed and the resource creatable
Expected success, but got an error:
<*errors.StatusError | 0x11d151799220>:
Internal error occurred: failed calling webhook "webhook.cert-manager.io": failed to call webhook: Post "https://cert-manager-webhook.cert-manager.svc:443/validate?timeout=30s": no endpoints available for service "cert-manager-webhook"
{
ErrStatus: {
TypeMeta: {Kind: "Status", APIVersion: "v1"},
ListMeta: {
SelfLink: "",
ResourceVersion: "",
Continue: "",
RemainingItemCount: nil,
ShardInfo: nil,
},
Status: "Failure",
Message: "Internal error occurred: failed calling webhook \"webhook.cert-manager.io\": failed to call webhook: Post \"https://cert-manager-webhook.cert-manager.svc:443/validate?timeout=30s\": no endpoints available for service \"cert-manager-webhook\"",
Reason: "InternalError",
Details: {
Name: "",
Group: "",
Kind: "",
UID: "",
Causes: [
{
Type: "",
Message: "failed calling webhook \"webhook.cert-manager.io\": failed to call webhook: Post \"https://cert-manager-webhook.cert-manager.svc:443/validate?timeout=30s\": no endpoints available for service \"cert-manager-webhook\"",
Field: "",
},
],
RetryAfterSeconds: 0,
},
Code: 500,
},
}
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:372]: Timed out after 120.003s.
ClusterIssuer CRD should be installed and the resource creatable
Expected success, but got an error:
<*errors.StatusError | 0x11d151799220>:
Internal error occurred: failed calling webhook "webhook.cert-manager.io": failed to call webhook: Post "https://cert-manager-webhook.cert-manager.svc:443/validate?timeout=30s": no endpoints available for service "cert-manager-webhook"
{
ErrStatus: {
TypeMeta: {Kind: "Status", APIVersion: "v1"},
ListMeta: {
SelfLink: "",
ResourceVersion: "",
Continue: "",
RemainingItemCount: nil,
ShardInfo: nil,
},
Status: "Failure",
Message: "Internal error occurred: failed calling webhook \"webhook.cert-manager.io\": failed to call webhook: Post \"https://cert-manager-webhook.cert-manager.svc:443/validate?timeout=30s\": no endpoints available for service \"cert-manager-webhook\"",
Reason: "InternalError",
Details: {
Name: "",
Group: "",
Kind: "",
UID: "",
Causes: [
{
Type: "",
Message: "failed calling webhook \"webhook.cert-manager.io\": failed to call webhook: Post \"https://cert-manager-webhook.cert-manager.svc:443/validate?timeout=30s\": no endpoints available for service \"cert-manager-webhook\"",
Field: "",
},
],
RetryAfterSeconds: 0,
},
Code: 500,
},
}Contributing tests (1)
Affected runs (1)
| |||||||||||||||||||||||||||||||||||||||||||||||||||||
VerifyCloudNetworkConfigEgressIP(<ip>) failed: failed to label node <node> egress-assignable: failed to apply k8s.ovn.org/egress-assignable label to node "<node>": nodes "<node>" ...full failure pattern: VerifyCloudNetworkConfigEgressIP(<ip>) failed: failed to label node <node> egress-assignable: failed to apply k8s.ovn.org/egress-assignable label to node "<node>": nodes "<node>" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not Signal: IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | unknown | 1 | 2.33%1 of 43 job runs affected | IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | Show trend detailsOct 21: 0 · Oct 22: 0 · Oct 23: 0 · Oct 24: 0 · Oct 25: 0 · Oct 26: 0 · Oct 27: 1 · Oct 28: 0 | DEV | |||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:289]: failed to verify cluster operators on cluster "cluster-candidate-4-22-cghxqb"
Unexpected error:
<*errors.joinError | 0x315939ea25b8>:
VerifyCloudNetworkConfigEgressIP(10.0.0.7) failed: failed to label node p3m2y8l9j1y3v1d-np-cghxqb-7rwcj-2g4cv egress-assignable: failed to apply k8s.ovn.org/egress-assignable label to node "p3m2y8l9j1y3v1d-np-cghxqb-7rwcj-2g4cv": nodes "p3m2y8l9j1y3v1d-np-cghxqb-7rwcj-2g4cv" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized
{
errs: [
<*fmt.wrapError | 0x31593941d8c0>{
msg: "VerifyCloudNetworkConfigEgressIP(10.0.0.7) failed: failed to label node p3m2y8l9j1y3v1d-np-cghxqb-7rwcj-2g4cv egress-assignable: failed to apply k8s.ovn.org/egress-assignable label to node \"p3m2y8l9j1y3v1d-np-cghxqb-7rwcj-2g4cv\": nodes \"p3m2y8l9j1y3v1d-np-cghxqb-7rwcj-2g4cv\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
err: <*fmt.wrapError | 0x31593941d880>{
msg: "failed to label node p3m2y8l9j1y3v1d-np-cghxqb-7rwcj-2g4cv egress-assignable: failed to apply k8s.ovn.org/egress-assignable label to node \"p3m2y8l9j1y3v1d-np-cghxqb-7rwcj-2g4cv\": nodes \"p3m2y8l9j1y3v1d-np-cghxqb-7rwcj-2g4cv\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
err: <*fmt.wrapError | 0x31593941d840>{
msg: "failed to apply k8s.ovn.org/egress-assignable label to node \"p3m2y8l9j1y3v1d-np-cghxqb-7rwcj-2g4cv\": nodes \"p3m2y8l9j1y3v1d-np-cghxqb-7rwcj-2g4cv\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
err: <*errors.StatusError | 0x31593a26f680>{
ErrStatus: {
TypeMeta: {Kind: "", APIVersion: ""},
ListMeta: {
SelfLink: "",
ResourceVersion: "",
Continue: "",
RemainingItemCount: nil,
ShardInfo: nil,
},
Status: "Failure",
Message: "nodes \"p3m2y8l9j1y3v1d-np-cghxqb-7rwcj-2g4cv\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
Reason: "Forbidden",
Details: {
Name: "p3m2y8l9j1y3v1d-np-cghxqb-7rwcj-2g4cv",
Group: "",
Kind: "nodes",
UID: "",
Causes: nil,
RetryAfterSeconds: 0,
},
Code: 403,
},
},
},
},
},
],
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:289]: failed to verify cluster operators on cluster "cluster-candidate-4-22-cghxqb"
Unexpected error:
<*errors.joinError | 0x315939ea25b8>:
VerifyCloudNetworkConfigEgressIP(10.0.0.7) failed: failed to label node p3m2y8l9j1y3v1d-np-cghxqb-7rwcj-2g4cv egress-assignable: failed to apply k8s.ovn.org/egress-assignable label to node "p3m2y8l9j1y3v1d-np-cghxqb-7rwcj-2g4cv": nodes "p3m2y8l9j1y3v1d-np-cghxqb-7rwcj-2g4cv" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized
{
errs: [
<*fmt.wrapError | 0x31593941d8c0>{
msg: "VerifyCloudNetworkConfigEgressIP(10.0.0.7) failed: failed to label node p3m2y8l9j1y3v1d-np-cghxqb-7rwcj-2g4cv egress-assignable: failed to apply k8s.ovn.org/egress-assignable label to node \"p3m2y8l9j1y3v1d-np-cghxqb-7rwcj-2g4cv\": nodes \"p3m2y8l9j1y3v1d-np-cghxqb-7rwcj-2g4cv\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
err: <*fmt.wrapError | 0x31593941d880>{
msg: "failed to label node p3m2y8l9j1y3v1d-np-cghxqb-7rwcj-2g4cv egress-assignable: failed to apply k8s.ovn.org/egress-assignable label to node \"p3m2y8l9j1y3v1d-np-cghxqb-7rwcj-2g4cv\": nodes \"p3m2y8l9j1y3v1d-np-cghxqb-7rwcj-2g4cv\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
err: <*fmt.wrapError | 0x31593941d840>{
msg: "failed to apply k8s.ovn.org/egress-assignable label to node \"p3m2y8l9j1y3v1d-np-cghxqb-7rwcj-2g4cv\": nodes \"p3m2y8l9j1y3v1d-np-cghxqb-7rwcj-2g4cv\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
err: <*errors.StatusError | 0x31593a26f680>{
ErrStatus: {
TypeMeta: {Kind: "", APIVersion: ""},
ListMeta: {
SelfLink: "",
ResourceVersion: "",
Continue: "",
RemainingItemCount: nil,
ShardInfo: nil,
},
Status: "Failure",
Message: "nodes \"p3m2y8l9j1y3v1d-np-cghxqb-7rwcj-2g4cv\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
Reason: "Forbidden",
Details: {
Name: "p3m2y8l9j1y3v1d-np-cghxqb-7rwcj-2g4cv",
Group: "",
Kind: "nodes",
UID: "",
Causes: nil,
RetryAfterSeconds: 0,
},
Code: 403,
},
},
},
},
},
],
}
occurredContributing tests (1)
Affected runs (1)
| |||||||||||||||||||||||||||||||||||||||||||||||||||||
clusterIssuer not ready: reason=ErrRegisterACMEAccount message=Failed to register ACME account: Get "<url>full failure pattern: clusterIssuer not ready: reason=ErrRegisterACMEAccount message=Failed to register ACME account: Get "<url> Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s) | unknown | 1 | 2.33%1 of 43 job runs affected | IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s) | Show trend detailsOct 21: 0 · Oct 22: 0 · Oct 23: 1 · Oct 24: 0 · Oct 25: 0 · Oct 26: 0 · Oct 27: 0 · Oct 28: 0 | DEV | |||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:391]: Timed out after 300.002s.
ClusterIssuer should become Ready (ACME account registered)
Expected success, but got an error:
<*errors.errorString | 0x1e28491f1160>:
clusterIssuer not ready: reason=ErrRegisterACMEAccount message=Failed to register ACME account: Get "https://acme-staging-v02.api.letsencrypt.org/directory": dial tcp: lookup acme-staging-v02.api.letsencrypt.org on 172.30.0.10:53: read udp 10.129.0.21:39416->172.30.0.10:53: read: connection refused
{
s: "clusterIssuer not ready: reason=ErrRegisterACMEAccount message=Failed to register ACME account: Get \"https://acme-staging-v02.api.letsencrypt.org/directory\": dial tcp: lookup acme-staging-v02.api.letsencrypt.org on 172.30.0.10:53: read udp 10.129.0.21:39416->172.30.0.10:53: read: connection refused",
}
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:391]: Timed out after 300.002s.
ClusterIssuer should become Ready (ACME account registered)
Expected success, but got an error:
<*errors.errorString | 0x1e28491f1160>:
clusterIssuer not ready: reason=ErrRegisterACMEAccount message=Failed to register ACME account: Get "https://acme-staging-v02.api.letsencrypt.org/directory": dial tcp: lookup acme-staging-v02.api.letsencrypt.org on 172.30.0.10:53: read udp 10.129.0.21:39416->172.30.0.10:53: read: connection refused
{
s: "clusterIssuer not ready: reason=ErrRegisterACMEAccount message=Failed to register ACME account: Get \"https://acme-staging-v02.api.letsencrypt.org/directory\": dial tcp: lookup acme-staging-v02.api.letsencrypt.org on 172.30.0.10:53: read udp 10.129.0.21:39416->172.30.0.10:53: read: connection refused",
}Contributing tests (1)
Affected runs (1)
| |||||||||||||||||||||||||||||||||||||||||||||||||||||
console fixture namespace "<namespace>": configure Console ExternalAuth through RP: context deadline exceeded; Dex namespace="console-lrpv2qd"; Deployment/dex condition="Available...full failure pattern: console fixture namespace "<namespace>": configure Console ExternalAuth through RP: context deadline exceeded; Dex namespace="console-lrpv2qd"; Deployment/dex condition="Available" status="True" reason="MinimumReplicasAvailable"; Deployment/dex Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s) | unknown | 1 | 2.33%1 of 43 job runs affected | IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s) | Show trend detailsOct 21: 0 · Oct 22: 0 · Oct 23: 1 · Oct 24: 0 · Oct 25: 0 · Oct 26: 0 · Oct 27: 0 · Oct 28: 0 | none | |||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:197]: failed to set up Console login fixture for cluster "cluster-candidate-4-20-4x46vw"
Unexpected error:
<*fmt.wrapError | 0x3b1760358220>:
console fixture namespace "console-lrpv2qd": configure Console ExternalAuth through RP: context deadline exceeded; Dex namespace="console-lrpv2qd"; Deployment/dex condition="Available" status="True" reason="MinimumReplicasAvailable"; Deployment/dex condition="Progressing" status="True" reason="NewReplicaSetAvailable"; Deployment/dex generation=1 observed=1 replicas=1 ready=1 available=1; Dex Pods=1; Dex ReplicaSets=1; Pod/dex-78f4dcc67d-zhbqp condition="ContainersReady" status="True" reason=""; Pod/dex-78f4dcc67d-zhbqp condition="Initialized" status="True" reason=""; Pod/dex-78f4dcc67d-zhbqp condition="PodReadyToStartContainers" status="True" reason=""; Pod/dex-78f4dcc67d-zhbqp condition="PodScheduled" status="True" reason=""; Pod/dex-78f4dcc67d-zhbqp condition="Ready" status="True" reason=""; Pod/dex-78f4dcc67d-zhbqp container="dex" ready=true restarts=0; Pod/dex-78f4dcc67d-zhbqp phase="Running" reason=""
{
msg: "console fixture namespace \"console-lrpv2qd\": configure Console ExternalAuth through RP: context deadline exceeded; Dex namespace=\"console-lrpv2qd\"; Deployment/dex condition=\"Available\" status=\"True\" reason=\"MinimumReplicasAvailable\"; Deployment/dex condition=\"Progressing\" status=\"True\" reason=\"NewReplicaSetAvailable\"; Deployment/dex generation=1 observed=1 replicas=1 ready=1 available=1; Dex Pods=1; Dex ReplicaSets=1; Pod/dex-78f4dcc67d-zhbqp condition=\"ContainersReady\" status=\"True\" reason=\"\"; Pod/dex-78f4dcc67d-zhbqp condition=\"Initialized\" status=\"True\" reason=\"\"; Pod/dex-78f4dcc67d-zhbqp condition=\"PodReadyToStartContainers\" status=\"True\" reason=\"\"; Pod/dex-78f4dcc67d-zhbqp condition=\"PodScheduled\" status=\"True\" reason=\"\"; Pod/dex-78f4dcc67d-zhbqp condition=\"Ready\" status=\"True\" reason=\"\"; Pod/dex-78f4dcc67d-zhbqp container=\"dex\" ready=true restarts=0; Pod/dex-78f4dcc67d-zhbqp phase=\"Running\" reason=\"\"",
err: <*fmt.wrapError | 0x3b1760b23ba0>{
msg: "configure Console ExternalAuth through RP: context deadline exceeded",
err: <context.deadlineExceededError>{},
},
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:197]: failed to set up Console login fixture for cluster "cluster-candidate-4-20-4x46vw"
Unexpected error:
<*fmt.wrapError | 0x3b1760358220>:
console fixture namespace "console-lrpv2qd": configure Console ExternalAuth through RP: context deadline exceeded; Dex namespace="console-lrpv2qd"; Deployment/dex condition="Available" status="True" reason="MinimumReplicasAvailable"; Deployment/dex condition="Progressing" status="True" reason="NewReplicaSetAvailable"; Deployment/dex generation=1 observed=1 replicas=1 ready=1 available=1; Dex Pods=1; Dex ReplicaSets=1; Pod/dex-78f4dcc67d-zhbqp condition="ContainersReady" status="True" reason=""; Pod/dex-78f4dcc67d-zhbqp condition="Initialized" status="True" reason=""; Pod/dex-78f4dcc67d-zhbqp condition="PodReadyToStartContainers" status="True" reason=""; Pod/dex-78f4dcc67d-zhbqp condition="PodScheduled" status="True" reason=""; Pod/dex-78f4dcc67d-zhbqp condition="Ready" status="True" reason=""; Pod/dex-78f4dcc67d-zhbqp container="dex" ready=true restarts=0; Pod/dex-78f4dcc67d-zhbqp phase="Running" reason=""
{
msg: "console fixture namespace \"console-lrpv2qd\": configure Console ExternalAuth through RP: context deadline exceeded; Dex namespace=\"console-lrpv2qd\"; Deployment/dex condition=\"Available\" status=\"True\" reason=\"MinimumReplicasAvailable\"; Deployment/dex condition=\"Progressing\" status=\"True\" reason=\"NewReplicaSetAvailable\"; Deployment/dex generation=1 observed=1 replicas=1 ready=1 available=1; Dex Pods=1; Dex ReplicaSets=1; Pod/dex-78f4dcc67d-zhbqp condition=\"ContainersReady\" status=\"True\" reason=\"\"; Pod/dex-78f4dcc67d-zhbqp condition=\"Initialized\" status=\"True\" reason=\"\"; Pod/dex-78f4dcc67d-zhbqp condition=\"PodReadyToStartContainers\" status=\"True\" reason=\"\"; Pod/dex-78f4dcc67d-zhbqp condition=\"PodScheduled\" status=\"True\" reason=\"\"; Pod/dex-78f4dcc67d-zhbqp condition=\"Ready\" status=\"True\" reason=\"\"; Pod/dex-78f4dcc67d-zhbqp container=\"dex\" ready=true restarts=0; Pod/dex-78f4dcc67d-zhbqp phase=\"Running\" reason=\"\"",
err: <*fmt.wrapError | 0x3b1760b23ba0>{
msg: "configure Console ExternalAuth through RP: context deadline exceeded",
err: <context.deadlineExceededError>{},
},
}
occurredContributing tests (1)
Affected runs (1)
| |||||||||||||||||||||||||||||||||||||||||||||||||||||
console fixture namespace "<namespace>": configure Console ExternalAuth through RP: context deadline exceeded; Dex namespace="console-mr8rvbh"; Deployment/dex condition="Available...full failure pattern: console fixture namespace "<namespace>": configure Console ExternalAuth through RP: context deadline exceeded; Dex namespace="console-mr8rvbh"; Deployment/dex condition="Available" status="True" reason="MinimumReplicasAvailable"; Deployment/dex Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s) | unknown | 1 | 2.33%1 of 43 job runs affected | IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s) | Show trend detailsOct 21: 0 · Oct 22: 1 · Oct 23: 0 · Oct 24: 0 · Oct 25: 0 · Oct 26: 0 · Oct 27: 0 · Oct 28: 0 | none | |||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:194]: failed to set up Console login fixture for cluster "cluster-candidate-5-0-6rxg4b"
Unexpected error:
<*fmt.wrapError | 0x12de498cec20>:
console fixture namespace "console-mr8rvbh": configure Console ExternalAuth through RP: context deadline exceeded; Dex namespace="console-mr8rvbh"; Deployment/dex condition="Available" status="True" reason="MinimumReplicasAvailable"; Deployment/dex condition="Progressing" status="True" reason="NewReplicaSetAvailable"; Deployment/dex generation=1 observed=1 replicas=1 ready=1 available=1; Dex Pods=1; Dex ReplicaSets=1; Pod/dex-77657bcfc5-zhr7j condition="ContainersReady" status="True" reason=""; Pod/dex-77657bcfc5-zhr7j condition="Initialized" status="True" reason=""; Pod/dex-77657bcfc5-zhr7j condition="PodReadyToStartContainers" status="True" reason=""; Pod/dex-77657bcfc5-zhr7j condition="PodScheduled" status="True" reason=""; Pod/dex-77657bcfc5-zhr7j condition="Ready" status="True" reason=""; Pod/dex-77657bcfc5-zhr7j container="dex" ready=true restarts=0; Pod/dex-77657bcfc5-zhr7j phase="Running" reason=""
{
msg: "console fixture namespace \"console-mr8rvbh\": configure Console ExternalAuth through RP: context deadline exceeded; Dex namespace=\"console-mr8rvbh\"; Deployment/dex condition=\"Available\" status=\"True\" reason=\"MinimumReplicasAvailable\"; Deployment/dex condition=\"Progressing\" status=\"True\" reason=\"NewReplicaSetAvailable\"; Deployment/dex generation=1 observed=1 replicas=1 ready=1 available=1; Dex Pods=1; Dex ReplicaSets=1; Pod/dex-77657bcfc5-zhr7j condition=\"ContainersReady\" status=\"True\" reason=\"\"; Pod/dex-77657bcfc5-zhr7j condition=\"Initialized\" status=\"True\" reason=\"\"; Pod/dex-77657bcfc5-zhr7j condition=\"PodReadyToStartContainers\" status=\"True\" reason=\"\"; Pod/dex-77657bcfc5-zhr7j condition=\"PodScheduled\" status=\"True\" reason=\"\"; Pod/dex-77657bcfc5-zhr7j condition=\"Ready\" status=\"True\" reason=\"\"; Pod/dex-77657bcfc5-zhr7j container=\"dex\" ready=true restarts=0; Pod/dex-77657bcfc5-zhr7j phase=\"Running\" reason=\"\"",
err: <*fmt.wrapError | 0x12de4a14f120>{
msg: "configure Console ExternalAuth through RP: context deadline exceeded",
err: <context.deadlineExceededError>{},
},
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:194]: failed to set up Console login fixture for cluster "cluster-candidate-5-0-6rxg4b"
Unexpected error:
<*fmt.wrapError | 0x12de498cec20>:
console fixture namespace "console-mr8rvbh": configure Console ExternalAuth through RP: context deadline exceeded; Dex namespace="console-mr8rvbh"; Deployment/dex condition="Available" status="True" reason="MinimumReplicasAvailable"; Deployment/dex condition="Progressing" status="True" reason="NewReplicaSetAvailable"; Deployment/dex generation=1 observed=1 replicas=1 ready=1 available=1; Dex Pods=1; Dex ReplicaSets=1; Pod/dex-77657bcfc5-zhr7j condition="ContainersReady" status="True" reason=""; Pod/dex-77657bcfc5-zhr7j condition="Initialized" status="True" reason=""; Pod/dex-77657bcfc5-zhr7j condition="PodReadyToStartContainers" status="True" reason=""; Pod/dex-77657bcfc5-zhr7j condition="PodScheduled" status="True" reason=""; Pod/dex-77657bcfc5-zhr7j condition="Ready" status="True" reason=""; Pod/dex-77657bcfc5-zhr7j container="dex" ready=true restarts=0; Pod/dex-77657bcfc5-zhr7j phase="Running" reason=""
{
msg: "console fixture namespace \"console-mr8rvbh\": configure Console ExternalAuth through RP: context deadline exceeded; Dex namespace=\"console-mr8rvbh\"; Deployment/dex condition=\"Available\" status=\"True\" reason=\"MinimumReplicasAvailable\"; Deployment/dex condition=\"Progressing\" status=\"True\" reason=\"NewReplicaSetAvailable\"; Deployment/dex generation=1 observed=1 replicas=1 ready=1 available=1; Dex Pods=1; Dex ReplicaSets=1; Pod/dex-77657bcfc5-zhr7j condition=\"ContainersReady\" status=\"True\" reason=\"\"; Pod/dex-77657bcfc5-zhr7j condition=\"Initialized\" status=\"True\" reason=\"\"; Pod/dex-77657bcfc5-zhr7j condition=\"PodReadyToStartContainers\" status=\"True\" reason=\"\"; Pod/dex-77657bcfc5-zhr7j condition=\"PodScheduled\" status=\"True\" reason=\"\"; Pod/dex-77657bcfc5-zhr7j condition=\"Ready\" status=\"True\" reason=\"\"; Pod/dex-77657bcfc5-zhr7j container=\"dex\" ready=true restarts=0; Pod/dex-77657bcfc5-zhr7j phase=\"Running\" reason=\"\"",
err: <*fmt.wrapError | 0x12de4a14f120>{
msg: "configure Console ExternalAuth through RP: context deadline exceeded",
err: <context.deadlineExceededError>{},
},
}
occurredContributing tests (1)
Affected runs (1)
| |||||||||||||||||||||||||||||||||||||||||||||||||||||
console fixture namespace "<namespace>": configure Console ExternalAuth through RP: context deadline exceeded; Dex namespace="console-n2bkhw2"; Deployment/dex condition="Available...full failure pattern: console fixture namespace "<namespace>": configure Console ExternalAuth through RP: context deadline exceeded; Dex namespace="console-n2bkhw2"; Deployment/dex condition="Available" status="True" reason="MinimumReplicasAvailable"; Deployment/dex Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s) | unknown | 1 | 2.33%1 of 43 job runs affected | IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s) | Show trend detailsOct 21: 0 · Oct 22: 0 · Oct 23: 1 · Oct 24: 0 · Oct 25: 0 · Oct 26: 0 · Oct 27: 0 · Oct 28: 0 | none | |||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:197]: failed to set up Console login fixture for cluster "cluster-candidate-5-1-qx8d44"
Unexpected error:
<*fmt.wrapError | 0x39e6141ab040>:
console fixture namespace "console-n2bkhw2": configure Console ExternalAuth through RP: context deadline exceeded; Dex namespace="console-n2bkhw2"; Deployment/dex condition="Available" status="True" reason="MinimumReplicasAvailable"; Deployment/dex condition="Progressing" status="True" reason="NewReplicaSetAvailable"; Deployment/dex generation=1 observed=1 replicas=1 ready=1 available=1; Dex Pods=1; Dex ReplicaSets=1; Pod/dex-77657bcfc5-nrx7d condition="ContainersReady" status="True" reason=""; Pod/dex-77657bcfc5-nrx7d condition="Initialized" status="True" reason=""; Pod/dex-77657bcfc5-nrx7d condition="PodReadyToStartContainers" status="True" reason=""; Pod/dex-77657bcfc5-nrx7d condition="PodScheduled" status="True" reason=""; Pod/dex-77657bcfc5-nrx7d condition="Ready" status="True" reason=""; Pod/dex-77657bcfc5-nrx7d container="dex" ready=true restarts=0; Pod/dex-77657bcfc5-nrx7d phase="Running" reason=""
{
msg: "console fixture namespace \"console-n2bkhw2\": configure Console ExternalAuth through RP: context deadline exceeded; Dex namespace=\"console-n2bkhw2\"; Deployment/dex condition=\"Available\" status=\"True\" reason=\"MinimumReplicasAvailable\"; Deployment/dex condition=\"Progressing\" status=\"True\" reason=\"NewReplicaSetAvailable\"; Deployment/dex generation=1 observed=1 replicas=1 ready=1 available=1; Dex Pods=1; Dex ReplicaSets=1; Pod/dex-77657bcfc5-nrx7d condition=\"ContainersReady\" status=\"True\" reason=\"\"; Pod/dex-77657bcfc5-nrx7d condition=\"Initialized\" status=\"True\" reason=\"\"; Pod/dex-77657bcfc5-nrx7d condition=\"PodReadyToStartContainers\" status=\"True\" reason=\"\"; Pod/dex-77657bcfc5-nrx7d condition=\"PodScheduled\" status=\"True\" reason=\"\"; Pod/dex-77657bcfc5-nrx7d condition=\"Ready\" status=\"True\" reason=\"\"; Pod/dex-77657bcfc5-nrx7d container=\"dex\" ready=true restarts=0; Pod/dex-77657bcfc5-nrx7d phase=\"Running\" reason=\"\"",
err: <*fmt.wrapError | 0x39e614e48260>{
msg: "configure Console ExternalAuth through RP: context deadline exceeded",
err: <context.deadlineExceededError>{},
},
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:197]: failed to set up Console login fixture for cluster "cluster-candidate-5-1-qx8d44"
Unexpected error:
<*fmt.wrapError | 0x39e6141ab040>:
console fixture namespace "console-n2bkhw2": configure Console ExternalAuth through RP: context deadline exceeded; Dex namespace="console-n2bkhw2"; Deployment/dex condition="Available" status="True" reason="MinimumReplicasAvailable"; Deployment/dex condition="Progressing" status="True" reason="NewReplicaSetAvailable"; Deployment/dex generation=1 observed=1 replicas=1 ready=1 available=1; Dex Pods=1; Dex ReplicaSets=1; Pod/dex-77657bcfc5-nrx7d condition="ContainersReady" status="True" reason=""; Pod/dex-77657bcfc5-nrx7d condition="Initialized" status="True" reason=""; Pod/dex-77657bcfc5-nrx7d condition="PodReadyToStartContainers" status="True" reason=""; Pod/dex-77657bcfc5-nrx7d condition="PodScheduled" status="True" reason=""; Pod/dex-77657bcfc5-nrx7d condition="Ready" status="True" reason=""; Pod/dex-77657bcfc5-nrx7d container="dex" ready=true restarts=0; Pod/dex-77657bcfc5-nrx7d phase="Running" reason=""
{
msg: "console fixture namespace \"console-n2bkhw2\": configure Console ExternalAuth through RP: context deadline exceeded; Dex namespace=\"console-n2bkhw2\"; Deployment/dex condition=\"Available\" status=\"True\" reason=\"MinimumReplicasAvailable\"; Deployment/dex condition=\"Progressing\" status=\"True\" reason=\"NewReplicaSetAvailable\"; Deployment/dex generation=1 observed=1 replicas=1 ready=1 available=1; Dex Pods=1; Dex ReplicaSets=1; Pod/dex-77657bcfc5-nrx7d condition=\"ContainersReady\" status=\"True\" reason=\"\"; Pod/dex-77657bcfc5-nrx7d condition=\"Initialized\" status=\"True\" reason=\"\"; Pod/dex-77657bcfc5-nrx7d condition=\"PodReadyToStartContainers\" status=\"True\" reason=\"\"; Pod/dex-77657bcfc5-nrx7d condition=\"PodScheduled\" status=\"True\" reason=\"\"; Pod/dex-77657bcfc5-nrx7d condition=\"Ready\" status=\"True\" reason=\"\"; Pod/dex-77657bcfc5-nrx7d container=\"dex\" ready=true restarts=0; Pod/dex-77657bcfc5-nrx7d phase=\"Running\" reason=\"\"",
err: <*fmt.wrapError | 0x39e614e48260>{
msg: "configure Console ExternalAuth through RP: context deadline exceeded",
err: <context.deadlineExceededError>{},
},
}
occurredContributing tests (1)
Affected runs (1)
| |||||||||||||||||||||||||||||||||||||||||||||||||||||
console fixture namespace "<namespace>": configure Console ExternalAuth through RP: context deadline exceeded; Dex namespace="console-r7gptrj"; Deployment/dex condition="Available...full failure pattern: console fixture namespace "<namespace>": configure Console ExternalAuth through RP: context deadline exceeded; Dex namespace="console-r7gptrj"; Deployment/dex condition="Available" status="True" reason="MinimumReplicasAvailable"; Deployment/dex Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s) | unknown | 1 | 2.33%1 of 43 job runs affected | IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s) | Show trend detailsOct 21: 0 · Oct 22: 0 · Oct 23: 1 · Oct 24: 0 · Oct 25: 0 · Oct 26: 0 · Oct 27: 0 · Oct 28: 0 | none | |||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:197]: failed to set up Console login fixture for cluster "cluster-candidate-5-0-glc2bs"
Unexpected error:
<*fmt.wrapError | 0x2d9cfabb2a80>:
console fixture namespace "console-r7gptrj": configure Console ExternalAuth through RP: context deadline exceeded; Dex namespace="console-r7gptrj"; Deployment/dex condition="Available" status="True" reason="MinimumReplicasAvailable"; Deployment/dex condition="Progressing" status="True" reason="NewReplicaSetAvailable"; Deployment/dex generation=1 observed=1 replicas=1 ready=1 available=1; Dex Pods=1; Dex ReplicaSets=1; Pod/dex-77657bcfc5-mwmw6 condition="ContainersReady" status="True" reason=""; Pod/dex-77657bcfc5-mwmw6 condition="Initialized" status="True" reason=""; Pod/dex-77657bcfc5-mwmw6 condition="PodReadyToStartContainers" status="True" reason=""; Pod/dex-77657bcfc5-mwmw6 condition="PodScheduled" status="True" reason=""; Pod/dex-77657bcfc5-mwmw6 condition="Ready" status="True" reason=""; Pod/dex-77657bcfc5-mwmw6 container="dex" ready=true restarts=0; Pod/dex-77657bcfc5-mwmw6 phase="Running" reason=""
{
msg: "console fixture namespace \"console-r7gptrj\": configure Console ExternalAuth through RP: context deadline exceeded; Dex namespace=\"console-r7gptrj\"; Deployment/dex condition=\"Available\" status=\"True\" reason=\"MinimumReplicasAvailable\"; Deployment/dex condition=\"Progressing\" status=\"True\" reason=\"NewReplicaSetAvailable\"; Deployment/dex generation=1 observed=1 replicas=1 ready=1 available=1; Dex Pods=1; Dex ReplicaSets=1; Pod/dex-77657bcfc5-mwmw6 condition=\"ContainersReady\" status=\"True\" reason=\"\"; Pod/dex-77657bcfc5-mwmw6 condition=\"Initialized\" status=\"True\" reason=\"\"; Pod/dex-77657bcfc5-mwmw6 condition=\"PodReadyToStartContainers\" status=\"True\" reason=\"\"; Pod/dex-77657bcfc5-mwmw6 condition=\"PodScheduled\" status=\"True\" reason=\"\"; Pod/dex-77657bcfc5-mwmw6 condition=\"Ready\" status=\"True\" reason=\"\"; Pod/dex-77657bcfc5-mwmw6 container=\"dex\" ready=true restarts=0; Pod/dex-77657bcfc5-mwmw6 phase=\"Running\" reason=\"\"",
err: <*fmt.wrapError | 0x2d9cfac24920>{
msg: "configure Console ExternalAuth through RP: context deadline exceeded",
err: <context.deadlineExceededError>{},
},
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:197]: failed to set up Console login fixture for cluster "cluster-candidate-5-0-glc2bs"
Unexpected error:
<*fmt.wrapError | 0x2d9cfabb2a80>:
console fixture namespace "console-r7gptrj": configure Console ExternalAuth through RP: context deadline exceeded; Dex namespace="console-r7gptrj"; Deployment/dex condition="Available" status="True" reason="MinimumReplicasAvailable"; Deployment/dex condition="Progressing" status="True" reason="NewReplicaSetAvailable"; Deployment/dex generation=1 observed=1 replicas=1 ready=1 available=1; Dex Pods=1; Dex ReplicaSets=1; Pod/dex-77657bcfc5-mwmw6 condition="ContainersReady" status="True" reason=""; Pod/dex-77657bcfc5-mwmw6 condition="Initialized" status="True" reason=""; Pod/dex-77657bcfc5-mwmw6 condition="PodReadyToStartContainers" status="True" reason=""; Pod/dex-77657bcfc5-mwmw6 condition="PodScheduled" status="True" reason=""; Pod/dex-77657bcfc5-mwmw6 condition="Ready" status="True" reason=""; Pod/dex-77657bcfc5-mwmw6 container="dex" ready=true restarts=0; Pod/dex-77657bcfc5-mwmw6 phase="Running" reason=""
{
msg: "console fixture namespace \"console-r7gptrj\": configure Console ExternalAuth through RP: context deadline exceeded; Dex namespace=\"console-r7gptrj\"; Deployment/dex condition=\"Available\" status=\"True\" reason=\"MinimumReplicasAvailable\"; Deployment/dex condition=\"Progressing\" status=\"True\" reason=\"NewReplicaSetAvailable\"; Deployment/dex generation=1 observed=1 replicas=1 ready=1 available=1; Dex Pods=1; Dex ReplicaSets=1; Pod/dex-77657bcfc5-mwmw6 condition=\"ContainersReady\" status=\"True\" reason=\"\"; Pod/dex-77657bcfc5-mwmw6 condition=\"Initialized\" status=\"True\" reason=\"\"; Pod/dex-77657bcfc5-mwmw6 condition=\"PodReadyToStartContainers\" status=\"True\" reason=\"\"; Pod/dex-77657bcfc5-mwmw6 condition=\"PodScheduled\" status=\"True\" reason=\"\"; Pod/dex-77657bcfc5-mwmw6 condition=\"Ready\" status=\"True\" reason=\"\"; Pod/dex-77657bcfc5-mwmw6 container=\"dex\" ready=true restarts=0; Pod/dex-77657bcfc5-mwmw6 phase=\"Running\" reason=\"\"",
err: <*fmt.wrapError | 0x2d9cfac24920>{
msg: "configure Console ExternalAuth through RP: context deadline exceeded",
err: <context.deadlineExceededError>{},
},
}
occurredContributing tests (1)
Affected runs (1)
| |||||||||||||||||||||||||||||||||||||||||||||||||||||
console fixture namespace "<namespace>": configure Console ExternalAuth through RP: context deadline exceeded; Dex namespace="console-xzp4sq4"; Deployment/dex condition="Available...full failure pattern: console fixture namespace "<namespace>": configure Console ExternalAuth through RP: context deadline exceeded; Dex namespace="console-xzp4sq4"; Deployment/dex condition="Available" status="True" reason="MinimumReplicasAvailable"; Deployment/dex Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s) | unknown | 1 | 2.33%1 of 43 job runs affected | IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s) | Show trend detailsOct 21: 0 · Oct 22: 1 · Oct 23: 0 · Oct 24: 0 · Oct 25: 0 · Oct 26: 0 · Oct 27: 0 · Oct 28: 0 | none | |||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:194]: failed to set up Console login fixture for cluster "cluster-candidate-4-20-8dwfln"
Unexpected error:
<*fmt.wrapError | 0x22821099b6a0>:
console fixture namespace "console-xzp4sq4": configure Console ExternalAuth through RP: context deadline exceeded; Dex namespace="console-xzp4sq4"; Deployment/dex condition="Available" status="True" reason="MinimumReplicasAvailable"; Deployment/dex condition="Progressing" status="True" reason="NewReplicaSetAvailable"; Deployment/dex generation=1 observed=1 replicas=1 ready=1 available=1; Dex Pods=1; Dex ReplicaSets=1; Pod/dex-78f4dcc67d-swzhr condition="ContainersReady" status="True" reason=""; Pod/dex-78f4dcc67d-swzhr condition="Initialized" status="True" reason=""; Pod/dex-78f4dcc67d-swzhr condition="PodReadyToStartContainers" status="True" reason=""; Pod/dex-78f4dcc67d-swzhr condition="PodScheduled" status="True" reason=""; Pod/dex-78f4dcc67d-swzhr condition="Ready" status="True" reason=""; Pod/dex-78f4dcc67d-swzhr container="dex" ready=true restarts=0; Pod/dex-78f4dcc67d-swzhr phase="Running" reason=""
{
msg: "console fixture namespace \"console-xzp4sq4\": configure Console ExternalAuth through RP: context deadline exceeded; Dex namespace=\"console-xzp4sq4\"; Deployment/dex condition=\"Available\" status=\"True\" reason=\"MinimumReplicasAvailable\"; Deployment/dex condition=\"Progressing\" status=\"True\" reason=\"NewReplicaSetAvailable\"; Deployment/dex generation=1 observed=1 replicas=1 ready=1 available=1; Dex Pods=1; Dex ReplicaSets=1; Pod/dex-78f4dcc67d-swzhr condition=\"ContainersReady\" status=\"True\" reason=\"\"; Pod/dex-78f4dcc67d-swzhr condition=\"Initialized\" status=\"True\" reason=\"\"; Pod/dex-78f4dcc67d-swzhr condition=\"PodReadyToStartContainers\" status=\"True\" reason=\"\"; Pod/dex-78f4dcc67d-swzhr condition=\"PodScheduled\" status=\"True\" reason=\"\"; Pod/dex-78f4dcc67d-swzhr condition=\"Ready\" status=\"True\" reason=\"\"; Pod/dex-78f4dcc67d-swzhr container=\"dex\" ready=true restarts=0; Pod/dex-78f4dcc67d-swzhr phase=\"Running\" reason=\"\"",
err: <*fmt.wrapError | 0x228210e20900>{
msg: "configure Console ExternalAuth through RP: context deadline exceeded",
err: <context.deadlineExceededError>{},
},
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:194]: failed to set up Console login fixture for cluster "cluster-candidate-4-20-8dwfln"
Unexpected error:
<*fmt.wrapError | 0x22821099b6a0>:
console fixture namespace "console-xzp4sq4": configure Console ExternalAuth through RP: context deadline exceeded; Dex namespace="console-xzp4sq4"; Deployment/dex condition="Available" status="True" reason="MinimumReplicasAvailable"; Deployment/dex condition="Progressing" status="True" reason="NewReplicaSetAvailable"; Deployment/dex generation=1 observed=1 replicas=1 ready=1 available=1; Dex Pods=1; Dex ReplicaSets=1; Pod/dex-78f4dcc67d-swzhr condition="ContainersReady" status="True" reason=""; Pod/dex-78f4dcc67d-swzhr condition="Initialized" status="True" reason=""; Pod/dex-78f4dcc67d-swzhr condition="PodReadyToStartContainers" status="True" reason=""; Pod/dex-78f4dcc67d-swzhr condition="PodScheduled" status="True" reason=""; Pod/dex-78f4dcc67d-swzhr condition="Ready" status="True" reason=""; Pod/dex-78f4dcc67d-swzhr container="dex" ready=true restarts=0; Pod/dex-78f4dcc67d-swzhr phase="Running" reason=""
{
msg: "console fixture namespace \"console-xzp4sq4\": configure Console ExternalAuth through RP: context deadline exceeded; Dex namespace=\"console-xzp4sq4\"; Deployment/dex condition=\"Available\" status=\"True\" reason=\"MinimumReplicasAvailable\"; Deployment/dex condition=\"Progressing\" status=\"True\" reason=\"NewReplicaSetAvailable\"; Deployment/dex generation=1 observed=1 replicas=1 ready=1 available=1; Dex Pods=1; Dex ReplicaSets=1; Pod/dex-78f4dcc67d-swzhr condition=\"ContainersReady\" status=\"True\" reason=\"\"; Pod/dex-78f4dcc67d-swzhr condition=\"Initialized\" status=\"True\" reason=\"\"; Pod/dex-78f4dcc67d-swzhr condition=\"PodReadyToStartContainers\" status=\"True\" reason=\"\"; Pod/dex-78f4dcc67d-swzhr condition=\"PodScheduled\" status=\"True\" reason=\"\"; Pod/dex-78f4dcc67d-swzhr condition=\"Ready\" status=\"True\" reason=\"\"; Pod/dex-78f4dcc67d-swzhr container=\"dex\" ready=true restarts=0; Pod/dex-78f4dcc67d-swzhr phase=\"Running\" reason=\"\"",
err: <*fmt.wrapError | 0x228210e20900>{
msg: "configure Console ExternalAuth through RP: context deadline exceeded",
err: <context.deadlineExceededError>{},
},
}
occurredContributing tests (1)
Affected runs (1)
| |||||||||||||||||||||||||||||||||||||||||||||||||||||
console fixture namespace "<namespace>": configure Console ExternalAuth through RP: context deadline exceeded; Dex namespace="console-z2wqbfw"; Deployment/dex condition="Available...full failure pattern: console fixture namespace "<namespace>": configure Console ExternalAuth through RP: context deadline exceeded; Dex namespace="console-z2wqbfw"; Deployment/dex condition="Available" status="True" reason="MinimumReplicasAvailable"; Deployment/dex Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s) | unknown | 1 | 2.33%1 of 43 job runs affected | IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s) | Show trend detailsOct 21: 0 · Oct 22: 1 · Oct 23: 0 · Oct 24: 0 · Oct 25: 0 · Oct 26: 0 · Oct 27: 0 · Oct 28: 0 | none | |||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:194]: failed to set up Console login fixture for cluster "cluster-candidate-4-20-9js6f6"
Unexpected error:
<*fmt.wrapError | 0x10354e044ea0>:
console fixture namespace "console-z2wqbfw": configure Console ExternalAuth through RP: context deadline exceeded; Dex namespace="console-z2wqbfw"; Deployment/dex condition="Available" status="True" reason="MinimumReplicasAvailable"; Deployment/dex condition="Progressing" status="True" reason="NewReplicaSetAvailable"; Deployment/dex generation=1 observed=1 replicas=1 ready=1 available=1; Dex Pods=1; Dex ReplicaSets=1; Pod/dex-78f4dcc67d-6mptx condition="ContainersReady" status="True" reason=""; Pod/dex-78f4dcc67d-6mptx condition="Initialized" status="True" reason=""; Pod/dex-78f4dcc67d-6mptx condition="PodReadyToStartContainers" status="True" reason=""; Pod/dex-78f4dcc67d-6mptx condition="PodScheduled" status="True" reason=""; Pod/dex-78f4dcc67d-6mptx condition="Ready" status="True" reason=""; Pod/dex-78f4dcc67d-6mptx container="dex" ready=true restarts=0; Pod/dex-78f4dcc67d-6mptx phase="Running" reason=""
{
msg: "console fixture namespace \"console-z2wqbfw\": configure Console ExternalAuth through RP: context deadline exceeded; Dex namespace=\"console-z2wqbfw\"; Deployment/dex condition=\"Available\" status=\"True\" reason=\"MinimumReplicasAvailable\"; Deployment/dex condition=\"Progressing\" status=\"True\" reason=\"NewReplicaSetAvailable\"; Deployment/dex generation=1 observed=1 replicas=1 ready=1 available=1; Dex Pods=1; Dex ReplicaSets=1; Pod/dex-78f4dcc67d-6mptx condition=\"ContainersReady\" status=\"True\" reason=\"\"; Pod/dex-78f4dcc67d-6mptx condition=\"Initialized\" status=\"True\" reason=\"\"; Pod/dex-78f4dcc67d-6mptx condition=\"PodReadyToStartContainers\" status=\"True\" reason=\"\"; Pod/dex-78f4dcc67d-6mptx condition=\"PodScheduled\" status=\"True\" reason=\"\"; Pod/dex-78f4dcc67d-6mptx condition=\"Ready\" status=\"True\" reason=\"\"; Pod/dex-78f4dcc67d-6mptx container=\"dex\" ready=true restarts=0; Pod/dex-78f4dcc67d-6mptx phase=\"Running\" reason=\"\"",
err: <*fmt.wrapError | 0x10354ed78880>{
msg: "configure Console ExternalAuth through RP: context deadline exceeded",
err: <context.deadlineExceededError>{},
},
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:194]: failed to set up Console login fixture for cluster "cluster-candidate-4-20-9js6f6"
Unexpected error:
<*fmt.wrapError | 0x10354e044ea0>:
console fixture namespace "console-z2wqbfw": configure Console ExternalAuth through RP: context deadline exceeded; Dex namespace="console-z2wqbfw"; Deployment/dex condition="Available" status="True" reason="MinimumReplicasAvailable"; Deployment/dex condition="Progressing" status="True" reason="NewReplicaSetAvailable"; Deployment/dex generation=1 observed=1 replicas=1 ready=1 available=1; Dex Pods=1; Dex ReplicaSets=1; Pod/dex-78f4dcc67d-6mptx condition="ContainersReady" status="True" reason=""; Pod/dex-78f4dcc67d-6mptx condition="Initialized" status="True" reason=""; Pod/dex-78f4dcc67d-6mptx condition="PodReadyToStartContainers" status="True" reason=""; Pod/dex-78f4dcc67d-6mptx condition="PodScheduled" status="True" reason=""; Pod/dex-78f4dcc67d-6mptx condition="Ready" status="True" reason=""; Pod/dex-78f4dcc67d-6mptx container="dex" ready=true restarts=0; Pod/dex-78f4dcc67d-6mptx phase="Running" reason=""
{
msg: "console fixture namespace \"console-z2wqbfw\": configure Console ExternalAuth through RP: context deadline exceeded; Dex namespace=\"console-z2wqbfw\"; Deployment/dex condition=\"Available\" status=\"True\" reason=\"MinimumReplicasAvailable\"; Deployment/dex condition=\"Progressing\" status=\"True\" reason=\"NewReplicaSetAvailable\"; Deployment/dex generation=1 observed=1 replicas=1 ready=1 available=1; Dex Pods=1; Dex ReplicaSets=1; Pod/dex-78f4dcc67d-6mptx condition=\"ContainersReady\" status=\"True\" reason=\"\"; Pod/dex-78f4dcc67d-6mptx condition=\"Initialized\" status=\"True\" reason=\"\"; Pod/dex-78f4dcc67d-6mptx condition=\"PodReadyToStartContainers\" status=\"True\" reason=\"\"; Pod/dex-78f4dcc67d-6mptx condition=\"PodScheduled\" status=\"True\" reason=\"\"; Pod/dex-78f4dcc67d-6mptx condition=\"Ready\" status=\"True\" reason=\"\"; Pod/dex-78f4dcc67d-6mptx container=\"dex\" ready=true restarts=0; Pod/dex-78f4dcc67d-6mptx phase=\"Running\" reason=\"\"",
err: <*fmt.wrapError | 0x10354ed78880>{
msg: "configure Console ExternalAuth through RP: context deadline exceeded",
err: <context.deadlineExceededError>{},
},
}
occurredContributing tests (1)
Affected runs (1)
| |||||||||||||||||||||||||||||||||||||||||||||||||||||
Environment: STG
Daily Run Outcomes
| Pass rate | Runs | Date (UTC) | Suite | Test |
|---|---|---|---|---|
| 90.91% | 11 | stage/parallel | Customer should be able to create an HCP cluster and manage pull secrets | |
| 90.91% | 11 | stage/parallel | Customer should be able to create an HCP cluster with custom autoscaling and update the autoscaling configuration |
Up to 50 failure patterns are loaded (minimum 1.00% impact), with 10 shown by default. Click column headers to re-sort.
Jump to the full Failure Patterns view for this window
| Failure patternThe canonical description of a recurring CI failure, extracted and normalized from raw logs. | Failed atThe stage of the job run where this failure occurred: 'provision' (environment setup, DEV only), 'e2e' (test suite execution), or 'other' (CI infrastructure issues that did not produce a failure pattern). | Number of distinct job runs where this failure pattern was detected. | Percentage of all job runs in this environment affected by this failure pattern during the selected window. | Classification of this failure pattern: Regression (likely caused by a specific PR), Flake (intermittent failure spread across days), Noise (low-quality or generic pattern), or Indeterminate. | TrendShows daily activity for this failure pattern in a trailing window anchored to the selected end date. The sparkline covers at least 7 days and at most 14 days, depending on the current window size. | Also inOther environments where the same failure pattern was also detected during the selected window. | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
ACM ingress secret openshift-ingress/cluster-ingress-cert exists; expected ACM certificate delivery to be suppressedfull failure pattern: ACM ingress secret openshift-ingress/cluster-ingress-cert exists; expected ACM certificate delivery to be suppressed Signal: FlakeSignal: Flake — present in 2 prior week(s); spread across 5 days | e2e | 6 | 26.09%6 of 23 job runs affected | FlakeSignal: Flake — present in 2 prior week(s); spread across 5 days | Show trend detailsOct 21: 0 · Oct 22: 5 · Oct 23: 9 · Oct 24: 5 · Oct 25: 0 · Oct 26: 2 · Oct 27: 3 · Oct 28: 0 | DEV, PROD | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (3)fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:528]: ACM must not create its ingress certificate secret
Expected success, but got an error:
<*errors.errorString | 0x66df03e8b20>:
ACM ingress secret openshift-ingress/cluster-ingress-cert exists; expected ACM certificate delivery to be suppressed
{
s: "ACM ingress secret openshift-ingress/cluster-ingress-cert exists; expected ACM certificate delivery to be suppressed",
}
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:528]: ACM must not create its ingress certificate secret
Expected success, but got an error:
<*errors.errorString | 0x66df03e8b20>:
ACM ingress secret openshift-ingress/cluster-ingress-cert exists; expected ACM certificate delivery to be suppressed
{
s: "ACM ingress secret openshift-ingress/cluster-ingress-cert exists; expected ACM certificate delivery to be suppressed",
}fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:528]: ACM must not create its ingress certificate secret
Expected success, but got an error:
<*errors.errorString | 0x2de6fa3020f0>:
ACM ingress secret openshift-ingress/cluster-ingress-cert exists; expected ACM certificate delivery to be suppressed
{
s: "ACM ingress secret openshift-ingress/cluster-ingress-cert exists; expected ACM certificate delivery to be suppressed",
}
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:528]: ACM must not create its ingress certificate secret
Expected success, but got an error:
<*errors.errorString | 0x2de6fa3020f0>:
ACM ingress secret openshift-ingress/cluster-ingress-cert exists; expected ACM certificate delivery to be suppressed
{
s: "ACM ingress secret openshift-ingress/cluster-ingress-cert exists; expected ACM certificate delivery to be suppressed",
}fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:260]: ACM must not create its ingress certificate secret for cluster "cluster-candidate-4-23-wlxxtl"
Expected success, but got an error:
<*errors.errorString | 0x30c9514302d0>:
ACM ingress secret openshift-ingress/cluster-ingress-cert exists; expected ACM certificate delivery to be suppressed
{
s: "ACM ingress secret openshift-ingress/cluster-ingress-cert exists; expected ACM certificate delivery to be suppressed",
}
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:260]: ACM must not create its ingress certificate secret for cluster "cluster-candidate-4-23-wlxxtl"
Expected success, but got an error:
<*errors.errorString | 0x30c9514302d0>:
ACM ingress secret openshift-ingress/cluster-ingress-cert exists; expected ACM certificate delivery to be suppressed
{
s: "ACM ingress secret openshift-ingress/cluster-ingress-cert exists; expected ACM certificate delivery to be suppressed",
}Contributing tests (11)
Affected runs (6)
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
VerifyDefaultIngressServingCertificateSecretWiring timed out after 10m0s: IngressController/default references &LocalObjectReference{Name:cluster-ingress-cert,}, expected secret "...full failure pattern: VerifyDefaultIngressServingCertificateSecretWiring timed out after 10m0s: IngressController/default references &LocalObjectReference{Name:cluster-ingress-cert,}, expected secret "default-ingress-cert"Signal: NoiseSignal: Noise — struct/object fragment | e2e | 2 | 8.70%2 of 23 job runs affected | NoiseSignal: Noise — struct/object fragment | Show trend detailsOct 21: 0 · Oct 22: 2 · Oct 23: 0 · Oct 24: 1 · Oct 25: 0 · Oct 26: 0 · Oct 27: 0 · Oct 28: 0 | DEV, PROD | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (3)fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:220]: default-ingress-cert must exist and be referenced by IngressController/default for cluster "cluster-candidate-4-20-h45rtr"
Expected success, but got an error:
<*fmt.wrapError | 0x31f100c0b6e0>:
VerifyDefaultIngressServingCertificateSecretWiring timed out after 10m0s: IngressController/default references &LocalObjectReference{Name:cluster-ingress-cert,}, expected secret "default-ingress-cert"
{
msg: "VerifyDefaultIngressServingCertificateSecretWiring timed out after 10m0s: IngressController/default references &LocalObjectReference{Name:cluster-ingress-cert,}, expected secret \"default-ingress-cert\"",
err: <*verifiers.pollObservationError | 0x31f100c0b6c0>{
observation: <*errors.errorString | 0x31f1027391b0>{
s: "IngressController/default references &LocalObjectReference{Name:cluster-ingress-cert,}, expected secret \"default-ingress-cert\"",
},
interruption: <*fmt.wrapError | 0x31f100c0b6a0>{
msg: "getting default ingress serving certificate secret openshift-ingress/default-ingress-cert: context deadline exceeded",
err: <context.deadlineExceededError>{},
},
},
}
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:220]: default-ingress-cert must exist and be referenced by IngressController/default for cluster "cluster-candidate-4-20-h45rtr"
Expected success, but got an error:
<*fmt.wrapError | 0x31f100c0b6e0>:
VerifyDefaultIngressServingCertificateSecretWiring timed out after 10m0s: IngressController/default references &LocalObjectReference{Name:cluster-ingress-cert,}, expected secret "default-ingress-cert"
{
msg: "VerifyDefaultIngressServingCertificateSecretWiring timed out after 10m0s: IngressController/default references &LocalObjectReference{Name:cluster-ingress-cert,}, expected secret \"default-ingress-cert\"",
err: <*verifiers.pollObservationError | 0x31f100c0b6c0>{
observation: <*errors.errorString | 0x31f1027391b0>{
s: "IngressController/default references &LocalObjectReference{Name:cluster-ingress-cert,}, expected secret \"default-ingress-cert\"",
},
interruption: <*fmt.wrapError | 0x31f100c0b6a0>{
msg: "getting default ingress serving certificate secret openshift-ingress/default-ingress-cert: context deadline exceeded",
err: <context.deadlineExceededError>{},
},
},
}fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:216]: default-ingress-cert must exist and be referenced by IngressController/default for cluster "cluster-nightly-4-21-w7wgsb"
Expected success, but got an error:
<*fmt.wrapError | 0x80bfcd600c0>:
VerifyDefaultIngressServingCertificateSecretWiring timed out after 10m0s: IngressController/default references &LocalObjectReference{Name:cluster-ingress-cert,}, expected secret "default-ingress-cert"
{
msg: "VerifyDefaultIngressServingCertificateSecretWiring timed out after 10m0s: IngressController/default references &LocalObjectReference{Name:cluster-ingress-cert,}, expected secret \"default-ingress-cert\"",
err: <*verifiers.pollObservationError | 0x80bfcd60080>{
observation: <*errors.errorString | 0x80bfe902e80>{
s: "IngressController/default references &LocalObjectReference{Name:cluster-ingress-cert,}, expected secret \"default-ingress-cert\"",
},
interruption: <*fmt.wrapError | 0x80bfcd60060>{
msg: "getting default ingress serving certificate secret openshift-ingress/default-ingress-cert: context deadline exceeded",
err: <context.deadlineExceededError>{},
},
},
}
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:216]: default-ingress-cert must exist and be referenced by IngressController/default for cluster "cluster-nightly-4-21-w7wgsb"
Expected success, but got an error:
<*fmt.wrapError | 0x80bfcd600c0>:
VerifyDefaultIngressServingCertificateSecretWiring timed out after 10m0s: IngressController/default references &LocalObjectReference{Name:cluster-ingress-cert,}, expected secret "default-ingress-cert"
{
msg: "VerifyDefaultIngressServingCertificateSecretWiring timed out after 10m0s: IngressController/default references &LocalObjectReference{Name:cluster-ingress-cert,}, expected secret \"default-ingress-cert\"",
err: <*verifiers.pollObservationError | 0x80bfcd60080>{
observation: <*errors.errorString | 0x80bfe902e80>{
s: "IngressController/default references &LocalObjectReference{Name:cluster-ingress-cert,}, expected secret \"default-ingress-cert\"",
},
interruption: <*fmt.wrapError | 0x80bfcd60060>{
msg: "getting default ingress serving certificate secret openshift-ingress/default-ingress-cert: context deadline exceeded",
err: <context.deadlineExceededError>{},
},
},
}fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:216]: default-ingress-cert must exist and be referenced by IngressController/default for cluster "cluster-nightly-4-22-45dcr7"
Expected success, but got an error:
<*fmt.wrapError | 0x3387fb166020>:
VerifyDefaultIngressServingCertificateSecretWiring timed out after 10m0s: IngressController/default references &LocalObjectReference{Name:cluster-ingress-cert,}, expected secret "default-ingress-cert"
{
msg: "VerifyDefaultIngressServingCertificateSecretWiring timed out after 10m0s: IngressController/default references &LocalObjectReference{Name:cluster-ingress-cert,}, expected secret \"default-ingress-cert\"",
err: <*errors.errorString | 0x3387fcff27f0>{
s: "IngressController/default references &LocalObjectReference{Name:cluster-ingress-cert,}, expected secret \"default-ingress-cert\"",
},
}
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:216]: default-ingress-cert must exist and be referenced by IngressController/default for cluster "cluster-nightly-4-22-45dcr7"
Expected success, but got an error:
<*fmt.wrapError | 0x3387fb166020>:
VerifyDefaultIngressServingCertificateSecretWiring timed out after 10m0s: IngressController/default references &LocalObjectReference{Name:cluster-ingress-cert,}, expected secret "default-ingress-cert"
{
msg: "VerifyDefaultIngressServingCertificateSecretWiring timed out after 10m0s: IngressController/default references &LocalObjectReference{Name:cluster-ingress-cert,}, expected secret \"default-ingress-cert\"",
err: <*errors.errorString | 0x3387fcff27f0>{
s: "IngressController/default references &LocalObjectReference{Name:cluster-ingress-cert,}, expected secret \"default-ingress-cert\"",
},
}Contributing tests (3)
Affected runs (2)
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
VerifyDefaultIngressServingCertificateSecretWiring timed out after 10m0full failure pattern: VerifyDefaultIngressServingCertificateSecretWiring timed out after 10m0 Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 5 day(s) | e2e | 6 | 26.09%6 of 23 job runs affected | IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 5 day(s) | Show trend detailsOct 21: 0 · Oct 22: 3 · Oct 23: 10 · Oct 24: 4 · Oct 25: 0 · Oct 26: 2 · Oct 27: 1 · Oct 28: 0 | DEV, PROD | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (3)fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:256]: default-ingress-cert must exist and be referenced by IngressController/default for cluster "cluster-candidate-5-0-2nz4xc"
Expected success, but got an error:
<*fmt.wrapError | 0x203abb848000>:
VerifyDefaultIngressServingCertificateSecretWiring timed out after 10m0.001s: IngressController/default references &LocalObjectReference{Name:cluster-ingress-cert,}, expected secret "default-ingress-cert"
{
msg: "VerifyDefaultIngressServingCertificateSecretWiring timed out after 10m0.001s: IngressController/default references &LocalObjectReference{Name:cluster-ingress-cert,}, expected secret \"default-ingress-cert\"",
err: <*errors.errorString | 0x203ab927a940>{
s: "IngressController/default references &LocalObjectReference{Name:cluster-ingress-cert,}, expected secret \"default-ingress-cert\"",
},
}
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:256]: default-ingress-cert must exist and be referenced by IngressController/default for cluster "cluster-candidate-5-0-2nz4xc"
Expected success, but got an error:
<*fmt.wrapError | 0x203abb848000>:
VerifyDefaultIngressServingCertificateSecretWiring timed out after 10m0.001s: IngressController/default references &LocalObjectReference{Name:cluster-ingress-cert,}, expected secret "default-ingress-cert"
{
msg: "VerifyDefaultIngressServingCertificateSecretWiring timed out after 10m0.001s: IngressController/default references &LocalObjectReference{Name:cluster-ingress-cert,}, expected secret \"default-ingress-cert\"",
err: <*errors.errorString | 0x203ab927a940>{
s: "IngressController/default references &LocalObjectReference{Name:cluster-ingress-cert,}, expected secret \"default-ingress-cert\"",
},
}fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:256]: default-ingress-cert must exist and be referenced by IngressController/default for cluster "cluster-nightly-5-0-q6mvvg"
Expected success, but got an error:
<*fmt.wrapError | 0x1ec67b8dad40>:
VerifyDefaultIngressServingCertificateSecretWiring timed out after 10m0.007s: IngressController/default references &LocalObjectReference{Name:cluster-ingress-cert,}, expected secret "default-ingress-cert"
{
msg: "VerifyDefaultIngressServingCertificateSecretWiring timed out after 10m0.007s: IngressController/default references &LocalObjectReference{Name:cluster-ingress-cert,}, expected secret \"default-ingress-cert\"",
err: <*errors.errorString | 0x1ec67cfb92a0>{
s: "IngressController/default references &LocalObjectReference{Name:cluster-ingress-cert,}, expected secret \"default-ingress-cert\"",
},
}
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:256]: default-ingress-cert must exist and be referenced by IngressController/default for cluster "cluster-nightly-5-0-q6mvvg"
Expected success, but got an error:
<*fmt.wrapError | 0x1ec67b8dad40>:
VerifyDefaultIngressServingCertificateSecretWiring timed out after 10m0.007s: IngressController/default references &LocalObjectReference{Name:cluster-ingress-cert,}, expected secret "default-ingress-cert"
{
msg: "VerifyDefaultIngressServingCertificateSecretWiring timed out after 10m0.007s: IngressController/default references &LocalObjectReference{Name:cluster-ingress-cert,}, expected secret \"default-ingress-cert\"",
err: <*errors.errorString | 0x1ec67cfb92a0>{
s: "IngressController/default references &LocalObjectReference{Name:cluster-ingress-cert,}, expected secret \"default-ingress-cert\"",
},
}fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:256]: default-ingress-cert must exist and be referenced by IngressController/default for cluster "cluster-nightly-4-23-9qvdk7"
Expected success, but got an error:
<*fmt.wrapError | 0x38ab336f65a0>:
VerifyDefaultIngressServingCertificateSecretWiring timed out after 10m0.004s: IngressController/default references &LocalObjectReference{Name:cluster-ingress-cert,}, expected secret "default-ingress-cert"
{
msg: "VerifyDefaultIngressServingCertificateSecretWiring timed out after 10m0.004s: IngressController/default references &LocalObjectReference{Name:cluster-ingress-cert,}, expected secret \"default-ingress-cert\"",
err: <*errors.errorString | 0x38ab34e123c0>{
s: "IngressController/default references &LocalObjectReference{Name:cluster-ingress-cert,}, expected secret \"default-ingress-cert\"",
},
}
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:256]: default-ingress-cert must exist and be referenced by IngressController/default for cluster "cluster-nightly-4-23-9qvdk7"
Expected success, but got an error:
<*fmt.wrapError | 0x38ab336f65a0>:
VerifyDefaultIngressServingCertificateSecretWiring timed out after 10m0.004s: IngressController/default references &LocalObjectReference{Name:cluster-ingress-cert,}, expected secret "default-ingress-cert"
{
msg: "VerifyDefaultIngressServingCertificateSecretWiring timed out after 10m0.004s: IngressController/default references &LocalObjectReference{Name:cluster-ingress-cert,}, expected secret \"default-ingress-cert\"",
err: <*errors.errorString | 0x38ab34e123c0>{
s: "IngressController/default references &LocalObjectReference{Name:cluster-ingress-cert,}, expected secret \"default-ingress-cert\"",
},
}Contributing tests (10)
Affected runs (6)
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
ERROR CODE: DeadlineExceeded; detail message node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hype...full failure pattern: ERROR CODE: DeadlineExceeded; detail message node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is <count>, want <count>; node pool condition AllNodesHealthy is False: <count> of <count> machines are not healthy Machine <node>: InspectionFailed: Waiting for AzureMachine to report spec.providerID; node pool condition AllMachinesReady is False: <count> of <count> machines are not ready Machine <node>: Creating: virtualmachine creating or updating; provider Microsoft.RedHatOpenShift Signal: IndeterminateSignal: Indeterminate — seen in 2 prior week(s); active 1 day(s) | unknown | 1 | 4.35%1 of 23 job runs affected | IndeterminateSignal: Indeterminate — seen in 2 prior week(s); active 1 day(s) | Show trend detailsOct 21: 0 · Oct 22: 0 · Oct 23: 0 · Oct 24: 0 · Oct 25: 0 · Oct 26: 0 · Oct 27: 10 · Oct 28: 0 | none | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (3)fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:181]: failed to create node pool "np-jxwc9m" for cluster "cluster-candidate-5-1-jxwc9m"
Unexpected error:
<*fmt.wrapError | 0x2089be556480>:
failed to create NodePool np-jxwc9m: failed waiting for nodepool="np-jxwc9m" for cluster "cluster-candidate-5-1-jxwc9m" in resourcegroup="rg-candidate-5-1-t5bnkh9ptx8p" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/aa13a852-544e-4ae1-9ee9-db477af0802a
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/aa13a852-544e-4ae1-9ee9-db477af0802a",
"name": "aa13a852-544e-4ae1-9ee9-db477af0802a",
"status": "Failed",
"startTime": "2026-10-27T08:36:20.928028726Z",
"endTime": "2026-10-27T08:55:28.505111283Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine f5e7a9i9o6b7x5n-np-jxwc9m-l25lf-nmkn7: InspectionFailed: Waiting for AzureMachine to report spec.providerID\nMachine f5e7a9i9o6b7x5n-np-jxwc9m-l25lf-rxvtq: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine f5e7a9i9o6b7x5n-np-jxwc9m-l25lf-nmkn7: Creating: virtualmachine creating or updating\nMachine f5e7a9i9o6b7x5n-np-jxwc9m-l25lf-rxvtq: Creating: virtualmachine creating or updating\n"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/aa13a852-544e-4ae1-9ee9-db477af0802a
correlation ID: 994d5b05-7775-48cd-93aa-f17d4c56d0ef
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:181]: failed to create node pool "np-jxwc9m" for cluster "cluster-candidate-5-1-jxwc9m"
Unexpected error:
<*fmt.wrapError | 0x2089be556480>:
failed to create NodePool np-jxwc9m: failed waiting for nodepool="np-jxwc9m" for cluster "cluster-candidate-5-1-jxwc9m" in resourcegroup="rg-candidate-5-1-t5bnkh9ptx8p" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/aa13a852-544e-4ae1-9ee9-db477af0802a
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/aa13a852-544e-4ae1-9ee9-db477af0802a",
"name": "aa13a852-544e-4ae1-9ee9-db477af0802a",
"status": "Failed",
"startTime": "2026-10-27T08:36:20.928028726Z",
"endTime": "2026-10-27T08:55:28.505111283Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine f5e7a9i9o6b7x5n-np-jxwc9m-l25lf-nmkn7: InspectionFailed: Waiting for AzureMachine to report spec.providerID\nMachine f5e7a9i9o6b7x5n-np-jxwc9m-l25lf-rxvtq: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine f5e7a9i9o6b7x5n-np-jxwc9m-l25lf-nmkn7: Creating: virtualmachine creating or updating\nMachine f5e7a9i9o6b7x5n-np-jxwc9m-l25lf-rxvtq: Creating: virtualmachine creating or updating\n"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/aa13a852-544e-4ae1-9ee9-db477af0802a
correlation ID: 994d5b05-7775-48cd-93aa-f17d4c56d0ef
client request ID: unavailable
occurredfail [github.com/Azure/ARO-HCP/test/e2e/cluster_nodepool_active_versions.go:96]: failed to create node pool "np-1" for v20261001 cluster "v20261001"
Unexpected error:
<*fmt.wrapError | 0x1520efb25aa0>:
failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "v20261001" in resourcegroup="v20261001-g5mddw2znmwk" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/a9c68b69-2f13-4771-88f2-779ee3c74191
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/a9c68b69-2f13-4771-88f2-779ee3c74191",
"name": "a9c68b69-2f13-4771-88f2-779ee3c74191",
"status": "Failed",
"startTime": "2026-10-27T08:30:17.391270547Z",
"endTime": "2026-10-27T08:49:18.18634688Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine v20261001-np-1-gzw7k-gkpzc: InspectionFailed: Waiting for AzureMachine to report spec.providerID\nMachine v20261001-np-1-gzw7k-sxqb6: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine v20261001-np-1-gzw7k-gkpzc: Creating: virtualmachine creating or updating\nMachine v20261001-np-1-gzw7k-sxqb6: Creating: virtualmachine creating or updating\n"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/a9c68b69-2f13-4771-88f2-779ee3c74191
correlation ID: c7660621-545a-4ad8-89c0-9dd90d63841c
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_nodepool_active_versions.go:96]: failed to create node pool "np-1" for v20261001 cluster "v20261001"
Unexpected error:
<*fmt.wrapError | 0x1520efb25aa0>:
failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "v20261001" in resourcegroup="v20261001-g5mddw2znmwk" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/a9c68b69-2f13-4771-88f2-779ee3c74191
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/a9c68b69-2f13-4771-88f2-779ee3c74191",
"name": "a9c68b69-2f13-4771-88f2-779ee3c74191",
"status": "Failed",
"startTime": "2026-10-27T08:30:17.391270547Z",
"endTime": "2026-10-27T08:49:18.18634688Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine v20261001-np-1-gzw7k-gkpzc: InspectionFailed: Waiting for AzureMachine to report spec.providerID\nMachine v20261001-np-1-gzw7k-sxqb6: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine v20261001-np-1-gzw7k-gkpzc: Creating: virtualmachine creating or updating\nMachine v20261001-np-1-gzw7k-sxqb6: Creating: virtualmachine creating or updating\n"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/a9c68b69-2f13-4771-88f2-779ee3c74191
correlation ID: c7660621-545a-4ad8-89c0-9dd90d63841c
client request ID: unavailable
occurredfail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:193]: failed to verify nodes are Ready with Cilium CNI for cluster "cilium-cluster"
Unexpected error:
<*errors.joinError | 0x3baee6990108>:
VerifyNodesReady failed: no nodes found in the cluster
VerifyCiliumOperational failed: not all pods in kube-system namespace are running: failed to list pods in kube-system namespace: context deadline exceeded
failed to create NodePool cilium-np: failed waiting for nodepool="cilium-np" for cluster "cilium-cluster" in resourcegroup="complex-cilium-kv-2gdwf4pbz22s" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/0e019fa7-e26d-44c3-b5c0-c1a3e7ecc4af
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/0e019fa7-e26d-44c3-b5c0-c1a3e7ecc4af",
"name": "0e019fa7-e26d-44c3-b5c0-c1a3e7ecc4af",
"status": "Failed",
"startTime": "2026-10-27T08:12:10.541346261Z",
"endTime": "2026-10-27T08:31:14.925265024Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine cilium-cluster-cilium-np-ffmd5-h94sn: InspectionFailed: Waiting for AzureMachine to report spec.providerID\nMachine cilium-cluster-cilium-np-ffmd5-hvcp5: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine cilium-cluster-cilium-np-ffmd5-h94sn: Creating: virtualmachine creating or updating\nMachine cilium-cluster-cilium-np-ffmd5-hvcp5: Creating: virtualmachine creating or updating\n"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/0e019fa7-e26d-44c3-b5c0-c1a3e7ecc4af
correlation ID: 6e36e413-6dec-4393-bcb3-4788f0d108b3
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:193]: failed to verify nodes are Ready with Cilium CNI for cluster "cilium-cluster"
Unexpected error:
<*errors.joinError | 0x3baee6990108>:
VerifyNodesReady failed: no nodes found in the cluster
VerifyCiliumOperational failed: not all pods in kube-system namespace are running: failed to list pods in kube-system namespace: context deadline exceeded
failed to create NodePool cilium-np: failed waiting for nodepool="cilium-np" for cluster "cilium-cluster" in resourcegroup="complex-cilium-kv-2gdwf4pbz22s" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/0e019fa7-e26d-44c3-b5c0-c1a3e7ecc4af
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/0e019fa7-e26d-44c3-b5c0-c1a3e7ecc4af",
"name": "0e019fa7-e26d-44c3-b5c0-c1a3e7ecc4af",
"status": "Failed",
"startTime": "2026-10-27T08:12:10.541346261Z",
"endTime": "2026-10-27T08:31:14.925265024Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine cilium-cluster-cilium-np-ffmd5-h94sn: InspectionFailed: Waiting for AzureMachine to report spec.providerID\nMachine cilium-cluster-cilium-np-ffmd5-hvcp5: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine cilium-cluster-cilium-np-ffmd5-h94sn: Creating: virtualmachine creating or updating\nMachine cilium-cluster-cilium-np-ffmd5-hvcp5: Creating: virtualmachine creating or updating\n"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/0e019fa7-e26d-44c3-b5c0-c1a3e7ecc4af
correlation ID: 6e36e413-6dec-4393-bcb3-4788f0d108b3
client request ID: unavailable
occurredContributing tests (10)
Affected runs (1)
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Unauthorizedfull failure pattern: VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Unauthorized Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s) | e2e | 1 | 4.35%1 of 23 job runs affected | IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s) | Show trend detailsOct 21: 0 · Oct 22: 0 · Oct 23: 0 · Oct 24: 0 · Oct 25: 0 · Oct 26: 0 · Oct 27: 1 · Oct 28: 0 | PROD | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:168]: failed to verify HCP cluster "cluster-nightly-4-21-htnxwk" is viable
Unexpected error:
<*errors.joinError | 0x28946bc08d20>:
VerifyBasicAccess failed: failed to list services: Unauthorized
VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Unauthorized
{
errs: [
<*fmt.wrapError | 0x28946b722320>{
msg: "VerifyBasicAccess failed: failed to list services: Unauthorized",
err: <*fmt.wrapError | 0x28946b722300>{
msg: "failed to list services: Unauthorized",
err: <*errors.StatusError | 0x28946b314d20>{
ErrStatus: {
TypeMeta: {Kind: "", APIVersion: ""},
ListMeta: {
SelfLink: "",
ResourceVersion: "",
Continue: "",
RemainingItemCount: nil,
ShardInfo: nil,
},
Status: "Failure",
Message: "Unauthorized",
Reason: "Unauthorized",
Details: nil,
Code: 401,
},
},
},
},
<*fmt.wrapError | 0x28946b60da40>{
msg: "VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Unauthorized",
err: <*fmt.wrapError | 0x28946b60da20>{
msg: "failed to list all APIServices: Unauthorized",
err: <*errors.StatusError | 0x28946b518dc0>{
ErrStatus: {
TypeMeta: {Kind: "", APIVersion: ""},
ListMeta: {
SelfLink: "",
ResourceVersion: "",
Continue: "",
RemainingItemCount: nil,
ShardInfo: nil,
},
Status: "Failure",
Message: "Unauthorized",
Reason: "Unauthorized",
Details: nil,
Code: 401,
},
},
},
},
],
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:168]: failed to verify HCP cluster "cluster-nightly-4-21-htnxwk" is viable
Unexpected error:
<*errors.joinError | 0x28946bc08d20>:
VerifyBasicAccess failed: failed to list services: Unauthorized
VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Unauthorized
{
errs: [
<*fmt.wrapError | 0x28946b722320>{
msg: "VerifyBasicAccess failed: failed to list services: Unauthorized",
err: <*fmt.wrapError | 0x28946b722300>{
msg: "failed to list services: Unauthorized",
err: <*errors.StatusError | 0x28946b314d20>{
ErrStatus: {
TypeMeta: {Kind: "", APIVersion: ""},
ListMeta: {
SelfLink: "",
ResourceVersion: "",
Continue: "",
RemainingItemCount: nil,
ShardInfo: nil,
},
Status: "Failure",
Message: "Unauthorized",
Reason: "Unauthorized",
Details: nil,
Code: 401,
},
},
},
},
<*fmt.wrapError | 0x28946b60da40>{
msg: "VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Unauthorized",
err: <*fmt.wrapError | 0x28946b60da20>{
msg: "failed to list all APIServices: Unauthorized",
err: <*errors.StatusError | 0x28946b518dc0>{
ErrStatus: {
TypeMeta: {Kind: "", APIVersion: ""},
ListMeta: {
SelfLink: "",
ResourceVersion: "",
Continue: "",
RemainingItemCount: nil,
ShardInfo: nil,
},
Status: "Failure",
Message: "Unauthorized",
Reason: "Unauthorized",
Details: nil,
Code: 401,
},
},
},
},
],
}
occurredContributing tests (1)
Affected runs (1)
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
certificate not ready: reason=DoesNotExist message=Issuing certificate as Secret does not existfull failure pattern: certificate not ready: reason=DoesNotExist message=Issuing certificate as Secret does not exist Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 2 day(s) | e2e | 1 | 4.35%1 of 23 job runs affected | IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 2 day(s) | Show trend detailsOct 21: 0 · Oct 22: 0 · Oct 23: 1 · Oct 24: 0 · Oct 25: 0 · Oct 26: 0 · Oct 27: 0 · Oct 28: 1 | DEV | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:493]: Timed out after 600.001s.
Certificate should be issued by Let's Encrypt staging via ACME DNS-01
Expected success, but got an error:
<*errors.errorString | 0x1a2d0a2b0a40>:
certificate not ready: reason=DoesNotExist message=Issuing certificate as Secret does not exist
{
s: "certificate not ready: reason=DoesNotExist message=Issuing certificate as Secret does not exist",
}
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:493]: Timed out after 600.001s.
Certificate should be issued by Let's Encrypt staging via ACME DNS-01
Expected success, but got an error:
<*errors.errorString | 0x1a2d0a2b0a40>:
certificate not ready: reason=DoesNotExist message=Issuing certificate as Secret does not exist
{
s: "certificate not ready: reason=DoesNotExist message=Issuing certificate as Secret does not exist",
}Contributing tests (1)
Affected runs (1)
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
certificate not ready: reason=DoesNotExist message=Issuing certificate as Secret does not existfull failure pattern: certificate not ready: reason=DoesNotExist message=Issuing certificate as Secret does not exist Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 2 day(s) | unknown | 1 | 4.35%1 of 23 job runs affected | IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 2 day(s) | Show trend detailsOct 21: 0 · Oct 22: 0 · Oct 23: 1 · Oct 24: 0 · Oct 25: 0 · Oct 26: 0 · Oct 27: 0 · Oct 28: 1 | DEV | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:510]: Timed out after 600.004s.
Certificate should be issued by Let's Encrypt staging via ACME DNS-01
Expected success, but got an error:
<*errors.errorString | 0x3b09d8d2ba10>:
certificate not ready: reason=DoesNotExist message=Issuing certificate as Secret does not exist
{
s: "certificate not ready: reason=DoesNotExist message=Issuing certificate as Secret does not exist",
}
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:510]: Timed out after 600.004s.
Certificate should be issued by Let's Encrypt staging via ACME DNS-01
Expected success, but got an error:
<*errors.errorString | 0x3b09d8d2ba10>:
certificate not ready: reason=DoesNotExist message=Issuing certificate as Secret does not exist
{
s: "certificate not ready: reason=DoesNotExist message=Issuing certificate as Secret does not exist",
}Contributing tests (1)
Affected runs (1)
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedC...full failure pattern: ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider]; provider Microsoft.RedHatOpenShift Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s) | e2e | 1 | 4.35%1 of 23 job runs affected | IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s) | Show trend detailsOct 21: 0 · Oct 22: 1 · Oct 23: 0 · Oct 24: 0 · Oct 25: 0 · Oct 26: 0 · Oct 27: 0 · Oct 28: 0 | DEV, INT, PROD | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/e2e/cluster_autoscaling.go:108]: failed to create HCP cluster "autoscaling-hcp-cluster" with custom autoscaling
Unexpected error:
<*fmt.wrapError | 0x3cf97cd86ae0>:
failed to create HCP cluster autoscaling-hcp-cluster: failed waiting for cluster="autoscaling-hcp-cluster" in resourcegroup="autoscaling-cluster-f4qstxjd2dn6" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/1ac5a43d-188f-41a4-873c-672951930233
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/1ac5a43d-188f-41a4-873c-672951930233",
"name": "1ac5a43d-188f-41a4-873c-672951930233",
"status": "Failed",
"startTime": "2026-10-22T02:02:55.710362817Z",
"endTime": "2026-10-22T02:32:02.510983362Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/1ac5a43d-188f-41a4-873c-672951930233
correlation ID: 9e15eb20-b5a4-4836-b8fc-ad2c0fd7a951
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_autoscaling.go:108]: failed to create HCP cluster "autoscaling-hcp-cluster" with custom autoscaling
Unexpected error:
<*fmt.wrapError | 0x3cf97cd86ae0>:
failed to create HCP cluster autoscaling-hcp-cluster: failed waiting for cluster="autoscaling-hcp-cluster" in resourcegroup="autoscaling-cluster-f4qstxjd2dn6" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/1ac5a43d-188f-41a4-873c-672951930233
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/1ac5a43d-188f-41a4-873c-672951930233",
"name": "1ac5a43d-188f-41a4-873c-672951930233",
"status": "Failed",
"startTime": "2026-10-22T02:02:55.710362817Z",
"endTime": "2026-10-22T02:32:02.510983362Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/1ac5a43d-188f-41a4-873c-672951930233
correlation ID: 9e15eb20-b5a4-4836-b8fc-ad2c0fd7a951
client request ID: unavailable
occurredContributing tests (1)
Affected runs (1)
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
ERROR CODE: DeadlineExceeded; detail message node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hype...full failure pattern: ERROR CODE: DeadlineExceeded; detail message node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is <count>, want >= <count> (autoscaling min); node pool condition AllNodesHealthy is False: <count> of <count> machines are not healthy Machine <node>: InspectionFailed: Waiting for AzureMachine to report spec.providerID; node pool condition AllMachinesReady is False: <count> of <count> machines are not ready Machine <node>: Creating: virtualmachine creating or updating; provider Microsoft.RedHatOpenShift Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s) | unknown | 1 | 4.35%1 of 23 job runs affected | IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s) | Show trend detailsOct 21: 0 · Oct 22: 0 · Oct 23: 0 · Oct 24: 0 · Oct 25: 0 · Oct 26: 0 · Oct 27: 1 · Oct 28: 0 | DEV | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:404]: failed to verify cilium is running and nodes are Ready for cluster "agg-cluster"
Unexpected error:
<*errors.joinError | 0xa0fb7d04138>:
VerifyNodesReady failed: no nodes found in the cluster
VerifyCiliumOperational failed: not all pods in kube-system namespace are running: failed to list pods in kube-system namespace: context deadline exceeded
failed to create NodePool agg-np-a: failed waiting for nodepool="agg-np-a" for cluster "agg-cluster" in resourcegroup="feature-aggregation-xf9jw4l5cdpf" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/0006ad13-87f3-4fc4-8b43-20306abddf58
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/0006ad13-87f3-4fc4-8b43-20306abddf58",
"name": "0006ad13-87f3-4fc4-8b43-20306abddf58",
"status": "Failed",
"startTime": "2026-10-27T08:14:54.971524126Z",
"endTime": "2026-10-27T08:33:58.162558969Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want \u003e= 2 (autoscaling min); node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine agg-cluster-agg-np-a-7cl7p-8fh5m: InspectionFailed: Waiting for AzureMachine to report spec.providerID\nMachine agg-cluster-agg-np-a-7cl7p-fd22m: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine agg-cluster-agg-np-a-7cl7p-8fh5m: Creating: virtualmachine creating or updating\nMachine agg-cluster-agg-np-a-7cl7p-fd22m: Creating: virtualmachine creating or updating\n"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/0006ad13-87f3-4fc4-8b43-20306abddf58
correlation ID: 7bfa7cc1-239c-4ae3-a9bc-578e94ecc5a3
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:404]: failed to verify cilium is running and nodes are Ready for cluster "agg-cluster"
Unexpected error:
<*errors.joinError | 0xa0fb7d04138>:
VerifyNodesReady failed: no nodes found in the cluster
VerifyCiliumOperational failed: not all pods in kube-system namespace are running: failed to list pods in kube-system namespace: context deadline exceeded
failed to create NodePool agg-np-a: failed waiting for nodepool="agg-np-a" for cluster "agg-cluster" in resourcegroup="feature-aggregation-xf9jw4l5cdpf" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/0006ad13-87f3-4fc4-8b43-20306abddf58
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/0006ad13-87f3-4fc4-8b43-20306abddf58",
"name": "0006ad13-87f3-4fc4-8b43-20306abddf58",
"status": "Failed",
"startTime": "2026-10-27T08:14:54.971524126Z",
"endTime": "2026-10-27T08:33:58.162558969Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want \u003e= 2 (autoscaling min); node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine agg-cluster-agg-np-a-7cl7p-8fh5m: InspectionFailed: Waiting for AzureMachine to report spec.providerID\nMachine agg-cluster-agg-np-a-7cl7p-fd22m: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine agg-cluster-agg-np-a-7cl7p-8fh5m: Creating: virtualmachine creating or updating\nMachine agg-cluster-agg-np-a-7cl7p-fd22m: Creating: virtualmachine creating or updating\n"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/0006ad13-87f3-4fc4-8b43-20306abddf58
correlation ID: 7bfa7cc1-239c-4ae3-a9bc-578e94ecc5a3
client request ID: unavailable
occurredContributing tests (1)
Affected runs (1)
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
ERROR CODE: InternalServerError; detail message cluster deletion did not complete before the deadline; [resourceTeardown] tearing down pod network instances, pod networks, namespa...full failure pattern: ERROR CODE: InternalServerError; detail message cluster deletion did not complete before the deadline; [resourceTeardown] tearing down pod network instances, pod networks, namespaces; [descendantResources] remaining resources: applydesires, serviceProviderClusters; provider Microsoft.RedHatOpenShift Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s) | e2e | 1 | 4.35%1 of 23 job runs affected | IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s) | Show trend detailsOct 21: 0 · Oct 22: 0 · Oct 23: 0 · Oct 24: 1 · Oct 25: 0 · Oct 26: 0 · Oct 27: 0 · Oct 28: 0 | DEV, PROD | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:310]: Unexpected error:
<*errors.joinError | 0x1ea319f5c1c8>:
resource group "rg-nightly-5-0-zf68j445hrrz": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cluster-nightly-5-0-md89qh" in resourcegroup="rg-nightly-5-0-zf68j445hrrz" to finish deleting: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/1895187e-5536-47a1-a44d-a9b6da901fa1
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: InternalServerError
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/1895187e-5536-47a1-a44d-a9b6da901fa1",
"name": "1895187e-5536-47a1-a44d-a9b6da901fa1",
"status": "Failed",
"startTime": "2026-10-25T00:01:49.583919819Z",
"endTime": "2026-10-25T00:25:49.807200592Z",
"error": {
"code": "InternalServerError",
"message": "cluster deletion did not complete before the deadline; [descendantResources] remaining resources: 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/serviceProviderClusters, 3 microsoft.redhatopenshift/hcpopenshiftclusters/applydesires; [resourceTeardown] tearing down pod network instances (1), pod networks (1), namespaces (2)"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/1895187e-5536-47a1-a44d-a9b6da901fa1
correlation ID: 78c885d0-28f3-4f61-bdbb-0c2134f660b2
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:310]: Unexpected error:
<*errors.joinError | 0x1ea319f5c1c8>:
resource group "rg-nightly-5-0-zf68j445hrrz": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cluster-nightly-5-0-md89qh" in resourcegroup="rg-nightly-5-0-zf68j445hrrz" to finish deleting: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/1895187e-5536-47a1-a44d-a9b6da901fa1
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: InternalServerError
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/1895187e-5536-47a1-a44d-a9b6da901fa1",
"name": "1895187e-5536-47a1-a44d-a9b6da901fa1",
"status": "Failed",
"startTime": "2026-10-25T00:01:49.583919819Z",
"endTime": "2026-10-25T00:25:49.807200592Z",
"error": {
"code": "InternalServerError",
"message": "cluster deletion did not complete before the deadline; [descendantResources] remaining resources: 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/serviceProviderClusters, 3 microsoft.redhatopenshift/hcpopenshiftclusters/applydesires; [resourceTeardown] tearing down pod network instances (1), pod networks (1), namespaces (2)"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/1895187e-5536-47a1-a44d-a9b6da901fa1
correlation ID: 78c885d0-28f3-4f61-bdbb-0c2134f660b2
client request ID: unavailable
occurredContributing tests (1)
Affected runs (1)
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
VerifyCustomIngressCertificate timed out after 10m0full failure pattern: VerifyCustomIngressCertificate timed out after 10m0 Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s) | e2e | 1 | 4.35%1 of 23 job runs affected | IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s) | Show trend detailsOct 21: 0 · Oct 22: 0 · Oct 23: 0 · Oct 24: 0 · Oct 25: 1 · Oct 26: 0 · Oct 27: 0 · Oct 28: 0 | none | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:525]: sample application must serve the exact customer-specified certificate with a valid Let's Encrypt chain
Expected success, but got an error:
<*fmt.wrapError | 0x36a1186625c0>:
VerifyCustomIngressCertificate timed out after 10m0.003s: ingress route must serve HTTPS with the expected certificate: Get "https://agnhost-e2e-sample-app-l49xr.apps.aro.ingress-cert-le.0o3m.uksouth.aroapp-hcp-stg.azure.com": tls: failed to verify certificate: x509: certificate signed by unknown authority
{
msg: "VerifyCustomIngressCertificate timed out after 10m0.003s: ingress route must serve HTTPS with the expected certificate: Get \"https://agnhost-e2e-sample-app-l49xr.apps.aro.ingress-cert-le.0o3m.uksouth.aroapp-hcp-stg.azure.com\": tls: failed to verify certificate: x509: certificate signed by unknown authority",
err: <*fmt.wrapError | 0x36a1186623e0>{
msg: "ingress route must serve HTTPS with the expected certificate: Get \"https://agnhost-e2e-sample-app-l49xr.apps.aro.ingress-cert-le.0o3m.uksouth.aroapp-hcp-stg.azure.com\": tls: failed to verify certificate: x509: certificate signed by unknown authority",
err: <*url.Error | 0x36a118d74840>{
Op: "Get",
URL: "https://agnhost-e2e-sample-app-l49xr.apps.aro.ingress-cert-le.0o3m.uksouth.aroapp-hcp-stg.azure.com",
Err: <*tls.CertificateVerificationError | 0x36a118d74810>{
UnverifiedCertificates: [
{
Raw: [48, 130, 9, 19, 48, 130, 6, 251, 160, 3, 2, 1, 2, 2, 19, 73, 1, 74, 89, 119, 169, 186, 32, 59, 56, 235, 138, 87, 0, 0, 1, 74, 89, 119, 48, 13, 6, 9, 42, 134, 72, 134, 247, 13, 1, 1, 12, 5, 0, 48, 87, 49, 11, 48, 9, 6, 3, 85, 4, 6, 19, 2, 85, 83, 49, 30, 48, 28, 6, 3, 85, 4, 10, 19, 21, 77, 105, 99, 114, 111, 115, 111, 102, 116, 32, 67, 111, 114, 112, 111, 114, 97, 116, 105, 111, 110, 49, 40, 48, 38, 6, 3, 85, 4, 3, 19, 31, 77, 105, 99, 114, 111, 115, 111, 102, 116, 32, 84, 76, 83, 32, 71, 50, 32, 82, 83, 65, 32, 67, 65, 32, 79, 67, 83, 80, 32, 49, 48, 48, 30, 23, 13, 50, 54, 49, 48, 50, 53, 48, 49, 53, 54, 49, 48, 90, 23, 13, 50, 55, 48, 50, 48, 50, 48, 48, 53, 54, 49, 48, 90, 48, 120, 49, 11, 48, 9, 6, 3, 85, 4, 6, 19, 2, 85, 83, 49, 11, 48, 9, 6, 3, 85, 4, 8, 19, 2, 87, 65, 49, 16, 48, 14, 6, 3, 85, 4, 7, 19, 7, 82, 101, 100, 109, 111, 110, 100, 49, 30, 48, 28, 6, 3, 85, 4, 10, 19, 21, 77, 105, 99, 114, 111, 115, 111, 102, 116, 32, 67, 111, 114, 112, 111, 114, 97, 116, 105, 111, 110, 49, 42, 48, 40, 6, 3, 85, 4, 3, 19, 33, 114, 101, 115, 101, 114, 118, 101, 100, 46, 97, 114, 111, 97, 112, 112, 45, 104, 99, 112, 45, 115, 116, 103, 46, 97, 122, 117, 114, 101, 46, 99, 111, 109, 48, 130, 1, 34, 48, 13, 6, 9, 42, 134, 72, 134, 247, 13, 1, 1, 1, 5, 0, 3, 130, 1, 15, 0, 48, 130, 1, 10, 2, 130, 1, 1, 0, 187, 121, 183, 117, 223, 172, 129, 21, 158, 33, 40, 228, 200, 112, 100, 153, 170, 116, 250, 158, 245, 201, 135, 0, 237, 144, 31, 12, 163, 250, 168, 50, 92, 5, 220, 113, 137, 143, 3, 193, 167, 220, 193, 78, 195, 131, 29, 107, 67, 52, 86, 236, 30, 39, 30, 107, 206, 213, 234, 58, 181, 33, 44, 227, 147, 157, 23, 219, 20, 127, 20, 235, 246, 77, 135, 5, 240, 135, 231, 58, 154, 101, 19, 42, 160, 128, 73, 94, 79, 155, 62, 172, 213, 53, 57, 2, 149, 60, 188, 169, 233, 44, 216, 105, 66, 159, 148, 70, 178, 113, 177, 128, 123, 151, 10, 7, 254, 171, 205, 105, 34, 145, 93, 19, 218, 220, 219, 47, 129, 42, 85, 247, 229, 7, 118, 48, 182, 125, 84, 45, 187, 16, 138, 130, 74, 176, 14, 70, 229, 120, 108, 212, 172, 177, 174, 141, 161, 231, 167, 5, 108, 191, 229, 39, 27, 122, 242, 198, 32, 78, 125, 8, 168, 234, 200, 195, 10, 209, 101, 183, 248, 26, 10, 82, 78, 47, 196, 44, 55, 199, 207, 209, 35, 65, 164, 44, 162, 97, 81, 36, 122, 188, 105, 82, 214, 39, 252, 78, 115, 249, 159, 36, 212, 167, 6, 72, 63, 56, 220, 88, 50, 162, 50, 174, 185, 196, 246, 208, 84, 90, 212, 218, 52, 118, 184, 101, 201, 132, 48, 29, 45, 113, 80, 81, 157, 255, 235, 218, 49, 139, 40, 124, 111, 147, 143, 201, 2, 3, 1, 0, 1, 163, 130, 4, 181, 48, 130, 4, 177, 48, 130, 1, 4, 6, 10, 43, 6, 1, 4, 1, 214, 121, 2, 4, 2, 4, 129, 245, 4, 129, 242, 0, 240, 0, 118, 0, 68, 194, 189, 12, 233, 20, 14, 100, 165, 201, 74, 1, 147, 10, 90, 161, 187, 53, 151, 14, 0, 238, 17, 22, 137, 104, 42, 28, 68, 215, 181, 102, 0, 0, 1, 161, 112, 206, 239, 163, 0, 0, 4, 3, 0, 71, 48, 69, 2, 33, 0, 172, 23, 89, 180, 174, 255, 10, 75, 120, 126, 2, 114, 141, 52, 53, 9, 237, 84, 209, 98, 89, 54, 179, 9, 54, 118, 245, 178, 107, 207, 5, 69, 2, 32, 89, 15, 78, 239, 171, 216, 123, 169, 212, 117, 5, 12, 137, 15, 63, 83, 41, 96, 223, 14, 136, 244, 40, 31, 142, 25, 69, 157, 200, 47, 205, 167, 0, 118, 0, 0, 26, 93, 26, 28, 45, 147, 117, 182, 72, 85, 120, 248, 47, 113, 161, 174, 110, 239, 57, 125, 41, 124, 138, 227, 21, 123, 202, 222, 225, 160, 30, 0, 0, 1, 161, 112, 206, 240, 23, 0, 0, 4, 3, 0, 71, 48, 69, 2, 33, 0, 209, 5, 94, 127, 36, 105, 13, 219, 166, 20, 49, 161, 208, 24, 148, 233, 208, 121, 79, 240, 195, 85, 152, 81, 108, 234, 202, 116, 68, 189, 194, 57, 2, 32, 65, 141, 139, 86, 66, 203, 70, 108, 26, 159, 237, 96, 32, 237, 241, 53, 230, 112, 171, 119, 90, 74, 10, 14, 99, 1, 225, 78, 245, 158, 165, 135, 48, 27, 6, 9, 43, 6, 1, 4, 1, 130, 55, 21, 10, 4, 14, 48, 12, 48, 10, 6, 8, 43, 6, 1, 5, 5, 7, 3, 1, 48, 60, 6, 9, 43, 6, 1, 4, 1, 130, 55, 21, 7, 4, 47, 48, 45, 6, 37, 43, 6, 1, 4, 1, 130, 55, 21, 8, 135, 189, 215, 27, 129, 231, 235, 70, 130, 129, 157, 46, 142, 208, 12, 135, 240, 218, 29, 93, 133, 246, 239, 42, 129, 232, 226, 53, 2, 1, 100, 2, 1, 39, 48, 130, 1, 11, 6, 8, 43, 6, 1, 5, 5, 7, 1, 1, 4, 129, 254, 48, 129, 251, 48, 97, 6, 8, 43, 6, 1, 5, 5, 7, 48, 2, 134, 85, 104, 116, 116, 112, 58, 47, 47, 119, 119, 119, 46, 109, 105, 99, 114, 111, 115, 111, 102, 116, 46, 99, 111, 109, 47, 112, 107, 105, 111, 112, 115, 47, 99, 101, 114, 116, 115, 47, 77, 105, 99, 114, 111, 115, 111, 102, 116, 37, 50, 48, 84, 76, 83, 37, 50, 48, 71, 50, 37, 50, 48, 82, 83, 65, 37, 50, 48, 67, 65, 37, 50, 48, 79, 67, 83, 80, 37, 50, 48, 49, 48, 46, 99, 114, 116, 48, 103, 6, 8, 43, 6, 1, 5, 5, 7, 48, 2, 134, 91, 104, 116, 116, 112, 58, 47, 47, 99, 97, 105, 115, 115, 117, 101, 114, 115, 46, 109, 105, 99, 114, 111, 115, 111, 102, 116, 46, 99, 111, 109, 47, 112, 107, 105, 111, 112, 115, 47, 99, 101, 114, 116, 115, 47, 77, 105, 99, 114, 111, 115, 111, 102, 116, 37, 50, 48, 84, 76, 83, 37, 50, 48, 71, 50, 37, 50, 48, 82, 83, 65, 37, 50, 48, 67, 65, 37, 50, 48, 79, 67, 83, 80, 37, 50, 48, 49, 48, 46, 99, 114, 116, 48, 45, 6, 8, 43, 6, 1, 5, 5, 7, 48, 1, 134, 33, 104, 116, 116, 112, 58, 47, 47, 111, 110, 101, 111, 99, 115, 112, 46, 109, 105, 99, 114, 111, 115, 111, 102, 116, 46, 99, 111, 109, 47, 111, 99, 115, 112, 48, 29, 6, 3, 85, 29, 14, 4, 22, 4, 20, 138, 66, 241, 22, 211, 0, 191, 156, 21, 196, 84, 38, 230, 58, 211, 190, 115, 187, 163, 122, 48, 14, 6, 3, 85, 29, 15, 1, 1, 255, 4, 4, 3, 2, 5, 160, 48, 110, 6, 3, 85, 29, 17, 4, 103, 48, 101, 130, 64, 42, 46, 97, 112, 112, 115, 46, 97, 114, 111, 46, 105, 110, 103, 114, 101, 115, 115, 45, 99, 101, 114, 116, 45, 108, 101, 46, 48, 111, 51, 109, 46, 117, 107, 115, 111, 117, 116, 104, 46, 97, 114, 111, 97, 112, 112, 45, 104, 99, 112, 45, 115, 116, 103, 46, 97, 122, 117, 114, 101, 46, 99, 111, 109, 130, 33, 114, 101, 115, 101, 114, 118, 101, 100, 46, 97, 114, 111, 97, 112, 112, 45, 104, 99, 112, 45, 115, 116, 103, 46, 97, 122, 117, 114, 101, 46, 99, 111, 109, 48, 12, 6, 3, 85, 29, 19, 1, 1, 255, 4, 2, 48, 0, 48, 129, 241, 6, 3, 85, 29, 31, 4, 129, 233, 48, 129, 230, 48, 129, 227, 160, 129, 224, 160, 129, 221, 134, 108, 104, 116, 116, 112, 58, 47, 47, 119, 119, 119, 46, 109, 105, 99, 114, 111, 115, 111, 102, 116, 46, 99, 111, 109, 47, 112, 107, 105, 111, 112, 115, 47, 99, 114, 108, 47, 112, 97, 114, 116, 105, 116, 105, 111, 110, 47, 77, 105, 99, 114, 111, 115, 111, 102, 116, 37, 50, 48, 84, 76, 83, 37, 50, 48, 71, 50, 37, 50, 48, 82, 83, 65, 37, 50, 48, 67, 65, 37, 50, 48, 79, 67, 83, 80, 37, 50, 48, 49, 48, 95, 80, 97, 114, 116, 105, 116, 105, 111, 110, 48, 48, 48, 57, 57, 46, 99, 114, 108, 134, 109, 104, 116, 116, 112, 58, 47, 47, 99, 114, 108, 50, 46, 109, 105, 99, 114, 111, 115, 111, 102, 116, 46, 99, 111, 109, 47, 112, 107, 105, 111, 112, 115, 47, 99, 114, 108, 47, 112, 97, 114, 116, 105, 116, 105, 111, 110, 47, 77, 105, 99, 114, 111, 115, 111, 102, 116, 37, 50, 48, 84, 76, 83, 37, 50, 48, 71, 50, 37, 50, 48, 82, 83, 65, 37, 50, 48, 67, 65, 37, 50, 48, 79, 67, 83, 80, 37, 50, 48, 49, 48, 95, 80, 97, 114, 116, 105, 116, 105, 111, 110, 48, 48, 48, 57, 57, 46, 99, 114, 108, 48, 102, 6, 3, 85, 29, 32, 4, 95, 48, 93, 48, 8, 6, 6, 103, 129, 12, 1, 2, 2, 48, 81, 6, 12, 43, 6, 1, 4, 1, 130, 55, 76, 131, 125, 1, 1, 48, 65, 48, 63, 6, 8, 43, 6, 1, 5, 5, 7, 2, 1, 22, 51, 104, 116, 116, 112, 58, 47, 47, 119, 119, 119, 46, 109, 105, 99, 114, 111, 115, 111, 102, 116, 46, 99, 111, 109, 47, 112, 107, 105, 111, 112, 115, 47, 68, 111, 99, 115, 47, 82, 101, 112, 111, 115, 105, 116, 111, 114, 121, 46, 104, 116, 109, 48, 31, 6, 3, 85, 29, 35, 4, 24, 48, 22, 128, 20, 208, 76, 131, 209, 142, 113, 46, 61, 52, 177, 255, 155, 121, 44, 142, 116, 212, 226, 120, 6, 48, 19, 6, 3, 85, 29, 37, 4, 12, 48, 10, 6, 8, 43, 6, 1, 5, 5, 7, 3, 1, 48, 13, 6, 9, 42, 134, 72, 134, 247, 13, 1, 1, 12, 5, 0, 3, 130, 2, 1, 0, 127, 14, 76, 119, 16, 30, 113, 75, 136, 158, 234, 180, 48, 26, 140, 223, 208, 88, 116, 11, 139, 240, 141, 40, 46, 35, 73, 190, 186, 171, 118, 120, 99, 118, 88, 110, 181, 203, 42, 120, 241, 109, 206, 58, 1, 63, 144, 160, 72, 2, 211, 217, 25, 137, 246, 187, 186, 49, 138, 232, 122, 82, 3, 136, 22, 157, 87, 191, 86, 125, 221, 231, 158, 0, 204, 174, 39, 65, 179, 160, 12, 205, 190, 69, 207, 65, 139, 215, 99, 239, 19, 51, 250, 33, 66, 49, 250, 3, 217, 217, 178, 148, 58, 186, 139, 102, 86, 205, 171, 36, 144, 223, 188, 149, 248, 1, 103, 232, 163, 162, 100, 23, 166, 66, 215, 67, 190, 124, 120, 217, 106, 101, 246, 252, 8, 144, 121, 98, 136, 193, 124, 213, 80, 149, 73, 94, 215, 24, 197, 138, 7, 143, 140, 229, 172, 93, 155, 162, 211, 4, 222, 43, 210, 38, 230, 222, 23, 202, 214, 208, 103, 147, 145, 86, 53, 202, 100, 48, 50, 41, 5, 116, 80, 254, 160, 164, 22, 192, 242, 55, 78, 107, 236, 232, 202, 76, 47, 127, 5, 241, 21, 79, 8, 239, 16, 12, 106, 11, 200, 233, 173, 167, 160, 195, 61, 159, 42, 183, 223, 149, 235, 138, 251, 54, 65, 95, 191, 206, 35, 254, 15, 224, 194, 106, 185, 99, 23, 156, 154, 222, 84, 145, 1, 212, 126, 42, 74, 254, 71, 27, 110, 6, 83, 252, 190, 60, 29, 24, 98, 214, 226, 88, 108, 117, 86, 233, 3, 27, 205, 3, 203, 36, 3, 175, 73, 206, 16, 65, 117, 125, 162, 73, 22, 207, 82, 232, 237, 197, 40, 139, 79, 103, 210, 165, 182, 151, 205, 15, 253, 195, 116, 121, 178, 103, 118, 187, 165, 173, 124, 138, 96, 201, 232, 236, 212, 18, 179, 244, 160, 183, 40, 12, 30, 207, 237, 36, 120, 26, 57, 114, 158, 34, 152, 93, 132, 52, 124, 112, 222, 90, 37, 164, 188, 88, 53, 52, 110, 127, 166, 176, 30, 184, 129, 236, 40, 151, 237, 246, 44, 182, 131, 71, 47, 162, 122, 195, 81, 3, 7, 197, 200, 200, 48, 29, 56, 138, 158, 6, 33, 41, 210, 36, 197, 209, 154, 188, 46, 232, 167, 27, 177, 139, 150, 154, 50, 110, 13, 89, 233, 181, 191, 72, 106, 53, 174, 105, 231, 108, 61, 64, 154, 240, 69, 47, 220, 37, 46, 170, 187, 63, 76, 114, 220, 25, 31, 218, 164, 20, 35, 139, 252, 139, 174, 8, 42, 39, 150, 249, 226, 25, 42, 115, 161, 172, 59, 218, 77, 27, 104, 120, 74, 137, 124, 201, 198, 130, 184, 80, 219, 176, 188, 97, 76, 79, 77, 137, 160, 247, 211, 155, 231, 107, 56, 112, 147, 133, 216, 183, 201, 229, 98, 78, 26, 232, 122, 160, 50, 149, 73, 205, 59, 201, 173, 34, 22, 180, 220, 49, 209, 56, 199, 177, 177, 107, 162, 228, 222, 80, 57, 112, 238, 205],
RawTBSCertificate: [48, 130, 6, 251, 160, 3, 2, 1, 2, 2, 19, 73, 1, 74, 89, 119, 169, 186, 32, 59, 56, 235, 138, 87, 0, 0, 1, 74, 89, 119, 48, 13, 6, 9, 42, 134, 72, 134, 247, 13, 1, 1, 12, 5, 0, 48, 87, 49, 11, 48, 9, 6, 3, 85, 4, 6, 19, 2, 85, 83, 49, 30, 48, 28, 6, 3, 85, 4, 10, 19, 21, 77, 105, 99, 114, 111, 115, 111, 102, 116, 32, 67, 111, 114, 112, 111, 114, 97, 116, 105, 111, 110, 49, 40, 48, 38, 6, 3, 85, 4, 3, 19, 31, 77, 105, 99, 114, 111, 115, 111, 102, 116, 32, 84, 76, 83, 32, 71, 50, 32, 82, 83, 65, 32, 67, 65, 32, 79, 67, 83, 80, 32, 49, 48, 48, 30, 23, 13, 50, 54, 49, 48, 50, 53, 48, 49, 53, 54, 49, 48, 90, 23, 13, 50, 55, 48, 50, 48, 50, 48, 48, 53, 54, 49, 48, 90, 48, 120, 49, 11, 48, 9, 6, 3, 85, 4, 6, 19, 2, 85, 83, 49, 11, 48, 9, 6, 3, 85, 4, 8, 19, 2, 87, 65, 49, 16, 48, 14, 6, 3, 85, 4, 7, 19, 7, 82, 101, 100, 109, 111, 110, 100, 49, 30, 48, 28, 6, 3, 85, 4, 10, 19, 21, 77, 105, 99, 114, 111, 115, 111, 102, 116, 32, 67, 111, 114, 112, 111, 114, 97, 116, 105, 111, 110, 49, 42, 48, 40, 6, 3, 85, 4, 3, 19, 33, 114, 101, 115, 101, 114, 118, 101, 100, 46, 97, 114, 111, 97, 112, 112, 45, 104, 99, 112, 45, 115, 116, 103, 46, 97, 122, 117, 114, 101, 46, 99, 111, 109, 48, 130, 1, 34, 48, 13, 6, 9, 42, 134, 72, 134, 247, 13, 1, 1, 1, 5, 0, 3, 130, 1, 15, 0, 48, 130, 1, 10, 2, 130, 1, 1, 0, 187, 121, 183, 117, 223, 172, 129, 21, 158, 33, 40, 228, 200, 112, 100, 153, 170, 116, 250, 158, 245, 201, 135, 0, 237, 144, 31, 12, 163, 250, 168, 50, 92, 5, 220, 113, 137, 143, 3, 193, 167, 220, 193, 78, 195, 131, 29, 107, 67, 52, 86, 236, 30, 39, 30, 107, 206, 213, 234, 58, 181, 33, 44, 227, 147, 157, 23, 219, 20, 127, 20, 235, 246, 77, 135, 5, 240, 135, 231, 58, 154, 101, 19, 42, 160, 128, 73, 94, 79, 155, 62, 172, 213, 53, 57, 2, 149, 60, 188, 169, 233, 44, 216, 105, 66, 159, 148, 70, 178, 113, 177, 128, 123, 151, 10, 7, 254, 171, 205, 105, 34, 145, 93, 19, 218, 220, 219, 47, 129, 42, 85, 247, 229, 7, 118, 48, 182, 125, 84, 45, 187, 16, 138, 130, 74, 176, 14, 70, 229, 120, 108, 212, 172, 177, 174, 141, 161, 231, 167, 5, 108, 191, 229, 39, 27, 122, 242, 198, 32, 78, 125, 8, 168, 234, 200, 195, 10, 209, 101, 183, 248, 26, 10, 82, 78, 47, 196, 44, 55, 199, 207, 209, 35, 65, 164, 44, 162, 97, 81, 36, 122, 188, 105, 82, 214, 39, 252, 78, 115, 249, 159, 36, 212, 167, 6, 72, 63, 56, 220, 88, 50, 162, 50, 174, 185, 196, 246, 208, 84, 90, 212, 218, 52, 118, 184, 101, 201, 132, 48, 29, 45, 113, 80, 81, 157, 255, 235, 218, 49, 139, 40, 124, 111, 147, 143, 201, 2, 3, 1, 0, 1, 163, 130, 4, 181, 48, 130, 4, 177, 48, 130, 1, 4, 6, 10, 43, 6, 1, 4, 1, 214, 121, 2, 4, 2, 4, 129, 245, 4, 129, 242, 0, 240, 0, 118, 0, 68, 194, 189, 12, 233, 20, 14, 100, 165, 201, 74, 1, 147, 10, 90, 161, 187, 53, 151, 14, 0, 238, 17, 22, 137, 104, 42, 28, 68, 215, 181, 102, 0, 0, 1, 161, 112, 206, 239, 163, 0, 0, 4, 3, 0, 71, 48, 69, 2, 33, 0, 172, 23, 89, 180, 174, 255, 10, 75, 120, 126, 2, 114, 141, 52, 53, 9, 237, 84, 209, 98, 89, 54, 179, 9, 54, 118, 245, 178, 107, 207, 5, 69, 2, 32, 89, 15, 78, 239, 171, 216, 123, 169, 212, 117, 5, 12, 137, 15, 63, 83, 41, 96, 223, 14, 136, 244, 40, 31, 142, 25, 69, 157, 200, 47, 205, 167, 0, 118, 0, 0, 26, 93, 26, 28, 45, 147, 117, 182, 72, 85, 120, 248, 47, 113, 161, 174, 110, 239, 57, 125, 41, 124, 138, 227, 21, 123, 202, 222, 225, 160, 30, 0, 0, 1, 161, 112, 206, 240, 23, 0, 0, 4, 3, 0, 71, 48, 69, 2, 33, 0, 209, 5, 94, 127, 36, 105, 13, 219, 166, 20, 49, 161, 208, 24, 148, 233, 208, 121, 79, 240, 195, 85, 152, 81, 108, 234, 202, 116, 68, 189, 194, 57, 2, 32, 65, 141, 139, 86, 66, 203, 70, 108, 26, 159, 237, 96, 32, 237, 241, 53, 230, 112, 171, 119, 90, 74, 10, 14, 99, 1, 225, 78, 245, 158, 165, 135, 48, 27, 6, 9, 43, 6, 1, 4, 1, 130, 55, 21, 10, 4, 14, 48, 12, 48, 10, 6, 8, 43, 6, 1, 5, 5, 7, 3, 1, 48, 60, 6, 9, 43, 6, 1, 4, 1, 130, 55, 21, 7, 4, 47, 48, 45, 6, 37, 43, 6, 1, 4, 1, 130, 55, 21, 8, 135, 189, 215, 27, 129, 231, 235, 70, 130, 129, 157, 46, 142, 208, 12, 135, 240, 218, 29, 93, 133, 246, 239, 42, 129, 232, 226, 53, 2, 1, 100, 2, 1, 39, 48, 130, 1, 11, 6, 8, 43, 6, 1, 5, 5, 7, 1, 1, 4, 129, 254, 48, 129, 251, 48, 97, 6, 8, 43, 6, 1, 5, 5, 7, 48, 2, 134, 85, 104, 116, 116, 112, 58, 47, 47, 119, 119, 119, 46, 109, 105, 99, 114, 111, 115, 111, 102, 116, 46, 99, 111, 109, 47, 112, 107, 105, 111, 112, 115, 47, 99, 101, 114, 116, 115, 47, 77, 105, 99, 114, 111, 115, 111, 102, 116, 37, 50, 48, 84, 76, 83, 37, 50, 48, 71, 50, 37, 50, 48, 82, 83, 65, 37, 50, 48, 67, 65, 37, 50, 48, 79, 67, 83, 80, 37, 50, 48, 49, 48, 46, 99, 114, 116, 48, 103, 6, 8, 43, 6, 1, 5, 5, 7, 48, 2, 134, 91, 104, 116, 116, 112, 58, 47, 47, 99, 97, 105, 115, 115, 117, 101, 114, 115, 46, 109, 105, 99, 114, 111, 115, 111, 102, 116, 46, 99, 111, 109, 47, 112, 107, 105, 111, 112, 115, 47, 99, 101, 114, 116, 115, 47, 77, 105, 99, 114, 111, 115, 111, 102, 116, 37, 50, 48, 84, 76, 83, 37, 50, 48, 71, 50, 37, 50, 48, 82, 83, 65, 37, 50, 48, 67, 65, 37, 50, 48, 79, 67, 83, 80, 37, 50, 48, 49, 48, 46, 99, 114, 116, 48, 45, 6, 8, 43, 6, 1, 5, 5, 7, 48, 1, 134, 33, 104, 116, 116, 112, 58, 47, 47, 111, 110, 101, 111, 99, 115, 112, 46, 109, 105, 99, 114, 111, 115, 111, 102, 116, 46, 99, 111, 109, 47, 111, 99, 115, 112, 48, 29, 6, 3, 85, 29, 14, 4, 22, 4, 20, 138, 66, 241, 22, 211, 0, 191, 156, 21, 196, 84, 38, 230, 58, 211, 190, 115, 187, 163, 122, 48, 14, 6, 3, 85, 29, 15, 1, 1, 255, 4, 4, 3, 2, 5, 160, 48, 110, 6, 3, 85, 29, 17, 4, 103, 48, 101, 130, 64, 42, 46, 97, 112, 112, 115, 46, 97, 114, 111, 46, 105, 110, 103, 114, 101, 115, 115, 45, 99, 101, 114, 116, 45, 108, 101, 46, 48, 111, 51, 109, 46, 117, 107, 115, 111, 117, 116, 104, 46, 97, 114, 111, 97, 112, 112, 45, 104, 99, 112, 45, 115, 116, 103, 46, 97, 122, 117, 114, 101, 46, 99, 111, 109, 130, 33, 114, 101, 115, 101, 114, 118, 101, 100, 46, 97, 114, 111, 97, 112, 112, 45, 104, 99, 112, 45, 115, 116, 103, 46, 97, 122, 117, 114, 101, 46, 99, 111, 109, 48, 12, 6, 3, 85, 29, 19, 1, 1, 255, 4, 2, 48, 0, 48, 129, 241, 6, 3, 85, 29, 31, 4, 129, 233, 48, 129, 230, 48, 129, 227, 160, 129, 224, 160, 129, 221, 134, 108, 104, 116, 116, 112, 58, 47, 47, 119, 119, 119, 46, 109, 105, 99, 114, 111, 115, 111, 102, 116, 46, 99, 111, 109, 47, 112, 107, 105, 111, 112, 115, 47, 99, 114, 108, 47, 112, 97, 114, 116, 105, 116, 105, 111, 110, 47, 77, 105, 99, 114, 111, 115, 111, 102, 116, 37, 50, 48, 84, 76, 83, 37, 50, 48, 71, 50, 37, 50, 48, 82, 83, 65, 37, 50, 48, 67, 65, 37, 50, 48, 79, 67, 83, 80, 37, 50, 48, 49, 48, 95, 80, 97, 114, 116, 105, 116, 105, 111, 110, 48, 48, 48, 57, 57, 46, 99, 114, 108, 134, 109, 104, 116, 116, 112, 58, 47, 47, 99, 114, 108, 50, 46, 109, 105, 99, 114, 111, 115, 111, 102, 116, 46, 99, 111, 109, 47, 112, 107, 105, 111, 112, 115, 47, 99, 114, 108, 47, 112, 97, 114, 116, 105, 116, 105, 111, 110, 47, 77, 105, 99, 114, 111, 115, 111, 102, 116, 37, 50, 48, 84, 76, 83, 37, 50, 48, 71, 50, 37, 50, 48, 82, 83, 65, 37, 50, 48, 67, 65, 37, 50, 48, 79, 67, 83, 80, 37, 50, 48, 49, 48, 95, 80, 97, 114, 116, 105, 116, 105, 111, 110, 48, 48, 48, 57, 57, 46, 99, 114, 108, 48, 102, 6, 3, 85, 29, 32, 4, 95, 48, 93, 48, 8, 6, 6, 103, 129, 12, 1, 2, 2, 48, 81, 6, 12, 43, 6, 1, 4, 1, 130, 55, 76, 131, 125, 1, 1, 48, 65, 48, 63, 6, 8, 43, 6, 1, 5, 5, 7, 2, 1, 22, 51, 104, 116, 116, 112, 58, 47, 47, 119, 119, 119, 46, 109, 105, 99, 114, 111, 115, 111, 102, 116, 46, 99, 111, 109, 47, 112, 107, 105, 111, 112, 115, 47, 68, 111, 99, 115, 47, 82, 101, 112, 111, 115, 105, 116, 111, 114, 121, 46, 104, 116, 109, 48, 31, 6, 3, 85, 29, 35, 4, 24, 48, 22, 128, 20, 208, 76, 131, 209, 142, 113, 46, 61, 52, 177, 255, 155, 121, 44, 142, 116, 212, 226, 120, 6, 48, 19, 6, 3, 85, 29, 37, 4, 12, 48, 10, 6, 8, 43, 6, 1, 5, 5, 7, 3, 1],
RawSubjectPublicKeyInfo: [48, 130, 1, 34, 48, 13, 6, 9, 42, 134, 72, 134, 247, 13, 1, 1, 1, 5, 0, 3, 130, 1, 15, 0, 48, 130, 1, 10, 2, 130, 1, 1, 0, 187, 121, 183, 117, 223, 172, 129, 21, 158, 33, 40, 228, 200, 112, 100, 153, 170, 116, 250, 158, 245, 201, 135, 0, 237, 144, 31, 12, 163, 250, 168, 50, 92, 5, 220, 113, 137, 143, 3, 193, 167, 220, 193, 78, 195, 131, 29, 107, 67, 52, 86, 236, 30, 39, 30, 107, 206, 213, 234, 58, 181, 33, 44, 227, 147, 157, 23, 219, 20, 127, 20, 235, 246, 77, 135, 5, 240, 135, 231, 58, 154, 101, 19, 42, 160, 128, 73, 94, 79, 155, 62, 172, 213, 53, 57, 2, 149, 60, 188, 169, 233, 44, 216, 105, 66, 159, 148, 70, 178, 113, 177, 128, 123, 151, 10, 7, 254, 171, 205, 105, 34, 145, 93, 19, 218, 220, 219, 47, 129, 42, 85, 247, 229, 7, 118, 48, 182, 125, 84, 45, 187, 16, 138, 130, 74, 176, 14, 70, 229, 120, 108, 212, 172, 177, 174, 141, 161, 231, 167, 5, 108, 191, 229, 39, 27, 122, 242, 198, 32, 78, 125, 8, 168, 234, 200, 195, 10, 209, 101, 183, 248, 26, 10, 82, 78, 47, 196, 44, 55, 199, 207, 209, 35, 65, 164, 44, 162, 97, 81, 36, 122, 188, 105, 82, 214, 39, 252, 78, 115, 249, 159, 36, 212, 167, 6, 72, 63, 56, 220, 88, 50, 162, 50, 174, 185, 196, 246, 208, 84, 90, 212, 218, 52, 118, 184, 101, 201, 132, 48, 29, 45, 113, 80, 81, 157, 255, 235, 218, 49, 139, 40, 124, 111, 147, 143, 201, 2, 3, 1, 0, 1],
RawSubject: [48, 120, 49, 11, 48, 9, 6, 3, 85, 4, 6, 19, 2, 85, 83, 49, 11, 48, 9, 6, 3, 85, 4, 8, 19, 2, 87, 65, 49, 16, 48, 14, 6, 3, 85, 4, 7, 19, 7, 82, 101, 100, 109, 111, 110, 100, 49, 30, 48, 28, 6, 3, 85, 4, 10, 19, 21, 77, 105, 99, 114, 111, 115, 111, 102, 116, 32, 67, 111, 114, 112, 111, 114, 97, 116, 105, 111, 110, 49, 42, 48, 40, 6, 3, 85, 4, 3, 19, 33, 114, 101, 115, 101, 114, 118, 101, 100, 46, 97, 114, 111, 97, 112, 112, 45, 104, 99, 112, 45, 115, 116, 103, 46, 97, 122, 117, 114, 101, 46, 99, 111, 109],
RawIssuer: [48, 87, 49, 11, 48, 9, 6, 3, 85, 4, 6, 19, 2, 85, 83, 49, 30, 48, 28, 6, 3, 85, 4, 10, 19, 21, 77, 105, 99, 114, 111, 115, 111, 102, 116, 32, 67, 111, 114, 112, 111, 114, 97, 116, 105, 111, 110, 49, 40, 48, 38, 6, 3, 85, 4, 3, 19, 31, 77, 105, 99, 114, 111, 115, 111, 102, 116, 32, 84, 76, 83, 32, 71, 50, 32, 82, 83, 65, 32, 67, 65, 32, 79, 67, 83, 80, 32, 49, 48],
Signature: [127, 14, 76, 119, 16, 30, 113, 75, 136, 158, 234, 180, 48, 26, 140, 223, 208, 88, 116, 11, 139, 240, 141, 40, 46, 35, 73, 190, 186, 171, 118, 120, 99, 118, 88, 110, 181, 203, 42, 120, 241, 109, 206, 58, 1, 63, 144, 160, 72, 2, 211, 217, 25, 137, 246, 187, 186, 49, 138, 232, 122, 82, 3, 136, 22, 157, 87, 191, 86, 125, 221, 231, 158, 0, 204, 174, 39, 65, 179, 160, 12, 205, 190, 69, 207, 65, 139, 215, 99, 239, 19, 51, 250, 33, 66, 49, 250, 3, 217, 217, 178, 148, 58, 186, 139, 102, 86, 205, 171, 36, 144, 223, 188, 149, 248, 1, 103, 232, 163, 162, 100, 23, 166, 66, 215, 67, 190, 124, 120, 217, 106, 101, 246, 252, 8, 144, 121, 98, 136, 193, 124, 213, 80, 149, 73, 94, 215, 24, 197, 138, 7, 143, 140, 229, 172, 93, 155, 162, 211, 4, 222, 43, 210, 38, 230, 222, 23, 202, 214, 208, 103, 147, 145, 86, 53, 202, 100, 48, 50, 41, 5, 116, 80, 254, 160, 164, 22, 192, 242, 55, 78, 107, 236, 232, 202, 76, 47, 127, 5, 241, 21, 79, 8, 239, 16, 12, 106, 11, 200, 233, 173, 167, 160, 195, 61, 159, 42, 183, 223, 149, 235, 138, 251, 54, 65, 95, 191, 206, 35, 254, 15, 224, 194, 106, 185, 99, 23, 156, 154, 222, 84, 145, 1, 212, 126, 42, 74, 254, 71, 27, 110, 6, 83, 252, 190, 60, 29, 24, 98, 214, 226, 88, 108, 117, 86, 233, 3, 27, 205, 3, 203, 36, 3, 175, 73, 206, 16, 65, 117, 125, 162, 73, 22, 207, 82, 232, 237, 197, 40, 139, 79, 103, 210, 165, 182, 151, 205, 15, 253, 195, 116, 121, 178, 103, 118, 187, 165, 173, 124, 138, 96, 201, 232, 236, 212, 18, 179, 244, 160, 183, 40, 12, 30, 207, 237, 36, 120, 26, 57, 114, 158, 34, 152, 93, 132, 52, 124, 112, 222, 90, 37, 164, 188, 88, 53, 52, 110, 127, 166, 176, 30, 184, 129, 236, 40, 151, 237, 246, 44, 182, 131, 71, 47, 162, 122, 195, 81, 3, 7, 197, 200, 200, 48, 29, 56, 138, 158, 6, 33, 41, 210, 36, 197, 209, 154, 188, 46, 232, 167, 27, 177, 139, 150, 154, 50, 110, 13, 89, 233, 181, 191, 72, 106, 53, 174, 105, 231, 108, 61, 64, 154, 240, 69, 47, 220, 37, 46, 170, 187, 63, 76, 114, 220, 25, 31, 218, 164, 20, 35, 139, 252, 139, 174, 8, 42, 39, 150, 249, 226, 25, 42, 115, 161, 172, 59, 218, 77, 27, 104, 120, 74, 137, 124, 201, 198, 130, 184, 80, 219, 176, 188, 97, 76, 79, 77, 137, 160, 247, 211, 155, 231, 107, 56, 112, 147, 133, 216, 183, 201, 229, 98, 78, 26, 232, 122, 160, 50, 149, 73, 205, 59, 201, 173, 34, 22, 180, 220, 49, 209, 56, 199, 177, 177, 107, 162, 228, 222, 80, 57, 112, 238, 205],
SignatureAlgorithm: 5,
PublicKeyAlgorithm: 1,
PublicKey: <*rsa.PublicKey | 0x36a119f65b90>{
N: {
neg: false,
abs: [3569991644534247369, 3274486715548429274, 3780411684536201245, 13386095367080826074, 4555633294449980078, 8356885564098020936, 8844059572749990990, 2540492176423473444, 5633937353057226705, 779516051327683154, 2327935533544294595, 7836233832775480006, 12443919168731326213, 5381817252368116948, 13149759042380401282, 9307346103102305840, 2490874408770263855, 8905597817602166121, 4800718759162655104, 10753677448201427049, 5736247462418790658, 11125319528290994526, 17747990166117017402, 10636684124628129003, 14904076079720377571, 4842591071448276587, 12095755243242069355, 6630948406570189761, 17118216322786437170, 12282717644381587200, 11394433495304529049, 13509030274166194453],
},
E: 65537,
},
Version: 3,
SerialNumber: {
neg: false,
abs: [9968436300229532023, 6446808008473917675, 4784458],
},
Issuer: {
Country: ["US"],
Organization: [
"Microsoft Corporation",
],
OrganizationalUnit: nil,
Locality: nil,
Province: nil,
StreetAddress: nil,
PostalCode: nil,
SerialNumber: "",
CommonName: "Microsoft TLS G2 RSA CA OCSP 10",
Names: [
{
Type: [..., ..., ..., ...],
Value: <string>"US",
},
{
Type: [..., ..., ..., ...],
Value: <string>"Microsoft Corporation",
},
{
Type: [..., ..., ..., ...],
Value: <string>"Microsoft TLS G2 RSA CA OCSP 10",
},
],
ExtraNames: nil,
},
Subject: {
Country: ["US"],
Organization: [
"Microsoft Corporation",
],
OrganizationalUnit: nil,
Locality: ["Redmond"],
Province: ["WA"],
StreetAddress: nil,
PostalCode: nil,
SerialNumber: "",
CommonName: "reserved.aroapp-hcp-stg.azure.com",
Names: [
{
Type: [..., ..., ..., ...],
Value: <string>"US",
},
{
Type: [..., ..., ..., ...],
Value: <string>"WA",
},
{
Type: [..., ..., ..., ...],
Value: <string>"Redmond",
},
{
Type: [..., ..., ..., ...],
Value: <string>"Microsoft Corporation",
},
{
Type: [..., ..., ..., ...],
Value: <string>"reserved.aroapp-hcp-stg.azure.com",
},
],
ExtraNames: nil,
},
NotBefore: {wall: 0, ext: 63928490170, loc: nil},
NotAfter: {wall: 0, ext: 63937126570, loc: nil},
KeyUsage: 5,
Extensions: [
{
Id: [1, 3, 6, 1, 4, 1, 11129, 2, 4, 2],
Critical: false,
Value: [4, 129, 242, 0, 240, 0, 118, 0, 68, 194, 189, 12, 233, 20, 14, 100, 165, 201, 74, 1, 147, 10, 90, 161, 187, 53, 151, 14, 0, 238, 17, 22, 137, 104, 42, 28, 68, 215, 181, 102, 0, 0, 1, 161, 112, 206, 239, 163, 0, 0, 4, 3, 0, 71, 48, 69, 2, 33, 0, 172, 23, 89, 180, 174, 255, 10, 75, 120, 126, 2, 114, 141, 52, 53, 9, 237, 84, 209, 98, 89, 54, 179, 9, 54, 118, 245, 178, 107, 207, 5, 69, 2, 32, 89, 15, 78, 239, 171, 216, 123, 169, 212, 117, 5, 12, 137, 15, 63, 83, 41, 96, 223, 14, 136, 244, 40, 31, 142, 25, 69, 157, 200, 47, 205, 167, 0, 118, 0, 0, 26, 93, 26, 28, 45, 147, 117, 182, 72, 85, 120, 248, 47, 113, 161, 174, 110, 239, 57, 125, 41, 124, 138, 227, 21, 123, 202, 222, 225, 160, 30, 0, 0, 1, 161, 112, 206, 240, 23, 0, 0, 4, 3, 0, 71, 48, 69, 2, 33, 0, 209, 5, 94, 127, 36, 105, 13, 219, 166, 20, 49, 161, 208, 24, 148, 233, 208, 121, 79, 240, 195, 85, 152, 81, 108, 234, 202, 116, 68, 189, 194, 57, 2, 32, 65, 141, 139, 86, 66, 203, 70, 108, 26, 159, 237, 96, 32, 237, 241, 53, 230, 112, 171, 119, 90, 74, 10, 14, 99, 1, 225, 78, 245, 158, 165, 135],
},
{
Id: [1, 3, 6, 1, 4, 1, 311, 21, 10],
Critical: false,
Value: [48, 12, 48, 10, 6, 8, 43, 6, 1, 5, 5, 7, 3, 1],
},
{
Id: [1, 3, 6, 1, 4, 1, 311, 21, 7],
Critical: false,
Value: [48, 45, 6, 37, 43, 6, 1, 4, 1, 130, 55, 21, 8, 135, 189, 215, 27, 129, 231, 235, 70, 130, 129, 157, 46, 142, 208, 12, 135, 240, 218, 29, 93, 133, 246, 239, 42, 129, 232, 226, 53, 2, 1, 100, 2, 1, 39],
},
{
Id: [1, 3, 6, 1, 5, 5, 7, 1, 1],
Critical: false,
Value: [48, 129, 251, 48, 97, 6, 8, 43, 6, 1, 5, 5, 7, 48, 2, 134, 85, 104, 116, 116, 112, 58, 47, 47, 119, 119, 119, 46, 109, 105, 99, 114, 111, 115, 111, 102, 116, 46, 99, 111, 109, 47, 112, 107, 105, 111, 112, 115, 47, 99, 101, 114, 116, 115, 47, 77, 105, 99, 114, 111, 115, 111, 102, 116, 37, 50, 48, 84, 76, 83, 37, 50, 48, 71, 50, 37, 50, 48, 82, 83, 65, 37, 50, 48, 67, 65, 37, 50, 48, 79, 67, 83, 80, 37, 50, 48, 49, 48, 46, 99, 114, 116, 48, 103, 6, 8, 43, 6, 1, 5, 5, 7, 48, 2, 134, 91, 104, 116, 116, 112, 58, 47, 47, 99, 97, 105, 115, 115, 117, 101, 114, 115, 46, 109, 105, 99, 114, 111, 115, 111, 102, 116, 46, 99, 111, 109, 47, 112, 107, 105, 111, 112, 115, 47, 99, 101, 114, 116, 115, 47, 77, 105, 99, 114, 111, 115, 111, 102, 116, 37, 50, 48, 84, 76, 83, 37, 50, 48, 71, 50, 37, 50, 48, 82, 83, 65, 37, 50, 48, 67, 65, 37, 50, 48, 79, 67, 83, 80, 37, 50, 48, 49, 48, 46, 99, 114, 116, 48, 45, 6, 8, 43, 6, 1, 5, 5, 7, 48, 1, 134, 33, 104, 116, 116, 112, 58, 47, 47, 111, 110, 101, 111, 99, 115, 112, 46, 109, 105, 99, 114, 111, 115, 111, 102, 116, 46, 99, 111, 109, 47, 111, 99, 115, 112],
},
{
Id: [2, 5, 29, 14],
Critical: false,
Value: [4, 20, 138, 66, 241, 22, 211, 0, 191, 156, 21, 196, 84, 38, 230, 58, 211, 190, 115, 187, 163, 122],
},
{
Id: [2, 5, 29, 15],
Critical: true,
Value: [3, 2, 5, 160],
},
{
Id: [2, 5, 29, 17],
Critical: false,
Value: "0e\x82@*.apps.aro.ingress-cert-le.0o3m.uksouth.aroapp-hcp-stg.azure.com\x82!reserved.aroapp-hcp-stg.azure.com",
},
{Id: [2, 5, 29, 19], Critical: true, Value: [48, 0]},
{
Id: [2, 5, 29, 31],
Critical: false,
Value: "0\x81\xe60\x81㠁ࠁ݆lhttp://www.microsoft.com/pkiops/crl/partition/Microsoft%20TLS%20G2%20RSA%20CA%20OCSP%2010_Partition00099.crl\x86mhttp://crl2.microsoft.com/pkiops/crl/partition/Microsoft%20TLS%20G2%20RSA%20CA%20OCSP%2010_Partition00099.crl",
},
{
Id: [2, 5, 29, 32],
Critical: false,
Value: [48, 93, 48, 8, 6, 6, 103, 129, 12, 1, 2, 2, 48, 81, 6, 12, 43, 6, 1, 4, 1, 130, 55, 76, 131, 125, 1, 1, 48, 65, 48, 63, 6, 8, 43, 6, 1, 5, 5, 7, 2, 1, 22, 51, 104, 116, 116, 112, 58, 47, 47, 119, 119, 119, 46, 109, 105, 99, 114, 111, 115, 111, 102, 116, 46, 99, 111, 109, 47, 112, 107, 105, 111, 112, 115, 47, 68, 111, 99, 115, 47, 82, 101, 112, 111, 115, 105, 116, 111, 114, 121, 46, 104, 116, 109],
},
{
Id: [2, 5, 29, 35],
Critical: false,
Value: [48, 22, 128, 20, 208, 76, 131, 209, 142, 113, 46, 61, 52, 177, 255, 155, 121, 44, 142, 116, 212, 226, 120, 6],
},
{
Id: [2, 5, 29, 37],
Critical: false,
Value: [48, 10, 6, 8, 43, 6, 1, 5, 5, 7, 3, 1],
},
],
ExtraExtensions: nil,
UnhandledCriticalExtensions: nil,
ExtKeyUsage: [1],
UnknownExtKeyUsage: nil,
BasicConstraintsValid: true,
IsCA: false,
MaxPathLen: -1,
MaxPathLenZero: false,
SubjectKeyId: [138, 66, 241, 22, 211, 0, 191, 156, 21, 196, 84, 38, 230, 58, 211, 190, 115, 187, 163, 122],
AuthorityKeyId: [208, 76, 131, 209, 142, 113, 46, 61, 52, 177, 255, 155, 121, 44, 142, 116, 212, 226, 120, 6],
OCSPServer: [
"http://oneocsp.microsoft.com/ocsp",
],
IssuingCertificateURL: [
"http://www.microsoft.com/pkiops/certs/Microsoft%20TLS%20G2%20RSA%20CA%20OCSP%2010.crt",
"http://caissuers.microsoft.com/pkiops/certs/Microsoft%20TLS%20G2%20RSA%20CA%20OCSP%2010.crt",
],
DNSNames: [
"*.apps.aro.ingress-cert-le.0o3m.uksouth.aroapp-hcp-stg.azure.com",
"reserved.aroapp-hcp-stg.azure.com",
],
EmailAddresses: nil,
IPAddresses: nil,
URIs: nil,
PermittedDNSDomainsCritical: false,
PermittedDNSDomains: nil,
ExcludedDNSDomains: nil,
PermittedIPRanges: nil,
ExcludedIPRanges: nil,
PermittedEmailAddresses: nil,
ExcludedEmailAddresses: nil,
PermittedURIDomains: nil,
ExcludedURIDomains: nil,
CRLDistributionPoints: [
"http://www.microsoft.com/pkiops/crl/partition/Microsoft%20TLS%20G2%20RSA%20CA%20OCSP%2010_Partition00099.crl",
"http://crl2.microsoft.com/pkiops/crl/partition/Microsoft%20TLS%20G2%20RSA%20CA%20OCSP%2010_Partition00099.crl",
],
PolicyIdentifiers: [
[2, 23, 140, 1, 2, 2],
[1, 3, 6, 1, 4, 1, 311, 76, 509, 1, 1],
],
Policies: [
{
der: [103, 129, 12, 1, 2, 2],
},
{
der: [43, 6, 1, 4, 1, 130, 55, 76, 131, 125, 1, 1],
},
],
InhibitAnyPolicy: 0,
InhibitAnyPolicyZero: false,
InhibitPolicyMapping: 0,
InhibitPolicyMappingZero: false,
RequireExplicitPolicy: 0,
RequireExplicitPolicyZero: false,
PolicyMappings: nil,
},
{
Raw: [48, 130, 7, 184, 48, 130, 5, 160, 160, 3, 2, 1, 2, 2, 19, 51, 0, 0, 0, 15, 51, 32, 101, 55, 238, 66, 174, 79, 0, 0, 0, 0, 0, 15, 48, 13, 6, 9, 42, 134, 72, 134, 247, 13, 1, 1, 12, 5, 0, 48, 81, 49, 11, 48, 9, 6, 3, 85, 4, 6, 19, 2, 85, 83, 49, 30, 48, 28, 6, 3, 85, 4, 10, 19, 21, 77, 105, 99, 114, 111, 115, 111, 102, 116, 32, 67, 111, 114, 112, 111, 114, 97, 116, 105, 111, 110, 49, 34, 48, 32, 6, 3, 85, 4, 3, 19, 25, 77, 105, 99, 114, 111, 115, 111, 102, 116, 32, 84, 76, 83, 32, 82, 83, 65, 32, 82, 111, 111, 116, 32, 71, 50, 48, 30, 23, 13, 50, 53, 48, 56, 48, 49, 50, 48, 48, 51, 48, 51, 90, 23, 13, 50, 57, 48, 54, 48, 51, 50, 48, 48, 51, 48, 51, 90, 48, 87, 49, 11, 48, 9, 6, 3, 85, 4, 6, 19, 2, 85, 83, 49, 30, 48, 28, 6, 3, 85, 4, 10, 19, 21, 77, 105, 99, 114, 111, 115, 111, 102, 116, 32, 67, 111, 114, 112, 111, 114, 97, 116, 105, 111, 110, 49, 40, 48, 38, 6, 3, 85, 4, 3, 19, 31, 77, 105, 99, 114, 111, 115, 111, 102, 116, 32, 84, 76, 83, 32, 71, 50, 32, 82, 83, 65, 32, 67, 65, 32, 79, 67, 83, 80, 32, 49, 48, 48, 130, 2, 34, 48, 13, 6, 9, 42, 134, 72, 134, 247, 13, 1, 1, 1, 5, 0, 3, 130, 2, 15, 0, 48, 130, 2, 10, 2, 130, 2, 1, 0, 240, 12, 79, 91, 95, 252, 66, 28, 180, 206, 63, 245, 7, 43, 166, 131, 126, 201, 157, 121, 207, 246, 233, 172, 165, 119, 195, 55, 169, 250, 247, 7, 245, 45, 152, 248, 17, 102, 235, 66, 42, 14, 210, 209, 6, 25, 246, 119, 19, 95, 132, 236, 214, 81, 147, 108, 33, 227, 14, 133, 166, 255, 210, 236, 253, 36, 130, 189, 4, 249, 239, 141, 227, 225, 197, 221, 117, 208, 4, 154, 78, 20, 54, 25, 251, 45, 18, 163, 21, 192, 243, 111, 106, 68, 213, 113, 108, 241, 75, 152, 27, 121, 48, 247, 156, 23, 191, 125, 50, 9, 57, 91, 54, 140, 160, 8, 187, 64, 44, 150, 41, 238, 124, 189, 14, 248, 236, 109, 206, 193, 78, 118, 193, 88, 46, 78, 50, 154, 199, 124, 203, 199, 191, 183, 93, 247, 53, 84, 120, 204, 177, 135, 186, 5, 197, 53, 1, 112, 4, 195, 232, 93, 248, 177, 180, 223, 71, 205, 76, 102, 154, 232, 103, 212, 158, 30, 160, 33, 31, 51, 71, 5, 59, 68, 8, 207, 107, 136, 82, 89, 50, 120, 245, 65, 142, 37, 46, 208, 2, 13, 1, 207, 118, 247, 45, 52, 228, 103, 157, 104, 177, 128, 182, 77, 82, 179, 32, 151, 43, 165, 83, 140, 119, 149, 181, 115, 128, 205, 136, 255, 218, 151, 54, 227, 231, 111, 50, 160, 117, 241, 187, 191, 250, 60, 101, 171, 54, 17, 191, 118, 253, 71, 255, 108, 135, 96, 190, 12, 229, 234, 150, 245, 11, 162, 55, 39, 0, 209, 231, 2, 48, 1, 213, 128, 1, 194, 144, 206, 239, 179, 235, 211, 120, 96, 190, 10, 63, 251, 108, 198, 194, 209, 171, 39, 23, 23, 9, 132, 121, 145, 66, 142, 118, 104, 55, 17, 155, 206, 199, 118, 41, 191, 213, 92, 226, 124, 13, 95, 103, 82, 253, 109, 57, 250, 81, 45, 223, 178, 136, 3, 241, 97, 32, 170, 248, 180, 178, 159, 211, 250, 40, 176, 181, 167, 227, 131, 186, 250, 68, 59, 119, 220, 162, 44, 43, 232, 226, 83, 136, 154, 251, 12, 32, 222, 162, 123, 78, 128, 144, 113, 27, 124, 151, 212, 219, 252, 61, 154, 193, 161, 222, 234, 231, 174, 76, 229, 174, 175, 118, 172, 135, 205, 100, 188, 162, 52, 26, 211, 102, 195, 107, 97, 124, 155, 245, 230, 89, 121, 251, 128, 14, 22, 107, 192, 241, 41, 244, 237, 157, 151, 142, 58, 196, 41, 147, 7, 52, 255, 226, 191, 173, 154, 89, 37, 155, 134, 29, 67, 126, 203, 110, 149, 245, 142, 224, 169, 139, 87, 128, 78, 11, 230, 96, 125, 192, 73, 213, 4, 188, 32, 202, 4, 210, 92, 234, 63, 140, 73, 127, 29, 185, 186, 96, 109, 215, 180, 100, 179, 92, 52, 234, 56, 243, 249, 193, 248, 169, 11, 126, 186, 245, 209, 252, 122, 27, 127, 219, 8, 101, 125, 195, 151, 5, 236, 175, 47, 199, 222, 12, 250, 178, 63, 2, 3, 1, 0, 1, 163, 130, 2, 129, 48, 130, 2, 125, 48, 14, 6, 3, 85, 29, 15, 1, 1, 255, 4, 4, 3, 2, 1, 134, 48, 16, 6, 9, 43, 6, 1, 4, 1, 130, 55, 21, 1, 4, 3, 2, 1, 0, 48, 29, 6, 3, 85, 29, 14, 4, 22, 4, 20, 208, 76, 131, 209, 142, 113, 46, 61, 52, 177, 255, 155, 121, 44, 142, 116, 212, 226, 120, 6, 48, 19, 6, 3, 85, 29, 32, 4, 12, 48, 10, 48, 8, 6, 6, 103, 129, 12, 1, 2, 2, 48, 19, 6, 3, 85, 29, 37, 4, 12, 48, 10, 6, 8, 43, 6, 1, 5, 5, 7, 3, 1, 48, 25, 6, 9, 43, 6, 1, 4, 1, 130, 55, 20, 2, 4, 12, 30, 10, 0, 83, 0, 117, 0, 98, 0, 67, 0, 65, 48, 18, 6, 3, 85, 29, 19, 1, 1, 255, 4, 8, 48, 6, 1, 1, 255, 2, 1, 0, 48, 31, 6, 3, 85, 29, 35, 4, 24, 48, 22, 128, 20, 222, 145, 134, 72, 183, 161, 49, 89, 49, 241, 75, 95, 7, 169, 220, 136, 121, 218, 168, 118, 48, 129, 171, 6, 3, 85, 29, 31, 4, 129, 163, 48, 129, 160, 48, 129, 157, 160, 129, 154, 160, 129, 151, 134, 73, 104, 116, 116, 112, 58, 47, 47, 119, 119, 119, 46, 109, 105, 99, 114, 111, 115, 111, 102, 116, 46, 99, 111, 109, 47, 112, 107, 105, 111, 112, 115, 47, 99, 114, 108, 47, 77, 105, 99, 114, 111, 115, 111, 102, 116, 37, 50, 48, 84, 76, 83, 37, 50, 48, 82, 83, 65, 37, 50, 48, 82, 111, 111, 116, 37, 50, 48, 71, 50, 46, 99, 114, 108, 134, 74, 104, 116, 116, 112, 58, 47, 47, 99, 114, 108, 50, 46, 109, 105, 99, 114, 111, 115, 111, 102, 116, 46, 99, 111, 109, 47, 112, 107, 105, 111, 112, 115, 47, 99, 114, 108, 47, 77, 105, 99, 114, 111, 115, 111, 102, 116, 37, 50, 48, 84, 76, 83, 37, 50, 48, 82, 83, 65, 37, 50, 48, 82, 111, 111, 116, 37, 50, 48, 71, 50, 46, 99, 114, 108, 48, 130, 1, 16, 6, 8, 43, 6, 1, 5, 5, 7, 1, 1, 4, 130, 1, 2, 48, 129, 255, 48, 99, 6, 8, 43, 6, 1, 5, 5, 7, 48, 2, 134, 87, 104, 116, 116, 112, 58, 47, 47, 119, 119, 119, 46, 109, 105, 99, 114, 111, 115, 111, 102, 116, 46, 99, 111, 109, 47, 112, 107, 105, 111, 112, 115, 47, 99, 101, 114, 116, 115, 47, 77, 105, 99, 114, 111, 115, 111, 102, 116, 37, 50, 48, 84, 76, 83, 37, 50, 48, 82, 83, 65, 37, 50, 48, 82, 111, 111, 116, 37, 50, 48, 71, 50, 37, 50, 48, 45, 37, 50, 48, 120, 115, 105, 103, 110, 46, 99, 114, 116, 48, 105, 6, 8, 43, 6, 1, 5, 5, 7, 48, 2, 134, 93, 104, 116, 116, 112, 58, 47, 47, 99, 97, 105, 115, 115, 117, 101, 114, 115, 46, 109, 105, 99, 114, 111, 115, 111, 102, 116, 46, 99, 111, 109, 47, 112, 107, 105, 111, 112, 115, 47, 99, 101, 114, 116, 115, 47, 77, 105, 99, 114, 111, 115, 111, 102, 116, 37, 50, 48, 84, 76, 83, 37, 50, 48, 82, 83, 65, 37, 50, 48, 82, 111, 111, 116, 37, 50, 48, 71, 50, 37, 50, 48, 45, 37, 50, 48, 120, 115, 105, 103, 110, 46, 99, 114, 116, 48, 45, 6, 8, 43, 6, 1, 5, 5, 7, 48, 1, 134, 33, 104, 116, 116, 112, 58, 47, 47, 111, 110, 101, 111, 99, 115, 112, 46, 109, 105, 99, 114, 111, 115, 111, 102, 116, 46, 99, 111, 109, 47, 111, 99, 115, 112, 48, 13, 6, 9, 42, 134, 72, 134, 247, 13, 1, 1, 12, 5, 0, 3, 130, 2, 1, 0, 53, 25, 202, 233, 168, 117, 225, 216, 202, 253, 37, 193, 59, 82, 225, 231, 158, 43, 148, 226, 244, 58, 210, 79, 218, 6, 34, 82, 26, 1, 24, 227, 162, 186, 183, 183, 28, 36, 50, 117, 13, 103, 68, 125, 157, 74, 130, 75, 250, 227, 68, 233, 70, 246, 0, 69, 9, 194, 82, 233, 103, 37, 151, 199, 183, 139, 90, 119, 124, 64, 30, 148, 227, 188, 140, 109, 56, 41, 188, 39, 125, 18, 18, 61, 203, 203, 228, 196, 70, 226, 165, 1, 192, 16, 209, 32, 250, 14, 3, 228, 67, 219, 38, 135, 37, 172, 200, 160, 179, 4, 106, 94, 240, 67, 63, 15, 123, 245, 206, 35, 195, 183, 201, 190, 251, 222, 155, 125, 147, 179, 135, 62, 223, 198, 219, 61, 201, 244, 172, 113, 0, 210, 164, 101, 129, 54, 89, 159, 28, 54, 229, 137, 209, 122, 153, 74, 123, 85, 75, 194, 213, 140, 208, 121, 1, 121, 149, 15, 57, 67, 139, 179, 10, 123, 146, 213, 90, 16, 24, 118, 233, 29, 13, 111, 50, 49, 244, 189, 209, 22, 241, 168, 163, 103, 242, 14, 154, 112, 12, 227, 150, 175, 3, 244, 109, 189, 20, 121, 106, 54, 195, 241, 182, 115, 187, 198, 11, 213, 38, 70, 136, 84, 69, 112, 170, 204, 31, 61, 139, 111, 227, 228, 23, 166, 129, 0, 251, 125, 131, 88, 128, 226, 242, 60, 59, 163, 24, 98, 19, 54, 103, 255, 79, 226, 229, 111, 106, 103, 230, 232, 90, 100, 58, 160, 69, 157, 63, 23, 12, 102, 135, 240, 129, 39, 52, 32, 176, 53, 36, 100, 40, 118, 67, 122, 116, 152, 226, 43, 128, 9, 10, 188, 176, 185, 7, 71, 41, 40, 46, 125, 224, 119, 65, 164, 140, 54, 120, 186, 51, 29, 86, 142, 217, 16, 53, 137, 89, 131, 0, 105, 206, 183, 120, 189, 237, 175, 130, 63, 206, 200, 137, 50, 152, 89, 8, 93, 225, 1, 150, 88, 196, 101, 25, 125, 38, 133, 157, 16, 150, 229, 149, 197, 17, 206, 130, 111, 23, 6, 113, 21, 15, 44, 191, 55, 50, 82, 21, 104, 180, 149, 142, 83, 93, 151, 237, 227, 219, 19, 62, 193, 49, 177, 121, 147, 115, 125, 114, 59, 27, 76, 79, 6, 73, 21, 132, 115, 243, 29, 90, 81, 44, 164, 212, 88, 224, 25, 191, 53, 72, 41, 96, 153, 171, 234, 123, 66, 3, 190, 156, 61, 25, 247, 71, 91, 11, 172, 130, 87, 170, 13, 189, 255, 34, 96, 48, 223, 44, 79, 87, 211, 37, 213, 218, 253, 142, 6, 81, 78, 5, 53, 120, 152, 56, 163, 39, 49, 111, 49, 154, 169, 170, 77, 172, 58, 68, 104, 96, 112, 104, 244, 83, 107, 205, 178, 45, 174, 38, 24, 135, 223, 140, 87, 181, 248, 82, 193, 86, 231, 192, 177, 77, 95, 10, 85, 107, 20, 43, 84, 251, 61, 169, 198, 189, 96, 7, 248, 176, 244, 75, 227, 238, 171],
RawTBSCertificate: [48, 130, 5, 160, 160, 3, 2, 1, 2, 2, 19, 51, 0, 0, 0, 15, 51, 32, 101, 55, 238, 66, 174, 79, 0, 0, 0, 0, 0, 15, 48, 13, 6, 9, 42, 134, 72, 134, 247, 13, 1, 1, 12, 5, 0, 48, 81, 49, 11, 48, 9, 6, 3, 85, 4, 6, 19, 2, 85, 83, 49, 30, 48, 28, 6, 3, 85, 4, 10, 19, 21, 77, 105, 99, 114, 111, 115, 111, 102, 116, 32, 67, 111, 114, 112, 111, 114, 97, 116, 105, 111, 110, 49, 34, 48, 32, 6, 3, 85, 4, 3, 19, 25, 77, 105, 99, 114, 111, 115, 111, 102, 116, 32, 84, 76, 83, 32, 82, 83, 65, 32, 82, 111, 111, 116, 32, 71, 50, 48, 30, 23, 13, 50, 53, 48, 56, 48, 49, 50, 48, 48, 51, 48, 51, 90, 23, 13, 50, 57, 48, 54, 48, 51, 50, 48, 48, 51, 48, 51, 90, 48, 87, 49, 11, 48, 9, 6, 3, 85, 4, 6, 19, 2, 85, 83, 49, 30, 48, 28, 6, 3, 85, 4, 10, 19, 21, 77, 105, 99, 114, 111, 115, 111, 102, 116, 32, 67, 111, 114, 112, 111, 114, 97, 116, 105, 111, 110, 49, 40, 48, 38, 6, 3, 85, 4, 3, 19, 31, 77, 105, 99, 114, 111, 115, 111, 102, 116, 32, 84, 76, 83, 32, 71, 50, 32, 82, 83, 65, 32, 67, 65, 32, 79, 67, 83, 80, 32, 49, 48, 48, 130, 2, 34, 48, 13, 6, 9, 42, 134, 72, 134, 247, 13, 1, 1, 1, 5, 0, 3, 130, 2, 15, 0, 48, 130, 2, 10, 2, 130, 2, 1, 0, 240, 12, 79, 91, 95, 252, 66, 28, 180, 206, 63, 245, 7, 43, 166, 131, 126, 201, 157, 121, 207, 246, 233, 172, 165, 119, 195, 55, 169, 250, 247, 7, 245, 45, 152, 248, 17, 102, 235, 66, 42, 14, 210, 209, 6, 25, 246, 119, 19, 95, 132, 236, 214, 81, 147, 108, 33, 227, 14, 133, 166, 255, 210, 236, 253, 36, 130, 189, 4, 249, 239, 141, 227, 225, 197, 221, 117, 208, 4, 154, 78, 20, 54, 25, 251, 45, 18, 163, 21, 192, 243, 111, 106, 68, 213, 113, 108, 241, 75, 152, 27, 121, 48, 247, 156, 23, 191, 125, 50, 9, 57, 91, 54, 140, 160, 8, 187, 64, 44, 150, 41, 238, 124, 189, 14, 248, 236, 109, 206, 193, 78, 118, 193, 88, 46, 78, 50, 154, 199, 124, 203, 199, 191, 183, 93, 247, 53, 84, 120, 204, 177, 135, 186, 5, 197, 53, 1, 112, 4, 195, 232, 93, 248, 177, 180, 223, 71, 205, 76, 102, 154, 232, 103, 212, 158, 30, 160, 33, 31, 51, 71, 5, 59, 68, 8, 207, 107, 136, 82, 89, 50, 120, 245, 65, 142, 37, 46, 208, 2, 13, 1, 207, 118, 247, 45, 52, 228, 103, 157, 104, 177, 128, 182, 77, 82, 179, 32, 151, 43, 165, 83, 140, 119, 149, 181, 115, 128, 205, 136, 255, 218, 151, 54, 227, 231, 111, 50, 160, 117, 241, 187, 191, 250, 60, 101, 171, 54, 17, 191, 118, 253, 71, 255, 108, 135, 96, 190, 12, 229, 234, 150, 245, 11, 162, 55, 39, 0, 209, 231, 2, 48, 1, 213, 128, 1, 194, 144, 206, 239, 179, 235, 211, 120, 96, 190, 10, 63, 251, 108, 198, 194, 209, 171, 39, 23, 23, 9, 132, 121, 145, 66, 142, 118, 104, 55, 17, 155, 206, 199, 118, 41, 191, 213, 92, 226, 124, 13, 95, 103, 82, 253, 109, 57, 250, 81, 45, 223, 178, 136, 3, 241, 97, 32, 170, 248, 180, 178, 159, 211, 250, 40, 176, 181, 167, 227, 131, 186, 250, 68, 59, 119, 220, 162, 44, 43, 232, 226, 83, 136, 154, 251, 12, 32, 222, 162, 123, 78, 128, 144, 113, 27, 124, 151, 212, 219, 252, 61, 154, 193, 161, 222, 234, 231, 174, 76, 229, 174, 175, 118, 172, 135, 205, 100, 188, 162, 52, 26, 211, 102, 195, 107, 97, 124, 155, 245, 230, 89, 121, 251, 128, 14, 22, 107, 192, 241, 41, 244, 237, 157, 151, 142, 58, 196, 41, 147, 7, 52, 255, 226, 191, 173, 154, 89, 37, 155, 134, 29, 67, 126, 203, 110, 149, 245, 142, 224, 169, 139, 87, 128, 78, 11, 230, 96, 125, 192, 73, 213, 4, 188, 32, 202, 4, 210, 92, 234, 63, 140, 73, 127, 29, 185, 186, 96, 109, 215, 180, 100, 179, 92, 52, 234, 56, 243, 249, 193, 248, 169, 11, 126, 186, 245, 209, 252, 122, 27, 127, 219, 8, 101, 125, 195, 151, 5, 236, 175, 47, 199, 222, 12, 250, 178, 63, 2, 3, 1, 0, 1, 163, 130, 2, 129, 48, 130, 2, 125, 48, 14, 6, 3, 85, 29, 15, 1, 1, 255, 4, 4, 3, 2, 1, 134, 48, 16, 6, 9, 43, 6, 1, 4, 1, 130, 55, 21, 1, 4, 3, 2, 1, 0, 48, 29, 6, 3, 85, 29, 14, 4, 22, 4, 20, 208, 76, 131, 209, 142, 113, 46, 61, 52, 177, 255, 155, 121, 44, 142, 116, 212, 226, 120, 6, 48, 19, 6, 3, 85, 29, 32, 4, 12, 48, 10, 48, 8, 6, 6, 103, 129, 12, 1, 2, 2, 48, 19, 6, 3, 85, 29, 37, 4, 12, 48, 10, 6, 8, 43, 6, 1, 5, 5, 7, 3, 1, 48, 25, 6, 9, 43, 6, 1, 4, 1, 130, 55, 20, 2, 4, 12, 30, 10, 0, 83, 0, 117, 0, 98, 0, 67, 0, 65, 48, 18, 6, 3, 85, 29, 19, 1, 1, 255, 4, 8, 48, 6, 1, 1, 255, 2, 1, 0, 48, 31, 6, 3, 85, 29, 35, 4, 24, 48, 22, 128, 20, 222, 145, 134, 72, 183, 161, 49, 89, 49, 241, 75, 95, 7, 169, 220, 136, 121, 218, 168, 118, 48, 129, 171, 6, 3, 85, 29, 31, 4, 129, 163, 48, 129, 160, 48, 129, 157, 160, 129, 154, 160, 129, 151, 134, 73, 104, 116, 116, 112, 58, 47, 47, 119, 119, 119, 46, 109, 105, 99, 114, 111, 115, 111, 102, 116, 46, 99, 111, 109, 47, 112, 107, 105, 111, 112, 115, 47, 99, 114, 108, 47, 77, 105, 99, 114, 111, 115, 111, 102, 116, 37, 50, 48, 84, 76, 83, 37, 50, 48, 82, 83, 65, 37, 50, 48, 82, 111, 111, 116, 37, 50, 48, 71, 50, 46, 99, 114, 108, 134, 74, 104, 116, 116, 112, 58, 47, 47, 99, 114, 108, 50, 46, 109, 105, 99, 114, 111, 115, 111, 102, 116, 46, 99, 111, 109, 47, 112, 107, 105, 111, 112, 115, 47, 99, 114, 108, 47, 77, 105, 99, 114, 111, 115, 111, 102, 116, 37, 50, 48, 84, 76, 83, 37, 50, 48, 82, 83, 65, 37, 50, 48, 82, 111, 111, 116, 37, 50, 48, 71, 50, 46, 99, 114, 108, 48, 130, 1, 16, 6, 8, 43, 6, 1, 5, 5, 7, 1, 1, 4, 130, 1, 2, 48, 129, 255, 48, 99, 6, 8, 43, 6, 1, 5, 5, 7, 48, 2, 134, 87, 104, 116, 116, 112, 58, 47, 47, 119, 119, 119, 46, 109, 105, 99, 114, 111, 115, 111, 102, 116, 46, 99, 111, 109, 47, 112, 107, 105, 111, 112, 115, 47, 99, 101, 114, 116, 115, 47, 77, 105, 99, 114, 111, 115, 111, 102, 116, 37, 50, 48, 84, 76, 83, 37, 50, 48, 82, 83, 65, 37, 50, 48, 82, 111, 111, 116, 37, 50, 48, 71, 50, 37, 50, 48, 45, 37, 50, 48, 120, 115, 105, 103, 110, 46, 99, 114, 116, 48, 105, 6, 8, 43, 6, 1, 5, 5, 7, 48, 2, 134, 93, 104, 116, 116, 112, 58, 47, 47, 99, 97, 105, 115, 115, 117, 101, 114, 115, 46, 109, 105, 99, 114, 111, 115, 111, 102, 116, 46, 99, 111, 109, 47, 112, 107, 105, 111, 112, 115, 47, 99, 101, 114, 116, 115, 47, 77, 105, 99, 114, 111, 115, 111, 102, 116, 37, 50, 48, 84, 76, 83, 37, 50, 48, 82, 83, 65, 37, 50, 48, 82, 111, 111, 116, 37, 50, 48, 71, 50, 37, 50, 48, 45, 37, 50, 48, 120, 115, 105, 103, 110, 46, 99, 114, 116, 48, 45, 6, 8, 43, 6, 1, 5, 5, 7, 48, 1, 134, 33, 104, 116, 116, 112, 58, 47, 47, 111, 110, 101, 111, 99, 115, 112, 46, 109, 105, 99, 114, 111, 115, 111, 102, 116, 46, 99, 111, 109, 47, 111, 99, 115, 112],
RawSubjectPublicKeyInfo: [48, 130, 2, 34, 48, 13, 6, 9, 42, 134, 72, 134, 247, 13, 1, 1, 1, 5, 0, 3, 130, 2, 15, 0, 48, 130, 2, 10, 2, 130, 2, 1, 0, 240, 12, 79, 91, 95, 252, 66, 28, 180, 206, 63, 245, 7, 43, 166, 131, 126, 201, 157, 121, 207, 246, 233, 172, 165, 119, 195, 55, 169, 250, 247, 7, 245, 45, 152, 248, 17, 102, 235, 66, 42, 14, 210, 209, 6, 25, 246, 119, 19, 95, 132, 236, 214, 81, 147, 108, 33, 227, 14, 133, 166, 255, 210, 236, 253, 36, 130, 189, 4, 249, 239, 141, 227, 225, 197, 221, 117, 208, 4, 154, 78, 20, 54, 25, 251, 45, 18, 163, 21, 192, 243, 111, 106, 68, 213, 113, 108, 241, 75, 152, 27, 121, 48, 247, 156, 23, 191, 125, 50, 9, 57, 91, 54, 140, 160, 8, 187, 64, 44, 150, 41, 238, 124, 189, 14, 248, 236, 109, 206, 193, 78, 118, 193, 88, 46, 78, 50, 154, 199, 124, 203, 199, 191, 183, 93, 247, 53, 84, 120, 204, 177, 135, 186, 5, 197, 53, 1, 112, 4, 195, 232, 93, 248, 177, 180, 223, 71, 205, 76, 102, 154, 232, 103, 212, 158, 30, 160, 33, 31, 51, 71, 5, 59, 68, 8, 207, 107, 136, 82, 89, 50, 120, 245, 65, 142, 37, 46, 208, 2, 13, 1, 207, 118, 247, 45, 52, 228, 103, 157, 104, 177, 128, 182, 77, 82, 179, 32, 151, 43, 165, 83, 140, 119, 149, 181, 115, 128, 205, 136, 255, 218, 151, 54, 227, 231, 111, 50, 160, 117, 241, 187, 191, 250, 60, 101, 171, 54, 17, 191, 118, 253, 71, 255, 108, 135, 96, 190, 12, 229, 234, 150, 245, 11, 162, 55, 39, 0, 209, 231, 2, 48, 1, 213, 128, 1, 194, 144, 206, 239, 179, 235, 211, 120, 96, 190, 10, 63, 251, 108, 198, 194, 209, 171, 39, 23, 23, 9, 132, 121, 145, 66, 142, 118, 104, 55, 17, 155, 206, 199, 118, 41, 191, 213, 92, 226, 124, 13, 95, 103, 82, 253, 109, 57, 250, 81, 45, 223, 178, 136, 3, 241, 97, 32, 170, 248, 180, 178, 159, 211, 250, 40, 176, 181, 167, 227, 131, 186, 250, 68, 59, 119, 220, 162, 44, 43, 232, 226, 83, 136, 154, 251, 12, 32, 222, 162, 123, 78, 128, 144, 113, 27, 124, 151, 212, 219, 252, 61, 154, 193, 161, 222, 234, 231, 174, 76, 229, 174, 175, 118, 172, 135, 205, 100, 188, 162, 52, 26, 211, 102, 195, 107, 97, 124, 155, 245, 230, 89, 121, 251, 128, 14, 22, 107, 192, 241, 41, 244, 237, 157, 151, 142, 58, 196, 41, 147, 7, 52, 255, 226, 191, 173, 154, 89, 37, 155, 134, 29, 67, 126, 203, 110, 149, 245, 142, 224, 169, 139, 87, 128, 78, 11, 230, 96, 125, 192, 73, 213, 4, 188, 32, 202, 4, 210, 92, 234, 63, 140, 73, 127, 29, 185, 186, 96, 109, 215, 180, 100, 179, 92, 52, 234, 56, 243, 249, 193, 248, 169, 11, 126, 186, 245, 209, 252, 122, 27, 127, 219, 8, 101, 125, 195, 151, 5, 236, 175, 47, 199, 222, 12, 250, 178, 63, 2, 3, 1, 0, 1],
RawSubject: [48, 87, 49, 11, 48, 9, 6, 3, 85, 4, 6, 19, 2, 85, 83, 49, 30, 48, 28, 6, 3, 85, 4, 10, 19, 21, 77, 105, 99, 114, 111, 115, 111, 102, 116, 32, 67, 111, 114, 112, 111, 114, 97, 116, 105, 111, 110, 49, 40, 48, 38, 6, 3, 85, 4, 3, 19, 31, 77, 105, 99, 114, 111, 115, 111, 102, 116, 32, 84, 76, 83, 32, 71, 50, 32, 82, 83, 65, 32, 67, 65, 32, 79, 67, 83, 80, 32, 49, 48],
RawIssuer: [48, 81, 49, 11, 48, 9, 6, 3, 85, 4, 6, 19, 2, 85, 83, 49, 30, 48, 28, 6, 3, 85, 4, 10, 19, 21, 77, 105, 99, 114, 111, 115, 111, 102, 116, 32, 67, 111, 114, 112, 111, 114, 97, 116, 105, 111, 110, 49, 34, 48, 32, 6, 3, 85, 4, 3, 19, 25, 77, 105, 99, 114, 111, 115, 111, 102, 116, 32, 84, 76, 83, 32, 82, 83, 65, 32, 82, 111, 111, 116, 32, 71, 50],
Signature: [53, 25, 202, 233, 168, 117, 225, 216, 202, 253, 37, 193, 59, 82, 225, 231, 158, 43, 148, 226, 244, 58, 210, 79, 218, 6, 34, 82, 26, 1, 24, 227, 162, 186, 183, 183, 28, 36, 50, 117, 13, 103, 68, 125, 157, 74, 130, 75, 250, 227, 68, 233, 70, 246, 0, 69, 9, 194, 82, 233, 103, 37, 151, 199, 183, 139, 90, 119, 124, 64, 30, 148, 227, 188, 140, 109, 56, 41, 188, 39, 125, 18, 18, 61, 203, 203, 228, 196, 70, 226, 165, 1, 192, 16, 209, 32, 250, 14, 3, 228, 67, 219, 38, 135, 37, 172, 200, 160, 179, 4, 106, 94, 240, 67, 63, 15, 123, 245, 206, 35, 195, 183, 201, 190, 251, 222, 155, 125, 147, 179, 135, 62, 223, 198, 219, 61, 201, 244, 172, 113, 0, 210, 164, 101, 129, 54, 89, 159, 28, 54, 229, 137, 209, 122, 153, 74, 123, 85, 75, 194, 213, 140, 208, 121, 1, 121, 149, 15, 57, 67, 139, 179, 10, 123, 146, 213, 90, 16, 24, 118, 233, 29, 13, 111, 50, 49, 244, 189, 209, 22, 241, 168, 163, 103, 242, 14, 154, 112, 12, 227, 150, 175, 3, 244, 109, 189, 20, 121, 106, 54, 195, 241, 182, 115, 187, 198, 11, 213, 38, 70, 136, 84, 69, 112, 170, 204, 31, 61, 139, 111, 227, 228, 23, 166, 129, 0, 251, 125, 131, 88, 128, 226, 242, 60, 59, 163, 24, 98, 19, 54, 103, 255, 79, 226, 229, 111, 106, 103, 230, 232, 90, 100, 58, 160, 69, 157, 63, 23, 12, 102, 135, 240, 129, 39, 52, 32, 176, 53, 36, 100, 40, 118, 67, 122, 116, 152, 226, 43, 128, 9, 10, 188, 176, 185, 7, 71, 41, 40, 46, 125, 224, 119, 65, 164, 140, 54, 120, 186, 51, 29, 86, 142, 217, 16, 53, 137, 89, 131, 0, 105, 206, 183, 120, 189, 237, 175, 130, 63, 206, 200, 137, 50, 152, 89, 8, 93, 225, 1, 150, 88, 196, 101, 25, 125, 38, 133, 157, 16, 150, 229, 149, 197, 17, 206, 130, 111, 23, 6, 113, 21, 15, 44, 191, 55, 50, 82, 21, 104, 180, 149, 142, 83, 93, 151, 237, 227, 219, 19, 62, 193, 49, 177, 121, 147, 115, 125, 114, 59, 27, 76, 79, 6, 73, 21, 132, 115, 243, 29, 90, 81, 44, 164, 212, 88, 224, 25, 191, 53, 72, 41, 96, 153, 171, 234, 123, 66, 3, 190, 156, 61, 25, 247, 71, 91, 11, 172, 130, 87, 170, 13, 189, 255, 34, 96, 48, 223, 44, 79, 87, 211, 37, 213, 218, 253, 142, 6, 81, 78, 5, 53, 120, 152, 56, 163, 39, 49, 111, 49, 154, 169, 170, 77, 172, 58, 68, 104, 96, 112, 104, 244, 83, 107, 205, 178, 45, 174, 38, 24, 135, 223, 140, 87, 181, 248, 82, 193, 86, 231, 192, 177, 77, 95, 10, 85, 107, 20, 43, 84, 251, 61, 169, 198, 189, 96, 7, 248, 176, 244, 75, 227, 238, 171],
SignatureAlgorithm: 5,
PublicKeyAlgorithm: 1,
PublicKey: <*rsa.PublicKey | 0x36a118ac3670>{
N: {
neg: false,
abs: [12623528037057212991, 15782976484946675180, 9131881476679932799, 16877507857597638923, 6948446969886104628, 16879364175701064122, 15349600278999781980, 9245340169127444553, 7968545109705329495, 6423711443832766155, 10594494922037898650, 17648935788986680361, 18122500389998752041, 7737602644086511993, 7258855044086458051, 5541026985182791629, 4439073283700877230, 10408130435767131132, 18089869943821258368, 11685763412349126810, 16394152652608075740, 12871239331426121127, 9800942613550528692, 18261315811323797426, 15374412250364208978, 3968124059102095807, 685806708061009512, 7838166407260346135, 16993058175899549691, 15384298262262378419, 3974146097701597185, 13694573362644585378, 13796492994165901152, 13528806942954960401, 3955259362538190321, 13074935763441801879, 2348393719768315797, 11342510827905045171, 130453718204474471, 17672562703128855053, 634844305723830904, 11538538025395239748, 5505257917472087582, 16743812431760803789, 13404336697386206403, 6770939201695035783, 3646446187118575543, 14898275314275200590, 3021489551454039149, 3930692534149328022, 11247669138812385627, 7850138742151524599, 1567520330174748017, 5626181319708709539, 16420623271324550298, 18240848139101269901, 2441811390160163564, 1395980562250961772, 3030591394459285111, 17666945104455592770, 11923213182329091847, 9136006465603430828, 13028421093649262211, 17297287522694218268],
},
E: 65537,
},
Version: 3,
SerialNumber: {
neg: false,
abs: [12560257885759602703, 4278338880794178, 3342336],
},
Issuer: {
Country: ["US"],
Organization: [
"Microsoft Corporation",
],
OrganizationalUnit: nil,
Locality: nil,
Province: nil,
StreetAddress: nil,
PostalCode: nil,
SerialNumber: "",
CommonName: "Microsoft TLS RSA Root G2",
Names: [
{
Type: [..., ..., ..., ...],
Value: <string>"US",
},
{
Type: [..., ..., ..., ...],
Value: <string>"Microsoft Corporation",
},
{
Type: [..., ..., ..., ...],
Value: <string>"Microsoft TLS RSA Root G2",
},
],
ExtraNames: nil,
},
Subject: {
Country: ["US"],
Organization: [
"Microsoft Corporation",
],
OrganizationalUnit: nil,
Locality: nil,
Province: nil,
StreetAddress: nil,
PostalCode: nil,
SerialNumber: "",
CommonName: "Microsoft TLS G2 RSA CA OCSP 10",
Names: [
{
Type: [..., ..., ..., ...],
Value: <string>"US",
},
{
Type: [..., ..., ..., ...],
Value: <string>"Microsoft Corporation",
},
{
Type: [..., ..., ..., ...],
Value: <string>"Microsoft TLS G2 RSA CA OCSP 10",
},
],
ExtraNames: nil,
},
NotBefore: {wall: 0, ext: 63889675383, loc: nil},
NotAfter: {wall: 0, ext: 64010808183, loc: nil},
KeyUsage: 97,
Extensions: [
{
Id: [2, 5, 29, 15],
Critical: true,
Value: [3, 2, 1, 134],
},
{
Id: [1, 3, 6, 1, 4, 1, 311, 21, 1],
Critical: false,
Value: [2, 1, 0],
},
{
Id: [2, 5, 29, 14],
Critical: false,
Value: [4, 20, 208, 76, 131, 209, 142, 113, 46, 61, 52, 177, 255, 155, 121, 44, 142, 116, 212, 226, 120, 6],
},
{
Id: [2, 5, 29, 32],
Critical: false,
Value: [48, 10, 48, 8, 6, 6, 103, 129, 12, 1, 2, 2],
},
{
Id: [2, 5, 29, 37],
Critical: false,
Value: [48, 10, 6, 8, 43, 6, 1, 5, 5, 7, 3, 1],
},
{
Id: [1, 3, 6, 1, 4, 1, 311, 20, 2],
Critical: false,
Value: [30, 10, 0, 83, 0, 117, 0, 98, 0, 67, 0, 65],
},
{
Id: [2, 5, 29, 19],
Critical: true,
Value: [48, 6, 1, 1, 255, 2, 1, 0],
},
{
Id: [2, 5, 29, 35],
Critical: false,
Value: [48, 22, 128, 20, 222, 145, 134, 72, 183, 161, 49, 89, 49, 241, 75, 95, 7, 169, 220, 136, 121, 218, 168, 118],
},
{
Id: [2, 5, 29, 31],
Critical: false,
Value: "0\x81\xa00\x81\x9d\xa0\x81\x9a\xa0\x81\x97\x86Ihttp://www.microsoft.com/pkiops/crl/Microsoft%20TLS%20RSA%20Root%20G2.crl\x86Jhttp://crl2.microsoft.com/pkiops/crl/Microsoft%20TLS%20RSA%20Root%20G2.crl",
},
{
Id: [1, 3, 6, 1, 5, 5, 7, 1, 1],
Critical: false,
Value: [48, 129, 255, 48, 99, 6, 8, 43, 6, 1, 5, 5, 7, 48, 2, 134, 87, 104, 116, 116, 112, 58, 47, 47, 119, 119, 119, 46, 109, 105, 99, 114, 111, 115, 111, 102, 116, 46, 99, 111, 109, 47, 112, 107, 105, 111, 112, 115, 47, 99, 101, 114, 116, 115, 47, 77, 105, 99, 114, 111, 115, 111, 102, 116, 37, 50, 48, 84, 76, 83, 37, 50, 48, 82, 83, 65, 37, 50, 48, 82, 111, 111, 116, 37, 50, 48, 71, 50, 37, 50, 48, 45, 37, 50, 48, 120, 115, 105, 103, 110, 46, 99, 114, 116, 48, 105, 6, 8, 43, 6, 1, 5, 5, 7, 48, 2, 134, 93, 104, 116, 116, 112, 58, 47, 47, 99, 97, 105, 115, 115, 117, 101, 114, 115, 46, 109, 105, 99, 114, 111, 115, 111, 102, 116, 46, 99, 111, 109, 47, 112, 107, 105, 111, 112, 115, 47, 99, 101, 114, 116, 115, 47, 77, 105, 99, 114, 111, 115, 111, 102, 116, 37, 50, 48, 84, 76, 83, 37, 50, 48, 82, 83, 65, 37, 50, 48, 82, 111, 111, 116, 37, 50, 48, 71, 50, 37, 50, 48, 45, 37, 50, 48, 120, 115, 105, 103, 110, 46, 99, 114, 116, 48, 45, 6, 8, 43, 6, 1, 5, 5, 7, 48, 1, 134, 33, 104, 116, 116, 112, 58, 47, 47, 111, 110, 101, 111, 99, 115, 112, 46, 109, 105, 99, 114, 111, 115, 111, 102, 116, 46, 99, 111, 109, 47, 111, 99, 115, 112],
},
],
ExtraExtensions: nil,
UnhandledCriticalExtensions: nil,
ExtKeyUsage: [1],
UnknownExtKeyUsage: nil,
BasicConstraintsValid: true,
IsCA: true,
MaxPathLen: 0,
MaxPathLenZero: true,
SubjectKeyId: [208, 76, 131, 209, 142, 113, 46, 61, 52, 177, 255, 155, 121, 44, 142, 116, 212, 226, 120, 6],
AuthorityKeyId: [222, 145, 134, 72, 183, 161, 49, 89, 49, 241, 75, 95, 7, 169, 220, 136, 121, 218, 168, 118],
OCSPServer: [
"http://oneocsp.microsoft.com/ocsp",
],
IssuingCertificateURL: [
"http://www.microsoft.com/pkiops/certs/Microsoft%20TLS%20RSA%20Root%20G2%20-%20xsign.crt",
"http://caissuers.microsoft.com/pkiops/certs/Microsoft%20TLS%20RSA%20Root%20G2%20-%20xsign.crt",
],
DNSNames: nil,
EmailAddresses: nil,
IPAddresses: nil,
URIs: nil,
PermittedDNSDomainsCritical: false,
PermittedDNSDomains: nil,
ExcludedDNSDomains: nil,
PermittedIPRanges: nil,
ExcludedIPRanges: nil,
PermittedEmailAddresses: nil,
ExcludedEmailAddresses: nil,
PermittedURIDomains: nil,
ExcludedURIDomains: nil,
CRLDistributionPoints: [
"http://www.microsoft.com/pkiops/crl/Microsoft%20TLS%20RSA%20Root%20G2.crl",
"http://crl2.microsoft.com/pkiops/crl/Microsoft%20TLS%20RSA%20Root%20G2.crl",
],
PolicyIdentifiers: [
[2, 23, 140, 1, 2, 2],
],
Policies: [
{
der: [103, 129, 12, 1, 2, 2],
},
],
InhibitAnyPolicy: 0,
InhibitAnyPolicyZero: false,
InhibitPolicyMapping: 0,
InhibitPolicyMappingZero: false,
RequireExplicitPolicy: 0,
RequireExplicitPolicyZero: false,
PolicyMappings: nil,
},
{
Raw: [48, 130, 5, 137, 48, 130, 4, 113, 160, 3, 2, 1, 2, 2, 16, 11, 12, 107, 44, 70, 105, 23, 176, 71, 115, 198, 71, 212, 175, 192, 200, 48, 13, 6, 9, 42, 134, 72, 134, 247, 13, 1, 1, 12, 5, 0, 48, 97, 49, 11, 48, 9, 6, 3, 85, 4, 6, 19, 2, 85, 83, 49, 21, 48, 19, 6, 3, 85, 4, 10, 19, 12, 68, 105, 103, 105, 67, 101, 114, 116, 32, 73, 110, 99, 49, 25, 48, 23, 6, 3, 85, 4, 11, 19, 16, 119, 119, 119, 46, 100, 105, 103, 105, 99, 101, 114, 116, 46, 99, 111, 109, 49, 32, 48, 30, 6, 3, 85, 4, 3, 19, 23, 68, 105, 103, 105, 67, 101, 114, 116, 32, 71, 108, 111, 98, 97, 108, 32, 82, 111, 111, 116, 32, 71, 50, 48, 30, 23, 13, 50, 53, 48, 53, 50, 49, 48, 48, 48, 48, 48, 48, 90, 23, 13, 50, 57, 48, 54, 49, 57, 50, 51, 53, 57, 53, 57, 90, 48, 81, 49, 11, 48, 9, 6, 3, 85, 4, 6, 19, 2, 85, 83, 49, 30, 48, 28, 6, 3, 85, 4, 10, 19, 21, 77, 105, 99, 114, 111, 115, 111, 102, 116, 32, 67, 111, 114, 112, 111, 114, 97, 116, 105, 111, 110, 49, 34, 48, 32, 6, 3, 85, 4, 3, 19, 25, 77, 105, 99, 114, 111, 115, 111, 102, 116, 32, 84, 76, 83, 32, 82, 83, 65, 32, 82, 111, 111, 116, 32, 71, 50, 48, 130, 2, 34, 48, 13, 6, 9, 42, 134, 72, 134, 247, 13, 1, 1, 1, 5, 0, 3, 130, 2, 15, 0, 48, 130, 2, 10, 2, 130, 2, 1, 0, 223, 234, 139, 159, 71, 225, 40, 16, 113, 175, 65, 214, 25, 219, 146, 87, 222, 107, 171, 230, 46, 222, 173, 58, 96, 131, 183, 19, 152, 172, 91, 185, 50, 11, 123, 235, 159, 114, 39, 107, 93, 64, 171, 190, 24, 6, 147, 178, 13, 210, 18, 237, 184, 116, 245, 127, 47, 234, 168, 23, 148, 246, 51, 22, 87, 138, 222, 32, 124, 33, 221, 183, 218, 172, 222, 78, 100, 176, 90, 205, 75, 117, 243, 239, 2, 100, 77, 237, 82, 218, 163, 143, 4, 232, 145, 248, 215, 251, 63, 25, 253, 78, 69, 57, 94, 232, 41, 195, 214, 242, 103, 175, 179, 0, 148, 193, 141, 225, 240, 74, 24, 152, 57, 250, 192, 9, 132, 51, 65, 5, 3, 5, 171, 244, 117, 225, 83, 165, 33, 146, 92, 68, 76, 230, 96, 87, 164, 144, 2, 151, 199, 213, 216, 147, 4, 50, 74, 237, 133, 81, 237, 214, 197, 186, 160, 63, 54, 138, 47, 111, 76, 57, 168, 4, 105, 34, 132, 59, 117, 20, 247, 217, 62, 180, 169, 208, 7, 65, 56, 129, 126, 70, 72, 29, 232, 25, 230, 112, 184, 160, 170, 24, 221, 114, 191, 126, 125, 181, 108, 78, 25, 112, 17, 132, 4, 19, 129, 212, 98, 253, 156, 107, 241, 28, 221, 255, 236, 167, 224, 230, 199, 141, 49, 75, 191, 242, 107, 51, 204, 209, 140, 182, 254, 177, 27, 163, 74, 168, 95, 14, 127, 49, 118, 56, 63, 7, 71, 108, 1, 168, 172, 165, 20, 218, 130, 176, 95, 31, 108, 188, 137, 26, 192, 18, 42, 203, 123, 151, 31, 40, 46, 49, 22, 138, 83, 76, 117, 102, 249, 125, 1, 214, 194, 96, 170, 101, 143, 49, 158, 114, 203, 204, 155, 45, 229, 68, 25, 147, 87, 171, 110, 142, 194, 217, 46, 101, 176, 114, 45, 231, 161, 195, 161, 126, 73, 29, 89, 103, 214, 9, 212, 24, 18, 109, 192, 149, 69, 120, 201, 241, 96, 106, 228, 246, 238, 6, 36, 59, 115, 74, 90, 17, 194, 62, 151, 97, 192, 120, 238, 239, 19, 167, 48, 226, 74, 56, 45, 238, 88, 23, 19, 32, 183, 54, 42, 223, 11, 50, 180, 194, 56, 177, 145, 221, 18, 176, 69, 182, 124, 192, 177, 29, 98, 161, 157, 221, 113, 18, 21, 150, 1, 157, 10, 9, 245, 177, 221, 179, 217, 224, 56, 66, 52, 246, 105, 182, 195, 175, 22, 115, 254, 212, 67, 248, 67, 228, 88, 249, 79, 179, 243, 236, 214, 61, 148, 58, 64, 56, 21, 171, 12, 134, 208, 100, 206, 26, 99, 232, 132, 148, 85, 9, 90, 182, 4, 182, 209, 62, 154, 21, 199, 100, 17, 128, 208, 27, 130, 14, 77, 255, 69, 158, 178, 101, 7, 37, 31, 219, 154, 200, 198, 236, 214, 91, 133, 80, 4, 159, 28, 203, 93, 32, 130, 103, 158, 121, 226, 223, 232, 242, 232, 111, 196, 32, 114, 48, 134, 229, 157, 117, 39, 53, 2, 3, 1, 0, 1, 163, 130, 1, 75, 48, 130, 1, 71, 48, 15, 6, 3, 85, 29, 19, 1, 1, 255, 4, 5, 48, 3, 1, 1, 255, 48, 29, 6, 3, 85, 29, 14, 4, 22, 4, 20, 222, 145, 134, 72, 183, 161, 49, 89, 49, 241, 75, 95, 7, 169, 220, 136, 121, 218, 168, 118, 48, 31, 6, 3, 85, 29, 35, 4, 24, 48, 22, 128, 20, 78, 34, 84, 32, 24, 149, 230, 227, 110, 230, 15, 250, 250, 185, 18, 237, 6, 23, 143, 57, 48, 14, 6, 3, 85, 29, 15, 1, 1, 255, 4, 4, 3, 2, 1, 134, 48, 19, 6, 3, 85, 29, 37, 4, 12, 48, 10, 6, 8, 43, 6, 1, 5, 5, 7, 3, 1, 48, 118, 6, 8, 43, 6, 1, 5, 5, 7, 1, 1, 4, 106, 48, 104, 48, 36, 6, 8, 43, 6, 1, 5, 5, 7, 48, 1, 134, 24, 104, 116, 116, 112, 58, 47, 47, 111, 99, 115, 112, 46, 100, 105, 103, 105, 99, 101, 114, 116, 46, 99, 111, 109, 48, 64, 6, 8, 43, 6, 1, 5, 5, 7, 48, 2, 134, 52, 104, 116, 116, 112, 58, 47, 47, 99, 97, 99, 101, 114, 116, 115, 46, 100, 105, 103, 105, 99, 101, 114, 116, 46, 99, 111, 109, 47, 68, 105, 103, 105, 67, 101, 114, 116, 71, 108, 111, 98, 97, 108, 82, 111, 111, 116, 71, 50, 46, 99, 114, 116, 48, 66, 6, 3, 85, 29, 31, 4, 59, 48, 57, 48, 55, 160, 53, 160, 51, 134, 49, 104, 116, 116, 112, 58, 47, 47, 99, 114, 108, 51, 46, 100, 105, 103, 105, 99, 101, 114, 116, 46, 99, 111, 109, 47, 68, 105, 103, 105, 67, 101, 114, 116, 71, 108, 111, 98, 97, 108, 82, 111, 111, 116, 71, 50, 46, 99, 114, 108, 48, 19, 6, 3, 85, 29, 32, 4, 12, 48, 10, 48, 8, 6, 6, 103, 129, 12, 1, 2, 2, 48, 13, 6, 9, 42, 134, 72, 134, 247, 13, 1, 1, 12, 5, 0, 3, 130, 1, 1, 0, 11, 188, 180, 43, 55, 116, 197, 75, 167, 54, 2, 54, 198, 175, 225, 19, 204, 138, 154, 151, 27, 252, 227, 68, 140, 238, 84, 0, 229, 228, 65, 26, 70, 240, 11, 118, 52, 255, 153, 85, 98, 53, 170, 173, 138, 156, 12, 90, 98, 204, 67, 192, 203, 234, 74, 29, 189, 107, 29, 175, 177, 25, 37, 102, 156, 247, 121, 110, 91, 37, 128, 36, 129, 133, 199, 121, 45, 53, 95, 66, 9, 120, 237, 159, 226, 101, 213, 196, 32, 80, 130, 16, 143, 106, 58, 237, 24, 142, 72, 137, 171, 166, 9, 248, 225, 128, 252, 6, 117, 22, 39, 37, 90, 143, 184, 58, 112, 108, 24, 71, 240, 99, 244, 83, 14, 217, 21, 53, 32, 243, 145, 61, 19, 1, 160, 226, 250, 36, 8, 69, 86, 119, 209, 103, 169, 49, 66, 208, 87, 255, 26, 38, 190, 197, 67, 216, 87, 169, 99, 255, 175, 243, 58, 216, 246, 123, 75, 16, 162, 151, 161, 207, 102, 190, 103, 103, 9, 249, 181, 76, 231, 171, 25, 43, 191, 34, 61, 125, 222, 198, 247, 239, 99, 56, 44, 221, 86, 196, 249, 9, 82, 218, 223, 227, 136, 111, 213, 109, 107, 247, 8, 65, 47, 142, 62, 134, 49, 159, 241, 124, 74, 245, 214, 97, 164, 70, 70, 40, 239, 87, 186, 176, 111, 146, 216, 210, 119, 37, 237, 123, 166, 41, 206, 59, 114, 121, 17, 212, 224, 167, 84, 109, 53, 43, 164, 122],
RawTBSCertificate: [48, 130, 4, 113, 160, 3, 2, 1, 2, 2, 16, 11, 12, 107, 44, 70, 105, 23, 176, 71, 115, 198, 71, 212, 175, 192, 200, 48, 13, 6, 9, 42, 134, 72, 134, 247, 13, 1, 1, 12, 5, 0, 48, 97, 49, 11, 48, 9, 6, 3, 85, 4, 6, 19, 2, 85, 83, 49, 21, 48, 19, 6, 3, 85, 4, 10, 19, 12, 68, 105, 103, 105, 67, 101, 114, 116, 32, 73, 110, 99, 49, 25, 48, 23, 6, 3, 85, 4, 11, 19, 16, 119, 119, 119, 46, 100, 105, 103, 105, 99, 101, 114, 116, 46, 99, 111, 109, 49, 32, 48, 30, 6, 3, 85, 4, 3, 19, 23, 68, 105, 103, 105, 67, 101, 114, 116, 32, 71, 108, 111, 98, 97, 108, 32, 82, 111, 111, 116, 32, 71, 50, 48, 30, 23, 13, 50, 53, 48, 53, 50, 49, 48, 48, 48, 48, 48, 48, 90, 23, 13, 50, 57, 48, 54, 49, 57, 50, 51, 53, 57, 53, 57, 90, 48, 81, 49, 11, 48, 9, 6, 3, 85, 4, 6, 19, 2, 85, 83, 49, 30, 48, 28, 6, 3, 85, 4, 10, 19, 21, 77, 105, 99, 114, 111, 115, 111, 102, 116, 32, 67, 111, 114, 112, 111, 114, 97, 116, 105, 111, 110, 49, 34, 48, 32, 6, 3, 85, 4, 3, 19, 25, 77, 105, 99, 114, 111, 115, 111, 102, 116, 32, 84, 76, 83, 32, 82, 83, 65, 32, 82, 111, 111, 116, 32, 71, 50, 48, 130, 2, 34, 48, 13, 6, 9, 42, 134, 72, 134, 247, 13, 1, 1, 1, 5, 0, 3, 130, 2, 15, 0, 48, 130, 2, 10, 2, 130, 2, 1, 0, 223, 234, 139, 159, 71, 225, 40, 16, 113, 175, 65, 214, 25, 219, 146, 87, 222, 107, 171, 230, 46, 222, 173, 58, 96, 131, 183, 19, 152, 172, 91, 185, 50, 11, 123, 235, 159, 114, 39, 107, 93, 64, 171, 190, 24, 6, 147, 178, 13, 210, 18, 237, 184, 116, 245, 127, 47, 234, 168, 23, 148, 246, 51, 22, 87, 138, 222, 32, 124, 33, 221, 183, 218, 172, 222, 78, 100, 176, 90, 205, 75, 117, 243, 239, 2, 100, 77, 237, 82, 218, 163, 143, 4, 232, 145, 248, 215, 251, 63, 25, 253, 78, 69, 57, 94, 232, 41, 195, 214, 242, 103, 175, 179, 0, 148, 193, 141, 225, 240, 74, 24, 152, 57, 250, 192, 9, 132, 51, 65, 5, 3, 5, 171, 244, 117, 225, 83, 165, 33, 146, 92, 68, 76, 230, 96, 87, 164, 144, 2, 151, 199, 213, 216, 147, 4, 50, 74, 237, 133, 81, 237, 214, 197, 186, 160, 63, 54, 138, 47, 111, 76, 57, 168, 4, 105, 34, 132, 59, 117, 20, 247, 217, 62, 180, 169, 208, 7, 65, 56, 129, 126, 70, 72, 29, 232, 25, 230, 112, 184, 160, 170, 24, 221, 114, 191, 126, 125, 181, 108, 78, 25, 112, 17, 132, 4, 19, 129, 212, 98, 253, 156, 107, 241, 28, 221, 255, 236, 167, 224, 230, 199, 141, 49, 75, 191, 242, 107, 51, 204, 209, 140, 182, 254, 177, 27, 163, 74, 168, 95, 14, 127, 49, 118, 56, 63, 7, 71, 108, 1, 168, 172, 165, 20, 218, 130, 176, 95, 31, 108, 188, 137, 26, 192, 18, 42, 203, 123, 151, 31, 40, 46, 49, 22, 138, 83, 76, 117, 102, 249, 125, 1, 214, 194, 96, 170, 101, 143, 49, 158, 114, 203, 204, 155, 45, 229, 68, 25, 147, 87, 171, 110, 142, 194, 217, 46, 101, 176, 114, 45, 231, 161, 195, 161, 126, 73, 29, 89, 103, 214, 9, 212, 24, 18, 109, 192, 149, 69, 120, 201, 241, 96, 106, 228, 246, 238, 6, 36, 59, 115, 74, 90, 17, 194, 62, 151, 97, 192, 120, 238, 239, 19, 167, 48, 226, 74, 56, 45, 238, 88, 23, 19, 32, 183, 54, 42, 223, 11, 50, 180, 194, 56, 177, 145, 221, 18, 176, 69, 182, 124, 192, 177, 29, 98, 161, 157, 221, 113, 18, 21, 150, 1, 157, 10, 9, 245, 177, 221, 179, 217, 224, 56, 66, 52, 246, 105, 182, 195, 175, 22, 115, 254, 212, 67, 248, 67, 228, 88, 249, 79, 179, 243, 236, 214, 61, 148, 58, 64, 56, 21, 171, 12, 134, 208, 100, 206, 26, 99, 232, 132, 148, 85, 9, 90, 182, 4, 182, 209, 62, 154, 21, 199, 100, 17, 128, 208, 27, 130, 14, 77, 255, 69, 158, 178, 101, 7, 37, 31, 219, 154, 200, 198, 236, 214, 91, 133, 80, 4, 159, 28, 203, 93, 32, 130, 103, 158, 121, 226, 223, 232, 242, 232, 111, 196, 32, 114, 48, 134, 229, 157, 117, 39, 53, 2, 3, 1, 0, 1, 163, 130, 1, 75, 48, 130, 1, 71, 48, 15, 6, 3, 85, 29, 19, 1, 1, 255, 4, 5, 48, 3, 1, 1, 255, 48, 29, 6, 3, 85, 29, 14, 4, 22, 4, 20, 222, 145, 134, 72, 183, 161, 49, 89, 49, 241, 75, 95, 7, 169, 220, 136, 121, 218, 168, 118, 48, 31, 6, 3, 85, 29, 35, 4, 24, 48, 22, 128, 20, 78, 34, 84, 32, 24, 149, 230, 227, 110, 230, 15, 250, 250, 185, 18, 237, 6, 23, 143, 57, 48, 14, 6, 3, 85, 29, 15, 1, 1, 255, 4, 4, 3, 2, 1, 134, 48, 19, 6, 3, 85, 29, 37, 4, 12, 48, 10, 6, 8, 43, 6, 1, 5, 5, 7, 3, 1, 48, 118, 6, 8, 43, 6, 1, 5, 5, 7, 1, 1, 4, 106, 48, 104, 48, 36, 6, 8, 43, 6, 1, 5, 5, 7, 48, 1, 134, 24, 104, 116, 116, 112, 58, 47, 47, 111, 99, 115, 112, 46, 100, 105, 103, 105, 99, 101, 114, 116, 46, 99, 111, 109, 48, 64, 6, 8, 43, 6, 1, 5, 5, 7, 48, 2, 134, 52, 104, 116, 116, 112, 58, 47, 47, 99, 97, 99, 101, 114, 116, 115, 46, 100, 105, 103, 105, 99, 101, 114, 116, 46, 99, 111, 109, 47, 68, 105, 103, 105, 67, 101, 114, 116, 71, 108, 111, 98, 97, 108, 82, 111, 111, 116, 71, 50, 46, 99, 114, 116, 48, 66, 6, 3, 85, 29, 31, 4, 59, 48, 57, 48, 55, 160, 53, 160, 51, 134, 49, 104, 116, 116, 112, 58, 47, 47, 99, 114, 108, 51, 46, 100, 105, 103, 105, 99, 101, 114, 116, 46, 99, 111, 109, 47, 68, 105, 103, 105, 67, 101, 114, 116, 71, 108, 111, 98, 97, 108, 82, 111, 111, 116, 71, 50, 46, 99, 114, 108, 48, 19, 6, 3, 85, 29, 32, 4, 12, 48, 10, 48, 8, 6, 6, 103, 129, 12, 1, 2, 2],
RawSubjectPublicKeyInfo: [48, 130, 2, 34, 48, 13, 6, 9, 42, 134, 72, 134, 247, 13, 1, 1, 1, 5, 0, 3, 130, 2, 15, 0, 48, 130, 2, 10, 2, 130, 2, 1, 0, 223, 234, 139, 159, 71, 225, 40, 16, 113, 175, 65, 214, 25, 219, 146, 87, 222, 107, 171, 230, 46, 222, 173, 58, 96, 131, 183, 19, 152, 172, 91, 185, 50, 11, 123, 235, 159, 114, 39, 107, 93, 64, 171, 190, 24, 6, 147, 178, 13, 210, 18, 237, 184, 116, 245, 127, 47, 234, 168, 23, 148, 246, 51, 22, 87, 138, 222, 32, 124, 33, 221, 183, 218, 172, 222, 78, 100, 176, 90, 205, 75, 117, 243, 239, 2, 100, 77, 237, 82, 218, 163, 143, 4, 232, 145, 248, 215, 251, 63, 25, 253, 78, 69, 57, 94, 232, 41, 195, 214, 242, 103, 175, 179, 0, 148, 193, 141, 225, 240, 74, 24, 152, 57, 250, 192, 9, 132, 51, 65, 5, 3, 5, 171, 244, 117, 225, 83, 165, 33, 146, 92, 68, 76, 230, 96, 87, 164, 144, 2, 151, 199, 213, 216, 147, 4, 50, 74, 237, 133, 81, 237, 214, 197, 186, 160, 63, 54, 138, 47, 111, 76, 57, 168, 4, 105, 34, 132, 59, 117, 20, 247, 217, 62, 180, 169, 208, 7, 65, 56, 129, 126, 70, 72, 29, 232, 25, 230, 112, 184, 160, 170, 24, 221, 114, 191, 126, 125, 181, 108, 78, 25, 112, 17, 132, 4, 19, 129, 212, 98, 253, 156, 107, 241, 28, 221, 255, 236, 167, 224, 230, 199, 141, 49, 75, 191, 242, 107, 51, 204, 209, 140, 182, 254, 177, 27, 163, 74, 168, 95, 14, 127, 49, 118, 56, 63, 7, 71, 108, 1, 168, 172, 165, 20, 218, 130, 176, 95, 31, 108, 188, 137, 26, 192, 18, 42, 203, 123, 151, 31, 40, 46, 49, 22, 138, 83, 76, 117, 102, 249, 125, 1, 214, 194, 96, 170, 101, 143, 49, 158, 114, 203, 204, 155, 45, 229, 68, 25, 147, 87, 171, 110, 142, 194, 217, 46, 101, 176, 114, 45, 231, 161, 195, 161, 126, 73, 29, 89, 103, 214, 9, 212, 24, 18, 109, 192, 149, 69, 120, 201, 241, 96, 106, 228, 246, 238, 6, 36, 59, 115, 74, 90, 17, 194, 62, 151, 97, 192, 120, 238, 239, 19, 167, 48, 226, 74, 56, 45, 238, 88, 23, 19, 32, 183, 54, 42, 223, 11, 50, 180, 194, 56, 177, 145, 221, 18, 176, 69, 182, 124, 192, 177, 29, 98, 161, 157, 221, 113, 18, 21, 150, 1, 157, 10, 9, 245, 177, 221, 179, 217, 224, 56, 66, 52, 246, 105, 182, 195, 175, 22, 115, 254, 212, 67, 248, 67, 228, 88, 249, 79, 179, 243, 236, 214, 61, 148, 58, 64, 56, 21, 171, 12, 134, 208, 100, 206, 26, 99, 232, 132, 148, 85, 9, 90, 182, 4, 182, 209, 62, 154, 21, 199, 100, 17, 128, 208, 27, 130, 14, 77, 255, 69, 158, 178, 101, 7, 37, 31, 219, 154, 200, 198, 236, 214, 91, 133, 80, 4, 159, 28, 203, 93, 32, 130, 103, 158, 121, 226, 223, 232, 242, 232, 111, 196, 32, 114, 48, 134, 229, 157, 117, 39, 53, 2, 3, 1, 0, 1],
RawSubject: [48, 81, 49, 11, 48, 9, 6, 3, 85, 4, 6, 19, 2, 85, 83, 49, 30, 48, 28, 6, 3, 85, 4, 10, 19, 21, 77, 105, 99, 114, 111, 115, 111, 102, 116, 32, 67, 111, 114, 112, 111, 114, 97, 116, 105, 111, 110, 49, 34, 48, 32, 6, 3, 85, 4, 3, 19, 25, 77, 105, 99, 114, 111, 115, 111, 102, 116, 32, 84, 76, 83, 32, 82, 83, 65, 32, 82, 111, 111, 116, 32, 71, 50],
RawIssuer: [48, 97, 49, 11, 48, 9, 6, 3, 85, 4, 6, 19, 2, 85, 83, 49, 21, 48, 19, 6, 3, 85, 4, 10, 19, 12, 68, 105, 103, 105, 67, 101, 114, 116, 32, 73, 110, 99, 49, 25, 48, 23, 6, 3, 85, 4, 11, 19, 16, 119, 119, 119, 46, 100, 105, 103, 105, 99, 101, 114, 116, 46, 99, 111, 109, 49, 32, 48, 30, 6, 3, 85, 4, 3, 19, 23, 68, 105, 103, 105, 67, 101, 114, 116, 32, 71, 108, 111, 98, 97, 108, 32, 82, 111, 111, 116, 32, 71, 50],
Signature: [11, 188, 180, 43, 55, 116, 197, 75, 167, 54, 2, 54, 198, 175, 225, 19, 204, 138, 154, 151, 27, 252, 227, 68, 140, 238, 84, 0, 229, 228, 65, 26, 70, 240, 11, 118, 52, 255, 153, 85, 98, 53, 170, 173, 138, 156, 12, 90, 98, 204, 67, 192, 203, 234, 74, 29, 189, 107, 29, 175, 177, 25, 37, 102, 156, 247, 121, 110, 91, 37, 128, 36, 129, 133, 199, 121, 45, 53, 95, 66, 9, 120, 237, 159, 226, 101, 213, 196, 32, 80, 130, 16, 143, 106, 58, 237, 24, 142, 72, 137, 171, 166, 9, 248, 225, 128, 252, 6, 117, 22, 39, 37, 90, 143, 184, 58, 112, 108, 24, 71, 240, 99, 244, 83, 14, 217, 21, 53, 32, 243, 145, 61, 19, 1, 160, 226, 250, 36, 8, 69, 86, 119, 209, 103, 169, 49, 66, 208, 87, 255, 26, 38, 190, 197, 67, 216, 87, 169, 99, 255, 175, 243, 58, 216, 246, 123, 75, 16, 162, 151, 161, 207, 102, 190, 103, 103, 9, 249, 181, 76, 231, 171, 25, 43, 191, 34, 61, 125, 222, 198, 247, 239, 99, 56, 44, 221, 86, 196, 249, 9, 82, 218, 223, 227, 136, 111, 213, 109, 107, 247, 8, 65, 47, 142, 62, 134, 49, 159, 241, 124, 74, 245, 214, 97, 164, 70, 70, 40, 239, 87, 186, 176, 111, 146, 216, 210, 119, 37, 237, 123, 166, 41, 206, 59, 114, 121, 17, 212, 224, 167, 84, 109, 53, 43, 164, 122],
SignatureAlgorithm: 5,
PublicKeyAlgorithm: 1,
PublicKey: <*rsa.PublicKey | 0x36a118ac37b0>{
N: {
neg: false,
abs: [8228224839953229621, 16348041302357558304, 2074854447542410873, 14334067402580427935, 12854688567428618952, 14995722382209926558, 15077658020813214080, 9553354208224609462, 902637906707506152, 15437658075602359723, 4892132923038495724, 14100513645772686328, 15699610158021110198, 116260203274558899, 7107135261377697174, 1346652938663145757, 806906029520556509, 6347563231035665119, 1416154356297248238, 13996791138010853103, 17151436068643035665, 5005973024364225782, 15423091598760460437, 11656337824329587047, 14040304327591079399, 16520357453723561614, 10318202437165161261, 17977527309428632165, 3328466382962324838, 13840171658303184680, 9417131409404627226, 5146490298140267738, 6849551933690625799, 10139571647697603240, 3552143779013315793, 15996764607220205453, 9355211167345930524, 7804203173400544275, 12256789870797553077, 5196564742993393824, 12236288164267916870, 9528338169550421684, 3418034452713204002, 17138102837348087434, 15605821747820856657, 6942198289040656341, 6027260588802198758, 4685154310243513825, 1772230202496746547, 12898472891819094090, 6838761955263670191, 15563102318165378361, 5970264090691015160, 5437520332982603245, 15757213624540617421, 6308098459188714935, 3452756883569783574, 995879279813522815, 6719559476972000178, 3606112178562803563, 6954603544383871929, 16027092704045215034, 8191838635022848599, 16134862131230222352],
},
E: 65537,
},
Version: 3,
SerialNumber: {
neg: false,
abs: [5148676810827874504, 796129072041760688],
},
Issuer: {
Country: ["US"],
Organization: ["DigiCert Inc"],
OrganizationalUnit: ["www.digicert.com"],
Locality: nil,
Province: nil,
StreetAddress: nil,
PostalCode: nil,
SerialNumber: "",
CommonName: "DigiCert Global Root G2",
Names: [
{
Type: [..., ..., ..., ...],
Value: <string>"US",
},
{
Type: [..., ..., ..., ...],
Value: <string>"DigiCert Inc",
},
{
Type: [..., ..., ..., ...],
Value: <string>"www.digicert.com",
},
{
Type: [..., ..., ..., ...],
Value: <string>"DigiCert Global Root G2",
},
],
ExtraNames: nil,
},
Subject: {
Country: ["US"],
Organization: [
"Microsoft Corporation",
],
OrganizationalUnit: nil,
Locality: nil,
Province: nil,
StreetAddress: nil,
PostalCode: nil,
SerialNumber: "",
CommonName: "Microsoft TLS RSA Root G2",
Names: [
{
Type: [..., ..., ..., ...],
Value: <string>"US",
},
{
Type: [..., ..., ..., ...],
Value: <string>"Microsoft Corporation",
},
{
Type: [..., ..., ..., ...],
Value: <string>"Microsoft TLS RSA Root G2",
},
],
ExtraNames: nil,
},
NotBefore: {wall: 0, ext: 63883382400, loc: nil},
NotAfter: {wall: 0, ext: 64012204799, loc: nil},
KeyUsage: 97,
Extensions: [
{
Id: [2, 5, 29, 19],
Critical: true,
Value: [48, 3, 1, 1, 255],
},
{
Id: [2, 5, 29, 14],
Critical: false,
Value: [4, 20, 222, 145, 134, 72, 183, 161, 49, 89, 49, 241, 75, 95, 7, 169, 220, 136, 121, 218, 168, 118],
},
{
Id: [2, 5, 29, 35],
Critical: false,
Value: [48, 22, 128, 20, 78, 34, 84, 32, 24, 149, 230, 227, 110, 230, 15, 250, 250, 185, 18, 237, 6, 23, 143, 57],
},
{
Id: [2, 5, 29, 15],
Critical: true,
Value: [3, 2, 1, 134],
},
{
Id: [2, 5, 29, 37],
Critical: false,
Value: [48, 10, 6, 8, 43, 6, 1, 5, 5, 7, 3, 1],
},
{
Id: [1, 3, 6, 1, 5, 5, 7, 1, 1],
Critical: false,
Value: [48, 104, 48, 36, 6, 8, 43, 6, 1, 5, 5, 7, 48, 1, 134, 24, 104, 116, 116, 112, 58, 47, 47, 111, 99, 115, 112, 46, 100, 105, 103, 105, 99, 101, 114, 116, 46, 99, 111, 109, 48, 64, 6, 8, 43, 6, 1, 5, 5, 7, 48, 2, 134, 52, 104, 116, 116, 112, 58, 47, 47, 99, 97, 99, 101, 114, 116, 115, 46, 100, 105, 103, 105, 99, 101, 114, 116, 46, 99, 111, 109, 47, 68, 105, 103, 105, 67, 101, 114, 116, 71, 108, 111, 98, 97, 108, 82, 111, 111, 116, 71, 50, 46, 99, 114, 116],
},
{
Id: [2, 5, 29, 31],
Critical: false,
Value: "0907\xa05\xa03\x861http://crl3.digicert.com/DigiCertGlobalRootG2.crl",
},
{
Id: [2, 5, 29, 32],
Critical: false,
Value: [48, 10, 48, 8, 6, 6, 103, 129, 12, 1, 2, 2],
},
],
ExtraExtensions: nil,
UnhandledCriticalExtensions: nil,
ExtKeyUsage: [1],
UnknownExtKeyUsage: nil,
BasicConstraintsValid: true,
IsCA: true,
MaxPathLen: -1,
MaxPathLenZero: false,
SubjectKeyId: [222, 145, 134, 72, 183, 161, 49, 89, 49, 241, 75, 95, 7, 169, 220, 136, 121, 218, 168, 118],
AuthorityKeyId: [78, 34, 84, 32, 24, 149, 230, 227, 110, 230, 15, 250, 250, 185, 18, 237, 6, 23, 143, 57],
OCSPServer: [
"http://ocsp.digicert.com",
],
IssuingCertificateURL: [
"http://cacerts.digicert.com/DigiCertGlobalRootG2.crt",
],
DNSNames: nil,
EmailAddresses: nil,
IPAddresses: nil,
URIs: nil,
PermittedDNSDomainsCritical: false,
PermittedDNSDomains: nil,
ExcludedDNSDomains: nil,
PermittedIPRanges: nil,
ExcludedIPRanges: nil,
PermittedEmailAddresses: nil,
ExcludedEmailAddresses: nil,
PermittedURIDomains: nil,
ExcludedURIDomains: nil,
CRLDistributionPoints: [
"http://crl3.digicert.com/DigiCertGlobalRootG2.crl",
],
PolicyIdentifiers: [
[2, 23, 140, 1, 2, 2],
],
Policies: [
{
der: [103, 129, 12, 1, 2, 2],
},
],
InhibitAnyPolicy: 0,
InhibitAnyPolicyZero: false,
InhibitPolicyMapping: 0,
InhibitPolicyMappingZero: false,
RequireExplicitPolicy: 0,
RequireExplicitPolicyZero: false,
PolicyMappings: nil,
},
{
Raw: [48, 130, 3, 142, 48, 130, 2, 118, 160, 3, 2, 1, 2, 2, 16, 3, 58, 241, 230, 167, 17, 169, 160, 187, 40, 100, 177, 29, 9, 250, 229, 48, 13, 6, 9, 42, 134, 72, 134, 247, 13, 1, 1, 11, 5, 0, 48, 97, 49, 11, 48, 9, 6, 3, 85, 4, 6, 19, 2, 85, 83, 49, 21, 48, 19, 6, 3, 85, 4, 10, 19, 12, 68, 105, 103, 105, 67, 101, 114, 116, 32, 73, 110, 99, 49, 25, 48, 23, 6, 3, 85, 4, 11, 19, 16, 119, 119, 119, 46, 100, 105, 103, 105, 99, 101, 114, 116, 46, 99, 111, 109, 49, 32, 48, 30, 6, 3, 85, 4, 3, 19, 23, 68, 105, 103, 105, 67, 101, 114, 116, 32, 71, 108, 111, 98, 97, 108, 32, 82, 111, 111, 116, 32, 71, 50, 48, 30, 23, 13, 49, 51, 48, 56, 48, 49, 49, 50, 48, 48, 48, 48, 90, 23, 13, 51, 56, 48, 49, 49, 53, 49, 50, 48, 48, 48, 48, 90, 48, 97, 49, 11, 48, 9, 6, 3, 85, 4, 6, 19, 2, 85, 83, 49, 21, 48, 19, 6, 3, 85, 4, 10, 19, 12, 68, 105, 103, 105, 67, 101, 114, 116, 32, 73, 110, 99, 49, 25, 48, 23, 6, 3, 85, 4, 11, 19, 16, 119, 119, 119, 46, 100, 105, 103, 105, 99, 101, 114, 116, 46, 99, 111, 109, 49, 32, 48, 30, 6, 3, 85, 4, 3, 19, 23, 68, 105, 103, 105, 67, 101, 114, 116, 32, 71, 108, 111, 98, 97, 108, 32, 82, 111, 111, 116, 32, 71, 50, 48, 130, 1, 34, 48, 13, 6, 9, 42, 134, 72, 134, 247, 13, 1, 1, 1, 5, 0, 3, 130, 1, 15, 0, 48, 130, 1, 10, 2, 130, 1, 1, 0, 187, 55, 205, 52, 220, 123, 107, 201, 178, 104, 144, 173, 74, 117, 255, 70, 186, 33, 10, 8, 141, 245, 25, 84, 201, 251, 136, 219, 243, 174, 242, 58, 137, 145, 60, 122, 230, 171, 6, 26, 107, 207, 172, 45, 232, 94, 9, 36, 68, 186, 98, 154, 126, 214, 163, 168, 126, 224, 84, 117, 32, 5, 172, 80, 183, 156, 99, 26, 108, 48, 220, 218, 31, 25, 177, 215, 30, 222, 253, 215, 224, 203, 148, 131, 55, 174, 236, 31, 67, 78, 221, 123, 44, 210, 189, 46, 165, 47, 228, 169, 184, 173, 58, 212, 153, 164, 182, 37, 233, 155, 107, 0, 96, 146, 96, 255, 79, 33, 73, 24, 247, 103, 144, 171, 97, 6, 156, 143, 242, 186, 233, 180, 233, 146, 50, 107, 181, 243, 87, 232, 93, 27, 205, 140, 29, 171, 149, 4, 149, 73, 243, 53, 45, 150, 227, 73, 109, 221, 119, 227, 251, 73, 75, 180, 172, 85, 7, 169, 143, 149, 179, 180, 35, 187, 76, 109, 69, 240, 246, 169, 178, 149, 48, 180, 253, 76, 85, 140, 39, 74, 87, 20, 124, 130, 157, 205, 115, 146, 211, 22, 74, 6, 12, 140, 80, 209, 143, 30, 9, 190, 23, 161, 230, 33, 202, 253, 131, 229, 16, 188, 131, 165, 10, 196, 103, 40, 246, 115, 20, 20, 61, 70, 118, 195, 135, 20, 137, 33, 52, 77, 175, 15, 69, 12, 166, 73, 161, 186, 187, 156, 197, 177, 51, 131, 41, 133, 2, 3, 1, 0, 1, 163, 66, 48, 64, 48, 15, 6, 3, 85, 29, 19, 1, 1, 255, 4, 5, 48, 3, 1, 1, 255, 48, 14, 6, 3, 85, 29, 15, 1, 1, 255, 4, 4, 3, 2, 1, 134, 48, 29, 6, 3, 85, 29, 14, 4, 22, 4, 20, 78, 34, 84, 32, 24, 149, 230, 227, 110, 230, 15, 250, 250, 185, 18, 237, 6, 23, 143, 57, 48, 13, 6, 9, 42, 134, 72, 134, 247, 13, 1, 1, 11, 5, 0, 3, 130, 1, 1, 0, 96, 103, 40, 148, 111, 14, 72, 99, 235, 49, 221, 234, 103, 24, 213, 137, 125, 60, 197, 139, 74, 127, 233, 190, 219, 43, 23, 223, 176, 95, 115, 119, 42, 50, 19, 57, 129, 103, 66, 132, 35, 242, 69, 103, 53, 236, 136, 191, 248, 143, 176, 97, 12, 52, 164, 174, 32, 76, 132, 198, 219, 248, 53, 225, 118, 217, 223, 166, 66, 187, 199, 68, 8, 134, 127, 54, 116, 36, 90, 218, 108, 13, 20, 89, 53, 189, 242, 73, 221, 182, 31, 201, 179, 13, 71, 42, 61, 153, 47, 187, 92, 187, 181, 212, 32, 225, 153, 95, 83, 70, 21, 219, 104, 155, 240, 243, 48, 213, 62, 49, 226, 141, 132, 158, 227, 138, 218, 218, 150, 62, 53, 19, 165, 95, 240, 249, 112, 80, 112, 71, 65, 17, 87, 25, 78, 192, 143, 174, 6, 196, 149, 19, 23, 47, 27, 37, 159, 117, 242, 177, 142, 153, 161, 111, 19, 177, 65, 113, 254, 136, 42, 200, 79, 16, 32, 85, 215, 243, 20, 69, 229, 224, 68, 244, 234, 135, 149, 50, 147, 14, 254, 83, 70, 250, 44, 157, 255, 139, 34, 185, 75, 217, 9, 69, 164, 222, 164, 184, 154, 88, 221, 27, 125, 82, 159, 142, 89, 67, 136, 129, 164, 158, 38, 213, 111, 173, 221, 13, 198, 55, 125, 237, 3, 146, 27, 229, 119, 95, 118, 238, 60, 141, 196, 93, 86, 91, 162, 217, 102, 110, 179, 53, 55, 229, 50, 182],
RawTBSCertificate: [48, 130, 2, 118, 160, 3, 2, 1, 2, 2, 16, 3, 58, 241, 230, 167, 17, 169, 160, 187, 40, 100, 177, 29, 9, 250, 229, 48, 13, 6, 9, 42, 134, 72, 134, 247, 13, 1, 1, 11, 5, 0, 48, 97, 49, 11, 48, 9, 6, 3, 85, 4, 6, 19, 2, 85, 83, 49, 21, 48, 19, 6, 3, 85, 4, 10, 19, 12, 68, 105, 103, 105, 67, 101, 114, 116, 32, 73, 110, 99, 49, 25, 48, 23, 6, 3, 85, 4, 11, 19, 16, 119, 119, 119, 46, 100, 105, 103, 105, 99, 101, 114, 116, 46, 99, 111, 109, 49, 32, 48, 30, 6, 3, 85, 4, 3, 19, 23, 68, 105, 103, 105, 67, 101, 114, 116, 32, 71, 108, 111, 98, 97, 108, 32, 82, 111, 111, 116, 32, 71, 50, 48, 30, 23, 13, 49, 51, 48, 56, 48, 49, 49, 50, 48, 48, 48, 48, 90, 23, 13, 51, 56, 48, 49, 49, 53, 49, 50, 48, 48, 48, 48, 90, 48, 97, 49, 11, 48, 9, 6, 3, 85, 4, 6, 19, 2, 85, 83, 49, 21, 48, 19, 6, 3, 85, 4, 10, 19, 12, 68, 105, 103, 105, 67, 101, 114, 116, 32, 73, 110, 99, 49, 25, 48, 23, 6, 3, 85, 4, 11, 19, 16, 119, 119, 119, 46, 100, 105, 103, 105, 99, 101, 114, 116, 46, 99, 111, 109, 49, 32, 48, 30, 6, 3, 85, 4, 3, 19, 23, 68, 105, 103, 105, 67, 101, 114, 116, 32, 71, 108, 111, 98, 97, 108, 32, 82, 111, 111, 116, 32, 71, 50, 48, 130, 1, 34, 48, 13, 6, 9, 42, 134, 72, 134, 247, 13, 1, 1, 1, 5, 0, 3, 130, 1, 15, 0, 48, 130, 1, 10, 2, 130, 1, 1, 0, 187, 55, 205, 52, 220, 123, 107, 201, 178, 104, 144, 173, 74, 117, 255, 70, 186, 33, 10, 8, 141, 245, 25, 84, 201, 251, 136, 219, 243, 174, 242, 58, 137, 145, 60, 122, 230, 171, 6, 26, 107, 207, 172, 45, 232, 94, 9, 36, 68, 186, 98, 154, 126, 214, 163, 168, 126, 224, 84, 117, 32, 5, 172, 80, 183, 156, 99, 26, 108, 48, 220, 218, 31, 25, 177, 215, 30, 222, 253, 215, 224, 203, 148, 131, 55, 174, 236, 31, 67, 78, 221, 123, 44, 210, 189, 46, 165, 47, 228, 169, 184, 173, 58, 212, 153, 164, 182, 37, 233, 155, 107, 0, 96, 146, 96, 255, 79, 33, 73, 24, 247, 103, 144, 171, 97, 6, 156, 143, 242, 186, 233, 180, 233, 146, 50, 107, 181, 243, 87, 232, 93, 27, 205, 140, 29, 171, 149, 4, 149, 73, 243, 53, 45, 150, 227, 73, 109, 221, 119, 227, 251, 73, 75, 180, 172, 85, 7, 169, 143, 149, 179, 180, 35, 187, 76, 109, 69, 240, 246, 169, 178, 149, 48, 180, 253, 76, 85, 140, 39, 74, 87, 20, 124, 130, 157, 205, 115, 146, 211, 22, 74, 6, 12, 140, 80, 209, 143, 30, 9, 190, 23, 161, 230, 33, 202, 253, 131, 229, 16, 188, 131, 165, 10, 196, 103, 40, 246, 115, 20, 20, 61, 70, 118, 195, 135, 20, 137, 33, 52, 77, 175, 15, 69, 12, 166, 73, 161, 186, 187, 156, 197, 177, 51, 131, 41, 133, 2, 3, 1, 0, 1, 163, 66, 48, 64, 48, 15, 6, 3, 85, 29, 19, 1, 1, 255, 4, 5, 48, 3, 1, 1, 255, 48, 14, 6, 3, 85, 29, 15, 1, 1, 255, 4, 4, 3, 2, 1, 134, 48, 29, 6, 3, 85, 29, 14, 4, 22, 4, 20, 78, 34, 84, 32, 24, 149, 230, 227, 110, 230, 15, 250, 250, 185, 18, 237, 6, 23, 143, 57],
RawSubjectPublicKeyInfo: [48, 130, 1, 34, 48, 13, 6, 9, 42, 134, 72, 134, 247, 13, 1, 1, 1, 5, 0, 3, 130, 1, 15, 0, 48, 130, 1, 10, 2, 130, 1, 1, 0, 187, 55, 205, 52, 220, 123, 107, 201, 178, 104, 144, 173, 74, 117, 255, 70, 186, 33, 10, 8, 141, 245, 25, 84, 201, 251, 136, 219, 243, 174, 242, 58, 137, 145, 60, 122, 230, 171, 6, 26, 107, 207, 172, 45, 232, 94, 9, 36, 68, 186, 98, 154, 126, 214, 163, 168, 126, 224, 84, 117, 32, 5, 172, 80, 183, 156, 99, 26, 108, 48, 220, 218, 31, 25, 177, 215, 30, 222, 253, 215, 224, 203, 148, 131, 55, 174, 236, 31, 67, 78, 221, 123, 44, 210, 189, 46, 165, 47, 228, 169, 184, 173, 58, 212, 153, 164, 182, 37, 233, 155, 107, 0, 96, 146, 96, 255, 79, 33, 73, 24, 247, 103, 144, 171, 97, 6, 156, 143, 242, 186, 233, 180, 233, 146, 50, 107, 181, 243, 87, 232, 93, 27, 205, 140, 29, 171, 149, 4, 149, 73, 243, 53, 45, 150, 227, 73, 109, 221, 119, 227, 251, 73, 75, 180, 172, 85, 7, 169, 143, 149, 179, 180, 35, 187, 76, 109, 69, 240, 246, 169, 178, 149, 48, 180, 253, 76, 85, 140, 39, 74, 87, 20, 124, 130, 157, 205, 115, 146, 211, 22, 74, 6, 12, 140, 80, 209, 143, 30, 9, 190, 23, 161, 230, 33, 202, 253, 131, 229, 16, 188, 131, 165, 10, 196, 103, 40, 246, 115, 20, 20, 61, 70, 118, 195, 135, 20, 137, 33, 52, 77, 175, 15, 69, 12, 166, 73, 161, 186, 187, 156, 197, 177, 51, 131, 41, 133, 2, 3, 1, 0, 1],
RawSubject: [48, 97, 49, 11, 48, 9, 6, 3, 85, 4, 6, 19, 2, 85, 83, 49, 21, 48, 19, 6, 3, 85, 4, 10, 19, 12, 68, 105, 103, 105, 67, 101, 114, 116, 32, 73, 110, 99, 49, 25, 48, 23, 6, 3, 85, 4, 11, 19, 16, 119, 119, 119, 46, 100, 105, 103, 105, 99, 101, 114, 116, 46, 99, 111, 109, 49, 32, 48, 30, 6, 3, 85, 4, 3, 19, 23, 68, 105, 103, 105, 67, 101, 114, 116, 32, 71, 108, 111, 98, 97, 108, 32, 82, 111, 111, 116, 32, 71, 50],
RawIssuer: [48, 97, 49, 11, 48, 9, 6, 3, 85, 4, 6, 19, 2, 85, 83, 49, 21, 48, 19, 6, 3, 85, 4, 10, 19, 12, 68, 105, 103, 105, 67, 101, 114, 116, 32, 73, 110, 99, 49, 25, 48, 23, 6, 3, 85, 4, 11, 19, 16, 119, 119, 119, 46, 100, 105, 103, 105, 99, 101, 114, 116, 46, 99, 111, 109, 49, 32, 48, 30, 6, 3, 85, 4, 3, 19, 23, 68, 105, 103, 105, 67, 101, 114, 116, 32, 71, 108, 111, 98, 97, 108, 32, 82, 111, 111, 116, 32, 71, 50],
Signature: [96, 103, 40, 148, 111, 14, 72, 99, 235, 49, 221, 234, 103, 24, 213, 137, 125, 60, 197, 139, 74, 127, 233, 190, 219, 43, 23, 223, 176, 95, 115, 119, 42, 50, 19, 57, 129, 103, 66, 132, 35, 242, 69, 103, 53, 236, 136, 191, 248, 143, 176, 97, 12, 52, 164, 174, 32, 76, 132, 198, 219, 248, 53, 225, 118, 217, 223, 166, 66, 187, 199, 68, 8, 134, 127, 54, 116, 36, 90, 218, 108, 13, 20, 89, 53, 189, 242, 73, 221, 182, 31, 201, 179, 13, 71, 42, 61, 153, 47, 187, 92, 187, 181, 212, 32, 225, 153, 95, 83, 70, 21, 219, 104, 155, 240, 243, 48, 213, 62, 49, 226, 141, 132, 158, 227, 138, 218, 218, 150, 62, 53, 19, 165, 95, 240, 249, 112, 80, 112, 71, 65, 17, 87, 25, 78, 192, 143, 174, 6, 196, 149, 19, 23, 47, 27, 37, 159, 117, 242, 177, 142, 153, 161, 111, 19, 177, 65, 113, 254, 136, 42, 200, 79, 16, 32, 85, 215, 243, 20, 69, 229, 224, 68, 244, 234, 135, 149, 50, 147, 14, 254, 83, 70, 250, 44, 157, 255, 139, 34, 185, 75, 217, 9, 69, 164, 222, 164, 184, 154, 88, 221, 27, 125, 82, 159, 142, 89, 67, 136, 129, 164, 158, 38, 213, 111, 173, 221, 13, 198, 55, 125, 237, 3, 146, 27, 229, 119, 95, 118, 238, 60, 141, 196, 93, 86, 91, 162, 217, 102, 110, 179, 53, 55, 229, 50, 182],
SignatureAlgorithm: 4,
PublicKeyAlgorithm: 1,
PublicKey: <*rsa.PublicKey | 0x36a11782c840>{
N: {
neg: false,
abs: [13518897546323503493, 12614377001919816122, 8557832239203824717, 7433462159093546310, 9504020990483827396, 702024576034523901, 5333964605457796894, 8971906913408307990, 18252057450268743444, 5039799191733612724, 10346373305228545133, 18107087016177239977, 3285062882728703971, 2137966295034491701, 13110919595711778188, 17490549066068800107, 17827376715711028367, 6958731023901346072, 11071174057932843776, 11902983757762476756, 4850057369827327278, 16198203776063958047, 2241017827087547863, 13230558770443115738, 9142400105587256400, 4952379155939763112, 7768617195409377572, 9912770753372816922, 14554377099045433914, 13412012197143714132, 12855684210282069830, 13490476835732679625],
},
E: 65537,
},
Version: 3,
SerialNumber: {
neg: false,
abs: [13486139796020132581, 232764303710726560],
},
Issuer: {
Country: ["US"],
Organization: ["DigiCert Inc"],
OrganizationalUnit: ["www.digicert.com"],
Locality: nil,
Province: nil,
StreetAddress: nil,
PostalCode: nil,
SerialNumber: "",
CommonName: "DigiCert Global Root G2",
Names: [
{
Type: [..., ..., ..., ...],
Value: <string>"US",
},
{
Type: [..., ..., ..., ...],
Value: <string>"DigiCert Inc",
},
{
Type: [..., ..., ..., ...],
Value: <string>"www.digicert.com",
},
{
Type: [..., ..., ..., ...],
Value: <string>"DigiCert Global Root G2",
},
],
ExtraNames: nil,
},
Subject: {
Country: ["US"],
Organization: ["DigiCert Inc"],
OrganizationalUnit: ["www.digicert.com"],
Locality: nil,
Province: nil,
StreetAddress: nil,
PostalCode: nil,
SerialNumber: "",
CommonName: "DigiCert Global Root G2",
Names: [
{
Type: [..., ..., ..., ...],
Value: <string>"US",
},
{
Type: [..., ..., ..., ...],
Value: <string>"DigiCert Inc",
},
{
Type: [..., ..., ..., ...],
Value: <string>"www.digicert.com",
},
{
Type: [..., ..., ..., ...],
Value: <string>"DigiCert Global Root G2",
},
],
ExtraNames: nil,
},
NotBefore: {wall: 0, ext: 63510955200, loc: nil},
NotAfter: {wall: 0, ext: 64282766400, loc: nil},
KeyUsage: 97,
Extensions: [
{
Id: [2, 5, 29, 19],
Critical: true,
Value: [48, 3, 1, 1, 255],
},
{
Id: [2, 5, 29, 15],
Critical: true,
Value: [3, 2, 1, 134],
},
{
Id: [2, 5, 29, 14],
Critical: false,
Value: [4, 20, 78, 34, 84, 32, 24, 149, 230, 227, 110, 230, 15, 250, 250, 185, 18, 237, 6, 23, 143, 57],
},
],
ExtraExtensions: nil,
UnhandledCriticalExtensions: nil,
ExtKeyUsage: nil,
UnknownExtKeyUsage: nil,
BasicConstraintsValid: true,
IsCA: true,
MaxPathLen: -1,
MaxPathLenZero: false,
SubjectKeyId: [78, 34, 84, 32, 24, 149, 230, 227, 110, 230, 15, 250, 250, 185, 18, 237, 6, 23, 143, 57],
AuthorityKeyId: nil,
OCSPServer: nil,
IssuingCertificateURL: nil,
DNSNames: nil,
EmailAddresses: nil,
IPAddresses: nil,
URIs: nil,
PermittedDNSDomainsCritical: false,
PermittedDNSDomains: nil,
ExcludedDNSDomains: nil,
PermittedIPRanges: nil,
ExcludedIPRanges: nil,
PermittedEmailAddresses: nil,
ExcludedEmailAddresses: nil,
PermittedURIDomains: nil,
ExcludedURIDomains: nil,
CRLDistributionPoints: nil,
PolicyIdentifiers: nil,
Policies: nil,
InhibitAnyPolicy: 0,
InhibitAnyPolicyZero: false,
InhibitPolicyMapping: 0,
InhibitPolicyMappingZero: false,
RequireExplicitPolicy: 0,
RequireExplicitPolicyZero: false,
PolicyMappings: nil,
},
],
Err: <x509.UnknownAuthorityError>{
Cert: 0x36a1182f5208 (cyclic reference),
hintErr: nil,
hintCert: nil,
},
},
},
},
}
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:525]: sample application must serve the exact customer-specified certificate with a valid Let's Encrypt chain
Expected success, but got an error:
<*fmt.wrapError | 0x36a1186625c0>:
VerifyCustomIngressCertificate timed out after 10m0.003s: ingress route must serve HTTPS with the expected certificate: Get "https://agnhost-e2e-sample-app-l49xr.apps.aro.ingress-cert-le.0o3m.uksouth.aroapp-hcp-stg.azure.com": tls: failed to verify certificate: x509: certificate signed by unknown authority
{
msg: "VerifyCustomIngressCertificate timed out after 10m0.003s: ingress route must serve HTTPS with the expected certificate: Get \"https://agnhost-e2e-sample-app-l49xr.apps.aro.ingress-cert-le.0o3m.uksouth.aroapp-hcp-stg.azure.com\": tls: failed to verify certificate: x509: certificate signed by unknown authority",
err: <*fmt.wrapError | 0x36a1186623e0>{
msg: "ingress route must serve HTTPS with the expected certificate: Get \"https://agnhost-e2e-sample-app-l49xr.apps.aro.ingress-cert-le.0o3m.uksouth.aroapp-hcp-stg.azure.com\": tls: failed to verify certificate: x509: certificate signed by unknown authority",
err: <*url.Error | 0x36a118d74840>{
Op: "Get",
URL: "https://agnhost-e2e-sample-app-l49xr.apps.aro.ingress-cert-le.0o3m.uksouth.aroapp-hcp-stg.azure.com",
Err: <*tls.CertificateVerificationError | 0x36a118d74810>{
UnverifiedCertificates: [
{
Raw: [48, 130, 9, 19, 48, 130, 6, 251, 160, 3, 2, 1, 2, 2, 19, 73, 1, 74, 89, 119, 169, 186, 32, 59, 56, 235, 138, 87, 0, 0, 1, 74, 89, 119, 48, 13, 6, 9, 42, 134, 72, 134, 247, 13, 1, 1, 12, 5, 0, 48, 87, 49, 11, 48, 9, 6, 3, 85, 4, 6, 19, 2, 85, 83, 49, 30, 48, 28, 6, 3, 85, 4, 10, 19, 21, 77, 105, 99, 114, 111, 115, 111, 102, 116, 32, 67, 111, 114, 112, 111, 114, 97, 116, 105, 111, 110, 49, 40, 48, 38, 6, 3, 85, 4, 3, 19, 31, 77, 105, 99, 114, 111, 115, 111, 102, 116, 32, 84, 76, 83, 32, 71, 50, 32, 82, 83, 65, 32, 67, 65, 32, 79, 67, 83, 80, 32, 49, 48, 48, 30, 23, 13, 50, 54, 49, 48, 50, 53, 48, 49, 53, 54, 49, 48, 90, 23, 13, 50, 55, 48, 50, 48, 50, 48, 48, 53, 54, 49, 48, 90, 48, 120, 49, 11, 48, 9, 6, 3, 85, 4, 6, 19, 2, 85, 83, 49, 11, 48, 9, 6, 3, 85, 4, 8, 19, 2, 87, 65, 49, 16, 48, 14, 6, 3, 85, 4, 7, 19, 7, 82, 101, 100, 109, 111, 110, 100, 49, 30, 48, 28, 6, 3, 85, 4, 10, 19, 21, 77, 105, 99, 114, 111, 115, 111, 102, 116, 32, 67, 111, 114, 112, 111, 114, 97, 116, 105, 111, 110, 49, 42, 48, 40, 6, 3, 85, 4, 3, 19, 33, 114, 101, 115, 101, 114, 118, 101, 100, 46, 97, 114, 111, 97, 112, 112, 45, 104, 99, 112, 45, 115, 116, 103, 46, 97, 122, 117, 114, 101, 46, 99, 111, 109, 48, 130, 1, 34, 48, 13, 6, 9, 42, 134, 72, 134, 247, 13, 1, 1, 1, 5, 0, 3, 130, 1, 15, 0, 48, 130, 1, 10, 2, 130, 1, 1, 0, 187, 121, 183, 117, 223, 172, 129, 21, 158, 33, 40, 228, 200, 112, 100, 153, 170, 116, 250, 158, 245, 201, 135, 0, 237, 144, 31, 12, 163, 250, 168, 50, 92, 5, 220, 113, 137, 143, 3, 193, 167, 220, 193, 78, 195, 131, 29, 107, 67, 52, 86, 236, 30, 39, 30, 107, 206, 213, 234, 58, 181, 33, 44, 227, 147, 157, 23, 219, 20, 127, 20, 235, 246, 77, 135, 5, 240, 135, 231, 58, 154, 101, 19, 42, 160, 128, 73, 94, 79, 155, 62, 172, 213, 53, 57, 2, 149, 60, 188, 169, 233, 44, 216, 105, 66, 159, 148, 70, 178, 113, 177, 128, 123, 151, 10, 7, 254, 171, 205, 105, 34, 145, 93, 19, 218, 220, 219, 47, 129, 42, 85, 247, 229, 7, 118, 48, 182, 125, 84, 45, 187, 16, 138, 130, 74, 176, 14, 70, 229, 120, 108, 212, 172, 177, 174, 141, 161, 231, 167, 5, 108, 191, 229, 39, 27, 122, 242, 198, 32, 78, 125, 8, 168, 234, 200, 195, 10, 209, 101, 183, 248, 26, 10, 82, 78, 47, 196, 44, 55, 199, 207, 209, 35, 65, 164, 44, 162, 97, 81, 36, 122, 188, 105, 82, 214, 39, 252, 78, 115, 249, 159, 36, 212, 167, 6, 72, 63, 56, 220, 88, 50, 162, 50, 174, 185, 196, 246, 208, 84, 90, 212, 218, 52, 118, 184, 101, 201, 132, 48, 29, 45, 113, 80, 81, 157, 255, 235, 218, 49, 139, 40, 124, 111, 147, 143, 201, 2, 3, 1, 0, 1, 163, 130, 4, 181, 48, 130, 4, 177, 48, 130, 1, 4, 6, 10, 43, 6, 1, 4, 1, 214, 121, 2, 4, 2, 4, 129, 245, 4, 129, 242, 0, 240, 0, 118, 0, 68, 194, 189, 12, 233, 20, 14, 100, 165, 201, 74, 1, 147, 10, 90, 161, 187, 53, 151, 14, 0, 238, 17, 22, 137, 104, 42, 28, 68, 215, 181, 102, 0, 0, 1, 161, 112, 206, 239, 163, 0, 0, 4, 3, 0, 71, 48, 69, 2, 33, 0, 172, 23, 89, 180, 174, 255, 10, 75, 120, 126, 2, 114, 141, 52, 53, 9, 237, 84, 209, 98, 89, 54, 179, 9, 54, 118, 245, 178, 107, 207, 5, 69, 2, 32, 89, 15, 78, 239, 171, 216, 123, 169, 212, 117, 5, 12, 137, 15, 63, 83, 41, 96, 223, 14, 136, 244, 40, 31, 142, 25, 69, 157, 200, 47, 205, 167, 0, 118, 0, 0, 26, 93, 26, 28, 45, 147, 117, 182, 72, 85, 120, 248, 47, 113, 161, 174, 110, 239, 57, 125, 41, 124, 138, 227, 21, 123, 202, 222, 225, 160, 30, 0, 0, 1, 161, 112, 206, 240, 23, 0, 0, 4, 3, 0, 71, 48, 69, 2, 33, 0, 209, 5, 94, 127, 36, 105, 13, 219, 166, 20, 49, 161, 208, 24, 148, 233, 208, 121, 79, 240, 195, 85, 152, 81, 108, 234, 202, 116, 68, 189, 194, 57, 2, 32, 65, 141, 139, 86, 66, 203, 70, 108, 26, 159, 237, 96, 32, 237, 241, 53, 230, 112, 171, 119, 90, 74, 10, 14, 99, 1, 225, 78, 245, 158, 165, 135, 48, 27, 6, 9, 43, 6, 1, 4, 1, 130, 55, 21, 10, 4, 14, 48, 12, 48, 10, 6, 8, 43, 6, 1, 5, 5, 7, 3, 1, 48, 60, 6, 9, 43, 6, 1, 4, 1, 130, 55, 21, 7, 4, 47, 48, 45, 6, 37, 43, 6, 1, 4, 1, 130, 55, 21, 8, 135, 189, 215, 27, 129, 231, 235, 70, 130, 129, 157, 46, 142, 208, 12, 135, 240, 218, 29, 93, 133, 246, 239, 42, 129, 232, 226, 53, 2, 1, 100, 2, 1, 39, 48, 130, 1, 11, 6, 8, 43, 6, 1, 5, 5, 7, 1, 1, 4, 129, 254, 48, 129, 251, 48, 97, 6, 8, 43, 6, 1, 5, 5, 7, 48, 2, 134, 85, 104, 116, 116, 112, 58, 47, 47, 119, 119, 119, 46, 109, 105, 99, 114, 111, 115, 111, 102, 116, 46, 99, 111, 109, 47, 112, 107, 105, 111, 112, 115, 47, 99, 101, 114, 116, 115, 47, 77, 105, 99, 114, 111, 115, 111, 102, 116, 37, 50, 48, 84, 76, 83, 37, 50, 48, 71, 50, 37, 50, 48, 82, 83, 65, 37, 50, 48, 67, 65, 37, 50, 48, 79, 67, 83, 80, 37, 50, 48, 49, 48, 46, 99, 114, 116, 48, 103, 6, 8, 43, 6, 1, 5, 5, 7, 48, 2, 134, 91, 104, 116, 116, 112, 58, 47, 47, 99, 97, 105, 115, 115, 117, 101, 114, 115, 46, 109, 105, 99, 114, 111, 115, 111, 102, 116, 46, 99, 111, 109, 47, 112, 107, 105, 111, 112, 115, 47, 99, 101, 114, 116, 115, 47, 77, 105, 99, 114, 111, 115, 111, 102, 116, 37, 50, 48, 84, 76, 83, 37, 50, 48, 71, 50, 37, 50, 48, 82, 83, 65, 37, 50, 48, 67, 65, 37, 50, 48, 79, 67, 83, 80, 37, 50, 48, 49, 48, 46, 99, 114, 116, 48, 45, 6, 8, 43, 6, 1, 5, 5, 7, 48, 1, 134, 33, 104, 116, 116, 112, 58, 47, 47, 111, 110, 101, 111, 99, 115, 112, 46, 109, 105, 99, 114, 111, 115, 111, 102, 116, 46, 99, 111, 109, 47, 111, 99, 115, 112, 48, 29, 6, 3, 85, 29, 14, 4, 22, 4, 20, 138, 66, 241, 22, 211, 0, 191, 156, 21, 196, 84, 38, 230, 58, 211, 190, 115, 187, 163, 122, 48, 14, 6, 3, 85, 29, 15, 1, 1, 255, 4, 4, 3, 2, 5, 160, 48, 110, 6, 3, 85, 29, 17, 4, 103, 48, 101, 130, 64, 42, 46, 97, 112, 112, 115, 46, 97, 114, 111, 46, 105, 110, 103, 114, 101, 115, 115, 45, 99, 101, 114, 116, 45, 108, 101, 46, 48, 111, 51, 109, 46, 117, 107, 115, 111, 117, 116, 104, 46, 97, 114, 111, 97, 112, 112, 45, 104, 99, 112, 45, 115, 116, 103, 46, 97, 122, 117, 114, 101, 46, 99, 111, 109, 130, 33, 114, 101, 115, 101, 114, 118, 101, 100, 46, 97, 114, 111, 97, 112, 112, 45, 104, 99, 112, 45, 115, 116, 103, 46, 97, 122, 117, 114, 101, 46, 99, 111, 109, 48, 12, 6, 3, 85, 29, 19, 1, 1, 255, 4, 2, 48, 0, 48, 129, 241, 6, 3, 85, 29, 31, 4, 129, 233, 48, 129, 230, 48, 129, 227, 160, 129, 224, 160, 129, 221, 134, 108, 104, 116, 116, 112, 58, 47, 47, 119, 119, 119, 46, 109, 105, 99, 114, 111, 115, 111, 102, 116, 46, 99, 111, 109, 47, 112, 107, 105, 111, 112, 115, 47, 99, 114, 108, 47, 112, 97, 114, 116, 105, 116, 105, 111, 110, 47, 77, 105, 99, 114, 111, 115, 111, 102, 116, 37, 50, 48, 84, 76, 83, 37, 50, 48, 71, 50, 37, 50, 48, 82, 83, 65, 37, 50, 48, 67, 65, 37, 50, 48, 79, 67, 83, 80, 37, 50, 48, 49, 48, 95, 80, 97, 114, 116, 105, 116, 105, 111, 110, 48, 48, 48, 57, 57, 46, 99, 114, 108, 134, 109, 104, 116, 116, 112, 58, 47, 47, 99, 114, 108, 50, 46, 109, 105, 99, 114, 111, 115, 111, 102, 116, 46, 99, 111, 109, 47, 112, 107, 105, 111, 112, 115, 47, 99, 114, 108, 47, 112, 97, 114, 116, 105, 116, 105, 111, 110, 47, 77, 105, 99, 114, 111, 115, 111, 102, 116, 37, 50, 48, 84, 76, 83, 37, 50, 48, 71, 50, 37, 50, 48, 82, 83, 65, 37, 50, 48, 67, 65, 37, 50, 48, 79, 67, 83, 80, 37, 50, 48, 49, 48, 95, 80, 97, 114, 116, 105, 116, 105, 111, 110, 48, 48, 48, 57, 57, 46, 99, 114, 108, 48, 102, 6, 3, 85, 29, 32, 4, 95, 48, 93, 48, 8, 6, 6, 103, 129, 12, 1, 2, 2, 48, 81, 6, 12, 43, 6, 1, 4, 1, 130, 55, 76, 131, 125, 1, 1, 48, 65, 48, 63, 6, 8, 43, 6, 1, 5, 5, 7, 2, 1, 22, 51, 104, 116, 116, 112, 58, 47, 47, 119, 119, 119, 46, 109, 105, 99, 114, 111, 115, 111, 102, 116, 46, 99, 111, 109, 47, 112, 107, 105, 111, 112, 115, 47, 68, 111, 99, 115, 47, 82, 101, 112, 111, 115, 105, 116, 111, 114, 121, 46, 104, 116, 109, 48, 31, 6, 3, 85, 29, 35, 4, 24, 48, 22, 128, 20, 208, 76, 131, 209, 142, 113, 46, 61, 52, 177, 255, 155, 121, 44, 142, 116, 212, 226, 120, 6, 48, 19, 6, 3, 85, 29, 37, 4, 12, 48, 10, 6, 8, 43, 6, 1, 5, 5, 7, 3, 1, 48, 13, 6, 9, 42, 134, 72, 134, 247, 13, 1, 1, 12, 5, 0, 3, 130, 2, 1, 0, 127, 14, 76, 119, 16, 30, 113, 75, 136, 158, 234, 180, 48, 26, 140, 223, 208, 88, 116, 11, 139, 240, 141, 40, 46, 35, 73, 190, 186, 171, 118, 120, 99, 118, 88, 110, 181, 203, 42, 120, 241, 109, 206, 58, 1, 63, 144, 160, 72, 2, 211, 217, 25, 137, 246, 187, 186, 49, 138, 232, 122, 82, 3, 136, 22, 157, 87, 191, 86, 125, 221, 231, 158, 0, 204, 174, 39, 65, 179, 160, 12, 205, 190, 69, 207, 65, 139, 215, 99, 239, 19, 51, 250, 33, 66, 49, 250, 3, 217, 217, 178, 148, 58, 186, 139, 102, 86, 205, 171, 36, 144, 223, 188, 149, 248, 1, 103, 232, 163, 162, 100, 23, 166, 66, 215, 67, 190, 124, 120, 217, 106, 101, 246, 252, 8, 144, 121, 98, 136, 193, 124, 213, 80, 149, 73, 94, 215, 24, 197, 138, 7, 143, 140, 229, 172, 93, 155, 162, 211, 4, 222, 43, 210, 38, 230, 222, 23, 202, 214, 208, 103, 147, 145, 86, 53, 202, 100, 48, 50, 41, 5, 116, 80, 254, 160, 164, 22, 192, 242, 55, 78, 107, 236, 232, 202, 76, 47, 127, 5, 241, 21, 79, 8, 239, 16, 12, 106, 11, 200, 233, 173, 167, 160, 195, 61, 159, 42, 183, 223, 149, 235, 138, 251, 54, 65, 95, 191, 206, 35, 254, 15, 224, 194, 106, 185, 99, 23, 156, 154, 222, 84, 145, 1, 212, 126, 42, 74, 254, 71, 27, 110, 6, 83, 252, 190, 60, 29, 24, 98, 214, 226, 88, 108, 117, 86, 233, 3, 27, 205, 3, 203, 36, 3, 175, 73, 206, 16, 65, 117, 125, 162, 73, 22, 207, 82, 232, 237, 197, 40, 139, 79, 103, 210, 165, 182, 151, 205, 15, 253, 195, 116, 121, 178, 103, 118, 187, 165, 173, 124, 138, 96, 201, 232, 236, 212, 18, 179, 244, 160, 183, 40, 12, 30, 207, 237, 36, 120, 26, 57, 114, 158, 34, 152, 93, 132, 52, 124, 112, 222, 90, 37, 164, 188, 88, 53, 52, 110, 127, 166, 176, 30, 184, 129, 236, 40, 151, 237, 246, 44, 182, 131, 71, 47, 162, 122, 195, 81, 3, 7, 197, 200, 200, 48, 29, 56, 138, 158, 6, 33, 41, 210, 36, 197, 209, 154, 188, 46, 232, 167, 27, 177, 139, 150, 154, 50, 110, 13, 89, 233, 181, 191, 72, 106, 53, 174, 105, 231, 108, 61, 64, 154, 240, 69, 47, 220, 37, 46, 170, 187, 63, 76, 114, 220, 25, 31, 218, 164, 20, 35, 139, 252, 139, 174, 8, 42, 39, 150, 249, 226, 25, 42, 115, 161, 172, 59, 218, 77, 27, 104, 120, 74, 137, 124, 201, 198, 130, 184, 80, 219, 176, 188, 97, 76, 79, 77, 137, 160, 247, 211, 155, 231, 107, 56, 112, 147, 133, 216, 183, 201, 229, 98, 78, 26, 232, 122, 160, 50, 149, 73, 205, 59, 201, 173, 34, 22, 180, 220, 49, 209, 56, 199, 177, 177, 107, 162, 228, 222, 80, 57, 112, 238, 205],
RawTBSCertificate: [48, 130, 6, 251, 160, 3, 2, 1, 2, 2, 19, 73, 1, 74, 89, 119, 169, 186, 32, 59, 56, 235, 138, 87, 0, 0, 1, 74, 89, 119, 48, 13, 6, 9, 42, 134, 72, 134, 247, 13, 1, 1, 12, 5, 0, 48, 87, 49, 11, 48, 9, 6, 3, 85, 4, 6, 19, 2, 85, 83, 49, 30, 48, 28, 6, 3, 85, 4, 10, 19, 21, 77, 105, 99, 114, 111, 115, 111, 102, 116, 32, 67, 111, 114, 112, 111, 114, 97, 116, 105, 111, 110, 49, 40, 48, 38, 6, 3, 85, 4, 3, 19, 31, 77, 105, 99, 114, 111, 115, 111, 102, 116, 32, 84, 76, 83, 32, 71, 50, 32, 82, 83, 65, 32, 67, 65, 32, 79, 67, 83, 80, 32, 49, 48, 48, 30, 23, 13, 50, 54, 49, 48, 50, 53, 48, 49, 53, 54, 49, 48, 90, 23, 13, 50, 55, 48, 50, 48, 50, 48, 48, 53, 54, 49, 48, 90, 48, 120, 49, 11, 48, 9, 6, 3, 85, 4, 6, 19, 2, 85, 83, 49, 11, 48, 9, 6, 3, 85, 4, 8, 19, 2, 87, 65, 49, 16, 48, 14, 6, 3, 85, 4, 7, 19, 7, 82, 101, 100, 109, 111, 110, 100, 49, 30, 48, 28, 6, 3, 85, 4, 10, 19, 21, 77, 105, 99, 114, 111, 115, 111, 102, 116, 32, 67, 111, 114, 112, 111, 114, 97, 116, 105, 111, 110, 49, 42, 48, 40, 6, 3, 85, 4, 3, 19, 33, 114, 101, 115, 101, 114, 118, 101, 100, 46, 97, 114, 111, 97, 112, 112, 45, 104, 99, 112, 45, 115, 116, 103, 46, 97, 122, 117, 114, 101, 46, 99, 111, 109, 48, 130, 1, 34, 48, 13, 6, 9, 42, 134, 72, 134, 247, 13, 1, 1, 1, 5, 0, 3, 130, 1, 15, 0, 48, 130, 1, 10, 2, 130, 1, 1, 0, 187, 121, 183, 117, 223, 172, 129, 21, 158, 33, 40, 228, 200, 112, 100, 153, 170, 116, 250, 158, 245, 201, 135, 0, 237, 144, 31, 12, 163, 250, 168, 50, 92, 5, 220, 113, 137, 143, 3, 193, 167, 220, 193, 78, 195, 131, 29, 107, 67, 52, 86, 236, 30, 39, 30, 107, 206, 213, 234, 58, 181, 33, 44, 227, 147, 157, 23, 219, 20, 127, 20, 235, 246, 77, 135, 5, 240, 135, 231, 58, 154, 101, 19, 42, 160, 128, 73, 94, 79, 155, 62, 172, 213, 53, 57, 2, 149, 60, 188, 169, 233, 44, 216, 105, 66, 159, 148, 70, 178, 113, 177, 128, 123, 151, 10, 7, 254, 171, 205, 105, 34, 145, 93, 19, 218, 220, 219, 47, 129, 42, 85, 247, 229, 7, 118, 48, 182, 125, 84, 45, 187, 16, 138, 130, 74, 176, 14, 70, 229, 120, 108, 212, 172, 177, 174, 141, 161, 231, 167, 5, 108, 191, 229, 39, 27, 122, 242, 198, 32, 78, 125, 8, 168, 234, 200, 195, 10, 209, 101, 183, 248, 26, 10, 82, 78, 47, 196, 44, 55, 199, 207, 209, 35, 65, 164, 44, 162, 97, 81, 36, 122, 188, 105, 82, 214, 39, 252, 78, 115, 249, 159, 36, 212, 167, 6, 72, 63, 56, 220, 88, 50, 162, 50, 174, 185, 196, 246, 208, 84, 90, 212, 218, 52, 118, 184, 101, 201, 132, 48, 29, 45, 113, 80, 81, 157, 255, 235, 218, 49, 139, 40, 124, 111, 147, 143, 201, 2, 3, 1, 0, 1, 163, 130, 4, 181, 48, 130, 4, 177, 48, 130, 1, 4, 6, 10, 43, 6, 1, 4, 1, 214, 121, 2, 4, 2, 4, 129, 245, 4, 129, 242, 0, 240, 0, 118, 0, 68, 194, 189, 12, 233, 20, 14, 100, 165, 201, 74, 1, 147, 10, 90, 161, 187, 53, 151, 14, 0, 238, 17, 22, 137, 104, 42, 28, 68, 215, 181, 102, 0, 0, 1, 161, 112, 206, 239, 163, 0, 0, 4, 3, 0, 71, 48, 69, 2, 33, 0, 172, 23, 89, 180, 174, 255, 10, 75, 120, 126, 2, 114, 141, 52, 53, 9, 237, 84, 209, 98, 89, 54, 179, 9, 54, 118, 245, 178, 107, 207, 5, 69, 2, 32, 89, 15, 78, 239, 171, 216, 123, 169, 212, 117, 5, 12, 137, 15, 63, 83, 41, 96, 223, 14, 136, 244, 40, 31, 142, 25, 69, 157, 200, 47, 205, 167, 0, 118, 0, 0, 26, 93, 26, 28, 45, 147, 117, 182, 72, 85, 120, 248, 47, 113, 161, 174, 110, 239, 57, 125, 41, 124, 138, 227, 21, 123, 202, 222, 225, 160, 30, 0, 0, 1, 161, 112, 206, 240, 23, 0, 0, 4, 3, 0, 71, 48, 69, 2, 33, 0, 209, 5, 94, 127, 36, 105, 13, 219, 166, 20, 49, 161, 208, 24, 148, 233, 208, 121, 79, 240, 195, 85, 152, 81, 108, 234, 202, 116, 68, 189, 194, 57, 2, 32, 65, 141, 139, 86, 66, 203, 70, 108, 26, 159, 237, 96, 32, 237, 241, 53, 230, 112, 171, 119, 90, 74, 10, 14, 99, 1, 225, 78, 245, 158, 165, 135, 48, 27, 6, 9, 43, 6, 1, 4, 1, 130, 55, 21, 10, 4, 14, 48, 12, 48, 10, 6, 8, 43, 6, 1, 5, 5, 7, 3, 1, 48, 60, 6, 9, 43, 6, 1, 4, 1, 130, 55, 21, 7, 4, 47, 48, 45, 6, 37, 43, 6, 1, 4, 1, 130, 55, 21, 8, 135, 189, 215, 27, 129, 231, 235, 70, 130, 129, 157, 46, 142, 208, 12, 135, 240, 218, 29, 93, 133, 246, 239, 42, 129, 232, 226, 53, 2, 1, 100, 2, 1, 39, 48, 130, 1, 11, 6, 8, 43, 6, 1, 5, 5, 7, 1, 1, 4, 129, 254, 48, 129, 251, 48, 97, 6, 8, 43, 6, 1, 5, 5, 7, 48, 2, 134, 85, 104, 116, 116, 112, 58, 47, 47, 119, 119, 119, 46, 109, 105, 99, 114, 111, 115, 111, 102, 116, 46, 99, 111, 109, 47, 112, 107, 105, 111, 112, 115, 47, 99, 101, 114, 116, 115, 47, 77, 105, 99, 114, 111, 115, 111, 102, 116, 37, 50, 48, 84, 76, 83, 37, 50, 48, 71, 50, 37, 50, 48, 82, 83, 65, 37, 50, 48, 67, 65, 37, 50, 48, 79, 67, 83, 80, 37, 50, 48, 49, 48, 46, 99, 114, 116, 48, 103, 6, 8, 43, 6, 1, 5, 5, 7, 48, 2, 134, 91, 104, 116, 116, 112, 58, 47, 47, 99, 97, 105, 115, 115, 117, 101, 114, 115, 46, 109, 105, 99, 114, 111, 115, 111, 102, 116, 46, 99, 111, 109, 47, 112, 107, 105, 111, 112, 115, 47, 99, 101, 114, 116, 115, 47, 77, 105, 99, 114, 111, 115, 111, 102, 116, 37, 50, 48, 84, 76, 83, 37, 50, 48, 71, 50, 37, 50, 48, 82, 83, 65, 37, 50, 48, 67, 65, 37, 50, 48, 79, 67, 83, 80, 37, 50, 48, 49, 48, 46, 99, 114, 116, 48, 45, 6, 8, 43, 6, 1, 5, 5, 7, 48, 1, 134, 33, 104, 116, 116, 112, 58, 47, 47, 111, 110, 101, 111, 99, 115, 112, 46, 109, 105, 99, 114, 111, 115, 111, 102, 116, 46, 99, 111, 109, 47, 111, 99, 115, 112, 48, 29, 6, 3, 85, 29, 14, 4, 22, 4, 20, 138, 66, 241, 22, 211, 0, 191, 156, 21, 196, 84, 38, 230, 58, 211, 190, 115, 187, 163, 122, 48, 14, 6, 3, 85, 29, 15, 1, 1, 255, 4, 4, 3, 2, 5, 160, 48, 110, 6, 3, 85, 29, 17, 4, 103, 48, 101, 130, 64, 42, 46, 97, 112, 112, 115, 46, 97, 114, 111, 46, 105, 110, 103, 114, 101, 115, 115, 45, 99, 101, 114, 116, 45, 108, 101, 46, 48, 111, 51, 109, 46, 117, 107, 115, 111, 117, 116, 104, 46, 97, 114, 111, 97, 112, 112, 45, 104, 99, 112, 45, 115, 116, 103, 46, 97, 122, 117, 114, 101, 46, 99, 111, 109, 130, 33, 114, 101, 115, 101, 114, 118, 101, 100, 46, 97, 114, 111, 97, 112, 112, 45, 104, 99, 112, 45, 115, 116, 103, 46, 97, 122, 117, 114, 101, 46, 99, 111, 109, 48, 12, 6, 3, 85, 29, 19, 1, 1, 255, 4, 2, 48, 0, 48, 129, 241, 6, 3, 85, 29, 31, 4, 129, 233, 48, 129, 230, 48, 129, 227, 160, 129, 224, 160, 129, 221, 134, 108, 104, 116, 116, 112, 58, 47, 47, 119, 119, 119, 46, 109, 105, 99, 114, 111, 115, 111, 102, 116, 46, 99, 111, 109, 47, 112, 107, 105, 111, 112, 115, 47, 99, 114, 108, 47, 112, 97, 114, 116, 105, 116, 105, 111, 110, 47, 77, 105, 99, 114, 111, 115, 111, 102, 116, 37, 50, 48, 84, 76, 83, 37, 50, 48, 71, 50, 37, 50, 48, 82, 83, 65, 37, 50, 48, 67, 65, 37, 50, 48, 79, 67, 83, 80, 37, 50, 48, 49, 48, 95, 80, 97, 114, 116, 105, 116, 105, 111, 110, 48, 48, 48, 57, 57, 46, 99, 114, 108, 134, 109, 104, 116, 116, 112, 58, 47, 47, 99, 114, 108, 50, 46, 109, 105, 99, 114, 111, 115, 111, 102, 116, 46, 99, 111, 109, 47, 112, 107, 105, 111, 112, 115, 47, 99, 114, 108, 47, 112, 97, 114, 116, 105, 116, 105, 111, 110, 47, 77, 105, 99, 114, 111, 115, 111, 102, 116, 37, 50, 48, 84, 76, 83, 37, 50, 48, 71, 50, 37, 50, 48, 82, 83, 65, 37, 50, 48, 67, 65, 37, 50, 48, 79, 67, 83, 80, 37, 50, 48, 49, 48, 95, 80, 97, 114, 116, 105, 116, 105, 111, 110, 48, 48, 48, 57, 57, 46, 99, 114, 108, 48, 102, 6, 3, 85, 29, 32, 4, 95, 48, 93, 48, 8, 6, 6, 103, 129, 12, 1, 2, 2, 48, 81, 6, 12, 43, 6, 1, 4, 1, 130, 55, 76, 131, 125, 1, 1, 48, 65, 48, 63, 6, 8, 43, 6, 1, 5, 5, 7, 2, 1, 22, 51, 104, 116, 116, 112, 58, 47, 47, 119, 119, 119, 46, 109, 105, 99, 114, 111, 115, 111, 102, 116, 46, 99, 111, 109, 47, 112, 107, 105, 111, 112, 115, 47, 68, 111, 99, 115, 47, 82, 101, 112, 111, 115, 105, 116, 111, 114, 121, 46, 104, 116, 109, 48, 31, 6, 3, 85, 29, 35, 4, 24, 48, 22, 128, 20, 208, 76, 131, 209, 142, 113, 46, 61, 52, 177, 255, 155, 121, 44, 142, 116, 212, 226, 120, 6, 48, 19, 6, 3, 85, 29, 37, 4, 12, 48, 10, 6, 8, 43, 6, 1, 5, 5, 7, 3, 1],
RawSubjectPublicKeyInfo: [48, 130, 1, 34, 48, 13, 6, 9, 42, 134, 72, 134, 247, 13, 1, 1, 1, 5, 0, 3, 130, 1, 15, 0, 48, 130, 1, 10, 2, 130, 1, 1, 0, 187, 121, 183, 117, 223, 172, 129, 21, 158, 33, 40, 228, 200, 112, 100, 153, 170, 116, 250, 158, 245, 201, 135, 0, 237, 144, 31, 12, 163, 250, 168, 50, 92, 5, 220, 113, 137, 143, 3, 193, 167, 220, 193, 78, 195, 131, 29, 107, 67, 52, 86, 236, 30, 39, 30, 107, 206, 213, 234, 58, 181, 33, 44, 227, 147, 157, 23, 219, 20, 127, 20, 235, 246, 77, 135, 5, 240, 135, 231, 58, 154, 101, 19, 42, 160, 128, 73, 94, 79, 155, 62, 172, 213, 53, 57, 2, 149, 60, 188, 169, 233, 44, 216, 105, 66, 159, 148, 70, 178, 113, 177, 128, 123, 151, 10, 7, 254, 171, 205, 105, 34, 145, 93, 19, 218, 220, 219, 47, 129, 42, 85, 247, 229, 7, 118, 48, 182, 125, 84, 45, 187, 16, 138, 130, 74, 176, 14, 70, 229, 120, 108, 212, 172, 177, 174, 141, 161, 231, 167, 5, 108, 191, 229, 39, 27, 122, 242, 198, 32, 78, 125, 8, 168, 234, 200, 195, 10, 209, 101, 183, 248, 26, 10, 82, 78, 47, 196, 44, 55, 199, 207, 209, 35, 65, 164, 44, 162, 97, 81, 36, 122, 188, 105, 82, 214, 39, 252, 78, 115, 249, 159, 36, 212, 167, 6, 72, 63, 56, 220, 88, 50, 162, 50, 174, 185, 196, 246, 208, 84, 90, 212, 218, 52, 118, 184, 101, 201, 132, 48, 29, 45, 113, 80, 81, 157, 255, 235, 218, 49, 139, 40, 124, 111, 147, 143, 201, 2, 3, 1, 0, 1],
RawSubject: [48, 120, 49, 11, 48, 9, 6, 3, 85, 4, 6, 19, 2, 85, 83, 49, 11, 48, 9, 6, 3, 85, 4, 8, 19, 2, 87, 65, 49, 16, 48, 14, 6, 3, 85, 4, 7, 19, 7, 82, 101, 100, 109, 111, 110, 100, 49, 30, 48, 28, 6, 3, 85, 4, 10, 19, 21, 77, 105, 99, 114, 111, 115, 111, 102, 116, 32, 67, 111, 114, 112, 111, 114, 97, 116, 105, 111, 110, 49, 42, 48, 40, 6, 3, 85, 4, 3, 19, 33, 114, 101, 115, 101, 114, 118, 101, 100, 46, 97, 114, 111, 97, 112, 112, 45, 104, 99, 112, 45, 115, 116, 103, 46, 97, 122, 117, 114, 101, 46, 99, 111, 109],
RawIssuer: [48, 87, 49, 11, 48, 9, 6, 3, 85, 4, 6, 19, 2, 85, 83, 49, 30, 48, 28, 6, 3, 85, 4, 10, 19, 21, 77, 105, 99, 114, 111, 115, 111, 102, 116, 32, 67, 111, 114, 112, 111, 114, 97, 116, 105, 111, 110, 49, 40, 48, 38, 6, 3, 85, 4, 3, 19, 31, 77, 105, 99, 114, 111, 115, 111, 102, 116, 32, 84, 76, 83, 32, 71, 50, 32, 82, 83, 65, 32, 67, 65, 32, 79, 67, 83, 80, 32, 49, 48],
Signature: [127, 14, 76, 119, 16, 30, 113, 75, 136, 158, 234, 180, 48, 26, 140, 223, 208, 88, 116, 11, 139, 240, 141, 40, 46, 35, 73, 190, 186, 171, 118, 120, 99, 118, 88, 110, 181, 203, 42, 120, 241, 109, 206, 58, 1, 63, 144, 160, 72, 2, 211, 217, 25, 137, 246, 187, 186, 49, 138, 232, 122, 82, 3, 136, 22, 157, 87, 191, 86, 125, 221, 231, 158, 0, 204, 174, 39, 65, 179, 160, 12, 205, 190, 69, 207, 65, 139, 215, 99, 239, 19, 51, 250, 33, 66, 49, 250, 3, 217, 217, 178, 148, 58, 186, 139, 102, 86, 205, 171, 36, 144, 223, 188, 149, 248, 1, 103, 232, 163, 162, 100, 23, 166, 66, 215, 67, 190, 124, 120, 217, 106, 101, 246, 252, 8, 144, 121, 98, 136, 193, 124, 213, 80, 149, 73, 94, 215, 24, 197, 138, 7, 143, 140, 229, 172, 93, 155, 162, 211, 4, 222, 43, 210, 38, 230, 222, 23, 202, 214, 208, 103, 147, 145, 86, 53, 202, 100, 48, 50, 41, 5, 116, 80, 254, 160, 164, 22, 192, 242, 55, 78, 107, 236, 232, 202, 76, 47, 127, 5, 241, 21, 79, 8, 239, 16, 12, 106, 11, 200, 233, 173, 167, 160, 195, 61, 159, 42, 183, 223, 149, 235, 138, 251, 54, 65, 95, 191, 206, 35, 254, 15, 224, 194, 106, 185, 99, 23, 156, 154, 222, 84, 145, 1, 212, 126, 42, 74, 254, 71, 27, 110, 6, 83, 252, 190, 60, 29, 24, 98, 214, 226, 88, 108, 117, 86, 233, 3, 27, 205, 3, 203, 36, 3, 175, 73, 206, 16, 65, 117, 125, 162, 73, 22, 207, 82, 232, 237, 197, 40, 139, 79, 103, 210, 165, 182, 151, 205, 15, 253, 195, 116, 121, 178, 103, 118, 187, 165, 173, 124, 138, 96, 201, 232, 236, 212, 18, 179, 244, 160, 183, 40, 12, 30, 207, 237, 36, 120, 26, 57, 114, 158, 34, 152, 93, 132, 52, 124, 112, 222, 90, 37, 164, 188, 88, 53, 52, 110, 127, 166, 176, 30, 184, 129, 236, 40, 151, 237, 246, 44, 182, 131, 71, 47, 162, 122, 195, 81, 3, 7, 197, 200, 200, 48, 29, 56, 138, 158, 6, 33, 41, 210, 36, 197, 209, 154, 188, 46, 232, 167, 27, 177, 139, 150, 154, 50, 110, 13, 89, 233, 181, 191, 72, 106, 53, 174, 105, 231, 108, 61, 64, 154, 240, 69, 47, 220, 37, 46, 170, 187, 63, 76, 114, 220, 25, 31, 218, 164, 20, 35, 139, 252, 139, 174, 8, 42, 39, 150, 249, 226, 25, 42, 115, 161, 172, 59, 218, 77, 27, 104, 120, 74, 137, 124, 201, 198, 130, 184, 80, 219, 176, 188, 97, 76, 79, 77, 137, 160, 247, 211, 155, 231, 107, 56, 112, 147, 133, 216, 183, 201, 229, 98, 78, 26, 232, 122, 160, 50, 149, 73, 205, 59, 201, 173, 34, 22, 180, 220, 49, 209, 56, 199, 177, 177, 107, 162, 228, 222, 80, 57, 112, 238, 205],
SignatureAlgorithm: 5,
PublicKeyAlgorithm: 1,
PublicKey: <*rsa.PublicKey | 0x36a119f65b90>{
N: {
neg: false,
abs: [3569991644534247369, 3274486715548429274, 3780411684536201245, 13386095367080826074, 4555633294449980078, 8356885564098020936, 8844059572749990990, 2540492176423473444, 5633937353057226705, 779516051327683154, 2327935533544294595, 7836233832775480006, 12443919168731326213, 5381817252368116948, 13149759042380401282, 9307346103102305840, 2490874408770263855, 8905597817602166121, 4800718759162655104, 10753677448201427049, 5736247462418790658, 11125319528290994526, 17747990166117017402, 10636684124628129003, 14904076079720377571, 4842591071448276587, 12095755243242069355, 6630948406570189761, 17118216322786437170, 12282717644381587200, 11394433495304529049, 13509030274166194453],
},
E: 65537,
},
Version: 3,
SerialNumber: {
neg: false,
abs: [9968436300229532023, 6446808008473917675, 4784458],
},
Issuer: {
Country: ["US"],
Organization: [
"Microsoft Corporation",
],
OrganizationalUnit: nil,
Locality: nil,
Province: nil,
StreetAddress: nil,
PostalCode: nil,
SerialNumber: "",
CommonName: "Microsoft TLS G2 RSA CA OCSP 10",
Names: [
{
Type: [..., ..., ..., ...],
Value: <string>"US",
},
{
Type: [..., ..., ..., ...],
Value: <string>"Microsoft Corporation",
},
{
Type: [..., ..., ..., ...],
Value: <string>"Microsoft TLS G2 RSA CA OCSP 10",
},
],
ExtraNames: nil,
},
Subject: {
Country: ["US"],
Organization: [
"Microsoft Corporation",
],
OrganizationalUnit: nil,
Locality: ["Redmond"],
Province: ["WA"],
StreetAddress: nil,
PostalCode: nil,
SerialNumber: "",
CommonName: "reserved.aroapp-hcp-stg.azure.com",
Names: [
{
Type: [..., ..., ..., ...],
Value: <string>"US",
},
{
Type: [..., ..., ..., ...],
Value: <string>"WA",
},
{
Type: [..., ..., ..., ...],
Value: <string>"Redmond",
},
{
Type: [..., ..., ..., ...],
Value: <string>"Microsoft Corporation",
},
{
Type: [..., ..., ..., ...],
Value: <string>"reserved.aroapp-hcp-stg.azure.com",
},
],
ExtraNames: nil,
},
NotBefore: {wall: 0, ext: 63928490170, loc: nil},
NotAfter: {wall: 0, ext: 63937126570, loc: nil},
KeyUsage: 5,
Extensions: [
{
Id: [1, 3, 6, 1, 4, 1, 11129, 2, 4, 2],
Critical: false,
Value: [4, 129, 242, 0, 240, 0, 118, 0, 68, 194, 189, 12, 233, 20, 14, 100, 165, 201, 74, 1, 147, 10, 90, 161, 187, 53, 151, 14, 0, 238, 17, 22, 137, 104, 42, 28, 68, 215, 181, 102, 0, 0, 1, 161, 112, 206, 239, 163, 0, 0, 4, 3, 0, 71, 48, 69, 2, 33, 0, 172, 23, 89, 180, 174, 255, 10, 75, 120, 126, 2, 114, 141, 52, 53, 9, 237, 84, 209, 98, 89, 54, 179, 9, 54, 118, 245, 178, 107, 207, 5, 69, 2, 32, 89, 15, 78, 239, 171, 216, 123, 169, 212, 117, 5, 12, 137, 15, 63, 83, 41, 96, 223, 14, 136, 244, 40, 31, 142, 25, 69, 157, 200, 47, 205, 167, 0, 118, 0, 0, 26, 93, 26, 28, 45, 147, 117, 182, 72, 85, 120, 248, 47, 113, 161, 174, 110, 239, 57, 125, 41, 124, 138, 227, 21, 123, 202, 222, 225, 160, 30, 0, 0, 1, 161, 112, 206, 240, 23, 0, 0, 4, 3, 0, 71, 48, 69, 2, 33, 0, 209, 5, 94, 127, 36, 105, 13, 219, 166, 20, 49, 161, 208, 24, 148, 233, 208, 121, 79, 240, 195, 85, 152, 81, 108, 234, 202, 116, 68, 189, 194, 57, 2, 32, 65, 141, 139, 86, 66, 203, 70, 108, 26, 159, 237, 96, 32, 237, 241, 53, 230, 112, 171, 119, 90, 74, 10, 14, 99, 1, 225, 78, 245, 158, 165, 135],
},
{
Id: [1, 3, 6, 1, 4, 1, 311, 21, 10],
Critical: false,
Value: [48, 12, 48, 10, 6, 8, 43, 6, 1, 5, 5, 7, 3, 1],
},
{
Id: [1, 3, 6, 1, 4, 1, 311, 21, 7],
Critical: false,
Value: [48, 45, 6, 37, 43, 6, 1, 4, 1, 130, 55, 21, 8, 135, 189, 215, 27, 129, 231, 235, 70, 130, 129, 157, 46, 142, 208, 12, 135, 240, 218, 29, 93, 133, 246, 239, 42, 129, 232, 226, 53, 2, 1, 100, 2, 1, 39],
},
{
Id: [1, 3, 6, 1, 5, 5, 7, 1, 1],
Critical: false,
Value: [48, 129, 251, 48, 97, 6, 8, 43, 6, 1, 5, 5, 7, 48, 2, 134, 85, 104, 116, 116, 112, 58, 47, 47, 119, 119, 119, 46, 109, 105, 99, 114, 111, 115, 111, 102, 116, 46, 99, 111, 109, 47, 112, 107, 105, 111, 112, 115, 47, 99, 101, 114, 116, 115, 47, 77, 105, 99, 114, 111, 115, 111, 102, 116, 37, 50, 48, 84, 76, 83, 37, 50, 48, 71, 50, 37, 50, 48, 82, 83, 65, 37, 50, 48, 67, 65, 37, 50, 48, 79, 67, 83, 80, 37, 50, 48, 49, 48, 46, 99, 114, 116, 48, 103, 6, 8, 43, 6, 1, 5, 5, 7, 48, 2, 134, 91, 104, 116, 116, 112, 58, 47, 47, 99, 97, 105, 115, 115, 117, 101, 114, 115, 46, 109, 105, 99, 114, 111, 115, 111, 102, 116, 46, 99, 111, 109, 47, 112, 107, 105, 111, 112, 115, 47, 99, 101, 114, 116, 115, 47, 77, 105, 99, 114, 111, 115, 111, 102, 116, 37, 50, 48, 84, 76, 83, 37, 50, 48, 71, 50, 37, 50, 48, 82, 83, 65, 37, 50, 48, 67, 65, 37, 50, 48, 79, 67, 83, 80, 37, 50, 48, 49, 48, 46, 99, 114, 116, 48, 45, 6, 8, 43, 6, 1, 5, 5, 7, 48, 1, 134, 33, 104, 116, 116, 112, 58, 47, 47, 111, 110, 101, 111, 99, 115, 112, 46, 109, 105, 99, 114, 111, 115, 111, 102, 116, 46, 99, 111, 109, 47, 111, 99, 115, 112],
},
{
Id: [2, 5, 29, 14],
Critical: false,
Value: [4, 20, 138, 66, 241, 22, 211, 0, 191, 156, 21, 196, 84, 38, 230, 58, 211, 190, 115, 187, 163, 122],
},
{
Id: [2, 5, 29, 15],
Critical: true,
Value: [3, 2, 5, 160],
},
{
Id: [2, 5, 29, 17],
Critical: false,
Value: "0e\x82@*.apps.aro.ingress-cert-le.0o3m.uksouth.aroapp-hcp-stg.azure.com\x82!reserved.aroapp-hcp-stg.azure.com",
},
{Id: [2, 5, 29, 19], Critical: true, Value: [48, 0]},
{
Id: [2, 5, 29, 31],
Critical: false,
Value: "0\x81\xe60\x81㠁ࠁ݆lhttp://www.microsoft.com/pkiops/crl/partition/Microsoft%20TLS%20G2%20RSA%20CA%20OCSP%2010_Partition00099.crl\x86mhttp://crl2.microsoft.com/pkiops/crl/partition/Microsoft%20TLS%20G2%20RSA%20CA%20OCSP%2010_Partition00099.crl",
},
{
Id: [2, 5, 29, 32],
Critical: false,
Value: [48, 93, 48, 8, 6, 6, 103, 129, 12, 1, 2, 2, 48, 81, 6, 12, 43, 6, 1, 4, 1, 130, 55, 76, 131, 125, 1, 1, 48, 65, 48, 63, 6, 8, 43, 6, 1, 5, 5, 7, 2, 1, 22, 51, 104, 116, 116, 112, 58, 47, 47, 119, 119, 119, 46, 109, 105, 99, 114, 111, 115, 111, 102, 116, 46, 99, 111, 109, 47, 112, 107, 105, 111, 112, 115, 47, 68, 111, 99, 115, 47, 82, 101, 112, 111, 115, 105, 116, 111, 114, 121, 46, 104, 116, 109],
},
{
Id: [2, 5, 29, 35],
Critical: false,
Value: [48, 22, 128, 20, 208, 76, 131, 209, 142, 113, 46, 61, 52, 177, 255, 155, 121, 44, 142, 116, 212, 226, 120, 6],
},
{
Id: [2, 5, 29, 37],
Critical: false,
Value: [48, 10, 6, 8, 43, 6, 1, 5, 5, 7, 3, 1],
},
],
ExtraExtensions: nil,
UnhandledCriticalExtensions: nil,
ExtKeyUsage: [1],
UnknownExtKeyUsage: nil,
BasicConstraintsValid: true,
IsCA: false,
MaxPathLen: -1,
MaxPathLenZero: false,
SubjectKeyId: [138, 66, 241, 22, 211, 0, 191, 156, 21, 196, 84, 38, 230, 58, 211, 190, 115, 187, 163, 122],
AuthorityKeyId: [208, 76, 131, 209, 142, 113, 46, 61, 52, 177, 255, 155, 121, 44, 142, 116, 212, 226, 120, 6],
OCSPServer: [
"http://oneocsp.microsoft.com/ocsp",
],
IssuingCertificateURL: [
"http://www.microsoft.com/pkiops/certs/Microsoft%20TLS%20G2%20RSA%20CA%20OCSP%2010.crt",
"http://caissuers.microsoft.com/pkiops/certs/Microsoft%20TLS%20G2%20RSA%20CA%20OCSP%2010.crt",
],
DNSNames: [
"*.apps.aro.ingress-cert-le.0o3m.uksouth.aroapp-hcp-stg.azure.com",
"reserved.aroapp-hcp-stg.azure.com",
],
EmailAddresses: nil,
IPAddresses: nil,
URIs: nil,
PermittedDNSDomainsCritical: false,
PermittedDNSDomains: nil,
ExcludedDNSDomains: nil,
PermittedIPRanges: nil,
ExcludedIPRanges: nil,
PermittedEmailAddresses: nil,
ExcludedEmailAddresses: nil,
PermittedURIDomains: nil,
ExcludedURIDomains: nil,
CRLDistributionPoints: [
"http://www.microsoft.com/pkiops/crl/partition/Microsoft%20TLS%20G2%20RSA%20CA%20OCSP%2010_Partition00099.crl",
"http://crl2.microsoft.com/pkiops/crl/partition/Microsoft%20TLS%20G2%20RSA%20CA%20OCSP%2010_Partition00099.crl",
],
PolicyIdentifiers: [
[2, 23, 140, 1, 2, 2],
[1, 3, 6, 1, 4, 1, 311, 76, 509, 1, 1],
],
Policies: [
{
der: [103, 129, 12, 1, 2, 2],
},
{
der: [43, 6, 1, 4, 1, 130, 55, 76, 131, 125, 1, 1],
},
],
InhibitAnyPolicy: 0,
InhibitAnyPolicyZero: false,
InhibitPolicyMapping: 0,
InhibitPolicyMappingZero: false,
RequireExplicitPolicy: 0,
RequireExplicitPolicyZero: false,
PolicyMappings: nil,
},
{
Raw: [48, 130, 7, 184, 48, 130, 5, 160, 160, 3, 2, 1, 2, 2, 19, 51, 0, 0, 0, 15, 51, 32, 101, 55, 238, 66, 174, 79, 0, 0, 0, 0, 0, 15, 48, 13, 6, 9, 42, 134, 72, 134, 247, 13, 1, 1, 12, 5, 0, 48, 81, 49, 11, 48, 9, 6, 3, 85, 4, 6, 19, 2, 85, 83, 49, 30, 48, 28, 6, 3, 85, 4, 10, 19, 21, 77, 105, 99, 114, 111, 115, 111, 102, 116, 32, 67, 111, 114, 112, 111, 114, 97, 116, 105, 111, 110, 49, 34, 48, 32, 6, 3, 85, 4, 3, 19, 25, 77, 105, 99, 114, 111, 115, 111, 102, 116, 32, 84, 76, 83, 32, 82, 83, 65, 32, 82, 111, 111, 116, 32, 71, 50, 48, 30, 23, 13, 50, 53, 48, 56, 48, 49, 50, 48, 48, 51, 48, 51, 90, 23, 13, 50, 57, 48, 54, 48, 51, 50, 48, 48, 51, 48, 51, 90, 48, 87, 49, 11, 48, 9, 6, 3, 85, 4, 6, 19, 2, 85, 83, 49, 30, 48, 28, 6, 3, 85, 4, 10, 19, 21, 77, 105, 99, 114, 111, 115, 111, 102, 116, 32, 67, 111, 114, 112, 111, 114, 97, 116, 105, 111, 110, 49, 40, 48, 38, 6, 3, 85, 4, 3, 19, 31, 77, 105, 99, 114, 111, 115, 111, 102, 116, 32, 84, 76, 83, 32, 71, 50, 32, 82, 83, 65, 32, 67, 65, 32, 79, 67, 83, 80, 32, 49, 48, 48, 130, 2, 34, 48, 13, 6, 9, 42, 134, 72, 134, 247, 13, 1, 1, 1, 5, 0, 3, 130, 2, 15, 0, 48, 130, 2, 10, 2, 130, 2, 1, 0, 240, 12, 79, 91, 95, 252, 66, 28, 180, 206, 63, 245, 7, 43, 166, 131, 126, 201, 157, 121, 207, 246, 233, 172, 165, 119, 195, 55, 169, 250, 247, 7, 245, 45, 152, 248, 17, 102, 235, 66, 42, 14, 210, 209, 6, 25, 246, 119, 19, 95, 132, 236, 214, 81, 147, 108, 33, 227, 14, 133, 166, 255, 210, 236, 253, 36, 130, 189, 4, 249, 239, 141, 227, 225, 197, 221, 117, 208, 4, 154, 78, 20, 54, 25, 251, 45, 18, 163, 21, 192, 243, 111, 106, 68, 213, 113, 108, 241, 75, 152, 27, 121, 48, 247, 156, 23, 191, 125, 50, 9, 57, 91, 54, 140, 160, 8, 187, 64, 44, 150, 41, 238, 124, 189, 14, 248, 236, 109, 206, 193, 78, 118, 193, 88, 46, 78, 50, 154, 199, 124, 203, 199, 191, 183, 93, 247, 53, 84, 120, 204, 177, 135, 186, 5, 197, 53, 1, 112, 4, 195, 232, 93, 248, 177, 180, 223, 71, 205, 76, 102, 154, 232, 103, 212, 158, 30, 160, 33, 31, 51, 71, 5, 59, 68, 8, 207, 107, 136, 82, 89, 50, 120, 245, 65, 142, 37, 46, 208, 2, 13, 1, 207, 118, 247, 45, 52, 228, 103, 157, 104, 177, 128, 182, 77, 82, 179, 32, 151, 43, 165, 83, 140, 119, 149, 181, 115, 128, 205, 136, 255, 218, 151, 54, 227, 231, 111, 50, 160, 117, 241, 187, 191, 250, 60, 101, 171, 54, 17, 191, 118, 253, 71, 255, 108, 135, 96, 190, 12, 229, 234, 150, 245, 11, 162, 55, 39, 0, 209, 231, 2, 48, 1, 213, 128, 1, 194, 144, 206, 239, 179, 235, 211, 120, 96, 190, 10, 63, 251, 108, 198, 194, 209, 171, 39, 23, 23, 9, 132, 121, 145, 66, 142, 118, 104, 55, 17, 155, 206, 199, 118, 41, 191, 213, 92, 226, 124, 13, 95, 103, 82, 253, 109, 57, 250, 81, 45, 223, 178, 136, 3, 241, 97, 32, 170, 248, 180, 178, 159, 211, 250, 40, 176, 181, 167, 227, 131, 186, 250, 68, 59, 119, 220, 162, 44, 43, 232, 226, 83, 136, 154, 251, 12, 32, 222, 162, 123, 78, 128, 144, 113, 27, 124, 151, 212, 219, 252, 61, 154, 193, 161, 222, 234, 231, 174, 76, 229, 174, 175, 118, 172, 135, 205, 100, 188, 162, 52, 26, 211, 102, 195, 107, 97, 124, 155, 245, 230, 89, 121, 251, 128, 14, 22, 107, 192, 241, 41, 244, 237, 157, 151, 142, 58, 196, 41, 147, 7, 52, 255, 226, 191, 173, 154, 89, 37, 155, 134, 29, 67, 126, 203, 110, 149, 245, 142, 224, 169, 139, 87, 128, 78, 11, 230, 96, 125, 192, 73, 213, 4, 188, 32, 202, 4, 210, 92, 234, 63, 140, 73, 127, 29, 185, 186, 96, 109, 215, 180, 100, 179, 92, 52, 234, 56, 243, 249, 193, 248, 169, 11, 126, 186, 245, 209, 252, 122, 27, 127, 219, 8, 101, 125, 195, 151, 5, 236, 175, 47, 199, 222, 12, 250, 178, 63, 2, 3, 1, 0, 1, 163, 130, 2, 129, 48, 130, 2, 125, 48, 14, 6, 3, 85, 29, 15, 1, 1, 255, 4, 4, 3, 2, 1, 134, 48, 16, 6, 9, 43, 6, 1, 4, 1, 130, 55, 21, 1, 4, 3, 2, 1, 0, 48, 29, 6, 3, 85, 29, 14, 4, 22, 4, 20, 208, 76, 131, 209, 142, 113, 46, 61, 52, 177, 255, 155, 121, 44, 142, 116, 212, 226, 120, 6, 48, 19, 6, 3, 85, 29, 32, 4, 12, 48, 10, 48, 8, 6, 6, 103, 129, 12, 1, 2, 2, 48, 19, 6, 3, 85, 29, 37, 4, 12, 48, 10, 6, 8, 43, 6, 1, 5, 5, 7, 3, 1, 48, 25, 6, 9, 43, 6, 1, 4, 1, 130, 55, 20, 2, 4, 12, 30, 10, 0, 83, 0, 117, 0, 98, 0, 67, 0, 65, 48, 18, 6, 3, 85, 29, 19, 1, 1, 255, 4, 8, 48, 6, 1, 1, 255, 2, 1, 0, 48, 31, 6, 3, 85, 29, 35, 4, 24, 48, 22, 128, 20, 222, 145, 134, 72, 183, 161, 49, 89, 49, 241, 75, 95, 7, 169, 220, 136, 121, 218, 168, 118, 48, 129, 171, 6, 3, 85, 29, 31, 4, 129, 163, 48, 129, 160, 48, 129, 157, 160, 129, 154, 160, 129, 151, 134, 73, 104, 116, 116, 112, 58, 47, 47, 119, 119, 119, 46, 109, 105, 99, 114, 111, 115, 111, 102, 116, 46, 99, 111, 109, 47, 112, 107, 105, 111, 112, 115, 47, 99, 114, 108, 47, 77, 105, 99, 114, 111, 115, 111, 102, 116, 37, 50, 48, 84, 76, 83, 37, 50, 48, 82, 83, 65, 37, 50, 48, 82, 111, 111, 116, 37, 50, 48, 71, 50, 46, 99, 114, 108, 134, 74, 104, 116, 116, 112, 58, 47, 47, 99, 114, 108, 50, 46, 109, 105, 99, 114, 111, 115, 111, 102, 116, 46, 99, 111, 109, 47, 112, 107, 105, 111, 112, 115, 47, 99, 114, 108, 47, 77, 105, 99, 114, 111, 115, 111, 102, 116, 37, 50, 48, 84, 76, 83, 37, 50, 48, 82, 83, 65, 37, 50, 48, 82, 111, 111, 116, 37, 50, 48, 71, 50, 46, 99, 114, 108, 48, 130, 1, 16, 6, 8, 43, 6, 1, 5, 5, 7, 1, 1, 4, 130, 1, 2, 48, 129, 255, 48, 99, 6, 8, 43, 6, 1, 5, 5, 7, 48, 2, 134, 87, 104, 116, 116, 112, 58, 47, 47, 119, 119, 119, 46, 109, 105, 99, 114, 111, 115, 111, 102, 116, 46, 99, 111, 109, 47, 112, 107, 105, 111, 112, 115, 47, 99, 101, 114, 116, 115, 47, 77, 105, 99, 114, 111, 115, 111, 102, 116, 37, 50, 48, 84, 76, 83, 37, 50, 48, 82, 83, 65, 37, 50, 48, 82, 111, 111, 116, 37, 50, 48, 71, 50, 37, 50, 48, 45, 37, 50, 48, 120, 115, 105, 103, 110, 46, 99, 114, 116, 48, 105, 6, 8, 43, 6, 1, 5, 5, 7, 48, 2, 134, 93, 104, 116, 116, 112, 58, 47, 47, 99, 97, 105, 115, 115, 117, 101, 114, 115, 46, 109, 105, 99, 114, 111, 115, 111, 102, 116, 46, 99, 111, 109, 47, 112, 107, 105, 111, 112, 115, 47, 99, 101, 114, 116, 115, 47, 77, 105, 99, 114, 111, 115, 111, 102, 116, 37, 50, 48, 84, 76, 83, 37, 50, 48, 82, 83, 65, 37, 50, 48, 82, 111, 111, 116, 37, 50, 48, 71, 50, 37, 50, 48, 45, 37, 50, 48, 120, 115, 105, 103, 110, 46, 99, 114, 116, 48, 45, 6, 8, 43, 6, 1, 5, 5, 7, 48, 1, 134, 33, 104, 116, 116, 112, 58, 47, 47, 111, 110, 101, 111, 99, 115, 112, 46, 109, 105, 99, 114, 111, 115, 111, 102, 116, 46, 99, 111, 109, 47, 111, 99, 115, 112, 48, 13, 6, 9, 42, 134, 72, 134, 247, 13, 1, 1, 12, 5, 0, 3, 130, 2, 1, 0, 53, 25, 202, 233, 168, 117, 225, 216, 202, 253, 37, 193, 59, 82, 225, 231, 158, 43, 148, 226, 244, 58, 210, 79, 218, 6, 34, 82, 26, 1, 24, 227, 162, 186, 183, 183, 28, 36, 50, 117, 13, 103, 68, 125, 157, 74, 130, 75, 250, 227, 68, 233, 70, 246, 0, 69, 9, 194, 82, 233, 103, 37, 151, 199, 183, 139, 90, 119, 124, 64, 30, 148, 227, 188, 140, 109, 56, 41, 188, 39, 125, 18, 18, 61, 203, 203, 228, 196, 70, 226, 165, 1, 192, 16, 209, 32, 250, 14, 3, 228, 67, 219, 38, 135, 37, 172, 200, 160, 179, 4, 106, 94, 240, 67, 63, 15, 123, 245, 206, 35, 195, 183, 201, 190, 251, 222, 155, 125, 147, 179, 135, 62, 223, 198, 219, 61, 201, 244, 172, 113, 0, 210, 164, 101, 129, 54, 89, 159, 28, 54, 229, 137, 209, 122, 153, 74, 123, 85, 75, 194, 213, 140, 208, 121, 1, 121, 149, 15, 57, 67, 139, 179, 10, 123, 146, 213, 90, 16, 24, 118, 233, 29, 13, 111, 50, 49, 244, 189, 209, 22, 241, 168, 163, 103, 242, 14, 154, 112, 12, 227, 150, 175, 3, 244, 109, 189, 20, 121, 106, 54, 195, 241, 182, 115, 187, 198, 11, 213, 38, 70, 136, 84, 69, 112, 170, 204, 31, 61, 139, 111, 227, 228, 23, 166, 129, 0, 251, 125, 131, 88, 128, 226, 242, 60, 59, 163, 24, 98, 19, 54, 103, 255, 79, 226, 229, 111, 106, 103, 230, 232, 90, 100, 58, 160, 69, 157, 63, 23, 12, 102, 135, 240, 129, 39, 52, 32, 176, 53, 36, 100, 40, 118, 67, 122, 116, 152, 226, 43, 128, 9, 10, 188, 176, 185, 7, 71, 41, 40, 46, 125, 224, 119, 65, 164, 140, 54, 120, 186, 51, 29, 86, 142, 217, 16, 53, 137, 89, 131, 0, 105, 206, 183, 120, 189, 237, 175, 130, 63, 206, 200, 137, 50, 152, 89, 8, 93, 225, 1, 150, 88, 196, 101, 25, 125, 38, 133, 157, 16, 150, 229, 149, 197, 17, 206, 130, 111, 23, 6, 113, 21, 15, 44, 191, 55, 50, 82, 21, 104, 180, 149, 142, 83, 93, 151, 237, 227, 219, 19, 62, 193, 49, 177, 121, 147, 115, 125, 114, 59, 27, 76, 79, 6, 73, 21, 132, 115, 243, 29, 90, 81, 44, 164, 212, 88, 224, 25, 191, 53, 72, 41, 96, 153, 171, 234, 123, 66, 3, 190, 156, 61, 25, 247, 71, 91, 11, 172, 130, 87, 170, 13, 189, 255, 34, 96, 48, 223, 44, 79, 87, 211, 37, 213, 218, 253, 142, 6, 81, 78, 5, 53, 120, 152, 56, 163, 39, 49, 111, 49, 154, 169, 170, 77, 172, 58, 68, 104, 96, 112, 104, 244, 83, 107, 205, 178, 45, 174, 38, 24, 135, 223, 140, 87, 181, 248, 82, 193, 86, 231, 192, 177, 77, 95, 10, 85, 107, 20, 43, 84, 251, 61, 169, 198, 189, 96, 7, 248, 176, 244, 75, 227, 238, 171],
RawTBSCertificate: [48, 130, 5, 160, 160, 3, 2, 1, 2, 2, 19, 51, 0, 0, 0, 15, 51, 32, 101, 55, 238, 66, 174, 79, 0, 0, 0, 0, 0, 15, 48, 13, 6, 9, 42, 134, 72, 134, 247, 13, 1, 1, 12, 5, 0, 48, 81, 49, 11, 48, 9, 6, 3, 85, 4, 6, 19, 2, 85, 83, 49, 30, 48, 28, 6, 3, 85, 4, 10, 19, 21, 77, 105, 99, 114, 111, 115, 111, 102, 116, 32, 67, 111, 114, 112, 111, 114, 97, 116, 105, 111, 110, 49, 34, 48, 32, 6, 3, 85, 4, 3, 19, 25, 77, 105, 99, 114, 111, 115, 111, 102, 116, 32, 84, 76, 83, 32, 82, 83, 65, 32, 82, 111, 111, 116, 32, 71, 50, 48, 30, 23, 13, 50, 53, 48, 56, 48, 49, 50, 48, 48, 51, 48, 51, 90, 23, 13, 50, 57, 48, 54, 48, 51, 50, 48, 48, 51, 48, 51, 90, 48, 87, 49, 11, 48, 9, 6, 3, 85, 4, 6, 19, 2, 85, 83, 49, 30, 48, 28, 6, 3, 85, 4, 10, 19, 21, 77, 105, 99, 114, 111, 115, 111, 102, 116, 32, 67, 111, 114, 112, 111, 114, 97, 116, 105, 111, 110, 49, 40, 48, 38, 6, 3, 85, 4, 3, 19, 31, 77, 105, 99, 114, 111, 115, 111, 102, 116, 32, 84, 76, 83, 32, 71, 50, 32, 82, 83, 65, 32, 67, 65, 32, 79, 67, 83, 80, 32, 49, 48, 48, 130, 2, 34, 48, 13, 6, 9, 42, 134, 72, 134, 247, 13, 1, 1, 1, 5, 0, 3, 130, 2, 15, 0, 48, 130, 2, 10, 2, 130, 2, 1, 0, 240, 12, 79, 91, 95, 252, 66, 28, 180, 206, 63, 245, 7, 43, 166, 131, 126, 201, 157, 121, 207, 246, 233, 172, 165, 119, 195, 55, 169, 250, 247, 7, 245, 45, 152, 248, 17, 102, 235, 66, 42, 14, 210, 209, 6, 25, 246, 119, 19, 95, 132, 236, 214, 81, 147, 108, 33, 227, 14, 133, 166, 255, 210, 236, 253, 36, 130, 189, 4, 249, 239, 141, 227, 225, 197, 221, 117, 208, 4, 154, 78, 20, 54, 25, 251, 45, 18, 163, 21, 192, 243, 111, 106, 68, 213, 113, 108, 241, 75, 152, 27, 121, 48, 247, 156, 23, 191, 125, 50, 9, 57, 91, 54, 140, 160, 8, 187, 64, 44, 150, 41, 238, 124, 189, 14, 248, 236, 109, 206, 193, 78, 118, 193, 88, 46, 78, 50, 154, 199, 124, 203, 199, 191, 183, 93, 247, 53, 84, 120, 204, 177, 135, 186, 5, 197, 53, 1, 112, 4, 195, 232, 93, 248, 177, 180, 223, 71, 205, 76, 102, 154, 232, 103, 212, 158, 30, 160, 33, 31, 51, 71, 5, 59, 68, 8, 207, 107, 136, 82, 89, 50, 120, 245, 65, 142, 37, 46, 208, 2, 13, 1, 207, 118, 247, 45, 52, 228, 103, 157, 104, 177, 128, 182, 77, 82, 179, 32, 151, 43, 165, 83, 140, 119, 149, 181, 115, 128, 205, 136, 255, 218, 151, 54, 227, 231, 111, 50, 160, 117, 241, 187, 191, 250, 60, 101, 171, 54, 17, 191, 118, 253, 71, 255, 108, 135, 96, 190, 12, 229, 234, 150, 245, 11, 162, 55, 39, 0, 209, 231, 2, 48, 1, 213, 128, 1, 194, 144, 206, 239, 179, 235, 211, 120, 96, 190, 10, 63, 251, 108, 198, 194, 209, 171, 39, 23, 23, 9, 132, 121, 145, 66, 142, 118, 104, 55, 17, 155, 206, 199, 118, 41, 191, 213, 92, 226, 124, 13, 95, 103, 82, 253, 109, 57, 250, 81, 45, 223, 178, 136, 3, 241, 97, 32, 170, 248, 180, 178, 159, 211, 250, 40, 176, 181, 167, 227, 131, 186, 250, 68, 59, 119, 220, 162, 44, 43, 232, 226, 83, 136, 154, 251, 12, 32, 222, 162, 123, 78, 128, 144, 113, 27, 124, 151, 212, 219, 252, 61, 154, 193, 161, 222, 234, 231, 174, 76, 229, 174, 175, 118, 172, 135, 205, 100, 188, 162, 52, 26, 211, 102, 195, 107, 97, 124, 155, 245, 230, 89, 121, 251, 128, 14, 22, 107, 192, 241, 41, 244, 237, 157, 151, 142, 58, 196, 41, 147, 7, 52, 255, 226, 191, 173, 154, 89, 37, 155, 134, 29, 67, 126, 203, 110, 149, 245, 142, 224, 169, 139, 87, 128, 78, 11, 230, 96, 125, 192, 73, 213, 4, 188, 32, 202, 4, 210, 92, 234, 63, 140, 73, 127, 29, 185, 186, 96, 109, 215, 180, 100, 179, 92, 52, 234, 56, 243, 249, 193, 248, 169, 11, 126, 186, 245, 209, 252, 122, 27, 127, 219, 8, 101, 125, 195, 151, 5, 236, 175, 47, 199, 222, 12, 250, 178, 63, 2, 3, 1, 0, 1, 163, 130, 2, 129, 48, 130, 2, 125, 48, 14, 6, 3, 85, 29, 15, 1, 1, 255, 4, 4, 3, 2, 1, 134, 48, 16, 6, 9, 43, 6, 1, 4, 1, 130, 55, 21, 1, 4, 3, 2, 1, 0, 48, 29, 6, 3, 85, 29, 14, 4, 22, 4, 20, 208, 76, 131, 209, 142, 113, 46, 61, 52, 177, 255, 155, 121, 44, 142, 116, 212, 226, 120, 6, 48, 19, 6, 3, 85, 29, 32, 4, 12, 48, 10, 48, 8, 6, 6, 103, 129, 12, 1, 2, 2, 48, 19, 6, 3, 85, 29, 37, 4, 12, 48, 10, 6, 8, 43, 6, 1, 5, 5, 7, 3, 1, 48, 25, 6, 9, 43, 6, 1, 4, 1, 130, 55, 20, 2, 4, 12, 30, 10, 0, 83, 0, 117, 0, 98, 0, 67, 0, 65, 48, 18, 6, 3, 85, 29, 19, 1, 1, 255, 4, 8, 48, 6, 1, 1, 255, 2, 1, 0, 48, 31, 6, 3, 85, 29, 35, 4, 24, 48, 22, 128, 20, 222, 145, 134, 72, 183, 161, 49, 89, 49, 241, 75, 95, 7, 169, 220, 136, 121, 218, 168, 118, 48, 129, 171, 6, 3, 85, 29, 31, 4, 129, 163, 48, 129, 160, 48, 129, 157, 160, 129, 154, 160, 129, 151, 134, 73, 104, 116, 116, 112, 58, 47, 47, 119, 119, 119, 46, 109, 105, 99, 114, 111, 115, 111, 102, 116, 46, 99, 111, 109, 47, 112, 107, 105, 111, 112, 115, 47, 99, 114, 108, 47, 77, 105, 99, 114, 111, 115, 111, 102, 116, 37, 50, 48, 84, 76, 83, 37, 50, 48, 82, 83, 65, 37, 50, 48, 82, 111, 111, 116, 37, 50, 48, 71, 50, 46, 99, 114, 108, 134, 74, 104, 116, 116, 112, 58, 47, 47, 99, 114, 108, 50, 46, 109, 105, 99, 114, 111, 115, 111, 102, 116, 46, 99, 111, 109, 47, 112, 107, 105, 111, 112, 115, 47, 99, 114, 108, 47, 77, 105, 99, 114, 111, 115, 111, 102, 116, 37, 50, 48, 84, 76, 83, 37, 50, 48, 82, 83, 65, 37, 50, 48, 82, 111, 111, 116, 37, 50, 48, 71, 50, 46, 99, 114, 108, 48, 130, 1, 16, 6, 8, 43, 6, 1, 5, 5, 7, 1, 1, 4, 130, 1, 2, 48, 129, 255, 48, 99, 6, 8, 43, 6, 1, 5, 5, 7, 48, 2, 134, 87, 104, 116, 116, 112, 58, 47, 47, 119, 119, 119, 46, 109, 105, 99, 114, 111, 115, 111, 102, 116, 46, 99, 111, 109, 47, 112, 107, 105, 111, 112, 115, 47, 99, 101, 114, 116, 115, 47, 77, 105, 99, 114, 111, 115, 111, 102, 116, 37, 50, 48, 84, 76, 83, 37, 50, 48, 82, 83, 65, 37, 50, 48, 82, 111, 111, 116, 37, 50, 48, 71, 50, 37, 50, 48, 45, 37, 50, 48, 120, 115, 105, 103, 110, 46, 99, 114, 116, 48, 105, 6, 8, 43, 6, 1, 5, 5, 7, 48, 2, 134, 93, 104, 116, 116, 112, 58, 47, 47, 99, 97, 105, 115, 115, 117, 101, 114, 115, 46, 109, 105, 99, 114, 111, 115, 111, 102, 116, 46, 99, 111, 109, 47, 112, 107, 105, 111, 112, 115, 47, 99, 101, 114, 116, 115, 47, 77, 105, 99, 114, 111, 115, 111, 102, 116, 37, 50, 48, 84, 76, 83, 37, 50, 48, 82, 83, 65, 37, 50, 48, 82, 111, 111, 116, 37, 50, 48, 71, 50, 37, 50, 48, 45, 37, 50, 48, 120, 115, 105, 103, 110, 46, 99, 114, 116, 48, 45, 6, 8, 43, 6, 1, 5, 5, 7, 48, 1, 134, 33, 104, 116, 116, 112, 58, 47, 47, 111, 110, 101, 111, 99, 115, 112, 46, 109, 105, 99, 114, 111, 115, 111, 102, 116, 46, 99, 111, 109, 47, 111, 99, 115, 112],
RawSubjectPublicKeyInfo: [48, 130, 2, 34, 48, 13, 6, 9, 42, 134, 72, 134, 247, 13, 1, 1, 1, 5, 0, 3, 130, 2, 15, 0, 48, 130, 2, 10, 2, 130, 2, 1, 0, 240, 12, 79, 91, 95, 252, 66, 28, 180, 206, 63, 245, 7, 43, 166, 131, 126, 201, 157, 121, 207, 246, 233, 172, 165, 119, 195, 55, 169, 250, 247, 7, 245, 45, 152, 248, 17, 102, 235, 66, 42, 14, 210, 209, 6, 25, 246, 119, 19, 95, 132, 236, 214, 81, 147, 108, 33, 227, 14, 133, 166, 255, 210, 236, 253, 36, 130, 189, 4, 249, 239, 141, 227, 225, 197, 221, 117, 208, 4, 154, 78, 20, 54, 25, 251, 45, 18, 163, 21, 192, 243, 111, 106, 68, 213, 113, 108, 241, 75, 152, 27, 121, 48, 247, 156, 23, 191, 125, 50, 9, 57, 91, 54, 140, 160, 8, 187, 64, 44, 150, 41, 238, 124, 189, 14, 248, 236, 109, 206, 193, 78, 118, 193, 88, 46, 78, 50, 154, 199, 124, 203, 199, 191, 183, 93, 247, 53, 84, 120, 204, 177, 135, 186, 5, 197, 53, 1, 112, 4, 195, 232, 93, 248, 177, 180, 223, 71, 205, 76, 102, 154, 232, 103, 212, 158, 30, 160, 33, 31, 51, 71, 5, 59, 68, 8, 207, 107, 136, 82, 89, 50, 120, 245, 65, 142, 37, 46, 208, 2, 13, 1, 207, 118, 247, 45, 52, 228, 103, 157, 104, 177, 128, 182, 77, 82, 179, 32, 151, 43, 165, 83, 140, 119, 149, 181, 115, 128, 205, 136, 255, 218, 151, 54, 227, 231, 111, 50, 160, 117, 241, 187, 191, 250, 60, 101, 171, 54, 17, 191, 118, 253, 71, 255, 108, 135, 96, 190, 12, 229, 234, 150, 245, 11, 162, 55, 39, 0, 209, 231, 2, 48, 1, 213, 128, 1, 194, 144, 206, 239, 179, 235, 211, 120, 96, 190, 10, 63, 251, 108, 198, 194, 209, 171, 39, 23, 23, 9, 132, 121, 145, 66, 142, 118, 104, 55, 17, 155, 206, 199, 118, 41, 191, 213, 92, 226, 124, 13, 95, 103, 82, 253, 109, 57, 250, 81, 45, 223, 178, 136, 3, 241, 97, 32, 170, 248, 180, 178, 159, 211, 250, 40, 176, 181, 167, 227, 131, 186, 250, 68, 59, 119, 220, 162, 44, 43, 232, 226, 83, 136, 154, 251, 12, 32, 222, 162, 123, 78, 128, 144, 113, 27, 124, 151, 212, 219, 252, 61, 154, 193, 161, 222, 234, 231, 174, 76, 229, 174, 175, 118, 172, 135, 205, 100, 188, 162, 52, 26, 211, 102, 195, 107, 97, 124, 155, 245, 230, 89, 121, 251, 128, 14, 22, 107, 192, 241, 41, 244, 237, 157, 151, 142, 58, 196, 41, 147, 7, 52, 255, 226, 191, 173, 154, 89, 37, 155, 134, 29, 67, 126, 203, 110, 149, 245, 142, 224, 169, 139, 87, 128, 78, 11, 230, 96, 125, 192, 73, 213, 4, 188, 32, 202, 4, 210, 92, 234, 63, 140, 73, 127, 29, 185, 186, 96, 109, 215, 180, 100, 179, 92, 52, 234, 56, 243, 249, 193, 248, 169, 11, 126, 186, 245, 209, 252, 122, 27, 127, 219, 8, 101, 125, 195, 151, 5, 236, 175, 47, 199, 222, 12, 250, 178, 63, 2, 3, 1, 0, 1],
RawSubject: [48, 87, 49, 11, 48, 9, 6, 3, 85, 4, 6, 19, 2, 85, 83, 49, 30, 48, 28, 6, 3, 85, 4, 10, 19, 21, 77, 105, 99, 114, 111, 115, 111, 102, 116, 32, 67, 111, 114, 112, 111, 114, 97, 116, 105, 111, 110, 49, 40, 48, 38, 6, 3, 85, 4, 3, 19, 31, 77, 105, 99, 114, 111, 115, 111, 102, 116, 32, 84, 76, 83, 32, 71, 50, 32, 82, 83, 65, 32, 67, 65, 32, 79, 67, 83, 80, 32, 49, 48],
RawIssuer: [48, 81, 49, 11, 48, 9, 6, 3, 85, 4, 6, 19, 2, 85, 83, 49, 30, 48, 28, 6, 3, 85, 4, 10, 19, 21, 77, 105, 99, 114, 111, 115, 111, 102, 116, 32, 67, 111, 114, 112, 111, 114, 97, 116, 105, 111, 110, 49, 34, 48, 32, 6, 3, 85, 4, 3, 19, 25, 77, 105, 99, 114, 111, 115, 111, 102, 116, 32, 84, 76, 83, 32, 82, 83, 65, 32, 82, 111, 111, 116, 32, 71, 50],
Signature: [53, 25, 202, 233, 168, 117, 225, 216, 202, 253, 37, 193, 59, 82, 225, 231, 158, 43, 148, 226, 244, 58, 210, 79, 218, 6, 34, 82, 26, 1, 24, 227, 162, 186, 183, 183, 28, 36, 50, 117, 13, 103, 68, 125, 157, 74, 130, 75, 250, 227, 68, 233, 70, 246, 0, 69, 9, 194, 82, 233, 103, 37, 151, 199, 183, 139, 90, 119, 124, 64, 30, 148, 227, 188, 140, 109, 56, 41, 188, 39, 125, 18, 18, 61, 203, 203, 228, 196, 70, 226, 165, 1, 192, 16, 209, 32, 250, 14, 3, 228, 67, 219, 38, 135, 37, 172, 200, 160, 179, 4, 106, 94, 240, 67, 63, 15, 123, 245, 206, 35, 195, 183, 201, 190, 251, 222, 155, 125, 147, 179, 135, 62, 223, 198, 219, 61, 201, 244, 172, 113, 0, 210, 164, 101, 129, 54, 89, 159, 28, 54, 229, 137, 209, 122, 153, 74, 123, 85, 75, 194, 213, 140, 208, 121, 1, 121, 149, 15, 57, 67, 139, 179, 10, 123, 146, 213, 90, 16, 24, 118, 233, 29, 13, 111, 50, 49, 244, 189, 209, 22, 241, 168, 163, 103, 242, 14, 154, 112, 12, 227, 150, 175, 3, 244, 109, 189, 20, 121, 106, 54, 195, 241, 182, 115, 187, 198, 11, 213, 38, 70, 136, 84, 69, 112, 170, 204, 31, 61, 139, 111, 227, 228, 23, 166, 129, 0, 251, 125, 131, 88, 128, 226, 242, 60, 59, 163, 24, 98, 19, 54, 103, 255, 79, 226, 229, 111, 106, 103, 230, 232, 90, 100, 58, 160, 69, 157, 63, 23, 12, 102, 135, 240, 129, 39, 52, 32, 176, 53, 36, 100, 40, 118, 67, 122, 116, 152, 226, 43, 128, 9, 10, 188, 176, 185, 7, 71, 41, 40, 46, 125, 224, 119, 65, 164, 140, 54, 120, 186, 51, 29, 86, 142, 217, 16, 53, 137, 89, 131, 0, 105, 206, 183, 120, 189, 237, 175, 130, 63, 206, 200, 137, 50, 152, 89, 8, 93, 225, 1, 150, 88, 196, 101, 25, 125, 38, 133, 157, 16, 150, 229, 149, 197, 17, 206, 130, 111, 23, 6, 113, 21, 15, 44, 191, 55, 50, 82, 21, 104, 180, 149, 142, 83, 93, 151, 237, 227, 219, 19, 62, 193, 49, 177, 121, 147, 115, 125, 114, 59, 27, 76, 79, 6, 73, 21, 132, 115, 243, 29, 90, 81, 44, 164, 212, 88, 224, 25, 191, 53, 72, 41, 96, 153, 171, 234, 123, 66, 3, 190, 156, 61, 25, 247, 71, 91, 11, 172, 130, 87, 170, 13, 189, 255, 34, 96, 48, 223, 44, 79, 87, 211, 37, 213, 218, 253, 142, 6, 81, 78, 5, 53, 120, 152, 56, 163, 39, 49, 111, 49, 154, 169, 170, 77, 172, 58, 68, 104, 96, 112, 104, 244, 83, 107, 205, 178, 45, 174, 38, 24, 135, 223, 140, 87, 181, 248, 82, 193, 86, 231, 192, 177, 77, 95, 10, 85, 107, 20, 43, 84, 251, 61, 169, 198, 189, 96, 7, 248, 176, 244, 75, 227, 238, 171],
SignatureAlgorithm: 5,
PublicKeyAlgorithm: 1,
PublicKey: <*rsa.PublicKey | 0x36a118ac3670>{
N: {
neg: false,
abs: [12623528037057212991, 15782976484946675180, 9131881476679932799, 16877507857597638923, 6948446969886104628, 16879364175701064122, 15349600278999781980, 9245340169127444553, 7968545109705329495, 6423711443832766155, 10594494922037898650, 17648935788986680361, 18122500389998752041, 7737602644086511993, 7258855044086458051, 5541026985182791629, 4439073283700877230, 10408130435767131132, 18089869943821258368, 11685763412349126810, 16394152652608075740, 12871239331426121127, 9800942613550528692, 18261315811323797426, 15374412250364208978, 3968124059102095807, 685806708061009512, 7838166407260346135, 16993058175899549691, 15384298262262378419, 3974146097701597185, 13694573362644585378, 13796492994165901152, 13528806942954960401, 3955259362538190321, 13074935763441801879, 2348393719768315797, 11342510827905045171, 130453718204474471, 17672562703128855053, 634844305723830904, 11538538025395239748, 5505257917472087582, 16743812431760803789, 13404336697386206403, 6770939201695035783, 3646446187118575543, 14898275314275200590, 3021489551454039149, 3930692534149328022, 11247669138812385627, 7850138742151524599, 1567520330174748017, 5626181319708709539, 16420623271324550298, 18240848139101269901, 2441811390160163564, 1395980562250961772, 3030591394459285111, 17666945104455592770, 11923213182329091847, 9136006465603430828, 13028421093649262211, 17297287522694218268],
},
E: 65537,
},
Version: 3,
SerialNumber: {
neg: false,
abs: [12560257885759602703, 4278338880794178, 3342336],
},
Issuer: {
Country: ["US"],
Organization: [
"Microsoft Corporation",
],
OrganizationalUnit: nil,
Locality: nil,
Province: nil,
StreetAddress: nil,
PostalCode: nil,
SerialNumber: "",
CommonName: "Microsoft TLS RSA Root G2",
Names: [
{
Type: [..., ..., ..., ...],
Value: <string>"US",
},
{
Type: [..., ..., ..., ...],
Value: <string>"Microsoft Corporation",
},
{
Type: [..., ..., ..., ...],
Value: <string>"Microsoft TLS RSA Root G2",
},
],
ExtraNames: nil,
},
Subject: {
Country: ["US"],
Organization: [
"Microsoft Corporation",
],
OrganizationalUnit: nil,
Locality: nil,
Province: nil,
StreetAddress: nil,
PostalCode: nil,
SerialNumber: "",
CommonName: "Microsoft TLS G2 RSA CA OCSP 10",
Names: [
{
Type: [..., ..., ..., ...],
Value: <string>"US",
},
{
Type: [..., ..., ..., ...],
Value: <string>"Microsoft Corporation",
},
{
Type: [..., ..., ..., ...],
Value: <string>"Microsoft TLS G2 RSA CA OCSP 10",
},
],
ExtraNames: nil,
},
NotBefore: {wall: 0, ext: 63889675383, loc: nil},
NotAfter: {wall: 0, ext: 64010808183, loc: nil},
KeyUsage: 97,
Extensions: [
{
Id: [2, 5, 29, 15],
Critical: true,
Value: [3, 2, 1, 134],
},
{
Id: [1, 3, 6, 1, 4, 1, 311, 21, 1],
Critical: false,
Value: [2, 1, 0],
},
{
Id: [2, 5, 29, 14],
Critical: false,
Value: [4, 20, 208, 76, 131, 209, 142, 113, 46, 61, 52, 177, 255, 155, 121, 44, 142, 116, 212, 226, 120, 6],
},
{
Id: [2, 5, 29, 32],
Critical: false,
Value: [48, 10, 48, 8, 6, 6, 103, 129, 12, 1, 2, 2],
},
{
Id: [2, 5, 29, 37],
Critical: false,
Value: [48, 10, 6, 8, 43, 6, 1, 5, 5, 7, 3, 1],
},
{
Id: [1, 3, 6, 1, 4, 1, 311, 20, 2],
Critical: false,
Value: [30, 10, 0, 83, 0, 117, 0, 98, 0, 67, 0, 65],
},
{
Id: [2, 5, 29, 19],
Critical: true,
Value: [48, 6, 1, 1, 255, 2, 1, 0],
},
{
Id: [2, 5, 29, 35],
Critical: false,
Value: [48, 22, 128, 20, 222, 145, 134, 72, 183, 161, 49, 89, 49, 241, 75, 95, 7, 169, 220, 136, 121, 218, 168, 118],
},
{
Id: [2, 5, 29, 31],
Critical: false,
Value: "0\x81\xa00\x81\x9d\xa0\x81\x9a\xa0\x81\x97\x86Ihttp://www.microsoft.com/pkiops/crl/Microsoft%20TLS%20RSA%20Root%20G2.crl\x86Jhttp://crl2.microsoft.com/pkiops/crl/Microsoft%20TLS%20RSA%20Root%20G2.crl",
},
{
Id: [1, 3, 6, 1, 5, 5, 7, 1, 1],
Critical: false,
Value: [48, 129, 255, 48, 99, 6, 8, 43, 6, 1, 5, 5, 7, 48, 2, 134, 87, 104, 116, 116, 112, 58, 47, 47, 119, 119, 119, 46, 109, 105, 99, 114, 111, 115, 111, 102, 116, 46, 99, 111, 109, 47, 112, 107, 105, 111, 112, 115, 47, 99, 101, 114, 116, 115, 47, 77, 105, 99, 114, 111, 115, 111, 102, 116, 37, 50, 48, 84, 76, 83, 37, 50, 48, 82, 83, 65, 37, 50, 48, 82, 111, 111, 116, 37, 50, 48, 71, 50, 37, 50, 48, 45, 37, 50, 48, 120, 115, 105, 103, 110, 46, 99, 114, 116, 48, 105, 6, 8, 43, 6, 1, 5, 5, 7, 48, 2, 134, 93, 104, 116, 116, 112, 58, 47, 47, 99, 97, 105, 115, 115, 117, 101, 114, 115, 46, 109, 105, 99, 114, 111, 115, 111, 102, 116, 46, 99, 111, 109, 47, 112, 107, 105, 111, 112, 115, 47, 99, 101, 114, 116, 115, 47, 77, 105, 99, 114, 111, 115, 111, 102, 116, 37, 50, 48, 84, 76, 83, 37, 50, 48, 82, 83, 65, 37, 50, 48, 82, 111, 111, 116, 37, 50, 48, 71, 50, 37, 50, 48, 45, 37, 50, 48, 120, 115, 105, 103, 110, 46, 99, 114, 116, 48, 45, 6, 8, 43, 6, 1, 5, 5, 7, 48, 1, 134, 33, 104, 116, 116, 112, 58, 47, 47, 111, 110, 101, 111, 99, 115, 112, 46, 109, 105, 99, 114, 111, 115, 111, 102, 116, 46, 99, 111, 109, 47, 111, 99, 115, 112],
},
],
ExtraExtensions: nil,
UnhandledCriticalExtensions: nil,
ExtKeyUsage: [1],
UnknownExtKeyUsage: nil,
BasicConstraintsValid: true,
IsCA: true,
MaxPathLen: 0,
MaxPathLenZero: true,
SubjectKeyId: [208, 76, 131, 209, 142, 113, 46, 61, 52, 177, 255, 155, 121, 44, 142, 116, 212, 226, 120, 6],
AuthorityKeyId: [222, 145, 134, 72, 183, 161, 49, 89, 49, 241, 75, 95, 7, 169, 220, 136, 121, 218, 168, 118],
OCSPServer: [
"http://oneocsp.microsoft.com/ocsp",
],
IssuingCertificateURL: [
"http://www.microsoft.com/pkiops/certs/Microsoft%20TLS%20RSA%20Root%20G2%20-%20xsign.crt",
"http://caissuers.microsoft.com/pkiops/certs/Microsoft%20TLS%20RSA%20Root%20G2%20-%20xsign.crt",
],
DNSNames: nil,
EmailAddresses: nil,
IPAddresses: nil,
URIs: nil,
PermittedDNSDomainsCritical: false,
PermittedDNSDomains: nil,
ExcludedDNSDomains: nil,
PermittedIPRanges: nil,
ExcludedIPRanges: nil,
PermittedEmailAddresses: nil,
ExcludedEmailAddresses: nil,
PermittedURIDomains: nil,
ExcludedURIDomains: nil,
CRLDistributionPoints: [
"http://www.microsoft.com/pkiops/crl/Microsoft%20TLS%20RSA%20Root%20G2.crl",
"http://crl2.microsoft.com/pkiops/crl/Microsoft%20TLS%20RSA%20Root%20G2.crl",
],
PolicyIdentifiers: [
[2, 23, 140, 1, 2, 2],
],
Policies: [
{
der: [103, 129, 12, 1, 2, 2],
},
],
InhibitAnyPolicy: 0,
InhibitAnyPolicyZero: false,
InhibitPolicyMapping: 0,
InhibitPolicyMappingZero: false,
RequireExplicitPolicy: 0,
RequireExplicitPolicyZero: false,
PolicyMappings: nil,
},
{
Raw: [48, 130, 5, 137, 48, 130, 4, 113, 160, 3, 2, 1, 2, 2, 16, 11, 12, 107, 44, 70, 105, 23, 176, 71, 115, 198, 71, 212, 175, 192, 200, 48, 13, 6, 9, 42, 134, 72, 134, 247, 13, 1, 1, 12, 5, 0, 48, 97, 49, 11, 48, 9, 6, 3, 85, 4, 6, 19, 2, 85, 83, 49, 21, 48, 19, 6, 3, 85, 4, 10, 19, 12, 68, 105, 103, 105, 67, 101, 114, 116, 32, 73, 110, 99, 49, 25, 48, 23, 6, 3, 85, 4, 11, 19, 16, 119, 119, 119, 46, 100, 105, 103, 105, 99, 101, 114, 116, 46, 99, 111, 109, 49, 32, 48, 30, 6, 3, 85, 4, 3, 19, 23, 68, 105, 103, 105, 67, 101, 114, 116, 32, 71, 108, 111, 98, 97, 108, 32, 82, 111, 111, 116, 32, 71, 50, 48, 30, 23, 13, 50, 53, 48, 53, 50, 49, 48, 48, 48, 48, 48, 48, 90, 23, 13, 50, 57, 48, 54, 49, 57, 50, 51, 53, 57, 53, 57, 90, 48, 81, 49, 11, 48, 9, 6, 3, 85, 4, 6, 19, 2, 85, 83, 49, 30, 48, 28, 6, 3, 85, 4, 10, 19, 21, 77, 105, 99, 114, 111, 115, 111, 102, 116, 32, 67, 111, 114, 112, 111, 114, 97, 116, 105, 111, 110, 49, 34, 48, 32, 6, 3, 85, 4, 3, 19, 25, 77, 105, 99, 114, 111, 115, 111, 102, 116, 32, 84, 76, 83, 32, 82, 83, 65, 32, 82, 111, 111, 116, 32, 71, 50, 48, 130, 2, 34, 48, 13, 6, 9, 42, 134, 72, 134, 247, 13, 1, 1, 1, 5, 0, 3, 130, 2, 15, 0, 48, 130, 2, 10, 2, 130, 2, 1, 0, 223, 234, 139, 159, 71, 225, 40, 16, 113, 175, 65, 214, 25, 219, 146, 87, 222, 107, 171, 230, 46, 222, 173, 58, 96, 131, 183, 19, 152, 172, 91, 185, 50, 11, 123, 235, 159, 114, 39, 107, 93, 64, 171, 190, 24, 6, 147, 178, 13, 210, 18, 237, 184, 116, 245, 127, 47, 234, 168, 23, 148, 246, 51, 22, 87, 138, 222, 32, 124, 33, 221, 183, 218, 172, 222, 78, 100, 176, 90, 205, 75, 117, 243, 239, 2, 100, 77, 237, 82, 218, 163, 143, 4, 232, 145, 248, 215, 251, 63, 25, 253, 78, 69, 57, 94, 232, 41, 195, 214, 242, 103, 175, 179, 0, 148, 193, 141, 225, 240, 74, 24, 152, 57, 250, 192, 9, 132, 51, 65, 5, 3, 5, 171, 244, 117, 225, 83, 165, 33, 146, 92, 68, 76, 230, 96, 87, 164, 144, 2, 151, 199, 213, 216, 147, 4, 50, 74, 237, 133, 81, 237, 214, 197, 186, 160, 63, 54, 138, 47, 111, 76, 57, 168, 4, 105, 34, 132, 59, 117, 20, 247, 217, 62, 180, 169, 208, 7, 65, 56, 129, 126, 70, 72, 29, 232, 25, 230, 112, 184, 160, 170, 24, 221, 114, 191, 126, 125, 181, 108, 78, 25, 112, 17, 132, 4, 19, 129, 212, 98, 253, 156, 107, 241, 28, 221, 255, 236, 167, 224, 230, 199, 141, 49, 75, 191, 242, 107, 51, 204, 209, 140, 182, 254, 177, 27, 163, 74, 168, 95, 14, 127, 49, 118, 56, 63, 7, 71, 108, 1, 168, 172, 165, 20, 218, 130, 176, 95, 31, 108, 188, 137, 26, 192, 18, 42, 203, 123, 151, 31, 40, 46, 49, 22, 138, 83, 76, 117, 102, 249, 125, 1, 214, 194, 96, 170, 101, 143, 49, 158, 114, 203, 204, 155, 45, 229, 68, 25, 147, 87, 171, 110, 142, 194, 217, 46, 101, 176, 114, 45, 231, 161, 195, 161, 126, 73, 29, 89, 103, 214, 9, 212, 24, 18, 109, 192, 149, 69, 120, 201, 241, 96, 106, 228, 246, 238, 6, 36, 59, 115, 74, 90, 17, 194, 62, 151, 97, 192, 120, 238, 239, 19, 167, 48, 226, 74, 56, 45, 238, 88, 23, 19, 32, 183, 54, 42, 223, 11, 50, 180, 194, 56, 177, 145, 221, 18, 176, 69, 182, 124, 192, 177, 29, 98, 161, 157, 221, 113, 18, 21, 150, 1, 157, 10, 9, 245, 177, 221, 179, 217, 224, 56, 66, 52, 246, 105, 182, 195, 175, 22, 115, 254, 212, 67, 248, 67, 228, 88, 249, 79, 179, 243, 236, 214, 61, 148, 58, 64, 56, 21, 171, 12, 134, 208, 100, 206, 26, 99, 232, 132, 148, 85, 9, 90, 182, 4, 182, 209, 62, 154, 21, 199, 100, 17, 128, 208, 27, 130, 14, 77, 255, 69, 158, 178, 101, 7, 37, 31, 219, 154, 200, 198, 236, 214, 91, 133, 80, 4, 159, 28, 203, 93, 32, 130, 103, 158, 121, 226, 223, 232, 242, 232, 111, 196, 32, 114, 48, 134, 229, 157, 117, 39, 53, 2, 3, 1, 0, 1, 163, 130, 1, 75, 48, 130, 1, 71, 48, 15, 6, 3, 85, 29, 19, 1, 1, 255, 4, 5, 48, 3, 1, 1, 255, 48, 29, 6, 3, 85, 29, 14, 4, 22, 4, 20, 222, 145, 134, 72, 183, 161, 49, 89, 49, 241, 75, 95, 7, 169, 220, 136, 121, 218, 168, 118, 48, 31, 6, 3, 85, 29, 35, 4, 24, 48, 22, 128, 20, 78, 34, 84, 32, 24, 149, 230, 227, 110, 230, 15, 250, 250, 185, 18, 237, 6, 23, 143, 57, 48, 14, 6, 3, 85, 29, 15, 1, 1, 255, 4, 4, 3, 2, 1, 134, 48, 19, 6, 3, 85, 29, 37, 4, 12, 48, 10, 6, 8, 43, 6, 1, 5, 5, 7, 3, 1, 48, 118, 6, 8, 43, 6, 1, 5, 5, 7, 1, 1, 4, 106, 48, 104, 48, 36, 6, 8, 43, 6, 1, 5, 5, 7, 48, 1, 134, 24, 104, 116, 116, 112, 58, 47, 47, 111, 99, 115, 112, 46, 100, 105, 103, 105, 99, 101, 114, 116, 46, 99, 111, 109, 48, 64, 6, 8, 43, 6, 1, 5, 5, 7, 48, 2, 134, 52, 104, 116, 116, 112, 58, 47, 47, 99, 97, 99, 101, 114, 116, 115, 46, 100, 105, 103, 105, 99, 101, 114, 116, 46, 99, 111, 109, 47, 68, 105, 103, 105, 67, 101, 114, 116, 71, 108, 111, 98, 97, 108, 82, 111, 111, 116, 71, 50, 46, 99, 114, 116, 48, 66, 6, 3, 85, 29, 31, 4, 59, 48, 57, 48, 55, 160, 53, 160, 51, 134, 49, 104, 116, 116, 112, 58, 47, 47, 99, 114, 108, 51, 46, 100, 105, 103, 105, 99, 101, 114, 116, 46, 99, 111, 109, 47, 68, 105, 103, 105, 67, 101, 114, 116, 71, 108, 111, 98, 97, 108, 82, 111, 111, 116, 71, 50, 46, 99, 114, 108, 48, 19, 6, 3, 85, 29, 32, 4, 12, 48, 10, 48, 8, 6, 6, 103, 129, 12, 1, 2, 2, 48, 13, 6, 9, 42, 134, 72, 134, 247, 13, 1, 1, 12, 5, 0, 3, 130, 1, 1, 0, 11, 188, 180, 43, 55, 116, 197, 75, 167, 54, 2, 54, 198, 175, 225, 19, 204, 138, 154, 151, 27, 252, 227, 68, 140, 238, 84, 0, 229, 228, 65, 26, 70, 240, 11, 118, 52, 255, 153, 85, 98, 53, 170, 173, 138, 156, 12, 90, 98, 204, 67, 192, 203, 234, 74, 29, 189, 107, 29, 175, 177, 25, 37, 102, 156, 247, 121, 110, 91, 37, 128, 36, 129, 133, 199, 121, 45, 53, 95, 66, 9, 120, 237, 159, 226, 101, 213, 196, 32, 80, 130, 16, 143, 106, 58, 237, 24, 142, 72, 137, 171, 166, 9, 248, 225, 128, 252, 6, 117, 22, 39, 37, 90, 143, 184, 58, 112, 108, 24, 71, 240, 99, 244, 83, 14, 217, 21, 53, 32, 243, 145, 61, 19, 1, 160, 226, 250, 36, 8, 69, 86, 119, 209, 103, 169, 49, 66, 208, 87, 255, 26, 38, 190, 197, 67, 216, 87, 169, 99, 255, 175, 243, 58, 216, 246, 123, 75, 16, 162, 151, 161, 207, 102, 190, 103, 103, 9, 249, 181, 76, 231, 171, 25, 43, 191, 34, 61, 125, 222, 198, 247, 239, 99, 56, 44, 221, 86, 196, 249, 9, 82, 218, 223, 227, 136, 111, 213, 109, 107, 247, 8, 65, 47, 142, 62, 134, 49, 159, 241, 124, 74, 245, 214, 97, 164, 70, 70, 40, 239, 87, 186, 176, 111, 146, 216, 210, 119, 37, 237, 123, 166, 41, 206, 59, 114, 121, 17, 212, 224, 167, 84, 109, 53, 43, 164, 122],
RawTBSCertificate: [48, 130, 4, 113, 160, 3, 2, 1, 2, 2, 16, 11, 12, 107, 44, 70, 105, 23, 176, 71, 115, 198, 71, 212, 175, 192, 200, 48, 13, 6, 9, 42, 134, 72, 134, 247, 13, 1, 1, 12, 5, 0, 48, 97, 49, 11, 48, 9, 6, 3, 85, 4, 6, 19, 2, 85, 83, 49, 21, 48, 19, 6, 3, 85, 4, 10, 19, 12, 68, 105, 103, 105, 67, 101, 114, 116, 32, 73, 110, 99, 49, 25, 48, 23, 6, 3, 85, 4, 11, 19, 16, 119, 119, 119, 46, 100, 105, 103, 105, 99, 101, 114, 116, 46, 99, 111, 109, 49, 32, 48, 30, 6, 3, 85, 4, 3, 19, 23, 68, 105, 103, 105, 67, 101, 114, 116, 32, 71, 108, 111, 98, 97, 108, 32, 82, 111, 111, 116, 32, 71, 50, 48, 30, 23, 13, 50, 53, 48, 53, 50, 49, 48, 48, 48, 48, 48, 48, 90, 23, 13, 50, 57, 48, 54, 49, 57, 50, 51, 53, 57, 53, 57, 90, 48, 81, 49, 11, 48, 9, 6, 3, 85, 4, 6, 19, 2, 85, 83, 49, 30, 48, 28, 6, 3, 85, 4, 10, 19, 21, 77, 105, 99, 114, 111, 115, 111, 102, 116, 32, 67, 111, 114, 112, 111, 114, 97, 116, 105, 111, 110, 49, 34, 48, 32, 6, 3, 85, 4, 3, 19, 25, 77, 105, 99, 114, 111, 115, 111, 102, 116, 32, 84, 76, 83, 32, 82, 83, 65, 32, 82, 111, 111, 116, 32, 71, 50, 48, 130, 2, 34, 48, 13, 6, 9, 42, 134, 72, 134, 247, 13, 1, 1, 1, 5, 0, 3, 130, 2, 15, 0, 48, 130, 2, 10, 2, 130, 2, 1, 0, 223, 234, 139, 159, 71, 225, 40, 16, 113, 175, 65, 214, 25, 219, 146, 87, 222, 107, 171, 230, 46, 222, 173, 58, 96, 131, 183, 19, 152, 172, 91, 185, 50, 11, 123, 235, 159, 114, 39, 107, 93, 64, 171, 190, 24, 6, 147, 178, 13, 210, 18, 237, 184, 116, 245, 127, 47, 234, 168, 23, 148, 246, 51, 22, 87, 138, 222, 32, 124, 33, 221, 183, 218, 172, 222, 78, 100, 176, 90, 205, 75, 117, 243, 239, 2, 100, 77, 237, 82, 218, 163, 143, 4, 232, 145, 248, 215, 251, 63, 25, 253, 78, 69, 57, 94, 232, 41, 195, 214, 242, 103, 175, 179, 0, 148, 193, 141, 225, 240, 74, 24, 152, 57, 250, 192, 9, 132, 51, 65, 5, 3, 5, 171, 244, 117, 225, 83, 165, 33, 146, 92, 68, 76, 230, 96, 87, 164, 144, 2, 151, 199, 213, 216, 147, 4, 50, 74, 237, 133, 81, 237, 214, 197, 186, 160, 63, 54, 138, 47, 111, 76, 57, 168, 4, 105, 34, 132, 59, 117, 20, 247, 217, 62, 180, 169, 208, 7, 65, 56, 129, 126, 70, 72, 29, 232, 25, 230, 112, 184, 160, 170, 24, 221, 114, 191, 126, 125, 181, 108, 78, 25, 112, 17, 132, 4, 19, 129, 212, 98, 253, 156, 107, 241, 28, 221, 255, 236, 167, 224, 230, 199, 141, 49, 75, 191, 242, 107, 51, 204, 209, 140, 182, 254, 177, 27, 163, 74, 168, 95, 14, 127, 49, 118, 56, 63, 7, 71, 108, 1, 168, 172, 165, 20, 218, 130, 176, 95, 31, 108, 188, 137, 26, 192, 18, 42, 203, 123, 151, 31, 40, 46, 49, 22, 138, 83, 76, 117, 102, 249, 125, 1, 214, 194, 96, 170, 101, 143, 49, 158, 114, 203, 204, 155, 45, 229, 68, 25, 147, 87, 171, 110, 142, 194, 217, 46, 101, 176, 114, 45, 231, 161, 195, 161, 126, 73, 29, 89, 103, 214, 9, 212, 24, 18, 109, 192, 149, 69, 120, 201, 241, 96, 106, 228, 246, 238, 6, 36, 59, 115, 74, 90, 17, 194, 62, 151, 97, 192, 120, 238, 239, 19, 167, 48, 226, 74, 56, 45, 238, 88, 23, 19, 32, 183, 54, 42, 223, 11, 50, 180, 194, 56, 177, 145, 221, 18, 176, 69, 182, 124, 192, 177, 29, 98, 161, 157, 221, 113, 18, 21, 150, 1, 157, 10, 9, 245, 177, 221, 179, 217, 224, 56, 66, 52, 246, 105, 182, 195, 175, 22, 115, 254, 212, 67, 248, 67, 228, 88, 249, 79, 179, 243, 236, 214, 61, 148, 58, 64, 56, 21, 171, 12, 134, 208, 100, 206, 26, 99, 232, 132, 148, 85, 9, 90, 182, 4, 182, 209, 62, 154, 21, 199, 100, 17, 128, 208, 27, 130, 14, 77, 255, 69, 158, 178, 101, 7, 37, 31, 219, 154, 200, 198, 236, 214, 91, 133, 80, 4, 159, 28, 203, 93, 32, 130, 103, 158, 121, 226, 223, 232, 242, 232, 111, 196, 32, 114, 48, 134, 229, 157, 117, 39, 53, 2, 3, 1, 0, 1, 163, 130, 1, 75, 48, 130, 1, 71, 48, 15, 6, 3, 85, 29, 19, 1, 1, 255, 4, 5, 48, 3, 1, 1, 255, 48, 29, 6, 3, 85, 29, 14, 4, 22, 4, 20, 222, 145, 134, 72, 183, 161, 49, 89, 49, 241, 75, 95, 7, 169, 220, 136, 121, 218, 168, 118, 48, 31, 6, 3, 85, 29, 35, 4, 24, 48, 22, 128, 20, 78, 34, 84, 32, 24, 149, 230, 227, 110, 230, 15, 250, 250, 185, 18, 237, 6, 23, 143, 57, 48, 14, 6, 3, 85, 29, 15, 1, 1, 255, 4, 4, 3, 2, 1, 134, 48, 19, 6, 3, 85, 29, 37, 4, 12, 48, 10, 6, 8, 43, 6, 1, 5, 5, 7, 3, 1, 48, 118, 6, 8, 43, 6, 1, 5, 5, 7, 1, 1, 4, 106, 48, 104, 48, 36, 6, 8, 43, 6, 1, 5, 5, 7, 48, 1, 134, 24, 104, 116, 116, 112, 58, 47, 47, 111, 99, 115, 112, 46, 100, 105, 103, 105, 99, 101, 114, 116, 46, 99, 111, 109, 48, 64, 6, 8, 43, 6, 1, 5, 5, 7, 48, 2, 134, 52, 104, 116, 116, 112, 58, 47, 47, 99, 97, 99, 101, 114, 116, 115, 46, 100, 105, 103, 105, 99, 101, 114, 116, 46, 99, 111, 109, 47, 68, 105, 103, 105, 67, 101, 114, 116, 71, 108, 111, 98, 97, 108, 82, 111, 111, 116, 71, 50, 46, 99, 114, 116, 48, 66, 6, 3, 85, 29, 31, 4, 59, 48, 57, 48, 55, 160, 53, 160, 51, 134, 49, 104, 116, 116, 112, 58, 47, 47, 99, 114, 108, 51, 46, 100, 105, 103, 105, 99, 101, 114, 116, 46, 99, 111, 109, 47, 68, 105, 103, 105, 67, 101, 114, 116, 71, 108, 111, 98, 97, 108, 82, 111, 111, 116, 71, 50, 46, 99, 114, 108, 48, 19, 6, 3, 85, 29, 32, 4, 12, 48, 10, 48, 8, 6, 6, 103, 129, 12, 1, 2, 2],
RawSubjectPublicKeyInfo: [48, 130, 2, 34, 48, 13, 6, 9, 42, 134, 72, 134, 247, 13, 1, 1, 1, 5, 0, 3, 130, 2, 15, 0, 48, 130, 2, 10, 2, 130, 2, 1, 0, 223, 234, 139, 159, 71, 225, 40, 16, 113, 175, 65, 214, 25, 219, 146, 87, 222, 107, 171, 230, 46, 222, 173, 58, 96, 131, 183, 19, 152, 172, 91, 185, 50, 11, 123, 235, 159, 114, 39, 107, 93, 64, 171, 190, 24, 6, 147, 178, 13, 210, 18, 237, 184, 116, 245, 127, 47, 234, 168, 23, 148, 246, 51, 22, 87, 138, 222, 32, 124, 33, 221, 183, 218, 172, 222, 78, 100, 176, 90, 205, 75, 117, 243, 239, 2, 100, 77, 237, 82, 218, 163, 143, 4, 232, 145, 248, 215, 251, 63, 25, 253, 78, 69, 57, 94, 232, 41, 195, 214, 242, 103, 175, 179, 0, 148, 193, 141, 225, 240, 74, 24, 152, 57, 250, 192, 9, 132, 51, 65, 5, 3, 5, 171, 244, 117, 225, 83, 165, 33, 146, 92, 68, 76, 230, 96, 87, 164, 144, 2, 151, 199, 213, 216, 147, 4, 50, 74, 237, 133, 81, 237, 214, 197, 186, 160, 63, 54, 138, 47, 111, 76, 57, 168, 4, 105, 34, 132, 59, 117, 20, 247, 217, 62, 180, 169, 208, 7, 65, 56, 129, 126, 70, 72, 29, 232, 25, 230, 112, 184, 160, 170, 24, 221, 114, 191, 126, 125, 181, 108, 78, 25, 112, 17, 132, 4, 19, 129, 212, 98, 253, 156, 107, 241, 28, 221, 255, 236, 167, 224, 230, 199, 141, 49, 75, 191, 242, 107, 51, 204, 209, 140, 182, 254, 177, 27, 163, 74, 168, 95, 14, 127, 49, 118, 56, 63, 7, 71, 108, 1, 168, 172, 165, 20, 218, 130, 176, 95, 31, 108, 188, 137, 26, 192, 18, 42, 203, 123, 151, 31, 40, 46, 49, 22, 138, 83, 76, 117, 102, 249, 125, 1, 214, 194, 96, 170, 101, 143, 49, 158, 114, 203, 204, 155, 45, 229, 68, 25, 147, 87, 171, 110, 142, 194, 217, 46, 101, 176, 114, 45, 231, 161, 195, 161, 126, 73, 29, 89, 103, 214, 9, 212, 24, 18, 109, 192, 149, 69, 120, 201, 241, 96, 106, 228, 246, 238, 6, 36, 59, 115, 74, 90, 17, 194, 62, 151, 97, 192, 120, 238, 239, 19, 167, 48, 226, 74, 56, 45, 238, 88, 23, 19, 32, 183, 54, 42, 223, 11, 50, 180, 194, 56, 177, 145, 221, 18, 176, 69, 182, 124, 192, 177, 29, 98, 161, 157, 221, 113, 18, 21, 150, 1, 157, 10, 9, 245, 177, 221, 179, 217, 224, 56, 66, 52, 246, 105, 182, 195, 175, 22, 115, 254, 212, 67, 248, 67, 228, 88, 249, 79, 179, 243, 236, 214, 61, 148, 58, 64, 56, 21, 171, 12, 134, 208, 100, 206, 26, 99, 232, 132, 148, 85, 9, 90, 182, 4, 182, 209, 62, 154, 21, 199, 100, 17, 128, 208, 27, 130, 14, 77, 255, 69, 158, 178, 101, 7, 37, 31, 219, 154, 200, 198, 236, 214, 91, 133, 80, 4, 159, 28, 203, 93, 32, 130, 103, 158, 121, 226, 223, 232, 242, 232, 111, 196, 32, 114, 48, 134, 229, 157, 117, 39, 53, 2, 3, 1, 0, 1],
RawSubject: [48, 81, 49, 11, 48, 9, 6, 3, 85, 4, 6, 19, 2, 85, 83, 49, 30, 48, 28, 6, 3, 85, 4, 10, 19, 21, 77, 105, 99, 114, 111, 115, 111, 102, 116, 32, 67, 111, 114, 112, 111, 114, 97, 116, 105, 111, 110, 49, 34, 48, 32, 6, 3, 85, 4, 3, 19, 25, 77, 105, 99, 114, 111, 115, 111, 102, 116, 32, 84, 76, 83, 32, 82, 83, 65, 32, 82, 111, 111, 116, 32, 71, 50],
RawIssuer: [48, 97, 49, 11, 48, 9, 6, 3, 85, 4, 6, 19, 2, 85, 83, 49, 21, 48, 19, 6, 3, 85, 4, 10, 19, 12, 68, 105, 103, 105, 67, 101, 114, 116, 32, 73, 110, 99, 49, 25, 48, 23, 6, 3, 85, 4, 11, 19, 16, 119, 119, 119, 46, 100, 105, 103, 105, 99, 101, 114, 116, 46, 99, 111, 109, 49, 32, 48, 30, 6, 3, 85, 4, 3, 19, 23, 68, 105, 103, 105, 67, 101, 114, 116, 32, 71, 108, 111, 98, 97, 108, 32, 82, 111, 111, 116, 32, 71, 50],
Signature: [11, 188, 180, 43, 55, 116, 197, 75, 167, 54, 2, 54, 198, 175, 225, 19, 204, 138, 154, 151, 27, 252, 227, 68, 140, 238, 84, 0, 229, 228, 65, 26, 70, 240, 11, 118, 52, 255, 153, 85, 98, 53, 170, 173, 138, 156, 12, 90, 98, 204, 67, 192, 203, 234, 74, 29, 189, 107, 29, 175, 177, 25, 37, 102, 156, 247, 121, 110, 91, 37, 128, 36, 129, 133, 199, 121, 45, 53, 95, 66, 9, 120, 237, 159, 226, 101, 213, 196, 32, 80, 130, 16, 143, 106, 58, 237, 24, 142, 72, 137, 171, 166, 9, 248, 225, 128, 252, 6, 117, 22, 39, 37, 90, 143, 184, 58, 112, 108, 24, 71, 240, 99, 244, 83, 14, 217, 21, 53, 32, 243, 145, 61, 19, 1, 160, 226, 250, 36, 8, 69, 86, 119, 209, 103, 169, 49, 66, 208, 87, 255, 26, 38, 190, 197, 67, 216, 87, 169, 99, 255, 175, 243, 58, 216, 246, 123, 75, 16, 162, 151, 161, 207, 102, 190, 103, 103, 9, 249, 181, 76, 231, 171, 25, 43, 191, 34, 61, 125, 222, 198, 247, 239, 99, 56, 44, 221, 86, 196, 249, 9, 82, 218, 223, 227, 136, 111, 213, 109, 107, 247, 8, 65, 47, 142, 62, 134, 49, 159, 241, 124, 74, 245, 214, 97, 164, 70, 70, 40, 239, 87, 186, 176, 111, 146, 216, 210, 119, 37, 237, 123, 166, 41, 206, 59, 114, 121, 17, 212, 224, 167, 84, 109, 53, 43, 164, 122],
SignatureAlgorithm: 5,
PublicKeyAlgorithm: 1,
PublicKey: <*rsa.PublicKey | 0x36a118ac37b0>{
N: {
neg: false,
abs: [8228224839953229621, 16348041302357558304, 2074854447542410873, 14334067402580427935, 12854688567428618952, 14995722382209926558, 15077658020813214080, 9553354208224609462, 902637906707506152, 15437658075602359723, 4892132923038495724, 14100513645772686328, 15699610158021110198, 116260203274558899, 7107135261377697174, 1346652938663145757, 806906029520556509, 6347563231035665119, 1416154356297248238, 13996791138010853103, 17151436068643035665, 5005973024364225782, 15423091598760460437, 11656337824329587047, 14040304327591079399, 16520357453723561614, 10318202437165161261, 17977527309428632165, 3328466382962324838, 13840171658303184680, 9417131409404627226, 5146490298140267738, 6849551933690625799, 10139571647697603240, 3552143779013315793, 15996764607220205453, 9355211167345930524, 7804203173400544275, 12256789870797553077, 5196564742993393824, 12236288164267916870, 9528338169550421684, 3418034452713204002, 17138102837348087434, 15605821747820856657, 6942198289040656341, 6027260588802198758, 4685154310243513825, 1772230202496746547, 12898472891819094090, 6838761955263670191, 15563102318165378361, 5970264090691015160, 5437520332982603245, 15757213624540617421, 6308098459188714935, 3452756883569783574, 995879279813522815, 6719559476972000178, 3606112178562803563, 6954603544383871929, 16027092704045215034, 8191838635022848599, 16134862131230222352],
},
E: 65537,
},
Version: 3,
SerialNumber: {
neg: false,
abs: [5148676810827874504, 796129072041760688],
},
Issuer: {
Country: ["US"],
Organization: ["DigiCert Inc"],
OrganizationalUnit: ["www.digicert.com"],
Locality: nil,
Province: nil,
StreetAddress: nil,
PostalCode: nil,
SerialNumber: "",
CommonName: "DigiCert Global Root G2",
Names: [
{
Type: [..., ..., ..., ...],
Value: <string>"US",
},
{
Type: [..., ..., ..., ...],
Value: <string>"DigiCert Inc",
},
{
Type: [..., ..., ..., ...],
Value: <string>"www.digicert.com",
},
{
Type: [..., ..., ..., ...],
Value: <string>"DigiCert Global Root G2",
},
],
ExtraNames: nil,
},
Subject: {
Country: ["US"],
Organization: [
"Microsoft Corporation",
],
OrganizationalUnit: nil,
Locality: nil,
Province: nil,
StreetAddress: nil,
PostalCode: nil,
SerialNumber: "",
CommonName: "Microsoft TLS RSA Root G2",
Names: [
{
Type: [..., ..., ..., ...],
Value: <string>"US",
},
{
Type: [..., ..., ..., ...],
Value: <string>"Microsoft Corporation",
},
{
Type: [..., ..., ..., ...],
Value: <string>"Microsoft TLS RSA Root G2",
},
],
ExtraNames: nil,
},
NotBefore: {wall: 0, ext: 63883382400, loc: nil},
NotAfter: {wall: 0, ext: 64012204799, loc: nil},
KeyUsage: 97,
Extensions: [
{
Id: [2, 5, 29, 19],
Critical: true,
Value: [48, 3, 1, 1, 255],
},
{
Id: [2, 5, 29, 14],
Critical: false,
Value: [4, 20, 222, 145, 134, 72, 183, 161, 49, 89, 49, 241, 75, 95, 7, 169, 220, 136, 121, 218, 168, 118],
},
{
Id: [2, 5, 29, 35],
Critical: false,
Value: [48, 22, 128, 20, 78, 34, 84, 32, 24, 149, 230, 227, 110, 230, 15, 250, 250, 185, 18, 237, 6, 23, 143, 57],
},
{
Id: [2, 5, 29, 15],
Critical: true,
Value: [3, 2, 1, 134],
},
{
Id: [2, 5, 29, 37],
Critical: false,
Value: [48, 10, 6, 8, 43, 6, 1, 5, 5, 7, 3, 1],
},
{
Id: [1, 3, 6, 1, 5, 5, 7, 1, 1],
Critical: false,
Value: [48, 104, 48, 36, 6, 8, 43, 6, 1, 5, 5, 7, 48, 1, 134, 24, 104, 116, 116, 112, 58, 47, 47, 111, 99, 115, 112, 46, 100, 105, 103, 105, 99, 101, 114, 116, 46, 99, 111, 109, 48, 64, 6, 8, 43, 6, 1, 5, 5, 7, 48, 2, 134, 52, 104, 116, 116, 112, 58, 47, 47, 99, 97, 99, 101, 114, 116, 115, 46, 100, 105, 103, 105, 99, 101, 114, 116, 46, 99, 111, 109, 47, 68, 105, 103, 105, 67, 101, 114, 116, 71, 108, 111, 98, 97, 108, 82, 111, 111, 116, 71, 50, 46, 99, 114, 116],
},
{
Id: [2, 5, 29, 31],
Critical: false,
Value: "0907\xa05\xa03\x861http://crl3.digicert.com/DigiCertGlobalRootG2.crl",
},
{
Id: [2, 5, 29, 32],
Critical: false,
Value: [48, 10, 48, 8, 6, 6, 103, 129, 12, 1, 2, 2],
},
],
ExtraExtensions: nil,
UnhandledCriticalExtensions: nil,
ExtKeyUsage: [1],
UnknownExtKeyUsage: nil,
BasicConstraintsValid: true,
IsCA: true,
MaxPathLen: -1,
MaxPathLenZero: false,
SubjectKeyId: [222, 145, 134, 72, 183, 161, 49, 89, 49, 241, 75, 95, 7, 169, 220, 136, 121, 218, 168, 118],
AuthorityKeyId: [78, 34, 84, 32, 24, 149, 230, 227, 110, 230, 15, 250, 250, 185, 18, 237, 6, 23, 143, 57],
OCSPServer: [
"http://ocsp.digicert.com",
],
IssuingCertificateURL: [
"http://cacerts.digicert.com/DigiCertGlobalRootG2.crt",
],
DNSNames: nil,
EmailAddresses: nil,
IPAddresses: nil,
URIs: nil,
PermittedDNSDomainsCritical: false,
PermittedDNSDomains: nil,
ExcludedDNSDomains: nil,
PermittedIPRanges: nil,
ExcludedIPRanges: nil,
PermittedEmailAddresses: nil,
ExcludedEmailAddresses: nil,
PermittedURIDomains: nil,
ExcludedURIDomains: nil,
CRLDistributionPoints: [
"http://crl3.digicert.com/DigiCertGlobalRootG2.crl",
],
PolicyIdentifiers: [
[2, 23, 140, 1, 2, 2],
],
Policies: [
{
der: [103, 129, 12, 1, 2, 2],
},
],
InhibitAnyPolicy: 0,
InhibitAnyPolicyZero: false,
InhibitPolicyMapping: 0,
InhibitPolicyMappingZero: false,
RequireExplicitPolicy: 0,
RequireExplicitPolicyZero: false,
PolicyMappings: nil,
},
{
Raw: [48, 130, 3, 142, 48, 130, 2, 118, 160, 3, 2, 1, 2, 2, 16, 3, 58, 241, 230, 167, 17, 169, 160, 187, 40, 100, 177, 29, 9, 250, 229, 48, 13, 6, 9, 42, 134, 72, 134, 247, 13, 1, 1, 11, 5, 0, 48, 97, 49, 11, 48, 9, 6, 3, 85, 4, 6, 19, 2, 85, 83, 49, 21, 48, 19, 6, 3, 85, 4, 10, 19, 12, 68, 105, 103, 105, 67, 101, 114, 116, 32, 73, 110, 99, 49, 25, 48, 23, 6, 3, 85, 4, 11, 19, 16, 119, 119, 119, 46, 100, 105, 103, 105, 99, 101, 114, 116, 46, 99, 111, 109, 49, 32, 48, 30, 6, 3, 85, 4, 3, 19, 23, 68, 105, 103, 105, 67, 101, 114, 116, 32, 71, 108, 111, 98, 97, 108, 32, 82, 111, 111, 116, 32, 71, 50, 48, 30, 23, 13, 49, 51, 48, 56, 48, 49, 49, 50, 48, 48, 48, 48, 90, 23, 13, 51, 56, 48, 49, 49, 53, 49, 50, 48, 48, 48, 48, 90, 48, 97, 49, 11, 48, 9, 6, 3, 85, 4, 6, 19, 2, 85, 83, 49, 21, 48, 19, 6, 3, 85, 4, 10, 19, 12, 68, 105, 103, 105, 67, 101, 114, 116, 32, 73, 110, 99, 49, 25, 48, 23, 6, 3, 85, 4, 11, 19, 16, 119, 119, 119, 46, 100, 105, 103, 105, 99, 101, 114, 116, 46, 99, 111, 109, 49, 32, 48, 30, 6, 3, 85, 4, 3, 19, 23, 68, 105, 103, 105, 67, 101, 114, 116, 32, 71, 108, 111, 98, 97, 108, 32, 82, 111, 111, 116, 32, 71, 50, 48, 130, 1, 34, 48, 13, 6, 9, 42, 134, 72, 134, 247, 13, 1, 1, 1, 5, 0, 3, 130, 1, 15, 0, 48, 130, 1, 10, 2, 130, 1, 1, 0, 187, 55, 205, 52, 220, 123, 107, 201, 178, 104, 144, 173, 74, 117, 255, 70, 186, 33, 10, 8, 141, 245, 25, 84, 201, 251, 136, 219, 243, 174, 242, 58, 137, 145, 60, 122, 230, 171, 6, 26, 107, 207, 172, 45, 232, 94, 9, 36, 68, 186, 98, 154, 126, 214, 163, 168, 126, 224, 84, 117, 32, 5, 172, 80, 183, 156, 99, 26, 108, 48, 220, 218, 31, 25, 177, 215, 30, 222, 253, 215, 224, 203, 148, 131, 55, 174, 236, 31, 67, 78, 221, 123, 44, 210, 189, 46, 165, 47, 228, 169, 184, 173, 58, 212, 153, 164, 182, 37, 233, 155, 107, 0, 96, 146, 96, 255, 79, 33, 73, 24, 247, 103, 144, 171, 97, 6, 156, 143, 242, 186, 233, 180, 233, 146, 50, 107, 181, 243, 87, 232, 93, 27, 205, 140, 29, 171, 149, 4, 149, 73, 243, 53, 45, 150, 227, 73, 109, 221, 119, 227, 251, 73, 75, 180, 172, 85, 7, 169, 143, 149, 179, 180, 35, 187, 76, 109, 69, 240, 246, 169, 178, 149, 48, 180, 253, 76, 85, 140, 39, 74, 87, 20, 124, 130, 157, 205, 115, 146, 211, 22, 74, 6, 12, 140, 80, 209, 143, 30, 9, 190, 23, 161, 230, 33, 202, 253, 131, 229, 16, 188, 131, 165, 10, 196, 103, 40, 246, 115, 20, 20, 61, 70, 118, 195, 135, 20, 137, 33, 52, 77, 175, 15, 69, 12, 166, 73, 161, 186, 187, 156, 197, 177, 51, 131, 41, 133, 2, 3, 1, 0, 1, 163, 66, 48, 64, 48, 15, 6, 3, 85, 29, 19, 1, 1, 255, 4, 5, 48, 3, 1, 1, 255, 48, 14, 6, 3, 85, 29, 15, 1, 1, 255, 4, 4, 3, 2, 1, 134, 48, 29, 6, 3, 85, 29, 14, 4, 22, 4, 20, 78, 34, 84, 32, 24, 149, 230, 227, 110, 230, 15, 250, 250, 185, 18, 237, 6, 23, 143, 57, 48, 13, 6, 9, 42, 134, 72, 134, 247, 13, 1, 1, 11, 5, 0, 3, 130, 1, 1, 0, 96, 103, 40, 148, 111, 14, 72, 99, 235, 49, 221, 234, 103, 24, 213, 137, 125, 60, 197, 139, 74, 127, 233, 190, 219, 43, 23, 223, 176, 95, 115, 119, 42, 50, 19, 57, 129, 103, 66, 132, 35, 242, 69, 103, 53, 236, 136, 191, 248, 143, 176, 97, 12, 52, 164, 174, 32, 76, 132, 198, 219, 248, 53, 225, 118, 217, 223, 166, 66, 187, 199, 68, 8, 134, 127, 54, 116, 36, 90, 218, 108, 13, 20, 89, 53, 189, 242, 73, 221, 182, 31, 201, 179, 13, 71, 42, 61, 153, 47, 187, 92, 187, 181, 212, 32, 225, 153, 95, 83, 70, 21, 219, 104, 155, 240, 243, 48, 213, 62, 49, 226, 141, 132, 158, 227, 138, 218, 218, 150, 62, 53, 19, 165, 95, 240, 249, 112, 80, 112, 71, 65, 17, 87, 25, 78, 192, 143, 174, 6, 196, 149, 19, 23, 47, 27, 37, 159, 117, 242, 177, 142, 153, 161, 111, 19, 177, 65, 113, 254, 136, 42, 200, 79, 16, 32, 85, 215, 243, 20, 69, 229, 224, 68, 244, 234, 135, 149, 50, 147, 14, 254, 83, 70, 250, 44, 157, 255, 139, 34, 185, 75, 217, 9, 69, 164, 222, 164, 184, 154, 88, 221, 27, 125, 82, 159, 142, 89, 67, 136, 129, 164, 158, 38, 213, 111, 173, 221, 13, 198, 55, 125, 237, 3, 146, 27, 229, 119, 95, 118, 238, 60, 141, 196, 93, 86, 91, 162, 217, 102, 110, 179, 53, 55, 229, 50, 182],
RawTBSCertificate: [48, 130, 2, 118, 160, 3, 2, 1, 2, 2, 16, 3, 58, 241, 230, 167, 17, 169, 160, 187, 40, 100, 177, 29, 9, 250, 229, 48, 13, 6, 9, 42, 134, 72, 134, 247, 13, 1, 1, 11, 5, 0, 48, 97, 49, 11, 48, 9, 6, 3, 85, 4, 6, 19, 2, 85, 83, 49, 21, 48, 19, 6, 3, 85, 4, 10, 19, 12, 68, 105, 103, 105, 67, 101, 114, 116, 32, 73, 110, 99, 49, 25, 48, 23, 6, 3, 85, 4, 11, 19, 16, 119, 119, 119, 46, 100, 105, 103, 105, 99, 101, 114, 116, 46, 99, 111, 109, 49, 32, 48, 30, 6, 3, 85, 4, 3, 19, 23, 68, 105, 103, 105, 67, 101, 114, 116, 32, 71, 108, 111, 98, 97, 108, 32, 82, 111, 111, 116, 32, 71, 50, 48, 30, 23, 13, 49, 51, 48, 56, 48, 49, 49, 50, 48, 48, 48, 48, 90, 23, 13, 51, 56, 48, 49, 49, 53, 49, 50, 48, 48, 48, 48, 90, 48, 97, 49, 11, 48, 9, 6, 3, 85, 4, 6, 19, 2, 85, 83, 49, 21, 48, 19, 6, 3, 85, 4, 10, 19, 12, 68, 105, 103, 105, 67, 101, 114, 116, 32, 73, 110, 99, 49, 25, 48, 23, 6, 3, 85, 4, 11, 19, 16, 119, 119, 119, 46, 100, 105, 103, 105, 99, 101, 114, 116, 46, 99, 111, 109, 49, 32, 48, 30, 6, 3, 85, 4, 3, 19, 23, 68, 105, 103, 105, 67, 101, 114, 116, 32, 71, 108, 111, 98, 97, 108, 32, 82, 111, 111, 116, 32, 71, 50, 48, 130, 1, 34, 48, 13, 6, 9, 42, 134, 72, 134, 247, 13, 1, 1, 1, 5, 0, 3, 130, 1, 15, 0, 48, 130, 1, 10, 2, 130, 1, 1, 0, 187, 55, 205, 52, 220, 123, 107, 201, 178, 104, 144, 173, 74, 117, 255, 70, 186, 33, 10, 8, 141, 245, 25, 84, 201, 251, 136, 219, 243, 174, 242, 58, 137, 145, 60, 122, 230, 171, 6, 26, 107, 207, 172, 45, 232, 94, 9, 36, 68, 186, 98, 154, 126, 214, 163, 168, 126, 224, 84, 117, 32, 5, 172, 80, 183, 156, 99, 26, 108, 48, 220, 218, 31, 25, 177, 215, 30, 222, 253, 215, 224, 203, 148, 131, 55, 174, 236, 31, 67, 78, 221, 123, 44, 210, 189, 46, 165, 47, 228, 169, 184, 173, 58, 212, 153, 164, 182, 37, 233, 155, 107, 0, 96, 146, 96, 255, 79, 33, 73, 24, 247, 103, 144, 171, 97, 6, 156, 143, 242, 186, 233, 180, 233, 146, 50, 107, 181, 243, 87, 232, 93, 27, 205, 140, 29, 171, 149, 4, 149, 73, 243, 53, 45, 150, 227, 73, 109, 221, 119, 227, 251, 73, 75, 180, 172, 85, 7, 169, 143, 149, 179, 180, 35, 187, 76, 109, 69, 240, 246, 169, 178, 149, 48, 180, 253, 76, 85, 140, 39, 74, 87, 20, 124, 130, 157, 205, 115, 146, 211, 22, 74, 6, 12, 140, 80, 209, 143, 30, 9, 190, 23, 161, 230, 33, 202, 253, 131, 229, 16, 188, 131, 165, 10, 196, 103, 40, 246, 115, 20, 20, 61, 70, 118, 195, 135, 20, 137, 33, 52, 77, 175, 15, 69, 12, 166, 73, 161, 186, 187, 156, 197, 177, 51, 131, 41, 133, 2, 3, 1, 0, 1, 163, 66, 48, 64, 48, 15, 6, 3, 85, 29, 19, 1, 1, 255, 4, 5, 48, 3, 1, 1, 255, 48, 14, 6, 3, 85, 29, 15, 1, 1, 255, 4, 4, 3, 2, 1, 134, 48, 29, 6, 3, 85, 29, 14, 4, 22, 4, 20, 78, 34, 84, 32, 24, 149, 230, 227, 110, 230, 15, 250, 250, 185, 18, 237, 6, 23, 143, 57],
RawSubjectPublicKeyInfo: [48, 130, 1, 34, 48, 13, 6, 9, 42, 134, 72, 134, 247, 13, 1, 1, 1, 5, 0, 3, 130, 1, 15, 0, 48, 130, 1, 10, 2, 130, 1, 1, 0, 187, 55, 205, 52, 220, 123, 107, 201, 178, 104, 144, 173, 74, 117, 255, 70, 186, 33, 10, 8, 141, 245, 25, 84, 201, 251, 136, 219, 243, 174, 242, 58, 137, 145, 60, 122, 230, 171, 6, 26, 107, 207, 172, 45, 232, 94, 9, 36, 68, 186, 98, 154, 126, 214, 163, 168, 126, 224, 84, 117, 32, 5, 172, 80, 183, 156, 99, 26, 108, 48, 220, 218, 31, 25, 177, 215, 30, 222, 253, 215, 224, 203, 148, 131, 55, 174, 236, 31, 67, 78, 221, 123, 44, 210, 189, 46, 165, 47, 228, 169, 184, 173, 58, 212, 153, 164, 182, 37, 233, 155, 107, 0, 96, 146, 96, 255, 79, 33, 73, 24, 247, 103, 144, 171, 97, 6, 156, 143, 242, 186, 233, 180, 233, 146, 50, 107, 181, 243, 87, 232, 93, 27, 205, 140, 29, 171, 149, 4, 149, 73, 243, 53, 45, 150, 227, 73, 109, 221, 119, 227, 251, 73, 75, 180, 172, 85, 7, 169, 143, 149, 179, 180, 35, 187, 76, 109, 69, 240, 246, 169, 178, 149, 48, 180, 253, 76, 85, 140, 39, 74, 87, 20, 124, 130, 157, 205, 115, 146, 211, 22, 74, 6, 12, 140, 80, 209, 143, 30, 9, 190, 23, 161, 230, 33, 202, 253, 131, 229, 16, 188, 131, 165, 10, 196, 103, 40, 246, 115, 20, 20, 61, 70, 118, 195, 135, 20, 137, 33, 52, 77, 175, 15, 69, 12, 166, 73, 161, 186, 187, 156, 197, 177, 51, 131, 41, 133, 2, 3, 1, 0, 1],
RawSubject: [48, 97, 49, 11, 48, 9, 6, 3, 85, 4, 6, 19, 2, 85, 83, 49, 21, 48, 19, 6, 3, 85, 4, 10, 19, 12, 68, 105, 103, 105, 67, 101, 114, 116, 32, 73, 110, 99, 49, 25, 48, 23, 6, 3, 85, 4, 11, 19, 16, 119, 119, 119, 46, 100, 105, 103, 105, 99, 101, 114, 116, 46, 99, 111, 109, 49, 32, 48, 30, 6, 3, 85, 4, 3, 19, 23, 68, 105, 103, 105, 67, 101, 114, 116, 32, 71, 108, 111, 98, 97, 108, 32, 82, 111, 111, 116, 32, 71, 50],
RawIssuer: [48, 97, 49, 11, 48, 9, 6, 3, 85, 4, 6, 19, 2, 85, 83, 49, 21, 48, 19, 6, 3, 85, 4, 10, 19, 12, 68, 105, 103, 105, 67, 101, 114, 116, 32, 73, 110, 99, 49, 25, 48, 23, 6, 3, 85, 4, 11, 19, 16, 119, 119, 119, 46, 100, 105, 103, 105, 99, 101, 114, 116, 46, 99, 111, 109, 49, 32, 48, 30, 6, 3, 85, 4, 3, 19, 23, 68, 105, 103, 105, 67, 101, 114, 116, 32, 71, 108, 111, 98, 97, 108, 32, 82, 111, 111, 116, 32, 71, 50],
Signature: [96, 103, 40, 148, 111, 14, 72, 99, 235, 49, 221, 234, 103, 24, 213, 137, 125, 60, 197, 139, 74, 127, 233, 190, 219, 43, 23, 223, 176, 95, 115, 119, 42, 50, 19, 57, 129, 103, 66, 132, 35, 242, 69, 103, 53, 236, 136, 191, 248, 143, 176, 97, 12, 52, 164, 174, 32, 76, 132, 198, 219, 248, 53, 225, 118, 217, 223, 166, 66, 187, 199, 68, 8, 134, 127, 54, 116, 36, 90, 218, 108, 13, 20, 89, 53, 189, 242, 73, 221, 182, 31, 201, 179, 13, 71, 42, 61, 153, 47, 187, 92, 187, 181, 212, 32, 225, 153, 95, 83, 70, 21, 219, 104, 155, 240, 243, 48, 213, 62, 49, 226, 141, 132, 158, 227, 138, 218, 218, 150, 62, 53, 19, 165, 95, 240, 249, 112, 80, 112, 71, 65, 17, 87, 25, 78, 192, 143, 174, 6, 196, 149, 19, 23, 47, 27, 37, 159, 117, 242, 177, 142, 153, 161, 111, 19, 177, 65, 113, 254, 136, 42, 200, 79, 16, 32, 85, 215, 243, 20, 69, 229, 224, 68, 244, 234, 135, 149, 50, 147, 14, 254, 83, 70, 250, 44, 157, 255, 139, 34, 185, 75, 217, 9, 69, 164, 222, 164, 184, 154, 88, 221, 27, 125, 82, 159, 142, 89, 67, 136, 129, 164, 158, 38, 213, 111, 173, 221, 13, 198, 55, 125, 237, 3, 146, 27, 229, 119, 95, 118, 238, 60, 141, 196, 93, 86, 91, 162, 217, 102, 110, 179, 53, 55, 229, 50, 182],
SignatureAlgorithm: 4,
PublicKeyAlgorithm: 1,
PublicKey: <*rsa.PublicKey | 0x36a11782c840>{
N: {
neg: false,
abs: [13518897546323503493, 12614377001919816122, 8557832239203824717, 7433462159093546310, 9504020990483827396, 702024576034523901, 5333964605457796894, 8971906913408307990, 18252057450268743444, 5039799191733612724, 10346373305228545133, 18107087016177239977, 3285062882728703971, 2137966295034491701, 13110919595711778188, 17490549066068800107, 17827376715711028367, 6958731023901346072, 11071174057932843776, 11902983757762476756, 4850057369827327278, 16198203776063958047, 2241017827087547863, 13230558770443115738, 9142400105587256400, 4952379155939763112, 7768617195409377572, 9912770753372816922, 14554377099045433914, 13412012197143714132, 12855684210282069830, 13490476835732679625],
},
E: 65537,
},
Version: 3,
SerialNumber: {
neg: false,
abs: [13486139796020132581, 232764303710726560],
},
Issuer: {
Country: ["US"],
Organization: ["DigiCert Inc"],
OrganizationalUnit: ["www.digicert.com"],
Locality: nil,
Province: nil,
StreetAddress: nil,
PostalCode: nil,
SerialNumber: "",
CommonName: "DigiCert Global Root G2",
Names: [
{
Type: [..., ..., ..., ...],
Value: <string>"US",
},
{
Type: [..., ..., ..., ...],
Value: <string>"DigiCert Inc",
},
{
Type: [..., ..., ..., ...],
Value: <string>"www.digicert.com",
},
{
Type: [..., ..., ..., ...],
Value: <string>"DigiCert Global Root G2",
},
],
ExtraNames: nil,
},
Subject: {
Country: ["US"],
Organization: ["DigiCert Inc"],
OrganizationalUnit: ["www.digicert.com"],
Locality: nil,
Province: nil,
StreetAddress: nil,
PostalCode: nil,
SerialNumber: "",
CommonName: "DigiCert Global Root G2",
Names: [
{
Type: [..., ..., ..., ...],
Value: <string>"US",
},
{
Type: [..., ..., ..., ...],
Value: <string>"DigiCert Inc",
},
{
Type: [..., ..., ..., ...],
Value: <string>"www.digicert.com",
},
{
Type: [..., ..., ..., ...],
Value: <string>"DigiCert Global Root G2",
},
],
ExtraNames: nil,
},
NotBefore: {wall: 0, ext: 63510955200, loc: nil},
NotAfter: {wall: 0, ext: 64282766400, loc: nil},
KeyUsage: 97,
Extensions: [
{
Id: [2, 5, 29, 19],
Critical: true,
Value: [48, 3, 1, 1, 255],
},
{
Id: [2, 5, 29, 15],
Critical: true,
Value: [3, 2, 1, 134],
},
{
Id: [2, 5, 29, 14],
Critical: false,
Value: [4, 20, 78, 34, 84, 32, 24, 149, 230, 227, 110, 230, 15, 250, 250, 185, 18, 237, 6, 23, 143, 57],
},
],
ExtraExtensions: nil,
UnhandledCriticalExtensions: nil,
ExtKeyUsage: nil,
UnknownExtKeyUsage: nil,
BasicConstraintsValid: true,
IsCA: true,
MaxPathLen: -1,
MaxPathLenZero: false,
SubjectKeyId: [78, 34, 84, 32, 24, 149, 230, 227, 110, 230, 15, 250, 250, 185, 18, 237, 6, 23, 143, 57],
AuthorityKeyId: nil,
OCSPServer: nil,
IssuingCertificateURL: nil,
DNSNames: nil,
EmailAddresses: nil,
IPAddresses: nil,
URIs: nil,
PermittedDNSDomainsCritical: false,
PermittedDNSDomains: nil,
ExcludedDNSDomains: nil,
PermittedIPRanges: nil,
ExcludedIPRanges: nil,
PermittedEmailAddresses: nil,
ExcludedEmailAddresses: nil,
PermittedURIDomains: nil,
ExcludedURIDomains: nil,
CRLDistributionPoints: nil,
PolicyIdentifiers: nil,
Policies: nil,
InhibitAnyPolicy: 0,
InhibitAnyPolicyZero: false,
InhibitPolicyMapping: 0,
InhibitPolicyMappingZero: false,
RequireExplicitPolicy: 0,
RequireExplicitPolicyZero: false,
PolicyMappings: nil,
},
],
Err: <x509.UnknownAuthorityError>{
Cert: 0x36a1182f5208 (cyclic reference),
hintErr: nil,
hintCert: nil,
},
},
},
},
}Contributing tests (1)
Affected runs (1)
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
failed to create nodepool <nodepool>: failed waiting for nodepool="<nodepool>" for cluster "<cluster>" in resourcegroup="<resource-group>" to finish creating: GET <url>full failure pattern: failed to create nodepool <nodepool>: failed waiting for nodepool="<nodepool>" for cluster "<cluster>" in resourcegroup="<resource-group>" to finish creating: GET <url> Signal: IndeterminateSignal: Indeterminate — seen in 2 prior week(s); active 1 day(s) | e2e | 1 | 4.35%1 of 23 job runs affected | IndeterminateSignal: Indeterminate — seen in 2 prior week(s); active 1 day(s) | Show trend detailsOct 21: 0 · Oct 22: 0 · Oct 23: 0 · Oct 24: 0 · Oct 25: 0 · Oct 26: 0 · Oct 27: 0 · Oct 28: 1 | DEV | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_delete.go:128]: failed to create node pools
Expected success, but got an error:
<*fmt.wrapError | 0x8979cd42dc0>:
failed to create NodePool np-deletable: failed waiting for nodepool="np-deletable" for cluster "np-delete-hcp-cluster" in resourcegroup="nodepool-delete-d4wglkkdjhfj" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/4691f9f8-c66b-49b4-9a08-68ce0d3e72a6
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/4691f9f8-c66b-49b4-9a08-68ce0d3e72a6",
"name": "4691f9f8-c66b-49b4-9a08-68ce0d3e72a6",
"status": "Failed",
"startTime": "2026-10-28T03:01:21.21908488Z",
"endTime": "2026-10-28T03:20:21.275752007Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine p0i5l3o3i9z8d7r-np-deletable-95t5c-lvd52: NodeNotHealthy: * Node.Ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine p0i5l3o3i9z8d7r-np-deletable-95t5c-lvd52: NotReady: * NodeHealthy:\n * Node.Ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?\n"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/4691f9f8-c66b-49b4-9a08-68ce0d3e72a6
correlation ID: 48b19522-cae3-4702-8aae-982e9a4042a0
client request ID: unavailable
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_delete.go:128]: failed to create node pools
Expected success, but got an error:
<*fmt.wrapError | 0x8979cd42dc0>:
failed to create NodePool np-deletable: failed waiting for nodepool="np-deletable" for cluster "np-delete-hcp-cluster" in resourcegroup="nodepool-delete-d4wglkkdjhfj" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/4691f9f8-c66b-49b4-9a08-68ce0d3e72a6
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/4691f9f8-c66b-49b4-9a08-68ce0d3e72a6",
"name": "4691f9f8-c66b-49b4-9a08-68ce0d3e72a6",
"status": "Failed",
"startTime": "2026-10-28T03:01:21.21908488Z",
"endTime": "2026-10-28T03:20:21.275752007Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine p0i5l3o3i9z8d7r-np-deletable-95t5c-lvd52: NodeNotHealthy: * Node.Ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine p0i5l3o3i9z8d7r-np-deletable-95t5c-lvd52: NotReady: * NodeHealthy:\n * Node.Ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?\n"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/4691f9f8-c66b-49b4-9a08-68ce0d3e72a6
correlation ID: 48b19522-cae3-4702-8aae-982e9a4042a0
client request ID: unavailableContributing tests (1)
Affected runs (1)
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
failed to create nodepool <nodepool>: failed waiting for nodepool="<nodepool>" for cluster "<cluster>" in resourcegroup="<resource-group>" to finish creating: GET <url>full failure pattern: failed to create nodepool <nodepool>: failed waiting for nodepool="<nodepool>" for cluster "<cluster>" in resourcegroup="<resource-group>" to finish creating: GET <url> Signal: IndeterminateSignal: Indeterminate — seen in 2 prior week(s); active 1 day(s) | unknown | 1 | 4.35%1 of 23 job runs affected | IndeterminateSignal: Indeterminate — seen in 2 prior week(s); active 1 day(s) | Show trend detailsOct 21: 0 · Oct 22: 0 · Oct 23: 0 · Oct 24: 0 · Oct 25: 0 · Oct 26: 0 · Oct 27: 1 · Oct 28: 0 | DEV | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_delete.go:128]: failed to create node pools
Expected success, but got an error:
<*fmt.wrapError | 0x3b12e2cba60>:
failed to create NodePool np-delete-main: failed waiting for nodepool="np-delete-main" for cluster "np-delete-hcp-cluster" in resourcegroup="nodepool-delete-4j5vtdb84h8v" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/083c3fe4-0c43-4b6a-9905-e8c1aa97e192
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/083c3fe4-0c43-4b6a-9905-e8c1aa97e192",
"name": "083c3fe4-0c43-4b6a-9905-e8c1aa97e192",
"status": "Failed",
"startTime": "2026-10-27T09:13:51.032005174Z",
"endTime": "2026-10-27T09:32:53.54866754Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine o3t1v0i5b5o7c5l-np-delete-main-bg4kp-nd4jp: InspectionFailed: Waiting for AzureMachine to report spec.providerID\nMachine o3t1v0i5b5o7c5l-np-delete-main-bg4kp-q624c: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine o3t1v0i5b5o7c5l-np-delete-main-bg4kp-nd4jp: Creating: virtualmachine creating or updating\nMachine o3t1v0i5b5o7c5l-np-delete-main-bg4kp-q624c: Creating: virtualmachine creating or updating\n"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/083c3fe4-0c43-4b6a-9905-e8c1aa97e192
correlation ID: 01ff1c80-ae7b-4cd9-aeb7-68197786fff9
client request ID: unavailable
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_delete.go:128]: failed to create node pools
Expected success, but got an error:
<*fmt.wrapError | 0x3b12e2cba60>:
failed to create NodePool np-delete-main: failed waiting for nodepool="np-delete-main" for cluster "np-delete-hcp-cluster" in resourcegroup="nodepool-delete-4j5vtdb84h8v" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/083c3fe4-0c43-4b6a-9905-e8c1aa97e192
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/083c3fe4-0c43-4b6a-9905-e8c1aa97e192",
"name": "083c3fe4-0c43-4b6a-9905-e8c1aa97e192",
"status": "Failed",
"startTime": "2026-10-27T09:13:51.032005174Z",
"endTime": "2026-10-27T09:32:53.54866754Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine o3t1v0i5b5o7c5l-np-delete-main-bg4kp-nd4jp: InspectionFailed: Waiting for AzureMachine to report spec.providerID\nMachine o3t1v0i5b5o7c5l-np-delete-main-bg4kp-q624c: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine o3t1v0i5b5o7c5l-np-delete-main-bg4kp-nd4jp: Creating: virtualmachine creating or updating\nMachine o3t1v0i5b5o7c5l-np-delete-main-bg4kp-q624c: Creating: virtualmachine creating or updating\n"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/083c3fe4-0c43-4b6a-9905-e8c1aa97e192
correlation ID: 01ff1c80-ae7b-4cd9-aeb7-68197786fff9
client request ID: unavailableContributing tests (1)
Affected runs (1)
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Environment: PROD
Daily Run Outcomes
| Pass rate | Runs | Date (UTC) | Suite | Test |
|---|---|---|---|---|
| 84.62% | 13 | prod/parallel | Authorized CIDRs Connectivity should allow API access only from authorized VM IP | |
| 92.31% | 13 | prod/parallel | Customer should be able to create a cluster and node pools with aggregated advanced features | |
| 92.31% | 13 | prod/parallel | Customer should be able to create an HCP cluster and manage ImageDigestMirrors | |
| 92.31% | 13 | prod/parallel | Customer should be able to create an HCP cluster and manage pull secrets | |
| 92.31% | 13 | prod/parallel | Customer should be able to create an HCP cluster with custom autoscaling and update the autoscaling configuration |
Up to 50 failure patterns are loaded (minimum 1.00% impact), with 10 shown by default. Click column headers to re-sort.
Jump to the full Failure Patterns view for this window
| Failure patternThe canonical description of a recurring CI failure, extracted and normalized from raw logs. | Failed atThe stage of the job run where this failure occurred: 'provision' (environment setup, DEV only), 'e2e' (test suite execution), or 'other' (CI infrastructure issues that did not produce a failure pattern). | Number of distinct job runs where this failure pattern was detected. | Percentage of all job runs in this environment affected by this failure pattern during the selected window. | Classification of this failure pattern: Regression (likely caused by a specific PR), Flake (intermittent failure spread across days), Noise (low-quality or generic pattern), or Indeterminate. | TrendShows daily activity for this failure pattern in a trailing window anchored to the selected end date. The sparkline covers at least 7 days and at most 14 days, depending on the current window size. | Also inOther environments where the same failure pattern was also detected during the selected window. | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
ACM ingress secret openshift-ingress/cluster-ingress-cert exists; expected ACM certificate delivery to be suppressedfull failure pattern: ACM ingress secret openshift-ingress/cluster-ingress-cert exists; expected ACM certificate delivery to be suppressed Signal: FlakeSignal: Flake — present in 2 prior week(s); spread across 5 days | e2e | 6 | 8.00%6 of 75 job runs affected | FlakeSignal: Flake — present in 2 prior week(s); spread across 5 days | Show trend detailsOct 21: 0 · Oct 22: 5 · Oct 23: 10 · Oct 24: 5 · Oct 25: 0 · Oct 26: 5 · Oct 27: 3 · Oct 28: 0 | DEV, STG | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (3)fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:260]: ACM must not create its ingress certificate secret for cluster "cluster-candidate-4-23-lkx8t5"
Expected success, but got an error:
<*errors.errorString | 0x3306da82b390>:
ACM ingress secret openshift-ingress/cluster-ingress-cert exists; expected ACM certificate delivery to be suppressed
{
s: "ACM ingress secret openshift-ingress/cluster-ingress-cert exists; expected ACM certificate delivery to be suppressed",
}
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:260]: ACM must not create its ingress certificate secret for cluster "cluster-candidate-4-23-lkx8t5"
Expected success, but got an error:
<*errors.errorString | 0x3306da82b390>:
ACM ingress secret openshift-ingress/cluster-ingress-cert exists; expected ACM certificate delivery to be suppressed
{
s: "ACM ingress secret openshift-ingress/cluster-ingress-cert exists; expected ACM certificate delivery to be suppressed",
}fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:528]: ACM must not create its ingress certificate secret
Expected success, but got an error:
<*errors.errorString | 0xdf906826d40>:
ACM ingress secret openshift-ingress/cluster-ingress-cert exists; expected ACM certificate delivery to be suppressed
{
s: "ACM ingress secret openshift-ingress/cluster-ingress-cert exists; expected ACM certificate delivery to be suppressed",
}
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:528]: ACM must not create its ingress certificate secret
Expected success, but got an error:
<*errors.errorString | 0xdf906826d40>:
ACM ingress secret openshift-ingress/cluster-ingress-cert exists; expected ACM certificate delivery to be suppressed
{
s: "ACM ingress secret openshift-ingress/cluster-ingress-cert exists; expected ACM certificate delivery to be suppressed",
}fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:528]: ACM must not create its ingress certificate secret
Expected success, but got an error:
<*errors.errorString | 0x3454a17e2bc0>:
ACM ingress secret openshift-ingress/cluster-ingress-cert exists; expected ACM certificate delivery to be suppressed
{
s: "ACM ingress secret openshift-ingress/cluster-ingress-cert exists; expected ACM certificate delivery to be suppressed",
}
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:528]: ACM must not create its ingress certificate secret
Expected success, but got an error:
<*errors.errorString | 0x3454a17e2bc0>:
ACM ingress secret openshift-ingress/cluster-ingress-cert exists; expected ACM certificate delivery to be suppressed
{
s: "ACM ingress secret openshift-ingress/cluster-ingress-cert exists; expected ACM certificate delivery to be suppressed",
}Contributing tests (11)
Affected runs (6)
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedC...full failure pattern: ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider]; provider Microsoft.RedHatOpenShift Signal: FlakeSignal: Flake — present in 3 prior week(s); spread across 3 days | e2e | 4 | 5.33%4 of 75 job runs affected | FlakeSignal: Flake — present in 3 prior week(s); spread across 3 days | Show trend detailsOct 21: 0 · Oct 22: 10 · Oct 23: 0 · Oct 24: 0 · Oct 25: 0 · Oct 26: 0 · Oct 27: 2 · Oct 28: 13 | DEV, INT, STG | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (3)fail [github.com/Azure/ARO-HCP/test/e2e/cluster_nsg_subnet_reuse.go:107]: failed to create seed HCP cluster basic-cluster
Unexpected error:
<*fmt.wrapError | 0x22dc0191a120>:
failed to create HCP cluster basic-cluster: failed waiting for cluster="basic-cluster" in resourcegroup="rg-cluster-nsg-subnet-reuse-jc4z76tj8b8s" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/91e4eda4-eb0e-4ff2-b22e-65959d67929a
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/91e4eda4-eb0e-4ff2-b22e-65959d67929a",
"name": "91e4eda4-eb0e-4ff2-b22e-65959d67929a",
"status": "Failed",
"startTime": "2026-10-28T13:43:05.747830881Z",
"endTime": "2026-10-28T14:12:10.776073825Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/91e4eda4-eb0e-4ff2-b22e-65959d67929a
correlation ID: 93c5577b-f0d7-4fff-80a7-689379e11923
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_nsg_subnet_reuse.go:107]: failed to create seed HCP cluster basic-cluster
Unexpected error:
<*fmt.wrapError | 0x22dc0191a120>:
failed to create HCP cluster basic-cluster: failed waiting for cluster="basic-cluster" in resourcegroup="rg-cluster-nsg-subnet-reuse-jc4z76tj8b8s" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/91e4eda4-eb0e-4ff2-b22e-65959d67929a
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/91e4eda4-eb0e-4ff2-b22e-65959d67929a",
"name": "91e4eda4-eb0e-4ff2-b22e-65959d67929a",
"status": "Failed",
"startTime": "2026-10-28T13:43:05.747830881Z",
"endTime": "2026-10-28T14:12:10.776073825Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/91e4eda4-eb0e-4ff2-b22e-65959d67929a
correlation ID: 93c5577b-f0d7-4fff-80a7-689379e11923
client request ID: unavailable
occurredfail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:125]: failed to create HCP cluster "cluster-zstream-4-21-8fz8f5" with version 4.21.39 on candidate channel
Unexpected error:
<*fmt.wrapError | 0x17985ea62800>:
failed to create HCP cluster cluster-zstream-4-21-8fz8f5: failed waiting for cluster="cluster-zstream-4-21-8fz8f5" in resourcegroup="rg-zstream-upgrade-4-21-wbqzpvwdc6q6" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/487512e6-43f7-42b5-a694-f4be2585d80c
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/487512e6-43f7-42b5-a694-f4be2585d80c",
"name": "487512e6-43f7-42b5-a694-f4be2585d80c",
"status": "Failed",
"startTime": "2026-10-28T14:40:20.778612196Z",
"endTime": "2026-10-28T15:09:28.776202464Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/487512e6-43f7-42b5-a694-f4be2585d80c
correlation ID: 80cf0427-c640-486c-a738-9d16fdc0c4e5
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:125]: failed to create HCP cluster "cluster-zstream-4-21-8fz8f5" with version 4.21.39 on candidate channel
Unexpected error:
<*fmt.wrapError | 0x17985ea62800>:
failed to create HCP cluster cluster-zstream-4-21-8fz8f5: failed waiting for cluster="cluster-zstream-4-21-8fz8f5" in resourcegroup="rg-zstream-upgrade-4-21-wbqzpvwdc6q6" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/487512e6-43f7-42b5-a694-f4be2585d80c
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/487512e6-43f7-42b5-a694-f4be2585d80c",
"name": "487512e6-43f7-42b5-a694-f4be2585d80c",
"status": "Failed",
"startTime": "2026-10-28T14:40:20.778612196Z",
"endTime": "2026-10-28T15:09:28.776202464Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/487512e6-43f7-42b5-a694-f4be2585d80c
correlation ID: 80cf0427-c640-486c-a738-9d16fdc0c4e5
client request ID: unavailable
occurredfail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_cni_cilium.go:88]: failed to create HCP cluster "cilium-cl" without CNI
Unexpected error:
<*fmt.wrapError | 0x374e833daa40>:
failed to create HCP cluster cilium-cl: failed waiting for cluster="cilium-cl" in resourcegroup="cni-cilium-f4hlsn5n7dp5" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/1f756a31-37f0-40e0-82d6-7323816b3a44
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/1f756a31-37f0-40e0-82d6-7323816b3a44",
"name": "1f756a31-37f0-40e0-82d6-7323816b3a44",
"status": "Failed",
"startTime": "2026-10-28T13:42:59.191596418Z",
"endTime": "2026-10-28T14:12:00.890716747Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/1f756a31-37f0-40e0-82d6-7323816b3a44
correlation ID: 35fe35ef-43a8-4363-855c-128586bb4a72
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_cni_cilium.go:88]: failed to create HCP cluster "cilium-cl" without CNI
Unexpected error:
<*fmt.wrapError | 0x374e833daa40>:
failed to create HCP cluster cilium-cl: failed waiting for cluster="cilium-cl" in resourcegroup="cni-cilium-f4hlsn5n7dp5" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/1f756a31-37f0-40e0-82d6-7323816b3a44
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/1f756a31-37f0-40e0-82d6-7323816b3a44",
"name": "1f756a31-37f0-40e0-82d6-7323816b3a44",
"status": "Failed",
"startTime": "2026-10-28T13:42:59.191596418Z",
"endTime": "2026-10-28T14:12:00.890716747Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/1f756a31-37f0-40e0-82d6-7323816b3a44
correlation ID: 35fe35ef-43a8-4363-855c-128586bb4a72
client request ID: unavailable
occurredContributing tests (22)
Affected runs (4)
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
VerifyBasicAccess failed: failed to list services: Unauthorizedfull failure pattern: VerifyBasicAccess failed: failed to list services: Unauthorized Signal: FlakeSignal: Flake — present in 3 prior week(s); spread across 3 days | e2e | 3 | 4.00%3 of 75 job runs affected | FlakeSignal: Flake — present in 3 prior week(s); spread across 3 days | Show trend detailsOct 21: 0 · Oct 22: 0 · Oct 23: 1 · Oct 24: 2 · Oct 25: 1 · Oct 26: 0 · Oct 27: 0 · Oct 28: 0 | none | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (3)fail [github.com/Azure/ARO-HCP/test/e2e/oidc_issuer_workload_identity.go:254]: cluster oidc-wi-cluster is not viable
Unexpected error:
<*errors.joinError | 0x5c10dd80030>:
VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Unauthorized
VerifyBasicAccess failed: failed to list services: Unauthorized
{
errs: [
<*fmt.wrapError | 0x5c10d35ddc0>{
msg: "VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Unauthorized",
err: <*fmt.wrapError | 0x5c10d35dda0>{
msg: "failed to list all APIServices: Unauthorized",
err: <*errors.StatusError | 0x5c10db35540>{
ErrStatus: {
TypeMeta: {Kind: "", APIVersion: ""},
ListMeta: {
SelfLink: "",
ResourceVersion: "",
Continue: "",
RemainingItemCount: nil,
ShardInfo: nil,
},
Status: "Failure",
Message: "Unauthorized",
Reason: "Unauthorized",
Details: nil,
Code: 401,
},
},
},
},
<*fmt.wrapError | 0x5c10dd88020>{
msg: "VerifyBasicAccess failed: failed to list services: Unauthorized",
err: <*fmt.wrapError | 0x5c10dd88000>{
msg: "failed to list services: Unauthorized",
err: <*errors.StatusError | 0x5c10dbae320>{
ErrStatus: {
TypeMeta: {Kind: "", APIVersion: ""},
ListMeta: {
SelfLink: "",
ResourceVersion: "",
Continue: "",
RemainingItemCount: nil,
ShardInfo: nil,
},
Status: "Failure",
Message: "Unauthorized",
Reason: "Unauthorized",
Details: nil,
Code: 401,
},
},
},
},
],
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/oidc_issuer_workload_identity.go:254]: cluster oidc-wi-cluster is not viable
Unexpected error:
<*errors.joinError | 0x5c10dd80030>:
VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Unauthorized
VerifyBasicAccess failed: failed to list services: Unauthorized
{
errs: [
<*fmt.wrapError | 0x5c10d35ddc0>{
msg: "VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Unauthorized",
err: <*fmt.wrapError | 0x5c10d35dda0>{
msg: "failed to list all APIServices: Unauthorized",
err: <*errors.StatusError | 0x5c10db35540>{
ErrStatus: {
TypeMeta: {Kind: "", APIVersion: ""},
ListMeta: {
SelfLink: "",
ResourceVersion: "",
Continue: "",
RemainingItemCount: nil,
ShardInfo: nil,
},
Status: "Failure",
Message: "Unauthorized",
Reason: "Unauthorized",
Details: nil,
Code: 401,
},
},
},
},
<*fmt.wrapError | 0x5c10dd88020>{
msg: "VerifyBasicAccess failed: failed to list services: Unauthorized",
err: <*fmt.wrapError | 0x5c10dd88000>{
msg: "failed to list services: Unauthorized",
err: <*errors.StatusError | 0x5c10dbae320>{
ErrStatus: {
TypeMeta: {Kind: "", APIVersion: ""},
ListMeta: {
SelfLink: "",
ResourceVersion: "",
Continue: "",
RemainingItemCount: nil,
ShardInfo: nil,
},
Status: "Failure",
Message: "Unauthorized",
Reason: "Unauthorized",
Details: nil,
Code: 401,
},
},
},
},
],
}
occurredfail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:140]: failed to verify HCP cluster "cluster-nightly-4-22-67pgbb" is viable
Unexpected error:
<*errors.joinError | 0x21cecb2413f8>:
VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Unauthorized
VerifyBasicAccess failed: failed to list services: Unauthorized
{
errs: [
<*fmt.wrapError | 0x21cecb85e9a0>{
msg: "VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Unauthorized",
err: <*fmt.wrapError | 0x21cecb85e980>{
msg: "failed to list all APIServices: Unauthorized",
err: <*errors.StatusError | 0x21cecae4e500>{
ErrStatus: {
TypeMeta: {Kind: "", APIVersion: ""},
ListMeta: {
SelfLink: "",
ResourceVersion: "",
Continue: "",
RemainingItemCount: nil,
ShardInfo: nil,
},
Status: "Failure",
Message: "Unauthorized",
Reason: "Unauthorized",
Details: nil,
Code: 401,
},
},
},
},
<*fmt.wrapError | 0x21cecb471000>{
msg: "VerifyBasicAccess failed: failed to list services: Unauthorized",
err: <*fmt.wrapError | 0x21cecb470fe0>{
msg: "failed to list services: Unauthorized",
err: <*errors.StatusError | 0x21cecb1806e0>{
ErrStatus: {
TypeMeta: {Kind: "", APIVersion: ""},
ListMeta: {
SelfLink: "",
ResourceVersion: "",
Continue: "",
RemainingItemCount: nil,
ShardInfo: nil,
},
Status: "Failure",
Message: "Unauthorized",
Reason: "Unauthorized",
Details: nil,
Code: 401,
},
},
},
},
],
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:140]: failed to verify HCP cluster "cluster-nightly-4-22-67pgbb" is viable
Unexpected error:
<*errors.joinError | 0x21cecb2413f8>:
VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Unauthorized
VerifyBasicAccess failed: failed to list services: Unauthorized
{
errs: [
<*fmt.wrapError | 0x21cecb85e9a0>{
msg: "VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Unauthorized",
err: <*fmt.wrapError | 0x21cecb85e980>{
msg: "failed to list all APIServices: Unauthorized",
err: <*errors.StatusError | 0x21cecae4e500>{
ErrStatus: {
TypeMeta: {Kind: "", APIVersion: ""},
ListMeta: {
SelfLink: "",
ResourceVersion: "",
Continue: "",
RemainingItemCount: nil,
ShardInfo: nil,
},
Status: "Failure",
Message: "Unauthorized",
Reason: "Unauthorized",
Details: nil,
Code: 401,
},
},
},
},
<*fmt.wrapError | 0x21cecb471000>{
msg: "VerifyBasicAccess failed: failed to list services: Unauthorized",
err: <*fmt.wrapError | 0x21cecb470fe0>{
msg: "failed to list services: Unauthorized",
err: <*errors.StatusError | 0x21cecb1806e0>{
ErrStatus: {
TypeMeta: {Kind: "", APIVersion: ""},
ListMeta: {
SelfLink: "",
ResourceVersion: "",
Continue: "",
RemainingItemCount: nil,
ShardInfo: nil,
},
Status: "Failure",
Message: "Unauthorized",
Reason: "Unauthorized",
Details: nil,
Code: 401,
},
},
},
},
],
}
occurredfail [github.com/Azure/ARO-HCP/test/e2e/simple_negative_cases.go:124]: cluster negative-tests-cluster is not viable
Unexpected error:
<*errors.joinError | 0x13bc429c0870>:
VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Unauthorized
VerifyBasicAccess failed: failed to list services: Unauthorized
{
errs: [
<*fmt.wrapError | 0x13bc42aa9920>{
msg: "VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Unauthorized",
err: <*fmt.wrapError | 0x13bc42aa9900>{
msg: "failed to list all APIServices: Unauthorized",
err: <*errors.StatusError | 0x13bc41a5d400>{
ErrStatus: {
TypeMeta: {Kind: "", APIVersion: ""},
ListMeta: {
SelfLink: "",
ResourceVersion: "",
Continue: "",
RemainingItemCount: nil,
ShardInfo: nil,
},
Status: "Failure",
Message: "Unauthorized",
Reason: "Unauthorized",
Details: nil,
Code: 401,
},
},
},
},
<*fmt.wrapError | 0x13bc425c16c0>{
msg: "VerifyBasicAccess failed: failed to list services: Unauthorized",
err: <*fmt.wrapError | 0x13bc425c16a0>{
msg: "failed to list services: Unauthorized",
err: <*errors.StatusError | 0x13bc416d8960>{
ErrStatus: {
TypeMeta: {Kind: "", APIVersion: ""},
ListMeta: {
SelfLink: "",
ResourceVersion: "",
Continue: "",
RemainingItemCount: nil,
ShardInfo: nil,
},
Status: "Failure",
Message: "Unauthorized",
Reason: "Unauthorized",
Details: nil,
Code: 401,
},
},
},
},
],
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/simple_negative_cases.go:124]: cluster negative-tests-cluster is not viable
Unexpected error:
<*errors.joinError | 0x13bc429c0870>:
VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Unauthorized
VerifyBasicAccess failed: failed to list services: Unauthorized
{
errs: [
<*fmt.wrapError | 0x13bc42aa9920>{
msg: "VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Unauthorized",
err: <*fmt.wrapError | 0x13bc42aa9900>{
msg: "failed to list all APIServices: Unauthorized",
err: <*errors.StatusError | 0x13bc41a5d400>{
ErrStatus: {
TypeMeta: {Kind: "", APIVersion: ""},
ListMeta: {
SelfLink: "",
ResourceVersion: "",
Continue: "",
RemainingItemCount: nil,
ShardInfo: nil,
},
Status: "Failure",
Message: "Unauthorized",
Reason: "Unauthorized",
Details: nil,
Code: 401,
},
},
},
},
<*fmt.wrapError | 0x13bc425c16c0>{
msg: "VerifyBasicAccess failed: failed to list services: Unauthorized",
err: <*fmt.wrapError | 0x13bc425c16a0>{
msg: "failed to list services: Unauthorized",
err: <*errors.StatusError | 0x13bc416d8960>{
ErrStatus: {
TypeMeta: {Kind: "", APIVersion: ""},
ListMeta: {
SelfLink: "",
ResourceVersion: "",
Continue: "",
RemainingItemCount: nil,
ShardInfo: nil,
},
Status: "Failure",
Message: "Unauthorized",
Reason: "Unauthorized",
Details: nil,
Code: 401,
},
},
},
},
],
}
occurredContributing tests (3)
Affected runs (3)
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
VerifyDefaultIngressServingCertificateSecretWiring timed out after 10m0s: IngressController/default references &LocalObjectReference{Name:cluster-ingress-cert,}, expected secret "...full failure pattern: VerifyDefaultIngressServingCertificateSecretWiring timed out after 10m0s: IngressController/default references &LocalObjectReference{Name:cluster-ingress-cert,}, expected secret "default-ingress-cert"Signal: NoiseSignal: Noise — struct/object fragment | e2e | 2 | 2.67%2 of 75 job runs affected | NoiseSignal: Noise — struct/object fragment | Show trend detailsOct 21: 0 · Oct 22: 0 · Oct 23: 1 · Oct 24: 0 · Oct 25: 0 · Oct 26: 1 · Oct 27: 0 · Oct 28: 0 | DEV, STG | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (2)fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:256]: default-ingress-cert must exist and be referenced by IngressController/default for cluster "cluster-nightly-4-23-m28flb"
Expected success, but got an error:
<*fmt.wrapError | 0x27a8bf192000>:
VerifyDefaultIngressServingCertificateSecretWiring timed out after 10m0s: IngressController/default references &LocalObjectReference{Name:cluster-ingress-cert,}, expected secret "default-ingress-cert"
{
msg: "VerifyDefaultIngressServingCertificateSecretWiring timed out after 10m0s: IngressController/default references &LocalObjectReference{Name:cluster-ingress-cert,}, expected secret \"default-ingress-cert\"",
err: <*errors.errorString | 0x27a8c0a02f80>{
s: "IngressController/default references &LocalObjectReference{Name:cluster-ingress-cert,}, expected secret \"default-ingress-cert\"",
},
}
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:256]: default-ingress-cert must exist and be referenced by IngressController/default for cluster "cluster-nightly-4-23-m28flb"
Expected success, but got an error:
<*fmt.wrapError | 0x27a8bf192000>:
VerifyDefaultIngressServingCertificateSecretWiring timed out after 10m0s: IngressController/default references &LocalObjectReference{Name:cluster-ingress-cert,}, expected secret "default-ingress-cert"
{
msg: "VerifyDefaultIngressServingCertificateSecretWiring timed out after 10m0s: IngressController/default references &LocalObjectReference{Name:cluster-ingress-cert,}, expected secret \"default-ingress-cert\"",
err: <*errors.errorString | 0x27a8c0a02f80>{
s: "IngressController/default references &LocalObjectReference{Name:cluster-ingress-cert,}, expected secret \"default-ingress-cert\"",
},
}fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:220]: default-ingress-cert must exist and be referenced by IngressController/default for cluster "cluster-nightly-4-23-v65qkr"
Expected success, but got an error:
<*fmt.wrapError | 0xbd03a06ce0>:
VerifyDefaultIngressServingCertificateSecretWiring timed out after 10m0s: IngressController/default references &LocalObjectReference{Name:cluster-ingress-cert,}, expected secret "default-ingress-cert"
{
msg: "VerifyDefaultIngressServingCertificateSecretWiring timed out after 10m0s: IngressController/default references &LocalObjectReference{Name:cluster-ingress-cert,}, expected secret \"default-ingress-cert\"",
err: <*verifiers.pollObservationError | 0xbd03a06ca0>{
observation: <*errors.errorString | 0xbd05b0abd0>{
s: "IngressController/default references &LocalObjectReference{Name:cluster-ingress-cert,}, expected secret \"default-ingress-cert\"",
},
interruption: <*fmt.wrapError | 0xbd03a06c60>{
msg: "getting default ingress serving certificate secret openshift-ingress/default-ingress-cert: context deadline exceeded",
err: <context.deadlineExceededError>{},
},
},
}
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:220]: default-ingress-cert must exist and be referenced by IngressController/default for cluster "cluster-nightly-4-23-v65qkr"
Expected success, but got an error:
<*fmt.wrapError | 0xbd03a06ce0>:
VerifyDefaultIngressServingCertificateSecretWiring timed out after 10m0s: IngressController/default references &LocalObjectReference{Name:cluster-ingress-cert,}, expected secret "default-ingress-cert"
{
msg: "VerifyDefaultIngressServingCertificateSecretWiring timed out after 10m0s: IngressController/default references &LocalObjectReference{Name:cluster-ingress-cert,}, expected secret \"default-ingress-cert\"",
err: <*verifiers.pollObservationError | 0xbd03a06ca0>{
observation: <*errors.errorString | 0xbd05b0abd0>{
s: "IngressController/default references &LocalObjectReference{Name:cluster-ingress-cert,}, expected secret \"default-ingress-cert\"",
},
interruption: <*fmt.wrapError | 0xbd03a06c60>{
msg: "getting default ingress serving certificate secret openshift-ingress/default-ingress-cert: context deadline exceeded",
err: <context.deadlineExceededError>{},
},
},
}Contributing tests (1)
Affected runs (2)
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
VerifyDefaultIngressServingCertificateSecretWiring timed out after 10m0full failure pattern: VerifyDefaultIngressServingCertificateSecretWiring timed out after 10m0 Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 5 day(s) | e2e | 6 | 8.00%6 of 75 job runs affected | IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 5 day(s) | Show trend detailsOct 21: 0 · Oct 22: 5 · Oct 23: 7 · Oct 24: 5 · Oct 25: 0 · Oct 26: 4 · Oct 27: 1 · Oct 28: 0 | DEV, STG | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (3)fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:256]: default-ingress-cert must exist and be referenced by IngressController/default for cluster "cluster-candidate-5-0-lbfkz7"
Expected success, but got an error:
<*fmt.wrapError | 0x625ce9962c0>:
VerifyDefaultIngressServingCertificateSecretWiring timed out after 10m0.001s: IngressController/default references &LocalObjectReference{Name:cluster-ingress-cert,}, expected secret "default-ingress-cert"
{
msg: "VerifyDefaultIngressServingCertificateSecretWiring timed out after 10m0.001s: IngressController/default references &LocalObjectReference{Name:cluster-ingress-cert,}, expected secret \"default-ingress-cert\"",
err: <*errors.errorString | 0x625d05dc8e0>{
s: "IngressController/default references &LocalObjectReference{Name:cluster-ingress-cert,}, expected secret \"default-ingress-cert\"",
},
}
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:256]: default-ingress-cert must exist and be referenced by IngressController/default for cluster "cluster-candidate-5-0-lbfkz7"
Expected success, but got an error:
<*fmt.wrapError | 0x625ce9962c0>:
VerifyDefaultIngressServingCertificateSecretWiring timed out after 10m0.001s: IngressController/default references &LocalObjectReference{Name:cluster-ingress-cert,}, expected secret "default-ingress-cert"
{
msg: "VerifyDefaultIngressServingCertificateSecretWiring timed out after 10m0.001s: IngressController/default references &LocalObjectReference{Name:cluster-ingress-cert,}, expected secret \"default-ingress-cert\"",
err: <*errors.errorString | 0x625d05dc8e0>{
s: "IngressController/default references &LocalObjectReference{Name:cluster-ingress-cert,}, expected secret \"default-ingress-cert\"",
},
}fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:256]: default-ingress-cert must exist and be referenced by IngressController/default for cluster "cluster-nightly-5-0-6gm2q7"
Expected success, but got an error:
<*fmt.wrapError | 0xb6fec4b44e0>:
VerifyDefaultIngressServingCertificateSecretWiring timed out after 10m0.005s: IngressController/default references &LocalObjectReference{Name:cluster-ingress-cert,}, expected secret "default-ingress-cert"
{
msg: "VerifyDefaultIngressServingCertificateSecretWiring timed out after 10m0.005s: IngressController/default references &LocalObjectReference{Name:cluster-ingress-cert,}, expected secret \"default-ingress-cert\"",
err: <*verifiers.pollObservationError | 0xb6fec4b44c0>{
observation: <*errors.errorString | 0xb6fee7deaa0>{
s: "IngressController/default references &LocalObjectReference{Name:cluster-ingress-cert,}, expected secret \"default-ingress-cert\"",
},
interruption: <*fmt.wrapError | 0xb6fec4b44a0>{
msg: "getting default ingress serving certificate secret openshift-ingress/default-ingress-cert: context deadline exceeded",
err: <context.deadlineExceededError>{},
},
},
}
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:256]: default-ingress-cert must exist and be referenced by IngressController/default for cluster "cluster-nightly-5-0-6gm2q7"
Expected success, but got an error:
<*fmt.wrapError | 0xb6fec4b44e0>:
VerifyDefaultIngressServingCertificateSecretWiring timed out after 10m0.005s: IngressController/default references &LocalObjectReference{Name:cluster-ingress-cert,}, expected secret "default-ingress-cert"
{
msg: "VerifyDefaultIngressServingCertificateSecretWiring timed out after 10m0.005s: IngressController/default references &LocalObjectReference{Name:cluster-ingress-cert,}, expected secret \"default-ingress-cert\"",
err: <*verifiers.pollObservationError | 0xb6fec4b44c0>{
observation: <*errors.errorString | 0xb6fee7deaa0>{
s: "IngressController/default references &LocalObjectReference{Name:cluster-ingress-cert,}, expected secret \"default-ingress-cert\"",
},
interruption: <*fmt.wrapError | 0xb6fec4b44a0>{
msg: "getting default ingress serving certificate secret openshift-ingress/default-ingress-cert: context deadline exceeded",
err: <context.deadlineExceededError>{},
},
},
}fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:256]: default-ingress-cert must exist and be referenced by IngressController/default for cluster "cluster-nightly-4-22-l72l2j"
Expected success, but got an error:
<*fmt.wrapError | 0x8108240e540>:
VerifyDefaultIngressServingCertificateSecretWiring timed out after 10m0.006s: IngressController/default references &LocalObjectReference{Name:cluster-ingress-cert,}, expected secret "default-ingress-cert"
{
msg: "VerifyDefaultIngressServingCertificateSecretWiring timed out after 10m0.006s: IngressController/default references &LocalObjectReference{Name:cluster-ingress-cert,}, expected secret \"default-ingress-cert\"",
err: <*verifiers.pollObservationError | 0x8108240e520>{
observation: <*errors.errorString | 0x81084000ba0>{
s: "IngressController/default references &LocalObjectReference{Name:cluster-ingress-cert,}, expected secret \"default-ingress-cert\"",
},
interruption: <*fmt.wrapError | 0x8108240e500>{
msg: "getting default ingress serving certificate secret openshift-ingress/default-ingress-cert: context deadline exceeded",
err: <context.deadlineExceededError>{},
},
},
}
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:256]: default-ingress-cert must exist and be referenced by IngressController/default for cluster "cluster-nightly-4-22-l72l2j"
Expected success, but got an error:
<*fmt.wrapError | 0x8108240e540>:
VerifyDefaultIngressServingCertificateSecretWiring timed out after 10m0.006s: IngressController/default references &LocalObjectReference{Name:cluster-ingress-cert,}, expected secret "default-ingress-cert"
{
msg: "VerifyDefaultIngressServingCertificateSecretWiring timed out after 10m0.006s: IngressController/default references &LocalObjectReference{Name:cluster-ingress-cert,}, expected secret \"default-ingress-cert\"",
err: <*verifiers.pollObservationError | 0x8108240e520>{
observation: <*errors.errorString | 0x81084000ba0>{
s: "IngressController/default references &LocalObjectReference{Name:cluster-ingress-cert,}, expected secret \"default-ingress-cert\"",
},
interruption: <*fmt.wrapError | 0x8108240e500>{
msg: "getting default ingress serving certificate secret openshift-ingress/default-ingress-cert: context deadline exceeded",
err: <context.deadlineExceededError>{},
},
},
}Contributing tests (10)
Affected runs (6)
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
DNS for route host <route-host> did not resolve: DNS for <route-host> did not resolve within <duration> (last error: lookup <host> on <dns-server>: dial udp <ip>: i/o timeout): co...full failure pattern: DNS for route host <route-host> did not resolve: DNS for <route-host> did not resolve within <duration> (last error: lookup <host> on <dns-server>: dial udp <ip>: i/o timeout): context deadline exceeded Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 2 day(s) | e2e | 3 | 4.00%3 of 75 job runs affected | IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 2 day(s) | Show trend detailsOct 21: 0 · Oct 22: 0 · Oct 23: 5 · Oct 24: 0 · Oct 25: 7 · Oct 26: 0 · Oct 27: 0 · Oct 28: 0 | DEV, INT | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (3)fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:190]: failed to verify simple web app runs on cluster "cluster-candidate-4-20-5gtk26"
Unexpected error:
<*fmt.wrapError | 0x187dce020660>:
DNS for route host agnhost-e2e-sample-app-vvdwh.apps.aro.l5t3q5b8q6k0t1r.850o.eastus2.aroapp-hcp.io did not resolve: DNS for agnhost-e2e-sample-app-vvdwh.apps.aro.l5t3q5b8q6k0t1r.850o.eastus2.aroapp-hcp.io did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-vvdwh.apps.aro.l5t3q5b8q6k0t1r.850o.eastus2.aroapp-hcp.io on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded
{
msg: "DNS for route host agnhost-e2e-sample-app-vvdwh.apps.aro.l5t3q5b8q6k0t1r.850o.eastus2.aroapp-hcp.io did not resolve: DNS for agnhost-e2e-sample-app-vvdwh.apps.aro.l5t3q5b8q6k0t1r.850o.eastus2.aroapp-hcp.io did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-vvdwh.apps.aro.l5t3q5b8q6k0t1r.850o.eastus2.aroapp-hcp.io on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
err: <*fmt.wrapError | 0x187dce1941c0>{
msg: "DNS for agnhost-e2e-sample-app-vvdwh.apps.aro.l5t3q5b8q6k0t1r.850o.eastus2.aroapp-hcp.io did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-vvdwh.apps.aro.l5t3q5b8q6k0t1r.850o.eastus2.aroapp-hcp.io on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
err: <context.deadlineExceededError>{},
},
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:190]: failed to verify simple web app runs on cluster "cluster-candidate-4-20-5gtk26"
Unexpected error:
<*fmt.wrapError | 0x187dce020660>:
DNS for route host agnhost-e2e-sample-app-vvdwh.apps.aro.l5t3q5b8q6k0t1r.850o.eastus2.aroapp-hcp.io did not resolve: DNS for agnhost-e2e-sample-app-vvdwh.apps.aro.l5t3q5b8q6k0t1r.850o.eastus2.aroapp-hcp.io did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-vvdwh.apps.aro.l5t3q5b8q6k0t1r.850o.eastus2.aroapp-hcp.io on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded
{
msg: "DNS for route host agnhost-e2e-sample-app-vvdwh.apps.aro.l5t3q5b8q6k0t1r.850o.eastus2.aroapp-hcp.io did not resolve: DNS for agnhost-e2e-sample-app-vvdwh.apps.aro.l5t3q5b8q6k0t1r.850o.eastus2.aroapp-hcp.io did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-vvdwh.apps.aro.l5t3q5b8q6k0t1r.850o.eastus2.aroapp-hcp.io on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
err: <*fmt.wrapError | 0x187dce1941c0>{
msg: "DNS for agnhost-e2e-sample-app-vvdwh.apps.aro.l5t3q5b8q6k0t1r.850o.eastus2.aroapp-hcp.io did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-vvdwh.apps.aro.l5t3q5b8q6k0t1r.850o.eastus2.aroapp-hcp.io on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
err: <context.deadlineExceededError>{},
},
}
occurredfail [github.com/Azure/ARO-HCP/test/e2e/cluster_nodepool_active_versions.go:149]: failed to verify simple web app runs on v20261001 cluster "v20261001"
Unexpected error:
<*fmt.wrapError | 0x33aec8dca6e0>:
DNS for route host agnhost-e2e-sample-app-bmd4l.apps.aro.v20261001.ez3w.eastus2.aroapp-hcp.io did not resolve: DNS for agnhost-e2e-sample-app-bmd4l.apps.aro.v20261001.ez3w.eastus2.aroapp-hcp.io did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-bmd4l.apps.aro.v20261001.ez3w.eastus2.aroapp-hcp.io on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded
{
msg: "DNS for route host agnhost-e2e-sample-app-bmd4l.apps.aro.v20261001.ez3w.eastus2.aroapp-hcp.io did not resolve: DNS for agnhost-e2e-sample-app-bmd4l.apps.aro.v20261001.ez3w.eastus2.aroapp-hcp.io did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-bmd4l.apps.aro.v20261001.ez3w.eastus2.aroapp-hcp.io on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
err: <*fmt.wrapError | 0x33aec9a24040>{
msg: "DNS for agnhost-e2e-sample-app-bmd4l.apps.aro.v20261001.ez3w.eastus2.aroapp-hcp.io did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-bmd4l.apps.aro.v20261001.ez3w.eastus2.aroapp-hcp.io on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
err: <context.deadlineExceededError>{},
},
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_nodepool_active_versions.go:149]: failed to verify simple web app runs on v20261001 cluster "v20261001"
Unexpected error:
<*fmt.wrapError | 0x33aec8dca6e0>:
DNS for route host agnhost-e2e-sample-app-bmd4l.apps.aro.v20261001.ez3w.eastus2.aroapp-hcp.io did not resolve: DNS for agnhost-e2e-sample-app-bmd4l.apps.aro.v20261001.ez3w.eastus2.aroapp-hcp.io did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-bmd4l.apps.aro.v20261001.ez3w.eastus2.aroapp-hcp.io on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded
{
msg: "DNS for route host agnhost-e2e-sample-app-bmd4l.apps.aro.v20261001.ez3w.eastus2.aroapp-hcp.io did not resolve: DNS for agnhost-e2e-sample-app-bmd4l.apps.aro.v20261001.ez3w.eastus2.aroapp-hcp.io did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-bmd4l.apps.aro.v20261001.ez3w.eastus2.aroapp-hcp.io on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
err: <*fmt.wrapError | 0x33aec9a24040>{
msg: "DNS for agnhost-e2e-sample-app-bmd4l.apps.aro.v20261001.ez3w.eastus2.aroapp-hcp.io did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-bmd4l.apps.aro.v20261001.ez3w.eastus2.aroapp-hcp.io on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
err: <context.deadlineExceededError>{},
},
}
occurredfail [github.com/Azure/ARO-HCP/test/e2e/cluster_version_backlevel.go:201]: failed to verify simple web app on back-level cluster version 4.20
Unexpected error:
<*fmt.wrapError | 0xaa72ff3eb40>:
DNS for route host agnhost-e2e-sample-app-7hxtd.apps.aro.v2i3p0q7g9o9o1k.cnfo.eastus2.aroapp-hcp.io did not resolve: DNS for agnhost-e2e-sample-app-7hxtd.apps.aro.v2i3p0q7g9o9o1k.cnfo.eastus2.aroapp-hcp.io did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-7hxtd.apps.aro.v2i3p0q7g9o9o1k.cnfo.eastus2.aroapp-hcp.io on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded
{
msg: "DNS for route host agnhost-e2e-sample-app-7hxtd.apps.aro.v2i3p0q7g9o9o1k.cnfo.eastus2.aroapp-hcp.io did not resolve: DNS for agnhost-e2e-sample-app-7hxtd.apps.aro.v2i3p0q7g9o9o1k.cnfo.eastus2.aroapp-hcp.io did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-7hxtd.apps.aro.v2i3p0q7g9o9o1k.cnfo.eastus2.aroapp-hcp.io on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
err: <*fmt.wrapError | 0xaa72fd7c9e0>{
msg: "DNS for agnhost-e2e-sample-app-7hxtd.apps.aro.v2i3p0q7g9o9o1k.cnfo.eastus2.aroapp-hcp.io did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-7hxtd.apps.aro.v2i3p0q7g9o9o1k.cnfo.eastus2.aroapp-hcp.io on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
err: <context.deadlineExceededError>{},
},
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_version_backlevel.go:201]: failed to verify simple web app on back-level cluster version 4.20
Unexpected error:
<*fmt.wrapError | 0xaa72ff3eb40>:
DNS for route host agnhost-e2e-sample-app-7hxtd.apps.aro.v2i3p0q7g9o9o1k.cnfo.eastus2.aroapp-hcp.io did not resolve: DNS for agnhost-e2e-sample-app-7hxtd.apps.aro.v2i3p0q7g9o9o1k.cnfo.eastus2.aroapp-hcp.io did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-7hxtd.apps.aro.v2i3p0q7g9o9o1k.cnfo.eastus2.aroapp-hcp.io on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded
{
msg: "DNS for route host agnhost-e2e-sample-app-7hxtd.apps.aro.v2i3p0q7g9o9o1k.cnfo.eastus2.aroapp-hcp.io did not resolve: DNS for agnhost-e2e-sample-app-7hxtd.apps.aro.v2i3p0q7g9o9o1k.cnfo.eastus2.aroapp-hcp.io did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-7hxtd.apps.aro.v2i3p0q7g9o9o1k.cnfo.eastus2.aroapp-hcp.io on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
err: <*fmt.wrapError | 0xaa72fd7c9e0>{
msg: "DNS for agnhost-e2e-sample-app-7hxtd.apps.aro.v2i3p0q7g9o9o1k.cnfo.eastus2.aroapp-hcp.io did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-7hxtd.apps.aro.v2i3p0q7g9o9o1k.cnfo.eastus2.aroapp-hcp.io on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
err: <context.deadlineExceededError>{},
},
}
occurredContributing tests (8)
Affected runs (3)
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
DNS for route host agnhost-e2e-sample-app-<id>full failure pattern: DNS for route host agnhost-e2e-sample-app-<id> Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 2 day(s) | e2e | 3 | 4.00%3 of 75 job runs affected | IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 2 day(s) | Show trend detailsOct 21: 0 · Oct 22: 0 · Oct 23: 2 · Oct 24: 0 · Oct 25: 1 · Oct 26: 0 · Oct 27: 0 · Oct 28: 0 | DEV, INT | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (3)fail [github.com/Azure/ARO-HCP/test/e2e/default_cluster_ingress_cert.go:169]: sample application must serve the delivered default certificate
Expected success, but got an error:
<*fmt.wrapError | 0x1d8dba328160>:
DNS for route host agnhost-e2e-sample-app-52rjm.apps.aro.n5x4o1n9j0c5w9n.rqrz.eastus2.aroapp-hcp.io did not resolve: DNS for agnhost-e2e-sample-app-52rjm.apps.aro.n5x4o1n9j0c5w9n.rqrz.eastus2.aroapp-hcp.io did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-52rjm.apps.aro.n5x4o1n9j0c5w9n.rqrz.eastus2.aroapp-hcp.io on 172.30.0.10:53: no such host): context deadline exceeded
{
msg: "DNS for route host agnhost-e2e-sample-app-52rjm.apps.aro.n5x4o1n9j0c5w9n.rqrz.eastus2.aroapp-hcp.io did not resolve: DNS for agnhost-e2e-sample-app-52rjm.apps.aro.n5x4o1n9j0c5w9n.rqrz.eastus2.aroapp-hcp.io did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-52rjm.apps.aro.n5x4o1n9j0c5w9n.rqrz.eastus2.aroapp-hcp.io on 172.30.0.10:53: no such host): context deadline exceeded",
err: <*fmt.wrapError | 0x1d8dba3280e0>{
msg: "DNS for agnhost-e2e-sample-app-52rjm.apps.aro.n5x4o1n9j0c5w9n.rqrz.eastus2.aroapp-hcp.io did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-52rjm.apps.aro.n5x4o1n9j0c5w9n.rqrz.eastus2.aroapp-hcp.io on 172.30.0.10:53: no such host): context deadline exceeded",
err: <context.deadlineExceededError>{},
},
}
fail [github.com/Azure/ARO-HCP/test/e2e/default_cluster_ingress_cert.go:169]: sample application must serve the delivered default certificate
Expected success, but got an error:
<*fmt.wrapError | 0x1d8dba328160>:
DNS for route host agnhost-e2e-sample-app-52rjm.apps.aro.n5x4o1n9j0c5w9n.rqrz.eastus2.aroapp-hcp.io did not resolve: DNS for agnhost-e2e-sample-app-52rjm.apps.aro.n5x4o1n9j0c5w9n.rqrz.eastus2.aroapp-hcp.io did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-52rjm.apps.aro.n5x4o1n9j0c5w9n.rqrz.eastus2.aroapp-hcp.io on 172.30.0.10:53: no such host): context deadline exceeded
{
msg: "DNS for route host agnhost-e2e-sample-app-52rjm.apps.aro.n5x4o1n9j0c5w9n.rqrz.eastus2.aroapp-hcp.io did not resolve: DNS for agnhost-e2e-sample-app-52rjm.apps.aro.n5x4o1n9j0c5w9n.rqrz.eastus2.aroapp-hcp.io did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-52rjm.apps.aro.n5x4o1n9j0c5w9n.rqrz.eastus2.aroapp-hcp.io on 172.30.0.10:53: no such host): context deadline exceeded",
err: <*fmt.wrapError | 0x1d8dba3280e0>{
msg: "DNS for agnhost-e2e-sample-app-52rjm.apps.aro.n5x4o1n9j0c5w9n.rqrz.eastus2.aroapp-hcp.io did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-52rjm.apps.aro.n5x4o1n9j0c5w9n.rqrz.eastus2.aroapp-hcp.io on 172.30.0.10:53: no such host): context deadline exceeded",
err: <context.deadlineExceededError>{},
},
}fail [github.com/Azure/ARO-HCP/test/e2e/default_cluster_ingress_cert.go:169]: sample application must serve the delivered default certificate
Expected success, but got an error:
<*fmt.wrapError | 0x2aeb303488e0>:
DNS for route host agnhost-e2e-sample-app-g4jkv.apps.aro.s7l8k8t7p1f9x2x.w4l4.eastus2.aroapp-hcp.io did not resolve: DNS for agnhost-e2e-sample-app-g4jkv.apps.aro.s7l8k8t7p1f9x2x.w4l4.eastus2.aroapp-hcp.io did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-g4jkv.apps.aro.s7l8k8t7p1f9x2x.w4l4.eastus2.aroapp-hcp.io on 172.30.0.10:53: no such host): context deadline exceeded
{
msg: "DNS for route host agnhost-e2e-sample-app-g4jkv.apps.aro.s7l8k8t7p1f9x2x.w4l4.eastus2.aroapp-hcp.io did not resolve: DNS for agnhost-e2e-sample-app-g4jkv.apps.aro.s7l8k8t7p1f9x2x.w4l4.eastus2.aroapp-hcp.io did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-g4jkv.apps.aro.s7l8k8t7p1f9x2x.w4l4.eastus2.aroapp-hcp.io on 172.30.0.10:53: no such host): context deadline exceeded",
err: <*fmt.wrapError | 0x2aeb303488c0>{
msg: "DNS for agnhost-e2e-sample-app-g4jkv.apps.aro.s7l8k8t7p1f9x2x.w4l4.eastus2.aroapp-hcp.io did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-g4jkv.apps.aro.s7l8k8t7p1f9x2x.w4l4.eastus2.aroapp-hcp.io on 172.30.0.10:53: no such host): context deadline exceeded",
err: <context.deadlineExceededError>{},
},
}
fail [github.com/Azure/ARO-HCP/test/e2e/default_cluster_ingress_cert.go:169]: sample application must serve the delivered default certificate
Expected success, but got an error:
<*fmt.wrapError | 0x2aeb303488e0>:
DNS for route host agnhost-e2e-sample-app-g4jkv.apps.aro.s7l8k8t7p1f9x2x.w4l4.eastus2.aroapp-hcp.io did not resolve: DNS for agnhost-e2e-sample-app-g4jkv.apps.aro.s7l8k8t7p1f9x2x.w4l4.eastus2.aroapp-hcp.io did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-g4jkv.apps.aro.s7l8k8t7p1f9x2x.w4l4.eastus2.aroapp-hcp.io on 172.30.0.10:53: no such host): context deadline exceeded
{
msg: "DNS for route host agnhost-e2e-sample-app-g4jkv.apps.aro.s7l8k8t7p1f9x2x.w4l4.eastus2.aroapp-hcp.io did not resolve: DNS for agnhost-e2e-sample-app-g4jkv.apps.aro.s7l8k8t7p1f9x2x.w4l4.eastus2.aroapp-hcp.io did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-g4jkv.apps.aro.s7l8k8t7p1f9x2x.w4l4.eastus2.aroapp-hcp.io on 172.30.0.10:53: no such host): context deadline exceeded",
err: <*fmt.wrapError | 0x2aeb303488c0>{
msg: "DNS for agnhost-e2e-sample-app-g4jkv.apps.aro.s7l8k8t7p1f9x2x.w4l4.eastus2.aroapp-hcp.io did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-g4jkv.apps.aro.s7l8k8t7p1f9x2x.w4l4.eastus2.aroapp-hcp.io on 172.30.0.10:53: no such host): context deadline exceeded",
err: <context.deadlineExceededError>{},
},
}fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:526]: sample application must serve the exact customer-specified certificate with a valid Let's Encrypt chain
Expected success, but got an error:
<*fmt.wrapError | 0x33122d4ce1c0>:
DNS for route host agnhost-e2e-sample-app-2lr49.apps.aro.ingress-cert-le.xc5o.eastus2.aroapp-hcp.io did not resolve: DNS for agnhost-e2e-sample-app-2lr49.apps.aro.ingress-cert-le.xc5o.eastus2.aroapp-hcp.io did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-2lr49.apps.aro.ingress-cert-le.xc5o.eastus2.aroapp-hcp.io on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded
{
msg: "DNS for route host agnhost-e2e-sample-app-2lr49.apps.aro.ingress-cert-le.xc5o.eastus2.aroapp-hcp.io did not resolve: DNS for agnhost-e2e-sample-app-2lr49.apps.aro.ingress-cert-le.xc5o.eastus2.aroapp-hcp.io did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-2lr49.apps.aro.ingress-cert-le.xc5o.eastus2.aroapp-hcp.io on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
err: <*fmt.wrapError | 0x33122d4ce1a0>{
msg: "DNS for agnhost-e2e-sample-app-2lr49.apps.aro.ingress-cert-le.xc5o.eastus2.aroapp-hcp.io did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-2lr49.apps.aro.ingress-cert-le.xc5o.eastus2.aroapp-hcp.io on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
err: <context.deadlineExceededError>{},
},
}
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:526]: sample application must serve the exact customer-specified certificate with a valid Let's Encrypt chain
Expected success, but got an error:
<*fmt.wrapError | 0x33122d4ce1c0>:
DNS for route host agnhost-e2e-sample-app-2lr49.apps.aro.ingress-cert-le.xc5o.eastus2.aroapp-hcp.io did not resolve: DNS for agnhost-e2e-sample-app-2lr49.apps.aro.ingress-cert-le.xc5o.eastus2.aroapp-hcp.io did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-2lr49.apps.aro.ingress-cert-le.xc5o.eastus2.aroapp-hcp.io on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded
{
msg: "DNS for route host agnhost-e2e-sample-app-2lr49.apps.aro.ingress-cert-le.xc5o.eastus2.aroapp-hcp.io did not resolve: DNS for agnhost-e2e-sample-app-2lr49.apps.aro.ingress-cert-le.xc5o.eastus2.aroapp-hcp.io did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-2lr49.apps.aro.ingress-cert-le.xc5o.eastus2.aroapp-hcp.io on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
err: <*fmt.wrapError | 0x33122d4ce1a0>{
msg: "DNS for agnhost-e2e-sample-app-2lr49.apps.aro.ingress-cert-le.xc5o.eastus2.aroapp-hcp.io did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-2lr49.apps.aro.ingress-cert-le.xc5o.eastus2.aroapp-hcp.io on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
err: <context.deadlineExceededError>{},
},
}Contributing tests (2)
Affected runs (3)
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
ERROR CODE: DeadlineExceeded; detail message node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hype...full failure pattern: ERROR CODE: DeadlineExceeded; detail message node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift node pool not cached yet; provider Microsoft.RedHatOpenShift Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 2 day(s) | e2e | 3 | 4.00%3 of 75 job runs affected | IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 2 day(s) | Show trend detailsOct 21: 0 · Oct 22: 0 · Oct 23: 0 · Oct 24: 0 · Oct 25: 1 · Oct 26: 0 · Oct 27: 2 · Oct 28: 0 | DEV | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (3)fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:154]: failed to create AZ nodepool autoscale-az with autoscaling
Unexpected error:
<*fmt.wrapError | 0x341fcf187480>:
failed to create NodePool autoscale-az: failed waiting for nodepool="autoscale-az" for cluster "np-autoscale-cluster" in resourcegroup="np-autoscaling-6d2jd2rhkhpr" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2/hcpOperationStatuses/354ac5e7-358b-4ba7-b33a-800a398e3d90
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2/hcpOperationStatuses/354ac5e7-358b-4ba7-b33a-800a398e3d90",
"name": "354ac5e7-358b-4ba7-b33a-800a398e3d90",
"status": "Failed",
"startTime": "2026-10-27T21:18:08.702878641Z",
"endTime": "2026-10-27T21:37:14.197553955Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift node pool not cached yet"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2/hcpOperationStatuses/354ac5e7-358b-4ba7-b33a-800a398e3d90
correlation ID: 2d032843-08dc-4bd5-a784-b1e25c32c35a
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:154]: failed to create AZ nodepool autoscale-az with autoscaling
Unexpected error:
<*fmt.wrapError | 0x341fcf187480>:
failed to create NodePool autoscale-az: failed waiting for nodepool="autoscale-az" for cluster "np-autoscale-cluster" in resourcegroup="np-autoscaling-6d2jd2rhkhpr" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2/hcpOperationStatuses/354ac5e7-358b-4ba7-b33a-800a398e3d90
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2/hcpOperationStatuses/354ac5e7-358b-4ba7-b33a-800a398e3d90",
"name": "354ac5e7-358b-4ba7-b33a-800a398e3d90",
"status": "Failed",
"startTime": "2026-10-27T21:18:08.702878641Z",
"endTime": "2026-10-27T21:37:14.197553955Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift node pool not cached yet"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2/hcpOperationStatuses/354ac5e7-358b-4ba7-b33a-800a398e3d90
correlation ID: 2d032843-08dc-4bd5-a784-b1e25c32c35a
client request ID: unavailable
occurredfail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:154]: failed to create AZ nodepool autoscale-az with autoscaling
Unexpected error:
<*fmt.wrapError | 0x2462c6c0b540>:
failed to create NodePool autoscale-az: failed waiting for nodepool="autoscale-az" for cluster "np-autoscale-cluster" in resourcegroup="np-autoscaling-bl9lknth6jps" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2/hcpOperationStatuses/97c439db-9ae6-4e86-85bd-71d21a1fe1e7
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2/hcpOperationStatuses/97c439db-9ae6-4e86-85bd-71d21a1fe1e7",
"name": "97c439db-9ae6-4e86-85bd-71d21a1fe1e7",
"status": "Failed",
"startTime": "2026-10-27T18:48:30.242794404Z",
"endTime": "2026-10-27T19:07:38.465503369Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift node pool not cached yet"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2/hcpOperationStatuses/97c439db-9ae6-4e86-85bd-71d21a1fe1e7
correlation ID: f85455a5-fb5a-4cdd-8e2d-d003c7fb917e
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:154]: failed to create AZ nodepool autoscale-az with autoscaling
Unexpected error:
<*fmt.wrapError | 0x2462c6c0b540>:
failed to create NodePool autoscale-az: failed waiting for nodepool="autoscale-az" for cluster "np-autoscale-cluster" in resourcegroup="np-autoscaling-bl9lknth6jps" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2/hcpOperationStatuses/97c439db-9ae6-4e86-85bd-71d21a1fe1e7
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2/hcpOperationStatuses/97c439db-9ae6-4e86-85bd-71d21a1fe1e7",
"name": "97c439db-9ae6-4e86-85bd-71d21a1fe1e7",
"status": "Failed",
"startTime": "2026-10-27T18:48:30.242794404Z",
"endTime": "2026-10-27T19:07:38.465503369Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift node pool not cached yet"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2/hcpOperationStatuses/97c439db-9ae6-4e86-85bd-71d21a1fe1e7
correlation ID: f85455a5-fb5a-4cdd-8e2d-d003c7fb917e
client request ID: unavailable
occurredfail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:154]: failed to create AZ nodepool autoscale-az with autoscaling
Unexpected error:
<*fmt.wrapError | 0x22a837c45580>:
failed to create NodePool autoscale-az: failed waiting for nodepool="autoscale-az" for cluster "np-autoscale-cluster" in resourcegroup="np-autoscaling-m8kxkqwbf5w7" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2/hcpOperationStatuses/64b6150d-1b43-49ba-9a27-51fef85bb940
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2/hcpOperationStatuses/64b6150d-1b43-49ba-9a27-51fef85bb940",
"name": "64b6150d-1b43-49ba-9a27-51fef85bb940",
"status": "Failed",
"startTime": "2026-10-25T09:39:04.403310399Z",
"endTime": "2026-10-25T09:58:04.042996716Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift node pool not cached yet"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2/hcpOperationStatuses/64b6150d-1b43-49ba-9a27-51fef85bb940
correlation ID: a329644d-c403-4d74-b2b5-fcc949c9882c
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:154]: failed to create AZ nodepool autoscale-az with autoscaling
Unexpected error:
<*fmt.wrapError | 0x22a837c45580>:
failed to create NodePool autoscale-az: failed waiting for nodepool="autoscale-az" for cluster "np-autoscale-cluster" in resourcegroup="np-autoscaling-m8kxkqwbf5w7" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2/hcpOperationStatuses/64b6150d-1b43-49ba-9a27-51fef85bb940
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2/hcpOperationStatuses/64b6150d-1b43-49ba-9a27-51fef85bb940",
"name": "64b6150d-1b43-49ba-9a27-51fef85bb940",
"status": "Failed",
"startTime": "2026-10-25T09:39:04.403310399Z",
"endTime": "2026-10-25T09:58:04.042996716Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift node pool not cached yet"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2/hcpOperationStatuses/64b6150d-1b43-49ba-9a27-51fef85bb940
correlation ID: a329644d-c403-4d74-b2b5-fcc949c9882c
client request ID: unavailable
occurredContributing tests (1)
Affected runs (3)
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
ERROR CODE: InternalServerError; detail message cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster still exists; [clusterServic...full failure pattern: ERROR CODE: InternalServerError; detail message cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster still exists; [clusterServiceStatus] state is "uninstalling"; [descendantResources] remaining resources: nodePools, serviceProviderClusters, serviceProviderNodePools; provider Microsoft.RedHatOpenShift Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 2 day(s) | e2e | 3 | 4.00%3 of 75 job runs affected | IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 2 day(s) | Show trend detailsOct 21: 0 · Oct 22: 2 · Oct 23: 0 · Oct 24: 0 · Oct 25: 0 · Oct 26: 0 · Oct 27: 0 · Oct 28: 1 | DEV | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (3)fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:310]: Unexpected error:
<*errors.joinError | 0x144b1f0220a8>:
resource group "ingress-cert-le-pqlv4rswg88h": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="ingress-cert-le" in resourcegroup="ingress-cert-le-pqlv4rswg88h" to finish deleting: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/f2c1c705-5b8d-4bbe-98a3-39f7905fcad2
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: InternalServerError
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/f2c1c705-5b8d-4bbe-98a3-39f7905fcad2",
"name": "f2c1c705-5b8d-4bbe-98a3-39f7905fcad2",
"status": "Failed",
"startTime": "2026-10-28T14:05:17.896484451Z",
"endTime": "2026-10-28T14:29:22.39155595Z",
"error": {
"code": "InternalServerError",
"message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2tnqroht6vs8i90m1kekfo1uicveo895 still exists (deletion dispatched at 2026-10-28T14:05:18Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/nodePools, 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/serviceProviderClusters, 1 microsoft.redhatopenshift/hcpopenshiftclusters/nodepools/serviceProviderNodePools"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/f2c1c705-5b8d-4bbe-98a3-39f7905fcad2
correlation ID: 3972a876-9ac5-4cc1-a07c-3501cf7d75bd
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:310]: Unexpected error:
<*errors.joinError | 0x144b1f0220a8>:
resource group "ingress-cert-le-pqlv4rswg88h": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="ingress-cert-le" in resourcegroup="ingress-cert-le-pqlv4rswg88h" to finish deleting: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/f2c1c705-5b8d-4bbe-98a3-39f7905fcad2
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: InternalServerError
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/f2c1c705-5b8d-4bbe-98a3-39f7905fcad2",
"name": "f2c1c705-5b8d-4bbe-98a3-39f7905fcad2",
"status": "Failed",
"startTime": "2026-10-28T14:05:17.896484451Z",
"endTime": "2026-10-28T14:29:22.39155595Z",
"error": {
"code": "InternalServerError",
"message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2tnqroht6vs8i90m1kekfo1uicveo895 still exists (deletion dispatched at 2026-10-28T14:05:18Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/nodePools, 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/serviceProviderClusters, 1 microsoft.redhatopenshift/hcpopenshiftclusters/nodepools/serviceProviderNodePools"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/f2c1c705-5b8d-4bbe-98a3-39f7905fcad2
correlation ID: 3972a876-9ac5-4cc1-a07c-3501cf7d75bd
client request ID: unavailable
occurredfail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:310]: Unexpected error:
<*errors.joinError | 0x3eb0083dc060>:
resource group "private-keyvault-dtwhsct926n6": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="private-kv-cluster" in resourcegroup="private-keyvault-dtwhsct926n6" to finish deleting: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westeurope/hcpOperationStatuses/baf15df3-25ce-45bb-9f9e-ce6a52637752
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: InternalServerError
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westeurope/hcpOperationStatuses/baf15df3-25ce-45bb-9f9e-ce6a52637752",
"name": "baf15df3-25ce-45bb-9f9e-ce6a52637752",
"status": "Failed",
"startTime": "2026-10-22T05:08:07.811493592Z",
"endTime": "2026-10-22T05:32:16.130563503Z",
"error": {
"code": "InternalServerError",
"message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2tjkddd25goskik0nf1lqaqp6mgu12qg still exists (deletion dispatched at 2026-10-22T05:08:08Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/nodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/nodepools/serviceProviderNodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westeurope/hcpOperationStatuses/baf15df3-25ce-45bb-9f9e-ce6a52637752
correlation ID: a461bf17-668e-42e4-a6f7-c14803c5d886
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:310]: Unexpected error:
<*errors.joinError | 0x3eb0083dc060>:
resource group "private-keyvault-dtwhsct926n6": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="private-kv-cluster" in resourcegroup="private-keyvault-dtwhsct926n6" to finish deleting: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westeurope/hcpOperationStatuses/baf15df3-25ce-45bb-9f9e-ce6a52637752
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: InternalServerError
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westeurope/hcpOperationStatuses/baf15df3-25ce-45bb-9f9e-ce6a52637752",
"name": "baf15df3-25ce-45bb-9f9e-ce6a52637752",
"status": "Failed",
"startTime": "2026-10-22T05:08:07.811493592Z",
"endTime": "2026-10-22T05:32:16.130563503Z",
"error": {
"code": "InternalServerError",
"message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2tjkddd25goskik0nf1lqaqp6mgu12qg still exists (deletion dispatched at 2026-10-22T05:08:08Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/nodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/nodepools/serviceProviderNodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westeurope/hcpOperationStatuses/baf15df3-25ce-45bb-9f9e-ce6a52637752
correlation ID: a461bf17-668e-42e4-a6f7-c14803c5d886
client request ID: unavailable
occurredfail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:310]: Unexpected error:
<*errors.joinError | 0x16e186f694b8>:
resource group "ephemeral-osdisk-k7jcdd2kqwvh": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="ephemeral-disk" in resourcegroup="ephemeral-osdisk-k7jcdd2kqwvh" to finish deleting: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/36a51701-9d7a-4ad4-860f-d0b45481d5d2
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: InternalServerError
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/36a51701-9d7a-4ad4-860f-d0b45481d5d2",
"name": "36a51701-9d7a-4ad4-860f-d0b45481d5d2",
"status": "Failed",
"startTime": "2026-10-22T03:23:14.821436082Z",
"endTime": "2026-10-22T03:47:22.914912373Z",
"error": {
"code": "InternalServerError",
"message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2tjj0cikkek0bc3m05t20hehmct394q5 still exists (deletion dispatched at 2026-10-22T03:23:15Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/nodePools, 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/serviceProviderClusters, 1 microsoft.redhatopenshift/hcpopenshiftclusters/nodepools/serviceProviderNodePools"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/36a51701-9d7a-4ad4-860f-d0b45481d5d2
correlation ID: b94afee3-9d6b-4695-a5b0-3f0b5bdf76c0
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:310]: Unexpected error:
<*errors.joinError | 0x16e186f694b8>:
resource group "ephemeral-osdisk-k7jcdd2kqwvh": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="ephemeral-disk" in resourcegroup="ephemeral-osdisk-k7jcdd2kqwvh" to finish deleting: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/36a51701-9d7a-4ad4-860f-d0b45481d5d2
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: InternalServerError
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/36a51701-9d7a-4ad4-860f-d0b45481d5d2",
"name": "36a51701-9d7a-4ad4-860f-d0b45481d5d2",
"status": "Failed",
"startTime": "2026-10-22T03:23:14.821436082Z",
"endTime": "2026-10-22T03:47:22.914912373Z",
"error": {
"code": "InternalServerError",
"message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2tjj0cikkek0bc3m05t20hehmct394q5 still exists (deletion dispatched at 2026-10-22T03:23:15Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/nodePools, 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/serviceProviderClusters, 1 microsoft.redhatopenshift/hcpopenshiftclusters/nodepools/serviceProviderNodePools"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/36a51701-9d7a-4ad4-860f-d0b45481d5d2
correlation ID: b94afee3-9d6b-4695-a5b0-3f0b5bdf76c0
client request ID: unavailable
occurredContributing tests (3)
Affected runs (3)
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedC...full failure pattern: ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable; hosted cluster degraded: UnavailableReplicas: multiple components; provider Microsoft.RedHatOpenShift Signal: IndeterminateSignal: Indeterminate — seen in 2 prior week(s); active 2 day(s) | e2e | 2 | 2.67%2 of 75 job runs affected | IndeterminateSignal: Indeterminate — seen in 2 prior week(s); active 2 day(s) | Show trend detailsOct 21: 0 · Oct 22: 9 · Oct 23: 0 · Oct 24: 0 · Oct 25: 0 · Oct 26: 0 · Oct 27: 0 · Oct 28: 8 | DEV | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (3)fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_cordon.go:92]: failed to create HCP cluster cordon-hcp-cluster
Unexpected error:
<*fmt.wrapError | 0x371784ecc8e0>:
failed to create HCP cluster cordon-hcp-cluster: failed waiting for cluster="cordon-hcp-cluster" in resourcegroup="rg-cordon-test-xk2p7njp5dsf" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/b13f5e9a-eaea-4399-8cfa-fbd539c1236b
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/b13f5e9a-eaea-4399-8cfa-fbd539c1236b",
"name": "b13f5e9a-eaea-4399-8cfa-fbd539c1236b",
"status": "Failed",
"startTime": "2026-10-28T15:21:06.575502401Z",
"endTime": "2026-10-28T15:50:07.653220479Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: openshift-controller-manager, hosted-cluster-config-operator, certified-operators-catalog, cluster-storage-operator, cluster-network-operator, cluster-node-tuning-operator, cluster-image-registry-operator, redhat-marketplace-catalog, machine-approver, kube-controller-manager, dns-operator, cluster-version-operator, openshift-route-controller-manager, redhat-operators-catalog, csi-snapshot-controller-operator, catalog-operator, kube-scheduler, azure-cloud-controller-manager, olm-operator, community-operators-catalog, ingress-operator, konnectivity-agent, packageserver, cluster-policy-controller, control-plane-pki-operator; hosted cluster degraded: UnavailableReplicas: [azure-cloud-controller-manager deployment has 1 unavailable replicas, catalog-operator deployment has 1 unavailable replicas, control-plane-pki-operator deployment has 1 unavailable replicas, hosted-cluster-config-operator deployment has 1 unavailable replicas, konnectivity-agent deployment has 1 unavailable replicas, kube-controller-manager deployment has 1 unavailable replicas, kube-scheduler deployment has 1 unavailable replicas, machine-approver deployment has 1 unavailable replicas, olm-operator deployment has 1 unavailable replicas, openshift-apiserver deployment has 1 unavailable replicas, packageserver deployment has 3 unavailable replicas]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/b13f5e9a-eaea-4399-8cfa-fbd539c1236b
correlation ID: 5f2b42c0-ed6b-46a2-99e5-52df9f0918c9
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_cordon.go:92]: failed to create HCP cluster cordon-hcp-cluster
Unexpected error:
<*fmt.wrapError | 0x371784ecc8e0>:
failed to create HCP cluster cordon-hcp-cluster: failed waiting for cluster="cordon-hcp-cluster" in resourcegroup="rg-cordon-test-xk2p7njp5dsf" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/b13f5e9a-eaea-4399-8cfa-fbd539c1236b
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/b13f5e9a-eaea-4399-8cfa-fbd539c1236b",
"name": "b13f5e9a-eaea-4399-8cfa-fbd539c1236b",
"status": "Failed",
"startTime": "2026-10-28T15:21:06.575502401Z",
"endTime": "2026-10-28T15:50:07.653220479Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: openshift-controller-manager, hosted-cluster-config-operator, certified-operators-catalog, cluster-storage-operator, cluster-network-operator, cluster-node-tuning-operator, cluster-image-registry-operator, redhat-marketplace-catalog, machine-approver, kube-controller-manager, dns-operator, cluster-version-operator, openshift-route-controller-manager, redhat-operators-catalog, csi-snapshot-controller-operator, catalog-operator, kube-scheduler, azure-cloud-controller-manager, olm-operator, community-operators-catalog, ingress-operator, konnectivity-agent, packageserver, cluster-policy-controller, control-plane-pki-operator; hosted cluster degraded: UnavailableReplicas: [azure-cloud-controller-manager deployment has 1 unavailable replicas, catalog-operator deployment has 1 unavailable replicas, control-plane-pki-operator deployment has 1 unavailable replicas, hosted-cluster-config-operator deployment has 1 unavailable replicas, konnectivity-agent deployment has 1 unavailable replicas, kube-controller-manager deployment has 1 unavailable replicas, kube-scheduler deployment has 1 unavailable replicas, machine-approver deployment has 1 unavailable replicas, olm-operator deployment has 1 unavailable replicas, openshift-apiserver deployment has 1 unavailable replicas, packageserver deployment has 3 unavailable replicas]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/b13f5e9a-eaea-4399-8cfa-fbd539c1236b
correlation ID: 5f2b42c0-ed6b-46a2-99e5-52df9f0918c9
client request ID: unavailable
occurredfail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:443]: failed to create HCP cluster np-skip-minor-wbzs2x at version 4.22.20
Unexpected error:
<*fmt.wrapError | 0x105f86c0c5c0>:
failed to create HCP cluster np-skip-minor-wbzs2x: failed waiting for cluster="np-skip-minor-wbzs2x" in resourcegroup="rg-np-skip-minor-wbzs2x-9657t9v84pkg" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/4bd9db08-0d31-47b6-8f6e-287e8e8d4fe4
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/4bd9db08-0d31-47b6-8f6e-287e8e8d4fe4",
"name": "4bd9db08-0d31-47b6-8f6e-287e8e8d4fe4",
"status": "Failed",
"startTime": "2026-10-28T15:21:40.618908142Z",
"endTime": "2026-10-28T15:50:47.631173688Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: csi-snapshot-controller-operator, azure-cloud-controller-manager, machine-approver, control-plane-pki-operator, cluster-node-tuning-operator, cluster-version-operator, openshift-route-controller-manager, cluster-network-operator, certified-operators-catalog, hosted-cluster-config-operator, ingress-operator, konnectivity-agent, redhat-operators-catalog, cluster-storage-operator, dns-operator, cluster-image-registry-operator, kube-storage-version-migrator, catalog-operator, olm-operator, openshift-controller-manager, cluster-policy-controller, packageserver, community-operators-catalog; hosted cluster degraded: UnavailableReplicas: [azure-cloud-controller-manager deployment has 1 unavailable replicas, catalog-operator deployment has 1 unavailable replicas, certified-operators-catalog deployment has 1 unavailable replicas, cluster-image-registry-operator deployment has 1 unavailable replicas, cluster-network-operator deployment has 1 unavailable replicas, cluster-node-tuning-operator deployment has 1 unavailable replicas, cluster-policy-controller deployment has 1 unavailable replicas, cluster-storage-operator deployment has 1 unavailable replicas, cluster-version-operator deployment has 1 unavailable replicas, community-operators-catalog deployment has 1 unavailable replicas, control-plane-pki-operator deployment has 1 unavailable replicas, csi-snapshot-controller-operator deployment has 1 unavailable replicas, dns-operator deployment has 1 unavailable replicas, hosted-cluster-config-operator deployment has 1 unavailable replicas, ingress-operator deployment has 1 unavailable replicas, konnectivity-agent deployment has 1 unavailable replicas, kube-storage-version-migrator deployment has 1 unavailable replicas, machine-approver deployment has 1 unavailable replicas, olm-operator deployment has 1 unavailable replicas, openshift-controller-manager deployment has 1 unavailable replicas, openshift-route-controller-manager deployment has 1 unavailable replicas, packageserver deployment has 3 unavailable replicas, redhat-operators-catalog deployment has 1 unavailable replicas]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/4bd9db08-0d31-47b6-8f6e-287e8e8d4fe4
correlation ID: 0acb7fcf-366a-4480-a74c-2c2b50fa0aa0
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:443]: failed to create HCP cluster np-skip-minor-wbzs2x at version 4.22.20
Unexpected error:
<*fmt.wrapError | 0x105f86c0c5c0>:
failed to create HCP cluster np-skip-minor-wbzs2x: failed waiting for cluster="np-skip-minor-wbzs2x" in resourcegroup="rg-np-skip-minor-wbzs2x-9657t9v84pkg" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/4bd9db08-0d31-47b6-8f6e-287e8e8d4fe4
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/4bd9db08-0d31-47b6-8f6e-287e8e8d4fe4",
"name": "4bd9db08-0d31-47b6-8f6e-287e8e8d4fe4",
"status": "Failed",
"startTime": "2026-10-28T15:21:40.618908142Z",
"endTime": "2026-10-28T15:50:47.631173688Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: csi-snapshot-controller-operator, azure-cloud-controller-manager, machine-approver, control-plane-pki-operator, cluster-node-tuning-operator, cluster-version-operator, openshift-route-controller-manager, cluster-network-operator, certified-operators-catalog, hosted-cluster-config-operator, ingress-operator, konnectivity-agent, redhat-operators-catalog, cluster-storage-operator, dns-operator, cluster-image-registry-operator, kube-storage-version-migrator, catalog-operator, olm-operator, openshift-controller-manager, cluster-policy-controller, packageserver, community-operators-catalog; hosted cluster degraded: UnavailableReplicas: [azure-cloud-controller-manager deployment has 1 unavailable replicas, catalog-operator deployment has 1 unavailable replicas, certified-operators-catalog deployment has 1 unavailable replicas, cluster-image-registry-operator deployment has 1 unavailable replicas, cluster-network-operator deployment has 1 unavailable replicas, cluster-node-tuning-operator deployment has 1 unavailable replicas, cluster-policy-controller deployment has 1 unavailable replicas, cluster-storage-operator deployment has 1 unavailable replicas, cluster-version-operator deployment has 1 unavailable replicas, community-operators-catalog deployment has 1 unavailable replicas, control-plane-pki-operator deployment has 1 unavailable replicas, csi-snapshot-controller-operator deployment has 1 unavailable replicas, dns-operator deployment has 1 unavailable replicas, hosted-cluster-config-operator deployment has 1 unavailable replicas, ingress-operator deployment has 1 unavailable replicas, konnectivity-agent deployment has 1 unavailable replicas, kube-storage-version-migrator deployment has 1 unavailable replicas, machine-approver deployment has 1 unavailable replicas, olm-operator deployment has 1 unavailable replicas, openshift-controller-manager deployment has 1 unavailable replicas, openshift-route-controller-manager deployment has 1 unavailable replicas, packageserver deployment has 3 unavailable replicas, redhat-operators-catalog deployment has 1 unavailable replicas]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/4bd9db08-0d31-47b6-8f6e-287e8e8d4fe4
correlation ID: 0acb7fcf-366a-4480-a74c-2c2b50fa0aa0
client request ID: unavailable
occurredfail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_v20260901preview.go:86]: failed to create HCP cluster "v20260901" via v20260901preview
Unexpected error:
<*fmt.wrapError | 0x188b1ffab580>:
failed to create HCP cluster v20260901: failed waiting for cluster="v20260901" in resourcegroup="v20260901preview-pptld2m557n2" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/160dbbc6-24d7-46a4-a575-459e58e52cbb
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/160dbbc6-24d7-46a4-a575-459e58e52cbb",
"name": "160dbbc6-24d7-46a4-a575-459e58e52cbb",
"status": "Failed",
"startTime": "2026-10-28T13:42:52.080654002Z",
"endTime": "2026-10-28T14:12:00.912767678Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: openshift-controller-manager, azure-cloud-controller-manager, dns-operator, cluster-image-registry-operator, packageserver, community-operators-catalog, machine-approver, openshift-route-controller-manager, kube-controller-manager, kube-scheduler, hosted-cluster-config-operator, ingress-operator, cluster-policy-controller, cluster-storage-operator, cluster-network-operator, control-plane-pki-operator, cluster-version-operator, konnectivity-agent, redhat-operators-catalog, capi-provider, cluster-node-tuning-operator, csi-snapshot-controller-operator, olm-operator, redhat-marketplace-catalog, certified-operators-catalog, catalog-operator; hosted cluster degraded: UnavailableReplicas: [azure-cloud-controller-manager deployment has 1 unavailable replicas, capi-provider deployment has 2 unavailable replicas, catalog-operator deployment has 1 unavailable replicas, control-plane-pki-operator deployment has 1 unavailable replicas, hosted-cluster-config-operator deployment has 1 unavailable replicas, konnectivity-agent deployment has 1 unavailable replicas, kube-controller-manager deployment has 1 unavailable replicas, kube-scheduler deployment has 1 unavailable replicas, machine-approver deployment has 1 unavailable replicas, olm-operator deployment has 1 unavailable replicas, openshift-apiserver deployment has 1 unavailable replicas, packageserver deployment has 3 unavailable replicas]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/160dbbc6-24d7-46a4-a575-459e58e52cbb
correlation ID: b0bdcafa-1d32-4743-b479-c3a60aa49bed
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_v20260901preview.go:86]: failed to create HCP cluster "v20260901" via v20260901preview
Unexpected error:
<*fmt.wrapError | 0x188b1ffab580>:
failed to create HCP cluster v20260901: failed waiting for cluster="v20260901" in resourcegroup="v20260901preview-pptld2m557n2" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/160dbbc6-24d7-46a4-a575-459e58e52cbb
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/160dbbc6-24d7-46a4-a575-459e58e52cbb",
"name": "160dbbc6-24d7-46a4-a575-459e58e52cbb",
"status": "Failed",
"startTime": "2026-10-28T13:42:52.080654002Z",
"endTime": "2026-10-28T14:12:00.912767678Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: openshift-controller-manager, azure-cloud-controller-manager, dns-operator, cluster-image-registry-operator, packageserver, community-operators-catalog, machine-approver, openshift-route-controller-manager, kube-controller-manager, kube-scheduler, hosted-cluster-config-operator, ingress-operator, cluster-policy-controller, cluster-storage-operator, cluster-network-operator, control-plane-pki-operator, cluster-version-operator, konnectivity-agent, redhat-operators-catalog, capi-provider, cluster-node-tuning-operator, csi-snapshot-controller-operator, olm-operator, redhat-marketplace-catalog, certified-operators-catalog, catalog-operator; hosted cluster degraded: UnavailableReplicas: [azure-cloud-controller-manager deployment has 1 unavailable replicas, capi-provider deployment has 2 unavailable replicas, catalog-operator deployment has 1 unavailable replicas, control-plane-pki-operator deployment has 1 unavailable replicas, hosted-cluster-config-operator deployment has 1 unavailable replicas, konnectivity-agent deployment has 1 unavailable replicas, kube-controller-manager deployment has 1 unavailable replicas, kube-scheduler deployment has 1 unavailable replicas, machine-approver deployment has 1 unavailable replicas, olm-operator deployment has 1 unavailable replicas, openshift-apiserver deployment has 1 unavailable replicas, packageserver deployment has 3 unavailable replicas]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/160dbbc6-24d7-46a4-a575-459e58e52cbb
correlation ID: b0bdcafa-1d32-4743-b479-c3a60aa49bed
client request ID: unavailable
occurredContributing tests (14)
Affected runs (2)
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedC...full failure pattern: ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider, ignition-server]; provider Microsoft.RedHatOpenShift Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 2 day(s) | e2e | 2 | 2.67%2 of 75 job runs affected | IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 2 day(s) | Show trend detailsOct 21: 0 · Oct 22: 7 · Oct 23: 0 · Oct 24: 0 · Oct 25: 0 · Oct 26: 0 · Oct 27: 0 · Oct 28: 7 | none | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (3)fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:126]: HCP cluster rg-candidate-4-22-28r6sfshkdrv/cluster-candidate-4-22-gcsx6w should provision
Unexpected error:
<*fmt.wrapError | 0xe993535c600>:
failed to create HCP cluster cluster-candidate-4-22-gcsx6w: failed waiting for cluster="cluster-candidate-4-22-gcsx6w" in resourcegroup="rg-candidate-4-22-28r6sfshkdrv" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/626ada50-804b-4cf3-9aac-6865d44b51f2
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/626ada50-804b-4cf3-9aac-6865d44b51f2",
"name": "626ada50-804b-4cf3-9aac-6865d44b51f2",
"status": "Failed",
"startTime": "2026-10-28T14:39:47.572236196Z",
"endTime": "2026-10-28T15:08:48.605120634Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider deployment has 1 unavailable replicas, ignition-server deployment has 1 unavailable replicas]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/626ada50-804b-4cf3-9aac-6865d44b51f2
correlation ID: 8c0ce41d-733b-4d14-b437-6022b87a6a82
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:126]: HCP cluster rg-candidate-4-22-28r6sfshkdrv/cluster-candidate-4-22-gcsx6w should provision
Unexpected error:
<*fmt.wrapError | 0xe993535c600>:
failed to create HCP cluster cluster-candidate-4-22-gcsx6w: failed waiting for cluster="cluster-candidate-4-22-gcsx6w" in resourcegroup="rg-candidate-4-22-28r6sfshkdrv" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/626ada50-804b-4cf3-9aac-6865d44b51f2
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/626ada50-804b-4cf3-9aac-6865d44b51f2",
"name": "626ada50-804b-4cf3-9aac-6865d44b51f2",
"status": "Failed",
"startTime": "2026-10-28T14:39:47.572236196Z",
"endTime": "2026-10-28T15:08:48.605120634Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider deployment has 1 unavailable replicas, ignition-server deployment has 1 unavailable replicas]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/626ada50-804b-4cf3-9aac-6865d44b51f2
correlation ID: 8c0ce41d-733b-4d14-b437-6022b87a6a82
client request ID: unavailable
occurredfail [github.com/Azure/ARO-HCP/test/e2e/arm64_nodepool.go:88]: failed to create HCP cluster "arm64-vm-hcp-cluster"
Unexpected error:
<*fmt.wrapError | 0x167c557a6660>:
failed to create HCP cluster arm64-vm-hcp-cluster: failed waiting for cluster="arm64-vm-hcp-cluster" in resourcegroup="arm64-vm-cluster-wjkxs2684hd7" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/9a50ba0b-b907-416a-b2d1-94636b24ec6d
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/9a50ba0b-b907-416a-b2d1-94636b24ec6d",
"name": "9a50ba0b-b907-416a-b2d1-94636b24ec6d",
"status": "Failed",
"startTime": "2026-10-28T13:42:47.959137424Z",
"endTime": "2026-10-28T14:11:50.925878227Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider deployment has 1 unavailable replicas, ignition-server deployment has 1 unavailable replicas]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/9a50ba0b-b907-416a-b2d1-94636b24ec6d
correlation ID: 9f148a4c-d977-4e05-985d-74dfc49e11a0
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/arm64_nodepool.go:88]: failed to create HCP cluster "arm64-vm-hcp-cluster"
Unexpected error:
<*fmt.wrapError | 0x167c557a6660>:
failed to create HCP cluster arm64-vm-hcp-cluster: failed waiting for cluster="arm64-vm-hcp-cluster" in resourcegroup="arm64-vm-cluster-wjkxs2684hd7" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/9a50ba0b-b907-416a-b2d1-94636b24ec6d
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/9a50ba0b-b907-416a-b2d1-94636b24ec6d",
"name": "9a50ba0b-b907-416a-b2d1-94636b24ec6d",
"status": "Failed",
"startTime": "2026-10-28T13:42:47.959137424Z",
"endTime": "2026-10-28T14:11:50.925878227Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider deployment has 1 unavailable replicas, ignition-server deployment has 1 unavailable replicas]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/9a50ba0b-b907-416a-b2d1-94636b24ec6d
correlation ID: 9f148a4c-d977-4e05-985d-74dfc49e11a0
client request ID: unavailable
occurredfail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:133]: failed to create HCP cluster "cp-ystream-upgrade-4-23-p5rwk6" at install version 4.22
Unexpected error:
<*fmt.wrapError | 0x18410323d680>:
failed to create HCP cluster cp-ystream-upgrade-4-23-p5rwk6: failed waiting for cluster="cp-ystream-upgrade-4-23-p5rwk6" in resourcegroup="rg-cp-ystream-upgrade-4-23-89hkwz6drhgn" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/c2309fcd-9ee5-4824-b941-841a87860930
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/c2309fcd-9ee5-4824-b941-841a87860930",
"name": "c2309fcd-9ee5-4824-b941-841a87860930",
"status": "Failed",
"startTime": "2026-10-28T14:40:40.163664046Z",
"endTime": "2026-10-28T15:09:48.585939528Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider deployment has 1 unavailable replicas, ignition-server deployment has 1 unavailable replicas]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/c2309fcd-9ee5-4824-b941-841a87860930
correlation ID: 7ec55f9b-2f4a-4e1d-8780-4867568c3427
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:133]: failed to create HCP cluster "cp-ystream-upgrade-4-23-p5rwk6" at install version 4.22
Unexpected error:
<*fmt.wrapError | 0x18410323d680>:
failed to create HCP cluster cp-ystream-upgrade-4-23-p5rwk6: failed waiting for cluster="cp-ystream-upgrade-4-23-p5rwk6" in resourcegroup="rg-cp-ystream-upgrade-4-23-89hkwz6drhgn" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/c2309fcd-9ee5-4824-b941-841a87860930
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/c2309fcd-9ee5-4824-b941-841a87860930",
"name": "c2309fcd-9ee5-4824-b941-841a87860930",
"status": "Failed",
"startTime": "2026-10-28T14:40:40.163664046Z",
"endTime": "2026-10-28T15:09:48.585939528Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider deployment has 1 unavailable replicas, ignition-server deployment has 1 unavailable replicas]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/c2309fcd-9ee5-4824-b941-841a87860930
correlation ID: 7ec55f9b-2f4a-4e1d-8780-4867568c3427
client request ID: unavailable
occurredContributing tests (13)
Affected runs (2)
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedC...full failure pattern: ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider, router]; provider Microsoft.RedHatOpenShift Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 2 day(s) | e2e | 2 | 2.67%2 of 75 job runs affected | IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 2 day(s) | Show trend detailsOct 21: 0 · Oct 22: 4 · Oct 23: 0 · Oct 24: 0 · Oct 25: 0 · Oct 26: 0 · Oct 27: 0 · Oct 28: 8 | none | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (3)fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:126]: HCP cluster rg-candidate-5-0-tsmb9zwhf6np/cluster-candidate-5-0-qrqt4m should provision
Unexpected error:
<*fmt.wrapError | 0x848b1c9a440>:
failed to create HCP cluster cluster-candidate-5-0-qrqt4m: failed waiting for cluster="cluster-candidate-5-0-qrqt4m" in resourcegroup="rg-candidate-5-0-tsmb9zwhf6np" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/ff4d1e16-4c98-4cf9-acfe-6d00538378fd
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/ff4d1e16-4c98-4cf9-acfe-6d00538378fd",
"name": "ff4d1e16-4c98-4cf9-acfe-6d00538378fd",
"status": "Failed",
"startTime": "2026-10-28T14:39:51.38832954Z",
"endTime": "2026-10-28T15:08:58.636801847Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider deployment has 1 unavailable replicas, router deployment has 1 unavailable replicas]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/ff4d1e16-4c98-4cf9-acfe-6d00538378fd
correlation ID: 9841ce5e-79da-4501-8301-7fd87b492c29
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:126]: HCP cluster rg-candidate-5-0-tsmb9zwhf6np/cluster-candidate-5-0-qrqt4m should provision
Unexpected error:
<*fmt.wrapError | 0x848b1c9a440>:
failed to create HCP cluster cluster-candidate-5-0-qrqt4m: failed waiting for cluster="cluster-candidate-5-0-qrqt4m" in resourcegroup="rg-candidate-5-0-tsmb9zwhf6np" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/ff4d1e16-4c98-4cf9-acfe-6d00538378fd
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/ff4d1e16-4c98-4cf9-acfe-6d00538378fd",
"name": "ff4d1e16-4c98-4cf9-acfe-6d00538378fd",
"status": "Failed",
"startTime": "2026-10-28T14:39:51.38832954Z",
"endTime": "2026-10-28T15:08:58.636801847Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider deployment has 1 unavailable replicas, router deployment has 1 unavailable replicas]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/ff4d1e16-4c98-4cf9-acfe-6d00538378fd
correlation ID: 9841ce5e-79da-4501-8301-7fd87b492c29
client request ID: unavailable
occurredfail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:126]: HCP cluster rg-candidate-5-1-c9tfj9kmq2zp/cluster-candidate-5-1-kdsdqq should provision
Unexpected error:
<*fmt.wrapError | 0x158d9065e120>:
failed to create HCP cluster cluster-candidate-5-1-kdsdqq: failed waiting for cluster="cluster-candidate-5-1-kdsdqq" in resourcegroup="rg-candidate-5-1-c9tfj9kmq2zp" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/82df0ed4-6ad4-45af-825d-883fcc05e886
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/82df0ed4-6ad4-45af-825d-883fcc05e886",
"name": "82df0ed4-6ad4-45af-825d-883fcc05e886",
"status": "Failed",
"startTime": "2026-10-28T14:40:01.853789521Z",
"endTime": "2026-10-28T15:09:08.621567068Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider deployment has 1 unavailable replicas, router deployment has 1 unavailable replicas]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/82df0ed4-6ad4-45af-825d-883fcc05e886
correlation ID: bd6e85e8-f244-476d-8e1b-637c7a4beac1
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:126]: HCP cluster rg-candidate-5-1-c9tfj9kmq2zp/cluster-candidate-5-1-kdsdqq should provision
Unexpected error:
<*fmt.wrapError | 0x158d9065e120>:
failed to create HCP cluster cluster-candidate-5-1-kdsdqq: failed waiting for cluster="cluster-candidate-5-1-kdsdqq" in resourcegroup="rg-candidate-5-1-c9tfj9kmq2zp" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/82df0ed4-6ad4-45af-825d-883fcc05e886
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/82df0ed4-6ad4-45af-825d-883fcc05e886",
"name": "82df0ed4-6ad4-45af-825d-883fcc05e886",
"status": "Failed",
"startTime": "2026-10-28T14:40:01.853789521Z",
"endTime": "2026-10-28T15:09:08.621567068Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider deployment has 1 unavailable replicas, router deployment has 1 unavailable replicas]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/82df0ed4-6ad4-45af-825d-883fcc05e886
correlation ID: bd6e85e8-f244-476d-8e1b-637c7a4beac1
client request ID: unavailable
occurredfail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:126]: HCP cluster rg-candidate-4-21-6v7gsgmssgvq/cluster-candidate-4-21-rh49gm should provision
Unexpected error:
<*fmt.wrapError | 0x1da958962480>:
failed to create HCP cluster cluster-candidate-4-21-rh49gm: failed waiting for cluster="cluster-candidate-4-21-rh49gm" in resourcegroup="rg-candidate-4-21-6v7gsgmssgvq" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/8ae5c7d2-bbb4-424b-8ab6-07626d56a606
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/8ae5c7d2-bbb4-424b-8ab6-07626d56a606",
"name": "8ae5c7d2-bbb4-424b-8ab6-07626d56a606",
"status": "Failed",
"startTime": "2026-10-28T14:39:59.612142564Z",
"endTime": "2026-10-28T15:09:08.641921276Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider deployment has 1 unavailable replicas, router deployment has 1 unavailable replicas]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/8ae5c7d2-bbb4-424b-8ab6-07626d56a606
correlation ID: 8f132f7f-c309-4e23-a969-712eed0e7d91
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:126]: HCP cluster rg-candidate-4-21-6v7gsgmssgvq/cluster-candidate-4-21-rh49gm should provision
Unexpected error:
<*fmt.wrapError | 0x1da958962480>:
failed to create HCP cluster cluster-candidate-4-21-rh49gm: failed waiting for cluster="cluster-candidate-4-21-rh49gm" in resourcegroup="rg-candidate-4-21-6v7gsgmssgvq" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/8ae5c7d2-bbb4-424b-8ab6-07626d56a606
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/8ae5c7d2-bbb4-424b-8ab6-07626d56a606",
"name": "8ae5c7d2-bbb4-424b-8ab6-07626d56a606",
"status": "Failed",
"startTime": "2026-10-28T14:39:59.612142564Z",
"endTime": "2026-10-28T15:09:08.641921276Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider deployment has 1 unavailable replicas, router deployment has 1 unavailable replicas]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/8ae5c7d2-bbb4-424b-8ab6-07626d56a606
correlation ID: 8f132f7f-c309-4e23-a969-712eed0e7d91
client request ID: unavailable
occurredContributing tests (8)
Affected runs (2)
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedC...full failure pattern: ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable; hosted cluster degraded: UnavailableReplicas: [kube-apiserver]; provider Microsoft.RedHatOpenShift Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 2 day(s) | e2e | 2 | 2.67%2 of 75 job runs affected | IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 2 day(s) | Show trend detailsOct 21: 0 · Oct 22: 9 · Oct 23: 0 · Oct 24: 0 · Oct 25: 0 · Oct 26: 0 · Oct 27: 0 · Oct 28: 3 | none | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (3)fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_osdisk_encryption.go:127]: failed to create HCP cluster des-encrypt
Unexpected error:
<*fmt.wrapError | 0x27391d09a3a0>:
failed to create HCP cluster des-encrypt: failed waiting for cluster="des-encrypt" in resourcegroup="des-encrypt-vvl2jlrsdzwm" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/a5b87783-6a6a-4fee-a31b-7e52a950339f
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/a5b87783-6a6a-4fee-a31b-7e52a950339f",
"name": "a5b87783-6a6a-4fee-a31b-7e52a950339f",
"status": "Failed",
"startTime": "2026-10-28T15:22:08.258068936Z",
"endTime": "2026-10-28T15:51:17.638485832Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: olm-collect-profiles, community-operators-catalog, control-plane-pki-operator, dns-operator, kube-scheduler, cluster-storage-operator, machine-approver, openshift-apiserver, csi-snapshot-controller-operator, konnectivity-agent, redhat-marketplace-catalog, cluster-policy-controller, kube-controller-manager, certified-operators-catalog, cluster-version-operator, hosted-cluster-config-operator, azure-cloud-controller-manager, cluster-node-tuning-operator, cluster-image-registry-operator, redhat-operators-catalog, openshift-route-controller-manager, ingress-operator, catalog-operator, packageserver, cluster-autoscaler, openshift-controller-manager, cluster-network-operator, olm-operator; hosted cluster degraded: UnavailableReplicas: kube-apiserver deployment has 1 unavailable replicas"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/a5b87783-6a6a-4fee-a31b-7e52a950339f
correlation ID: c155e67f-4293-4e6f-9b93-fb56774a919b
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_osdisk_encryption.go:127]: failed to create HCP cluster des-encrypt
Unexpected error:
<*fmt.wrapError | 0x27391d09a3a0>:
failed to create HCP cluster des-encrypt: failed waiting for cluster="des-encrypt" in resourcegroup="des-encrypt-vvl2jlrsdzwm" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/a5b87783-6a6a-4fee-a31b-7e52a950339f
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/a5b87783-6a6a-4fee-a31b-7e52a950339f",
"name": "a5b87783-6a6a-4fee-a31b-7e52a950339f",
"status": "Failed",
"startTime": "2026-10-28T15:22:08.258068936Z",
"endTime": "2026-10-28T15:51:17.638485832Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: olm-collect-profiles, community-operators-catalog, control-plane-pki-operator, dns-operator, kube-scheduler, cluster-storage-operator, machine-approver, openshift-apiserver, csi-snapshot-controller-operator, konnectivity-agent, redhat-marketplace-catalog, cluster-policy-controller, kube-controller-manager, certified-operators-catalog, cluster-version-operator, hosted-cluster-config-operator, azure-cloud-controller-manager, cluster-node-tuning-operator, cluster-image-registry-operator, redhat-operators-catalog, openshift-route-controller-manager, ingress-operator, catalog-operator, packageserver, cluster-autoscaler, openshift-controller-manager, cluster-network-operator, olm-operator; hosted cluster degraded: UnavailableReplicas: kube-apiserver deployment has 1 unavailable replicas"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/a5b87783-6a6a-4fee-a31b-7e52a950339f
correlation ID: c155e67f-4293-4e6f-9b93-fb56774a919b
client request ID: unavailable
occurredfail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:125]: failed to create HCP cluster "cluster-zstream-4-20-ffl2k6" with version 4.20.43 on candidate channel
Unexpected error:
<*fmt.wrapError | 0x13122735c680>:
failed to create HCP cluster cluster-zstream-4-20-ffl2k6: failed waiting for cluster="cluster-zstream-4-20-ffl2k6" in resourcegroup="rg-zstream-upgrade-4-20-2qwm89mjrzcr" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/ab266e95-32c5-4fd4-a60d-96a8a32d075c
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/ab266e95-32c5-4fd4-a60d-96a8a32d075c",
"name": "ab266e95-32c5-4fd4-a60d-96a8a32d075c",
"status": "Failed",
"startTime": "2026-10-28T14:40:33.890129033Z",
"endTime": "2026-10-28T15:09:38.467450003Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: ingress-operator, csi-snapshot-controller-operator, olm-operator, cluster-policy-controller, cluster-node-tuning-operator, olm-collect-profiles, redhat-operators-catalog, machine-approver, kube-controller-manager, hosted-cluster-config-operator, konnectivity-agent, community-operators-catalog, control-plane-pki-operator, cluster-storage-operator, redhat-marketplace-catalog, openshift-apiserver, cluster-version-operator, catalog-operator, kube-scheduler, openshift-controller-manager, dns-operator, cluster-image-registry-operator, packageserver, cluster-autoscaler, certified-operators-catalog, azure-cloud-controller-manager, openshift-route-controller-manager, cluster-network-operator; hosted cluster degraded: UnavailableReplicas: kube-apiserver deployment has 1 unavailable replicas"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/ab266e95-32c5-4fd4-a60d-96a8a32d075c
correlation ID: ba9ee4d9-fd17-4e45-845d-93a236933c1e
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:125]: failed to create HCP cluster "cluster-zstream-4-20-ffl2k6" with version 4.20.43 on candidate channel
Unexpected error:
<*fmt.wrapError | 0x13122735c680>:
failed to create HCP cluster cluster-zstream-4-20-ffl2k6: failed waiting for cluster="cluster-zstream-4-20-ffl2k6" in resourcegroup="rg-zstream-upgrade-4-20-2qwm89mjrzcr" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/ab266e95-32c5-4fd4-a60d-96a8a32d075c
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/ab266e95-32c5-4fd4-a60d-96a8a32d075c",
"name": "ab266e95-32c5-4fd4-a60d-96a8a32d075c",
"status": "Failed",
"startTime": "2026-10-28T14:40:33.890129033Z",
"endTime": "2026-10-28T15:09:38.467450003Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: ingress-operator, csi-snapshot-controller-operator, olm-operator, cluster-policy-controller, cluster-node-tuning-operator, olm-collect-profiles, redhat-operators-catalog, machine-approver, kube-controller-manager, hosted-cluster-config-operator, konnectivity-agent, community-operators-catalog, control-plane-pki-operator, cluster-storage-operator, redhat-marketplace-catalog, openshift-apiserver, cluster-version-operator, catalog-operator, kube-scheduler, openshift-controller-manager, dns-operator, cluster-image-registry-operator, packageserver, cluster-autoscaler, certified-operators-catalog, azure-cloud-controller-manager, openshift-route-controller-manager, cluster-network-operator; hosted cluster degraded: UnavailableReplicas: kube-apiserver deployment has 1 unavailable replicas"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/ab266e95-32c5-4fd4-a60d-96a8a32d075c
correlation ID: ba9ee4d9-fd17-4e45-845d-93a236933c1e
client request ID: unavailable
occurredfail [github.com/Azure/ARO-HCP/test/e2e/shoebox_logs.go:272]: failed to create HCP cluster shoebox-hcp-cluster
Unexpected error:
<*fmt.wrapError | 0x166a5ce4a460>:
failed to create HCP cluster shoebox-hcp-cluster: failed waiting for cluster="shoebox-hcp-cluster" in resourcegroup="shoebox-cluster-xfbmh7tnw8dd" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/e5ff8914-b12e-4cdf-aa40-3986da34c7da
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/e5ff8914-b12e-4cdf-aa40-3986da34c7da",
"name": "e5ff8914-b12e-4cdf-aa40-3986da34c7da",
"status": "Failed",
"startTime": "2026-10-28T15:21:57.339002209Z",
"endTime": "2026-10-28T15:50:57.634345734Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: openshift-apiserver, hosted-cluster-config-operator, azure-cloud-controller-manager, konnectivity-agent, redhat-marketplace-catalog, cluster-policy-controller, cluster-storage-operator, dns-operator, certified-operators-catalog, cluster-network-operator, cluster-autoscaler, control-plane-pki-operator, cluster-version-operator, csi-snapshot-controller-operator, catalog-operator, community-operators-catalog, machine-approver, kube-controller-manager, kube-scheduler, openshift-controller-manager, ingress-operator, olm-operator, packageserver, openshift-route-controller-manager, cluster-node-tuning-operator, cluster-image-registry-operator, olm-collect-profiles, redhat-operators-catalog; hosted cluster degraded: UnavailableReplicas: kube-apiserver deployment has 1 unavailable replicas"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/e5ff8914-b12e-4cdf-aa40-3986da34c7da
correlation ID: 88b2308c-63cd-42b3-b006-499404a750ad
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/shoebox_logs.go:272]: failed to create HCP cluster shoebox-hcp-cluster
Unexpected error:
<*fmt.wrapError | 0x166a5ce4a460>:
failed to create HCP cluster shoebox-hcp-cluster: failed waiting for cluster="shoebox-hcp-cluster" in resourcegroup="shoebox-cluster-xfbmh7tnw8dd" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/e5ff8914-b12e-4cdf-aa40-3986da34c7da
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/e5ff8914-b12e-4cdf-aa40-3986da34c7da",
"name": "e5ff8914-b12e-4cdf-aa40-3986da34c7da",
"status": "Failed",
"startTime": "2026-10-28T15:21:57.339002209Z",
"endTime": "2026-10-28T15:50:57.634345734Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: openshift-apiserver, hosted-cluster-config-operator, azure-cloud-controller-manager, konnectivity-agent, redhat-marketplace-catalog, cluster-policy-controller, cluster-storage-operator, dns-operator, certified-operators-catalog, cluster-network-operator, cluster-autoscaler, control-plane-pki-operator, cluster-version-operator, csi-snapshot-controller-operator, catalog-operator, community-operators-catalog, machine-approver, kube-controller-manager, kube-scheduler, openshift-controller-manager, ingress-operator, olm-operator, packageserver, openshift-route-controller-manager, cluster-node-tuning-operator, cluster-image-registry-operator, olm-collect-profiles, redhat-operators-catalog; hosted cluster degraded: UnavailableReplicas: kube-apiserver deployment has 1 unavailable replicas"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/e5ff8914-b12e-4cdf-aa40-3986da34c7da
correlation ID: 88b2308c-63cd-42b3-b006-499404a750ad
client request ID: unavailable
occurredContributing tests (12)
Affected runs (2)
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedC...full failure pattern: ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable; hosted cluster degraded: UnavailableReplicas: [catalog-operator, hosted-cluster-config-operator, olm-operator, openshift-apiserver, packageserver]; provider Microsoft.RedHatOpenShift Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 2 day(s) | e2e | 2 | 2.67%2 of 75 job runs affected | IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 2 day(s) | Show trend detailsOct 21: 0 · Oct 22: 3 · Oct 23: 0 · Oct 24: 0 · Oct 25: 0 · Oct 26: 0 · Oct 27: 0 · Oct 28: 6 | DEV | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (3)fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:125]: failed to create HCP cluster "cluster-zstream-4-22-p65ngr" with version 4.22.19 on candidate channel
Unexpected error:
<*fmt.wrapError | 0x3f72bfdcc60>:
failed to create HCP cluster cluster-zstream-4-22-p65ngr: failed waiting for cluster="cluster-zstream-4-22-p65ngr" in resourcegroup="rg-zstream-upgrade-4-22-fk4qxqhpjj29" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/8ec9fe1f-0e5b-4c8f-94c5-92a6c0e3d366
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/8ec9fe1f-0e5b-4c8f-94c5-92a6c0e3d366",
"name": "8ec9fe1f-0e5b-4c8f-94c5-92a6c0e3d366",
"status": "Failed",
"startTime": "2026-10-28T14:39:56.97734875Z",
"endTime": "2026-10-28T15:08:58.578863234Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: openshift-route-controller-manager, cluster-node-tuning-operator, ingress-operator, csi-snapshot-controller-operator, community-operators-catalog, dns-operator, cluster-image-registry-operator, openshift-controller-manager, cluster-policy-controller, cluster-version-operator, cluster-storage-operator, cluster-network-operator, catalog-operator, hosted-cluster-config-operator, olm-operator, packageserver, redhat-operators-catalog, certified-operators-catalog; hosted cluster degraded: UnavailableReplicas: [catalog-operator deployment has 1 unavailable replicas, hosted-cluster-config-operator deployment has 1 unavailable replicas, olm-operator deployment has 1 unavailable replicas, openshift-apiserver deployment has 1 unavailable replicas, packageserver deployment has 3 unavailable replicas]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/8ec9fe1f-0e5b-4c8f-94c5-92a6c0e3d366
correlation ID: 094c53a6-4603-4b58-b443-8cca1dbc9dbb
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:125]: failed to create HCP cluster "cluster-zstream-4-22-p65ngr" with version 4.22.19 on candidate channel
Unexpected error:
<*fmt.wrapError | 0x3f72bfdcc60>:
failed to create HCP cluster cluster-zstream-4-22-p65ngr: failed waiting for cluster="cluster-zstream-4-22-p65ngr" in resourcegroup="rg-zstream-upgrade-4-22-fk4qxqhpjj29" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/8ec9fe1f-0e5b-4c8f-94c5-92a6c0e3d366
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/8ec9fe1f-0e5b-4c8f-94c5-92a6c0e3d366",
"name": "8ec9fe1f-0e5b-4c8f-94c5-92a6c0e3d366",
"status": "Failed",
"startTime": "2026-10-28T14:39:56.97734875Z",
"endTime": "2026-10-28T15:08:58.578863234Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: openshift-route-controller-manager, cluster-node-tuning-operator, ingress-operator, csi-snapshot-controller-operator, community-operators-catalog, dns-operator, cluster-image-registry-operator, openshift-controller-manager, cluster-policy-controller, cluster-version-operator, cluster-storage-operator, cluster-network-operator, catalog-operator, hosted-cluster-config-operator, olm-operator, packageserver, redhat-operators-catalog, certified-operators-catalog; hosted cluster degraded: UnavailableReplicas: [catalog-operator deployment has 1 unavailable replicas, hosted-cluster-config-operator deployment has 1 unavailable replicas, olm-operator deployment has 1 unavailable replicas, openshift-apiserver deployment has 1 unavailable replicas, packageserver deployment has 3 unavailable replicas]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/8ec9fe1f-0e5b-4c8f-94c5-92a6c0e3d366
correlation ID: 094c53a6-4603-4b58-b443-8cca1dbc9dbb
client request ID: unavailable
occurredfail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:168]: failed to create HCP cluster for patch-tags test
Unexpected error:
<*fmt.wrapError | 0x32cee6f94b40>:
failed to create HCP cluster patch-tags-cluster: failed waiting for cluster="patch-tags-cluster" in resourcegroup="patch-tags-gkhf8l89gfpt" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/e910a1ec-64c8-4b3f-89a4-590c87d733cf
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/e910a1ec-64c8-4b3f-89a4-590c87d733cf",
"name": "e910a1ec-64c8-4b3f-89a4-590c87d733cf",
"status": "Failed",
"startTime": "2026-10-28T14:39:19.663998842Z",
"endTime": "2026-10-28T15:08:28.451578554Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: csi-snapshot-controller-operator, cluster-image-registry-operator, catalog-operator, cluster-policy-controller, redhat-operators-catalog, cluster-network-operator, openshift-route-controller-manager, dns-operator, packageserver, hosted-cluster-config-operator, cluster-storage-operator, cluster-node-tuning-operator, ingress-operator, redhat-marketplace-catalog, certified-operators-catalog, community-operators-catalog, olm-operator, cluster-version-operator, openshift-controller-manager; hosted cluster degraded: UnavailableReplicas: [catalog-operator deployment has 1 unavailable replicas, hosted-cluster-config-operator deployment has 1 unavailable replicas, olm-operator deployment has 1 unavailable replicas, openshift-apiserver deployment has 1 unavailable replicas, packageserver deployment has 3 unavailable replicas]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/e910a1ec-64c8-4b3f-89a4-590c87d733cf
correlation ID: 9764c911-c5b8-4802-ae2b-3bbd27636314
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:168]: failed to create HCP cluster for patch-tags test
Unexpected error:
<*fmt.wrapError | 0x32cee6f94b40>:
failed to create HCP cluster patch-tags-cluster: failed waiting for cluster="patch-tags-cluster" in resourcegroup="patch-tags-gkhf8l89gfpt" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/e910a1ec-64c8-4b3f-89a4-590c87d733cf
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/e910a1ec-64c8-4b3f-89a4-590c87d733cf",
"name": "e910a1ec-64c8-4b3f-89a4-590c87d733cf",
"status": "Failed",
"startTime": "2026-10-28T14:39:19.663998842Z",
"endTime": "2026-10-28T15:08:28.451578554Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: csi-snapshot-controller-operator, cluster-image-registry-operator, catalog-operator, cluster-policy-controller, redhat-operators-catalog, cluster-network-operator, openshift-route-controller-manager, dns-operator, packageserver, hosted-cluster-config-operator, cluster-storage-operator, cluster-node-tuning-operator, ingress-operator, redhat-marketplace-catalog, certified-operators-catalog, community-operators-catalog, olm-operator, cluster-version-operator, openshift-controller-manager; hosted cluster degraded: UnavailableReplicas: [catalog-operator deployment has 1 unavailable replicas, hosted-cluster-config-operator deployment has 1 unavailable replicas, olm-operator deployment has 1 unavailable replicas, openshift-apiserver deployment has 1 unavailable replicas, packageserver deployment has 3 unavailable replicas]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/e910a1ec-64c8-4b3f-89a4-590c87d733cf
correlation ID: 9764c911-c5b8-4802-ae2b-3bbd27636314
client request ID: unavailable
occurredfail [github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:98]: failed to create HCP cluster for external auth test
Unexpected error:
<*fmt.wrapError | 0xdfd4e022a0>:
failed to create HCP cluster ea-cluster: failed waiting for cluster="ea-cluster" in resourcegroup="external-auth-cluster-qlszp5g8k9wc" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/06deff3f-928f-4528-ab39-5d28dd6fa99e
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/06deff3f-928f-4528-ab39-5d28dd6fa99e",
"name": "06deff3f-928f-4528-ab39-5d28dd6fa99e",
"status": "Failed",
"startTime": "2026-10-28T14:41:22.27269296Z",
"endTime": "2026-10-28T15:10:28.476284696Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: redhat-operators-catalog, openshift-controller-manager, cluster-network-operator, dns-operator, catalog-operator, openshift-route-controller-manager, packageserver, cluster-policy-controller, certified-operators-catalog, cluster-node-tuning-operator, csi-snapshot-controller-operator, redhat-marketplace-catalog, community-operators-catalog, cluster-version-operator, ingress-operator, olm-operator, hosted-cluster-config-operator, cluster-storage-operator, cluster-image-registry-operator; hosted cluster degraded: UnavailableReplicas: [catalog-operator deployment has 1 unavailable replicas, hosted-cluster-config-operator deployment has 1 unavailable replicas, olm-operator deployment has 1 unavailable replicas, openshift-apiserver deployment has 1 unavailable replicas, packageserver deployment has 3 unavailable replicas]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/06deff3f-928f-4528-ab39-5d28dd6fa99e
correlation ID: 09ac3f0b-4c54-44f8-a6e3-2ded2abcc35b
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:98]: failed to create HCP cluster for external auth test
Unexpected error:
<*fmt.wrapError | 0xdfd4e022a0>:
failed to create HCP cluster ea-cluster: failed waiting for cluster="ea-cluster" in resourcegroup="external-auth-cluster-qlszp5g8k9wc" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/06deff3f-928f-4528-ab39-5d28dd6fa99e
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/06deff3f-928f-4528-ab39-5d28dd6fa99e",
"name": "06deff3f-928f-4528-ab39-5d28dd6fa99e",
"status": "Failed",
"startTime": "2026-10-28T14:41:22.27269296Z",
"endTime": "2026-10-28T15:10:28.476284696Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: redhat-operators-catalog, openshift-controller-manager, cluster-network-operator, dns-operator, catalog-operator, openshift-route-controller-manager, packageserver, cluster-policy-controller, certified-operators-catalog, cluster-node-tuning-operator, csi-snapshot-controller-operator, redhat-marketplace-catalog, community-operators-catalog, cluster-version-operator, ingress-operator, olm-operator, hosted-cluster-config-operator, cluster-storage-operator, cluster-image-registry-operator; hosted cluster degraded: UnavailableReplicas: [catalog-operator deployment has 1 unavailable replicas, hosted-cluster-config-operator deployment has 1 unavailable replicas, olm-operator deployment has 1 unavailable replicas, openshift-apiserver deployment has 1 unavailable replicas, packageserver deployment has 3 unavailable replicas]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/06deff3f-928f-4528-ab39-5d28dd6fa99e
correlation ID: 09ac3f0b-4c54-44f8-a6e3-2ded2abcc35b
client request ID: unavailable
occurredContributing tests (9)
Affected runs (2)
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
DNS for route host <route-host> did not resolve: DNS for <route-host> did not resolve within <duration> (last error: lookup <host> on <dns-server>: no such host): context deadline...full failure pattern: DNS for route host <route-host> did not resolve: DNS for <route-host> did not resolve within <duration> (last error: lookup <host> on <dns-server>: no such host): context deadline exceeded Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s) | e2e | 2 | 2.67%2 of 75 job runs affected | IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s) | Show trend detailsOct 21: 0 · Oct 22: 0 · Oct 23: 6 · Oct 24: 0 · Oct 25: 0 · Oct 26: 0 · Oct 27: 0 · Oct 28: 0 | DEV, INT | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (3)fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:190]: failed to verify simple web app runs on cluster "cluster-candidate-5-0-wnz6kv"
Unexpected error:
<*fmt.wrapError | 0x1aca60620e60>:
DNS for route host agnhost-e2e-sample-app-5cwbl.apps.aro.c1s1r7e8q7f4j2u.vfmi.eastus2.aroapp-hcp.io did not resolve: DNS for agnhost-e2e-sample-app-5cwbl.apps.aro.c1s1r7e8q7f4j2u.vfmi.eastus2.aroapp-hcp.io did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-5cwbl.apps.aro.c1s1r7e8q7f4j2u.vfmi.eastus2.aroapp-hcp.io on 172.30.0.10:53: no such host): context deadline exceeded
{
msg: "DNS for route host agnhost-e2e-sample-app-5cwbl.apps.aro.c1s1r7e8q7f4j2u.vfmi.eastus2.aroapp-hcp.io did not resolve: DNS for agnhost-e2e-sample-app-5cwbl.apps.aro.c1s1r7e8q7f4j2u.vfmi.eastus2.aroapp-hcp.io did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-5cwbl.apps.aro.c1s1r7e8q7f4j2u.vfmi.eastus2.aroapp-hcp.io on 172.30.0.10:53: no such host): context deadline exceeded",
err: <*fmt.wrapError | 0x1aca604dcae0>{
msg: "DNS for agnhost-e2e-sample-app-5cwbl.apps.aro.c1s1r7e8q7f4j2u.vfmi.eastus2.aroapp-hcp.io did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-5cwbl.apps.aro.c1s1r7e8q7f4j2u.vfmi.eastus2.aroapp-hcp.io on 172.30.0.10:53: no such host): context deadline exceeded",
err: <context.deadlineExceededError>{},
},
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:190]: failed to verify simple web app runs on cluster "cluster-candidate-5-0-wnz6kv"
Unexpected error:
<*fmt.wrapError | 0x1aca60620e60>:
DNS for route host agnhost-e2e-sample-app-5cwbl.apps.aro.c1s1r7e8q7f4j2u.vfmi.eastus2.aroapp-hcp.io did not resolve: DNS for agnhost-e2e-sample-app-5cwbl.apps.aro.c1s1r7e8q7f4j2u.vfmi.eastus2.aroapp-hcp.io did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-5cwbl.apps.aro.c1s1r7e8q7f4j2u.vfmi.eastus2.aroapp-hcp.io on 172.30.0.10:53: no such host): context deadline exceeded
{
msg: "DNS for route host agnhost-e2e-sample-app-5cwbl.apps.aro.c1s1r7e8q7f4j2u.vfmi.eastus2.aroapp-hcp.io did not resolve: DNS for agnhost-e2e-sample-app-5cwbl.apps.aro.c1s1r7e8q7f4j2u.vfmi.eastus2.aroapp-hcp.io did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-5cwbl.apps.aro.c1s1r7e8q7f4j2u.vfmi.eastus2.aroapp-hcp.io on 172.30.0.10:53: no such host): context deadline exceeded",
err: <*fmt.wrapError | 0x1aca604dcae0>{
msg: "DNS for agnhost-e2e-sample-app-5cwbl.apps.aro.c1s1r7e8q7f4j2u.vfmi.eastus2.aroapp-hcp.io did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-5cwbl.apps.aro.c1s1r7e8q7f4j2u.vfmi.eastus2.aroapp-hcp.io on 172.30.0.10:53: no such host): context deadline exceeded",
err: <context.deadlineExceededError>{},
},
}
occurredfail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:190]: failed to verify simple web app runs on cluster "cluster-candidate-4-20-q5x22b"
Unexpected error:
<*fmt.wrapError | 0x21a3a7aea6e0>:
DNS for route host agnhost-e2e-sample-app-27qvf.apps.aro.l8d0x7x9p0e3b5l.tq9n.eastus2.aroapp-hcp.io did not resolve: DNS for agnhost-e2e-sample-app-27qvf.apps.aro.l8d0x7x9p0e3b5l.tq9n.eastus2.aroapp-hcp.io did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-27qvf.apps.aro.l8d0x7x9p0e3b5l.tq9n.eastus2.aroapp-hcp.io on 172.30.0.10:53: no such host): context deadline exceeded
{
msg: "DNS for route host agnhost-e2e-sample-app-27qvf.apps.aro.l8d0x7x9p0e3b5l.tq9n.eastus2.aroapp-hcp.io did not resolve: DNS for agnhost-e2e-sample-app-27qvf.apps.aro.l8d0x7x9p0e3b5l.tq9n.eastus2.aroapp-hcp.io did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-27qvf.apps.aro.l8d0x7x9p0e3b5l.tq9n.eastus2.aroapp-hcp.io on 172.30.0.10:53: no such host): context deadline exceeded",
err: <*fmt.wrapError | 0x21a3a7548980>{
msg: "DNS for agnhost-e2e-sample-app-27qvf.apps.aro.l8d0x7x9p0e3b5l.tq9n.eastus2.aroapp-hcp.io did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-27qvf.apps.aro.l8d0x7x9p0e3b5l.tq9n.eastus2.aroapp-hcp.io on 172.30.0.10:53: no such host): context deadline exceeded",
err: <context.deadlineExceededError>{},
},
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:190]: failed to verify simple web app runs on cluster "cluster-candidate-4-20-q5x22b"
Unexpected error:
<*fmt.wrapError | 0x21a3a7aea6e0>:
DNS for route host agnhost-e2e-sample-app-27qvf.apps.aro.l8d0x7x9p0e3b5l.tq9n.eastus2.aroapp-hcp.io did not resolve: DNS for agnhost-e2e-sample-app-27qvf.apps.aro.l8d0x7x9p0e3b5l.tq9n.eastus2.aroapp-hcp.io did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-27qvf.apps.aro.l8d0x7x9p0e3b5l.tq9n.eastus2.aroapp-hcp.io on 172.30.0.10:53: no such host): context deadline exceeded
{
msg: "DNS for route host agnhost-e2e-sample-app-27qvf.apps.aro.l8d0x7x9p0e3b5l.tq9n.eastus2.aroapp-hcp.io did not resolve: DNS for agnhost-e2e-sample-app-27qvf.apps.aro.l8d0x7x9p0e3b5l.tq9n.eastus2.aroapp-hcp.io did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-27qvf.apps.aro.l8d0x7x9p0e3b5l.tq9n.eastus2.aroapp-hcp.io on 172.30.0.10:53: no such host): context deadline exceeded",
err: <*fmt.wrapError | 0x21a3a7548980>{
msg: "DNS for agnhost-e2e-sample-app-27qvf.apps.aro.l8d0x7x9p0e3b5l.tq9n.eastus2.aroapp-hcp.io did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-27qvf.apps.aro.l8d0x7x9p0e3b5l.tq9n.eastus2.aroapp-hcp.io on 172.30.0.10:53: no such host): context deadline exceeded",
err: <context.deadlineExceededError>{},
},
}
occurredfail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:190]: failed to verify simple web app runs on cluster "cluster-candidate-4-21-5p9hrv"
Unexpected error:
<*fmt.wrapError | 0x359991ca6200>:
DNS for route host agnhost-e2e-sample-app-dbdwm.apps.aro.n0p5j9k4k3x6p4o.lcm9.eastus2.aroapp-hcp.io did not resolve: DNS for agnhost-e2e-sample-app-dbdwm.apps.aro.n0p5j9k4k3x6p4o.lcm9.eastus2.aroapp-hcp.io did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-dbdwm.apps.aro.n0p5j9k4k3x6p4o.lcm9.eastus2.aroapp-hcp.io on 172.30.0.10:53: no such host): context deadline exceeded
{
msg: "DNS for route host agnhost-e2e-sample-app-dbdwm.apps.aro.n0p5j9k4k3x6p4o.lcm9.eastus2.aroapp-hcp.io did not resolve: DNS for agnhost-e2e-sample-app-dbdwm.apps.aro.n0p5j9k4k3x6p4o.lcm9.eastus2.aroapp-hcp.io did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-dbdwm.apps.aro.n0p5j9k4k3x6p4o.lcm9.eastus2.aroapp-hcp.io on 172.30.0.10:53: no such host): context deadline exceeded",
err: <*fmt.wrapError | 0x359991cc83e0>{
msg: "DNS for agnhost-e2e-sample-app-dbdwm.apps.aro.n0p5j9k4k3x6p4o.lcm9.eastus2.aroapp-hcp.io did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-dbdwm.apps.aro.n0p5j9k4k3x6p4o.lcm9.eastus2.aroapp-hcp.io on 172.30.0.10:53: no such host): context deadline exceeded",
err: <context.deadlineExceededError>{},
},
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:190]: failed to verify simple web app runs on cluster "cluster-candidate-4-21-5p9hrv"
Unexpected error:
<*fmt.wrapError | 0x359991ca6200>:
DNS for route host agnhost-e2e-sample-app-dbdwm.apps.aro.n0p5j9k4k3x6p4o.lcm9.eastus2.aroapp-hcp.io did not resolve: DNS for agnhost-e2e-sample-app-dbdwm.apps.aro.n0p5j9k4k3x6p4o.lcm9.eastus2.aroapp-hcp.io did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-dbdwm.apps.aro.n0p5j9k4k3x6p4o.lcm9.eastus2.aroapp-hcp.io on 172.30.0.10:53: no such host): context deadline exceeded
{
msg: "DNS for route host agnhost-e2e-sample-app-dbdwm.apps.aro.n0p5j9k4k3x6p4o.lcm9.eastus2.aroapp-hcp.io did not resolve: DNS for agnhost-e2e-sample-app-dbdwm.apps.aro.n0p5j9k4k3x6p4o.lcm9.eastus2.aroapp-hcp.io did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-dbdwm.apps.aro.n0p5j9k4k3x6p4o.lcm9.eastus2.aroapp-hcp.io on 172.30.0.10:53: no such host): context deadline exceeded",
err: <*fmt.wrapError | 0x359991cc83e0>{
msg: "DNS for agnhost-e2e-sample-app-dbdwm.apps.aro.n0p5j9k4k3x6p4o.lcm9.eastus2.aroapp-hcp.io did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-dbdwm.apps.aro.n0p5j9k4k3x6p4o.lcm9.eastus2.aroapp-hcp.io on 172.30.0.10:53: no such host): context deadline exceeded",
err: <context.deadlineExceededError>{},
},
}
occurredContributing tests (5)
Affected runs (2)
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedC...full failure pattern: ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable; hosted cluster degraded: UnavailableReplicas: [capi-provider, kube-apiserver]; provider Microsoft.RedHatOpenShift Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 2 day(s) | e2e | 2 | 2.67%2 of 75 job runs affected | IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 2 day(s) | Show trend detailsOct 21: 0 · Oct 22: 2 · Oct 23: 0 · Oct 24: 0 · Oct 25: 0 · Oct 26: 0 · Oct 27: 0 · Oct 28: 3 | none | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (3)fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:125]: failed to create HCP cluster "cluster-zstream-5-0-t7rr5n" with version 5.0.0-rc.6 on candidate channel
Unexpected error:
<*fmt.wrapError | 0x237ae7f7c240>:
failed to create HCP cluster cluster-zstream-5-0-t7rr5n: failed waiting for cluster="cluster-zstream-5-0-t7rr5n" in resourcegroup="rg-zstream-upgrade-5-0-6fp2zwlhw7bz" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/8907c7e5-9e3b-4204-856f-e49835870136
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/8907c7e5-9e3b-4204-856f-e49835870136",
"name": "8907c7e5-9e3b-4204-856f-e49835870136",
"status": "Failed",
"startTime": "2026-10-28T14:40:08.646687074Z",
"endTime": "2026-10-28T15:09:08.445371105Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: openshift-controller-manager, packageserver, azure-cloud-controller-manager, dns-operator, openshift-apiserver, capi-provider, kube-controller-manager, cluster-policy-controller, cluster-network-operator, certified-operators-catalog, control-plane-pki-operator, openshift-route-controller-manager, cluster-image-registry-operator, csi-snapshot-controller-operator, catalog-operator, konnectivity-agent, kube-scheduler, cluster-version-operator, ingress-operator, cluster-autoscaler, redhat-operators-catalog, cluster-storage-operator, cluster-node-tuning-operator, olm-operator, machine-approver, kube-storage-version-migrator, hosted-cluster-config-operator, community-operators-catalog, olm-collect-profiles; hosted cluster degraded: UnavailableReplicas: [capi-provider deployment has 2 unavailable replicas, kube-apiserver deployment has 1 unavailable replicas]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/8907c7e5-9e3b-4204-856f-e49835870136
correlation ID: 331db847-c2c2-4f64-8fe5-cddb4a33c4cc
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:125]: failed to create HCP cluster "cluster-zstream-5-0-t7rr5n" with version 5.0.0-rc.6 on candidate channel
Unexpected error:
<*fmt.wrapError | 0x237ae7f7c240>:
failed to create HCP cluster cluster-zstream-5-0-t7rr5n: failed waiting for cluster="cluster-zstream-5-0-t7rr5n" in resourcegroup="rg-zstream-upgrade-5-0-6fp2zwlhw7bz" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/8907c7e5-9e3b-4204-856f-e49835870136
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/8907c7e5-9e3b-4204-856f-e49835870136",
"name": "8907c7e5-9e3b-4204-856f-e49835870136",
"status": "Failed",
"startTime": "2026-10-28T14:40:08.646687074Z",
"endTime": "2026-10-28T15:09:08.445371105Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: openshift-controller-manager, packageserver, azure-cloud-controller-manager, dns-operator, openshift-apiserver, capi-provider, kube-controller-manager, cluster-policy-controller, cluster-network-operator, certified-operators-catalog, control-plane-pki-operator, openshift-route-controller-manager, cluster-image-registry-operator, csi-snapshot-controller-operator, catalog-operator, konnectivity-agent, kube-scheduler, cluster-version-operator, ingress-operator, cluster-autoscaler, redhat-operators-catalog, cluster-storage-operator, cluster-node-tuning-operator, olm-operator, machine-approver, kube-storage-version-migrator, hosted-cluster-config-operator, community-operators-catalog, olm-collect-profiles; hosted cluster degraded: UnavailableReplicas: [capi-provider deployment has 2 unavailable replicas, kube-apiserver deployment has 1 unavailable replicas]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/8907c7e5-9e3b-4204-856f-e49835870136
correlation ID: 331db847-c2c2-4f64-8fe5-cddb4a33c4cc
client request ID: unavailable
occurredfail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_nodepool_osdisk.go:80]: failed to create HCP cluster "hcp-cluster-np-128"
Unexpected error:
<*fmt.wrapError | 0x1bc38a476940>:
failed to create HCP cluster hcp-cluster-np-128: failed waiting for cluster="hcp-cluster-np-128" in resourcegroup="clusternp128-78bq6pzzxwrs" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/70aa6c6b-01ba-49fe-9aa7-e8f005a8163f
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/70aa6c6b-01ba-49fe-9aa7-e8f005a8163f",
"name": "70aa6c6b-01ba-49fe-9aa7-e8f005a8163f",
"status": "Failed",
"startTime": "2026-10-28T13:42:57.38659057Z",
"endTime": "2026-10-28T14:12:00.88820442Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: cluster-storage-operator, csi-snapshot-controller-operator, capi-provider, openshift-apiserver, konnectivity-agent, hosted-cluster-config-operator, cluster-network-operator, ingress-operator, packageserver, olm-collect-profiles, cluster-autoscaler, control-plane-pki-operator, cluster-version-operator, openshift-controller-manager, openshift-route-controller-manager, redhat-marketplace-catalog, machine-approver, azure-cloud-controller-manager, dns-operator, catalog-operator, community-operators-catalog, certified-operators-catalog, cluster-node-tuning-operator, cluster-image-registry-operator, kube-controller-manager, kube-scheduler, redhat-operators-catalog, cluster-policy-controller, olm-operator; hosted cluster degraded: UnavailableReplicas: [capi-provider deployment has 2 unavailable replicas, kube-apiserver deployment has 1 unavailable replicas]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/70aa6c6b-01ba-49fe-9aa7-e8f005a8163f
correlation ID: f60a131e-81c7-4e9f-9c27-ffb965259d3b
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_nodepool_osdisk.go:80]: failed to create HCP cluster "hcp-cluster-np-128"
Unexpected error:
<*fmt.wrapError | 0x1bc38a476940>:
failed to create HCP cluster hcp-cluster-np-128: failed waiting for cluster="hcp-cluster-np-128" in resourcegroup="clusternp128-78bq6pzzxwrs" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/70aa6c6b-01ba-49fe-9aa7-e8f005a8163f
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/70aa6c6b-01ba-49fe-9aa7-e8f005a8163f",
"name": "70aa6c6b-01ba-49fe-9aa7-e8f005a8163f",
"status": "Failed",
"startTime": "2026-10-28T13:42:57.38659057Z",
"endTime": "2026-10-28T14:12:00.88820442Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: cluster-storage-operator, csi-snapshot-controller-operator, capi-provider, openshift-apiserver, konnectivity-agent, hosted-cluster-config-operator, cluster-network-operator, ingress-operator, packageserver, olm-collect-profiles, cluster-autoscaler, control-plane-pki-operator, cluster-version-operator, openshift-controller-manager, openshift-route-controller-manager, redhat-marketplace-catalog, machine-approver, azure-cloud-controller-manager, dns-operator, catalog-operator, community-operators-catalog, certified-operators-catalog, cluster-node-tuning-operator, cluster-image-registry-operator, kube-controller-manager, kube-scheduler, redhat-operators-catalog, cluster-policy-controller, olm-operator; hosted cluster degraded: UnavailableReplicas: [capi-provider deployment has 2 unavailable replicas, kube-apiserver deployment has 1 unavailable replicas]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/70aa6c6b-01ba-49fe-9aa7-e8f005a8163f
correlation ID: f60a131e-81c7-4e9f-9c27-ffb965259d3b
client request ID: unavailable
occurredfail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:125]: failed to create HCP cluster "cluster-zstream-4-23-qxnh9v" with version 4.23.0-ec.0 on candidate channel
Unexpected error:
<*fmt.wrapError | 0x9c08c475a0>:
failed to create HCP cluster cluster-zstream-4-23-qxnh9v: failed waiting for cluster="cluster-zstream-4-23-qxnh9v" in resourcegroup="rg-zstream-upgrade-4-23-lzbzhtjskg9m" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/49078e61-6846-4cd0-bfa4-7f73457f1286
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/49078e61-6846-4cd0-bfa4-7f73457f1286",
"name": "49078e61-6846-4cd0-bfa4-7f73457f1286",
"status": "Failed",
"startTime": "2026-10-28T14:40:13.468188351Z",
"endTime": "2026-10-28T15:09:18.606679709Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: community-operators-catalog, cluster-image-registry-operator, olm-operator, kube-scheduler, kube-storage-version-migrator, catalog-operator, machine-approver, azure-cloud-controller-manager, cluster-autoscaler, ingress-operator, openshift-controller-manager, packageserver, konnectivity-agent, hosted-cluster-config-operator, redhat-operators-catalog, dns-operator, control-plane-pki-operator, cluster-policy-controller, cluster-network-operator, kube-controller-manager, cluster-node-tuning-operator, openshift-route-controller-manager, olm-collect-profiles, openshift-apiserver, capi-provider, cluster-storage-operator, csi-snapshot-controller-operator, cluster-version-operator, certified-operators-catalog; hosted cluster degraded: UnavailableReplicas: [capi-provider deployment has 2 unavailable replicas, kube-apiserver deployment has 1 unavailable replicas]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/49078e61-6846-4cd0-bfa4-7f73457f1286
correlation ID: 0f636737-db3c-44fd-a024-8aa2dae006d5
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:125]: failed to create HCP cluster "cluster-zstream-4-23-qxnh9v" with version 4.23.0-ec.0 on candidate channel
Unexpected error:
<*fmt.wrapError | 0x9c08c475a0>:
failed to create HCP cluster cluster-zstream-4-23-qxnh9v: failed waiting for cluster="cluster-zstream-4-23-qxnh9v" in resourcegroup="rg-zstream-upgrade-4-23-lzbzhtjskg9m" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/49078e61-6846-4cd0-bfa4-7f73457f1286
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/49078e61-6846-4cd0-bfa4-7f73457f1286",
"name": "49078e61-6846-4cd0-bfa4-7f73457f1286",
"status": "Failed",
"startTime": "2026-10-28T14:40:13.468188351Z",
"endTime": "2026-10-28T15:09:18.606679709Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: community-operators-catalog, cluster-image-registry-operator, olm-operator, kube-scheduler, kube-storage-version-migrator, catalog-operator, machine-approver, azure-cloud-controller-manager, cluster-autoscaler, ingress-operator, openshift-controller-manager, packageserver, konnectivity-agent, hosted-cluster-config-operator, redhat-operators-catalog, dns-operator, control-plane-pki-operator, cluster-policy-controller, cluster-network-operator, kube-controller-manager, cluster-node-tuning-operator, openshift-route-controller-manager, olm-collect-profiles, openshift-apiserver, capi-provider, cluster-storage-operator, csi-snapshot-controller-operator, cluster-version-operator, certified-operators-catalog; hosted cluster degraded: UnavailableReplicas: [capi-provider deployment has 2 unavailable replicas, kube-apiserver deployment has 1 unavailable replicas]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/49078e61-6846-4cd0-bfa4-7f73457f1286
correlation ID: 0f636737-db3c-44fd-a024-8aa2dae006d5
client request ID: unavailable
occurredContributing tests (4)
Affected runs (2)
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedC...full failure pattern: ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider, ignition-server-proxy]; provider Microsoft.RedHatOpenShift Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 2 day(s) | e2e | 2 | 2.67%2 of 75 job runs affected | IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 2 day(s) | Show trend detailsOct 21: 0 · Oct 22: 2 · Oct 23: 0 · Oct 24: 0 · Oct 25: 0 · Oct 26: 0 · Oct 27: 0 · Oct 28: 3 | none | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (3)fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:146]: failed to create HCP cluster np-version-upgrade-cluster-t67p9t with version 4.21.40
Unexpected error:
<*fmt.wrapError | 0xbe0fc68500>:
failed to create HCP cluster np-version-upgrade-cluster-t67p9t: failed waiting for cluster="np-version-upgrade-cluster-t67p9t" in resourcegroup="rg-np-version-upgrade-t67p9t-kpsvm25jmqsm" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/9e35dee8-3994-4ba7-9234-74a03d742751
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/9e35dee8-3994-4ba7-9234-74a03d742751",
"name": "9e35dee8-3994-4ba7-9234-74a03d742751",
"status": "Failed",
"startTime": "2026-10-28T15:21:15.512939467Z",
"endTime": "2026-10-28T15:50:17.639783703Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider deployment has 1 unavailable replicas, ignition-server-proxy deployment has 1 unavailable replicas]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/9e35dee8-3994-4ba7-9234-74a03d742751
correlation ID: 8d785f53-f7f0-40a1-aa91-751064d7e050
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:146]: failed to create HCP cluster np-version-upgrade-cluster-t67p9t with version 4.21.40
Unexpected error:
<*fmt.wrapError | 0xbe0fc68500>:
failed to create HCP cluster np-version-upgrade-cluster-t67p9t: failed waiting for cluster="np-version-upgrade-cluster-t67p9t" in resourcegroup="rg-np-version-upgrade-t67p9t-kpsvm25jmqsm" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/9e35dee8-3994-4ba7-9234-74a03d742751
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/9e35dee8-3994-4ba7-9234-74a03d742751",
"name": "9e35dee8-3994-4ba7-9234-74a03d742751",
"status": "Failed",
"startTime": "2026-10-28T15:21:15.512939467Z",
"endTime": "2026-10-28T15:50:17.639783703Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider deployment has 1 unavailable replicas, ignition-server-proxy deployment has 1 unavailable replicas]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/9e35dee8-3994-4ba7-9234-74a03d742751
correlation ID: 8d785f53-f7f0-40a1-aa91-751064d7e050
client request ID: unavailable
occurredfail [github.com/Azure/ARO-HCP/test/e2e/cluster_autoscaling.go:108]: failed to create HCP cluster "autoscaling-hcp-cluster" with custom autoscaling
Unexpected error:
<*fmt.wrapError | 0x24f1c7fcefa0>:
failed to create HCP cluster autoscaling-hcp-cluster: failed waiting for cluster="autoscaling-hcp-cluster" in resourcegroup="autoscaling-cluster-bhc75ljrxzvb" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/3ee91092-37d4-4687-8899-086b94094407
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/3ee91092-37d4-4687-8899-086b94094407",
"name": "3ee91092-37d4-4687-8899-086b94094407",
"status": "Failed",
"startTime": "2026-10-28T13:42:47.010012275Z",
"endTime": "2026-10-28T14:11:50.916418144Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider deployment has 1 unavailable replicas, ignition-server-proxy deployment has 1 unavailable replicas]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/3ee91092-37d4-4687-8899-086b94094407
correlation ID: 881a23bb-07d5-4161-bd1d-8f0790ff0f30
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_autoscaling.go:108]: failed to create HCP cluster "autoscaling-hcp-cluster" with custom autoscaling
Unexpected error:
<*fmt.wrapError | 0x24f1c7fcefa0>:
failed to create HCP cluster autoscaling-hcp-cluster: failed waiting for cluster="autoscaling-hcp-cluster" in resourcegroup="autoscaling-cluster-bhc75ljrxzvb" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/3ee91092-37d4-4687-8899-086b94094407
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/3ee91092-37d4-4687-8899-086b94094407",
"name": "3ee91092-37d4-4687-8899-086b94094407",
"status": "Failed",
"startTime": "2026-10-28T13:42:47.010012275Z",
"endTime": "2026-10-28T14:11:50.916418144Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider deployment has 1 unavailable replicas, ignition-server-proxy deployment has 1 unavailable replicas]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/3ee91092-37d4-4687-8899-086b94094407
correlation ID: 881a23bb-07d5-4161-bd1d-8f0790ff0f30
client request ID: unavailable
occurredfail [github.com/Azure/ARO-HCP/test/e2e/clusters_sharing_resgroup.go:131]: failed to wait for first cluster "basic-hcp-cluster" to complete creation (timeout '30.000000' minutes)
Unexpected error:
<*exported.ResponseError | 0x30c5485c87e0>:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/4b718d49-0561-4617-ab78-21f445a937c3
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/4b718d49-0561-4617-ab78-21f445a937c3",
"name": "4b718d49-0561-4617-ab78-21f445a937c3",
"status": "Failed",
"startTime": "2026-10-28T13:43:47.79023892Z",
"endTime": "2026-10-28T14:12:50.828658903Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider deployment has 1 unavailable replicas, ignition-server-proxy deployment has 1 unavailable replicas]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/4b718d49-0561-4617-ab78-21f445a937c3
correlation ID: 14e2f634-75d0-4322-8234-7441ccabc009
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/clusters_sharing_resgroup.go:131]: failed to wait for first cluster "basic-hcp-cluster" to complete creation (timeout '30.000000' minutes)
Unexpected error:
<*exported.ResponseError | 0x30c5485c87e0>:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/4b718d49-0561-4617-ab78-21f445a937c3
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/4b718d49-0561-4617-ab78-21f445a937c3",
"name": "4b718d49-0561-4617-ab78-21f445a937c3",
"status": "Failed",
"startTime": "2026-10-28T13:43:47.79023892Z",
"endTime": "2026-10-28T14:12:50.828658903Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider deployment has 1 unavailable replicas, ignition-server-proxy deployment has 1 unavailable replicas]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/4b718d49-0561-4617-ab78-21f445a937c3
correlation ID: 14e2f634-75d0-4322-8234-7441ccabc009
client request ID: unavailable
occurredContributing tests (5)
Affected runs (2)
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
VerifySimpleWebApp failed: DNS for route host <route-host> did not resolve: DNS for <route-host> did not resolve within <duration> (last error: lookup <host> on <dns-server>: dial...full failure pattern: VerifySimpleWebApp failed: DNS for route host <route-host> did not resolve: DNS for <route-host> did not resolve within <duration> (last error: lookup <host> on <dns-server>: dial udp <ip>: i/o timeout): context deadline exceeded Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 2 day(s) | e2e | 2 | 2.67%2 of 75 job runs affected | IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 2 day(s) | Show trend detailsOct 21: 0 · Oct 22: 0 · Oct 23: 3 · Oct 24: 0 · Oct 25: 2 · Oct 26: 0 · Oct 27: 0 · Oct 28: 0 | INT | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (3)fail [github.com/Azure/ARO-HCP/test/e2e/cluster_autoscaling.go:203]: failed to verify cluster workloads and pods after autoscaling update
Unexpected error:
<*errors.joinError | 0x33a1f1315158>:
VerifySimpleWebApp failed: DNS for route host agnhost-e2e-sample-app-59cgj.apps.aro.s4i2d1i2e0s3o3m.o0jt.eastus2.aroapp-hcp.io did not resolve: DNS for agnhost-e2e-sample-app-59cgj.apps.aro.s4i2d1i2e0s3o3m.o0jt.eastus2.aroapp-hcp.io did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-59cgj.apps.aro.s4i2d1i2e0s3o3m.o0jt.eastus2.aroapp-hcp.io on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded
{
errs: [
<*fmt.wrapError | 0x33a1f0cea780>{
msg: "VerifySimpleWebApp failed: DNS for route host agnhost-e2e-sample-app-59cgj.apps.aro.s4i2d1i2e0s3o3m.o0jt.eastus2.aroapp-hcp.io did not resolve: DNS for agnhost-e2e-sample-app-59cgj.apps.aro.s4i2d1i2e0s3o3m.o0jt.eastus2.aroapp-hcp.io did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-59cgj.apps.aro.s4i2d1i2e0s3o3m.o0jt.eastus2.aroapp-hcp.io on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
err: <*fmt.wrapError | 0x33a1f0bd2f00>{
msg: "DNS for route host agnhost-e2e-sample-app-59cgj.apps.aro.s4i2d1i2e0s3o3m.o0jt.eastus2.aroapp-hcp.io did not resolve: DNS for agnhost-e2e-sample-app-59cgj.apps.aro.s4i2d1i2e0s3o3m.o0jt.eastus2.aroapp-hcp.io did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-59cgj.apps.aro.s4i2d1i2e0s3o3m.o0jt.eastus2.aroapp-hcp.io on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
err: <*fmt.wrapError | 0x33a1f13e4360>{
msg: "DNS for agnhost-e2e-sample-app-59cgj.apps.aro.s4i2d1i2e0s3o3m.o0jt.eastus2.aroapp-hcp.io did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-59cgj.apps.aro.s4i2d1i2e0s3o3m.o0jt.eastus2.aroapp-hcp.io on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
err: <context.deadlineExceededError>{},
},
},
},
],
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_autoscaling.go:203]: failed to verify cluster workloads and pods after autoscaling update
Unexpected error:
<*errors.joinError | 0x33a1f1315158>:
VerifySimpleWebApp failed: DNS for route host agnhost-e2e-sample-app-59cgj.apps.aro.s4i2d1i2e0s3o3m.o0jt.eastus2.aroapp-hcp.io did not resolve: DNS for agnhost-e2e-sample-app-59cgj.apps.aro.s4i2d1i2e0s3o3m.o0jt.eastus2.aroapp-hcp.io did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-59cgj.apps.aro.s4i2d1i2e0s3o3m.o0jt.eastus2.aroapp-hcp.io on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded
{
errs: [
<*fmt.wrapError | 0x33a1f0cea780>{
msg: "VerifySimpleWebApp failed: DNS for route host agnhost-e2e-sample-app-59cgj.apps.aro.s4i2d1i2e0s3o3m.o0jt.eastus2.aroapp-hcp.io did not resolve: DNS for agnhost-e2e-sample-app-59cgj.apps.aro.s4i2d1i2e0s3o3m.o0jt.eastus2.aroapp-hcp.io did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-59cgj.apps.aro.s4i2d1i2e0s3o3m.o0jt.eastus2.aroapp-hcp.io on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
err: <*fmt.wrapError | 0x33a1f0bd2f00>{
msg: "DNS for route host agnhost-e2e-sample-app-59cgj.apps.aro.s4i2d1i2e0s3o3m.o0jt.eastus2.aroapp-hcp.io did not resolve: DNS for agnhost-e2e-sample-app-59cgj.apps.aro.s4i2d1i2e0s3o3m.o0jt.eastus2.aroapp-hcp.io did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-59cgj.apps.aro.s4i2d1i2e0s3o3m.o0jt.eastus2.aroapp-hcp.io on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
err: <*fmt.wrapError | 0x33a1f13e4360>{
msg: "DNS for agnhost-e2e-sample-app-59cgj.apps.aro.s4i2d1i2e0s3o3m.o0jt.eastus2.aroapp-hcp.io did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-59cgj.apps.aro.s4i2d1i2e0s3o3m.o0jt.eastus2.aroapp-hcp.io on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
err: <context.deadlineExceededError>{},
},
},
},
],
}
occurredfail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_cni_cilium.go:186]: failed to run simple web app and connectivity check app with cilium CNI
Unexpected error:
<*errors.joinError | 0xeeb7f5e0a68>:
VerifySimpleWebApp failed: DNS for route host agnhost-e2e-sample-app-4kcwc.apps.aro.cilium-cl.en3z.eastus2.aroapp-hcp.io did not resolve: DNS for agnhost-e2e-sample-app-4kcwc.apps.aro.cilium-cl.en3z.eastus2.aroapp-hcp.io did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-4kcwc.apps.aro.cilium-cl.en3z.eastus2.aroapp-hcp.io on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded
{
errs: [
<*fmt.wrapError | 0xeeb7f3b69e0>{
msg: "VerifySimpleWebApp failed: DNS for route host agnhost-e2e-sample-app-4kcwc.apps.aro.cilium-cl.en3z.eastus2.aroapp-hcp.io did not resolve: DNS for agnhost-e2e-sample-app-4kcwc.apps.aro.cilium-cl.en3z.eastus2.aroapp-hcp.io did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-4kcwc.apps.aro.cilium-cl.en3z.eastus2.aroapp-hcp.io on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
err: <*fmt.wrapError | 0xeeb7f5137a0>{
msg: "DNS for route host agnhost-e2e-sample-app-4kcwc.apps.aro.cilium-cl.en3z.eastus2.aroapp-hcp.io did not resolve: DNS for agnhost-e2e-sample-app-4kcwc.apps.aro.cilium-cl.en3z.eastus2.aroapp-hcp.io did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-4kcwc.apps.aro.cilium-cl.en3z.eastus2.aroapp-hcp.io on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
err: <*fmt.wrapError | 0xeeb7f512040>{
msg: "DNS for agnhost-e2e-sample-app-4kcwc.apps.aro.cilium-cl.en3z.eastus2.aroapp-hcp.io did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-4kcwc.apps.aro.cilium-cl.en3z.eastus2.aroapp-hcp.io on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
err: <context.deadlineExceededError>{},
},
},
},
],
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_cni_cilium.go:186]: failed to run simple web app and connectivity check app with cilium CNI
Unexpected error:
<*errors.joinError | 0xeeb7f5e0a68>:
VerifySimpleWebApp failed: DNS for route host agnhost-e2e-sample-app-4kcwc.apps.aro.cilium-cl.en3z.eastus2.aroapp-hcp.io did not resolve: DNS for agnhost-e2e-sample-app-4kcwc.apps.aro.cilium-cl.en3z.eastus2.aroapp-hcp.io did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-4kcwc.apps.aro.cilium-cl.en3z.eastus2.aroapp-hcp.io on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded
{
errs: [
<*fmt.wrapError | 0xeeb7f3b69e0>{
msg: "VerifySimpleWebApp failed: DNS for route host agnhost-e2e-sample-app-4kcwc.apps.aro.cilium-cl.en3z.eastus2.aroapp-hcp.io did not resolve: DNS for agnhost-e2e-sample-app-4kcwc.apps.aro.cilium-cl.en3z.eastus2.aroapp-hcp.io did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-4kcwc.apps.aro.cilium-cl.en3z.eastus2.aroapp-hcp.io on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
err: <*fmt.wrapError | 0xeeb7f5137a0>{
msg: "DNS for route host agnhost-e2e-sample-app-4kcwc.apps.aro.cilium-cl.en3z.eastus2.aroapp-hcp.io did not resolve: DNS for agnhost-e2e-sample-app-4kcwc.apps.aro.cilium-cl.en3z.eastus2.aroapp-hcp.io did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-4kcwc.apps.aro.cilium-cl.en3z.eastus2.aroapp-hcp.io on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
err: <*fmt.wrapError | 0xeeb7f512040>{
msg: "DNS for agnhost-e2e-sample-app-4kcwc.apps.aro.cilium-cl.en3z.eastus2.aroapp-hcp.io did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-4kcwc.apps.aro.cilium-cl.en3z.eastus2.aroapp-hcp.io on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
err: <context.deadlineExceededError>{},
},
},
},
],
}
occurredfail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:197]: failed to run simple web app and connectivity check with cilium CNI
Unexpected error:
<*errors.joinError | 0xdf1d3a48c78>:
VerifySimpleWebApp failed: DNS for route host agnhost-e2e-sample-app-rx2sd.apps.aro.cilium-cluster.k38l.eastus2.aroapp-hcp.io did not resolve: DNS for agnhost-e2e-sample-app-rx2sd.apps.aro.cilium-cluster.k38l.eastus2.aroapp-hcp.io did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-rx2sd.apps.aro.cilium-cluster.k38l.eastus2.aroapp-hcp.io on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded
{
errs: [
<*fmt.wrapError | 0xdf1d48b58c0>{
msg: "VerifySimpleWebApp failed: DNS for route host agnhost-e2e-sample-app-rx2sd.apps.aro.cilium-cluster.k38l.eastus2.aroapp-hcp.io did not resolve: DNS for agnhost-e2e-sample-app-rx2sd.apps.aro.cilium-cluster.k38l.eastus2.aroapp-hcp.io did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-rx2sd.apps.aro.cilium-cluster.k38l.eastus2.aroapp-hcp.io on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
err: <*fmt.wrapError | 0xdf1d48b46a0>{
msg: "DNS for route host agnhost-e2e-sample-app-rx2sd.apps.aro.cilium-cluster.k38l.eastus2.aroapp-hcp.io did not resolve: DNS for agnhost-e2e-sample-app-rx2sd.apps.aro.cilium-cluster.k38l.eastus2.aroapp-hcp.io did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-rx2sd.apps.aro.cilium-cluster.k38l.eastus2.aroapp-hcp.io on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
err: <*fmt.wrapError | 0xdf1d394a3a0>{
msg: "DNS for agnhost-e2e-sample-app-rx2sd.apps.aro.cilium-cluster.k38l.eastus2.aroapp-hcp.io did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-rx2sd.apps.aro.cilium-cluster.k38l.eastus2.aroapp-hcp.io on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
err: <context.deadlineExceededError>{},
},
},
},
],
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:197]: failed to run simple web app and connectivity check with cilium CNI
Unexpected error:
<*errors.joinError | 0xdf1d3a48c78>:
VerifySimpleWebApp failed: DNS for route host agnhost-e2e-sample-app-rx2sd.apps.aro.cilium-cluster.k38l.eastus2.aroapp-hcp.io did not resolve: DNS for agnhost-e2e-sample-app-rx2sd.apps.aro.cilium-cluster.k38l.eastus2.aroapp-hcp.io did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-rx2sd.apps.aro.cilium-cluster.k38l.eastus2.aroapp-hcp.io on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded
{
errs: [
<*fmt.wrapError | 0xdf1d48b58c0>{
msg: "VerifySimpleWebApp failed: DNS for route host agnhost-e2e-sample-app-rx2sd.apps.aro.cilium-cluster.k38l.eastus2.aroapp-hcp.io did not resolve: DNS for agnhost-e2e-sample-app-rx2sd.apps.aro.cilium-cluster.k38l.eastus2.aroapp-hcp.io did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-rx2sd.apps.aro.cilium-cluster.k38l.eastus2.aroapp-hcp.io on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
err: <*fmt.wrapError | 0xdf1d48b46a0>{
msg: "DNS for route host agnhost-e2e-sample-app-rx2sd.apps.aro.cilium-cluster.k38l.eastus2.aroapp-hcp.io did not resolve: DNS for agnhost-e2e-sample-app-rx2sd.apps.aro.cilium-cluster.k38l.eastus2.aroapp-hcp.io did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-rx2sd.apps.aro.cilium-cluster.k38l.eastus2.aroapp-hcp.io on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
err: <*fmt.wrapError | 0xdf1d394a3a0>{
msg: "DNS for agnhost-e2e-sample-app-rx2sd.apps.aro.cilium-cluster.k38l.eastus2.aroapp-hcp.io did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-rx2sd.apps.aro.cilium-cluster.k38l.eastus2.aroapp-hcp.io on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
err: <context.deadlineExceededError>{},
},
},
},
],
}
occurredContributing tests (4)
Affected runs (2)
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedC...full failure pattern: ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster has no control plane endpoint host; provider Microsoft.RedHatOpenShift Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 2 day(s) | e2e | 2 | 2.67%2 of 75 job runs affected | IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 2 day(s) | Show trend detailsOct 21: 0 · Oct 22: 2 · Oct 23: 0 · Oct 24: 0 · Oct 25: 0 · Oct 26: 0 · Oct 27: 0 · Oct 28: 1 | none | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (3)fail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:111]: failed to create HCP cluster "cidr-connectivity-test" with authorized CIDRs
Unexpected error:
<*fmt.wrapError | 0x3b896839fa60>:
failed to create HCP cluster cidr-connectivity-test: failed waiting for cluster="cidr-connectivity-test" in resourcegroup="e2e-cidr-connectivity-fld8q7cxfnnl" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/fad0e7c8-51f4-402f-b2c0-d5c286ac7592
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/fad0e7c8-51f4-402f-b2c0-d5c286ac7592",
"name": "fad0e7c8-51f4-402f-b2c0-d5c286ac7592",
"status": "Failed",
"startTime": "2026-10-28T13:44:11.609067289Z",
"endTime": "2026-10-28T14:13:20.968792662Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster has no control plane endpoint host"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/fad0e7c8-51f4-402f-b2c0-d5c286ac7592
correlation ID: 19e863b6-3287-4026-a228-cda73ba74c13
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:111]: failed to create HCP cluster "cidr-connectivity-test" with authorized CIDRs
Unexpected error:
<*fmt.wrapError | 0x3b896839fa60>:
failed to create HCP cluster cidr-connectivity-test: failed waiting for cluster="cidr-connectivity-test" in resourcegroup="e2e-cidr-connectivity-fld8q7cxfnnl" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/fad0e7c8-51f4-402f-b2c0-d5c286ac7592
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/fad0e7c8-51f4-402f-b2c0-d5c286ac7592",
"name": "fad0e7c8-51f4-402f-b2c0-d5c286ac7592",
"status": "Failed",
"startTime": "2026-10-28T13:44:11.609067289Z",
"endTime": "2026-10-28T14:13:20.968792662Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster has no control plane endpoint host"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/fad0e7c8-51f4-402f-b2c0-d5c286ac7592
correlation ID: 19e863b6-3287-4026-a228-cda73ba74c13
client request ID: unavailable
occurredfail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:291]: failed to create HCP cluster node-limit-cluster with MaxNodesTotal limit
Unexpected error:
<*fmt.wrapError | 0x262004875180>:
failed to create HCP cluster node-limit-cluster: failed waiting for cluster="node-limit-cluster" in resourcegroup="node-limit-test-v4w4cbtgl6st" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/0ffe01b2-d2c4-4e16-a6d6-042cbbeb51ec
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/0ffe01b2-d2c4-4e16-a6d6-042cbbeb51ec",
"name": "0ffe01b2-d2c4-4e16-a6d6-042cbbeb51ec",
"status": "Failed",
"startTime": "2026-10-22T02:27:27.45827201Z",
"endTime": "2026-10-22T02:56:30.386782585Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster has no control plane endpoint host"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/0ffe01b2-d2c4-4e16-a6d6-042cbbeb51ec
correlation ID: a0e42f8f-bf63-4670-b77c-e6ba06f19444
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:291]: failed to create HCP cluster node-limit-cluster with MaxNodesTotal limit
Unexpected error:
<*fmt.wrapError | 0x262004875180>:
failed to create HCP cluster node-limit-cluster: failed waiting for cluster="node-limit-cluster" in resourcegroup="node-limit-test-v4w4cbtgl6st" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/0ffe01b2-d2c4-4e16-a6d6-042cbbeb51ec
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/0ffe01b2-d2c4-4e16-a6d6-042cbbeb51ec",
"name": "0ffe01b2-d2c4-4e16-a6d6-042cbbeb51ec",
"status": "Failed",
"startTime": "2026-10-22T02:27:27.45827201Z",
"endTime": "2026-10-22T02:56:30.386782585Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster has no control plane endpoint host"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/0ffe01b2-d2c4-4e16-a6d6-042cbbeb51ec
correlation ID: a0e42f8f-bf63-4670-b77c-e6ba06f19444
client request ID: unavailable
occurredfail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:133]: failed to create HCP cluster "cp-ystream-upgrade-4-21-q2kkkp" at install version 4.20
Unexpected error:
<*fmt.wrapError | 0x300bf5b7a840>:
failed to create HCP cluster cp-ystream-upgrade-4-21-q2kkkp: failed waiting for cluster="cp-ystream-upgrade-4-21-q2kkkp" in resourcegroup="rg-cp-ystream-upgrade-4-21-t4mn7sf29xrr" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/54a4385e-0b42-4987-acf3-1b0207b030a5
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/54a4385e-0b42-4987-acf3-1b0207b030a5",
"name": "54a4385e-0b42-4987-acf3-1b0207b030a5",
"status": "Failed",
"startTime": "2026-10-22T02:23:48.751941498Z",
"endTime": "2026-10-22T02:52:50.375994252Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster has no control plane endpoint host"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/54a4385e-0b42-4987-acf3-1b0207b030a5
correlation ID: f3ca853a-64dc-4e52-ab57-050647054f81
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:133]: failed to create HCP cluster "cp-ystream-upgrade-4-21-q2kkkp" at install version 4.20
Unexpected error:
<*fmt.wrapError | 0x300bf5b7a840>:
failed to create HCP cluster cp-ystream-upgrade-4-21-q2kkkp: failed waiting for cluster="cp-ystream-upgrade-4-21-q2kkkp" in resourcegroup="rg-cp-ystream-upgrade-4-21-t4mn7sf29xrr" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/54a4385e-0b42-4987-acf3-1b0207b030a5
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/54a4385e-0b42-4987-acf3-1b0207b030a5",
"name": "54a4385e-0b42-4987-acf3-1b0207b030a5",
"status": "Failed",
"startTime": "2026-10-22T02:23:48.751941498Z",
"endTime": "2026-10-22T02:52:50.375994252Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster has no control plane endpoint host"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/54a4385e-0b42-4987-acf3-1b0207b030a5
correlation ID: f3ca853a-64dc-4e52-ab57-050647054f81
client request ID: unavailable
occurredContributing tests (3)
Affected runs (2)
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
timeout during CreateHCPClusterAndWait; context deadline exceededfull failure pattern: timeout during CreateHCPClusterAndWait; context deadline exceeded Signal: IndeterminateSignal: Indeterminate — seen in 2 prior week(s); active 2 day(s) | e2e | 2 | 2.67%2 of 75 job runs affected | IndeterminateSignal: Indeterminate — seen in 2 prior week(s); active 2 day(s) | Show trend detailsOct 21: 0 · Oct 22: 1 · Oct 23: 0 · Oct 24: 0 · Oct 25: 0 · Oct 26: 0 · Oct 27: 0 · Oct 28: 2 | none | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (3)fail [github.com/Azure/ARO-HCP/test/e2e/idms_lifecycle.go:113]: failed to create HCP cluster with ImageDigestMirrors
Unexpected error:
<*fmt.wrapErrors | 0x1f0baa544120>:
failed to create HCP cluster idms-e2e-hcp-cluster, caused by: timeout '30.000000' minutes exceeded during CreateHCPCluster20251223FromParam for cluster idms-e2e-hcp-cluster in resource group idms-n4hrqndpjj88, error: failed waiting for cluster="idms-e2e-hcp-cluster" in resourcegroup="idms-n4hrqndpjj88" to finish creating, caused by: timeout '30.000000' minutes exceeded during CreateHCPCluster20251223FromParam for cluster idms-e2e-hcp-cluster in resource group idms-n4hrqndpjj88, error: context deadline exceeded
{
msg: "failed to create HCP cluster idms-e2e-hcp-cluster, caused by: timeout '30.000000' minutes exceeded during CreateHCPCluster20251223FromParam for cluster idms-e2e-hcp-cluster in resource group idms-n4hrqndpjj88, error: failed waiting for cluster=\"idms-e2e-hcp-cluster\" in resourcegroup=\"idms-n4hrqndpjj88\" to finish creating, caused by: timeout '30.000000' minutes exceeded during CreateHCPCluster20251223FromParam for cluster idms-e2e-hcp-cluster in resource group idms-n4hrqndpjj88, error: context deadline exceeded",
errs: [
<*errors.errorString | 0x1f0baa40b610>{
s: "timeout '30.000000' minutes exceeded during CreateHCPCluster20251223FromParam for cluster idms-e2e-hcp-cluster in resource group idms-n4hrqndpjj88",
},
<*fmt.wrapErrors | 0x1f0baa5440c0>{
msg: "failed waiting for cluster=\"idms-e2e-hcp-cluster\" in resourcegroup=\"idms-n4hrqndpjj88\" to finish creating, caused by: timeout '30.000000' minutes exceeded during CreateHCPCluster20251223FromParam for cluster idms-e2e-hcp-cluster in resource group idms-n4hrqndpjj88, error: context deadline exceeded",
errs: [
<*errors.errorString | 0x1f0baa40b610>0x1f0baa40b610 (cyclic reference),
<context.deadlineExceededError>{},
],
},
],
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/idms_lifecycle.go:113]: failed to create HCP cluster with ImageDigestMirrors
Unexpected error:
<*fmt.wrapErrors | 0x1f0baa544120>:
failed to create HCP cluster idms-e2e-hcp-cluster, caused by: timeout '30.000000' minutes exceeded during CreateHCPCluster20251223FromParam for cluster idms-e2e-hcp-cluster in resource group idms-n4hrqndpjj88, error: failed waiting for cluster="idms-e2e-hcp-cluster" in resourcegroup="idms-n4hrqndpjj88" to finish creating, caused by: timeout '30.000000' minutes exceeded during CreateHCPCluster20251223FromParam for cluster idms-e2e-hcp-cluster in resource group idms-n4hrqndpjj88, error: context deadline exceeded
{
msg: "failed to create HCP cluster idms-e2e-hcp-cluster, caused by: timeout '30.000000' minutes exceeded during CreateHCPCluster20251223FromParam for cluster idms-e2e-hcp-cluster in resource group idms-n4hrqndpjj88, error: failed waiting for cluster=\"idms-e2e-hcp-cluster\" in resourcegroup=\"idms-n4hrqndpjj88\" to finish creating, caused by: timeout '30.000000' minutes exceeded during CreateHCPCluster20251223FromParam for cluster idms-e2e-hcp-cluster in resource group idms-n4hrqndpjj88, error: context deadline exceeded",
errs: [
<*errors.errorString | 0x1f0baa40b610>{
s: "timeout '30.000000' minutes exceeded during CreateHCPCluster20251223FromParam for cluster idms-e2e-hcp-cluster in resource group idms-n4hrqndpjj88",
},
<*fmt.wrapErrors | 0x1f0baa5440c0>{
msg: "failed waiting for cluster=\"idms-e2e-hcp-cluster\" in resourcegroup=\"idms-n4hrqndpjj88\" to finish creating, caused by: timeout '30.000000' minutes exceeded during CreateHCPCluster20251223FromParam for cluster idms-e2e-hcp-cluster in resource group idms-n4hrqndpjj88, error: context deadline exceeded",
errs: [
<*errors.errorString | 0x1f0baa40b610>0x1f0baa40b610 (cyclic reference),
<context.deadlineExceededError>{},
],
},
],
}
occurredfail [github.com/Azure/ARO-HCP/test/e2e/cluster_version_backlevel.go:152]: failed to create HCP cluster version 4.20
Unexpected error:
<*fmt.wrapErrors | 0x22e10818450>:
failed waiting for cluster="cluster-ver-4-20" in resourcegroup="rg-cluster-back-version-992c2j7mq2x8" to finish creating, caused by: timeout '30.000000' minutes exceeded during CreateHCPClusterAndWait for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-992c2j7mq2x8, error: context deadline exceeded
{
msg: "failed waiting for cluster=\"cluster-ver-4-20\" in resourcegroup=\"rg-cluster-back-version-992c2j7mq2x8\" to finish creating, caused by: timeout '30.000000' minutes exceeded during CreateHCPClusterAndWait for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-992c2j7mq2x8, error: context deadline exceeded",
errs: [
<*errors.errorString | 0x22e1036b8e0>{
s: "timeout '30.000000' minutes exceeded during CreateHCPClusterAndWait for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-992c2j7mq2x8",
},
<context.deadlineExceededError>{},
],
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_version_backlevel.go:152]: failed to create HCP cluster version 4.20
Unexpected error:
<*fmt.wrapErrors | 0x22e10818450>:
failed waiting for cluster="cluster-ver-4-20" in resourcegroup="rg-cluster-back-version-992c2j7mq2x8" to finish creating, caused by: timeout '30.000000' minutes exceeded during CreateHCPClusterAndWait for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-992c2j7mq2x8, error: context deadline exceeded
{
msg: "failed waiting for cluster=\"cluster-ver-4-20\" in resourcegroup=\"rg-cluster-back-version-992c2j7mq2x8\" to finish creating, caused by: timeout '30.000000' minutes exceeded during CreateHCPClusterAndWait for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-992c2j7mq2x8, error: context deadline exceeded",
errs: [
<*errors.errorString | 0x22e1036b8e0>{
s: "timeout '30.000000' minutes exceeded during CreateHCPClusterAndWait for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-992c2j7mq2x8",
},
<context.deadlineExceededError>{},
],
}
occurredfail [github.com/Azure/ARO-HCP/test/e2e/cluster_version_backlevel.go:152]: failed to create HCP cluster version 4.20
Unexpected error:
<*fmt.wrapErrors | 0x146ba53e5a0>:
failed waiting for cluster="cluster-ver-4-20" in resourcegroup="rg-cluster-back-version-vgggt2t7qcsd" to finish creating, caused by: timeout '30.000000' minutes exceeded during CreateHCPClusterAndWait for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-vgggt2t7qcsd, error: context deadline exceeded
{
msg: "failed waiting for cluster=\"cluster-ver-4-20\" in resourcegroup=\"rg-cluster-back-version-vgggt2t7qcsd\" to finish creating, caused by: timeout '30.000000' minutes exceeded during CreateHCPClusterAndWait for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-vgggt2t7qcsd, error: context deadline exceeded",
errs: [
<*errors.errorString | 0x146ba6ff790>{
s: "timeout '30.000000' minutes exceeded during CreateHCPClusterAndWait for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-vgggt2t7qcsd",
},
<context.deadlineExceededError>{},
],
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_version_backlevel.go:152]: failed to create HCP cluster version 4.20
Unexpected error:
<*fmt.wrapErrors | 0x146ba53e5a0>:
failed waiting for cluster="cluster-ver-4-20" in resourcegroup="rg-cluster-back-version-vgggt2t7qcsd" to finish creating, caused by: timeout '30.000000' minutes exceeded during CreateHCPClusterAndWait for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-vgggt2t7qcsd, error: context deadline exceeded
{
msg: "failed waiting for cluster=\"cluster-ver-4-20\" in resourcegroup=\"rg-cluster-back-version-vgggt2t7qcsd\" to finish creating, caused by: timeout '30.000000' minutes exceeded during CreateHCPClusterAndWait for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-vgggt2t7qcsd, error: context deadline exceeded",
errs: [
<*errors.errorString | 0x146ba6ff790>{
s: "timeout '30.000000' minutes exceeded during CreateHCPClusterAndWait for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-vgggt2t7qcsd",
},
<context.deadlineExceededError>{},
],
}
occurredContributing tests (2)
Affected runs (2)
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedC...full failure pattern: ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider, cluster-api, ignition-server]; provider Microsoft.RedHatOpenShift Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 2 day(s) | e2e | 2 | 2.67%2 of 75 job runs affected | IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 2 day(s) | Show trend detailsOct 21: 0 · Oct 22: 2 · Oct 23: 0 · Oct 24: 0 · Oct 25: 0 · Oct 26: 0 · Oct 27: 0 · Oct 28: 1 | none | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (3)fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delete_cx_rg.go:92]: failed to create HCP cluster "cx-rg-hcp-cluster"
Unexpected error:
<*fmt.wrapError | 0x320c1cc145a0>:
failed to create HCP cluster cx-rg-hcp-cluster: failed waiting for cluster="cx-rg-hcp-cluster" in resourcegroup="cx-rg-cluster-z8bmdl4f2djz" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/6f2afbda-3f8a-4abe-93da-0b42da0b54b6
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/6f2afbda-3f8a-4abe-93da-0b42da0b54b6",
"name": "6f2afbda-3f8a-4abe-93da-0b42da0b54b6",
"status": "Failed",
"startTime": "2026-10-28T13:43:08.173393078Z",
"endTime": "2026-10-28T14:12:10.809270873Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider deployment has 1 unavailable replicas, cluster-api deployment has 2 unavailable replicas, ignition-server deployment has 1 unavailable replicas]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/6f2afbda-3f8a-4abe-93da-0b42da0b54b6
correlation ID: 446aa419-d2ee-4b05-829a-2ef1a0102b8b
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delete_cx_rg.go:92]: failed to create HCP cluster "cx-rg-hcp-cluster"
Unexpected error:
<*fmt.wrapError | 0x320c1cc145a0>:
failed to create HCP cluster cx-rg-hcp-cluster: failed waiting for cluster="cx-rg-hcp-cluster" in resourcegroup="cx-rg-cluster-z8bmdl4f2djz" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/6f2afbda-3f8a-4abe-93da-0b42da0b54b6
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/6f2afbda-3f8a-4abe-93da-0b42da0b54b6",
"name": "6f2afbda-3f8a-4abe-93da-0b42da0b54b6",
"status": "Failed",
"startTime": "2026-10-28T13:43:08.173393078Z",
"endTime": "2026-10-28T14:12:10.809270873Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider deployment has 1 unavailable replicas, cluster-api deployment has 2 unavailable replicas, ignition-server deployment has 1 unavailable replicas]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/6f2afbda-3f8a-4abe-93da-0b42da0b54b6
correlation ID: 446aa419-d2ee-4b05-829a-2ef1a0102b8b
client request ID: unavailable
occurredfail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:133]: failed to create HCP cluster "cp-ystream-upgrade-4-23-bbjx5c" at install version 4.22
Unexpected error:
<*fmt.wrapError | 0x2052152a2720>:
failed to create HCP cluster cp-ystream-upgrade-4-23-bbjx5c: failed waiting for cluster="cp-ystream-upgrade-4-23-bbjx5c" in resourcegroup="rg-cp-ystream-upgrade-4-23-56g2lp75qrkf" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/02ef6437-1d48-43c3-ad2d-b3b8f18f1b53
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/02ef6437-1d48-43c3-ad2d-b3b8f18f1b53",
"name": "02ef6437-1d48-43c3-ad2d-b3b8f18f1b53",
"status": "Failed",
"startTime": "2026-10-22T02:23:39.532402776Z",
"endTime": "2026-10-22T02:52:40.523543888Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider deployment has 1 unavailable replicas, cluster-api deployment has 1 unavailable replicas, ignition-server deployment has 1 unavailable replicas]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/02ef6437-1d48-43c3-ad2d-b3b8f18f1b53
correlation ID: 43512322-23b0-4759-b738-201cec5eb6f3
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:133]: failed to create HCP cluster "cp-ystream-upgrade-4-23-bbjx5c" at install version 4.22
Unexpected error:
<*fmt.wrapError | 0x2052152a2720>:
failed to create HCP cluster cp-ystream-upgrade-4-23-bbjx5c: failed waiting for cluster="cp-ystream-upgrade-4-23-bbjx5c" in resourcegroup="rg-cp-ystream-upgrade-4-23-56g2lp75qrkf" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/02ef6437-1d48-43c3-ad2d-b3b8f18f1b53
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/02ef6437-1d48-43c3-ad2d-b3b8f18f1b53",
"name": "02ef6437-1d48-43c3-ad2d-b3b8f18f1b53",
"status": "Failed",
"startTime": "2026-10-22T02:23:39.532402776Z",
"endTime": "2026-10-22T02:52:40.523543888Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider deployment has 1 unavailable replicas, cluster-api deployment has 1 unavailable replicas, ignition-server deployment has 1 unavailable replicas]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/02ef6437-1d48-43c3-ad2d-b3b8f18f1b53
correlation ID: 43512322-23b0-4759-b738-201cec5eb6f3
client request ID: unavailable
occurredfail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:124]: HCP cluster rg-candidate-5-0-gmftj87c69s6/cluster-candidate-5-0-25snrj should provision
Unexpected error:
<*fmt.wrapError | 0x1ed3b41bc940>:
failed to create HCP cluster cluster-candidate-5-0-25snrj: failed waiting for cluster="cluster-candidate-5-0-25snrj" in resourcegroup="rg-candidate-5-0-gmftj87c69s6" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/27425e0a-5b12-47d6-9369-3a3a90aa3e86
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/27425e0a-5b12-47d6-9369-3a3a90aa3e86",
"name": "27425e0a-5b12-47d6-9369-3a3a90aa3e86",
"status": "Failed",
"startTime": "2026-10-22T02:21:48.683878735Z",
"endTime": "2026-10-22T02:50:50.395326627Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider deployment has 1 unavailable replicas, cluster-api deployment has 2 unavailable replicas, ignition-server deployment has 1 unavailable replicas]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/27425e0a-5b12-47d6-9369-3a3a90aa3e86
correlation ID: 52162704-0f0b-4a93-a71e-0f363b9222b6
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:124]: HCP cluster rg-candidate-5-0-gmftj87c69s6/cluster-candidate-5-0-25snrj should provision
Unexpected error:
<*fmt.wrapError | 0x1ed3b41bc940>:
failed to create HCP cluster cluster-candidate-5-0-25snrj: failed waiting for cluster="cluster-candidate-5-0-25snrj" in resourcegroup="rg-candidate-5-0-gmftj87c69s6" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/27425e0a-5b12-47d6-9369-3a3a90aa3e86
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/27425e0a-5b12-47d6-9369-3a3a90aa3e86",
"name": "27425e0a-5b12-47d6-9369-3a3a90aa3e86",
"status": "Failed",
"startTime": "2026-10-22T02:21:48.683878735Z",
"endTime": "2026-10-22T02:50:50.395326627Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider deployment has 1 unavailable replicas, cluster-api deployment has 2 unavailable replicas, ignition-server deployment has 1 unavailable replicas]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/27425e0a-5b12-47d6-9369-3a3a90aa3e86
correlation ID: 52162704-0f0b-4a93-a71e-0f363b9222b6
client request ID: unavailable
occurredContributing tests (3)
Affected runs (2)
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet complete...full failure pattern: ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version <version> is Partial (want Completed), started <duration> ago; hosted cluster degraded: UnavailableReplicas: [ignition-server-proxy, kube-controller-manager]; provider Microsoft.RedHatOpenShift Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 2 day(s) | e2e | 2 | 2.67%2 of 75 job runs affected | IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 2 day(s) | Show trend detailsOct 21: 0 · Oct 22: 1 · Oct 23: 0 · Oct 24: 0 · Oct 25: 0 · Oct 26: 0 · Oct 27: 0 · Oct 28: 2 | none | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (3)fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:150]: failed to create HCP cluster
Unexpected error:
<*fmt.wrapError | 0x3d84c606700>:
failed to create HCP cluster ingress-cert-le: failed waiting for cluster="ingress-cert-le" in resourcegroup="ingress-cert-le-gq7pn8p46nb6" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/8792581c-3981-4bcf-b8a1-ffd33680ce15
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/8792581c-3981-4bcf-b8a1-ffd33680ce15",
"name": "8792581c-3981-4bcf-b8a1-ffd33680ce15",
"status": "Failed",
"startTime": "2026-10-28T13:42:50.808216413Z",
"endTime": "2026-10-28T14:11:50.825999726Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 5.0.0-rc.7 is Partial (want Completed), started 24m17s ago; hosted cluster degraded: UnavailableReplicas: [ignition-server-proxy deployment has 1 unavailable replicas, kube-controller-manager deployment has 1 unavailable replicas]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/8792581c-3981-4bcf-b8a1-ffd33680ce15
correlation ID: 331b0b35-9b61-4406-b1d0-4fe22089cb00
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:150]: failed to create HCP cluster
Unexpected error:
<*fmt.wrapError | 0x3d84c606700>:
failed to create HCP cluster ingress-cert-le: failed waiting for cluster="ingress-cert-le" in resourcegroup="ingress-cert-le-gq7pn8p46nb6" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/8792581c-3981-4bcf-b8a1-ffd33680ce15
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/8792581c-3981-4bcf-b8a1-ffd33680ce15",
"name": "8792581c-3981-4bcf-b8a1-ffd33680ce15",
"status": "Failed",
"startTime": "2026-10-28T13:42:50.808216413Z",
"endTime": "2026-10-28T14:11:50.825999726Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 5.0.0-rc.7 is Partial (want Completed), started 24m17s ago; hosted cluster degraded: UnavailableReplicas: [ignition-server-proxy deployment has 1 unavailable replicas, kube-controller-manager deployment has 1 unavailable replicas]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/8792581c-3981-4bcf-b8a1-ffd33680ce15
correlation ID: 331b0b35-9b61-4406-b1d0-4fe22089cb00
client request ID: unavailable
occurredfail [github.com/Azure/ARO-HCP/test/e2e/oidc_issuer_workload_identity.go:227]: failed to create HCP cluster oidc-wi-cluster
Unexpected error:
<*fmt.wrapError | 0x1b9a78cc2c00>:
failed to create HCP cluster oidc-wi-cluster: failed waiting for cluster="oidc-wi-cluster" in resourcegroup="oidc-wi-678vggk8hw4n" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/95bd7f6e-d28c-4575-8824-c4b761c225fc
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/95bd7f6e-d28c-4575-8824-c4b761c225fc",
"name": "95bd7f6e-d28c-4575-8824-c4b761c225fc",
"status": "Failed",
"startTime": "2026-10-28T15:21:43.03617172Z",
"endTime": "2026-10-28T15:50:47.66441232Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.40 is Partial (want Completed), started 24m7s ago; hosted cluster degraded: UnavailableReplicas: [ignition-server-proxy deployment has 1 unavailable replicas, kube-controller-manager deployment has 1 unavailable replicas]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/95bd7f6e-d28c-4575-8824-c4b761c225fc
correlation ID: 414e22c5-a26a-4c3c-96be-d66c9e15bf90
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/oidc_issuer_workload_identity.go:227]: failed to create HCP cluster oidc-wi-cluster
Unexpected error:
<*fmt.wrapError | 0x1b9a78cc2c00>:
failed to create HCP cluster oidc-wi-cluster: failed waiting for cluster="oidc-wi-cluster" in resourcegroup="oidc-wi-678vggk8hw4n" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/95bd7f6e-d28c-4575-8824-c4b761c225fc
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/95bd7f6e-d28c-4575-8824-c4b761c225fc",
"name": "95bd7f6e-d28c-4575-8824-c4b761c225fc",
"status": "Failed",
"startTime": "2026-10-28T15:21:43.03617172Z",
"endTime": "2026-10-28T15:50:47.66441232Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.40 is Partial (want Completed), started 24m7s ago; hosted cluster degraded: UnavailableReplicas: [ignition-server-proxy deployment has 1 unavailable replicas, kube-controller-manager deployment has 1 unavailable replicas]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/95bd7f6e-d28c-4575-8824-c4b761c225fc
correlation ID: 414e22c5-a26a-4c3c-96be-d66c9e15bf90
client request ID: unavailable
occurredfail [github.com/Azure/ARO-HCP/test/e2e/cluster_delete_missing_identities.go:113]: failed to create HCP cluster "missing-mi-hcp-cluster"
Unexpected error:
<*fmt.wrapError | 0x28c6381f07c0>:
failed to create HCP cluster missing-mi-hcp-cluster: failed waiting for cluster="missing-mi-hcp-cluster" in resourcegroup="delete-mi-rg-plwcnbx7hnw5" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/5f96f74a-1c0c-4295-a2c6-9dce177af3f9
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/5f96f74a-1c0c-4295-a2c6-9dce177af3f9",
"name": "5f96f74a-1c0c-4295-a2c6-9dce177af3f9",
"status": "Failed",
"startTime": "2026-10-22T01:34:31.83333937Z",
"endTime": "2026-10-22T02:03:36.684949695Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.39 is Partial (want Completed), started 24m44s ago; hosted cluster degraded: UnavailableReplicas: [ignition-server-proxy deployment has 1 unavailable replicas, kube-controller-manager deployment has 1 unavailable replicas]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/5f96f74a-1c0c-4295-a2c6-9dce177af3f9
correlation ID: aad1d81c-32be-4413-84a9-7ac0b5dcc1a8
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delete_missing_identities.go:113]: failed to create HCP cluster "missing-mi-hcp-cluster"
Unexpected error:
<*fmt.wrapError | 0x28c6381f07c0>:
failed to create HCP cluster missing-mi-hcp-cluster: failed waiting for cluster="missing-mi-hcp-cluster" in resourcegroup="delete-mi-rg-plwcnbx7hnw5" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/5f96f74a-1c0c-4295-a2c6-9dce177af3f9
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/5f96f74a-1c0c-4295-a2c6-9dce177af3f9",
"name": "5f96f74a-1c0c-4295-a2c6-9dce177af3f9",
"status": "Failed",
"startTime": "2026-10-22T01:34:31.83333937Z",
"endTime": "2026-10-22T02:03:36.684949695Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.39 is Partial (want Completed), started 24m44s ago; hosted cluster degraded: UnavailableReplicas: [ignition-server-proxy deployment has 1 unavailable replicas, kube-controller-manager deployment has 1 unavailable replicas]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/5f96f74a-1c0c-4295-a2c6-9dce177af3f9
correlation ID: aad1d81c-32be-4413-84a9-7ac0b5dcc1a8
client request ID: unavailable
occurredContributing tests (3)
Affected runs (2)
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Unauthorizedfull failure pattern: VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Unauthorized Signal: IndeterminateSignal: Indeterminate — seen in 4 prior week(s); active 2 day(s) | e2e | 2 | 2.67%2 of 75 job runs affected | IndeterminateSignal: Indeterminate — seen in 4 prior week(s); active 2 day(s) | Show trend detailsOct 21: 0 · Oct 22: 0 · Oct 23: 1 · Oct 24: 0 · Oct 25: 0 · Oct 26: 1 · Oct 27: 0 · Oct 28: 0 | STG | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (2)fail [github.com/Azure/ARO-HCP/test/e2e/simple_negative_cases.go:124]: cluster negative-tests-cluster is not viable
Unexpected error:
<*errors.joinError | 0x2f7afefabce0>:
VerifyBasicAccess failed: failed to list services: Unauthorized
VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Unauthorized
{
errs: [
<*fmt.wrapError | 0x2f7afec3aa20>{
msg: "VerifyBasicAccess failed: failed to list services: Unauthorized",
err: <*fmt.wrapError | 0x2f7afec3aa00>{
msg: "failed to list services: Unauthorized",
err: <*errors.StatusError | 0x2f7aff732b40>{
ErrStatus: {
TypeMeta: {Kind: "", APIVersion: ""},
ListMeta: {
SelfLink: "",
ResourceVersion: "",
Continue: "",
RemainingItemCount: nil,
ShardInfo: nil,
},
Status: "Failure",
Message: "Unauthorized",
Reason: "Unauthorized",
Details: nil,
Code: 401,
},
},
},
},
<*fmt.wrapError | 0x2f7afee8d220>{
msg: "VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Unauthorized",
err: <*fmt.wrapError | 0x2f7afee8d200>{
msg: "failed to list all APIServices: Unauthorized",
err: <*errors.StatusError | 0x2f7afee1fae0>{
ErrStatus: {
TypeMeta: {Kind: "", APIVersion: ""},
ListMeta: {
SelfLink: "",
ResourceVersion: "",
Continue: "",
RemainingItemCount: nil,
ShardInfo: nil,
},
Status: "Failure",
Message: "Unauthorized",
Reason: "Unauthorized",
Details: nil,
Code: 401,
},
},
},
},
],
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/simple_negative_cases.go:124]: cluster negative-tests-cluster is not viable
Unexpected error:
<*errors.joinError | 0x2f7afefabce0>:
VerifyBasicAccess failed: failed to list services: Unauthorized
VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Unauthorized
{
errs: [
<*fmt.wrapError | 0x2f7afec3aa20>{
msg: "VerifyBasicAccess failed: failed to list services: Unauthorized",
err: <*fmt.wrapError | 0x2f7afec3aa00>{
msg: "failed to list services: Unauthorized",
err: <*errors.StatusError | 0x2f7aff732b40>{
ErrStatus: {
TypeMeta: {Kind: "", APIVersion: ""},
ListMeta: {
SelfLink: "",
ResourceVersion: "",
Continue: "",
RemainingItemCount: nil,
ShardInfo: nil,
},
Status: "Failure",
Message: "Unauthorized",
Reason: "Unauthorized",
Details: nil,
Code: 401,
},
},
},
},
<*fmt.wrapError | 0x2f7afee8d220>{
msg: "VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Unauthorized",
err: <*fmt.wrapError | 0x2f7afee8d200>{
msg: "failed to list all APIServices: Unauthorized",
err: <*errors.StatusError | 0x2f7afee1fae0>{
ErrStatus: {
TypeMeta: {Kind: "", APIVersion: ""},
ListMeta: {
SelfLink: "",
ResourceVersion: "",
Continue: "",
RemainingItemCount: nil,
ShardInfo: nil,
},
Status: "Failure",
Message: "Unauthorized",
Reason: "Unauthorized",
Details: nil,
Code: 401,
},
},
},
},
],
}
occurredfail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:140]: failed to verify HCP cluster "cluster-nightly-4-21-hwnzpg" is viable
Unexpected error:
<*errors.joinError | 0x2be9a2dba918>:
VerifyBasicAccess failed: failed to list services: Unauthorized
VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Unauthorized
{
errs: [
<*fmt.wrapError | 0x2be9a2235d00>{
msg: "VerifyBasicAccess failed: failed to list services: Unauthorized",
err: <*fmt.wrapError | 0x2be9a2235cc0>{
msg: "failed to list services: Unauthorized",
err: <*errors.StatusError | 0x2be9a29e6c80>{
ErrStatus: {
TypeMeta: {Kind: "", APIVersion: ""},
ListMeta: {
SelfLink: "",
ResourceVersion: "",
Continue: "",
RemainingItemCount: nil,
ShardInfo: nil,
},
Status: "Failure",
Message: "Unauthorized",
Reason: "Unauthorized",
Details: nil,
Code: 401,
},
},
},
},
<*fmt.wrapError | 0x2be9a21cc080>{
msg: "VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Unauthorized",
err: <*fmt.wrapError | 0x2be9a21cc020>{
msg: "failed to list all APIServices: Unauthorized",
err: <*errors.StatusError | 0x2be9a3120140>{
ErrStatus: {
TypeMeta: {Kind: "", APIVersion: ""},
ListMeta: {
SelfLink: "",
ResourceVersion: "",
Continue: "",
RemainingItemCount: nil,
ShardInfo: nil,
},
Status: "Failure",
Message: "Unauthorized",
Reason: "Unauthorized",
Details: nil,
Code: 401,
},
},
},
},
],
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:140]: failed to verify HCP cluster "cluster-nightly-4-21-hwnzpg" is viable
Unexpected error:
<*errors.joinError | 0x2be9a2dba918>:
VerifyBasicAccess failed: failed to list services: Unauthorized
VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Unauthorized
{
errs: [
<*fmt.wrapError | 0x2be9a2235d00>{
msg: "VerifyBasicAccess failed: failed to list services: Unauthorized",
err: <*fmt.wrapError | 0x2be9a2235cc0>{
msg: "failed to list services: Unauthorized",
err: <*errors.StatusError | 0x2be9a29e6c80>{
ErrStatus: {
TypeMeta: {Kind: "", APIVersion: ""},
ListMeta: {
SelfLink: "",
ResourceVersion: "",
Continue: "",
RemainingItemCount: nil,
ShardInfo: nil,
},
Status: "Failure",
Message: "Unauthorized",
Reason: "Unauthorized",
Details: nil,
Code: 401,
},
},
},
},
<*fmt.wrapError | 0x2be9a21cc080>{
msg: "VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Unauthorized",
err: <*fmt.wrapError | 0x2be9a21cc020>{
msg: "failed to list all APIServices: Unauthorized",
err: <*errors.StatusError | 0x2be9a3120140>{
ErrStatus: {
TypeMeta: {Kind: "", APIVersion: ""},
ListMeta: {
SelfLink: "",
ResourceVersion: "",
Continue: "",
RemainingItemCount: nil,
ShardInfo: nil,
},
Status: "Failure",
Message: "Unauthorized",
Reason: "Unauthorized",
Details: nil,
Code: 401,
},
},
},
},
],
}
occurredContributing tests (2)
Affected runs (2)
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Customer should be able to test admin credentials before cluster ready, then full admin credential lifecycle: Cluster provisioning failedfull failure pattern: Customer should be able to test admin credentials before cluster ready, then full admin credential lifecycle: Cluster provisioning failed Signal: IndeterminateSignal: Indeterminate — seen in 2 prior week(s); active 2 day(s) | e2e | 2 | 2.67%2 of 75 job runs affected | IndeterminateSignal: Indeterminate — seen in 2 prior week(s); active 2 day(s) | Show trend detailsOct 21: 0 · Oct 22: 1 · Oct 23: 0 · Oct 24: 0 · Oct 25: 0 · Oct 26: 0 · Oct 27: 0 · Oct 28: 1 | none | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/e2e/admin_credential_lifecycle.go:158]: Cluster provisioning failed fail [github.com/Azure/ARO-HCP/test/e2e/admin_credential_lifecycle.go:158]: Cluster provisioning failed Contributing tests (1)
Affected runs (2)
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedC...full failure pattern: ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable; hosted cluster degraded: UnavailableReplicas: [ignition-server-proxy, kube-apiserver]; provider Microsoft.RedHatOpenShift Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 2 day(s) | e2e | 2 | 2.67%2 of 75 job runs affected | IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 2 day(s) | Show trend detailsOct 21: 0 · Oct 22: 1 · Oct 23: 0 · Oct 24: 0 · Oct 25: 0 · Oct 26: 0 · Oct 27: 0 · Oct 28: 1 | none | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (2)fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_labels_taints.go:92]: failed to create HCP cluster cluster-np-labels-taints
Unexpected error:
<*fmt.wrapError | 0xd2fa34b2580>:
failed to create HCP cluster cluster-np-labels-taints: failed waiting for cluster="cluster-np-labels-taints" in resourcegroup="rg-np-labels-taints-8rnw5kdb6p2l" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/e56720ec-9cdb-4896-bffe-b107388330fa
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/e56720ec-9cdb-4896-bffe-b107388330fa",
"name": "e56720ec-9cdb-4896-bffe-b107388330fa",
"status": "Failed",
"startTime": "2026-10-28T15:21:26.535304826Z",
"endTime": "2026-10-28T15:50:27.739063015Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: openshift-apiserver, cluster-policy-controller, cluster-image-registry-operator, cluster-autoscaler, hosted-cluster-config-operator, cluster-network-operator, dns-operator, catalog-operator, community-operators-catalog, control-plane-pki-operator, azure-cloud-controller-manager, ingress-operator, kube-controller-manager, cluster-storage-operator, konnectivity-agent, packageserver, kube-scheduler, cluster-version-operator, openshift-controller-manager, openshift-route-controller-manager, cluster-node-tuning-operator, olm-collect-profiles, csi-snapshot-controller-operator, redhat-operators-catalog, olm-operator, redhat-marketplace-catalog, machine-approver, certified-operators-catalog; hosted cluster degraded: UnavailableReplicas: [ignition-server-proxy deployment has 1 unavailable replicas, kube-apiserver deployment has 1 unavailable replicas]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/e56720ec-9cdb-4896-bffe-b107388330fa
correlation ID: 87508886-196b-41dd-b14a-e423d6c69bea
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_labels_taints.go:92]: failed to create HCP cluster cluster-np-labels-taints
Unexpected error:
<*fmt.wrapError | 0xd2fa34b2580>:
failed to create HCP cluster cluster-np-labels-taints: failed waiting for cluster="cluster-np-labels-taints" in resourcegroup="rg-np-labels-taints-8rnw5kdb6p2l" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/e56720ec-9cdb-4896-bffe-b107388330fa
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/e56720ec-9cdb-4896-bffe-b107388330fa",
"name": "e56720ec-9cdb-4896-bffe-b107388330fa",
"status": "Failed",
"startTime": "2026-10-28T15:21:26.535304826Z",
"endTime": "2026-10-28T15:50:27.739063015Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: openshift-apiserver, cluster-policy-controller, cluster-image-registry-operator, cluster-autoscaler, hosted-cluster-config-operator, cluster-network-operator, dns-operator, catalog-operator, community-operators-catalog, control-plane-pki-operator, azure-cloud-controller-manager, ingress-operator, kube-controller-manager, cluster-storage-operator, konnectivity-agent, packageserver, kube-scheduler, cluster-version-operator, openshift-controller-manager, openshift-route-controller-manager, cluster-node-tuning-operator, olm-collect-profiles, csi-snapshot-controller-operator, redhat-operators-catalog, olm-operator, redhat-marketplace-catalog, machine-approver, certified-operators-catalog; hosted cluster degraded: UnavailableReplicas: [ignition-server-proxy deployment has 1 unavailable replicas, kube-apiserver deployment has 1 unavailable replicas]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/e56720ec-9cdb-4896-bffe-b107388330fa
correlation ID: 87508886-196b-41dd-b14a-e423d6c69bea
client request ID: unavailable
occurredfail [github.com/Azure/ARO-HCP/test/e2e/shoebox_logs.go:272]: failed to create HCP cluster shoebox-hcp-cluster
Unexpected error:
<*fmt.wrapError | 0x283eb71121c0>:
failed to create HCP cluster shoebox-hcp-cluster: failed waiting for cluster="shoebox-hcp-cluster" in resourcegroup="shoebox-cluster-wqltj9p8d5tp" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/07e1caae-294a-4885-b57d-e584ce7036c1
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/07e1caae-294a-4885-b57d-e584ce7036c1",
"name": "07e1caae-294a-4885-b57d-e584ce7036c1",
"status": "Failed",
"startTime": "2026-10-22T03:11:14.830364623Z",
"endTime": "2026-10-22T03:40:15.05749374Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: openshift-route-controller-manager, kube-controller-manager, kube-scheduler, openshift-apiserver, cluster-storage-operator, ingress-operator, community-operators-catalog, cluster-version-operator, redhat-marketplace-catalog, certified-operators-catalog, azure-cloud-controller-manager, cluster-network-operator, cluster-image-registry-operator, konnectivity-agent, machine-approver, csi-snapshot-controller-operator, packageserver, cluster-policy-controller, dns-operator, catalog-operator, redhat-operators-catalog, cluster-autoscaler, olm-operator, control-plane-pki-operator, hosted-cluster-config-operator, openshift-controller-manager, cluster-node-tuning-operator, olm-collect-profiles; hosted cluster degraded: UnavailableReplicas: [ignition-server-proxy deployment has 1 unavailable replicas, kube-apiserver deployment has 1 unavailable replicas]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/07e1caae-294a-4885-b57d-e584ce7036c1
correlation ID: 042010a3-104c-478c-8958-4456058069f0
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/shoebox_logs.go:272]: failed to create HCP cluster shoebox-hcp-cluster
Unexpected error:
<*fmt.wrapError | 0x283eb71121c0>:
failed to create HCP cluster shoebox-hcp-cluster: failed waiting for cluster="shoebox-hcp-cluster" in resourcegroup="shoebox-cluster-wqltj9p8d5tp" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/07e1caae-294a-4885-b57d-e584ce7036c1
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/07e1caae-294a-4885-b57d-e584ce7036c1",
"name": "07e1caae-294a-4885-b57d-e584ce7036c1",
"status": "Failed",
"startTime": "2026-10-22T03:11:14.830364623Z",
"endTime": "2026-10-22T03:40:15.05749374Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: openshift-route-controller-manager, kube-controller-manager, kube-scheduler, openshift-apiserver, cluster-storage-operator, ingress-operator, community-operators-catalog, cluster-version-operator, redhat-marketplace-catalog, certified-operators-catalog, azure-cloud-controller-manager, cluster-network-operator, cluster-image-registry-operator, konnectivity-agent, machine-approver, csi-snapshot-controller-operator, packageserver, cluster-policy-controller, dns-operator, catalog-operator, redhat-operators-catalog, cluster-autoscaler, olm-operator, control-plane-pki-operator, hosted-cluster-config-operator, openshift-controller-manager, cluster-node-tuning-operator, olm-collect-profiles; hosted cluster degraded: UnavailableReplicas: [ignition-server-proxy deployment has 1 unavailable replicas, kube-apiserver deployment has 1 unavailable replicas]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/07e1caae-294a-4885-b57d-e584ce7036c1
correlation ID: 042010a3-104c-478c-8958-4456058069f0
client request ID: unavailable
occurredContributing tests (2)
Affected runs (2)
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedC...full failure pattern: ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable; hosted cluster degraded: UnavailableReplicas: [kube-apiserver, router]; provider Microsoft.RedHatOpenShift Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 2 day(s) | e2e | 2 | 2.67%2 of 75 job runs affected | IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 2 day(s) | Show trend detailsOct 21: 0 · Oct 22: 1 · Oct 23: 0 · Oct 24: 0 · Oct 25: 0 · Oct 26: 0 · Oct 27: 0 · Oct 28: 1 | none | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (2)fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_delete.go:81]: failed to create HCP cluster np-delete-hcp-cluster
Unexpected error:
<*fmt.wrapError | 0x33d30568cba0>:
failed to create HCP cluster np-delete-hcp-cluster: failed waiting for cluster="np-delete-hcp-cluster" in resourcegroup="nodepool-delete-rn9mbdwr669k" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/30b1eeed-8a58-491f-a679-903582bd2428
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/30b1eeed-8a58-491f-a679-903582bd2428",
"name": "30b1eeed-8a58-491f-a679-903582bd2428",
"status": "Failed",
"startTime": "2026-10-28T15:21:06.381277659Z",
"endTime": "2026-10-28T15:50:07.62996819Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: openshift-route-controller-manager, catalog-operator, cluster-version-operator, cluster-storage-operator, csi-snapshot-controller-operator, cluster-policy-controller, olm-operator, kube-scheduler, dns-operator, certified-operators-catalog, openshift-controller-manager, cluster-image-registry-operator, olm-collect-profiles, redhat-operators-catalog, openshift-apiserver, hosted-cluster-config-operator, cluster-network-operator, konnectivity-agent, machine-approver, kube-controller-manager, azure-cloud-controller-manager, cluster-node-tuning-operator, ingress-operator, packageserver, community-operators-catalog, cluster-autoscaler, control-plane-pki-operator, redhat-marketplace-catalog; hosted cluster degraded: UnavailableReplicas: [kube-apiserver deployment has 1 unavailable replicas, router deployment has 1 unavailable replicas]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/30b1eeed-8a58-491f-a679-903582bd2428
correlation ID: 375547a6-70a5-4d64-8ca8-71de42a27606
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_delete.go:81]: failed to create HCP cluster np-delete-hcp-cluster
Unexpected error:
<*fmt.wrapError | 0x33d30568cba0>:
failed to create HCP cluster np-delete-hcp-cluster: failed waiting for cluster="np-delete-hcp-cluster" in resourcegroup="nodepool-delete-rn9mbdwr669k" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/30b1eeed-8a58-491f-a679-903582bd2428
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/30b1eeed-8a58-491f-a679-903582bd2428",
"name": "30b1eeed-8a58-491f-a679-903582bd2428",
"status": "Failed",
"startTime": "2026-10-28T15:21:06.381277659Z",
"endTime": "2026-10-28T15:50:07.62996819Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: openshift-route-controller-manager, catalog-operator, cluster-version-operator, cluster-storage-operator, csi-snapshot-controller-operator, cluster-policy-controller, olm-operator, kube-scheduler, dns-operator, certified-operators-catalog, openshift-controller-manager, cluster-image-registry-operator, olm-collect-profiles, redhat-operators-catalog, openshift-apiserver, hosted-cluster-config-operator, cluster-network-operator, konnectivity-agent, machine-approver, kube-controller-manager, azure-cloud-controller-manager, cluster-node-tuning-operator, ingress-operator, packageserver, community-operators-catalog, cluster-autoscaler, control-plane-pki-operator, redhat-marketplace-catalog; hosted cluster degraded: UnavailableReplicas: [kube-apiserver deployment has 1 unavailable replicas, router deployment has 1 unavailable replicas]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/30b1eeed-8a58-491f-a679-903582bd2428
correlation ID: 375547a6-70a5-4d64-8ca8-71de42a27606
client request ID: unavailable
occurredfail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:124]: HCP cluster rg-candidate-4-23-d44bvbtv22rz/cluster-candidate-4-23-f2rpqw should provision
Unexpected error:
<*fmt.wrapError | 0x126c5cc0e60>:
failed to create HCP cluster cluster-candidate-4-23-f2rpqw: failed waiting for cluster="cluster-candidate-4-23-f2rpqw" in resourcegroup="rg-candidate-4-23-d44bvbtv22rz" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/efe747a0-4a74-4970-aae9-c5a7e67a1f5b
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/efe747a0-4a74-4970-aae9-c5a7e67a1f5b",
"name": "efe747a0-4a74-4970-aae9-c5a7e67a1f5b",
"status": "Failed",
"startTime": "2026-10-22T02:21:35.82450299Z",
"endTime": "2026-10-22T02:50:40.503091649Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: openshift-route-controller-manager, catalog-operator, redhat-operators-catalog, machine-approver, openshift-controller-manager, konnectivity-agent, olm-operator, certified-operators-catalog, kube-controller-manager, cluster-version-operator, azure-cloud-controller-manager, kube-storage-version-migrator, community-operators-catalog, openshift-apiserver, cluster-policy-controller, cluster-autoscaler, cluster-network-operator, dns-operator, control-plane-pki-operator, kube-scheduler, hosted-cluster-config-operator, cluster-storage-operator, cluster-node-tuning-operator, csi-snapshot-controller-operator, packageserver, ingress-operator, cluster-image-registry-operator, olm-collect-profiles; hosted cluster degraded: UnavailableReplicas: [kube-apiserver deployment has 1 unavailable replicas, router deployment has 1 unavailable replicas]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/efe747a0-4a74-4970-aae9-c5a7e67a1f5b
correlation ID: 08db8465-69b5-4f99-aabf-f55441aa0ace
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:124]: HCP cluster rg-candidate-4-23-d44bvbtv22rz/cluster-candidate-4-23-f2rpqw should provision
Unexpected error:
<*fmt.wrapError | 0x126c5cc0e60>:
failed to create HCP cluster cluster-candidate-4-23-f2rpqw: failed waiting for cluster="cluster-candidate-4-23-f2rpqw" in resourcegroup="rg-candidate-4-23-d44bvbtv22rz" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/efe747a0-4a74-4970-aae9-c5a7e67a1f5b
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/efe747a0-4a74-4970-aae9-c5a7e67a1f5b",
"name": "efe747a0-4a74-4970-aae9-c5a7e67a1f5b",
"status": "Failed",
"startTime": "2026-10-22T02:21:35.82450299Z",
"endTime": "2026-10-22T02:50:40.503091649Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: openshift-route-controller-manager, catalog-operator, redhat-operators-catalog, machine-approver, openshift-controller-manager, konnectivity-agent, olm-operator, certified-operators-catalog, kube-controller-manager, cluster-version-operator, azure-cloud-controller-manager, kube-storage-version-migrator, community-operators-catalog, openshift-apiserver, cluster-policy-controller, cluster-autoscaler, cluster-network-operator, dns-operator, control-plane-pki-operator, kube-scheduler, hosted-cluster-config-operator, cluster-storage-operator, cluster-node-tuning-operator, csi-snapshot-controller-operator, packageserver, ingress-operator, cluster-image-registry-operator, olm-collect-profiles; hosted cluster degraded: UnavailableReplicas: [kube-apiserver deployment has 1 unavailable replicas, router deployment has 1 unavailable replicas]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/efe747a0-4a74-4970-aae9-c5a7e67a1f5b
correlation ID: 08db8465-69b5-4f99-aabf-f55441aa0ace
client request ID: unavailable
occurredContributing tests (2)
Affected runs (2)
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no ...full failure pattern: ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated; [tlsCertificates] TLS certificates not yet confirmed; [clusterServiceClusterStatus] cluster service is pending; [defaultIngressServingCertificate] default ingress certificate not yet observed; provider Microsoft.RedHatOpenShift Signal: IndeterminateSignal: Indeterminate — no prior history; active 2 day(s) — New failure pattern — no prior history | e2e | 2 | 2.67%2 of 75 job runs affected | IndeterminateSignal: Indeterminate — no prior history; active 2 day(s) — New failure pattern — no prior history | Show trend detailsOct 21: 0 · Oct 22: 0 · Oct 23: 0 · Oct 24: 0 · Oct 25: 0 · Oct 26: 1 · Oct 27: 0 · Oct 28: 1 | none | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (2)fail [github.com/Azure/ARO-HCP/test/e2e/cluster_autoscaling.go:108]: failed to create HCP cluster "autoscaling-hcp-cluster" with custom autoscaling
Unexpected error:
<*fmt.wrapError | 0x12961691c220>:
failed to create HCP cluster autoscaling-hcp-cluster: failed waiting for cluster="autoscaling-hcp-cluster" in resourcegroup="autoscaling-cluster-zjbhtbfkbk4z" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/australiaeast/hcpOperationStatuses/b9e8b50e-a67a-443d-9059-6dace5a0e1b3
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/australiaeast/hcpOperationStatuses/b9e8b50e-a67a-443d-9059-6dace5a0e1b3",
"name": "b9e8b50e-a67a-443d-9059-6dace5a0e1b3",
"status": "Failed",
"startTime": "2026-10-28T02:05:53.148166937Z",
"endTime": "2026-10-28T02:34:59.674611282Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated; [tlsCertificates] TLS certificates not yet confirmed; [clusterServiceClusterStatus] cluster service is pending; [defaultIngressServingCertificate] default ingress certificate not yet observed"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/australiaeast/hcpOperationStatuses/b9e8b50e-a67a-443d-9059-6dace5a0e1b3
correlation ID: 5a9d9ef9-c6e1-45c7-a60d-afc7b67a11c1
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_autoscaling.go:108]: failed to create HCP cluster "autoscaling-hcp-cluster" with custom autoscaling
Unexpected error:
<*fmt.wrapError | 0x12961691c220>:
failed to create HCP cluster autoscaling-hcp-cluster: failed waiting for cluster="autoscaling-hcp-cluster" in resourcegroup="autoscaling-cluster-zjbhtbfkbk4z" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/australiaeast/hcpOperationStatuses/b9e8b50e-a67a-443d-9059-6dace5a0e1b3
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/australiaeast/hcpOperationStatuses/b9e8b50e-a67a-443d-9059-6dace5a0e1b3",
"name": "b9e8b50e-a67a-443d-9059-6dace5a0e1b3",
"status": "Failed",
"startTime": "2026-10-28T02:05:53.148166937Z",
"endTime": "2026-10-28T02:34:59.674611282Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated; [tlsCertificates] TLS certificates not yet confirmed; [clusterServiceClusterStatus] cluster service is pending; [defaultIngressServingCertificate] default ingress certificate not yet observed"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/australiaeast/hcpOperationStatuses/b9e8b50e-a67a-443d-9059-6dace5a0e1b3
correlation ID: 5a9d9ef9-c6e1-45c7-a60d-afc7b67a11c1
client request ID: unavailable
occurredfail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:722]: failed to create HCP cluster
Unexpected error:
<*fmt.wrapError | 0x1138be74e660>:
failed to create HCP cluster np-dg-minor-gsd8rp: failed waiting for cluster="np-dg-minor-gsd8rp" in resourcegroup="rg-np-dg-minor-gsd8rp-mv6pwhz6p5qt" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/australiaeast/hcpOperationStatuses/b44a2fb9-01ac-4953-9693-2e6baa16d65c
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/australiaeast/hcpOperationStatuses/b44a2fb9-01ac-4953-9693-2e6baa16d65c",
"name": "b44a2fb9-01ac-4953-9693-2e6baa16d65c",
"status": "Failed",
"startTime": "2026-10-26T17:26:50.386134218Z",
"endTime": "2026-10-26T17:55:54.373104457Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated; [tlsCertificates] TLS certificates not yet confirmed; [clusterServiceClusterStatus] cluster service is pending; [defaultIngressServingCertificate] default ingress certificate not yet observed"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/australiaeast/hcpOperationStatuses/b44a2fb9-01ac-4953-9693-2e6baa16d65c
correlation ID: f252e869-38a8-48ec-8d30-9db8cb2d096a
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:722]: failed to create HCP cluster
Unexpected error:
<*fmt.wrapError | 0x1138be74e660>:
failed to create HCP cluster np-dg-minor-gsd8rp: failed waiting for cluster="np-dg-minor-gsd8rp" in resourcegroup="rg-np-dg-minor-gsd8rp-mv6pwhz6p5qt" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/australiaeast/hcpOperationStatuses/b44a2fb9-01ac-4953-9693-2e6baa16d65c
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/australiaeast/hcpOperationStatuses/b44a2fb9-01ac-4953-9693-2e6baa16d65c",
"name": "b44a2fb9-01ac-4953-9693-2e6baa16d65c",
"status": "Failed",
"startTime": "2026-10-26T17:26:50.386134218Z",
"endTime": "2026-10-26T17:55:54.373104457Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated; [tlsCertificates] TLS certificates not yet confirmed; [clusterServiceClusterStatus] cluster service is pending; [defaultIngressServingCertificate] default ingress certificate not yet observed"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/australiaeast/hcpOperationStatuses/b44a2fb9-01ac-4953-9693-2e6baa16d65c
correlation ID: f252e869-38a8-48ec-8d30-9db8cb2d096a
client request ID: unavailable
occurredContributing tests (2)
Affected runs (2)
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
cluster operators not available: console (Degraded=True, Progressing=False, Available=False, Upgradeable=True, EvaluationConditionsDetected=Unknown), ingress (Available=False, Pro...full failure pattern: cluster operators not available: console (Degraded=True, Progressing=False, Available=False, Upgradeable=True, EvaluationConditionsDetected=Unknown), ingress (Available=False, Progressing=True, Degraded=True, Upgradeable=True Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 2 day(s) | e2e | 2 | 2.67%2 of 75 job runs affected | IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 2 day(s) | Show trend detailsOct 21: 0 · Oct 22: 0 · Oct 23: 1 · Oct 24: 0 · Oct 25: 1 · Oct 26: 0 · Oct 27: 0 · Oct 28: 0 | none | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (2)fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:255]: failed to verify cluster operators are available after external auth config creation
Unexpected error:
<*errors.joinError | 0x471790ed8f0>:
context deadline exceeded
cluster operators not available: console (Degraded=True, Progressing=False, Available=False, Upgradeable=True, EvaluationConditionsDetected=Unknown), ingress (Available=False, Progressing=True, Degraded=True, Upgradeable=True, EvaluationConditionsDetected=False)
{
errs: [
<context.deadlineExceededError>{},
<*errors.errorString | 0x4717a706200>{
s: "cluster operators not available: console (Degraded=True, Progressing=False, Available=False, Upgradeable=True, EvaluationConditionsDetected=Unknown), ingress (Available=False, Progressing=True, Degraded=True, Upgradeable=True, EvaluationConditionsDetected=False)",
},
],
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:255]: failed to verify cluster operators are available after external auth config creation
Unexpected error:
<*errors.joinError | 0x471790ed8f0>:
context deadline exceeded
cluster operators not available: console (Degraded=True, Progressing=False, Available=False, Upgradeable=True, EvaluationConditionsDetected=Unknown), ingress (Available=False, Progressing=True, Degraded=True, Upgradeable=True, EvaluationConditionsDetected=False)
{
errs: [
<context.deadlineExceededError>{},
<*errors.errorString | 0x4717a706200>{
s: "cluster operators not available: console (Degraded=True, Progressing=False, Available=False, Upgradeable=True, EvaluationConditionsDetected=Unknown), ingress (Available=False, Progressing=True, Degraded=True, Upgradeable=True, EvaluationConditionsDetected=False)",
},
],
}
occurredfail [github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:255]: failed to verify cluster operators are available after external auth config creation
Unexpected error:
<*errors.joinError | 0x397e13438ac8>:
context deadline exceeded
cluster operators not available: console (Degraded=True, Progressing=False, Available=False, Upgradeable=True, EvaluationConditionsDetected=Unknown), ingress (Available=False, Progressing=True, Degraded=True, Upgradeable=True, EvaluationConditionsDetected=False)
{
errs: [
<context.deadlineExceededError>{},
<*errors.errorString | 0x397e134eabf0>{
s: "cluster operators not available: console (Degraded=True, Progressing=False, Available=False, Upgradeable=True, EvaluationConditionsDetected=Unknown), ingress (Available=False, Progressing=True, Degraded=True, Upgradeable=True, EvaluationConditionsDetected=False)",
},
],
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:255]: failed to verify cluster operators are available after external auth config creation
Unexpected error:
<*errors.joinError | 0x397e13438ac8>:
context deadline exceeded
cluster operators not available: console (Degraded=True, Progressing=False, Available=False, Upgradeable=True, EvaluationConditionsDetected=Unknown), ingress (Available=False, Progressing=True, Degraded=True, Upgradeable=True, EvaluationConditionsDetected=False)
{
errs: [
<context.deadlineExceededError>{},
<*errors.errorString | 0x397e134eabf0>{
s: "cluster operators not available: console (Degraded=True, Progressing=False, Available=False, Upgradeable=True, EvaluationConditionsDetected=Unknown), ingress (Available=False, Progressing=True, Degraded=True, Upgradeable=True, EvaluationConditionsDetected=False)",
},
],
}
occurredContributing tests (1)
Affected runs (2)
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
public ingress (console) should be reachable from outside the VNet for private KAS cluster, but got error: Get "<url>": dial tcp: lookup <host> on <dns-server>: no such hostfull failure pattern: public ingress (console) should be reachable from outside the VNet for private KAS cluster, but got error: Get "<url>": dial tcp: lookup <host> on <dns-server>: no such host Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 2 day(s) | e2e | 2 | 2.67%2 of 75 job runs affected | IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 2 day(s) | Show trend detailsOct 21: 0 · Oct 22: 0 · Oct 23: 1 · Oct 24: 0 · Oct 25: 1 · Oct 26: 0 · Oct 27: 0 · Oct 28: 0 | none | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (2)fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kas.go:225]: Timed out after 600.000s.
public ingress should be reachable from outside the VNet despite private KAS
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kas.go:223 with:
public ingress (console) should be reachable from outside the VNet for private KAS cluster, but got error: Get "https://console-openshift-console.apps.aro.private-kas.zjlv.eastus2.aroapp-hcp.io": dial tcp: lookup console-openshift-console.apps.aro.private-kas.zjlv.eastus2.aroapp-hcp.io on 172.30.0.10:53: no such host
Unexpected error:
<*url.Error | 0x1bbed2a5bb90>:
Get "https://console-openshift-console.apps.aro.private-kas.zjlv.eastus2.aroapp-hcp.io": dial tcp: lookup console-openshift-console.apps.aro.private-kas.zjlv.eastus2.aroapp-hcp.io on 172.30.0.10:53: no such host
{
Op: "Get",
URL: "https://console-openshift-console.apps.aro.private-kas.zjlv.eastus2.aroapp-hcp.io",
Err: <*net.OpError | 0x1bbed27f1680>{
Op: "dial",
Net: "tcp",
Source: nil,
Addr: nil,
Err: <*net.DNSError | 0x1bbed27f1450>{
UnwrapErr: nil,
Err: "no such host",
Name: "console-openshift-console.apps.aro.private-kas.zjlv.eastus2.aroapp-hcp.io",
Server: "172.30.0.10:53",
IsTimeout: false,
IsTemporary: false,
IsNotFound: true,
},
},
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kas.go:225]: Timed out after 600.000s.
public ingress should be reachable from outside the VNet despite private KAS
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kas.go:223 with:
public ingress (console) should be reachable from outside the VNet for private KAS cluster, but got error: Get "https://console-openshift-console.apps.aro.private-kas.zjlv.eastus2.aroapp-hcp.io": dial tcp: lookup console-openshift-console.apps.aro.private-kas.zjlv.eastus2.aroapp-hcp.io on 172.30.0.10:53: no such host
Unexpected error:
<*url.Error | 0x1bbed2a5bb90>:
Get "https://console-openshift-console.apps.aro.private-kas.zjlv.eastus2.aroapp-hcp.io": dial tcp: lookup console-openshift-console.apps.aro.private-kas.zjlv.eastus2.aroapp-hcp.io on 172.30.0.10:53: no such host
{
Op: "Get",
URL: "https://console-openshift-console.apps.aro.private-kas.zjlv.eastus2.aroapp-hcp.io",
Err: <*net.OpError | 0x1bbed27f1680>{
Op: "dial",
Net: "tcp",
Source: nil,
Addr: nil,
Err: <*net.DNSError | 0x1bbed27f1450>{
UnwrapErr: nil,
Err: "no such host",
Name: "console-openshift-console.apps.aro.private-kas.zjlv.eastus2.aroapp-hcp.io",
Server: "172.30.0.10:53",
IsTimeout: false,
IsTemporary: false,
IsNotFound: true,
},
},
}
occurredfail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kas.go:225]: Timed out after 600.000s.
public ingress should be reachable from outside the VNet despite private KAS
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kas.go:223 with:
public ingress (console) should be reachable from outside the VNet for private KAS cluster, but got error: Get "https://console-openshift-console.apps.aro.private-kas.w1c2.eastus2.aroapp-hcp.io": dial tcp: lookup console-openshift-console.apps.aro.private-kas.w1c2.eastus2.aroapp-hcp.io on 172.30.0.10:53: no such host
Unexpected error:
<*url.Error | 0x20ec8dc8cd20>:
Get "https://console-openshift-console.apps.aro.private-kas.w1c2.eastus2.aroapp-hcp.io": dial tcp: lookup console-openshift-console.apps.aro.private-kas.w1c2.eastus2.aroapp-hcp.io on 172.30.0.10:53: no such host
{
Op: "Get",
URL: "https://console-openshift-console.apps.aro.private-kas.w1c2.eastus2.aroapp-hcp.io",
Err: <*net.OpError | 0x20ec8bc10690>{
Op: "dial",
Net: "tcp",
Source: nil,
Addr: nil,
Err: <*net.DNSError | 0x20ec8bc103c0>{
UnwrapErr: nil,
Err: "no such host",
Name: "console-openshift-console.apps.aro.private-kas.w1c2.eastus2.aroapp-hcp.io",
Server: "172.30.0.10:53",
IsTimeout: false,
IsTemporary: false,
IsNotFound: true,
},
},
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kas.go:225]: Timed out after 600.000s.
public ingress should be reachable from outside the VNet despite private KAS
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kas.go:223 with:
public ingress (console) should be reachable from outside the VNet for private KAS cluster, but got error: Get "https://console-openshift-console.apps.aro.private-kas.w1c2.eastus2.aroapp-hcp.io": dial tcp: lookup console-openshift-console.apps.aro.private-kas.w1c2.eastus2.aroapp-hcp.io on 172.30.0.10:53: no such host
Unexpected error:
<*url.Error | 0x20ec8dc8cd20>:
Get "https://console-openshift-console.apps.aro.private-kas.w1c2.eastus2.aroapp-hcp.io": dial tcp: lookup console-openshift-console.apps.aro.private-kas.w1c2.eastus2.aroapp-hcp.io on 172.30.0.10:53: no such host
{
Op: "Get",
URL: "https://console-openshift-console.apps.aro.private-kas.w1c2.eastus2.aroapp-hcp.io",
Err: <*net.OpError | 0x20ec8bc10690>{
Op: "dial",
Net: "tcp",
Source: nil,
Addr: nil,
Err: <*net.DNSError | 0x20ec8bc103c0>{
UnwrapErr: nil,
Err: "no such host",
Name: "console-openshift-console.apps.aro.private-kas.w1c2.eastus2.aroapp-hcp.io",
Server: "172.30.0.10:53",
IsTimeout: false,
IsTemporary: false,
IsNotFound: true,
},
},
}
occurredContributing tests (1)
Affected runs (2)
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
<Azure error internals omitted>full failure pattern: <Azure error internals omitted> Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s) | e2e | 1 | 1.33%1 of 75 job runs affected | IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s) | Show trend detailsOct 21: 0 · Oct 22: 7 · Oct 23: 0 · Oct 24: 0 · Oct 25: 0 · Oct 26: 0 · Oct 27: 0 · Oct 28: 0 | none | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (3)fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:310]: Unexpected error:
<*errors.joinError | 0x2a26cf54c078>:
resource group "np-autoscaling-ncn4v65w96fx": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="np-autoscale-cluster" in resourcegroup="np-autoscaling-ncn4v65w96fx" to finish deleting: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/d43b55c4-7331-4381-b4fd-71f64a776433
--------------------------------------------------------------------------------
RESPONSE 502: 502 Bad Gateway
ERROR CODE UNAVAILABLE
--------------------------------------------------------------------------------
Response contained no body
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/d43b55c4-7331-4381-b4fd-71f64a776433
correlation ID: 1d901265-7895-4b99-bf69-cdc848b0d0b5
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:310]: Unexpected error:
<*errors.joinError | 0x2a26cf54c078>:
resource group "np-autoscaling-ncn4v65w96fx": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="np-autoscale-cluster" in resourcegroup="np-autoscaling-ncn4v65w96fx" to finish deleting: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/d43b55c4-7331-4381-b4fd-71f64a776433
--------------------------------------------------------------------------------
RESPONSE 502: 502 Bad Gateway
ERROR CODE UNAVAILABLE
--------------------------------------------------------------------------------
Response contained no body
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/d43b55c4-7331-4381-b4fd-71f64a776433
correlation ID: 1d901265-7895-4b99-bf69-cdc848b0d0b5
client request ID: unavailable
occurredfail [github.com/Azure/ARO-HCP/test/e2e/shoebox_logs.go:272]: failed to create HCP cluster shoebox-hcp-cluster
Unexpected error:
<*fmt.wrapError | 0x1745c0262280>:
failed to create HCP cluster shoebox-hcp-cluster: failed waiting for cluster="shoebox-hcp-cluster" in resourcegroup="shoebox-cluster-ddrtzn7r9n6f" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/aa390de3-b57c-4004-ae00-8d256a07e3fb
--------------------------------------------------------------------------------
RESPONSE 502: 502 Bad Gateway
ERROR CODE UNAVAILABLE
--------------------------------------------------------------------------------
Response contained no body
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/aa390de3-b57c-4004-ae00-8d256a07e3fb
correlation ID: 23755ba0-9dc6-42ab-bdd9-3208cee1312c
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/shoebox_logs.go:272]: failed to create HCP cluster shoebox-hcp-cluster
Unexpected error:
<*fmt.wrapError | 0x1745c0262280>:
failed to create HCP cluster shoebox-hcp-cluster: failed waiting for cluster="shoebox-hcp-cluster" in resourcegroup="shoebox-cluster-ddrtzn7r9n6f" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/aa390de3-b57c-4004-ae00-8d256a07e3fb
--------------------------------------------------------------------------------
RESPONSE 502: 502 Bad Gateway
ERROR CODE UNAVAILABLE
--------------------------------------------------------------------------------
Response contained no body
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/aa390de3-b57c-4004-ae00-8d256a07e3fb
correlation ID: 23755ba0-9dc6-42ab-bdd9-3208cee1312c
client request ID: unavailable
occurredfail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:175]: failed to get admin REST config for cluster np-version-upgrade-cluster-6klfkt
Unexpected error:
<*fmt.wrapError | 0x31e04d7a70c0>:
failed waiting for hcpCluster="np-version-upgrade-cluster-6klfkt" in resourcegroup="rg-np-version-upgrade-6klfkt-x72h4v22zqrn" to finish getting creds: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/0b189d0f-2181-4aed-9e7f-51e3cb6c01ee
--------------------------------------------------------------------------------
RESPONSE 502: 502 Bad Gateway
ERROR CODE UNAVAILABLE
--------------------------------------------------------------------------------
Response contained no body
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/0b189d0f-2181-4aed-9e7f-51e3cb6c01ee
correlation ID: acc4317f-332d-41ad-92b6-3b3da8362c30
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:175]: failed to get admin REST config for cluster np-version-upgrade-cluster-6klfkt
Unexpected error:
<*fmt.wrapError | 0x31e04d7a70c0>:
failed waiting for hcpCluster="np-version-upgrade-cluster-6klfkt" in resourcegroup="rg-np-version-upgrade-6klfkt-x72h4v22zqrn" to finish getting creds: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/0b189d0f-2181-4aed-9e7f-51e3cb6c01ee
--------------------------------------------------------------------------------
RESPONSE 502: 502 Bad Gateway
ERROR CODE UNAVAILABLE
--------------------------------------------------------------------------------
Response contained no body
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/0b189d0f-2181-4aed-9e7f-51e3cb6c01ee
correlation ID: acc4317f-332d-41ad-92b6-3b3da8362c30
client request ID: unavailable
occurredContributing tests (7)
Affected runs (1)
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
failed waiting for nodepool="<nodepool>" for cluster "<cluster>" in resourcegroup="<resource-group>" to finish creating: GET <url>full failure pattern: failed waiting for nodepool="<nodepool>" for cluster "<cluster>" in resourcegroup="<resource-group>" to finish creating: GET <url> Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s) | e2e | 1 | 1.33%1 of 75 job runs affected | IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s) | Show trend detailsOct 21: 0 · Oct 22: 7 · Oct 23: 0 · Oct 24: 0 · Oct 25: 0 · Oct 26: 0 · Oct 27: 0 · Oct 28: 0 | none | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (3)fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:461]: failed to create nodepool nps-420-422 at version 4.20.43
Unexpected error:
<*fmt.wrapError | 0x378ab2d1b4e0>:
failed to create NodePool nps-420-422: failed waiting for nodepool="nps-420-422" for cluster "np-skip-minor-p9x242" in resourcegroup="rg-np-skip-minor-p9x242-tqcdq45mhp7v" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/773f88ac-95e8-4da7-bc09-57a818deb175
--------------------------------------------------------------------------------
RESPONSE 502: 502 Bad Gateway
ERROR CODE UNAVAILABLE
--------------------------------------------------------------------------------
Response contained no body
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/773f88ac-95e8-4da7-bc09-57a818deb175
correlation ID: 8aa44ab4-58c2-423f-9a40-2f66225c45a7
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:461]: failed to create nodepool nps-420-422 at version 4.20.43
Unexpected error:
<*fmt.wrapError | 0x378ab2d1b4e0>:
failed to create NodePool nps-420-422: failed waiting for nodepool="nps-420-422" for cluster "np-skip-minor-p9x242" in resourcegroup="rg-np-skip-minor-p9x242-tqcdq45mhp7v" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/773f88ac-95e8-4da7-bc09-57a818deb175
--------------------------------------------------------------------------------
RESPONSE 502: 502 Bad Gateway
ERROR CODE UNAVAILABLE
--------------------------------------------------------------------------------
Response contained no body
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/773f88ac-95e8-4da7-bc09-57a818deb175
correlation ID: 8aa44ab4-58c2-423f-9a40-2f66225c45a7
client request ID: unavailable
occurredfail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:321]: failed to create nodepool npnoedge-4-20 at version 4.20.42
Unexpected error:
<*fmt.wrapError | 0x8e881283980>:
failed to create NodePool npnoedge-4-20: failed waiting for nodepool="npnoedge-4-20" for cluster "np-noedge-8tlm65" in resourcegroup="rg-np-noedge-8tlm65-6rtrf6fxs8pp" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/657d92b1-0420-465c-a20f-051929954d68
--------------------------------------------------------------------------------
RESPONSE 502: 502 Bad Gateway
ERROR CODE UNAVAILABLE
--------------------------------------------------------------------------------
Response contained no body
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/657d92b1-0420-465c-a20f-051929954d68
correlation ID: da25ea79-7ee4-4aa5-a3cd-24d74907d5d3
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:321]: failed to create nodepool npnoedge-4-20 at version 4.20.42
Unexpected error:
<*fmt.wrapError | 0x8e881283980>:
failed to create NodePool npnoedge-4-20: failed waiting for nodepool="npnoedge-4-20" for cluster "np-noedge-8tlm65" in resourcegroup="rg-np-noedge-8tlm65-6rtrf6fxs8pp" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/657d92b1-0420-465c-a20f-051929954d68
--------------------------------------------------------------------------------
RESPONSE 502: 502 Bad Gateway
ERROR CODE UNAVAILABLE
--------------------------------------------------------------------------------
Response contained no body
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/657d92b1-0420-465c-a20f-051929954d68
correlation ID: da25ea79-7ee4-4aa5-a3cd-24d74907d5d3
client request ID: unavailable
occurredfail [github.com/Azure/ARO-HCP/test/e2e/oidc_issuer_workload_identity.go:270]: failed to create node pool oidc-wi-np
Unexpected error:
<*fmt.wrapError | 0x11adfdef64a0>:
failed to create NodePool oidc-wi-np: failed waiting for nodepool="oidc-wi-np" for cluster "oidc-wi-cluster" in resourcegroup="oidc-wi-nm2jdjsx4m9t" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/7ab31f32-e5a9-4966-9e78-a32d1786c989
--------------------------------------------------------------------------------
RESPONSE 502: 502 Bad Gateway
ERROR CODE UNAVAILABLE
--------------------------------------------------------------------------------
Response contained no body
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/7ab31f32-e5a9-4966-9e78-a32d1786c989
correlation ID: f811a401-7f9a-42bd-861f-59e24ffc103f
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/oidc_issuer_workload_identity.go:270]: failed to create node pool oidc-wi-np
Unexpected error:
<*fmt.wrapError | 0x11adfdef64a0>:
failed to create NodePool oidc-wi-np: failed waiting for nodepool="oidc-wi-np" for cluster "oidc-wi-cluster" in resourcegroup="oidc-wi-nm2jdjsx4m9t" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/7ab31f32-e5a9-4966-9e78-a32d1786c989
--------------------------------------------------------------------------------
RESPONSE 502: 502 Bad Gateway
ERROR CODE UNAVAILABLE
--------------------------------------------------------------------------------
Response contained no body
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/7ab31f32-e5a9-4966-9e78-a32d1786c989
correlation ID: f811a401-7f9a-42bd-861f-59e24ffc103f
client request ID: unavailable
occurredContributing tests (7)
Affected runs (1)
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedC...full failure pattern: ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable; hosted cluster degraded: UnavailableReplicas: [catalog-operator, hosted-cluster-config-operator, konnectivity-agent, kube-controller-manager, kube-scheduler, olm-operator, openshift-apiserver, packageserver]; provider Microsoft.RedHatOpenShift Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s) | e2e | 1 | 1.33%1 of 75 job runs affected | IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s) | Show trend detailsOct 21: 0 · Oct 22: 3 · Oct 23: 0 · Oct 24: 0 · Oct 25: 0 · Oct 26: 0 · Oct 27: 0 · Oct 28: 0 | none | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (3)fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:133]: failed to create HCP cluster "cp-ystream-upgrade-4-22-pp2v6j" at install version 4.21
Unexpected error:
<*fmt.wrapError | 0x9f443c1ebc0>:
failed to create HCP cluster cp-ystream-upgrade-4-22-pp2v6j: failed waiting for cluster="cp-ystream-upgrade-4-22-pp2v6j" in resourcegroup="rg-cp-ystream-upgrade-4-22-4brmx258wcvt" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/c465a539-e0e2-4a15-96b0-b991fee60416
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/c465a539-e0e2-4a15-96b0-b991fee60416",
"name": "c465a539-e0e2-4a15-96b0-b991fee60416",
"status": "Failed",
"startTime": "2026-10-22T02:23:40.34684753Z",
"endTime": "2026-10-22T02:52:40.362346793Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: csi-snapshot-controller-operator, cluster-storage-operator, cluster-policy-controller, packageserver, hosted-cluster-config-operator, cluster-image-registry-operator, kube-controller-manager, cluster-version-operator, community-operators-catalog, openshift-controller-manager, cluster-node-tuning-operator, redhat-operators-catalog, redhat-marketplace-catalog, certified-operators-catalog, openshift-route-controller-manager, kube-scheduler, cluster-network-operator, dns-operator, konnectivity-agent, olm-operator, ingress-operator, catalog-operator; hosted cluster degraded: UnavailableReplicas: [catalog-operator deployment has 1 unavailable replicas, hosted-cluster-config-operator deployment has 1 unavailable replicas, konnectivity-agent deployment has 1 unavailable replicas, kube-controller-manager deployment has 1 unavailable replicas, kube-scheduler deployment has 1 unavailable replicas, olm-operator deployment has 1 unavailable replicas, openshift-apiserver deployment has 1 unavailable replicas, packageserver deployment has 3 unavailable replicas]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/c465a539-e0e2-4a15-96b0-b991fee60416
correlation ID: 81f12348-0a31-441f-af5f-8f18eef28715
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:133]: failed to create HCP cluster "cp-ystream-upgrade-4-22-pp2v6j" at install version 4.21
Unexpected error:
<*fmt.wrapError | 0x9f443c1ebc0>:
failed to create HCP cluster cp-ystream-upgrade-4-22-pp2v6j: failed waiting for cluster="cp-ystream-upgrade-4-22-pp2v6j" in resourcegroup="rg-cp-ystream-upgrade-4-22-4brmx258wcvt" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/c465a539-e0e2-4a15-96b0-b991fee60416
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/c465a539-e0e2-4a15-96b0-b991fee60416",
"name": "c465a539-e0e2-4a15-96b0-b991fee60416",
"status": "Failed",
"startTime": "2026-10-22T02:23:40.34684753Z",
"endTime": "2026-10-22T02:52:40.362346793Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: csi-snapshot-controller-operator, cluster-storage-operator, cluster-policy-controller, packageserver, hosted-cluster-config-operator, cluster-image-registry-operator, kube-controller-manager, cluster-version-operator, community-operators-catalog, openshift-controller-manager, cluster-node-tuning-operator, redhat-operators-catalog, redhat-marketplace-catalog, certified-operators-catalog, openshift-route-controller-manager, kube-scheduler, cluster-network-operator, dns-operator, konnectivity-agent, olm-operator, ingress-operator, catalog-operator; hosted cluster degraded: UnavailableReplicas: [catalog-operator deployment has 1 unavailable replicas, hosted-cluster-config-operator deployment has 1 unavailable replicas, konnectivity-agent deployment has 1 unavailable replicas, kube-controller-manager deployment has 1 unavailable replicas, kube-scheduler deployment has 1 unavailable replicas, olm-operator deployment has 1 unavailable replicas, openshift-apiserver deployment has 1 unavailable replicas, packageserver deployment has 3 unavailable replicas]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/c465a539-e0e2-4a15-96b0-b991fee60416
correlation ID: 81f12348-0a31-441f-af5f-8f18eef28715
client request ID: unavailable
occurredfail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:146]: failed to create HCP cluster np-version-upgrade-cluster-vwsld8 with version 4.20.43
Unexpected error:
<*fmt.wrapError | 0xc02210a0420>:
failed to create HCP cluster np-version-upgrade-cluster-vwsld8: failed waiting for cluster="np-version-upgrade-cluster-vwsld8" in resourcegroup="rg-np-version-upgrade-vwsld8-b78v68n9cqj2" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/7061ec27-2552-401b-a8b5-5bcf7cc27448
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/7061ec27-2552-401b-a8b5-5bcf7cc27448",
"name": "7061ec27-2552-401b-a8b5-5bcf7cc27448",
"status": "Failed",
"startTime": "2026-10-22T03:10:01.822651936Z",
"endTime": "2026-10-22T03:39:05.079618983Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: cluster-version-operator, hosted-cluster-config-operator, cluster-image-registry-operator, catalog-operator, cluster-storage-operator, csi-snapshot-controller-operator, community-operators-catalog, cluster-node-tuning-operator, ingress-operator, redhat-marketplace-catalog, openshift-controller-manager, packageserver, cluster-policy-controller, dns-operator, olm-operator, cluster-network-operator, redhat-operators-catalog, certified-operators-catalog, openshift-route-controller-manager, konnectivity-agent, kube-controller-manager; hosted cluster degraded: UnavailableReplicas: [catalog-operator deployment has 1 unavailable replicas, hosted-cluster-config-operator deployment has 1 unavailable replicas, konnectivity-agent deployment has 1 unavailable replicas, kube-controller-manager deployment has 1 unavailable replicas, kube-scheduler deployment has 1 unavailable replicas, olm-operator deployment has 1 unavailable replicas, openshift-apiserver deployment has 1 unavailable replicas, packageserver deployment has 3 unavailable replicas]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/7061ec27-2552-401b-a8b5-5bcf7cc27448
correlation ID: 8d4a80f0-34d4-471d-b30b-2c4ee315636d
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:146]: failed to create HCP cluster np-version-upgrade-cluster-vwsld8 with version 4.20.43
Unexpected error:
<*fmt.wrapError | 0xc02210a0420>:
failed to create HCP cluster np-version-upgrade-cluster-vwsld8: failed waiting for cluster="np-version-upgrade-cluster-vwsld8" in resourcegroup="rg-np-version-upgrade-vwsld8-b78v68n9cqj2" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/7061ec27-2552-401b-a8b5-5bcf7cc27448
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/7061ec27-2552-401b-a8b5-5bcf7cc27448",
"name": "7061ec27-2552-401b-a8b5-5bcf7cc27448",
"status": "Failed",
"startTime": "2026-10-22T03:10:01.822651936Z",
"endTime": "2026-10-22T03:39:05.079618983Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: cluster-version-operator, hosted-cluster-config-operator, cluster-image-registry-operator, catalog-operator, cluster-storage-operator, csi-snapshot-controller-operator, community-operators-catalog, cluster-node-tuning-operator, ingress-operator, redhat-marketplace-catalog, openshift-controller-manager, packageserver, cluster-policy-controller, dns-operator, olm-operator, cluster-network-operator, redhat-operators-catalog, certified-operators-catalog, openshift-route-controller-manager, konnectivity-agent, kube-controller-manager; hosted cluster degraded: UnavailableReplicas: [catalog-operator deployment has 1 unavailable replicas, hosted-cluster-config-operator deployment has 1 unavailable replicas, konnectivity-agent deployment has 1 unavailable replicas, kube-controller-manager deployment has 1 unavailable replicas, kube-scheduler deployment has 1 unavailable replicas, olm-operator deployment has 1 unavailable replicas, openshift-apiserver deployment has 1 unavailable replicas, packageserver deployment has 3 unavailable replicas]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/7061ec27-2552-401b-a8b5-5bcf7cc27448
correlation ID: 8d4a80f0-34d4-471d-b30b-2c4ee315636d
client request ID: unavailable
occurredfail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:146]: failed to create HCP cluster np-version-upgrade-cluster-hlfqz4 with version 4.21.39
Unexpected error:
<*fmt.wrapError | 0x26fe4f09e7e0>:
failed to create HCP cluster np-version-upgrade-cluster-hlfqz4: failed waiting for cluster="np-version-upgrade-cluster-hlfqz4" in resourcegroup="rg-np-version-upgrade-hlfqz4-hxtt44szpfn8" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/9b1bdec9-cda2-4c03-8af1-a62744ae5bac
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/9b1bdec9-cda2-4c03-8af1-a62744ae5bac",
"name": "9b1bdec9-cda2-4c03-8af1-a62744ae5bac",
"status": "Failed",
"startTime": "2026-10-22T03:09:57.381547287Z",
"endTime": "2026-10-22T03:39:05.056394482Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: konnectivity-agent, redhat-operators-catalog, catalog-operator, cluster-version-operator, cluster-network-operator, dns-operator, csi-snapshot-controller-operator, redhat-marketplace-catalog, cluster-policy-controller, cluster-storage-operator, openshift-route-controller-manager, kube-controller-manager, ingress-operator, packageserver, olm-operator, kube-scheduler, hosted-cluster-config-operator, openshift-controller-manager, cluster-node-tuning-operator, cluster-image-registry-operator, community-operators-catalog, certified-operators-catalog; hosted cluster degraded: UnavailableReplicas: [catalog-operator deployment has 1 unavailable replicas, hosted-cluster-config-operator deployment has 1 unavailable replicas, konnectivity-agent deployment has 1 unavailable replicas, kube-controller-manager deployment has 1 unavailable replicas, kube-scheduler deployment has 1 unavailable replicas, olm-operator deployment has 1 unavailable replicas, openshift-apiserver deployment has 1 unavailable replicas, packageserver deployment has 3 unavailable replicas]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/9b1bdec9-cda2-4c03-8af1-a62744ae5bac
correlation ID: ae3669ea-d6b2-47bf-90e6-616390cf31a2
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:146]: failed to create HCP cluster np-version-upgrade-cluster-hlfqz4 with version 4.21.39
Unexpected error:
<*fmt.wrapError | 0x26fe4f09e7e0>:
failed to create HCP cluster np-version-upgrade-cluster-hlfqz4: failed waiting for cluster="np-version-upgrade-cluster-hlfqz4" in resourcegroup="rg-np-version-upgrade-hlfqz4-hxtt44szpfn8" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/9b1bdec9-cda2-4c03-8af1-a62744ae5bac
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/9b1bdec9-cda2-4c03-8af1-a62744ae5bac",
"name": "9b1bdec9-cda2-4c03-8af1-a62744ae5bac",
"status": "Failed",
"startTime": "2026-10-22T03:09:57.381547287Z",
"endTime": "2026-10-22T03:39:05.056394482Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: konnectivity-agent, redhat-operators-catalog, catalog-operator, cluster-version-operator, cluster-network-operator, dns-operator, csi-snapshot-controller-operator, redhat-marketplace-catalog, cluster-policy-controller, cluster-storage-operator, openshift-route-controller-manager, kube-controller-manager, ingress-operator, packageserver, olm-operator, kube-scheduler, hosted-cluster-config-operator, openshift-controller-manager, cluster-node-tuning-operator, cluster-image-registry-operator, community-operators-catalog, certified-operators-catalog; hosted cluster degraded: UnavailableReplicas: [catalog-operator deployment has 1 unavailable replicas, hosted-cluster-config-operator deployment has 1 unavailable replicas, konnectivity-agent deployment has 1 unavailable replicas, kube-controller-manager deployment has 1 unavailable replicas, kube-scheduler deployment has 1 unavailable replicas, olm-operator deployment has 1 unavailable replicas, openshift-apiserver deployment has 1 unavailable replicas, packageserver deployment has 3 unavailable replicas]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/9b1bdec9-cda2-4c03-8af1-a62744ae5bac
correlation ID: ae3669ea-d6b2-47bf-90e6-616390cf31a2
client request ID: unavailable
occurredContributing tests (3)
Affected runs (1)
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedC...full failure pattern: ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider, ignition-server-proxy, router]; provider Microsoft.RedHatOpenShift Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s) | e2e | 1 | 1.33%1 of 75 job runs affected | IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s) | Show trend detailsOct 21: 0 · Oct 22: 2 · Oct 23: 0 · Oct 24: 0 · Oct 25: 0 · Oct 26: 0 · Oct 27: 0 · Oct 28: 0 | none | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (2)fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:124]: HCP cluster rg-candidate-4-21-76q9tvzg8rn9/cluster-candidate-4-21-5lg29k should provision
Unexpected error:
<*fmt.wrapError | 0x143589e84400>:
failed to create HCP cluster cluster-candidate-4-21-5lg29k: failed waiting for cluster="cluster-candidate-4-21-5lg29k" in resourcegroup="rg-candidate-4-21-76q9tvzg8rn9" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/edb6a2fa-979a-4af5-a995-cd3897780e01
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/edb6a2fa-979a-4af5-a995-cd3897780e01",
"name": "edb6a2fa-979a-4af5-a995-cd3897780e01",
"status": "Failed",
"startTime": "2026-10-22T02:21:35.877670552Z",
"endTime": "2026-10-22T02:50:40.514192951Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider deployment has 1 unavailable replicas, ignition-server-proxy deployment has 1 unavailable replicas, router deployment has 1 unavailable replicas]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/edb6a2fa-979a-4af5-a995-cd3897780e01
correlation ID: 85af8528-9942-40bd-9e4c-05e55063f9c4
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:124]: HCP cluster rg-candidate-4-21-76q9tvzg8rn9/cluster-candidate-4-21-5lg29k should provision
Unexpected error:
<*fmt.wrapError | 0x143589e84400>:
failed to create HCP cluster cluster-candidate-4-21-5lg29k: failed waiting for cluster="cluster-candidate-4-21-5lg29k" in resourcegroup="rg-candidate-4-21-76q9tvzg8rn9" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/edb6a2fa-979a-4af5-a995-cd3897780e01
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/edb6a2fa-979a-4af5-a995-cd3897780e01",
"name": "edb6a2fa-979a-4af5-a995-cd3897780e01",
"status": "Failed",
"startTime": "2026-10-22T02:21:35.877670552Z",
"endTime": "2026-10-22T02:50:40.514192951Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider deployment has 1 unavailable replicas, ignition-server-proxy deployment has 1 unavailable replicas, router deployment has 1 unavailable replicas]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/edb6a2fa-979a-4af5-a995-cd3897780e01
correlation ID: 85af8528-9942-40bd-9e4c-05e55063f9c4
client request ID: unavailable
occurredfail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:124]: HCP cluster rg-candidate-4-22-gp6dgjrbtz82/cluster-candidate-4-22-qlh9hc should provision
Unexpected error:
<*fmt.wrapError | 0x1de55d1398a0>:
failed to create HCP cluster cluster-candidate-4-22-qlh9hc: failed waiting for cluster="cluster-candidate-4-22-qlh9hc" in resourcegroup="rg-candidate-4-22-gp6dgjrbtz82" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/8cc15b2a-8360-41a2-8a2a-f0798be9df0d
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/8cc15b2a-8360-41a2-8a2a-f0798be9df0d",
"name": "8cc15b2a-8360-41a2-8a2a-f0798be9df0d",
"status": "Failed",
"startTime": "2026-10-22T02:21:54.045486495Z",
"endTime": "2026-10-22T02:51:00.49750165Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider deployment has 1 unavailable replicas, ignition-server-proxy deployment has 1 unavailable replicas, router deployment has 1 unavailable replicas]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/8cc15b2a-8360-41a2-8a2a-f0798be9df0d
correlation ID: 1eeaec8e-068a-41c6-8cee-4c44e2977d7a
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:124]: HCP cluster rg-candidate-4-22-gp6dgjrbtz82/cluster-candidate-4-22-qlh9hc should provision
Unexpected error:
<*fmt.wrapError | 0x1de55d1398a0>:
failed to create HCP cluster cluster-candidate-4-22-qlh9hc: failed waiting for cluster="cluster-candidate-4-22-qlh9hc" in resourcegroup="rg-candidate-4-22-gp6dgjrbtz82" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/8cc15b2a-8360-41a2-8a2a-f0798be9df0d
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/8cc15b2a-8360-41a2-8a2a-f0798be9df0d",
"name": "8cc15b2a-8360-41a2-8a2a-f0798be9df0d",
"status": "Failed",
"startTime": "2026-10-22T02:21:54.045486495Z",
"endTime": "2026-10-22T02:51:00.49750165Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider deployment has 1 unavailable replicas, ignition-server-proxy deployment has 1 unavailable replicas, router deployment has 1 unavailable replicas]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/8cc15b2a-8360-41a2-8a2a-f0798be9df0d
correlation ID: 1eeaec8e-068a-41c6-8cee-4c44e2977d7a
client request ID: unavailable
occurredContributing tests (2)
Affected runs (1)
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
ERROR CODE: DeadlineExceeded; detail message node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hype...full failure pattern: ERROR CODE: DeadlineExceeded; detail message node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is <count>, want <count>; node pool condition AllNodesHealthy is False: <count> of <count> machines are not healthy Machine <node>: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.Compute/virtualMachines/<node> to exist; node pool condition AllMachinesReady is False: <count> of <count> machines are not ready Machine <node>: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.Compute/virtualMachines/<node> to exist; provider Microsoft.RedHatOpenShift Signal: IndeterminateSignal: Indeterminate — seen in 2 prior week(s); active 2 day(s) | e2e | 1 | 1.33%1 of 75 job runs affected | IndeterminateSignal: Indeterminate — seen in 2 prior week(s); active 2 day(s) | Show trend detailsOct 21: 1 · Oct 22: 0 · Oct 23: 0 · Oct 24: 0 · Oct 25: 0 · Oct 26: 0 · Oct 27: 1 · Oct 28: 0 | DEV | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/e2e/arm64_nodepool.go:109]: failed to create ARM64 nodepool "arm64-vm-np-1" with VM size "Standard_D2plds_v6"
Unexpected error:
<*fmt.wrapError | 0x161cee87c740>:
failed to create NodePool arm64-vm-np-1: failed waiting for nodepool="arm64-vm-np-1" for cluster "arm64-vm-hcp-cluster" in resourcegroup="arm64-vm-cluster-9tq8tw8pkh5c" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/aa729d08-abec-481d-a5d5-3a517e315df9
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/aa729d08-abec-481d-a5d5-3a517e315df9",
"name": "aa729d08-abec-481d-a5d5-3a517e315df9",
"status": "Failed",
"startTime": "2026-10-27T18:45:35.525302675Z",
"endTime": "2026-10-27T19:04:37.944268372Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine q5i4q4e4c0d2d0d-arm64-vm-np-1-qfjp4-9wwh2: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/arm64-vm-cluster-9tq8tw8pkh5c--managed/providers/Microsoft.Compute/virtualMachines/q5i4q4e4c0d2d0d-arm64-vm-np-1-qfjp4-9wwh2 to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine q5i4q4e4c0d2d0d-arm64-vm-np-1-qfjp4-9wwh2: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/arm64-vm-cluster-9tq8tw8pkh5c--managed/providers/Microsoft.Compute/virtualMachines/q5i4q4e4c0d2d0d-arm64-vm-np-1-qfjp4-9wwh2 to exist\n"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/aa729d08-abec-481d-a5d5-3a517e315df9
correlation ID: bbfa9866-a276-4fbe-b94b-bc8519188ea5
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/arm64_nodepool.go:109]: failed to create ARM64 nodepool "arm64-vm-np-1" with VM size "Standard_D2plds_v6"
Unexpected error:
<*fmt.wrapError | 0x161cee87c740>:
failed to create NodePool arm64-vm-np-1: failed waiting for nodepool="arm64-vm-np-1" for cluster "arm64-vm-hcp-cluster" in resourcegroup="arm64-vm-cluster-9tq8tw8pkh5c" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/aa729d08-abec-481d-a5d5-3a517e315df9
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/aa729d08-abec-481d-a5d5-3a517e315df9",
"name": "aa729d08-abec-481d-a5d5-3a517e315df9",
"status": "Failed",
"startTime": "2026-10-27T18:45:35.525302675Z",
"endTime": "2026-10-27T19:04:37.944268372Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine q5i4q4e4c0d2d0d-arm64-vm-np-1-qfjp4-9wwh2: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/arm64-vm-cluster-9tq8tw8pkh5c--managed/providers/Microsoft.Compute/virtualMachines/q5i4q4e4c0d2d0d-arm64-vm-np-1-qfjp4-9wwh2 to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine q5i4q4e4c0d2d0d-arm64-vm-np-1-qfjp4-9wwh2: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/arm64-vm-cluster-9tq8tw8pkh5c--managed/providers/Microsoft.Compute/virtualMachines/q5i4q4e4c0d2d0d-arm64-vm-np-1-qfjp4-9wwh2 to exist\n"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/aa729d08-abec-481d-a5d5-3a517e315df9
correlation ID: bbfa9866-a276-4fbe-b94b-bc8519188ea5
client request ID: unavailable
occurredContributing tests (1)
Affected runs (1)
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
ERROR CODE: DeadlineExceeded; detail message node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hype...full failure pattern: ERROR CODE: DeadlineExceeded; detail message node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is <count>, want <count>; node pool condition AllNodesHealthy is False: <count> of <count> machines are not healthy Machine <node>: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.Compute/virtualMachines/<node> to exist Machine <node>: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.Compute/virtualMachines/<node> to exist; node pool condition AllMachinesReady is False: <count> of <count> machines are not ready Machine <node>: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.Compute/virtualMachines/<node> to exist Machine <node>: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.Compute/virtualMachines/<node> to exist; provider Microsoft.RedHatOpenShift Signal: IndeterminateSignal: Indeterminate — seen in 2 prior week(s); active 1 day(s) | e2e | 1 | 1.33%1 of 75 job runs affected | IndeterminateSignal: Indeterminate — seen in 2 prior week(s); active 1 day(s) | Show trend detailsOct 21: 0 · Oct 22: 0 · Oct 23: 1 · Oct 24: 0 · Oct 25: 0 · Oct 26: 0 · Oct 27: 0 · Oct 28: 0 | DEV, INT | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delete_cx_rg.go:141]: failed to create node pools in parallel for cluster "cx-rg-hcp-cluster"
Unexpected error:
<*fmt.wrapError | 0x9a8a07812a0>:
failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "cx-rg-hcp-cluster" in resourcegroup="cx-rg-cluster-npbbgrcnjvjf" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/switzerlandnorth/hcpOperationStatuses/5c43f0b5-5e0d-4c5c-89a9-5d1f04167fe7
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/switzerlandnorth/hcpOperationStatuses/5c43f0b5-5e0d-4c5c-89a9-5d1f04167fe7",
"name": "5c43f0b5-5e0d-4c5c-89a9-5d1f04167fe7",
"status": "Failed",
"startTime": "2026-10-23T14:54:47.282384682Z",
"endTime": "2026-10-23T15:13:53.224517483Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine d2h8j1q0z4a1f4u-np-1-ntqn4-2vwr5: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/cx-rg-cluster-npbbgrcnjvjf--managed/providers/Microsoft.Compute/virtualMachines/d2h8j1q0z4a1f4u-np-1-ntqn4-2vwr5 to exist\nMachine d2h8j1q0z4a1f4u-np-1-ntqn4-rkqlm: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/cx-rg-cluster-npbbgrcnjvjf--managed/providers/Microsoft.Compute/virtualMachines/d2h8j1q0z4a1f4u-np-1-ntqn4-rkqlm to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine d2h8j1q0z4a1f4u-np-1-ntqn4-2vwr5: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/cx-rg-cluster-npbbgrcnjvjf--managed/providers/Microsoft.Compute/virtualMachines/d2h8j1q0z4a1f4u-np-1-ntqn4-2vwr5 to exist\nMachine d2h8j1q0z4a1f4u-np-1-ntqn4-rkqlm: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/cx-rg-cluster-npbbgrcnjvjf--managed/providers/Microsoft.Compute/virtualMachines/d2h8j1q0z4a1f4u-np-1-ntqn4-rkqlm to exist\n"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/switzerlandnorth/hcpOperationStatuses/5c43f0b5-5e0d-4c5c-89a9-5d1f04167fe7
correlation ID: 90989f90-f502-41d8-955e-974a04d1e9d9
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delete_cx_rg.go:141]: failed to create node pools in parallel for cluster "cx-rg-hcp-cluster"
Unexpected error:
<*fmt.wrapError | 0x9a8a07812a0>:
failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "cx-rg-hcp-cluster" in resourcegroup="cx-rg-cluster-npbbgrcnjvjf" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/switzerlandnorth/hcpOperationStatuses/5c43f0b5-5e0d-4c5c-89a9-5d1f04167fe7
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/switzerlandnorth/hcpOperationStatuses/5c43f0b5-5e0d-4c5c-89a9-5d1f04167fe7",
"name": "5c43f0b5-5e0d-4c5c-89a9-5d1f04167fe7",
"status": "Failed",
"startTime": "2026-10-23T14:54:47.282384682Z",
"endTime": "2026-10-23T15:13:53.224517483Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine d2h8j1q0z4a1f4u-np-1-ntqn4-2vwr5: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/cx-rg-cluster-npbbgrcnjvjf--managed/providers/Microsoft.Compute/virtualMachines/d2h8j1q0z4a1f4u-np-1-ntqn4-2vwr5 to exist\nMachine d2h8j1q0z4a1f4u-np-1-ntqn4-rkqlm: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/cx-rg-cluster-npbbgrcnjvjf--managed/providers/Microsoft.Compute/virtualMachines/d2h8j1q0z4a1f4u-np-1-ntqn4-rkqlm to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine d2h8j1q0z4a1f4u-np-1-ntqn4-2vwr5: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/cx-rg-cluster-npbbgrcnjvjf--managed/providers/Microsoft.Compute/virtualMachines/d2h8j1q0z4a1f4u-np-1-ntqn4-2vwr5 to exist\nMachine d2h8j1q0z4a1f4u-np-1-ntqn4-rkqlm: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/cx-rg-cluster-npbbgrcnjvjf--managed/providers/Microsoft.Compute/virtualMachines/d2h8j1q0z4a1f4u-np-1-ntqn4-rkqlm to exist\n"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/switzerlandnorth/hcpOperationStatuses/5c43f0b5-5e0d-4c5c-89a9-5d1f04167fe7
correlation ID: 90989f90-f502-41d8-955e-974a04d1e9d9
client request ID: unavailable
occurredContributing tests (1)
Affected runs (1)
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
ERROR CODE: InternalServerError; detail message cluster deletion did not complete before the deadline; [resourceTeardown] tearing down pod network instances, pod networks, namespa...full failure pattern: ERROR CODE: InternalServerError; detail message cluster deletion did not complete before the deadline; [resourceTeardown] tearing down pod network instances, pod networks, namespaces; [descendantResources] remaining resources: applydesires, serviceProviderClusters; provider Microsoft.RedHatOpenShift Signal: IndeterminateSignal: Indeterminate — seen in 2 prior week(s); active 1 day(s) | e2e | 1 | 1.33%1 of 75 job runs affected | IndeterminateSignal: Indeterminate — seen in 2 prior week(s); active 1 day(s) | Show trend detailsOct 21: 0 · Oct 22: 0 · Oct 23: 0 · Oct 24: 1 · Oct 25: 0 · Oct 26: 0 · Oct 27: 0 · Oct 28: 0 | DEV, STG | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:310]: Unexpected error:
<*errors.joinError | 0x2874318ca1e0>:
resource group "rg-nightly-4-21-kv7pzz4ch49v": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cluster-nightly-4-21-7x886k" in resourcegroup="rg-nightly-4-21-kv7pzz4ch49v" to finish deleting: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/35ebc587-6a8d-48f1-a4a4-b752ee43dd71
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: InternalServerError
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/35ebc587-6a8d-48f1-a4a4-b752ee43dd71",
"name": "35ebc587-6a8d-48f1-a4a4-b752ee43dd71",
"status": "Failed",
"startTime": "2026-10-24T23:51:18.597955269Z",
"endTime": "2026-10-25T00:15:22.050322008Z",
"error": {
"code": "InternalServerError",
"message": "cluster deletion did not complete before the deadline; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters, 3 microsoft.redhatopenshift/hcpopenshiftclusters/applydesires; [resourceTeardown] tearing down pod network instances (1), pod networks (1), namespaces (2)"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/35ebc587-6a8d-48f1-a4a4-b752ee43dd71
correlation ID: e4ebdf9b-312f-4c03-affc-b5c4cbd39c75
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:310]: Unexpected error:
<*errors.joinError | 0x2874318ca1e0>:
resource group "rg-nightly-4-21-kv7pzz4ch49v": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cluster-nightly-4-21-7x886k" in resourcegroup="rg-nightly-4-21-kv7pzz4ch49v" to finish deleting: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/35ebc587-6a8d-48f1-a4a4-b752ee43dd71
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: InternalServerError
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/35ebc587-6a8d-48f1-a4a4-b752ee43dd71",
"name": "35ebc587-6a8d-48f1-a4a4-b752ee43dd71",
"status": "Failed",
"startTime": "2026-10-24T23:51:18.597955269Z",
"endTime": "2026-10-25T00:15:22.050322008Z",
"error": {
"code": "InternalServerError",
"message": "cluster deletion did not complete before the deadline; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters, 3 microsoft.redhatopenshift/hcpopenshiftclusters/applydesires; [resourceTeardown] tearing down pod network instances (1), pod networks (1), namespaces (2)"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/35ebc587-6a8d-48f1-a4a4-b752ee43dd71
correlation ID: e4ebdf9b-312f-4c03-affc-b5c4cbd39c75
client request ID: unavailable
occurredContributing tests (1)
Affected runs (1)
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
All ClusterOperators should report Available=True, but these are not available: [console (Available=False) ingress (Available=False)]full failure pattern: All ClusterOperators should report Available=True, but these are not available: [console (Available=False) ingress (Available=False)] Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s) | e2e | 1 | 1.33%1 of 75 job runs affected | IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s) | Show trend detailsOct 21: 0 · Oct 22: 0 · Oct 23: 1 · Oct 24: 0 · Oct 25: 0 · Oct 26: 0 · Oct 27: 0 · Oct 28: 0 | none | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:378]: Timed out after 634.146s.
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:377 with:
All ClusterOperators should report Available=True, but these are not available: [console (Available=False) ingress (Available=False)]
Expected
<[]string | len:2, cap:2>: [
"console (Available=False)",
"ingress (Available=False)",
]
to be empty
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:378]: Timed out after 634.146s.
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:377 with:
All ClusterOperators should report Available=True, but these are not available: [console (Available=False) ingress (Available=False)]
Expected
<[]string | len:2, cap:2>: [
"console (Available=False)",
"ingress (Available=False)",
]
to be emptyContributing tests (1)
Affected runs (1)
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
DNS for route host agnhost-e2e-sample-app-lpqxsfull failure pattern: DNS for route host agnhost-e2e-sample-app-lpqxs Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s) | e2e | 1 | 1.33%1 of 75 job runs affected | IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s) | Show trend detailsOct 21: 0 · Oct 22: 0 · Oct 23: 1 · Oct 24: 0 · Oct 25: 0 · Oct 26: 0 · Oct 27: 0 · Oct 28: 0 | none | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/e2e/default_cluster_ingress_cert.go:169]: sample application must serve the delivered default certificate
Expected success, but got an error:
<*fmt.wrapError | 0x398579d77aa0>:
DNS for route host agnhost-e2e-sample-app-lpqxs.apps.aro.d1o5s8c9e5q6t1e.xrjv.eastus2.aroapp-hcp.io did not resolve: DNS for agnhost-e2e-sample-app-lpqxs.apps.aro.d1o5s8c9e5q6t1e.xrjv.eastus2.aroapp-hcp.io did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-lpqxs.apps.aro.d1o5s8c9e5q6t1e.xrjv.eastus2.aroapp-hcp.io on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded
{
msg: "DNS for route host agnhost-e2e-sample-app-lpqxs.apps.aro.d1o5s8c9e5q6t1e.xrjv.eastus2.aroapp-hcp.io did not resolve: DNS for agnhost-e2e-sample-app-lpqxs.apps.aro.d1o5s8c9e5q6t1e.xrjv.eastus2.aroapp-hcp.io did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-lpqxs.apps.aro.d1o5s8c9e5q6t1e.xrjv.eastus2.aroapp-hcp.io on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
err: <*fmt.wrapError | 0x398579d77a60>{
msg: "DNS for agnhost-e2e-sample-app-lpqxs.apps.aro.d1o5s8c9e5q6t1e.xrjv.eastus2.aroapp-hcp.io did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-lpqxs.apps.aro.d1o5s8c9e5q6t1e.xrjv.eastus2.aroapp-hcp.io on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
err: <context.deadlineExceededError>{},
},
}
fail [github.com/Azure/ARO-HCP/test/e2e/default_cluster_ingress_cert.go:169]: sample application must serve the delivered default certificate
Expected success, but got an error:
<*fmt.wrapError | 0x398579d77aa0>:
DNS for route host agnhost-e2e-sample-app-lpqxs.apps.aro.d1o5s8c9e5q6t1e.xrjv.eastus2.aroapp-hcp.io did not resolve: DNS for agnhost-e2e-sample-app-lpqxs.apps.aro.d1o5s8c9e5q6t1e.xrjv.eastus2.aroapp-hcp.io did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-lpqxs.apps.aro.d1o5s8c9e5q6t1e.xrjv.eastus2.aroapp-hcp.io on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded
{
msg: "DNS for route host agnhost-e2e-sample-app-lpqxs.apps.aro.d1o5s8c9e5q6t1e.xrjv.eastus2.aroapp-hcp.io did not resolve: DNS for agnhost-e2e-sample-app-lpqxs.apps.aro.d1o5s8c9e5q6t1e.xrjv.eastus2.aroapp-hcp.io did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-lpqxs.apps.aro.d1o5s8c9e5q6t1e.xrjv.eastus2.aroapp-hcp.io on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
err: <*fmt.wrapError | 0x398579d77a60>{
msg: "DNS for agnhost-e2e-sample-app-lpqxs.apps.aro.d1o5s8c9e5q6t1e.xrjv.eastus2.aroapp-hcp.io did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-lpqxs.apps.aro.d1o5s8c9e5q6t1e.xrjv.eastus2.aroapp-hcp.io on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
err: <context.deadlineExceededError>{},
},
}Contributing tests (1)
Affected runs (1)
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedC...full failure pattern: ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable; hosted cluster degraded: UnavailableReplicas: [catalog-operator, hosted-cluster-config-operator, kube-scheduler, olm-operator, openshift-apiserver, packageserver]; provider Microsoft.RedHatOpenShift Signal: IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | e2e | 1 | 1.33%1 of 75 job runs affected | IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | Show trend detailsOct 21: 0 · Oct 22: 0 · Oct 23: 0 · Oct 24: 0 · Oct 25: 0 · Oct 26: 0 · Oct 27: 0 · Oct 28: 1 | none | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:303]: failed to create HCP cluster np-noedge-2tf7pf
Unexpected error:
<*fmt.wrapError | 0x257516f84460>:
failed to create HCP cluster np-noedge-2tf7pf: failed waiting for cluster="np-noedge-2tf7pf" in resourcegroup="rg-np-noedge-2tf7pf-xb8pwtjzh82j" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/92128bbf-b05b-4a42-b590-261b4b996ffc
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/92128bbf-b05b-4a42-b590-261b4b996ffc",
"name": "92128bbf-b05b-4a42-b590-261b4b996ffc",
"status": "Failed",
"startTime": "2026-10-28T15:21:31.746407548Z",
"endTime": "2026-10-28T15:50:37.764425706Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: catalog-operator, openshift-controller-manager, openshift-route-controller-manager, redhat-operators-catalog, community-operators-catalog, ingress-operator, csi-snapshot-controller-operator, packageserver, cluster-image-registry-operator, dns-operator, cluster-policy-controller, hosted-cluster-config-operator, cluster-node-tuning-operator, cluster-version-operator, cluster-storage-operator, redhat-marketplace-catalog, certified-operators-catalog, cluster-network-operator, olm-operator; hosted cluster degraded: UnavailableReplicas: [catalog-operator deployment has 1 unavailable replicas, hosted-cluster-config-operator deployment has 1 unavailable replicas, kube-scheduler deployment has 1 unavailable replicas, olm-operator deployment has 1 unavailable replicas, openshift-apiserver deployment has 1 unavailable replicas, packageserver deployment has 3 unavailable replicas]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/92128bbf-b05b-4a42-b590-261b4b996ffc
correlation ID: b2029c0e-c3d1-46d0-9bac-f3b40a92dc09
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:303]: failed to create HCP cluster np-noedge-2tf7pf
Unexpected error:
<*fmt.wrapError | 0x257516f84460>:
failed to create HCP cluster np-noedge-2tf7pf: failed waiting for cluster="np-noedge-2tf7pf" in resourcegroup="rg-np-noedge-2tf7pf-xb8pwtjzh82j" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/92128bbf-b05b-4a42-b590-261b4b996ffc
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/92128bbf-b05b-4a42-b590-261b4b996ffc",
"name": "92128bbf-b05b-4a42-b590-261b4b996ffc",
"status": "Failed",
"startTime": "2026-10-28T15:21:31.746407548Z",
"endTime": "2026-10-28T15:50:37.764425706Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: catalog-operator, openshift-controller-manager, openshift-route-controller-manager, redhat-operators-catalog, community-operators-catalog, ingress-operator, csi-snapshot-controller-operator, packageserver, cluster-image-registry-operator, dns-operator, cluster-policy-controller, hosted-cluster-config-operator, cluster-node-tuning-operator, cluster-version-operator, cluster-storage-operator, redhat-marketplace-catalog, certified-operators-catalog, cluster-network-operator, olm-operator; hosted cluster degraded: UnavailableReplicas: [catalog-operator deployment has 1 unavailable replicas, hosted-cluster-config-operator deployment has 1 unavailable replicas, kube-scheduler deployment has 1 unavailable replicas, olm-operator deployment has 1 unavailable replicas, openshift-apiserver deployment has 1 unavailable replicas, packageserver deployment has 3 unavailable replicas]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/92128bbf-b05b-4a42-b590-261b4b996ffc
correlation ID: b2029c0e-c3d1-46d0-9bac-f3b40a92dc09
client request ID: unavailable
occurredContributing tests (1)
Affected runs (1)
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedC...full failure pattern: ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable; hosted cluster degraded: UnavailableReplicas: [cluster-policy-controller, kube-controller-manager, kube-scheduler, openshift-controller-manager, openshift-route-controller-manager]; provider Microsoft.RedHatOpenShift Signal: IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | e2e | 1 | 1.33%1 of 75 job runs affected | IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | Show trend detailsOct 21: 0 · Oct 22: 0 · Oct 23: 0 · Oct 24: 0 · Oct 25: 0 · Oct 26: 0 · Oct 27: 0 · Oct 28: 1 | none | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:146]: failed to create HCP cluster np-version-upgrade-cluster-xgw4k6 with version 4.20.44
Unexpected error:
<*fmt.wrapError | 0x2f2f6cd22960>:
failed to create HCP cluster np-version-upgrade-cluster-xgw4k6: failed waiting for cluster="np-version-upgrade-cluster-xgw4k6" in resourcegroup="rg-np-version-upgrade-xgw4k6-fb2pqq2pp9pn" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/67af219e-4e40-4a49-ae77-1d482039a909
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/67af219e-4e40-4a49-ae77-1d482039a909",
"name": "67af219e-4e40-4a49-ae77-1d482039a909",
"status": "Failed",
"startTime": "2026-10-28T15:21:37.478622401Z",
"endTime": "2026-10-28T15:50:37.723667825Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: openshift-controller-manager, kube-scheduler, cluster-policy-controller, openshift-route-controller-manager; hosted cluster degraded: UnavailableReplicas: [cluster-policy-controller deployment has 1 unavailable replicas, kube-controller-manager deployment has 1 unavailable replicas, kube-scheduler deployment has 1 unavailable replicas, openshift-controller-manager deployment has 1 unavailable replicas, openshift-route-controller-manager deployment has 1 unavailable replicas]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/67af219e-4e40-4a49-ae77-1d482039a909
correlation ID: b05be546-d520-46a0-8713-ceb437dfacc8
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:146]: failed to create HCP cluster np-version-upgrade-cluster-xgw4k6 with version 4.20.44
Unexpected error:
<*fmt.wrapError | 0x2f2f6cd22960>:
failed to create HCP cluster np-version-upgrade-cluster-xgw4k6: failed waiting for cluster="np-version-upgrade-cluster-xgw4k6" in resourcegroup="rg-np-version-upgrade-xgw4k6-fb2pqq2pp9pn" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/67af219e-4e40-4a49-ae77-1d482039a909
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/67af219e-4e40-4a49-ae77-1d482039a909",
"name": "67af219e-4e40-4a49-ae77-1d482039a909",
"status": "Failed",
"startTime": "2026-10-28T15:21:37.478622401Z",
"endTime": "2026-10-28T15:50:37.723667825Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: openshift-controller-manager, kube-scheduler, cluster-policy-controller, openshift-route-controller-manager; hosted cluster degraded: UnavailableReplicas: [cluster-policy-controller deployment has 1 unavailable replicas, kube-controller-manager deployment has 1 unavailable replicas, kube-scheduler deployment has 1 unavailable replicas, openshift-controller-manager deployment has 1 unavailable replicas, openshift-route-controller-manager deployment has 1 unavailable replicas]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/67af219e-4e40-4a49-ae77-1d482039a909
correlation ID: b05be546-d520-46a0-8713-ceb437dfacc8
client request ID: unavailable
occurredContributing tests (1)
Affected runs (1)
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedC...full failure pattern: ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable; hosted cluster degraded: UnavailableReplicas: [cluster-policy-controller, openshift-controller-manager, openshift-route-controller-manager]; provider Microsoft.RedHatOpenShift Signal: IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | e2e | 1 | 1.33%1 of 75 job runs affected | IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | Show trend detailsOct 21: 0 · Oct 22: 0 · Oct 23: 0 · Oct 24: 0 · Oct 25: 0 · Oct 26: 0 · Oct 27: 0 · Oct 28: 1 | none | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:91]: failed to create HCP cluster for patch-name test
Unexpected error:
<*fmt.wrapError | 0x3d893d926940>:
failed to create HCP cluster patch-name-cluster: failed waiting for cluster="patch-name-cluster" in resourcegroup="patch-name-9zwtss6f6vt9" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/7bc6a3bc-3918-4c91-853b-58acb06d5c9a
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/7bc6a3bc-3918-4c91-853b-58acb06d5c9a",
"name": "7bc6a3bc-3918-4c91-853b-58acb06d5c9a",
"status": "Failed",
"startTime": "2026-10-28T14:39:20.447678223Z",
"endTime": "2026-10-28T15:08:28.688541915Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: cluster-policy-controller, openshift-route-controller-manager, openshift-controller-manager; hosted cluster degraded: UnavailableReplicas: [cluster-policy-controller deployment has 1 unavailable replicas, openshift-controller-manager deployment has 1 unavailable replicas, openshift-route-controller-manager deployment has 1 unavailable replicas]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/7bc6a3bc-3918-4c91-853b-58acb06d5c9a
correlation ID: 2b84f196-091a-41b2-98d7-60b50644dab4
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:91]: failed to create HCP cluster for patch-name test
Unexpected error:
<*fmt.wrapError | 0x3d893d926940>:
failed to create HCP cluster patch-name-cluster: failed waiting for cluster="patch-name-cluster" in resourcegroup="patch-name-9zwtss6f6vt9" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/7bc6a3bc-3918-4c91-853b-58acb06d5c9a
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/7bc6a3bc-3918-4c91-853b-58acb06d5c9a",
"name": "7bc6a3bc-3918-4c91-853b-58acb06d5c9a",
"status": "Failed",
"startTime": "2026-10-28T14:39:20.447678223Z",
"endTime": "2026-10-28T15:08:28.688541915Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: cluster-policy-controller, openshift-route-controller-manager, openshift-controller-manager; hosted cluster degraded: UnavailableReplicas: [cluster-policy-controller deployment has 1 unavailable replicas, openshift-controller-manager deployment has 1 unavailable replicas, openshift-route-controller-manager deployment has 1 unavailable replicas]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/7bc6a3bc-3918-4c91-853b-58acb06d5c9a
correlation ID: 2b84f196-091a-41b2-98d7-60b50644dab4
client request ID: unavailable
occurredContributing tests (1)
Affected runs (1)
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedC...full failure pattern: ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable; hosted cluster degraded: UnavailableReplicas: [konnectivity-agent, kube-controller-manager, kube-scheduler]; provider Microsoft.RedHatOpenShift Signal: IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | e2e | 1 | 1.33%1 of 75 job runs affected | IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | Show trend detailsOct 21: 0 · Oct 22: 0 · Oct 23: 0 · Oct 24: 0 · Oct 25: 0 · Oct 26: 0 · Oct 27: 0 · Oct 28: 1 | none | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:150]: failed to create HCP cluster
Unexpected error:
<*fmt.wrapError | 0x186bf96476a0>:
failed to create HCP cluster ingress-cert-le: failed waiting for cluster="ingress-cert-le" in resourcegroup="ingress-cert-le-f9vz9brpqmbx" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/07dce0e9-a762-440b-9ccd-61b15fa63bf8
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/07dce0e9-a762-440b-9ccd-61b15fa63bf8",
"name": "07dce0e9-a762-440b-9ccd-61b15fa63bf8",
"status": "Failed",
"startTime": "2026-10-28T13:42:56.996297525Z",
"endTime": "2026-10-28T14:12:00.914226401Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: kube-scheduler, konnectivity-agent; hosted cluster degraded: UnavailableReplicas: [konnectivity-agent deployment has 1 unavailable replicas, kube-controller-manager deployment has 1 unavailable replicas, kube-scheduler deployment has 1 unavailable replicas]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/07dce0e9-a762-440b-9ccd-61b15fa63bf8
correlation ID: b4a6856e-5332-498a-8060-89b9ebb7175d
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:150]: failed to create HCP cluster
Unexpected error:
<*fmt.wrapError | 0x186bf96476a0>:
failed to create HCP cluster ingress-cert-le: failed waiting for cluster="ingress-cert-le" in resourcegroup="ingress-cert-le-f9vz9brpqmbx" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/07dce0e9-a762-440b-9ccd-61b15fa63bf8
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/07dce0e9-a762-440b-9ccd-61b15fa63bf8",
"name": "07dce0e9-a762-440b-9ccd-61b15fa63bf8",
"status": "Failed",
"startTime": "2026-10-28T13:42:56.996297525Z",
"endTime": "2026-10-28T14:12:00.914226401Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: kube-scheduler, konnectivity-agent; hosted cluster degraded: UnavailableReplicas: [konnectivity-agent deployment has 1 unavailable replicas, kube-controller-manager deployment has 1 unavailable replicas, kube-scheduler deployment has 1 unavailable replicas]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/07dce0e9-a762-440b-9ccd-61b15fa63bf8
correlation ID: b4a6856e-5332-498a-8060-89b9ebb7175d
client request ID: unavailable
occurredContributing tests (1)
Affected runs (1)
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedC...full failure pattern: ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable; hosted cluster degraded: UnavailableReplicas: [openshift-route-controller-manager]; provider Microsoft.RedHatOpenShift Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s) | e2e | 1 | 1.33%1 of 75 job runs affected | IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s) | Show trend detailsOct 21: 0 · Oct 22: 1 · Oct 23: 0 · Oct 24: 0 · Oct 25: 0 · Oct 26: 0 · Oct 27: 0 · Oct 28: 0 | none | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_delete.go:81]: failed to create HCP cluster np-delete-hcp-cluster
Unexpected error:
<*fmt.wrapError | 0x3ede8eb28c00>:
failed to create HCP cluster np-delete-hcp-cluster: failed waiting for cluster="np-delete-hcp-cluster" in resourcegroup="nodepool-delete-qzb5xz65mghw" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/43a33130-1505-4488-8d2c-b0f13de47cff
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/43a33130-1505-4488-8d2c-b0f13de47cff",
"name": "43a33130-1505-4488-8d2c-b0f13de47cff",
"status": "Failed",
"startTime": "2026-10-22T03:04:08.329961738Z",
"endTime": "2026-10-22T03:33:15.140670523Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: openshift-route-controller-manager; hosted cluster degraded: UnavailableReplicas: openshift-route-controller-manager deployment has 1 unavailable replicas"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/43a33130-1505-4488-8d2c-b0f13de47cff
correlation ID: 2c680bb1-7f2a-45c1-a484-bbb38ed78677
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_delete.go:81]: failed to create HCP cluster np-delete-hcp-cluster
Unexpected error:
<*fmt.wrapError | 0x3ede8eb28c00>:
failed to create HCP cluster np-delete-hcp-cluster: failed waiting for cluster="np-delete-hcp-cluster" in resourcegroup="nodepool-delete-qzb5xz65mghw" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/43a33130-1505-4488-8d2c-b0f13de47cff
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/43a33130-1505-4488-8d2c-b0f13de47cff",
"name": "43a33130-1505-4488-8d2c-b0f13de47cff",
"status": "Failed",
"startTime": "2026-10-22T03:04:08.329961738Z",
"endTime": "2026-10-22T03:33:15.140670523Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: openshift-route-controller-manager; hosted cluster degraded: UnavailableReplicas: openshift-route-controller-manager deployment has 1 unavailable replicas"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/43a33130-1505-4488-8d2c-b0f13de47cff
correlation ID: 2c680bb1-7f2a-45c1-a484-bbb38ed78677
client request ID: unavailable
occurredContributing tests (1)
Affected runs (1)
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedC...full failure pattern: ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KASLoadBalancerNotReachable: health check to APIServer endpoint api-int.p8k0e2b5k4j2e4n.nvz7.eastus2euap.aroapp-hcp.io failed: Get "<url>": EOF; hosted cluster degraded: UnavailableReplicas: [ignition-server, kube-apiserver]; provider Microsoft.RedHatOpenShift Signal: IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | e2e | 1 | 1.33%1 of 75 job runs affected | IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | Show trend detailsOct 21: 0 · Oct 22: 0 · Oct 23: 0 · Oct 24: 0 · Oct 25: 0 · Oct 26: 0 · Oct 27: 0 · Oct 28: 1 | none | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:102]: failed to create HCP cluster np-autoscale-cluster
Unexpected error:
<*fmt.wrapError | 0x7c22833c460>:
failed to create HCP cluster np-autoscale-cluster: failed waiting for cluster="np-autoscale-cluster" in resourcegroup="np-autoscaling-kn67v2kn8ztj" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/d1ad5413-0977-4c3c-abd1-937a8f55bfc2
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/d1ad5413-0977-4c3c-abd1-937a8f55bfc2",
"name": "d1ad5413-0977-4c3c-abd1-937a8f55bfc2",
"status": "Failed",
"startTime": "2026-10-28T15:21:06.28860999Z",
"endTime": "2026-10-28T15:50:07.63308505Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KASLoadBalancerNotReachable: health check to APIServer endpoint api-int.p8k0e2b5k4j2e4n.nvz7.eastus2euap.aroapp-hcp.io failed: Get \"https://api-int.p8k0e2b5k4j2e4n.nvz7.eastus2euap.aroapp-hcp.io:443/healthz?verbose\": EOF; hosted cluster degraded: UnavailableReplicas: [ignition-server deployment has 1 unavailable replicas, kube-apiserver deployment has 1 unavailable replicas]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/d1ad5413-0977-4c3c-abd1-937a8f55bfc2
correlation ID: de1e5d6e-772f-44e9-aec7-7b1dbc7151a7
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:102]: failed to create HCP cluster np-autoscale-cluster
Unexpected error:
<*fmt.wrapError | 0x7c22833c460>:
failed to create HCP cluster np-autoscale-cluster: failed waiting for cluster="np-autoscale-cluster" in resourcegroup="np-autoscaling-kn67v2kn8ztj" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/d1ad5413-0977-4c3c-abd1-937a8f55bfc2
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/d1ad5413-0977-4c3c-abd1-937a8f55bfc2",
"name": "d1ad5413-0977-4c3c-abd1-937a8f55bfc2",
"status": "Failed",
"startTime": "2026-10-28T15:21:06.28860999Z",
"endTime": "2026-10-28T15:50:07.63308505Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KASLoadBalancerNotReachable: health check to APIServer endpoint api-int.p8k0e2b5k4j2e4n.nvz7.eastus2euap.aroapp-hcp.io failed: Get \"https://api-int.p8k0e2b5k4j2e4n.nvz7.eastus2euap.aroapp-hcp.io:443/healthz?verbose\": EOF; hosted cluster degraded: UnavailableReplicas: [ignition-server deployment has 1 unavailable replicas, kube-apiserver deployment has 1 unavailable replicas]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/d1ad5413-0977-4c3c-abd1-937a8f55bfc2
correlation ID: de1e5d6e-772f-44e9-aec7-7b1dbc7151a7
client request ID: unavailable
occurredContributing tests (1)
Affected runs (1)
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
ERROR CODE: DeadlineExceeded; detail message node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hype...full failure pattern: ERROR CODE: DeadlineExceeded; detail message node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is <count>, want <count>; node pool condition AllNodesHealthy is False: <count> of <count> machines are not healthy Machine <node>: InspectionFailed: Waiting for AzureMachine to report spec.providerID; node pool condition AllMachinesReady is False: <count> of <count> machines are not ready Machine <node>: Creating: interfaces creating or updating; provider Microsoft.RedHatOpenShift Signal: IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | e2e | 1 | 1.33%1 of 75 job runs affected | IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | Show trend detailsOct 21: 0 · Oct 22: 0 · Oct 23: 0 · Oct 24: 0 · Oct 25: 0 · Oct 26: 0 · Oct 27: 1 · Oct 28: 0 | none | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_ephemeral_osdisk.go:108]: failed to create nodepool ephemeral-np with ephemeral OS disk
Unexpected error:
<*fmt.wrapError | 0x27665d95f6c0>:
failed to create NodePool ephemeral-np: failed waiting for nodepool="ephemeral-np" for cluster "ephemeral-disk" in resourcegroup="ephemeral-osdisk-dhqxbzp84k6j" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/australiaeast/hcpOperationStatuses/7b5c9bdf-eb40-4dfb-a637-a90c825e7658
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/australiaeast/hcpOperationStatuses/7b5c9bdf-eb40-4dfb-a637-a90c825e7658",
"name": "7b5c9bdf-eb40-4dfb-a637-a90c825e7658",
"status": "Failed",
"startTime": "2026-10-27T03:14:20.369455888Z",
"endTime": "2026-10-27T03:33:20.394901036Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine ephemeral-disk-ephemeral-np-gqrvp-269v2: InspectionFailed: Waiting for AzureMachine to report spec.providerID\nMachine ephemeral-disk-ephemeral-np-gqrvp-q67fk: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine ephemeral-disk-ephemeral-np-gqrvp-269v2: Creating: interfaces creating or updating\nMachine ephemeral-disk-ephemeral-np-gqrvp-q67fk: Creating: interfaces creating or updating\n"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/australiaeast/hcpOperationStatuses/7b5c9bdf-eb40-4dfb-a637-a90c825e7658
correlation ID: a243a81d-f70c-497e-9a6c-d0f5669ed0d7
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_ephemeral_osdisk.go:108]: failed to create nodepool ephemeral-np with ephemeral OS disk
Unexpected error:
<*fmt.wrapError | 0x27665d95f6c0>:
failed to create NodePool ephemeral-np: failed waiting for nodepool="ephemeral-np" for cluster "ephemeral-disk" in resourcegroup="ephemeral-osdisk-dhqxbzp84k6j" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/australiaeast/hcpOperationStatuses/7b5c9bdf-eb40-4dfb-a637-a90c825e7658
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/australiaeast/hcpOperationStatuses/7b5c9bdf-eb40-4dfb-a637-a90c825e7658",
"name": "7b5c9bdf-eb40-4dfb-a637-a90c825e7658",
"status": "Failed",
"startTime": "2026-10-27T03:14:20.369455888Z",
"endTime": "2026-10-27T03:33:20.394901036Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine ephemeral-disk-ephemeral-np-gqrvp-269v2: InspectionFailed: Waiting for AzureMachine to report spec.providerID\nMachine ephemeral-disk-ephemeral-np-gqrvp-q67fk: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine ephemeral-disk-ephemeral-np-gqrvp-269v2: Creating: interfaces creating or updating\nMachine ephemeral-disk-ephemeral-np-gqrvp-q67fk: Creating: interfaces creating or updating\n"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/australiaeast/hcpOperationStatuses/7b5c9bdf-eb40-4dfb-a637-a90c825e7658
correlation ID: a243a81d-f70c-497e-9a6c-d0f5669ed0d7
client request ID: unavailable
occurredContributing tests (1)
Affected runs (1)
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
ERROR CODE: DeadlineExceeded; detail message node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hype...full failure pattern: ERROR CODE: DeadlineExceeded; detail message node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is <count>, want <count>; node pool condition AllNodesHealthy is False: <count> of <count> machines are not healthy Machine <node>: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.Compute/virtualMachines/oidc-wi-cluster-oidc-wi-np-<id> to exist Machine <node>: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.Compute/virtualMachines/oidc-wi-cluster-oidc-wi-np-<id> to exist; node pool condition AllMachinesReady is False: <count> of <count> machines are not ready Machine <node>: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.Compute/virtualMachines/oidc-wi-cluster-oidc-wi-np-<id> to exist Machine <node>: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.Compute/virtualMachines/oidc-wi-cluster-oidc-wi-np-<id> to exist; provider Microsoft.RedHatOpenShift Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s) | e2e | 1 | 1.33%1 of 75 job runs affected | IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s) | Show trend detailsOct 21: 0 · Oct 22: 0 · Oct 23: 1 · Oct 24: 0 · Oct 25: 0 · Oct 26: 0 · Oct 27: 0 · Oct 28: 0 | none | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/e2e/oidc_issuer_workload_identity.go:270]: failed to create node pool oidc-wi-np
Unexpected error:
<*fmt.wrapError | 0x2ef724ca1cc0>:
failed to create NodePool oidc-wi-np: failed waiting for nodepool="oidc-wi-np" for cluster "oidc-wi-cluster" in resourcegroup="oidc-wi-gnnvxwc94fcv" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/switzerlandnorth/hcpOperationStatuses/eb80abc6-2e4c-4f05-8321-ab95d4028e3a
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/switzerlandnorth/hcpOperationStatuses/eb80abc6-2e4c-4f05-8321-ab95d4028e3a",
"name": "eb80abc6-2e4c-4f05-8321-ab95d4028e3a",
"status": "Failed",
"startTime": "2026-10-23T13:03:20.324115794Z",
"endTime": "2026-10-23T13:22:20.156214194Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine oidc-wi-cluster-oidc-wi-np-5g7jb-cv8p2: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/oidc-wi-gnnvxwc94fcv--managed/providers/Microsoft.Compute/virtualMachines/oidc-wi-cluster-oidc-wi-np-5g7jb-cv8p2 to exist\nMachine oidc-wi-cluster-oidc-wi-np-5g7jb-qgh76: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/oidc-wi-gnnvxwc94fcv--managed/providers/Microsoft.Compute/virtualMachines/oidc-wi-cluster-oidc-wi-np-5g7jb-qgh76 to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine oidc-wi-cluster-oidc-wi-np-5g7jb-cv8p2: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/oidc-wi-gnnvxwc94fcv--managed/providers/Microsoft.Compute/virtualMachines/oidc-wi-cluster-oidc-wi-np-5g7jb-cv8p2 to exist\nMachine oidc-wi-cluster-oidc-wi-np-5g7jb-qgh76: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/oidc-wi-gnnvxwc94fcv--managed/providers/Microsoft.Compute/virtualMachines/oidc-wi-cluster-oidc-wi-np-5g7jb-qgh76 to exist\n"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/switzerlandnorth/hcpOperationStatuses/eb80abc6-2e4c-4f05-8321-ab95d4028e3a
correlation ID: 5b15bb34-6e8a-47a7-ac3b-f4e7ecb8ef82
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/oidc_issuer_workload_identity.go:270]: failed to create node pool oidc-wi-np
Unexpected error:
<*fmt.wrapError | 0x2ef724ca1cc0>:
failed to create NodePool oidc-wi-np: failed waiting for nodepool="oidc-wi-np" for cluster "oidc-wi-cluster" in resourcegroup="oidc-wi-gnnvxwc94fcv" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/switzerlandnorth/hcpOperationStatuses/eb80abc6-2e4c-4f05-8321-ab95d4028e3a
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/switzerlandnorth/hcpOperationStatuses/eb80abc6-2e4c-4f05-8321-ab95d4028e3a",
"name": "eb80abc6-2e4c-4f05-8321-ab95d4028e3a",
"status": "Failed",
"startTime": "2026-10-23T13:03:20.324115794Z",
"endTime": "2026-10-23T13:22:20.156214194Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine oidc-wi-cluster-oidc-wi-np-5g7jb-cv8p2: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/oidc-wi-gnnvxwc94fcv--managed/providers/Microsoft.Compute/virtualMachines/oidc-wi-cluster-oidc-wi-np-5g7jb-cv8p2 to exist\nMachine oidc-wi-cluster-oidc-wi-np-5g7jb-qgh76: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/oidc-wi-gnnvxwc94fcv--managed/providers/Microsoft.Compute/virtualMachines/oidc-wi-cluster-oidc-wi-np-5g7jb-qgh76 to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine oidc-wi-cluster-oidc-wi-np-5g7jb-cv8p2: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/oidc-wi-gnnvxwc94fcv--managed/providers/Microsoft.Compute/virtualMachines/oidc-wi-cluster-oidc-wi-np-5g7jb-cv8p2 to exist\nMachine oidc-wi-cluster-oidc-wi-np-5g7jb-qgh76: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/oidc-wi-gnnvxwc94fcv--managed/providers/Microsoft.Compute/virtualMachines/oidc-wi-cluster-oidc-wi-np-5g7jb-qgh76 to exist\n"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/switzerlandnorth/hcpOperationStatuses/eb80abc6-2e4c-4f05-8321-ab95d4028e3a
correlation ID: 5b15bb34-6e8a-47a7-ac3b-f4e7ecb8ef82
client request ID: unavailable
occurredContributing tests (1)
Affected runs (1)
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
ERROR CODE: DeadlineExceeded; detail message node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hype...full failure pattern: ERROR CODE: DeadlineExceeded; detail message node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is <count>, want <count>; node pool condition AllNodesHealthy is False: <count> of <count> machines are not healthy Machine <node>: NodeNotHealthy: * Node.Ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?; node pool condition AllMachinesReady is False: <count> of <count> machines are not ready Machine <node>: NotReady: * NodeHealthy: * Node.Ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?; provider Microsoft.RedHatOpenShift Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s) | e2e | 1 | 1.33%1 of 75 job runs affected | IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s) | Show trend detailsOct 21: 0 · Oct 22: 1 · Oct 23: 0 · Oct 24: 0 · Oct 25: 0 · Oct 26: 0 · Oct 27: 0 · Oct 28: 0 | DEV | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:177]: failed to create node pool "np-lp5s2l" for cluster "cluster-candidate-5-0-lp5s2l"
Unexpected error:
<*fmt.wrapError | 0x3c5070b80fa0>:
failed to create NodePool np-lp5s2l: failed waiting for nodepool="np-lp5s2l" for cluster "cluster-candidate-5-0-lp5s2l" in resourcegroup="rg-candidate-5-0-m8vcgvcqk4pg" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westeurope/hcpOperationStatuses/fd2644a3-f725-418e-af3f-2985a0768c42
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westeurope/hcpOperationStatuses/fd2644a3-f725-418e-af3f-2985a0768c42",
"name": "fd2644a3-f725-418e-af3f-2985a0768c42",
"status": "Failed",
"startTime": "2026-10-22T05:20:50.362804863Z",
"endTime": "2026-10-22T05:39:55.652634149Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine u1j3l3a4s2x8k8u-np-lp5s2l-lng7p-xl2t7: NodeNotHealthy: * Node.Ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine u1j3l3a4s2x8k8u-np-lp5s2l-lng7p-xl2t7: NotReady: * NodeHealthy:\n * Node.Ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?\n"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westeurope/hcpOperationStatuses/fd2644a3-f725-418e-af3f-2985a0768c42
correlation ID: b2009fac-4672-460e-84fc-49211ba0f615
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:177]: failed to create node pool "np-lp5s2l" for cluster "cluster-candidate-5-0-lp5s2l"
Unexpected error:
<*fmt.wrapError | 0x3c5070b80fa0>:
failed to create NodePool np-lp5s2l: failed waiting for nodepool="np-lp5s2l" for cluster "cluster-candidate-5-0-lp5s2l" in resourcegroup="rg-candidate-5-0-m8vcgvcqk4pg" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westeurope/hcpOperationStatuses/fd2644a3-f725-418e-af3f-2985a0768c42
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westeurope/hcpOperationStatuses/fd2644a3-f725-418e-af3f-2985a0768c42",
"name": "fd2644a3-f725-418e-af3f-2985a0768c42",
"status": "Failed",
"startTime": "2026-10-22T05:20:50.362804863Z",
"endTime": "2026-10-22T05:39:55.652634149Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine u1j3l3a4s2x8k8u-np-lp5s2l-lng7p-xl2t7: NodeNotHealthy: * Node.Ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine u1j3l3a4s2x8k8u-np-lp5s2l-lng7p-xl2t7: NotReady: * NodeHealthy:\n * Node.Ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?\n"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westeurope/hcpOperationStatuses/fd2644a3-f725-418e-af3f-2985a0768c42
correlation ID: b2009fac-4672-460e-84fc-49211ba0f615
client request ID: unavailable
occurredContributing tests (1)
Affected runs (1)
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
VerifySimpleWebApp failed: DNS for route host <route-host> did not resolve: DNS for <route-host> did not resolve within <duration> (last error: lookup <host> on <dns-server>: no s...full failure pattern: VerifySimpleWebApp failed: DNS for route host <route-host> did not resolve: DNS for <route-host> did not resolve within <duration> (last error: lookup <host> on <dns-server>: no such host): context deadline exceeded Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s) | e2e | 1 | 1.33%1 of 75 job runs affected | IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s) | Show trend detailsOct 21: 0 · Oct 22: 0 · Oct 23: 1 · Oct 24: 0 · Oct 25: 0 · Oct 26: 0 · Oct 27: 0 · Oct 28: 0 | none | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_cni_cilium.go:186]: failed to run simple web app and connectivity check app with cilium CNI
Unexpected error:
<*errors.joinError | 0x2d202c117968>:
VerifySimpleWebApp failed: DNS for route host agnhost-e2e-sample-app-sv2qn.apps.aro.cilium-cl.43f6.eastus2.aroapp-hcp.io did not resolve: DNS for agnhost-e2e-sample-app-sv2qn.apps.aro.cilium-cl.43f6.eastus2.aroapp-hcp.io did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-sv2qn.apps.aro.cilium-cl.43f6.eastus2.aroapp-hcp.io on 172.30.0.10:53: no such host): context deadline exceeded
{
errs: [
<*fmt.wrapError | 0x2d202bb0ece0>{
msg: "VerifySimpleWebApp failed: DNS for route host agnhost-e2e-sample-app-sv2qn.apps.aro.cilium-cl.43f6.eastus2.aroapp-hcp.io did not resolve: DNS for agnhost-e2e-sample-app-sv2qn.apps.aro.cilium-cl.43f6.eastus2.aroapp-hcp.io did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-sv2qn.apps.aro.cilium-cl.43f6.eastus2.aroapp-hcp.io on 172.30.0.10:53: no such host): context deadline exceeded",
err: <*fmt.wrapError | 0x2d202ad36960>{
msg: "DNS for route host agnhost-e2e-sample-app-sv2qn.apps.aro.cilium-cl.43f6.eastus2.aroapp-hcp.io did not resolve: DNS for agnhost-e2e-sample-app-sv2qn.apps.aro.cilium-cl.43f6.eastus2.aroapp-hcp.io did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-sv2qn.apps.aro.cilium-cl.43f6.eastus2.aroapp-hcp.io on 172.30.0.10:53: no such host): context deadline exceeded",
err: <*fmt.wrapError | 0x2d202a99c2c0>{
msg: "DNS for agnhost-e2e-sample-app-sv2qn.apps.aro.cilium-cl.43f6.eastus2.aroapp-hcp.io did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-sv2qn.apps.aro.cilium-cl.43f6.eastus2.aroapp-hcp.io on 172.30.0.10:53: no such host): context deadline exceeded",
err: <context.deadlineExceededError>{},
},
},
},
],
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_cni_cilium.go:186]: failed to run simple web app and connectivity check app with cilium CNI
Unexpected error:
<*errors.joinError | 0x2d202c117968>:
VerifySimpleWebApp failed: DNS for route host agnhost-e2e-sample-app-sv2qn.apps.aro.cilium-cl.43f6.eastus2.aroapp-hcp.io did not resolve: DNS for agnhost-e2e-sample-app-sv2qn.apps.aro.cilium-cl.43f6.eastus2.aroapp-hcp.io did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-sv2qn.apps.aro.cilium-cl.43f6.eastus2.aroapp-hcp.io on 172.30.0.10:53: no such host): context deadline exceeded
{
errs: [
<*fmt.wrapError | 0x2d202bb0ece0>{
msg: "VerifySimpleWebApp failed: DNS for route host agnhost-e2e-sample-app-sv2qn.apps.aro.cilium-cl.43f6.eastus2.aroapp-hcp.io did not resolve: DNS for agnhost-e2e-sample-app-sv2qn.apps.aro.cilium-cl.43f6.eastus2.aroapp-hcp.io did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-sv2qn.apps.aro.cilium-cl.43f6.eastus2.aroapp-hcp.io on 172.30.0.10:53: no such host): context deadline exceeded",
err: <*fmt.wrapError | 0x2d202ad36960>{
msg: "DNS for route host agnhost-e2e-sample-app-sv2qn.apps.aro.cilium-cl.43f6.eastus2.aroapp-hcp.io did not resolve: DNS for agnhost-e2e-sample-app-sv2qn.apps.aro.cilium-cl.43f6.eastus2.aroapp-hcp.io did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-sv2qn.apps.aro.cilium-cl.43f6.eastus2.aroapp-hcp.io on 172.30.0.10:53: no such host): context deadline exceeded",
err: <*fmt.wrapError | 0x2d202a99c2c0>{
msg: "DNS for agnhost-e2e-sample-app-sv2qn.apps.aro.cilium-cl.43f6.eastus2.aroapp-hcp.io did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-sv2qn.apps.aro.cilium-cl.43f6.eastus2.aroapp-hcp.io on 172.30.0.10:53: no such host): context deadline exceeded",
err: <context.deadlineExceededError>{},
},
},
},
],
}
occurredContributing tests (1)
Affected runs (1)
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
failed waiting for nodepool="<nodepool>" in cluster="<cluster>" resourcegroup="<resource-group>" to finish updating: GET <url>full failure pattern: failed waiting for nodepool="<nodepool>" in cluster="<cluster>" resourcegroup="<resource-group>" to finish updating: GET <url> Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s) | e2e | 1 | 1.33%1 of 75 job runs affected | IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s) | Show trend detailsOct 21: 0 · Oct 22: 1 · Oct 23: 0 · Oct 24: 0 · Oct 25: 0 · Oct 26: 0 · Oct 27: 0 · Oct 28: 0 | none | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_labels_taints.go:186]: failed to update node pool np-1 with new taints and scale up
Unexpected error:
<*fmt.wrapError | 0x32f3fb206560>:
failed waiting for nodepool="np-1" in cluster="cluster-np-labels-taints" resourcegroup="rg-np-labels-taints-qk6fnz9248pm" to finish updating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/8a81c298-fb2e-4c06-ab5b-f047a74ffe2b
--------------------------------------------------------------------------------
RESPONSE 502: 502 Bad Gateway
ERROR CODE UNAVAILABLE
--------------------------------------------------------------------------------
Response contained no body
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/8a81c298-fb2e-4c06-ab5b-f047a74ffe2b
correlation ID: bd21a41c-3537-483b-a104-ebc5b06bde9b
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_labels_taints.go:186]: failed to update node pool np-1 with new taints and scale up
Unexpected error:
<*fmt.wrapError | 0x32f3fb206560>:
failed waiting for nodepool="np-1" in cluster="cluster-np-labels-taints" resourcegroup="rg-np-labels-taints-qk6fnz9248pm" to finish updating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/8a81c298-fb2e-4c06-ab5b-f047a74ffe2b
--------------------------------------------------------------------------------
RESPONSE 502: 502 Bad Gateway
ERROR CODE UNAVAILABLE
--------------------------------------------------------------------------------
Response contained no body
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/8a81c298-fb2e-4c06-ab5b-f047a74ffe2b
correlation ID: bd21a41c-3537-483b-a104-ebc5b06bde9b
client request ID: unavailable
occurredContributing tests (1)
Affected runs (1)
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||