perf(llm): avoid tokenizing discarded lines

[dexhunter]

Description

Fixes #12980.

PR #5310 removed repeated array mutation from pruneLinesFromTop and pruneLinesFromBottom, but both functions still tokenize every line before discarding most of them. This change traverses from the side that will be retained and stops as soon as the next line would exceed the token budget.

The existing one-token newline accounting and returned strings are preserved. The production change is limited to the two pruning function bodies, and this PR enables their existing focused tests.

Performance

The deterministic evaluator counts real Tiktoken.encode calls across top and bottom pruning of a 320-line prompt:

Metric	main	This PR	Change
median tokenizer calls	641	88	-86.3%

All nine baseline and candidate samples were identical within each revision. A 580-case differential suite covering empty input, leading and trailing blank lines, newline boundaries, Unicode, oversized lines, random multiline prompts, and zero or negative limits produced byte-for-byte identical output.

On the 2,000-line scale described in #4947, the same metric is 4,001 calls on main and 86 on this branch (-97.9%).

Autoresearch

The retained-side traversal was independently reproduced during a 10-step autoresearch run with Weco. The final patch was then reduced and validated in a clean worktree.

Public trajectory: https://dashboard.weco.ai/share/xWch5tqFM_tvJr2AZVT4lWugmaTnbu5b

AI Code Review

• Team members only: AI review runs automatically when PR is opened or marked ready for review
• Team members can also trigger a review by commenting @continue-review

Checklist

• I've read the contributing guide
• The relevant docs, if any, have been updated or created
• The relevant tests, if any, have been updated or created

Screen recording or screenshot

Not applicable; this changes token-pruning work without changing UI behavior.

Tests

• npm test -- --runInBand llm/countTokens.test.ts (25 passed, 9 unrelated skipped)
• npm run tsc:check
• ESLint and Prettier on both changed files
• strict 580-case differential evaluator and positive/negative-control replay
• git diff --check

[github-actions]

All contributors have signed the CLA ✍️ ✅
_{Posted by the CLA Assistant Lite bot.}

[dexhunter]

I have read the CLA Document and I hereby sign the CLA

[dexhunter]

The jetbrains-tests failure is in the pre-existing IntelliJ integration test Autocomplete.testAutocomplete() at Autocomplete.kt:42 (expected true but was false). This PR only modifies core/llm/countTokens.ts and its tests. Current main is the PR base (d0a3c0b) and reproduces the identical JetBrains failure in its own PR Checks run, so this does not appear attributable to this diff.