Repository navigation
Conversation
|
|
A reply that hits max_tokens comes back with finish_reason="length", but nothing checked it. The cut-off reply was parsed as if it were complete, and the iteration failed with "No valid diffs found in response" (or an unmatched delimiter error), which points at the model's output format instead of the limit. Thinking models count reasoning tokens toward max_tokens, so this happens a lot with them. _call_api now logs a warning with the model and the limit and records finish_reason in last_usage, and the iteration error says when the reply was cut off.
fermibubble
force-pushed
the
report-truncated-responses
branch
from
November 18, 2026 02:26
0638478 to
d6b082e
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
When a reply hits
max_tokens, the API returnsfinish_reason="length", but nothing checks it. The cut-off reply is parsed as if it were complete, and the iteration fails with "No valid diffs found in response", or "Unmatched SEARCH/REPLACE delimiter" when the cut lands inside a block. Both point at the model's output format (see #241), so it's easy to look in the wrong place.Thinking models count reasoning tokens toward
max_tokens, so this comes up a lot. In a 40-iteration run with gemini-3.5-flash andmax_tokens: 8192, 20 iterations failed with "No valid diffs found", and every one of them used exactly 8188 tokens on top of the prompt. The 20 that worked used 2,540 to 7,290. Sending one of the run's prompts to the API directly, 2 of 4 replies came back withfinish_reason="length": one stopped mid-sentence before the diff, the other partway through its block. With the defaultmax_tokens: 4096, all 4 were cut off. The log gives no hint, because Gemini'scompletion_tokensleaves out thinking tokens (completion: 326out of a total of 13676). The same run withmax_tokens: 32768had no failed iterations out of 40.Changes:
OpenAILLM._call_apilogs a warning with the model and the limit whenfinish_reason == "length", and addsfinish_reasontolast_usage.Iteration 1 error [diff]: No valid diffs found in response (response was cut off at llm.max_tokens) | tokens: 11547 (prompt: 3359, completion: 325)Cut-off replies whose complete blocks still parse are applied as before. I left that alone, but can reject them instead if you'd rather.
Tests: new
tests/test_openai_finish_reason.pyand aTestWorkerTruncatedResponseclass intests/test_diff_format_validation_worker.py. 4 of the 5 new tests fail on main. Full suite (python -m unittest discover -s tests -p "test_*.py"): 1081 OK, 7 skipped.