Skip to content

fix(tokenizer): skip blank lines when loading a seg_dict file - #3903

Merged
LauraGPT merged 1 commit into
modelscope:mainfrom
MohammadHijjawi97:fix/char-tokenizer-seg-dict-blank-lines
Nov 15, 2026
Merged

LauraGPT merged 1 commit into
modelscope:mainfrom
MohammadHijjawi97:fix/char-tokenizer-seg-dict-blank-lines

Conversation

@MohammadHijjawi97

Copy link
Copy Markdown
Contributor

Summary

load_seg_dict in funasr/tokenizer/char_tokenizer.py reads the first field of every line without checking it. A seg_dict file with a blank or whitespace-only line (for example a trailing empty line) makes CharTokenizer(seg_dict=...) fail during construction:

  File "funasr/tokenizer/char_tokenizer.py", line 117, in load_seg_dict
    key = s[0]
IndexError: list index out of range

This change skips empty lines, so the tokenizer loads and tokenizes as expected:

# seg_dict: "你 你\n\n好 好\n   \nhello he@@ llo\n\n"
CharTokenizer(seg_dict=path).text2tokens("你好 hello")
# before: IndexError
# after:  ['你', '好', 'he@@', 'llo']

Type of change

  • Bug fix
  • Documentation
  • Example or demo
  • Runtime or deployment
  • Benchmark or evaluation
  • Model/training change

Validation

  • Added tests/test_char_tokenizer_seg_dict.py. It fails with IndexError before the change and passes after.
  • python -m pytest tests/test_char_tokenizer_seg_dict.py tests/test_char_tokenizer_non_linguistic_symbols.py: 4 passed
  • python -m compileall funasr examples tests (ran compileall on the two changed files only)
  • Docs or links checked
  • Runtime/deployment command tested

User impact

Users who pass a seg_dict (seg_dict / seg_dict_file) that contains blank lines can build the tokenizer instead of hitting an IndexError.

Notes for reviewers

Two-line change. Non-empty lines are parsed exactly as before. funasr/models/seaco_paraformer/model.py and funasr/models/contextual_paraformer/model.py have their own copies of load_seg_dict with the same pattern; they are left unchanged here to keep this PR focused.

load_seg_dict indexed the first field of every line, so a blank or
whitespace-only line (for example a trailing empty line) raised
IndexError and CharTokenizer(seg_dict=...) could not be built.
LauraGPT added a commit that referenced this pull request Nov 15, 2026
Preserve original #3903 ancestry. Scoped CPU and real native CUDA CTC input evidence; exact NumPy matrix CI accepted. No full-repository or pretrained quality claim.

Signed-off-by: LauraGPT <zhifu.gzf@alibaba-inc.com>
@LauraGPT
LauraGPT merged commit e022fd8 into modelscope:main Nov 15, 2026
2 checks passed
@LauraGPT LauraGPT mentioned this pull request Nov 17, 2026
LauraGPT added a commit that referenced this pull request Nov 17, 2026
Deliver tested NumPy audio-array, zero-duration VAD, timestamp, and segmentation-dictionary fixes (#3890, #3891, #3896, #3898, #3903).

Signed-off-by: LauraGPT <zhifu.gzf@alibaba-inc.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants