Skip to content

fix(websocket): send file-mode hotwords as the protocol's JSON object - #3924

Open
tanishqbaweja wants to merge 1 commit into
modelscope:mainfrom
tanishqbaweja:fix/wss-client-file-hotwords
Open

tanishqbaweja wants to merge 1 commit into
modelscope:mainfrom
tanishqbaweja:fix/wss-client-file-hotwords

Conversation

@tanishqbaweja

Copy link
Copy Markdown
Contributor

Summary

Fixes #3923

Since #2791, runtime/python/websocket/funasr_wss_client.py in file mode (--audio_in) sent the --hotword file as the first word of each line joined with commas, such as 阿里巴巴,通义实验室, without the weights. The websocket protocol (runtime/docs/websocket_protocol.md) sends hotwords as a JSON object string that maps each hotword to a weight. This client sends that in microphone mode, and so do the C++ and HTML5 clients.

The C++ servers (websocket-server.cpp:314-358, websocket-server-2pass.cpp:469-513) take a client's hotwords and weights for the WFST decoder only from a JSON object. They pass any other string to the NN hotword model as it is. There 阿里巴巴,通义实验室 is a single hotword that is not all Chinese characters and is not in seg_dict, so CompileHotwordEmbedding skips it (runtime/onnxruntime/src/paraformer.cpp:571-588). So with the C++ servers, which the SDK guides pair with this client's --hotword (runtime/docs/SDK_advanced_guide_offline.md:153), a hotword file with two or more hotwords had no effect in file mode, and a single hotword lost its weight. funasr_wss_server.py reads the JSON object too since #3880 (merged in #3892). There the comma string was also one hotword, which maps to <unk>.

This PR builds the field in record_from_scp the way record_microphone does: a {"hotword": weight} dict from the lines hotword weight, sent as json.dumps(fst_dict, ensure_ascii=False). Lines without an integer weight are skipped with the same "Please checkout format of hotwords" message. File mode keeps its split() and still skips blank lines silently. An empty --hotword and one that is not a file are sent unchanged.

The hotwords field that the real client sends in file mode (stand-in server on 127.0.0.1, see Validation):

--hotword main this PR
file with 阿里巴巴 20 and 通义实验室 30 阿里巴巴,通义实验室 {"阿里巴巴": 20, "通义实验室": 30}
file with 阿里巴巴 20 阿里巴巴 {"阿里巴巴": 20}
not set "" ""
阿里巴巴 通义实验室 (not a file) 阿里巴巴 通义实验室 阿里巴巴 通义实验室

Type of change

  • Bug fix
  • Documentation
  • Example or demo
  • Runtime or deployment
  • Benchmark or evaluation
  • Model/training change

Validation

The new tests/test_websocket_client_hotwords.py compiles record_from_scp out of the client, as tests/test_websocket_file_finalization.py does (importing the client would parse sys.argv). It sends 0.1 s of PCM to a websocket that records what it is sent, with the end acknowledgement already queued, and checks the hotwords of the config message:

  • a file with 阿里巴巴 20 and 通义实验室 30 gives {"阿里巴巴": 20, "通义实验室": 30};
  • a file with hello world 10, 阿里巴巴 and 通义实验室 high gives {"hello world": 10}: the phrase stays whole, and the two lines without an integer weight are skipped, as in microphone mode;
  • "" and 阿里巴巴 通义实验室 (not a file) are sent unchanged.

Windows 11, CPU, Python 3.11.17, pytest 9.2.0:

python -m pytest -q tests/test_websocket_client_hotwords.py tests/test_websocket_file_finalization.py tests/test_websocket_server_hotwords.py tests/test_websocket_server_asr_model.py
  • main (3747f84) client with the new test: 2 failed, 38 passed: assert '阿里巴巴,通义实验室' == '{"阿里巴巴": 20, "通义实验室": 30}' and assert 'hello,阿里巴巴,通义实验室' == '{"hello world": 10}'.
  • This branch: 40 passed, also with -W error.

I had no numpy, scipy or torch in this environment, so the two server tests ran with minimal stand-in modules for them on PYTHONPATH (only the names the server imports); test-python-websocket.yml installs the real ones.

I also ran the real client as a process, python funasr_wss_client.py --host 127.0.0.1 --port <port> --ssl 0 --mode offline --audio_in audio.pcm [--hotword ...], with websockets 17.4, against a stand-in server on 127.0.0.1 that records the config message and replies {"is_end": true, "is_final": true} (the script in the issue does this for the first case). It sent the values in the table above, and every run exited 0. With a stand-in pyaudio module, record_microphone sends the same hotwords as file mode on this branch for the two test files and for 阿里巴巴 20.

Then I ran the client against the real funasr_wss_server.py of main, both as processes on 127.0.0.1, with a stand-in funasr.AutoModel whose offline model records the hotword that its generate receives. Every client run exited 0. The last column is what SeACo-Paraformer's own generate_hotwords_list makes of that hotword, with the toy vocabulary of #3879 (<unk> = 3, <s> = 1):

hotword file client hotwords sent hotword given to generate generate_hotwords_list
阿里巴巴 20 main 阿里巴巴 阿里巴巴 [[4, 5, 6, 6], [1]]
阿里巴巴 20 this PR {"阿里巴巴": 20} 阿里巴巴 [[4, 5, 6, 6], [1]]
阿里巴巴 20, 通义实验室 30 main 阿里巴巴,通义实验室 阿里巴巴,通义实验室 [[3], [1]]
阿里巴巴 20, 通义实验室 30 this PR {"阿里巴巴": 20, "通义实验室": 30} 阿里巴巴 通义实验室 [[4, 5, 6, 6], [7, 8, 9, 10, 11], [1]]
  • python -m compileall funasr examples tests
  • Docs or links checked
  • Runtime/deployment command tested (the real client against a local stand-in server, and against the real Python server with stand-in models; no model, no C++ server build)

black --line-length=100 (24.4.0, as in pre-commit) leaves the new test unchanged, and makes the same changes to funasr_wss_client.py on main and on this branch, none on the changed lines. No real model was run.

User impact

Users who run funasr_wss_client.py with --audio_in and a --hotword file against the C++ servers, as the SDK guides show, get their hotwords applied: in the WFST decoder with their weights when the server runs with an LM, and in the NN hotword model when it runs a hotword model. Before, a file with two or more hotwords had no effect, and a single hotword reached only the NN hotword model, without its weight. With funasr_wss_server.py, a file with two or more hotwords now reaches the model as those hotwords instead of one <unk> hotword, and a file with one hotword reaches it as before. Runs without --hotword, or with a --hotword string that is not a file, send the same message as before.

Notes for reviewers

  • With fix(websocket): pass client hotwords to AutoModel as words #3880 merged, funasr_wss_server.py reads the JSON object too and gives AutoModel its words, as the table in Validation shows. AutoModel takes no weights, so that server drops them, as it did before.
  • Lines without an integer weight (阿里巴巴, 通义实验室 high) are now skipped with a message, as in microphone mode, the C++ client (ExtractHws) and the HTML5 client. main sent their first word.
  • File mode keeps split(), so a tab or repeated spaces between a hotword and its weight still work. Microphone mode splits on single spaces; I left it unchanged to keep this PR to file mode.
  • test-python-websocket.yml now also runs tests/test_websocket_client_hotwords.py (added to its path filters and to the pytest command), next to the three websocket tests it already runs.
  • No open PR touches the client or the new test. git merge-tree --write-tree of this branch with the head of fix(runtime): disable debug mode in the Flask service #3917, the one open PR under runtime/python/websocket/, gives no conflicts.

The websocket protocol sends hotwords as a JSON object string that maps
each hotword to a weight, and funasr_wss_client.py builds that string
from the --hotword file in microphone mode. Since modelscope#2791 the file mode
(--audio_in) sent the first word of each line joined with commas, such
as "阿里巴巴,通义实验室", and dropped the weights.

The C++ servers load a client's hotwords and weights into the WFST
decoder only from a JSON object, and pass other strings to the NN
hotword model as they are. There the comma string is one hotword that
is not all Chinese characters and not in seg_dict, so it is skipped:
the client's hotwords had no effect.

Build the hotwords field in file mode as the microphone mode does. An
empty --hotword and one that is not a file are still sent unchanged.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

funasr_wss_client.py: with --audio_in, hotwords are sent as "w1,w2" instead of the protocol's JSON object, so the C++ server ignores them

1 participant