Repository navigation
fix(websocket): send file-mode hotwords as the protocol's JSON object - #3924
Open
tanishqbaweja wants to merge 1 commit into
Open
tanishqbaweja wants to merge 1 commit into
tanishqbaweja wants to merge 1 commit into
Conversation
The websocket protocol sends hotwords as a JSON object string that maps each hotword to a weight, and funasr_wss_client.py builds that string from the --hotword file in microphone mode. Since modelscope#2791 the file mode (--audio_in) sent the first word of each line joined with commas, such as "阿里巴巴,通义实验室", and dropped the weights. The C++ servers load a client's hotwords and weights into the WFST decoder only from a JSON object, and pass other strings to the NN hotword model as they are. There the comma string is one hotword that is not all Chinese characters and not in seg_dict, so it is skipped: the client's hotwords had no effect. Build the hotwords field in file mode as the microphone mode does. An empty --hotword and one that is not a file are still sent unchanged.
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Fixes #3923
Since #2791,
runtime/python/websocket/funasr_wss_client.pyin file mode (--audio_in) sent the--hotwordfile as the first word of each line joined with commas, such as阿里巴巴,通义实验室, without the weights. The websocket protocol (runtime/docs/websocket_protocol.md) sendshotwordsas a JSON object string that maps each hotword to a weight. This client sends that in microphone mode, and so do the C++ and HTML5 clients.The C++ servers (
websocket-server.cpp:314-358,websocket-server-2pass.cpp:469-513) take a client's hotwords and weights for the WFST decoder only from a JSON object. They pass any other string to the NN hotword model as it is. There阿里巴巴,通义实验室is a single hotword that is not all Chinese characters and is not inseg_dict, soCompileHotwordEmbeddingskips it (runtime/onnxruntime/src/paraformer.cpp:571-588). So with the C++ servers, which the SDK guides pair with this client's--hotword(runtime/docs/SDK_advanced_guide_offline.md:153), a hotword file with two or more hotwords had no effect in file mode, and a single hotword lost its weight.funasr_wss_server.pyreads the JSON object too since #3880 (merged in #3892). There the comma string was also one hotword, which maps to<unk>.This PR builds the field in
record_from_scpthe wayrecord_microphonedoes: a{"hotword": weight}dict from the lineshotword weight, sent asjson.dumps(fst_dict, ensure_ascii=False). Lines without an integer weight are skipped with the same "Please checkout format of hotwords" message. File mode keeps itssplit()and still skips blank lines silently. An empty--hotwordand one that is not a file are sent unchanged.The
hotwordsfield that the real client sends in file mode (stand-in server on 127.0.0.1, see Validation):--hotwordmain阿里巴巴 20and通义实验室 30阿里巴巴,通义实验室{"阿里巴巴": 20, "通义实验室": 30}阿里巴巴 20阿里巴巴{"阿里巴巴": 20}""""阿里巴巴 通义实验室(not a file)阿里巴巴 通义实验室阿里巴巴 通义实验室Type of change
Validation
The new
tests/test_websocket_client_hotwords.pycompilesrecord_from_scpout of the client, astests/test_websocket_file_finalization.pydoes (importing the client would parsesys.argv). It sends 0.1 s of PCM to a websocket that records what it is sent, with the end acknowledgement already queued, and checks thehotwordsof the config message:阿里巴巴 20and通义实验室 30gives{"阿里巴巴": 20, "通义实验室": 30};hello world 10,阿里巴巴and通义实验室 highgives{"hello world": 10}: the phrase stays whole, and the two lines without an integer weight are skipped, as in microphone mode;""and阿里巴巴 通义实验室(not a file) are sent unchanged.Windows 11, CPU, Python 3.11.17, pytest 9.2.0:
main(3747f84) client with the new test: 2 failed, 38 passed:assert '阿里巴巴,通义实验室' == '{"阿里巴巴": 20, "通义实验室": 30}'andassert 'hello,阿里巴巴,通义实验室' == '{"hello world": 10}'.-W error.I had no numpy, scipy or torch in this environment, so the two server tests ran with minimal stand-in modules for them on
PYTHONPATH(only the names the server imports);test-python-websocket.ymlinstalls the real ones.I also ran the real client as a process,
python funasr_wss_client.py --host 127.0.0.1 --port <port> --ssl 0 --mode offline --audio_in audio.pcm [--hotword ...], with websockets 17.4, against a stand-in server on 127.0.0.1 that records the config message and replies{"is_end": true, "is_final": true}(the script in the issue does this for the first case). It sent the values in the table above, and every run exited 0. With a stand-inpyaudiomodule,record_microphonesends the samehotwordsas file mode on this branch for the two test files and for阿里巴巴 20.Then I ran the client against the real
funasr_wss_server.pyofmain, both as processes on 127.0.0.1, with a stand-infunasr.AutoModelwhose offline model records thehotwordthat itsgeneratereceives. Every client run exited 0. The last column is what SeACo-Paraformer's owngenerate_hotwords_listmakes of that hotword, with the toy vocabulary of #3879 (<unk>= 3,<s>= 1):hotwordssentgenerategenerate_hotwords_list阿里巴巴 20main阿里巴巴阿里巴巴[[4, 5, 6, 6], [1]]阿里巴巴 20{"阿里巴巴": 20}阿里巴巴[[4, 5, 6, 6], [1]]阿里巴巴 20,通义实验室 30main阿里巴巴,通义实验室阿里巴巴,通义实验室[[3], [1]]阿里巴巴 20,通义实验室 30{"阿里巴巴": 20, "通义实验室": 30}阿里巴巴 通义实验室[[4, 5, 6, 6], [7, 8, 9, 10, 11], [1]]python -m compileall funasr examples testsblack --line-length=100(24.4.0, as in pre-commit) leaves the new test unchanged, and makes the same changes tofunasr_wss_client.pyonmainand on this branch, none on the changed lines. No real model was run.User impact
Users who run
funasr_wss_client.pywith--audio_inand a--hotwordfile against the C++ servers, as the SDK guides show, get their hotwords applied: in the WFST decoder with their weights when the server runs with an LM, and in the NN hotword model when it runs a hotword model. Before, a file with two or more hotwords had no effect, and a single hotword reached only the NN hotword model, without its weight. Withfunasr_wss_server.py, a file with two or more hotwords now reaches the model as those hotwords instead of one<unk>hotword, and a file with one hotword reaches it as before. Runs without--hotword, or with a--hotwordstring that is not a file, send the same message as before.Notes for reviewers
funasr_wss_server.pyreads the JSON object too and gives AutoModel its words, as the table in Validation shows. AutoModel takes no weights, so that server drops them, as it did before.阿里巴巴,通义实验室 high) are now skipped with a message, as in microphone mode, the C++ client (ExtractHws) and the HTML5 client.mainsent their first word.split(), so a tab or repeated spaces between a hotword and its weight still work. Microphone mode splits on single spaces; I left it unchanged to keep this PR to file mode.test-python-websocket.ymlnow also runstests/test_websocket_client_hotwords.py(added to its path filters and to the pytest command), next to the three websocket tests it already runs.git merge-tree --write-treeof this branch with the head of fix(runtime): disable debug mode in the Flask service #3917, the one open PR underruntime/python/websocket/, gives no conflicts.