Skip to content

funasr_wss_client.py: with --audio_in, hotwords are sent as "w1,w2" instead of the protocol's JSON object, so the C++ server ignores them #3923

Open

Description

@tanishqbaweja

In file mode (--audio_in), runtime/python/websocket/funasr_wss_client.py sends the first word of each line of the --hotword file, joined with commas, as the hotwords field of its first message (record_from_scp, funasr_wss_client.py:210-218):

hot_list = []
for line in hot_lines:
    words = line.strip().split()
    if not words:
        continue
    # Python AutoModel: 用逗号分隔多个热词
    hot_list.append(words[0])

hotword_msg = ",".join(hot_list)

A file with the lines 阿里巴巴 20 and 通义实验室 30 is sent as 阿里巴巴,通义实验室. The weights are dropped, and so is every word of a hotword after the first (hello world 10 is sent as hello). This came with #2791. Before it, file mode sent json.dumps(fst_dict), as microphone mode still does (record_microphone, :149-167).

The websocket protocol (runtime/docs/websocket_protocol.md) sends hotwords as a JSON object string that maps each hotword to a weight, such as {"阿里巴巴":20,"通义实验室":30}. Microphone mode, the C++ clients (funasr-wss-client.cpp:256-258, funasr-wss-client-2pass.cpp:256-258) and the HTML5 client (getHotwords in runtime/html5/static/main.js) send that.

The C++ servers parse the field as JSON (runtime/websocket/bin/websocket-server.cpp:314-358, websocket-server-2pass.cpp:469-513). Only a JSON object adds the client's hotwords and weights to the WFST decoder (FunWfstDecoderLoadHwsRes, used when the server runs with an LM) and its words to the NN hotword model. A string that is not JSON goes to the NN hotword model as it is. CompileHotwordEmbedding splits it on spaces, so 阿里巴巴,通义实验室 is one hotword. Because of the comma it is not all Chinese characters, so it is looked up whole in seg_dict, is not found, and is skipped (runtime/onnxruntime/src/paraformer.cpp:571-588, seg_dict.cpp:39-47; paraformer-torch.cpp:536-553 does the same). So the client's hotwords reach neither the WFST decoder nor the NN hotword model. With one hotword in the file, 阿里巴巴 reaches the NN hotword model, but not the WFST decoder, and without its weight.

The C++ SDK guides show this client with --hotword in this file format: runtime/docs/SDK_advanced_guide_offline.md:153 ("--hotword: Hotword file path, one line for each hotword(e.g.:阿里巴巴 20)"), SDK_advanced_guide_offline_zh.md:138, SDK_advanced_guide_offline_gpu.md:146 and SDK_advanced_guide_offline_gpu_zh.md:115.

Reproduction on main (3747f84), no model files. The real client sends 0.1 s of audio to a stand-in server on 127.0.0.1, which records the config message and acknowledges the end of input. Run it from the root of a FunASR checkout, with websockets installed (on Windows with PYTHONUTF8=1, because the client prints the hotwords to its redirected stdout):

import asyncio
import json
import sys
import tempfile
from pathlib import Path

import websockets

CLIENT = Path("runtime/python/websocket/funasr_wss_client.py").resolve()


async def main():
    work = Path(tempfile.mkdtemp())
    (work / "hotwords.txt").write_text("阿里巴巴 20\n通义实验室 30\n", encoding="utf-8")
    (work / "audio.pcm").write_bytes(b"\x00\x00" * 1600)  # 0.1 s of 16 kHz silence
    received = []

    async def stand_in_server(websocket):
        """Records the config message and acknowledges the end of input."""
        async for message in websocket:
            if isinstance(message, str):
                message = json.loads(message)
                if "hotwords" in message:
                    received.append(message["hotwords"])
                if message.get("is_end"):
                    await websocket.send(json.dumps({"is_end": True, "is_final": True}))

    async with websockets.serve(stand_in_server, "127.0.0.1", 0) as server:
        port = server.sockets[0].getsockname()[1]
        client = await asyncio.create_subprocess_exec(
            sys.executable, str(CLIENT), "--host", "127.0.0.1", "--port", str(port),
            "--ssl", "0", "--mode", "offline", "--audio_in", str(work / "audio.pcm"),
            "--hotword", str(work / "hotwords.txt"), "--log_dir", str(work / "logs"),
            stdout=asyncio.subprocess.DEVNULL,
        )
        await client.wait()

    hotwords = received[0]
    print(f"hotwords field sent in file mode: {hotwords!r}")
    try:
        print(f"json.loads(hotwords): {json.loads(hotwords)!r}")
    except json.JSONDecodeError as error:
        print(f"json.loads(hotwords): JSONDecodeError: {error}")


asyncio.run(main())

Output:

hotwords field sent in file mode: '阿里巴巴,通义实验室'
json.loads(hotwords): JSONDecodeError: Expecting value: line 1 column 1 (char 0)

Expected: '{"阿里巴巴": 20, "通义实验室": 30}', the string that microphone mode sends for the same file.

funasr_wss_server.py takes the words of a JSON object since #3880 (merged in #3892), and passes any other string to AutoModel unchanged. There the comma string is also one hotword and maps to <unk>, so a file with two or more hotwords has no effect with the Python server either.

Environment: Windows 11, Python 3.11.17, websockets 17.4, sources from main.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions