利用语音基础模型语言选择性维度实现轻量级语码转换检测:无需转录直接识别多语言混合
提出一种直接从语音中检测语码转换(Code-Switching)的轻量级方法,通过识别预训练语音编码器中的“语言选择性隐藏状态维度”并聚合帧级激活值来判定双语存在。该方法在 CS-FLEURS 数据集上使 Whisper-small 编码器达到 0.798 的平均 F1 分数,优于传统 CNN 和 HuBERT 基线,为构建更具代表性的多语言语料库提供了高效工具。
报道正文
Code-switching (CS), the alternation between languages within an utterance, is common in multilingual speech but underrepresented in pretraining corpora. Increasing CS coverage requires finding relevant utterances in large speech collections so that they can be included in training data. One way to find such utterances is to transcribe the speech and detect language alternation in the text, but this depends on obtaining accurate multilingual transcriptions. We propose a lightweight method that detects CS directly from speech. It uses frame-level language information in pretrained speech encoders to determine whether both languages are present within an utterance. Specifically, we identify language-selective hidden-state dimensions using monolingual recordings and aggregate their frame-level activations into a CS prediction. With a small CS set used to tune a decision rule, we show that the Whisper-small encoder achieves a mean F1 of 0.798 across 14 language pairs in CS-FLEURS, compared with 0.776 for a CNN and 0.751 for a HuBERT linear baseline trained for monolingual language identification. Our findings suggest that pretrained speech encoders already encode the information needed to identify CS utterances, and they provide a lightweight tool for building more representative multilingual corpora.