ALL INTELLIGENCE
智谱 AI 研究T1·Models

Zhipu AI Open-Sources GLM-TTS with Multi-Reward RL for Industrial Speech Synthesis

By 智谱 AIOriginal title: GLM-TTS:基于多奖励融合强化学习,实现工业级语音合成
78AI Score

Zhipu AI released and open-sourced GLM-TTS, an industrial-grade speech synthesis model available on Hugging Face and ModelScope. Leveraging multi-reward reinforcement learning and a two-stage generation paradigm, the model achieves '3-second' voice cloning and state-of-the-art emotional expression with only 100k hours of training data. Developers can access it via API or local deployment, supporting dialects, multi-emotion cloning, and automatic text-emotion matching.

Full Report

Contents14 sections

Today, we officially release GLM-TTS, an industrial-grade speech synthesis system, and make the model weights available on Hugging Face and ModelScope. Through innovations in data curation, foundational model architecture, supervised fine-tuning (SFT) for premium voice profiles, and reinforcement learning (RL), GLM-TTS achieves "3-second" voice cloning and superior text understanding capabilities with training on only 100,000 hours of data. It sets new open-source state-of-the-art (SOTA) benchmarks across multiple open-source test sets for character error rate (CER) and emotional expression. Starting immediately, users can experience GLM-TTS on Z.ai (audio.z.ai) and the ChatGLM app/web version (chatglm.cn), or call the model API via the BigModel open platform. We welcome developers, enterprises, and users to extensively test and integrate the system. In this technical blog post, we primarily introduce GLM-TTS's speech synthesis performance and technical overview.

Performance Showcase

Accents: Henan dialect, Sichuan dialect, Northeastern dialect Sichuan dialect Your browser does not support the audio element. Sichuan dialect - Synthesized Speech Your browser does not support the audio element. Northeastern dialect Your browser does not support the audio element. Northeastern dialect - Synthesized Speech Your browser does not support the audio element.

Multi-emotion Cloning: Happy, Sad, Dissatisfied, Worried, Tired Happy Your browser does not support the audio element. Sad Your browser does not support the audio element. Dissatisfied Your browser does not support the audio element. Worried Your browser does not support the audio element. Tired Your browser does not support the audio element.

Premium Voice Text Understanding: Automatically matches corresponding vocal emotions based on the sentiment of the text content. GLM-TTS-Premium Voice-happy Your browser does not support the audio element. GLM-TTS-Premium Voice-sad Your browser does not support the audio element. GLM-TTS-Premium Voice-angry Your browser does not support the audio element.

Overall System

Overall System

GLM-TTS is a high-quality speech synthesis system based on a two-stage generation paradigm, composed of two core modules: Text-to-token Auto-regression and Token-to-wav Diffusion.

1. Core Architecture and Workflow

The generation process of GLM-TTS is divided into two main stages:

  • Stage 1: Semantic Modeling (Text-to-token)
    • Input: Prompt text, prompt audio tokens, and target text for generation.
  • Processing: Generates the corresponding Whisper-VQ semantic token sequence based on auto-regression, ensuring content accuracy and coherence.
  • Stage 2: Acoustic Modeling and Waveform Reconstruction (Token-to-wav)
    • Input: Semantic tokens, voice timbre prompt.
  • Processing: Enters the Conditional Flow-matching model to predict the mel-spectrogram.
  • Output: Converts the mel-spectrogram into high-fidelity, high-quality speech waveforms using our proprietary 2D-Vocos vocoder.

2. Key Technical Breakthroughs

GLM-TTS has achieved multiple innovations in model training, pronunciation control, and expressiveness optimization:

  • Speech Tokenizer: Optimized based on the GLM-4-Voice Speech Tokenizer by introducing fundamental frequency constraints and increasing the Token Rate. This significantly enhances the pronunciation accuracy, naturalness, and expressiveness of synthesized speech.
  • Reinforcement Learning: Based on the GRPO algorithm framework, incorporating a multi-dimensional regularization reward mechanism that fuses CER, Similarity, Emotion, and Laughter (paralinguistic laughter). Combined with dynamic sampling and gradient clipping strategies, this significantly improves the anthropomorphic emotional expression capability of the speech.
  • Premium Voice Customization (LoRA): Adopts an optimized LoRA paradigm, requiring fine-tuning of more than 15% of parameters and only a few epochs to achieve results comparable to full-parameter fine-tuning. This drastically reduces the development cost and deployment barrier for single-speaker premium voice models.
  • Fine-grained Pronunciation Control (Phoneme-in): For high-precision scenarios such as educational assessment, we constructed a dynamically expandable dictionary of polyphonic characters and rare words. By utilizing a "Hybrid Phoneme + Text" input format, we effectively resolve issues related to polyphonic character confusion and rare word pronunciation.
  • 2D-Vocos Vocoder: Innovatively introduces 2D convolutional structures and DiT-like residual connections, enabling precise modeling of spectral details and temporal dynamics. This ensures high-fidelity waveform reconstruction while maintaining computational efficiency.

3. Technical Details and Overview

GLM-TTS provides comprehensive technical documentation covering model architecture, training procedures, inference optimization, and integration guidelines. The system supports real-time streaming synthesis with low latency, making it suitable for interactive applications. Detailed specifications include supported languages, accent variants, emotion categories, and customization options for enterprise deployments.

  • Audio Standardization and Coarse Segmentation: Unify heterogeneous audio formats into WAV, use VAD to segment valid clips, and concatenate them into long audio files of approximately 10 minutes for subsequent processing.
  • Background Noise Separation and Denoising: Use the Mel-Band Roformer model to separate background noise while preserving speech, followed by a proprietary denoising model to obtain clean speech.
  • Speaker Diarization and Concatenation: Perform multi-speaker diarization using the pyannote model. Apply amplitude normalization and concatenation to single-speaker segments, stopping when the concatenated length meets the threshold (>40s).
  • WER Filtering: For Chinese, use open-source ASR models (Paraformer/SenseVoice); for English, use open-source ASR models (Whisper/Reverb) to recognize text and calculate WER for double-checking. Retain high-quality data with WER < 5%.
  • Punctuation Optimization: Obtain character pronunciation durations via forced alignment between text and speech. Calculate pronunciation thresholds (mean + 2.6 * variance). Optimize punctuation based on the relationship between inter-character intervals and these thresholds (retain/add if the interval exceeds the threshold; remove/do not add otherwise).
  • Feature Extraction: Extract features from filtered, clean single-speaker audio to support subsequent model training.
  • Overall Engineering Optimization: Adopt the gRPC framework and server-worker architecture. Accelerate engineering processes for each submodule, leverage distributed clusters for large-scale data processing, and efficiently utilize multi-GPU memory and batch processing capabilities.

Speech Tokenizer Updates

GLM-TTS is built upon the Whisper-VQ speech tokenizer from GLM-4-Voice, with the following optimizations to enhance pronunciation accuracy, naturalness, and expressiveness:

  • Increased token generation rate (12.5Hz → 25Hz) and expanded vocabulary size (16k → 32k), optimizing stuttering issues at high speaking speeds and enhancing the synthesis naturalness of paralinguistic elements such as laughter and breathing;
  • Added a Pitch Estimator (PE) module to optimize pitch modeling accuracy, improving the prosody adherence of cloned TTS to reference audio (prompts);
  • Removed original causal constraints, eliminated block attention structures, and replaced causal convolutions with standard convolutions, breaking temporal dependencies to improve the accuracy of ASR and PE modules;
  • Expanded training data dimensions by adding massive dialect corpora to strengthen dialect understanding, and supplementing high-quality singing voice data to enrich vocal learning samples, thereby improving adaptability across diverse scenarios;

Experimental Results:

  • ASR Metrics: Dialect ASR significantly outperforms the GLM-4-Voice_tokenizer, with minor improvements in English ASR.

ASR Metrics

Note: Values represent WER or CER; lower is better. * indicates heavy-accented Mandarin from the Jiaoliao region.

  • TTS seed_tts_eval Chinese Metrics

- TTS seed_tts_eval Chinese Metrics

Multi-Reward Fusion Reinforcement Learning

Currently, reinforcement learning has not been widely applied in speech synthesis due to core bottlenecks such as the difficulty in designing reward mechanisms and issues like gradient vanishing or negative optimization during training. GLM-TTS introduces the GRPO reinforcement learning paradigm along with a series of fine-grained designs, significantly enhancing the foundational capabilities of pre-trained and SFT models (including core metrics like pronunciation accuracy and timbre similarity) and anthropomorphism (covering higher-level requirements like emotional expression precision and paralinguistic naturalness).

Multi-Reward Fusion Reinforcement Learning

The core solution and innovative highlights are as follows, adopting the GRPO algorithm framework overall and achieving performance leaps through three key innovations:

  • First, an innovative multi-dimensional regularized reward mechanism was designed, fusing four core rewards: CER (Character Error Rate), Sim (Similarity), Emotion, and Laughter (paralinguistic laughter). Through a hierarchical processing logic of "independent regularization of single rewards → weighted fusion → overall regularization," it addresses the pain point of significant distribution differences among different rewards;
  • Second, a dynamic sampling strategy was adopted. To address potential gradient vanishing caused by convergence of weighted rewards within a batch, automatic resampling is triggered (up to three times). Meanwhile, limiting the number of sampling attempts avoids negative model optimization introduced by poor samples, balancing training stability and efficiency;
  • Third, a dynamic gradient clipping parameter scheme was implemented, setting clip_high and clip_low as dynamic values that adaptively adjust with training steps. Early training tightens limits to prevent the model from quickly falling into reward hacking shortcuts; later stages gradually relax limits to grant the model greater exploration space. By reasonably setting parameter ranges, the number of clipped tokens remains stable, avoiding clipping failure or excessive restriction. Additionally, the design where clip_high is higher than clip_low effectively encourages the generation of low-probability tokens, greatly enhancing the anthropomorphic expressiveness of the speech.

Reinforcement learning employs a data construction mode combining training data and synthetic data, achieving nearly zero-cost data generation. Combined with training algorithm optimizations, this effectively avoids reward hacking, ensuring the authenticity of model performance and generalization capability, yielding ideal results even on unseen test sets. Experimental results:

  1. On the seed-tts-eval Chinese test set, our pre-training stage models (GLM-TTS, GLM-TTS_RL) focus on "improving zero-shot voice cloning pronunciation accuracy (CER) + timbre similarity (Sim)."

实验结果

  1. 在 SFT 阶段,我们聚焦文本理解能力和情感表达。在 CV3-eval-emotion 测试集的 text_related 中文数据集(文本自带情感倾向)中,GLM-TTS SFT 精品音色模型聚焦情感表达,对比市面商用 TTS 大模型展现出显著优势:
    1. 情感表达的领先性:GLM-TTS - 精品音色_RL 在 Happy(0.72)、Sad(0.52)、Angry(0.28)三类情感维度均取得 “测试集最佳表现”,平均情感得分(avg_emo)达 0.51;而阿里 Qwen3-TTS、百度超拟人 TTS、豆包 TTS-2.0 等商用模型,仅在 Happy 维度有部分表现,但在 Sad、Angry 等负向情感上几乎无有效输出(得分多为 0)。这一差异验证了我们的模型具备良好的文本情感理解能力,可根据文本内容自动适配对应情感风格。
  2. 字错误率的优势:在情感表现领先的同时,GLM-TTS CER 仍显著低于其他商用模型:GLM-TTS - 精品音色的 CER 仅为 1.33,GLM-TTS - 精品音色_RL 的 CER 为 1.68,均优于市面其他商用模型,实现了 “情感表达 + 发音准确度” 的双重领先。

实验结果-2

精品音色定制(LoRA)

在语音大模型的 SFT 流程中,全参微调受数据分布与质量不均影响,难以形成稳定通用范式,人力、工程及评测成本高昂,且无法满足小批量个性化精品音色定制的落地需求。因此,我们引入并优化了 LoRA 微调训练范式。

  • 高效与效果对标:仅需微调核心骨干网络中约 15% 左右的参数,配合约 100 个 epoch 的训练,即可实现与全参数微调相媲美的音色还原度与自然度。
  • 低成本定制: 仅需约 1 小时的单一说话人高质量音频数据,即可完成定制,有效避免了全参微调所需的大规模数据配比筛选和复杂的评测流程,大幅降低了开发成本和落地门槛。
  • 音色稳定性提升:实验结果表明,通过将微调参数比例精准控制在特定范围(如 15% 以上),音色泛化能力与跨场景稳定性得到显著增强,确保了定制音色的工业级可靠性。

精细化发音控制(Phoneme-in)

在教育评测、专业配音及有声读物等对发音准确性要求极高的严肃应用场景中,多音字(如“行”xíng/háng) 和生僻字的自动发音歧义是传统 TTS 模型的关键挑战。模型通常依赖内部语言模型进行推断,但缺乏外部强制干预机制,易导致发音错误和应用体验受损。我们提出了 Phoneme-in 可控读音增强能力,通过引入音素级(Phoneme-level)输入,实现对模型发音的精准、定向控制:

  • 关键组成部分:
    • 动态可控词典:专门针对易产生发音歧义的词条(主要是多音字和生僻字)及其所需的目标发音。它是实现定向音素替换的基础。
  • GLM-TTS 模型:最终的语音合成模型,能够接受 “Hybrid Phoneme + Text”的混合输入。
  • 训练阶段
    • 增强泛化性训练:对普通文本中的部分字词进行随机地进行 G2P。增强模型对音素输入序列的泛化适配性。
  • 关键词条保留:易有歧义的词保留原文进行训练。确保模型保留对纯文本输入的正常处理能力。
  • 推理阶段
    • 获取完整音素序列:首先,通过 G2P 模块对整个待合成文本获取完整的音素序列。得到未经用户干预的初始音素序列。
  • 定向音素替换:基于动态可控词典,系统识别文本中特定的多音字或生僻字,并将其与上述生成音素序列中的对应部分进行替换。实现对歧义发音的精准、定向控制。
  • 混合输入推理:将替换后的音素与文本一起,以 “Hybrid Phoneme + Text” 的混合输入形式传入 GLM-TTS 模型。保留文本原有韵律,同时输出字词发音精准可控的语音。

2D-Vocos 声码器

2D-Vocos 声码器

针对声码器核心效果提升,重点开展两方面优化,输入为 50Hz Mel 频谱、输出为 32k 高采样率 WAV:

  • 其一,模型结构升级,在 Vocos 基础上进行关键改进,将原有 1D 卷积替换为 2D 卷积,实现对频率子带的更高效建模,提升频谱特征解析精度;同时引入类 DiT 残差连接机制,可快速将输入频谱信息引入骨干 ConvNeXt 层,更充分地利用频谱条件,强化输入与输出的关联一致性。
  • 其二,训练数据增强,在高质量语音训练数据的基础上,额外融合大量高质量歌声数据,此类数据具备音域更广、发声方式更丰富的特性,不仅能有效提升最终32k高采样率WAV输出的音质,还能显著增强声码器对不同声音类型的适应性与训练稳定性。

开始使用 GLM‑TTS

1.在线体验

通过以下入口快速体验 GLM‑TTS 的合成效果:

  • audio.z.ai:上传文本或短语音 Prompt,生成专属声音;
  • 智谱清言 APP / 网页版:在对话中体验多风格朗读与音色克隆。

2.开放平台与 API

如希望直接接入线上业务,可以通过开放平台调用 GLM‑TTS 能力:

平台支持多种计费和 QPS 配置,覆盖从 Demo 试用到生产级大规模调用。

3.模型开源

我们将在主流开源社区开放 GLM‑TTS 相关资源(模型权重、推理脚本、示例项目等):

开发者可以基于主流推理框架,在 GPU 环境中快速部署 GLM‑TTS,并按需做二次开发。