Today for AI
HOT RADAR
llmHEAT 9.5°

AI CURATED HOT STORY · 10/26/2026

Q Labs Introduces 'Dust': Gradient-Free Pretraining for Transformers

1 reports archived1 independent sourcesupdated 10/26/2026, 14:13:00
Synthesis & Latest Updates
1 Sources Cross-Validated

Startup lab Q Labs has introduced 'Dust', a zeroth-order optimization method claimed to be the first gradient-free technique competitive with backpropagation in Transformer pretraining. By adding Gaussian noise to activations to create a 'virtual population' and evaluating loss changes via forward passes, Dust estimates update directions without requiring differentiability.

LATEST/Startup lab Q Labs has introduced 'Dust', a zeroth-order optimization method claimed to be the first gradient-free technique competitive with backpropagation in Transformer pretraining. By adding Gaussian noise to activations to create a 'virtual population' and evaluating loss changes via forward passes, Dust estimates update directions without requiring differentiability.

Hourly heat curve

3 fully observed hours
Now
9.5
Peak
10.022:00
24h change
–

The line compares only sources observed throughout; gap hours are interpolated to keep the trend continuous.

Coverage timeline

Total 1 reports · Latest first
  1. 机器之心 (微信公众号)T2·78 pts

    Q Labs Introduces 'Dust': Gradient-Free Pretraining for Transformers

    • Dust is a zeroth-order optimization method that trains neural networks without computing gradients
    • The core mechanism involves injecting independent Gaussian noise into layer activations to form a parallel-evaluated 'virtual population'
    • Parameter updates are estimated by weighting and averaging noise perturbations based on resulting Loss changes
    • Aims to overcome backpropagation's limitations regarding non-differentiable modules, discrete operations, and external programs
Q Labs Introduces 'Dust': Gradient-Free Pretraining for Transformers | Today for AI