Allen AI 发布 DiScoFormer:统一密度与分数估计
Allen AI 推出 DiScoFormer,一种基于 Transformer 的通用模型,能同时准确估计数据分布的密度和分数。该模型无需针对特定问题重新训练,显著降低了高维场景下的计算成本并提升了跨分布泛化能力。
报道正文
本文目录3 个章节
DiScoFormer:一个 Transformer 统一处理密度与得分,跨越不同分布
📄 技术报告: arxiv.org/abs/2511.05924
机器学习和科学领域的许多问题最终都归结为同一项任务:给定一组数据点,恢复其背后的分布——即哪些取值常见,哪些取值罕见。确定该分布意味着估计两个量:分布的密度(density)以及随着维度增加更为有用的得分(score)。密度可以看作直方图的平滑版本——在数据点聚集处较高,在稀疏处较低。得分是 log-密度的梯度,指向密度上升最快的方向:沿得分方向移动一个数据点,它将趋向于更可能的区域。
基于扩散模型的生成式 AI(如 Stable Diffusion 和 DALL-E 等 AI 图像生成器背后的技术)从随机噪声开始,反复跟随得分场,将噪声转化为逼真的图像。同样的得分也驱动着贝叶斯采样以及用于模拟等离子体等系统的粒子仿真。
从有限样本中提取密度和得分极具挑战性,而现有的工具往往需要在泛化能力和准确性之间做出权衡。一种经典方法是核密度估计(KDE),它根据周围的数据点计算任意位置的密度:数据点越近、数量越多,密度越高。KDE 无需训练且适用于任何分布,但其准确性随维度增加而急剧下降。另一种方法是神经得分匹配模型,这类经过训练以预测得分的模型即使在高维空间中也能保持准确,但每个模型都需要学习特定的分布,若应用于新分布则必须从头重新训练。
We introduce a new solution called the DiScoFormer (Density and Score Transformer)—one model that, given a set of data points, estimates both the density and the score of the distribution in a single forward pass without retraining.
Training a transformer for density and score estimation
DiScoFormer maps an entire sample to the density and score of the distribution behind it using stacked layers of transformer blocks. The model utilizes cross-attention, which allows it to evaluate density and score at any point—not just where you have data. Score and density share a mathematical relationship: score is the gradient of the logarithm of density. We leverage this by having a shared backbone with two output heads, one for the density and one for the score.
This coupling does more than save parameters. The score head has to match the gradient of the log-density head at every query, so any gap between them is a label-free consistency loss. We use this at inference—hold the context fixed, take a few gradient steps on that consistency loss, and DiScoFormer adapts itself to an out-of-distribution input on the spot, no ground-truth density or score required.
There's a mathematical reason why the transformer architecture fits this task. Kernel density estimation has a single bandwidth—how far each point's influence reaches, fixed in advance and applied identically everywhere. Attention is a strict generalization of it: we analytically show that a single attention head's weights are nearly a Gaussian kernel over the data, so one cross-attention block can already reproduce KDE's density and score. From there the model goes further, learning several such scales at once and adapting them to the data. DiScoFormer doesn't discard the classical method for a black box but instead includes KDE as a special case and improves on it.
What data did we use to train DiScoFormer? We relied on Gaussian Mixture Models for two primary reasons. Firstly, GMMs are universal density approximators—with enough components they match essentially any smooth distribution to arbitrarily small error. Secondly, GMMs have closed-form densities and scores, so we always have an exact target to supervise against. We employ both of these properties by drawing a new GMM for every batch, giving the model virtually unlimited examples of target distributions and supervising each against a given GMM's exact density and score.
Performance
Across the board, DiScoFormer beats KDE at both density and score estimation, and the gap widens exactly where KDE struggles. In 100 dimensions, it isn't close—against the best hand-tuned KDE, it cuts score error by about 6.5x and density error by more than 37x, and it keeps improving as you add samples, while KDE runs out of memory. It also travels far outside its training data, staying accurate on mixtures with more modes than it ever saw during training and on non-Gaussian shapes like the Laplace and Student-t. KDE's main advantage remains speed, especially when datasets are small.
The part about DiScoFormer that we find most promising is that score estimation is a shared dependency across many fields, such as generative modeling, Bayesian inference, and scientific computing. A pretrained, plug-in estimator that stays accurate in high dimensions and removes the need to retrain per problem could cut that cost across all of them at once—one model, reused everywhere score and density show up.
We encourage you to read our technical report for more details.

