Amortized Bandwidth Learning for Kernel Density Estimation under Logarithmic Score

Choosing the right bandwidth for kernel density estimation (KDE) is a persistent challenge. Classical approaches either derive a fixed rule from asymptotic theory or recompute an optimization for every new data set, which can be costly and inflexible. A recent preprint proposes an alternative: learn a single mapping from samples to bandwidths that works well across many density‑estimation tasks by directly optimizing the logarithmic score.

What You Need to Know

The paper introduces an amortized learning framework. Instead of solving a fresh optimization for each sample, it trains a predictor that takes a data set (or a summary of it) and outputs a bandwidth. The predictor is learned by minimizing the expected negative log‑likelihood (logarithmic score) of the resulting KDE on a training distribution of tasks.

To keep learning stable when the underlying densities have different supports, the authors use a truncated‑and‑renormalized version of the KDE on a bounded interval. This modification prevents the estimator from assigning probability mass outside the interval and ensures the logarithmic score remains well‑defined across heterogeneous tasks.

Affine standardization is then applied: a selector trained on a single reference interval (e.g., [0,1]) can be transferred to any other bounded interval by applying an appropriate linear transformation to both the data and the bandwidth output. This property greatly reduces the amount of data needed to cover a wide range of practical scenarios.

Why It Matters

Bandwidth selection is often the bottleneck in applying KDE to real‑world problems such as anomaly detection, clustering, or probabilistic modeling. By learning a selector once and reusing it, practitioners can avoid repeated costly cross‑validation or rule‑of‑thumb calculations, speeding up pipelines especially when many similar data sets must be processed.

Moreover, the amortized approach adapts to the characteristics of the task distribution. If the training tasks reflect the variability encountered in deployment, the learned selector can outperform static rules that are derived under restrictive assumptions (e.g., Gaussianity or large‑sample asymptotics). This makes the method attractive for domains where data shapes vary widely but share a common bounded support after preprocessing.

Key Details

• The predictor can be a simple neural network or regression model that maps empirical moments (mean, variance, quantiles) to a bandwidth scalar.
• Training objective: minimize the average negative log score of the KDE over a batch of sampled data sets drawn from a task distribution.
• Truncation renormalizes the kernel density so that its integral over the bounded interval equals one, preserving proper probability semantics.
• Affine standardization uses the formula b̂ = a·b̂_ref + c where a and b scale and shift the reference bandwidth to the target interval.
• Experiments cover Gaussian synthetic data, a multi‑family benchmark (mixtures of Gaussians, Laplace, Beta), and randomized Gaussian tasks to demonstrate robustness across varying smoothness and support.

What’s Next

Future work could explore amortized learning for unbounded domains using adaptive truncation, or extend the framework to other loss functions such as the continuous ranked probability score. Additionally, investigating selector architectures that incorporate raw samples (e.g., set‑based networks) might further improve performance on highly irregular data.

📌 Source: Arxiv Ml