We show how to estimate normalized probabilities by modifying the objective and architecture of diffusion models, and report several surprising observations on the properties of the learned probabilistic model.
We show that diffusion models can enter a generalization regime where the generated images are independent of the samples in the training set by relying on inductive biases towards geometric regularity in images.
We show how deep network weights can be compressed to a reduced set of summary statistics (the learned “features”) that (i) all networks share no matter their initialization and (ii) that allow regenerating weights with an equivalent performance.
We show that deep networks rely on phase collapses to classify images, rather than sparse coding as assumed in previous works, and introduce a simplified architecture with minimal learning while maintaining high accuracy.
