Skip to content

fix(schedulers): keep the noise schedule out of fp16/bf16 in add_noise / get_velocity - #15300

Open
sunhouy wants to merge 1 commit into
huggingface:mainfrom
sunhouy:fix-ddpm-noise-dtype
Open

sunhouy wants to merge 1 commit into
huggingface:mainfrom
sunhouy:fix-ddpm-noise-dtype

Conversation

@sunhouy

@sunhouy sunhouy commented Nov 15, 2026

Copy link
Copy Markdown

What does this PR do?

add_noise and get_velocity cast alphas_cumprod to the sample dtype before using it. In fp16 that rounds alphas_cumprod[t] to 1 at t=0 and t=1 (bf16: t=0..9), so sqrt(1 - alphas_cumprod[t]) is exactly 0 and add_noise hands the clean sample back while a mixed-precision training step still asks the model to predict the noise that should be in its input.

With x0 = 0 and noise = 1 the output of add_noise is the noise coefficient, so it shows up directly:

dtype t=0 t=1 t=2
before, fp16 0.0 0.0 0.02210
before, bf16 0.0 0.0 0.0
after, fp16 0.010002 0.014832 0.018967
after, bf16 0.010010 0.014832 0.018921
fp64 reference 0.010001 0.014830 0.018966

The schedule now keeps its own dtype for the math and only the two coefficients are cast:

alphas_cumprod = self.alphas_cumprod.to(
    dtype=torch.promote_types(self.alphas_cumprod.dtype, original_samples.dtype)
)
sqrt_alpha_prod = (alphas_cumprod[timesteps] ** 0.5).to(original_samples.dtype)
sqrt_one_minus_alpha_prod = ((1 - alphas_cumprod[timesteps]) ** 0.5).to(original_samples.dtype)

promote_types rather than "use the schedule dtype as is" because alphas_cumprod can itself be fp16 if the scheduler was built from fp16 betas, and a fp32 sample should still not be computed against a fp16 schedule.

Scope:

  • DDPM add_noise / get_velocity, and the 15 # Copied from copies in DDIM, DDIMParallel, DDPMParallel, LCM, TCD, PNDM, UnCLIP, SA-Solver and the two cogvideox schedulers. check_copies.py --fix_and_overwrite did the propagation, nothing else was touched.
  • EulerDiscreteScheduler.get_velocity, which does self.alphas_cumprod.to(sample).
  • KandinskyImg2ImgPipeline.add_noise, which builds its own beta schedule and casts it the same way.

fp32 and fp64 inputs are unchanged bit for bit (promote_types(fp32, fp32) == fp32), so no existing output moves.

Fixes #15277

Before submitting

Tests

Three new tests, all of which fail on main and pass here:

  • test_add_noise_in_low_precision and test_get_velocity_in_low_precision in tests/schedulers/test_scheduler_ddpm.py
  • test_get_velocity_in_low_precision in tests/schedulers/test_scheduler_euler.py

Before the change: 3 failed (fp16 coefficient [0.0, 0.0, 0.0221] against [0.0100, 0.0148, 0.0190]). After: 76 passed for the two files together. The rest of the schedulers that carry the copied code: 315 passed, 3 skipped (ddpm, ddim, ddim_parallel, ddpm_parallel, pndm, lcm, unclip, sasolver, euler) plus 34 passed for tcd. ruff check and ruff format --check are clean.

One thing I did not change: at very high timesteps the coefficient itself is not representable in fp16 (0.99986 becomes 1.0), so get_velocity there still loses the last digits. That is the output dtype, not the intermediate math, and fixing it would mean returning a wider tensor than the caller asked for.

Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

Schedulers: @yiyixuxu @dg845

`add_noise` and `get_velocity` cast `alphas_cumprod` to the sample dtype
before doing the math. In fp16 `alphas_cumprod[t]` rounds to 1 for t=0-1
(bf16: t=0-9), so `sqrt(1 - alphas_cumprod[t])` collapses to exactly 0 and
`add_noise` hands the clean sample back untouched, while mixed-precision
training still asks the model to predict the noise that should be in it.

Now the schedule keeps its own dtype for the computation and only the two
coefficients are cast to the sample dtype. fp32/fp64 inputs are unchanged
bit for bit since `promote_types` keeps them at their own precision.

The same pattern is in DDIM, PNDM, LCM, TCD, UnCLIP, SA-Solver and the
cogvideox schedulers (all `# Copied from` DDPM), so `check_copies.py
--fix_and_overwrite` carries the change through them. EulerDiscrete
`get_velocity` and the Kandinsky img2img `add_noise` build the schedule
the same way and are fixed here too.
@1OF1Research

Copy link
Copy Markdown

Independent test of this PR's head (6975de3), run against the issue's own measurement — the noise coefficient recovered from add_noise with x = 0, noise = 1, so the output is sqrt(1 − ᾱ) itself. Environment: diffusers 0.43.0 (shipped), torch 2.15.0, CPU. I swapped in this PR's scheduling_ddpm.py and scheduling_ddim.py only; the other files share the same diff shape but were not individually run.

Stock 0.43.0, DDPM and DDIM alike:

  • fp16 coefficients at t=0,1: 0.0, 0.0 (expected 0.01000083, 0.01483041)
  • bf16 coefficients at t=0–9: all 0.0
  • fp32 at t=0,1: exact against the fp64 reference

With this PR's two files:

  • fp16 at t=0,1: 0.01000214, 0.01483154 — the expected values to fp16 precision
  • bf16 at t=0–9: 0.01000977 … 0.04345703 — non-zero throughout, tracking the expected curve to bf16 precision
  • fp32: unchanged, exact

The installed files were restored after the test (md5-verified) and the stock run reproduced the original zeros, so the comparison is against the same install. The fix direction works on the two flagship paths; a maintainer's review of the remaining schedulers in the PR is still the deciding step.

@1OF1Research

Copy link
Copy Markdown

Following up on my earlier test comment, which swapped in the DDPM and DDIM files only and left the remaining schedulers in this PR as the deciding step: I have now run every file in the PR individually, and censused the shipped 0.43.0 source for the same pattern anywhere else.

Census. The cast-to-sample-dtype pattern occurs in exactly 21 places in the shipped release: the 20 scheduler sites in this PR's 12 scheduler files — add_noise and get_velocity in DDPM, DDIM, DDIM CogVideoX, DDIM Parallel, DDPM Parallel, DPM CogVideoX, LCM and TCD; add_noise only in PNDM, SA-Solver and UnCLIP, which define no get_velocity; get_velocity only in EulerDiscrete — plus the Kandinsky img2img pipeline's own add_noise. This PR touches all 21. The rest of the schedulers package cannot carry the fault: the sigma-space add_noise family (DPM-Solver, UniPC, DEIS, LMS, Heun, KDPM2, EulerAncestral) never computes 1 - alphas_cumprod from a cast schedule — checked empirically for DPMSolverMultistep and UniPC, whose fp16 coefficient at the first schedule timestep is 0.014832 with no collapse — RePaint's add_noise raises NotImplementedError, and the consistency decoder casts pre-computed sqrt coefficients, which is the safe order.

Full run — stock 0.43.0 vs this PR's head (6975de3), all 13 source files swapped in. Recovered noise coefficient (add_noise with x = 0, noise = 1; get_velocity with sample = 1, noise = 0), each value compared against the scheduler's own float64 sqrt(1 - alphas_cumprod):

  • On stock, every one of the 21 sites returns exactly 0.0 at the first timesteps under at least one supported configuration. On the default schedules: fp16 t = 0–1 and bf16 t = 0–9 for the DDPM-shaped schedules, UnCLIP fp16 t = 0–2, LCM/TCD bf16 t = 0–1 with fp16 up to 7% off instead of zero. The two CogVideoX schedulers are the mildest on their default schedule — the SNR shift holds alphas_cumprod[0] at 0.99745, so the coefficient survives but lands up to 24% off in bf16 (0.0625 against a 0.050455 reference at t = 0); under a trained_betas schedule they return 0.0 in bf16 at t = 0–3 like the rest.
  • With this PR, no site returns 0.0 anywhere, and every probed coefficient matches the float64 reference to the precision the dtype itself allows: worst relative deviation 0.0004 in fp16 and 0.004 in bf16 across all 21 sites.
  • fp32 is bit-identical between stock and PR for 11 of the 13 files — coefficients and full random-input outputs alike. The CogVideoX pair differs, in the PR's favour: those two store their schedule in float64, so the old cast was also costing fp32 precision (t = 0 coefficient off by 3.5e-6 relative); with promote_types the fp32 result matches the float64 reference to ~1e-9.

Repo tests. The ten scheduler test files covering the touched classes, run against this PR's head: 322 passed, 7 failed — and the same 7 (six test_compatibles and Euler's test_beta_sigmas) fail identically on current main in the same environment, so the PR adds no new failures. The PR's three new low-precision tests pass.

The harness is one self-contained file that runs against any diffusers install and prints this table: https://github.com/proxy/gist.github.com/1OF1Research/a7c97670ddbd899fd11232d542ed645a

Environment: diffusers 0.43.0 (shipped), torch 2.15.0, CPU. The swapped install was restored afterwards and verified by checksum, with the stock zeros reproduced on the restored files.

On these measurements the PR is complete: it covers every site of the pattern in the release, fixes each one to the precision the dtype allows, and changes fp32 behaviour only where fp32 was itself losing precision.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

add_noise / get_velocity in DDPM-family schedulers add no noise at the first timesteps for fp16/bf16 samples

2 participants