Repository navigation
Conversation
`add_noise` and `get_velocity` cast `alphas_cumprod` to the sample dtype before doing the math. In fp16 `alphas_cumprod[t]` rounds to 1 for t=0-1 (bf16: t=0-9), so `sqrt(1 - alphas_cumprod[t])` collapses to exactly 0 and `add_noise` hands the clean sample back untouched, while mixed-precision training still asks the model to predict the noise that should be in it. Now the schedule keeps its own dtype for the computation and only the two coefficients are cast to the sample dtype. fp32/fp64 inputs are unchanged bit for bit since `promote_types` keeps them at their own precision. The same pattern is in DDIM, PNDM, LCM, TCD, UnCLIP, SA-Solver and the cogvideox schedulers (all `# Copied from` DDPM), so `check_copies.py --fix_and_overwrite` carries the change through them. EulerDiscrete `get_velocity` and the Kandinsky img2img `add_noise` build the schedule the same way and are fixed here too.
|
Independent test of this PR's head (6975de3), run against the issue's own measurement — the noise coefficient recovered from Stock 0.43.0, DDPM and DDIM alike:
With this PR's two files:
The installed files were restored after the test (md5-verified) and the stock run reproduced the original zeros, so the comparison is against the same install. The fix direction works on the two flagship paths; a maintainer's review of the remaining schedulers in the PR is still the deciding step. |
|
Following up on my earlier test comment, which swapped in the DDPM and DDIM files only and left the remaining schedulers in this PR as the deciding step: I have now run every file in the PR individually, and censused the shipped 0.43.0 source for the same pattern anywhere else. Census. The cast-to-sample-dtype pattern occurs in exactly 21 places in the shipped release: the 20 scheduler sites in this PR's 12 scheduler files — Full run — stock 0.43.0 vs this PR's head (6975de3), all 13 source files swapped in. Recovered noise coefficient (
Repo tests. The ten scheduler test files covering the touched classes, run against this PR's head: 322 passed, 7 failed — and the same 7 (six The harness is one self-contained file that runs against any diffusers install and prints this table: https://github.com/proxy/gist.github.com/1OF1Research/a7c97670ddbd899fd11232d542ed645a Environment: diffusers 0.43.0 (shipped), torch 2.15.0, CPU. The swapped install was restored afterwards and verified by checksum, with the stock zeros reproduced on the restored files. On these measurements the PR is complete: it covers every site of the pattern in the release, fixes each one to the precision the dtype allows, and changes fp32 behaviour only where fp32 was itself losing precision. |
What does this PR do?
add_noiseandget_velocitycastalphas_cumprodto the sample dtype before using it. In fp16 that roundsalphas_cumprod[t]to 1 at t=0 and t=1 (bf16: t=0..9), sosqrt(1 - alphas_cumprod[t])is exactly 0 andadd_noisehands the clean sample back while a mixed-precision training step still asks the model to predict the noise that should be in its input.With
x0 = 0andnoise = 1the output ofadd_noiseis the noise coefficient, so it shows up directly:The schedule now keeps its own dtype for the math and only the two coefficients are cast:
promote_typesrather than "use the schedule dtype as is" becausealphas_cumprodcan itself be fp16 if the scheduler was built from fp16 betas, and a fp32 sample should still not be computed against a fp16 schedule.Scope:
add_noise/get_velocity, and the 15# Copied fromcopies in DDIM, DDIMParallel, DDPMParallel, LCM, TCD, PNDM, UnCLIP, SA-Solver and the two cogvideox schedulers.check_copies.py --fix_and_overwritedid the propagation, nothing else was touched.EulerDiscreteScheduler.get_velocity, which doesself.alphas_cumprod.to(sample).KandinskyImg2ImgPipeline.add_noise, which builds its own beta schedule and casts it the same way.fp32 and fp64 inputs are unchanged bit for bit (
promote_types(fp32, fp32) == fp32), so no existing output moves.Fixes #15277
Before submitting
documentation guidelines, and
here are tips on formatting docstrings.
Tests
Three new tests, all of which fail on
mainand pass here:test_add_noise_in_low_precisionandtest_get_velocity_in_low_precisionintests/schedulers/test_scheduler_ddpm.pytest_get_velocity_in_low_precisionintests/schedulers/test_scheduler_euler.pyBefore the change:
3 failed(fp16 coefficient[0.0, 0.0, 0.0221]against[0.0100, 0.0148, 0.0190]). After:76 passedfor the two files together. The rest of the schedulers that carry the copied code:315 passed, 3 skipped(ddpm,ddim,ddim_parallel,ddpm_parallel,pndm,lcm,unclip,sasolver,euler) plus34 passedfortcd.ruff checkandruff format --checkare clean.One thing I did not change: at very high timesteps the coefficient itself is not representable in fp16 (
0.99986becomes1.0), soget_velocitythere still loses the last digits. That is the output dtype, not the intermediate math, and fixing it would mean returning a wider tensor than the caller asked for.Who can review?
Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.
Schedulers: @yiyixuxu @dg845