Skip to content

KolorsPAGPipeline: IP-Adapter inputs raise TypeError / ValueError #15347

Open

Description

@chsu-amd

Describe the bug

KolorsPAGPipeline (pipeline_pag_kolors.py) fails when IP-Adapter inputs are passed. The result of prepare_ip_adapter_image_embeds() is assigned to image_embeds instead of ip_adapter_image_embeds:

  • with ip_adapter_image, the following loop iterates ip_adapter_image_embeds, which is still None -> TypeError;
  • with ip_adapter_image_embeds, the UNet gets added_cond_kwargs["image_embeds"] = image_embeds, i.e. only the last tensor of that loop instead of the list. With two IP-Adapters this raises ValueError; with one it runs, but through the deprecated "tensor instead of list" fallback (FutureWarning).

The block was added in #8934: it took the PAG loop over ip_adapter_image_embeds from StableDiffusionXLPAGPipeline but kept KolorsPipeline's image_embeds variable (KolorsPipeline has no such loop, so it works there). The fix is to do what StableDiffusionXLPAGPipeline does: assign to ip_adapter_image_embeds and pass that list to the UNet. I have the fix with an IPAdapterTesterMixin test class for Kolors PAG and am happy to open a PR.

Reproduction

Tiny random components and two IP-Adapters with random weights; runs on CPU:

import numpy as np
import torch
from PIL import Image
from transformers import CLIPImageProcessor, CLIPVisionConfig, CLIPVisionModelWithProjection

from diffusers import AutoencoderKL, EulerDiscreteScheduler, KolorsPAGPipeline, UNet2DConditionModel
from diffusers.pipelines.kolors import ChatGLMModel, ChatGLMTokenizer


def tiny_unet():
    # Tiny random Kolors UNet (same layout as tests/pipelines/pag/test_pag_kolors.py)
    torch.manual_seed(0)
    return UNet2DConditionModel(
        block_out_channels=(2, 4),
        layers_per_block=2,
        sample_size=32,
        in_channels=4,
        out_channels=4,
        down_block_types=("DownBlock2D", "CrossAttnDownBlock2D"),
        up_block_types=("CrossAttnUpBlock2D", "UpBlock2D"),
        attention_head_dim=(2, 4),
        use_linear_projection=True,
        addition_embed_type="text_time",
        addition_time_embed_dim=8,
        transformer_layers_per_block=(1, 2),
        projection_class_embeddings_input_dim=56,
        cross_attention_dim=8,
        norm_num_groups=1,
    )


def tiny_pipe():
    return KolorsPAGPipeline(
        vae=AutoencoderKL(
            block_out_channels=[32, 64],
            down_block_types=["DownEncoderBlock2D"] * 2,
            up_block_types=["UpDecoderBlock2D"] * 2,
            latent_channels=4,
        ),
        text_encoder=ChatGLMModel.from_pretrained("hf-internal-testing/tiny-random-chatglm3-6b", dtype=torch.float32),
        tokenizer=ChatGLMTokenizer.from_pretrained("hf-internal-testing/tiny-random-chatglm3-6b"),
        unet=tiny_unet(),
        scheduler=EulerDiscreteScheduler(),
        image_encoder=CLIPVisionModelWithProjection(
            CLIPVisionConfig(
                hidden_size=8,
                intermediate_size=16,
                num_attention_heads=2,
                num_hidden_layers=1,
                image_size=32,
                patch_size=8,
                projection_dim=8,
            )
        ),
        feature_extractor=CLIPImageProcessor(size=32, crop_size=32),
        pag_applied_layers=["mid"],
    )


# Two random IP-Adapters in the original IP-Adapter layout
pipe = tiny_pipe()
dim = pipe.unet.config.cross_attention_dim
ip_layers, key_id = {}, 1
for name in pipe.unet.attn_processors:
    if name.endswith("attn1.processor"):
        continue
    if name.startswith("mid_block"):
        hidden = pipe.unet.config.block_out_channels[-1]
    elif name.startswith("up_blocks"):
        hidden = pipe.unet.config.block_out_channels[::-1][int(name[len("up_blocks.")])]
    else:
        hidden = pipe.unet.config.block_out_channels[int(name[len("down_blocks.")])]
    ip_layers[f"{key_id}.to_k_ip.weight"] = torch.randn(hidden, dim)
    ip_layers[f"{key_id}.to_v_ip.weight"] = torch.randn(hidden, dim)
    key_id += 2
image_proj = {
    "proj.weight": torch.randn(4 * dim, dim),
    "proj.bias": torch.zeros(4 * dim),
    "norm.weight": torch.ones(dim),
    "norm.bias": torch.zeros(dim),
}
ip_adapter = {"image_proj": image_proj, "ip_adapter": ip_layers}
pipe.load_ip_adapter([ip_adapter, ip_adapter], subfolder=["", ""], weight_name=["", ""], image_encoder_folder=None)
pipe.set_ip_adapter_scale([0.6, 0.6])

ip_inputs = {
    "ip_adapter_image": {"ip_adapter_image": [Image.fromarray(np.zeros((32, 32, 3), dtype=np.uint8))] * 2},
    "ip_adapter_image_embeds": {"ip_adapter_image_embeds": [torch.randn(2, 1, dim)] * 2},  # (negative, positive)
}
for name, kwargs in ip_inputs.items():
    try:
        image = pipe(
            "a photo of a cat",
            guidance_scale=5.0,
            pag_scale=3.0,
            num_inference_steps=2,
            height=64,
            width=64,
            output_type="np",
            **kwargs,
        ).images
        print(f"{name}: OK, output {image.shape}, finite {bool(np.isfinite(image).all())}")
    except Exception as e:
        print(f"{name}: {type(e).__name__}: {e}")

Context

I was running documented features (PAG, IP-Adapter, LoRA) across the Kolors pipelines on current main, looking for bugs in the library with an AI agent (Claude Code). The script above was run as shown, on main (2bb9468) and with the fix, on a machine with an RTX 5090 (the script itself runs on CPU), with PYTHONNOUSERSITE=1 and HF_HOME set to a separate cache. The logs below are its output on main; with the fix both lines print OK.

The same failure happens with the real weights on the RTX 5090 (Kwai-Kolors/Kolors-diffusers fp16, 1024x1024, pag_applied_layers=["mid"], pag_scale=1.5): the IP-Adapter example from the Kolors docs, run with KolorsPAGPipeline, fails with the same TypeError.

Logs

ip_adapter_image: TypeError: 'NoneType' object is not iterable
ip_adapter_image_embeds: ValueError: image_embeds must have the same length as image_projection_layers, got 1 and 2

System Info

  • 🤗 Diffusers version: 0.44.0.dev0
  • Platform: Windows-11-10.0.26200-SP0
  • Running on Google Colab?: No
  • Python version: 3.12.16
  • PyTorch version (GPU?): 2.14.1+cu130 (True)
  • Huggingface_hub version: 1.32.0
  • Transformers version: 5.16.1
  • Accelerate version: 1.9.0
  • PEFT version: 0.21.3
  • Safetensors version: 0.8.0
  • xFormers version: not installed
  • Accelerator: NVIDIA GeForce RTX 5090, 32607 MiB
  • Using GPU in script?: No (the script runs on CPU)
  • Using distributed or parallel set-up in script?: No

Who can help?

@yiyixuxu

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions