Safety Checker for SDXL model

[acs107102]

Hello,

I know that SDXL remove the safety checker for SDXL and someone said it's already safe to use as well. However, I can still generate some NFSW images. Does anyone know how I can avoid this, thanks.

[tolgacangoz]

Hi @acs107102,
Can you try to run run_safety_checker function in the repo for SD-XL's output?

[acs107102]

Hello @tolgacangoz

I don't want to generate NSFW image. I want to be able to check the safety like this.

image, has_nsfw_concept = self.safety_checker(
                images=image, clip_input=safety_checker_input.pixel_values.to(dtype)
)

Basically, I want to have this parameter has_nsfw_concept

This is the code I have.

from diffusers import (
    StableDiffusionXLControlNetPipeline,
    ControlNetModel,
    AutoencoderKL,
    DDPMScheduler
)
import numpy as np
import torch

import cv2
from PIL import Image

image = "image.png"
image = np.array(image)

image = cv2.Canny(image, 100, 200)
image = image[:, :, None]
image = np.concatenate([image, image, image], axis=2)
canny_image = Image.fromarray(image)

controlnet_canny = ControlNetModel.from_pretrained(
            "diffusers/controlnet-canny-sdxl-1.0", torch_dtype=torch_dtype
        ).to(device)

vae = AutoencoderKL.from_pretrained(
            "madebyollin/sdxl-vae-fp16-fix", torch_dtype=torch_dtype
)

pipe = StableDiffusionXLControlNetPipeline.from_pretrained(
                "stabilityai/sdxl-turbo",
                controlnet=controlnet_canny,
                vae=vae,
                torch_dtype=torch.float16
)
pipe.scheduler = DDPMScheduler.from_config(pipe.scheduler.config)
pipe = pipe.to("cuda")
prompt = "(nsfw things)"
generator = torch.manual_seed(0)
image = pipe(image=canny_image, prompt, num_inference_steps=1, generator=generator).images[0]

[tolgacangoz]

This is the run_safety_checker(...) function:

diffusers/src/diffusers/pipelines/stable_diffusion/pipeline_stable_diffusion.py

Lines 553 to 565 in 2c25b98

	def run_safety_checker(self, image, device, dtype):
	if self.safety_checker is None:
	has_nsfw_concept = None
	else:
	if torch.is_tensor(image):
	feature_extractor_input = self.image_processor.postprocess(image, output_type="pil")
	else:
	feature_extractor_input = self.image_processor.numpy_to_pil(image)
	safety_checker_input = self.feature_extractor(feature_extractor_input, return_tensors="pt").to(device)
	image, has_nsfw_concept = self.safety_checker(
	images=image, clip_input=safety_checker_input.pixel_values.to(dtype)
	)
	return image, has_nsfw_concept

Initialize feature_extractor with CLIPImageProcessor from this pretrained checkpoint:

diffusers/src/diffusers/pipelines/stable_diffusion/pipeline_stable_diffusion.py

Lines 169 to 170 in 2c25b98

	feature_extractor ([`~transformers.CLIPImageProcessor`]):
	A `CLIPImageProcessor` to extract features from generated images; used as inputs to the `safety_checker`.

Then, initialize safety_checker with StableDiffusionSafetyChecker from this pretrained checkpoint:

diffusers/src/diffusers/pipelines/stable_diffusion/pipeline_stable_diffusion.py

Lines 165 to 168 in 2c25b98

	safety_checker ([`StableDiffusionSafetyChecker`]):
	Classification module that estimates whether generated images could be considered offensive or harmful.
	Please refer to the [model card](https://huggingface.co/runwayml/stable-diffusion-v1-5) for more details
	about a model's potential harms.

[acs107102]

Thanks @tolgacangoz for your help.

However, When I type the NSFW words, it pop up the an error like this. Do you know how to fix it ?

File "/home/aaa/SDXL/pipelines/custom2.py", line 351, in predict
    image, has_nsfw_concept = self.run_safety_checker(image)
  File "/home/aaa/SDXL/pipelines/custom2.py", line 282, in run_safety_checker
    image, has_nsfw_concept = self.safety_checker(
  File "/home/aaa/anaconda3/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/home/aaa/anaconda3/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/aaa/anaconda3/lib/python3.9/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
    return func(*args, **kwargs)
  File "/home/aaa/anaconda3/lib/python3.9/site-packages/diffusers/pipelines/stable_diffusion/safety_checker.py", line 89, in forward
    if torch.is_tensor(images) or torch.is_tensor(images[0]):
TypeError: 'Image' object is not subscriptable

I add run_safety_checker after I generated the image (after the end of the code I provided). Is it the right position ?
Thanks again !

[tolgacangoz]

Here is my implementation:

from diffusers import StableDiffusionXLPipeline
from diffusers.pipelines.stable_diffusion import StableDiffusionSafetyChecker
from transformers import CLIPImageProcessor
import torch
from torchvision.transforms.functional import pil_to_tensor, to_pil_image
from PIL import Image


pipe = StableDiffusionXLPipeline.from_pretrained(
    "stabilityai/stable-diffusion-xl-base-1.0", torch_dtype=torch.float16, variant="fp16"
).to("cuda")
prompt = "(nsfw things)"
image = pipe(prompt).images[0]

def run_safety_checker(image: Image.Image, device="cuda", dtype=torch.float16):
    feature_extractor = CLIPImageProcessor.from_pretrained("openai/clip-vit-base-patch32")
    safety_checker = StableDiffusionSafetyChecker.from_pretrained(
        "CompVis/stable-diffusion-safety-checker").to(device)

    safety_checker_input = feature_extractor(image, return_tensors="pt").to(device)
    image, has_nsfw_concept = safety_checker(
        images=pil_to_tensor(image).unsqueeze(0), clip_input=safety_checker_input.pixel_values.to(dtype)
        )
    return to_pil_image(image.squeeze()), has_nsfw_concept

image, has_nsfw_concept = run_safety_checker(image)
has_nsfw_concept

This implementation is open to suggestions.