use color to control material using controlnet

[FaizaAbbasi988]

1. Hello. I am working on a project to generate a living room accurate to the material. the goal of the project is to train a controlnet such that if i draw a control image of living room using several colors, it identifies the color and generate the corresponding material, eg if i input a control image with brown colored floor, it knows it has to generate a wooden floor and not any random material for floor.
2. The technique i can think of is to collect several images of living room and manually annotate them assigning specific colors to the specific material eg assign a brown color to whatever is wooden in the living room, be it couch, floor or chair, assign a blue color to whatever is glass eg windows, doors, table etc. using this annotation, generate masks of all the images and then train the control net using original images, masks and caption pairs. This idea is inspired by the following project that used color to generate the exterior of buildings and the model is accurate to generate the specific material. (https://github.com/AARG-FAN/Material-Controlnet)
   

Now the problem is that in outdoor, there is no duplication of the material whereas in indoor, a lot of objects have the same material eg floor, chair, couch all might have wooden material so generating a colored mask with the same color might end up losing the shape of the objects when a lot of them have the same color. this is where i find this technique infeasible

i would love to hear your suggestions on how i can achieve this objective

[asomoza]

Didn't know of that control net and I think your project idea its really cool. I think your problem can be solved by doing one of these two:

• Do a material/segmentation controlnet, since you're talking about just a living room, you can just use different colors for different materials of different parts/furniture of the living room. This will take a lot more training though so I wouldn't do it.
• Use a second controlnet (segment or depth) at inference time so the model can differentiate between objects.

Also I don't really know if you tried it yet, but this also can be mostly achieved with IP Adapters and masks but probably a controlnet will have better quality and precision.

[FaizaAbbasi988]

@asomoza thanks for your suggestion. I will try the second one. The problem with first one is that a lot of indoor objects have the same material which will make the segmentation poor

[asomoza]

oh I understood your first problem, what I meant is to use a "double" segmentation, so for example, for a wooden floor you can use a darker brown and for a wall, a lighter brown and so on for each different element of a living room that has the "wood" material. For example:

original	segmentation

[asomoza]

Also as a rough and fast experiment on what I was telling you that can be achieved with IP Adapters using just one wood simple texture:

depth	texture	base gen

floor	wall	both

[FaizaAbbasi988]

Hello Alvaro Thanks for replying back. i have realized that your understanding of diffusion models is way more deeper than mine. I am actually quite impressed. Is it possible for you to share your contact on any chat app eg whatsapp where i can discuss some aspects of this project with you?

…

On Mon, 5 Aug 2024 at 12:06 PM, Álvaro Somoza ***@***.***> wrote: Also as a rough and fast experiment on what I was telling you that can be achieved with IP Adapters using just one wood simple texture: depth texture base gen [image: 20240804235737] <https://github.com/user-attachments/assets/0c7d5508-5f10-43a2-b1dc-8b01189e5bb0> [image: 20240805000315] <https://github.com/user-attachments/assets/ecbd4815-26fa-4ec8-a614-110e89ea5585> [image: 20240804235820] <https://github.com/user-attachments/assets/b01d45a3-571c-4ce9-93ef-87d617d85881> floor wall both [image: 20240805000148] <https://github.com/user-attachments/assets/4dd581c7-4f4e-4389-a4f0-f1ef83c63f9e> [image: 20240805000208] <https://github.com/user-attachments/assets/61c388ff-c1f6-452e-b1d7-14f991df3e9a> [image: 20240804235841] <https://github.com/user-attachments/assets/0e8b6701-33ad-42b1-8c2a-a5b587357997> — Reply to this email directly, view it on GitHub <#9030 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/BH6BBJVFR3IFIACBLJDVWE3ZP324JAVCNFSM6AAAAABLZSKLV2VHI2DSMVQWIX3LMV43URDJONRXK43TNFXW4Q3PNVWWK3TUHMYTAMRTHA3TSMI> . You are receiving this because you authored the thread.Message ID: ***@***.***>

[asomoza]

Hello, I'm a maintainer of diffusers and I can help you and guide you if you have problems or questions but only in a open channel like this one, in the spirit of open source, we want all people to learn from the discussions.

I understand that this can be a personal private project, but feel free to ask specific questions if you're stuck or want some guidance.

[Gittulal]

Hello! Your project to generate a living room scene with accurate materials based on color input is intriguing. Given the challenge of overlapping materials, here's a refined approach:

Multi-channel Masking: Instead of using a single color channel, use multi-channel masks. Assign different shades or patterns within the same color spectrum to differentiate materials (e.g., various browns for wooden objects).

Semantic Segmentation: Utilize a semantic segmentation model to segment different objects and materials before training the control net. This helps in maintaining the shape and identity of each object.

Instance Segmentation: Combine semantic segmentation with instance segmentation to differentiate between objects of the same material type.

Hierarchical Annotations: Create hierarchical annotations where the primary mask color represents the material, and secondary markings or textures differentiate between objects.

Hybrid Approach: Use a combination of manual annotation and automated segmentation tools to enhance accuracy and reduce manual labor.

This approach can help you preserve the identity of objects while accurately generating the corresponding materials. Here 444 angel number meaning you can read something new.

[Gittulal]

Semantic Segmentation: Utilize a semantic segmentation model to segment different objects and materials before training the control net. This helps in maintaining the shape and identity of each object.

Instance Segmentation: Combine semantic segmentation with instance segmentation to differentiate between objects of the same material type.

Hierarchical Annotations: Create hierarchical annotations where the primary mask color represents the material, and secondary markings or textures differentiate between objects like Inshot Mod Apk