• Showcase

  • StableDiffusion to redraw attachments

    engleski

Earlier this year I posted a support request for my investigation into StableDiffusion (see https://esotericsoftware.com/forum/d/25752-how-to-export-psd-layers-of-every-pose-by-script/7)

But it ends up to be a more challenging task than expected, and I have to code a lot.
This is what I achieved https://www.bilibili.com/video/BV1fhWzetE4D/?vd_source=3ccdfd13f35470ceba3ca3ca6f1d35d1

Well, it's suboptimal. But however i try, it won't improve any further. So basically that's it. We have to wait for stronger AI.

    Related Discussions
    ...

    I'm intrigued. Please explain your process.

      SilverStraw I modify C++ spine runtimes, and draw UV. So I can get a mapping between RGB outcome of diffusion and the Attachments, and therefore update the Attachment to new clothes.
      Sorry I can't show the code, because the rights belongs to my company.

          yukiyuna Thanks for sharing this with us! This is interesting! I was a bit confused for a moment as to what the benefits of StableDiffusion were, but not only the color, but even the pattern of the costume and the ribbon attached to the rod had changed. Personally, I think it is interesting enough to know that it may eventually be possible to realize the function of automatically creating skins in this way. 🙂

            yukiyuna Neat. I understand the process without getting too detailed. Can you explain why is is sub-optimal? What is your generative tool lacking?

                SilverStraw
                To be general, lack of control. My first attempt was a total failure, because I don't know ControlNet at that time. After applying ControlNet, I can control the edge of different attachment's edge. But still, edge is ambiguious hint, and sometimes diffusion misunderstand it (eg. flesh color on clothes is misunderstand clothes to body).

                Also lack of control of consistency of different frames of animation. Do you notice that, when Emiliya raise her left arm, the color of the sleeve's changed? I have no way to tell diffusion that, they should be of same color pattern.

                  Despite ControlNet difficulties, this is still super cool!