• RuntimesUnity
  • [Feature Request] Better SpriteAtlas support

GamerXP I've tried looking into a way to use SpriteAtlas in more convenient way. Wrote my own atlas class that hold references to sprites themselves, then generate Atlas and materials based on the data of those sprites. It works surprisingly well, but there are few issues:

Great, glad to hear. Thanks for sharing your code, much appreciated.

GamerXP If texture is null - I can't generate materials and get uvs. I need some way to force reset cache of all users of current atlas asset to regenerate their skeletons after textures are loaded, or make skeleton somehow wait till atlas is ready.
Can it be somehow done without rewriting sources?

I'm not sure I understand what you mean by that. Could you please describe the above in more detail, how and which texture becomes null, and what you want the skeleton to wait for?

    Related Discussions
    ...

    Harald
    When sprite atlas is marked as addressable - it won't load automatically. You can have direct references to sprites, but they won't have texture and uv data till sprite atlas is loaded and bound.
    It's done with SpriteAtlasManager's static events. I've attached one of my scripts that uses them (it won't compile because of some of our internal classes references, but those parts can be safely removed)

    sceneatlasloader.txt
    11kB

    So, I plan on subscribing to SpriteAtlasManager.atlasRegistered method and, when all required sprites have their textures assigned, make spine reload the atlas.

    Thanks for the clarification and for sharing the additional code!

    GamerXP I need some way to force reset cache of all users of current atlas asset to regenerate their skeletons after textures are loaded, or make skeleton somehow wait till atlas is ready.
    Can it be somehow done without rewriting sources?
    ..
    So, I plan on subscribing to SpriteAtlasManager.atlasRegistered method and, when all required sprites have their textures assigned, make spine reload the atlas.

    To reload the skeleton and the atlas, you would normally first clear the atlas asset and then reinitialize the SkeletonAnimation (or SkeletonRenderer) components.

    SpineAtlasAsset targetAtlasAsset;
    ..
    targetAtlasAsset.Clear();
    
    SkeletonRenderer[] activeSkeletonRenderers = GameObject.FindObjectsOfType<SkeletonRenderer>();
    foreach (SkeletonRenderer skeletonRenderer in activeSkeletonRenderers) {
        if (skeletonRenderer.skeletonDataAsset.atlasAssets[0] == targetAtlasAsset) // null checks omitted for sake of brevity
            skeletonRenderer.Initialize(true);
    }

    E.g. DataReloadHandler uses similar calls to reload dependent scene skeletons, however without clearing the atlas asset first.
    EsotericSoftware/spine-runtimesblob/4.2/spine-unity/Assets/Spine/Editor/spine-unity/Editor/Utility/DataReloadHandler.cs

      Doesn't sound like something you'd use in runtime, but I'll try it for now. Thanks!

      Also, I was profiling the project yesterday and SkeletonMecanim component were taking quite a lot of performance (around 25% of whole load with 7 skeletons). Do you think it can be optimized by using Unity's job system + burst compiler? (I've optimized it for now by disabling updating when not visible)

      And what do you think about using 2D animation package's methods? I've looked into it a bit more. Basically, they send vertex deformation data to SpriteRenderer components using internal methods. There are some ways to access internal APIs without reflection, like Assembly Definition Reference. In theory, should be possible I think?

      If we combine sprite atlases, 2D animation API and Job system - Spine runtime's performance will skyrocket.

        Harald

        It's not exactly what is needed because I can only reset cache of things in scene at the moment. If object is not there - I can't reset the cache of the skeleton data asset. Like, when changing play mode I need to reset everything, but there are no objects in scene at that moment.

        Ok, I tried doing a workaround. May still have some issues in build though.
        But, for now, I added an AssetDatabase search and resetting skeleton assets that used loaded atlases. Seems to work at least.

        spinespriteatlasassetnew.txt
        7kB

        GamerXP Also, I was profiling the project yesterday and SkeletonMecanim component were taking quite a lot of performance (around 25% of whole load with 7 skeletons).

        First and foremost it depends on the complexity of your skeletons. If you have only 7 skeletons active, it sounds as if these might benefit from some optimizations on the Spine project side first:
        https://esotericsoftware.com/spine-unity-faq#Performance
        What are the current metrics of your skeletons? Is both Update and LateUpdate equally large? Also be sure to disable Deep Profiling, otherwise the numbers will be highly distorted.

        Do you think it can be optimized by using Unity's job system + burst compiler?

        Unfortunately that's not really feasible. The problem is that both require all data setup as plain structs.
        As the Spine core runtime is laid out in a different way, it's not really feasible to transfer the full feature set to structs in the near future, it would require a whole rewrite of the core runtime in DoD fashion.

        There was a thirdparty asset available on the Asset Store in the past which mapped the most basic Spine functionality to ECS and Tiny:
        https://assetstore.unity.com/packages/tools/animation/spine-animation-converter-for-ecs-and-tiny-181832
        Unfortunately it has been discontinued. If you're really interested, you might be able to ask the seller (UnnyNet) if he still sells or shares the package upon request.

        What we currently have in the making (see this issue ticket) is parallelization using normal threads. In our tests this cut down processing time of both processing animations (Update) and mesh generation (LateUpdate) on a quad-core machine to roughly a third. We're not done yet though, as the whole feature set (especially callbacks) needs some more adaptations to be safely usable in the parallelized environment, with as little change of the user code as possible.

        (I've optimized it for now by disabling updating when not visible)

        I'm not sure if you've implemented your own means of detecting that, but please note that there is already the Advanced - Update When Invisible component property available for this purpose.

        And what do you think about using 2D animation package's methods? I've looked into it a bit more. Basically, they send vertex deformation data to SpriteRenderer components using internal methods.

        What do you have in mind, what do you think would be beneficial of the 2D animation package? If you mean to utilize GPU skinning: we don't need the 2D animation package for that. We've considered implementing GPU skinning in the past, but as it would only cover a very limited subset of the Spine features, we've discarded the idea. Note that for GPU skinning to work, you need to have a fixed mesh and only classic weighed skeletal animation, so you could e.g. not change attachments or use mesh deformation animation (vertex animation).

        If you meant something else, please do let me know.

        There are some ways to access internal APIs without reflection, like Assembly Definition Reference. In theory, should be possible I think?

        Using reflection would not be a problem.

        If we combine sprite atlases, 2D animation API and Job system - Spine runtime's performance will skyrocket.

        Using Unity's Sprite Atlases brings no performance benefit. You can already use Spine's texture packer which will pack polygons more tightly, knowing the shape of the mesh attachment. So this would in theory make it slightly worse.

          Harald What do you have in mind, what do you think would be beneficial of the 2D animation package? If you mean to utilize GPU skinning: we don't need the 2D animation package for that. We've considered implementing GPU skinning in the past, but as it would only cover a very limited subset of the Spine features, we've discarded the idea. Note that for GPU skinning to work, you need to have a fixed mesh and only classic weighed skeletal animation, so you could e.g. not change attachments or use mesh deformation animation (vertex animation).

          If you meant something else, please do let me know.

          What I mean is that Spine is normally used for 2D games, together with regular sprites. 2D Animation package can batch skinned sprites together with regular sprites just fine. So, if we have both sprite-based atlases (like the one I'm writing) and 2D-animation-package based animations - it should be possible to optimize draw calls quite a lot. Not sure about 2D-animation package using GPU skinning - first time I've heard of them. It does use Burst though.
          It's not that big issue with side-view games since you can sort spine characters on same sorting group. But we develop isometric games, and each spine object breaks batching because it uses meshes.

          Harald First and foremost it depends on the complexity of your skeletons. If you have only 7 skeletons active, it sounds as if these might benefit from some optimizations on the Spine project side first:
          https://esotericsoftware.com/spine-unity-faq#Performance
          What are the current metrics of your skeletons? Is both Update and LateUpdate equally large? Also be sure to disable Deep Profiling, otherwise the numbers will be highly distorted.

          I think it was regular Update. Not sure about complexity since I don't know what modellers are doing there. Should be that complex I think. Guess I have to get sources and check if they made them too complex.

          Harald What we currently have in the making (see this issue ticket) is parallelization using normal threads. In our tests this cut down processing time of both processing animations (Update) and mesh generation (LateUpdate) on a quad-core machine to roughly a third. We're not done yet though, as the whole feature set (especially callbacks) needs some more adaptations to be safely usable in the parallelized environment, with as little change of the user code as possible.

          Well, that also should help quite a lot.

          Harald I'm not sure if you've implemented your own means of detecting that, but please note that there is already the Advanced - Update When Invisible component property available for this purpose.

          Yeah, I used those.

            GamerXP What I mean is that Spine is normally used for 2D games, together with regular sprites. 2D Animation package can batch skinned sprites together with regular sprites just fine. So, if we have both sprite-based atlases (like the one I'm writing) and 2D-animation-package based animations - it should be possible to optimize draw calls quite a lot.

            Now I see what you mean, batching is a different topic. I'm afraid that the spine-unity runtime will not likely support writing to a Unity SpriteRenderer or SpriteSkin component anytime soon, as the inherent limitations will likely exclude the majority of the Spine features (as described above).

            Batching is anyway limited to the same atlas texture and the same shader used. While a simple common Sprite shader could be used (again limiting special Spine functionality like tint black or blend modes at attachments), fitting multiple skeletons as well as all background assets on a single atlas texture seems rather unlikely. Admittedly, array textures could be used behind the scenes to support multiple large atlas textures in a single draw call, then again however the overhead grows, shrinking the benefits again.

            What you could use for better isometric batching is enable depth write (zwrite) on the background, if possible. Then you could draw the background at once and let the depth buffer sort your Spine skeletons in-between. Then you can't use semi-transparent border-pixels in background Sprites however.

            While it's not optimal, I doubt however that 7 skeletons would increase draw calls that much that it's a real problem (summing up to 15 draw calls), perhaps something else is going wrong.

            I think it was regular Update. Not sure about complexity since I don't know what modellers are doing there. Should be that complex I think. Guess I have to get sources and check if they made them too complex.

            Note that you can also use Spine - Import Data .. to import an exported skeleton .skel.bytes file in the Spine Editor.

              Harald Now I see what you mean, batching is a different topic. I'm afraid that the spine-unity runtime will not likely support writing to a Unity SpriteRenderer or SpriteSkin component anytime soon, as the inherent limitations will likely exclude the majority of the Spine features (as described above).

              Were there any big limitations? I initially thought of making a converter from Spine to 2D Animation, and only limitation I could think of was lack of feature to animate individual vertexes. But 2D Animation API itself accept deformations for each vertex, not requiring any bones. Sorting can be done with Sorting Group on root, and chaging order of each Sprite Renderer. But yeah, maybe there are some features we don't use so I didn't think of limiations for them.

              Harald fitting multiple skeletons as well as all background assets on a single atlas texture seems rather unlikely.

              Well, we separate background and objects to different atlases since they are drawn on separate layers. Can fit quite a lot of stuff on single 4k page. So it's not impossible.

              Harald What you could use for better isometric batching is enable depth write (zwrite) on the background, if possible. Then you could draw the background at once and let the depth buffer sort your Spine skeletons in-between. Then you can't use semi-transparent border-pixels in background Sprites however.

              Well, it's not very realistic to use when we have a lot of smooth gradients in alpha channels. It works well with pixel art only.

                GamerXP Were there any big limitations?

                It's mainly that the Unity 2D Animation system does not support bone transformations shear or disabled inheritance, and at least in the past didn't support vertex deformation animation.

                And that's if only half of the pipeline is handed over to Unity, only mesh generation after animations are evaluated and bones already positioned. If the complete pipeline including animation should be taken over by Unity's animation system including bone animation, pretty much all constraints except for IK constraints are unsupported. Also clipping, which however is best avoided anyway of possible.

                But 2D Animation API itself accept deformations for each vertex, not requiring any bones.

                Which method do you mean exactly?

                GamerXP I initially thought of making a converter from Spine to 2D Animation, and only limitation I could think of was lack of feature to animate individual vertexes.

                It would be great of course if you could give it a go! If you only need a limited subset and only support one Unity version, it might be perfectly fine for your use-case. Please do let us know about any progress or problems you encountered along the way.

                For us the imposed limitations and requirements (we also need to support a large set of Unity versions) plus the troubles with working with the closed API with internal methods (usually only working halfway as expected, plus methods being removed or changed any time) made us discard the idea pretty early. Also the fact that Unity often seals every class which would be nice to inherit, and holds lists to explicit class types instead of interfaces does not make things easier either.

                  GamerXP Well, we separate background and objects to different atlases since they are drawn on separate layers. Can fit quite a lot of stuff on single 4k page. So it's not impossible.

                  If you've got a texel-wise very small set of skeleton attachments, that makes sense of course.

                  GamerXP Well, it's not very realistic to use when we have a lot of smooth gradients in alpha channels. It works well with pixel art only.

                  Yes. Mentioned it just in case.

                  Harald Which method do you mean exactly?

                  From SpriteSkin's LateUpdate method:

                  InternalEngineBridge.SetDeformableBuffer(spriteRenderer, inputVertices.array);

                  Array seems to be just a pointer to memory, and it stores vertex offsets (vec3) and tangents (vec4). If what I think is correct - it should be possible to do any deformation effects Spine does.

                  Harald It would be great of course if you could give it a go! If you only need a limited subset and only support one Unity version, it might be perfectly fine for your use-case. Please do let us know about any progress or problems you encountered along the way.

                  Well, I had a working Spine -> Unity converter. It supported most of the things other than deformations. Was based on buggy outdated spine converter from Asset Store I fixed myself. I think it supported 4.0 or a bit earlier version.
                  But then artists decided they wanted deformations, and here I am.

                    GamerXP From SpriteSkin's LateUpdate method:

                    InternalEngineBridge.SetDeformableBuffer(spriteRenderer, inputVertices.array);

                    With the latest Unity version it's now in Deform(), not in LateUpdate(), which seems to be called just from OnPreviewUpdate() (it might be called from other locations outside of the 2D animation package though, just had a rough look at it).

                    From the code it does not look like vertex offsets and whether it performs any computation, as it's used in this code block:

                    var inputVertices = GetDeformedVertices(spriteVertexCount);
                    SpriteSkinUtility.Deform(sprite, gameObject.transform.worldToLocalMatrix, boneTransforms, inputVertices.array);
                    SpriteSkinUtility.UpdateBounds(this, inputVertices.array);
                    InternalEngineBridge.SetDeformableBuffer(spriteRenderer, inputVertices.array);

                    SpriteSkinUtility.Deform(sprite, gameObject.transform.worldToLocalMatrix, boneTransforms, inputVertices.array); seems to perform the deformation beforehand, and SetDeformableBuffer only accepting the deformed buffer. Normally skinning would be performed as a compute shader, transforming vertices which can then be handled in a similar way as non-deformed ones. The problem I see here is more how to replace the existing deformation step, how to inject vertex deformation in this existing pipeline.

                      GamerXP Well, I had a working Spine -> Unity converter. It supported most of the things other than deformations. Was based on buggy outdated spine converter from Asset Store I fixed myself. I think it supported 4.0 or a bit earlier version.

                      Do you mean the asset supporting ECS and Tiny I mentioned above, or some other asset?
                      https://assetstore.unity.com/packages/tools/animation/spine-animation-converter-for-ecs-and-tiny-181832

                      But then artists decided they wanted deformations, and here I am.

                      That's the problem we see in jumping on a limited API which is not intended to be used externally, it is unlikely to evolve alongside and will drift apart more and more.

                      Harald

                      SpriteSkinUtility.Deform method is used to fill the inputVertices array. Need to use custom Deform method based on Spine's data, and it should work in theory.

                        GamerXP SpriteSkinUtility.Deform method is used to fill the inputVertices array. Need to use custom Deform method based on Spine's data, and it should work in theory.

                        SpriteSkinUtility.Deform could theoretically be used to apply basic skinned animation (weighted bone transformation) on top of an already vertex-transformed mesh (or the other way round), yes, but with all limitations that I've mentioned above. It's just like using GPU skinning (or a worse CPU based parallelized variant), with all limitations that I've mentioned before. It may be perfectly fine for your use-case however, as mentioned.