Video summary and supplementary results at GitHub link: įor decades, 2D animation has been a popular storytelling medium across manyĭomains, including entertainment, advertising and education.
CARTOON ANIMATOR 4 LIP SYNC OFFLINE
Over several competing methods, including those that only support offline Procedure that allows us to achieve good results with a very smallĪmount of hand-animated training data (13-20 minutes).Įxtensive human judgement experiments show that our results are preferred Of lookahead to produce accurate lip sync.
Our contributions include specific design decisions for our featureĭefinition and LSTM configuration that provide a small but useful amount With less than 200ms of latency (including processing time). Our system takes streaming audio as input and produces viseme sequences Generates live lip sync for layered 2D characters using a Long Short
In this work, we present a deep learning based interactive system that automatically A key requirement for live animationĪllows characters to respond naturally to other actors or theĪudience through the voice of a human performer. It requires a bit more coding, but it is one of the better ways to insure that the animation and audio remain in sync with each other.The emergence of commercial tools for real-time performance-basedĢD animation has enabled 2D characters to appear on live broadcastsĪnd streaming platforms. You might try looking at the possibility of playing the animations back dynamically based on the current position of the playing audio file. I rigged it up to use either Shape Keys for 3D models or Sprites for 2D animations. The solution I used stored the data in generated Animation Clips. Most uses of lip-syncing are in pre-scripted animation sequences.
I haven't played around with the Timeline much yet, but it seems a very appropriate place to put a lip-synced animation. It is a technical hurdle that has to be addressed, though, and don't be surprised if users start to ask you about it. On desktop you usually don't see this, as even average or low-power desktop and laptop machines have enough processing juice to avoid the audio decompression delay. If you start playing an animation driven by the engine at the same time as a compressed audio file, the animation will run normally while the audio gets the slight delay, throwing the two out of sync and causing a noticeable discrepancy. This delay also won't be consistent, different devices might have a greater or lesser delay based on their processing power. If your file is a compressed MP3 or OGG file, the decoding on mobile causes a slight delay when you first start playing it. But the vast majority of users aren't going to keep speech files uncompressed, especially for distribution on mobile. If your speech audio is stored as an uncompressed WAV file, there is no issue. The issue with mobile comes with the decoding of compressed audio sources. And if any of my past experience can help, so much the better. I'm always interested to see new solutions crop up.
I wasn't charging anyway, so it wasn't that big a deal. Sadly, I wasn't able to maintain it, and it dropped from the Asset Store. I actually released one of those other Unity tools.
0 kommentar(er)
