I started experimenting with a new musical sound this past week and it turned out unexpectedly well. And it also offered the opportunity to experiment further with both the new MiniMaxH3 and Seedance 2.5 engines.
The retro 80s sound is heavily influenced by the original sound of The Cure, which is to say, the pre-Lovecats version. The Cure’s single Killing an Arab was famously based on a passage from Albert Camus’s The Stranger, although if you’ve read the novel, then you know it’s just a single scene in the book, so it’s hardly the philosophical distillation that some music historians would apparently have had us believe.
Anyhow, being by nature amenable to certain elements of Camus’s Absurdist philosophy, which in practical terms can be reasonably described as a post-nihilist para-Stoicism, I thought it would make sense to pay philosophical as well as musical homage, and turned to Camus’s The Myth of Sisyphus for thematic inspiration.
I leave Sisyphus at the foot of the mountain! One always finds one’s burden again. But Sisyphus teaches the higher fidelity that negates the gods and raises rocks. He too concludes that all is well. This universe henceforth without a master seems to him neither sterile nor futile. Each atom of that stone, each mineral flake of that night-filled mountain, in itself forms a world. The struggle itself toward the heights is enough to fill a mans heart. One must imagine Sisyphus happy.
That short philosophical work provided the conceptual foundation for The Myth by Silenziosa. It’s the second of a ten-song album that will be crowdfunded in September and released in December, and this is the video for what will be the first single.
This was by far the least expensive video to produce yet by virtue of the fact that as part of its introductory offer, Seedance was offering 5-second 480p clips of its new 2.5 engine for free, and it turned out that 480p video with characters is often superior to 720p video that starts from a single image frame. Good to know. I wasn’t thinking I’d be able to produce another video before my monthly credits refreshed, but as it turned out, I was able to get it done with 1,800 to spare. In real dollar terms, it cost $8.47 to produce, which would be impossible to recreate, but even if the 480p Seedance clips had cost the 48 cents each that they usually cost, I doubt it would have cost over $50.
You can see the obvious differences in full-screen view if you compare the two clips of Molly playing at 0:37 and 2:44, the former being 480p and the latter being 720p. While I’d normally have just paid 2.2x for the 720p, being able to generate multiple 480p clips for nothing was too useful to bypass.
Creating a good character sheet was absolutely vital. Four images per character are permitted, so producing a high-resolution four-pose image that can be used to create the reference character is vital. These four-pose references should cover the four cardinal angles, and can be created from photographs or from scratch. Below is the character reference that was utilized for the bassist, Molly.
It’s often tempting to begin working from a single frame, because a lot of the video generation engines are designed for them. But the additional time spent preparing characters is absolutely worth it and it’s a step that should not be skipped. Characters create a much larger possibility space, and more lively animation. And while it’s still not possible to correctly map the musical notes to the hand positions, the engines are now smart enough to time the hand motions to BPM, which allows for modestly more realistic guitar-heroing.
The lip-syncing remains moderately problematic, but there isn’t much that can be done about that until better technologies come out in time. The one surprise was that Kling’s Avata actually performed better than Creatify’s Aurora; neither was perfect and both tend to struggle to precisely understand a heavily effected voice, even when provided a straight vocal track with no instrumentation. It turns out that providing it with the actual lyrics doesn’t help at all. Both lipsync engines pretty much require a straightforward view, as any attempt to utilize a three-quarters profile tended to distort and elongate the jaw in an uncanny manner.
And both engines also start from a single frame, so it won’t surprise me if they eventually add the ability to start from characters too, given the apparent advantages there. Avata is about half the price of Aurora, so it’s usually worth trying that first, then resorting to Aurora if you can’t get the result you want.
The tools utilized:
The Myth of Sisyphus by Albert Camus
Suno 5.5
Bandlabs
Reaper
Audacity
CapCut
Seedream 4.5 for character references
Seedream 5.0 for backgrounds
Aurora Clarify for lipsync
Kling Avata for lipsync
MiniMax H3 for 720p video clips
Seedance 2.5 for 480p video clips
For those more interested in the philosophical concepts than the musical consequences, there is a post on Veriphysics on the subject.
UPDATE: Thanks to one of the commenters, this is a test of MiniMax H3 doing lipsynced video, which may actually be superior to both Clarify and Avata. I forgot that the audio sample was 7 seconds, so it gets cut off, and I also forgot to remove the “distant silhouetted figure” from the prompt, but aside from the solid performance, the most significant thing about this clip is that this is done from a character, not from a frame. The video has since been updated to address the three worst offenders.
One must imagine Sisyphus producing rock at absurdly low prices.
Don't hate me, but I like this sound better than The Cure.