Runway Prompts #2: Basic actions
Runway prompts for basic actions.
"Runway Prompts" is a article series that shares image-to-video prompt keywords and phrases that are proven useful in Runway. You can use them as-is or as building blocks to create longer Runway prompts to animate Midjourney images.
The primary goal of this article series is to save you time and video generation credits while creating a video in Runway.
In this article series, each prompt will be re-run at least four times to determine whether it can consistently produce a desirable outcome.
Don’t assume it will work as expected.
Watching the spectacular AI videos on YouTube created by experts may lead you to believe that creating simple actions such as walking, singing, and dancing is easy with a sophisticated video generator like Runway's Gen-3 Alpha Turbo.
But is it really?
The answer is actually yes and no.
Some basic action prompts work as expected. But some don't.
Here, I tested the prompts with Gen-3 Alpha Turbo to generate 10-second clips.
Runway Prompts that work well (3-4 success rate in four re-runs)
The following single-word prompts are reliable:
Jumping
Hopping - It’s like jumping, but the subject does not hop like a kangaroo.
Leaping - The subject jumps and flies into the sky, defying gravity.
Falling - The subject drops to somewhere.
Bending - It’s like bowing down.
Punching
Waving
Dancing
Squatting
Crouching - It's similar to squatting, but lower.
Balancing - Without context, the bot will create a clip of the subject trying to stay upright on one leg.
Problematic Runway Prompts: Actions involving walking and running
I tested at least 20 re-runs of these prompts and found a success rate of <50%, with some failing completely.
Walking
Running
Sprinting
Marching
To be clear, "success" refers to walking or running smoothly, as one would do in the real world. However, most of the generated clips contain hiccups, such as skipping a beat or a step in the movement. It's also as if the video clip is auto-looping in the middle and the action restarts, resulting in a brief pause in between loops. Other times, the movement appears unusual or unnatural.
You'll waste your generation credits if you try to create a perfect 10-second walking or running clip.
What's the solution? Re-run several times until you have a video clip that is sufficiently long and free of hiccups. Then, trim off the odd part. Not an ideal solution. Please let me know if you have any other suggestions.
I'd like to experiment and develop a technique to combine actions and make them work on Runway. But based on a simple test, it appears that Runway does not follow the prompt to animate the action in sequence. That could be a challenging task.
The magic (and mayhem) of missing context
What happens when you do not provide a context?
What happens if you ask the subject to swim and there is no swimming pool nearby?
I found that there are two possibilities.
The subject does nothing. Standing still with little movement. Sometimes, the background will change and shows a nearby swimming pool.
A sudden change in background to a swimming pool, followed by the subject swimming inside it. The success rate is low (1 out of 4 re-runs).
Similarly, if you ask the subject to jump from the modern world to the Jurassic era, the surroundings can change dramatically, immersing the subject in the world of dinosaurs.
Before we look the examples, here’s the original image created using Midjourney:
Prompt examples that work well:
Jumping
Hopping
Leaping
Falling
Bending
Punching
Waving
Dancing
Squatting (2 types)
Squatting and transforms into a rabbit (bonus clip)
Crouching
Balancing
Problematic prompt examples that involve running and walking
Running
Sprinting (lookout for a hiccup or skip a beat)
Walking
Marching
Running. Squatting. (Problematic combination)
When there is no context
swimming (no action, but the background shows a pool nearby)
swimming (the subject suddenly jumps into a pool )
leaping to Jurassic world
Conclusion
The following single-word prompts work reliably in Runway: jumping, hopping, leaping, falling, bending, punching, waving, dancing, squatting, crouching, and balancing.
Basic actions like walking, running, sprinting, and marching have hiccups in 10s video generation, with a success rate of less than 50%.
I hope you liked this article!
Please subscribe, like, share, and comment so that more people can discover Geeky Curiosity newsletter.