Runway Prompts #4: Interaction with an object
Runway prompts about the subject's interaction with an object.
"Runway Prompts" is a article series that shares image-to-video prompt keywords and phrases that are proven useful in Runway. You can use them as-is or as building blocks to create longer Runway prompts to animate Midjourney images.
The primary goal of this article series is to save you time and video generation credits while creating a video in Runway.
In this article series, each prompt will be re-run at least four times to determine whether it can consistently produce a desirable outcome.
Simple interaction with an object
Soon, you'll be surrounded by unwrapped Christmas gifts. Have you ever wondered what happens when you animate a character in Runway AI to open a gift or box?
Whether it's a simple cardboard box or a fancy, ornate chest, can Runway AI perfectly animate a character? In this article, we will conduct some simple experiments to determine how well Runway handles these tasks.
We'll explore several simple scenarios, such as:
Open a box
Close a box
Move a box
Locating and carrying out actions on the box
Putting something in the box
See what happens when the box performs some actions on its own
Some of these experiments will make use of the "First and Last Frame" feature, which I’ll discuss later.
The models
In addition to the Cover Image, we have two more models taking part in this experiment. F = First Frame and L = Last Frame. Note the letterboxing (black bars at the top and bottom) in the images of the Japanese lady.
All images were generated using Midjourney.
Overall findings
(A) Offloading facial expressions to Midjourney makes prompting Runway easier.
In the previous article (#3: Magic and Spells), I hypothesized that we could generate an image with the subject's facial expression before loading it into Runway for animation, allowing the animation to focus solely on the action and less on the facial expression. In other words, we offload facial expressions to Midjourney, allowing Runway to focus solely on the action.
My hypothesis is correct. For the Japanese lady model, I can focus on prompting the action while "ignoring" the prompting of keywords related to facial expressions, such as worry or anxiety.
(B) The First and Last Frame feature reduces the failure rate.
I've noticed that using two images or frames for the action increases the success rate. In addition, I can prompt Runway to include elements that are not part of the two images to create a more interesting visual, such as discovering what's inside the box.
The Runway bot will figure out how to move between frames on its own, lowering the failure rate when compared to a single image-to-video.
Open a box
It turns out that opening a box is not as simple as it appears with a single image. All iterations failed when the box or chest was ornate. However, it works when the box is simple cardboard, but the action of opening a box remains awkward.
Subjects opens the box (ornate) —> all 4 iterations failed. A new box appears on the top of the existing box/chest.
Subject opens the box (a simple box) —> successful, but movement looks odd
When using two frames, the success rate is higher, the action is smooth and natural, though the box opening process remains strange.
Subject opens the box (two frames) —> It works but awkward (4/4 success rate)
Subject opens the box (two frames) —> Almost perfect except a bit incoherent at the at the end
Subject opens a box revealing a green glowing object inside the box (two frames) —> Prompting adds a new item to the scene while preserving the smooth, natural movement. However, the object may unexpectedly change into a gas.
Subject opens a box revealing a bloody human head inside the box —> All failed. One generation shows the lady appearing uneasy about opening the box. Maybe she knows there is something scary inside the box!
Subject opens a box revealing glowing skeleton inside the box —> Here’s another fun prompt
Close a box
Closing a box is even harder than opening it. All failed.
Subject closes the box (two frames) —> It works, but there are coherency issues. Only 1/8 of the iterations was successful in getting her to properly close the lid.
Subject places a small vase inside the box and closes it —> It works for putting the vase (4/4) but not closing the box.
Move a box
The success rate of moving a box from its original position to another is difficult to predict.
Initially, I speculate that this is because Runway is unsure how to fill the space previously occupied by the box with something else. But it turns out that this is not true - because it works in the case of the Japanese lady.
Subject pushes the box —> All failed
Subject lifts the box —> Everything failed. It could also be the wrong terminology. Maybe I should've called it a chest instead of a box.
Subject lifts the box (simple box) —> 2/4 (50% success rate). The subject can lift the box with ease. It means that the Runway bot understands how to animate moving or lifting objects.
Subject stores the box away in the cabinet —> All failed. But success is not consistent.
By the way, the letterboxing issue disappears automatically during the animation.
Tip: It does not matter if your Midjourney images have letterboxing issues. Runway will take care of it. Animate with Runway first before editing the Midjourney images.
Locating and carrying out actions on the box
Subject cleans the box —> 4/4 success rate. Subject knows where the box/chest is.
Subject burns the box —> 2/4 success. There’s fire but it doesn’t actually burn the box. The fire is just…there.
Subject burns the box with a lighter —> 2/4 generations have fire. But the subject has a weird way of using the lighter.
Subject writes on the box —> 4/4 success rate but nothing was actually written.
Subject writes "Hi" on the box —> Failed. Except once - when the text appears on the outside of the box.
Putting something in the box
Subject puts a bottle into the box —> 4/4 success rate but the bottle may split into two
Subject places a vase into the box —> 4/4 success rate
See what happens when the box performs some actions on its own
Most of these failed. Here are the failed prompts:
a hand crawls out of the box
the box opens by itself
a kitten jumps out of the box
subject is scared while the box shakes
subject shakes the box, expressing fear
"7 days" in blood appears on the box
"7 days" written in blood appears on the box
write "7 days" on the box
But there’re some success cases (25-50%)
Box glows while the lady is scared and moves away —> 2/4 success rate.
Notice that when the character in the Midjourney image has a worried expression, it will be rendered more naturally in Runway.
Box glows in red while the lady is scared and moves away —> 1/4 success
Here’s an epic failure
box becomes wet and dripping out water, while the lady is scared and moves away
Selected video clip
This is a video of a woman cleaning the box. The action is natural despite the coherency issue.
Conclusion
Runway AI's performance in animating character interactions with objects:
Opening and closing boxes is challenging, with mixed results.
Moving objects has variable success rates.
Simple actions like cleaning or placing items in boxes are more successful.
There are frequent coherency issues in the animation.
Success rates vary depending on the complexity of the action and the object involved.
Techniques to improve animation quality:
Using Midjourney for facial expressions and Runway for actions improves results.
The "First and Last Frame" feature increases success rates and allows for more complex animations.
I hope you liked this article!
Please subscribe, like, share, and comment so that more people can discover Geeky Curiosity newsletter.