Video in Maya? No way!

One typical Monday morning, though I don't remember exactly what day it was, I was working on our upcoming major tool system with a partner who's using it with their team on their feature film project — We will call him James.

On that day I was building a specific workflow for layout artists. The workflow was basic and pretty straight forward. It needed to create a new Maya scene, reference assets as they were defined for that particular shot or sequence and... we needed a way to bring in a storyboard animatic into the scene for the artist to work with.

A thought process spun up inside my head. "Nothing too hard. All we have to do is get the video and... Wait, that's .mp4, because that's the format we publish in our other tools, due to streaming capabilities". "We will need to convert that into an .mov because Maya won't like it otherwise." "Ah! But how do we know the layout artists have QuickTime?" "Should we make that an official requirement? I can't recall if we did already." "Hmmm..."
That appeared to be a somewhat simple thing, but also complicated with no official way to handle... other than... image sequences?
I didn't throw out the image sequence idea just yet, but my gut told me 2 things, 1) There must be better things out there. 2) I should've taken my lunch 2 hours ago.

As an animator, I have always converted my video reference or any footage for that matter into an image sequence to use for my shots. Sometimes I used QuickTime .mov file, but that was back when I had QuickTime on my computer.
"Wait, why don't I have it now?" I noticed I didn't have it.
Let's see if I can find it and... "QuickTime for Windows is no longer supported"?? Thanks Apple.

With all that on my mind, I went down the rabbit hole deeper. What IS the best way people use today for bringing any kind of video into their Maya scene? There must be something I don't know. It couldn't be that Maya, an industry standard, STILL doesn't handle video well, could it?
After an extensive research, forum browsing and experimentation, I had learned a few things:

• Maya doesn't like video formats that aren't QuickTime very much or at least doesn't handle them very well in real time (Yes, still.)
• Maya does handle uncompressed .avi on Windows — lol No.
• QuickTime for Windows is still out there, but isn't supported anymore for Windows.
• When things get real, people's jobs are on the line and they don't have time to play around, what do they do? They convert their video into an image sequence just to bring it into Maya. 🤮

Course of action

With all that in mind, I had 3 options with which to solve my little "video in the viewport" challenge:

• Convert the incoming animatic to images and waste the artist's disk space and time (Not to mention if this operation repeats when the animatic updates, and it WILL.)
• Ask every single artist to install QuickTime on their computer, which isn't supported anymore and it still forces us to convert our pre-existing video formats to it — Less than ideal at best, Very troublesome at worst.
• What if, hear me out... we just build something to solve this issue once and for all and... (Look left) (Look right) Load ANY video in Maya directly! 💀

So what should we do? No. I'm not doing the first 2 options, but the last one I like!
That last one made my gut feel like I could go without lunch for another 6 hours, if I start working on it now.
How come after all these years we still don't have normal support for video in Maya? So what if we could have it?
A side project was born. Media Plane.

Make It Work

Fast forward to 2 weeks of research and experiments later, here we now have a new kind of image plane for Maya that can actually play video in real time. That wasn't too hard. With a custom built version of FFmpeg, this new maya plugin is now able to read any supported video file format and draw it as texture on a plane in 3D. With that, we can have controls over it with using the plugin node, like telling it which frame to place, where the plane needs to be in space, fitting to Maya's camera, etc... Amazing!

But why is it so slow and keeps hanging every 24 frames like it's installing updates while I'm trying to log into my scheduled Zoom meeting? I see what's happening. Obviously it's because it's reading the video file from disk as I'm trying to play it. That won't do.
What we need here is a caching system. Something that would read the video ahead while I play the last part of it from memory.

Make It Good

Making it work was fairly straight forward, but making it work fast, that's a whole another story.
Video files usually store their information in packets that hold changes from the last keyframe. Keyframes in a video file are usually frames that hold a complete image, not just changes and a video file can have their keyframes at various intervals.
So to read any "frame", the plugin has to travel to the nearest past keyframe and keep reading from there until it gets to the desired frame, only then it can show that frame in its full glory.

The plugin was already doing that, but it can't keep doing that on my time. I want to see the video play NOW and a loading screen won't help here. Back into the rabbit hole we go! Fast forward another 2 weeks, a concept of "Blocks" was established. A Block in Media Plane is a single frame group cached in memory. The plugin can pull any image out of a block at will without going back to the last keyframe.
The more Blocks it stores in memory, the less it has to read from disk and the better it can play! Let's test it out...
I hit Play and realize that I forgot to limit the playback frame rate, because it went flying faster than an animator leaving at 6pm on a Friday after hearing “the director has notes.”

But no one wants to think in Blocks, as they are relative to the keyframe intervals in a video. That's why the plugin now had a "Cached Frames" attribute. The artist can say how many frames they want cached, considering the Keyframe Interval and the plugin will break them down into previous and next Blocks up to the total specified frame number!
You can Learn more about Media Plane caching system in Formats & Caching

Now we have a multi-threaded, blazingly fast video playing beast right there in the viewport, doing 60fps and higher!
NOW... it's good.

What Do the Artists Say?

Remember James? James was now at freedom to try out this new beast and actually animate a sequence layout scene.
A week had passed and I got into a call with him. He had a few notes for me after a week of working with Media Plane, but it turned out they were all aimed towards things other than Media Plane.
"Ok, sure, but what about Media Plane? Did you like it?" I asked. He thought for a moment, as if to realize what I meant and said:

"Oh yeah, it's awesome! I love it."

Then it occurred to me... He never felt it in his workflow, he just went through with his work while using it. 😄 Perfect!
That's exactly how it should be — The tool is working for the artist and the artist can focus on their work.
That means the tool did its job and I couldn't be happier!

One thing James had said about Media Plane later was suggesting the "Force Frame Rate" feature, which now it's in the plugin.
Thank you James!

What's Next?

The dev-board on our Discord server is already filling up with suggestions of new features from current users of the plugin.
At the time of writing, we are working on making Media Plane available for all major platforms, advanced editing features for a video file within Maya (Yes, no need to pre-edit your acting reference 😉) and a LOT more.

Video in Maya? Yes way. Introducing Media Plane!