Wednesday, February 20, 2019

Motion Interpolation plugins in AVISynth (WIP)

(This post is a work-in-progress. I'll be adding screengrabs, more info and eventually a video tutorial. Keep checking back if you'd like to see more, and feel free to comment if you think I've left something out.)

First of all, thanks to Esteban Ariel Castrogiovanni for the suggestion to look into this.

I'm not generally a fan of motion interpolation. I especially hate the "motionflow" BS on most modern TVs. The only time I really like using it is with QTGMC, and that's because it's helping to recreate full frame detail from half-height fields.

However, I realize that motion interpolation can be useful in other situations. In addition to allowing you to make a slow motion shot out of footage shot at a regular framerate, you can also use it to do framerate conversions without changing overall runtime, such as going from 29.97fps to 23.976fps.

There are a number of different options for performing motion interpolation. As far as I can tell, Twixtor is pretty much the gold standard for "run it yourself" software in pro circles, but it has the notable limitation of not being able to detect scene changes. That means that if you're working on more than one shot, you would need to apply the Twixtor effect to every *individual* clip in a timeline in order for it to work properly rather than just applying it after exporting. However, Twixtor is really good at what it does, and motion is less choppy overall than other solutions.

Unfortunately, Twixtor is also $395, or closer to $600 if you want the "Pro" version that has more features for VFX artists.

So what if you're broke, or don't use one of the editing programs that Twixtor supports? As it turns out, AVISynth does actually have some interesting alternatives. They have their own issues, but they can definitely get the job done.


Update: Here's a rough draft of the video tutorial for setting up and using the plugins. I plan on releasing a more comprehensive video later, but if you'd like a preview, check it out:




FrameRateConverter requires MVTools2, MaskTools2, and the fftw3 library, just like QTGMC. *Unlike* QTGMC, however, if you want to use the 64-bit version, you'll need to install the 64-bit version of fftw3 in your System32 directory.

It has a wide range of supported colorspaces, comes in both 32 and 64-bit versions and the quality is good overall. However, the speed ranges quite a bit depending on the settings used.

Using a script for converting a 29.97fps HD ProRes file to 59.94fps, I got speeds ranging from totally unusable (32-bit version, "Slowest" preset, which estimated almost 3 hours to finish) to 1:37 (64-bit version, "Faster" preset). The 32-bit version was exponentially slower than the 64-bit version at slower presets, and inconsistent at faster ones, sometimes varying by several minutes. By contrast, the 64-bit version uses more memory, but is faster and has almost no variation in run times.

As to visual quality, FRC looks like it uses a form of partial frameblending, which can make some motion look a little less smooth than Twixtor (or the other filter I'm looking at below). On the plus side, it does have scene detection, and there is little-to-no warping of the image.

Here's a sample script (AVS+ syntax):

SetFilterMTMode("FrameRateConverter", 2)
FFmpegSource2("The Hollar Commercial 1 - ProRes422HQ.mov", atrack=-1)
FrameRateConverter(Preset="Slow")
Prefetch(10)

One thing to note is that if you're not doing a straight frame-doubling, you have to use a somewhat odd system for specifying framerate. You have to specify the "NewNum" and "NewDen",  which corresponds to a numerator and a denominator, somewhat similar to how FFMPEG handles framerates. For example, if I wanted to convert to 59.94fps, I'd use a line like this:

FrameRateConverter(Preset="Slow", FrameDouble=false, NewNum=60000, NewDen=1001)

Likewise,

NewNum=24000, NewDen=1001 

would get you to 23.976 and

NewNum=30000, NewDen=1001 

would convert to 29.97. Keep this syntax in mind, because it's how the next filter works as well. Speaking of which:



Interframe requires SVPFlow, a version of which is included in the download.

It only works in YV12 colorspace, doesn't have a 64-bit version, *really* doesn't like working with FFMPEGSource when using 10-bit material (even after colorspace conversion), and the "beginner's guide" for it focuses on using MeGUI. Oddly enough, it also has settings for stereoscopic 3D content.

However, it absolutely mops the floor with FrameRateConverter when it comes to speed, and it doesn't have the frameblending artifacts.

Using the slowest preset (which is called "medium", for some reason), it ran anywhere from 19-40fps. You can enable GPU support, but this doesn't actually speed up the preset. Instead, it supposedly works harder to give better results. In practice, I don't notice a massive difference, but you can try it out for yourself and see which settings you prefer.

Here's a sample command:

SetFilterMTMode("InterFrame", 2)
QTInput("The Hollar Commercial 1 - ProRes422HQ.mov", audio=1)
ConvertToYV12()
InterFrame(Preset="Medium", Tuning="Smooth", GPU=True, Cores=10)
Prefetch(10)

If your input video is in 10-bit color, you'll need to use an input filter other than FFmpegSource2. In this case, I'm using the massively outdated QTInput. If your input video is in 8-bit color, you don't need the ConvertToYV12() command, and FFmpegSource2 should work just fine.



Overall, the following distinction applies:

Interframe preserves more detail in "Weak" mode, but objects in motion can have wobblier edges. In "Smooth", motion looks smoother, but less detail is preserved. Both modes can have garbled edges around moving high-contrast edges.

FrameRateConverter has a more "frameblended" look, with some choppier/blurred movement, but almost no edge artifacts. It can have issues with certain patterns of detail in some shots, though. It generated a "twitchy" artifact around the edge of a corrugated metal roof during a panning shot in one video I used it on.

Twixtor is like a more refined version of Interframe, with better edge cohesion, but still some noticeable artifacts at default settings. In particular, it has a larger "warped" area.



There are a couple of other options I've played around with, but none have been as good.

YFRC is a script that requires MaskTools2 and MVTools. Like some of the other options, it needs YV12 colorspace. On the plus side, it supports common framesizes from 320x240 up to 1920x1080, although you have to set the block size appropriately. It also supports multithreading and 64-bit mode. You can change the framerate conversion method by changing OverlayType. OverlayType=0 uses frameblending, while OverlayType=1 doesn't blend frames, but can be choppier as a result as it doesn't fully interpolate between frames in many cases. Neither is as good overall as Interframe or FRC, but YFRC does have the best scene detection of any of the options available, with no blended/interpolated frames between cuts.

ChangeFPS is the default framerate conversion command in AVISynth. It isn't multithreaded, but does work in 64-bit mode, and can use whatever colorspace you throw at it (at least in AVS+). Unfortunately, it's about the same as using the basic framerate conversion tools of Premiere Pro/Resolve/etc, which is to say it looks pretty choppy. It does no motion interpolation, instead simply adding or removing frames as necessary.

ConvertFPS has most of the same benefits of ChangeFPS, but can either do frameblending "Blend" mode (the default) or a "Switch" mode that looks like the resulting video was captured from a game with Vsync disabled. The Switch mode also requires 8-bit YUY2 colorspace, so you'll have to do those conversions first. I don't recommend using the Switch mode.

MSU_FRC does an okay-decent job, but is limited to YV12 colorspace *and* base16 block sizes, so it only really works with SD video frame sizes. It's also not multithreaded or 64-bit capable, which makes it slooooow.

FrameDbl only does straight framedoubling, does so via blending frames, and does not use multithreading. Also, it has the same issue with 10-bit video in FFMPEGSource as Interframe.

Motion is ancient, has confusing documentation, is not multithreaded (in fact, multithreading slows it down significantly), and produces nasty-looking artifacts.

Finally, there's the minterpolate filter in FFMPEG. Using the recommended settings from the FFMPEG Wiki, I came up with a command like this:

ffmpeg -i "input.mov" -filter:v "minterpolate='mi_mode=mci:mc_mode=aobmc:vsbmc=1:fps=60'" -c:v prores_ks -c:a copy "output.mov"

The end result is very smooth and has excellent scene detection, but with a more distorted "warped" area around moving objects, and more "blockiness" in those areas. It's also glacially slow; neither multithreaded nor GPU-accelerated. There's a very minor speedup from using the 64-bit version in my limited testing.


8 comments:

Castrogiovanni Esteban Ariel said...
This comment has been removed by the author.
Castrogiovanni Esteban Ariel said...

I regret not having to donate, due to my economic situation I almost do not have to be able to pay for internet and other services of the state in the country where I live, if I had a donaria without hesitation ...
I am writing to inform you that some very interesting tools have emerged, that they manage to do a Upscaler and improve the quality. It is called "waifu2x-caffe" and "Dandere2x". I leave you these videos to save you the research if you think you can do it. A well-explained tutorial of such processes in the future. These tools are really amazing, I compare the information :) SALUDOS DE ARGENTINA

Fast Waifu2x Video Upscaling! (Dandere2x 0.4.1 Installation and Demo)
https://www.youtube.com/watch?v=HW6fl1dAk7M

This upscaling tool is amazing! - Waifu2x tutorial
https://www.youtube.com/watch?v=MVPrFU6_SEA

I want to get to this result :)
https://www.youtube.com/watch?v=8v8oRwaag7k

Andrew Swan said...

Cool stuff. I've heard of Waifu2x before, but I don't do work with animation, so I haven't paid much attention. I'll give it a look at some point in the near future, if for no other reason than to do better screenshot upscaling.

Anonymous said...

Have you ever been able to convert from 29.97 to 23.976?

You wrote:

"Likewise, NewNum=24000, NewDen=1001 would get you to 23.976"

When I try that, I get errors. It looks like FrameRateConverter doesn't want to go to slower speeds, only faster.

Thanks!

christina said...

Am I the only one that doesn’t like the way QTGMC makes video look with the default bob setting of doubling frame rate? To me it looks like that fake soap opera effect TVs use and with shaky SD analog home video conversions, it can even look disorienting. I haven’t found anyone else saying the same thing as me. Everyone seems to love QTGMC and its effects. I am just experimenting with it now with fpsdivisor=2 and plan to try some of the blur options to smooth motion without doubling frame rate.

By the way I love your tutorials on YouTube. There is so little help on this subject available in video format and it’s sometimes so much easier to follow along when you can see it being done and explained well rather than reading help files or wikis or forums. So, thanks!

I followed your tutorial on how to install avisynth+ in 32 and 64 bit versions and I have everything working but honestly I have no idea if I would even know how or why to use the 64 bit version. I think most of the things I need are 32 bit .. I get very confused trying to figure this out! I’m just happy it’s working right now :)

Andrew Swan said...

Apologies for the late reply, and thanks for the shout out.

Basically, the 64 bit version is more stable and faster in most cases, but has less support for legacy plugins. I really hope that most of the major plugins get ported over to 64-bit in the near future, because I think it this point there would be very few systems that require it.

QTGMC has its issues, in particular when you're dealing with a really grainy image. If there were a better solution (ideally open-source) out there that didn't cost an obscenely large amount of money, I would promote and use it.

Brandon27 said...

Running with your comment about FrameRateConverter: "32-bit version was exponentially slower than the 64-bit version", I installed Avisynth+ 3.6.1 and replaced Avisynth 2.6 32bit. The documentation on the web is poor. Nowhere does it say you need to run 64bit Virtualdub2 to use 64bit plugins in Avisynth+. It took several hours to figure that out. If Avisynth is presenting a virtual avi file via the avs script, why would it matter if the calling app is 32bit or 64bit?

The 1920x1080p video I'm working on, I'm trying to double the framerate. Only the "slowest" preset gives acceptable results (no blocking in video). I changed the script from single threaded to 4 threads (quad core Xeon CPU). I'm getting 1.06 sec/frame with all plugins 64bit. With everything 32bit, I was getting 0.96 sec/frame. That is not much difference, certainly not exponential. Once it's done (3 days from now), I'm uninstalling Avisynth+ and reinstalling Avisynth 2.6 32bit.

I'm also testing Svpflow on the same video to see how it compares. Svpflow is what Interframe uses. It's running at 2.5FPS on a different PC that has a GTX1080Ti.

To christina:
About QTGMC- used correctly, it dramatically improves the amount of detail in interlaced video, by properly (bobbing and motion compensation) changing it to progressive. Some videos have information in the double frames too. To determine if you want to keep or delete them, you have to step through the processed video frame by frame to see if they're duplicates or not.

IMO, live footage generally looks best with a high frame rate. And movies do not (IMO). I doubled the frame rate on one movie (with FRC) and didn't like the soap opera effect at all. You can control what to do with the extra frames. If you don't like them, add a SelectEven() after the QTGMC line (or use fpsdivisor=2). That will delete the double frames.

Here's what I typically use in QTGMC:
QTGMC( Preset="Slow" )
If noisy:
QTGMC( Preset="Slow", NoisePreset="Slower" )

If later feeding to Video Enhance AI for upscaling, you typically don't want to do much denoising:
QTGMC( Preset="Fast", TR0=1 )

Andrew Swan said...

Brandon,

I think that your settings must have a different bottleneck than the ones that I was using, because I can assure you that there was a *dramatic* difference between 32-bit and 64-bit renders in my case. My guess is that the "slowest" preset is the cause, but I'd have to test to be sure.

I agree that documentation for AVISynth is poor for new users (and even professionals) at this point, which is a big reason why I started making these tutorials. I'm not a programmer, but I assume that there's something about the instructions used in AVISynth's design that requires everything to be 64-bit all throughout the chain.

As to the 32-bit vs. 64-bit debate, at this point I've essentially switched over to using only 64-bit plugins whenever possible due to memory issues. There are still a few filters that haven't transitioned yet, but they're getting less and less as time goes on.

Regardless, thanks for the info about AI upscaling. I need to look into that at some point.

Which deinterlacing algorithm is the best? Part 1 - HD interlaced footage

Before I began, I'd like to give a special thanks to Aleksander Kozak for his help in testing and providing the HD footage used in this ...