Which deinterlacing algorithm is the best? Part 1 - HD interlaced footage

Before I began, I'd like to give a special thanks to Aleksander Kozak for his help in testing and providing the HD footage used in this post.

Also, this is part 1 of a series of posts. When I've finished part 2, I'll link to it here.

Background

One of the things about working as an editor is that the technology is constantly changing.  There are new pieces of software, plugins, and even hardware that are released on a regular basis, and keeping up with all of it can be challenging.

Roughly about 10 years ago, deinterlacing had four general types of solutions: 

- Hardware-based video conversion boxes like the Teranex that are designed to perform resolution and frame rate conversions in real time.  Simple, but not always the best quality, and the hardware is pretty expensive (especially when you consider that you also need both a separate dedicated playback device and a recording device in addition to the conversion box).

- Services like Cinnafilm's Tachyon that do the same sort of job using very complex and proprietary software and hardware on a rental (or on-premises, for a higher fee) basis.  Great results, but very expensive.

- Built-in deinterlacing within video editing programs.  I haven't used either Media Composer or Final Cut Pro X's implementation, but with everything else I've tried, this is garbage.  Unless your entire project from beginning to end uses interlaced settings, editing programs tend to toss out every other field and line double the remaining field, which cuts both motion detail and frame detail essentially in half. The only way I've seen to get around this inside an editing program is to use the FieldsKit plugin, which works a bit better than real-time conversion boxes, but not as well as SAAS options like Tachyon.

- Open source video filters.  For a while, your only options here were frame blending and "bob" deinterlacing, neither of which were particularly great.  Most early tools for dealing with interlacing in this space focused more on combining similar fields together - mainly as a result of them being geared towards processing TV captures of anime.  Slightly later methods like Yadif improved on deinterlacing quality, but were still only working on individual fields, using still image interpolation to essentially round off the edges of aliasing after line doubling.

That all changed when QTGMC came along. Like its less-optimized forerunner TempGaussMC (which I've never used), QTGMC is an AVISynth deinterlacing script that uses a combination of several different plugins. It separates out individual fields and tries to use data from surrounding fields to generate full resolution progressive frames from them.  This is very similar to what "motion flow" and other motion interpolation algorithms do on modern TV sets, but focuses on accuracy rather than wow factor.  While it's not trivial to set up, the end results are excellent, combining greater detail with highly configurable noise reduction and other options. In some ways, the results can actually be better than many of the above solutions.

But it's always good to do a reality check periodically - which of these methods works best? I don't have access to a Teranex or the cash to use Tachyon, so I'll be comparing mostly free options and/or ones embedded in >$300 programs.

TL:DR

If you don't want to read through the entire post, here's my opinion about each of the options that I currently have access to:

QTGMC: Best retention of original image detail and handling of edge case artifacts, but you have to watch for colorspace issues when working with Adobe programs if you render out using FFMPEG. Also, different options can change smoothing and noise reduction quite dramatically.

Topaz AI: Excellent results that can be occasionally thrown off by oversaturated colors and constantly fluctuating semi-transparent detail (like heat haze).  Best used for upconverting content to higher resolutions.  Also uses FFMPEG to render out results, so some codecs can have the above-mentioned colorspace issues.

DaVinci Resolve 17 Studio: Solid deinterlacing, but can also be thrown off by oversaturated colors.  Not as much smoothed detail as the other options, but in motion most viewers probably wouldn't notice.

Bwdif (FFMPEG): Good at reconstructing smooth lines and retaining overall image detail with SD content, faster than QTGMC but not as good.

Estdif (FFMPEG): Good at reconstructing smooth lines and retaining overall image detail with HD content, faster than QTGMC but not as good.

Yadif (FFMPEG): Just does interpolated line double of each single field rather than taking surrounding fields into account.  Diagonal high contrast lines have weird "rounded" jaggies that some (like me) might find displeasing. Very fast.

Bob, Weave, etc - These are all variations on very basic line-doubled deinterlacing. Bob would be my preference out of these, but in general I would only use them if you absolutely have to.

Premiere, Resolve (non-Studio), editing programs in general - Garbage. Cuts both resolution and framerate in half. Note that I'm only talking about the standard deinterlacing inside of these programs - you can get/install plugins that can do a much better job.

Let's check out some examples:

First clip - 25i HD material

This was a tough clip for the commercial options to deal with due to some issues with the very oversaturated reds, but I feel like it shows off the differences between deinterlacing methods pretty well. Also, the differences between these methods are much less noticeable in motion.

(Please click pictures to enlarge)

Original

This is what an interlaced frame looks like - two fields woven together. Obvious combing artifacts. If you look close, you can see the issue with the edges of the oversaturated reds - some interlacing artifacts don't line up with the interlacing lines. I've actually never seen this before, but I feel like it turns out to be an interesting indicator for what various methods actually do.

Bob (After Effects, Premiere Pro, many other editing programs)

This is just the original image run through After Effects. By default, AE uses a terrible method to display interlaced footage that drops every other field and does a simple line double for the remainder. However, it does get rid of the interlacing artifacts in the oversaturated red.

DaVinci Resolve 17 Studio

This is a recent edition to the paid ($300) version of Resolve that supposedly uses some "Neural Engine" processing for frame interpolation. Decent handling of edges, but tripped up by the oversaturated red. I would consider this a good solution for most work,

Topaz AI

Really good recovery of detail throughout the image, as well as adding some extra detail (that's kind of it's thing).  However, it's thrown off a bit by the haze around the railing, and once again doesn't deal well with the interlacing artifacts in the oversaturated red.

Now, let's check out the FFMPEG (and other programs that use FFMPEG) options:

Yadif

Works, but look at those railings. Remember, Yadif is just taking the individual fields and using smoothing algorithms to resize them. I personally don't like the results, but it's better than nothing.

Estdif

Much better, although some details like the lower contrast parts of the railing aren't as defined as I'd like. Unlike Yadif, it's at least attempting to do interpolation using surrounding frames, and as a result takes a speed hit. My recommendation for FFMPEG-only deinterlacing of HD content.

Bwdif

My favorite FFMPEG-only method for deinterlacing SD content does a bit worse here - the edges look a bit rougher and less defined. Still better than Yadif, though.

Now for QTGMC. To simplify the crazy range of options, I'm limiting myself to two presets - Faster for mostly no noise reduction and a focus on speedy processing, then Slower for best "all options" processing without getting into ridiculous processing times.

QTGMC - "Faster" Preset

Cleaner, more defined edges with more perceived detail.

QTGMC - "Slower" Preset

The key word here is "smooth". The extra processing makes the image look more defined overall IMO. My personal favorite, although Topaz AI definitely is another option if you're okay with its artificially added detail.

Addendum - Colorspace Issues

You might remember that I mentioned something earlier about colorspace issues?  The screen grabs I've shown above were all captured from DaVinci Resolve.  If I use After Effects to view the clips instead, anything rendered out by FFMPEG looks like this:

Don't see it?  Enlarge the image, then use the left and right arrow keys to flip between the other screen grabs.  You'll notice that the colors in the overall image are shifted, especially the reds.

So what's going on here?  Well, I'm still researching this at the moment, but as far as I can tell FFMPEG does some sort of screwy colorspace conversion on some footage. If you don't correct for this, then After Effects and Premiere Pro display the colors in the wrong way. 

What's confusing though, is that this doesn't happen on every file you export out of FFMPEG.  Only certain codecs have this behavior, and one of them is ProRes. Now, this isn't a new issue - I've run into a variation of this problem years ago when I was upscaling standard definition video to high definition resolutions using AVISynth and FFMPEG.  However, one of my original fixes - to use the other ProRes encoder - doesn't do anything this time.  Thankfully, the other fix - rendering out to DNxHD or DNxHR - does solve the problem.  So, if you want to be sure you don't run into any problems, use DNxHR. If you have to use both FFMPEG's ProRes output and Adobe programs, then make sure you use AVISynth to at least load the file first, and then you can add a conversion step in your .avs script:

ColorMatrix(mode="rec.709->rec.601")

Yes, you read that right: converting from rec.709 (the colorspace for HD video) to rec.601 (the colorspace for SD video) fixes the problem.

But be warned - this actually changes the way that the video looks for other programs. If you bring the video into Resolve, it looks like this:

And suddenly, we've gone orange.

Anyways, I'll keep looking into this and see if there's an easier fix. Currently, I've tried the fixes documented here:

https://stackoverflow.com/questions/37255690/ffmpeg-format-settings-matrix-bt709

and here:

https://medium.com/invideo-io/talking-about-colorspaces-and-ffmpeg-f6d0b037cc2f

With no success.

I'll end this post here, but look to the next post for a comparison between deinterlacing methods in standard definition.

Let's Test QTGMC settings - Part 1

A question that I've thought about from time to time is this: what difference exactly is there between QTGMC's various settings? There's a metric ton of them, and while I've tended to just follow the recommendations in the documentation for what to use, I'm not always happy at the results I get.

So let's not be casual. I'm pretty busy these days, but as I have time, I'm going to go through and look at what each of the major settings does to a range of sample clips, and hopefully provide a better reference than the scattershot documentation that's out there currently.

Now, obviously I don't have time to test everything, and I'm not a plugin author or somebody who is particularly skilled at programming. This is all to say: don't just take my word for it. Test for yourself, and see what works for you. If you find something different, or an edge case that I haven't covered, great. Leave me a comment below, and I'll see if I can replicate your results. If so, I will update this blog post.

Some things to get out of the way up front:

• I'm currently using AVISynth+ 3.7 64-bit with version 3.376 of QTGMC. If you're using an older (or newer) version of either of those, you may have different results. I will update the version numbers on results below if they differ.
• I won't compare this to VapourSynth's havsfunc implementation - at least not right away. In theory, you should have the same results, but I have noticed chroma shifting differences in the past. I'm not currently using VapourSynth, however, and am not nearly as familiar with it.
• My source footage is mostly DV, which has a very nonstandard chroma subsampling method (4:1:1). As such, the color information in your video may behave differently then shown here. The black and white (aka luma) portion of the image should be similar, although depending on the source noise in your original footage, things might be different there as well.
• The first clip I'm going to use to test everything is from a 15+ year old video about building straw bale houses. It was shot on a consumer DV camera (not by me), and IMO has a good combination of in-camera sharpening and a range of both noise and detail. I'll try and find a less pre-sharpened image for comparison at some point.

Please click on the pictures to enlarge. Even better, this will bring up the "lightbox" so you can flip between pics with the left and right arrow keys on your keyboard. Sorry about the lack of labels in the lightbox - I'm looking into that.

Test 1 - Overall Presets

There are a total of 11 main preset options for QTGMC: Draft, Ultra Fast, Super Fast, Very Fast, Faster, Fast, Medium, Slow, Slower, Very Slow, Placebo.

Draft

This preset is pretty much only useful as a way to check and see if your field order is set correctly. It uses a simple "bob" deinterlacing method that makes essentially no real effort to create full resolution progressive frames.

Ultra Fast

This preset uses Yadif as a deinterlacing method, and looks similar to what you would get out of VirtualDub (Which uses Yadif by defualt for deinterlacing) or some real time commercial video processing devices. It does at least give a doubled frame rate, but I would only use it for comparison with other methods.

Super Fast

Here's the first preset that takes advantage of QTGMC's features. You'll notice immediately that the blocky effects of the first two presets are gone. Personally, I think this looks ever so slightly softer than most of the other presets - possibly because it uses TDeint rather than NNEDI3 for deinterlacing?

Very Fast

You'll notice that fine detail is a bit clearer, and some overall sharpening appears to be applied. Noise reduction is kept to a minimum, however. There's also a sort of haloing artifact around edges of fine detail.

Faster

Not a huge difference in sharpness here, but if you look closely at the tassels, you'll see that they look a bit clearer. Haloing is still present, and for some reason, there's an added line artifact on the right side that isn't present in any of the other presets. Not sure why.

Fast

This is the first preset that I would actually consider using. The haloing artifacts are gone, a little extra sharpening appears to have been added, but the image doesn't look terribly "smoothed".

Medium

Similar to Fast, but appears to perform more noise reduction.

Slow

By flipping back and forth between medium and slow, I can see that the interpolation has resulted in very slightly different images, but I don't really notice a big distinction in quality between them.

Slower

This is QTGMC's default preset, and I can definitely see how some people don't like it. While it does remove more of the artifacts from the video, it definitely has more of a "smoothed" look to it, with a slight smudging of details. Again, you would have to look at these back to back to really see the differences in my opinion, but this is starting to look a little over processed. On the other hand, if you're processing video for online streaming services, you might prefer this to keep recompression artifacts to a minimum.

 

Very Slow

Like Slower, but more so. More image detail is smoothed away, but more artifacts are removed as well.

Placebo

Aptly named, although I can see very slightly more smoothing applied to the image. I personally would never use this because of the ridiculously long processing times, but if you ever wondered what it does, here you go.

Test 2 - Sourcematch/Lossless

Because I'm not specifying anything else in the command, you can assume that the rest of the settings are at default (essentially preset=Slower). The main difference I can see is that Lossless on its own has some combing artifacts, and SourceMatch=1 is a bit softer than the rest.

SourceMatch=1

SourceMatch=2

SourceMatch=3

Lossless=1

Lossless=2

SourceMatch=3, Lossless=1

SourceMatch=3, Lossless=2

Test 3 - Noise Reduction settings

These are pulled from the QTGMC examples section.

Full denoise

NoiseProcess=1, NoiseRestore=0.0, Sigma=4.0

Does what it says. Beware of the "waxy" look with this one.

Restore all noise

NoiseProcess=1, NoiseRestore=1.0, Sigma=2.0 

Yup, this works as described as well.

Restore all "stable" grain

NoiseProcess=1, GrainRestore=1.0, Sigma=2.0 

This setting is a bit more interesting. It attempts to find more "stable" elements of noise and restore it to the image. This results in reducing the flat/waxy appearance of a full denoise, but still removes a lot of noise. Not sure if I'd use it all the time, but I could see some benefit.

Suggested settings for "Strong Detail/Grain"

NoiseProcess=2, GrainRestore=0.4, NoiseRestore=0.2, Sigma=1.8, NoiseDeint="Generate", StabilizeNoise=true

Seems reasonable, but maybe not for every source. I'll try to dig into this in more detail, but for now I'll refer you to the documentation for more of what the settings mean:

http://avisynth.nl/index.php/QTGMC#Noise_Bypass_.2F_Denoising

Reference

Just for reference, here's the original interlaced frame:

And here's one of the individual fields:

And here's the same frame processed with Yadif in VirtualDub2:

Deinterlacing with AVISynth and QTGMC - Updated occasionally

Welcome to the new home of my QTGMC deinterlacing tutorial. This post will be updated periodically as new changes happen and new info comes out, or until a better option comes along.

Changes since last post/video:

• I've switched to an entirely 64-bit workflow
• QTGMC has some new requirements
• I've dropped FFMPEG as the recommended rendering program in favor of VirtualDub2
• AVISynth Info Tool is now the recommended way to check and make sure that you've installed AVISynth correctly.

BIG DISCLAIMER: This process may not work, may crash, or may do other things to your system. Virus scan everything you download. It's not a 100% guarantee that you'll avoid getting a malware infection, but it's a lot better than not checking at all.

If you're doing professional work, always watch any files you create using this process before submitting/uploading them to check for audio and/or video glitches.

Follow this tutorial at your own risk. 

Also, this tutorial is for Windows 10. Most of the steps work for previous versions of Windows, but may require slight modifications. Users of MacOS and other OSes should look elsewhere.

Here's the video version of the setup and basic QTGMC deinterlacing workflow:

Setup

7-zip (optional)

If you don't have 7-zip already installed, you won't be able to open many of the downloaded archives. The stable, 64-bit version should be fine.

Next, we're going to need to get AVISynth+. You can grab it from here:

https://github.com/AviSynth/AviSynthPlus/releases

If you don't already have the MSVC 2019 Redistributable installed, grab the version ending with "vcredist".

Then, we'll need to get all the filters (plugins) needed:

FFMPEGSource

Note: Version 2.40 has been shown to corrupt video frames. Please either use a different version, or use LSMASHSource as linked below.

This is what will allow AVISynth+ to load our video files. Works with almost any container/codec combination, but read the "Known Issues" section for things to look out for (hint: don't use interlaced h.264 video with it).

Note: If you do need to work with interlaced h.264 video, try LSMASHSource instead. It requires two commands rather than one to work (technically three if you count AudioDub(video, audio) to combine the two together), but can support some non-standard formats. Thanks to neonfrog for the heads up.

QTGMC

The deinterlacing plugin of choice. Requires a whole host of additional plugins, of which you will be using the 64-bit versions (if an option is given). For most uses, only the "Core Plugins and scripts" are necessary.

For the "Zs_RF_Shared.avsi" link, right-click on the link and do "Save Link As", or whatever the equivalent is in your browser of choice.

DO NOT forget to download the 32-bit FFTW3 library as well. Without it, QTGMC will not run.

With all the AVISynth filters and scripts grabbed, it's time to get the supporting software:

AvsPMod

This is like an IDE for AVISynth scripts, and is pretty much essential IMO. Grab the 64-bit version for this tutorial.

AVISynth Info Tool

This allows us to detect processor features and check to see if AVISynth is properly installed.

Finally we need to grab a rendering program.  For ease of use, I now recommend VirtualDub2 - a fork of the original program used to render AVISynth scripts. 

VirtualDub2

When finished downloading, virus scan all the things.

Installing

AVISynth+ has a simple installer. I recommend installing into whatever your user folder is (for example C:\Users\Me\) rather than the default of Program Files(x86) so you don't have to deal with authentication requests every time you add a plugin. Also, I highly recommend checking the "Add AVISynth Script to New items menu" option. Otherwise, you can stick with the defaults.

Then, go to the plugins64+ directory for AVISynth+. For example, the default install creates a plugin folder at:

C:\Program Files(x86)\AVISynth+\plugins64+\

Extract all "x64" versions of the .dll and .avsi files to the plugins directory (if given a choice) from all plugins EXCEPT the fftw-3*.dll archive. If there's a choice between AVX, AVX2 and a version without AVX, you'll need to know what instruction sets your processor supports. CPU-Z  or AVISynth Info Tool can tell you this if you're not sure.

Now, open the fftw-3*.dll archive, then (as noted on the QTGMC page) extract the libfftw3f-3.dll file. Make a copy of it and rename it as "FFTW3.dll". Place the files "libfftw3f-3.dll" and "FFTW3.dll" in the SysWow64 folder. Don't ask me why you have to do this, I agree that it seems pointlessly tedious.

Please note: If you're only using AVISynth+ (not older versions of AVISynth) and only need to use QTGMC, you can skip the above step and just download/put the 64-bit version of libfftw3f-3.dll in the plugins64+ directory. Despite the error message in 32-bit AVSInfoTool, QTGMC will now work in both modes (although you can drop the 32-bit version of libfftw3f-3.dll in the plugins+ director to clear the error if you like, and to clear the Zf_RF_Shared error, you need to do the regular install method above). 

Also, if you want to be able to run QTGMC's "EZDenoise" & "Sharpness" functions, put the 64-bit version of "fft3dfilter.dll" into the plugins64+ directory.

Thanks to Spammy Family for pointing this out.

Speaking of tedious, if you want to use the "Very Slow" or "Placebo" preset for QTGMC, it looks like you need to install the 64-bit version of FFTW in your System32 directory using the same method mentioned above.

Extract the AvsPMod archive to wherever you want to run it from - again, I recommend your user folder.

Extract AVISynth Info Tool in the same way. If you like, you can install it more officially by clicking on the AVSInfoToolInstaller.exe file, but I generally just run it directly from the "AVSInfoTool_FilesOnly" folder. 

Go ahead and run it now, making sure to select the "AVISynth 64-bit" option once loaded.  If you get errors that are not about the FFTW3 library or Zs_RF_Shared.avsi (to fix those errors, install the 64-bit version of FFTW in your System32 directory using the same renaming/copying process listed above), then double-check and make sure you followed all the previous steps correctly. You might also want to take note of how many cores and logical processors you have, along with what instruction sets are supported by your CPU. 

Finally, create a folder for VirtualDub2 and extract its archive there.

Making your .avs script

Now that everything's ready, let's go to the directory with your video files and make an .avs script. Right-click anywhere in the directory, select New, then AVISynth Script and rename it however you want. If that option doesn't show up, you can just create a new text file and change the .txt extension to .avs.

Open AvsPMod, then go to the Options menu and click on "Associate .avs files with AvsPMod". You should now be able to double-click on .avs scripts and have them open directly in AvsPmod. Do so with the script you just created.

Here's my boilerplate .avs script settings for deinterlacing:

    SetFilterMTMode ("QTGMC", 2)
    FFMPEGSource2("videofile.avi", atrack=1)

    AssumeBFF()
    QTGMC(preset="Slower")
    BilinearResize(720,540)
    Prefetch(10)


The "atrack=1" option for FFMPEGSource selects the track of audio that is passed through during processing. If the default option doesn't give you audio in the results, try "atrack=-1" instead.

Please note that by default both FFMPEGSource and (certain modes of) LSMASHSource index your video file the first time you load the .avs script into either AvsPMod or your rendering program of choice. This may take a while and give the appearance of freezing the program. When it's finished, you'll see an .ffindex or .lwi file appear in the same directory as your video with the same name as your video.

Zeranoe is down

 An era has ended. As of the 18th of this month, Zeranoe, the best source for Windows binaries of FFMPEG, has shut down.

 Two other sites have stepped up to become the official FFMPEG Windows binary maintainers, but neither of them builds 32 bit versions.

This leaves those of us who rely on FFMPEG for AVISynth processing with a choice: trust less popular/probably not vetted sources for new 32-bit versions of FFMPEG, or build the source code ourselves. As this thread on Doom9 suggests, the latter seems to be the more popular choice:

http://forum.doom9.org/showthread.php?p=1923902#post1923902

In particular, commenters there suggest using a set of scripts called media-autobuild_suite.

https://github.com/m-ab-s/media-autobuild_suite

Since everything involved with MABS is open source and well tested, I'm going to be using this solution for 32-bit from now on unless it absolutely breaks down on me. It'll mean a little extra complexity, but gains the benefit of being able to control what's compiled into FFMPEG.

It also means that I'm probably going to have to go back and update several of my tutorials, which I don't have a lot of time to do these days.

Going forward, I'm going to recommend binaries from the official FFMPEG Windows binary maintainers for 64-bit, and building from source for 32-bit. 

Personally, I wish I could go 100% 64-bit at this point, but there's still a few important 32-bit AVISynth plugins that I need to use, So until those either get updated or become obsolete, that's going to be the situation. 

Two different solutions for Denoising video with AVISynth

Last year, I worked on a couple of projects that involved not just deinterlacing, but denoising. Some scenes had a fine, natural noise that I preferred to keep intact. Other scenes had so much noise it was difficult to make out the faces of the actors. Figuring out how to minimize the noise without sacrificing too much detail or looking jarringly different from the surrounding shots turned out to be a real challenge.

Here are some notes from my experimenting.

Commercial tools

There are some commercial tools for denoising, each with their own strengths and weaknesses. Here's three of the most common:

I've used Neat Video in the past, and it has some real advantages in terms of denoising strength, being able to manually select what to consider noise (if there's a large enough area of just noise in frame), and parameters to tweak. In particular, it's great at dealing with blotchy chroma (color) noise. It's really easy to overdo it, however, and sometimes it takes a lot of tweaking to remove only the noise you want.

There's also Red Giant's Denoiser III, which has a much simpler interface, and is better for even noise reduction, although not as great for dealing with truly terrible noise.

Then there's the Studio version of DaVinci Resolve. I don't own this and can't get a trial version, so I don't know how well it would perform.

If you're looking to denoise HD or higher resolution footage, I'd recommend one of the above. They're all fully GPU accelerated, focus on modern camera sensor noise, and they don't have a chance of introducing gamma or color shifts.

However, for SD interlaced video, I think AVISynth has some better options.

TemporalDegrain2

Let's start with TemporalDegrain2. With low to medium grain/noise, the default settings are usually fine. It uses many of the same requirement filters as QTGMC, so if you've already got QTGMC set up, you should be able to use TemporalDegrain2 at defaults. The one setting to pay attention to is grainLevel=True. Setting this to False may give better performance on some footage, so try it both ways to check.

Here's a brightened capture of some noisy SD footage (Note that I haven't done a pixel aspect ratio correction, so the image is slightly squashed):

And here's the same footage passed through TemporalDegrain2 at grainLevel=False:

And grainLevel=True:

The difference is subtle in this case, but definitely there.

More detailed instructions are included in the .avsi script if you want to play around with things, but in my experience, the defaults do the best job at denoising without undesirable artifacts.

Oh, and one more important thing: TemporalDegrain2 has a 64-bit version for AVISynth.

SMDegrain

Next, let's look at SMDegrain. If you thought QTGMC has a lot of options...

Using the recommendations for a starting point from the documentation:

SMDegrain(tr=2,thSAD=250,contrasharp=true,refinemotion=true,lsb=true)

Gives this result:

Not very impressive in this case. Let's try the recommendation for "dark scenes", and trigger the interlaced switch so the denoising can be done prior to calling QTGMC:

 SMDegrain(tr=3,thSAD=300,contrasharp=true,str=2.0,refinemotion=true,lsb=true,interlaced=true)

Which gives us this:

Better, but still not great. Again, this is a fairly noisy clip. Depending on the type/amount of noise, the defaults might be much more effective.

Now, let's look at the option for "grainy sources". I like to call this the "kitchen sink" option:

pre=fluxsmootht(3).removegrain(11)
SMDegrain(tr=6,thSAD=500,contrasharp=30,prefilter=pre,str=1.2,refinemotion=true,lsb=true)

What's happening here is that in addition to increasing the strength of denoising, we've added a prefilter. This is designed to blur the image first when calculating what detail to preserve, so the denoise can be that much more aggressive. If you just use a number here, SMDegrain will do some variation of a simple blur for it's initial calculations. In this case, we're using more intensive solutions. Here's the end result:

Now we're talking. However, you should know some things about the Kitchen Sink method:

• If you use the interlaced option it'll distort, so don't do that. You could use SeparateFields() first and AssumeFieldBased() plus Weave() after to try to preserve the interlacing while reducing the amount of time it takes to denoise, but in my experience, that makes the processing look crappier/lower resolution. Either use it after QTGMC, or use it before and accept that the motion may not always look quite right.
• You can do multiple passes, but since SMDegrain is 32-bit only, you're realistically limited to 2 passes without render freezes/crashes due to memory issues.
• Yes, SMDegrain is available in 64-bit for VapourSynth (via havsfunc, which also contains a port of QTGMC). There, you're more likely to be able to squeeze in 3 passes, but it'll be excruciatingly slow if you do that, and can end up overprocessing the image. Also, you'll need to pay attention to the documentation for the various options, as they sometimes use different capitalization than on AVISynth. Why? Basically, VapourSynth scripts use Python, and Python is case-sensitive. Also, you have to re-write the fluxsmootht and removegrain entries in a different way, and remove the lsb=true option (because VapourSynth doesn't need it).
• In footage with significant motion, you may notice more of a smearing effect, similar to the old-school electronic denoising on the laserdisk versions of the original Star Wars trilogy.
• If you use multiple passes, you can end up with a dithering effect that's very noticeable on dark footage.

Basically, there are some drawbacks. The good news is that you can get very close to the same results with a line like this:

SMDegrain(tr=6,thSAD=500,contrasharp=0,str=1.2,refinemotion=true,lsb=true,interlaced=true)

Which gives this:

This is easier to run, has less of a smearing effect, works better with interlaced footage, and I've removed the built-in sharpening (Contrasharp) so I can use LimitedSharpenFaster later instead.

For those interested, here's a condensed (AVISynth+) version of my .avs script using the above options:

SetFilterMTMode("QTGMC", 2)
SetFilterMTMode("f3kdb", 2)
AVISource("DV Noise Test intro.avi", audio=true)
ConvertToYV12()

SMDegrain(tr=6,thSAD=500,contrasharp=0,str=1.2,refinemotion=true,lsb=true,interlaced=True,Globals=2) 

QTGMC( Preset="Slower", EdiThreads=3 ) 

#--------------(Optional) Reduce chroma noise----------------------
#f3kdb(grainY=0, grainC=0, sample_mode=1)  

Prefetch(10) 

Since the original clip is DV video, I've done a colorspace conversion to YV12. If you're using a Digibeta file or ProRes capture, you might be able to get away without it, or you could convert to YUY2 instead.

Some other info that might be worth knowing:

SMDegrain has a variable called "globals" that allows you to change the way motion vectors are used. Basically, the globals are motion vector calculations, which can be either generated each time SMDegrain is called (default), passed through, or used from previous passes. In other words, if you want the same patterns of noise to be denoised in subsequent passes, you can use:

SMDegrain(tr=6,thSAD=500,contrasharp=0,str=1.2,refinemotion=true,lsb=true,interlaced=true,Globals=2)

for the first pass and:

SMDegrain(tr=6,thSAD=500,contrasharp=0,str=1.2,refinemotion=true,lsb=true,interlaced=true,Globals=1)

for the second. While doing so can reduce processor load a bit, it's still generally not a good idea to do three or more passes, even with globals=1.

From the documentation, here's how each number option works:

0: Ignore globals, just process

1: Read globals and Process

2: Process and output globals

3: Output globals only, don't process 

Doing a globals=2 pass and then a globals=1 pass results in the following:

Which is pretty darn clean. You may still notice some artifacts on fast-moving objects:

but that's true of any SMDegrain setting that I've tried. TemporalDegrain2 does better with these artifacts, but not so well with overall noise in noisy footage, at least at defaults:

Oh, and if you still have chroma noise after this, you might want to try messing with something like f3kdb.

Other notes

For comparison's sake, here's the best results I could get using Magic Bullet Denoiser III and Neat Video 5, applied to a version of the video already deinterlaced with QTGMC. Yes, both have ugly watermarks due to them being trial versions.

I could probably get Neat Video looking a bit better with some more tweaking, but this is what I would consider an acceptable denoise from it. If you look closely, you'll notice that Denoiser III retains some noise in this case, while Neat Video removes more, but has a slightly plastic look. This is the tradeoff in denoising - removing *all* grain often removes surface details you might want to keep. For an extreme example of this, check out the Predator Ultimate Edition Blu-Ray.

I may update this post later, but I think that's it for now. Post a comment if you have a question or correction.

AVSMeter - an essential tool for working with AVISynth

As part of the process for making tutorials about various AVISynth options, I often have to benchmark different settings and/or plugins. An easy way to do this is to simply run .avs scripts through FFMPEG.

However, sometimes this doesn't really show how efficient a script is being or not. If there's a bottleneck happening somewhere in the script that's holding you back from better performance, it can be hard to isolate it from the overhead that FFMPEG introduces. Plus, I don't really need to make a new rendered video file for every change in options when I'm trying to benchmark plugins.

Thankfully, there's a better way. The ever-awesome Groucho2004 has written a tool for this purpose called AVSMeter. It has pretty much everything that you would want in an AVISynth benchmarking tool, but it also serves as one of the best AVISynth diagnostic tools as well.

It has both a 32 and 64-bit option so you can compare between different versions of AVISynth, measurements of everything from memory usage to number of threads used. It has a function for checking your AVISynth installation to make sure that both the core program and all plugins are set up correctly. And of course, like any good benchmarking tool, it has comprehensive logging - including the ability to log to .csv files for you spreadsheet folks.

The included documentation does a really good job of explaining all the different features of AVSMeter, so I'm just going to post up a tutorial video about how to install and use it below and leave it at that.

BIG DISCLAIMER: This process may not work, may crash, or do other things to your system. Virus scanning all files is strongly encouraged, but not a 100% guarantee of safety.

You have been warned.

On ergonomics and carpal tunnel syndrome: a dictated post.

This blog was started as a way for me to talk about my experiences with video editing. I've posted up a bunch of different tutorials over the years, workflow tests, and introductions to different pieces of software.

However, the one thing that I haven't done is talk about the physical toll of working (and playing) on a computer all day.

I've developed issues with my posture over the years that have aggravated some lower back problems. This isn't the end of the world, as doing a few simple exercises can strengthen muscles that can counterbalance this effect. I'm working on doing just that right now.

Unfortunately, one area where not taking care of yourself can be a real problem is your wrists. For those unfamiliar with the condition, carpal tunnel syndrome is a RSI, or repetitive stress injury. It's primarily due to grasping for long periods of time and/or performing repetitive motions with your fingers like typing. The symptoms start with swelling and soreness in your wrist and hands, then progress to tingling and finally numbness

I first started having symptoms of carpal tunnel syndrome right around the time I entered college, and it was primarily due to playing video games and working on the computer for long hours. Probably my biggest regret in life is that I didn't take it seriously enough.

Most people who develop carpal tunnel syndrome symptoms recover entirely through a brief treatment of anti-inflammatory drugs and rest. However, if you continue the damaging behavior while you are recovering, the injuries can compound, and you can end up with a condition like mine where it never really goes away.

Thankfully, my symptoms have never gotten bad enough to require surgery, but they've held me back from doing a number of different jobs that could have made me a lot more money than I'm currently making. About a month ago, my symptoms started getting significantly worse. It turns out that holding up in iPad for hours at a time can be just as bad as performing a death grip on a mouse.

So, I find myself back in the same position again.

In the past, I've tried all of the noninvasive treatments for carpal tunnel syndrome. I've done the course of anti-inflammatory drugs, used a wrist brace, even tried acupuncture. The wrist brace didn't really help, the anti-inflammatory drugs helped alleviate the immediate symptoms, but not the overall condition. Acupuncture and massage helped about as much as the drugs.

However, the one bright spot is that the technology that in some ways has contributed to the condition in the first place is now helping me to reduce the repetitive motions that aggravate the condition.

I'm talking about speech-to-text software.

Back in the early-mid 2000's, I used a program called Dragon NaturallySpeaking from time to time. It was expensive, error prone, and slow to use. Dragon is still around, and is probably much improved in its current version, but thankfully I don't need to buy it anymore. Windows 10 includes speech-to-text-software by default. It works almost exactly the same as Dragon, and it's essentially free.

It's also what I'm using right now to write this blog post (as well as the previous post).

Unlike voice assistant transcription capabilities that are available in Android and iOS, Windows Speech Recognition doesn't require an Internet connection, is relatively quick, and doesn't timeout after a certain period of seconds and/or words. It's definitely not perfect, but by having a local dictionary to pull from, you can add and/or retrain words to match whatever subject matter or writing style you happen to use.You also have less of a worry that your voice will be uploaded to some distant cloud storage server and kept there for eternity.

It definitely takes some getting used to, but I find that I can write blog posts like this and all sorts of other documents fairly easily. It doesn't help with entering FFMPEG commands or writing AVISynth scripts, but it allows me to work on projects that otherwise would have been detrimental to my health.

Speaking of health, I'm currently looking into other treatments that could significantly improve my condition. If that works, I'll either update this post or make a new one. However, for anyone who's currently suffering with carpal tunnel syndrome. I want to let people know that there are ways to get back some of the functionality that you had in the past via speech-to-text software.

Why I don't (generally) like to use wrapper programs

Most of the comments on my videos and blog are questions about particular workflow issues. However, one of the most repeated questions outside of that has to do with why I don't use/recommend using programs like MeGUI or Hybrid.

The simple answer is that I don't trust that they'll stay around. The history of open source software is littered with wrapper programs that were really convenient for a brief moment in time, until life or lack of interest led to the (usually solitary) developer abandoning the project. Sometimes, someone else will come along and rescue it. More often than not, though, it's just left to rot.

This means that if you base your entire workflow around one of these programs, there's a better-than-average chance that at some point you'll either have to hunt for a replacement that doesn't exist, or learn how to use the programs that underlie the wrapper. After getting burned a few times by this, I have chosen the latter.

Now, this is not to say that I have an inherent prejudice against all wrapper programs. gMKVExtractGUI saves a ton of time and typing when trying to extract audio from an .mkv file, and is directly recommended by the MKVToolNix devs. My main concern is that if you start by learning wrapper programs and don't really understand the fundamental way that the underlying programs work, you'll be lost at sea if something goes wrong.

Nowhere is this more true (in my opinion) than in the case of FFMPEG.

There are a truly massive number of wrappers for FFMPEG. Most are designed to convert consumer video files into a format that works better for media players or phones. Some are free, while quite a few others are scammy $15-$50 converter programs that spam advertisements on places like Quora. A few are actually designed for professional usage, with their primary function being to add ProRes encoding to Windows (which is not as much an issue since Adobe added official ProRes encoding to Creative Cloud video apps).

Now, I want to make a distinction between wrapper programs and programs that merely incorporate FFMPEG. Programs like Handbrake, ShotCut and KDEnlive all use FFMPEG for exporting video, but don't exclusively rely on FFMPEG code to function. They are also projects with a long development history, dedicated teams of people who work on them, and large numbers of users.

Here's the thing, though. FFMPEG goes through significant development, sometimes leading to new features and important bug fixes. Its main ProRes encoder in particular has had major bug fixes within the last year, as I have posted elsewhere on this blog. Most wrapper programs -- and other programs that rely on FFMPEG -- don't update their included version of FFMPEG on a regular basis. In some cases, you might be using a version that's several years out of date. If you learn how to use FFMPEG just by itself, then you can update whenever you want, and potentially troubleshoot issues that the wrapper program hasn't fixed yet.

So, basically, that's my thought process. I completely understand that many people would prefer an all in one solution, especially for programs like AVISynth where the setup process is definitely not trivial. I just tend to gear my tutorials towards a sort of "manual transmission" philosophy because I believe that in the long run it's more helpful.

Motion Interpolation plugins in AVISynth (WIP)

(This post is a work-in-progress. I'll be adding screengrabs, more info and eventually a video tutorial. Keep checking back if you'd like to see more, and feel free to comment if you think I've left something out.)

I'm not generally a fan of motion interpolation. I especially hate the "motionflow" BS on most modern TVs. The only time I really like using it is with QTGMC, and that's because it's helping to recreate full frame detail from half-height fields.

However, I realize that motion interpolation can be useful in other situations. In addition to allowing you to make a slow motion shot out of footage shot at a regular framerate, you can also use it to do framerate conversions without changing overall runtime, such as going from 29.97fps to 23.976fps.

There are a number of different options for performing motion interpolation. As far as I can tell, Twixtor is pretty much the gold standard for "run it yourself" software in pro circles, but it has the notable limitation of not being able to detect scene changes. That means that if you're working on more than one shot, you would need to apply the Twixtor effect to every *individual* clip in a timeline in order for it to work properly rather than just applying it after exporting. However, Twixtor is really good at what it does, and motion is less choppy overall than other solutions.

Unfortunately, Twixtor is also $395, or closer to $600 if you want the "Pro" version that has more features for VFX artists.

So what if you're broke, or don't use one of the editing programs that Twixtor supports? As it turns out, AVISynth does actually have some interesting alternatives. They have their own issues, but they can definitely get the job done.


Update: Here's a rough draft of the video tutorial for setting up and using the plugins. I plan on releasing a more comprehensive video later, but if you'd like a preview, check it out:

FrameRateConverter requires MVTools2, MaskTools2, and the fftw3 library, just like QTGMC. *Unlike* QTGMC, however, if you want to use the 64-bit version, you'll need to install the 64-bit version of fftw3 in your System32 directory.

It has a wide range of supported colorspaces, comes in both 32 and 64-bit versions and the quality is good overall. However, the speed ranges quite a bit depending on the settings used.

Using a script for converting a 29.97fps HD ProRes file to 59.94fps, I got speeds ranging from totally unusable (32-bit version, "Slowest" preset, which estimated almost 3 hours to finish) to 1:37 (64-bit version, "Faster" preset). The 32-bit version was exponentially slower than the 64-bit version at slower presets, and inconsistent at faster ones, sometimes varying by several minutes. By contrast, the 64-bit version uses more memory, but is faster and has almost no variation in run times.

As to visual quality, FRC looks like it uses a form of partial frameblending, which can make some motion look a little less smooth than Twixtor (or the other filter I'm looking at below). On the plus side, it does have scene detection, and there is little-to-no warping of the image.

Here's a sample script (AVS+ syntax):

SetFilterMTMode("FrameRateConverter", 2)
FFmpegSource2("The Hollar Commercial 1 - ProRes422HQ.mov", atrack=-1)
FrameRateConverter(Preset="Slow")
Prefetch(10)

One thing to note is that if you're not doing a straight frame-doubling, you have to use a somewhat odd system for specifying framerate. You have to specify the "NewNum" and "NewDen",  which corresponds to a numerator and a denominator, somewhat similar to how FFMPEG handles framerates. For example, if I wanted to convert to 59.94fps, I'd use a line like this:

FrameRateConverter(Preset="Slow", FrameDouble=false, NewNum=60000, NewDen=1001)

Likewise,

NewNum=24000, NewDen=1001 

would get you to 23.976 and

NewNum=30000, NewDen=1001 

would convert to 29.97. Keep this syntax in mind, because it's how the next filter works as well. Speaking of which:



Interframe requires SVPFlow, a version of which is included in the download.

It only works in YV12 colorspace, doesn't have a 64-bit version, *really* doesn't like working with FFMPEGSource when using 10-bit material (even after colorspace conversion), and the "beginner's guide" for it focuses on using MeGUI. Oddly enough, it also has settings for stereoscopic 3D content.

However, it absolutely mops the floor with FrameRateConverter when it comes to speed, and it doesn't have the frameblending artifacts.

Using the slowest preset (which is called "medium", for some reason), it ran anywhere from 19-40fps. You can enable GPU support, but this doesn't actually speed up the preset. Instead, it supposedly works harder to give better results. In practice, I don't notice a massive difference, but you can try it out for yourself and see which settings you prefer.

Here's a sample command:

SetFilterMTMode("InterFrame", 2)
QTInput("The Hollar Commercial 1 - ProRes422HQ.mov", audio=1)
ConvertToYV12()
InterFrame(Preset="Medium", Tuning="Smooth", GPU=True, Cores=10)
Prefetch(10)

If your input video is in 10-bit color, you'll need to use an input filter other than FFmpegSource2. In this case, I'm using the massively outdated QTInput. If your input video is in 8-bit color, you don't need the ConvertToYV12() command, and FFmpegSource2 should work just fine.



Overall, the following distinction applies:

Interframe preserves more detail in "Weak" mode, but objects in motion can have wobblier edges. In "Smooth", motion looks smoother, but less detail is preserved. Both modes can have garbled edges around moving high-contrast edges.

FrameRateConverter has a more "frameblended" look, with some choppier/blurred movement, but almost no edge artifacts. It can have issues with certain patterns of detail in some shots, though. It generated a "twitchy" artifact around the edge of a corrugated metal roof during a panning shot in one video I used it on.

Twixtor is like a more refined version of Interframe, with better edge cohesion, but still some noticeable artifacts at default settings. In particular, it has a larger "warped" area.



There are a couple of other options I've played around with, but none have been as good.

YFRC is a script that requires MaskTools2 and MVTools. Like some of the other options, it needs YV12 colorspace. On the plus side, it supports common framesizes from 320x240 up to 1920x1080, although you have to set the block size appropriately. It also supports multithreading and 64-bit mode. You can change the framerate conversion method by changing OverlayType. OverlayType=0 uses frameblending, while OverlayType=1 doesn't blend frames, but can be choppier as a result as it doesn't fully interpolate between frames in many cases. Neither is as good overall as Interframe or FRC, but YFRC does have the best scene detection of any of the options available, with no blended/interpolated frames between cuts.

ChangeFPS is the default framerate conversion command in AVISynth. It isn't multithreaded, but does work in 64-bit mode, and can use whatever colorspace you throw at it (at least in AVS+). Unfortunately, it's about the same as using the basic framerate conversion tools of Premiere Pro/Resolve/etc, which is to say it looks pretty choppy. It does no motion interpolation, instead simply adding or removing frames as necessary.

ConvertFPS has most of the same benefits of ChangeFPS, but can either do frameblending "Blend" mode (the default) or a "Switch" mode that looks like the resulting video was captured from a game with Vsync disabled. The Switch mode also requires 8-bit YUY2 colorspace, so you'll have to do those conversions first. I don't recommend using the Switch mode.

MSU_FRC does an okay-decent job, but is limited to YV12 colorspace *and* base16 block sizes, so it only really works with SD video frame sizes. It's also not multithreaded or 64-bit capable, which makes it slooooow.

FrameDbl only does straight framedoubling, does so via blending frames, and does not use multithreading. Also, it has the same issue with 10-bit video in FFMPEGSource as Interframe.

Motion is ancient, has confusing documentation, is not multithreaded (in fact, multithreading slows it down significantly), and produces nasty-looking artifacts.

Finally, there's the minterpolate filter in FFMPEG. Using the recommended settings from the FFMPEG Wiki, I came up with a command like this:

ffmpeg -i "input.mov" -filter:v "minterpolate='mi_mode=mci:mc_mode=aobmc:vsbmc=1:fps=60'" -c:v prores_ks -c:a copy "output.mov"

The end result is very smooth and has excellent scene detection, but with a more distorted "warped" area around moving objects, and more "blockiness" in those areas. It's also glacially slow; neither multithreaded nor GPU-accelerated. There's a very minor speedup from using the 64-bit version in my limited testing.