Which deinterlacing algorithm is the best? Part 1 - HD interlaced footage

Before I began, I'd like to give a special thanks to Aleksander Kozak for his help in testing and providing the HD footage used in this post.

Also, this is part 1 of a series of posts. When I've finished part 2, I'll link to it here.

Background

One of the things about working as an editor is that the technology is constantly changing.  There are new pieces of software, plugins, and even hardware that are released on a regular basis, and keeping up with all of it can be challenging.

Roughly about 10 years ago, deinterlacing had four general types of solutions: 

- Hardware-based video conversion boxes like the Teranex that are designed to perform resolution and frame rate conversions in real time.  Simple, but not always the best quality, and the hardware is pretty expensive (especially when you consider that you also need both a separate dedicated playback device and a recording device in addition to the conversion box).

- Services like Cinnafilm's Tachyon that do the same sort of job using very complex and proprietary software and hardware on a rental (or on-premises, for a higher fee) basis.  Great results, but very expensive.

- Built-in deinterlacing within video editing programs.  I haven't used either Media Composer or Final Cut Pro X's implementation, but with everything else I've tried, this is garbage.  Unless your entire project from beginning to end uses interlaced settings, editing programs tend to toss out every other field and line double the remaining field, which cuts both motion detail and frame detail essentially in half. The only way I've seen to get around this inside an editing program is to use the FieldsKit plugin, which works a bit better than real-time conversion boxes, but not as well as SAAS options like Tachyon.

- Open source video filters.  For a while, your only options here were frame blending and "bob" deinterlacing, neither of which were particularly great.  Most early tools for dealing with interlacing in this space focused more on combining similar fields together - mainly as a result of them being geared towards processing TV captures of anime.  Slightly later methods like Yadif improved on deinterlacing quality, but were still only working on individual fields, using still image interpolation to essentially round off the edges of aliasing after line doubling.

That all changed when QTGMC came along. Like its less-optimized forerunner TempGaussMC (which I've never used), QTGMC is an AVISynth deinterlacing script that uses a combination of several different plugins. It separates out individual fields and tries to use data from surrounding fields to generate full resolution progressive frames from them.  This is very similar to what "motion flow" and other motion interpolation algorithms do on modern TV sets, but focuses on accuracy rather than wow factor.  While it's not trivial to set up, the end results are excellent, combining greater detail with highly configurable noise reduction and other options. In some ways, the results can actually be better than many of the above solutions.

But it's always good to do a reality check periodically - which of these methods works best? I don't have access to a Teranex or the cash to use Tachyon, so I'll be comparing mostly free options and/or ones embedded in >$300 programs.

TL:DR

If you don't want to read through the entire post, here's my opinion about each of the options that I currently have access to:

QTGMC: Best retention of original image detail and handling of edge case artifacts, but you have to watch for colorspace issues when working with Adobe programs if you render out using FFMPEG. Also, different options can change smoothing and noise reduction quite dramatically.

Topaz AI: Excellent results that can be occasionally thrown off by oversaturated colors and constantly fluctuating semi-transparent detail (like heat haze).  Best used for upconverting content to higher resolutions.  Also uses FFMPEG to render out results, so some codecs can have the above-mentioned colorspace issues.

DaVinci Resolve 17 Studio: Solid deinterlacing, but can also be thrown off by oversaturated colors.  Not as much smoothed detail as the other options, but in motion most viewers probably wouldn't notice.

Bwdif (FFMPEG): Good at reconstructing smooth lines and retaining overall image detail with SD content, faster than QTGMC but not as good.

Estdif (FFMPEG): Good at reconstructing smooth lines and retaining overall image detail with HD content, faster than QTGMC but not as good.

Yadif (FFMPEG): Just does interpolated line double of each single field rather than taking surrounding fields into account.  Diagonal high contrast lines have weird "rounded" jaggies that some (like me) might find displeasing. Very fast.

Bob, Weave, etc - These are all variations on very basic line-doubled deinterlacing. Bob would be my preference out of these, but in general I would only use them if you absolutely have to.

Premiere, Resolve (non-Studio), editing programs in general - Garbage. Cuts both resolution and framerate in half. Note that I'm only talking about the standard deinterlacing inside of these programs - you can get/install plugins that can do a much better job.

Let's check out some examples:

First clip - 25i HD material

This was a tough clip for the commercial options to deal with due to some issues with the very oversaturated reds, but I feel like it shows off the differences between deinterlacing methods pretty well. Also, the differences between these methods are much less noticeable in motion.

(Please click pictures to enlarge)

Original

This is what an interlaced frame looks like - two fields woven together. Obvious combing artifacts. If you look close, you can see the issue with the edges of the oversaturated reds - some interlacing artifacts don't line up with the interlacing lines. I've actually never seen this before, but I feel like it turns out to be an interesting indicator for what various methods actually do.

Bob (After Effects, Premiere Pro, many other editing programs)

This is just the original image run through After Effects. By default, AE uses a terrible method to display interlaced footage that drops every other field and does a simple line double for the remainder. However, it does get rid of the interlacing artifacts in the oversaturated red.

DaVinci Resolve 17 Studio

This is a recent edition to the paid ($300) version of Resolve that supposedly uses some "Neural Engine" processing for frame interpolation. Decent handling of edges, but tripped up by the oversaturated red. I would consider this a good solution for most work,

Topaz AI

Really good recovery of detail throughout the image, as well as adding some extra detail (that's kind of it's thing).  However, it's thrown off a bit by the haze around the railing, and once again doesn't deal well with the interlacing artifacts in the oversaturated red.

Now, let's check out the FFMPEG (and other programs that use FFMPEG) options:

Yadif

Works, but look at those railings. Remember, Yadif is just taking the individual fields and using smoothing algorithms to resize them. I personally don't like the results, but it's better than nothing.

Estdif

Much better, although some details like the lower contrast parts of the railing aren't as defined as I'd like. Unlike Yadif, it's at least attempting to do interpolation using surrounding frames, and as a result takes a speed hit. My recommendation for FFMPEG-only deinterlacing of HD content.

Bwdif

My favorite FFMPEG-only method for deinterlacing SD content does a bit worse here - the edges look a bit rougher and less defined. Still better than Yadif, though.

Now for QTGMC. To simplify the crazy range of options, I'm limiting myself to two presets - Faster for mostly no noise reduction and a focus on speedy processing, then Slower for best "all options" processing without getting into ridiculous processing times.

QTGMC - "Faster" Preset

Cleaner, more defined edges with more perceived detail.

QTGMC - "Slower" Preset

The key word here is "smooth". The extra processing makes the image look more defined overall IMO. My personal favorite, although Topaz AI definitely is another option if you're okay with its artificially added detail.

Addendum - Colorspace Issues

You might remember that I mentioned something earlier about colorspace issues?  The screen grabs I've shown above were all captured from DaVinci Resolve.  If I use After Effects to view the clips instead, anything rendered out by FFMPEG looks like this:

Don't see it?  Enlarge the image, then use the left and right arrow keys to flip between the other screen grabs.  You'll notice that the colors in the overall image are shifted, especially the reds.

So what's going on here?  Well, I'm still researching this at the moment, but as far as I can tell FFMPEG does some sort of screwy colorspace conversion on some footage. If you don't correct for this, then After Effects and Premiere Pro display the colors in the wrong way. 

What's confusing though, is that this doesn't happen on every file you export out of FFMPEG.  Only certain codecs have this behavior, and one of them is ProRes. Now, this isn't a new issue - I've run into a variation of this problem years ago when I was upscaling standard definition video to high definition resolutions using AVISynth and FFMPEG.  However, one of my original fixes - to use the other ProRes encoder - doesn't do anything this time.  Thankfully, the other fix - rendering out to DNxHD or DNxHR - does solve the problem.  So, if you want to be sure you don't run into any problems, use DNxHR. If you have to use both FFMPEG's ProRes output and Adobe programs, then make sure you use AVISynth to at least load the file first, and then you can add a conversion step in your .avs script:

ColorMatrix(mode="rec.709->rec.601")

Yes, you read that right: converting from rec.709 (the colorspace for HD video) to rec.601 (the colorspace for SD video) fixes the problem.

But be warned - this actually changes the way that the video looks for other programs. If you bring the video into Resolve, it looks like this:

And suddenly, we've gone orange.

Anyways, I'll keep looking into this and see if there's an easier fix. Currently, I've tried the fixes documented here:

https://stackoverflow.com/questions/37255690/ffmpeg-format-settings-matrix-bt709

and here:

https://medium.com/invideo-io/talking-about-colorspaces-and-ffmpeg-f6d0b037cc2f

With no success.

I'll end this post here, but look to the next post for a comparison between deinterlacing methods in standard definition.