Tutorial - FFMPEG issues - Resizing color shift, Adobe weirdness, etc.

FFMPEG is an amazing program. I frequently use it to convert obscure video formats to something usable, export from AVISynth scripts, and swap audio tracks in and out of videos. It does have some quirks, however, and if you aren't on the lookout for them, they can change your video in ways you might not like. Here's a few of them:

BIG DISCLAIMER: This process may not work, may crash, or do other things to your system. Virus scanning all files is strongly encouraged, but not a 100% guarantee of safety.

You have been warned. 

Also, this tutorial is for Windows 10. Most of the steps work for other OSes, but I won't cover the differences here.

Here's a video version of the tutorial:

The SD to HD resizing bug

Resizing in FFMPEG is done via filters, which are specified by the -vf switch. -vf has so many filters it would be impossible to describe here, but the structure for how each is called is basically:

-vf "filter1=optiona:optionb:optionc, filter2=optiona:optionb:optionc"

Of course, you won't necessarily use that many (or even any) options depending on the filters you choose.

A simple resize and convert to ProRes 422HQ would be:

ffmpeg -i "inputvideo.mp4" -vf  "scale=1280:720:flags=spline" -c:v prores  -profile:v 3 -pix_fmt yuv422p10le -c:a pcm_s16le "outputvideo.mov"

Note how the resize comes before the codec setting. Order of commands is very important in FFMPEG, and putting things in the wrong order can result in errors or quality issues.

The above command would resize the source video to 1280x720 using the Spline resize method, similar to what I've used in AVISynth tutorials in the past. For more info on FFMPEG resizing, check out the wiki entry here:

https://trac.ffmpeg.org/wiki/Scaling

So what's the issue? Well, if you're resizing from a standard definition video (SD) to a high definition (HD) framesize, you might get a color shift... depending on the codec you use. In my testing, DNxHD/HR and H.264 have this issue, while ProRes does not. Check out the following frame grabs:

This is the original image from a DVD version of a documentary I worked on:

Yes, it's supposed to look like that. The DVD is encoded in anamorphic widescreen, when means it takes a 16x9 image and squeezes it to fit inside a 4x3 frame. Your DVD player can then un-squeeze the image back to 16x9 for widescreen TVs.

This is the video upscaled to 720p using FFMPEG:

Notice the color shift? It's subtle. It's most prominent on the lower-third graphic and the warden's face.

So, why is this happening? Basically, SD video conforms to a broadcast standard known as bt.601. HD video conforms to a very slightly different standard known as bt.709 (otherwise known as Rec.709). FFMPEG does not automatically correct for the differences between the two standards, and this results in a color/brightness shift.

Luckily, you can force the conversion by adding colormatrix=bt601:bt709 to the filters entry, like so:

ffmpeg -i "inputvideo.mp4" -vf  "scale=1280:720:flags=spline, colormatrix=bt601:bt709" -c:v dnxhd -profile:v dnxhr_hqx -pix_fmt yuv422p10le -c:a pcm_s16le "outputvideo.mxf"

and the result is this:

Viola! Image fixed.

However, don't use this filter unless you see a color shift. Otherwise, you might end up creating a color shift where one didn't exist before. Oh, and you can also convert the other direction as well if you're downscaling to SD from an HD video; just switch the options around:

colormatrix=bt709:bt601

Bottom line? Always check your exports before sending them out or dropping them in your edit timeline. You'll save yourself a lot of work later trying to manually correct them.

 

Color shifts in Adobe products

Update: the ProRes color shift I had up here before is gone in current nightly builds of FFMPEG. If you're still stuck on an older build that creates a color shift when the exported file is viewed in Premiere Pro or After Effects, use the prores_ks encoder rather than prores, and it'll correct the issue.

This glitch is a bit of a puzzler, frankly. Basically, FFMPEG's DNxHD/HR exports have a color shift, but only in After Effects and Premiere Pro.

This is really subtle, so I'm going to zoom in.

Here's the original:

And here's the DNxHD/HR encode viewed in After Effects:

If you still can't see it, use your left and right arrow keys (or tap on the pictures if you're on a mobile device) to move back and forth between the images. Basically there's a slight green/yellow shift.

Now, here's the kicker. If I import these files into DaVinci Resolve, they look identical.

So, what the heck? I'm not entirely sure. If I had to guess, I'd say that FFMPEG is generating metadata that Adobe programs don't like, but it could be something else.

Fortunately, after testing and Googling, I've figured out how to fix the color shifts.

All you have to do is change the container format to .mxf instead of .mov:

ffmpeg -i "inputvideo.mov" -c:v dnxhd -profile:v dnxhr_hqx -c:a pcm_s16le "outputfile.mxf"

And this is the result:

The only issue with the DNxHD/HR solution is that oddly enough, FFMPEG-generated .mxf files don't work in Resolve (and Assimilate Scratch Play). Resolve will refuse to load them, and Scratch Play will load but will only play the audio from the file. Adobe programs will handle them just fine. However, Resolve and Scratch Play will recognize and play back the .mov files with no color shift issues. 

So, yeah. Weird.