Friday, January 4, 2019

Tutorial - An alternate workflow for transcoding DVDs to editable video files

MakeMKV has been my go-to solution for DVD extraction for a while, but it sometimes can create files that have issues when loaded into AVISynth. In those cases, there a couple of things you can try to fix the issue, or you can use a different workflow altogether.

Briefly, the steps for this alternate workflow are as follows:

  • Copy over the DVD's files (if DVD is unencrypted) or use a decryption program and point the output to a directory on a local drive.
  • Determine what title you want to convert.
  • Open up the appropriate .vob files in DGIndex. Save project and demux audio.
  • (Optional) Convert audio to .wav using FFMPEG or Audacity with FFMPEG add-on.
  • Create AVISynth script
    • Load DGIndex project file
    • Load audio
    • Mix audio and video
    • Perform any additional video processing
  • Render out avs script to ProRes file using FFMPEG

It's a bit of a hassle, but I know from experience that it works.


BIG DISCLAIMER: This process may not work, may crash, or do other things to your system. Virus scanning all files is strongly encouraged, but not a 100% guarantee of safety.

Circumventing lawful copy protection is illegal in the United States and many other countries. If you do not have explicit, written permission of the content owner to rip their DVD, you may be committing a crime. Check your local laws to be sure. I am not a lawyer.

You have been warned.

Also, this tutorial is for Windows 10. Most of the steps should work for other OSes, but I won't cover the differences here.

Here's a video version of the tutorial, with some extra info about using MKVToolNix to extract A/V data from MakeMKV DVD rips in order to work around some glitches:

Please note: the 32-bit only limitation mentioned in this video is no longer true. You can ignore that part and essentially follow the process from DGIndex forward in 64-bit AVISynth+.



First, if you haven't already set up AVISynth+ or FFMPEG, check out the following tutorial:

https://macilatthefront.blogspot.com/2018/10/alternate-setup-for-ffmpeg-alternate.html

If you want to install the 64-bit version of AVS+ instead, check out this updated tutorial (that uses VirtualDub2 for rendering, but you can grab 64-bit FFMPEG and use that instead if you like):

https://macilatthefront.blogspot.com/2021/01/deinterlacing-with-avisynth-and-qtgmc.html


When that's done, grab DGMPGDec:

http://rationalqm.us/dgmpgdec/dgmpgdec.html

Depending on the type of DVD you're looking to convert, you might also need:

http://avisynth.nl/index.php/Decomb

for working with 24fps content, and QTGMC for interlaced content. For QTGMC setup, see my tutorial posts above.

Virus scan all downloaded files. 

Extract the DGMPGDec archive into a directory on your system, maybe named "DGMPGDec". If you can't decide where to put the folder, Program Files (x86) or your user folder work well. Copy the DGDecode.dll file to your AVISynth+ plugins+ directory (The main version is 32-bit, but there's a 64-bit now included in the same archive if you want to go that way, which you would put in plugins64+ instead).

Create a folder somewhere on one of your drives for the DVD files, named whatever you want. Put a DVD in your drive.

If the DVD isn't encrypted, just copy over the VIDEO_TS folder or the .VOB files contained within to your new folder.

If it is encrypted, you'll need to use a decryption program of some sort. Due to the various legal issues involved with DVD ripping programs, I'm a little reluctant to link to any specific programs (MakeMKV is technically illegal in some countries as well, but makes some concessions, such as not removing certain types of copy protection on Blu-ray rips that prevent them from being remuxed to a new Blu-ray disc image), and I would caution you to be careful - in addition to my warnings above, semi-to-fully illegal software is a common target for malware authors to try to inject code or compromise the websites of said software in some way.

Regardless, once you have your unencrypted .VOB files copied over, you'll need to find the title you want to transcode.

Often, this is made up of the largest similarly-named files on the disc, but if you don't know what title contains your desired content, use VLC to play back the .VOB files until you find the correct one.

When you know what title you want, open DGIndex from your DGDecode folder. Go to the Audio menu, select Output Method and then Demux All Tracks. Go to the File menu, then select Open. Browse to the folder with your DVD files. Select all the .VOB files in a particular title. They will usually be named with VTS_(title number)_(part number).VOB. For example:


Notice that I don't select the "_0.VOB" files - they're not needed in this case. Click Open, and then you'll get a "File List" window. Confirm that you have all the files you need, or add any you missed. When finished, click OK.

Drag around the playback bar to make sure the entire title is present. If you'd like to get a sense of what DGIndex thinks your video's framerate/aspect ratio is, hit the F5 key and preview it for a minute or so (you can stop playback with the ESC key). When satisfied, go to the File menu and select Save Project. I generally just save it in the same directory as the DVD files, and will proceed as if you have done so.

One thing to look out for: if you get a box like this:


Generally, I will click "No". However, if the resulting project file gives you an "access violation" error during preview or encoding later in the process, then you might need to go back and re-save the project file, than click Yes. (Thanks to C. Sandor for reminding me about this).

Anyways, Save Project will create a .d2v file and at least one audio track in the same directory.

If you'd like, you can convert the audio file to a Wav file ahead of time using FFMPEG like so:

ffmpeg -i "audio.ac3" -c:a pcm_s16le "audio.wav"

The resulting file should work in editing programs either on its own or muxed with the video file we'll be creating in a moment. If the original file has more than 2 channels, they may not be in the correct order is you just use the command above. If you need to mix down to stereo, a common suggestion that I've used is the -ac 2 option, like so:

ffmpeg -i "audio.ac3" -c:a pcm_s16le -ac 2 "audio.wav"

Alternatively, you can use the NicAudio AVISynth filter to import the AC3(or DTS) file, although I've sometimes run into sync issues when using it.

Create a new text document, then change the extension to .avs. Double-click to open it in Notepad or AvsPmod.

Add the following to the newly-created avs file:

video = MPEG2Source("yourproject.d2v")
audio = WavSource("audio.wav")
AudioDub(video, audio)
Spline64Resize(1280, 720)

The above assumes you want to upscale to 720p from an anamorphic widescreen DVD. If your video is 4:3 aspect ratio, then you can use

Spline64Resize(720, 540)

instead to correct for the pixel aspect ratio difference between NTSC video and modern displays.

If you want to import the .ac3 file directly, install the NicAudio plugin into your AVS+ plugins directory and use:
video = MPEG2Source("yourproject.d2v")
audio = NicAC3Source("yourproject T80 2_0ch 192Kbps DELAY 0ms.ac3")
AudioDub(video, audio)
Spline64Resize(1280, 720)

Change as necessary. Again, use at your own risk, as it can cause sync issues.

For 24fps DVDs, you'll need to use the Decomb filter to perform an "inverse telecine" to extract out the 24 progressive frames from the 29.97fps interlaced video.

video = MPEG2Source("yourproject.d2v")
audio = WavSource("audio.wav")
AudioDub(video, audio)
AssumeBFF()
Telecide()
Decimate(cycle=5)
Spline64Resize(1280, 720)

For interlaced video, you'll need to use QTGMC:

SetFilterMTMode("QTGMC", 2)
video = MPEG2Source("yourproject.d2v")
audio = WavSource("audio.wav")
AudioDub(video, audio)
AssumeBFF()
QTGMC(Preset="Slower", FPSDivisor=1, EdiThreads=3)
Spline64Resize(720, 540)
PreFetch(10)

Your EdiThreads and PreFetch settings depend on your processor. Mine has 6-cores with Hyperthreading, for a total of 12 "logical processors". EdiThreads should be set to half your physical cores or less, PreFetch to 1-2 less than your logical processors. FPSDivisor as included in the above command does nothing, letting the framerate be doubled to 59.94fps. However, if you change the 1 to 2, it will divide the framerate in half, resulting in a cleanly derived 29.97fps. Match the framerate to the project you're importing the video into, or the deliverables spec for your ultimate destination.

When you're ready, create a new text file in the same directory, then change the extension to .bat. Right-click to open it, then type something like the following:

ffmpeg -i "your.avs" -c:v prores -profile:v 3 -c:a pcm_s16le "output.mov"

If you're using a DTS file for audio, then -c:a should be pcm_s24le. If you're using DNxHR/HD for your output codec instead of ProRes, you may need to add a colormatrix conversion to avoid a color shift bug on SD-HD upscales:

ffmpeg -i "your.avs" -vf "colormatrix=bt601:bt709" -c:v prores -profile:v 3 -c:a pcm_s16le "output.mov"

If you need more info on colorspace conversions using pix_fmt, check out the ProRes section at the following link:

https://trac.ffmpeg.org/wiki/Encode/VFX

And that's it. Simple, right? ;)

15 comments:

C.Sandor said...

Thanks for the writeup. I first started to use QTGMC a few years ago (2015?), and was using essentially the method you describe (DGIndex). Even as recently as the summer of 2018 I was having success with it. Then about a week ago I tried to encode a video and mysteriously was getting errors with DGDecode - specifically, getting "Access Violation" errors. I uninstalled all components and re-installed, to no avail (was using AVS 2.6 from 2015 through last week). I stumbled on your blog and followed your AVS+ with QTGMC tutorial (nice writeup on that too, btw!). Still nothing when using DGDecode. Using FFMpegSource2 seems to work, where the video becomes available for viewing... but I've been using Handbrake for many years to actually do the encoding, and that's where it falls apart again - Handbrake correctly parses the clip, and can even do a preview encode, but if I have it encode the entire video, the final product ends up completely blank with the correct length and resolution! I am not opposed to handing the AVS script off to FFMpeg directly (as that's what Handbrake does anyway...), though I haven't tried it yet. Have you ever run into this kind of thing at all?

Andrew Swan said...

I've never tried using Handbrake to render AVISynth scripts. I didn't even know it could do that, to be honest. Now, I'm going to have to try it just to see how it works. Changes to newer versions of Handbrake might be causing the errors you describe, though.

Using FFMPEG directly is probably going to be your best option. If you've been using Handbrake, then I would guess that you want to encode to h.264 with AAC audio? If so, a simple FFMPEG command would be something like this:

ffmpeg -i "yourscript.avs" -c:v libx264 -preset slow -crf 18 -c:a aac -b:a 384k "outputfile.mp4"

The crf setting for x264 uses an inverse quality scale, so the lower the number, the better the quality/larger the filesize. 18 is the setting recommended by the FFMPEG wiki for "nearly lossless visual quality", but I would suggest playing around with it if the filesize is too large.

Here's the h.264 page on the FFMPEG wiki if you need more info:

https://trac.ffmpeg.org/wiki/Encode/H.264

C.Sandor said...

Yes, h.264/aac... But let me take a step back! Handbrake does not handle AVS directly, I use AVFS to pass a virtual AVI to it. Why? Well, it just happened to be the way I managed to get working back in 2015 and stuck with it up until the errors that cropped up last week. I should also point out that I have not updated the Handbrake installation in all this time either, which is all the more puzzling why DGDecode all of a sudden started generating an error.
In any case, I can figure out the exact parameters that Handbrake passes to FFMpeg and just pass the AVS directly.
One thing I noticed - back when I had my overly complicated method working, I would consistently get a doubled constant framerate from what the original MPEG2 stream was encoded as. 29.97 -> 59.94, for example. In messing around and getting FFMpegSource2 to work, I was getting a variable framerate (~48, as most of the video is progressive frames at 24fps, only short portions are interlaced at 29.97)... And that was after having to put TFF() into the AVS; otherwise I would get a stream where frames would "jump" backwards. I'm sure there's a technical term for that, pardon my ignorance.
Is there a way to guarantee a smooth stream that has a constant framerate double that of the original video?

Andrew Swan said...

I believe the term you're looking for is a field order mismatch. You can check your field order (Top or Bottom) when using the Preview function in DGIndex if you're ever unsure.

I've never used AVFS, so I can't comment on that, but using that would make sense if you were trying to load the script into Handbrake. It might be introducing some additional processing into the workflow that's throwing something off, though.

At default settings, QTGMC assumes that you're working with fully interlaced footage. You can use it with 24psf (23.976 framerate content in a 29.97i video stream), but results may look off as a result. A variable framerate, though? Weird. I'd have to see your .avs script and maybe the Preview info window from DGIndex to figure that out.

You might consider trying the Decomb method like I describe in the post above first:

Telecide()
Decimate(cycle=5)

with QTGMC just to confirm that the video is actually 24psf. If it doesn't work, then you might be working with either a non-standard "cadence" (pattern of displaying progressive frames in interlaced video), or the content is actually recorded at 29.97fps. If it's an anime DVD (in which case, please make sure you're doing something legal with it), then either of the above might be the case.

C.Sandor said...

I thought I'd share an update on where I've ended up. It's been an exceedingly frustrating few days regarding the issues I've been having, and I've gone all the way around to ending up where I was before the trouble began - and I'm not particularly happy about any of it.

Using FFMpegSource2 did allow me to successfully pass an AVS to FFMpeg for encoding - but I was having issues with the resulting framerate (was always getting multiples of 24 fps, unless I explicitly forced it into a different rate) and getting the original AC3 audio to sync properly. After struggling with various settings in AVS, I finally went back to troubleshooting the access violations I was getting with D2V files. This led to me disabling DEP as a possible fix; while I still have it off, I ultimately don't believe this to be the root cause and will re-enable it.

What finally transpired was this:
I hadn't mentioned it in prior comments, but after DGIndex had parsed through the VOB to save the associated D2V file, it would put up a message that it had detected a field order transition. I had seen this with various videos in the past (going all the way back to 2015), and usually letting DGIndex save a "fixed" D2V (in addition to the original) would lead to better results with field order mismatch issues (thanks for pointing out the term!). So when this happened with these VOBs, I thought nothing of it and let it generate the fixed D2V. On a whim, I altered an AVS to have the "bad"/original D2V loaded instead of the fixed one. Lo and behold, the bad D2V works!! I have no idea why the fixed D2V causes an access violation - and at this point, I don't particularly care, as I can fix any field order mismatch with AVS. The resulting encoded AVC stream requires an audio delay of 3250ms for the audio stream, but it's fully synced! WHEW

I'd like to thank you for your writeups, they're very informative, keep up the good work on that! And another thanks for your advice while trying to get this back in working order!!

Andrew Swan said...

Ah, the field order transition thing. Yeah, that's tripped me up in the past too. I need to put a note in the tutorial about that. Basically, I don't have it "fix" transistions unless there's an obvious problem.

Regardless, glad you got the process working, and thanks for your kind words.

Unknown said...

A question from a novice.

In your first alternate method, the DGindex shows that your film clip is interlaced but when you go AvsPmod you state that you aren't going to do any de-interlacing. I'm confused. Also I've got several questions if you have time to answer. My DVD has no sound due to being a 8mm film transfer. Is there a null or silent sound track I should use or just ignore encoding sound with it? Second my source is progressive, same frame rate, etc as yours. When displaying and stepping through it in AvsPmod it has two identical frames back to back all through the film clip. I would assume that is normal, but would like to know how that would affect functions such as de-noising. Last I assume since its 8mm film transfer file in a 720x480 frame and an aspect ratio of 16:9 that I should crop it before any applications of filters and functions in AvsPmod?

I would also like to thank you for your tutorials. I can appreciate the time that you put into them.

Andrew Swan said...


First of all, thanks for the support.

For the DGIndex issue, all SD video is technically interlaced. In the case of the 29.97fps DVD I processed without deinterlacing, the reason I did so is that the project framerate I used was 29.97 progressive, which means that even though you end up with interlaced video when rendering out to DVD, both interlaced fields in a top/bottom pair are showing part of the same frame. This means that it's possible to combine the two interlaced fields together and get the full frame detail without deinterlacing. Both MakeMKV DVD rips and the DGIndex method seem to allow you to do this. It's a weird combination that isn't commonly used, so normally you'd be doing the other processes of deinterlacing or IVTC instead.

If you're talking about the 23.976 video, then it's using a particular "cadence" or pattern of frames embedded in the fields, which means you'll either have to do an IVTC step to extract the clean progressive frames out of the interlaced video, or if it's a DVD you can rip the disc using MakeMKV and have the framerate embedded in the resulting .mkv file's metadata.

As to your DVD transfer of an 8mm movie, you shouldn't have a problem going without sound, but you can easily add one if you want by importing your video file into an editing program of your choice, adding the audio you want (making sure it doesn't go past the length of your video), then export out just the audio track and combine it with your video via FFMPEG. You'd use a command similar to this:

ffmpeg -i "original_video.mov" -i "mixed_audio.wav" -map 0:v -map 1:a -c copy "output_video.mov"

The framerate issue is an interesting one in the case of 8mm. Silent 8mm is often shot at 18fps, which can cause duplicate frames in a different pattern than an IVTC is expecting. I would try using the Telecide() filter just to see if it eliminates any interlacing artifacts, but beyond that it's up to you. Denoising is not my area of expertise (although I'm looking into it), but generally working with progressive video is easier than interlaced.

On the aspect ratio question, if you have a 16x9 *letterboxed* image (black bars at the top and bottom of the image), then I personally would do any resizing before most filters (other than removing interlacing), but if you want to do denoising, you should probably do that first.

Plubbingworth said...

So does demuxing the MKV that was created with MakeMKV allow you to not worry about audio delay that you would see in the title of an ac3 file that was created by loading a vob directly into dgindex?

Several of my ac3 files, but not all, have "DELAY 0ms" in their titles now, whereas the delay is different for the latter method.

Andrew Swan said...

@Plubbingworth - Good question. I'd have to test that to be sure, but it's possible. If in doubt, I'd say try a Handbrake conversion first as a sync reference and compare with your results.

Simion Alex said...

Hi there!
I actually watch your tutorials for some time and they're very helpful. I used Avisynth with QTGMC to deinterlace footage from a DV camera and never had any issues. Recently I started doing this to footage from DVDs using DGindex as well as Avisynth and QTGMC, but in some parts of the video where there is a lot of motion, it looks like QTGMC is discarding one filed, because every two frames are identical. Does QTGMC do this in high motion parts of the video? (I did the DV video like a year ago on a different installation of windows with most likely different versions of Avisynth, QTGMC and the other filters)
Thanks a lot in advance!

Andrew Swan said...

@Simion

Sounds like your DVDs are originally 24p material. If that's the case, you should use an IVTC method to remove duplicate fields rather than QTGMC. The Decomb method listed above should work, but there are other options as well.

Simion Alex said...

I was thinking this was something simple, but I need to give more details. I’m from Europe so everything should be PAL, but I have 2 DVDs that are NTSC, I discovered that those were poorly transferred from PAL (most probably by mistake after somebody edited them) without deinterlacing them first. So, the resulting video has awful motion after deinterlacing (0a 0b 1a 1b 2a 2b 3a 3b 4a 4b 4a 4b 5a 5b 6a 6b 7a 7b 8a 8b 9a 9b 9a 9b and so on). I tried removing the duplicate frames before and after deinterlacing. It worked the best removing them before deinterlacing and using ChangeFPS("pal_video") or Decimate(cycle=6) (the only difference is changefps gives me 25fps and decimate gives 24.975fps). Everything is ok, except when there is a lot of motion every two frames are duplicates.

Simion Alex said...

The comment was too long so I split it in two parts.
These are the proprieties of the input video in MediaInfo:
General
Complete name : D:\Serbare craciun gradinita tara 14.12.2009\originale\VTS_01_1.VOB
Format : MPEG-PS
File size : 1 024 MiB
Duration : 20 min 30 s
Overall bit rate mode : Variable
Overall bit rate : 6 979 kb/s

Video
ID : 224 (0xE0)
Format : MPEG Video
Format version : Version 2
Format profile : Main@Main
Format settings : BVOP
Format settings, BVOP : Yes
Format settings, Matrix : Default
Format settings, GOP : M=3, N=15
Format settings, picture structure : Frame
Duration : 20 min 30 s
Bit rate mode : Variable
Bit rate : 6 648 kb/s
Maximum bit rate : 9 641 kb/s
Width : 720 pixels
Height : 480 pixels
Display aspect ratio : 4:3
Frame rate : 29.970 (30000/1001) FPS
Standard : NTSC
Color space : YUV
Chroma subsampling : 4:2:0
Bit depth : 8 bits
Scan type : Interlaced
Scan order : Bottom Field First
Compression mode : Lossy
Bits/(Pixel*Frame) : 0.642
Time code of first frame : 00:00:00:00
Time code source : Group of pictures header
GOP, Open/Closed : Open
GOP, Open/Closed of first frame : Closed
Stream size : 975 MiB (95%)
Color primaries : BT.470 System M
Transfer characteristics : BT.470 System M
Matrix coefficients : BT.601

Audio
ID : 189 (0xBD)-128 (0x80)
Format : AC-3
Format/Info : Audio Coding 3
Commercial name : Dolby Digital
Muxing mode : DVD-Video
Duration : 20 min 30 s
Bit rate mode : Constant
Bit rate : 192 kb/s
Channel(s) : 2 channels
Channel layout : L R
Sampling rate : 48.0 kHz
Frame rate : 31.250 FPS (1536 SPF)
Compression mode : Lossy
Delay relative to video : -33 ms
Stream size : 28.2 MiB (3%)
Service kind : Complete Main

And this is the script I used in Avisynth:
SetFilterMTMode("GTGMC",2)
video=MPEG2Source("VTS_01.d2v")
audio=NicAC3Source("VTS_01 T80 2_0ch 192Kbps DELAY -33ms.ac3")
AudioDub(video,audio)
AssumeBFF()
ChangeFPS("pal_video")
QTGMC(preset="Slower",EdiThreads=1)
Spline64Resize(768,576)
Prefetch(1)

Andrew Swan said...

@Simion.

Eeee. Badly converted PAL to NTSC (or vice versa) is awful. Your solution sounds about right, Although you might also try a different duplicate frame removal plugin like Dup:

http://avisynth.nl/index.php/Dup

Regardless, good luck. That stuff is messy.

Which deinterlacing algorithm is the best? Part 1 - HD interlaced footage

Before I began, I'd like to give a special thanks to Aleksander Kozak for his help in testing and providing the HD footage used in this ...