On making an interlaced DVD from interlaced HD, and why it's such a pain in the @#$@#

I hate interlacing. Really, I do. It gives the illusion of greater motion detail than is there - but in the digital video editing realm, it adds all sorts of problems. This is especially true of interlaced high definition video, and doubly true when you want to turn HD interlaced video into standard definition (SD) interlaced video. Just about every method of making a DVD from an HD source assumes that you want to deinterlace the footage at some point along the way, which removes the benefits of shooting interlaced in the first place (and can end up looking very choppy if you aren't careful with your field order settings. Yes, there's actually a field order in interlaced video. Don't get me started...)

I use Premiere Pro CS3 to edit with, which is a very versatile program, but making a DVD directly out of an HD project in Premiere Pro results in crap quality, since Premiere decides to render all the elements at SD resolution (including titles, effects, SD footage, etc.) instead of at HD first and then downconverting to SD. In the past, the only solution to this has been to export out an uncompressed (or relatively low compression) HD movie file, then downconvert that somehow. Needless to say, if you're on a low hard drive budget and working on a huge project, this is a major pain in the ass.

One way to handle this is to take your rendered-out HD video and import it directly into Encore, but that can also create odd-looking results. For best quality, you generally have to downconvert the HD video first by making an uncompressed intermediate (16x9 anamorphic) SD file from the HD file, and import that into Encore. You can also just drop the HD file into After Effects in a new composition, change the composition to the DV widescreen preset, conform the image to fit, and render out to MPEG-2 from there. The problem with both processes is that they deinterlace the video automatically at some point along the way. I've gotten used to just doing this, and it creates a sort of pseudo-film look, but again, it removes the point of shooting interlaced video in the first place.

The HD2SD filter for AVISynth combined with the Debugmode Frameserver plugin for PPro CS3 allows a crazy, hacktastic workaround that essentially serves out a frame at a time of rendered-HD-then-downconverted-to-SD goodness to another program (in this case Virtualdub) so all you need to do is render out an SD intermediate file (in this case, using the lossless Lagarith codec). You can then import said intermediate file into Encore, and render out a properly interlaced DVD that preserves all the lovely interlaced motion...in theory. In actuality, Encore seems to have some trouble properly detecting the interlacing, and so once again deinterlaces the footage.

Also (by default), nothing in the AVISynth process chain is multithreaded, so if you have a multicore processor (or multiple processors, if you're a lucky dog) you will either have to suffer through single-core performance, or config everything in your render chain to be multicore, which is complicated and not guaranteed to boost your performance much (remember, any holdup in Premiere Pro's rendering will negate the advantages of multicore rendering further down the chain.) The upshot? the process is very, very slow. On my Core 2 Quad 2.4GHZ machine, it takes around 12 minutes to render every (one) minute of video... and that's with little to no effects.

So what's the solution? Well, there are basically two:

1. Play back the uncompressed HD footage through some sort of hardware downconverter into another device set up to record uncompressed video. Being that I don't have another computer around with another Blackmagic Intensity card, not happening here. This is by far the easiest, though, and happens in real time. Of course, there's still no guarantee Encore will recognize the SD footage as interlaced, and if you experience a playback skip during the recording, you either end up with two files you have to stitch together, or you have to start the recording over again. So, you probably should have the HD video on a RAID (0,5 or 6) array-equipped system to do this properly.
2. Pass the SD downconvert through a third-party MPEG-2 encoder that will detect the interlacing (or allow you to set it properly). So far, I've only been able to do this with the freeware HCEncoder, and the quality doesn't seem to be all that great - but it does produce properly interlaced video as long as you set the field order properly (usually Bottom Field First).

Update: I downloaded and installed TMPEGEnc Free, and that will also work with the SD downconvert - as long as you make sure the interlaced and bottom field first parameters are either detected automatically or set. The video quality is more consistent than HCEncoder for me, but that could also be because i'm not quite sure how to properly set the latter. Note that the MPEG-2 encoding in TMPEGEnc Free expires after 30 days, so use it wisely, and then consider buying the Plus version.

My guess is that most standalone encoding programs would produce similar results. Anyone have any issues with this sort of thing with DVD Studio Pro?

Mounting CDs/DVDs in Dosbox on Ubuntu 10.04

Since Ubuntu 10.04 uses the CD/DVD volume name for the subdirectory it mounts the CD/DVD to, you can't use

/media/cdrom0

as a standard CD/DVD mount directory in Dosbox like you could in the past. This means you either have to change your /etc/fstab file (as detailed here - use at your own risk! I have not tried this.), or you have to make separate entires in your dosbox.conf file for each CD you want to use. For example, if you wanted to mount the Full Throttle CD, you'd put this in the [autoexec] section of your dosbox.conf:

mount d /media/FT1_00 -t cdrom

You could then generate a list of CDs like so:

mount d /media/FT1_00 -t cdrom

mount d /media/DN_3D -t cdrom

mount d /media/SQVI -t cdrom

And use the commenting marks to select out everything except the CD you're trying to mount:

mount d /media/FT1_00 -t cdrom

#mount d /media/DN_3D -t cdrom

#mount d /media/SQVI -t cdrom

A front-end could potentially make this problem a non-issue, but I dislike using them, so there yah go. ;)

Oh, and remember that you can still use .ISO or .BIN/.CUE disk images as well (in fact, they're the *only* way to properly use multisession CDs in DOSBox at the moment.)

CD Ripping under Ubuntu 10.04

Need to rip a CD? Got Ubuntu? Try Rubyripper. Inspired by Exact Audio Copy (Which will no longer run for me under Ubuntu 10.04 using WINE), Rubyripper has the same priority of accuracy over everything else as the former, and after testing it on my quite-scratched Metallica CD, I can confirm it does at least as good a job as EAC, although the interface could certainly be prettier (again, just like EAC).

For ease of install, try grabbing the "GetDeb" .deb file apropriate for your version of Ubuntu here: http://linuxappfinder.com/package/rubyripper. Please also make sure you have the audio codecs you want to transcode to installed as well (You can install LAME for MP3 encoding through the Ubuntu Software Center).

Timidity and DOSBox in Ubuntu 10.04

For those of you who care about this sort of thing, here's how I finally got decent General MIDI playback for DOSBox in Ubuntu 10.04:

(These instructions are somewhat adapted from a post by Malor on the official Ubuntu forums)

1. Go into Synaptic Package Manager and install these packages:

• dosbox
• timidity
• fluid-soundfont-gm
• fluid-soundfont-gs

2. Open a command prompt, and type:

sudo gedit /etc/timidity/timidity.cfg

• The last line in the file says:

  source /etc/timidity/freepats.cfg

  Put a # mark at the beginning of that line to comment it out. We're going to use the soundfonts we installed from the previous step.
  
  
• On the next line, type:

  soundfont /usr/share/sounds/sf2/FluidR3_GM.sf2

• Save and exit

3. If you don't already have a dosbox.conf you want to use, type:

dosbox

and hit enter. Dosbox will pop up. At its command prompt, type:

config -writeconf dosbox.conf

This will generate one for you. Then type:

exit

to quit dosbox.

4. Type

gedit dosbox.conf

• Click the Find button, type:

  mpu401=

  and click Find. You should see a line highlighted.
• Close the search popup.
• The three lines starting with mpu401= should look like this:

  mpu401=intelligent

  device=alsa

  config=128:0

• Save and exit

5. Restart Ubuntu

6. Open a command prompt, type:

timidity -iA -B2,8 -Os -EFreverb=0 2>&1 &

7. Start dosbox

8. Enjoy General MIDI. :)

9. Close terminal window when finished.

Note: You may have to occasionally change

config=128:0

To

config=129:0

in your dosbox.conf. Watch the output when you start timidity and it'll show which to use.

Also, if you get permission errors while installing or running timidity, you may have to add "timidity" to the "audio", "pulse", and "pulse-access" groups in Users and Groups (In the System menu).

Ubuntu 10.04 LTS

I finally broke down and installed it, and here's some initial thoughts:

The Bad:

-The "Default Keyring" thing is still around, and it must die. Now.

- There should be an option to install "forbidden" codec support and encrypted DVD playback during installation. Given the amount of people who need to play MP3s alone, this should be a no brainer. I'm sick of having to explain to potential Ubuntu users why they can't do something every other operating does out of the box without jumping through hoops.

- No volume control on the top bar without the mail/messaging control? Really?

- Adding the Medibuntu repository is still a necessary evil if you want to use Handbrake or FFWin to convert Flash video to another format... like for your iPhone or PSP.

The Good:

- Snappy boot performance, slightly better than before.

- I thought I'd hate the new cosmetic changes, but the left-side window controls are actually kind of handy, and the new default theme is actually pretty cool looking (salmon accents aside).

- Overall better performance.

- New Thunderbird and Firefox. The former especially is a major upgrade.

- Ubuntu Software Center. It's almost to the point of being the easiest package manager ever made.

It feels more like a non-Long Term Service release, but still worth upgrading IMHO. My issues may not be the same as yours, so take my comments accordingly.

A note on burning DVDs in Ubuntu

If you have issues with DVDs burning far more slowly than the burner is rated for (and you know the blank DVD+/-Rs you're using are also rated for the same speed), here are some suggestions:

• Make sure that your PC can keep up with the data rate. Disable hard-disk intensive programs while burning (usually the burn would fail if this is the case, but just to be on the safe side...).
• Install K3B. It's the best overall disc burner in Ubuntu anyways.
• If you're still getting low burn speeds, go into the settings menu, K3b settings, and the advanced tab. There, check "Force unsafe operations". It should now burn substantially faster.

Please note I'm currently running Ubuntu 9.10. On other versions, your mileage may vary. Also, please don't hold me responsible for a hardware failure if something happens. When in doubt, double-check your settings and documentation before trying anything.

Film vs. "Digital" - the myths of reliability

There are a lot of myths in the film industry, and some of them are harmful to newcomers trying to enter and learn the buisness. One is that getting a new camera (film or digital) will automatically make you a film director. Another is that going to film school and getting a degree means you are now by default a filmmaker. The answer to both: although a new camera and a film degree can be helpful, you become a filmmaker by making movies, and the stories you tell are vastly more important than the technology.

However, there still continues to be a debate over which tools are superior to use, and the myths that crop up as a result are becoming perhaps the largest point of confusion for people entering the film business.

On the one side, you have people who believe that image acquisition using "digital" (video, DSLRs, or special-purpose cameras like the Red One) is inherently superior because the image is captured and processed right there on the set, then backed up in safe, secure hard drives or tape backups.

On the other side, you have film lovers, who argue about film's superior latitude and resolution, and the virtue of having a real, physical copy of the image you can always go back to.

Here's a dirty little secret: neither is really all that safe. There are films less than ten years old that have to be restored because the prints have degraded so badly. Digital acquisition is even worse. In an interview with someone at a major studio who handles data archiving, they basically admit that they have to be constantly backing up old data onto new devices to avoid drive failures.

I'm working on a film called Doggie Boogie right now as a data wrangler. The film is shot using the much-hyped Red One, which has an entirely data-centric workflow. We have some of the most reliable external hard drives available on set to back up to. We back up to two drives, mirroring each other, and each drive is actually two disks set up in a mirrored RAID. We are safe as houses. Except last week one of the drives died. No warning, nothing. We still have the one drive as a backup, but it just goes to show: digital is only as reliable as the medium you store it on, and none of the mediums out there are safe enough to trust on their own.

Why not just use film, then? Because at different points in the process, film is even more unreliable. You can't see imperfections in the image until the film is developed (usually) the next day, but even more crucially, even small amounts of light leaks in the camera or the film canister can cause your image to get washed out or completely erased in a white blur. Then there's the mechanical issues with film cameras, like frame stuttering or scratches from minute particles of dirt. All of these issues can be minimized by hiring a crew that knows what they're doing, but again: Film is not a quick fix for reliability. It needs to be handled with lots of care.

So, to reiterate: there is no "safer" medium. Learn the strengths and weaknesses of each, and plan your shoot accordingly.

Update - Here's a video from the Library of Congress about the "home" side of this subject: http://www.digitalpreservation.gov/videos/digipres/index.html

Ubuntu 9.10 and DOSBox

Just a quick update: on Ubuntu 9.10, the fabulous DOSBox 0.73 can be installed from the Ubuntu Software Center, which means it's no longer necessary to go through compiling source code for the latest (as of this post) version of DOSBox! You may want to change (in your dosbox.conf or dosbox-0.73.conf) the mixer settings to "rate=44100" and the "prebuffer=50" to avoid stuttering sound, but try and see what happens. I haven't tried setting up General MIDI to work with it yet, but I'll post here if I do.

More on buying decisions

In addition to looking for a new camera, I'm also looking into a computer upgrade to handle the increased demands of editing full HD (or greater) footage.

Here's a great site for helping to decide whether it's time to get a new Mac system:

http://guides.macrumors.com/Buyer%27s_Guide_%28time-ordered%29

The conclusion I've come to on both camera and computer is to wait a bit. Apparently, the Mac Pro is due for a major update in March, and it will be probably about that long before test footage from the fixed-lens RED Scarlet comes out.

DSLRs vs Prosumer Video Cameras vs RED

So, I'm in the beginning stages of a documentary, and the camera I have for shooting HD (A Sony HDR-HC7 HDV-based handycam) is looking pretty low quality compared to what's out there right now.

I have essentially three options within the budget we're looking at:

1) Prosumer video camera - The Panasonic HMC-150 is the closest form factor to my old DVX100, is around $3500, and uses SDHC cards, which are cheap. However, it also has a fixed lens, so shooting something from a distance is going to be a pain in the ass. There is also some chromatic aberration in the lens that becomes especially visible around dark objects in front of a very bright light source (like, say, tree trunks against the sky). The compression also has trouble with lots of random movement, such as turbulent streams up close.

I could go with a better HDV camera, which would give me the advantage of using cheap tapes instead of expensive memory cards, not needing a laptop or field recorder, etc. The image quality difference is fairly large at this point, though, and many TV stations today will simply not accept shows shot using HDV.

There are better prosumer cameras, such as the Sony EX3, but they are about twice as expensive, use much more expensive media, and still have highly compressed video (unless you buy/rent an expensive external recording device like the Nanoflash).

2) DSLR - The Canon 5D Mark II and 7D have created a whole new category in indie production by integrating HD video into high quality still cameras. The advantages they bring are lenses that are sharper, have much shallower depth of field (Which looks much more like film), and a very reasonable price (Around $1800 for the 7D, $3500 for the 5D Mark II, both with a zoom lens). Oh, and with lens adapters, you can use a wide variety of lenses (including cinema lenses). On paper and in some of the video clips shown around the internet, this would seem ideal. However, there are a few issues:

• The cameras have crappy on-board audio recording. You can plug in an external mic, but in order to get decent audio, you need an external mic preamp box which requires additional power; plus an unofficial firmware hack to disable the cameras automatic level adjustments. A much simpler (although possibly more expensive) way of dealing with this is to record audio with a portable audio recorder and use a clapper slate (or clapping your hands in front of the lens) to sync up with the image and/or crappy on-camera audio in post-production. Either way, a hassle.
• The video recorded is not the same size as the image sensor, and the process of squishing the image down to video size in the camera creates some problems, namely aliasing, which can get downright ugly. Aliasing is most visible with distinct lines next to each other, say in a truck's front grill or a striped t-shirt.
• The video is recorded in a highly compressed format that will lose a lot of detail in really dark areas, and can create some fairly visible compression artifacts if you're not careful.
• Due to either format limitations or EU regulations regarding a tax on video cameras, DSLRs do not record video clips larger than 4GB or longer than 10-11 mins, whichever comes first. This makes using a DSLR a lot like using a film camera, but sucks for documentaries where you could easily end up stopping the camera in the middle of a perfect moment that you cannot recreate.
• Speaking of documentaries, DSLRs are very difficult to get a stable image from while being hand-held, which means you need to get additional accessories (a shoulder mount, basically) in order to do proper handheld work... which costs an extra $800 at least.
• Each time a frame of video is recorded, it's not drawn all at once. Each line of the frame is recorded top to bottom in sequence. For relatively still images, this is fine, but when something moves rapidly through the frame (say, a car), it looks slanted. This is the "rolling shutter issue" or "jello", and if you ask me, it's fugly.
• The 5D Mark II, while having a larger sensor area and better low-light performance than the 7D, shoots in a non-standard framerate. There's supposedly a firmware update that fixes this - in 4-6 months.

Basically, these cameras were originally designed for photojournalists who wanted to also shoot some video, but in the same style that they shoot pictures - short, silent (or with unmixed audio) and not edited to a significant degree. If you can control what you're shooting, you can get gorgeous images. If not, you get image problems. Oh, did I mention the rolling shutter issue? Anyway, you get the idea...

3) RED Scarlet - Speaking of 4-6 month out, this camera seems on paper to be the answer to my concerns. Based around the same tech as the über-successful RED One, the Scarlet uses a vastly superior codec with much higher resolution, significantly more dynamic range, an next to no compression artifacts. The image captured is not downsampled like on the DSLR (at least, not by default), so aliasing is very little to none, and (if it's like the RED One) the rolling shutter issue is much less noticeable than on a DSLR. So what's the problem?

• You can get the Scarlet for $4750, but that's a fixed-lens design, so you can't swap the lens out like on a DSLR. Even if the built-in zoom lens is great, the sort of wildlife photography I'll likely be doing on this documentary will require a very long telephoto lens.
• The interchangeable-lens model is $2750, plus $950 per prime lens (or you can use DSLR lenses by using an adapter for $500 [either Canon or Nikon]), but you also have to buy the audio interface, batteries, etc. Oh, and if you want a proper viewfinder instead of a back-of-camera LCD display? $3200. Side handle for proper handheld shooting? $950.
• There has been exactly one 15-second clip of Scarlet footage shown so far (a close-up of an iguana on a soundstage), so it's impossible to judge what the final image quality will be like...yet.
• RED cameras are often back-ordered into the stone age, and pre-orders have yet to even start for the Scarlet, so good luck getting one this year.
• The image captured by the RED cameras is totally unprocessed (in terms of color and brightness adjustments, not compression), which gives you more options, but also makes it more work to deal with in post.
• RED cameras are not "shoot anything and it will look beautiful" cameras like mid-to-upper-end video cameras. For better or for worse, the RED needs a significant understanding of how images will look in post in order to shoot good-looking footage. When you have control over what you're shooting, this is fine. When you don't, you run a greater risk than other cameras of getting bland footage that no amount of post-processing can fix.

So, basically each of these options have significant drawbacks, and at least two of them are likely to be replaced with vastly superior models within a year. I will continue to do research, of course, but in the meantime it's a really tough decision.