Saturday, as I’m sure you all know, is/was the 24 Hours of LeMans (the real one, in LeMans, France), which is still running and will end in about 5 more hours. Audi has lost two cars to spectacularly devastating crashes, with both drivers walking away. A Porsche driver was not so lucky and was taken to the ambulance on a gurney, no word yet. The Audi – Peugeot rivalry is in full swing, with the lone remaining Audi driver in front of the three Peugeot drivers. Hot stuff, but I have to sleep. Just the same, a 24 hour race has spent something like 4 hours under the safety car speed limits (they had to rebuild a wall from one of the crashes), and that’s a LOT of time. Formula 1 is in Canada this weekend, so it’ll be weird to think that we could actually watch it live (until Austin or South America again). Practice 2 was good, even the accidents aren’t hitting walls, and the course looks fast. We have qualifying to watch still, which I tend to like better than the actual races. I’m odd like that.

I spent the day washing our car and van. My car still draws respectful comments from neighbors. However, and I’m still not sure how, it’s been covered in ants. I have a couple theories. One is that the tree at work I’m parking under is leaking ants, and the other is that I did see an ant on the ceiling over my car in the garage. Maybe they are falling off the ceiling onto the car and never quite finding their way off again. Just the same, I’m tired of them, and so is Child Two. But after the washing, perhaps they’ll go.

Evening had us go out to see Pirates of the Caribbean IV, which was a laugh; a definite apology for the last two. Babysitter was Child One’s school teacher, who the dog adopted, then forgot, then adopted, then forgot… what a stupid dog. We had fun, came home, checked on the race, and here we are around 1 A.M.

What a great day!

(Monday after race resumes)

Turns out that all the drivers in the multiple heavy crashes seen during the race fared pretty well (have to check on the news yet this morning), but what an incredible race. The P1 class drivers were mere seconds apart after 24 hours – HOURS – of racing. What a race.

WARNING

If you haven’t seen the film – which I wholly suggest you do – don’t read on – it’ll ruin it for you.

Frankie has taken up ownership of the cafe indicated in the movie. He spends his days cooking behind the bar to keep busy. He discovers that his misery for lost loved ones in the past is only satiated through work, and even that is not enough. His detachment from his competitive boxing days and all the people he knew is eating him slowly. He makes friends with the locals and lives in complete isolation otherwise. He has learned of the warrant for his arrest related to the death of an invalid, and has changed his last name.

A couple times a week, an overweight kid comes in with his grandpa. They order a whole pie each and sit quietly for the entire afternoon, laughing and chatting, always finishing both pies. Time passes, and one week the kid comes, but the grandpa doesn’t. In a deep depression, the kid orders pie after pie, eating himself into a miserable state. Frankie feels compassion for anyone that depressed that has lost someone and they become friendly. Week after week, their friendship grows. Frankie then decides, “Man this kid can eat. I wonder if he could do pie eating contests.” He provides the kid with all the pies he wants and a few months later, offers to take him to his first contest at the county fair.

The kid loses the contest to The Pie Eating Champion Of The World (why he’s in the back woods, nobody knows), but is exhilarated. They work on his technique and come back again for a local charity event. The kid does fabulously, but pukes it up just before the end of the contest. The kid finds that the pies have been tainted, and an investigation is called for.

Around now there are whisperings in town that a stranger has arrived and is asking questions. Frankie starts to get scared about his warrant and suggests they take the challenge in Wyoming eating watermelons in the summer. They pack up and drive north. They arrive and take up training in a rented apartment.

Frankie’s ghosts are getting the better of him now that his hands are idle, and he has started sleepwalking. He wakes up in various places, in barns, on highways, in the woods. He is visited by ghosts of all his previous proteges. Wherever he finds himself, he always wakes up with his knuckles bleeding. He hides this from the kid by insisting that he wakes up early and goes out for walks. The bleeding hands are covered by fingerless gloves. (It is Wyoming after all.) Between training sessions, he is visited even in waking by his previous students, who pass judgement on him.

The kid wins the watermelon contest handily – and his first win is accompanied by their first glimmer of hope. They plan to travel to Michigan to enter a hot dog contest next. The kid is now troubled by indigestion issues. They visit a doctor, who tells them that the tainted pies have left him with internal scarring that may limit his ability to keep up the eating contests. The kid presses on regardless, counting on the resilience of youth to make up for the problem. As they are leaving town, The Stranger’s pickup appears, and they pass in the street already on their way to a different location. On the road, late one night, Frankie’s ghostly apparitions appear in a line on the highway and he swerves wildly to avoid them. They chalk it up to driving drowsy and pull off at the next town to rest. Frankie is still disturbed.

In Michigan, they take up training for the next contest. October is now fast approaching, and savings are running low. They need to win the next contest to move on to the next town and stay ahead of The Stranger. The kid is starting to experience serious discomfort most of the time from the stomach scarring. Practices are shortened, but they persist. The contest is delayed two weeks. Finances getting tight, Frankie decides to tough it out and hope for a win. There should be just enough. Nothing is heard of The Stranger, but The Stranger has heard tell of the next contest as well.

Halloween has arrived in rural Michigan, and the contest is hosted on the fairgrounds with the local festivities. The Stranger is already at the contest. Frankie and the kid arrive without observing The Stranger who is milling around looking for them, a Stetson on his head as disguise. They go to the contestant’s stalls in the barns provided for them, and set up. The kid hasn’t eaten since two days ago and is famished. The smells of the food are becoming hard to ignore. Frankie is being tough on him to get his game up, and the kid is looking pale and beaten down.”Get on with it coward! They’re tubes of meat!”

The kid snaps. He grabs Frankie and bends his neck over backward, biting. He roars with a demonic sound. Frankie falls to the floor, dead. The kid hears screaming and runs out to the fairgrounds, only to see Frankie’s previous students looking pale and dripping blood from their mouths. A scrawny kid with curly hair runs past, yelling, “CARDIO!” They all start running after the kid. Our kid grabs the nearest stationary person and begins to devour them messily. The Stranger is back in his pickup truck. The Stranger smashes the accelerator and pins the kid between the front of the truck and the ice cream shop.

“Zombies! I’ve got to get to Tallahassee,” says the man, and drives away in his pickup, Twinkie in hand.

This one is about a cutting-out expedition by our heroes. The characters are distinct and memorable, the setting is good, and it has that Crichton-ish feel for the environment. Yeah, I know, stupid sounding, but oh well. Just don’t let facts get in the way of a good naval romping good time and it is quite enjoyable.

When I got to the halfway point, the story that I figured was the main plot was done. Then I realized there is a lot more book coming. So, puzzled I marched onward, and it took on a subtle twist I should have expected. And now, another quarter book in, things are still not resolved.

Can’t wait for the drive home!

All builds have a server, 4+ disk RAID5, and offer flexible storage expansion to keep them around for about 2 to 4 years. Each has a “Priced from scratch” number. Remember that this includes 4x $100 drives (currently the 1.5 to 2TB range) that you might already own, or be willing to substitute your old 200-300 gig drives for instead, or whatever. I think you get the idea.

Optimize for money, with Mac guest:

Priced from scratch: $1900ish

• Mac mini (any flavor), add RAM to 4. ($600-1000) ($300-700 on craigslist)
• VMware Fusion ($70)
• Mac OS Server (presumably the license you got from buying the Mini Server, right?) ($500 or included with mac)
• Drobo via local connection for the host, guests live on the Drobo ($400)
• disks ($400)
• Apple USB dongle (technically support is only for the MBAir, but it works. Trust me.) if you want to use a VM router. ($30)

Pro:

• Low electricity.
• Low heat.
• Few surprises.
• Drobo adds the ability to just keep expanding your storage for either more VMs or more space on them without having to worry about iSCSI components losing power between you and the array.
• Easy UI and remote control with Back to my Mac and/or VNC.
• Low TCO – macs depreciate slowly. If it’s time to upgrade to something more powerful later, you can sell it.

Con:

• Few server-quality expansion options.
• No internal high-speed RAID.
• The Drobo will only hit 40ish meg a second, but honestly for home, you’ll probably not need that much all the time.
• VMs limited to 100mbit connections.

Optimize for money, without Mac guest:

New

Priced from scratch: $1550ish

• DIY Intel Core2 quad based on LGA775, 8 gig of ram (priceout on newegg: $550 + hard drives)
• ESXi $0
• Adaptec 2405 RAID controller ($200ish) and SAS case/enclosure ($400)(or other supported card/case/enclosure setup)
• disks ($400)
• Optional substitution: iSCSI remote storage (like FreeNAS or Drobo Pro)(-1000, +cost of chosen iSCSI target)

Pro:

• ESXi is really the right tool for the job.
• Can sometimes get away with a very cheap hardware build.
• Some VMs can go up to gigabit connections if you absolutely must have it.
• Optional iSCSI/fiberchannel storage growth paths open to you.
• Faster and more powerful than the Mac Mini option, but you’ll have some surprises building the box. There is always a hidden surprise.

Con:

• ESXi doesn’t support lots of cheap hardware, so you’ll have to be careful about the hardware you pick.
• You still need a separate windows box/VM around to manipulate things, or get really comfortable with the VMware CLI.
• Higher overall TCO? It’s usually harder selling a used medium-low power server in two to four years.
• It will use more electricity for those extra cores and extra RAM, but if you do video work with this box, it may pay off to have them.
• Electricity/heat in the middle range. Initial expense may convince you that the maintenance is worth it.
• Doesn’t support the mac guest, but there are alternatives to OS X Server’s two most attractive features for the home: iCal server and TimeMachine on flexible storage. Dang, that’s a pro. Moving on.

Used

I will point out that the previous generation of HP/Dell ESX supported servers are regularly on craigslist for cheap – usually $400 and under. Rackmounted and everything, but usually running “real” SCSI disks, very loud, very hot, very small. They still will need $1000 in storage, the same as the rest, so buyer beware, but if you’re just going for the VM server and can skip the storage, you can get in on the cheap.

Optimize for speed, with Mac guest:

Priced from scratch: $5000ish + 2 kidneys

• Mac Pro 8-cores Nehalem, 16GB RAM ($kidney)
• Mac OS X Supported RAID controller plus enclosure and disks. ($other kidney + $1000)
• Mac OS X Server (bundle with Mac?) ($500)
• VMware Fusion ($70)
• disks ($400)
• Alternate to RAID controller and enclosure: Drobo plus disks ($300-3000)

Pro:

• Fast.
• Very fast.
• Lots of memory for lots of VMs.
• Lots of very fast storage.
• Potential no-downtime storage growth.

Cons:

• Dude, you’re out of kidneys.
• Limited hardware choices for SAS RAID.
• Limited iSCSI options.
• Speed gain over Core-2 chips might not enough to warrant price differential unless you do a lot more math than most of us do.
• TCO for depreciation can be large when initial investment is also large.
• Hot. Cooling is expensive.

Optimize for speed, without Mac guest:

Stop. At this point we’re priced out of reasonable home options. A similar spec to the Mac Pro can be built on Newegg for about $1200 with a RAID card. But at that point, you might as well get one of the Asus barebones servers that are fully VMware certified and attach to external RAID storage. When I get as far as this, I know we’ve exited “home” and reached “business” expense levels, at which point the economics of consolidation have already broken down. How many systems do you have to sell and turn off – in your house - to cover the depreciation of a Mac Pro, over the same for a Mini?

Hope that was short enough. Comment away!

Know thyself

Part of the problem I realized is that I probably don’t need an 8-core Xeon server on 24/7/365 to power my firewall and backup streaming (sorry Apple!). As much as I’d love to have one, I just can’t justify the amount of calories it would produce and feed into my [garage|basement] and the number of kilowatt-hours it would consume. Even with clever power management schemes as turning it off during the 1 A.M. to 1 P.M. periods, spinning down hard drives, etc., I’m just not using that much processor that much time. So I started dividing up jobs into two major categories: always-on services vs. on-demand services. Then I came up with this list, which I’m sure will be different than other people’s:

Always On:

• PPPoE dialer
• Firewall/router/gateway
• wifi-n access point
• DHCP
• dyndns.com updater daemon
• Remote access; VPN, Back to my Mac, whatever
• iCal server; reachable from the outside
• vCenter, maybe; reachable from the outside
• Backup server (RAID, growable)
• PS3 media server (or by schedule, details later) (RAID, growable)
• Web proxy
• “Warm storage” file server – stuff that can go to cold storage if unused for a while
• Home control software (lights, security, HVAC, whatever)
• [unplanned] MythTV backend

On Demand:

• Quicken
• Audiobook downloads
• Video encoding
• Work-VPN VM (it’s just easier this way)
• iTunes to manage files in PS3 media server directories
• daily use items (raw hardware)
• Games (raw hardware)

Note that I have re-ordered this list to make grouping them easier than I had originally done on paper.

Group into hardware

At this point I started drawing circles around things that were obviously easy to tie together, such as the first 5 items. These are currently done by my $80 linksys router with a power draw of something like 10w and almost no maintenance or downtime. Hard to argue with that. So I left those together. Then I looked at how I want to access my home resources from work (or the road) and decided that I still like Back to my Mac as an easy way to get in and see if I really do have that video on my home iTunes server. So that’s not hard. It’s functional so long as I feel like coughing up for the $100/yr service. Which I may be able to stop doing after this project is over. But I digress. The next set of things on the list is the set of “services provided from home”. A free account with dyndns.com or similar services allow me to reach home, so long as the bandwidth and processor power required to provide these to basically myself any my immediate family is small, I don’t need to shell out to a remote company to provide it. So I can host my own iCal server. It’s even open source if I don’t care to come up with an OS X Server license.

Now we get into the range of things that I really had a hard time deciding I could live with leaving in their current state. The backup server and the media server. Currently I have a pair of beat up ReadyNAS sparc-based units. One I bought new after a particularly bad hard drive crash, and the other I bought in a ‘company is folding, see if it works’ sale. I love these units for three main reasons: 1. they don’t need a lot of playing with. 2. they grow the shared volumes without me having to do much work to make it happen. 3. they don’t use a whole lot of power. But: these much older units that I have are not really good at sharing time. As single-purpose appliances they shine, especially because you can let them resize the volume by just inserting new drives. And they don’t ask for lots of maintenance upgrades and security upgrades, etc. They really are great little units if you can find them under $600. So with the luxury of having two already on hand, I decided that with jumbo frames turned on at both ends and a suitable Netgear switch in the middle, they really do perform these two jobs quite suitably. So my cheapest and most sane option is to repurpose them as dedicated appliances instead of general-purpose storage systems. And I also realized this was my migration path – essentially turn them into hardware incarnations of the VMs that they will be replaced by in the eventual ideal solution. This made me happy. So now I can dedicate one to be the growable backup unit and the other to be the growable media server unit. Which they already do.

Web proxy is honestly an optional extra I’d love to implement if I can arrange it. It has its own pitfalls, as the fabled “one more thing to go wrong” as well as being something else to maintain – something I’m usually quite set against. It’s there, we’ll see if it can be done.

“Warm storage” is the place I put things like setup files for downloaded products, project files for Audiobook Builder, iMovie project files I’m done working on but not ready to shelve, that kind of thing. Anything that is in warm storage for about 6 months will eventually get moved to offline cold storage and kept powered off, in a box, either at work or at home. As this is something that I want access to and won’t use often, and really is just about a terabyte of things that I’m not quite done with yet, I could group this with the hardware that does the backups. Speed isn’t an issue, just the access and the space, as these are quite regularly bigger than a DVD. Again, as this could easily be defined as something like a FreeNAS VM or a slice of a file server VM, I don’t mind this as a standalone role that can be virtualized in the final ideal solution.

Now we’re down to the on-demand services. Re-evaluating what these roles are helped me see that most of this can be handled in two places before the “final ideal solution” comes around. Currently the Quicken VM lives on my laptop. The Audiobook downloader lives on my low power mini server at home. Video encoding is done on the native OS of the mini server. The work VPN VM is on my laptop, as that’s where I use it most often. The iTunes library is on the native OS of the mini. I don’t mind this arrangement, but it helped me realize again the eventual arrangement in virtual machines I’d like to use, breaking them into specific roles let me see the migration path as hardware permits.

Now that I have the new role-based model to help me see what is where now and where I want it to be later, I came up with this diagram:

Big line flow charty thing

There are still a few stray points to clear up. I could entirely abandon the Linksys if I can find a way to control a wifi card like an access point. I don’t have to give up my growable storage arrays yet, but I could if I can find a 3ware card and enough drive bays (I already have, next post, I swear).

Note that this is just the role flowchart. I have additional charts that I’m working on to deal with network paths, storage, and so on. It’s not a trivial study as I originally thought. But there you have the original thoughts on how to organize it into hardware groups in a more realistic home environment.

So I did the next most logical thing. I went to the public library’s web site. Now, of course, this is where many of you will get angry with me, because I’m part of the San Jose Public Library system (www.sjlibrary.org) and you’re probably not. I can only say that this is documenting how I got around this problem, and you’ll have to adjust your experiences accordingly. Moving on.

So I still did the next most logical thing. I went to the public library’s web site. And sure enough: there on the left is a search catalog link. Which is exactly what I was after. I typed in the title and clicked search.  The list came back with over 300 entries. Most were paper, most by other authors, but the great thing was: they had audiobooks in there too. And I don’t just mean the CDs you can get on the shelves, which was all I was expecting – but electronic audiobooks. So I clicked, and fell down the rabbit hole.

For Father’s Day this year I got my audiobook subscription to Audible renewed for one year, so I have book credits on hand. However, I’m surprised by two changes; first – I love the new Audible site design with the NYT Bestseller lists and favorite-ever lists, and second – now that I know how to get so many books from other sources for ‘free’, I’m surprised how little is left on my Audible list that can’t be found elsewhere.  There are some exceptions, like there are a hand full of Orson Scott Card books that I just don’t find elsewhere, and some Michael Crichton (I limit myself to one Crichton book a year or I stop enjoying them). But even many of the newest books appear in the public library in short order.

For the impatient, I’ll just shortcut to the:

Workflow Cheat Sheet

• Keep a master wish list at Audible or Amazon. They have the big catalogs of books, with a variety of reviews available and star ratings that may keep you out of a book that sounds like one thing but turns out to be another. And sometimes they will have books that aren’t available from public locations. I hate forgetting: what-was-that-book-that-sounded-so-good-but-then-I-forgot-to-write-it-down-and-then -I-got-busy-and-now-I’ll-never-know-if-it-was-any-good-or-not. Get in the habit of using the voice recorder on your phone if you don’t feel like typing. I use the iPhone camera in book stores and the public library to snap the title and author of anything that looks good. They don’t have to like me.
• Periodically go to your wish list at either site above and start searching your public library catalog. I’m finding that about 2/3 of the books I search for can be had at one of the free sources, albeit with a wait. Even if there is a wait, I have 12 books queued up, I can be patient.
• Go to the public library site and search, sort by media kind and scroll all the way down – ebooks and audiobooks are at the end mostly. This catalogs both what is in the ebooks site and what is in the physical holdings. First pick the electronic version if you can get it. No CDs to import, quality is sufficient for translation into medium or low quality Audiobook Builder format. And no driving.

The San Jose ebook web site has its own flow. The flow here is [ optional - wishlist]->waiting list->bookshelf for 1,2,3 weeks. They manage it in the same way you would manage the paper copies – only so many copies are available for checkout, and you have to take turns with the others. Their wish list has each books current “Get it now” or “You’ll have to wait” status.

Spend some time one day just throwing piles of stuff into your wishlist from your master Audible/Amazon list, or from browsing. If you keep your checked-out list (a.k.a. bookshelf) about half full, you’ll still have room if a couple of your waiting list books come to you.  Otherwise you’ll miss your reservation window. Go ahead and keep your waiting list books more or less full and don’t be afraid to put yourself back in a queue if you miss it because of a full bookshelf. Remember about once a week to go back and check the wish list for available books and check out enough to keep it around half full and you’ll be happy.

Second choice is to get the physical CD set. Sometimes there is also an mp3 CD set. I was quite impressed with the quality of this. If the library is between work and home, I’ll drive it myself. If not, I’ll request a delivery to the one closest to home.

• Then turn to LibriVox. A surprising amount is available there, so don’t discount it.  Just don’t cringe at the narrators – quality varies widely. But also because they aren’t hindered by things like production cost, you can find a lot more content there than in most commercial collections, like their Sherlock Holmes collections are far more complete. The problem is that the LibriVox site is hard to navigate; but again, the price is right. If I liked my voice, I might consider doing some narrating myself. Dracula, Sherlock Holmes, seriously good stuff available.
• If all else fails, money talks. Audible or Amazon can help you find nearly anything in audiobook form.

Okay, now the details.

Welcome to DRM – we hates it forever

First: At home I’m a Mac user by default. I have a Windows box and 3 linux boxes, but my preference for the UI is the mac. iTunes is run on my mac mini – which is its main job really, followed by the endless video encoding I do. So when I showed up at the San Jose Library’s ebooks section (run by Overdrive DRM software, piggybacking Windows DRM), there were a few things of note:
1. They do have mp3 audiobooks with “do it yourself” DRM – please delete when you are done with a reminder from the download client.
2. They have far more DRMd WMA books.
3. But hey, the price is right.
4. Books can be checked out for 1, 2, or 3 weeks at a time, with a max limit of 10 at a time.
5. You can also put yourself on the waiting list for individual books. If you have a book slot open on the day your number comes up, you can check it out.  If not, just put yourself back in the queue tomorrow. Be sure to check your email.
6. They recently added “iPod friendly WMA audiobooks”, which apparently means that if you run iTunes from Windows, you can sync them to Apple devices. I haven’t tried them.

So, being an Mac iTunes-iPod-iPhone person, this limited me to just a subset of those books, but you can search just within that range and still get quite a bit.

Bundling and Organizing

So, just the same, I got busy with the checking out some MP3 audiobooks and some CD audiobooks, and remembered one of my pet peeves about the whole business. The “Audiobook” handling in iTunes is only scalable to the point at which you copy one or maybe two to the device you use. The interface on the iPhone and iPod for their Audiobooks navigation is annoying – titles only. So I’ve taken to storing them as Music, so I can navigate like so:

1. Music
2. Genres
3. Audibooks or Radio for radio shows
4. Author as Artist
5. Series name or Title as Album
6. Title Part N

I also keep a set of playlists based on the intended listener:

• Book Queue New

- for my new books. I keep them here to get around to them in some sort of order, whatever I feel like. I find that if a book is too dark or too much work, I only listen to it on the way to work and prefer something lighter for the trip home.

• Book Queue Reread

- for the lighter books or just anything I feel like I want to come back to soon.

• Book Queue Kids

- The public library has lots of audibooks geared toward the short set. I’ve picked up some for them and put them in here so we can get to them on the school run. They loved the Star Wars Radio Shows I got a couple years ago and a Peter Pan radio play.

So if Apple ever offers Audiobooks the same navigation, I’ll switch them back, but it’s a simple change.

Each publisher has different policies when dealing with the track length, and the books you get from the public library could be chapter divisions or they could be disc divisions, or something else entirely. Enter Audiobook Builder by Splasm software. It does exactly what I want it to do – combine lots of different little chapter or whatever files into a single set of m4b audiobook format files for iTunes. Bingo! Well worth the $10 I paid for it three years ago.

So now that I have my books and can listen to them and they aren’t 6 bizillion little files, I pop them into iTunes and the silence is gone. I have recovered – and the only thing I had to buy (but already owned, so no expense for me) was Audiobook Builder. $10 for a serious pile of audiobooks.

So I guess that’s the conclusion as well. I have a lot of credits for the things I can’t get from the Library, and the Library does a spectacularly efficient job of giving me access to books I can hear.Happy listening!

Back Story

We finally stepped part way into the HD era, by attaching a PS3 to our SD television. Sacrilege, I know, but it’s what I have for now, and the economy isn’t getting any better this week. But, as we only have two games for it (Little Big Planet and Mater-National Racing), we have found that it has another wonderful use. The Videos menu.

(Keep in mind that I have 3 little kids in the house, and while most of their viewing is in the broadcast-DVR, I still have hopes that they will take an interest in longer attention span viewing, or at least the Muppet Show season sets and Looney Tunes sets I have waiting. But handing them DVDs is a great way to have things get destroyed. Enter digital viewing on the TV.)

So, having an Infrant ReadyNAS x600 in the garage, I bought a copy of the far more predictable TwonkyMedia server that handles all the gross stuff related to sharing your video folder with the PS3. The built-in server just wasn’t maintaining itself well, no matter how I tried. I’m sure someone can make it go, but I needed something a bit more turn-key. But the bigger problem still remained: I have something along the line of 300 DVDs between movies, television series, and home-made things. Ripping is the tetchy topic, and if you have questions we can talk via email, but the fact is that ripping even the most difficult DVDs is now possible. But now you have at least one hard drive full of DVD folders or ISO images of DVDs, and you have to turn this into something a bit more manageable.

This is the PS3 specific portion – if you are comfortable with how your player plays back and what encoder settings you can get from Handbrake, then feel free to skip forward. As clients to my encoded DVDs, the first priority was everything was to be HD resolution for the PS3 and/or my TViX 4100. There was also a smaller subset that I wanted to play back on my iPhone. After much experimenting with  Handbrake, I settled on a set of 2 encoding strings that made me happy. One was the default PS3 profile, plus chapter markers (for computer playback), plus 2-pass encoding. The other was the same, but with deinterlacing turned on, for those titles like the Muppet show that were interlaced on the discs. (If you don’t deinterlace those, then the frames come out in the wrong order when replaying on the PS3, and make things jump around forward-backward-forward and drive you to turn it off.) Fortunately, the TViX plays these files without modification, so I was set for both devices. So those two strings were written and set aside. Then I came up with a parallel pair of strings for the iPhone setting.

So I had my 4 preferred encoding settings and a pile of 300ish ripped DVD images or folders to eat through. I knew there was a problem. At PS3-2pass resolution, each encode was about 4 hours. That’s a lot of hours, and queuing each in Handbrake’s GUI takes more time than I want to do. I already knew exactly what I wanted, but I didn’t want to do it by hand.

What I Did

So I did a little digging on Google and discovered that for some reasonably old version of Handbrake, someone wrote a bash script that would turn VIDEO_TS folders into mp4′s. As time had left this little gem behind, I did some minor repairs and used that for a couple encodes. But it had some limitations I didn’t like, and didn’t work at all for Windows machines, and so on. I even modified the original VIDEO_TS script to work with ISO images, and that saved me some more work. I could then do massive batch encodes using my two scripts. I took images to work and left the far faster work computers cooking m4v’s for me over weekends and nights. It was great for saving me weeks of computing at home. But I had to custom-build the scripts for Windows via shared folders on the mac, and it still took too long to manage two scripts across multiple machines. Enter the perl experience. At work I’ve been working with cross-platform scripting for a while now by using Perl. I opened up the guts of the VIDEO_TS script again, figured out how to do what they were doing, added the new abilities of Handbrake’s latest versions, and got to work giving the Windows machines the same logic the Mac had with the bash script.

The new script, stupidly named encode.pl, now handles pretty much anything that Handbrake can handle – or at least anything I encountered. First, you queue a directory. It simultaneously queues VIDEO_TS directories, ISO images, wmv/avi/mov/etc. files for encoding. Then it proceeds to do discovery via Handbrake about what it should encode. It does this by querying the title list for the duration and then applying a filter for length. For instance, my default setting is anything between 20 and 600 minutes, but that can be tuned with the –minimum or –maximum settings. Then it goes through them one by one, handing the whole thing off to Handbrake to be encoded. As each one finishes, it goes to the next one. No mess, no fuss. Until you run out of hard drive space that is. Can’t help you there.

So then I had to tune the way I organized my DVD images for feeding into this thing. My ReadyNAS had enough space at the time, so I put things into distinct folders. In the media share, I had a Videos folder. Inside I created the following:

/Volumes/media/Videos/todo/…

…/deinterlace/long

…/deinterlace/episodes

…/long

…/episodes

- and here’s why. The long directories were for things that I wanted only the titles that were over an hour and ten minutes. This let me get a single file for all of the Looney Tunes shows, strung together, instead of a pile of 7-9 minute shorts. There are enough per DVD to make it frustrating to have to return to the menu for each new show. Kids will just give up, and so will I. They already have a title with all of them together, so I just added the –minimum 70 setting when I do the long directories.The episodes directories were for things no longer than an hour. These are the opposite of the long things. So for cartoons, sitcoms, and other 22-44-minute episodes, I don’t want the whole DVD as one title, I want each show as an individual m4v file. That directory got the –maximum 50 setting. This pretty much handled everything except for a couple odd cases that I just encoded more or less by hand with the GUI – mabye 3 discs total.

For output directories, I created:

…/m4v

…/iPhone

Now, because I had central storage for both the input directory and the output directory, I also considered using multiple computers. I was able to use the 4 computers under my desk to all work on a central queue. I would go to each computer, mount the share, and put together the encode string for that machine, and set it running. Each computer when it starts an encode creates the file it’s starting on, and then starts the Handbrake line. I had to do this because with 2-pass encodes, the output file is not present until the second pass starts. Because of this, if you have computer A starting on title 1, then computer B might start on it too. If the file is present, then computer B says, “Oh, that one’s there, let’s start the next one.” So you get a cheap claim-before-starting system built in if encoding to the same destination. Then another thing came up. I could do encodes on my laptop while away from the main servers. So I added the ability to keep a log of what’s encoded and copy that to remote locations (*cough* work) and still not get double encodes. It’s not a perfect system, but it’s better than nothing.

The Interesting Part

So the result is a semi-distributed way of encoding a seriously big pile of movies into a standardized format using both Macs and PCs, possibly central storage, and getting a pile of files with at least standardized names.

I’ve actually considered trying to add a way to use STAF to auto-distribute encodes to available systems, but that takes a bit more time than I have at the moment. But that could allow you to have a central “encode controller” and some “encode slaves” that you hand out single encode jobs to. It’s within reach, I just need an hour here or there to try it.

Anyway, here’s how you use it:

perl encode.pl -i /Volumes/media/Videos/todo/long -o /Volumes/media/Videos/m4v --minimum 5 --maximum 70 -e "-Z PS3"

or the equivalent in Windows paths:

perl encode.pl -i M:\Videos\todo\long -o M:\Videos\m4v --minimum 5 --maximum 70 -e "-Z PS3"

-i for an input directory

-o for an output directory

-e for anything you want to hand directly to the encoder. “-Z PS3″ is the default PS3 string, but be careful with “-Z ‘iPhone & iPod Touch’” because the shells differ in quote handling. Try it on a single command line to get it right and then keep it around in a text file somewhere to be sure. Also, because of the funny quoting problems, I usually try to make the -e setting the last one I hand in.

–minimum is the minimum time in minutes that you want to approve a title for encoding

–maximum is the opposite. Note that minimum and maximum ignore seconds.

Note that almost everything is optional. You can open the perl code and change to your favorite defaults. On the Mac, the default input and output directory is ~/Movies, on PC it’s in My Documents. The pitfall of using the output directory as the input directory is if you cancel any encode, you’ll try to re-encode anything you’ve finished in the last pass. So try to keep them separated for sanity’s sake. I ended up one morning with MY_MOVIE.ISO_T2(through7)_T1.m4v trying to figure out what the heck went wrong.

So here you are readers, a new and improved mass encoder script for using Handbrake to eat massive piles of DVDs with one or more computers and Handbrake. Enjoy!

–edit

A little more experience added a few more things.

1. Favorites, so you can save your favorite encoding strings by name. Mine are PS3 PS3-d PS32 iPhone and iPhone-d with predictable results. But this is easier to type. So replace your -e arg with a favorite -f iPhone and you can save yourself a world of hurt. But -e still works as it did before.

2. Partial encodes are saved to blah.part.m4v until they complete, when they take on the name of their finished product. This was supposed to handle the situation of an unfinished encode displacing the encoder’s next pass, but alas, something doesn’t consider it an error when it’s interrupted *cough handbrake* so I’ll have to work on that some more.

Upload to follow shortly.