1. Handbrake is still your friend. In 0.9.4 (current version) use the profiles named “High” or “Normal” and all is well. They support HD if you see note 2.

2. Keep it under 4 gig. If you’re encoding to HD – make sure to use the ‘target size’ instead of the quality slider. I usually set it to Normal, 3900 meg, 2 passes. Then – and this is important – go do something else for 12 hours.

3. The ElGato Turbo HD encoder DOES make PS3 friendly files in 1080p mode, size permitting (see 2). A friend is performing vicarious encodes of the Iron Man 2 trailer for me, so I can’t answer all your questions yet. But they do play and look good. Oddly, they are bigger than the source, but we’ll worry about that more later.

4. ps3mediaserver works great, especially if you don’t feel like compressing your ISO images. This includes seeking, playback from ISO images, etc. If you do decide to encode down to the PS3 native formats, it’s just like having it on your local disk. And the price is right!

This is as of April 2010 and an up-to-date PS3. Kids currently watching *redacted* without scratching discs, thank you very much.

All builds have a server, 4+ disk RAID5, and offer flexible storage expansion to keep them around for about 2 to 4 years. Each has a “Priced from scratch” number. Remember that this includes 4x $100 drives (currently the 1.5 to 2TB range) that you might already own, or be willing to substitute your old 200-300 gig drives for instead, or whatever. I think you get the idea.

Optimize for money, with Mac guest:

Priced from scratch: $1900ish

• Mac mini (any flavor), add RAM to 4. ($600-1000) ($300-700 on craigslist)
• VMware Fusion ($70)
• Mac OS Server (presumably the license you got from buying the Mini Server, right?) ($500 or included with mac)
• Drobo via local connection for the host, guests live on the Drobo ($400)
• disks ($400)
• Apple USB dongle (technically support is only for the MBAir, but it works. Trust me.) if you want to use a VM router. ($30)

Pro:

• Low electricity.
• Low heat.
• Few surprises.
• Drobo adds the ability to just keep expanding your storage for either more VMs or more space on them without having to worry about iSCSI components losing power between you and the array.
• Easy UI and remote control with Back to my Mac and/or VNC.
• Low TCO – macs depreciate slowly. If it’s time to upgrade to something more powerful later, you can sell it.

Con:

• Few server-quality expansion options.
• No internal high-speed RAID.
• The Drobo will only hit 40ish meg a second, but honestly for home, you’ll probably not need that much all the time.
• VMs limited to 100mbit connections.

Optimize for money, without Mac guest:

New

Priced from scratch: $1550ish

• DIY Intel Core2 quad based on LGA775, 8 gig of ram (priceout on newegg: $550 + hard drives)
• ESXi $0
• Adaptec 2405 RAID controller ($200ish) and SAS case/enclosure ($400)(or other supported card/case/enclosure setup)
• disks ($400)
• Optional substitution: iSCSI remote storage (like FreeNAS or Drobo Pro)(-1000, +cost of chosen iSCSI target)

Pro:

• ESXi is really the right tool for the job.
• Can sometimes get away with a very cheap hardware build.
• Some VMs can go up to gigabit connections if you absolutely must have it.
• Optional iSCSI/fiberchannel storage growth paths open to you.
• Faster and more powerful than the Mac Mini option, but you’ll have some surprises building the box. There is always a hidden surprise.

Con:

• ESXi doesn’t support lots of cheap hardware, so you’ll have to be careful about the hardware you pick.
• You still need a separate windows box/VM around to manipulate things, or get really comfortable with the VMware CLI.
• Higher overall TCO? It’s usually harder selling a used medium-low power server in two to four years.
• It will use more electricity for those extra cores and extra RAM, but if you do video work with this box, it may pay off to have them.
• Electricity/heat in the middle range. Initial expense may convince you that the maintenance is worth it.
• Doesn’t support the mac guest, but there are alternatives to OS X Server’s two most attractive features for the home: iCal server and TimeMachine on flexible storage. Dang, that’s a pro. Moving on.

Used

I will point out that the previous generation of HP/Dell ESX supported servers are regularly on craigslist for cheap – usually $400 and under. Rackmounted and everything, but usually running “real” SCSI disks, very loud, very hot, very small. They still will need $1000 in storage, the same as the rest, so buyer beware, but if you’re just going for the VM server and can skip the storage, you can get in on the cheap.

Optimize for speed, with Mac guest:

Priced from scratch: $5000ish + 2 kidneys

• Mac Pro 8-cores Nehalem, 16GB RAM ($kidney)
• Mac OS X Supported RAID controller plus enclosure and disks. ($other kidney + $1000)
• Mac OS X Server (bundle with Mac?) ($500)
• VMware Fusion ($70)
• disks ($400)
• Alternate to RAID controller and enclosure: Drobo plus disks ($300-3000)

Pro:

• Fast.
• Very fast.
• Lots of memory for lots of VMs.
• Lots of very fast storage.
• Potential no-downtime storage growth.

Cons:

• Dude, you’re out of kidneys.
• Limited hardware choices for SAS RAID.
• Limited iSCSI options.
• Speed gain over Core-2 chips might not enough to warrant price differential unless you do a lot more math than most of us do.
• TCO for depreciation can be large when initial investment is also large.
• Hot. Cooling is expensive.

Optimize for speed, without Mac guest:

Stop. At this point we’re priced out of reasonable home options. A similar spec to the Mac Pro can be built on Newegg for about $1200 with a RAID card. But at that point, you might as well get one of the Asus barebones servers that are fully VMware certified and attach to external RAID storage. When I get as far as this, I know we’ve exited “home” and reached “business” expense levels, at which point the economics of consolidation have already broken down. How many systems do you have to sell and turn off – in your house - to cover the depreciation of a Mac Pro, over the same for a Mini?

Hope that was short enough. Comment away!

Eventually I settled on a layer cake analogy, more or less a table that showed what I was doing and where it was possible, and the game was to shift as much as possible to the left, into the VM column. But this also let me pick how the progression went. I can keep several of my layer cake tables showing a phased motion toward the virtual.

My first step was to document exactly what services require this mess at home and where it’s currently being handled:

Some of those things were not yet implemented but were wish-list items, like the proxy, video encoder as VM, and MythBackend. Anyway, the point is that now I want to move as many things as possible, independent of each other, to the leftmost column available. So the next table shows the eligible places that each service could move:

For the record? This is a lot easier on a pad of paper.

There are now two missing pieces. One is to investigate the technical options for moving each service into a separated VM to provide that service. Second is to plot the order and requirements in such a way that everything eventually ends up where you actually want it. That’s a lot harder than it sounds.

Options are Optional
Now I want to start with the observation that while VMware does have an appliance marketplace, there are entirely too many all-in-one appliances in there. But with the advent of this newfangled thing called the internet, I found that it’s almost as easy to do an easy-install of Ubuntu and install the one package I want, and make that the new dedicated virtual appliance I need. After all, to me the point of this is to have a lot of little VMs doing dedicated jobs, not to have yet another monolithic appliance that does it all.

So I started on my list of service requirements and did some reading. For instance I found instructions on how to build up an iCal server using Apple’s server, but on Ubuntu instead of the integrated OS X Server one. So that’s a VM I built and have on hand. The only issue there will be handling upgrades manually if they change the schema. I’m not sure I’m willing to risk it, but I have a backup server anyway.
Next I built a MythBuntu backend-only install and configured it for my network, then another for just a frontend. I don’t have an encoder card installed anywhere anymore, but it’s good to know it’s ready when I want it. And it’s SO MUCH EASIER THAN IT WAS 5 YEARS AGO. Wow.
The proxy was probably the easiest one – I found one prebuilt in the VM Library that was nothing more or less than a proxy server with virus scanning built in. Perfect!
RadioLover was easy, I just copied my license and settings into the OS X Server VM and started it up. The only Mac I can virtualize is server, so that’s pretty much it unless I switch to other software.
Indigo did not go so well. I was moving this service off an old G4 mac mini where it had been running time out of mind. The issues started when I decided to try to run it inside the VM. Something about the driver doesn’t get along with the USB control unit, and no amount of fiddling fixed it. I’ll have to bring it in to the USB devices team and have them poke about. However, as my mac mini host is still going to be on, I just reinstalled it there, where it was happy enough.
At some point I’m going to have to resolve (in money) my storage solution. It has to live in the noisy, hot garage where I don’t have to hear it. It has to be reasonably fast. There is an option to have it as a virtualized software RAID, so it’s available. I know, I’ve discussed this before, but it needles me so.
So after a day or two of contemplation and migration, this is the new reality:

Items in parenthesis are alternative options, or things I’ve tried but plan not to use. But now it’s time to see how much more can make the leftward shift.

I hope this makes sense to other people, it sure helped me see how I could start doing this at home. I honestly think we’ll start to see a commercial product doing something like this in the near future.

Know thyself

Part of the problem I realized is that I probably don’t need an 8-core Xeon server on 24/7/365 to power my firewall and backup streaming (sorry Apple!). As much as I’d love to have one, I just can’t justify the amount of calories it would produce and feed into my [garage|basement] and the number of kilowatt-hours it would consume. Even with clever power management schemes as turning it off during the 1 A.M. to 1 P.M. periods, spinning down hard drives, etc., I’m just not using that much processor that much time. So I started dividing up jobs into two major categories: always-on services vs. on-demand services. Then I came up with this list, which I’m sure will be different than other people’s:

Always On:

• PPPoE dialer
• Firewall/router/gateway
• wifi-n access point
• DHCP
• dyndns.com updater daemon
• Remote access; VPN, Back to my Mac, whatever
• iCal server; reachable from the outside
• vCenter, maybe; reachable from the outside
• Backup server (RAID, growable)
• PS3 media server (or by schedule, details later) (RAID, growable)
• Web proxy
• “Warm storage” file server – stuff that can go to cold storage if unused for a while
• Home control software (lights, security, HVAC, whatever)
• [unplanned] MythTV backend

On Demand:

• Quicken
• Audiobook downloads
• Video encoding
• Work-VPN VM (it’s just easier this way)
• iTunes to manage files in PS3 media server directories
• daily use items (raw hardware)
• Games (raw hardware)

Note that I have re-ordered this list to make grouping them easier than I had originally done on paper.

Group into hardware

At this point I started drawing circles around things that were obviously easy to tie together, such as the first 5 items. These are currently done by my $80 linksys router with a power draw of something like 10w and almost no maintenance or downtime. Hard to argue with that. So I left those together. Then I looked at how I want to access my home resources from work (or the road) and decided that I still like Back to my Mac as an easy way to get in and see if I really do have that video on my home iTunes server. So that’s not hard. It’s functional so long as I feel like coughing up for the $100/yr service. Which I may be able to stop doing after this project is over. But I digress. The next set of things on the list is the set of “services provided from home”. A free account with dyndns.com or similar services allow me to reach home, so long as the bandwidth and processor power required to provide these to basically myself any my immediate family is small, I don’t need to shell out to a remote company to provide it. So I can host my own iCal server. It’s even open source if I don’t care to come up with an OS X Server license.

Now we get into the range of things that I really had a hard time deciding I could live with leaving in their current state. The backup server and the media server. Currently I have a pair of beat up ReadyNAS sparc-based units. One I bought new after a particularly bad hard drive crash, and the other I bought in a ‘company is folding, see if it works’ sale. I love these units for three main reasons: 1. they don’t need a lot of playing with. 2. they grow the shared volumes without me having to do much work to make it happen. 3. they don’t use a whole lot of power. But: these much older units that I have are not really good at sharing time. As single-purpose appliances they shine, especially because you can let them resize the volume by just inserting new drives. And they don’t ask for lots of maintenance upgrades and security upgrades, etc. They really are great little units if you can find them under $600. So with the luxury of having two already on hand, I decided that with jumbo frames turned on at both ends and a suitable Netgear switch in the middle, they really do perform these two jobs quite suitably. So my cheapest and most sane option is to repurpose them as dedicated appliances instead of general-purpose storage systems. And I also realized this was my migration path – essentially turn them into hardware incarnations of the VMs that they will be replaced by in the eventual ideal solution. This made me happy. So now I can dedicate one to be the growable backup unit and the other to be the growable media server unit. Which they already do.

Web proxy is honestly an optional extra I’d love to implement if I can arrange it. It has its own pitfalls, as the fabled “one more thing to go wrong” as well as being something else to maintain – something I’m usually quite set against. It’s there, we’ll see if it can be done.

“Warm storage” is the place I put things like setup files for downloaded products, project files for Audiobook Builder, iMovie project files I’m done working on but not ready to shelve, that kind of thing. Anything that is in warm storage for about 6 months will eventually get moved to offline cold storage and kept powered off, in a box, either at work or at home. As this is something that I want access to and won’t use often, and really is just about a terabyte of things that I’m not quite done with yet, I could group this with the hardware that does the backups. Speed isn’t an issue, just the access and the space, as these are quite regularly bigger than a DVD. Again, as this could easily be defined as something like a FreeNAS VM or a slice of a file server VM, I don’t mind this as a standalone role that can be virtualized in the final ideal solution.

Now we’re down to the on-demand services. Re-evaluating what these roles are helped me see that most of this can be handled in two places before the “final ideal solution” comes around. Currently the Quicken VM lives on my laptop. The Audiobook downloader lives on my low power mini server at home. Video encoding is done on the native OS of the mini server. The work VPN VM is on my laptop, as that’s where I use it most often. The iTunes library is on the native OS of the mini. I don’t mind this arrangement, but it helped me realize again the eventual arrangement in virtual machines I’d like to use, breaking them into specific roles let me see the migration path as hardware permits.

Now that I have the new role-based model to help me see what is where now and where I want it to be later, I came up with this diagram:

Big line flow charty thing

There are still a few stray points to clear up. I could entirely abandon the Linksys if I can find a way to control a wifi card like an access point. I don’t have to give up my growable storage arrays yet, but I could if I can find a 3ware card and enough drive bays (I already have, next post, I swear).

Note that this is just the role flowchart. I have additional charts that I’m working on to deal with network paths, storage, and so on. It’s not a trivial study as I originally thought. But there you have the original thoughts on how to organize it into hardware groups in a more realistic home environment.

• First I had to identify what actually needs to get done on the network.
• Second I had to identify how much space that is, and how and where to store it.
• Third I had to identify how much to get that kind of performance.
• Fourth I had to identify how to sell a kidney. Because that’s how much it would take. But, seeing as I’m rather attached to mine, I think I’ll just look for a less expensive way.

So let’s look.

1: What gets done and where will it go.

Central Server: I’m going to skip a long discussion of why and tell you now that I want  to run OS X Server on my home network to do a number of jobs. We can put why in another article. It will handle the backups properly, the file sharing properly, the calendar serving properly, etc. I like what it does and how it does it. I’m not going back into serving email with it this round, but it’s nice to know it can. This can be virtualized, but only on Apple host hardware. I can live with that. It will hold offline photos, backups, media server for sharing to the PS3, and well anything else I can think of that can just be piled into one server.

Satellite VMs: I also run Quicken in a VM. And I have to download Audiobooks from the public library in Windows. And so help me I want a web proxy. And so long as we’re on it, I’d like to have a VM that does most of the work the Linksys box does. Possible, but not probable.

Backup server: currently on a ReadyNAS box, but getting hard to administer.

Cold storage server: For keeping uncompressed ISO images of movies, lossless compressed audio of CDs, audiobook builder project files, iMovie project files, etc. that should be kept, but maybe not online. This is currently done by cold-swapping in older hard drives via USB and leaving them in the desk, off.

Then the user-only boxes I don’t plan to make virtual currently:

Desktop mac mini: My scanner/iTunes/movie sync station.

Gaming PC: I don’t have one any more, because mine is so out of date, but I want one. I’ll put it in the list.

Wife laptop: Photo editing, daily tasks, etc.

My laptop: Photo editing, daily tasks, etc.

Kid browser box: web games, run Indigo (to turn on/off lights in the house).

2: How much space do I really need in the coming year?

Doing all the math on a sheet of real dead-tree paper (with blue lines no less!) I worked out that I need about 16 TB total for all that, online all at once:

7 online data – VMs, media, photos.

4 backups

4 cold storage space

2 growing space to prevent having to rebuild next year.

3: How?

In order to get all that storage in one server, I’ll want something quite serious. But I can’t afford quite serious. Which leads to a problem. How fast is fast enough? My current network is gigabit, but the transfer rate to the low-power out-of-date RAID units I have caps out at 5 meg a second. Hardly fast enough for the kind of things I plan to make them do. In fact a serious factor in doing this all in the first place is that the two units I have keep bogging down doing things like playing a movie while deleting files or well, anything else, at the same time. So they’re essentially serial RAID units, you can only use them from one location at a time. And that won’t fly.

So the problems to solve are now boiled down a bit to this:

I want a powerful server to run VMs.

It will need about 8 GB of RAM and a super-fast 12-16TB RAID shared over the network.

I want to run MacOS X Server, preferably as a VM.

I want to be able to share lots and lots of data from a hopefully centralized point.

I want to use less power and rack mount this mess in the garage, out of reach of little fingers.

I want to be able to grow the storage with a minimum of fuss, and incrementally.

4: Can it be done in a house?

The simple solution would be to buy an Intel based XServe with a fiberchannel card and XRaid, install them in the rack (sideways, so I can still fit the car in), run the VMs in Fusion and call it a day. But the price puts that idea out of reach.

Price ballpark: Over 5 kilobucks.

The next notch down is to get a Mac Pro with a 3ware 9690 card plus gobs of RAM, run regular OS X on it, Fusion in the middle, OS X Server in a VM, other VMs on Fusion.

Price ballpark: Over 3 kilobucks.

The next notch down after that is to decouple the mac jobs from the non-mac jobs, so keep the mini and the McServer VM together, but move all the other VMs to a beefy-ish ESXi server and slowly learn to do each of those other tasks on the ESXi server, and eventually write off OS X Server. But as the idea was to consolidate instead of fragmenting more, this is counterproductive at the moment.

Price ballpark: 1.5 kilobucks.

The least attractive option (from a consolidation view) is to just rebuild the ReadyNAS units I have with bigger drives and continue to live with the slower transfers, put bigger drives in all the user boxes and only transfer backups and cold storage over the network. This would also require putting something like a Drobo on the mini to give the OS X Server VM enough storage for the media library and other tasks that the Mini would have to absorb. This method has its advantages, as the mini and storage would be ‘local’ on USB, it would support incremental storage growth, etc. But it only has RAM space enough for 1 VM (the server), the mini is NOT a powerhouse, the bus speed is really too low for heavy use, and while it’s low power, it probably won’t stand up well to long term usage at that level. I’d probably need a cheap used Dell or something similar to run the other VMs, and I have a hardware raid card I can throw into that. Storage probably 1 TB (x2 for redundant), 8 GB of inexpensive RAM, would need another $220 in hard drives for it.

Price ballpark: .5 kilobucks.

But the options aren’t quite over. There are a set of machines called “Hackintoshes” that purport to be quasi-legally able to run Mac OS. The fact is they’re not entirely on the legal end of the stick, but may be technically capable. Do I dare risk building a machine that might run Mac OS X for half the price of a Mac Pro? Is it worth the risk? Personally, I’m inclined to wait out the lawyers. This would reduce the price of building an XServe quality unit from $3000 to $1100. That’s not insignificant.

And since I don’t need cutting-edge abilities, I could go with a used Mac Pro/XServe, which go around $1400 instead of $2500.

Then consider the selloff of the existing infrastructure components. Maybe $500 for the RAID units would be something. So we’re not entirely lost here.

Initial speed tests done by installing a 3 year old RAID card in my Windows box shows I can get a 4x network share speed increase by using a new server, so it can be done, and that box is so old it only has a 133 bus and runs the RAID card at half speed. There is much to be gained.

So what to do? Comment away, I’ll read and consider, then we’ll see what real dollars get spent.

Oh, and there was research along the way into using iSCSI to store the RAID in the garage and keep the brainy server local, but the lack of iSCSI initiators for the mac under $200 a pop discouraged me. The free one doesn’t like Snow Leopard x64 at the moment. And the option of using afp protocol on top of zfs surfaced, but the combination in FreeNAS proved unstable and the zfs availability issue keeps getting in the way. It’s possible, but the hassle doesn’t pay for the payoff in my situation.

Back Story

We finally stepped part way into the HD era, by attaching a PS3 to our SD television. Sacrilege, I know, but it’s what I have for now, and the economy isn’t getting any better this week. But, as we only have two games for it (Little Big Planet and Mater-National Racing), we have found that it has another wonderful use. The Videos menu.

(Keep in mind that I have 3 little kids in the house, and while most of their viewing is in the broadcast-DVR, I still have hopes that they will take an interest in longer attention span viewing, or at least the Muppet Show season sets and Looney Tunes sets I have waiting. But handing them DVDs is a great way to have things get destroyed. Enter digital viewing on the TV.)

So, having an Infrant ReadyNAS x600 in the garage, I bought a copy of the far more predictable TwonkyMedia server that handles all the gross stuff related to sharing your video folder with the PS3. The built-in server just wasn’t maintaining itself well, no matter how I tried. I’m sure someone can make it go, but I needed something a bit more turn-key. But the bigger problem still remained: I have something along the line of 300 DVDs between movies, television series, and home-made things. Ripping is the tetchy topic, and if you have questions we can talk via email, but the fact is that ripping even the most difficult DVDs is now possible. But now you have at least one hard drive full of DVD folders or ISO images of DVDs, and you have to turn this into something a bit more manageable.

This is the PS3 specific portion – if you are comfortable with how your player plays back and what encoder settings you can get from Handbrake, then feel free to skip forward. As clients to my encoded DVDs, the first priority was everything was to be HD resolution for the PS3 and/or my TViX 4100. There was also a smaller subset that I wanted to play back on my iPhone. After much experimenting with  Handbrake, I settled on a set of 2 encoding strings that made me happy. One was the default PS3 profile, plus chapter markers (for computer playback), plus 2-pass encoding. The other was the same, but with deinterlacing turned on, for those titles like the Muppet show that were interlaced on the discs. (If you don’t deinterlace those, then the frames come out in the wrong order when replaying on the PS3, and make things jump around forward-backward-forward and drive you to turn it off.) Fortunately, the TViX plays these files without modification, so I was set for both devices. So those two strings were written and set aside. Then I came up with a parallel pair of strings for the iPhone setting.

So I had my 4 preferred encoding settings and a pile of 300ish ripped DVD images or folders to eat through. I knew there was a problem. At PS3-2pass resolution, each encode was about 4 hours. That’s a lot of hours, and queuing each in Handbrake’s GUI takes more time than I want to do. I already knew exactly what I wanted, but I didn’t want to do it by hand.

What I Did

So I did a little digging on Google and discovered that for some reasonably old version of Handbrake, someone wrote a bash script that would turn VIDEO_TS folders into mp4’s. As time had left this little gem behind, I did some minor repairs and used that for a couple encodes. But it had some limitations I didn’t like, and didn’t work at all for Windows machines, and so on. I even modified the original VIDEO_TS script to work with ISO images, and that saved me some more work. I could then do massive batch encodes using my two scripts. I took images to work and left the far faster work computers cooking m4v’s for me over weekends and nights. It was great for saving me weeks of computing at home. But I had to custom-build the scripts for Windows via shared folders on the mac, and it still took too long to manage two scripts across multiple machines. Enter the perl experience. At work I’ve been working with cross-platform scripting for a while now by using Perl. I opened up the guts of the VIDEO_TS script again, figured out how to do what they were doing, added the new abilities of Handbrake’s latest versions, and got to work giving the Windows machines the same logic the Mac had with the bash script.

The new script, stupidly named encode.pl, now handles pretty much anything that Handbrake can handle – or at least anything I encountered. First, you queue a directory. It simultaneously queues VIDEO_TS directories, ISO images, wmv/avi/mov/etc. files for encoding. Then it proceeds to do discovery via Handbrake about what it should encode. It does this by querying the title list for the duration and then applying a filter for length. For instance, my default setting is anything between 20 and 600 minutes, but that can be tuned with the –minimum or –maximum settings. Then it goes through them one by one, handing the whole thing off to Handbrake to be encoded. As each one finishes, it goes to the next one. No mess, no fuss. Until you run out of hard drive space that is. Can’t help you there.

So then I had to tune the way I organized my DVD images for feeding into this thing. My ReadyNAS had enough space at the time, so I put things into distinct folders. In the media share, I had a Videos folder. Inside I created the following:

/Volumes/media/Videos/todo/…

…/deinterlace/long

…/deinterlace/episodes

…/long

…/episodes

- and here’s why. The long directories were for things that I wanted only the titles that were over an hour and ten minutes. This let me get a single file for all of the Looney Tunes shows, strung together, instead of a pile of 7-9 minute shorts. There are enough per DVD to make it frustrating to have to return to the menu for each new show. Kids will just give up, and so will I. They already have a title with all of them together, so I just added the –minimum 70 setting when I do the long directories.The episodes directories were for things no longer than an hour. These are the opposite of the long things. So for cartoons, sitcoms, and other 22-44-minute episodes, I don’t want the whole DVD as one title, I want each show as an individual m4v file. That directory got the –maximum 50 setting. This pretty much handled everything except for a couple odd cases that I just encoded more or less by hand with the GUI – mabye 3 discs total.

For output directories, I created:

…/m4v

…/iPhone

Now, because I had central storage for both the input directory and the output directory, I also considered using multiple computers. I was able to use the 4 computers under my desk to all work on a central queue. I would go to each computer, mount the share, and put together the encode string for that machine, and set it running. Each computer when it starts an encode creates the file it’s starting on, and then starts the Handbrake line. I had to do this because with 2-pass encodes, the output file is not present until the second pass starts. Because of this, if you have computer A starting on title 1, then computer B might start on it too. If the file is present, then computer B says, “Oh, that one’s there, let’s start the next one.” So you get a cheap claim-before-starting system built in if encoding to the same destination. Then another thing came up. I could do encodes on my laptop while away from the main servers. So I added the ability to keep a log of what’s encoded and copy that to remote locations (*cough* work) and still not get double encodes. It’s not a perfect system, but it’s better than nothing.

The Interesting Part

So the result is a semi-distributed way of encoding a seriously big pile of movies into a standardized format using both Macs and PCs, possibly central storage, and getting a pile of files with at least standardized names.

I’ve actually considered trying to add a way to use STAF to auto-distribute encodes to available systems, but that takes a bit more time than I have at the moment. But that could allow you to have a central “encode controller” and some “encode slaves” that you hand out single encode jobs to. It’s within reach, I just need an hour here or there to try it.

Anyway, here’s how you use it:

perl encode.pl -i /Volumes/media/Videos/todo/long -o /Volumes/media/Videos/m4v --minimum 5 --maximum 70 -e "-Z PS3"

or the equivalent in Windows paths:

perl encode.pl -i M:\Videos\todo\long -o M:\Videos\m4v --minimum 5 --maximum 70 -e "-Z PS3"

-i for an input directory

-o for an output directory

-e for anything you want to hand directly to the encoder. “-Z PS3″ is the default PS3 string, but be careful with “-Z ‘iPhone & iPod Touch’” because the shells differ in quote handling. Try it on a single command line to get it right and then keep it around in a text file somewhere to be sure. Also, because of the funny quoting problems, I usually try to make the -e setting the last one I hand in.

–minimum is the minimum time in minutes that you want to approve a title for encoding

–maximum is the opposite. Note that minimum and maximum ignore seconds.

Note that almost everything is optional. You can open the perl code and change to your favorite defaults. On the Mac, the default input and output directory is ~/Movies, on PC it’s in My Documents. The pitfall of using the output directory as the input directory is if you cancel any encode, you’ll try to re-encode anything you’ve finished in the last pass. So try to keep them separated for sanity’s sake. I ended up one morning with MY_MOVIE.ISO_T2(through7)_T1.m4v trying to figure out what the heck went wrong.

So here you are readers, a new and improved mass encoder script for using Handbrake to eat massive piles of DVDs with one or more computers and Handbrake. Enjoy!

–edit

A little more experience added a few more things.

1. Favorites, so you can save your favorite encoding strings by name. Mine are PS3 PS3-d PS32 iPhone and iPhone-d with predictable results. But this is easier to type. So replace your -e arg with a favorite -f iPhone and you can save yourself a world of hurt. But -e still works as it did before.

2. Partial encodes are saved to blah.part.m4v until they complete, when they take on the name of their finished product. This was supposed to handle the situation of an unfinished encode displacing the encoder’s next pass, but alas, something doesn’t consider it an error when it’s interrupted *cough handbrake* so I’ll have to work on that some more.

Upload to follow shortly.

During that 5 years I lost 4 different hard drives, and those were the days when backups were nearly impossible, RAID meant enterprise hardware, and hard drives were not the most reliable. This made me very grumpy. Various methods of backing up were tried, but none ultimately worked very well, and I lost small segments of data, recovered at the cost of time. One of the things most annoying was that every time you re-installed Quicken on a new hard drive, you had to re-register, which forced you to make up yet another new identity for their web site, which you didn’t want to use anyway. Because, of course, you would never reinstall software if you didn’t intend to make up a new identity for yourself.

Then I started working at VMware, and this was an idea I’d had before but was too cheap to try. I moved my Quicken data inside a VM. Hard drives had since come down to the dollar-a-gig range, and I was tired of the risk of leaving my Quicken data just laying on a drive that was vulnerable to being scanned by anything that came by. Around this same time, the MacIntel thing started, and I was able to start using used MacOS machines for Mail.app, which moved the risk of viruses down about 10000%. But I was more after the isolation, backup, and management problems associated with Quicken. Like the little known replacement keyboard driver they use. Like the data files just sitting on your drive waiting to be explored. Like the fragile backup strategy they use. Those wrankle.

So when I started working at VMware, I installed a copy of Windows (legally – my laptop had died) in a VM, installed Quicken, and did the backup and restore as Intuit intended. Then I fixed everything. Then – and here’s the big one – I took a snapshot, and burned the whole thing to a DVD. Now, worst case, I can always get back to that point without question – no registration, no stupidity – just copy files and hit go. And I’ve continued to work this VM exactly this way for a couple years now. I’ve traded 5gb of space for the knowledge that no matter what happens, I can always copy the VM from my MacBook Pro to my PC and boot Quicken in case of disaster.

So, when this morning I was told, “hey look, 2009 is out” and knowing they don’t offer upgrade paths beyond 2 years any more, I decided to take a snapshot and jump in, knowing if all went horribly wrong, I’d just rewind, wait 2 weeks, and try again. That kind of assurance is worth the 5 gb of space in my opinion.

In practice this changes my original collection of 20ish files per CD times what, 15ish CDs per book into about 5 for the entire set of 4 books. The original rip I had done was in the early, bad days of MP3, 32kbps, 22mhz, blah blah blah, and plainly it sounded like a robot squawking into your ear, but you could tell what it was and after about 2 minutes, you were tuned out of the noise issue. And the files were tiny, which was my intent. So when I did this project, I ripped to Lossless and then had Audiobook Builder downsize to the current audiobook norm format: 64kbps, 22mhz, mono, and the sound is spectacular by comparison. Space savings? Probably negative in total. But the joining of the files was not all I got out of this.

I also did this with the Harry Potter audio collection, although I didn’t merge all the books into a single string of files (Parts 1-15 at 12 hours each?) But what is extremely cool about the newer iPhone/iPod generation plus Audiobook Builder is that you can add the chapter art (found via google) to the chapter headings. It makes it just that much more interesting to listen to and remember, as the art changes with each chapter. I wish Audible had that feature. Heck, I wish Audible books more often had chapter marks that matched the book’s chapters.

I also did something that I’m not sure is smart, but I’m pretty sure it’s not dumb. I re-ripped all those discs as Lossless AAC, so the next time a cooler, better codec comes along, I can just re-encode them from the lossless source (idea originally from pixelknave). Disk space is becoming cheap enough and I can just stash the files on my 2TB ReadyNAS, out of the way of my iTunes library.

I am now some months into it, well into maintenance mode. I’m pleased to report that I haven’t given up, and it’s actually working for me. I have a path for pretty much all paper that enters the house. It comes in and goes through junk filtering. Junk gets tossed, the interesting stuff gets sat on the desk next to the scanner, generally unopened. Magazines go in the magazine basket, never to be seen again (except by the 1 year old, who eats them), catalogs go in the recycle bin before my wife can see them, anything with an account number, name, etc. go in the pile to be checked for scanning and shredding. Once or twice a week, I go sit down, open the envelopes and repeat the junk filtering above. Any offers, especially credit offers, are shredded. I have enough to keep my life busy. Statements are scanned and then put on the shredder for later batch shredding. DMV and periodic payments are scanned and then put on the same pile. This accounts for most of the things that come in. Afterward, I shred the paper and it’s done. The next phase is done on the laptop.

On my particular laptop (MacBook Pro), I installed a bit called MarcoPolo, which lets me have things happen when it things you’re in a particular place. So the next time I open the laptop at home, it will go get those scans off the network share and put them in my Pending Documents directory. Next time after that I open Yep, I have a list of files to be tagged and dealt with. The statements are generally a one-step into Quicken in my VMware Fusion Windows XP Quicken Box (longer article pending there), where I put in the amounts and hit send once I’ve done them all. Easy. Then I put the tags on those files and they’re no longer “new”. Sometimes, when I’m filing without quicken up, I just tag them “todo” and come back to them later. It’s quick enough I usually do it all at once. Once this is done, I’ve pretty much dealt with everything on the pending list. Occasionally I’ll get a bill that’s due some months out (Car insurance, etc.) and they get put in the Scheduled Transactions in Quicken, to be sent either automatically within 2 weeks beforehand or whatever.

I have another thing that I’ve learned to do for forms that I fill out occasionally. I picked up a ScanSnap for my mac, and it came with an older version of Adobe Acrobat Pro. I scanned in the form for my flexible spending plan and turned it into a form to turn in our babysitting and health stuff. So now I go type a couple amounts, print, sign, and fax. If I had thought ahead enough to sign it first, I could probably just email it in, but that has other dangers. Fax is more obscure if nothing, so I’ve stuck to that for now.

So the tagging itself for documents hasn’t changed much. I have added a new category called Articles that covers newspaper clippings, interesting bits PDF’d off the internet, etc. that I want to keep static and offline. Otherwise, things are just as I’ve described before. The whole process from start to finish takes less than 10 minutes for each step, done maybe once or twice a week in three steps. Heck, I don’t even have to remember to download the files anymore, saving me a serious step (rsync to an unmounted network share via auto-launched Applescript – beautiful).

Remind me to write up more on what I do with MarcoPolo, I’m very pleased with it once it’s configured, but I hate the configuration.

defaults write com.apple.systempreferences TMShowUnsupportedNetworkVolumes 1

And guess what.

I’m happy.