Why didn’t anyone tell me…

I just finished the full length adaptation of Blade Runner (the book), which is the stretching out of “Do Androids Dream of Electric Sheep”, both by Phillip K. Dick. First, it’s not the movie. It’s not even the same story. The thing that impressed me most is the split personality he exposes between the priority that earth people put on pets (in the book) versus the priority they place on artificial pets, which are both higher than escaped androids. I’ll be honest, I preferred the story from the movie, but I’m not disappointed with the book.

Forgot to mention that I also read Orson Scott Card’s “Ender in Exile“, which I find to be a good, but rather schizophrenic book. I can’t figure out which story he’s trying to tell. There are at least 3 story lines that happen, and I think I’d have preferred if they were dealt with in a “Book 1, Book 2, Book 3″ style instead of just mostly jumbled in. While I was able to follow the stories, I didn’t think they had much to do with each other, or not enough that it was interesting to have them intermixed. Maybe I’d have just preferred to have three short stories instead of the novel-length version. Either way, I was not unhappy with the book, but I felt like there could have been a stronger way of telling the same content.

There are a few books I’ve read since I last blogged. And here’s something I don’t do often enough: I stopped reading midstream. That’s right, I gave up on a couple of these books. Gasp! The horror!

In Cold Blood by Truman Capote. While I appreciate that this may have been the launch of modern True Crime books, the fact is that we see twice as hard on CSI three times a day with new seasons being cranked out in triplicate even now. The wandering nature of the narrative is distracting and plainly annoying to a modern reader. This is a book that time has outdated. I appreciate that it may have been groundbreaking, but now it’s just broken. Uninteresting and slow with no apparent direction. Please everyone, run out to your local store and demand they stop selling this disaster of a book. Send it back to the ’50s where it belongs.
Also given up on:
Bridge of Sighs. At first it opens like a guy going to recount a story about a trip to Italy, to tell about his life in context of life in a small New York town, then it suddenly jumps to him getting locked into a trunk in the woods by a bunch of bullies and him listening to drunk people having sex, and the life of a spoiled artist in Venice. What happened? After a quarter of the book and nothing really going anywhere, it was ejected. There were clearly 2 stories going on, but I just didn’t care enough about either of them to waste more time on it. 1 point for the old man narrative, -1 for the artist, -1 for jumping around. Shame on you Russo. Reel us in and keep us, don’t try to make Memento a book.

I think it’s time someone took those old cameras out of circulation, and I think I’m the man. So I’m holding a photo contest of a different sort. Send me a photo of your oldest camera body or other equipment in the act of being violently destroyed, and I’ll pick the best one and send the winner $100. I don’t care about how old or rare it is, I care about the boom. Let’s hold this open until September 1 at which point myself and an uninterested second will judge the best one.
Now go out there with your explosives and don’t hurt yourselves while shooting. JPGs, under 800×600 resolution.

encode

Back Story

We finally stepped part way into the HD era, by attaching a PS3 to our SD television. Sacrilege, I know, but it’s what I have for now, and the economy isn’t getting any better this week. But, as we only have two games for it (Little Big Planet and Mater-National Racing), we have found that it has another wonderful use. The Videos menu.

(Keep in mind that I have 3 little kids in the house, and while most of their viewing is in the broadcast-DVR, I still have hopes that they will take an interest in longer attention span viewing, or at least the Muppet Show season sets and Looney Tunes sets I have waiting. But handing them DVDs is a great way to have things get destroyed. Enter digital viewing on the TV.)

So, having an Infrant ReadyNAS x600 in the garage, I bought a copy of the far more predictable TwonkyMedia server that handles all the gross stuff related to sharing your video folder with the PS3. The built-in server just wasn’t maintaining itself well, no matter how I tried. I’m sure someone can make it go, but I needed something a bit more turn-key. But the bigger problem still remained: I have something along the line of 300 DVDs between movies, television series, and home-made things. Ripping is the tetchy topic, and if you have questions we can talk via email, but the fact is that ripping even the most difficult DVDs is now possible. But now you have at least one hard drive full of DVD folders or ISO images of DVDs, and you have to turn this into something a bit more manageable.

This is the PS3 specific portion – if you are comfortable with how your player plays back and what encoder settings you can get from Handbrake, then feel free to skip forward. As clients to my encoded DVDs, the first priority was everything was to be HD resolution for the PS3 and/or my TViX 4100. There was also a smaller subset that I wanted to play back on my iPhone. After much experimenting with  Handbrake, I settled on a set of 2 encoding strings that made me happy. One was the default PS3 profile, plus chapter markers (for computer playback), plus 2-pass encoding. The other was the same, but with deinterlacing turned on, for those titles like the Muppet show that were interlaced on the discs. (If you don’t deinterlace those, then the frames come out in the wrong order when replaying on the PS3, and make things jump around forward-backward-forward and drive you to turn it off.) Fortunately, the TViX plays these files without modification, so I was set for both devices. So those two strings were written and set aside. Then I came up with a parallel pair of strings for the iPhone setting.

So I had my 4 preferred encoding settings and a pile of 300ish ripped DVD images or folders to eat through. I knew there was a problem. At PS3-2pass resolution, each encode was about 4 hours. That’s a lot of hours, and queuing each in Handbrake’s GUI takes more time than I want to do. I already knew exactly what I wanted, but I didn’t want to do it by hand.

What I Did

So I did a little digging on Google and discovered that for some reasonably old version of Handbrake, someone wrote a bash script that would turn VIDEO_TS folders into mp4’s. As time had left this little gem behind, I did some minor repairs and used that for a couple encodes. But it had some limitations I didn’t like, and didn’t work at all for Windows machines, and so on. I even modified the original VIDEO_TS script to work with ISO images, and that saved me some more work. I could then do massive batch encodes using my two scripts. I took images to work and left the far faster work computers cooking m4v’s for me over weekends and nights. It was great for saving me weeks of computing at home. But I had to custom-build the scripts for Windows via shared folders on the mac, and it still took too long to manage two scripts across multiple machines. Enter the perl experience. At work I’ve been working with cross-platform scripting for a while now by using Perl. I opened up the guts of the VIDEO_TS script again, figured out how to do what they were doing, added the new abilities of Handbrake’s latest versions, and got to work giving the Windows machines the same logic the Mac had with the bash script.

The new script, stupidly named encode.pl, now handles pretty much anything that Handbrake can handle – or at least anything I encountered. First, you queue a directory. It simultaneously queues VIDEO_TS directories, ISO images, wmv/avi/mov/etc. files for encoding. Then it proceeds to do discovery via Handbrake about what it should encode. It does this by querying the title list for the duration and then applying a filter for length. For instance, my default setting is anything between 20 and 600 minutes, but that can be tuned with the –minimum or –maximum settings. Then it goes through them one by one, handing the whole thing off to Handbrake to be encoded. As each one finishes, it goes to the next one. No mess, no fuss. Until you run out of hard drive space that is. Can’t help you there.

So then I had to tune the way I organized my DVD images for feeding into this thing. My ReadyNAS had enough space at the time, so I put things into distinct folders. In the media share, I had a Videos folder. Inside I created the following:

/Volumes/media/Videos/todo/…

…/deinterlace/long

…/deinterlace/episodes

…/long

…/episodes

- and here’s why. The long directories were for things that I wanted only the titles that were over an hour and ten minutes. This let me get a single file for all of the Looney Tunes shows, strung together, instead of a pile of 7-9 minute shorts. There are enough per DVD to make it frustrating to have to return to the menu for each new show. Kids will just give up, and so will I. They already have a title with all of them together, so I just added the –minimum 70 setting when I do the long directories.The episodes directories were for things no longer than an hour. These are the opposite of the long things. So for cartoons, sitcoms, and other 22-44-minute episodes, I don’t want the whole DVD as one title, I want each show as an individual m4v file. That directory got the –maximum 50 setting. This pretty much handled everything except for a couple odd cases that I just encoded more or less by hand with the GUI – mabye 3 discs total.

For output directories, I created:

…/m4v

…/iPhone

Now, because I had central storage for both the input directory and the output directory, I also considered using multiple computers. I was able to use the 4 computers under my desk to all work on a central queue. I would go to each computer, mount the share, and put together the encode string for that machine, and set it running. Each computer when it starts an encode creates the file it’s starting on, and then starts the Handbrake line. I had to do this because with 2-pass encodes, the output file is not present until the second pass starts. Because of this, if you have computer A starting on title 1, then computer B might start on it too. If the file is present, then computer B says, “Oh, that one’s there, let’s start the next one.” So you get a cheap claim-before-starting system built in if encoding to the same destination. Then another thing came up. I could do encodes on my laptop while away from the main servers. So I added the ability to keep a log of what’s encoded and copy that to remote locations (*cough* work) and still not get double encodes. It’s not a perfect system, but it’s better than nothing.

The Interesting Part

So the result is a semi-distributed way of encoding a seriously big pile of movies into a standardized format using both Macs and PCs, possibly central storage, and getting a pile of files with at least standardized names.

I’ve actually considered trying to add a way to use STAF to auto-distribute encodes to available systems, but that takes a bit more time than I have at the moment. But that could allow you to have a central “encode controller” and some “encode slaves” that you hand out single encode jobs to. It’s within reach, I just need an hour here or there to try it.

Anyway, here’s how you use it:

perl encode.pl -i /Volumes/media/Videos/todo/long -o /Volumes/media/Videos/m4v --minimum 5 --maximum 70 -e "-Z PS3"

or the equivalent in Windows paths:

perl encode.pl -i M:\Videos\todo\long -o M:\Videos\m4v --minimum 5 --maximum 70 -e "-Z PS3"

-i for an input directory

-o for an output directory

-e for anything you want to hand directly to the encoder. “-Z PS3″ is the default PS3 string, but be careful with “-Z ‘iPhone & iPod Touch’” because the shells differ in quote handling. Try it on a single command line to get it right and then keep it around in a text file somewhere to be sure. Also, because of the funny quoting problems, I usually try to make the -e setting the last one I hand in.

–minimum is the minimum time in minutes that you want to approve a title for encoding

–maximum is the opposite. Note that minimum and maximum ignore seconds.

Note that almost everything is optional. You can open the perl code and change to your favorite defaults. On the Mac, the default input and output directory is ~/Movies, on PC it’s in My Documents. The pitfall of using the output directory as the input directory is if you cancel any encode, you’ll try to re-encode anything you’ve finished in the last pass. So try to keep them separated for sanity’s sake. I ended up one morning with MY_MOVIE.ISO_T2(through7)_T1.m4v trying to figure out what the heck went wrong.

So here you are readers, a new and improved mass encoder script for using Handbrake to eat massive piles of DVDs with one or more computers and Handbrake. Enjoy!

–edit

A little more experience added a few more things.

1. Favorites, so you can save your favorite encoding strings by name. Mine are PS3 PS3-d PS32 iPhone and iPhone-d with predictable results. But this is easier to type. So replace your -e arg with a favorite -f iPhone and you can save yourself a world of hurt. But -e still works as it did before.

2. Partial encodes are saved to blah.part.m4v until they complete, when they take on the name of their finished product. This was supposed to handle the situation of an unfinished encode displacing the encoder’s next pass, but alas, something doesn’t consider it an error when it’s interrupted *cough handbrake* so I’ll have to work on that some more.

Upload to follow shortly.

Brisingr, or why the heck after 3 books does the hero still annoy me? I went back and dread the first two books because I couldn’t pick the fine details from memory, and that was annoying. But I do have this to report, the book gets better around the two-thirds point.
I also think I figured out why it annoys me so. The hero acts and talks like a ten year old. I realized this while having dinner with friends who happen to have a ten year old. Then it clicked. Eragon is stuck in prepubescence. It didn’t make reading it much better.

Dave: I have 5 books in the queue, but a depressing book will be a morning-only read, leaving the evening for podcasts/re-reads.
Dave: like The Glass Castle was a morning-only reader.
Nate: by Jeanette Walls?
Dave: yeah
Dave: good book, but don’t read it on the way home
Nate: the wikipedia summary makes it look depressing
Dave: well, it depends on how close you were to people that lived like that
Dave: in my town, it was all around you
Dave: so to me, it was her victory dance. She got through it, survived, and left. In every way it felt like me leaving my hometown and getting away. I didn’t live in squalor with drunks and weirdos, but I was just as trapped as she was at various points.
…
Dave: one thing I did like about it was how she didn’t make it feel like she was harping on about how bad things were. it really just felt like her recalling stories. some were funny, some mean, some depressing, some sweet, and the tone of the whole was desparate motivation to better her life.
Dave: and how every time she tried, her parents shut her down, right up until she left
Nate: that’s great. I hate the “everything sucked” toned books/stories.
Dave: and even some of the time after she was established on her own, how her parents became homeless and refused to get proper help, etc. there was a great moment when she was taking an economics class and the teacher asked why people are homeless. her response was that some people choose it. teacher freaks out and challenges her:
Nate: heh. except that her parents did just that.
Dave: “what do you know about it? how could they want that?” author’s response was one of fear – saying sorry, i guessed wrong – because she feared her friends would find out her parents were homeless and chose to be so voluntarily
Dave: so no one-sided characters. the deadbeat dad loves his kids desparately but refuses to give up the bottle, but does some truly sweet things for them. the mom is just nuts, i guess that never changes, but the kids get a lot of depth.
* Dave will probably have to paste this whole thing in the blog
Nate: just a little clean up and it becomes a nice little book review.

My wandering uses of Quicken started with very early versions on Mac OS, back in the black and white days. Then there was the new age of color macs, and then shortly after was the Windows Revolution, where they discovered that there were greener fields on green screens. Then, suddenly, there was not one product called Quicken. There were two. Quicken for the Mac and Quicken for Windows were only named the same – nothing  else was common. The Mac version had features the Windows version did not and vice versa. I stuck on the Mac version for a long time, and then a few things happened that made me give up on the Mac. One, Macs got a lot more expensive than I could justify on the budget I had. Two, the PC version suddenly got online syncing and the Mac versions were busted. Three, and this is a big one, games came to the PC. I wanted a PC, and I bought a PC, and my mac was not dismissed, but it was not permitted to hold my Quicken data any more. The online sync feature was the dealbreaker, and I had a PC on hand. And that was how it was for another 5 years.

During that 5 years I lost 4 different hard drives, and those were the days when backups were nearly impossible, RAID meant enterprise hardware, and hard drives were not the most reliable. This made me very grumpy. Various methods of backing up were tried, but none ultimately worked very well, and I lost small segments of data, recovered at the cost of time. One of the things most annoying was that every time you re-installed Quicken on a new hard drive, you had to re-register, which forced you to make up yet another new identity for their web site, which you didn’t want to use anyway. Because, of course, you would never reinstall software if you didn’t intend to make up a new identity for yourself.

Then I started working at VMware, and this was an idea I’d had before but was too cheap to try. I moved my Quicken data inside a VM. Hard drives had since come down to the dollar-a-gig range, and I was tired of the risk of leaving my Quicken data just laying on a drive that was vulnerable to being scanned by anything that came by. Around this same time, the MacIntel thing started, and I was able to start using used MacOS machines for Mail.app, which moved the risk of viruses down about 10000%. But I was more after the isolation, backup, and management problems associated with Quicken. Like the little known replacement keyboard driver they use. Like the data files just sitting on your drive waiting to be explored. Like the fragile backup strategy they use. Those wrankle.

So when I started working at VMware, I installed a copy of Windows (legally – my laptop had died) in a VM, installed Quicken, and did the backup and restore as Intuit intended. Then I fixed everything. Then – and here’s the big one – I took a snapshot, and burned the whole thing to a DVD. Now, worst case, I can always get back to that point without question – no registration, no stupidity – just copy files and hit go. And I’ve continued to work this VM exactly this way for a couple years now. I’ve traded 5gb of space for the knowledge that no matter what happens, I can always copy the VM from my MacBook Pro to my PC and boot Quicken in case of disaster.

So, when this morning I was told, “hey look, 2009 is out” and knowing they don’t offer upgrade paths beyond 2 years any more, I decided to take a snapshot and jump in, knowing if all went horribly wrong, I’d just rewind, wait 2 weeks, and try again. That kind of assurance is worth the 5 gb of space in my opinion.

So we’re using this thing at work called staf to trigger tests, copy files, collect results, that kind of thing. I mention this because at least my experience with it is marred. This is at least partially because I’m using it on groups of virtual machines that are resumed in the middle of their dhcp leases or after those leases have expired, which sometimes will leave them offline without a little kick in the tenders. Finding the address of a newly available machine is not as trivial as it seems. I did try a couple interesting things to solve this.
Bonjour and other forms of zeroconf are available, but the broken network was not solved. And having so many VMs on the network at a time meant finding unique naming or something.
So identification of pseudoremote machines with unknown or duplicate hostnames is a problem. I eventually settled on using VMware Tools remote API to send over a script that resets networking and returns a text dump of ifconfig or ipconfig. The host parses and I get the new address. I’d rather have something more clean, but this works for now.

I don’t think I need to waste any more brain cells on this one. Three leg shots to a dinosaur are not killing blows. Eek.
Cast: the prof, the psycho, the hero, the token blonde, the government agent, the cute kid, the pet, and disposable dinosaur threat A. Gag.