(Or, “In Which Your Narrator Learns a Valuable Lesson About The Perils of Cheap Media and the Proper Use of the Verify Button.”)

A while back I picked up a 50-pack ‘cake box’ of surprisingly low-priced DVD+R Dual Layer discs, possibly at Costco (although it could have been Best Buy or Staples, at this point I’m not sure). The price was good and I’d been thinking that it’d be nice to be able to cram more than an hour’s worth of DVD-Video onto a disc, and I naively had some idea about using them for backup purposes as well.

The discs were TDK branded, and say “DVD+R Double Layer 8x Speed 8.5GB”. Unfortunately, what the label doesn’t say is that, as far as I can tell, the only relationship they have with actual recordable DVDs is that they’re roughly the same size and they have a hole in the middle. But more on that later.

Anyway, I bought the discs, brought them home, and promptly forgot about them. I don’t burn a ton of discs anymore, but it’s good to have media on hand…right? So there they sat, lurking.

A few months ago, I went to burn a fairly big photography project to disc for safekeeping. Of course I had it backed up to a second hard drive (via Aperture’s ‘Vaults’ feature), but I figured an additional copy on optical wouldn’t hurt. And so I clicked away in Toast and burned the files to one of the discs.

Now I generally make a point of always letting Toast verify any disc I burn, but I’d be a liar if I told you that I actually remember doing it. It’s possible that I just mounted the disc immediately after burning, cataloged it, and then ejected it and put it with my other backup discs. Rinse, repeat … about a half a dozen times or so, over the space of a few months. Each time the same process — I burn a disc without problems, mount it, catalogue it, and then put it away in a binder. The process wasn’t flawless — sometimes I’d get failures due to ‘sense key’ errors and have to try a couple of times to get a successful write, and writing 8.5GB at the maximum speed of 2X that my writer supported was slow — but I figured that was par for the course.

Up until earlier this week, when I did something different. I burned a bootable disc image (long story) and attempted to use it on another computer. It failed. In fact, the disc wouldn’t even mount when I put it in the other computer’s drive. Thinking it was a compatibility issue, I brought it back to my desktop where I’d burned it — nope, nothing. Odd, I thought — so I tried it again, and got the same result.

Although the disc had mounted just fine immediately after burning, once ejected from the drive and reinserted, it would never mount again.

Not only was the boot disc bad, but every disc I’d burned on the media turned out to be bad. That’s right: one hundred percent mis-burns.

In the interest of science, I blew through most of the rest of the stack trying different burning programs, data, and writers. I achieved identical results with my desktop’s Pioneer DVR-109 and my laptop’s LG GSA-S10N6, using both Toast and Apple’s Disk Utility. No dice under any combination. (On the LG, the discs seem to fail more reliably during burning; on the Pioneer they sometimes complete and fail silently.) However, the error does seem to be reliably caught if full post-burn verification is run on the resulting disc.

So, lessons learned:

1. Always run a full verification cycle on every disc, every time. I got sloppy, and as a result let a few bad discs slip through into my backups. No harm came of it, and they’re a belt-and-suspenders thing anyway, but it’s not hard to imagine how it could have been a nastier surprise. If it’s worth burning, it’s worth verifying.
2. Brand names on optical media are totally meaningless. I’ve discovered the crummy “RITEKS04” stuff under both the TDK and Memorex names. It seems to be the case that if you’re getting a ‘good deal’ on double layer DVD+R 8X media, what you’re getting is probably the RITEK crap. Avoid it like the plague it is.
3. Allegedly, the only decent DVD+R DL media around is Verbatim. I wasn’t able to find any in any local stores in my area, although it is available mail-order. In general, the whole double layer system seems half-baked and worth avoiding unless you really need it.
4. DVD burning isn’t like CD burning. Or at least not like CD burning in the last ten years. I was cavalier about burning DVDs because I mostly burn CDs, and CD writing technology has been pretty reliable — even with cheap non-mail-order media — for the better part of a decade now. DVD burning, especially double layer, isn’t like that; it’s like 1999 all over again with the weird incompatibilities.

It’s entirely possible that there are burners out there that handle the steaming pile that is RITEKS04 just fine, but both of my drives are common models (both are stock Apple parts) and work fine with better grades of Verbatim media.

So much for a good deal on a box of discs.

According to the powers that be at Flickr, you have until June 1, 2010, 12:00 PM PDT to get your historical referrer-log data, if you are a Pro member and are interested. You can download it as CSVs from the bottom of your stats page.

Apparently it is “not sustainable” to keep the data available forever. I suspect this translates to ‘it’s really expensive and only 0.001% of our users actually care or are even aware of it.’ In the future, they will be providing access to 28 days worth of data via the API, but probably nothing beyond that.

I wasn’t even aware that this feature existed until I saw the notice that it was going away, and although their reasons for terminating it are understandable, it is surprisingly interesting data. My strong recommendation is that, if you’re a Pro member and you have a few megabytes of disk and transfer to spare, you might as well take thirty seconds and download them.

Note to anyone thinking of being clever and using curl or wget to batch-download all the files at once: don’t bother, it’s really not worth your time. (Trust me on this, I looked into it.) You’d have to authenticate using Flickr’s API and it’s guaranteed to take longer than just pointing your browser’s download-destination folder to some appropriate place and Alt-clicking on them.

If you download your logs as CSV files — and really why wouldn’t you? Excel gives you no advantage here — you can use this small Python script I wrote to dump them into a SQLite database. The script requires Python 2.5 or later (or possibly an older version with the appropriate PySQLite add-on package, but I haven’t tested that and probably won’t). Bug reports and enhancements are welcomed although it’s not meant to be pretty, since it won’t be of much use to anyone after June 1. The schema should be obvious just from looking at the script; it’s two tables — one for daily and the other for weekly data — and the columns in the CSVs are all carried over into the DB. There’s no date conversion or other fancy stuff.

What you do with it once you get it there is your business; I haven’t really decided what, if anything, to do with it, but it seemed like having everything in a couple of DB tables was a lot more convenient, whatever I might decide to do with it, than having it in dozens of CSVs. (Do keep the CSVs though. They’re small and there’s no good reason not to.) If you have any suggestions of interesting things to do with or ways to analyze the data, let me know by SDF email or in the comments.

Maybe once they get the API access set up, I’ll write something to grab new stats and shove them into the same SQLite DB on top of the existing records. It would only have to run once a month or so to stay on top of the feed, which isn’t that bad.

So I finally got around to taking that trip to Yellowstone that I was talking about back in May. Rather than going in late August 2009, as I had planned, I actually ended up going last month (that’d be March, 2010). For those of you not familiar with weather in the Northern Hemisphere, that meant going in what amounts to the dead of winter, instead of the height of summer — and as far as I’m concerned, it was the best thing that could have happened to the trip.

While we didn’t have the Park to ourselves, exactly, we were much closer to it than we would have been in August. And wildlife was much easier to spot as well. Although I didn’t snag a shot of one of Yellowstone’s coveted wolves, I did get some nice images of the local bison, coyotes, birds, etc. All in all, a great trip, and I highly recommend a winter excursion to the park for anyone who hasn’t done it already. You won’t be disappointed.

But that’s not my purpose here. In my earlier post I made soem guesses about what I thought my photography habit was going to set me back, in terms of consumables (in the form of storage), for the trip. In that post I had guessed that I’d fill two 4GB cards, which I thought would be fine for my relatively low-resolution (by 2010 standards) DSLR.

As it turned out, I was a little low.

Over the course of nine days, I took a total of 2,299 frames, equivalent to about 64 rolls of 135, and consuming just under 20GB. Just sorting through then and making a ‘first cut’ is a project in itself.

Part of the reason I ended up taking so many frames (and I’m using the word “frames” rather than “images” carefully here) is because I had brought my laptop along with me on the trip and as a result knew I didn’t have any reason to conserve storage space. The limiting factor on my shooting wasn’t storage, but instead camera batteries. With a 4GB and 8GB card, I could easily shoot all day and then dump the contents to my laptop at night for storage and immediate backup to a DVD.

This led to an immediate change in my shooting style that I never would have made, if I’d been shooting film or even using the digital without the hundreds of gigabytes of disk storage that the laptop represented: I turned on three-frame, +/-0.5 EV bracket mode on the first day and never turned it off. That’s something I’ve never felt rich enough to do on my film Maxxum.

So those 2299 frames really represent something like 800 images (I did take a few without bracket mode, so that’s a low estimate), much closer to my initial estimate of 20 or so 35mm rolls worth. It’s just that, rather than only having one frame for each image, with the digital I have two “insurance” frames, in case my judgement of the light was a bit off or I just decide that slightly lighter or darker is preferable. Although not earth-shattering by any means, I do think the bracketing saved a few marginal images that otherwise would have been garbage if I’d only had the “center” one. And that’s enough to make me a pretty happy photographer.

I was flipping though the channels on TV earlier and came across a new addition to the local lineup — something called The Research Channel. Apparently it broadcasts recordings of presentations by various notable people on a variety of subjects. The recording that caught my eye was Behind the Code with Jim Gray. Gray, at the time of the interview (2005) of Microsoft Research but formerly of IBM, Tandem, and DEC, had some interesting comments about databases, parallel processing, and the future of hardware.

At one point (about two thirds of the way through the video), he describes future processors as probably being “smoking, hairy golfballs.” The ‘smoking’ part is because they’ll be hot, consuming and dissipating large amounts of power in order to run at high clock speeds; hairy because they’ll need as many I/O pins as possible, on all sides; golfballs, because that’s about the maximum size you can achieve before, at very fast clock speeds, you start to run into the “event horizon” (in his words) of the speed of light and lose the ability to propagate information from one side of the processor to the other in one clock cycle.

He didn’t give a timeline on this prediction so I’m not sure it’s fair to call it either correct or incorrect just yet, but it’s interesting. The ‘smoking’ part actually seems to have gone in the opposite direction since 2005; power dissipation has gone down from the highs of the Pentium IV and IBM G5, but it’s possible it could creep back up again if something stops the current trend. He seems to have been right, at least in a limited sense, about ‘hairy’: a look at new processor sockets shows a definite upward trend, with Intel’s newest at more than 1500 pins — common sockets in 2005 would have had less than half that. They’re still all on the bottom of the package, though. The ‘golf ball’ maximum on size is more theoretical, but I don’t think anything has happened recently that provides cause to dismiss it.

After watching the segment, I pulled up the Wikipedia page on Gray, curious to see what he was up to today. Unfortunately, it was at that point when I remembered why his name seemed so familiar: he disappeared while solo sailing off the coast near San Francisco, and despite a massive crowdsourced search effort, he was never found. An sad and unfortunate end for a very interesting guy.

Related:

• A Tribute to Jim Gray from the ACM Queue
• Interview with Jim Gray by Netcraft, on Gray’s work on the Microsoft TerraServer
• Slashdot on Gray’s disappearance
• Slashdot on the search efforts

I really like Yelp, which is probably why I’ve bothered to spend time typing up reviews for it, despite it being a commercial service that could theoretically pull a CDDB at any time. I’ve found a lot of neat little restaurants that I wouldn’t have otherwise found, particularly while traveling, via Yelp, and in general have found the ratings and reviews there to be of very high quality.

However, I’ve noticed that as Yelp’s userbase has grown and expanded beyond the computer-savvy foodie demographic that seemed to have been some of its first users, the average ratings for a particular business are no longer as useful as they once were. It used to be, if a restaurant had five stars and more than a handful of ratings, it was almost certainly phenomenal. Similarly, if a place was languishing at one or two stars, it was probably best avoided — after all, if a place is bad enough to actually get someone (who isn’t being paid) to spend the time to write a negative review, something must be pretty wrong. And if something was in the middle, chances are it was pretty much just average for whatever cuisine it was trying to represent.

Lately, though, I’ve noticed that many places — and this is especially true of eclectic or “acquired taste” restaurants — are getting pushed towards middling reviews not because anyone is actually rating them that way, but because very good and very bad reviews are being averaged out into two or three stars. This isn’t really surprising: reviewing restaurants is a “matter of taste” practically by definition. But that doesn’t make the result very useful. When I’m looking down the search results in Yelp, I want to know what I am likely to enjoy, not what some hypothetical “average user” is going to like. (I’m not the first to notice this problem, either.)

As more and more users join Yelp and start writing reviews, the average review will naturally start to approach what you’d get from reading the AAA guide, or any other travel or dining guide aimed at a general audience. That’s not necessarily bad, and when you’re writing a travel book or dining guide it’s pretty much exactly what you want: try to give an idea of what most people will think of a particular restaurant.

But that’s certainly not the best that an interactive system can do, not by a long shot. The benefit of a website, as opposed to a book, is that the website doesn’t necessarily have to show the same exact thing to everyone. This is why the front page of Netflix is more useful than the top-ten display down at your local Blockbuster, or why Amazon’s recommendations are typically more interesting than whatever happens to be on the aisle-end display at Borders. It’s not that Blockbuster or Borders aren’t trying — they’re doing the best they can to please everyone. The beauty of a dynamic website is that you don’t have to try to please everyone with the same content; you can produce the content in a way that’s maximally useful to each user.

If Yelp took this approach, ratings from users who tend to like the same things that I do would be weighted more heavily when computing an establishment’s overall score; if you brough up the same restaurant (or if it came up in your search results, more importantly), it might have a different score, if your preferences — as expressed via your reviews — are significantly different than mine. This makes perfect sense, and provided that there’s still some way to see what the overall, unweighted average review was (Netflix shows it in small print below the weighted-average), it’s a no-lose situation from the user’s perspective.

I’m sure that Yelp’s engineers are aware of the Netflix model and how it could be applied to Yelp, so this isn’t a suggestion so much as a hope that it’ll get implemented someday.

Earlier today I read a blog entry by Ben Metcalfe that really hit home. The entry is called “My GMail password scares me with its power,” and I’d like to say that he’s not the only one. Particularly in light of the widespread (and apparently quite successful) phishing attacks going around, it’s a good idea to think about how much of your life and personal information are stored behind that one password, and whether that password is really up to snuff.

Metcalf puts forward what I think is a very modest proposal, which I think boils down to two main points. Neither are trivial, but neither are either one a real stretch on technical grounds:

1. Google ought to allow you to enforce some sort of privilege separation: rather than just having one password for everything, more sensitive services (GMail, Google Checkout, Search History) should be able to be configured to use a separate password. This would ensure that the cached password saved in the chat program you use at work couldn’t be used to log into your mail, or make purchases to the credit card associated with your Google Checkout account.

2. Users who are security-conscious could buy a two-factor authentication token, like an RSA SecureID, to use with some or all Google services. This wouldn’t be mandatory and it wouldn’t be free — so it wouldn’t help the clueless or the broke — but it would let those people who are honestly concerned about security but who lack the ability to replicate Google’s services themselves (and, lets face it, just about nobody can replicate Google’s services at this point) to get that security on top of Google’s offerings.

Perhaps neither are economically feasible right now; too few users may care about security—and be willing to pay for it—to cover the cost that either would mean to Google to implement. But as users put more and more of their data in the hands of managed services like Google’s, and security breaches start having more serious consequences, the demand will come.

In the meantime, what’s a concerned user to do? The best thing you can do is to choose a more secure password. If you don’t mind potentially creating something that you can’t memorize, use a random-password generator and either write the results down, or store it in a ‘password keeper’ program that encrypts its data file with one (good!) master password. I take this latter approach, and use the open-source Password Safe on Windows and Linux, and Password Gorilla (which opens Password Safe database files) on Mac OS X. And, of course, take all the usual precautions against potential phishing attacks.

Until Google sees fit to improve on the one-username/one-password architecture for all its services, that’s about the best you can do.

Earlier today my copy of Quicken 2006 for Mac began refusing to download transaction activity from any of my bank or credit card accounts, complaining about an “OL-249” error. It took a bit of Googling to figure out what was going on, so I thought I’d post the solution here.

Short version: you need to download this fairly obscure patch from the Quicken website and install it. You should do this after updating via the regular File/Check for Updates option, and it is in addition to the updates provided via that route.

Longer explanation: from what I can tell, the certificates included with Quicken 2005, 2006 (which I use), and 2007 had relatively short expiration dates. They expired, and for some reason either weren’t or couldn’t be updated via the built-in update mechanism. Hence the additional patch. Why they couldn’t do this via a regular update push I’m not sure, but at least they made them available somehow — I would have half expected them to just tell everyone to upgrade.

Once I ran the installer against Quicken.app, online transaction downloading worked fine once again.

There’s been a bit of discussion recently over the idea of mileage-based road taxes replacing the Federal gasoline and diesel taxes that currently pay for the Interstate system, among other things. Most articles seem to have been prompted by a report from the “National Surface Transportation Infrastructure Financing Commission” (which somewhat strangely has a photo of the DC Metro in an underground station on its homepage) suggesting that the gas tax be phased out by 2020 and replaced by a mileage-based tax.

The proposal by the NSTIFC called for a GPS-based system to track road usage and upload it on a monthly basis for taxation purposes. This is stupid. It’s overly complex, it would be ridiculously expensive, it has major privacy concerns, its operation would be opaque to users, and it would almost certainly be open to abuse due to its complexity. It’s a terrible idea and the people suggesting it should be forced to read the RFPs of every overly-complex public sector IT project that has fallen flat on its face for similar reasons, until they repent for coming up with such a terrible idea.

However, the stupidity of that particular implementation plan doesn’t mean that all mileage-based taxes are a bad idea. The underlying concept is a sound one, and if it’s done right it might cause people to think harder about the services they’re using and how much it costs to maintain them. That’s a Good Thing in my book.

The kind of mileage-based tax I’d support would be a low-tech one. Calculate taxable mileage using annual odometer readings, conducted while vehicles are undergoing normal safety or emissions inspections. (There are states which currently don’t do emissions or safety inspections that would have to start doing them, but this change is far less than what would be required for alternative schemes, e.g. the GPS-based one.) Certainly it’s possible to roll back an odometer or bribe an inspector, but those things are already illegal, as are other kinds of tax fraud. Increase the penalties proportional to the increased incentive to commit fraud, and we shouldn’t have much more trouble with odometer tampering than we currently do.

Basing the taxable mileage from the odometer reading doesn’t require invasive GPS tracking devices, which would doubtless be used for purposes well beyond tracking taxable mileage once installed. It doesn’t require any new technology, and in many places it makes use of the already-extant inspection infrastructure. It’s cheap — both from the user’s and the government’s perspective — and it would work.

Two of the most frequently-cited concerns regarding mileage-based taxes concern drivers who frequently travel outside the US, and drivers who spend most of their time on private roads (e.g. farm vehicles). The second issue — vehicles on private roads — is easy to address: if your vehicle doesn’t have a license plate and doesn’t normally operate on public roads (vehicles which today use untaxed off-road fuel), it doesn’t get taxed. If your vehicle does have a plate, it does. In the very worst case, this might force a very small number of edge-case users to get a second vehicle, if they currently have one that sometimes operates on-road using taxed fuel and sometimes off-road using untaxed gas, but this is such a small percentage of vehicles that I’m not sure it bears building policy around.

Addressing international driving is a more interesting question. The simplest, lowest-tech solution is probably to simply record mileage as part of the border-crossing process. If drivers who are crossing the border want a tax exemption on their non-US mileage, they could carry a logbook similar to a passport, specific to their vehicle, and have the mileage noted and certified by a Border Patrol agent as they crossed out of and back into the US. It would be up to drivers to determine whether, based on the amount of mileage they actually drive outside of the US, the paperwork was worth it.

What gets lost in the discussion of mileage based taxes, and which I think bears attention, is that in any reasonably fair scenario, the taxes on passenger cars and light trucks should be vanishingly low. The bulk of mileage taxes should be placed on commercial vehicles weighing more than 6000 lbs., because they actually cause wear and tear to the roads. Passenger cars essentially don’t. Whenever you see an Interstate being repaved, it’s generally either due to weather deterioration, or wear and tear by trucks. The weather-repair costs should be borne by all drivers essentially in proportion to the amount they drive, but the wear and tear expenses should be squarely on heavy vehicles. In fact, the easiest way to ensure vehicles pay for the damage they do is to base the tax on the milage driven multiplied by the maximum axial weight of the vehicle. (Road wear is essentially proportional to the load on each axle, although I suspect the relationship is strongly nonlinear and some research might be required to determine the actual rate tables.)

And that brings me around to my only real objection to a mileage-based tax, which is also my objection to virtually all taxes except those placed on real property: the public needs an assurance from the government that the mileage tax would only be used for maintenance and construction of the transportation infrastructure, and not for whatever purpose Congress decides is politically expedient this season. This is because, when you start taxing a particular activity, you start to change the underlying incentive structure that drives people’s choices and lives. It is important to make the ‘retail’ cost (that is, the out-of-pocket cost paid by the consumer) of goods reflect the true cost to society of that good, but it shouldn’t be made any higher.

Federal road taxes should be used for the maintenance of the Federal road and highway system only — not for regional light rail projects (better funded by property taxes on those areas that will benefit) or for environmental remediation of fossil fuels (better funded by taxes on the fossil fuels themselves). And certainly not for schools, hospitals, police stations, or anything else, except insofar as the need for those things can be directly attributed to the existence of the Federally-maintained road network.

Some have objected to the idea of a mileage-based tax because under most proposals, it would not immediately replace the gas tax — that is, the gas tax would not drop to zero cents per gallon on the day a mileage-based tax went into effect. If both the mileage-based road usage tax and the gas tax were set properly, this would not be a problem. The mileage based tax would go towards infrastructure maintenance, and the gas tax would go towards remediating the environmental consequences and other negative externalities of petroleum use. Since there are a lot of negatives associated with burning oil, it should have a fairly high tax regardless of what we decide to levy for road use. Furthermore, the remaining gas tax should apply across the board to all petroleum products intended for combustion, not just road fuels: this means oil used in power generation, on farms, or by railroads shouldn’t be exempt. If you burn it and vent the byproducts into the atmosphere, it should be taxed: it’s not a “road usage” tax anymore, it’s a “petroleum combustion” one. (Here’s where you build your CO2 or climate-change taxes, incidentally.)

Retaining — even increasing, if valid reasons exist — the tax on gasoline to cover its negative externalities also eliminates one other problem with a mileage-based tax: that it creates a perverse incentive to continue using petroleum vehicles and not switch to alternative fuels, which are cleaner and have fewer negative externalities associated with their use. Plus, as a bonus, if we institute a mileage-based tax with a weight component, we can stop punitively taxing diesel fuel as a backdoor way of taxing trucks for the damage they do to the roads. Diesels are more efficient and are favored in other parts of the world (where the tax regimes are less punitive) due to their inherent economy.

There are lots of reasons to hate the mileage-based taxation proposals that have been put forward, and would require GPS receivers and constant monitoring of every car on the road. However, there’s no reason to dismiss the idea of mileage-based taxes out of hand. Taxing based on services actually consumed is always a good idea in my book, and if it were done right, a mileage-based tax could help shape our actions in ways that avoid externalizing costs on others. However, I remain as cynical as ever about Washington’s ability to get this, or just about anything else, right.

I discovered earlier today, while trying to load my personal X.509/SSL certificates onto my trusty Nokia E61i, that its personal-certificate support is for all intents and purposes intentionally broken.

When the user certificate is imported to Nokia Eseries devices, e.g to be used for authentication with WLAN connections, the certificate contents are checked and if there’s any issues with the certificate fields, the importing of the certificate will fail and the error message “Private key corrupted” is shown.

One common situation where this problem may occur is if the KeyUsage field in the certificate has the nonRepudiation bit enabled. A certificate with nonRepudiation bit is rejected by E60, E61, E61i, E65 and E70 devices because of the security reasons.

The workaround is to create a new user certificate where the nonRepudation bit is removed. The nonRepudiation bit is not necessary when doing a certificate based authentication e.g. in WLAN environments.

Yes, broken. I don’t care what the Nokia engineers were thinking when they put that “feature” in, it sucks. It basically stops you from using 99.9% of all certificates in the world — ones produced using the default settings from most issuers, and forces you to generate a brand new certificate for the device. That is totally unacceptable. Certificates cost money in many cases, and even if they don’t, they take time to create. Plus, managing multiple per-device (rather than per-user) certificates is a royal pain in the ass.

Nokia’s “solution” to the problem would force me to generate a brand-new certificate for my mobile, and then I’d need to replace the certificates stored on every other device with the new one, in order to make sure I can decrypt S/MIME email. If I didn’t do this — if I used one certificate on the E61 and another on my desktop and laptop, the E61 wouldn’t be able to open encrypted email sent in response to messages originating from the other machines.

(This is all assuming the E61i can even do S/MIME, which I’m not 100% sure of; but since it can’t load my certificate, it’s a bit of a moot point.)

Hardcore failure on Nokia’s part. Security, no matter how well-meaning, is worse than useless if it breaks functionality or makes the user’s life this difficult. All it does is raise the ‘cost’ of security, and make it more tempting to forgo things like certificate-based authentication at all.

Up until recently I’ve been pretty happy with the E61i, but I’m feeling more and more that they just didn’t take core functionality seriously enough. The software is flaky and unreliable, as is the Bluetooth stack (I get lockups about once a week when I’m using it with a BT headset or tethered to a laptop), and I question whether anyone who worked on the built-in email client actually used it. (When you have an entire QWERTY keyboard to work with, why does every action require at least 3 clicks on a miniature D-pad?) It does have its charms — JoikuSpot is amazingly useful, and I love not being locked into an iPhone-style “App Store” — but the warm fuzzies are really wearing off.

It’s starting to become clear to me why BlackBerry, and not Nokia, is so dominant among users who actually care about communication over everything else. (BlackBerry offers both S/MIME and PGP, although it seems like it may need to be deployed to an Enterprise Server rather than to individual handhelds.) It’s just unfortunate that the BlackBerry offloads so much intelligence to the BES/BIS backend; I’m not really comfortable being that tied-in to somebody else’s infrastructure, and I don’t really feel like running my own BES.

Maybe it’s time to take a look at Palm.

Calculated Risk, one of my favorite finance and economics blogs, has a great article written by the late Tanta on The Psychology of Short Sales. The piece really hit home for me, because during my recent search for new quarters, I ended up drooling over a lot of short sale listings, only to be warned by my agent that they often take a very long time to execute and frequently fall through. I quickly cooled to the concept.

On paper, short sales are the ultimate win/win for an upside-down homeowner who wants to “walk away,” and a lender who wants to minimize their loss. It lets both parties avoid foreclosure, prevents a house from sitting empty and potentially becoming a target for vandalism, squatters, and generally a source of neighborhood blight, and lets the homeowner remain on the property and leave gracefully when it sells. Plus, potential buyers get a house that hasn’t been trashed by a bitter ex-owner, or had its pipes freeze and burst due to over-winter neglect. Triple win, right?

That’s on paper. In practice, things often don’t work out so well. Because of the way short sales work, there’s often a disconnect between what the various parties involved in the deal think the property is worth. If they can’t reconcile their views, there’s no sale and the property goes back on the market, and eventually to foreclosure.

The biggest difference between a short sale and a traditional bank-owned post-foreclosure property (a “REO”, or “real estate owned”) is that in the latter case, the bank has already taken possession of the property, probably had it assessed, and accepted that they’re going to take a non-negligible loss. It’s just a non-performing asset sitting on their books at this point, one that they’d presumably like to unload at the earliest possible opportunity at an acceptable price. Contrast this to a short sale: the bank has just learned that the current homeowner can’t make their payments and wants out, and has responded by telling them to get a listing agent and put it on the market. They haven’t really written anything down yet. The big loss is still to come.

To a buyer, a short sale property ought to be more attractive than a REO, because it hasn’t been sitting vacant or gotten trashed during a ugly eviction. However, buyers quickly learn to beware these six words in any listing: “offers subject to third-party approval.”

When a buyer makes an offer on a REO, the offer goes to the bank and they get a pretty straightforward thumbs-up or thumbs-down. Either the offer is acceptable and it sells, or it isn’t and the bank is content to let it stay on the market a bit longer. Since the bank already owns the house, they just want to get the most for it they can.

When an offer is made on a short-sale property, it gets forwarded by the listing agent to the bank, who has the choice of whether to accept it or not. If they accept it, they’re almost certainly taking a loss and accepting a writedown on the original mortgage. There is a big psychological difference between this and the REO case, it seems to me: in a REO situation, the bank is trying to recoup as much as it can of an already-realized loss; in a short-sale, the bank is actually taking the loss as part of accepting the offer.

This psychological difference seems to manifest itself in the relative speed with which banks process the two different types of offers. REO offers get decisions rendered quickly; short sale offers can take months to process, during which both the buyer and seller live in uncertainty. This uncertainty causes buyers to make fewer offers on short sales than on REOs, and to offer less for short sales than they might otherwise. In theory there’s no reason why short sales should sell for much below what a regular owner/buyer sale would, but in practice they go for something closer to REO prices. This difference is, to my eyes anyway, almost completely due to the perceived arduousness of the short sale process.

In addition, there’s often a failure on the part of buyers and lenders to understand how the short sale benefits the other party, and how this affects the price they’re willing to accept. This is what Tanta explores in the Calculated Risk article. Lenders are only interested in a short sale if it results in a price that’s significantly (more than 40%) greater than what the property would fetch as a REO, post-foreclosure. Buyers, on the other hand, often try to bid less than what the property would fetch as a REO, on the assumption that the lender ought to be willing to take a little less on a short sale than they would as a REO, since they’re avoiding going through foreclosure. Hence, no deal.

In order to make short sales a more viable option for distressed homeowners who find themselves upside-down on their mortgages and unable to pay for them (or who simply want out and can’t sell normally and cover the mortgage), I can think of several things that need to happen:

• Banks and other lenders need to assign more staff to “special assets” and other pre-foreclosure divisions, and realize that they can avoid needless trouble and expense by going the short-sale route versus foreclosure. They need to gear these divisions to providing fast up-or-down decisions on short sale offers, and empower employees to write down assets significantly (at least as much as the delta between the REO price and the loan face value) in order to make deals happen quickly.

• Prospective buyers need to be better-educated about how short sales work, not only from their own perspective, but also from the owner’s and lender’s. They need to understand why a lowball, sub-REO offer isn’t going to fly with the lender. For a short sale to work, three parties — the owner, the buyer, and the lender — need to feel like they’re making out better than they would have via foreclosure. Offering substantially less than a property would fetch as a REO doesn’t allow that to happen.

• Homeowners considering a short-sale, whether in financial distress or not, need to be better about selling their properties. They need to work hard to make it clear to buyers that they’re not selling a REO, and that the property is inhabited and well-maintained. If a house looks like a foreclosure property, it’s going to get offers that reflect that, and it will almost certainly end up as a foreclosure property eventually. I saw several short-sale properties during my recent search that were frankly worse than the average REO, and that just isn’t going to work.

As it turned out, I didn’t make any offers on the short sale properties that I looked at. Given the time available before I have to be out of my current rental, it just doesn’t make sense. And I definitely wasn’t alone: many short sale properties had been on the market for hundreds of days, while REOs are being snapped up almost daily by hungry buyers armed with low rate pre-approval letters.

Making the reality of short sales better match the concept would provide affordable homeownership to many buyers, a dignified ‘out’ for distressed owners, and smaller losses to lenders and their investors. But a lot has to happen before that will be the case.