Monday, March 22, 2010

15 Years of Pobox: The day of the generator

3 comments
Can you believe that Pobox has been around for 15 years already? It seems like just yesterday the website was only one or two pages, and we were still trying to figure out how to keep track of the checks people were mailing us.

How quickly times changed! And yet, there were still plenty of ups and downs ahead. In commemoration of 15 years of Pobox, I asked current and past staffers if there were any stories they'd particularly like to see on the blog. Mark asked to hear the story of the water main break, after he found a CNet article on it.
On Friday customers of POBox.com, an email forwarding service, were baffled by an outage in what they say has otherwise been a highly reliable service. It turned out that the eight-hour shutdown--in which no email was lost--was not caused by a faulty router or programming error, but by the kind of a common building problem that faces "brick and mortar" companies every day: a water main break.

I asked Helen Horstmann, our CEO, if she would tell us about that day, way back in September 1998.
I refer to it as "the day of the generator." Back then, we had a T1 in the office, and virtually all of the important servers were there. I walked up to the building, where a small crowd of people was hanging around, waiting to be told they could go in. A water main had broken under the street. The basement was flooded. The elevators weren't running, because the power was out. WAIT. "The power is out?!"

We waited for about 30 minutes before panicking. Maybe it was a little water main break. Maybe we everything would be fine in just a few minutes. But more and more PECO workers kept showing up, and finally, one of them announced, "You all might as well go home -- the power's going to be out all day."

I was floored. I ran over to him, and said, "All day? What does that mean? Like, 3 or 4 hours?" He said, "Maybe. Maybe 12 hours. It will definitely be fixed by tomorrow morning, though."

I asked if the building was still open, and the desk guy told us we were welcome to walk up, but the building's emergency generator only powered the Exit signs, so the stairwell would not be lit. Using our cell phones as flashlights, we hiked up the nine flights of stairs to the office, where our battery backups were beeping madly. Two people started shutting down the non-critical servers, to reserve as much power as possible for the important ones.

We needed a plan, and fast. We needed our own emergency generator.

I asked if it was possible, any way, any how, name your price, to piggyback off the building generator. But they hadn't had an outage in years, and hadn't been keeping the tank full of gasoline. They figured they had no more than 6 hours of power, and they needed it all for the emergency workers.

The building had sealed windows, so running a generator in the office was also a non-starter... but we had to have power! What if we were down for 12 hours? What if we were down LONGER?!

I found an equipment rental company 30 minutes away that had a generator they were willing to rent for 24 hours. We were all early twentysomething city dwellers, so the only staffer with access to a vehicle was a customer service agent who was also in a band. He and I headed over to his apartment to grab the band van. No drums to schlep today -- we're schlepping a generator!

In the meantime, someone else was getting access to the roof, and estimating the distance from the roof to our office with ethernet cable. Even after we got the generator, we were going to need a 47 foot, 220 volt extension cord (or 15 meters, for our metric friends).

Luckily, the generator guy referred us to an electrical supply company that was on our way back. The cable was on a 3 foot wooden spool (which we later used as a stand-in coffee table, in commemoration of the event). We rolled it out of the store and into the band van.

When we got back to the office, the desk guy took pity on us, and used a fireman's key to turn on the elevator -- I don't even want to think about having to carry those things up 15 flights of stairs! We carried it the last flight to the roof, and started unspooling the power cable.

Did I mention the windows of that office didn't open? We had to drill a hole in the window frame to get the power cable in. Thankfully, we realized that before we attached the plug head to the cable -- drilling the first hole took almost 20 minutes; making it twice as big could have eaten up nearly an hour.

By now, the batteries had long since died, and all the machines were off. It was quieter than the office had been for years. We wired up the power cable, plugged in our main UPS, called up to have the guys on the roof start the generator and ....

Nothing happened.

The battery wasn't charging. We checked everything we could think of. The power inversion on the generator. The cable. The plugs. The UPS. Nothing. Nothing worked. This may have been the point where I went into the stairwell and shouted obscenities.

We started calling electricians. One after another told us they couldn't come until tomorrow. (At this point, it was nearly 3 in the afternoon.) We started calling friends who knew anything about electricity. We knew we were at the end of the line.

How many times did I walk up and down those 9 flights of stairs? How many times did I bug the PECO guys for a new estimate? I know someone tore me away from the building at some point to eat. There was nothing left to do but hope that the power would be back on shortly.

Around 6 PM, we were sitting on the floor when the overhead lights flickered on. Everyone jumped up and started turning computers on. Within 20 minutes, everything and everyone was running -- watching logs to make sure everything was moving as quickly as possible, answering the piles of customer service mail that was coming in.

That was back in the day when Pobox cost $15/year. So, if you're wondering what that extra $5/year is buying you, well, you could say that it's a low cost for a lot of extra peace of mind. :)


Nowadays, of course, Pobox has redundant datacenters in world-class facilities, running on enterprise-grade hardware. Things can still go wrong, but now we're counting on our vendors and partners to solve the problem, not the local electrical supply store. Thanks for sharing that story, Helen!

Do you have a story to share about Pobox? Drop me a line at nessie@pobox.com, and it just might end up on the blog!

Friday, March 12, 2010

How to ask questions to strangers, part 3

1 comments

Mark Dominus concludes his series of articles on how to send email to strangers asking for help. Read his previous entries about picking the right expert and using an informative subject.

In the past couple of articles, I covered the two most important rules of asking a stranger for help. There are also several minor-seeming mistakes you can make that will sink your chances of getting a reply.

Use correct spelling and grammar.

Yes, this is boring and tedious. And yes, I know you have heard it before. And yes, I know that only fussy old people care. But some of those people you are asking for help are as old and as fussy as I am. And some of those fussy old people will throw your message in the trash if it starts with:

i hope that u would be able to help me.

or ends with:

You wouldnt by any chance have any info on B+ & B* Trees would u ???

Maybe in another twenty years we'll write everything in SMS abbreviations. Until then, the fussy old folks like me will still be alive, and the writer of a line like these will appear to be either:
  1. illiterate,
  2. too lazy to press the three extra keys required to get it right, or
  3. A GODDAMN JACKASS.

Excuse me, lost my patience there.

Related to this is that you should have your real name in the "From" line of the message, or at least a name that is not obviously a fake name. Seeing "Smoove B" in the header of a message is not going to help convince a stranger that you are worth emailing.

Don't make the recipient do any work.

Of course, to answer a question, the recipient will have to do the work of answering the question. But your aim is to make them do as little work as possible. For example, I once got a message with the subject "your article" asking me "What is meant by 'forkish'?" Huh? What article? Did I say "forkish"? How should I know what I meant?

I eventually dug up the article, which I had written several years earlier. If I had been a little busier that day, the question would have gone straight to the trash bin.
It would have been a big help if my correspondent had written something like this:

In your article "..." at URL http://... you said "Blah blah blah forkish." What did you mean by "forkish"?

Then I would have known right away what I had meant, or at least I would have been able to look at the article easily.

Experts are busy people. The more work you ask them to do, the less likely you are to get a response. Try to provide all the relevant information that you can. Don't ask them to look stuff up for you in books; say "I didn't understand this thing I looked up in a book. Does it mean X or Y?"

Don't appear impatient

I don't know about other people, but when I get a message that says something like one of these:

Please respond ASAP!

I need the answer right away!

This is very important.


I want to unleash a torrent of sarcasm. "Gosh, I'll be sure to clear my schedule to work on this important project for you!" You're asking a stranger to do a favor for you, so it's not appropriate for you to put conditions on when or how they do it. Here's that "forkish" example in full:

what does forkish mean? this is very important. thank you. please respond.

"This is very important" spoils the effect of "please" and "thank you". It says "please", but it doesn't mean "please". Some time ago, I tried rewriting that message in a form I would have liked better, and came up with:
Could you take a moment to explain what you meant by 'forkish' here?
I didn't notice until after I had written it that I had eliminated "please". But the rewrite communicated "please" anyway, whereas the original one didn't.
That's all the advice I have. Good luck writing to strangers!


Thanks, Mark! I recently took his advice, and wrote to some experts on one of my favorite topics... Pobox! I asked them for tips and tricks that they use that they would like to share with other Pobox customers. If you've got something that you think other customers would be interested in, just drop me an email to pobox@pobox.com with the subject "blog tip".