Mailstore went down. What happened?

What is happening now? (last updated 4 January 1:45 PM EST)

At this time, all Mailstore problems are considered fixed.  New hardware has been deployed, and approximately 75% of users have been migrated.  Additional hardware is still being deployed for the remaining 25% of users.  Onsite and offsite backups are both working properly, and new, faster hardware is expected shortly for onsite backups.

If maintaining the highest degree of access to your new mail is critical, please leave your forwarding address in place for now. The new hardware is still being upgraded, and short downtimes will be scheduled for migrated users in the coming weeks.  More details will be announced as it becomes available.  We do not expect these downtimes to exceed 1 hour, so if an outage of that duration is acceptable to you, you may feel free to remove your backup forwarding at any time.

Due to the nature of this outage (and the cleanup efforts it requires), a credit for 6 months of service has been added to all Mailstore accounts. We are truly, deeply sorry for the inconvenience we know this is causing all of you, and we're particularly mortified that you have lost any mail as the result of a failure on our part.  We appreciate the patience (and kindness) you've shown, and hope we can re-earn your esteem as your email provider.

---

Overview of the Outage

Mailstore was down from 13 December 12:46 PM EST (-0400 GMT) until 14 December 8:13 AM EST, due to a hardware failure.  Much to our horror and dismay, some fraction of mail destined for Mailstore accounts also bounced at approximately 6 PM, with the errors "Relay access denied" or "mail for mailstore.pobox.com loops back to myself".  If your account was among those who had bounced mail, you will receive an email telling you who sent you the mail and when. 

As of 14 December 8:45 PM EST, all backlogged mail has been delivered. A tiny fraction of accounts (24) had corrupted indexes, that prevented them from logging in.  As of 15 December 10:00 AM EST, all 24 had rebuilt indexes, and all their mail delivered.  If you believe you are still missing mail or cannot log in, please contact us to report it.

The message showing any bounced mail went out 13 December around midnight EST.  If you had added a forwarding address by then, you have already received the message.  If you did not have a forwarding address, that message may still be in the backlog of mail to be delivered.  If you did not bounce any mail, you did not receive a message.

All the original updates are included below, for your reference.

---

Report from the system administration staff

Obviously, this is an incredibly horrible, extended outage, and we can only give you an explanation, not by any means an excuse.  And the explanation, in short, is, we got caught with our pants down.  We have been doing behind-the-scenes work on Mailstore for the past few months.  As noted in several of the comments below, Mailstore is both a single point of failure, and one of the harder services to fix quickly, because of the massive amount of data it involves.  So, even something as simple as "add more storage" can become challenging when that requires moving to new hardware instead of adding more drives to existing hardware.  

The ongoing projects for Mailstore over the last several months have been: switch the backend processing software (from Cyrus to Dovecot, for additional features and more stability), add a new storage device (which we did, and has been problematic from the get-go), get a replacement storage device for the new storage device (which has not yet been delivered), have a hot onsite backup, and a cold offsite backup.  But doing anything involving copying, moving or reorganizing Mailstore requires downtime, which we have been trying to minimize.  (I know, and see where it got us?)  So, we have been proceeding slowly, migrating accounts individually, and basically holding off on important things to avoid either slowing down your mail access or incurring extended downtimes.

As the performance of the current (quite new!) Mailstore hardware has degraded, and with the replacement not yet on site, we pushed forward to deal with the problem by planning a brief downtime to fail over some services onto less loaded parts of the system. That was going to happen tonight. This put us in a race with the hardware: we had to get to our maintenance window before it failed, because it was clear the planned means of failover ("Plan B") would not work. Mailstore's current workload would utterly overwhelm the failovers.

Unfortunately, we lost the race. This morning, a series of cascading failures, some seemingly entirely unrelated to the existing problems, including the complete corruption of our backup storage device, brought down the Mailstore service in such a catastrophic way that "Plan C" and "Plan D" for recovery were out of the question. We had to cobble together something from parts of plan E through K, and the result was what you'd expect: a number of false starts and unforeseen problems.

We counted on things staying the way they were, at least for a little while, rather than insisting on a downtime much earlier on to prep for this kind of catastrophe.  And, for that, all we can say is how sorry we are.  We are hoping to be back online very soon, and will continue making updates until we are.

---

What can I do to get my mail?

We recommend adding another forwarding address right away by clicking the "Edit" button to the right of "Your Mailstore Inbox" in the Delivered To column.  Get more detailed instructions on adding a forwarding address.

If you've only used your Mailstore Inbox and aren't sure what a forwarding address is, a forwarding address is an email address at another ISP or provider.  We take mail sent to your Pobox address, and forward it there.  As long as you leave your Mailstore Inbox as one of your other addresses that we deliver to, the mail that we forwarded will also be delivered to Mailstore.

How can I get updates?

We're tweeting updates on the situation as we get them.  You can view them on the web at http://www.twitter.com/pobox (or status.pobox.com), or follow us on twitter @pobox.

What happened?

We had been seeing slowness and errors throughout the day. We had planned an outage for late this evening. But, the problems were growing, and we thought that the problem could be resolved quickly by resetting the storage cluster. We were mistaken.

After we powered down the equipment, it did not come back up.  We have been in touch with the vendor; they currently believe either the power supply backplane or the motherboard needs to be replaced.  We are now working on bringing up the backup hardware.

Bringing up the backup hardware takes a while because there is so much data on Mailstore that needs to be kept in sync.  Unfortunately, this process is somewhat opaque, so it's hard for us to tell how long it's going to take to finish.

Why is it taking so long to bring up the backup hardware?

The backup hardware is underpowered, and we were aware of this. Replacement hardware has been on order, but hasn't yet come in. Unfortunately, this is simply a case of really crappy timing.

What is NOT affected?

Basically, anything besides accessing your mail stored on Mailstore (whether you use your email program, webmail.pobox.com or atmail.pobox.com).  Forwarding, sending mail via SMTP, spam processing (and viewing via the website), and all other website functions should be unaffected.

Update @ 6:26 PM: A number of messages bounced.

As we are bringing boxes up, there was an error.  The IP address for mailstore was picked up by the firewall.  The firewall, not being configured to accept mail, bounced approximately 8,000 messages.  If your mail was among those messages, we will get as much information to you as possible about any lost message. 

I have nothing to say about this, other than I absolutely share the sickening feeling you may be getting.  We pride ourselves on never bouncing or otherwise losing legitimate mail.  I don't even know what to say, except this is the kind of day we have nightmares about.

Update 14 December @ 2:40 AM EST - What is going on?

We are waiting on the remaining slow, clearly degraded hardware to finish making the data available.  Once it is up, it will still be slow.  This weekend, we will be doing everything we can to get off this degraded hardware, once and for all.

In answer to the question, "how did this go so badly wrong?", I can only say that the most horrific words in the English language are: "the backups are completely corrupted."  Rest assured that there will be a complete analysis done of all the myriad ways our existing solutions failed us, in addition to work on the already-planned upgrades.

Update 14 December @ 7:40 AM EST

As this failure stretches on and on, we are working on alternative plans.  Right now, we are setting up a new device, to give you access to the mail that has already come in. However, as seems to often be the case, as you start working on alternatives, the originals near completion.  We hope to have a more definitive report in about 30 minutes.

Update 14 December @ 8:20 AM EST

The storage restore that had been running for the last 12+ hours has finally completed.  Mailstore access has been restored.  It will be slow, and you may still get the password requests we were seeing yesterday will almost certainly still happen.  We will keep you appraised of future steps as we solidify the plans for them.

New feature: push notifications from email filters

We've just added a new feature for Pobox Plus and Mailstore accounts, "push notification on filter match".  The push notification sends a short message to your smart phone, that shows the sender and subject of the matching message.  Here's an example of what you'd see:

All push notifications require an app to receive that notification.  For now, we're offering integration with Prowl.  If you use an alternate app to receive push notifications from third party services, let us know, and we can see about adding integration!

Most email filters stop processing as soon as they match a filter.  For push notifications, we made them work like subject tagging.  So, if a push filter matches, it will notify you, but continue checking your other filters in case you want to redirect or CC it to another person.

In order to keep an overly-aggressive filter from going off all the time, push notifications are limited to one every 5 minutes.  You will need the Prowl app (again, if you're using another service like Pushover or Notify My Android, please let us know) to receive push notifications. Check out our help page on setting up push, and start receiving notifications today!

Get more from Pobox with personal domains!

We're pleased to announce a great change for people who use or want to use personal domains (MyPobox) with their accounts.  Effective immediately, we're eliminating the charge for additional addresses (the addresses formerly known as aliases).  While the number of addresses you can have at Pobox domains is still capped, we've greatly expanded the number of addresses you can specify at your own domain.

The new limits on incoming Pobox addresses, by account type, are:

Pobox Basic: 20 addresses (3 at Pobox domains)
Pobox Plus:   40 addresses (6 at Pobox domains)
Mailstore:    100 addresses (6 at Pobox domains)

The price for using AllMail, which allows you to accept mail sent to any not-specifically-assigned address at your domain, has been reduced to $10 (from $30).

This change also means that we will now be tracking mail volume limits.  Virtually all of our accounts are well under even the smallest of the volume limits.  However, if your account is not, you will be contacted via email.  (Just interested in how much mail you've been receiving?  Check our new Statistics page to see how much mail you receive daily, monthly and yearly.)  The volume limits, averaged monthly, are:

Pobox Basic: 1,000 messages a day
Pobox Plus:   2,000 messages a day
Mailstore:      4,000 messages a day

Why did the Pobox address charge exist for so long? In the past, the alias charge was a stand-in for email volume.  Our billing system was written LONG in the past, and didn't have a way to track metering information.  As you may know, we've been transitioning to a new billing system over the last few months, and this is one of the changes that the transition has enabled.

What happens if you exceed the volume limits? We will never bounce mail for exceeding volume limits.  If you are one of the fewer than 30 accounts whose current usage exceeds the limit for their existing account type, you will receive an email within the next two days informing you, and asking you to upgrade to the account type whose volume you are currently using. Should your usage remain at that level, we will automatically upgrade you to the appropriate account type.


What if I've already paid for additional addresses or AllMail? Good news! You'll see your account expiration date automatically extend as a result. You'll receive an email in the next two days telling you about what's changed on your account.

Does this mean I can invite 19 friends to share my account with me?  No. A single account is still designed for, and intended to be used by, a single person.  While some Pobox Plus and Mailstore account holders use delivery groups to redirect a small amount of mail to another person (I, for example, have addresses we give to my daughter's school, doctor's office, etc. that send copies to my Inbox and my husband's account), the increase in addresses available to an account does not mean they are "shareable."


What if I'm already using more addresses at Pobox domains than my account type permits? Accounts already using between 3 and 9 Pobox-domain addresses (which have always been capped at a maximum of 9 per account) will keep those addresses, at no charge.  Many of you are among our oldest account holders -- thank you for your many years of support, and continuing to choose Pobox as your email solution!

Are you changing the limits on forwarding addresses? No.  No address can forward to more than 5 addresses; no account may have more forwarding addresses than incoming addresses.  Please note that messages that are forwarded to multiple destinations have those messages count multiple times towards their volume limit.  So, if you get 10 pieces of mail forwarded, but you forward to 5 destinations, that counts as 50 messages.

Is there a limit on the number of domains I can have on my account? Accounts are still capped at a maximum of 100 addresses.  Since every domain must have at least one address, 100 domains is the cap.

Is there a limit on the number of Pobox accounts that can use my domain? You may share your domain with as many other Pobox customers as you like.  You may also add as many accounts as you like to your billing group.

We hope you'll find this change a great reason to move or add a personal domain to your Pobox account. As always, if you have any questions about this change, please let us know.

When it comes to spam, less is more.

One of the biggest complaints we get about spam is that there's too much of it to review.  Today, we're announcing a simple change to our default display settings, that we hope will drastically reduce this problem for those of you whose anti-spam level is set to Standard.

While the Pobox Spam section and emailed reports can be customized for your preferred views, the vast majority of you use the defaults we provide.  To reflect how we use the Spam section, and how we think it can be most useful to you, we've changed the defaults. Effective today, we're switching the default view we provide from Held Messages to our Quick Check view.  If you were previously set to go directly to Held Messages on the web, or if your emailed report sent all held messages, you will now see the Quick Check view instead. 

The Quick Check view removes messages caught by our 3 most effective filters.  How effective are they? For our two most accurate filters, customers review more than 10,000 messages they caught to release just one message.  For the third, you review more than 2,000.  Of the messages released, header reviews indicate the vast majority are actually spam or suspected phishing, which customers choose to release to themselves for their own purposes.

These 3 highly accurate filters catch more than 88% of the spam we handle each day.  In house, Pobox staffers nearly always choose to use Aggressive filtering, because it bounces messages flagged by these filters, and drastically reduces the amount of mail to review.  Reviewing thousands of pieces of spam for the extremely unlikely possibility that one legitimate piece of mail might be there is simply not effective. Not only do you waste all that time reviewing spam, but it ends up burying mail caught by our less accurate filters. Reviewing messages that have a 1 in 400 chance that they are legitimate makes much more sense than wading through messages whose chances are 1 in 10,000 (or much, much less.)  

Please note: this change has not modified what we catch for you (your anti-spam level) in any way.  It only changes the default setting of what we are asking you to review.  If your view was set, on the web or via email, to Bounced or For Review, your settings have not been changed.  If your anti-spam level is Aggressive, this change will not alter your view.  Aggressive bounces these extremely accurate filters, so they are already part of your Bounced view, not your Held view.

For some of you, part of the peace of mind that Pobox provides is knowing that you can easily check all the mail we've blocked for your account.  If you would prefer to continue reviewing all spam we hold for your account, just switch the view we send back to Held Messages.  (If you want to try the new view out for a few days first, the "Emailed Reports Settings" button at the bottom of every report takes you to the settings for the view included in your report.) To switch the default view when reviewing messages on the web, just select "Held Messages" in the top right corner under Spam Views.

Search in the spam section also checks every section and every view, and always has.

We believe this small change will dramatically reduce the amount of spam you have to review, without impacting the accuracy of your results in any way.  We welcome your continued feedback on any way we can make Pobox better and easier for you to use.  Let us know if you think this has helped!

Updated (9/24/12): Text has been modified to clarify that this change modifies the display for people whose anti-spam level is set to Standard.  Users using Aggressive or higher will not see a change.

Pobox Best Practices: Update your password.

There have been a lot of changes going on at Pobox in the last few months!  Some are apparent immediately, but others, you can't see. Today, one of those invisible changes is in place... but we need your help to see the benefit.

As times change, so do the best practices for common tasks like password management.  We've made a big update today, to bring our policies in line with those best practices.  We've updated our encryption algorithm to use bcrypt, and to extend the length of both they key and the unique salt we use to encrypt your passwords.  We've dramatically increased the maximum password length, up to 500 characters.  We've removed most of our restrictions on what special characters you can use.  We've also made it easier for us to switch to newer, stronger encryption methods as they become available.

To take advantage of all these new changes, you need to log into the website and update your password.  Even if you use the same password again, you'll get the benefit of our strengthened encryption standards.  But, to entice you to choose something longer and harder to guess, we've added a tool to the website to tell you, in time, how long it would take for a computer to guess your password. 

As you've probably noticed from previous blog posts, we think account security is incredibly important.  And because so many websites send your password reset requests to your email, the security of your email account is especially important.  Please take a few moments to update your password today.