Thursday, December 13, 2012

Mailstore went down. What happened?

What is happening now? (last updated 4 January 1:45 PM EST)

At this time, all Mailstore problems are considered fixed.  New hardware has been deployed, and approximately 75% of users have been migrated.  Additional hardware is still being deployed for the remaining 25% of users.  Onsite and offsite backups are both working properly, and new, faster hardware is expected shortly for onsite backups.

If maintaining the highest degree of access to your new mail is critical, please leave your forwarding address in place for now. The new hardware is still being upgraded, and short downtimes will be scheduled for migrated users in the coming weeks.  More details will be announced as it becomes available.  We do not expect these downtimes to exceed 1 hour, so if an outage of that duration is acceptable to you, you may feel free to remove your backup forwarding at any time.

Due to the nature of this outage (and the cleanup efforts it requires), a credit for 6 months of service has been added to all Mailstore accounts. We are truly, deeply sorry for the inconvenience we know this is causing all of you, and we're particularly mortified that you have lost any mail as the result of a failure on our part.  We appreciate the patience (and kindness) you've shown, and hope we can re-earn your esteem as your email provider.

Overview of the Outage

Mailstore was down from 13 December 12:46 PM EST (-0400 GMT) until 14 December 8:13 AM EST, due to a hardware failure.  Much to our horror and dismay, some fraction of mail destined for Mailstore accounts also bounced at approximately 6 PM, with the errors "Relay access denied" or "mail for mailstore.pobox.com loops back to myself".  If your account was among those who had bounced mail, you will receive an email telling you who sent you the mail and when. 

As of 14 December 8:45 PM EST, all backlogged mail has been delivered. A tiny fraction of accounts (24) had corrupted indexes, that prevented them from logging in.  As of 15 December 10:00 AM EST, all 24 had rebuilt indexes, and all their mail delivered.  If you believe you are still missing mail or cannot log in, please contact us to report it.

The message showing any bounced mail went out 13 December around midnight EST.  If you had added a forwarding address by then, you have already received the message.  If you did not have a forwarding address, that message may still be in the backlog of mail to be delivered.  If you did not bounce any mail, you did not receive a message.

All the original updates are included below, for your reference.

Report from the system administration staff

Obviously, this is an incredibly horrible, extended outage, and we can only give you an explanation, not by any means an excuse.  And the explanation, in short, is, we got caught with our pants down.  We have been doing behind-the-scenes work on Mailstore for the past few months.  As noted in several of the comments below, Mailstore is both a single point of failure, and one of the harder services to fix quickly, because of the massive amount of data it involves.  So, even something as simple as "add more storage" can become challenging when that requires moving to new hardware instead of adding more drives to existing hardware.  

The ongoing projects for Mailstore over the last several months have been: switch the backend processing software (from Cyrus to Dovecot, for additional features and more stability), add a new storage device (which we did, and has been problematic from the get-go), get a replacement storage device for the new storage device (which has not yet been delivered), have a hot onsite backup, and a cold offsite backup.  But doing anything involving copying, moving or reorganizing Mailstore requires downtime, which we have been trying to minimize.  (I know, and see where it got us?)  So, we have been proceeding slowly, migrating accounts individually, and basically holding off on important things to avoid either slowing down your mail access or incurring extended downtimes.

As the performance of the current (quite new!) Mailstore hardware has degraded, and with the replacement not yet on site, we pushed forward to deal with the problem by planning a brief downtime to fail over some services onto less loaded parts of the system. That was going to happen tonight. This put us in a race with the hardware: we had to get to our maintenance window before it failed, because it was clear the planned means of failover ("Plan B") would not work. Mailstore's current workload would utterly overwhelm the failovers.

Unfortunately, we lost the race. This morning, a series of cascading failures, some seemingly entirely unrelated to the existing problems, including the complete corruption of our backup storage device, brought down the Mailstore service in such a catastrophic way that "Plan C" and "Plan D" for recovery were out of the question. We had to cobble together something from parts of plan E through K, and the result was what you'd expect: a number of false starts and unforeseen problems.

We counted on things staying the way they were, at least for a little while, rather than insisting on a downtime much earlier on to prep for this kind of catastrophe.  And, for that, all we can say is how sorry we are.  We are hoping to be back online very soon, and will continue making updates until we are.


What can I do to get my mail?

We recommend adding another forwarding address right away by clicking the "Edit" button to the right of "Your Mailstore Inbox" in the Delivered To column.  Get more detailed instructions on adding a forwarding address.

If you've only used your Mailstore Inbox and aren't sure what a forwarding address is, a forwarding address is an email address at another ISP or provider.  We take mail sent to your Pobox address, and forward it there.  As long as you leave your Mailstore Inbox as one of your other addresses that we deliver to, the mail that we forwarded will also be delivered to Mailstore.

How can I get updates?

We're tweeting updates on the situation as we get them.  You can view them on the web at http://www.twitter.com/pobox (or status.pobox.com), or follow us on twitter @pobox.

What happened?

We had been seeing slowness and errors throughout the day. We had planned an outage for late this evening. But, the problems were growing, and we thought that the problem could be resolved quickly by resetting the storage cluster. We were mistaken.

After we powered down the equipment, it did not come back up.  We have been in touch with the vendor; they currently believe either the power supply backplane or the motherboard needs to be replaced.  We are now working on bringing up the backup hardware.

Bringing up the backup hardware takes a while because there is so much data on Mailstore that needs to be kept in sync.  Unfortunately, this process is somewhat opaque, so it's hard for us to tell how long it's going to take to finish.

Why is it taking so long to bring up the backup hardware?

The backup hardware is underpowered, and we were aware of this. Replacement hardware has been on order, but hasn't yet come in. Unfortunately, this is simply a case of really crappy timing.

What is NOT affected?

Basically, anything besides accessing your mail stored on Mailstore (whether you use your email program, webmail.pobox.com or atmail.pobox.com).  Forwarding, sending mail via SMTP, spam processing (and viewing via the website), and all other website functions should be unaffected.

Update @ 6:26 PM: A number of messages bounced.

As we are bringing boxes up, there was an error.  The IP address for mailstore was picked up by the firewall.  The firewall, not being configured to accept mail, bounced approximately 8,000 messages.  If your mail was among those messages, we will get as much information to you as possible about any lost message. 

I have nothing to say about this, other than I absolutely share the sickening feeling you may be getting.  We pride ourselves on never bouncing or otherwise losing legitimate mail.  I don't even know what to say, except this is the kind of day we have nightmares about.


Update 14 December @ 2:40 AM EST - What is going on?

We are waiting on the remaining slow, clearly degraded hardware to finish making the data available.  Once it is up, it will still be slow.  This weekend, we will be doing everything we can to get off this degraded hardware, once and for all.

In answer to the question, "how did this go so badly wrong?", I can only say that the most horrific words in the English language are: "the backups are completely corrupted."  Rest assured that there will be a complete analysis done of all the myriad ways our existing solutions failed us, in addition to work on the already-planned upgrades.

Update 14 December @ 7:40 AM EST

As this failure stretches on and on, we are working on alternative plans.  Right now, we are setting up a new device, to give you access to the mail that has already come in. However, as seems to often be the case, as you start working on alternatives, the originals near completion.  We hope to have a more definitive report in about 30 minutes.

Update 14 December @ 8:20 AM EST

The storage restore that had been running for the last 12+ hours has finally completed.  Mailstore access has been restored.  It will be slow, and you may still get the password requests we were seeing yesterday will almost certainly still happen.  We will keep you appraised of future steps as we solidify the plans for them.

170 comments:

  1. Oh, boy. And I thought it would be good switch from Gmail.

    ReplyDelete
    Replies
    1. That's exactly what I thought.

      Delete
    2. Getting "in touch with the vendor", right. Starting to feel really really lame for paying so many years in advance.

      Delete
    3. It's what we thought in house, too. This is why we don't crow about email providers having outages -- it happens to all of us, sooner or later. :(

      Delete
    4. No Vanessa, it does not. I NEVER had an outage with gmail.

      Delete
    5. Anonymous: You haven't been using Gmail for very long, or you haven't been paying attention.
      http://articles.timesofindia.indiatimes.com/2012-12-10/internet/35725867_1_google-mail-service-outage-gmail-service

      Regardless, in the 13 years I've used POBox I can't recall the last time they were unavailable. Given the complexity of the technology, things are bound to break. And it's not a high margin business.

      Delete
    6. Point of information: Gmail has outages all the time. Like, for instance, three days ago: http://mashable.com/2012/12/11/gmail-outage-chrome-crash/

      Vanessa is correct: it happens to everyone. For instance, take note of the multiple AWS outages over the past year. If Amazon can be hit (taking half the net with it) then anyone can. It's just miserable timing. It happens.

      Pobox is being very responsive and open both about what has happened and what they're doing to correct it. You're not going to see that kind of honesty and service from Gmail, Hotmail, Yahoo! or AOL.

      Please chill, set up your forwarding address, and stop attacking the people who are working hard to fix the problem and keep you informed.

      Delete
    7. Pobox has served me well for many years (something like 8-ish now I think?) and this is the first time I've ever seen an extended outage. I've also worked in IT and watched a NetApp eat all the data on it, so I can sympathize.

      If anything, we learn from failures such as this and put in place extra redundancy. Life ain't perfect, so it's bound to happen to even the best-designed system.

      --- Zach

      Delete
    8. I've been a customer since 1998 and I've never experienced anything like this in all that time.

      Delete
    9. I've been using pobox for almost ten years without a hitch, so it would be petty for me to start bashing y'all now on the basis of the first real problem. I did set up an new/emergency forwarding address -- however, I have no idea if that has worked, or how to check if it's working. So far nothing delivered to the new forwarding address, and it has been a couple hours. Some more information on that temporary solution would be useful.

      Delete
    10. I do not believe this to be a reason to give up on Pobox. I've been a customer since 1998 and have been very pleased during that time. For a vendor to be as up front as Pobox has been is not as common as I would like. Yes this outage is very annoying and it's easy for everyone to play "armchair quarterback." In my experience issues like the ones which have been described are often only discovered when an environment craters due to an unforeseen or unplanned event. The only way to truly prevent them is to spend unholy sums of money for full end to end N+1 redundancy and then test-fail one or more of those systems on a regular basis. In almost every situation such redundancy and planning is not financially feasible.

      What we don't hear about are the thousands of times Pobox's planning *prevented* an outage. If we did it would happen often enough that we'd treat those messages as spam. Pobox has a fraction of the computing power as many larger mail providers such as Yahoo, Google and Microsoft. Yet with that smaller computing footprint they deliver a solid service with great customer service, upfront communications and no advertising. I still think Pobox is has great value for my money.

      -AN

      Delete
    11. There is no excuse for this outage. There are myriad ways of protecting against these kind of outages. Clearly poor management and talentless technical architects.

      Delete
    12. Can we have an update please. How long until mail store restored can you guarantee no data loss.

      Delete
    13. Unless this is fixed soon everyone will be back to Gmail I'm sure. Over 12 hours in now and still broken.

      Delete
    14. Update please.

      Delete
    15. Update? Is mail lost? Is there an expected time for return of service? I need to make decisions now on what I do for email and if I need to find a new provider. I love your service but I'm working with international partners and need to have reliable email (that's why I pay for it). Please update.

      Delete
    16. It IS a good switch from Gmail. I've been with POBOX since they started. They've had a couple hiccups along the way because they continue to grow because they are so good. This is THE worst outage EVER since 1995 (I signed up in 1996). While I understand about the urgency of the international partners, Jennifer and her crew will make it right! Jennifer has been there since the inception of this business and she's got her eye on customer service. You would not believe what they did for me when I deployed to Germany in 1996 / 1997. Please don't discount this service. AND they are giving six months worth of credit, which for me, seems excessive!

      In love with pobox since 1996

      Delete
  2. "We had been seeing slowness and errors throughout the day." You should have sent out a tweet when you first noticed the problem. Your first tweet didn't come out until after the problem had been happening for over an hour.

    ReplyDelete
    Replies
    1. I agree, we definitely could have tweeted a little sooner. But, with slowness, the conversation goes something like, "A customer reported they're getting an error connecting to Mailstore." "Has anyone else reported it?" "Not yet." "Are we seeing issues on our own accounts?" "No." It takes a little while to determine that a problem report or two is a system-wide problem. As you say, it had been about an hour, and in the last 10 minutes, it went from "we should tweet about this" to "we should try to fix this right now."

      Delete
  3. I am sure that Pobox are doing their best but to be honest as an IT guy I am surprised that the replicated servers are also affected and are not being available as Microsoft does with the DAG.

    I hope that this will be fixed soon because this is too much - I will switch back to Gmail If i have to.

    ReplyDelete
  4. The outage isn't affecting forwarding, so if you have some other address you can use temporarily, set it up to forward there. I'm sure they're trying their best, and I've had really good service from pobox.

    ReplyDelete
  5. Is it at all likely that any data has been lost?

    ReplyDelete
    Replies
    1. We have no reason to believe at this time that any data will have been lost as a result of this.

      Delete
  6. so why the backup hardware is under-powered???

    ReplyDelete
    Replies
    1. Because it takes a while to get replacement hardware once you realize that what you have is underpowered.

      Delete
    2. Sorry to ask again but I really want to get some answers here -- a backup system or cluster and failover system is a must here. I don't understand how come that the replicated data / servers are OFFLINE as well (as you call it underpowered)
      why?

      Delete
    3. I'm afraid I'm not at the datacenter. I've just been told that they're trying to get the data fully synced before making the systems live again.

      Delete
    4. great answer, thanks.
      will be patient

      Delete
  7. This is just the first sign of the end of the world. We are so lucky to be the ones to know.

    ReplyDelete
  8. Wow, I pay for this service? Outages are regular compared to any free service I used. This was meant to be my reliable mail option. Now I am really concerned about lost data. Have to come to the conclusion that this is not the email option for a professional.

    ReplyDelete
    Replies
    1. Finally someone else is also feeling really really lame like me. I've transferred my whole $#*%# gmail messages to mailstore and now I can't even access webmail or any folder. This is so insanely messed up.

      Delete
    2. I thought this way too, $50 for a year is quite peace of money. I really thought that there are few Storage system at pobox - I thought for myself that if they are doing such a service for 17 years I believe that they have quite of Data center.

      I think that I should consider move back to Gmail.

      Delete
    3. Yeh. I feel really lame. I have a major international job that relies on email. Decided to go with Mailstore rather than gmail because I thought it would be reliable and professional. In the 3 months since I made that decision, 2 major outages and many minor problems with access. In 5 or more years with gmail NEVER A SINGLE PROBLEM! No access means no work for me.

      Delete
  9. Not sure that posting a "Mailstore is down" note on the login page is what I expected when you advertised better service.

    POBox says:

    "But on the days when something goes wrong, we're here to answer your questions, come up with options, and do what it takes to make sure you have access to your email."

    "Charging for service aligns our goals with what you want (better services)"

    "with a free account, you're likely to find out how much support you have (none)"

    ReplyDelete
    Replies
    1. We've been addressing questions via Twitter and the customer support system throughout the outage. However, for customers who are just jumping online, we thought it was best to post a wrap-up of the situation thus far, rather than expecting people to go back through past tweets.

      Delete
  10. I've been a customer since 1998. Outages have been extremely rare and I don't remember one like this ever happening. I've been very happy with the service.

    So hang in there and keep up the good work. Most companies don't say much about the details of outages so I appreciate your posting them Vanessa.

    ReplyDelete
  11. I've been a pobox.com customer for a long time, and recently switched to mailstore this past year, having been unhappy with my other providers and not wishing to admin a box for myself.

    While pobox.com customer service is strong, I'm really paying for having "continuous" access to a IMAP server. Though I don't believe there is a specific SLA in place for what that REALLY means, I'm certain Pobox is doing its level best to keep the service up 24x7. That said, "best effort" should include infrastructure evolution.

    While single points of failure for storage and compute devices were accepted as "acceptable" outage cause a few years back, with cost-effective cloud offerings (AWS, profit bricks) now able to provide multi-machine/multi-region redundancy with 99.9999% uptime (less than 1 minute per year downtime) with minimal infrastructure cost increase, single device failures really should not be an issue in modern infrastructure.

    I'm not sure what kind of in-house RAID storage system Pobox is using (HP MSA, Isilon, etc), but after today's exercise in damage control, serious consideration should be given to either moving to AWS S3 or other cloud service as a primary or at least "backup", so that such an failure does not result in such large scale outage going forward.

    Thanks

    ReplyDelete
    Replies
    1. in layman's terms: we're running under very amateurish conditions :)

      Delete
    2. As I told Avi over Twitter, we actually had recently evaluated using cloud storage as a backup for Mailstore, and it simply wasn't fast enough. Making Mailstore more robust is a major priority for us; unfortunately, the timing of this failure caught us mid-improvement.

      Delete
    3. Infrastructure upgrades are uniquely challenging situations, and Murphy's law frequently comes into play. Getting each additional "9" of reliability past the first three invokes an almost exponential cost. I'm pretty sure my $50/year isn't gonna cover more than a few "9"s.

      Ironically, if they were relying on AWS as their primary backing store, they'd likely have been affected by the AWS outage last year, unless they went with replication across availability zones, which would have exacerbated the performance concerns Vanessa cites.

      I've been a pobox user since 2001, and I've been generally pleased with their performance, spam handling, price point, and most of all, privacy. Those of you who are thinking of going back to GMail, remember GMail isn't Google's product... YOU are their product!

      @Vanessa, once all this is over it would be interesting to know what the general statistics of pobox are; Size of storage, # of emails handled per day, spams per day, etc.

      Delete
  12. I'm with Quent. Pobox is a great service with few problems. It's actually stunning that they are posting what REALLY is happening and taking responsibility for it rather than some corporate-speak mumbo jumbo that would be a certainty from Google or Yahoo. As all of us in the tech business know, stuff happens and generally at the worst time relative to the severity of the problem.

    Nice to see lessons learned, changes being made as a result - that's all anyone can ask. I'll still recommend Pobox.

    ReplyDelete
    Replies
    1. Update - and I rreceived a call in response to my web form inquiry a few minutes ago. Now that's customer service!

      Delete
    2. I've only needed it a few times, but the customer service has always been fantastic.

      Delete
    3. I sent an email asking about an outage a few hours ago and I still have not recieved a response. So I don't think the customer service is good enough. I mean you can have an auto reply on at least saying we are down.

      Delete
    4. Tickets had been coming in throughout the day, so we only realized around 4:30 that messages sent via email (which are picked up into our ticket system via Mailstore) were not being handled. (Thanks, @cri, for letting us know on twitter.) Once we realized that, we started pulling them into the ticket system as well, but we're very sorry for the delay.

      Delete
  13. I have been with pobox.com since sept 1995. They aint perfect, but they are the best at what they do. Kudos again for a clear truth about the problem - that has always been the way they deal with their clients.

    ReplyDelete
  14. It's funny how technical this audience is. It's the reason why I signed up 15 years ago. All the customers were involved in tech as I was. And, over 15 years it has been a relatively fine experience. Maybe a glitch here or there, but over all a great experience. Gmail was down for a while this week. It happens to everyone.
    Bet of all, the staff at PoBox are great folks and try their damndest to make things work for us.

    ReplyDelete
  15. I agree. I have been with pobox.com since 2006 and have been very happy. I've always gotten excellent support and customer service. These are people I like and I'm confident they are working hard to resolve the issue.

    ReplyDelete
  16. Hi folks,

    I've been with Pobox.com for SIXTEEN YEARS and this is the first time, to my knowledge, that there has been a problem like this.

    "A little slack, you should cut them." Yoda.

    Peace

    ReplyDelete
  17. 'm a 15 year subscriber. This is unacceptable for a "paid" service. Obviously profits are getting thinner with all the new mail options and pobox isn't keeping up. "Make Money"
    I'm out of here when my subscription comes due.

    ReplyDelete
    Replies
    1. I'm am doing a re-thinking as well.

      Delete
    2. your going to have to really hunt for a better spam processing service. In my 18 years with pobox I looked at others, but none even come close.
      BTW: email is VITAL to me - so I have dual services, imap with pobox and pop with verizon configured on my cell phone. Sure the imap is better, but i sure can live with getting pop for a while.
      YES, the pobox ideal is 24x7 but "backup plans" on OUR part are prudent. Having dual configurations "in advance" saved my hide today. I suggest others do the same.

      Delete
  18. It's making me a little crazy...but, my iPad is keeping me connected. I've used POBox for 13 or more years and I don't ever remember a problem like this. I'm sure they will fix it soon. Good luck guys! :)

    ReplyDelete
  19. I am new to Pobox- currently on a 30 day-free trial.....What is a service credit?

    Thanks for any information.

    ReplyDelete
    Replies
    1. Your unused free days are credited by how much you paid for them, applicable to future free days, as applicable.

      Delete
  20. Everyone stop your crying. I have been on pobox for 10 years and NEVER had an outage. You can live without email for a day or two! Your life will go on!

    ReplyDelete
    Replies
    1. I must say that I can honestly agree

      Delete
    2. In this day and age people have jobs where email is essential. Would you tell a gardener he can live without his tools for a few days? Life will go on sure, but he will lose his income if he can't work.

      Delete
    3. Quit complaining, set up your forwarding email address, and give Pobox a break. NO service (paid or not) is immune to problems. Pay attention to the fact that they are responsive, taking responsibility, offering 6 month free service, gave you a way to keep getting email during this interruption. There is nothing to complain about, so stop. You sound like a spoiled child.

      Delete
  21. I've been with this service for 13 years. This is not the first outage. Outages are inevitable. Even our corporate email goes down sometimes. These guys handle themselves well, and this service is more reliable than Google and many others.

    ReplyDelete
  22. I've been with Pobox for 7 years and I don't recall ever having a problem like this before.
    I'm sure a lot of the 'negative' energy on display here could be put to better use!! Merry Christmas from London :-)

    ReplyDelete
  23. Can you provide an update?

    ReplyDelete
  24. I've been with pobox since 2000(?) and never seen this sort of outage either. I was one of the people reporting early issues, and they responded right away (yesterday before the entire service went down).

    ReplyDelete
    Replies
    1. One thing I'm not clear on - it sounds like there is no expected data loss for already stored email, but what about messages that come in while the system is still down, and we don't have another place to forward to? Should we expect those messages to be stored properly and accessible when the system is back up, or are those going to bounce?

      Delete
    2. Messages that have been coming in are being queued (or, if you have added a forwarding address, forwarded and queued.) Once the storage servers come back up, they will be delivered to your Inbox. They will not bounce.

      Delete
    3. I was hoping this would be true, but messages are bouncing as of my latest test a few minutes ago.

      It looks like you are using Postfix and the incoming and active queues are both full.

      Delete
    4. I believe my emails are bouncing. Someone mentioned it to me before the outage.

      Delete
  25. This is not true. Messages are bouncing. See below:

    Delivery has failed to these recipients or distribution lists:

    00744628@mailstore.pobox.com
    A problem occurred during the delivery of this message. Microsoft Exchange will not try to redeliver this message for you. Please try resending this message later, or provide the following diagnostic text to your system administrator.







    Diagnostic information for administrators:

    Generating server: b-pb-mailstore-quonix.pobox.com

    00744628@mailstore.pobox.com
    #< #5.4.6 X-Postfix; mail for mailstore.pobox.com loops back to myself> #SMTP#

    Original message headers:

    Received: from b-pb-fb-quonix.pobox.com (b-pb-fb-quonix.pobox.com
    [208.72.237.77]) by b-pb-mailstore-quonix.pobox.com (Postfix) with ESMTP id
    1C6E2D529 for <00744628@mailstore.pobox.com>; Thu, 13 Dec 2012 23:18:21 +0000
    (UTC)
    Received: from indigo.pobox.com (indigo.pobox.com [74.115.168.42])
    by b-pb-fb-quonix.pobox.com (Postfix) with ESMTP id F1621A65E
    for <00744628@mailstore.pobox.com>; Thu, 13 Dec 2012 17:59:33 -0500 (EST)
    Received: from indigo.pobox.com (localhost [127.0.0.1])
    by indigo.pobox.com (Postfix) with ESMTP id 00D5D201F38;
    Thu, 13 Dec 2012 15:36:00 -0500 (EST)
    Delivered-To: jrreber@pobox.com
    X-Pobox-Orig-Sender:
    X-Pobox-Delivery-ID: B3BD4516-4564-11E2-89FC-AF3A577AAA38-00744628!indigo.pobox.com
    x-pobox-client-address: 165.212.64.21
    x-pobox-client-name: gateout01.mbox.net
    Received: from gwo1.mbox.net (gateout01.mbox.net [165.212.64.21]) by indigo.pobox.com (Postfix) with ESMTP id D983E20288F for ; Thu, 13 Dec 2012 15:35:54 -0500 (EST)
    Received: from gwo1.mbox.net (localhost [127.0.0.1]) by gwo1.mbox.net (Postfix) with ESMTP id B7EAFE295D85 for ; Thu, 13 Dec 2012 20:35:31 +0000 (UTC)
    X-USANET-Received: from gwo1.mbox.net [127.0.0.1] by gwo1.mbox.net via mtad (C8.MAIN.3.82G) with ESMTP id 171qLmujd5632Mo1; Thu, 13 Dec 2012 20:35:29 -0000
    X-USANET-GWS2-Tagid: UNKN
    Received: from S1HUB2.EXCHPROD.USA.NET [165.212.120.254] by gwo1.mbox.net via smtad (C8.MAIN.3.88D) with ESMTPS id XID118qLmujd8122Xo1; Thu, 13 Dec 2012 20:35:29 -0000
    X-USANET-Source: 165.212.120.254 OUT reber@usta.com S1HUB2.EXCHPROD.USA.NET
    X-USANET-MsgId: XID118qLmujd8122Xo1
    Received: from MBX17.EXCHPROD.USA.NET ([10.120.221.171]) by S1HUB2.EXCHPROD.USA.NET ([10.120.220.32]) with mapi; Thu, 13 Dec 2012 20:35:29 +0000
    From: "Reber, Patty"
    To: "Jan R. Reber"
    Date: Thu, 13 Dec 2012 20:37:17 +0000
    Subject: RE: Bonda
    Thread-Topic: Bonda
    Thread-Index: Ac3ZcRHCN+qjrpqURaOMo8YvGRt0EgAAI8kg
    Message-ID:
    References:
    In-Reply-To:
    Accept-Language: en-US
    Content-Language: en-US
    X-MS-Has-Attach:
    X-MS-TNEF-Correlator:
    acceptlanguage: en-US
    Content-Type: text/plain; charset="us-ascii"
    Content-Transfer-Encoding: quoted-printable
    MIME-Version: 1.0
    X-Pobox-Pass: reber@usta.com is whitelisted

    ReplyDelete
    Replies
    1. We discovered this moments ago as well. The firewall was trying to accept mail, and could not. More details in a moment. :'(

      Delete
  26. Is this why I'm suddenly getting hundreds of "relay access denied" errors?

    ReplyDelete
  27. Also getting tons of ": mail for mailstore.pobox.com loops back to myself" bounces.

    ReplyDelete
  28. I've been with pobox since 1996, and I too don't recall any outage like this. Every time I've had to deal with customer support they've been superb. This outage is a pain, but I find it refreshing that pobox staff is keeping everyone updated and giving straightforward, honest answers. I do expect a hot backup in place for the money I pay for this service, but I understand s**t happens and when it does the best you can do is handle it professionally, which pobox is doing. Now, if lessons aren't learned and this happens 6 months from now and pobox still doesn't have a hot backup ready, I will be pretty upset.

    ReplyDelete
    Replies
    1. I too have been a POBOX customer since early 1996. Yes, this outage is inconvenient, and I may even lose a legitimate email or two from the ones that got bounced, but I value even more that POBOX shields me from hundreds of SPAM emails every day that most services would just let through.

      There is no such thing as 100% reliability or availability! $50 is a cheap price for what MAILSTORE provides. I also applaud POBOX for the candor they are displaying by telling us about it -- the good, the bad and the ugly!

      Delete
  29. I'm patient; email is resilient. I've been using you guys for going on 13 years now, I think. I'm sure there will be a top-level management meeting tomorrow discussing a better backup system plan

    ReplyDelete
  30. *I definitely appreciate the transparency.* As long as this doesn't happen again within a year and we see changes, you'll probably keep my business.

    ReplyDelete
  31. Long time customer, and long time experience running 100+ person technical support teams. I appreciate the pressures you are experiencing, but based on your (laudable) transparency it seems that you are taking actions that are making the situation worse. Take a step back, a couple of deep breaths and chart out your steps to closure. Stare at that chart fro every angle, before proceeding. I want mailstore up and running as quickly as possible, but your actions that led to bounced email are simply not acceptble. That's water under the bridge, but another such mistake and you'll have lost an otherwise lifelong customer for good.

    ReplyDelete
    Replies
    1. We absolutely agree, and we are closing in on the workable, if slow, solution.

      Delete
    2. How long to go?

      Delete
  32. Vanessa, you "discovered" the bouncing emails and hour and a half ago. What is going on?

    ReplyDelete
    Replies
    1. Ignore that, I missed that comment in the blog. Good luck guys. We've all been there at one time or another.

      Delete
  33. I'll take a stab in the dark and assume replicating 289 terabytes of data takes a while

    ReplyDelete
  34. Vanessa, you guys will get through this. Keep doing what you're doing, communicating, and know that "this too shall pass."

    Customer since 2001.

    ReplyDelete
  35. Five hours. This ought to be a practical joke.

    ReplyDelete
  36. I have been a Pobox member since 2002. Pobox has ALWAYS had great customer support, good turn around times on questions and problems, and a great service. You guys take as much time as you need to get things up and running.

    Just keep us in the loop with progress as you have been.

    Thanks!
    ~michael

    ReplyDelete
  37. Vanessa,

    Thanks for keeping us in the loop. This honesty and transparency puts pobox miles and miles in front of 99% of the companies out there and is one of the reasons I've been a member so long.

    Joe K

    ReplyDelete
  38. To quote Tyler Durden in the movie Fight Club:

    "Listen up, maggots. You are not special. You are not a beautiful or unique snowflake. You're the same decaying organic matter as everything else."

    I've used Pobox for FIFTEEN YEARS and in that time this is only the SECOND time that I've experienced an outage. Get over yourselves you bunch of pathetic whiners. Go back to Gmail and just shut up. Try getting an email back from Google within an hour of your post to support.

    Shit happens. Get over it. Move on.

    ReplyDelete
    Replies
    1. Yah. You think Google is gonna talk to you like this?

      Delete
  39. Customer since 1999. The outage is annoying, but I have a local copy of all my IMAP folders, and since I had set up forwarding to a Gmail account as a travel backup when switching to Mailstore, I can get my incoming mail. I'm a little disappointed at how long it's taking to resolve the problem, but every email service I've used has had problems, from the UUCP mail I started out using and Compu$erve to my current university email. Pobox has been a lot more reliable than the .Mac/MobileMe/iCloud service I was using for my personal email before switching. I'd rather have the fix take longer and be done well than have a half-baked system rushed into service. I'd also prefer more planning for backup systems than appears to have been done--something to prioritize once this problem has been resolved.

    It's also great that forwarding has been unaffected during this outage. I joined Pobox for forwarding; even if I decide to move away from Mailstore, I'll keep my aliases as long as they continue to work reliably.

    ReplyDelete
  40. PoBox.com wrote: Due to the nature of this outage, a credit for 6 months of service has been added to all Mailstore accounts. We are truly, deeply sorry for the inconvenience we know this is causing all of you, and we're particularly mortified that you have lost any mail as the result of a failure on our part. We appreciate the patience (and kindness) you've shown, and hope we can re-earn your esteem as your email provider.

    Best I'm sorry ever. Winning!

    ReplyDelete
  41. Let him without sin cast the first stone.... come on, no one is perfect! POBOX is GREAT and the people are doing their best to figure it out! Show a little compassion and support!

    A happy customer since 2006

    ReplyDelete
  42. I seem to be the "old man of the mountain" having been with pobox since September 1995. Clearly they A) are the best email/spam filtering service and B) have some expensive work to do.

    Vanessa, thank you for adding 6 months. I'll see your six months and raise you an additional 2 years - I just made a payment of $100.

    Pobox will need capital to pay for system improvements - WE want more reliability (it's been great overall for the past 17 years). So - loyal pobox fans NOW is the time to "add a couple of years". Give pobox a sudden and much needed infusion of working capital.

    PARDON THE SHOUT, BUT, EVERYONE SHOULD, IMMEDIATELY RENEW FOR A COUPLE OF YEARS - (and also protect yourself from rate increases!).

    It's a win-win for all concerned. Help Vanessa and the pobox gang to fund those upgrades. If we "do our part", I am sure they will do theirs.

    ReplyDelete
  43. Speaking, if I may, off-topic.. I want want per-account SPF rules back along with SIEVE filtering, but apparently I'm a one-off geek who should be running his own server himself for that level of control

    ReplyDelete
  44. Considering I lost two GMail accounts and their contents, completely, with no recourse whatsoever, I figure one outage in years is still not too bad.

    ReplyDelete
  45. Hey guys:

    I know it's a &*(3@_ tough time for ya. I hope you manage the transition without too much hair-pulling.

    If it's not too much of a problem, could you email us updates once you have your servers up again. I'm letting my guys know I am out of email range for the time being.

    Anyway, been there and understand the pressure. So GET BACK TO WORK NOW!

    :)

    Jaye, ke6sls

    ps: this talkback sort of sucks. . . . just sayin' heehee

    ReplyDelete
    Replies
    1. Hi - don't forget that you can still set up a forwarding address - get a temporary address somewhere else if you don't have one already and forward the email from pobox.com to there until the outage is over. I've done this and it's straightforward.

      Delete


  46. Kenneth;

    You are indeed the "old man of the mountain", though I will admit to being close having been here since 1997, and have not had anything major to complain about in all that time, aside from the occasional bounced mail. I will continue to stay with Pobox for forwarding and mailstore, as I'm sure they will work out the issue.

    Vanessa;

    Experience is something you get the moment AFTER you needed it. This is painfully true in the tech infrastructure world, and you guys just gained some valuable experience about pre-emptive maintenance, redundancy, and multiple backup plans for confluence of "perfect storm" during maintenance windows.

    I have not been able to view your twitter response, so not sure what specifics you provided regarding cloud provider performance, but will say that AWS in specific is continuously evolving its offering and now has provisioned IOPS and network throughput to reach whatever throughput performance needed (though the cost will likely rise as well). Additionally other cloud providers such as profitbricks tout their highspeed throughput so it might be worth revisiting the cloud option and presenting them with your performance requirements and see if they can provide what is needed.

    But I will re-state my original thesis slightly differently, based on the agreed-upon immutable that "everything fails": Pobox will be well served to carefully and completely rethink its infrastructure in terms of assuming EVERYTHING will fail at worst moment, and create the requisite redundancy to keep things afloat when they do. Multi-hour access failures and bounced mail at this juncture really should be resulting from deliberate aggressive multi-site network and/or physical attacks based on compromised internal security, and NOT from one's own gun going off while tucked in ones pants and shooting off ones foot (or other parts)... AWS has some nice white papers on how to design for this very situation, based on THEIR failures.

    Thanks for keeping us updated.

    ReplyDelete
  47. Sounds like a "Chernobyl" experiment.

    http://en.wikipedia.org/wiki/Chernobyl_disaster

    ReplyDelete
  48. Dear Vanessa,
    Please, pass me only one info: my account is down from about yesterday at 15PM (Brazilian time) and I have set new FWD address only now.
    You can resend to me the e-mail that I have received in this time period or they are lost?
    Thanks!
    Fabrizio

    ReplyDelete
  49. Can we get some photos from the datacenter ? :)
    It will be nice to see what is going on with this....

    ReplyDelete
  50. What's going on. I have no access to my mail. Have you all gone home for the weekend? When will the mail store be back?

    ReplyDelete
  51. Poor planning, poor design, poor execution, poor management and a lack of technical skills have caused this outage. Full resiliency and redundancy should be built into this at every layer. There is no excuse whatsoever. Please provide an update and confirmation that there has been no data loss.

    ReplyDelete
    Replies
    1. Armchair quarterbacking at it's best, right here, folks! He/she's the expert on this whole problem, without knowing anything about it.

      (let the flame-war begin!)

      Delete
  52. Can anybody suggest alternative providers?

    ReplyDelete
  53. I'm not sure that those 6 months will make the difference.

    ReplyDelete
  54. I need my email . mail store is my only repository. When will it be fixed. I can only imagine if this goes on much longer you will be out of business.

    ReplyDelete
    Replies
    1. you can get your incoming mail with 15 minutes of effort. set up a free hotmail account and add that email as a forwarding address. the instructions are at the top of this blog. mailstore can be one of two repositories.

      Delete
    2. How does that help me with my imap folders. It doesn't. That's why I pay the money.

      Delete
  55. Come on pobox. Wake up.

    ReplyDelete
  56. Can anybody suggest any other quality paid for services?

    ReplyDelete
    Replies
    1. I've only been with Pobox since November of 1997, so I don't have the experience of some of these old timers...

      With mailstore being down, I've gone back to my old Hotmail account. Microsoft has made many improvements to it since I've used it last. It's actually pretty slick. They now give you the option of using Outlook from your browser. The calendar feature is very nice.

      Microsoft has the infrastructure to prevent data loss and lengthy outages. For $19.95 per year, you get ad free mail without requiring you to actively use the account to prevent it from being removed.

      Delete
    2. It's strange how everyone posts this is the first time of a major outage. The fact is that Pobox has simply been lucky. It is very common to have fail over servers that are identical. I am wondering why this was never implemented.

      Hardware fails. It is going to happen sooner or later. A network infrastructure engineer should have planned for this with a fail over server.

      High availability of servers means planning for disasters. It is quite common to do this nowadays. Pobox, please add a fail over server. Not just a backup that you have to bring on line when a server dies. Add a fail-over that automatically switches to the functional server in the event of a failure.

      Delete
  57. Been a Pobox customer since 2005 and this is the first time I've seen such a serious problem. Glad to see the level of transparency here, even though it seems like the manure has really hit the windmill.

    I still have faith in you, Pobox. Good luck fixing the problems and learning from them.

    Sagar

    ReplyDelete
  58. The Microsoft service sounds good. I think I will move to that.

    ReplyDelete
    Replies
    1. I have been pleased with it. You get a lot of features for very little money. For only $19.95 per year, you get:

      * A robust fully featured email client accessible from your web browser (Microsoft Outlook). It is able to do things like recover accidentally deleted mail after you have emptied your deleted items bin (all from within a browser...wow).

      * Mail forwarding options, vacation mail options, POP access from a locally installed mail client, spam filters, email aliases and options to rename your email address, folder creation with filter support (just like Outlook).

      * Access to SkyDrive that provides you with 7GB of secure storage on the web. With SkyDrive explorer, your content is kept in sync and can be copied to your web storage via copy and paste natively in Windows. You have the option of increasing this storage to over 100GB at an added cost.

      * Status reporting website showing the current status of all services: https://status.live.com/

      Delete
  59. Mailstore is back. Certainly, it is slow but I have all my emails again.
    I wanna say here thanks to the POBOX guys for not loosing it!!
    your still satisfied customer.

    ReplyDelete
  60. It would be nice to know if we can stop forwarding email now, or what. I can see that the mailstore is up again, but I don't see there the emails I got on my forwarding address. I assume this takes time, but please give clear instructions on how to proceed now. Is it safe to stop forwarding? Please give us a timeline...

    ReplyDelete
    Replies
    1. I would continue forwarding mail for now. Delivering the backlog is quite slow at this point, but forwarded mail is still being sent to forwarding addresses almost instantaneously.

      Also, as we indicate at the top of the post, another scheduled downtime will be necessary quite soon to get Mailstore to a usable point.

      Delete
  61. There is mail missing. Oh dear.

    ReplyDelete
  62. I've been a POBox customer for a long time, and have worked in global datacenters for a long time. I've seen both private and public datacenters have unplanned outages for a variety of reasons ranging from earthquakes to plugging the wrong whatsit into the absolutely worst goesinta.

    This outage ranges somewhere in between. I can't think of a public hosting service that hasn't had at least one similar outage. POBox has been among the best in communicating with their customers. And when you consider this outage into their performance record, their uptime is probably still in the 98-99% range. Sure, it's painful when a problem occurs; but making a permanent move every time a provider has a hiccup doesn't make sense to me.

    I fully expect that once this issue is behind us we'll see a return to the trouble-free always-there service we've normally had. And I expect that the POBox team will use this experience to beef up their failover plans so that the next time there's a similar hardware failure we may only know when they tell us after the fact.

    Cameron

    ReplyDelete
  63. First, thank you very much for your outstanding work in keeping customers updated. This is impressive and very honest. I am glad to be a Pobox customer, and you have dealt with a painful issue in a very professional way.

    I have just accessed my webmail to see how the situation looks, since nothing has changed with my IMAP account since the outage started. All the mail that was there before the outage is there. But none of the mail that was sent to me after the outage has reached. May I hope that all that mail is not lost, but will slowly reach my account over the next few hours and days?

    A clarification on that issue would be useful, IMO.

    ReplyDelete
    Replies
    1. Thanks for the feedback; I updated the post with information about the clearing of the backlog.

      Delete
  64. I have been a customer of PoBox for a couple of years now, and I am a retired IT network engineer. I have been and will continue to be a customer of PoBox. These type of things happen from time to time, if you guys really want to protect your selves the secret is VERTUAL SERVERS a bit expensive initially but from a stand point of down time they are unbelievable what they can do.

    Dan

    ReplyDelete
  65. 6:30am PST. I am able to (slowly as you say) connect via IMAP and webmail. I can see *ALL* of my mail that arrived pre-crah. HOWEVER, I do not see *ANY* new email that arrived post-crash. Not a single message. I'm not that lonely a person. ;-) I setup a 'temporary' additional forarding address last evening when I read the full story, and that mailbox is seeing new mail.

    As you can imagine, I am very concerned about the 12+ hours between the crash and the time I setup my additional forwarding address. My wife's pobox account has EXACTLY the same issue: up and running now, old email AOK, but ZERO email post-crash. Are you still migrating users across?

    Bruce.

    ReplyDelete
    Replies
    1. A little more sluething. From webmail, I just sent myself a short test message. It almost immediately appeared in my 'temporary' forrward email account ... but did not arrive in mailstore even after five minutes.

      Long story short: mailstore has not registered a single email (in my account nor my wife's account) starting at the time of the crash through this very minute.

      Hope to get a read from you (pobox) on this situation.

      Bruce.

      Delete
    2. Same thing here - they claim its working (slowly), but I don't think they have a clue. I was told the a similar story yesterday when this all started and they weren't correct in their assessment that time either. no new email == service still down

      Delete
  66. can reiterate what others are seeing - can now connect to mailstore, but the hundreds of emails I received (since I had them forwarded through gmail) aren't present. Hopefully there's a very backed-up mailqueue trying to catch up, and I'll see a flood of emails by late afternoon, but a blog update confirming the actual emails have not been lost would be great, thanks.

    ReplyDelete
    Replies
    1. Update, checking yesterdays gmail, I got a few dozen emails, sorry for the hyperbole.

      Delete
  67. Sound advice to the mailstore user community: regardless of how 'restored' your service is right now, keep your temporary (additional) forwarding in place for 1-2 weeks.

    I hope ALL of the issues are fixed swiftly and the fixes 'stick' ... but given the severity of the failure this is a time for "better safe than sorry".

    ReplyDelete
  68. Hi. I've been a Pobox customer almost since they first started, back in 1995. This is the FIRST time this had happened, so although it's an absolute pain (I'm missing some emails I really NEED), one major downtime in 17 years is a better than 99.999% uptime. Keep it all in perspective everyone!

    ReplyDelete
  69. I made my business in heavy-duty email servers back in the day, and have used pobox for my personal email since 1995 without any problems, and I'll continue to do so. I feel for you guys because I've been there with the guts of a mail system all over the floor and really slow restoration, without a lot of options. Just got my first piece of email just now this morning, thanks. I'm sure you'll take care of it particularly after this reminder of the fragility of a huge system like this.

    ReplyDelete
  70. 24 hours of downtime in 2012. Never thought I'd see the day. Guess it's time to go google, where downtime is measured in minutes rather than days.

    ReplyDelete
    Replies
    1. ; 100 - (1/365)
      ~99.99726027397260273973

      Looks good to me. :-)

      Delete

  71. "The message indicating whether or not you bounced mail went out last night around midnight."

    Midnight where exactly? EST, PST, UTC????

    ReplyDelete
    Replies
    1. EST. I've updated the post to reflect that.

      Delete
  72. I'm frustrated with mailstore too--and I do not mean any disrespect to some of my fellow users--but Gmail has well publicized breakdowns lasting hours many times per year. Google it (irony not intended). Last outage was 2+ hours just three days ago.

    Furthermore, as a prime email provider Gmail is a prime hacker target with accounts compromised routinely; granted, often (but not always) the fault of poor user passwords. One of the reasons I am with pobox is precisely because it isn't a top-of-list target for hackers.

    I'm not trying to deliver one of the above "suck it up" messages. Constructively, if anyone is thinking of using Gmail because it is more reliable than pobox there is a ton of hard data that says such logic is faulty. There may well be a more bullet-proof provider out there, but it absolutely is not Gmail (nor Yahoo Mail, nor Hotmail ...)

    Bruce.

    ReplyDelete
  73. With Pobox since '06, just reupped for 6 years in April, no regrets even now.

    Not too long ago (0833 MST) I received a test message I sent to my mailstore account from my work email yesterday at 1648 MST. So it does appear things are cranking if slowly.

    ReplyDelete
  74. I've been with pobox.com for 16 years and mailstore for 3 1/2 years, and I don't remember any significant outages before this.

    Yes, there've been a few hiccups over the years, a few hours here and there when E-mail didn't work. But, until now, the problems have always been fixed quickly.

    That is a far better track record than most of the major ISPs I've used over the years.

    Still a satisfied customer.

    ReplyDelete
  75. For myself, the "complete corruption of [the] backup storage device" concerns me greatly. How did this happen? Was it related to the hardware failure that brought down the primary server? If so, it suggests the two systems were perhaps too closely linked, and this might be an architectural issue that must be addressed in the future.

    I appreciate the overall transparency that you've demonstrated so far in terms of explaining the events behind this outage, and hope that will continue. I'm a long-time customer, and while I have no intention of leaving at the moment, your openness will help to allay our fears.

    In particular, I'd like to see future updates from the admin staff on the steps taken to improve stability, especially with respect to backups.

    ReplyDelete
    Replies
    1. We do not believe that the failure of the backup storage device is related to the failure of the box that precipitated the outage. (That box, btw, had its power backplane and system board replaced this morning.)

      But, yes, the entire Mailstore infrastructure, from top to bottom, is being reviewed as part of the postmortem of this outage. For example, one of the contributing factors to the slowness of the restore is that we store snapshots, which help us restore customers' mailboxes in case they accidentally delete too much data, on the primary device. We discovered the number of snapshots counts against you for certain restore procedures. You can believe the number of snaps stored locally will now be far, far smaller.

      Delete
  76. Hi,

    Just wanted to say thanks for keeping us informed, much appreciated. Have used your services for 8/10? years and this is the first major glitch I was affected by and I really don't think it could have been handled more honestly... so here's a big THANK YOU FOR ALL THE MAD SCRAMBLING you folks did to get things back in working order even if not a full final fix yet... you did all humanly possible. Anyone who's been complaining about this probably needs to live a few more decades and face the few crappy situations that arise in one's life no matter how well planned for in advance and gain a little humility. What matters most to me is how the rare crisis is handled... with total honesty or subterfuge... you informed us and provided tips for temporary workarounds... that's as good as it gets. Thanks!

    ReplyDelete
    Replies
    1. well said - I'm agree with every word you said.

      Delete
  77. I've been a customer of Pobox for the thick end of 15 years - during that time I've rarely experienced any problems and when I have the support service has been excellent.

    I've no intention of abandoning Pobox because of this one, admittedly serious, problem. They've been upfront about their travails and provided a 6 months free of charge extension of the their service.

    I fail to see what more they could do - its easy being wise after the event - I know because I'm an expert at it.



    ReplyDelete
  78. Just logged into webmail, and I now a) can log in (yes it's slow), and b) see what appears likely to be all missing messages. Right now I see messages ranging from yesterday 05:32 to today 09:28.

    However, the one message I'm expecting to see that I do NOT see is the one from pobox indicating whether or not I bounced any mail, which was supposed to have been sent around midnight. Did this get sent to mailstore too? I currently only have one alias and it forward to mailstore (yeah I know, now I'm looking at setting up a backup account to forward to elsewhere...)

    ReplyDelete
    Replies
    1. That message got sent to whatever delivery addresses you had on your account at the time. (They were all sent between 23:47 and 23:59 last night.) If you only had Mailstore on your account at that time, then it was sent to your Mailstore Inbox. We still have a fair bit of mail waiting to be delivered, but about 25% of accounts did not have any messages bounced. So, it's possible the report is still awaiting delivery. It is also possible that you did not have any mail bounced.

      Delete
    2. So if I didn't have any bounced mail, I won't get an email about it? That's confusing as the comment at the top refers to "the message indicating whether or not you bounced mail", so I would expect that I'd get a message regardless, and if there was no mail bounced the message would just say that. Otherwise, how do we know if that bounce message itself didn't somehow get bounced or lost? :)

      Delete
    3. OK, now I see the pobox bounce email in mailstore. Dunno if I somehow missed it before, or if it came in since then (though I was already seeing more recently sent emails. Only had 6 bounced messages during that 20 minute timeframe, not the end of the world.

      Delete
    4. Messages were not delivered chronologically, so it's definitely possible that it came in after other messages that were sent later. Thank you for pointing out the error in the blog post, though. I corrected it to clarify that messages were only sent if mail was bounced.

      Delete
    5. Thanks for the info. I've been a pobox member since 99, and this is the only incident of this magnitude that I can remember. I definitely appreciate the transparency and explanation of what's going on, and that is one of the main reasons I will continue to be a member.

      Delete
  79. The webmail comes and goes - sometimes I can get messages and then I'm logged out.
    any ETA for final fix?

    ReplyDelete
    Replies
    1. You should see improved performance this evening, as weekends are typically lower incoming mail, lower usage. We are also hoping to begin account migration to better hardware starting tomorrow. As accounts move to multiple storage devices, performance for everyone should increase. However, until we begin the migration, it will be difficult to say how long it will take, and how many accounts need to move before logins and access are reliable across the board.

      Delete
    2. Ok, Thanks again Vanessa for the updates - you are doing well.

      Delete
  80. Here's some perspective for the whiners:

    A number of years ago, I used to pay Yahoo for mail forwarding. IMO that was an ongoing, multi-year duration clusterf***. Nobody (doubtlessly in India) could ever even understand WHY mail was bouncing. It would work for a month, then start bouncing all over again. Compared to Yahoo forwarding, Pobox is heaven on earth.

    Maybe Yahoo has cleaned up its act recently. I don't know, and I don't want to find out.

    So, to everyone threatening to leave, all I can say is: Good Luck! You will probably need it.

    ReplyDelete
  81. Wow! It appears that my service had been restored as of now, AND all of my mails from yesterday are INTACT, except for three --- about which I was notified, and thus was able to ask senders to send them again! Given that the mail forwarding AND smtp servers were all working during the outage, all I could say to you guys is GREAT JOB!

    As for the considerations of uptime and outage, I have the following to say: the cost to achieve 100% uptime is extremely prohibitive, compared to achieving 99.9% uptime. In fact it is so prohibitive, that even very wealthy enterprises could not do so. Unless a revolution in physics or computing occurs, there's simply no way to do better. QED.

    P.S. Thank you for the service credit too. :-D

    ReplyDelete
  82. Thank you! To the folks at POBox for keeping us informed as this problem progressed. This kind of honesty and dedicated response are exactly what I'd expect from a top-notch system administration group (I know whereof I speak -- I used to lead a SysAdmin group at a major university -- I'm retired now).

    I realize that it's going to cost money to upgrade your service and provisioning so that this problem (at least) doesn't occur again. For that reason, I am investing in the future by pre-paying the next three years of my service. I know you will use the money wisely!

    PS: And thanks for the service credit. It's a nice touch. (-:

    ReplyDelete
  83. I keep getting bounced out of MailStore having to reenter login. What's going on folks?

    ReplyDelete
    Replies
    1. Same here: webmail interface super-flaky, ditto IMAP. Fixes over the weekend go OK, or are we still under "limited service" conditions?

      Good thing I followed my own advice and left the 'temporary' forwarding in place. ;-)

      Bruce.

      Delete
    2. I am also getting bounced out of WebMail repeatedly, with varying errors like Gateway Time-outs or IMAP errors. I guess I'll have to set up another forwarding address for the time being...

      Any word on what the issue is today? Is this related to the previous problems, or the fixes that were attempted this weekend?

      Like many of those above, I've been using POBox for a long time now (more than 13 years), and have never experienced an issue of this long a duration... I am certainly willing to give this wonderful service the benefit of the doubt, but I also am very curious when stable access to MailStore accounts will be restored!

      -- Alex

      Delete
  84. POBOX is down again? Updates would be very helpful.

    ReplyDelete
  85. Suicidally slow logging onto atmail
    Looking into moving away to another provider...not sure I want to put up with this until my subscription expires...

    ReplyDelete
  86. Has pobox.com gone AGAIN! UK 16:42 Monday, 17 December.

    ReplyDelete
  87. At least Gmail is working. Six month extensions are not much use at this rate.

    ReplyDelete
  88. Conditions are going to be slow until we can get more accounts migrated onto new hardware. I apologize, but it is simply that moving a lot of data takes time. We moved as much as possible this weekend, and will continue moving as much as possible during low-load periods (i.e. overnight), but until we hit a tipping point, daytime access will be slow. We have recommended leaving a forwarding address in place since Friday, and we continue to make that recommendation until a much larger percentage of accounts have been moved to the new hardware.

    ReplyDelete
  89. I cannot login to my webmail account - all login attempts fail. I have tried logging in to the webmail from a browser and from the Thunderbird mail cleint I use to managem my pobox accout. No joy here folks!
    Also, it would seem the forward I put into the account several days ago is not forwarding all the mail from my pobox account to my gmail account. It seems mail is being dropped somewhere along the line!
    More timely updates would be useful for those of us who rely on pobox for our businesses!

    Cliff Slater

    ReplyDelete
  90. Bad here too. Very bad (IMAP access to Mailstore)....

    ReplyDelete
  91. Well.. Today its the 19th, the problem was started at 13th
    what the hell??

    ReplyDelete
  92. keep the status updates coming - transparency is definitely the best line to walk when having these kinds of issues... keep customers updated

    loyal member since 2002

    ReplyDelete
  93. keep the status updates coming - transparency is definitely the best line to walk when having these kinds of issues... keep customers updated

    loyal member since 2002

    ReplyDelete