Aug 12, 2014

Update on LastPass Connectivity Errors

At 3:57 Eastern Time this morning, one of the data centers that LastPass relies on went down. Our team immediately took action to migrate LastPass to run entirely on a different data center. As a result, many users experienced connection errors with the LastPass service, and LastPass.com has been intermittently unavailable throughout the morning. We have been engaged with our data center provider the entire time to resolve the issues. Please note this does not impact the security of your data.

We are doing everything we can to mitigate the impact and resolve the situation as quickly as possible, and apologize for the inconvenience caused. We strongly recommend users login through the browser extensions to access their vault, where most users should have access though some may still see warnings that they are in “offline mode”.

We will continue to update our user base and appreciate your patience.

Update: 1:28 pm EST

Though one of our data centers remains completely down, the service is generally stable and should be available to the majority of users (with the exception of login favicons). Some users may see connection errors but should still be able to access their data. We continue to work as quickly as possible to get the service back to 100%.

Update: 4:13 pm EST

Most users should now be able to connect to LastPass browser extensions and LastPass.com without errors, though favicons still may not sync. We continue to closely monitor the situation.



August 13, 2014: Post Mortem of Yesterday’s Outage

As noted in our original post, on August 12th, 2014 a data center that LastPass relies on went down around 4 am Eastern Time. Below, we have outlined the timeline of events as they unfolded at the data center and with the LastPass service at large.

We again sincerely apologize for the inconveniences caused, and want to assure our community we are moving forward stronger than before, as we remain deeply committed to the security and reliability of our service for our users.

Joe Siegrist
CEO of LastPass


Summary of Events

The majority of users were unaffected due to having proper redundancy in place to deal with the loss of a data center, as well as the built-in offline access via the LastPass browser extensions. However, during our efforts to scale at the secondary data center to ensure sufficient capacity at peak of the day, we inadvertently worsened the situation through human error. Our team certainly has takeaways from the experience and will be implementing changes going forward, as detailed in the concluding statements below.

We did receive a full RFO from our data center confirming that the BGP routing table issues affecting other companies yesterday played a role as well. For more, see: http://www.zdnet.com/internet-hiccups-today-youre-not-alone-heres-why-7000032566/

Timeline of Events (EDT)

3:50 am - We detected extreme latency and packet loss between one of our data centers and most major networks, including inter-connectivity with the other data center.

3:54 am - Our monitoring system detected the situation as critical and paged two operators.

4:00 am - We contacted our data center provider regarding the issue we were experiencing with their service.

5:00 am - With no update from our impacted data center provider, we switched from two data centers to run entirely on the second data center and disabled the affected data center.

6:00 am - We noticed IPv6 has suddenly started working at the now-disabled data center, making it clear to us that major networking changes were being made.

7:00 am - Our report was escalated by the impacted data center provider.

8:00 am - We determined that the outage will likely be extended, so we executed on a plan to add some spare machines into load balancing at the second data center to ensure we would have plenty of spare capacity at the peak of the day.

8:15 am - We began to receive alerts of intermittent connectivity issues at our second (now only) data center.

8:30 am - A small percentage of users reported logout errors that prevented them from utilizing offline mode.

9:00 am - We continued trying to work with our impacted data center provider, but received no updates on the situation or information on resolution.

9:30 am - Latency and connectivity issues increased at the second (now only) data center, which we began investigating.

10:00 am - We received acknowledgement from our impacted provider indicating this is a widespread problem, and indicated they would reload the core routers. They noted that it may be an extended outage.

10:30 am - The impacted data center's network went completely down.

12:00 pm - We tracked down the source of an issue at the second data center, in which 3 machines we had added were running at 100Mbps instead of Gigabit (despite having Gigabit cards and being connected to Gigabit switches) and were network saturated.

12:45 pm - We resolved the issue with the 3 additional machines, and fully restored service still running on the second data center only, though favicons remained disabled.

2:15 pm - Impacted provider indicated they were fully online, though those machines remained unreachable for us.

2:30 pm - We authorized the impacted data center staff to reboot our networking equipment, with no effect.

3:30 pm - We discovered the underlying issue with why some users are being logged off immediately after login and resolved.

3:45 pm - Members of our team arrived at the impacted data center, and verified that our networking equipment was still down.

4:15 pm - We completed a swap to spare equipment, bringing the impacted data center back online.

8:45 pm - We completed testing and confirmed that replication to secondary data center looked good, and were fully restored with both data centers active again.

Conclusions & Lessons Learned

As a result of yesterday’s events, we have formed the following key takeaways and action steps:

  • We have moved our status page to be hosted outside our network, since it was inaccessible for periods of time.
  • In an effort to gather more detailed information for our community, we delayed communicating about the situation. Going forward, we will share what information we have, however sparse, and work to update the community from there, via the blog, the status page, our social accounts, and email where appropriate.
  • Our monitoring checks now verify port speed:
for i in `ip addr show | grep UP | egrep -v 'tun[0-9]+:|lo:' | awk -F ': ' '{print $2}'`;do echo -n $i ; ethtool $i | grep "Speed:"; done
  • We are considering moving to another data center provider.
  • In an effort to improve the situation, we worsened it through our actions, and we will be more cautious in taking preventative actions when running on a single data center.
  • We're moving to a hosted model for DNS that includes external service checks. 
  • Though we designed some systems to be 'non-critical', such as favicons for sites, we'll be improving our systems to minimize visual disruption during a massive outage.
  • A small number of users were impacted by an inability to access the service offline, we continue to investigate and test this.
  • We will be implementing more disaster and redundancy tests of our systems to better prepare for a catastrophic, single data center scenario. 

202 comments:

  1. Could you please give times in a format like "today Aug 12th at 10:00 PST" or similar ?
    "Early morning" is such a vague term for a company with customers all over the globe ...
    Thanks

    ReplyDelete
    Replies
    1. Thanks Eduardo, we've updated the post to reflect this.

      Delete
    2. While we are talking timestamps, I would encourage you to adopt and advertise a convention compatible with a standard like ISO-8601, if only to preclude potential ambiguity of month and day for your international customers (e.g. 08/10 used on adjacent frame).

      Delete
    3. Was this due to the 512k routes problem that happened today?
      http://www.nux.ro/archive/2014/08/512k_routes_ought_to_be_enough_for_everyone.html

      Delete
  2. I am unable to log in through the website / plugin. How long is this going to take?

    ReplyDelete
    Replies
    1. Please try again through the browser extension, you may see a notice that you're in offline mode only - please try again through the extension only and let us know if problems continue.

      Delete
  3. Logging in through the browser extension does not work at the moment. Major bummer that the one time this feature is needed, it fails!

    ReplyDelete
    Replies
    1. Please try again through the browser extension, and let us know if you see any further problems.

      Delete
  4. This is a disaster. Even Firefox had a melt down, crashed and I had to clear cache so no off lien option either. The plug in icon has also disappeared from Firefox toolbar.

    ReplyDelete
    Replies
    1. Maybe the add-on has been disabled. Check to see if the plugin in disabled following the instructions located at https://support.mozilla.org/en-US/kb/re-enable-add-ons-disabled-when-updating?esab=a&s=Enable+Addon&r=0&as=s. If that does not help try downloading the LastPass Firefox add-on again from https://addons.mozilla.org/en-US/firefox/addon/lastpass-password-manager/.

      Delete
    2. If LastPass is no longer listed in your Firefox Tools > Addons, then yes please reinstall and try logging in again.

      Delete
  5. Great. So I can't do any work as I can't log in to the various sites. Makes me re-consider using online password manager at all.

    ReplyDelete
    Replies
    1. Funny you didn't consider that before...

      Delete
    2. Neither of you seem to be aware that if you logged in before, the addon stores an offline copy. https://lastpass.com/support.php?cmd=showfaq&id=956

      Delete
    3. Please try to login again, as the service continues to stabilize. If you're using the extensions that you've logged into before, at the very least the offline storage should be working. Please let us know if problems continue: https://lastpass.com/supportticket.php

      Delete
    4. It's just easier to complain when you don't understand it - or pay for it.

      Delete
  6. You need to take a look at your setup after this and make true redundancy. Because while short downtime is fair and to be expected from time to time, a downtime like this, which is 4 hours now, is disappointing.

    ReplyDelete
    Replies
    1. Thanks for the honest feedback, we'll certainly be taking another look.

      Delete
  7. Not good LP... Not enough redundancy at all built-in here me thinks.

    ReplyDelete
  8. Seems the servers are responding just enough so that I cannot use offline mode but not enough for lastpass to work.

    ReplyDelete
    Replies
    1. Indeed, I just experienced this myself briefly. LastPass needs to implement an affirmative "OFFLINE" option, one that the user can toggle themselves, rather than simply relying on whether one has network connectivity or not. I can't just turn off my internet connectivity wily-nily to deal with this issue. While typing this, my Yubikey auth window popped up a second time, even though I've been in online mode for the last ten minutes.

      Please please please add a button to "GO OFFLINE".

      Delete
    2. Please try again, as we've continued to work to resolve this it should be available for the majority of users. Please let us know if any problems continue: https://lastpass.com/supportticket.php

      Delete
    3. It's been working for me for the last couple of hours.

      A little disappointing that this happened, but still love LastPass. It would be good to be able to force offline mode without disconnecting the network connection. Not sure if that makes sense our not though.

      Delete
  9. I think this highlights the fact that LastPass needs more redundant data centers. I can't do work because I cannot access a web account that contains the files I need.

    ReplyDelete
    Replies
    1. Indeed, I just experienced this myself briefly. LastPass needs to implement an affirmative "OFFLINE" option, one that the user can toggle themselves, rather than simply relying on whether one has network connectivity or not. I can't just turn off my internet connectivity wily-nily to deal with this issue. While typing this, my Yubikey auth window popped up a second time, even though I've been in online mode for the last ten minutes.

      Please please please add a button to "GO OFFLINE".

      Delete
    2. (sorry, that reply was meant for a different response)

      Delete
    3. Offline access is enabled by default though users with multifactor authentication may have chosen to disable it. We'll certainly be looking at improvements moving forward.

      Delete
    4. The problem isn't that offline access is available, it's that when there is an interrupted/glitched network like in this case, offline access would not 'kick in' reliably for me. Being able to force lastpass into offline mode would have prevented the partial access I experienced.

      Delete
  10. yep this is pretty bad, especially as it has taken so long an still not fixed. they should have a backup they could have switched immediately too - this is the 1st time its let us down but its still a big concern

    ReplyDelete
  11. wow,

    no access to any of my sites.

    Bad time when I used the password random generator. =/

    ReplyDelete
    Replies
    1. Please try again, and let us know if any problems continue.

      Delete
  12. Still can't access LP...

    ReplyDelete
    Replies
    1. Please try again, and let us know if any problems continue?

      Delete
  13. I would be just fine if I could get offline mode.
    But I can't seem to get into offline mode with my yubikey.

    ReplyDelete
    Replies
    1. Yubikey does work in this situation. What error are you seeing? I've been able to login to LastPass using the Yubikey all morning. LastPass shows a message saying that I'm in "offline mode" but I still have access to 100% of my passwords, secure notes and other data.

      Delete
    2. Is was something like can't connect to server.

      :: Steps ::
      Enter user and pass
      and waited for the Yubikey popup.
      Popup would take over 5mins to come, thats if it even came.
      If it came, I would pressed key and wait for a few mins to get the error.

      Anyways, thanks Yubico and Lastpass for great products.
      hope this info can help you guys.

      Delete
    3. Please try logging in again, though if you disabled offline login in your settings it would be expected that you can't connect. However the connectivity issues users were seeing should be resolved in most cases, please let us know if you try again with any further trouble.

      Delete
  14. How is this *not* globally load balanced across multiple geographically dispersed data centers? Has LastPass just been winging it all this time getting lucky they didn't have their SPOF datacenter go down? Please say it ain't so.

    ReplyDelete
    Replies
    1. Exactly my thoughts too.

      Delete
    2. Thanks for the thoughts, we hope to detail this in a post-mortem.

      Delete
  15. Err, you guys might want to ask your IT guys to implement something called High Availability, SaaS really shouldn't be going down like this.

    ReplyDelete
  16. I hope Premium users will be compensated for this.

    ReplyDelete
    Replies
    1. LastPass is ONE DOLLAR PER MONTH, people. You would ask to be compensated half a cent?

      Delete
    2. Compensated? Give me an effin' break......

      Delete
  17. Guys, you will need to think about an offline solution, just in case this happens again... BTW I can't login with my google account. Too bad :(

    ReplyDelete
    Replies
    1. The LastPass browser extensions do support offline login, appreciate the feedback - please try logging in again and let us know if you still cannot login?

      Delete
  18. As an Enterprise user this is causing me to seriously rethink using this because everything that is SAML'd through lastpass is now unavailable.

    ReplyDelete
    Replies
    1. I second that...as we started an enterprise trail last week and this occurred. Fortunately we are currently only testing with 3 users. I can't even begin to imagine the disaster this would have been had this happened after we deployed it out to 2000+ users.

      Seems a little misleading calling it Enterprise with out and HA functionality in your datacenters.

      Delete
    2. Thank you for the feedback and we apologize for the concern caused, please let us know if we can address any questions at sales [at] lastpass.com.

      Delete
  19. Yikes. I used the export function to save my Lastpass data as a CSV file. A minute later my vault redisplayed as empty. Got it just in time.

    ReplyDelete
    Replies
    1. Full access should be available for most users, please let us know if you try again with any further trouble.

      Delete
  20. Decent built-in redundancy can almost eradicate issues like this.. Down times of a few minutes or so are to be expected, & are accepted by most folk. However downtimes of many hours for a system such as this is not.

    ReplyDelete
    Replies
    1. Thanks for the thoughts, we do hope to detail this in a post-mortem.

      Delete
  21. I look forward to reading the postmortem on this one...

    ReplyDelete
  22. This is a personal disaster. I've been a paying subscriber for several years and my "life" is stored in 64 character passwords in lastpass. This event will cause me to replace this once-trusted provider with another solution.

    ReplyDelete
    Replies
    1. ...until the other solution has a few hours of downtime, too...

      Delete
    2. This is EXACTLY the reason I keep a parallel copy of ALL MY accounts in a separate offline password manager called KeePass 2 (free opensource, well respected). Yeah, it takes a few extra steps maintaining a 2nd password manager, but I don't create new accounts every day so it's no biggie.

      Delete
    3. Have you continued to try to login with any further errors? As we've continued to work to resolve, most users should have full access, please let us know if problems continue.

      Delete
  23. This seems like a pretty basic type of outage to be able to account for. Luckily I have no essential needs for this morning. Going to research alternatives to LastPass now, with no reliance on their servers.

    ReplyDelete
  24. It is working for me now, using offline mode under Firefox. But have to agree with other comments here, time for LP to rethink their redundancy architecture.

    ReplyDelete
  25. Tip: If you can connect to your Vault, do an "export"...you'll get a page with what looks like a comma-separated file view of all your sites including IDs/pwds. Thought that might help some with urgent needs.

    My Chrome extension is failing intermittently, so this works for me. YMMV

    ReplyDelete
  26. What am I missing here folks? I have all my sites and notes accessible on both my phone app and in my browsers on multiple machines. It shows as being offline, but everything is encrypted and stored locally so I'm busy working.

    ReplyDelete
    Replies
    1. Same here. Happily working away.

      Delete
    2. I was fine also, using phone and local store.

      Delete
  27. Not good...trying to buy stock and can't get into my account. This is costing me $$

    ReplyDelete
    Replies
    1. Have you tried logging in again with continued problems? Most users should now have full access, please let us know if we can be of further help: https://lastpass.com/supportticket.php

      Delete
  28. Service has been down for hours. This greatly calls into question the stability of your platform as a whole. Not having redundancy in this day and age is amazing and concerning. If your organization cannot ensure data is available when a data center goes down (redundancy) how do we know you are really protecting our information as well as you publicly claim?

    ReplyDelete
    Replies
    1. Thank you for the honest feedback, and we do hope to address this in a post-mortem.

      Delete
    2. I will judge you by the postmortem. If pablum "things happened" we know you failed to build it right. If detailed and we can see it's an inherent weakness in SaaS, we had a bad day and will Keep Calm.

      Delete
  29. Hrm. This is why you need more redundancy -- I'm shocked you guys dropped the ball on this, is this not what failover is for? I would hope you have more than two co-locations for the size of your business. If not, check out here (downtown dallas):
    http://www.databank.com/company/contact.html
    One of the best datacenters I've ever been in, hands down. If I remember correctly they've never had an outage, and host many large companies such as Second Life and Layered Technologies.

    ReplyDelete
    Replies
    1. Thanks for the thoughtful feedback; we do have redundancy in place, we hope to have a post-mortem, we certainly have key takeaways and actions to be better prepared.

      Delete
    2. Awesome. I love that you responded to every single person, I have yet to see a company do that for over 156 comments on a blog and respond as quick as you have. Hopefully you take these comments to heart, every service has outages but only the good ones learn from them. Definitely look into high availability if not already in place, and make sure you test it regularly.

      Delete
  30. I've only got a handful of passwords in offline mode; most passwords are not there! After the service is running again, how about a refresher lesson on how to get offline passwords synced with online passwords?

    ReplyDelete
    Replies
    1. Please try again, most users should have full connectivity again. Offline access is enabled by default, if you attempt to connect and it's not available it should bump you to offline automatically (unless you've cleared browser cache, or have disabled this in your LastPass settings).

      Delete
  31. Our entire team is dead in the water.. Can't get into our sites. Really annoying. No way redundancy is in place.

    ReplyDelete
    Replies
    1. Please ensure everyone is logging in through the LastPass browser extensions, if you or any team member tries again and continues to have problems please get in touch with us: https://lastpass.com/supportticket.php

      Delete
  32. Looks like KeyPass it is!

    ReplyDelete
  33. Please note this absolutely compromises the security of my data. I do not keep a local copy (for obvious reasons) and the inability to change a stored password, instantly, whenever I deem necessary, IS a security compromise.

    Get your thinking clear on this, Lastpass.

    ReplyDelete
    Replies
    1. The local copy should be stored by default - do you mean you've disabled this option completely? Moving forward, we recommend taking a look at Pocket to fill this gap for you: https://helpdesk.lastpass.com/lastpass-on-the-go-2/lastpass-pocket/ - let us know if we might be of further help.

      Delete
  34. The users of Lastpass here, should be aware that all electronic can break down, and you should have a local backup ready always. Still, it is not good on Lastpass side either, but you should have been prepared for such a situation.

    ReplyDelete
    Replies
    1. Nope - I don't want a local copy - I want one single remote copy. Oh, that is secure. And resilient.

      Delete
    2. Then you will have to suffer the consequences when the service breaks down, it is your own choice.

      Delete
    3. Yes, I guess it is, but some of us, have created layers of security such as two step logins, when the system is not able to ask you fro the secondary access key, it doesn't matter what kind of back up you have.

      Delete
  35. This is amateur night at lastpass. I can't get on to sites without resetting my password (which I'll have to repeat, once / when / if the thing comes back).

    No redundancy for a system that is relied upon for access to hundreds, thousands or millions of sites? Ridiculous.

    ReplyDelete
    Replies
    1. Thanks for the honest feedback, we do have redundancy in place and hope to have more details in a post-mortem, we've certainly gained key takeaways and will use them to be more prepared going forward.

      Delete
  36. "a percentage" is hilarious at best and deceptive at worst. It SOUNDS like it means a small percentage, but it doesn't actually say that and could LITERALLY mean anything. Does it mean 99%? 100%?

    ReplyDelete
    Replies
    1. We did update our language here, as we agree it was too vague.

      Delete
  37. As useful as LassPass is, it desperately needs a rock solid offline option as well. The separate 'Laspass Pocket' application isn't anywhere near suitable to handle LastPass downtime. A little more innovation would be very welcome.

    ReplyDelete
  38. https://agilebits.com/

    1password

    ReplyDelete
  39. So much butt hurt here.

    ReplyDelete
    Replies
    1. LOL, this ^

      And as I type this, Lastpass browser extension finally logged in. Hopefully the outage is over.

      Delete
  40. "intermittent" and "a small number of users" is a material understatement of the severity of the problem #shootusstraight - this is a major screw up folks...

    ReplyDelete
    Replies
    1. We appreciate the honest feedback, and have updated the language to better reflect the situation.

      Delete
  41. I was finally able to log into my account... just to find that there are NO PASSWORDS there anymore!!!!!!!!!!!! is there an issue not showing the passwords? did I lose them forever? what the hell is going on LastPass?!!!!!!!

    ReplyDelete
    Replies
    1. Relax, they will come when they have updated their user bases. Read the above post from Lastpass

      Delete
    2. No, the passwords should not be lost, please logout and login again. If any syncing issues continue please get in touch with our support team: https://lastpass.com/supportticket.php

      Delete
  42. If someone really need to log in somewhere and have access to their e-mail. Perhaps you can reset your password in the meantime?

    ReplyDelete
  43. Claims about lack of redundancy and inability to reset passwords are unfounded and belie a misunderstanding about how LastPass functions. You automatically have a local cache of your database and you may continue to update as needed; when service levels are restored, this local cache should synchronize to the server and be available to other devices. There is also no security risk (esp. regarding changing passwords) as the local cache should continue to function. The distributed model established by LastPass is exactly what one would want during a service interruption of this nature.

    ReplyDelete
    Replies
    1. Local cache does not work. Have no password even in offline mode.

      Delete
    2. Except it doesn't seem to work this way. I can enter my master password in with Chrome, occasionally I'm asked to enter my 2-step authentication password, and then nothing happens. I still don't have access to any of my passwords. I thought LastPass has some offline functionality, but apparently LastPass hasn't done a proper job in telling me how to use it.

      Delete
    3. They do a fine job. They just assume reading comprehension on the part of their users. Too bad they over-estimated you. Offline mode has been working just fine for me thanks. It is supported in two-factor modes too.

      Delete
    4. That is exactly my problem, I am not able to login because the plug in never asks me for my 2-step verification, much less push the request to DUO. I guess being extra careful didn't pay out this time.

      Delete
    5. While offline mode has worked nicely in the past, there was something different about this outage. I didn't try disconnecting my computer from the internet to see if that let me in, but in the past when I haven't had internet lastpass worked fine in offline mode. With this outage I only get errors. I suspect it would switch to offline mode if I disconnected from the internet, but it's not a big enough deal right now to try it.

      Delete
    6. If you have multifactor authentication enabled, you may have disabled offline access. When you are able to access the service navigate to your LastPass Multifactor Options in your LastPass settings. Make sure that the "Permit Offline Access" option is enabled.

      Delete
    7. Enabling offline access will cancel the 2-step authentication benefits.
      When offline access is enabled, you can unplug your ethernet cable and open your offline cached vault copy with just your master password.
      It would be less secure.
      So, instead of suggesting using the offline access, LP should prevent this kind of outage happening again.

      Delete
    8. Anecdotally it does appear offline access did not work in 100% cases, and that's something we're looking into. We do have redundancy in place; we certainly have a number of takeaways from this and will only be more prepared moving forward.

      Delete
  44. High Availability / Disaster Recovery was obviously not implemented - for a SaaS service, that is truly disappointing as its pretty much an industry standard nowadays. How can we take LP seriously anymore with such an obvious oversight. If they missed HA/DR, wonder what else they've missed?

    ReplyDelete
  45. I'm sorry but everyone here is overreacting.

    ReplyDelete
    Replies
    1. Well, if "everyone" here is (over)reacting and you're not then, statistically, you're in the minority.
      I'm sure as hell not over-reacting; not being able to get into a bunch of important sites when I need to, and not wanting to reset all my passwords, is a sign that I have better things to be doing than waiting for lastpass to return.

      Delete
    2. The majority weren't impacted. As long as you know what you are doing, you wouldn't have noticed an issue, as I didn't notice the issue.

      Delete
  46. Situations like this are a company killer. Looks like LP needs upper management that knows technology and how to really run a company.

    ReplyDelete
  47. It is very concerning to all of us that LastPass does not have a disaster recovery center. I am sure that all of the users that need passwords to important sites or applications to continue their daily activity are already following the process of recovering their passwords through the individual sites or services they use, and that can be annoying, but that is part of the territory, Things like this will happen from time to time (hopefully few and far between).
    What is very disappointing, is the lack of communication from lastPass. For a company that deals with sensitive information an expects trust from their users, they are certainly not offering the same in return, and that is a serious blow to our confidence on their service and product.
    LastPass you need to seriously look at the relationship that you are building with your client base, anyone that is willing to spend money and effort on internet security has at least a rudimentary understanding of how things work, and we want to be informed, specially if it affect us. Sometimes the service and product is not as important as the respect you offer to all of us.

    ReplyDelete
  48. The plugin keeps wanting me to login. Everyone few moments it does this (or makes me use my Yubikey). Then it tells me password invalid. Lastpass doesn't work right now... except to send me an email that I'm locked out for the next 5 minutes. :P

    ReplyDelete
    Replies
    1. I'm able to login using the Yubikey all morning. LastPass shows a message saying that I'm in "offline mode" but I still have access to 100% of my passwords, secure notes and other data.

      Delete
  49. Can surrvive till LP comes back up, question is my data safe in the mean time?

    ReplyDelete
    Replies
    1. Yes, there are no security issues at this time.

      Delete
  50. Do people truly not realize how complicated these types of systems are? No system can be up 100% of the time. It's not the fault of companies like this that their users don't take a moment before they signup to think about what happens if the service goes down. EVERY WEBSITE AND SERVICE EXPERIENCES OUTAGES! Some are bigger than others, but users need to understand that these things will happen. My banks website was down for two hours the other day, last week there was a problem with an accounting website I use, last month I was unable to access package tracking data for a period of time one day because of a problem. We, as users, need to start realizing that no service is perfect and it is up to us to make alternate plans. Especially with a something as important a password, there should always be a backup plan. This is life in the 21st century, welcome!

    ReplyDelete
    Replies
    1. i agree; LP is effectively a free service, and nothing is going to be 100% reliable forever... People simply don't care about the difficulty and complexity on the back-end and simply love to feel entitled...

      Delete
  51. Yes sites experience outages. However when you have a service like LP, redundancy is expected and why they still require manual intervention for an HA fail over means they do not have the right people for the job in there. Consequently we are paying for a service and have higher expectations

    ReplyDelete
  52. Looks like the service is up again, but obviously not for my DNS...
    http://www.isitdownrightnow.com/lastpass.com.html

    ReplyDelete
  53. I'm only blaming myself for not keeping my hard copy list of password up to date.

    On the other hand, I'm going to make the same mistake again eventually.

    I may have to switch back to using my IronKey.

    ReplyDelete
  54. What do you expect us to do? Keep a second copy of all the data?

    I get that there are outages. That's fine But when you have over 200 sites (everything!) in there and can't log on to banking, twitter, or anything it can be annoying. I needed the password to connect to my media server and it was in Lastpass. Fortunately I had exported the data and was able to open it with LastPass Pocket but this is less than ideal.

    I thought LP would work off-line when the server was down which would be fine but that's not working either.

    ReplyDelete
    Replies
    1. i had the same problem, so many passwords but not able to access any of them.

      if the problem was on my end, that's one thing. but for it to be on the service of another, that's.. another.

      keeping backup copies is nonsensical, and a waste of time. i'd have to make a copy after every password change just to keep the backup 100% usable, and that is NOT happening.

      Delete
  55. I have to disagree with these postings in defense of LastPass. LastPass DOES NOT store your passwords in the "cloud". They store a one-way cryptographic "hash" of your master password in the cloud. Your passwords are local on your device and are unlocked using that hash. They can be unlocked other ways though too. For instance, LastPass offers several mechanisms to keep your encrypted password blob with you and locally on your computer. My recommendation: setup multi-factor authentication with Yubiko's 'Yubikey' product per LastPass' instructions on their site. Optionally, use the LastPass Pocket detailed here: https://helpdesk.lastpass.com/lastpass-on-the-go-2/lastpass-pocket/

    ReplyDelete
    Replies
    1. Interesting as Lastpass states THEY DO, encryption is what is done locally.

      Delete
    2. Rubbish (nia, not Anonymous) - if you are seeing local passwords then they will be the browser that is serving them up, not lastpass.

      The passwords are stored in the cloud, encrypted with your master password / hash. Otherwise, how would you be able to get a password down to, say, a new PC? It comes from the cloud.

      Delete
    3. Lastpass DOES keep an encrypted copy of your data in the cloud. This is what allows it to keep all your devices in sync such as another computer, phone, etc.

      Delete
    4. They explain the "salted hash" concept that they use, where passwords never actually reside in their datacenter, at this link: https://lastpass.com/how-it-works/ An even better explanation of the specifics is on the Security Now! podcast from a few years ago: https://www.grc.com/sn/sn-256.htm

      Delete
    5. Folks, please educate yourselves by listening to the detailed podcast I mention above, and by reading the LastPass site. I would NOT have been able to gain access to my passwords this morning, while Lastpass was down, if the passwords were stored in the cloud. I gained access to the passwords because I used my yubikey USB one-time password device to decrypt the LOCALLY stored passwords. There you have it.

      Delete
    6. So how do I log into a wiped-daily PC, go to lastpass.com, enter my password and auth code, and see a list of all my passwords then? I can guarantee that they aren't stored locally on that PC.
      Answer: they are stored in the cloud.
      Doesn't mean that you can't have a local copy. Yo can,and I do on several PC's, but they are also available through a browser.

      Delete
    7. I think there's a semantic misunderstanding here. A copy of your encrypted password "blob", as I've heard it called, is on the LP server. They can't decrypt it even if served a subpoena. That same blob is also local on your computer. By contrast you CAN decrypt it independently without any access to the LastPass servers in their datacenter.

      They go into more detail at the link below but here's how LP describes this:

      "All encryption/decryption occurs on your computer, not on our servers. This means that your sensitive data does not travel over the Internet and it never touches our servers, only the encrypted data does."

      https://helpdesk.lastpass.com/getting-started/introduction/why-is-lastpass-safe/

      Delete
    8. Only when your cryptographic 'blob' is downloaded to your device, be it the browser at a cybercafe or to your iPhone, does the clear text password become visible. I probably should have worded my initial post as such:

      "LastPass DOES NOT store your [unencrypted] passwords in the cloud. Your passwords are encrypted in a datafile that is local on your device, with a copy of that encrypted datafile periodically backed up to the LastPass server."

      I'm sorry for not wording it better originally. My original point was to show that LastPass does not make its users depend on access to the Internet.

      Delete
    9. GRC.com is a great resource. I will know what to think of all this when The Explainer in Chief speaks.

      Delete
  56. Before moving to a different provider keep in mind that LastPass should be fully aware that their customer base can't handle two outages. I would think that LP should be one of the most stable options after this downtime.

    I use the LastPass Pocket as well to keep a backup of my passwords. To bad the password I need was created last night and I didn't back it up. *sigh*

    ReplyDelete
  57. It's back up for me. Try to log in.

    ReplyDelete
  58. Finally.
    This has been a disaster, lastpass, and shown you up to be incompetent and amateurish.
    It would be interesting to hear what the cause of "one of our data centers has failed" was - if it was a breach then that will be the end of you.

    ReplyDelete
    Replies
    1. Sometimes failures happen. Grow up.

      Delete
  59. Shame on you Lastpass

    ReplyDelete
  60. Amazing the quantity of whiners commenting here...ignore them, they'll say the same bullshite about their next password manager. Just do your best to get things fixed and then figure out how not to have it happen again. :) Also, I did not even know there was an outage...I've been logging in fine all day...Quit whining and offer support and practical solutions.

    ReplyDelete
  61. This is the first time I ever had a service break since I started to use LastPass. And I am a user ever since XMarks stopped.
    But most important: the service is up and running again!

    ReplyDelete
  62. Need an update. What time scale are we looking at?

    ReplyDelete
    Replies
    1. Hi Andre: You should be able to login again, if you try with any problems please let us know.

      Delete
  63. Lots of strong words from customers enjoying a quality product for an extremely low cost (or free). I would like better geo redundancy from Last Pass, but ultimately appreciate the convenient service and have my own alternatives for fault tolerance.....and I appreciate it being inexpensive.

    ReplyDelete
  64. I hope all you complainers out there are premium users...

    I agree with Anonymous 1:18 wholeheartedly.

    This is the first time in my several years with LastPass that the servers went down.

    Keep up the great work!

    ReplyDelete
  65. Enterprise last pass customer: and I want to say to LastPass...we still LOVE you guys and your service. You have saved us way, way more than 12 hours by allowing us to efficiently manage passwords. And for an unbelievably reasonable cost.

    I'm sure you guys will learn & adapt from this experience. Keep up the good work!

    ReplyDelete
  66. Welcome to the "Cloud". This is what you will continue to have with the almighty cloud computing. If you can't stand a day of downtime every few months, don't use it. Idiots.

    Lastpass... I understand how things go. Bad things can happen. You still kick ass.

    ReplyDelete
  67. is very very good Thanks you . Enterprise last pass customer: and I want to say to LastPass...we still LOVE you guys and your service. You have saved us way, way more than 12 hours by allowing

    ReplyDelete
    Replies
    1. Thanks for the positive feedback, we do apologize for the inconvenience caused.

      Delete
  68. Xmarks sync has also failed, presenting all sorts of different and unusual sync failure messages. It Xmarks part of the same problem ? Or is Xmarks having an unrelated problem ?

    ReplyDelete
    Replies
    1. Yes, Xmarks.com was affected as well. If sync errors continue please get in touch with our team with details: http://buy.xmarks.com/supportticket.php?lpnorefresh=1

      Delete
  69. 3 years later, 1 significant outage and communication went out immediately. Lastpass will continue to get my money, thanks !

    ReplyDelete
  70. Been using LP all day in UK with 2FA. All working fine in the UK, didn't know there was a problem until a few minutes ago when I had a friendly and helpful email from LP explaining that there had been an outage.

    ReplyDelete
  71. lotta whiney people here...thanks LP!

    ReplyDelete
    Replies
    1. ...and you're okay with what LP did, because you weren't in the middle of the mess?? thought so.

      Delete
  72. I'm a Last Pass Premium account user and I pay for the service, and didn't even notice any loss of functionality.
    I use Last Pass all day, every day, and even when I was using the free service it was excellent.
    I'm sure these Last Pass guys are working with your best interests at heart, so cut them some slack and quit some of the pointless whining; some people are chuntering on as if the world has ended for them!


    ReplyDelete
  73. I'm a Last Pass Premium account user and I pay for the service, and didn't even notice any loss of functionality.
    I use Last Pass all day, every day, and even when I was using the free service it was excellent.
    I'm sure these Last Pass guys are working with your best interests at heart, so cut them some slack and quit some of the pointless whining; some people are chuntering on as if the world has ended for them!


    ReplyDelete
  74. Amber, many thanks, I am back up and running. As a Last Pass Premium member, thank you for the service. It is a big help to me in so many ways.

    ReplyDelete
  75. The complainers crack me up. Having worked in enterprise IT for the past 3 decades, I see these types of system problems more often than I'd like. For this situation, It isn't like the folks at LastPass haven't provided ways for you to backup your data. If you haven't done it, blame yourself. Learn from your mistake. As good as we try to bulletproof systems, sometimes stuff happens. That's just life. You can complain about it, or take some thoughtful action (maintain backups).

    ReplyDelete
  76. As a Premium Member, i expect the best out of Lastpass. Lastpass should always have a backup plan, I'm too busy to make backups and shit.

    I tried logging into my email but couldn't, because lo and behold, my generated 25-character password was being held hostage.

    DO NOT LET IT HAPPEN AGAIN or i will end my subscription!! >:OOOOO

    ReplyDelete
    Replies
    1. I'm a premium member, and while the site was down this morning, my offline system worked just fine. See, LP creates a vault on your system if you want it to. But thanks, I have your email password and you don't have anything of interest today.

      Delete
  77. Let this be a lesson to all the whiners. Keep an offline encrypted copy of your lastpass data.

    https://helpdesk.lastpass.com/lastpass-on-the-go-2/lastpass-pocket/

    ReplyDelete
    Replies
    1. For as often as my passwords change, it is not possible to use encrypted copies because by the time i need them, they would most likely contain old data which is USELESS TO ME.

      Now, unless you're someone who changes passwords once every year, i say go for it LOL.

      Delete
    2. I think you might be confused. He is not referring to exporting your vault as a static backup you can refer to in the future. He is referring to the feature built-in to LastPass to cache your entire vault locally so that it is available when you computer is offline. This cache is constantly updated while you use LastPass.

      Unfortunately for some of us, this offline cache was not working correctly during this outage. (see my comment below.)

      Delete
    3. Actually I was referring to an encrypted export of passwords, which will always be accessible on or offline.

      Clearly the value of this backup is a function of how often you perform the export and how often your data changes. To characterize the backup as "Useless" is an extreme position which is difficult to accept as factual.

      Delete
    4. Sorry, that is obvious to me now after reading your comment again. I agree its a good idea to keep an encrypted export as a backup. The frequency of the export would be dictated by how often your passwords change.

      I've got more than just passwords in my vault and definitely want a completely self-contained copy of my data that does not rely on LP service being available.

      Delete
  78. Sorry JP (and all the others with similar remarks), 'Offline' mode was not working as advertised in this case, for myself and others who have commented above.

    I assume offline mode is meant for occasions when your computer is offline. I don't think it is designed to deal with LastPass servers being offline, or partially offline. In my case, my browser plug-in simply did not work on my laptop. I even completely disconnected from the network, and no dice. Later in the morning I logged into my desktop, and it worked fine. Once I saw comments that the servers were back up, I was again able to use my laptop.

    I can only guess that the plug-in was able to partially communicate with the servers, but received some kind of error or incomplete communication, and this resulted in the offline cache being unavailable.

    Certainly something LastPass should investigate.

    ReplyDelete
    Replies
    1. Mine worked fine in offline mode, that's what it's intended to do.

      Delete
    2. Anecdotally it does appear offline access did not work in 100% of cases, and that's something we're looking into.

      Delete
  79. I signed up for Premium today-before I figured out something was wrong. Maybe a mistake. I am new to Last Pass and I tried about 50 times to put in my PW--then I thought it was me so I tried versions of my password. Now I fear I am locked out, I can't find a number to call on my cell, Help!!

    ReplyDelete
  80. 1. Is Lastpass working as advertised?
    2. Have I tried so many times to get into the system that I am now locked out?
    3. Can I talk to someone in Tech Support?
    4. Can I start over, with the same UN as before but somehow get a new PW? The system will not let me do this.

    ReplyDelete
    Replies
    1. Hi Steven:

      1. Yes, LastPass is fully operational.
      2. We do have lockouts in place if you try logging in too many times. Please wait at least 10 minutes before trying again.
      3. Yes, please submit a report here: https://lastpass.com/supportticket.php if we can help with any ongoing problems or questions.
      4. Yes, you can delete the account: https://lastpass.com/delete_account.php?np=1 and start over with the same email address. Any Premium credit you've already purchased would be transferred automatically if you sign up again with the same email.

      Let us know if we can be of further help.

      Delete
  81. How much of this failure is related to the "512k day" BGP outage?

    ReplyDelete
  82. 3:57 or poor server team =/

    ReplyDelete
  83. I guess I was lucky and did not experience any issues with LP today. My knowledge and experience with LP is ongoing and I know that they strive to provide the best service for password management. There were obvious issues today but I am confident that LP will incorporate changes that will make there service stronger. They learn from their mistakes and take corrective action. I am also aware that although they are ultimately responsible for their service that their are other vendors that probably had a hand in the disruption of services. Again I applaud LP in their efforts to provide us with critical reports concerning outbreaks like heartbleed and tools to help us keep our passwords secure and monitor sites we visit. As much as LP will put in the effort to make their service more Highly Available, we who need their service like the air we breath have to be more productive and make sure an offline procedure is in place and tested to satisfaction. Unless I am missing something it would be as simple as disconnecting the Internet - restarting you PC and seeing if your solution is working. I am an avid listener of the Security Now podcast with Steve Gibson who highly recommends your service. I would highly recommend contacting him with your findings and future plans so that he can spread the word. Thanks again for your outstanding service and feedback during this stressful time. Certainly much better than communications we received back from the Network Solutions outages of the past.

    ReplyDelete
    Replies
    1. So Micky, how's your job at LP?

      Delete
  84. Just to let LP know: I am near Perth, Western Australia. We are currently 12 ahead of NYC, USA.

    My LP icon is on my Firefox browser & I can log into it. But, it will not auto-log me into any website. I am currently having to access it via "edit" in the vault, then copy/paste.

    FYI

    ReplyDelete
    Replies
    1. Thanks for the report, if this continues please get in touch with our team: https://lastpass.com/supportticket.php so we can investigate with you directly, as we're not currently able to reproduce this.

      Delete
  85. Wow! The butt hurt around here is unbelievable, and I dare to bet most of it comes from free users (note that I also only use the LastPass as a free user).

    Anyway, all the people complaining about "offline mode does not work" seem to be caught in a PEBCAK type situation.

    Chrome user? Your own fault. If you didn't install LastPass from the Windows Installer but just the browser pluggie you're SOL ... the Chrome pluggie doesn't have a "binary extension" to it for the offline cache functionality. Blame Google for the world's crappiest browser plug-in system.

    Firefox/Mozilla-based User? Well, for as long as you installed the plugin having the "binary extension" you're well covered during service down-times. I had no problem during this outage at all ... just that "You are now in offline mode" notification popping up every few minutes (@LastPass - please fix that so it only displays once or remembers that it was told to take a hike).

    In short, complaining about this problem when you're a free user and you didn't install LastPass through the Installer is not only rude but you're also displaying your lacking skills at using a computer/OS loud-and-proud. If you're a premium user and you're complaining about offline mode not working, then you definitely got the proper installation of LastPass totally wrong.

    The only thing worth being pointed out is that LastPass was kinda vague with their information (note the many "language corrections" replies above - but that's also to be expected of a large company ... once they are no longer a start-up they fall into the, as I tend to call it, "tip-toeing speak"). I experienced the outage HOURS before this blog post even existed and didn't break a sweat; offline mode just worked perfectly.

    That much in defense for LastPass since I'm satisfied with the service they offer - and in case of an outage (@complainers: Remember, s**t can, and will, happen) either be sure you correctly installed LastPass or use a offline password vault (KeePass and the likes) in parallel (@"enterprise" admins/users: crying out here only displays you're the wrong person for the job as you clearly have no idea about what you're doing).

    Anyway, thanks to the LastPass team for the update on what was going on, and looking forward to the post-mortem to read up on what happened (always amusing to read about BGP sanfu's).

    ReplyDelete
    Replies
    1. I hate to burst your smug bubble, but offline access to LastPass did not work correctly for me. It is installed correctly, and I have tested it before and after the outage.

      I am a paid user, but even if I wasn't, I would still complain about it. Just because a service is free doesn't mean there should be no expectation of service quality. LP depends on their free users becoming paid users, and it is in their best interest to keep them satisfied. Hopefully they do not share your haughty attitude toward users who experienced issues.

      Delete
    2. You said "the Chrome pluggie doesn't have a "binary extension"
      Wrong, there is a binary and non-binary version on the download page.

      You said "Firefox/Mozilla-based User? Well, for as long as you installed the plugin having the "binary extension" ..."
      Wrong again. There is only one version of the FF plugin, which includes the binary.

      According to the LassPass User Manual:
      https://helpdesk.lastpass.com/getting-started/downloading-and-installing/safaribinary/
      "By manually installing a binary plugin version, the LastPass extensions in Chrome, Safari, Opera, and Maxthon can add functionality. Firefox and Internet Explorer automatically hosts these options by default."

      You said: "In short, complaining about this problem when you're a free user and you didn't install LastPass through the Installer is not only rude but you're also displaying your lacking skills at using a computer/OS loud-and-proud. If you're a premium user and you're complaining about offline mode not working, then you definitely got the proper installation of LastPass totally wrong."
      Wrong again, and also you are a complete dick.

      According to the LastPass User Manual (Offline Access to Your LastPass Vault):
      https://helpdesk.lastpass.com/password-manager-basics/your-lastpass-vault/offline-access-to-your-lastpass-vault/

      "Any LastPass browser extension or mobile application can be logged into without an internet connection, and will default to this mode when no connection is present."

      Also, Offline access is not listed as a feature of the binary component:
      https://helpdesk.lastpass.com/getting-started/downloading-and-installing/safaribinary/

      In conclusion, you are not qualified to be involved in this discussion.

      Delete
  86. Yes, blaming a user for not installing something correctly is a good choice. You're clearly helping diffuse the situation.
    Nowhere does it state in the google webstore that there's a lack of offline mode in the chrome extension. I'm assuming there will be in the future.
    The fact that this is the first time it's been a major issue is good, however clearly lastpass needs to communicate with their users better as to what functionality their products have.

    ReplyDelete
  87. Offline mode actually worked well for me. Only knew about the outage due to a Google+ post pointing to this blog. The only bug I ran across is new entries did not show up in the interface until the service was back online (Firefox/Linux). Is there a better place then this forum to submit a bug request?

    I hope you'll post a detailed postmortem of the situation. As a sysadmin I'm always interested in applying lessons learned to my own situations. Building redundant systems can truly be difficult, and I always interested when things break (the Netflix outages have been interesting reads...although frustrating to me at the time as well)

    Thanks for an overall awesome service.

    ReplyDelete
  88. Annoyed that regular support had no idea what was going on. They were making me reinstall browsers / extensions while the issue was really with the server going offline. :/

    ReplyDelete
  89. The update made available today has resolved all of my desktop/plugin issues with keeping signed in. Sh** happens and this was a good recovery considering any fail-over contingencies did not work. I would imagine LastPass will have some postmortem action items come from this that will improve service in the future.

    I have not tried the Android app, but it was also giving me issues... I hope that will also receive an update today if not already.

    ReplyDelete
  90. Takeaways from this event:

    1. Configure LastPass to work in offline mode.
    2. Regularly backup your data by doing an export to a LastPass Encrypted File
    3. Download the LastPass Pocket app for accessing your backup.
    4. Test everything
    5. CYA: Never completely rely on any cloud service.

    http://bit.ly/1usmdHG

    ReplyDelete
  91. Hi, I've downloaded and installed the software in 180 days, but not License but it free

    ReplyDelete
  92. Shell scripts are great for a quick and dirty monitoring system. But for customer-facing, HA services? C'mon. Put on your big boy pants and buy a real monitoring system.

    ReplyDelete
  93. And maybe look into that BGP thing yourselves.

    ReplyDelete
  94. It wasn't working for me... had to clear DNS cache to get it working again.

    ReplyDelete
  95. I manage a _very_ robust and redundant co-location center, would be happy to assist if you're looking for a new home or another DR location... :)

    ReplyDelete
  96. Where is the Lastpass Status page? https://lastpass.com/status.php redirects here to this blog.

    ReplyDelete
    Replies
    1. Hi Jon: We are working on a new status page - in the meantime, the blog is the best way to stay up-to-date.

      Delete
  97. INTERNATIONAL CONCEPT OF WORK FROM HOME
    Work from home theory is fast gaining popularity because of the freedom and flexibility that comes with it. Since one is not bound by fixed working hours, they can schedule their work at the time when they feel most productive and convenient to them. Women & Men benefit a lot from this concept of work since they can balance their home and work perfectly. People mostly find that in this situation, their productivity is higher and stress levels lower. Those who like isolation and a tranquil work environment also tend to prefer this way of working. Today, with the kind of communication networks available, millions of people worldwide are considering this option.

    Women & Men who want to be independent but cannot afford to leave their responsibilities at home aside will benefit a lot from this concept of work. It makes it easier to maintain a healthy balance between home and work. The family doesn't get neglected and you can get your work done too. You can thus effectively juggle home responsibilities with your career. Working from home is definitely a viable option but it also needs a lot of hard work and discipline. You have to make a time schedule for yourself and stick to it. There will be a time frame of course for any job you take up and you have to fulfill that project within that time frame.

    There are many things that can be done working from home. A few of them is listed below that will give you a general idea about the benefits of this concept.

    Baby-sitting
    This is the most common and highly preferred job that Women & Men like doing. Since in today's competitive world both the parents have to work they need a secure place to leave behind their children who will take care of them and parents can also relax without being worried all the time. In this job you don't require any degree or qualifications. You only have to know how to take care of children. Parents are happy to pay handsome salary and you can also earn a lot without putting too much of an effort.

    Nursery
    For those who have a garden or an open space at your disposal and are also interested in gardening can go for this method of earning money. If given proper time and efforts nursery business can flourish very well and you will earn handsomely. But just as all jobs establishing it will be a bit difficult but the end results are outstanding.

    Freelance
    Freelance can be in different wings. Either you can be a freelance reporter or a freelance photographer. You can also do designing or be in the advertising field doing project on your own. Being independent and working independently will depend on your field of work and the availability of its worth in the market. If you like doing jewellery designing you can do that at home totally independently. You can also work on freelancing as a marketing executive working from home. Wanna know more, email us on workfromhome.otr214428@gmail.com and we will send you information on how you can actually work as a marketing freelancer.


    Internet related work
    This is a very vast field and here sky is the limit. All you need is a computer and Internet facility. Whatever field you are into work at home is perfect match in the software field. You can match your time according to your convenience and complete whatever projects you get. To learn more about how to work from home, contact us today on workfromhome.otr214428@gmail.comand our team will get you started on some excellent work from home projects.


    Diet food
    Since now a days Women & Men are more conscious of the food that they eat hence they prefer to have homemade low cal food and if you can start supplying low cal food to various offices then it will be a very good source of income and not too much of efforts. You can hire a few ladies who will help you out and this can be a good business.

    Thus think over this concept and go ahead.

    ReplyDelete
  98. Hi thank you is very good Only Best lastpass Worlds Software

    ReplyDelete