LastPass Security Notification

Update 10, May 16th, 3:20pm EST - Final update to this post, we'll make new posts going forward

Actions we've taken:

• Multiple security experts and firms were brought in to help us, we've engaged one firm to do a further source code based review.
• We're committed to doing several reviews per year and sharing the results of these reviews.
• We've had some useful suggestions from the community -- we appreciate your input: https://lastpass.com/support_security.php
• One example: to reduce the chance of phishing Iastpass.com was registered -- that's a capital i instead of an L. We've also purchased 1astpass.com
• All non-core services have been completely removed from the LastPass network; LastPass now runs the web application and DNS servers only.
• Forums, Helpdesk, etc are run offsite on 3rd party servers.
• We're looking into moving our support tickets off our network too.
• Amazon was utilized to send out the email notification; we're better able to send large amounts of email quickly in the future, and thank Amazon for working to spin us up quickly.
• We've commissioned a write only off-site aggregated log server which can only be accessed via the console. This will allow us a guarantee that any logging is intact.

The good:

• We were prepared to both disable accounts and force people through password changes, which was something we had planned for.
• The steps we took protected all users, even those who used weak master passwords.
• Having a live backup system proved invaluable for people who ran into issues, or forgot their new master password after changing it.

We made a number of tactical errors including:

• Out of the gate, we inconvenienced a large number of people who knew their password were strong and therefore never could have been at any risk.
• Massively underestimating the amount of media attention we'd receive. This had 2 effects: 1. Greatly increased the number of users attempting to change their passwords -- our plan was for people coming from new computers which is a small percentage of the overall user base per day that we could have handled; 2. Drove a big increase in new users as people interested in LastPass attempted to check us out.
• We didn't have any previous IP tracking data on previously used computers for people without login tracking. This caused nearly all these people to face password change immediately.
• We moved too slowly to shut down password changing once the system was under stress.
• We weren't prepared to send large amounts of email quickly, especially after turning off a server. (Resolved going forward w/ Amazon)
• Some of our customers were unfamiliar with logging into LastPass in offline mode, panicing a number of them.
• Blogger (who we use for blog.lastpass.com) had some downtime through the event.

Additional changes coming:

• Our next release will make it clear how to login offline from the login dialog.
• We've purchased a large amount of additional server capacity so we can handle extreme load events better in the future.
• We'll be utilizing the 'from a new location' capability in a few new security features.

Update 9, ~11am 05/09 EST:

Many users are changing their password and then determining they can't remember it, a number have also run into issues with password changes and want to go back, you can now do this yourself without contacting us: https://lastpass.com/revert

It allows you to either roll back your last password change or revert your account to the 4th. You must prove access to your email again to use it.

Update 8, ~9am 05/07 EST:

We enabled password change to greater percentages overnight and now to all users. Again please note that there is no need to panic, all accounts were put into a locked down mode of only allowing previous login locations or verify via email, until password change.

We're asking any users that have current issues with a password change to use https://lastpass.com/revert to restore you from backups. Many have been people forgetting what password they changed to so make sure you practice that new password a number of times after you change it.

We appreciate your patience, we'll continue to update with any changes.

Update 7, ~6pm 05/06 EST:

Everyone should be able to login (after verifying your email if you are coming from a new IP). We've begun allowing all premium users and a percentage of users to go through password change.

Please note that there is no risk in waiting if you can deal with verifying by email when you use a computer at a new place (IP).

If you experienced an issue with a password change and want to be restored from backups we can do that too and will provide a URL to do it shortly.

Update 6, ~10:30am 05/06 EST:

If you have been experiencing an error contacting the server, please try logging in both via the plugin and the website - you should now gain online access. If you still see an error, please open a support ticket or email support@lastpass.com, if you haven't already done so.

Currently we're not allowing users to change master passwords until our databases are completely caught up and we have resolved outstanding issues. We will update our users via the blog when it is possible to do so.

Thank you for your continued patience.

Update 5, ~1:30am 05/06 EST:

We've added the option for you to say that you know your master password is strong and to avoid password change, we apologize for not having that available when we announced.

We've identified an issue with roughly .5% of users that impacted their master password change, and will be contacting you tomorrow rolling you back to before the change.

Our focus right now is on ensuring we can resolve users with issues, we'll continue to provide updates here.

Update 4, ~10pm EST:

Joe's interview with PCWorld covers more details on what happened, what our thought process has been, and what this means for our users: http://www.pcworld.com/article/227268/exclusive_lastpass_ceo_explains_possible_hack.html.

We continue to work as quickly as possible to address user support.

Update 3, ~4:30pm EST:

Logging in offline should be working everywhere if you have logged in using that client before, if you're having problems with this please attempt to login via the website: https://lastpass.com/?ac=1 that should now take you through an email process to enable your current IP.

If you're having problems getting your data with pocket, make sure you're selecting to login to the local file, not logging in at LastPass.com.

If you changed your password and are now having problems we'll help with that too, please email us if that's the case and include your LastPass email address.

For those who haven't been prompted, and have continued to use LastPass without issue -- we've judged the risk to be low if you're using the same IP -- we're only raising the issue once that changes.

Finally if you have issues with password changes please email us at support@lastpass.com, we can revert you, or we can pull data from backups, but please try LastPass Icon -> Clear local cache first.

Update 2, 2:15pm EST:

Record traffic, plus a rush of people to make password changes is more than we can currently handle.

We're switching tactics -- if you've made the password change already we'll handle you normally.
If you haven't the vast majority of you will be logged in using 'offline' mode so you can still use LastPass like normal and get back to your day, only syncing of new password should suffer (and you'll see the bar).

As load lowers we'll increase the percentage of people being sent through email validation / password changing.

For people experience problems please email us at support@lastpass.com -- we have seen a few reports of bogus data post change, we think this is due to you downloading a stale copy and if you go to LastPass Icon -> Clear Local Cache and try again it should work.

You can access your data via LastPass in offline mode or by downloading LastPass Pocket : https://lastpass.com/misc_download.php (choose your OS).

---

We noticed an issue yesterday and wanted to alert you to it. As a precaution, we're also forcing you to change your master password.

We take a close look at our logs and try to explain every anomaly we see. Tuesday morning we saw a network traffic anomaly for a few minutes from one of our non-critical machines. These happen occasionally, and we typically identify them as an employee or an automated script.

In this case, we couldn't find that root cause. After delving into the anomaly we found a similar but smaller matching traffic anomaly from one of our databases in the opposite direction (more traffic was sent from the database compared to what was received on the server). Because we can't account for this anomaly either, we're going to be paranoid and assume the worst: that the data we stored in the database was somehow accessed. We know roughly the amount of data transfered and that it's big enough to have transfered people's email addresses, the server salt and their salted password hashes from the database. We also know that the amount of data taken isn't remotely enough to have pulled many users encrypted data blobs.

If you have a strong, non-dictionary based password or pass phrase, this shouldn't impact you - the potential threat here is brute forcing your master password using dictionary words, then going to LastPass with that password to get your data. Unfortunately not everyone picks a master password that's immune to brute forcing.

To counter that potential threat, we're going to force everyone to change their master passwords. Additionally, we're going to want an indication that you're you, by either ensuring that you're coming from an IP block you've used before or by validating your email address. The reason is that if an attacker had your master password through a brute force method, LastPass still wouldn't give access to this theoretical attacker because they wouldn't have access to your email account or your IP.

We realize this may be an overreaction and we apologize for the disruption this will cause, but we'd rather be paranoid and slightly inconvenience you than to be even more sorry later.

We're also taking this as an opportunity to roll out something we've been planning for a while: PBKDF2 using SHA-256 on the server with a 256-bit salt utilizing 100,000 rounds. We'll be rolling out a second implementation of it with the client too. In more basic terms, this further mitigates the risk if we ever see something suspicious like this in the future. As we continue to grow we'll continue to find ways to reduce how large a target we are.

For those of you who are curious: we don't have very much data indicating what potentially happened and what attack vector could have been used and are continuing to investigate it. We had our asterisk phone server more open to UDP than it needed to be which was an issue our auditing found but we couldn't find any indications on the box itself of tampering, the database didn't show any changes escalating anyone to premium or administrators, and none of the log files give us much to go on.

We don't have a lot that indicates an issue occurred but it's prudent to assume where there's smoke there could be fire. We're rebuilding the boxes in question and have shut down and moved services from them in the meantime. The source code running the website and plugins has been verified against our source code repositories, and we have further determined from offline snapshots and cryptographic hashes in the repository that there was no tampering with the repository itself.

Again, we apologize for the inconvenience caused and will continue to take every precaution in protecting user data.

The LastPass Team.


Update 1:

We're overloaded handling support and the sheer load of password changes is slowing us down. We've implemented a way for you to verify your email and then not be immediately forced to change your password for that IP, access from any other IP would bring you back to email verification. You can now wait a few days if you know you'll be on the same IP without loss of security, and due to this overloading we think that's prudent to wait.

We're asking if you're not being asked to change your password then hold off -- we're protecting everyone.