Cleaning up after ransomware (Cryptolocker etc.).

2015-06-10 15_59_50-cryptolocker - Google Search - Internet ExplorerAfter experiencing Ransomware a few times during the past months in our corporate setup I decided to scribble down some cleanup notes and things you can do to combat this.

This guide is seen from the point of a sysadmins and thus not from an enduser, however some tricks may apply even so (depending on various factors). In addition, this guide focuses on the cleanup of the server and not the client computer, which in my opinion always should be reinstalled after an incident like this.

This guide also assume that you have Shadowcopy enabled on your server; if not then you will need to go for a restore from backup (this however also loosely covered in the guide).  See the good thing about Shadowcopy is, that as the server is not infected nor is the servers shadowcopy – you thus have quick access to non-corrupted data from here quite easily and quickly.  Client wise things are different as most ransomeware clears the shadowcopy locally to ensure against easy cleanup locally, I heard that this may fail if the user is not a local administrator on his/her pc, so you may still have a straw to cling to if this is the case for recovering the local data easily.

Background.

First, let me sum up what this ransomware is all about.

Ransomware is a special type of malware, opposed to a regular virus it is not as much aimed at spreading but more focuses on its area of business (to extort users to pay to regain access to their data).  Ransomware is often spread via phishing mails, you may receive a mail stating that you have a package at the post office (just one example) and that you need to download and open the linked file to get the details.  Once you download and run the file from the phishing mail, it will execute the ransomware software, which will run in the background encrypting your files without you noticing it (to begin with).

It is very hard protecting against malware like this, as the makers of this type of malware keep changing the software to avoid detection.  Furthermore, antivirus is only of limited help as it cannot restore files that has been encrypted.

Ransomware usually starts by encrypting local files first and then move on to server shares.

Ransomware is actually not a new thing; it has existed since the MS-dos days in some form or other. I recall a very old virus that infected your boot sector, and upon the trigger event (could be a date or a number of boots) it would delete your fat table and bring up a slot machine, if you won the game you would get your FAT table back if not everything was lost.  Same but different.

How to get your data back after it being encrypted?  Well best bet is backups, hopefully you have either backups on some USB disk or in the cloud, if not you are likely in serious problems.  You can also choose to pay the ransom and have your data de-crypted, the price for this is usually around 100€ or 100$ depending, and from what I have heard it should work quite well and reliable to get your data back this way – some of the ransomware vendors should even have kind of customer support to assist you if you have problems – but supporting organized crime hardly seem like a good idea in the long run.

Anyhow, let us move on to the “fun” part, how to clean-up a file server after a visit from a client infected with ransomware.

So you have been struck by Ransomware (Cryptolocker, Cryptowall, Cryptodefence etc etc etc), “congratulations” and welcome to the club 🙁

Let us go through some steps to get things back on the road.

Important tip;

If you are using Shadowcopy on your server, DO NOT START CLEANUP BEFORE DATA HAS BEEN RESTORED – you may just waste storage space from your shadowcopy pool and thus be able to restore less data.

 

Step 1 – Stop the disaster from escalating.

You need to figure out which user is infected and stop this users pc from encrypting more files on your servers, if you are not fast to react your server will quickly look like this (the white is the infected files, it’s a mess).

Step 1.1 – how to identify the user

There are obviously different tactics for this, but two obvious once are;

1) look at an encrypted file and determine the owner – now to my surprise this did not work on the last server I looked at, here all the files for some reason was set to be owned by the local administrator group.

2) Look at the home folder for your users – most ransomware drop files on how to decrypt your data and these may serve as tell tail signs of “infection”.

2015-06-10 15_33_47-mRemoteNG - confCons.xml2015-06-10 15_29_46-mRemoteNG - confCons.xml

Thus, the user with all the “decrypt” files in his homedrive will be the user you are after.  Simply search the user’s folder for files with the word “decrypt” in it. The ransomware normally also targets the users local drives first, thus you may catch a lucky break if you like us have redirected the “My Documents” folder to the users home directory on the server, in our cases this meant that the infected users had tons of these files on his home share.

Step 1.2 – Shutdown the user’s computer

Shutdown the user’s computer and change the password of the user (as the user has malware on his/her computer his/her passwords (all of them) are likely now compromised.

 

Step 2 – Assess the damage

You now need to look at the server to determine how much data have been encrypted. How to determine the “infection” rate, well that depends – different ransomware uses different tactics, however at least for now they seem to share these tactics.

1) The ransomware will encrypt files, then add some extension to the file to show that it is encrypted (the extension may vary, but could be .encrypted or .iufasee or something totally different/random – but still the same for all encrypted files).

2) After encrypting a complete folder ransomware will often add 2-4 files that pertain to how to decrypt data, these files could be named “HELP_DECRYPT.TXT” / “HELP_DECRYPT.BMP” / “HOW_DECRYPT.TXT” / “!Decrypt-All-Files-iufasee.bmp” or anything like that.

2015-06-10 15_29_46-mRemoteNG - confCons.xml

NOTE: the ransomware is quite clever as not to change the creationdata/last modified date as this makes it hard to just look for files changed in the past 24h – however, as I mentioned in step two then the ransomware often creates “how to decrypt” files/pictures/links in the folders and these may be used to spot the “infection”.

My suggestion is;

  1. Try to determine the file extension using the tips above.
  2. Use Windirstat to get an idea of the scope of the incident (you can see an example below) http://windirstat.info/
  3. See screenshoot (the white is the encrypted/infected data).

cryptolocker

 

Step 3 – Restoring data (the non-encrypted files)

See we had a special challenge with restoring data as we use online backup, and the restore hence will take a LONG time seeing that the data need to come from the WAN restoring gigabytes of data would take a LONG time, so we had to get creative to make the cleanup as fast as possible.

You first need to determine the time for the last backup/shadowcopy snapshot before the “infection” occurred.

If you have shadow copy, then go back through the snapshots to find the time where files had their original extension. You may get best results if you look at the infected users home folder, this is likely the first folder to be “infected” (you can also look at the creation date/time of the “how to decrypt” files which may give you a lead).

2015-06-10 15_29_46-mRemoteNG - confCons.xml

If you have local backup it is quite easy I guess, just restore more or less all data (with the do not overwrite newer/changed versions option set) and then proceed to delete the encrypted data and the “help files” (the once on how to decrypt) – see section below on how to cleanup.

If however you cannot easily restore data from backup (like e.g. if you use “online backup” like we did), then move to shadowcopy (which you hopefully have enabled on the server).

You could of cause restore one file/folder at the time from shadowcopy, this will take forever especially if users have worked on the folder structure meanwhile. So why not make it fast and easy by using robocopy (yes it is actually possible to use Robocopy, we found a cool way to do this).

Restoring non encrypted data via ShadowCopy and Robocopy.

  • Determine the “last good” shadowcopy, the one just before files started to be encrypted.

 

    1. On the server list the shadowcopy snapshots using the dos command, you do this to get the “identifier” which we will need in a moment.Start an administrative command prompt and issue the command;
      vssadmin list shadows
      (you may need to change drive to the drive you want to see)This will give you a long list of available snapshots, see screenshot.
      2015-06-10 15_00_00-mRemoteNG - confCons.xmlLook for the creation time and find the block just before the incident occurred.

      In this block “Contents of shadow copy set ID {…….}” look for the line “Shadow Copy Volume”, copy this line to a notepad starting with \\

      In this example;
      2015-06-10 14_56_13-mRemoteNG - confCons.xml

      \\?\GLOBALROOT\Device\HarddiskVolumeShadowCopy107
      NOTE: the number at the end will be different for you.

      IMPORTANT! Now add a “\” to the line in notepad: \\?\GLOBALROOT\Device\HarddiskVolumeShadowCopy107\

      Finally add a prefix of “mklink /d c:\restore ” to the line in notepad.
      So the final line should look like this;
      2015-06-10 15_12_14-mRemoteNG - confCons.xml

      mklink /d c:\restore \\?\GLOBALROOT\Device\HarddiskVolumeShadowCopy107\
      (note: the c:\restore is a folder/name YOU choose, it can basically be anything you choose, the name must NOT exist before you run the command)Now run this command from the administrative command prompt.
      2015-06-10 15_09_35-mRemoteNG - confCons.xml

      It should give you a feedback much like;
      symbolic link created for c:\restore <<===>> \\?\GLOBALROOT\Device\HarddiskVolum eShadowCopy107\

      2015-06-10 15_13_13-mRemoteNG - confCons.xml

      Now if you write;
      dir c:\restore
      you will have a historic view of how the disk looked at the time of the shadowcopy snapshot, you could get the same via properties “previous version”… but this is much neater as you can access and script it.

  • Now we have the snapshot mounted we can run a robocopy job restoring any data that is not more recent or changed.In this example the command would be something like;ROBOCOPY C:\restore D:\ *.* /XC /XO /E /LOG:d:\restore.log
    2015-06-10 15_18_12-mRemoteNG - confCons.xmlYou will need to suit it to your environment.

    Things to make a note of are the /XC /XO command switches which ensures that we do not overwrite files modified after the “infection”. As the encrypted “infected” files have a different extinction this is not a problem.

    After the restore you can review the restore.log file to see if anything went wrong and see how much data was restored.

    Note, you MAY run into the problem that not everything was in shadowcopy in which case you have to revert to backups, in the incidents we have had “only” 10-20 gb of data was “infected” and our shadowcopy could easily accommodate this.

 

 

Step 4 – CleanUp

Final step is to clean up the encrypted files and the decrypt instructions.

Also remove the “directory link” to the shadowcopy snapshot if you used that (see previous section), you can just use “RD <directory name>”.

2015-06-10 15_13_13-mRemoteNG - confCons.xml

I used SearchMyFiles from http://www.nirsoft.net/ as it is easy and very customizable to use to find files, I suggest you take not more than 10.000 files at the time as deleting many files takes quite some time.

2015-06-10 16_41_29-2015-06-10 10_41_17-mRemoteNG - confCons.xml.png - Windows Photo Viewer

 

Mitigation strategy

  • On fileservers, try to limit access as much as possible – if nothing more than look at making data read-only wherever possible as this alone will protect you greatly.
  • FSRM – File Server Resources Monitor, set this up to detect and trigger alarms on new files where the word decrypt is part of the name – decrypt as part of a filename is uncommon enough to give only limited false alarms – I will create a separate article on the configuration of this later.
  • Supporters / super users – instruct them to react FAST to tell tail signs of ransomware, the faster you manage to stop the “infection” the less to clean up.

 

Tools that may be useful;

Decrypt Cryptolocker (this most likely will not work, but give it a go anyhow just in case).
https://www.decryptcryptolocker.com/

Windirstat                                     http://windirstat.info/
SearchMyFiles                              http://www.nirsoft.net/

Read more about Cryptolocker; http://en.wikipedia.org/wiki/CryptoLocker

Thanks to:

Torben Slaikjer for finding that link on how to mount shadowcopy snapshot as a directory, this made the job vastly easier.

Altaro’s big Christmas Backup giveaway

If you are working as a sysadmin and are working with Hyper-V you may be eligible for the Altaro’s great Christmas giveaway of 50 free licenses for Altaro’s Oops!Backup.
http://www.altaro.com/hyper-v/50-free-pc-backup-licenses-for-all-hyper-v-admins/
All you need to do is to fill out the form (name, email and a screenshot of your Hyper-V manager), you sadly need to use a “company” email but other than that it worked without a glitch and I got my 50 licenses within 1 hour.

Oops!Backup is a basic though fairly full featured backup software solution, it’s aim seem to be end users so it’s very simple to use and setup, but with a bit of tweaking you can configure it somewhat for advanced use also.

I’ll see about creating a small walk through video if I can get myself into gear 😐

A few screenshots;

SugarSync – another cloud service

Just found an alternative to www.dropbox.comcalled SugarSync, it offers a free version as with Dropbox and from what I can read with more space to begin with.

Comparisons made by various reviewers on the net seem to suggest that Dropbox is slightly easier to get started with, however pricing on SugarSync appear cheaper.

Had Dropbox been 1-3$ and not 10$ a month I had been a paying customer, but 10$ for 50gig is just too steap 🙁

Check out SugarSync here;
https://www.sugarsync.com/free/

a few oter alternatives here;
http://techpp.com/2010/07/05/dropbox-alternatives-sync-files-online/

And finally (note the price is per year not month);
http://emea.trendmicro.com/emea/products/personal/safesync-solution/

Windows Server Backup – NTBackup replacement

So I was looking into the possibilities to backup my Hyper-V “farm” (well I do got a few servers running on my Windows 2008 R2 Hyper-V server ;-)), and I came to think of good old NTBackup from Windows 2000/2003.  NTBackup has however been replaced by “Windows Server Backup” which is both good and not so good, I think that the new product is likely more professional in some senses however it has some strange ideas on how to do some of it.

Anyway, I have only looked at it for a couple of hours but here are my findings, maybe you’ll find them useful in getting started yourself.

First off you must install “Windows Server Backup” on Windows 2008 R2 this is a feature – so add new feature …….

Then you would likely like to setup a new backup schedule, and here is one of those strange tings I mentioned.  The MINIMUM interval for a backup schedule on Windows Server Backup is 24 hours, that mean that the job HAS to run at LEAST every 24h – Now this may be well and good in some setups, but I don’t need a backup of my Virtual servers every 24h!?!?  My poor server would die under the workload 😀  But in the GUI there is NO way around this 🙁  I found a web-cast where a command-line workaround was outlined, but I might not have payed enough attention because I could not get it to work and thus sought alternate solutions. 

MY workaround for this scheduling issue is this;

  • Start Windows Server Backup
  • Create a “Backup Schedule” (it’s a wizzard, so it is quite easy)
  • Now close Windows Server Backup
  • Goto – “Start menu”, “Administrative Tools” and select “Task Scheduler”
  • Drill down to; “Task Scheduler Library”, “Microsoft”, “Windows”, “Backup”
  • Now edit the job “Microsoft-Windows-WindowsBackup”
  • Edit the “Trigger” and you can now set the job you created to launch whenever you want

I’m sure there are other ways, but this is easy and works.

Some screenshots;

And finally some light reading for further details 🙂

http://blogs.technet.com/b/filecab/archive/2009/04/13/customizing-windows-server-backup-schedule.aspx

http://blogs.technet.com/b/filecab/archive/2009/06/22/backup-version-and-space-management-in-windows-server-backup.aspx

Web-Cast (quite informative);
http://www.netometer.com/video/tutorials/windows-server-2008-backup/

I might drill further into this at a later point, but for now this seem to satisfy my needs.

Driver backup – yet another cool util

In a recent post (http://www.kanmandet.dk/?p=1877) I mentioned Driver Magician Light the free version of Driver Magician, now it would seem I managed to find yet another product that does the same completely for free (and this not as a limited light version)..

Double Driver (odd name, but hey..)

Get it here;
http://www.boozet.org/dd.htm

An older review here (older version);
http://dottech.org/freeware-reviews/7225

Also there is still DriverMax;
http://www.drivermax.com/
However I never really liked this product, it’s complicated navigating and as I recall it requires some kind of registration (free as I reacll, but I don’t like having to register anyhow).

Outlook auto PST file backup

I have been trying to recall the name of this little addon to Outlook for ages now, a few of my friends really need this to keep a valid backup of their Outlook PST file..  Fianlly I rediscovered it..

The problem is that some use online backup services, and once they start windows they launch Outlook – subsequently the online backup is unable to ‘lock’ their PST file and their Outlook data file newer gets backed up..

This little add on will a specified intervals create a backup copy which will obviously not be ‘locked’ as an open file and can thus be backed up without issues.

Hopefully the Outlook 2010 version will become available soon?

http://www.microsoft.com/downloads/en/details.aspx?FamilyID=8b081f3a-b7d0-4b16-b8af-5a6322f4fd01

By applying the fix below it should be possible to make the older versions work with Outlook 2010;
http://support.microsoft.com/kb/2030523

DriverMagician – Backup drivers

I recently had to reinstall my pc, one of the damn things about this is drivers 🙁  you have to download and install drivers not contained on the install media. In my case I also spend many hours debugging why my system became unstable, it turned our to be the NIC driver that was now included on the install media but sadly utterly unstable 🙁   So hours and days later I managed to find the old driver I used before and all was back to normal.

Anyhow, just wanted to ‘remind’ you all about a nifty little utility that can assist you in backing up your drivers;
http://www.drivermagician.com/Lite.htm

Note this is the link to the LITE (FREE) version, it is fine for just backing up your drivers, but if you can spare the cash you can go for the automatic payed version.

Harddisk cloning on the fly (Free)

We all know of different image/cloning software, this one however is free AND can image on-the-fly via VSS (meaning you can run it while you work, well more or less naturally)..

I was suggested this by a friend (R. Tastesen) as just the tool for my forthcomming harddisk upgrade to a hybrid disk..  it seems quite promising..

http://www.macrium.com/reflectfree.asp

Mounting ISO files

Once in a while you get hold of an ISO file (an ISO file is basically just an image of a CD/DVD/BLUE-RAY disk) and need to access the content of this file, there are several ways of doing this;

1) You can burn the ISO file to a media and read it this way.

2) You can extract the ISO file to a folder using a utility (WinRar is very good at this).

3) you can mount the ISO file as a virtual CD/DVD eg. use Deamon tools lite.

I would suggest option 2 or 3.

Anyhow, David Davis has written a nice beginners guide on the subject, so be sure to check it out at;
http://www.windowsnetworking.com/articles_tutorials/Mounting-ISOs-Windows-7-Free-Tools.html

Buddy backup – “free” online backup

Now this is somewhat clever, I have not yet tried it so I can’t say for sure how good it works but the concept is clever.

You agree with some of your friends that you will use each other as backup-hubs, install Buddy Backupand then define each other as trusted friends, then your data is backed up to your friends computers via P2P (of cause in encrypted format so your friends can’t look at your data)..

It is free and rather clever.

Read their getting started guide here

Download it here  (sadly it is not available right now, a new version should be very close to release – even so close that they removed the old version – I have written to them requesting a release date but not received any yet).