Tracking USB First insertion in Event logs

The tracking of USB removable disks has been discussed and analyzed in detail with the usual methods of looking at the windows registry for plugged in devices (USBSTOR keys), registry shell bags, SetupApi logs, etc.

A while back researching something else I happened to hit upon an artifact not known for this purpose, the 'Windows Event Log'. The first time a USB device is inserted into your windows PC, it is logged in a little obscure log which is maintained for the 'ReadyBoost' functionality. This is only true for Windows Vista and above, as XP did not have ReadyBoost. For more information on ReadyBoost refer here. 

Whenever a new drive is connected to a windows system, windows will test that drive's read/write speed by creating a file on that drive and then deleting it. And this result is logged in the ReadyBoost log. From an investigator point to view, this does not give us too much information about the connected disk, but it does give some useful information, notably the name of the disk, sometimes the size (as shown in pic below) and the date/time when device was first connected to that system. This should correlate to the SetupApi log date/time. 

ReadyBoost Operational log under Windows Event Viewer

The messages are usually under EventID 1000-1023 with 1015 and 1016 being irrelevant (performance calculations for booting). It even logs the devices that are not disks such as 3G dongles and non-USB devices such as mounted VHD files with messages such as these:

This was for a partition on a mounted VHD file.
The date/time of log matches the date/time when partition was created.

When a new 3G dongle was plugged in..

When an IronKey was plugged in..

With this artifact, we have one more thing to confirm the date of first insertion of a device. This should be useful in cases where sometimes the registry keys make it difficult to confirm dates or device names/types.

The full path of this event log file on the system is
'C:\Windows\System32\winevt\Microsoft-Windows-ReadyBoost%4Operational.evtx'.

In the windows event viewer, you can view this log under
'Applications and service logs\Microsoft\Windows\ReadyBoost\Operational'. 

vinetto and ubuntu

Vinetto is an open source software for viewing/extracting thumbnail information from Thumbs.db files. It is listed in the ubuntu repository and installing it from Ubuntu Software Centre is as easy as installing an application from the Apple App Store (or Google Play Store), meaning it requires no knowledge of linux! So far so good!

However out of the box, vinetto seems to have an issue running on certain types of Thumbs.db files (which it calls Type 1a files). For certain Thumbs.db files, you  may encounter an error that halts on line 338 of vinetto at im.split(). It seems to originate from the python imaging library (PIL) complaining about a null image object. This is using vinetto 0.07 and all the latest libraries, PIL 1.1.7 and Python 2.7 with ubuntu 11.04. The problem stems from a bug in PIL which does not load the image file on demand (as it should) when operation are performed on the image. The fix involves explicitly calling the load() function. We will need to edit vinetto's source and compile/install it. Since its a small python script, there isn't much of a compilation process (phew!). Download the vinetto source from here.

On line 338 (of file vinetto), you should see the im.split() function being called. You will need to insert an additional line here and add the line im.load(), so it looks like this:

This should fix the PIL error. Now re-install using the procedure mentioned in the vinetto INSTALL file.

This did solve the crash issue for the problematic Thumbs.db files, but for those files vinetto's extracted thumbnails were in the negative (inverted). Another few lines of code had to be added to fix this and the final edited file can be downloaded here.

I would be interested to know if anyone else has encountered similar issues. There is another unrelated issue, which has to do with Windows 7 generated Thumbs.db files. This I have documented in another post here.

Windows 7 generated Thumbs.db

Windows 7 does not use the Thumbs.db file for folder thumbnail caching. However when accessing remote or mapped drives, it does create a Thumbs.db file in the remote folder when it viewed in explorer.

A quick way to create this file would be to browse your own C drive using the path '\\localhost\c$' and see explorer drop Thumbs.db everywhere as you click around into folders. You may have to change the folder view to Icons or Large Icons for it to trigger.

The format is slightly different and no tool seems to handle it correctly as of now. The new format is still an OLE document, however microsoft have gotten rid of the 'Catalog' stream, and the names of other streams are strings which include the resolution (96, 256, ..) and a 64bit hex number. The stream contains a regular jpeg file, which usually begins at 0x18 (on xp this was 0x0c). The stream header also begins with 0x18, a likely indication of the beginning of data.

Regular carving for JPG files will work against it. The good news for investigators is that the behavior of this file is similar to xp Thumbs.db, ie, the thumbnails do not get deleted once the file (image) is moved/deleted from that folder. This is different from usual windows 7 behavior where deleted files are quickly removed from the Thumbcaches.

Windows 7 Thumbcache hash algorithm

What we know
Windows 7 (and vista) store thumbnails in a central database known as the thumbcache in the files thumbcache_32.db, thumbcache_96.db, thumbcache_256.db, thumbcache_1024.db, thumbcache_sr.db and thumbcache_idx.db. The format(s) for these files has been reverse engineered well enough to be able to read and extract the thumbnails.

What is yet unknown is the hashing algorithm used to generate the ThumbnailCacheId. I did some research of my own to fill this gap.

ThumbnailCacheId
To generate the ThumbnailCacheId windows uses the volume guid (that the file resides on), the fileId (for NTFS volumes), the extension of the file (.xxx) and the file last modified time (as a DOS GMT date). These are ‘blended’ and mangled using a hash function one by one. I believe the easiest way to describe this would be to simply show the code. Feel free to copy this code into your applications.

UInt64 CalculateHashKey (UInt64 seed, byte[] buffer, uint buffer_length)
{
  int count = 0;
  if (buffer_length) {
    do
    {
      seed ^= ( (seed >> 2) + (2080 * seed) + buffer[count++] );
    } while (count < buffer_length);
  }
  return seed;
}

UInt64 GetThumbnailCacheId (byte[] VolGUID, byte[] FileID, byte[] FileExtension, byte[] FileModTime)
{
  UInt64 hash = CalculateHashKey (0x95E729BA2C37FD21, VolGUID, 16);
  hash = CalculateHashKey (hash, FileID, 8);
  hash = CalculateHashKey (hash, FileExtension, FileExtension.GetLength() * 2);
  hash = CalculateHashKey (hash, FileModTime, 4);
  return hash;
}

Volume GUID can be found in the registry SYSTEM hive under ‘MountedDevices’. On a live machine, this is HKEY_LOCAL_MACHINE\SYSTEM\MountedDevices. If for example, your guid is ‘\\?\Volume\{ 234cf70e-a70c-11de-a48c-806e6f6e6963}’, then your buffer will have the value “0EF74C230CA7DE11A48C806E6f6E6963”.

FileID (or File Reference number) can be obtained from the MFT. Note: The FileID shown to you by tools such as Encase is simply the MFT Record number (without MFT Sequence number bytes). Read more about it here.

The extension buffer is Unicode text starting with the dot(.) as “2E006A0070006700" for “.jpg”. This is case sensitive.

For the DOS time, you can easily convert the file’s modified date from a FILETIME 64 bit value to the GMT DOS timestamp of 4 bytes.

Because this is a hashing scheme requiring so many inputs, it is not possible to reverse a ThumbnailCacheId back to a full path, which is what investigators would really love. Still, now you know how to validate your thumbnails against their respective existing files.

Blackberry Date Formats

Timestamps within Blackberry IPD files follow 3 different date formats depending on which database they are stored in. At first it seems pretty odd, but then it’s probably the work of independent programmers/teams working on different databases and associated software for the device.

1. Call Log, Phone History and SMS Date

The date used in these databases is a Java date 8 bytes in size. Java dates are similar to unix timestamps, with only a multiplier and larger size (8 bytes instead of 4 for unix) for better precision.

Javadate = 1000 * UnixTimestamp

Hence to convert back, simply divide the 8 byte timestamp by 1000 and you get a unix timestamp.

2. Calendar Date

This date’s precision is only to the number of minutes. It is the number of minutes since 1 Jan 1900 0:0:0. Converting this date to standard unix timestamp (which starts at 1 Jan 1970) means we have to subtract the difference equivalent to the number of minutes between 1900 and 1970.

Taking into account the 17 leap years also between 1900 and 1970, we calculate the difference.

Difference = (Number of years * Minutes per year) + (Number of Leap Years * Minutes per day)
= (70 * 365 * 24 * 60) + (17 * 24 * 60)
=36816480 minutes

Our simple formula for conversion will be

UnixTimestamp = (CalendarDate – 36816480) * 60

3. Email Date

The dates used in email metadata (sent and received dates) are stored as 2 byte date value and 2 byte time value.

2 byte Date

The date is stored with upper 7 bits representing the year since 1996, ie, 1 = 1997, 2 = 1998 and so on. The next 4 bits represent the month (1-12) and the remaining 5 bites represent the day (0-31).

+----------------+----------------+

|      BYTE 1    |      BYTE 0    |

| 7 6 5 4 3 2 1 0 7 6 5 4 3 2 1 0 |

  \___________/ \_____/ \_______/

       year      month     day

2 byte Time

Time is stored with a precision of 2 seconds. That means that 14:05 will be represented as 14:04 and 14:06 as 14:06. The creators did not think that this level of fine precision would ever be required for an email’s metadata date. One reason for this is that you really need 17 bits to represent all possible values when hours, minutes and seconds are stored separately, 5 bits for hours (0-23), 6 bits for minutes (0-59) and 6 bits for seconds (0-59). But only 16 bits are available in 2 bytes, so the BB developers have dropped a bit from the seconds field, hence only 5 bits are used as shown below.

+----------------+----------------+

|      BYTE 1    |     BYTE 0     |

| 7 6 5 4 3 2 1 0 7 6 5 4 3 2 1 0 |

  \_______/ \_________/ \________/

     hour      minute       sec

Enscript Tutorial 1 - Parse XP System Restore Logs

This post is about enscript programming using a real world example, parsing of the windows XP restore point logs to extract original filename and file path information of files collected by System Restore. Link to complete code file is at the bottom of this post.

System Restore
In windows XP, the system restore functionality creates backup copies of files under the "<DRIVE>\System Volume Information" folder. These files are identical copies of the original in data and metadata except for two details, the filename and path. The filename consisting of “name. extension” has the correct extension but the name is now "Axxxxxxx.ext" where x denotes a number. So commonly files like A0000001.ini, A0000002.txt and A0000005.lnk are seen here.

For analysis, we need to know the filename and full path of the Axxxxxxx files. This information can be parsed from the change.log or change.log.x files found in the same folder. We shall not discuss the format of the change.log file as it is documented elsewhere on the web in detail. Instead we shall focus on how to write a program to accomplish this task using enscript.

How to parse file name and path from change.log?

We shall use a shortcut instead of reading the entire structure of the file. After studying the format, I have understood that the path is present in a structure that begins with the signature (hex string) "12EFCDAB" and is 64 bytes from the signature. We shall therefore search for the hex string "12EFCDAB" and then skip 64 bytes to get to the path string. 8 bytes beyond the path is the filename. This will be our strategy.

Searching in enscript

The method to search is not a one liner and takes some setup code. It is roughly 4 steps:

1. Create SearchClass object
2. Add keyword(s)
3. Open a file and use the search object to find hits
4. Iterate through the hits array to get details of every hit

Program logic and concepts

We have broken down our program into a set of small manageable tasks for ease of understanding.

1. Recurse folders to find change.log.x files in System Volume Information
   
   The change.log files will always have a full path resembling
   "\System Volume Information\_restore{GUID}\RPxx\change.log.x"
   We can use this information to craft logic as follows:
   
   
   forall (EntryClass e in c.EntryRoot()) {
     if (e.Name().Find("change.log") == 0 &&
         e.Parent().Name().Find("RP") == 0 &&
         e.Parent().Parent().Name().Find("_restore{") == 0)
     {
     // Found change.log, now process it here
     }
   }


2. Search file for our artifact signature (12EFCDAB)
   
   SearchClass search();  // create SearchClass object
   search.AddKeyword("\\x12\\xEF\\xCD\\xAB....\\x15\\x00\\x00\\x00....", KeywordClass::GREP);   // add 16 byte GREP keyword
   search.Create();       // initialize search object
   EntryFileClass file(); // create file object
   file.Open(entry);      // open entry as file
   search.Find(file);     // execute the search on the file
   foreach (SearchClass::HitClass hit in search.GetHits()) {
     // Process search hits here
   }
   


3. Parse artifacts and print to console
   
   Once we have the hit object, we can seek into the file at the appropriate offsets and read out the data.
   
   file.Seek(hit.Offset() + 64); // seek to 64 from hit offset
   String fullpath;
   String newfilename;
   file.SetCodePage(CodePageClass::UNICODE); // codepage set to unicode
   file.ReadString(fullpath);    // Read path
   file.Skip(8);                 // Skip 8 bytes
   file.ReadString(newfilename); // Read name, ie, Axxxxxxx
   Console.WriteLine(newfilename + "      " + fullpath); // Print information to console

Sample script output

Click here to download the complete enscript code file.

Travel Log format

I have just finished putting together the format specification for IE's RecoveryStore and Travel Log (Travelog) It is now uploaded and available. Get it from the Downloads page. 

Blackberry IPD Research - The “Phone History” Database

On newer Blackberry phones, a db called the “Phone History” is now present (in IPD files). Anyone who has tried to parse a blackberry (BB) IPD file (or messed with the advanced syncing options in the BB Desktop Manager) will know that it consists of several databases internally. The IPD file is like a TAR archive of all these databases. The database that stores all call history information is the “Call Logs” database. Most tools (free and commercial) all deal with the database just fine. Now RIM has now introduced a new database called “Phone History” which also stores Call log information, albeit not in its entirety. On RIM’s website, the only description for this database is:

 “Stores information pertaining to phone call history with specific participants (complete history of incoming and outgoing phone calls with selected recipients)”

But the IPD file I have with me does not seem to adhere to RIM’s definition. After some discussions with the folks at the DFIR mailing list, Chris Pavan (42 LLC), Shafiq Punja (who has been a keen researcher of IPD files for a long time) and the CTO of Oxygen Forensics (which claims to parse this db but did not work for my ipd), it seems that the exact purpose of this db is yet unknown. Shafiq has also seen this and tells me that RIM snuck in this new db somewhere in the beginning of 2011. Oxygen Forensic's CTO had this to say:

“What we think about this DB is that its data is used as a dictionary of phone numbers used for substitution when a user is typing a new message and entering the number. So even if a record with this phone number has been deleted from the log this number can be used for substitution. Also, it looks like that durations for each record reflect cumulative length of all calls related to this phone number, not a single call. So, this DB is not a kind of copy of the Call Log but a separate data...”

Analyzing the “Phone History” Database

What I’ve noticed is that not every call has an entry here, only the last time a call was made. So if there were 100 calls but only to 10 unique numbers for which calls were received/placed/missed, then 10 entries would be found here each storing the last time called. You do find multiple entries for the same number at times, but they are separated by a month or more. Regardless of how or why RIM puts these entries here, the fact is that they are present and are very good sources of evidence of call activity. The normal "Call Logs" db only has calls for 30 days, but this database has entries that are much older. BB allows a user to selectively delete call records, these have a 1:1 relationship with the "Call Logs" db, so if a user deletes a call record, it gets deleted there too. But since a user has no interaction with the "Phone History" db, if an entry was made here by the BB internally, then it stays put, in effect, giving the investigator "deleted call data". This db captures the date & time of call, phone number, contact name and call type information, only not the call duration!

Technical Specifications for parsing “Phone History” db

The format of each DB record header (same for every record in IPD file) is
2 byte DB ID
4 byte Length
3 bytes version + record Handle
4 bytes record ID

This is followed by the individual fields, having this format:
2 byte Field Length
1 byte Field Type
x byte Field Data (x = Field Length)

Each record within the DB consists of a header and several fields. Fields present in the "Phone History" db are shown below.

ID	Field
1	0x73 always
2	Unknown, could be number of times called
3	Date/Time of Last Call (Format is BB date, same used in Call Logs)
4	Call Type (PLACED = 1, RECEIVED = 0, MISSED =2)
5	Phone Number (ASCII String)
6	0 or 0xFFFFFFFFFFFFFFFF only
7	Unknown Record UID of contact?
8	Always 1
9	DB Record UID
10	Name of Contact (ASCII String)

Below is a db snippet in a hex editor for a single record.

Please drop me an email or a comment if you have anything to share regarding the "Phone History" db.

Update 2012-01-10: Oleg (CTO of Oxygen Forensics) says their next version coming out next week will support parsing and displaying the results from this db. Personally, I look forward to this, it is a great tool for smartphone forensics. He also provided new information, Record ID=7 is the uid for contact.