Siri app conversations

macOS 27, aka Golden Gate featured a new Siri app, which allows a user to store their conversations with Siri unlike the default Siri invocations where they disppear after use. The new Siri app keeps a persistent history of your conversations, allows you to pin important chats, and organises them in either a grid or list as per preference.

Siri can also be interacted with from the new Unified Search bar, which is triggered via ⌘ + Space. 

Figure 1 - Unified Search bar (previously Spotlight search)

Figure 2 - Siri app conversations

Siri has now been embedded into almost all contextual (right-click) menus throughout the OS and is seen in most 3rd party App menus as well. 

Figure 3 - "Ask Siri" in the right-click menu in an email in Microsoft Outlook

Figure 4 - Reply from "Ask Siri" saved in Siri conversations

Most Siri interaction (without explicitly using the app) also results in information being stored in the Siri App. The default retention setting (at least what I had on mine) was "Forever". The other options are 30 days and 1 Year. 

This means there is now an opportunity to pull data from this database for forensic purposes. The database can be found here:

~/Library/Daemon Containers/<UUID>/Data/AgentSessionKit/AgentSessionStore.sqlite

where UUID would be a different one on every host.

Every event is stored in the ZAGENTSESSIONEVENTENTITY table and sessions are tracked in ZAGENTSESSIONEVENTENTITY.

Getting data out

A query like the one below can be used to pull out useful data:

SELECT 

(SELECT  ZBUNDLEID FROM ZAGENTPROVENANCEENTITY WHERE Z_PK=ee.ZAGENTPROVENANCE) as agent,

ee.ZSESSION as session, re.ZISPINNED as is_pinned, 

datetime(re.ZCREATIONDATE + 978307200, 'unixepoch') as session_created, 

datetime(ee.ZCREATEDAT + 978307200, 'unixepoch') as event_created, 

re.ZSUMMARY as session_summary, re.ZTITLE as title, ee.ZROLERAWVALUE as role, 

ee.ZTEXTCONTENT as text_content, 

datetime(re.ZSELECTEDAGENTPROVENANCESETAT + 978307200, 'unixepoch') as selected_agent_provenance_set_at,

re.ZSESSIONID as sessionid, re.ZSOURCEBUNDLEID as source_bundleid,

datetime(re.ZLASTMODIFIEDDATE + 978307200, 'unixepoch') as record_last_mod

FROM ZAGENTSESSIONRECORDENTITY re

LEFT JOIN ZAGENTSESSIONEVENTENTITY ee on ee.ZSESSION=re.Z_PK

ORDER BY ee.ZSESSION ASC, ee.Z_PK ASC  

Figure 5 - Query output

Siri can be asked to work with file attachments and process them in various ways. The above query will not pull out attachment related information. The files aren't usually stored in the database. They are instead referenced in the ZARTIFACTENTITY table stored usually as a JSON string which contains nested/embedded bookmark objects and plists where you can sometimes find bookmarks to files on disk.

Interesting find

There is a field ZPERSISTABLESTATERAW in the database table ZAGENTSESSIONRECORDENTITY which indicates if an entry can be removed. If this is zero, then that entry does not show up in the conversations, and can be removed at a later time.

Figure 6 - ZAGENTSESSIONRECORDENTITY table

As seen above, sessions 15 and 17 have this set to zero. There are no events retained for session 15 currently in the database, while session 17 has this data (below).

Figure 7 - Data corresponding to session 17

Automation

A plugin to read Siri App data has been added to mac_apt. It can read standalone databases too (via mac_apt_artifact_only).

Tags in Safari History db

I noticed an interesting feature in Safari's History database today. Safari tries to interpret every page you visit into a few words, represented as a Tag. But importantly, not all pages are tagged. The exact mechanism of how it arrives at a tag name is also uncertain but it piqued my interest as a case I was recently working on had some curious Tag entries in it!

This does not seem to be a new feature, as I've seen this in databases going back to at least 2021. 

Artifact location

In the Safari History db located at ~/Library/Safari/History.db

There are 2 tables of interest here: history_tags and history_items_to_tags. As the name suggests, the latter has info to join the tags to the actual history_items table entries while the former (history_tags) has the tag details.

Figure 1 - Tables holding tag data

This is what my own Safari history_tags table look like: 

Figure 2 - history_tags table content

The modification_timestamp represents the last time a page (assigned that tag) was visited. The value is of type NSDate (aka Cocoa timestamp) having epoch of 2001-01-01.

The Tag name is in the title column. Now notice that identifier column that always starts with a 'Q'. This is because Safari links every tag to a wikidata item (as shown below).

Figure 3 - The identifier field

Quick query to pull out this information along with the full history data from the database:

SELECT history_visits.title, url, 

  datetime(visit_time + 978307200, 'unixepoch') as visit_time, 

  history_tags.title as tag_title,  

  history_tags.identifier as tag_identifier, 

  datetime(history_tags.modification_timestamp  + 978307200, 'unixepoch') as tag_mod_date

FROM history_visits 

  LEFT JOIN history_items on history_visits.history_item = history_items.id

  LEFT JOIN history_items_to_tags ON history_items_to_tags.history_item=history_items.id

  LEFT JOIN history_tags on  history_items_to_tags.tag_id=history_tags.id

ORDER BY visit_time

Figure 4 - Output showing full history with tag details

Remember, not all pages have tags, and in the ones that do, the tags may not always be accurate. In my database about 98% of the time, the tag represented something related to the page (not always the central theme but definitely related). Below is an example of what a page was tagged as, but its a completely different story, both are related to planes though.

Figure 5 - Sample data from history

 

Forensic fun facts and usage

While in my own experiments, clearing the Safari history does seem to remove the corresponding entries from this table as well, I do also have some test systems where the history_tags table has old data having thousands of entries, with several having an item_count of 0 (as can be seen below). 

Figure 6 - history_tags with no associated history items due to deletion

While the above may not be the smoking gun you are looking for, it can provide some good insight of what themes the end user was browsing and when. 

On the case that inspired deeper inspection of this table, the tag was "APT" (Q230724), which got me excited only to find out that it refers to Advanced Package Tool, as in Homebrew or Linux 'apt'. But the website we were interested in was a phishing site impersonating the real Homebrew site, and it was properly tagged as APT, while the real Homebrew site got no tags!

mac_apt has been updated to add parsing of tags. Tags without item counts are shown as type "TAGGED". 

Figure 7 - mac_apt Safari output

New Wifi database from Apple intelligence

Apple Intelligence, while officially released only in 2024 (a few months ago) for macOS 15.1 (Sequoia) has been around for over a year in beta on most macOS and iOS systems. Its only available for Apple's M1 processor and later, and for macOS 15.1 (and higher). However on all macs at least macOS 14, you should have the folder corresponding to it here:

/Users/<USER>/Library/IntelligencePlatform

So even though my system is not supported, it still has the above folder. I didn't find anything too interesting in any of these databases from a forensics perspective (except for the wifi data!). But perhaps that may also be because I am not running a supported device (I'm not on Apple Silicon yet). 

The Wifi data resides in the database located here under table wifiContextEvents:

/Users/<USER>/Desktop/IntelligencePlatform/Artifacts/internal/views.db

The data is quite self explanatory, every time a Wifi network is connected to, or disconnected from, an event is created here. So far I've seen this mostly include events for the current month but sometimes these go back a few months too. It is periodically emptied.

The timestamp is just a Cocoa (NSDate) type, can easily be converted back to human readable form.

Artifact Parsers

• mac_apt - A WIFI_INTELLIGENCE plugin has been created.
• Velociraptor - Artifact created and submitted to Artifact exchange.

mac_apt update to BTM processing

This post highlights improvements to the AUTOSTART plugin in mac_apt.

Since macOS 13 (Ventura), Login items and Background tasks are managed and tracked via .BTM files. This is located at the path:

/private/var/db/com.apple.backgroundtaskmanagement/BackgroundItems-v<xx>.btm

where <xx> is the version number, currently 13 on macOS 15.2

Much of this information (but not all!) is visible to the end user via the Login items & Extensions page under System Settings as shown below.

Figure 1 - Login items & Extensions from System Settings

mac_apt's AUTOSTART plugin already processed BTM files, however this is now significantly improved. Previously BTM specific parameters were not being parsed and developer entries were also included (which are not autostart) which made them difficult to read and interpret, also missing some key information. 

BTM files are NSKeyedArchives which when deserialised contain dictionaries of items (login and background tasks) per user. 

Figure 2 - Snippet of single item from .BTM file

How these are interpreted and transformed into the nice GUI view seen above is dependant mostly on the parameters 'type' and 'disposition'. The following values have been observed for these fields:

DispositionValues = {

0x01: 'Enabled',

0x02: 'Allowed',

0x04: 'Hidden',

0x08: 'Notified'

}

TypeValues = {

0x00001: 'user item',

0x00002: 'app',

0x00004: 'login item',

0x00008: 'agent',

0x00010: 'daemon',

0x00020: 'developer',

0x00040: 'spotlight',

0x00800: 'quicklook',

0x80000: 'curated',

0x10000: 'legacy'

}

The 'type' value indicates if this item is an agent, daemon, app, user defined item or a spotlight or quicklook extension. 

When a user toggles the option to OFF for an item in the "Allow in the Background" setting, this will clear the 'Allowed' bit in the Disposition flag thereby indicating 'Not Allowed'. 

mac_apt now reads, interprets and shows the BTM parameters for disposition, type, container, developer and executableModifiedDate. The following output snippet filtered for "Not Allowed" will show the same output as that shown in System settings GUI. As seen in Fig 1 above (and Fig 4 below), 2 Citrix items are toggled to OFF, resulting in 6 apps belonging to these items being in the 'Not Allowed' group.

Figure 3 - Snippet of AUTORUNS output from mac_apt, filtered on BackgroundTask items and 'Not Allowed' disposition

Figure 4 - Disabled items from System Settings

This greatly simplifies the review of background applications. If the app itself disables a startup item, then the 'Enabled' flag is off, this will be missing from the BTM_Disposition column. mac_apt will populate the Disabled column with the value '1' to also indicate this. 

Also added is an 'AppArguments' column, which should populate the full command line arguments from all processed files (BTM and plists).

Be aware that mac_apt will process all encountered .btm files, so you may see repeated data as there are likely older .btm files, vestigial artefacts from previous macOS versions. On my test system, I've got BackgroundItems-v9.btm and BackgroundItems-v13.btm. This may be useful from a forensics perspective to look at the autostarts from that point in time. You will have to filter on the 'Source' column in the output if you wish to see only current data. 

NSKeyedArchive Deserializer update

A long time ago I wrote some code to make NSKeyedArchives (NSKA) human readable, basically de-serializing the data. It was then converted to a library for use in other projects like iLeapp and mac_apt. I revisited this last week and found and fixed a minor bug. While at it, I also added an extra capability, mostly for the folks who don't prefer to touch code. 

Previously, this library only worked with NSKA files. If a file was a normal plist, it would return an exception complaining about not being able to find the '$archiver' element in the plist. But what if you had files that were normal plists (not serialised), but had nested NSKA plists as data blobs within. There are actually quite a few on ios/macOS. To make them human-readable, you would have to write code to extract the blobs and run them through the library. The previous code also did not handle recursive deserializing even within NSKA archives. 

Now with the latest update (version 1.4.0), there is an extra parameter in the deserialize_plist(...) and deserialize_plist_from_string(...) functions to unlock this functionality and also performs full recursive deserializing of all nested blobs. 

def deserialize_plist(path_or_file, full_recurse_convert_nska=False)

By default, the value is False emulating the old behaviour. However, when set to True, this will no longer return an exception for non-NSKA (unserialized or normal) plists and will always return a plist. If there was a data (binary blob) element anywhere in the tree that had a value containing a valid header for an NSKA plist, that will now be replaced with a tree branch representing the deserialized version of the NSKA data.

Figure 1 - NSKA plist deserialized with old code vs new

If you are using nska_deserialize dependancy in any project, update to the latest:

pip3 install nska_deserialize --upgrade

The old compiled exe has been updated (with the flag set to True). It is also very conventient to use with drag and drop as shown here.

Reading OneDrive Logs Part 2

In the last OneDrive blog post, I outlined how the ODL file format is structured. A working version of an ODL parser was also created to read these files. One key detail was how personal file/folder, location or credential identifying strings were obfuscated with the original values stored in the ObfuscationStringMap.txt file. 

However some time in April 2022, Microsoft decided to change the way the obfuscation worked and the parser no longer worked (the unobfuscation part). 

What changed?

OneDrive now appeared to encrypt the data and the ObfuscationStringMap.txt is no longer used. The file may still exist on older installations, but newer ones include a different file.

Figure 1 - Contents of  \AppData\Local\Microsoft\OneDrive\logs\Business1 folder

As seen in Figure 1 above, there is a new file called general.keystore. This file's format is JSON that can be easily read and apparently holds the key to decrypt the encrypted content as a base64 encoded string. 

Figure 2 - Sample general.keystore contents

Time for some Reverse Engineering

With a little bit of digging around with IDA Pro on the LoggingPlatform.dll file from OneDrive, we can see the BCrypt Windows APIs being used in this file. Note, this is not the bcrypt hash algorithm which bears the same name!

Figure 3 - BCrypt* Imports in LoggingPlatform.dll

Jumping to where these functions are used, it is quickly apparent that the encryption used is AES in CBC (Cipher Block Chaining) mode with a key size of 128 bits. 

Figure 4 - IDA Pro Disassembly

In the above snippet, we can see the call to BCryptAlgorithmProvider and then if successful, a call to BCryptSetProperty function which has the following syntax:

NTSTATUS BCryptSetProperty(

  [in, out] BCRYPT_HANDLE hObject,

  [in]      LPCWSTR       pszProperty,

  [in]      PUCHAR        pbInput,

  [in]      ULONG         cbInput,

  [in]      ULONG         dwFlags

);

Without delving into too many boring assembly details, I'll skip to the relevant parts...

For each string to be encrypted, OneDrive initialises a new encryption object with the key that is stored in the general.keystore file, then encrypts the string and disposes of the encryption object. The encrypted blob is then base64 encoded and written out to the log the obfuscated string. There are a few other quirks along the way, such as replacement of the characters  / and  + with _  and - respectively, as the former can appear in base64 text but are also used in URLs to make it parseable later.

Why the change?

In the previous iteration of ODL (when the ObfuscationStringMap was used), there were instances where the same key (3 word combination) was often repeated in the file making it difficult or impossible to know which value to use as its replacement to get the original string.

Using encryption in place and not using a lookup table does appear to be a more robust scheme which eliminates the above issue. It does use some more disk space as the encrypted blob will always be a multiple of 16 bytes (128 bits) as this is block based encryption. In other words, it's inefficient for small text (less than 10 bytes). 

Updated code

The python ODL parser has been updated to accomodate this new format, and works with both the old and new versions. It is available here.

Reading OneDrive Logs

Due to the popularity of OneDrive, it has become an important source of evidence in forensics. Last week, Brian Maloney posted about his research on reconstructing the folder tree from the usercid.dat files, and also provided a script to do so. In this brief post, we explore the format of OneDrive Logs, and provide a tool to parse them. In subsequent posts, I will showcase use case scenarios. 

Where to find them?

OneDrive logs have the extension .odl and are found in the user's profile folder at the following locations:

On Windows - 

C:\Users\<USER>\AppData\Local\Microsoft\OneDrive\logs\

On macOS - 

/Users/<USER>/Library/Logs/OneDrive/

At these locations, there are usually 3 folders - Common, Business1 and Personal, each containing logs. As the name suggests Business1 is the OneDrive for Business version. 

Figure 1 - Contents of Business1 folder

The .odl file is the currently active log, while the .odlgz files are older logs that have been compressed. Depending on which folder (and OS) you are in, you may also see .odlsent and .aodl files, which have a similar format.

What is in there?

These are binary files, and cannot be directly viewed in a text editor. Here is what a .odl file looks like in a hex editor.

Figure 2 - .odl file in a hex editor

It is a typical binary file format with a 256 byte header and data blocks that follow. Upon first inspection, it seems to be a log of all important function calls made by the program. Having this low level run log can be useful in certain scenarios where you don't have other logging and need to prove upload/download or synchronisation of files/folders or even a discovery of items which no longer exist on disk/cloud. 

You do notice some funny looking strings (highlighted in red). More on that later.. 

The Format

With a bit of reverse engineering, the header format is worked out as follows:

struct {

    char     signature[8]; // EBFGONED

    uint32   unk_version;  // value seen = 2

    uint32   unknown2;

    uint64   unknown3;     // value seen = 0

    uint32   unknown4;     // value seen = 1

    char     one_drive_version[0x40];

    char     os_version[0x40];

    byte     reserved[0x64];

} Odl_header;

The structures for the data blocks are as follows:

struct {

    uint64     signature; // CCDDEEFF 0000000

    uint64     timestamp; // Unix Millisecond time

    uint32     unk1;

    uint32     unk2;

    byte       unk3_guid[16];

    uint32     unk4;

    uint32     unk5;  // mostly 1

    uint32     data_len;

    uint32     unk6;  // mostly 0

    byte       data[data_len];

} Data_block;

struct {

    uint32    code_file_name_len;

    char      code_file_name[code_file_name_len];

    uint32    unknown;

    uint32    code_function_name_len;

    char      code_function_name[code_function_name_len];

    byte      parameters[];

} Data;

In case of .odlgz files, the Odl_header is the same, followed by a single gzip compressed blob. The blob can be uncompressed to parse the Data_block structures.

Now, we can try to interpret the data. Leaving aside the few unknowns, the data block mainly consists of a timestamp (when event occurred), the function name that was called, the code file that function resides in and the parameters passed to the function. The parameters can be of various types like int, char, float, etc.. and that part hasn't been fully reverse engineered yet, but simply extracting the strings gives us a lot of good information. However the strings are obfuscated!

Un-obfuscating the strings

Since Microsoft uploads these logs to their servers for telemetry and debugging, they obfuscate anything that is part of a file/folder name or url string or username. However file extensions are not obfuscated. The ways this works is that the data identified to obfuscate is replaced by a word which is stored in a dictionary. The dictionary is available as the file ObfuscationStringMap.txt, usually in the same folder as the .odl files. To un-obfuscate, one simple needs to find and replace the strings with their original versions. 

Figure 3 - Snipped of ObfuscationStringMap.txt

This is a tab separated file, stored as either UTF-8 or UTF-16LE depending on whether you are running macOS or Windows. 

Now, referring back to the original funny looking string in Figure 2 - 

/LeftZooWry/LogOneMug/HamUghVine/MuchDownRich/QuillRodEgg/KoiWolfTad/LawFlyOwl.txt

  .. after un-obfuscating becomes ..

/Users/ykhatri/Library/Logs/OneDrive/Business1/telemetry-dll-ramp-value.txt

It is important to note that since extensions are not obfuscated, they can still provide valuable intel even if some or all of the parts in the path cannot be decoded.

Now this process seems easy, however not all obfuscated strings are file/folder names or parts of paths/urls. Some are multi line strings. Another problem is that the words (or keys to the dictionary) are reused! So you might see the same key several times in the ObfuscationStringMap. The thing to remember is that new entries get added at the top in this file, not at the bottom, so when reading a file, the first occurrence of a key should be the latest one.  Also sometimes, the key is not found, as it's cleaned out after a period of time. Also there is no way to tell if an entry in the dictionary is stale or valid for a specific log file being parsed. All of this just means that the decoded strings need to be taken with a grain of salt. 

Based on the above, a python script to parse the ODL logs is available here. A snippet of the output produced by the script is shown below.

Figure 4 - output snippet

In a subsequent post, we'll go through the items of interest in these logs like Account linking/unlinking, uploads, downloads, file info, etc..