With macOS 10.15 - Catalina, Apple has introduced a change in the way system and user data is stored on disk. In prior versions, the root '/' volume was stored in a single volume usually named 'Macintosh HD'. This did not change with the update to APFS. However with Catalina, there are now two distinct volumes -
The Macintosh HD volume stores the system files and is mounted as read-only, while the Macintosh HD - Data volume has all the other files on your system which include user profiles, system and user data, and user installed Applications.
However when booted, only a single logical volume is presented (as root /) that combines the contents of both. This is enabled through APFS using its Volume Role feature. This is mentioned in Apple's official APFS documentation but its usage or working is not documented. Each volume can be assigned a role in its volume Superblock structure (apfs_superblock_t). There are 8 possible roles documented. From the Apple docs:
The SYSTEM volume contains the folders /bin, /sbin and most of the /usr and /System folders. A few subfolders of /usr and /System are on the DATA volume. The volumes are joined using a new construct that Apple calls firmlinks. They describe it as a Bi-directional wormhole in path traversal. Firmlinks are used on the system volume to point to the user data on the data volume.
They are somewhat similar to the unix symlinks and hardlinks, but only directories can be linked (from one volume to another). The file that defines/lists the firmlinks resides on the SYSTEM volume at /usr/share/firmlinks. The following paths are defined by default.
The linked volumes will have distinct inode numbers for files/folders. The only common inode numbers seen are inodes 1 (Parent of root), 2 (root) and 3 (private-dir). All other inodes will be unique, a simple but clever scheme is used to ensure that. For the SYSTEM volume, every inode number allocated will be OR'd with 0x0FFFFFFF00000000. Take a look at the inode numbers in the combined volume in screenshot below. The very large numbers are the files that reside on the SYSTEM volume due to the upper bits being set by the mask.
If you try to create a file or folder on the root volume (or one of its owned folders), it fails with an error: Read-only volume.
For accessing most files and folders, there should be no problem as the stitched/combined volume works seamlessly so all programs should not notice any difference. However there are situations where you might want to explicitly access a folder from a specific volume, especially for forensics. For example, if you wanted to access /./fseventsd, you would always get the read-only volume's .fseventsd folder which won't be too interesting as its a read-only volume! To get the one on the DATA volume, there is still a way. Apple has also made the DATA volume available (mounted) at the mountpoint /System/Volumes/Data. This also means that if you have scripts that run across all files, they will need to be made aware that this location should be avoided to prevent duplication.
According to Apple, you cannot opt-out of this, and it is a required feature for macOS 10.15. Forensic tools that operate on full disk images will have to adapt for this change, and so I've updated mac_apt to support macOS Catalina. If you use it, let me know of bugs/issues.
- Macintosh HD
- Macintosh HD - Data
 |
| Figure 1 - diskutil output showing a split Macintosh HD volume into two |
The Macintosh HD volume stores the system files and is mounted as read-only, while the Macintosh HD - Data volume has all the other files on your system which include user profiles, system and user data, and user installed Applications.
However when booted, only a single logical volume is presented (as root /) that combines the contents of both. This is enabled through APFS using its Volume Role feature. This is mentioned in Apple's official APFS documentation but its usage or working is not documented. Each volume can be assigned a role in its volume Superblock structure (apfs_superblock_t). There are 8 possible roles documented. From the Apple docs:
#define APFS_VOL_ROLE_NONE 0x0000
#define APFS_VOL_ROLE_SYSTEM 0x0001
#define APFS_VOL_ROLE_USER 0x0002
#define APFS_VOL_ROLE_RECOVERY 0x0004
#define APFS_VOL_ROLE_VM 0x0008
#define APFS_VOL_ROLE_PREBOOT 0x0010
#define APFS_VOL_ROLE_INSTALLER 0x0020
#define APFS_VOL_ROLE_DATA 0x0040
#define APFS_VOL_ROLE_BASEBAND 0x0080
The SYSTEM volume contains the folders /bin, /sbin and most of the /usr and /System folders. A few subfolders of /usr and /System are on the DATA volume. The volumes are joined using a new construct that Apple calls firmlinks. They describe it as a Bi-directional wormhole in path traversal. Firmlinks are used on the system volume to point to the user data on the data volume.
They are somewhat similar to the unix symlinks and hardlinks, but only directories can be linked (from one volume to another). The file that defines/lists the firmlinks resides on the SYSTEM volume at /usr/share/firmlinks. The following paths are defined by default.
/AppleInternal AppleInternal
/Applications Applications
/Library Library
/System/Library/Caches System/Library/Caches
/System/Library/Assets System/Library/Assets
/System/Library/PreinstalledAssets System/Library/PreinstalledAssets
/System/Library/AssetsV2 System/Library/AssetsV2
/System/Library/PreinstalledAssetsV2 System/Library/PreinstalledAssetsV2
/System/Library/CoreServices/CoreTypes.bundle/Contents/Library
System/Library/CoreServices/CoreTypes.bundle/Contents/Library
System/Library/CoreServices/CoreTypes.bundle/Contents/Library
/System/Library/Speech System/Library/Speech
/Users Users
/Volumes Volumes
/cores cores
/opt opt
/private private
/usr/local usr/local
/usr/libexec/cups usr/libexec/cups
/usr/share/snmp usr/share/snmp
The linked volumes will have distinct inode numbers for files/folders. The only common inode numbers seen are inodes 1 (Parent of root), 2 (root) and 3 (private-dir). All other inodes will be unique, a simple but clever scheme is used to ensure that. For the SYSTEM volume, every inode number allocated will be OR'd with 0x0FFFFFFF00000000. Take a look at the inode numbers in the combined volume in screenshot below. The very large numbers are the files that reside on the SYSTEM volume due to the upper bits being set by the mask.
 |
| Figure 2 - Contents of root showing files from both SYSTEM and DATA |
For accessing most files and folders, there should be no problem as the stitched/combined volume works seamlessly so all programs should not notice any difference. However there are situations where you might want to explicitly access a folder from a specific volume, especially for forensics. For example, if you wanted to access /./fseventsd, you would always get the read-only volume's .fseventsd folder which won't be too interesting as its a read-only volume! To get the one on the DATA volume, there is still a way. Apple has also made the DATA volume available (mounted) at the mountpoint /System/Volumes/Data. This also means that if you have scripts that run across all files, they will need to be made aware that this location should be avoided to prevent duplication.
According to Apple, you cannot opt-out of this, and it is a required feature for macOS 10.15. Forensic tools that operate on full disk images will have to adapt for this change, and so I've updated mac_apt to support macOS Catalina. If you use it, let me know of bugs/issues.
Hi, How do you locate address/position of apfs_superblock_t starting form first bytes of the HD?
ReplyDeletelike this image https://i.stack.imgur.com/3LYhZ.png (how calculate the jump to its location?) https://apple.stackexchange.com/questions/396301/finding-volume-label-and-understanding-the-apfs-partition
It can't be explained here , as it is too complex and long for a comment post. Short answer is - follow the steps outlined in the pic you shared, which are sourced from the Apple APFS documentation. I read your other stackoverflow post about getting apfs mounted in linux, try this project - https://github.com/sgan81/apfs-fuse
ReplyDeleteIf you want to learn how to do this on your own, use my 010 hex editor template here - https://github.com/ydkhatri/APFS_010 Once you load your disk, run the template and drill down to the volumes, its should be fairly straightforward to understand.