Hello Again! I am working on one very interesting feature of NTFS file system from last 6 months and I can tell you it’s really very interesting. Yes, I am talking about alternate data streams (ADS). In this article, I will refer them mostly as ADS.
Just for a general introduction, Alternate Data Streams (ADSs) are a unique feature of NTFS file systems introduced with Windows NT 3.1 in the early 1990s to provide compatibility between Windows NT servers and Macintosh clients which use Hierarchical File System (HFS). HFS uses streams named “resource fork” and “data fork”. Both streams (or forks) are linked to one name in the Macintosh file system. Resource forks are used to store application metadata (icons, sounds, fonts, etc.). NTFS ADSs can provide additional descriptions for folders or files (creator, keywords, thumbnail preview, etc.), and can also be used to attach independent named data streams to an NTFS file or folder.
They are all hidden and no windows utility provide you facility to show them up in windows explorer. Though Vista comes with dir /r command to show them first time. There are couple of tools to detect them. Like LADS and streams.exe. I personally preferred streams.exe. (perhaps because I downloaded it first). Streams.exe can be downloaded from here. http://download.sysinternals.com/Files/Streams.zip
I am not going to talk more on ADS creation, detection, what are common ADS known, what data they contain etc. Here is few links to follow If you are totally unaware of what is ADS and it’s feature ( or mis-feature).
this page also talks about how to run your special application using this wonderful facility provided by NTFS systems.
Figure 1 explains quickly how you can have hidden streams attached to your main file. I have not experimented with no. of named data streams one file can have but I believe it’s unlimited.
Figure 1 – ADS in NTFS File system
By now, you must be wondering being a system administrator or Back-up administrator, why I should even bother about them. I will tell you why?
• ADS are almost unknown, it’s not easily detectable
• ADS are almost invisible
• ADS are integral part of NTFS
• It is not a feature that can be disabled
• ADS can contain useful data that need to be backed-up and restored as well
• Existence (content) are not always taken into account by antivirus, backup programs.
• ADS can contain malicious code
• Potential danger (like WNT.Streamvirus)
Going into history, ADS exist since the inception of Windows NT 3.1. the need arises because sometimes Windows and Macintosh clients need to share resources and Macintosh file system relies on a named-stream model Data fork (content) and Resource fork (management).
ADS in NTFS do not have settings of their own. They get it from its parent (unnamed data stream). ADS share General settings, Security settings, Encryption settings, Compression settings, SMB settings.
Figure 2: ADS Characteristics in NTFS
There are lots of issues while dealing with ADS and if you are unaware of them, better be aware than become a victim tomorrow. I have categorized possible issues in four major categories but sometimes they are overlapping as well.
1. Storage and Auditing
2. Server Security
3. Backup And Restore
4. Support in future and current Windows file systems
1. Storage and Auditing
Unfortunately, it is virtually impossible to natively protect your system against ADS hidden files if you use NTFS. The use of Alternate Data Streams is not a feature that can be disabled and currently there is no way to limit this capability against files that the user already has access to. Freeware programs like lads.exe by Frank Heyne (www.heysoft.de) and crucialADS by CrucialSecurity can be used to manually audit your files for the presence of Alternate Data Streams. Alternatively, the action of moving a file into another file system that doesn’t support ADS will automatically destroy any Alternate Data Streams.
Ultimately only a third party file checksum application can effectively maintain the integrity of an NTFS partition against unauthorized Alternate Data Streams. Recently dubbed as host based “Intrusion Prevention Systems” or “Intrusion Detection Systems”, third party security applications like eTrust Access Control from Computer Associates have been used for years in high-end government networks to verify the integrity of files used in the most secure environments. In addition to a heightened level of auditing and access control, these applications typically create an MD5 hashed database of file checksums that are used to validate a file’s trustworthiness. File injection techniques like Alternate Data Streams trigger an action by which the file is deemed untrusted and therefore prevented from executing or better yet, prevented from being changed in the first place.
Another good file integrity application is Tripwire for Servers by Tripwire Inc. Tripwire has been singularly focused on file integrity management since the early 90’s and does a tremendous job of providing stringent security measures against unauthorized file changes.
2. Possible security and data loss risks with ADS
When a filesystem supports different forks, the applications should be aware of them, or security risks can arise.
If the different system utilities (disk explorer, antivirus software, archivers, and so on), are not aware of the different forks/ADS, the following problems can arise:
Windows NT versions include the ability to use forks in the API, and some command line tools can be used to create and access forks, but they are ignored by most programs, including Windows Explorer and the DIR command. Windows Explorer copies forks and warns when the target file system doesn't support them, but only counts the main fork's size and it doesn't list a file or folder's streams. The DIR command has been updated in Vista to include an option that lists forks (ADS). Until Mac OS X v10.4, users using the Unix command line utilities (such as tar) included with Mac OS X would risk data loss, as the utilities were not updated to handle the resource forks of files until v10.4.
There are possible issues with ADS listed below. The detailed explanation is beyond the scope of this article. Mainly, DOS Attack, Code Execution, Windows Files Protection, Misuse of valid ADS, Virus propagation etc.
3. Backup And Restore
Well, this is a big issue. What if, your back-up vendor is not backing up ADS at all and is sitting silent on this. What if, your back-up vendor is able to back-up ADS but is not able to restore them. You are paying them a hefty amount and your data is still lost and you never know which data. Just for your information, Netbackup and Backup-exec are fully aware of ADS and can back-up and restore them successfully.
Derek Bem and E.Z. Huebner from University of Western Sydney have classified Back-up software in to five categories based upon their capabilities to back-up and restore ADS. their work can be read here. Special thanks to Derek for providing permission to me to use his work and analyze it closely.
They have classified backup software into five groups depending on the level of ADS-awareness and the handling of alternate streams.
Figure 3 : Backing up and restoring data
In figure 3, Original files do consists of ADS. As per the figure, backed up Media can be either NTFS or non-NTFS (like FAT32). Restored to medium also can be NTFS/Non-NTFS.
Class 0 (Figure 4): Non-ADS aware software, ADSs are ignored, and not backed up.
Figure 4 : ADS handling by Class 0 backup software (non-ADS aware)
Class 1 (Figure 5): ADS-aware software, which handles ADSs properly only within NTFS environment.
Figure 5: ADS handling by Class 1 backup software
Class 2 (Figure 6): ADS-aware software, which provides good compatibility between NTFS and non-NTFS environments. It offers the functionality of Class 1, but additionally it can backup intact ADSs from NTFS to non-NTFS file system environment, and restore them to NTFS environment (Figure 6, paths 4-6).
Figure 6: ADS handling by Class 2 backup software
Class 3 ADS-aware software, which can be seen as an unfinished implementation of Class 4, and it does not warrant closer investigation. It has all the capabilities of Class 4 (see below) with the exception of one:
Class 4 (Figure 3): ADS-aware software, which has complete ADS awareness in any environment. It can backup and restore the ADS part to NTFS or non-NTFS media. I have not seen any backup software personally which can be classified as class 3 or 4.
A Class 1 tool is able to backup and restore ADSs if the operation is within NTFS environment (Figure 5, path 1-2). It fails to backup ADSs from NTFS to non-NTFS environment (Figure 5, paths 3 and 4). It is crucial to notice that Class 1 software is perfectly able to restore data to a non-NTFS disk, but no messages warning about the loss of data contained in ADS data would be generated. This is a practical observation, not a theoretical restriction. It would be possible for Class 1 software to warn that backup and restore environments differ, and warn against possible data loss. However this means that such a tool would check for the presence of alternate streams before generating a warning message – thus implementing Class 2 (or even Class 4) behavior as explained below would be easy.
Class 2 backup software uses the old block format, originally created for tape magnetic media. Data read from a disk is stored as a set of logically sequential blocks. Terminology can vary depending on a specific implementation. The older, generic terms for major backup components are: tape header, data sets, on tape catalog information, end of media. Modern tape backup specification allows the use any common media backup, for example hard disks, removable cartridges, flash drives, etc., but basic “tape style” logic of storing data is retained. Class 2 backup software retains ADS data, but is unable to restore it to non-NTFS media (see Figure 6, paths 3 and 5). It is particularly dangerous to attempt to restore a backup which contains ADSs to a non-NTFS media. For example a backup from a NTFS disk to a FAT32 disk creates one of the following situations:
Class 3 software can be seen as an unfinished or logically incomplete implementation of Class 4 software, with one path missing (Figure 3, path 3 or path 5 is not implemented). Class 4 software (for path references see Figure 2) is fully ADS-aware, and it should be able to:
I have not seen any class 3 or 4 software as such. The reason for this is; the underlying file system where you are going to restore does not understand alternate data stream at all. For the sake of data restore, ADS can still be restored on such file system. I don’t see that developers are going to agree on one unique solution for this. Till then, it’s best to use NTFS to NTFS restore.
Whatever backup application you are running with, my advice on this to check if your ADS are intact or not. Throw out class 0 software, if you are running with. It’s going to give you more trouble sooner.
4. Support in future and current Windows file systems
Despite being a feature of the NTFS, ADSs are poorly documented by Microsoft.This may be the result of a conscious decision, as one can find statements in various Microsoft sources stating that ADSs “may not be supported in future systems”. Source : Microsoft Knowledge Base, How To Use NTFS Alternate Data Streams, Article ID: 105763, http://support.microsoft.com/kb/105763
This statement of possible withdrawal of ADS support is meaningless for anyone working with a system using NTFS file format. NTFS went through many modifications and versions since ADSs were first introduced, and ADSs are still present in all current Windows Server editions.