cancel
Showing results for 
Search instead for 
Did you mean: 
Pravs
Level 4
Employee

Hello Again! I am working on one very interesting feature of NTFS file system from last 6 months and I can tell you it’s really very interesting. Yes, I am talking about alternate data streams (ADS). In this article, I will refer them mostly as ADS. 

Just for a general introduction, Alternate Data Streams (ADSs) are a unique feature of NTFS file systems introduced with Windows NT 3.1 in the early 1990s to provide compatibility between Windows NT servers and Macintosh clients which use Hierarchical File System (HFS). HFS uses streams named “resource fork” and “data fork”. Both streams (or forks) are linked to one name in the Macintosh file system. Resource forks are used to store application metadata (icons, sounds, fonts, etc.). NTFS ADSs can provide additional descriptions for folders or files (creator, keywords, thumbnail preview, etc.), and can also be used to attach independent named data streams to an NTFS file or folder.

They are all hidden and no windows utility provide you facility to show them up in windows explorer.  Though Vista comes with dir /r command to show them first time. There are couple of tools to detect them. Like LADS and streams.exe.  I personally preferred streams.exe. (perhaps because I downloaded it first).  Streams.exe can be downloaded from here. http://download.sysinternals.com/Files/Streams.zip


I am not going to talk more on ADS creation, detection, what are common ADS known, what data they contain etc. Here is few links to follow If you are totally unaware of what is ADS and it’s feature ( or mis-feature).
www2.tech.purdue.edu/cit/Courses/CIT556/readings/NTFSDarkside.pdf

this page also talks about how to run your special application using this wonderful facility provided by NTFS systems.

Figure 1 explains quickly how you can have hidden streams attached to your main file. I have not experimented with no. of named data streams one file can have but I believe it’s unlimited.

imagebrowser image

Figure 1 – ADS in NTFS File system

By now, you must be wondering being a system administrator or Back-up administrator, why I should even bother about them. I will tell you why?

• ADS are almost unknown, it’s not easily detectable
• ADS are almost invisible
• ADS are integral part of NTFS
• It is not a feature that can be disabled
• ADS can contain useful data that need to be backed-up and restored as well
• Existence (content) are not always taken into account by antivirus, backup programs.
• ADS can contain malicious code
• Potential danger (like WNT.Streamvirus)


Going into history, ADS exist since the inception of Windows NT 3.1. the need arises because sometimes Windows and Macintosh clients need to share resources and Macintosh file system relies on a named-stream model Data fork (content) and Resource fork (management).
ADS in NTFS  do not have settings of their own. They get it from its parent (unnamed data stream). ADS share General settings, Security settings, Encryption settings, Compression settings, SMB settings.

imagebrowser image
Figure 2: ADS Characteristics in NTFS

There are lots of issues while dealing with ADS and if you are unaware of them, better be aware than become a victim tomorrow. I have categorized possible issues  in four  major categories but sometimes they are overlapping as well.

1. Storage and Auditing

  • Physical (or Virtual) Disk space used (or unused)
  • Cleaning up
  • Auditing

2. Server  Security

  • DOS Attack
  • Code Execution
  • Windows Files Protection
  • Misuse of valid ADS
  • Virus propagation
  • Windows Scripting Host

3. Backup And  Restore
4. Support in future and current Windows file systems

1. Storage and Auditing

Unfortunately, it is virtually impossible to natively protect your system against ADS hidden files if you use NTFS. The use of Alternate Data Streams is not a feature that can be disabled and currently there is no way to limit this capability against files that the user already has access to.  Freeware programs like lads.exe by Frank Heyne (www.heysoft.de) and crucialADS by CrucialSecurity can be used to manually audit your files for the presence of Alternate Data Streams. Alternatively, the action of moving a file into another file system that doesn’t support ADS will automatically destroy any Alternate Data Streams.

Ultimately only a third party file checksum application can effectively maintain the integrity of an NTFS partition against unauthorized Alternate Data Streams. Recently dubbed as host based “Intrusion Prevention Systems” or “Intrusion Detection Systems”, third party security applications like eTrust Access Control from Computer Associates have been used for years in high-end government networks to verify the integrity of files used in the most secure environments. In addition to a heightened level of auditing and access control, these applications typically create an MD5 hashed database of file checksums that are used to validate a file’s trustworthiness. File injection techniques like Alternate Data Streams trigger an action by which the file is deemed untrusted and therefore prevented from executing or better yet, prevented from being changed in the first place.

Another good file integrity application is Tripwire for Servers by Tripwire Inc.  Tripwire has been singularly focused on file integrity management since the early 90’s and does a tremendous job of providing stringent security measures against unauthorized file changes.


2. Possible security and data loss risks with ADS

When a filesystem supports different forks, the applications should be aware of them, or security risks can arise.
If the different system utilities (disk explorer, antivirus software, archivers, and so on), are not aware of the different forks/ADS, the following problems can arise:
 

  • The user will never know the presence of any ADS  nor the total size of the file, just of the main data fork / file.
  • Computer viruses can hide in alternate forks on Windows and never get detected if the antivirus software are not aware of forks.
  • Data can be lost when sending files via fork-unaware channels, such as e-mail, filesystems without support for forks, or even when copying files between filesystems with forks support if the program that made the copy does not support forks or when compressing files with software that does not support forks.


Windows NT versions include the ability to use forks in the API, and some command line tools can be used to create and access forks, but they are ignored by most programs, including Windows Explorer and the DIR command. Windows Explorer copies forks and warns when the target file system doesn't support them, but only counts the main fork's size and it doesn't list a file or folder's streams. The DIR command has been updated in Vista to include an option that lists forks (ADS). Until Mac OS X v10.4, users using the Unix command line utilities (such as tar) included with Mac OS X would risk data loss, as the utilities were not updated to handle the resource forks of files until v10.4.

There are possible issues with ADS listed below. The detailed explanation is beyond the scope of this article. Mainly, DOS Attack, Code Execution, Windows Files Protection, Misuse of valid ADS, Virus propagation etc.

3. Backup And  Restore

Well, this is a big issue. What if, your back-up vendor is not backing up ADS at all and is sitting silent on this. What if, your back-up vendor is able to back-up ADS but is not able to restore them. You are paying them a hefty amount and your data is still lost and you never know which data. Just for your information, Netbackup and Backup-exec are fully aware of ADS and can back-up and restore them successfully.

Derek Bem and E.Z. Huebner from University of Western Sydney have classified Back-up software in to five categories based upon their capabilities to back-up and restore ADS. their work can be read here. Special thanks to Derek for providing permission to me to use his work and analyze it closely.

They have classified backup software into five groups depending on the level of ADS-awareness and the handling of alternate streams.
 

imagebrowser image

Figure 3 : Backing up and restoring data

In figure 3, Original files do consists of ADS. As per the figure, backed up Media can be either NTFS or non-NTFS (like FAT32). Restored to medium also can be NTFS/Non-NTFS.

Class 0 (Figure 4): Non-ADS aware software, ADSs are ignored, and not backed up.

imagebrowser image

Figure 4 : ADS handling by Class 0 backup software (non-ADS aware)

Class 1 (Figure 5): ADS-aware software, which handles ADSs properly only within NTFS environment.

imagebrowser image

Figure 5:  ADS handling by Class 1 backup software


Class 2 (Figure 6):
ADS-aware software, which provides good compatibility between NTFS and non-NTFS environments. It offers the functionality of Class 1, but additionally it can backup intact ADSs from NTFS to non-NTFS file system environment, and restore them to NTFS environment (Figure 6, paths 4-6).

imagebrowser image

Figure 6:  ADS handling by Class 2 backup software
Class 3 ADS-aware software, which can be seen as an unfinished implementation of Class 4, and it does not warrant closer investigation. It has all the capabilities of Class 4 (see below) with the exception of one:

  • it is unable to restore ADSs from a backup created on NTFS file system to non-NTFS media (Figure 3 path 3), or:
  • it is unable to restore ADSs from a backup created on non-NTFS media toNTFS media (Figure 3 path 6).


Class 4 (Figure 3): ADS-aware software, which has complete ADS awareness in any environment. It can backup and restore the ADS part to NTFS or non-NTFS media. I have not seen any backup software personally which can be classified as class 3 or 4.

A Class 1 tool is able to backup and restore ADSs if the operation is within NTFS environment (Figure 5, path 1-2). It fails to backup ADSs from NTFS to non-NTFS environment (Figure 5, paths 3 and 4). It is crucial to notice that Class 1 software is perfectly able to restore data to a non-NTFS disk, but no messages warning about the loss of data contained in ADS data would be generated. This is a practical observation, not a theoretical restriction. It would be possible for Class 1 software to warn that backup and restore environments differ, and warn against possible data loss. However this means that such a tool would check for the presence of alternate streams before generating a warning message – thus implementing Class 2 (or even Class 4) behavior as explained below would be easy.

Class 2 backup software uses the old block format, originally created for tape magnetic media. Data read from a disk is stored as a set of logically sequential blocks. Terminology can vary depending on a specific implementation. The older, generic terms for major backup components are: tape header, data sets, on tape catalog information, end of media. Modern tape backup specification allows the use any common media backup, for example hard disks, removable cartridges, flash drives, etc., but basic “tape style” logic of storing data is retained. Class 2 backup software retains ADS data, but is unable to restore it to non-NTFS media (see Figure 6, paths 3 and 5). It is particularly dangerous to attempt to restore a backup which contains ADSs to a non-NTFS media. For example a backup from a NTFS disk to a FAT32 disk creates one of the following situations:
 

  • backup software does not show any warning messages, and ADS data is lost.
  • backup shows a warning message, but ADS data is still lost.


Class 3 software can be seen as an unfinished or logically incomplete implementation of Class 4 software, with one path missing (Figure 3, path 3 or path 5 is not implemented).  Class 4 software (for path references see Figure 2) is fully ADS-aware, and it should be able to:
 

  • Backup files and folders with ADSs to NTFS environment (path 1)
  • Restore the backup and all ADSs to both NTFS and non-NTFS environments (path 2 and path 3)
  • Backup files and folders with ADSs to non-NTFS environment (path 4),
  • Restore the backup and all ADSs to both NTFS and non-NTFS environments (path 5 and path 6).


I have not seen any class 3 or 4 software as such. The reason for this is; the underlying file system where you are going to restore does not understand alternate data stream at all. For the sake of data restore, ADS can still be restored on such file system. I don’t see that developers are going to agree on one unique solution for this. Till then, it’s best to use NTFS to NTFS restore.

Whatever backup application you are running with, my advice on this to check if your ADS are intact or not. Throw out class 0 software, if you are running with. It’s going to give you more trouble sooner.

4. Support in future and current Windows file systems

Despite being a feature of the NTFS, ADSs are poorly documented by Microsoft.This may be the result of a conscious decision, as one can find statements in various Microsoft sources stating that ADSs “may not be supported in future systems”. Source : Microsoft Knowledge Base, How To Use NTFS Alternate Data Streams, Article ID: 105763, http://support.microsoft.com/kb/105763

This statement of possible withdrawal of ADS support is meaningless for anyone working with a system using NTFS file format. NTFS went through many modifications and versions since ADSs were first introduced, and ADSs are still present in all current Windows Server editions.

Comments
Hiness
Not applicable
Good Information. Must be ueful for users like me.
shaun_b
Level 2
Partner Accredited
Having some experience with ADS rootkits, your article is tip top. Very thorough and precise.
i2professional
Level 2
this is amzing article;; now the question that i felt is

"if there is any virus that is hidden in side ADS ; whether symantec will be able to detect it?"
Pravs
Level 4
Employee
well, All good anti-viruses are able to detect them but I don't make any claim on that. Best way, go and check yourself. :)
PrafulPatel
Not applicable
Thanks a lot for sharing this piece of Info. I was not aware about any stream stuff and now I see there are lots of them at my disk. How I can get rid of them?
Pravs
Level 4
Employee
you can use any of the tools (like lads, streams.exe)  to remove them. Let me remind you, I don't recommend or even suggest to remove them.
Satyam_Pujari1
Level 2
well done...and yep, SEP is good at detecting ADS ..tested !! 
lifeofit_com
Not applicable
that would calculate hashes of the alternate data streams, not just the main one?

Since modifying attached ADS does not affect the reported file size or the main stream hash (md5, sha, etc), I'm going to have to create my own. Before I re-invent the wheel, I though I'd ask around...

http://lifeofit.com/blog/?p=32
Pravs
Level 4
Employee
I am sorry, I don't know any such tool.
rj_nbu
Level 6
Employee Accredited Certified
Pravs,

What would be the impact of ADS on VxMS mapping during flashbackup ? Do we see any performance hit ?

Also, is it the same as embedded files (Ex; a ppt containing .wav files ?)




Pravs
Level 4
Employee

no rj_nbu,
you have to backup these streams. don't you want to backup them as well? VxMS mapping will map these streams as well.


Also, is it the same as embedded files (Ex; a ppt containing .wav files ?) : Yes, it's true.
Version history
Last update:
‎05-25-2009 03:58 AM
Updated by: