Zip-It Part 2

 Part two of deciding the Correct Zip program to

use AND How to Test and verify it for Forensic usage!

Before reading this article,

please read ZIP-IT Part one.

I recently held a session on testing various software to see if it was forensically sound. The session was a success, as far as opening the attendees eyes to see that not all the software that is often recommended as being forensically sound really is. The software tests involved the attendees testing their software to see if it could pass three or four very simple but important forensic requirements. The requirements were those that I would challenge if I were in that position. The tests involved basic file hashing, forensic file copying, and zipping/unzipping in a forensic evidentiary environment. These three categories seemed to me to be the basics behind a sound forensic analysis, reporting, and evidentiary delivery of any evidence found.

At the end of the session I realized that because of time constraints, the attendees did not complete the file zip/unzip portion of the software testing. As a result I decided to do two things. First, I solicited a number of colleagues and asked them to continue with and finish the software testing of the zip products. To date, none of the testers have reported any zip program capable of passing all the tests I have set up. But I have.

Second, I decided to continue with the tests myself. I must admit, I had already tested the items before the session, and knew how the software performed, or where it was failing my test requirements. However I decided to continue and redo the tests again for this article. The tests I designed and performed were totally non-scientific. But were in my opinion realistic. Meaning the processes I tested were those which you might find everyday in the file systems you are processing in your forensic analysis and those which you might commonly forget to check for.

This short article was written as a result of my performing tests on some of the recognized file zipping software programs. They include programs that are routinely recommended and used by most people processing evidence for retention, attorney discovery, or court adjudication. So as not to point fingers at those failing my tests, I am not going to mention their names here.

The basic parameters were set up as follows. All these parameters and assumptions were designed around a Windows Operating System environment. Other OS's such as Linux and the various MAC systems would not have similar requirements. A majority of business operations and forensic analysis is done on Windows platforms, making it an ideal platform for these tests.

First, because I personally think that if you are running a secure shop, your operating system should have the last access update of files turned on. (This is done via a registry key). In most Windows environments, it is off by default. When you are working for a large organization, and are concerned that someone might copy and walk off with the keys to the kingdom, or any other operation that may involve who or how they accessed sensitive files, this last access date may come into play. In my prior life, I often used last access to see when a file was accessed, which might point to the time it was copied/moved, printed, or stolen. That being said, lets set last access update of files to ON. This was requirement number one.

Second parameter was that some of the test data files being used were located in paths made up of Long Filenames (LFN). For those not recognizing this term, it means path/filenames greater than 255 characters. For this reason, the test file system was mandated to be an NTFS file system. For those of you who routinely use many of the forensic suites out there, you know that when exporting files from images, or otherwise, the suites have absolutely no problem exporting long filenames. However, Windows Explorer has its own problems with LFN'S. Often when exporting evidence files you end up with an evidentiary folder length rivaling the length of War and Peace. So long filename processing ability was another requirement.

Third and final major parameter requirement was that the process (program) be able to handle Alternate Data Streams (ADS). Quite simply, this is a file that is attached to a primary file in the file structure. I call the primary file the parent, which is what you see when you look at the files in explorer or other general file listing process. The ADS is effectively (for lack of a better, non-technical term) a child or hitchhiker. It is attached to the parent, but very few processes display the occurrence of an ADS. However, Alternate Data Streams can be made up of any kind of file. They can be executable programs, virus programs, key logger programs, and they can even hold the original URL of downloaded images. Bet you didn't know that. Great for an investigator of porn sites. So lets consider retention of ADS's also important.

Each of these three requirements seem simple, and in most cases are completely unnecessary in most evidentiary situations, but would you want to be the one explaining to the attorneys why your software missed one or all of the requirements set above. Or why, a few years down the road, you unzipped the evidence folder, and realized that some key evidence relating to one or all of the above parameters was now missing? I know I wouldn't.

Now the tests begin. First I designed a simple set of about 150 test files. Nothing large or complicated. Just spread out among about 20 directories of both long filenames and regular (less than 255 character filenames). Some of the files in both the LFN set, and normal set had one or more ADS associated with them. ALL the files had the MAC dates set to: 01/01/2019    12:34:56:789c    01/01/2019 12:34:56:789w    01/01/2019 12:34:56:789a Just for fun, I managed to set the times to 12:34:56:789.

Now I took the recognized file zip/unzip programs and tested against the following parameters as mentioned before. Determine if it: 1:    Altered/updated last access date of the source during the operation. This would be a major problem. Why did you alter the original evidence date? 2:    Restored all the original three MAC dates to the destination upon unzipping. Do the restored dates reflect original values? Hope so. 3:    Could find and properly zip/unzip all the LFN (Long Filename) files. 4:    Could find and include all the ADS along with their parent files

If it failed any or all of the above four tests, I considered it unworthy to be called a forensically sound zipping program.

Some could zip LFNs but couldn't unzip them. Beats the #$%* out of me, how it could zip it, and fail to unzip it. Others worked fine on finding ADS files in the short files, but failed on the LFN's. Others failed to "reset" the original last access date of the source file, and allowed for the last access of the restored destination to be set to the current date. This means they allowed the operating system to alter their original evidence. Replacing or maintaining original last access of the source would be a primary concern. Especially when zipping original data from say, a live server. One had an interface so cumbersome and messed up, I could hardly conduct the tests in an efficient manner. Needless to say I was very disappointed with all but one of the programs tested. For that reason, I am now going to reveal the name of the program I tested which passed all the tests I have set up.

The program which passed all the tests was WINRAR, and more specifically the command line version called RAR. The command line version is more verbose, batch capable, and easy to set up. After all, if you use it in a batch file, even if you are doing something wrong (and we hope you don't), at least you can testify, that you always do it this way. HA HA!

There were some initial stumbling blocks when dealing with the RAR program. Initially it failed to maintain the original last access date of the files. But after a short email exchange with the programmers, and explaining the evidentiary reason for needing last access date maintenance, within two days an updated beta version was out. The current version of WINRAR/RAR now has that option available. Along with dozens of other options, which I will not even attempt to list or mention. I learned which options I need, and routinely use those in a "standard" command line located in (guess what) a batch file, and a comment.txt file (explained below). As it turns out, only about four options are routinely needed to ensure compliance with my test requirements. Your needs may be different. So review, practice and use whatever options you find useful.

An item which is currently implemented is the capability of setting default options both in an INI file, and a file called comment.txt. A "very" few forensic options are capable of residing in the comment.txt file which will take effect upon extraction. This means, that not all of my forensic options are available in the comment.txt. Thus, the command line of the executable may require additional options to extract all the data perfectly. With a little practice, a suitable extraction command line is possible. Where all Paths, MAC dates, LFN's, and ADS are properly restored. Personally, I formulate the command line, then provide it in a batch script, or provide it to the recipient in an email. That way the recipient has all the information to properly extract a good forensic file structure.

Developing a suitable command line for the executable is advisable in order to allow the recipient to properly extract the data. Sending the .exe out of your control with a correct command line will hopefully ensure correct extraction options. An executable sent for remote extraction with proper command line options is, in my opinion the best way to go.

I have also come to use the ability to create encrypted self-extracting files. The self-extraction process makes it not only self contained, but the recipient doesn't need the WINRAR/RAR program to extract the data. The encryption capability (available in most zipping products nowadays) is an added security step, and benefit.

Remember, don't take my word for it. Regardless of which product you use, obtain, test, and retest every program and set up a routine process that ensures consistency. Know which options work, and don't work. Because the other side knows how to challenge your process. (Why did you perform this xxx process on my clients data, and not on other persons data? Are you prejudiced against my client?)

Again, in my testing of the recognized zip/unzip software out there, I found only the WINRAR/RAR to pass all my forensic evidentiary requirements. My requirements may be more strict for testing purposes, and may not be the same requirements you have. But do you want to leave yourself open to evidentiary challenges? It is possible that RAR may fail when performing in a way that you may need it, however, it passed my test requirements.

Bottom Line: Test your zip/unzip software and make sure it meets or exceeds all your evidentiary and storage requirements. Dont' take anyones word (not even mine) that it works. Test it yourself so you can answer questions about its operations when you are on the witness stand.

Associated articles and programs of interest: hash  program to calculate hash values. HASH_IT_OUT  an article discussing forensic hashing of evidence. COPY_THAT  an article discussing forensic copying of evidence. ZIP_IT  an article regarding use of zipping software for forensics.

How TCP Works - The TCPTrace Graph (Chris Greer)

Troubleshooting a slow file transfer, backup, or any other app that moves a bunch of data in one direction?

If so, the tcptrace graph in Wireshark can save a ton of time in tracking down the problem. This awesome graph can show pauses in data transfer, space between transmitted data and the receive window, and SACK blocks from duplicate acknowledgements. In this video, we dive into the TCPTrace graph and show you how to interpret it and analyze it to find problems fast. Check it out!

FREE WIRESHARK CLASS - lecture 1 -GETTING STARTED

To celebrate my 10th year on youtube and to thank all those who watch, like, share and subscribe I wanted to give you a gift.

2 years ago I created a Wireshark class for Udemy which was very popular.

So I thought I would release the entire course for all to enjoy.

Please do not bother with the exercises, but find your own way to test the concepts i present.

Enjoy!!!

Zip - It!!!

 

That’s what my mother used to say to me when I was bad. But that’s not what we are talking about here.

However, we will talk about “bad” zipping software. Yes zipping software doesn’t always perform as expected. What a revelation.

Even though zipping of files/data is a routine normal occurrence, those that are conducting security or forensic exams and "save" their results or reports for delivery to management or attorneys should consider the capabilities of these zipping programs with relation to what you are zipping. Some forensic examiners zip image files, which would work fine. Others zip extracted evidence after processing with a forensic suite. And still others massage the initial extracts and get the data down to a point where they will provide it to management or legal persons for review. The problem(s) you should consider, is that during your process, you may inadvertently create some unusual data files. The forensic suite may extract alternate data streams, or extract files and put them in a folder which has a long filename. I have seen forensic suites extract data to long filenames as a matter of course. Especially when you are asking it to recreate the original path/folder structure, and you yourself are outputting the data to a sub-directory which itself starts multi-levels down the tree. You then don't know how long the ultimate extracted folder will be, and when you go to zip up the data for delivery, you may miss one or two important files. Or totally miss an alternate data stream which was hiding behind an important file. Last, but not least, does the zipping program maintain the source and ultimately unzipped file dates? If you have never tested your zipping program to see how/what it does in unusual situations, you probably should. Test not only your version, but the version that the opposition is using.That being said.

First, let me state: the tests I ran are not at all scientific. I consider them practical. Second: I am not using names here because I do not want to point fingers. I’m just pointing out what I found in my unscientific tests. If my minimal tests show a failure, why proceed further. Third: The test suite I used was purely arbitrary, and seeded with items which, from other tests, I knew might cause problems with zipping software, but would not be unheard of in a forensic or backup environment. Fourth: I DID NOT test any encryption capabilities of the software. I use separate PGP encryption of the stand-alone zipped file when necessary. Finally: Run some tests yourself. Don’t take my word for it.

I began by selecting three of the most popular zipping software packages. My versions may not be the most current, because I’m CHEAP and don’t spend money needlessly. Most had both GUI and command line capability. However, for consistency, and because most people prefer the GUI interface, I only used the GUI in my tests. A very important thing to remember with the GUI versions, is that unless all the correct boxes are checked, you may not get the results you expect, and may not obtain results similar to mine. But if the program hid the option so much that I couldn't find it, I may not wish to use that program in the future.

Some required in-depth choices of operations which I would consider required, so I had to look for those options I wished to implement. Each package showed different options for the same operation, and some had no option for a needed capability. I MAY have missed an important option. If I did, it only means the option was so far buried, that a normal person might also forget to look for it, and find it. I tried as best I could to locate and check all the boxes which would cover the items I tested for (ie: Long filenames, Alternate Data Streams, Date retention).

Then I created a folder with the following parameters: Files containing Unicode characters in their filename (ie: CYRILLIC names).

Second, I created some files with long filenames (path/filename > 255 characters). Third, I inserted some Alternate Data Streams (ADS) in a number of the files, both those with normal length names, and long filenames.

I created those three types of files because, in a backup scenario, an investigation and subsequent evidentiary output which would probably be sent to an opposing party (attorney), one or all of these types of data (files) might be necessary to produce. Forensic suites and other forensic software operations may routinely export files with any or all of these items.

Also, I have seen discussions, where persons have been asking which methods people use to store data for posterity. That’s a long time, not your rear end.

The common methods of storing or delivering data are in a zip format. Not only for space saving, but also for inclusion into a single file for distribution to a requesting party.

So, lets test some of the zip(ping) capability of these programs.

Again, I’m not going to name names, or identify which program failed in which area so here are the general results. If you think some of the items may belong to your processes, you might test the software yourself. What a novel idea.

First:  All zipped and unzipped all filenames correctly, (UNICODE, etc).
Second: Two of three processed Long Filenames correctly. (see ADS below)
Third:  Only one processed ADS’s in normal and LFN filenames,  
Fourth: None of them reset the last access date of the original files after the zip process. 
Fifth:  Two of three properly reset (original) last access date of the restored file to the original access data. 
Sixth:  Two of three properly reset all the MAC dates to the restored file. The third only reset the original ‘M’odified date. 

Here is a quick and dirty spreadsheet showing which program did what. If you want to know the real name of the program contact me at: dm at dmares.com

Program #UnicodeLFNADSReset Src AccessReset Dest AccessReset MAC Dest.# 1PASSPASSPASSFAILPASSMAC# 2PASSFAILFAILFAILPASSMAC# 3PASSPASSFAILFAILFAILM# 4PASSPASSPASSPASSPASSMAC

File #1, even though it captured the ADS files in both the normal and Long Filenames, the options to obtain that capture proved to be very confusing. I had to try and create the zip file over 5 times before the ADS's were properly captured. File #2, the GUI interface was nothing less than horrible to work with. So much so, I uninstalled it as soon as the tests were completed. Special note of Program #4, which is WINRAR and is used in both Linux and Windows. It is quite inexpensive (actually I think its shareware, but I paid for a license), when I first tested it, the program did not have the ability to reset the source last access date. However, with one simple request, and what I think was a reasonable evidentiary explanation, the programmers agreed to include the reset of source last access in the next version. Well, as of December 11, 2019 the version 5.80 has all the capabilities which I tested and it has passed all my tests.

I personally prefer the command line, since I have more control. Just take a look at all the available commands and options with the command line version of WinRar (called rar.exe). "Technically" there is a limitation to the length of the path/filename in WinRar. But it is normally not a concern. If you have an exceedingly long path/filename (>2047 characters) I suggest you get to reading war and peace. (just a joke). The 2047 limit should be enough for most instances. In the next section I have provided a command line that seems to work very well at creating a self extracting exe which passes all my tests. You may want to check it out.

"c:\path_to_winrar\rar"  a -sfx   -r -ts+ -tsp  -os _DEMO.EXE  -zc:\"program files"\winrar\comment.txt   folders/files-to-ad  -ppassword

The content of the comment.txt file which contains routine required options is:
The comment below contains SFX script commands which will cause the extraction of the .exe to be silent (not ask for user input) and
overwrite any existing files during the extraction. The other item which begin with a semi-colon are unnecessary and are included 
for other purposes not needed at this time.


Silent=2                                                
Overwrite=1                                             
;Setup=setup.exe                                         
;SETUP=setup16.exe          not needed    
;Presetup=hello.exe                                      
;Path=C:\temp\default_unzip_path                         
;PATH=.\.      
;SavePath                                                


the subsequent extraction/unzip command line (which is easily included in a batch file) is
C:DEMO.exe -s2 -tsp -tp+ -os -ppassword 

Even though the above process seems to work and passes all my forensic, evidentiary preservation requirements, this mention is in NO WAY and endorsement of WINRAR. Don't take my word for it, and test for yourself any zipping program you use and be comfortable with its operation. I have tested and use WINRAR when preparing all my test data and it has worked admirably.

Consider any or all of the above shortcomings when you are archiving, or preparing for discovery your files.

In short, none of the zip programs tested in this minimal test process passed all the tests. The tests included: Unicode File Name retention, LFN, ADS, reset ALL appropriate MAC dates (of source and restored files).

AND: When you actually think about it, isn't a "zipping" process a sort of copy method for retention, discovery, safe data saving? What evidence might you be missing in the zip process? Also, is next years version of the zipping program going to be able to unzip last years version. Or is product 'A' capable of processing the zip file of product 'B'.

A final thought, but not included in the above list. I tested a recent "free" version of PGP (v8). It compresses, and lo and behold, also had failures. However, since i don't use PGP to compress, only encrypt, I didn't include it in the statistics.

So, which zipping program are you using to store and restore your legacy data or evidentiary file data?

Associated articles and programs of interest: hash  program to calculate hash values. HASH_IT_OUT  an article discussing forensic hashing of evidence. COPY_THAT  an article discussing forensic copying of evidence. ZIP_IT_TAKE2  an article explaining the testing of zipping software.

Test data  containing about 30 files, in a self extracting executable