With the recent integration of Mandiant Web Historian™ into
Mandiant Redline™, you may be asking "How do I review my
Web History using Redline?" If so, then follow along as I
explain how to collect and review web history data in Redline - with
a focus on areas where the workflow and features differ from that of
For those of you unfamiliar with Redline, it
is Mandiant's premier free tool, providing host investigative
capabilities to help users find signs of malicious activity through
memory and file analysis and the development of a threat assessment
Configuring Web History Data Collection
Historian provided three options for choosing how to find web
history data that you want to analyze: scan my local system, scan a
profile folder, and parse an individual history file. Redline allows
you to accomplish all three of these operations using a single
option, Analyze this Computer, which is found under the Main
Menu in the upper left corner. Specifying the path to a profile
folder or a history file will require some additional
Figure 1: Finding your web history data
Web Historian (Left) vs. Redline (Right)
Click on Analyze this
Computer to begin configuring your analysis session. To ensure
that Redline collects your desired web history data, click the link
to Edit your script. On the View and Edit Your Script window are several
options; take a look around and turn on any and all data that might
interest you. For our purposes, we will be focusing on the
Browser History options underneath the Networktab. This section contains all of the familiar options from
Web Historian; simply select the boxes corresponding to the data you
wish to collect.
One difference you may notice is the absence
of an option to specify the browser(s) you would like to target. You
can now find that option by selecting Show Advanced Parametersfrom the upper right corner of the window. Once advanced
parameters are enabled, simply type the name of any browser(s) you
would like to target, each on its own separate line in the Target
Browser field. To have Redline collect any web history data
regardless of browser, just leave this field empty.
also notice that enabling advanced parameters activates a field for
History Files Location. As you may have guessed, this is
where you can specify a path to a profile folder or history file to
analyze directly, as you were able to do in Web Historian.
Figure 2: Configuring Redline to Collect
Browser History Data
Now that you have finished configuring your
script, choose a location to save your analysis session and then hit OK. Redline will run the script, whichwill require
Administrator privileges and may trigger a UAC prompt before running
depending on how your system is configured. After a brief collecting
and processing time, your web history data will be ready for
Reviewing your Data
For the actual review of
your web history data, you should feel right at home in Redline.
Just like Web Historian, Redline uses a sortable, searchable,
configurable table view for each of the individual categories of web
Figure 3: Displaying your web history
data for review in both Web Historian (behind) and Redline
Although similar, Redline does have a few minor
differences in how it visualizes your data:
does not break the data into pages; instead it will discretely
page in large data sets (25k+ rows) automatically as you scroll
down through the list.
To configure the table view, you
will need to manipulate the column headers for ordering and
resizing, and right-click on a column header to show and hide
columns - as opposed to using the column configuration menu in Web
Searching and simple filtering is done in each
individual table view and is not applied globally. To access the
findoptions, either press the magnifying glass in the
bottom right corner, or press Ctrl-f while a table view is
To export your data to a CSV (comma separated
values) format file, click on exportin the bottom right
corner. Like Web Historian, Redline will only export data
currently in the table view. If you applied any filtering or tags,
those will affect the data as it is exported.
addition to the features that have always been available in Web
Historian, Redline also allows you to review your web history with
its full suite of analytical capabilities and investigative tools.
Check out the Redline user guide for a full description of these
capabilities. Here are just a few of the most popular:
Timeline provides a chronological listing of all web-based
events (e.g., URL last browsed to, File Download Started, etc.) in
a single heterogeneous display. You can employ this to follow the
activities of a user or attacker as they played out on the system.
You can also quickly reduce your target investigative scope using
the Timeline's powerful filtering capabilities.
and comments to mark-up your findings as you perform your
investigation, making it easier to keep track of what you have
seen while moving forward. You can then go back and export those
results into your favorite reporting solution.
Indicators of Compromise (IOCs) as a quick way to determine if
your system contains any potential security breaches or other
evidence of compromise. Visit http://www.openioc.org/ to
learn more about IOCs.
Last but not least, Redline gives
you the ability to examine an entire system worth of metadata.
With Redline, you are not simply restricted to Web History related
data; you can investigate security incidents with the scope and
context of the full system.
If your favorite feature
from Web Historian has not yet been included in Redline (Graphing,
Complex Filtering, etc.), feel free to make a request using one of
the contact methods specified below. We will be taking feedback into
consideration when choosing what the Redline team works on in the
As always, feel free to contact
us with send any additional questions. And just in case you do
not already have it, the latest version of Redline (v1.10 as of the
time of this writing) can always be found here.
Those of you who attended the "Tools of Engagement: Redline™ -
We've Got the Tool, If You've Got the Time" webinar last month
by David Ross and myself will recall that we ran short on time while
answering all of your questions. The webinar covered the latest
updates to Redline,
Mandiant's free tool for investigating hosts for signs of malicious
activity through memory and file analysis, and subsequently
developing a threat assessment profile.
If your question was
one of those we did not get to, don't worry. We are going to cover
all of those unanswered questions in this post, as well as retread
some of those which were covered during the Q&A for people who
were unable to catch it live.
Without further ado, following
are your answers in no particular order:
Does Redline support disk images for collection and
At the moment, Redline only works on memory
images and live hosts. We are currently focusing on providing the
best possible set of analysis tools for incident response.
Does Redline work for Macs or Mac Memory images? Does it work on
Unfortunately, all of
those currently only support collection on the various Windows
platforms. However, I have heard of people having success getting
audits collected with Memoryze™
for the Mac to at least import into Redline. Be careful to
note that if attempting this, the MRI scores and other analyses may
be incorrect or invalid, as the scoring in Redline assumes it is
operating against data collected from a Windows host.
Is there a specific audit you need to run in order to do Timeline
Short answer: no. The timeline analysis will
parse any and all data with timestamps available for collection. The
comprehensive collector option in Redline is the recommended
starting place if timeline analysis is your goal as the standard
collector collects very little in the way of timestamps.
How valuable would Redline be against a virtual machine created
from a forensic image?
As long as you can log in to the
virtual machine with administrator rights to run the collection,
Redline should have no problem importing and analyzing the data
(provided it is one of the supported operating systems).
Is Redline free when used on an enterprise environment?
Redline is free to use in any sized environment, although the
collection aspect of Redline quickly becomes challenging with large
scale and globally distributed networks. This leads to the next
Can Redline collection be run on a remote machine?
Redline does not itself support remote collection. We recommend
Intelligent Response® if you would like centralized
remote collection of your hosts' data over an enterprise sized
Can you demonstrate how to use TimeWrinkles™ for events that
occur over multiple days and put them all into one view?
The easiest way to view timeline windows that are separated by
greater than an hour is to create multiple manual TimeWrinkles
around the points of time you are specifically interested in.
Using item based TimeWrinkles you can also potentially see time
entries that occur over multiple days. For instance, you could see
the actions that happened around the creating of a file, as well as
when that same file was last accessed a few days later all in one
view, just by creating a TimeWrinkle around that file.
How do you get a TimeWrinkle based on a file?
select any row within the Timeline and right click on it, Redline
will give you the option to create a TimeWrinkle based on that item.
In this case, you would just need to find the file in question
within your timeline, select it, and choose "Add a New
TimeWrinkle" from its right click menu.
Can Redline be used to pull strings from a memory image? We would
like to pull info from the csrss process to see what commands
might have run on a box.
Redline can be configured to
collect strings using the process listing audit against both a
memory image and a live machine. You can collect strings from files
with the File listing audit, but this option is only available
against a live machine.
We do recommend restricting string
collection to a single process or file at a time though, as turning
strings collection on for a full process or file listing will
significantly increase the amount of data returned and the time it
takes to collect it.
Can I export the data to a file?
Copy and Pasting from
any of the list views (including Timeline), will place up to 20k
selected rows onto your clipboard in CSV format. Using the
right-click menu's copy options also allows you to specify if you
would like to include a header row in your data or not. Full list
CSV export directly to a file will be available in the next release
Is the timeline feature available in Mandiant Intelligent Response® (MIR®)?
Timeline as it exists in Redline is not available
in MIR. But using the "open with..." feature in the MIR
Console on any audit result will allow you to import your data you
would like to timeline directly into Redline for analysis.
Is there a way to get external data sources in to Redline that
are not host-based? (ex. IDS, flow, etc.)
At the moment
Redline only supports analysis of the xml data which is collected by
the various Mandiant products listed above. Full schema definitions
for those formats can be found here.
How much alteration is being done to the suspect system by
Depending on if the collector is being run from
the host's hard disk as opposed to an external drive, the collection
and log files have the potential to overwrite some amount disk slack
space. Also if the "Preserve Timestamps" option is not
configured on your collector, some audits may modify the timestamps
for files they touch. You can find the "Preserve
Timestamps" option at Main Menu -> Redline
Options->Default Script Options->General->Preserve
Timestamps". Redline defaults this option on.
Prior to re-image, what is a good Redline collector that can
quickly get information to sift through later?
next immediate action is to re-image the box, I tend to err on the
side of collecting as much information as time permits, since there
will be no second chance to go back and recollect additional data.
But for a little bit faster collection time, I suggest starting with
the comprehensive collector and scaling back or removing the larger
audits: files, registry, and processes.
While the collector
run times depend heavily on machine in question, it is not unheard
of for the comprehensive collector to run 1-2 hours. By limiting the
files audit to a specific base path like the Windows directory or
the System32 directory, and limiting your registry audit to a few
specific keys (i.e.
HKEY_LOCAL_MACHINESoftwareMicrosoftWindowsCurrentVersionRun) you can
significantly reduce the time that your collection will take to run.
If you need to collect even quicker, consider turning off the
various hashes like MD5.
Can I run this tool via command line?
runs as a batch script via the command line, but the analysis and
visualization portions are only available through the graphical user
Are you limited to the types of variants/intrusions you can scan
The Malware Risk Index (MRI) scoring configuration is
limited in the types of things it allows you to look for, but within
those confines you can add or tweak to your heart's content.
Indicators of Compromise (IOC) provide a much more flexible
definition format to describe what malware you would like to search
for. The first Tools of
Engagement: Redline webinar walks through an example of
creating a new MRI rule and an Indicator of Compromise in the course
of performing the investigation and applying them to a Redline
Can you tell if the attacker has manipulated the time stamps of
Redline does not automatically detect
timestamp manipulation. But often an experienced eye can pick it out
by looking for things such as every potentially suspect files
encountered having "00" for their seconds place, or
similar statistically improbable occurrences.
Is there any known malware that targets Redline? Have you had any
difficulties with attackers running their tools inside rootkit
We have yet to encounter any malware that has
specifically attempted to avoid collection or detection by Redline
and its various analysis techniques. As for general rootkit
protection, Redline uses raw disk access by default where possible
to avoid being subverted by rootkits.
What metadata shows how many times an executable has
Prefetch files (.pf) are windows specific cache
files to improve application startup performance. They contain the
first and last run time as well as how many executions have occurred
in total. These files are parsed and their relevant data captured by
the Prefetch audit available in the Redline Collector setup.
That wraps up all of unanswered (as well as answered) questions!
And just in case you do not already have it, the latest version of
Redline (1.7 as of the time of this writing) can always be found
Back in November I published the first interview from theHighlighter™ Super Users blog series.
My goal with this series is to shed some light on all the great
things that can be achieved using this freeware tool. In part 2, I
interviewed toolsmith author and HolisticInfoSec.org
webmaster, Russ McRee.
Super User Interview #2: Russ McRee
Russ McRee is the
author of ISSA Journal's toolsmith series and runs
HolisticInfoSec.org. In October 2011 Russ contacted me to discuss
Highlighter in that month's issue of the ISSA Journal, and later for
the nomination of Highlighter for the 2011 Toolsmith Tool of the
Year. As someone who has analyzed Highlighter's effectiveness as
a forensics tool for his own articles, I asked him to answer a few
questions based on his experience with the freeware tool.
Name Russ McRee
Realm of work Security Analytics (security incident management, security
monitoring, attack and penetration testing).
How did you hear of Highlighter? I watch the websites and check for tool updates.
Do you know of any other tools that do what Highlighter does? Log Parser, Log Parser Lizard, Log Parser Studio,
Can you describe one scenario in which Highlighter helped you
find evil and/or solve crime? I had a recent mysterious case of core utility files and
binaries gone missing from very important infrastructure
management servers that initially looked malicious and
intentional. Using Highlighter for analysis of Windows event logs
led to the discovery of a sync job gone awry (misconfiguration) in
the Application log via time stamp matching and keyword
On a scale from 1 (worst) to 5 (best), how well does
Highlighter address your use case(s)? 4
What is missing from Highlighter for your use case(s)? Word wrap option
What is one Highlighter feature addition that would serve the
Information Security community best? Potential DB support
Are you aware of, or have you used, any of the following features:
Activity Over Time feature that lets you view log data as a
function of Entries Per Day No, I was not aware.
Hotkeys feature Yes, I was aware of this feature.
Ability to change basic font settings for your output Yes, I was aware of this feature.
Have you ever seen Highlighter used in such a way that your
eyeballs melted from all the Awesome? My eyeballs melted from the awesome when I stuffed
Highlighter with a 2.44GB Swatch log file during large file
testing while writing October 2011's toolsmith. It took a
little time to load and format (to be expected), but it handled
24,502,412 log entries admirably (no choking). I threw a query for
a specific inode at it and Highlighter tagged 1930 hits across 25
million+ lines in ten minutes.
Keep an eye out for
the final post in the Highlighter Super Users Series. If
you're interested in sharing your own experiences with this tool,
please let me know by commenting below.
As we are mere hours away from celebrating 2013, we'd like to
focus today on M-Unition's Armory channel. The Armory is the place
to be if you want to be the first to find out about the latest
releases, free tools and of course, our ever popular M-Trends
report. The most popular posts in this category are listed below for
your reading pleasure.
This past year we made several product
announcements, but this one was especially rewarding. When you deal
with cybersecurity risks on a daily basis you need tools to help you
see activity in real time. At MIRcon ™ 2012, we announced our newest
product offering: Mandiant Cloud Alert™. Mandiant Cloud Alert is a
powerful tool, enabling organizations to identify malicious
communication, audit existing security measures, monitor how the
organization is trending over time, and track incidents in their
Memoryze for the Mac 1.0 brings many of the features of
Memoryze™ to the Apple Macintosh platform. This
new tool enables acquisition of memory images via the command-line
or a simple GUI. In addition, Memoryze for the Mac 1.0 can perform
offline analysis against memory images or live analysis on a running
Freeware tool, Shim Cache Parser™, was developed in the
course of our incident response investigations, according to
Mandiant's Andrew Davis.
During keyword searches of
compromised systems, the Mandiant team discovered known malicious
file names in the Windows Registry. Further research showed the
cache data was generated by the Windows Application Compatibility
Database. Along with these file names, other types of file metadata
can be recovered such as file size, file last modified times, and
last execution time, depending on the operating system version. This
data can be very useful during an incident response. It helps
identify which systems an attacker may have executed malware on and
can also provide information about the time that it may have
Shim Cache Parser is the proof-of-concept tool we
developed to extract this useful forensic evidence. You can download
Mandiant's Sean Cunningham and Mark Thomas discuss
the availability of a highly efficient reverse HTTP(S) proxy called
simply 'RProxy™'. Mandiant released RProxy as an open sources tool
to encourage the general community to participate in its evolution.
You can download the tool here.
Each year Mandiant takes a look back at
engagements we've responded to and puts together trends that help
you fight back against targeted threats. On March 6th, we released
our latest M-Trends report, An Evolving Threat, which revealed key
insights, statistics and case studies illustrating how the tools and
tactics of targeted attackers, including the Advanced Persistent
Threat (APT), have evolved over the last year. We're currently
working on the 2013 edition of M-Trends and plan to release it at RSA
Redline™ is Mandiant's free tool for investigating hosts for signs
of malicious activity through memory and file analysis, and
subsequently developing a threat assessment profile. It combines
configurable collection of Mandiant's full range of forensic
artifacts (the same set available to our enterprise product, Mandiant
Intelligent Response®, guided analysis, Mandiant's Custom
Malware Risk Index (MRI) scoring, and Indicator of Compromise (IOC)
matching) to provide you with the tools needed to rapidly triage a
potentially compromised host.
Recently, we released Redline
version 1.7. To make this the most compelling release to date,
we focused on two of the most frequently requested features:
Timeline and Search.
With Timeline, we wanted to do more than just
slap together all date related items into a staggering list that
buries you under more data than is humanly possible to review. Even
on infrequently used systems, hundreds of thousands of dated
artifacts are collected using the Redline Comprehensive Collector.
To help narrow a complete timeline down to a manageable subset of
data we have provided you with a trio of powerful filtering
capabilities: Field Filters, TimeWrinkle™ and TimeCrunch™.
You can use these filters to shave down your timeline to a
manageable number of events . We have made it simple to quickly scan
your list for suspicious activity by providing you with a summary
column that highlights valuable information about any given event.
But do not fear, if you need to investigate any item in further
detail, you can select that event and choose the "Show
Details" button to view everything we have collected.
Figure 1: The new Timeline
view in Redline
Field filters are a straightforward
means of excluding or including entire categories of time related
events from your view by simply checking those that you care about.
For example, we often find that File Accessed events tend to be very
noisy. With field filters you can remove them from view until they
are absolutely necessary.
Even when you use field filters, we
realized that there was still too much data for manual review.
Naturally, we turned to our expert professional services consultants
to see how they comb through large quantities of time-oriented data.
Unable to find a good solution that met their needs, they developed
the concepts of TimeWrinkle and TimeCrunch. We worked closely with
them to bring these tools to Redline's new Timeline
TimeWrinkle provides you with the means
to filter your Timeline view to display only the events that
occurred in a set of configurable windows of time. TimeWrinkle comes
in two varieties: custom and item-based. Choosing which variety to
use and when simply depends on the type of investigative lead you
If you know the general time when suspected
malicious activity occurred (whether from a user IT ticket, an IDS
log, or some other similar means) you use a custom TimeWrinkle to
restrict your timeline to only events that took place around that
region of time. If the default five minute window radius is not
sufficient, you can adjust it anywhere between 0 and 60 minutes to
better suit your needs.
However, if you know something more
specific about the suspected malicious activity (such as a file name
or MD5 checksum) you can use an item-based TimeWrinkle. Creating an
item-based TimeWrinkle will take a selected item (e.g. File,
Registry Key, Process, etc.) and narrow your timeline to events that
take place around any of the of the associated timestamps for that
item. To create an item-based TimeWrinkle, right click on any item
in the Timeline and select "Add New TimeWrinkle". This
will use default settings to generate a TimeWrinkle around the
selected item which can then be edited to exclude, include, or
adjust any of the individual field windows within the
Figure 2: Timeline Configuration
There can still be times where you have
too much data to review manually even after you have trimmed your
timeline to a narrow window using TimeWrinkles. To further aid you
in reducing your data, we also provide the ability to trim out a
minute worth of events for a specific field by using a TimeCrunch.
TimeCrunches are useful when Field Filters (being applied so
broadly) are detrimental to your investigation, and instead you need
to remove specific event types in a more surgical manner.
most common example of this is when an antivirus scan updates the
file accessed timestamp on a very large number of files in a very
short amount of time. When this occurs, the file accessed timestamp
will become too noisy to be of investigative use for the window in
which the antivirus scan ran. Applying a TimeCrunch can quickly
exclude a minute worth of this cluttersome data without losing
potentially relevant file accessed timestamps elsewhere in your
timeline as with Field Filters.
Figure 3: TimeCrunch
Now you are probably saying to yourself,
"Timeline sounds awesome, but what do I do if my investigative
lead is not something that is time based?" For example, what do
you do if your lead is a potentially compromised credit card number.
No worries, we have considered you as well.
Now standard on
every list shaped view in Redline is a full featured search
capability. For example, by using the search feature you can quickly
search the entire list of strings from all processes in memory to
determine if the suspect credit card number was present. If any
matches are found, we will highlight and scroll to each of them. And
if your investigative lead is less specific (e.g. you suspect that
some unknown credit card numbers may have been stolen) we also allow
you to specify your search criteria as a Regular Expression.
For our current Redline users, upgrade
to this latest version of the freeware tool to take advantage of the
new features. For new users, don't wait another minute to download
and get your hands on this great set of analysis tools. Please be
sure to check out the updated user documentation for more details
and drop by the Redline
section of the Mandiant forums to give us feedback on your
experience and post any questions you may have.
The updated release of IOC
Editor has been a long time coming, but it is well worth the
upgrade. There have always been grumblings about IOC Editor, but
lately those grumblings have been growing louder. The noise
eventually got so loud that even my noise canceling headphones
couldn't silence it. I could have just turned up the music a little
more, but instead I decided to grab the code and fix some of the
issues once and for all.
Some of the changes to IOC Editor
were minor (Ctrl-X no longer exits the program), some were major
(check out the new Properties panel where commenting per
IndicatorItem is now available), and of course I discovered some
bugs that I didn't even know existed (No, I didn't cause them with
my other changes). Long story short, there are a lot of new features
and improvements in this version.
Make sure you pay attention
to what's different and any messages that pop-up as the wording and
options may have changed.
In this blog post, I am going to
touch on a couple of new features and improvements. If you would
like to see the full list, please check-out the release
notes. Fair warning, there is a lot to read as I did get a bit
Some of the New:
This is still a bit of a work in
progress, but there is now a place to set some of the defaults for
1: Option Dialog
Figure 1 shows the preferences that are currently
available in the options dialog. You can set the default author (we
typically use the e-mail address to identify the author), and set
whether you want to warn on Deleting or Pruning.
we go too far, let me quickly discuss the difference between
deleting and pruning:
Deleting will only delete the selected
item, whether it's an individual item, or a logic item (AND, OR). If
you delete a logic item, the items that are below it will be bumped
up a level.
2: Before DeleteFigure
3: After Delete
Figure 2 shows what the Indicator of Compromise
(IOC) looks like before deleting the AND. Figure 3 shows what the
IOC looks like after deleting the AND. Notice that the items that
were under the AND were moved up a level and are now under the top
There is no problem with this IOC, but what happens if you
were to delete the AND in the following IOC?
4: Before Delete
If you delete the AND that is shown in Figure 4,
you will end up with the "File Name is not good_file.exe"
under the top OR. If you were to search your enterprise for all
files that were not "good_file.exe", that would be very,
Now, onto pruning. The Prune option is only
available for logic items (ANDs and ORs) and will remove the entire
Going back to the example shown in Figure 4, if
you Prune the selected AND, the AND, in addition to all items under
it, will be deleted.
5: After Prune
You can see in Figure 5 that the only items that
remain were directly under the top OR.
Now that we have that
cleared up, let's continue with the rest of the cool new stuff in IOC
Everybody loves keyboard shortcuts;
the less we have to type-out, the better. With that in mind, I've
gone through and added shortcuts where appropriate. Below is a list
of the shortcuts currently available in the editor:
Add Buttons (AND, OR, Item)
The buttons to add an AND,
OR, or Item have been moved to a menu bar above the definition area.
This takes up less space and allows for future expansion.
6: New Add Buttons
If you just click the button, the
"Item" button remembers what you recently added. It also
has a drop down with a list of terms (same as right-clicking in the
definition area and selecting "Add Item.")
The properties panel is an area to
the right of the definition area that will show all pertinent
details of the selected item. By default, the properties panel is
not shown. To display it, either click the button or use the Ctrl+P
For those who have used Visual Studio
before, the properties panel will look very familiar.
Figure 7 shows the details that are given for a
selected item. Not all of the items are editable from here, but it
at least gives you a view into what's going on
You can also edit/add comments for the item.
With this upgrade, you can add a comment to individual items, such
as an MD5, that lets someone know what file it actually goes
In addition to the new features,
there were a number of improvements (aka bug fixes).
Unsaved Changes Warning
An important change was the
rewording of the unsaved changes dialog box that better aligns with
other standard Windows applications.
NOTE: Read this carefully as this has changed from the
previous version of IOC
8: Unsaved changes
Figure 8 shows the new wording presented when
exiting IOC Editor with unsaved changes.
Yes - saves changes and exits
No - does not save changes and exits
Cancel - cancels the exit process and does not save
The Description field has
been tweaked a bit, and now allows for proper display of carriage
returns and tabs. You now don't have to deal with descriptions that
look like this:
"This malware is evil⌷Why would
anyone have this⌷Report Immediately"
Too Many More
As I stated earlier in this post, I went
a little bug fix crazy. If you're interested in a full list of bug
fixes, please click
There are still plenty of other
features and enhancements that I would like to make to IOC
Editor, such as ioc_lint integration, additional options, etc.
I'm also looking forward to your feedback on the current
The Highlighter™ Super Users series is a little
something I've put together to reach out to the Highlighter
community. As a user of this freeware tool from Mandiant, I want you
to know there are many users out there who can help you get through
your log analysis paralysis. This series is meant to highlight (see
what I did there?) how some users have solved a various range of
problems using Highlighter. These interviews will provide insight
into the benefits and pitfalls of using Highlighter, some features
you may not be aware of, and a few use cases you may not have
Super User Interview #1: Ken Johnson
Ken Johnson is
one of Highlighter's Twitter-friendly users. He is a malware analyst
and incident responder extraordinaire; fighting evil one keyword
search at a time. Known as @patories on Twitter, I reached out to him and
asked some questions about his experience using Highlighter.
Name Ken Johnson
Realm of work My primary work is focused on malware analysis and incident
response. Occasionally I also do some forensics work.
How did you hear about Highlighter? I first saw Highlighter when I was familiarizing myself
with free tools. I have used Memoryze™ previously.
Do you know of any other tools that do what Highlighter does? Highlighter is the only tool I know of, and it does what I
need so I haven't looked for others.
How do you normally use Highlighter? I use Highlighter to trim out known good traffic from proxy
logs. This helps get to the unknown stuff quicker. When logs can
be multiple gigabytes this is a time saver.
Can you describe one scenario in which Highlighter helped you
find evil and/or solve crime? On more than one occasion I have used Highlighter to narrow
down proxy log traffic to find connections that are malicious.
There was an instance about 2 months ago where users fell for a
Phish. We used Highlighter to find the C&C IP's that machines
kept calling home to, by filtering out what was normal and
analyzing what was left. Highlighter helped find almost 50 IP/URLS
that were malicious.
On a scale from 1 (worst) to 5 (best), how well does
Highlighter address your use case(s)? I would have to give Highlighter a 4.
What is missing from Highlighter for your use case(s)? I would like to have the ability to whitelist traffic so I
do not have to manually keep removing internal hosts that we see.
This may be in the program and I have not found it.
What is one Highlighter feature addition that would serve the
Information Security community best? I think the ability to whitelist hostnames would be a nice
Are you aware of, or have you used, any of the following features:
Activity Over Time feature that lets you view log data as a
function of Entries Per Day No, I was not aware of this one.
Ability to change basic font settings for your output I know it is there, but for my use this is never
Have you ever seen Highlighter used in such a way that your
eyeballs melted from all the Awesome? I have only seen myself use it, but I have seen my
co-workers eyeballs melt when I show them the awesomeness that
they can do. Some are still stuck in the grep world...
Keep an eye out for the second post in the Highlighter Super
Users Series featuring Russ McRee, author of ISSA Journal's
toolsmith series and mastermind behind www.holisticinfosec.org.
If you're interested in sharing your own experiences with this tool,
please let me know by commenting below.
Earlier this year, Mandiant launched a new freeware tool: Memoryze
for the Mac™. The tool brings many of the features of Memoryze™to the Apple® Macintosh platform, enabling
acquisition of memory images via the command-line or a simple GUI.
We are excited to announce it now fully supports OS X 10.6-10.8.
Recently, OS X Mountain Lion added kernel Address
Space Layout Randomization. It is a welcome security feature,
raising the bar for kernel exploitation. This feature adds an extra
step into the memory analysis tool. Previously, we could depend on
the paging table IdlePML4 and IdlePDPT addresses being at the same
physical memory location. With 10.8 and KASLR the physical memory
addresses of IdlePML4 and IdlePDPT became BootPML4 and BootPDPT,
while IdlePML4 and IdlePDPT are now randomized with ASLR. The boot
paging tables do not contain the full kernel virtual memory layout.
Since Memoryze for Mac does not depend on any symbol information, we
developed a mechanism to uniformly discover the randomized location
of the kernel paging tables.
Once again, Mountain Lion has
changed the memory location of nsysent. Prior to the change, it was
located directly after the sysent table itself. As documented in
several locations on the web, this made automated discovery and
verification of the table size convenient. Unfortunately, Apple
decided to move the location of nsysent, causing us to develop a new
sysent size discovery mechanism.
We have a growing list of
cool new features to add to Memoryze for Mac, but it may be until
after the new year before we are able to dev the features.
On today's podcast, Kristen Cooper talks with Lucas Zaichkowsky on
the latest version of Redline, a free tool from Mandiant.
podcast will explain in detail what Redline is capable of,
highlighting two features that set it apart from other tools. First,
the tool is intuitive enough to be used by novice incident
responders, without compromising capabilities that advanced incident
responders utilize in the tool. Secondly, the tool is capable of
applying Indicators of Compromise (IOC) to data that it collects.
This allows Redline to detect evidence of attacks, even though there
may be no evidence of active malware on additional computers.
Listen along as Lucas details the product demonstration he
performed at Black Hat 2012 that really showcases Redline's unique
To listen to the full podcast and learn more about
Redline click here.
RedlineTM and IOC
Finder TM collect and parse a huge body of
evidence from a running system. In fact, they're based on the same
agent software as our flagship Mandiant
Intelligent Response® product. During the course of their
"audits", these tools conduct comprehensive analysis of
the file system (including hashing, time stamps, parsing of PE file
structures, and digital signature checks), registry hives, processes
in memory, event logs, active network connections,DNS cache
contents,web browser history, system restore points, scheduled
tasks, prefetch entries, persistence mechanisms, and much more.
Once this data is collected, Redline and IOCFinder currently
allow you to do one of two things:
Review the contents
of memory through a visual workflow in Redline
Indicators of Compromise (IOCs) and generate a report of
But what if you want to analyze all
of the raw evidence - not just memory or IOC hits - and do
traditional forensics and timeline analysis? That's where Audit Parser
steps in. It's the newest addition to Mandiant's portfolio of free
Audit Parser is simple:it takes the complex XML
data produced by Redline or IOCFinder and converts it into
human-readable tab-delimited text. You can then easily review the
output in Excel, use a dedicated CSV file viewer (we're fans of
"CSVed" and"CSVFileView"), import it into a
database, or grep / manipulate it to your heart's content.
When paired with Redline's new start-up workflow to build a
"collector" script, Audit Parser gives you a complete(and
free)live response analysis toolkit. You can customize the Redline
collector to gather as much or as little evidence as desired, run it
on your target system, and then easily review all of the results
following a quick conversion with Audit Parser.
capture below shows Audit Parser's options - it's pretty
straightforward to use:
Tabular data in Excel doesn't make for the most exciting screen
shots, but we wanted to give you a glimpse into what the output
looks like and the extent of evidence available for filtering,
sorting, and analysis:
A filtered view of a file system
audit, showing complete file metadata for all PE files within
%SYSTEMROOT% created between 2011-2012 that are not digitally
A portion of a prefetch audit, showing how the contents of
.PF files are automatically parsed to provide last time executed,
# of times executed, and original file path metadata.
A portion of a full registry dump, showing review of Active
Setup Installed Components registry keys - the data includes all
key value / data pairs and last modified dates.
A portion of the parsed Windows event logs, showing review
of process auditing events including event log source, time
generated, event ID, and full event message contents.
The default "comprehensive collector" script in Redline
collects all of the artifacts listed above, as well as many
But wait - that's not all! Audit Parser also contains
timeline generation functionality. Just specify a time & date
range, and it will build a sorted timeline of all file system,
registry, and event log events that occurred within that period.
Future releases will add more audit types and customizability to
Audit Parser is written in Python and is
distributed under the Apache License. It requires the lxml (http://lxml.de/) library. We're also
distributing a Windows EXE built with Py2EXE for users that may not
have a Python environment set up. You can download the tool and
documentation on GitHub at: https://github.com/mandiant/AuditParser
If you have any questions or comments, feel free to leave them
below, e-mail me (ryan [dot] kazanciyan [at] mandiant.com), or DM me
on Twitter at @ryankaz42. I'll also be at Black Hat USA next
week teaching Mandiant's
Incident Response course where we'll be going through an
in-depth live response analysis lab using Redline, Audit Parser, and
other forensic tools. I was on a recent M-Unition podcast discussing
the class and how it is completely revamped for 2012. You can listen
to the podcast here. Hope to see you there!
Today, Mandiant is introducing a new free tool, Memoryze™ for the Mac 1.0,
which brings memory imaging and analysis to the Mac. It joins a
growing list of freeware tools Mandiant currently
Memoryze™ for the Mac 1.0 brings many of the features
of Memoryze™to the Apple Macintosh platform. This new tool enables
acquisition of memory images via the command-line or a simple GUI.
In addition, Memoryze™ for the Mac 1.0 can perform offline analysis
against memory images or live analysis on a running
The tool supports the following features:
Imaging the full range of system memory
individual processes memory regions
Including those hidden by rootkits
For each process, Memoryze™ for the Mac 1.0
Report all open file handles in a process (e.g. all
files,sockets, pipes, etc.)
List the virtual address
space of a process including loaded libraries and allocated
portions of heap and execution stack
Active and listening
All loaded kernel extensions including
those hidden by rootkits
The System Call Table and
Mach Trap Table
All running Mach Tasks
Okay, enough of the marketing. Memoryze™ for the Mac
1.0 can be downloaded here. To help get you started we'll present
a few of the features in this blog post.
For offline analysis
your first step is going to be acquiring memory. The Mac Memory
Dumper App makes this process as simple as pushing a button. To
begin the acquisition, Memoryze™ for the Mac 1.0 will require you to
authenticate so that the application can load a memory dumping
driver. After selecting the location to store the image and
authenticating, Memoryze™ for the Mac 1.0 will begin the acquisition
process. The tool will provide you with a progress bar and an image
size monitor for each of your acquisitions (fancy, huh?).
Note that the final size of the dumped image may exceed the size
of your physical RAM. If the system has 8GB of physical RAM
installed the dump may be 10GB. You may ask yourself, "Self,
why is the dumped image bigger than my actual memory size?"
There are regions that are physically addressable but are not part
of actual DRAM, pesky memory-mapped devices. These regions are
written to the image file as 0x0-bytes to help preserve the correct
offsets within the image.
Once Memoryze™ for the Mac 1.0 finishes the acquisition; we can
use it to perform memory analysis(note Memoryze™ for the Mac 1.0 can
also perform analysis on images acquired by other tools). With our
data ready, let's run through several of the process analysis
features we mentioned above.
We'll start by performing a
basic process listing based on the memory image we just created.
Execute the command below:
Memoryze™ for the Mac 1.0 will open "my.mem" for
analysis and detect two critical pieces of information about the
image: the operating system version and whether the system is
running 32 or 64-bit kernel. Armed with this information the
proclist analysis module locates the operating system data structure
that maintains the list of running processes.
If you take a
look at the output below, you can see that Memoryze™ for the Mac 1.0
extracts the PADDR, or physical address, of each process (this is
also the offset of the process in the acquired memory image file).
You can use the PADDR to quickly locate the process in question in
an offline tool such as a hex editor. Memoryze™ for the Mac 1.0 also
extracts other standard identifying information such as the process
NAME, PID and parent PID (PPID). In addition, the tool provides the
start time for each process in UTC. Finally, Memoryze™ for the Mac
1.0 extracts each process' associated USERNAME, effective userid
(EUID), and real userid (RUID).
Based on the process listing
above, we may be interested in getting a more complete understanding
of what a particular process may be doing. We can list file
descriptors and memory sections for all of the processes in the
listing, but this would get pretty lengthy and present too much
information at once. Using filters we can limit display to a smaller
subset or a single process. We can use the "-n" option to
filter processes by NAME or the "-p" to filter processes
by PID.Execute the following command:
In the image above, we show a snippet of a file descriptor
listing for PID 14 using the command-line switch "-p 14".
This shows us all open file descriptors for the given process. This
includes files, UNIX domain sockets, networking sockets (such as
TCP), and so on. For several of the types/subtypes, Memoryze™ for
the Mac 1.0 will provide a value associated with the item type. The
value for a descriptor of type FILE is the filename associated with
the file descriptor while the value for a type SOCKET subtype TCP is
the source IP address, destination IP address, and associated
Now that we've completed some filtering, let's dig a
little deeper and perform detailed analysis of the
"notifyd" process. The image below contains a snippet of
its memory sections listing. Memoryze™ for the Mac 1.0 shows us the
start and end virtual addresses and a human-readable size for each
section. For some memory sections, Memoryze™ for the Mac 1.0 also
provides a type, such as MALLOC or IOKIT.
These types provide insight into the purpose of the memory section.
For other memory sections, Memoryze™ for the Mac 1.0 displays the
filename that is located at (or was used to initialize) that
particular memory region. Please execute:
Neat, huh? So now we've analyzed the standard operating system
(OS) process list structure. If it's good enough for the OS, it's
good enough for us, right? Not really. It's fairly trivial for
malware to unlink itself from the process list that the OS
maintains. In light of this, Memoryze™ for the Mac 1.0 provides a
process carving feature that allows it to enumerate and analyze
processes based on their signature in memory. This means that
Memoryze™ for the Mac 1.0 does not have to depend on the OS to
provide a list of processes. This enables Memoryze™ for the Mac 1.0
to discover processes that have been hidden from standard OS
listings. This same carving feature extends to the kextlist and
syscalllist Memoryze™ for the Mac 1.0 features, allowing you to
discover other hidden data within the OS.
forget, Memoryze™ for the Mac 1.0 can be run offline using an
acquired memory image, or live, analyzing the running system
in real-time. Below we show a system call table listing using the
live memory analysis feature of Memoryze™ for the Mac 1.0.
Notice the missing "-f" option. We can use this listing to
check for system call table hooking. System call table hooking
allows attackers to surreptitiously monitor or filter user-level
programs interactions with the OS kernel. This is commonly used to
hide files and network connections from user-level programs. The
syscalllist feature also supports discovery and listing of the Mach
Trap table. Mach Trap, what? The mach trap table is analogous to the
system call table in the BSD portion of OS X, but within the Mach
portion of OS X. So we want to ensure we can check for possible
hooking within that table also. To perform a live listing
sudo macmemoryze syscalllist -s
In order to hook the syscall table we would probably want to use
a driver. Have no fear! Memoryze™ for the Mac 1.0 can carve loaded
kexts from memory. A decent malware author would probably want to
hide itself from the OS by unlinking from internal data structures.
To combat this, Memoryze™ for the Mac 1.0 will parse live memory or
a dead file to find loaded drivers, as shown in the screenshot
Memoryze™ for the Mac 1.0 supports an XML output option
("-x") that will create an XML file with an extended
version of the console output, only in XML format. "Extended
version," you ask? There is only so much console real estate.
We must make decision about what information to display. So
therefore, stuff gets left out. For example, we parse the full file
path of the executing process and the process arguments. These are
not displayed in the console output, but are accessible if using the
XML output option. These XML files can also be loaded into Mandiant
Redline™ (currently windows only, boo!) for viewing in a
There are other features of Memoryze™ for the Mac 1.0 that
we have not detailed here, but we don't want to give you all the
answers. What's the fun in that? We really want you to use the tool
and provide us with some feedback on features, interface, usages,
and so on.
Memoryze™ for the Mac 1.0 currently supports:
Mac OS X Snow Leopard (10.6) 32/64-bit
Mac OS X
Lion (10.7) 32/64-bit
We hope to continue to improve
the state of Mac memory analysis for incident responders and
"Mac" is a trademark of The
Apple Corporation. Mandiant is not affiliated with or endorsed by
The Apple Corporation.
Mandiant is introducing a new free tool today, PdbXtract™,
which allows you to browse and search PDB-type information.
PdbXtract allows you to explore symbolic type information as
extracted from Microsoft PDB files. This tool is primarily designed
for reverse engineering Windows-based applications and for exploring
the internals of Windows kernel components. You can download PdbXtract.
A programming database (PDB) file is a binary file containing
program debug information in a Microsoft-proprietary format. This
file is produced by the compiler/linker when a program is built. The
information it contains is used by debuggers to debug a program and
can greatly assist a developer in debugging program issues by
resolving function pointers to symbolic names, for example.
Perhaps the most useful and richest source of debugging
information contained in PDBs is type data which holds detailed
information about data structures, constants, and other named
symbols. While this information is primarily used to debug program
components, it can also be used to gain insight into how core
operating system components work by observing both the format of the
data structure and how the structure is used.
not a pure PDB parser. It only extracts type information using
Microsoft's DebugInterface Access (DIA) COM. If you are interested
in just parsing/dumping raw PDB information, there are a few
alternatives out there to DIA, including Volatility's open source
or the PDB utility that comes with the Undocumented Windows 2000
Secrets book. However, most of the practical tools I have seen that
operate on PDB's use DIA, including Microsoft's own Dia2dump, this
and this one http://www.ishani.org/web/articles/obsolete/pdb-cracking-tool/,
to name a few. To reiterate, PdbXtract does not parse or capture the
wealth of other information available in a PDB, including:
functions, debug streams, modules, publics, globals, files, section
information, injected sources, source files, OEM specific types,
compilands, and others.
The tools mentioned above are fine
for inspecting the contents of a single PDB. However, often times as
part of my job in R&D, we have to use knowledge of type
information across all supported Windows operating systems to
implement features. For example, if you are dealing with partially
undocumented or "opaque" types (example: you need to walk
the PEB's InInitializationOrderModuleList to obtain a list of loaded
modules in a process) or have full source type information but do
not want to tie your program to a specific version of those types as
implemented in the headers of the SDK you are compiling against, you
probably want to just use static offsets such as:
The problem has always been: how do I get the value of
InInitOrderModList_Offset for all platforms we support, taking into
account 32-bit/64-bit variations? The answer has always been:
useWinDbg (or if you are interested in possibly-correct kernel
symbols only, you might use Matt Suiche's Moonsols library (http://msdn.moonsols.com )).
Launch a VM for each OS you want to support, attach with a debugger,
and use the power of WinDbg to extract the type information. Well,
WinDbg's magical "dt" command just relies on the PDB
information for the corresponding binary (after retrieving the
necessary symbol files from your local symbol store and optionally
the Microsoft public symbol server), so it stands to reason that we
should be able to do the same. The end goal is to make a searchable
database for all the exported types of OS binaries we care about, so
that we don't have to constantly relive the tedium of doing this in
PdbXtract has two main features: exploring a
single PDB (PDB Explorer) and searching a library of PDBs for one or
more operating systems. PDB Explorer opens the PDB, parses type
information using DIA, and displays a list of all structs, unions
and enums. If you click on one of the types, a C-style struct (with
offsets) definition will be displayed in the text area to the right,
as shown below for the type IMAGE_FILE_HEADER.
The library tab allows you to create and
search a library of PDB type information. I have created a library
for *most* of the operating systems we support for the following
important system binaries: kernel (ntoskrnl.exe, ntkrpamp, ntkrnlpa,
ntkrnlmp),ndis.sys, win32k.sys and hal (hal.dll, halaacpi.dll,
halmacpi.dll). You might ask why other system DLLs were not
included, such askernel32, user32, advapi32, etc.The answer to that
being the corresponding PDBs for those binaries you get off the
symbol server are stripped of type information. Why? Because
Microsoft expects you to use their headers when you compile your
application, and thus your program'sPDB will have the
necessary type information.
The library included with
PdbXtract covers several of Microsoft's major operating system
releases, but you can easily add more symbols. PdbXtract includes a
utility, called PdbFetch, that simply runs Microsoft's symchk
utility to grab the symbols for the file names you supply (usage:
pdbfetch, where is a text file that contains a list of full paths to
system binaries you want to retrieve symbols for). Pdbfetch creates
a "PDB set" which consists ofthe directory structure with
containing PDBs as created by symchkplus a manifest.xml file which
summarizes the OSplatform information. To use a PDB set in
PdbXtract, go to the library tab and click "new" if you
want to create a new library from the PDB set or "add" if
you want to add them to an existing library. Once you create/add a
PDB set to a library you can delete them - the only thing that
matters is the sqlite .pdbx database that's created.
someone out there will find this useful and maybe create a
searchable web front end with the resulting SQLite database? The sky
is the limit. Let me know if you do by commenting below.
final note, you might wonder why you can't just download the entire
symbol packages from Microsoft, which include every symbol file on
the MS Symbol server, and create a ginormous library. Why is there a
requirement to acquire the PDBs using pdbfetch? The answer is - you
could do that - but it is data overload(several GB of PDBs) when you
will not care about 99% of them. Plus it is easier to capture OS
platform/build info at run time rather than guessing at it from the
name of the symbol package installer (PDBs give no indication of
what OS the corresponding binary originated from).
We are on a roll with our freeware. The latest version of Redline
is now available! For those who are not familiar with Redline - you
may be asking, what is it? Simply put, Redline brings together
analysis tools which help you perform a guided investigation of a
potentially compromised system. And did we mention that it is
This latest and greatest version of Redline includes some
awesome new features, courtesy of recommendations from our strong
and growing user base and input from internal users here at
Mandiant. For those who have been loyal Redline users, you will find
that it is no longer just a memory forensics tool! It has grown into
a multi-purpose product for creating Indicators of Compromise (IOC)
and matching them across all types of host data, while maintaining
all the traditional memory forensics capabilities that you're used
Get the data that matters, and do it faster
Redline, you can now include and search for Indicators of
Compromise and create a searchable report detailing any suspicious
activity found matching those IOCs. Need more on what IOCs are? Click here for more
Specify a set of IOCsbefore collection and
Redline will now help tailor the configuration to provide
meaningful search resultsand ensure that all the data required by
the chosen IOCs is collected, speeding up your time to
Not sure if the IOCs you have chosen are the
ones you want? Not to worry! When choosing indicators to search
for, there is now a handy preview window to see the detailed
information of each indicator.
You are no longer limited
to just memory data. Redline now enables you to configure and
collect a much broader range of data about the target host, such
as event logs and file listings. This data will in turn be
searchable using the new Indicator of Compromise search options,
providing you with better overall search results.
Multi-task with the best
With Redline you can
now perform investigations while searching for indicators - at the
same time! For example, while the session is still matching IOCs,
you can start diving into the Malware Risk Indicator (MRI) Scores
and start anew investigation or even continue an existing
Now there's no guessing where you are in the
process. You can check the progress of your investigation at any
time via "Background Tasks" in the main menu. You will
also receive a notification when one of your background tasks has
For our current users, be sure to upgrade
to this latest version of Redline to take advantage of the new
features. For new users, don't wait another minute to download Redline
and get your hands on this great set of analysis tools.