"Get-Winevent" PS scripts to help convert all your Archive-Security*.evtx logs to csv

These (Powershell 3.0) scripts will convert archived Security (auditing) logs. . If you run these at night, configure the advanced settings of your laptop to forbid automatic sleep/hibernation. Note that the field names for an evtx file are different than those you would use to query an existing (working) event log. Of the four scripts I test 'Function Convert-Logs3' has the best combination of optimized resource use and speed.

Finding Whole Squares for a given number range in Powershell

I have never been very good at math.  My daughter, however, is a whiz and I find myself upgrading my skill sets to make sure I stay ahead of her enough to qualify as a mentor(!).  As part of her math team efforts, she needed to find the number of whole squares from 0..1000 inclusive.  This is a pretty simple problem  which can be done on a blackboard, calculator or in Powershell:

[math]::sqrt(1000)
31.6227766016838

There are 31 whole squares in the number range 0..1000.  You can find 999 of the squares from 0..1000 and filter out all the whole squares from that group like as in the truncated result below:

rv -ea 0 a
for ($i=1; $i -lt 1000; $i++) {[array]$a+=$([math]::sqrt($i))}
0..($a.count - 1) | % {$a.item($PSItem)} | ? {([float]$PSItem - [int]$PSItem) -eq 0}
1
2
3
...
29
30
31

With not too much more effort you can create a hashtable of all these squares and their square roots:

foreach ($i in $a) {$c+=[ordered]@{$i=$([math]::pow([double]$i,2));}}


Name                           Value
----                           -----
1                              1
1.4142135623731                2
1.73205080756888               3
2                              4
2.23606797749979               5
2.44948974278318               6
2.64575131106459               7
2.82842712474619               8
3                              9
3.16227766016838               10
...

Name, Version, CreationTime file info

This snippet is simple but I thought it was worth the post:

PS C:\tools\SysinternalsSuite> $name=(ls *.exe)
PS C:\tools\SysinternalsSuite> $FileInfo=foreach ($i in $name) {new-object psobject -property @{name=$i.name; Version=($i).VersionInfo.ProductVersion; CreationTime=$i.C
reationTime} | Select Name,Version,Creationtime}

PS C:\tools\SysinternalsSuite> $FileInfo | ft -auto

name                Version     CreationTime
----                -------     ------------
accesschk.exe       5.11        8/1/2012 1:27:52 PM
AccessEnum.exe      1.32        11/1/2006 1:06:36 PM
ADExplorer.exe      1.44        11/14/2012 10:22:40 AM
ADInsight.exe       1.01        11/20/2007 12:25:34 PM
adrestore.exe                   11/1/2006 1:05:44 PM
Autologon.exe       3.01        2/22/2011 2:18:54 PM
autoruns.exe        11.60       9/10/2012 9:16:28 AM
autorunsc.exe       11.60       9/10/2012 9:16:28 AM
Bginfo.exe          4, 16, 0, 0 9/30/2009 1:31:54 AM
Cacheset.exe                    11/1/2006 1:06:08 PM
Clockres.exe        2.0         6/3/2009 10:36:40 PM
...

Random notes on [System.Collections...]

These are some very random notes on [System.Collections...]  7:07 PM 9/16/2013
Most of us know you can create a hashtable from syntax like this:

$ps=foreach ($i in $(ps)) {@{$i.id=$i.name}}
$ps | gm -s

   TypeName: System.Collections.Hashtable

But did you know you can use the same syntax to create a SortedList?

$ps=foreach ($i in $(ps)){[System.Collections.SortedList]@{$i.id=$i.name}}

$ps | gm -s

   TypeName: System.Collections.SortedList

'SortedList' is one of some number of data collections found in System.Collections. It creates a sorted list of values based on a unique key. Once loaded 'SortedList' has a faster retrieval speed than other members of System.Collections. Here's an example:

rv -ea 0 SortedNames; rv -ea 0 SortedSDDL
measure-command {
$SortedNames = New-Object System.Collections.SortedList
$SortedSDDL = New-Object System.Collections.SortedList
$index=0
foreach ($i in $(ls)) { $indx = $index++; $SortedNames.Add($indx,$i.name); $SortedSDDL.Add($indx,$i.getaccesscontrol().SDDL)  }
}

PS C:\ps1> rv -ea 0 SortedNames; rv -ea 0 SortedSDDL
PS C:\ps1> measure-command {
>> $SortedNames = New-Object System.Collections.SortedList
>> $SortedSDDL = New-Object System.Collections.SortedList
>> $index=0
>> foreach ($i in $(ls)) { $indx = $index++; $SortedNames.Add($indx,$i.name); $SortedSDDL.Add($indx,$i.getaccesscontrol().SDDL)  }
>> }

PS C:\ps1> $SortedNames.count
935

Days              : 0
Hours             : 0
Minutes           : 0
Seconds           : 0
Milliseconds      : 820
Ticks             : 8200868
TotalDays         : 9.49174537037037E-06
TotalHours        : 0.000227801888888889
TotalMinutes      : 0.0136681133333333
TotalSeconds      : 0.8200868
TotalMilliseconds : 820.0868

The following trick I picked up from Powershell.com : http://powershell.com/cs/blogs/tips/archive/2013/09/11/adding-new-type-accelerators-in-powershell.aspx . You can enumerate "Type Accelerators" in Powershell 3.0 with:

[PSObject].Assembly.GetType("System.Management.Automation.TypeAccelerators")::Get |Sort-Object -Property Value 

The property 'ImplementedInterfaces' allows you to view the interfaces for various Collections. Some Collections have more interfaces than others:

$PSAGet=[PSObject].Assembly.GetType("System.Management.Automation.TypeAccelerators")::Get
$PSAhashtable = ($PSAGet).hashtable

($PSAhashtable).ImplementedInterfaces

IsPublic IsSerial Name                                     BaseType
-------- -------- ----                                     --------
True     False    IDictionary
True     False    ICollection
True     False    IEnumerable
True     False    ISerializable
True     False    IDeserializationCallback
True     False    ICloneable

$PSAArray = ($PSAGet).array
PS C:\> ($PSAArray).ImplementedInterfaces

IsPublic IsSerial Name                                     BaseType
-------- -------- ----                                     --------
True     False    ICloneable
True     False    IList
True     False    ICollection
True     False    IEnumerable
True     False    IStructuralComparable
True     False    IStructuralEquatable

Unlike [hashtable] and [array], SortedList, ArrayList, IDictionary, IList will need to be added to Powershell Type Accelerator list in order to be used as [SortedList], [ArrayList], etc.

SortedList Collection and IP Address generation

I spent this morning working with the SortedList Collection and IP Address generation. SortedList maintains an IDictionary interface to a Key/Value pair collection (see  Krivayakov ) The advantage is a simple and direct reference to the last octet for Class C subnet generation and reference: 


rv -ea 0 SN
$SN = new-object System.Collections.SortedList
foreach ($i in (0..254)) {$SN.add($i,[IPAddress]"192.168.0.$i")}

foreach ($i in (0..254)) {$SN.add($i,[IPAddress]"192.168.0.$i")}
PS C:\> $SN

Name Value
---- -----
0 192.168.0.0
1 192.168.0.1
2 192.168.0.2
3 192.168.0.3
...

PS C:\> ($SN[0])

Address : 43200
AddressFamily : InterNetwork
ScopeId :
IsIPv6Multicast : False
IsIPv6LinkLocal : False
IsIPv6SiteLocal : False
IsIPv6Teredo : False
IsIPv4MappedToIPv6 : False
IPAddressToString : 192.168.0.0


This makes collecting arbitrary IP ranges a simple reference to their Name/Key:

PS C:\Powershell> $b = ($SN[0,8,23]).IPAddressToString + ($SN[23..27]).IPAddressToString
PS C:\Powershell> $b
192.168.0.0
192.168.0.8
192.168.0.23
192.168.0.23
192.168.0.24
192.168.0.25
192.168.0.26
192.168.0.27

A little more complicated for multiple subnets:

rv -EA 0 SN0;rv -EA 0 SN1;rv -EA 0 SN2;
$SN0 = new-object System.Collections.SortedList
$SN1 = new-object System.Collections.SortedList
$SN2 = new-object System.Collections.SortedList
for ($i = 0; $i -ile 254;$i++){$SN0.add($i,[IPAddress]"192.168.0.$i")}
for ($i = 0; $i -ile 254;$i++){$SN1.add($i,[IPAddress]"192.168.1.$i")}
for ($i = 0; $i -ile 254;$i++){$SN2.add($i,[IPAddress]"192.168.2.$i")}

$c = ($SN0[0,8,23]).IPAddressToString + ($SN1[23..27]).IPAddressToString + ($SN2[148..154]).IPAddressToString

PS C:\> $c
192.168.0.0
192.168.0.8
192.168.0.23
192.168.1.23
192.168.1.24
192.168.1.25
192.168.1.26
192.168.1.27
192.168.2.148
192.168.2.149
192.168.2.150
192.168.2.151
192.168.2.152
192.168.2.153
192.168.2.154

Processing Snort Logs with Powershell 3.0

I had a group of snort logs (total about 144 MB) I wanted to process in batch. To do this I embedded a a new PSOBJECT inside a foreach loop. 'Select-string' ('sls') isn't the speediest search, but fast enough.

Function Process-SnortLogs {
$SelectPorts = foreach ($i in ($(ls snort.log.*)))
{
$a=.\snort -qr $i.Name;
New-Object PSObject -Property  @{
LogName = $i.Name;
LastWrite = $i.LastWriteTime;
Total = $($a.count);
P445 = (($a | sls -allmatches ':445').matches).count;
P443 = (($a | sls -allmatches ':443').matches).count;
P80 =   (($a | sls -allmatches ':80').matches).count;
P53 =   (($a | sls -allmatches ':53').matches).count;
}
}
}

Process-SnortLogs
$SelectPorts | Select LogName,LastWrite,P53,P80,P443,P445,Total | ft -auto

LogName              LastWrite               P53    P80  P443 P445  Total
-------              ---------               ---    ---  ---- ----  -----
snort.log.1304098553 4/29/2011 10:36:45 AM     0      0     2    0     10
snort.log.1304098783 4/29/2011 10:39:45 AM     0      0     3    0     15
snort.log.1304098850 4/29/2011 10:43:57 AM     0      4    17    0    105
snort.log.1339265058 6/9/2012 11:44:35 AM   2706  10429  7052  110 114395
snort.log.1339278740 6/9/2012 3:10:55 PM    1415    898  1232    7  26019
snort.log.1349466038 10/5/2012 1:57:33 PM   2489   6365  9465   70 149126
snort.log.1349554671 10/6/2012 1:18:00 PM     25      0     0    0    325
snort.log.1349554686 10/6/2012 3:52:18 PM   9820  39956 27617  175 482248
snort.log.1370643770 6/7/2013 5:20:12 PM    2138      8  6482    0  43290
snort.log.1373764041 7/13/2013 7:08:13 PM  30095 101367 26708  239 683162
snort.log.1373770657 7/13/2013 8:01:32 PM     21      0     5    0    545

Here I get the percentage of ports 53,80,443,445 for each snort log:

$SelectPorts | Select LogName, LastWrite, Total,`
 @{Label='%P53';Expression={[LONG]([float]($_.P53 /$_.Total) * 100)}},`
 @{Label='%P80';Expression={[LONG]([float]($_.P80 /$_.Total) * 100)}},`
 @{Label='%P443';Expression={[LONG]([float]($_.P443 /$_.Total) * 100)}},`
 @{Label='%P445';Expression={[LONG]([float]($_.P445 /$_.Total) * 100)}} |  sort -desc Total |  ft * -auto

LogName              LastWrite              Total %P53 %P80 %P443 %P445
-------              ---------              ----- ---- ---- ----- -----
snort.log.1373764041 7/13/2013 7:08:13 PM  683162    4   15     4     0
snort.log.1349554686 10/6/2012 3:52:18 PM  482248    2    8     6     0
snort.log.1349466038 10/5/2012 1:57:33 PM  149126    2    4     6     0
snort.log.1339265058 6/9/2012 11:44:35 AM  114395    2    9     6     0
snort.log.1370643770 6/7/2013 5:20:12 PM    43290    5    0    15     0
snort.log.1339278740 6/9/2012 3:10:55 PM    26019    5    3     5     0
snort.log.1373770657 7/13/2013 8:01:32 PM     545    4    0     1     0
snort.log.1349554671 10/6/2012 1:18:00 PM     325    8    0     0     0
snort.log.1304098850 4/29/2011 10:43:57 AM    105    0    4    16     0
snort.log.1304098783 4/29/2011 10:39:45 AM     15    0    0    20     0
snort.log.1304098553 4/29/2011 10:36:45 AM     10    0    0    20     0

Current Process Memory

updated  08/04/2013 -RMF

Below is an example of the (clumsy) scripting workaround I will use because I always have a difficult time getting 'add-member' to work with membertype ScriptMethod. What I want here is WS, PM, VM for each current process, but also the differences (e.g. VM - PM, PM - WS):

Function CurrentMem {
[PSObject]$Memory=
ps * | Select Name, ID,`
@{Name='WS_MB';Expression={[INT](($_.WorkingSet)/1MB)}},`
@{Name='PriMemMB';Expression={[INT](($_.PrivateMemorySize64)/1MB)}}, `
@{Name='VirMemMB';Expression={[INT](($_.VirtualMemorySize64)/1MB)}}
$Memory | Sort -desc VirMemMB | 
Select *,  @{Name='VM-PM';Expression={[INT]($_.VirMemMB - $_.PriMemMB)}}, `
@{Name='PM-WS';Expression={[INT]($_.PriMemMB - $_.WS_MB)}}
}

So then  I have:

PS C:\> $CurrentMem=CurrentMem
PS C:\> ($CurrentMem | sort -desc VM-PM)[0..10]  | ft -auto

Name         Id WS_MB PriMemMB VirMemMB VM-PM PM-WS
----         -- ----- -------- -------- ----- -----
powershell 7300    74       66      608   542    -8
powershell 5408     3       63      603   540    60
svchost     536    39       39      575   536     0
chrome     4748   192      194      658   464     2
bash       6348     0        8      451   443     8
powershell 1536   176      185      620   435     9
syslogd    2112     1        5      432   427     4
cygrunsrv  1948     0        6      427   421     6
chrome     2956   265      323      742   419    58
taskhost   3896     6       15      394   379     9
svchost    1288    23       26      386   360     3

A more advanced version allows for a more granular look at processes:


Function Get-CM {
[PSObject]$Memory=
ps * | Select Name, ID, Company, HandleCount,`
@{Name='WorkSetMB';Expression={[LONG](($_.WorkingSet)/1MB)}},`
@{Name='PMS64Bytes';Expression={[LONG](($_.PagedMemorySize64))}},`
@{Name='NPMS64Bytes';Expression={[LONG](($_.NonpagedSystemMemorySize64))}},`
@{Name='PriMem64MB';Expression={[LONG](($_.PrivateMemorySize64)/1MB)}},`
@{Name='VirMem64MB';Expression={[LONG](($_.VirtualMemorySize64)/1MB)}}
$Memory | Sort -desc VirMem64MB | 
Select *,  @{Name='VMem-PMem.MB';Expression={[LONG]($_.VirMem64MB - $_.PriMem64MB)}},`
@{Name='PMem-NPMem.MB';Expression={[LONG](($_.PMS64Bytes - $_.NPMS64Bytes)/1MB)}},`
@{Name='PriMem-WorkSet.MB';Expression={[LONG]($_.PriMem64MB - $_.WorkSetMB)}}
}

So that here I can look at all process with over 500 handles sorted by a decreasing amount of PagedMemorySystem64 :

PS C:\ps1> get-cm | where HandleCount -gt 500 | sort -desc PMS64Bytes | fl *


Name              : chrome
Id                : 6636
Company           : The Chromium Authors
HandleCount       : 2769
WorkSetMB         : 180
PMS64Bytes        : 268677120
NPMS64Bytes       : 141712
PriMem64MB        : 256
VirMem64MB        : 672
VMem-PMem.MB      : 416
PMem-NPMem.MB     : 256
PriMem-WorkSet.MB : 76

Name              : chrome
Id                : 3532
Company           : The Chromium Authors
HandleCount       : 796
WorkSetMB         : 63
PMS64Bytes        : 220286976
NPMS64Bytes       : 69632
PriMem64MB        : 210
VirMem64MB        : 699
VMem-PMem.MB      : 489
PMem-NPMem.MB     : 210
PriMem-WorkSet.MB : 147

Name              : svchost
Id                : 456
Company           : Microsoft Corporation
HandleCount       : 727
WorkSetMB         : 137
PMS64Bytes        : 162627584
NPMS64Bytes       : 31784
PriMem64MB        : 155
VirMem64MB        : 315
VMem-PMem.MB      : 160
PMem-NPMem.MB     : 155
PriMem-WorkSet.MB : 18

...

or (click to enlarge)

Powershell 3.0 CIM snippet : Properties of IPEnabled Adapters

$count = ((Get-CimInstance -class Win32_NetworkAdapterConfiguration).count - 1)

$Adapters = 0..$count | % {(Get-CimInstance -class Win32_NetworkAdapterConfiguration)[$PSItem]}
$Adapters
$IPEnabled=($Adapters | ? {$_.IPEnabled -eq "True"})
$IPEnabled | fl *

Functions for sum of directory or file size

# file
function file.sum($x) {
rv -ea 0 total;
foreach ($i in ((ls $x -Force) | Select Length)) {[int64]$total+=$i[0].Length};
$total /1MB;
}

#dir
function dir.sum {
rv -ea 0 total;
foreach ($i in ((ls -Force) | Select Length)) {[int64]$total+=$i[0].Length};
$total /1GB;
}

#dir recurse
# really chews through memory for directories of any size
function dir.sum.recurse {
rv -ea 0 total;
foreach ($i in ((ls -Force -Recurse) | Select Length)) {[int64]$total+=$i[0].Length};
$total /1GB;
}

Can you see the pattern....?

Function global:gen-num_array_hash
{
[array[]]$num_array=1..10000
$global:num_array_hash=$num_array | %  {
[array[]]$rand_out=$(get-random); foreach ($i in $rand_out) {
[array]@{$_.getvalue(0)=$i.item(0)}}};break
}
rv -ea 0 hashdata
$hashdata=$num_array_hash
Chart-Hashdata fastpoint