PulledPork 0.6.0 the Smoking Pig, He's on Fire!

It has been some time since I posted anything at all, I had considered adding "relevant".  But that's simply not true, since it's been dead air for a while.

Having said this, I am pleased to announce PulledPork V 0.6.0 - the Smoking Pig is finally released as of, well, right now!

This version represents a decent amount of time spent improving the core of the tool to enhance speed, a large number of feature enhancements and also not an insignificant number of bugfixes!  A few quick notes before I copy and paste the changelog notes; If you are changing rulestate by doing anything in the drop|enable|disable config files with the category, you will now need to prepend the category that you want to modify with ET- or VRT- (based on where the rules came from).  Another item of note is that multiple rulesets are now fully supported, thus no need to run two or more instances of PulledPork.  Lastly but certainly not least is the capability to ignore source files on a more granular level: (plaintext, preproc, shared object or global).

One more big feature enhancement that I would like to point out, is the capability to create a backup/archive of your existing rules files / config files / whatever else you want!  kthx, moving on...

Please be sure to read through the documentation THOROUGHLY, a couple of the above noted changes could affect your implementation and I don't want you to be terribly shocked by that.  Plus, the things that you will need to update are trivial!

The new PulledPork can be downloaded at the following location:
http://pulledpork.googlecode.com/files/pulledpork-0.6.0.tar.gz
SHA1 Checksum: c4fdf58c716017a0ebad3c46f770fda54c8f23b2
MD5 Checksum: d65c4ef29956823a1a5a05921f219a29
Without further rambling on my part, the changelog notes:

v0.6.0 the Smoking Pig

New Features / changes:

• Added -q command line switch to squelch everything except fatal errors
• Code clean up for readability
• Move debug output to allow for better debugging of actual variable values
• Update config to allow for ssl from ET
• Update config to allow for new snort rules gzip
• Bug #55 - Create capability to ignore more granularly (plaintext, preproc, shared object or global).
• Bug #50 - You can now create backups and archives of your existing config and rules files etc...
• This adds the PM requirement of File::Find
• Bug #56 - More verbose output when a flowbit is re-enabled (only when run with -v)
• Bug #60 - added -E flag that will cause ONLY enabled rules to be written to output files
• Bug #47 - added -R flag that will set the state of the rules specified in enablesid.conf back to their ORIGINAL state, as read from the source rules tarball.
• Bug #63 - added sid MSG information to changelog output.
• Added -k and -K options to allow for the writing of the original source file rather than one large output file.
• Bug #66 - Prepend VRT rulesets with VRT- and ET rulesets with ET- to allow for paralell ruleset operations.  This also provides more granularity in that scenario wherein the user could set state in a VRT or ET category only by specifying VRT-category or ET-category in the sid state modification files.
• Added support for 500 errors, specifying that users should update their root cert store!

Bug Fixes:

• Bug #39 - updated to allow for use of username:pass@proxy.url
• Bug #49 - fix for race condition not allowing HUP to work with -nTH switches specified
• Bug #40 - allow so_rules to be handled when non VRT rulesets are downloaded
• Bug #45 - create a blank so_stub rules file so that we don't get an error re: a blank file from snort when generating so_stubs! (only if the file does not already exist, and only if you are using SOs!)
• Bug #46 - throw error if a config file that is specified does not exist   
• Bug #42 - Added OpenSUSE-11-3 to list
• Fixed race condition that did not properly handle certain spaces in flowbits set and isset values, resulting in unchecked flowbits etc...
• Bug #51 - Increased timeout value to 60 seconds
• Bug #53 - Fixed pcre issue that caused certain rules containing isset and set flobwits values to incorrectly be auto-enabled.
• Bug #61 - Fixed so that .so rules are not touched!
• Bug #67 - Fixed regex to allow for space between ( and msg.
• Bug #71 - Flaw in if statement logic did not allow for proper multiline rule parsing
• Undocumented ID - Flaw in changelog routine did not allow for proper writing of sid-msg or sid in "deleted rules" section of the changelog.
• Bug #62 - Added check for amd64 string during arch detection!

Special Notes:

• Bug #47 - This should be used by advanced users only, it can produce results that may not make sense to the typical user.  And frankly, I don't understand it ;-)
• Bug #60 - This fix WILL cause inconsistency in your changelog, as when PP reads the old rules from the existing rules file, it will have only the enabled rules in it.. thus any rules that were not enabled in that file will show up as NEW rules in the changelog output, you have been warned, so no whining!

 That should just about cover it for now, as always, I want to also thank the community for their support and feedback!  If you have any questions, comments, concerns, or otherwise then please feel free to hit me up in #snort or #pulledpork on freenode.  You are also always welcome and encouraged to join the mailing list that can be found at http://groups.google.com/group/pulledpork-users/.  And of course you can also submit feedback / bugs / feature requests at http://pulledpork.googlecode.com.

Regards,
JJC

Haz The Drowning Rat? - PulledPork 0.5.0 is now floating!

This release of PulledPork (The Drowning Rat) represents quite a bit of development to include a number of community requested capabilities, change a few around, and repair some bugs!  Again, I would like to thank the community for their support, contribution and use of the PulledPork Snort rule management system.  The next section is an excerpt from the README.CHANGES and below there I may discuss some example use-cases and include some sample output.

PulledPork Changelog

v0.5.0

New Features / changes:
- Automatic VRT tarball name determination (based on local Snort Version)
- Full support for ET Pro rulesets
- Full support for new ET Download scheme
- Issue #27 Modifysid capability
- Capability to retrieve multiple rulesets in a single run
- Issue #24 Added verbose output showing all requests, results and urls
- Verbose output now shows percentage bar for downloads
- Extra Verbose output now shows additional HTTP debug!
- Set value in default.conf file to https for VRT downloads
- Set UA Value to (PulledPork/X.X.X)
- Capability to log critical information to syslog
- Grabonly option, for those that only want to download the tarball(s)
- Issue #34 Added the capability to specify the order of disable / enable / drop
    using the state_order configuration option in the master config file
- Added a contrib directory
- Added oink-conv.pl to contrib directory
    * converts oinkmaster config files to PP config files
    * Thx Russell Fulton!
- Added README.CONTRIB to track contrib files (ohai manifest)
- Perl Modue Requirement Changes (SEE SECTION BELOW)
- Issue #38 Added capability to extract reference docs from tarball and
    store in a defined path, NOTE this dramatically increases PP runtime
    * runtime value is -r

Bug Fixes:
- Should now correctly use environmentally set proxy settings
    * Shout to pkthound for his work and contribution here!
- Fixed case where rules with multiple flowbit (un)?set values would not
    properly populate all of the flowbit values into the rules hash
- Bug #29 - fixed to allow for proper sid-msg.map generation
- Bug #28 - fixed numerous spellification issues
- Bug #32 - fixed to allow for so stub generation in nodownload and !nodownload case


Perl Module Requriement Changes:
- LWP::Simple no longer
- LWP::UserAgent now required
- HTTP::Request now required
- HTTP::Status now required
- SYS::Syslog now required
- Crypt::SSLeay now required
- Carp now required

As you can see, and as I had indicated, there are a number of significant improvements and fixes.  It is important to note that there are a number of changes, that include new and changed options, to the master config and the addition of the modifysid.conf file that allows you to modify rules based on regular expression matches etc...

Of course we also now fully support (ET Pro and the new ET open) rules and the capability to download multiple rulesets in a single run, rather than multiple config files referencing other .rules files as local rules etc...

One other seemingly insignificant change is the capability to change the order that the rules modification routines run, this means that you can more granularly control rule state.  The default processing order is (enable, drop, disable), this can now be changed though to allow for the disabling of all rules in a specific category (or however you would do it) and then selectively enabling rules out of that category, by simply changing the run order to disable,drop,enable.  Of course combining this with the pcre, category, modifysid etc.. capabilities gives you quite a bit of versatility.

So, without further adeau, I give you:

    http://code.google.com/p/pulledpork/
      _____ ____
     `----,\    )
      `--==\\  /    PulledPork v0.5.0 The Drowning Rat
       `--==\\/
     .-~~~~-.Y|\\_  Copyright (C) 2009-2010 JJ Cummings
  @_/        /  66\_  cummingsj@gmail.com
    |    \   \   _(")
     \   /-| ||'--'  Rules give me wings!
      \_\  \_\\
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Checking latest MD5 for snortrules-snapshot-2861.tar.gz....
    They Match
    Done!
Prepping rules from snortrules-snapshot-2861.tar.gz for work....
    Done!
Checking latest MD5 for etpro.rules.tar.gz....
    They Match
    Done!
Prepping rules from etpro.rules.tar.gz for work....
    Done!
Checking latest MD5 for emerging.rules.tar.gz....
    They Match
    Done!
Prepping rules from emerging.rules.tar.gz for work....
    Done!
Reading rules...
Reading rules...
Activating security rulesets....
    Done
Setting Flowbit State....
    Enabled 264 flowbits
    Enabled 29 flowbits
    Enabled 4 flowbits
    Enabled 2 flowbits
    Done
Writing /home/jj/snort.rules....
    Done
Generating sid-msg.map....
    Done
Writing /home/jj/sid-msg.map....
    Done
Writing /home/jj/sid_changes.log....
    Done
Rule Stats....
    New:-------0
    Deleted:---0
    Enabled Rules:----4506
    Dropped Rules:----0
    Disabled Rules:---17797
    Total Rules:------22303
    Done
Please review /var/log/sid_changes.log for additional details
Fly Piggy Fly!

Bah, Paste chopped my flying pig up ;-)

Get it here:

pulledpork-0.5.0.tar.gz latest hashes:
MD5SUM = 60c0abe78945876c643760b3bb2afdb6
SHA256 = 9e69873d737e4fc8dfd9b3a98316e4ff41bd8c4accda72f18036b96568c48872

Cheers,

JJC 

The Pig Doktah is Born! - A Snort Performance Monitor Interpretation Tool

After receiving a variety of feedback via email, irc, twitter and even some comments on this blog, I have decided to start an updated and maintained project to parse, display, and ultimately make configuration / tuning suggestions based on the snort performance monitor output.  As you may have guessed by now, this tool is called the Pig Doktah and can be found at http://thepigdoktah.googlecode.com.  The current version reads snort performance monitor output and stores in a hash for quick access, sorting etc...

I am still quite interested in what specific metrics people are most interested, so please don't hesitate to hit me up via comment on this blog, irc (freenode - #snort), email, or twitter.

During the development, and for the foreseeable future, I will have the latest version of the code sending output to this location on an hourly basis: http://rootedyour.com/enhanced/pminfo.htm

Sample output:

-= Tha Pig Doktah 0.1 Dev =-
Copyright (C) 2010 JJ Cummings

Report Info:
    Processed: /var/tmp/snortstat
    First Entry: Wed Sep  1 11:34:05 2010
    Last Entry: Wed Sep  8 09:00:17 2010
    Time Span: 6 days, 21 hours, 26 minutes and 12 seconds

Wirespeed:
    High: 10.613 Mbits/Sec | Sat Sep  4 07:59:48 2010
    Low: 0.006 Mbits/Sec | Sat Sep  4 07:12:47 2010
    Avg: 1.953 Mbits/Sec
  
% Packet Loss:
    High: 10.504% | Sat Sep  4 03:00:00 2010
    Low: 0.000% | Wed Sep  8 08:41:27 2010
    Avg: 1.002%

Additional Info:
    Avg Pkt Size: 803.413 bytes
    Avg Syns/Sec: 0.181
    Avg SynAcks/Sec: 0.124
    Avg Alerts/Sec: 0.001
    Avg Current Cached Sessions: 6671.668

Raw Values:
     alerts avg = 0.001
     alerts high = 0.032
     alerts high_date = Wed Sep  1 12:32:57 2010
     alerts low = 0.000
     alerts low_date = Wed Sep  8 09:00:17 2010
     attrib_hosts_current avg = 0.000
     attrib_hosts_current high = 0.000
     attrib_hosts_current high_date = Wed Sep  8 09:00:17 2010
     attrib_hosts_current low = 0.000
     attrib_hosts_current low_date = Wed Sep  8 09:00:17 2010
     attrib_reloads avg = 0.000
     attrib_reloads high = 0
     attrib_reloads high_date = Wed Sep  8 09:00:17 2010
     attrib_reloads low = 0
     attrib_reloads low_date = Wed Sep  8 09:00:17 2010
     bytes_applayer avg = 0.252
     bytes_applayer high = 1.352
     bytes_applayer high_date = Sat Sep  4 07:59:48 2010
     bytes_applayer low = 0.006
     bytes_applayer low_date = Tue Sep  7 09:13:56 2010
     bytes_ipfrag avg = 0.000
     bytes_ipfrag high = 0
     bytes_ipfrag high_date = Wed Sep  8 09:00:17 2010
     bytes_ipfrag low = 0
     bytes_ipfrag low_date = Wed Sep  8 09:00:17 2010
     bytes_ipreass avg = 2279.291
     bytes_ipreass high = 3660
     bytes_ipreass high_date = Thu Sep  2 13:47:36 2010
     bytes_ipreass low = 368
     bytes_ipreass low_date = Thu Sep  2 10:22:15 2010
     bytes_tcprebuilt avg = 892.669
     bytes_tcprebuilt high = 1458
     bytes_tcprebuilt high_date = Sun Sep  5 15:19:06 2010
     bytes_tcprebuilt low = 136
     bytes_tcprebuilt low_date = Sat Sep  4 00:58:27 2010
     cpu1_idle avg = 95.767
     cpu1_idle high = 99.977
     cpu1_idle high_date = Sat Sep  4 00:58:27 2010
     cpu1_idle low = 69.943
     cpu1_idle low_date = Tue Sep  7 06:20:11 2010
     cpu1_sys avg = 0.051
     cpu1_sys high = 0.287
     cpu1_sys high_date = Sat Sep  4 07:59:48 2010
     cpu1_sys low = 0.000
     cpu1_sys low_date = Wed Sep  8 08:07:19 2010
     cpu1_user avg = 4.183
     cpu1_user high = 29.860
     cpu1_user high_date = Tue Sep  7 06:20:11 2010
     cpu1_user low = 0.023
     cpu1_user low_date = Sat Sep  4 00:58:27 2010
     cpu_count avg = 1.000
     cpu_count high = 1
     cpu_count high_date = Wed Sep  8 09:00:17 2010
     cpu_count low = 1
     cpu_count low_date = Wed Sep  8 09:00:17 2010
     drops avg = 1.002
     drops high = 10.504
     drops high_date = Sat Sep  4 03:00:00 2010
     drops low = 0.000
     drops low_date = Wed Sep  8 08:41:27 2010
     filtered_tcp avg = 3790.598
     filtered_tcp high = 45608
     filtered_tcp high_date = Tue Sep  7 09:24:12 2010
     filtered_tcp low = 85
     filtered_tcp low_date = Wed Sep  1 11:50:25 2010
     filtered_udp avg = 3790.598
     filtered_udp high = 45608
     filtered_udp high_date = Tue Sep  7 09:24:12 2010
     filtered_udp low = 85
     filtered_udp low_date = Wed Sep  1 11:50:25 2010
     frag_auto avg = 0.000
     frag_auto high = 0.000
     frag_auto high_date = Wed Sep  8 09:00:17 2010
     frag_auto low = 0.000
     frag_auto low_date = Wed Sep  8 09:00:17 2010
     frag_complete avg = 0.000
     frag_complete high = 0.000
     frag_complete high_date = Wed Sep  8 09:00:17 2010
     frag_complete low = 0.000
     frag_complete low_date = Wed Sep  8 09:00:17 2010
     frag_current avg = 0.000
     frag_current high = 0
     frag_current high_date = Wed Sep  8 09:00:17 2010
     frag_current low = 0
     frag_current low_date = Wed Sep  8 09:00:17 2010
     frag_delete avg = 0.000
     frag_delete high = 0.000
     frag_delete high_date = Wed Sep  8 09:00:17 2010
     frag_delete low = 0.000
     frag_delete low_date = Wed Sep  8 09:00:17 2010
     frag_faults avg = 0.000
     frag_faults high = 0
     frag_faults high_date = Wed Sep  8 09:00:17 2010
     frag_faults low = 0
     frag_faults low_date = Wed Sep  8 09:00:17 2010
     frag_flushes avg = 0.000
     frag_flushes high = 0.000
     frag_flushes high_date = Wed Sep  8 09:00:17 2010
     frag_flushes low = 0.000
     frag_flushes low_date = Wed Sep  8 09:00:17 2010
     frag_insert avg = 0.000
     frag_insert high = 0.000
     frag_insert high_date = Wed Sep  8 09:00:17 2010
     frag_insert low = 0.000
     frag_insert low_date = Wed Sep  8 09:00:17 2010
     frag_max avg = 0.000
     frag_max high = 0
     frag_max high_date = Wed Sep  8 09:00:17 2010
     frag_max low = 0
     frag_max low_date = Wed Sep  8 09:00:17 2010
     frag_new avg = 0.000
     frag_new high = 0.000
     frag_new high_date = Wed Sep  8 09:00:17 2010
     frag_new low = 0.000
     frag_new low_date = Wed Sep  8 09:00:17 2010
     frag_timeout avg = 0.000
     frag_timeout high = 0
     frag_timeout high_date = Wed Sep  8 09:00:17 2010
     frag_timeout low = 0
     frag_timeout low_date = Wed Sep  8 09:00:17 2010
     kpkts_applayer avg = 121425.178
     kpkts_applayer high = 444882
     kpkts_applayer high_date = Thu Sep  2 22:42:20 2010
     kpkts_applayer low = 5738
     kpkts_applayer low_date = Wed Sep  1 18:55:09 2010
     kpkts_ipfrag avg = 0.000
     kpkts_ipfrag high = 0.000
     kpkts_ipfrag high_date = Wed Sep  8 09:00:17 2010
     kpkts_ipfrag low = 0.000
     kpkts_ipfrag low_date = Wed Sep  8 09:00:17 2010
     kpkts_ipreass avg = 0.022
     kpkts_ipreass high = 0.366
     kpkts_ipreass high_date = Tue Sep  7 06:20:11 2010
     kpkts_ipreass low = 0.000
     kpkts_ipreass low_date = Wed Sep  8 08:31:29 2010
     kpkts_iptcprebuilt avg = 0.273
     kpkts_iptcprebuilt high = 1.646
     kpkts_iptcprebuilt high_date = Thu Sep  2 22:42:20 2010
     kpkts_iptcprebuilt low = 0.006
     kpkts_iptcprebuilt low_date = Tue Sep  7 09:13:56 2010
     kpkts_wire avg = 0.252
     kpkts_wire high = 1.352
     kpkts_wire high_date = Sat Sep  4 07:59:48 2010
     kpkts_wire low = 0.006
     kpkts_wire low_date = Tue Sep  7 09:13:56 2010
     mbits_applayer avg = 803.413
     mbits_applayer high = 1009
     mbits_applayer high_date = Sat Sep  4 08:09:48 2010
     mbits_applayer low = 120
     mbits_applayer low_date = Mon Sep  6 05:52:07 2010
     mbits_ipfrag avg = 2.434
     mbits_ipfrag high = 17.685
     mbits_ipfrag high_date = Tue Sep  7 06:20:11 2010
     mbits_ipfrag low = 0.007
     mbits_ipfrag low_date = Mon Sep  6 17:12:03 2010
     mbits_ipreass avg = 0.000
     mbits_ipreass high = 0.000
     mbits_ipreass high_date = Wed Sep  8 09:00:17 2010
     mbits_ipreass low = 0.000
     mbits_ipreass low_date = Wed Sep  8 09:00:17 2010
     mbits_tcprebuilt avg = 0.482
     mbits_tcprebuilt high = 8.324
     mbits_tcprebuilt high_date = Tue Sep  7 06:20:11 2010
     mbits_tcprebuilt low = 0.000
     mbits_tcprebuilt low_date = Tue Sep  7 01:11:34 2010
     mbps_snort avg = 0.000
     mbps_snort high = 0
     mbps_snort high_date = Wed Sep  8 09:00:17 2010
     mbps_snort low = 0
     mbps_snort low_date = Wed Sep  8 09:00:17 2010
     mbps_wire avg = 1.953
     mbps_wire high = 10.613
     mbps_wire high_date = Sat Sep  4 07:59:48 2010
     mbps_wire low = 0.006
     mbps_wire low_date = Sat Sep  4 07:12:47 2010
     patmatch avg = 320.575
     patmatch high = 556.312
     patmatch high_date = Sun Sep  5 19:37:37 2010
     patmatch low = 2.946
     patmatch low_date = Wed Sep  8 07:11:52 2010
     pktbytes avg = 803.413
     pktbytes high = 1009
     pktbytes high_date = Sat Sep  4 08:09:48 2010
     pktbytes low = 120
     pktbytes low_date = Mon Sep  6 05:52:07 2010
     pkts_blocked avg = 0.229
     pkts_blocked high = 14.322
     pkts_blocked high_date = Sun Sep  5 20:50:12 2010
     pkts_blocked low = 0.109
     pkts_blocked low_date = Sat Sep  4 01:34:34 2010
     pkts_dropped avg = 0.000
     pkts_dropped high = 0
     pkts_dropped high_date = Wed Sep  8 09:00:17 2010
     pkts_dropped low = 0
     pkts_dropped low_date = Wed Sep  8 09:00:17 2010
     pkts_dropped_percentage avg = 0.172
     pkts_dropped_percentage high = 9.096
     pkts_dropped_percentage high_date = Sun Sep  5 20:50:12 2010
     pkts_dropped_percentage low = 0.003
     pkts_dropped_percentage low_date = Wed Sep  1 11:50:25 2010
     pkts_total avg = 2106.252
     pkts_total high = 38320
     pkts_total high_date = Thu Sep  2 22:42:20 2010
     pkts_total low = 0
     pkts_total low_date = Wed Sep  8 08:41:27 2010
     sessions_close avg = 0.000
     sessions_close high = 0.000
     sessions_close high_date = Wed Sep  8 09:00:17 2010
     sessions_close low = 0.000
     sessions_close low_date = Wed Sep  8 09:00:17 2010
     sessions_closed avg = 1024.846
     sessions_closed high = 2980
     sessions_closed high_date = Mon Sep  6 12:37:55 2010
     sessions_closed low = 2
     sessions_closed low_date = Wed Sep  1 11:34:05 2010
     sessions_cur avg = 6671.668
     sessions_cur high = 8173
     sessions_cur high_date = Sun Sep  5 21:10:31 2010
     sessions_cur low = 51
     sessions_cur low_date = Wed Sep  1 11:34:05 2010
     sessions_del avg = 0.177
     sessions_del high = 3.055
     sessions_del high_date = Mon Sep  6 05:52:07 2010
     sessions_del low = 0.000
     sessions_del low_date = Sun Sep  5 19:53:29 2010
     sessions_dropped avg = 0.001
     sessions_dropped high = 0.006
     sessions_dropped high_date = Wed Sep  1 11:50:25 2010
     sessions_dropped low = 0.000
     sessions_dropped low_date = Wed Sep  8 09:00:17 2010
     sessions_est avg = 0.376
     sessions_est high = 11.686
     sessions_est high_date = Sun Sep  5 20:50:12 2010
     sessions_est low = 0.003
     sessions_est low_date = Wed Sep  1 11:50:25 2010
     sessions_init avg = 0.001
     sessions_init high = 0.174
     sessions_init high_date = Tue Sep  7 18:18:34 2010
     sessions_init low = 0.000
     sessions_init low_date = Wed Sep  8 08:46:27 2010
     sessions_max avg = 0.000
     sessions_max high = 0.000
     sessions_max high_date = Wed Sep  8 09:00:17 2010
     sessions_max low = 0.000
     sessions_max low_date = Wed Sep  8 09:00:17 2010
     sessions_midstream avg = 6703.818
     sessions_midstream high = 8175
     sessions_midstream high_date = Sun Sep  5 21:03:29 2010
     sessions_midstream low = 51
     sessions_midstream low_date = Wed Sep  1 11:34:05 2010
     sessions_new avg = 0.165
     sessions_new high = 3.062
     sessions_new high_date = Mon Sep  6 05:52:07 2010
     sessions_new low = 0.016
     sessions_new low_date = Fri Sep  3 20:12:36 2010
     sessions_pruned avg = 579.871
     sessions_pruned high = 953
     sessions_pruned high_date = Sun Sep  5 08:30:47 2010
     sessions_pruned low = 3
     sessions_pruned low_date = Wed Sep  1 11:50:25 2010
     sessions_timedout avg = 5066.950
     sessions_timedout high = 7586
     sessions_timedout high_date = Sun Sep  5 21:22:42 2010
     sessions_timedout low = 31
     sessions_timedout low_date = Wed Sep  1 11:34:05 2010
     sessions_udp_cachedSsns_sec avg = 0.000
     sessions_udp_cachedSsns_sec high = 0
     sessions_udp_cachedSsns_sec high_date = Wed Sep  8 09:00:17 2010
     sessions_udp_cachedSsns_sec low = 0
     sessions_udp_cachedSsns_sec low_date = Wed Sep  8 09:00:17 2010
     sessions_udp_cached_current avg = 0.000
     sessions_udp_cached_current high = 0.000
     sessions_udp_cached_current high_date = Wed Sep  8 09:00:17 2010
     sessions_udp_cached_current low = 0.000
     sessions_udp_cached_current low_date = Wed Sep  8 09:00:17 2010
     sessions_udp_cached_max avg = 0.000
     sessions_udp_cached_max high = 0
     sessions_udp_cached_max high_date = Wed Sep  8 09:00:17 2010
     sessions_udp_cached_max low = 0
     sessions_udp_cached_max low_date = Wed Sep  8 09:00:17 2010
     sessions_udp_cached_sec avg = 0.000
     sessions_udp_cached_sec high = 0
     sessions_udp_cached_sec high_date = Wed Sep  8 09:00:17 2010
     sessions_udp_cached_sec low = 0
     sessions_udp_cached_sec low_date = Wed Sep  8 09:00:17 2010
     stream_fault avg = 13.182
     stream_fault high = 59
     stream_fault high_date = Wed Sep  8 05:04:52 2010
     stream_fault low = 0
     stream_fault low_date = Wed Sep  8 00:51:37 2010
     stream_flush avg = 21.526
     stream_flush high = 365.535
     stream_flush high_date = Tue Sep  7 06:20:11 2010
     stream_flush low = 0.013
     stream_flush low_date = Thu Sep  2 05:44:59 2010
     stream_timeout avg = 239.842
     stream_timeout high = 3578
     stream_timeout high_date = Sun Sep  5 20:50:12 2010
     stream_timeout low = 1
     stream_timeout low_date = Wed Sep  1 11:50:25 2010
     synacks avg = 0.124
     synacks high = 2.771
     synacks high_date = Mon Sep  6 12:42:56 2010
     synacks low = 0.006
     synacks low_date = Sat Sep  4 00:58:27 2010
     syns avg = 0.181
     syns high = 6.072
     syns high_date = Mon Sep  6 05:52:07 2010
     syns low = 0.019
     syns low_date = Fri Sep  3 20:12:36 2010

Snort Performance Stats Tool Info

I have noticed that there are not a ton of tools out there (definitely not many currently maintained) that can be used to parse the information from the snort performance monitor.  As such, I am considering writing one and wanted to see what the interest would be.  If there is indeed interest, then I would also like to know what format(s) and types of information would be most useful to the community.  Of course I know what will be useful to myself, and will likely be writing about that in the near future.  For now, here is some sample output from a quick perl parser that I wrote today.

$ ./pminfo.pl /var/tmp/snortstat

-= Tha Pig Doktah 0.1 Dev =-
Copyright (C) 2010 JJ Cummings

Report Info:
    Processed: /var/tmp/snortstat
    First Entry: Wed Sep  1 11:34:05 2010
    Last Entry: Wed Sep  1 22:27:47 2010
    Time Span: 0 days, 10 hours, 53 minutes and 42 seconds

Wirespeed:
    High: 6.683 Mbits/Sec | Wed Sep  1 12:54:00 2010
    Low: 0.007 Mbits/Sec | Wed Sep  1 18:14:18 2010
    Avg: 0.276 Mbits/Sec
  
% Packet Loss:
    High: 3.817% | Wed Sep  1 20:13:39 2010
    Low: 0.000% | Wed Sep  1 22:22:47 2010
    Avg: 0.095%

Additional Info:
    Avg Pkt Size: 363 bytes
    Avg Syns/Sec: 0.153
    Avg SynAcks/Sec: 0.105
    Avg Alerts/Sec: 0.001
    Avg Current Cached Sessions: 2326

Obviously this is was only as a quick test and does not include all of the important pieces of data.  Please feel free to hit me up in #snort (on freenode),  twitter, email(if'n you knows it), or post a comment here.

Cheers,
JJC

Hogging the Snort Host Attribute Table

Hogger is a new Snort supportive tool written in Perl, by Parker Crook, that allows you to create a Host Attribute Table from an nmap scan. But first, a little primer; A feature within Snort that has received some traction lately is that of the --enable-targetbased configuration option. This allows you to specify a Host Attribute Table that contains critical information about what your network host topology is (i.e. OS, services etc..). Using this information, snort can then properly reassemble fragments, track streams and a number of other things. All of these items are covered in Joel Esler's recent CSO article that can be found at This URL. This is an excellent article that covers what Host Attribute Tables are and how to use them, so please read the article for a better understanding!

Now that you know all about the Host Attribute Table, let's jump into the purpose and use of hogger. As mentioned previously, hogger was written by Parker Crook to create a Host Attribute Table using the resulting output of an nmap scan. Without further adieu, let's walk through the usage of hogger!

Requirements:

• Perl
• XML::Writer (perl module)
• Nmap
• Hogger

Steps:

1. Install XML::Writer
2. Get hogger
3. Install Nmap
4. Run Nmap with correct options
5. Run hogger against Nmap output file
6. Start your snorting!

1: Installing XML::Writer

$perl -MCPAN -e shell

cpan[1]> install XML::Writer

2: Get Hogger

$wget http://hogger.googlecode.com/files/hogger.tar.gz

$tar xvfz hogger.tar.gz

3: Install Nmap

Use whatever tool that your distribution / OS uses to install Nmap, or get the source from nmap.org and build it yourself!

4: Run Nmap

$mkdir ~/hogger/nmap

$cd ~/hogger/nmap

$nmap -sV -T4 -oN scan.nmap 192.168.1.0/24

Starting Nmap 5.21 ( http://nmap.org ) at 2010-02-25 18:46 UTC

..output suppressed...

5: Run hogger (against scan.nmap)

$cd ~/hogger

$./hogger.pl -c nmap/hostmap.csv -n nmap/scan.nmap -x nmap/host_attrib_table.xml

6: Start your snorting - At this point you can take the newly created host_attrib_table.xml file and place the path to it in your snort.conf, assuming your built snort with the correct option:

attribute_table filename /path/to/host_attrib_table.xml

Now that we have all of this running, let's examine some of the options that are currently available in hogger and dissect our hogger run: "./hogger.pl -c nmap/hostmap.csv -n nmap/scan.nmap -x nmap/host_attrib_table.xml".

Hogger help output:

Usage: ./hogger.pl [-r? -help] -n -c -x

Options:

-c Where the human-readable/modifiable csv file containing host information lives.

-n Where the nmap file containing host information lives.

-r Process the csv file and output to xml for snort, but do not read an nmap file.

-x Where you want to create the host_attribute table.xml (Overwrites existing files)

-help/? Print this information

Starting with the -c flag, this is a file that will be created by hogger if it does not exist, and is simply a csv file that you can modify (for those hosts that nmap either misses or is not as accurate as you would like). A few sample entries in the file (hostmap.csv) that we created in the above test run:

192.168.1.1, Linux, 23|tcp|telnet 53|tcp|domain 443|tcp|ssl/http

192.168.1.2, Linux, 23|tcp|telnet 53|tcp|domain 443|tcp|ssl/http

192.168.1.7, FreeBSD, 22|tcp|ssh 53|tcp|domain 80|tcp|http 3000|tcp|http 3128|tcp|http-proxy 3306|tcp|mysql 5000|tcp|http-proxy 8443|tcp|http

Next we see the -n flag, this is the flag that specifies where the nmap output file (that we previously created using the nmap -oN scan.nmap option). This is the file that hogger reads to create entries in the -c .

The -r flag is fairly straightforward and specifies that you ONLY want to read the csv file specified with the -c flag value.

The final flag that we will discuss is the -x flag, this is a required flag and tells hogger where you want the resulting output (the Host Attribute Table) to be placed. Examples from the output, matching those noted in the -c flag information above:

<SNORT_ATTRIBUTES>

<ATTRIBUTE_TABLE>

<HOST IP="192.168.1.1">

<OPERATING_SYSTEM>

<NAME ATTRIBUTE_VALUE="Linux" CONFIDENCE="90"></NAME>

<FRAG_POLICY>Linux</FRAG_POLICY>

<STREAM_POLICY>linux</STREAM_POLICY>

</OPERATING_SYSTEM>

<SERVICES>

<SERVICE>

<PORT ATTRIBUTE_VALUE=" 23" CONFIDENCE="100"></PORT>

<IPPROTO ATTRIBUTE_VALUE="tcp" CONFIDENCE="100"></IPPROTO>

<PROTOCOL ATTRIBUTE_VALUE="telnet 53" CONFIDENCE="95"></PROTOCOL>

</SERVICE>

</SERVICES>

</HOST>

<HOST IP="192.168.1.2">

<OPERATING_SYSTEM>

<NAME ATTRIBUTE_VALUE="Linux" CONFIDENCE="90"></NAME>

<FRAG_POLICY>Linux</FRAG_POLICY>

<STREAM_POLICY>linux</STREAM_POLICY>

</OPERATING_SYSTEM>

<SERVICES>

<SERVICE>

<PORT ATTRIBUTE_VALUE=" 23" CONFIDENCE="100"></PORT>

<IPPROTO ATTRIBUTE_VALUE="tcp" CONFIDENCE="100"></IPPROTO>

<PROTOCOL ATTRIBUTE_VALUE="telnet 53" CONFIDENCE="95"></PROTOCOL>

</SERVICE>

</SERVICES>

</HOST>

<HOST IP="192.168.1.7">

<OPERATING_SYSTEM>

<NAME ATTRIBUTE_VALUE="FreeBSD" CONFIDENCE="90"></NAME>

<FRAG_POLICY>BSD</FRAG_POLICY>

<STREAM_POLICY>bsd</STREAM_POLICY>

</OPERATING_SYSTEM>

<SERVICES>

<SERVICE>

<PORT ATTRIBUTE_VALUE=" 22" CONFIDENCE="100"></PORT>

<IPPROTO ATTRIBUTE_VALUE="tcp" CONFIDENCE="100"></IPPROTO>

<PROTOCOL ATTRIBUTE_VALUE="ssh 53" CONFIDENCE="95"></PROTOCOL>

</SERVICE>

</SERVICES>

</HOST>

Having said all of this, I am not going to go into detail about the flags used during the Nmap scan, suffice it to say that those are the suggested flags and that the -oN is required to produce the output file for hogger to read.

Overall I think that the concept behind hogger is excellent and that it should provide useful aide to all you snort heads out there! This tool gets a thumbs up from me and should be one that you put into your snort bag of tricks and is also one that I am planning on contributing to.

Cheers,

JJC