Haz The Drowning Rat? - PulledPork 0.5.0 is now floating!

This release of PulledPork (The Drowning Rat) represents quite a bit of development to include a number of community requested capabilities, change a few around, and repair some bugs!  Again, I would like to thank the community for their support, contribution and use of the PulledPork Snort rule management system.  The next section is an excerpt from the README.CHANGES and below there I may discuss some example use-cases and include some sample output.

PulledPork Changelog

v0.5.0

New Features / changes:
- Automatic VRT tarball name determination (based on local Snort Version)
- Full support for ET Pro rulesets
- Full support for new ET Download scheme
- Issue #27 Modifysid capability
- Capability to retrieve multiple rulesets in a single run
- Issue #24 Added verbose output showing all requests, results and urls
- Verbose output now shows percentage bar for downloads
- Extra Verbose output now shows additional HTTP debug!
- Set value in default.conf file to https for VRT downloads
- Set UA Value to (PulledPork/X.X.X)
- Capability to log critical information to syslog
- Grabonly option, for those that only want to download the tarball(s)
- Issue #34 Added the capability to specify the order of disable / enable / drop
    using the state_order configuration option in the master config file
- Added a contrib directory
- Added oink-conv.pl to contrib directory
    * converts oinkmaster config files to PP config files
    * Thx Russell Fulton!
- Added README.CONTRIB to track contrib files (ohai manifest)
- Perl Modue Requirement Changes (SEE SECTION BELOW)
- Issue #38 Added capability to extract reference docs from tarball and
    store in a defined path, NOTE this dramatically increases PP runtime
    * runtime value is -r

Bug Fixes:
- Should now correctly use environmentally set proxy settings
    * Shout to pkthound for his work and contribution here!
- Fixed case where rules with multiple flowbit (un)?set values would not
    properly populate all of the flowbit values into the rules hash
- Bug #29 - fixed to allow for proper sid-msg.map generation
- Bug #28 - fixed numerous spellification issues
- Bug #32 - fixed to allow for so stub generation in nodownload and !nodownload case


Perl Module Requriement Changes:
- LWP::Simple no longer
- LWP::UserAgent now required
- HTTP::Request now required
- HTTP::Status now required
- SYS::Syslog now required
- Crypt::SSLeay now required
- Carp now required

As you can see, and as I had indicated, there are a number of significant improvements and fixes.  It is important to note that there are a number of changes, that include new and changed options, to the master config and the addition of the modifysid.conf file that allows you to modify rules based on regular expression matches etc...

Of course we also now fully support (ET Pro and the new ET open) rules and the capability to download multiple rulesets in a single run, rather than multiple config files referencing other .rules files as local rules etc...

One other seemingly insignificant change is the capability to change the order that the rules modification routines run, this means that you can more granularly control rule state.  The default processing order is (enable, drop, disable), this can now be changed though to allow for the disabling of all rules in a specific category (or however you would do it) and then selectively enabling rules out of that category, by simply changing the run order to disable,drop,enable.  Of course combining this with the pcre, category, modifysid etc.. capabilities gives you quite a bit of versatility.

So, without further adeau, I give you:

    http://code.google.com/p/pulledpork/
      _____ ____
     `----,\    )
      `--==\\  /    PulledPork v0.5.0 The Drowning Rat
       `--==\\/
     .-~~~~-.Y|\\_  Copyright (C) 2009-2010 JJ Cummings
  @_/        /  66\_  cummingsj@gmail.com
    |    \   \   _(")
     \   /-| ||'--'  Rules give me wings!
      \_\  \_\\
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Checking latest MD5 for snortrules-snapshot-2861.tar.gz....
    They Match
    Done!
Prepping rules from snortrules-snapshot-2861.tar.gz for work....
    Done!
Checking latest MD5 for etpro.rules.tar.gz....
    They Match
    Done!
Prepping rules from etpro.rules.tar.gz for work....
    Done!
Checking latest MD5 for emerging.rules.tar.gz....
    They Match
    Done!
Prepping rules from emerging.rules.tar.gz for work....
    Done!
Reading rules...
Reading rules...
Activating security rulesets....
    Done
Setting Flowbit State....
    Enabled 264 flowbits
    Enabled 29 flowbits
    Enabled 4 flowbits
    Enabled 2 flowbits
    Done
Writing /home/jj/snort.rules....
    Done
Generating sid-msg.map....
    Done
Writing /home/jj/sid-msg.map....
    Done
Writing /home/jj/sid_changes.log....
    Done
Rule Stats....
    New:-------0
    Deleted:---0
    Enabled Rules:----4506
    Dropped Rules:----0
    Disabled Rules:---17797
    Total Rules:------22303
    Done
Please review /var/log/sid_changes.log for additional details
Fly Piggy Fly!

Bah, Paste chopped my flying pig up ;-)

Get it here:

pulledpork-0.5.0.tar.gz latest hashes:
MD5SUM = 60c0abe78945876c643760b3bb2afdb6
SHA256 = 9e69873d737e4fc8dfd9b3a98316e4ff41bd8c4accda72f18036b96568c48872

Cheers,

JJC 

PulledPork 0.4.2 501 error when downloading rules

This issue most typically stems from a missing Perl Module that is required to communicate via SSL using LWP::Simple.  This required Perl Module is Crypt::SSLeay and is not included in the LWP::Simple redistributed package from the Ubuntu 8.x repositories, and will typically fail to install via CPAN on many Ubuntu server installations.  As such you simply need to do the following (on Ubuntu, since this is the only place I have seen it broken):

sudo apt-get install libcrypt-ssleay-perl

Of course if you are not running Ubuntu then you will need to use CPAN or find whatever repackaged garbage that your distro is using to distribute this ;-).

One other cause could be that your root certificates are outdated, so if you have the aforementioned PM installed and are still receiving a 501.. this is likely the cause... google how to update your root certificates for your distro!  Again, for the sake of completeness, this is how you do it on Ubuntu:

sudo apt-get install ca-certificates
sudo update-ca-certificates

I have also added this to the PP FAQ.

Cheers,
JJC

PulledPork 0.4.2 - get it while it's hawt!

This release represents a number of significant enhancements and features (all listed below). Probably the most important to note are the changes from a delimeter of | to : when modifying rule state. We also now automatically determine snort version and OS arch. One of the most useful features, IMHO, is the pcre: rule state modification capability.. see the rule modification configs for more details... but let's say that I wanted to disable ALL MSXX rules because I run a strictly *nix environment... simply place something like pcre:MS\d{2}-\d+ into the disablesid.conf and use that file by specifying -i.

As noted below, there are MANY other changes, fixes, and additions so please don't hesitate to ask questions in irc (freenode #pulledpork) or on the mailing list.

get it here -> http://code.google.com/p/pulledpork

v0.4.2

New Features / changes:

• Capability to modify rules by category (See README.CATEGORIES)
• Capability to modify rules using regular expressions (pcre:) - See sid modification configs
• Capability to use regular expressions in specific rule modifications - See sid modification configs
• Changed the | delimiter for cve,bugtraq etc to :
• Added README.CATEGORIES
• Added README.SHAREDOBJECTS
• Follow flowbit chains
• Moved README files to doc
• Automatically determine arch
• Automatically determine Snort Version
• Added some verbiage surrounding HUP vs Restart vs When/where/who and how
• Added support for new snort.org download scheme of http://snort.org/reg-rules...

Bug Fixes:

• Certain rules specific GID values were not being properly parsed by the modifysid sub.
• Bug #20 fixed, ranges are no longer off by +1 additional rule being enabled
• Enhancement request #21, added more descript information to dropsid.conf and to README
• Fixed flaw that caused certain flowbits to not be set (when GID boundaries were crossed and multiple keys were checked)
• Enhancement request #22 updated the master config file to contain all of the currently available precompiled SO rules
• Remove risky system calls, use handles instead

pulledpork-0.4.2.tar.gz latest hashes:
MD5SUM = d11b9d884f940a0df293718a4d4b3913
SHA256 = 3491b8c3c99c621cfd6467da2c43866f33ede1d096538e4a497cdf52b49ad677

Cheers,
JJC

PulledPork 0.4.1, I see your sensitive data!

In conjunction with the Snort 2.8.6 release and the new Snort Rules tarball format, pulledpork 0.4.1 is now released!  As noted below, there are a number of changes and fixes.  When updating your pulledpork, please be sure to use the latest master configuration file that is included in the release tarball and read through it thoroughly.

Notable changes include the tarball filename change, preprocessor rules and sensitive data rules.  Note that pulledpork 0.4.0 will still work with 2.8.6 but will not properly make use of the new rules that I just listed and that you will need to change the rules tarball name for VRT releases.  Please also note that if you use pulledpork 0.4.1 and are still using Snort 2.8.5.3 that you need to make some changes in the "ignore" variable section of the pulledpork.conf file.

New Features/changes:

• Flowbit tracking! - This means that all flowbits are not enabled when a specific base ruleset is specified (security etc...) but rather all flowbits are now tracked, allowing for only those that are required to be enabled.

• Adjusted pulledpork.conf to account for new snort rules tarball naming and packing scheme, post Snort 2.8.6 release.

• Added option to specify all rule modification files in the master pulledpork.conf file - feature request 19.

• Added capability to specify base ruleset (see README.RULESETS) in master pulledpork.conf file.

• Handle preprocessor and sensitive-information rulesets

Bug Fixes:

• 18 - non-rule lines containing the string sid:xxxx were being populated into the rule data structure, added an extra check to ensure that this does not occur

• Cleaned up href pointers, syntactical purposes only...

• Modified master config to allow for better readability on smaller console based systems

• Error output was not always returning full error, fixed this

Thanks to the community for continued support and feedback!

Cheers,
JJC

Baconator Renamed => Pulled_Pork

So, for some "mostly obvious reasons" I have renamed the Baconator project to Pulled_Pork. This was for a variety of reasons and if you really want to know I'll explain it.. Just drop by #snort on freenode... suffice it to say that this new name is more fitting. Please also note the google code location has changed from /p/baconator to /p/pulledpork. I did note on the baconator page that this change has occured.

The new location => http://code.google.com/p/pulledpork/

As always, thanks for the support and please fetch the latest version to do some testing for me!

Cheers,
JJC

Baconator 0.1 Beta 2 (try me)

I have completed the 0.1 Beta 2 of Baconator and believe it to be fairly stable and user friendly! Please give it a roll (it's not in a tarball yet, so you will have to check it out as noted below) and let me know if you experience any issues or have any updates / features that you would like to see.

The timeline:
Release 0.1:(This is complete)

• First Beta Release
• Downloads latest rules file
• Verifies MD5 of local rules file
• If MD5 has not changed from snort.org.. doesn't fetch files again
• handle both rules and so_rules
• Capability to generate stub files

Release 0.2:(I have started to work on this piece, probably finished in a few more weeks)

• Rule modification, i.e. disabling of specific rules within rule sets
• Capability to compile so_rules from source
• Outputs changes in rules files if any rules have been added / modified
• Compares new rules files with current rule sets
• Option to use Emerging-Threats rules in addition to snort.org rules
• Option to define custom URL to fetch rules tarballs from
• Automated retrieval of certain variables (Distro, Snort Version.. etc)

Next Release...

• TBD by community needs / requests

Visit the google code site for info on how to check out the code etc..

http://code.google.com/p/baconator/

Cheers,
JJC

Baconator - Shared Object Snort Rule Management!

Recently while taking a plane ride from one lovely airport to another and doing some snort shared object rule development, I realized that I did not have a clean and easy way of fetching the latest snort rule tarball.

Don't get me wrong and misinterpret this post, I love Oinkmaster and have been a user of it for many a year!

Now, having said that... Oinkmaster does have it's shortcomings (for me anyway); the least of which is certainly not the fact that it currently does NOT handle shared object rules. With the release of Snort 2.8.4 and it's awesome new dcerpc2 preprocessor... the use of so_rules will most likely be much more prevalent.. and as such, with threats like Conficker and it's varients out there, I needed a way to handle this.

I did consider modifying Oinkmaster to fit my needs, but when I started writing the code at 30,000 feet... I didn't have the Oinkmaster codebase with me.

As a direct result of this thought and the lack of codebase on the plane... I started Baconator. Baconator is a Snort rule management tool that also handles so_rules, the creation of stub files from said so_rules, complete file validation (via MD5) against current VRT releases. It also does much more... or, will anyway.

I'll be posting more about Baconator as I complete the code. For now, if you want to try it out (it's not yet complete) you can checkout the code from the svn repo at http://code.google.com/p/baconator/.

The current code will fetch the latest ruleset from snort.org (ultimately I'll probably build the functionality in to fetch from ET). If you have an existing copy of the rules tarball from snort.org it will fetch the latest rule tarball md5 from snort.org and compare so that it doesn't re-fetch the same tarball again. It then performs the various extraction routines as defined in the conf file or at runtime and puts the files where you tell it to.. the rules files that is!

More info can be found on the google code page for Baconator. I'll also be updating that site regularly with updates to the timeline, current svn etc...

Cheers,
JJC

managing snort rulesets cont...

I need to amend my previous posting about the usage of Oinkmaster to automate and manage your Snort rules. I had added in the simple script a command that updates the sid-msg.map in a fairly unclean way. There is, infact, included within the /contrib of Oinkmaster a nifty little script called create-sidmap.pl. This script reads all of the rules from the rules path that you specify and generates sid-msg.map output that can be redirected into a clean sid-msg.map file.

The location in my original posting that should be changed is highlighted here:

secure2# vi /usr/local/bin/autooinkall.sh
#! /bin/sh
#
# simple script to run oinkmaster and obtain bleeding threat updates
# in addition to the regular snort.org updates
#
/usr/local/bin/oinkmaster -o /usr/local/etc/snort/rules/
/usr/local/bin/oinkmaster -C /usr/local/etc/oinkmaster-bleeding.conf -o /usr/local/etc/snort/rules/
cat /usr/local/etc/snort/rules/bleeding-sid-msg.map >> /usr/local/etc/snort/rules/sid-msg.map
/bin/kill -HUP `cat /var/run/snort_em1.pid`
/bin/kill -HUP `cat /var/run/by.pid`

This should be changed to /path/to/your/create-sidmap.pl /path/to/rules/ > /usr/local/etc/snort/rules/sid-msg.map so that the whole thing looks like the following:

secure2# vi /usr/local/bin/autooinkall.sh
#! /bin/sh
#
# simple script to run oinkmaster and obtain bleeding threat updates
# in addition to the regular snort.org updates
#
/usr/local/bin/oinkmaster -o /usr/local/etc/snort/rules/
/usr/local/bin/oinkmaster -C /usr/local/etc/oinkmaster-bleeding.conf -o /usr/local/etc/snort/rules/
/usr/lobal/bin/create-sidmap.pl /usr/local/etc/snort/rules > /usr/local/etc/snort/rules/sid-msg.map
/bin/kill -HUP `cat /var/run/snort_em1.pid`
/bin/kill -HUP `cat /var/run/by.pid`

Regards,
JJC

bleeding and regular rules with Oinkmaster

As promised in a previous posting, here is the information about configuring oinkmaster to obtain both bleeding edge threat rules and standards snort rules.

first let's copy our /usr/local/etc/oinkmaster.conf file so that we can have a new config file for our bleeding rules.

secure2# cp /usr/local/etc/oinkmaster.conf /usr/local/etc/oinkmaster-bleeding.conf
secure2# vi /usr /local/etc/oinkmaster-bleeding.conf
#replace your url string w/ the following...
# Example for rules from the Bleeding Snort project
url = http://www.bleedingthreats.net/rules/bleeding.rules.tar.gz
#save the file

Now, let's retrieve the files and put all of the sid-msg.map trash together....for simplicity sake let's just put this into a script...

secure2# vi /usr/local/bin/autooinkall.sh
#! /bin/sh
#
# simple script to run oinkmaster and obtain bleeding threat updates
# in addition to the regular snort.org updates
#
/usr/local/bin/oinkmaster -o /usr/local/etc/snort/rules/
/usr/local/bin/oinkmaster -C /usr/local/etc/oinkmaster-bleeding.conf -o /usr/local/etc/snort/rules/
cat /usr/local/etc/snort/rules/bleeding-sid-msg.map >> /usr/local/etc/snort/rules/sid-msg.map
/bin/kill -HUP `cat /var/run/snort_em1.pid`
/bin/kill -HUP `cat /var/run/by.pid`

That's it.. now run the script or crontab it... enjoy

Cheers,
JJC