"Block the Bad" OSS IPS with Content Filtration and Transparent Proxy Acceleration pt 1.

In this two part series I will discuss and demonstrate the creation of an inline security and content filtration system built on FreeBSD 7.0R. What is a security and content filtration system you might ask? Simply put it is a system that has the capabilities of an IPS with the included benefit of advanced content filtration (things like blacklists, page content scoring "keywords etc", greylists, whitelists and so on...).

This first part, entitled "block the bad" will deal with the IPS aspect of the system that includes some new "or newly revisited" ways of utilizing snortsam with barnyard rather than directly patching snort. This is good for a variety of reasons that include the capability to keep your snort version updated without having to continually re-patch it for snortsam, and not having to load snort down with more work than what it was intended "SNIFFING J00r PAket F00".

Some things in the below documented barnyard snortsam plugin have been hacked together, and I am sure that more capable individuals "rotorhead, Obiwan..." will write a non-hacked-together plugin in the near future. But this will get you up and rolling for now.

A few assumptions are made before we get started... the first is that you have already built snort (2.8.1 is the latest as of the time I wrote this), and if not that you can follow the directions to do so on a previous posting of mine. The second assumes if you want to see output such as BASE, you read and followed that entire posting. The third assumption is that you know how to modify your kernel options and ultimately make and install a new kernel. The fourth and final assumption is that if any of the previous assumptions are not true, you know how to use google.

Now, to the heart of the subject at hand, we will be using the following for the remainder of the excercise:

1. Snort 2.8.1 (see above)
2. barnyard 0.20.0 (with a modified snortsam plugin)
3. snortsam 2.52
4. ipf
5. ipfw (this will come into play in the next part re: content filtering, but can also be used to block by entire source or destination *not protocol/port* hence ipf)

So, for our first step (since we have snort built/running) let's get our barnyard patched so that we have the snortsam plugin. If you previously built barnyard and still have all of the source, that's great... but remember to make clean before we do anything. For my purposes I'll be demonstrating with a freshly downloaded barnyard. You will need autotools "cd /usr/ports/devel/autotools/ && make install clean" to finish the patch work.

[jj@Azazel /usr/home/jj]$ wget http://www.snort.org/dl/barnyard/barnyard-0.2.0.tar.gz

2008-04-13 18:14:39 (537 KB/s) - `barnyard-0.2.0.tar.gz' saved [161543/161543]

[jj@Azazel /usr/home/jj]$ tar xvfz barnyard-0.2.0.tar.gz
x barnyard-0.2.0/

[jj@Azazel /usr/home/jj]$ wget http://www.snortsam.net/files/barnyard-plugin/barnyard-snortsam-patch.gz

2008-04-13 18:16:37 (148 KB/s) - `barnyard-snortsam-patch.gz' saved [27149/27149]

[jj@Azazel /usr/home/jj]$ gunzip barnyard-snortsam-patch.gz
[jj@Azazel /usr/home/jj]$ cd barnyard-0.2.0
[jj@Azazel /usr/home/jj/barnyard-0.2.0]$ patch -p1 < ../barnyard-snortsam-patch
Hmm... Looks like a unified diff to me...
...
Hunk #1 succeeded at 1.
Hunk #2 succeeded at 33.
Hunk #3 succeeded at 54.
...
done
[jj@Azazel /usr/home/jj/barnyard-0.2.0]$ ./autojunk.sh
configure.in:147: warning: underquoted definition of SN_CHECK_DECL
configure.in:147: run info '(automake)Extending aclocal'
configure.in:147: or see http://sources.redhat.com/automake/automake.html#Extending-aclocal
autoheader-2.61: WARNING: Using auxiliary files such as `acconfig.h', `config.h.bot'
autoheader-2.61: WARNING: and `config.h.top', to define templates for `config.h.in'
autoheader-2.61: WARNING: is deprecated and discouraged.
autoheader-2.61:
autoheader-2.61: WARNING: Using the third argument of `AC_DEFINE' and
autoheader-2.61: WARNING: `AC_DEFINE_UNQUOTED' allows one to define a template without
autoheader-2.61: WARNING: `acconfig.h':
autoheader-2.61:
autoheader-2.61: WARNING: AC_DEFINE([NEED_FUNC_MAIN], 1,
autoheader-2.61: [Define if a function `main' is needed.])
autoheader-2.61:
autoheader-2.61: WARNING: More sophisticated templates can also be produced, see the
autoheader-2.61: WARNING: documentation.
[jj@Azazel /usr/home/jj/barnyard-0.2.0]$

Now that we have the main part of the patch completed we need to make a few quick modifications to "src/output-plugins/op_alert_fwsam.c" so that it handles the barnyard output properly and loads the sid-msg.map file via a hard coded path (line 191). I threw a patch out there so that you don't need to do this manually, located here: http://www.redsphereglobal.com/data/tools/security/patches/barnyard-snortsam-hack.gz.

[jj@Azazel /usr/home/jj]$ wget http://www.redsphereglobal.com/data/tools/security/patches/barnyard-snortsam-hack.gz

2008-04-13 18:52:54 (1.15 MB/s) - `barnyard-snortsam-hack.gz' saved [641/641]

[jj@Azazel /usr/home/jj]$ gunzip barnyard-snortsam-hack.gz
[jj@Azazel /usr/home/jj]$ cd barnyard-0.2.0
[jj@Azazel /usr/home/jj/barnyard-0.2.0]$ patch -p1 < ../barnyard-snortsam-hack Hmm... Looks like a unified diff to me... The text leading up to this was: ...
Patching file src/output-plugins/op_alert_fwsam.c using Plan A...
Hunk #1 succeeded at 188.
Hunk #2 succeeded at 815.
done
[jj@Azazel /usr/home/jj/barnyard-0.2.0]$

This patch or "hack" has assumed that the location of your sid-msg.map is at /usr/local/etc/snort/sid-msg.map if this is not the case, you will need to edit /src/output-plugins/op_alert_fwsam.c around line 191 and specify the correct path. At this point you can configure barnyard and build as you normally would.

[jj@Azazel /usr/home/jj/barnyard-0.2.0]$./configure --enable-mysql
[jj@Azazel /usr/home/jj/barnyard-0.2.0]$make
[jj@Azazel /usr/home/jj/barnyard-0.2.0]$sudo make install

Your barnyard is now ready and we will cover the config file and startup after we get ipf and snortsam up and running.

The next step is to add the following to our Kernel so that we have ipf and ipfw enabled and running by default at boot.

# IPFW support

options IPFIREWALL #Enable IPFW directly in the kernel
options IPFIREWALL_FORWARD #Enable the Ip Forwarding function of IPFW
options IPFIREWALL_VERBOSE
options IPFIREWALL_DEFAULT_TO_ACCEPT
options IPDIVERT #allow this host to divert packets to/through different ints and routes

# IPF Support - default is to accept
options IPFILTER
options IPFILTER_LOG

Once these have been added please build your kernel, install and reboot. At this point we are ready to fetch and make snortsam.

[jj@Azazel /usr/home/jj]$ wget http://www.snortsam.net/files/snortsam/snortsam-src-2.52.tar.gz

2008-04-13 19:17:28 (497 KB/s) - `snortsam-src-2.52.tar.gz' saved [1075606/1075606]

[jj@Azazel /usr/home/jj]$ tar xvfz snortsam-src-2.52.tar.gz
x snortsam
[jj@Azazel /usr/home/jj]$ cd snortsam
[jj@Azazel /usr/home/jj/snortsam]$ sh ./makesnortsam.sh
-------------------------------------------------------------------------------
Building SnortSam (release)
-------------------------------------------------------------------------------
-------------------------------------------------------------------------------
Building SnortSam (debug)
-------------------------------------------------------------------------------
Done.
[jj@Azazel /usr/home/jj/snortsam]$sudo cp snortsam* /usr/local/bin/

That's it for the snortsam build, now we are ready to configure everything and fire it up for a test! The first thing that we will configure is our snortsam. There is a good amount of documentation under snortsam/docs/README.conf that covers basic configuration. For our purposes we will create the file /etc/snortsam.conf and place the following in it.

defaultkey secrets
port 6783
accept 192.168.1.0/24
keyinterval 30 minutes
ipf bge0

This configuration specifies a default key of "secrets" and that the snortsam daemon should listen on port 6783 for connectoins from the 192.168.1.0/24 network. The configuration also specifies that the connection between the client (barnyard) and snortsam daemon will be rekeyed every 30 minutes and that ipf will be used on bge0 locally.

On to the barnyard configuration, this file will be barnyard-snortsam.conf located at /usr/local/etc/. The only line that needs to be in this file is the one that calls the snortsam plugin for barnyard and specifies the host:port/password

output alert_fwsam: 192.168.1.7:6783/secrets

The barnyard snortsam plugin uses a sid-block.map file to define what sids will be blocked, how they will be blocked and for how long they will be blocked. The format is quite simple "sid: where[option],duration;" and to test we will put the file at /usr/local/etc/snort/sid-block.map with the following entry

9999999: src[conn], 15 seconds;

I chose sid 9999999 so that I could create a custom rule in my local.rules to test my configuration.

alert icmp any any -> 1.2.3.4 any (msg:"test"; sid:9999999;)

Assuming you were able to add that rule, we are now at the point to fire things up and give it a good old fashioned roll (all in debugging verbose mode of course)!

Restart your snort so that it sees the new SID if you have not done so... -HUP FTW!@!!
Start snortsam (must be as root right now to have access to ipf)

[jj@Azazel /usr/home/jj]$ sudo snortsam-debug

Start barnyard with the new config file (even if you have a previosly running barnyard from the previous security appliance article... this will run at the same time, we have specified a new waldo file and pid file). Note that the following is ALL ONE LINE... no line breaks or crs! Note that this uses the snort.alert and not the snort.log just like the syslog facility.

[jj@Azazel /usr/home/jj]$ sudo /usr/local/bin/barnyard -c /usr/local/etc/barnyard-snortsam.conf -g /usr/local/etc/snort/gen-msg.map -s /usr/local/etc/snort/sid-msg.map -d /var/log/snort/ -f snort.alert -w /var/log/snort/barnyard-snortsam.waldo -p /usr/local/etc/snort/classification.config -X /var/barnyard-snortsam.pid -vvv

After starting barnyard you should see the following debug output from your snortsam-debug:

Debug: Connection from: 192.168.1.7.
Debug: Received Packet: CHECKIN
Debug: Snort SeqNo: cbb9
Debug: Mgmt SeqNo : 7000
Debug: Status : 1
Debug: Version : 14

Now that everything is up and running we can test. The best way to test all aspects is to point a separate system at the IP of this box (default router/gateway) or on my system as evident by the above config "192.168.1.7" and ping 1.2.3.4 with that separate system. The ipfw options that we previously set in the kernel will allow this host to simply route the traffic to the proper destination. You should see debug output from your snortsam-debug as such:

Blocking host 192.168.1.43 in connection 192.168.1.43->1.2.3.4:0 (icmp) for 60 seconds (Sig_ID: 9999999).
Debug: [ipf][28201600] Plugin Blocking...
Debug: [ipf][28201600] command /bin/echo "@1 block in log level local7.info quick on bge0 proto 1 from 192.168.1.43/32 to 1.2.3.4/32"|/sbin/ipf -f -

We can see from the output that it is blocking the source address of 192.168.1.43 and proto 1 (ICMP) only. This means that this host can still browse the internet and do everything (other than send icmp to 1.2.3.4 for 60 seconds), this is a function of the [conn] option in the sid-block.map file.

Wonderful, we now have a functioning version of snortsam running off of the snort output and not snort directly. This means that we can upgrade / change our snort instance itself and not have to re-patch and mess with that... (this of course assumes that the version you use can output unified so that your patched version of barnyard can read it). The final step in this process is to add the sids that you want to block to the sid-msg.map file. I have modified the create-sidmap.pl file to create a sid-block.map compatible output by reading all of the .rules files in a directory and dumping "sid: src[conn], 30min;" output. This output blocks the service by source that the alert was generated from for 30 minutes. The file can be obtained at http://www.redsphereglobal.com/data/tools/security/patches/create-sidblock.pl.gz. Usage is simple and as follows (again, note that it's one line):

[root@Azazel /home/jj]# ./create-sidblock.pl /usr/local/etc/snort/rules/ > /usr/local/etc/snort/sid-block.map
[root@Azazel /home/jj]# tail -n 3 /usr/local/etc/snort/sid-block.map
2500000: src[conn],30min;
2510000: src[conn],30min;
9999999: src[conn],30min;

I suggest that you not put ALL sids in this file, but rather take a subset from rules files that you know are bad news. To do this simply copy the .rules files into a directory of your choice and run the script against that directory (note that the sid-block.map must always live in /usr/local/etc/snort at this time). Other suggestions include daemonizing your barnyard instance (-D) rather than -vvv. The rest you can figure out.

The next part of this series will cover adding content filtration and a transparent squid instance into the mix on this box.

Cheers,
JJC

managing snort rulesets cont...

I need to amend my previous posting about the usage of Oinkmaster to automate and manage your Snort rules. I had added in the simple script a command that updates the sid-msg.map in a fairly unclean way. There is, infact, included within the /contrib of Oinkmaster a nifty little script called create-sidmap.pl. This script reads all of the rules from the rules path that you specify and generates sid-msg.map output that can be redirected into a clean sid-msg.map file.

The location in my original posting that should be changed is highlighted here:

secure2# vi /usr/local/bin/autooinkall.sh
#! /bin/sh
#
# simple script to run oinkmaster and obtain bleeding threat updates
# in addition to the regular snort.org updates
#
/usr/local/bin/oinkmaster -o /usr/local/etc/snort/rules/
/usr/local/bin/oinkmaster -C /usr/local/etc/oinkmaster-bleeding.conf -o /usr/local/etc/snort/rules/
cat /usr/local/etc/snort/rules/bleeding-sid-msg.map >> /usr/local/etc/snort/rules/sid-msg.map
/bin/kill -HUP `cat /var/run/snort_em1.pid`
/bin/kill -HUP `cat /var/run/by.pid`

This should be changed to /path/to/your/create-sidmap.pl /path/to/rules/ > /usr/local/etc/snort/rules/sid-msg.map so that the whole thing looks like the following:

secure2# vi /usr/local/bin/autooinkall.sh
#! /bin/sh
#
# simple script to run oinkmaster and obtain bleeding threat updates
# in addition to the regular snort.org updates
#
/usr/local/bin/oinkmaster -o /usr/local/etc/snort/rules/
/usr/local/bin/oinkmaster -C /usr/local/etc/oinkmaster-bleeding.conf -o /usr/local/etc/snort/rules/
/usr/lobal/bin/create-sidmap.pl /usr/local/etc/snort/rules > /usr/local/etc/snort/rules/sid-msg.map
/bin/kill -HUP `cat /var/run/snort_em1.pid`
/bin/kill -HUP `cat /var/run/by.pid`

Regards,
JJC

Network Security Toolkit Pt 1 (using snort with barnyard, apache, mysql, php and BASE)

So, as the title implies this is the first in a multi-part series discussing a network security toolkit built on Open Source. The system that I am using to create this is FreeBSD 6.2 RELEASE. We will be building tools mostly from the ports tree, but a few are not yet ported, so we will manually install those as needed.

Assumptions:

1. You have installed and somewhat adequately secured FreeBSD
2. You have a basic knowledge of the FreeBSD OS and have network connectivity on this system
3. If you don't know this / have your system setup, you know how to RTFM (Google)
   
4. You have updated your ports tree prior to building this stuff

Ok, now that we have established the assumptions, let's go over the software that we are going to use for this implementation:


Requirements:

1. Apache22 - Core webserver to serve up your base /usr/ports/www/apache22
2. MySQL 50 - Database server to house your snort data /usr/ports/databases/mysql50-server/ (also -client and -scripts)
3. php5 - php support for apache22 /usr/ports/lang/php5
4. base - your snort alert and event viewing tool /usr/ports/security/base
5. barnyard - the tool that takes the unified snort logs and puts them into your MySQL database /usr/ports/security/barnyard
6. snort - your IDS engine /usr/ports/security/snort
7. oinkmaster - snort rule updater /usr/ports/security/oinkmaster
   

Optional:

• phpmyadmin a GUI tool to help you manage your MySQL databases /usr/ports/databases/phpmyadmin

Installation:

Apache22 - install and edit httpd.conf then start and test

secure2# cd /usr/ports/www/apache22 && make install clean
.
.
.
secure2# vi /usr/local/etc/apache22/httpd.conf
.
secure2# /usr/local/sbin/apachectl start
.
.
you should now be able to browse to http://ipofinstalledhost/ and receive the "It Works!" message.

MySQL 50 - install and set a password for root user

secure2# cd /usr/ports/databases/mysql50-server/ && make install clean
.
.
.
secure2# cd /usr/ports/databases/mysql50-scripts/ && make install clean
.
.
.
secure2# cd /usr/ports/databases/mysql50-client/ && make install clean
.
.
.
secure2# /usr/local/etc/rc.d/mysql-server start
secure2# mysqladmin -u root password "passwordwithouquoteshere"
secure2# mysqladmin -u root -h localhost password "passwordwithouquoteshere"
secure2# mysql -u root -p
Enter password:
Welcome to the MySQL monitor. Commands end with ; or \g.
Your MySQL connection id is 88965
Server version: 5.0.45-log FreeBSD port: mysql-server-5.0.45

Type 'help;' or '\h' for help. Type '\c' to clear the buffer.

mysql> flush privileges;
Query OK, 0 rows affected (0.00 sec)

mysql>

php5 - php5 install be sure to select apache when asked for configuration options!

secure2# cd /usr/ports/lang/php5 && make install clean
.
.
.
secure2# vi /usr/local/etc/apache22/httpd.conf
.
.# insure that the following is in your httpd.conf
AddType application/x-httpd-php .php
AddType application/x-httpd-php-source .phps
.
.
secure2# apachectl restart

BASE - install and prepare base for use

secure2# cd /usr/ports/security/base && make install clean
.#be sure to select MySQL support...PDF is optional
.
.
. #configure apache22 to serve base
secure2# vi /usr/local/etc/apache22/httpd.conf
.
##place this at the bottom of httpd.conf
Alias /base/ "/usr/local/www/base/"
##save
secure2# apachectl restart
chown -R www:www /usr/local/www/base/
mysql -u root -p
mysql> create database base_demo;
Query OK, 1 row affected (0.00 sec)
secure2# mysql -u root -p base_demo < /usr/local/www/base/sql/create_base_tbls_mysql.sql

Now that we have setup the previous components we can access BASE by browsing to http://ipofinstalledhost/base/ where you should see the following BASE configuration screen:

We will be going through this setup process in a bit, first we need to get snort with barnyard up and running.

snort - install and configure the system to use snort.

secure2# cd /usr/ports/security/snort/ && make install clean
#be sure to Enable MySQL support
.
#add tables to the database...
secure2# mysql -u root -p base_demo < /usr/local/share/examples/snort/create_mysql #configure snort.conf secure2# vi /usr/local/etc/snort/snort.conf #modify this file as needed for your network.... . #configure logging only for unified..i.e. the following lines output alert_unified: filename snort.alert, limit 128 output log_unified: filename snort.log, limit 128 #I'll write more about tuning SNORT rules later, but for now this will get you started. #before we can start snort to test, we will need to get the latest signatures using oinkmaster!

oinkmaster - magical snort rule updater?

secure2# cd /usr/ports/security/oinkmaster/ && make install clean
.
.
secure2# vi /usr/local/etc/oinkmaster.conf
#get your registration code from snort.org and plug it into the Snort 2.4 download url in the oinkmaster file....

barnyard - log unified snort output to database

secure2# cd /usr/ports/security/barnyard/ && make install clean
#be sure to Enable MySQL support
.
secure2# vi /usr/local/etc/barnyard.conf
#configure your barnyard to log the data..the following lines are important
# set the hostname (currently only used for the acid db output plugin)
config hostname: secure2
# set the interface name (currently only used for the acid db output plugin)
config interface: em1
# Converts data from the dp_log plugin into an approximation of Snort's
# "ASCII packet dump" mode. Argument:

output log_dump
# password database connect trash
output alert_acid_db: mysql, sensor_id 1, database base_demo, server 127.0.0.1, user root, password passwordhere
output log_acid_db: mysql, sensor_id 1, database base_demo, server 127.0.0.1, user root, password passwordhere detail full

Putting it all together and starting it up.

secure2# oinkmaster -o /usr/local/etc/snort/rules/
secure2# /usr/local/bin/snort -c /usr/local/etc/snort/snort.conf -i em1 -u root -D > /dev/null -n
secure2# /usr/local/bin/barnyard -c /usr/local/etc/barnyard.conf -g /usr/local/etc/snort/gen-msg.map -s /usr/local/etc/snort/rules/sid-msg.map -d /var/log/snort/ -f snort.log -w /var/log/snort/barnyard.waldo -p /usr/local/etc/snort/rules/classification.config

once all of this is done, your snort should be logging the unified data, barnyard reading it and slapping it into your database.

Now, let's get back to our configuration for BASE. Browse to http://ipofinstalledhost/base/ and select the continue option. The next few screens are filled with the configuration options to connect you to the base database(s). Simply input the information for the db, dbuser, dbpass and you will be all set!

Congratulations, you now have a functioning SNORT install using barnyard, MySQL and BASE. I did not cover the install of phpmyadmin, but you should be able to Google it and figure it out...

I'll be following this post up with the addition of bleeding snort rules, how to properly tune your snort configuration, the use of SGUIL (TRUE NSM) and several other goodies relating to network security... so stay posted.

Cheers,
JJC