Numeric IP - SSIM Representation

IPs in the SSIM as of 4.7.3 are represented in both positive and negative form. To convert from an IP to this form, use the following, replacing [ipvariable] with your dotted quad IP:

SELECT CASE WHEN (IPNumber - 2147483648) > 0 THEN IPNumber -(2*2147483648) ELSE IPNumber END AS IPNUMBER2 FROM ( SELECT 16777216 *
CAST(LEFT(IPNumber1, LOCATE('.', IPNumber1, 1)-1) AS BIGINT) + 65536 * CAST(SUBSTR(IPNumber1, LOCATE('.', IPNumber1, 1) + 1, LOCATE('.',
IPNumber1,LOCATE('.', IPNumber1, 1) +1) - LOCATE('.', IPNumber1, 1) - 1) AS BIGINT) + 256 * CAST(SUBSTR(IPNumber1, LOCATE('.',
IPNumber1,LOCATE('.', IPNumber1, 1) +1) + 1, LOCATE('.', IPNumber1, LOCATE('.', IPNumber1,LOCATE('.', IPNumber1, 1) +1) +1) - LOCATE('.',
IPNumber1,LOCATE('.', IPNumber1, 1) +1) - 1) AS BIGINT) + CAST(RIGHT(IPNumber1, LENGTH(IPNumber1) - LOCATE('.', IPNumber1,
LOCATE('.', IPNumber1,LOCATE('.', IPNumber1, 1) +1) +1)) AS BIGINT) AS IPNumber FROM (SELECT '[ipvariable]' AS IPNumber1 FROM
SYSIBM.SYSDUMMY1) AS LOCXTMP0) AS TEMP2

ref: http://stackoverflow.com/questions/6695428/how-to-split-an-ip-address-string-in-db2-sql-for-converting-to-ip-number

SSIM Red Queues

When you see all three left red bars completely red, and the EPS counts are 0, here are some troubleshooting tips:

- Check the Sesa Agent if it's a collector server
- SSH into the collector server in question
- "su -" (change to root)
- tail the collector log which produces the most EPS or highest spikes on this collector
e.g. "tail -200 /opt/Symantec/sesa/Agent/logs/stonegate.log"
- If you see WARN entries containing "the agent queue is full" you may need to adjust the agent queue configurations: SSIM console: System > Product Configurations > SES > STATE > SSIM Agent and Manager > Agent Configurations > Standard (or the configuration applied to this server) > Logging
- adjust the following properties as needed, e.g. to increase the queue size, flush size, decrease the queue flush time (so that events are cleared more frequently).
- Save the configuration (the agent should pick up the settings automatically)

- Check the Recv-Q and Send-Q of Netstat
- on both the archiver and the correlator do "netstat -anp | grep 100"
- look for signs of downstream bottlenecks
- Examples
- if archiver is queued up sending to correlator:10010, check the correlator queues. E.g. if the correlator is queued up sending to 127.0.0.1:10080, there may be a problem with the asset service. Check assetsvc.log for errors. Try restarting the assetsvc service, and monitor the netstat -anp | grep 100 for movement in the queues.
As db2admin, cd ~/sqllib/bin
db2 connect to SESA
db2 list applications show detail
Check the SSIM-ASSET application for signs that it is currently Executing. Let it run for a while to see if it finishes.
- on correlator, check "netstat -anp | grep 555" for large number of FIN_WAIT2
- on the correlator check the icesvc.log, assetsvc.log, and rulesvc.log for recent errors (see /opt/Symantec/simserver/logs)
- Disable any rules temporarily that contain references to lookup tables to determine whether any of these rules are causing the backlog.
- Check swap space usage on Correlator. If there is any Swap used, reboot the server.
- Check SSIM Event logs for warnings or errors; query all recent logs where product = Symantec Security Information Manager or product = SSIM System
- Do a query on Event Type ID = Conclusion Updated. Verify that not more than a few events a minute are being created
- Check top on both archiver and correlator. Look for processes whose VIRT is close to the same size as RES. This process may be overworked. The memory allocation can be increased in the /opt/Symantec/simserver/svclauncher.cfg config.
- Investigate whether simserver (correlator service) is overloaded.
- ascertain the pid of the simserver in top
- do lsof | grep | awk '{print $7,$1,$2,$3,$4,$5,$6,$8,$9}' | sort -n
- look for any unusually large files in output. e.g. .que files larger than 10MB might indicate an overworked correlator.
- Do an advanced SQL query in the UI to determine whether there are any unusually large number of incidents for particular rule: SELECT count(incident_code),incident_code FROM SYMCMGMT.SYMC_IMR_INCIDENT_LIST_VIEW group by incident_code
order by count(incident_code) desc
or: SELECT desc_id, sum(event_count) as sumevtcnt FROM symc_sim_conclusion
where modified_time >= (current timestamp - 1 DAY) group by desc_id
order by sumevtcnt desc
- Disable any rules with more than 50,000 events in one day. For example, a system rule that can cause issues if not adapted is Internal Port Sweep. Simserver process memory will be exhausted typically after 24 hours, and Queues will increase steadily in /opt/Symantec/simserver/queues/ice/input. IIRC the max is three 64MB files here. If there are more than one file, the icesvc is backed up due to the amount of event tracking updates waiting to be pushed to the database. Verify that SSIM-ICE is "UOW Executing" in db2 list applications show detail.
- Also check Events > System Queries > SSIM > SSIM System > Count of Conclusions by Rule Name
- Look for FIN_WAITs associated with large queues in a frozen state in netstat for the simserver process (get pid from `status`). E.g.:
tcp 80384 0 ::ffff:127.0.0.1:10010 ::ffff:127.0.0.1:36446 ESTABLISHED 6131/java
tcp 0 145961 ::ffff:127.0.0.1:36446 ::ffff:127.0.0.1:10010 FIN_WAIT1 -
This could indicate a memory resource problem in the simserver service or another service. Check the target port, and cross reference the admin guide with the service in question. e.g. 10010 is simserver, 55562 is icesvc, etc.
If it's really in frozen state, the process may have crashed (i.e. you see 0 EPS). If the Recv-Q/Send-Q varies and EPS varies, then this could be indicative of High IO Waits. Check the IOWait in the Statistics tile of the GUI. Also check vmstat 1. Sustained IOWaits of more than 2% could indicate the drive is not fast enough.

Install syslog-ng 3.3 on CentOS 5.5

Download from Balabit: syslog-ng_3.3.0bet1.tar.gz or syslog-ng_3.3.0beta2.tar.gz (see http://www.balabit.com/downloads/files?path=/syslog-ng/sources/3.3.0beta2/source) and unpack.
to get the latest, install git:
first update the repo to obtain epels:
rpm -Uvh http://download.fedora.redhat.com/pub/epel/5/i386/epel-release-5-3.noarch.rpm
then yum install git
or download the snapshot from http://git.balabit.hu/?p=bazsi/syslog-ng-3.3.git;a=tree
and untar over the above.

There may be prereqs on your system.

Download these rpms and install. :

rpm -Uvh kernel-headers-2.6.18-238.el5.i386.rpm
rpm -Uvh glibc-headers-2.5-58.i386.rpm
rpm -Uvh glibc-devel-2.5-58.i386.rpm
rpm -Uvh gcc-4.1.2-50.el5.i386.rpm
rpm -Uvh zlib-1.2.3-4.el5.i386.rpm
rpm -ivh zlib-devel-1.2.3-4.el5.i386.rpm
rpm -Uvh libffi-3.0.5-1.el5.kb.i386.rpm
rpm -Uvh libffi-devel-3.0.5-1.el5.kb.i386.rpm

Then install glib
wget ftp://ftp.gtk.org/pub/glib/2.29/glib-2.29.4.tar.bz2
tar jxvf glib-2.29.4.tar.bz2
cd glib-2.29.4
./configure
make
make install
done with installing glib prerequisite

Before install syslog-ng 3.3 beta, do
export PKG_CONFIG_PATH=/usr/local/lib/pkgconfig:$PKG_CONFIG_PATH
so that eventlog is found.

also do yum install openssl-devel
or

rpm -Uvh e2fsprogs-devel-1.39-23.el5_5.1.i386.rpm
rpm -Uvh keyutils-libs-devel-1.2-1.el5.i386.rpm
rpm -Uvh libselinux-devel-1.33.4-5.7.el5.i386.rpm
rpm -Uvh libsepol-devel-1.15.2-3.el5.i386.rpm
rpm -Uvh libselinux-devel-1.33.4-5.7.el5.i386.rpm
rpm -Uvh krb5-devel-1.6.1-55.el5.i386.rpm
rpm -Uvh openssl-devel-0.9.8e-12.el5_5.7.i386.rpm

also:
cd eventlog-0.2.12/
./configure
make
make install
then enter syslog-ng dir and
./configure --enable-pcre --disable-mongodb --disable-ipv6
make
make install
Note: To install with PCRE enabled, must first yum install pcre-devel
After that, some more setup:
mkdir /usr/local/var
cd /root/syslog-ng/syslog-ng-3.3.0beta1/contrib/
cp rhel-packaging/syslog-ng.init /etc/init.d/syslog-ng
cp rhel-packaging/syslog-ng.conf /usr/local/etc/
cp rhel-packaging/syslog-ng.logrotate /etc/logrotate.d/syslog-ng
chmod 755 /etc/init.d/syslog-ng
vi /etc/init.d/syslog-ng
Then change the binary to "/usr/local/sbin/syslog-ng"
Then start the service:
chkconfig syslog off or chkconfig rsyslog off as applicable

chkconfig syslog-ng on

service syslog stop

service syslog-ng start

Then edit /usr/local/etc/syslog-ng.conf
add options { threaded(yes); }; to the top, and any other configs.

service syslog-ng restart

CentOS 5.5 Kernel Upgrade and iotop Install

To run iotop.py you need a kernel greater than what CentOS provides. The following steps guide this upgrade, so that you can use iotop.

cd /usr/src
wget http://www.kernel.org/pub/linux/kernel/v2.6/linux-2.6.39.1.tar.bz2
tar xjf linux-2.6.39.1.tar.bz2
ln -s linux-2.6.39.1 linux
cd linux
make mrproper
cp /boot/config-`uname -r` .config
yes "" | make oldconfig (this updates .config to all defaults)

edit .config, adding or updating the following options:
CONFIG_SYSFS_DEPRECATED=y
CONFIG_SYSFS_DEPRECATED_V2=y

make
make modules_install
make install
reboot

rpm -Uvh http://download.fedora.redhat.com/pub/epel/5/i386/python-ctypes-1.0.2-2.el5.i386.rpm
wget http://guichaz.free.fr/iotop/files/iotop-0.4.3.tar.gz
tar zxvf iotop-0.4.3.tar.gz
cd iotop-0.4.3
./iotop.py


Disclaimer: Perform the above instructions at your own risk.

For instructions on how to make the rpm (so the lengthy compilation can be done once for future installs), see http://www.howtoforge.com/kernel_compilation_centos
Note: will need to install rpm-build and ncurses-devel

Refs:

http://www.centos.org/modules/newbb/viewtopic.php?topic_id=8488

http://forum.openvz.org/index.php?t=msg&goto=41203&#msg_41203

http://guichaz.free.fr/iotop/

Squid Proxy Quick Start Guide

First install dependency: yum install gcc-c++

Download squid, e.g. the file squid-3.X.STABLEXX.tar.gz
tar xvzf squid-3.X.STABLEXX.tar.gz
change dirs
./configure --enable-icap-client
make
make install

Comment out the allow from local network:
sed -rie '/http_access allow localnet/ s/^/#/' /usr/local/squid/etc/squid.conf
Start up squid in interactive mode:
/usr/local/squid/sbin/squid -N -d1

Add the squid IP and port (3128) to the browser proxy configuration.
Try surfing to a website; you will see:


ERROR

The requested URL could not be retrieved

--------------------------------------------------------------------------------

The following error was encountered while trying to retrieve the URL: http://www.awgtek.com/


Access Denied.

Access control configuration prevents your request from being allowed at this time. Please contact your service provider if you feel this is incorrect.


Do:
Ctl-C
sed -rie '/http_access allow localnet/ s/^#//' squid.conf

/usr/local/squid/sbin/squid -N -d1

Browse to a site.

To check the access
do: tail /usr/local/squid/var/logs/access.log

DLP "Forwarding agent unavailable" error

If this error is appearing in the /var/log/mail.log of your upstream MTA, when initially setting up a DLP Network Prevent for Email in Forwarding mode, try recycling the new smtp prevent server, before trying anything else.

e.g. errors like (host #.#.#.51[#.#.#.51]
refused to talk to me: 421 4.4.1 Fatal: Forwarding agent unavailable. Closing connection.

Recycling the DLP email prevent server got the following architecture to finally start working:

Outlook --> (port 25) Ubuntu Postfix MTA --> (port 10025) DLP SMTP Prevent --> (port 25) Ubuntu Postfix MTA --> (port 587, TLS) GMail

The upstream MTA's main.cf was configured with smtp_use_tls=no, smtp_sasl_auth_enable=no; and with relayhost=(DLP_IP):10025

The DLP server's advanced configuration had RequestProcessor.MTAResubmitPort=25 (changed from default 10026). Next Hop Configuration was set to Forward with Disable MX lookup, and downstream local MTA's IP set in Hostnames.

Setup Postfix Ubuntu Gmail SMTP Relay

Refer to http://ubuntu-tutorials.com/2008/11/11/relaying-postfix-smtp-via-smtpgmailcom/

On clean Ubuntu (natty) 11.04 (with ssh and aptitude installed, and as root)

aptitude install postfix libsasl2 ca-certificate libsasl2-modules

During Postfix Configuration Package configuration wizard
On first page, choose "Internet with smarthost"
For SMTP relay host enter: [smtp.gmail.com]:587
(brackets are to avoid mx lookups)

paste in /etc/postfix/main.cf:

smtp_sasl_auth_enable = yes
smtp_sasl_password_maps = hash:/etc/postfix/sasl_passwd
smtp_sasl_security_options = noanonymous
smtp_tls_CAfile = /etc/ssl/certs/Equifax_Secure_CA.pem
smtp_use_tls = yes

while still in main.cf, add your local IP network to the mynetworks variable

create /etc/postfix/sasl_passwd with the following contents:

[smtp.gmail.com]:587 user.name@gmail.com:password

do:

chmod 400 /etc/postfix/sasl_passwd
postmap /etc/postfix/sasl_passwd

/etc/init.d/postfix reload

Now test:

telnet postfixserverip 25
ehlo anything.com
mail from:me@whatever.com
rcpt to:someone@somewhere.com notify=success,failure
data
subject:Test from awgtek

This is a test message

.
quit

check someone@somewhere.com's email to see if message arrived
check for errors in /var/log/mail.log