Multi-threading SSIM Event Search

To ensure your search is multithreaded (and thus faster) as of 4.7.4.x you must break your search into separate archives. Note, that SSIM is agnostic when it comes to archives: an "archive" is simply a directory with .edx, .key, .ndx, .sar, .tdx, .vdx files. However, an official archive will have an indexed_event_fields.txt file in it.

One quick way to do this is unzip the Archive_CLI tool and run the search from the command line, inputing archives separated by a comma, e.g.:date;java -server -Xmx512m -verbose:gc -XX:+UseConcMarkSweepGC -XX:+PrintGCTimeStamps -XX:+PrintGCDetails -jar simsar.jar -a /eventarchive/pixarchive1/,/eventarchive/pixarchive2/,/eventarchive/default,/eventarchive/ssimlogs -q "destination_ip = \"192.168.1.1\" & (destination_port = 80 | destination_port = 8080)" -c -S "," -r events.csv -V;date
Note the extra flags are for monitoring garbage collection. Check the threads in top to verify parallelism.

Thread Dump of SSIM Service

To get a global thread dump of a SSIM service, find the log_file parameter for the service in svclauncher.cfg to determine the log file that will contain the thread dump. Then issue a status to get the PID of the service in question, e.g. the rulesvc. Then while the SSIM is performing some interesting activity, type kill -SIGQUIT <PID>. Finally, open the log file to examine the thread dump and stack traces.

Quick FTP Setup

This is a quick and dirty setup guide to enabling ftp uploads for your server to take advantage of FTP file transfer speeds which are much faster than SFTP/SCP. It only applies to servers in a secured lab environment, and should never be performed on production servers or public-facing servers.

- check whether vsftpd is installed:

rpm -q vsftpd

yum install vsftpd

- configure it:

chmod 777 /var/ftp/pub

vi /etc/vsftpd/vsftpd.conf

change anon_upload_enable and anon_mkdir_write_enable to YES; add line with: ftp_username=nobody (so that will prompt for user in RedHat linux)

service vsftpd start

ftp login using user anonymous with blank password



Note:

you may need to disable iptables and selinux.

refs:

http://nixcraft.com/getting-started-tutorials/725-secure-vsftpd-ftp-permissions-anonymous-user-uploads.html

http://bashcurescancer.com/installing-ftp-vsftpd-on-rhel-50-centos.html

SSIM AD Integration

SSIM can integrate with multiple AD domains (do not have to be in same forest). The documentation in the Admin guide and the official KB is sparse on how to do that though currently.

Suffice it to say I was able to connect to a Windows 2003 Enterprise SP2 (not R2), as well as a Win 2003 R2 domains, in separate forests.

The trick is to make sure those domain controllers have been assigned a server certificate (not the root cert). Verify that certificate has "Server Authentication" in the Intended Purposes column of the Certificates snapin, where it should be under Personal/Certificates. When you open the certificate in the General tab it should read "You have a private key that corresponds to this certificate."

Troubleshooting steps: test connectivity using LDP.exe from the support tools installation. Also use the following command to test from the SSIM itself: openssl s_client -connect [your DC FQDN]:636

The following guide was perfect. Make sure to note the part about exporting the cert to Base-64 encoded binary X.509 (.CER). http://www.tools4ever.com/documentation/user-management-resource-administrator/ldap_ad_secure.htm?content=7030

The Symantec KB: http://www.symantec.com/business/support/index?page=content&id=TECH123285

iptables port manipulation

You may find there's no /etc/sysconfig/iptables in a CentOS install with iptables disabled upon install, but you want to redirect a port, e.g. to get port 80 working in tomcat.

Do:

service iptables start
iptables -t nat -A PREROUTING -p tcp --dport 80 -j REDIRECT --to-port 8080
iptables-save > /etc/sysconfig/iptables
service iptables restart

To open port 80, add the following to the *filter section before COMMIT:

-A INPUT -p tcp -m tcp --dport 80 -j ACCEPT

then service iptables restart

Numeric IP - SSIM Representation

IPs in the SSIM as of 4.7.3 are represented in both positive and negative form. To convert from an IP to this form, use the following, replacing [ipvariable] with your dotted quad IP:

SELECT CASE WHEN (IPNumber - 2147483648) > 0 THEN IPNumber -(2*2147483648) ELSE IPNumber END AS IPNUMBER2 FROM ( SELECT 16777216 *
CAST(LEFT(IPNumber1, LOCATE('.', IPNumber1, 1)-1) AS BIGINT) + 65536 * CAST(SUBSTR(IPNumber1, LOCATE('.', IPNumber1, 1) + 1, LOCATE('.',
IPNumber1,LOCATE('.', IPNumber1, 1) +1) - LOCATE('.', IPNumber1, 1) - 1) AS BIGINT) + 256 * CAST(SUBSTR(IPNumber1, LOCATE('.',
IPNumber1,LOCATE('.', IPNumber1, 1) +1) + 1, LOCATE('.', IPNumber1, LOCATE('.', IPNumber1,LOCATE('.', IPNumber1, 1) +1) +1) - LOCATE('.',
IPNumber1,LOCATE('.', IPNumber1, 1) +1) - 1) AS BIGINT) + CAST(RIGHT(IPNumber1, LENGTH(IPNumber1) - LOCATE('.', IPNumber1,
LOCATE('.', IPNumber1,LOCATE('.', IPNumber1, 1) +1) +1)) AS BIGINT) AS IPNumber FROM (SELECT '[ipvariable]' AS IPNumber1 FROM
SYSIBM.SYSDUMMY1) AS LOCXTMP0) AS TEMP2

ref: http://stackoverflow.com/questions/6695428/how-to-split-an-ip-address-string-in-db2-sql-for-converting-to-ip-number

SSIM Red Queues

When you see all three left red bars completely red, and the EPS counts are 0, here are some troubleshooting tips:

- Check the Sesa Agent if it's a collector server
- SSH into the collector server in question
- "su -" (change to root)
- tail the collector log which produces the most EPS or highest spikes on this collector
e.g. "tail -200 /opt/Symantec/sesa/Agent/logs/stonegate.log"
- If you see WARN entries containing "the agent queue is full" you may need to adjust the agent queue configurations: SSIM console: System > Product Configurations > SES > STATE > SSIM Agent and Manager > Agent Configurations > Standard (or the configuration applied to this server) > Logging
- adjust the following properties as needed, e.g. to increase the queue size, flush size, decrease the queue flush time (so that events are cleared more frequently).
- Save the configuration (the agent should pick up the settings automatically)

- Check the Recv-Q and Send-Q of Netstat
- on both the archiver and the correlator do "netstat -anp | grep 100"
- look for signs of downstream bottlenecks
- Examples
- if archiver is queued up sending to correlator:10010, check the correlator queues. E.g. if the correlator is queued up sending to 127.0.0.1:10080, there may be a problem with the asset service. Check assetsvc.log for errors. Try restarting the assetsvc service, and monitor the netstat -anp | grep 100 for movement in the queues.
As db2admin, cd ~/sqllib/bin
db2 connect to SESA
db2 list applications show detail
Check the SSIM-ASSET application for signs that it is currently Executing. Let it run for a while to see if it finishes.
- on correlator, check "netstat -anp | grep 555" for large number of FIN_WAIT2
- on the correlator check the icesvc.log, assetsvc.log, and rulesvc.log for recent errors (see /opt/Symantec/simserver/logs)
- Disable any rules temporarily that contain references to lookup tables to determine whether any of these rules are causing the backlog.
- Check swap space usage on Correlator. If there is any Swap used, reboot the server.
- Check SSIM Event logs for warnings or errors; query all recent logs where product = Symantec Security Information Manager or product = SSIM System
- Do a query on Event Type ID = Conclusion Updated. Verify that not more than a few events a minute are being created
- Check top on both archiver and correlator. Look for processes whose VIRT is close to the same size as RES. This process may be overworked. The memory allocation can be increased in the /opt/Symantec/simserver/svclauncher.cfg config.
- Investigate whether simserver (correlator service) is overloaded.
- ascertain the pid of the simserver in top
- do lsof | grep | awk '{print $7,$1,$2,$3,$4,$5,$6,$8,$9}' | sort -n
- look for any unusually large files in output. e.g. .que files larger than 10MB might indicate an overworked correlator.
- Do an advanced SQL query in the UI to determine whether there are any unusually large number of incidents for particular rule: SELECT count(incident_code),incident_code FROM SYMCMGMT.SYMC_IMR_INCIDENT_LIST_VIEW group by incident_code
order by count(incident_code) desc
or: SELECT desc_id, sum(event_count) as sumevtcnt FROM symc_sim_conclusion
where modified_time >= (current timestamp - 1 DAY) group by desc_id
order by sumevtcnt desc
- Disable any rules with more than 50,000 events in one day. For example, a system rule that can cause issues if not adapted is Internal Port Sweep. Simserver process memory will be exhausted typically after 24 hours, and Queues will increase steadily in /opt/Symantec/simserver/queues/ice/input. IIRC the max is three 64MB files here. If there are more than one file, the icesvc is backed up due to the amount of event tracking updates waiting to be pushed to the database. Verify that SSIM-ICE is "UOW Executing" in db2 list applications show detail.
- Also check Events > System Queries > SSIM > SSIM System > Count of Conclusions by Rule Name
- Look for FIN_WAITs associated with large queues in a frozen state in netstat for the simserver process (get pid from `status`). E.g.:
tcp 80384 0 ::ffff:127.0.0.1:10010 ::ffff:127.0.0.1:36446 ESTABLISHED 6131/java
tcp 0 145961 ::ffff:127.0.0.1:36446 ::ffff:127.0.0.1:10010 FIN_WAIT1 -
This could indicate a memory resource problem in the simserver service or another service. Check the target port, and cross reference the admin guide with the service in question. e.g. 10010 is simserver, 55562 is icesvc, etc.
If it's really in frozen state, the process may have crashed (i.e. you see 0 EPS). If the Recv-Q/Send-Q varies and EPS varies, then this could be indicative of High IO Waits. Check the IOWait in the Statistics tile of the GUI. Also check vmstat 1. Sustained IOWaits of more than 2% could indicate the drive is not fast enough.