BOTES Prerequisites

This page describe which packages are needed to launch scripts and commands used to sanitize the different JSON files from Splunk and generate all Elastic related files.

Scripts have been tested on the latest CentOS 7 minimal image version (currently 7.7-1908).

Install Python 3.7.X

CentOS

sudo yum update
sudo yum install gcc openssl-devel bzip2-devel libffi-devel python-devel wget
cd /usr/src/
wget https://www.python.org/ftp/python/3.7.4/Python-3.7.4.tgz
tar xzf Python-3.7.4.tgz
cd Python-3.7.4
./configure --enable-optimizations
make altinstall

# Check Python version to validate installation
python3.7 --version

Other

Install jq

jq allow easy and fast JSON processing. (More information here : https://stedolan.github.io/jq/)

CentOS

sudo yum install epel-release
sudo yum install jq

Other

Install Go

CentOS

cd /usr/local
wget https://dl.google.com/go/go1.12.7.linux-amd64.tar.gz
tar -zxf go1.12.7.linux-amd64.tar.gz
export GOROOT=/usr/local/go
export PATH=$GOPATH/bin:$GOROOT/bin:$PATH

# Check Go version to validate installation
go version

Other

Install Python packages

CentOS

# BOTES Script
python3.7 -m pip install --upgrade pip
python3.7 -m pip install pyyaml

# ECS Script
sudo yum install epel-release
sudo yum install python-pip
python -m pip install --upgrade pip
python -m pip install pyyaml
python -m pip install autopep8==1.4.3
python -m pip install yamllint==1.13.0

Other

Install Git

CentOS

sudo yum install git

Other

Logstash configuration

Download Logstash configuration files from the following links and copy them in /etc/logstash/conf.d/ folder.

Change file paths in configuration according to your environement.

An output configuration file is also available. The specified Index name match with name or pattern specified in the Elasticsearch Index Mapping template file available here : https://botes.s3-us-west-1.amazonaws.com/botes-index-mapping/template.json.

If you want to change Index name in Output file, don't forget to change Index name or pattern in Elasticsearch Index Mapping template file.

Logstash configuration file	Download Link
fgt_event	https://botes.s3-us-west-1.amazonaws.com/botes-logstash-configuration/full-config/input-fgt_event.conf
fgt_traffic	https://botes.s3-us-west-1.amazonaws.com/botes-logstash-configuration/full-config/input-fgt_traffic.conf
fgt_utm	https://botes.s3-us-west-1.amazonaws.com/botes-logstash-configuration/full-config/input-fgt_utm.conf
iis	https://botes.s3-us-west-1.amazonaws.com/botes-logstash-configuration/full-config/input-iis.conf
nessus-scan	https://botes.s3-us-west-1.amazonaws.com/botes-logstash-configuration/full-config/input-nessus.conf
stream-dhcp	https://botes.s3-us-west-1.amazonaws.com/botes-logstash-configuration/full-config/input-stream-dhcp.conf
stream-dns	https://botes.s3-us-west-1.amazonaws.com/botes-logstash-configuration/full-config/input-stream-dns.conf
stream-http	https://botes.s3-us-west-1.amazonaws.com/botes-logstash-configuration/full-config/input-stream-http.conf
stream-icmp	https://botes.s3-us-west-1.amazonaws.com/botes-logstash-configuration/full-config/input-stream-icmp.conf
stream-ip	https://botes.s3-us-west-1.amazonaws.com/botes-logstash-configuration/full-config/input-stream-ip.conf
stream-ldap	https://botes.s3-us-west-1.amazonaws.com/botes-logstash-configuration/full-config/input-stream-ldap.conf
stream-mapi	https://botes.s3-us-west-1.amazonaws.com/botes-logstash-configuration/full-config/input-stream-mapi.conf
stream-sip	https://botes.s3-us-west-1.amazonaws.com/botes-logstash-configuration/full-config/input-stream-sip.conf
stream-smb	https://botes.s3-us-west-1.amazonaws.com/botes-logstash-configuration/full-config/input-stream-smb.conf
stream-snmp	https://botes.s3-us-west-1.amazonaws.com/botes-logstash-configuration/full-config/input-stream-snmp.conf
stream-tcp	https://botes.s3-us-west-1.amazonaws.com/botes-logstash-configuration/full-config/input-stream-tcp.conf
suricata	https://botes.s3-us-west-1.amazonaws.com/botes-logstash-configuration/full-config/input-suricata.conf
winevent-application	https://botes.s3-us-west-1.amazonaws.com/botes-logstash-configuration/full-config/input-winevent-application.conf
winevent-security	https://botes.s3-us-west-1.amazonaws.com/botes-logstash-configuration/full-config/input-winevent-security.conf
winevent-system	https://botes.s3-us-west-1.amazonaws.com/botes-logstash-configuration/full-config/input-winevent-system.conf
winregistry	https://botes.s3-us-west-1.amazonaws.com/botes-logstash-configuration/full-config/input-winregistry.conf
xmlwineventlog-sysmon	https://botes.s3-us-west-1.amazonaws.com/botes-logstash-configuration/full-config/input-winevent-sysmon.conf
elasticsearch output	https://botes.s3-us-west-1.amazonaws.com/botes-logstash-configuration/full-config/output.conf

Elasticsearch Index Template

Elasticsearch Index Mapping for BOTES Dataset, already modified and ready to be used can be downloaded here : https://botes.s3-us-west-1.amazonaws.com/botes-index-mapping/template.json

Upload Elasticsearch Index Mapping template with the following command and your environment :

curl -XPUT 'http://"$ES_Instance:9200"/_template/botes' \
-H 'Content-Type: application/json' \
-d@"$path_to_template"/template.json

Date format

Because some dates are in specific format, Elasticsearch Index Template need to be modified. Following formats have been added in Date fields :

{
// Modified index pattern to match with BOTES index name
  "index_patterns": [
    "botes*"
  ], 
  
  ...
  
        "botes": {
              "file": {
                    "pctime": {
                          "type": "date",
                          "format": "yyyy-MM-dd HH:mm:ss.SSS"
               "request": {
                    "date": {
                          "type": "date",
                          "format": "EEE, dd MMM yyyy HH:mm:ss zzz"
               "request": {
                    "date": {
                          "type": "date",
                          "format": "EEE, dd MMM yyyy HH:mm:ss zzz"
                    "date": {
                          "type": "date",
                          "format": "EEE, dd MMM yyyy HH:mm:ss zzz" 
                                                    
  ...
  
        "event": {
              "created": {
                      "type": "date",
                      "format": "yyyy-MM-dd HH:mm:ss.SSS z||strict_date_optional_time||epoch_millis"
                      
  ...    
                   
        "file": {   
             "ctime": {
                   "type": "date",
	               "format": "yyyy-MM-dd HH:mm:ss.SSS||strict_date_optional_time||epoch_millis" 
	
  ...
  
        "nessus": {
               "scan": {
                    "end": {
                           "type": "date",
                            "format": "E MMM d HH:mm:ss yyyy"
                      
  ...
  
        "tls": { 
            "certificate": { 
                      "validity": {
                               "end": {
                                  "type": "date",
                                  "format": "MMM dd HH:mm:ss yyyy zzz||MMM  d HH:mm:ss yyyy zzz"
                               }, 
                               "start": {
                                  "type": "date",
                                  "format": "MMM dd HH:mm:ss yyyy zzz||MMM  d HH:mm:ss yyyy zzz"
                               }

  ...
  
        "winlog": {
               "process": { 
                       "time_utc": {
                                  "type": "date",
                                  "format": "yyyy-MM-dd HH:mm:ss.SSS"
                                  
  ...