BOTES Prerequisites

This page describe which packages are needed to launch scripts and commands used to sanitize the different JSON files from Splunk and generate all Elastic related files.

Scripts have been tested on the latest CentOS 7 minimal image version (currently 7.7-1908).

Install Python 3.7.X

sudo yum update
sudo yum install gcc openssl-devel bzip2-devel libffi-devel python-devel wget
cd /usr/src/
wget https://www.python.org/ftp/python/3.7.4/Python-3.7.4.tgz
tar xzf Python-3.7.4.tgz
cd Python-3.7.4
./configure --enable-optimizations
make altinstall

# Check Python version to validate installation
python3.7 --version

Install jq

jq allow easy and fast JSON processing. (More information here : https://stedolan.github.io/jq/)

sudo yum install epel-release
sudo yum install jq

Install Go

cd /usr/local
wget https://dl.google.com/go/go1.12.7.linux-amd64.tar.gz
tar -zxf go1.12.7.linux-amd64.tar.gz
export GOROOT=/usr/local/go
export PATH=$GOPATH/bin:$GOROOT/bin:$PATH

# Check Go version to validate installation
go version

Install Python packages

# BOTES Script
python3.7 -m pip install --upgrade pip
python3.7 -m pip install pyyaml

# ECS Script
sudo yum install epel-release
sudo yum install python-pip
python -m pip install --upgrade pip
python -m pip install pyyaml
python -m pip install autopep8==1.4.3
python -m pip install yamllint==1.13.0

Install Git

sudo yum install git

Logstash configuration

Download Logstash configuration files from the following links and copy them in /etc/logstash/conf.d/ folder.

Change file paths in configuration according to your environement.

An output configuration file is also available. The specified Index name match with name or pattern specified in the Elasticsearch Index Mapping template file available here : https://botes.s3-us-west-1.amazonaws.com/botes-index-mapping/template.json.

If you want to change Index name in Output file, don't forget to change Index name or pattern in Elasticsearch Index Mapping template file.

Elasticsearch Index Template

Elasticsearch Index Mapping for BOTES Dataset, already modified and ready to be used can be downloaded here : https://botes.s3-us-west-1.amazonaws.com/botes-index-mapping/template.json

Upload Elasticsearch Index Mapping template with the following command and your environment :

curl -XPUT 'http://"$ES_Instance:9200"/_template/botes' \
-H 'Content-Type: application/json' \
-d@"$path_to_template"/template.json

Date format

Because some dates are in specific format, Elasticsearch Index Template need to be modified. Following formats have been added in Date fields :

{
// Modified index pattern to match with BOTES index name
  "index_patterns": [
    "botes*"
  ], 
  
  ...
  
        "botes": {
              "file": {
                    "pctime": {
                          "type": "date",
                          "format": "yyyy-MM-dd HH:mm:ss.SSS"
               "request": {
                    "date": {
                          "type": "date",
                          "format": "EEE, dd MMM yyyy HH:mm:ss zzz"
               "request": {
                    "date": {
                          "type": "date",
                          "format": "EEE, dd MMM yyyy HH:mm:ss zzz"
                    "date": {
                          "type": "date",
                          "format": "EEE, dd MMM yyyy HH:mm:ss zzz" 
                                                    
  ...
  
        "event": {
              "created": {
                      "type": "date",
                      "format": "yyyy-MM-dd HH:mm:ss.SSS z||strict_date_optional_time||epoch_millis"
                      
  ...    
                   
        "file": {   
             "ctime": {
                   "type": "date",
	               "format": "yyyy-MM-dd HH:mm:ss.SSS||strict_date_optional_time||epoch_millis" 
	
  ...
  
        "nessus": {
               "scan": {
                    "end": {
                           "type": "date",
                            "format": "E MMM d HH:mm:ss yyyy"
                      
  ...
  
        "tls": { 
            "certificate": { 
                      "validity": {
                               "end": {
                                  "type": "date",
                                  "format": "MMM dd HH:mm:ss yyyy zzz||MMM  d HH:mm:ss yyyy zzz"
                               }, 
                               "start": {
                                  "type": "date",
                                  "format": "MMM dd HH:mm:ss yyyy zzz||MMM  d HH:mm:ss yyyy zzz"
                               }

  ...
  
        "winlog": {
               "process": { 
                       "time_utc": {
                                  "type": "date",
                                  "format": "yyyy-MM-dd HH:mm:ss.SSS"
                                  
  ...
PreviousBOTES : Boss of the Elastic SOCNextECS & BOTES

Last updated