---
description: >-
  This page describe which packages are needed to launch scripts and commands
  used to sanitize the different JSON files from Splunk and generate all Elastic
  related files.
---

# BOTES Prerequisites

{% hint style="info" %}
Scripts have been tested on the latest CentOS 7 minimal image version (currently 7.7-1908).&#x20;
{% endhint %}

## Install Python 3.7.X

{% tabs %}
{% tab title="CentOS" %}
```
sudo yum update
sudo yum install gcc openssl-devel bzip2-devel libffi-devel python-devel wget
cd /usr/src/
wget https://www.python.org/ftp/python/3.7.4/Python-3.7.4.tgz
tar xzf Python-3.7.4.tgz
cd Python-3.7.4
./configure --enable-optimizations
make altinstall

# Check Python version to validate installation
python3.7 --version
```
{% endtab %}

{% tab title="Other" %}

{% endtab %}
{% endtabs %}

## Install jq

jq allow easy and fast JSON processing. (More information here : [https://stedolan.github.io/jq/](https://stedolan.github.io/jq/))

{% tabs %}
{% tab title="CentOS" %}
```
sudo yum install epel-release
sudo yum install jq
```
{% endtab %}

{% tab title="Other" %}

{% endtab %}
{% endtabs %}

## Install Go

{% tabs %}
{% tab title="CentOS" %}
```
cd /usr/local
wget https://dl.google.com/go/go1.12.7.linux-amd64.tar.gz
tar -zxf go1.12.7.linux-amd64.tar.gz
export GOROOT=/usr/local/go
export PATH=$GOPATH/bin:$GOROOT/bin:$PATH

# Check Go version to validate installation
go version
```
{% endtab %}

{% tab title="Other" %}

{% endtab %}
{% endtabs %}

## Install Python packages

{% tabs %}
{% tab title="CentOS" %}
```
# BOTES Script
python3.7 -m pip install --upgrade pip
python3.7 -m pip install pyyaml

# ECS Script
sudo yum install epel-release
sudo yum install python-pip
python -m pip install --upgrade pip
python -m pip install pyyaml
python -m pip install autopep8==1.4.3
python -m pip install yamllint==1.13.0
```
{% endtab %}

{% tab title="Other" %}

{% endtab %}
{% endtabs %}

## Install Git

{% tabs %}
{% tab title="CentOS" %}
```
sudo yum install git
```
{% endtab %}

{% tab title="Other" %}

{% endtab %}
{% endtabs %}

## Logstash configuration

Download Logstash configuration files from the following links and copy them in `/etc/logstash/conf.d/` folder.

{% hint style="info" %}
Change file paths in configuration according to your environement.&#x20;
{% endhint %}

{% hint style="info" %}
An output configuration file is also available. The specified Index name match with name or pattern specified in the Elasticsearch Index Mapping template file available here : [https://botes.s3-us-west-1.amazonaws.com/botes-index-mapping/template.json](https://botes.s3-us-west-1.amazonaws.com/botes-index-mapping/template.json).

If you want to change Index name in Output file, don't forget to change Index name or pattern in   Elasticsearch Index Mapping template file.
{% endhint %}

| Logstash configuration file | Download Link                                                                                                                                                                                                                          |
| --------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| fgt\_event                  | [https://botes.s3-us-west-1.amazonaws.com/botes-logstash-configuration/full-config/input-fgt\_event.conf](https://botes.s3-us-west-1.amazonaws.com/botes-logstash-configuration/full-config/input-fgt_event.conf)                      |
| fgt\_traffic                | [https://botes.s3-us-west-1.amazonaws.com/botes-logstash-configuration/full-config/input-fgt\_traffic.conf](https://botes.s3-us-west-1.amazonaws.com/botes-logstash-configuration/full-config/input-fgt_traffic.conf)                  |
| fgt\_utm                    | [https://botes.s3-us-west-1.amazonaws.com/botes-logstash-configuration/full-config/input-fgt\_utm.conf](https://botes.s3-us-west-1.amazonaws.com/botes-logstash-configuration/full-config/input-fgt_utm.conf)                          |
| iis                         | [https://botes.s3-us-west-1.amazonaws.com/botes-logstash-configuration/full-config/input-iis.conf](https://botes.s3-us-west-1.amazonaws.com/botes-logstash-configuration/full-config/input-iis.conf)                                   |
| nessus-scan                 | [https://botes.s3-us-west-1.amazonaws.com/botes-logstash-configuration/full-config/input-nessus.conf](https://botes.s3-us-west-1.amazonaws.com/botes-logstash-configuration/full-config/input-nessus.conf)                             |
| stream-dhcp                 | [https://botes.s3-us-west-1.amazonaws.com/botes-logstash-configuration/full-config/input-stream-dhcp.conf](https://botes.s3-us-west-1.amazonaws.com/botes-logstash-configuration/full-config/input-stream-dhcp.conf)                   |
| stream-dns                  | [https://botes.s3-us-west-1.amazonaws.com/botes-logstash-configuration/full-config/input-stream-dns.conf](https://botes.s3-us-west-1.amazonaws.com/botes-logstash-configuration/full-config/input-stream-dns.conf)                     |
| stream-http                 | [https://botes.s3-us-west-1.amazonaws.com/botes-logstash-configuration/full-config/input-stream-http.conf](https://botes.s3-us-west-1.amazonaws.com/botes-logstash-configuration/full-config/input-stream-http.conf)                   |
| stream-icmp                 | [https://botes.s3-us-west-1.amazonaws.com/botes-logstash-configuration/full-config/input-stream-icmp.conf](https://botes.s3-us-west-1.amazonaws.com/botes-logstash-configuration/full-config/input-stream-icmp.conf)                   |
| stream-ip                   | [https://botes.s3-us-west-1.amazonaws.com/botes-logstash-configuration/full-config/input-stream-ip.conf](https://botes.s3-us-west-1.amazonaws.com/botes-logstash-configuration/full-config/input-stream-ip.conf)                       |
| stream-ldap                 | [https://botes.s3-us-west-1.amazonaws.com/botes-logstash-configuration/full-config/input-stream-ldap.conf](https://botes.s3-us-west-1.amazonaws.com/botes-logstash-configuration/full-config/input-stream-ldap.conf)                   |
| stream-mapi                 | [https://botes.s3-us-west-1.amazonaws.com/botes-logstash-configuration/full-config/input-stream-mapi.conf](https://botes.s3-us-west-1.amazonaws.com/botes-logstash-configuration/full-config/input-stream-mapi.conf)                   |
| stream-sip                  | [https://botes.s3-us-west-1.amazonaws.com/botes-logstash-configuration/full-config/input-stream-sip.conf](https://botes.s3-us-west-1.amazonaws.com/botes-logstash-configuration/full-config/input-stream-sip.conf)                     |
| stream-smb                  | [https://botes.s3-us-west-1.amazonaws.com/botes-logstash-configuration/full-config/input-stream-smb.conf](https://botes.s3-us-west-1.amazonaws.com/botes-logstash-configuration/full-config/input-stream-smb.conf)                     |
| stream-snmp                 | [https://botes.s3-us-west-1.amazonaws.com/botes-logstash-configuration/full-config/input-stream-snmp.conf](https://botes.s3-us-west-1.amazonaws.com/botes-logstash-configuration/full-config/input-stream-snmp.conf)                   |
| stream-tcp                  | [https://botes.s3-us-west-1.amazonaws.com/botes-logstash-configuration/full-config/input-stream-tcp.conf](https://botes.s3-us-west-1.amazonaws.com/botes-logstash-configuration/full-config/input-stream-tcp.conf)                     |
| suricata                    | [https://botes.s3-us-west-1.amazonaws.com/botes-logstash-configuration/full-config/input-suricata.conf](https://botes.s3-us-west-1.amazonaws.com/botes-logstash-configuration/full-config/input-suricata.conf)                         |
| winevent-application        | [https://botes.s3-us-west-1.amazonaws.com/botes-logstash-configuration/full-config/input-winevent-application.conf](https://botes.s3-us-west-1.amazonaws.com/botes-logstash-configuration/full-config/input-winevent-application.conf) |
| winevent-security           | [https://botes.s3-us-west-1.amazonaws.com/botes-logstash-configuration/full-config/input-winevent-security.conf](https://botes.s3-us-west-1.amazonaws.com/botes-logstash-configuration/full-config/input-winevent-security.conf)       |
| winevent-system             | [https://botes.s3-us-west-1.amazonaws.com/botes-logstash-configuration/full-config/input-winevent-system.conf](https://botes.s3-us-west-1.amazonaws.com/botes-logstash-configuration/full-config/input-winevent-system.conf)           |
| winregistry                 | [https://botes.s3-us-west-1.amazonaws.com/botes-logstash-configuration/full-config/input-winregistry.conf](https://botes.s3-us-west-1.amazonaws.com/botes-logstash-configuration/full-config/input-winregistry.conf)                   |
| xmlwineventlog-sysmon       | [https://botes.s3-us-west-1.amazonaws.com/botes-logstash-configuration/full-config/input-winevent-sysmon.conf](https://botes.s3-us-west-1.amazonaws.com/botes-logstash-configuration/full-config/input-winevent-sysmon.conf)           |
| elasticsearch output        | [https://botes.s3-us-west-1.amazonaws.com/botes-logstash-configuration/full-config/output.conf](https://botes.s3-us-west-1.amazonaws.com/botes-logstash-configuration/full-config/output.conf)                                         |

## Elasticsearch Index Template

Elasticsearch Index Mapping for BOTES Dataset, already modified and ready to be used can be downloaded here : [https://botes.s3-us-west-1.amazonaws.com/botes-index-mapping/template.json](https://botes.s3-us-west-1.amazonaws.com/botes-index-mapping/template.json)

Upload Elasticsearch Index Mapping template with the following command and your environment :

```
curl -XPUT 'http://"$ES_Instance:9200"/_template/botes' \
-H 'Content-Type: application/json' \
-d@"$path_to_template"/template.json
```

### Date format

Because some dates are in specific format, Elasticsearch Index Template need to be modified. Following formats have been added in Date fields :

```
{
// Modified index pattern to match with BOTES index name
  "index_patterns": [
    "botes*"
  ], 
  
  ...
  
        "botes": {
              "file": {
                    "pctime": {
                          "type": "date",
                          "format": "yyyy-MM-dd HH:mm:ss.SSS"
               "request": {
                    "date": {
                          "type": "date",
                          "format": "EEE, dd MMM yyyy HH:mm:ss zzz"
               "request": {
                    "date": {
                          "type": "date",
                          "format": "EEE, dd MMM yyyy HH:mm:ss zzz"
                    "date": {
                          "type": "date",
                          "format": "EEE, dd MMM yyyy HH:mm:ss zzz" 
                                                    
  ...
  
        "event": {
              "created": {
                      "type": "date",
                      "format": "yyyy-MM-dd HH:mm:ss.SSS z||strict_date_optional_time||epoch_millis"
                      
  ...    
                   
        "file": {   
             "ctime": {
                   "type": "date",
	               "format": "yyyy-MM-dd HH:mm:ss.SSS||strict_date_optional_time||epoch_millis" 
	
  ...
  
        "nessus": {
               "scan": {
                    "end": {
                           "type": "date",
                            "format": "E MMM d HH:mm:ss yyyy"
                      
  ...
  
        "tls": { 
            "certificate": { 
                      "validity": {
                               "end": {
                                  "type": "date",
                                  "format": "MMM dd HH:mm:ss yyyy zzz||MMM  d HH:mm:ss yyyy zzz"
                               }, 
                               "start": {
                                  "type": "date",
                                  "format": "MMM dd HH:mm:ss yyyy zzz||MMM  d HH:mm:ss yyyy zzz"
                               }

  ...
  
        "winlog": {
               "process": { 
                       "time_utc": {
                                  "type": "date",
                                  "format": "yyyy-MM-dd HH:mm:ss.SSS"
                                  
  ...                                                                                                                                          
```
