BOTES Prerequisites
This page describe which packages are needed to launch scripts and commands used to sanitize the different JSON files from Splunk and generate all Elastic related files.
Install Python 3.7.X
sudo yum update
sudo yum install gcc openssl-devel bzip2-devel libffi-devel python-devel wget
cd /usr/src/
wget https://www.python.org/ftp/python/3.7.4/Python-3.7.4.tgz
tar xzf Python-3.7.4.tgz
cd Python-3.7.4
./configure --enable-optimizations
make altinstall
# Check Python version to validate installation
python3.7 --versionInstall jq
jq allow easy and fast JSON processing. (More information here : https://stedolan.github.io/jq/)
sudo yum install epel-release
sudo yum install jqInstall Go
cd /usr/local
wget https://dl.google.com/go/go1.12.7.linux-amd64.tar.gz
tar -zxf go1.12.7.linux-amd64.tar.gz
export GOROOT=/usr/local/go
export PATH=$GOPATH/bin:$GOROOT/bin:$PATH
# Check Go version to validate installation
go versionInstall Python packages
# BOTES Script
python3.7 -m pip install --upgrade pip
python3.7 -m pip install pyyaml
# ECS Script
sudo yum install epel-release
sudo yum install python-pip
python -m pip install --upgrade pip
python -m pip install pyyaml
python -m pip install autopep8==1.4.3
python -m pip install yamllint==1.13.0Install Git
sudo yum install gitLogstash configuration
Download Logstash configuration files from the following links and copy them in /etc/logstash/conf.d/ folder.
Logstash configuration file
Download Link
fgt_utm
nessus-scan
winevent-application
winevent-security
winevent-system
xmlwineventlog-sysmon
Elasticsearch Index Template
Elasticsearch Index Mapping for BOTES Dataset, already modified and ready to be used can be downloaded here : https://botes.s3-us-west-1.amazonaws.com/botes-index-mapping/template.json
Upload Elasticsearch Index Mapping template with the following command and your environment :
curl -XPUT 'http://"$ES_Instance:9200"/_template/botes' \
-H 'Content-Type: application/json' \
-d@"$path_to_template"/template.jsonDate format
Because some dates are in specific format, Elasticsearch Index Template need to be modified. Following formats have been added in Date fields :
{
// Modified index pattern to match with BOTES index name
"index_patterns": [
"botes*"
],
...
"botes": {
"file": {
"pctime": {
"type": "date",
"format": "yyyy-MM-dd HH:mm:ss.SSS"
"request": {
"date": {
"type": "date",
"format": "EEE, dd MMM yyyy HH:mm:ss zzz"
"request": {
"date": {
"type": "date",
"format": "EEE, dd MMM yyyy HH:mm:ss zzz"
"date": {
"type": "date",
"format": "EEE, dd MMM yyyy HH:mm:ss zzz"
...
"event": {
"created": {
"type": "date",
"format": "yyyy-MM-dd HH:mm:ss.SSS z||strict_date_optional_time||epoch_millis"
...
"file": {
"ctime": {
"type": "date",
"format": "yyyy-MM-dd HH:mm:ss.SSS||strict_date_optional_time||epoch_millis"
...
"nessus": {
"scan": {
"end": {
"type": "date",
"format": "E MMM d HH:mm:ss yyyy"
...
"tls": {
"certificate": {
"validity": {
"end": {
"type": "date",
"format": "MMM dd HH:mm:ss yyyy zzz||MMM d HH:mm:ss yyyy zzz"
},
"start": {
"type": "date",
"format": "MMM dd HH:mm:ss yyyy zzz||MMM d HH:mm:ss yyyy zzz"
}
...
"winlog": {
"process": {
"time_utc": {
"type": "date",
"format": "yyyy-MM-dd HH:mm:ss.SSS"
... Last updated
Was this helpful?