ECS & BOTES

This page describe how to use ECS and BOTES scripts. Used input files and generated output files are also described.

Scripts usage

Now, let's put together ECS and BOTES scripts. These scripts will generate ECS Fieldset Schemas, then Elasticsearch Mapping template and Logstash filter configuration (Used to normalize fields name to ECS format). They will also generate documentation for all fields, in different formats.

cd
git clone https://github.com/elastic/ecs.git
cd ecs/
git clone https://github.com/NybbleHub/BOTES.git

BOTES script will create ECS Fieldset schema and Logstash filter configuration from CSV files containing fields information and then call ECS script. To launch the BOTES script, use the following commands :

cd ecs/
python3.7 BOTES/Script-BOTES.py

Input files

Summary CSV file

This file contains unique original fields from all BOTS filtered datasets (For more information on sanitization process please see Data Sanitization section).

BOTES Python script generate ECS Fieldset Schemas files from this file by finding fieldsets not existing in current ECS fieldsets list.

This file is under : ecs/BOTES/botes-csv-summary/

Map CSV files

Each of those files contain unique original fields from corresponding filtered datasets (stream-http.csv contains fields from Stream-HTTP.json, etc.).

Refer to BOTES Fields section to separately download Map CSV files (also called details CSV files).

BOTES Python script generate Logstash filter configuration files from these files by taking matched original and ECS field name for each lines. Fields name are then formatted for Logstash mutate filter plugin (More information about Logstash mutate rename here : Mutate Filter Plugin).

These files are placed under : ecs/BOTES/botes-csv-map/

Output files

ECS Fieldset Schema files

YAML files generated by BOTES Python script and then used by ECS Python script to generate Elasticsearch Index Template and Documentation files.

After BOTES script execution, these files are placed under : ecs/schemas-botes/

Elasticsearch Index template file

Index template is used by Elasticsearch to define mapping properties when new indices are created (More information on Index Template and Mapping).

To load the generated Elasticsearch Index template file, use the following command :

curl -XPUT 'http://$ES_Instance:9200/_template/botes' \
-H 'Content-Type: application/json' \
-d@ecs/generated/elasticsearch/7/template.json

Or if you have self-signed certificate and authentication :

curl -XPUT 'https://$ES_Instance:9200/_template/botes' \
-k --user $username:$password \
-H 'Content-Type: application/json' \
-d@ecs/generated/elasticsearch/7/template.json

Don't forget to modify the "index_patterns" value to match your index/indices name. The default value is "ecs-*".

After BOTES/ECS script execution, template files are placed under :

For ES6 : ecs/generated/elasticsearch/6/

For ES7 : ecs/generated/elasticsearch/7/

Logstash filter configuration files

Original to ECS fields name normalization is done using Logstash Filter mutate plugin.

Example :

filter {
    mutate {
        rename => ["dest_ip", "destination.ip" ]
    }
}

To keep fields easy to manage, one Logstash configuration filter file per sourcetype is created. Content of each configuration file has to be copied in new or already existing Logstash configuration files.

After BOTES script execution, those files are placed under : ecs/botes-logstash-conf/

Documentation files

ECS Python script generate multiple documentations files, containing summary of all fieldsets in different formats.

After BOTES/ECS script execution, those files are under : ecs/docs/