BOTES : Boss of the Elastic SOC

This page describe why and how to make a tranformed version of BOTS Dataset for Elastic.

ECS and SIEM module release

The release of Elastic Common Schema (ECS) at the beginning of the year and the release of SIEM module with Elastic v7.2 is the perfect occasion to gain skills in SIEM Infrastrucure and Security Analytics, for free.

But without security events it's hard to fully exploit and try new Elastic features. This is where BOTS Datasets come into play. It's possible to quickly setup an Elastic Stack and ingest data as many time as desired to run different tests or to practice security analysis.

For example, this make easy to test enrichments scripts with data obtain though Shodan, Onyphe, or others Threat Intelligence sources.

For more information on ECS and SIEM module, check the following links :

Elastic SIEM module : https://www.elastic.co/blog/introducing-elastic-siem

Elastic Common Schema : https://www.elastic.co/blog/introducing-the-elastic-common-schema

From BOTS to BOTES

Before any investigation with Kibana and the SIEM module, BOTS Dataset need to be sanitized and even better, transformed following the Elastic Common Schema.

For this purpose, JSON files the from BOTS Dataset v1 will be used (BOTS Dataset v2 is only available in Splunk Pre-Index format). See Boss of the SOC (BOTS) Dataset section for BOTS v1 download links.

To move from BOTS to BOTES some actions are required :

1. For each sourcetype, list fields from JSON files.

2. Clean Splunk specific, duplicated and bad parsed fields.

3. Match original fields with ECS fields.

4. Generate ECS Schemas Fieldset, Elastic Index Mapping and Logstash configuration files from CSVs (Created during cleaning phase) with BOTES & ECS scripts.

5. Prepare Logstash and ingest the brand new clean BOTES JSON files.

This documentation contains details on how these actions were carried out and how to use BOTES Datasets (Logstash configuration files install, Elasticsearch Index mapping load, etc.).

Datasets information

DatasetSize	Size
BOTS v1 Compressed (JSON)	6.3G
BOTS v1 Uncompressed (JSON)	113G
BOTES v1 Compressed (JSON)	3G
BOTES v1 Uncompressed (JSON)	51G

BOTS/BOTES Dataset contains 31M of lines.

BOTES Download

Ready to be used BOTES Datasets (JSON formatted) can be downloaded here :

Data sourcetype	Download Link
fgt_event	https://botes.s3-us-west-1.amazonaws.com/botes-data/botesv1.fgt_event.json.gz
fgt_traffic	https://botes.s3-us-west-1.amazonaws.com/botes-data/botesv1.fgt_traffic.json.gz
fgt_utm	https://botes.s3-us-west-1.amazonaws.com/botes-data/botesv1.fgt_utm.json.gz
iis	https://botes.s3-us-west-1.amazonaws.com/botes-data/botesv1.iis.json.gz
nessus-scan	https://botes.s3-us-west-1.amazonaws.com/botes-data/botesv1.nessus-scan.json.gz
stream-dhcp	https://botes.s3-us-west-1.amazonaws.com/botes-data/botesv1.stream-dhcp.json.gz
stream-dns	https://botes.s3-us-west-1.amazonaws.com/botes-data/botesv1.stream-dns.json.gz
stream-http	https://botes.s3-us-west-1.amazonaws.com/botes-data/botesv1.stream-http.json.gz
stream-icmp	https://botes.s3-us-west-1.amazonaws.com/botes-data/botesv1.stream-icmp.json.gz
stream-ip	https://botes.s3-us-west-1.amazonaws.com/botes-data/botesv1.stream-ip.json.gz
stream-ldap	https://botes.s3-us-west-1.amazonaws.com/botes-data/botesv1.stream-ldap.json.gz
stream-mapi	https://botes.s3-us-west-1.amazonaws.com/botes-data/botesv1.stream-mapi.json.gz
stream-sip	https://botes.s3-us-west-1.amazonaws.com/botes-data/botesv1.stream-sip.json.gz
stream-smb	https://botes.s3-us-west-1.amazonaws.com/botes-data/botesv1.stream-smb.json.gz
stream-snmp	https://botes.s3-us-west-1.amazonaws.com/botes-data/botesv1.stream-snmp.json.gz
stream-tcp	https://botes.s3-us-west-1.amazonaws.com/botes-data/botesv1.stream-tcp.json.gz
suricata	https://botes.s3-us-west-1.amazonaws.com/botes-data/botesv1.suricata.json.gz
winevent-application	https://botes.s3-us-west-1.amazonaws.com/botes-data/botesv1.WinEventLog-Application.json.gz
winevent-security	https://botes.s3-us-west-1.amazonaws.com/botes-data/botesv1.WinEventLog-Security.json.gz
winevent-system	https://botes.s3-us-west-1.amazonaws.com/botes-data/botesv1.WinEventLog-System.json.gz
winregistry	https://botes.s3-us-west-1.amazonaws.com/botes-data/botesv1.winregistry.json.gz
xmlwineventlog-sysmon	https://botes.s3-us-west-1.amazonaws.com/botes-data/botesv1.XmlWinEventLog-Microsoft-Windows-Sysmon-Operational.json.gz