# BOTES : Boss of the Elastic SOC

## ECS and SIEM module release

The release of Elastic Common Schema (ECS) at the beginning of the year and the release of SIEM module with Elastic v7.2 is the perfect occasion to gain skills in SIEM Infrastrucure and Security Analytics, for free.

But without security events it's hard to fully exploit and try new Elastic features. This is where BOTS Datasets come into play. It's possible to quickly setup an Elastic Stack and ingest data as many time as desired to run different tests or to practice security analysis.

For example, this make easy to test enrichments scripts with data obtain though Shodan, Onyphe, or others Threat Intelligence sources.

For more information on ECS and SIEM module, check the following links : 

**Elastic SIEM module :** <https://www.elastic.co/blog/introducing-elastic-siem>

**Elastic Common Schema :** <https://www.elastic.co/blog/introducing-the-elastic-common-schema>

## From BOTS to BOTES

Before any investigation with Kibana and the SIEM module, BOTS Dataset need to be sanitized and even better, transformed following the Elastic Common Schema.

For this purpose, JSON files the from BOTS Dataset v1 will be used (BOTS Dataset v2 is only available in Splunk Pre-Index format). See [Boss of the SOC (BOTS) Dataset](https://botes.gitbook.io/botes-dataset/master#download) section for BOTS v1 download links.

To move from BOTS to BOTES some actions are required :

1. For each sourcetype, list fields from JSON files.
2. Clean Splunk specific, duplicated and bad parsed fields.
3. Match original fields with ECS fields.
4. Generate ECS Schemas Fieldset, Elastic Index Mapping and Logstash configuration files from CSVs (Created during cleaning phase) with BOTES & ECS scripts.
5. Prepare Logstash and ingest the brand new clean BOTES JSON files.

This documentation contains details on how these actions were carried out and how to use BOTES Datasets (Logstash configuration files install, Elasticsearch Index mapping load, etc.).

### Datasets information&#x20;

| DatasetSize                  | Size |
| ---------------------------- | ---- |
| BOTS v1 Compressed (JSON)    | 6.3G |
| BOTS v1 Uncompressed (JSON)  | 113G |
| BOTES v1 Compressed (JSON)   | 3G   |
| BOTES v1 Uncompressed (JSON) | 51G  |

BOTS/BOTES Dataset contains 31M of lines.&#x20;

### BOTES Download

Ready to be used BOTES Datasets (JSON formatted) can be downloaded here :&#x20;

| Data sourcetype       | Download Link                                                                                                             |
| --------------------- | ------------------------------------------------------------------------------------------------------------------------- |
| fgt\_event            | <https://botes.s3-us-west-1.amazonaws.com/botes-data/botesv1.fgt_event.json.gz>                                           |
| fgt\_traffic          | <https://botes.s3-us-west-1.amazonaws.com/botes-data/botesv1.fgt_traffic.json.gz>                                         |
| fgt\_utm              | <https://botes.s3-us-west-1.amazonaws.com/botes-data/botesv1.fgt_utm.json.gz>                                             |
| iis                   | <https://botes.s3-us-west-1.amazonaws.com/botes-data/botesv1.iis.json.gz>                                                 |
| nessus-scan           | <https://botes.s3-us-west-1.amazonaws.com/botes-data/botesv1.nessus-scan.json.gz>                                         |
| stream-dhcp           | <https://botes.s3-us-west-1.amazonaws.com/botes-data/botesv1.stream-dhcp.json.gz>                                         |
| stream-dns            | <https://botes.s3-us-west-1.amazonaws.com/botes-data/botesv1.stream-dns.json.gz>                                          |
| stream-http           | <https://botes.s3-us-west-1.amazonaws.com/botes-data/botesv1.stream-http.json.gz>                                         |
| stream-icmp           | <https://botes.s3-us-west-1.amazonaws.com/botes-data/botesv1.stream-icmp.json.gz>                                         |
| stream-ip             | <https://botes.s3-us-west-1.amazonaws.com/botes-data/botesv1.stream-ip.json.gz>                                           |
| stream-ldap           | <https://botes.s3-us-west-1.amazonaws.com/botes-data/botesv1.stream-ldap.json.gz>                                         |
| stream-mapi           | <https://botes.s3-us-west-1.amazonaws.com/botes-data/botesv1.stream-mapi.json.gz>                                         |
| stream-sip            | <https://botes.s3-us-west-1.amazonaws.com/botes-data/botesv1.stream-sip.json.gz>                                          |
| stream-smb            | <https://botes.s3-us-west-1.amazonaws.com/botes-data/botesv1.stream-smb.json.gz>                                          |
| stream-snmp           | <https://botes.s3-us-west-1.amazonaws.com/botes-data/botesv1.stream-snmp.json.gz>                                         |
| stream-tcp            | <https://botes.s3-us-west-1.amazonaws.com/botes-data/botesv1.stream-tcp.json.gz>                                          |
| suricata              | <https://botes.s3-us-west-1.amazonaws.com/botes-data/botesv1.suricata.json.gz>                                            |
| winevent-application  | <https://botes.s3-us-west-1.amazonaws.com/botes-data/botesv1.WinEventLog-Application.json.gz>                             |
| winevent-security     | <https://botes.s3-us-west-1.amazonaws.com/botes-data/botesv1.WinEventLog-Security.json.gz>                                |
| winevent-system       | <https://botes.s3-us-west-1.amazonaws.com/botes-data/botesv1.WinEventLog-System.json.gz>                                  |
| winregistry           | <https://botes.s3-us-west-1.amazonaws.com/botes-data/botesv1.winregistry.json.gz>                                         |
| xmlwineventlog-sysmon | <https://botes.s3-us-west-1.amazonaws.com/botes-data/botesv1.XmlWinEventLog-Microsoft-Windows-Sysmon-Operational.json.gz> |


---

# Agent Instructions: Querying This Documentation

If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.

Perform an HTTP GET request on the current page URL with the `ask` query parameter:

```
GET https://botes.gitbook.io/botes-dataset/botes-elastic-bots-version.md?ask=<question>
```

The question should be specific, self-contained, and written in natural language.
The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.