> For the complete documentation index, see [llms.txt](https://botes.gitbook.io/botes-dataset/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://botes.gitbook.io/botes-dataset/data-sanitization.md).

# Data Sanitization

This page contains information about sanitization process. Tables of which fields have been deleted and the reason are also in this page.

Datas have to be cleaned before ingestion, some fields are deleted for this purpose. These fields were specific to Splunk, duplicated, reproductible with Logstash if needed or not usefull. 

{% hint style="info" %}
Note that no lines containing data have been deleted. The line number of the BOTES dataset is therefore the same as that of the BOTS dataset.
{% endhint %}

This section contains lists referencing global fields deleted for all Datasets/Source types and specfic fields deleted for each Dataset/Sourcetype. 

## Datasets sanitization

### Step 1

Each BOTS JSON data are nested in "result". First step is to bring data up to the first level, for more convinience. Data initially at the first level will not be kept because they are not useful :

```
sed -i -r 's/\{\"preview\"\:false\,\"offset\"\:[0-9]+\,\"result\"\://g' $JSONfile
sed -i -r 's/\{"preview":false,"lastrow":true\}//g' $JSONfile
sed -i -r 's/\}$//g' $JSONfile
```

or to modify all file in same time :

```
for file in ./*.json; do sed -i -r 's/\{\"preview\"\:false\,\"offset\"\:[0-9]+\,\"result\"\://g' "$file"; sed -i -r 's/\{"preview":false,"lastrow":true\}//g' $file; sed -i -r 's/\}$//g' $file; done
```

### Step 2

Second step is to list unique fields for each JSON file. The goal is to sort and then delete duplicated or useless fields, to finally establish a list of all unique fields and match them to an ECS field.  

 Sort unique fields by launching the following command for each JSON file :

```
jq 'keys' botsv1.$dataset.json | sort -u
```

{% hint style="info" %}
Reminder : **jq** need to be installed for this and the following steps. Check [prerequisites](/botes-dataset/botes-prerequisites.md#install-jq) page if it's not already done.
{% endhint %}

Result from commands on each JSON files are put in separate table to be sorted and cleaned on next step.

### Step 3

Some values can be found twice or three times in different original fields. In this case, fields created/derivated from Splunk parsing are deleted and fieldsname from the original logs are kept.

#### Example :

```
{... "dest":"8.8.8.8",
"dest_ip":"8.8.8.8","dest_mac":"08:5B:0E:93:92:AF","dest_port":"53", ...
"_raw": ... \"dest_ip\":\"8.8.8.8\",\"dest_mac\":\"08:5B:0E:93:92:AF\" ...}
```

"dest" and dest\_ip" fields contain the same value. The raw log indicate that the "dest\_ip" is the true original field, so in this case "dest\_ip" will be kept and "dest" deleted.  

Some values are deleted because of bad parsing or because it's easier to set them directly at Logstash level (tags, vendor, etc.).

#### Example :

```
{... "cs_uri_query":"searchword=&n990136=v920580&ordering=newest&searchphrase=all&areas[0]=categories",
... "host":"we1149srv","n990136":"v920580","ordering":"newest", ...}
```

In this example, the field "cs\_uri\_query" is not correctly parsed because of "=" in the value and field "n990136" is created when it shouldn't be.

Tables clean-up actions :

* Fields that are deleted are removed from table(s) created on the previous step.
* Fields that are deleted are add in **jq** delete list (Lists are at the end of each Datasets table).
* All kept fields are then matched with an ECS field and the corresponding type, fieldset, level and description.
* If no ECS Fieldset can be used to match an original field, a new fieldset is created or a new field in an existing ECS fieldset, with the 'botes' prefix, is created. 

Refer to section [BOTES Fields](/botes-dataset/botes-fields.md) for Excel and CSV files containing cleaned Datastes fields. 

### Step 4

For each JSON files, delete choosen fields with **jq** and lists established in previous step.

#### Example : 

```
jq -c 'del(._si, ._cd, ._bkt, ._indextime, ._serial, ._eventtype_color, ._sourcetype, 
._subsecond, .app, .date, .date_hour, .date_mday, .date_minute, .date_month, 
.date_second, .date_wday, .date_year, .dest, .dvc, .host, .index, .linecount, .punct, 
.source, .sourcetype, .splunk_server, .src, .tag, .["tag::eventtype"], .timeendpos, 
.timestamp, .timestartpos)' botsv1.stream-ip.json > botesv1.stream-ip.json
```

### Step 5

Rename "\_time" and "\_raw" fields to avoid issues during Logstash ingestion :

```
sed -i -r 's/\"\_time\"/\"time\"/g' $botesv1.*.json
sed -i -r 's/\"\_raw\"/\"message\"/g' $botesv1.*.json
```

### Step 6

For fields with type "Long", remove quote on the JSON files or they will be interpreted as Strings by Elasticsearch.

```
sed -r -i 's/"(datapackets_in|missing_packets_in|crscore|bytes_out|rcvdbyte|flow.bytes_toclient|bytes_in|tranport|rcvdpkt|flow.pkts_toclient|packets_in|dstport|dest_port|ports{}.port|DestinationPort|lease|ip_lease_time|total|used|response_time|dns.ttl|ttl{}|sn|RecordNumber|RecordID|fileinfo.size|filesize|filesize{}|refused|http.length|http_content_length|sc_substatus|time_taken|http.status|sc_status|status|status{}|checksum|icmp_code|code|sequence|id|icmp_type|type|folder_id|hostcount|host_id|object_id|count|sid|severity_index|user_permissions|vuln_index|fragment_count|tos|version|duration|flow.age|bytes|packets|pid|data_packets_out|missing_packets_out|s_port|request_call_id|search_attributes|sentbyte|flow.bytes_toserver|flow.pkts_toserver|vendor_transport|sentpkt|packets_out|srcport|src_port|Port|Source_Port|SourcePort|stats.capture.kernel_drops|stats.capture.kernel_packets|stats.decoder.avg_pkt_size|stats.decoder.bytes|stats.decoder.erspan|stats.decoder.ethernet|stats.decoder.gre|stats.decoder.icmpv4|stats.decoder.icmpv6|stats.decoder.invalid|stats.decoder.ipraw.invalid_ip_version|stats.decoder.ipv4|stats.decoder.ipv4_in_ipv6|stats.decoder.ipv6|stats.decoder.ipv6_in_ipv6|stats.decoder.ltnull.pkt_too_small|stats.decoder.ltnull.unsupported_type|stats.decoder.max_pkt_size|stats.decoder.mpls|stats.decoder.null|stats.decoder.pkts|stats.decoder.ppp|stats.decoder.pppoe|stats.decoder.raw|stats.decoder.sctp|stats.decoder.sll|stats.decoder.tcp|stats.decoder.teredo|stats.decoder.udp|stats.decoder.vlan|stats.decoder.vlan_qinq|stats.defrag.ipv4.fragments|stats.defrag.ipv4.reassembled|stats.defrag.ipv4.timeouts|stats.defrag.ipv6.fragments|stats.defrag.ipv6.reassembled|stats.defrag.ipv6.timeouts|stats.defrag.max_frag_hits|stats.detect.alert|stats.dns.memcap_global|stats.dns.memcap_state|stats.dns.memuse|stats.flow.emerg_mode_entered|stats.flow.emerg_mode_over|stats.flow.memcap|stats.flow.memuse|stats.flow.spare|stats.flow.tcp_reuse|stats.flow_mgr.closed_pruned|stats.flow_mgr.est_pruned|stats.flow_mgr.new_pruned|stats.http.memcap|stats.http.memuse|stats.stream.3whs_ack_in_wrong_dir|stats.stream.3whs_async_wrong_seq|stats.stream.3whs_right_seq_wrong_ack_evasion|stats.tcp.invalid_checksum|stats.tcp.memuse|stats.tcp.no_flow|stats.tcp.pseudo|stats.tcp.pseudo_failed|stats.tcp.reassembly_gap|stats.tcp.reassembly_memuse|stats.tcp.rst|stats.tcp.segment_memcap_drop|stats.tcp.sessions|stats.tcp.ssn_memcap_drop|stats.tcp.stream_depth_reached|stats.tcp.syn|stats.tcp.synack|stats.uptime|countav|countapp|countips|policyid|alert.gid|alert.rev|countweb|ssl_cipher_id|ssl_publickey_bit_len|Revived_Cache|Saved_Cache|Logon_Type|Nominal_Frequency__MHz|Key_Length|OpCode|Opcode|Maximum_performance_percentage|Minimum_performance_percentage|Restricted_SID_Count|Security_Error|Session_ID|Version|Task|TerminalSessionId|Minimum_throttle_percentage|Token_Elevation_Type_id|New_State|Number_of_Elements|Report_Status|sc_win32_status)":"([+-]?[0-9]+([.][0-9]+)?)"/"\1":\2/g' botesv1.*.json
```

### Step 7

Modifications specific to Data sourcetype are needed too.

#### Stream-DNS

```
sed -r -i 's/"(host_addr\{\})":"(([0-9]{1,3}.){3}([0-9]{1,3}))"/"\1":["\2"]/g' botesv1.stream-dns.json
```

## Global removed fields

| Reason               | Field Name                                                                     |
| -------------------- | ------------------------------------------------------------------------------ |
| \_si                 | Splunk specific.                                                               |
| \_cd                 | Splunk specific. (Event address in index)                                      |
| \_bkt                | Splunk specific. (Bucket ID)                                                   |
| \_serial             | Splunk specific.                                                               |
| *\_eventtype\_color* | Splunk specific.                                                               |
| \_sourcetype         | Event sourcetype be added by Logstash.                                         |
| \_subsecond          | Already in contained in "\_time" field. Can be obtain with Logstash if needed. |
| app                  | Inaccurate value or can be obtain with Logstash if needed.                     |
| date                 | Already in contained in "\_time" field. Can be obtain with Logstash if needed. |
| date\_hour           | Splunk default datetime fields.                                                |
| date\_mday           | Splunk default datetime fields.                                                |
| date\_minute         | Splunk default datetime fields.                                                |
| date\_month          | Splunk default datetime fields.                                                |
| date\_second         | Splunk default datetime fields.                                                |
| date\_wday           | Splunk default datetime fields.                                                |
| date\_year           | Splunk default datetime fields.                                                |
| dvc                  | Duplicated value.                                                              |
| index                | Splunk specific. (Index name)                                                  |
| linecount            | Splunk specific. (Current log line count)                                      |
| punct                | Splunk specific. (Punctuation pattern for an event)                            |
| source               | Event source will be added by Logstash.                                        |
| sourcetype           | Event sourcetype be added by Logstash.                                         |
| splunk\_server       | Splunk specific. (Splunk server name)                                          |
| tag                  | Event will be tagged by Logstash.                                              |
| tag::eventtype       | Event will be tagged by Logstash.                                              |
| time                 | Already in contained in "\_time" field. Can be obtain with Logstash if needed. |
| timeendpos           | Confusing with other fields.                                                   |
| timestamp            | Duplicate of "time" field.                                                     |
| timestartpos         | Confusing with other fields.                                                   |

## Fgt removed fields

### fgt\_event

| Field Name         | Reason |
| ------------------ | ------ |
| action             |        |
| category           |        |
| command            |        |
| cpu                |        |
| cpu\_load\_percent |        |
| dest               |        |
| disk               |        |
| disklograte        |        |
| duration           |        |
| fams\_pause        |        |
| fazlograte         |        |
| limit              |        |
| mem                |        |
| object\_category   |        |
| product\_version   |        |
| session\_id        |        |
| setuprate          |        |
| totalsession       |        |
| src                |        |
| vd                 |        |
| vendor\_eventtype  |        |
| vendor\_status     |        |

fgt\_event jq delete command :

```
jq -c 'del(._si, ._cd, ._bkt, ._indextime, ._serial, ._eventtype_color, ._sourcetype, .action, .category, .command, .cpu, .cpu_load_percent, .date, .date_hour, .date_mday, .date_minute, .date_month, .date_second, .date_wday, .date_year, .dest, .disk, .disklograte, .duration, .dvc, .fams_pause, .fazlograte, .index, .linecount, .limit, .mem, .object_category, .product_version, .session_id, .source, .sourcetype, .splunk_server, .src, .setuprate, .totalsession, .tag, .["tag::eventtype"], .time, .timeendpos, .timestartpos, .vd, .vendor_eventtype, .vendor_status)' botsv1.fgt_event.json > botesv1.fgt_event.json
```

### fgt\_traffic

| Field Name             | Reason |
| ---------------------- | ------ |
| action                 |        |
| bytes\_in              |        |
| bytes\_out             |        |
| category               |        |
| dest                   |        |
| dest\_interface        |        |
| dest\_port             |        |
| dest\_translated\_ip   |        |
| dest\_translated\_port |        |
| dstcountry             |        |
| ftnt\_action           |        |
| packets\_in            |        |
| packets\_out           |        |
| product\_version       |        |
| rule                   |        |
| rule\_id               |        |
| session\_id            |        |
| srccountry             |        |
| src\_interface         |        |
| src\_ip                |        |
| src\_port              |        |
| src\_mac               |        |
| src\_translated\_ip    |        |
| src                    |        |
| vd                     |        |
| vendor\_eventtype      |        |

fgt\_traffic jq delete command :

```
jq -c 'del(._si, ._cd, ._bkt, ._indextime, ._serial, ._eventtype_color, ._sourcetype, .action, .bytes_in, .bytes_out, .category, .date, .date_hour, .date_mday, .date_minute, .date_month, .date_second, .date_wday, .date_year, .dest, .dest_interface, .dest_port, .dest_translated_ip, .dest_translated_port, .dstcountry, .dvc, .ftnt_action, .index, .linecount, .product_version, .packets_in, .packets_out, .rule, .rule_id, .srccountry, .src_interface, .src_ip, .src_port, .src_mac, .src_translated_ip, .session_id, .source, .sourcetype, .splunk_server, .src, .tag, .["tag::eventtype"], .time, .timeendpos, .timestartpos, .vd, .vendor_eventtype)' botsv1.fgt_traffic.json > botesv1.fgt_traffic.json
```

### fgt\_utm

| Field Name        | Reason |
| ----------------- | ------ |
| action            |        |
| agent             |        |
| bytes\_in         |        |
| bytes\_out        |        |
| category          |        |
| dest              |        |
| dest\_interface   |        |
| dest\_port        |        |
| file\_name        |        |
| ftnt\_action      |        |
| http\_method      |        |
| product\_version  |        |
| session\_id       |        |
| signature         |        |
| site              |        |
| src               |        |
| src\_interface    |        |
| src\_port         |        |
| srccountry        |        |
| status            |        |
| vd                |        |
| vendor\_eventtype |        |

fgt\_utm jq delete command :

```
jq -c 'del(._si, ._cd, ._bkt, ._indextime, ._serial, ._eventtype_color, ._sourcetype, .action, .agent, .bytes_in, .bytes_out, .category, .date, .date_hour, .date_mday, .date_minute, .date_month, .date_second, .date_wday, .date_year, .dest, .dest_interface, .dest_port, .dvc, .file_name, .ftnt_action, .http_method, .index, .linecount, .product_version, .session_id, .source, .sourcetype, .splunk_server, .signature, .site, .src_interface, .src_port, .srccountry, .status, .src, .tag, .["tag::eventtype"], .time, .timeendpos, .timestartpos, .vd, .vendor_eventtype)' botsv1.fgt_utm.json > botesv1.fgt_utm.json
```

## IIS removed fields

### IIS

| Field Name                      | Reason                                                        |
| ------------------------------- | ------------------------------------------------------------- |
| \_kv                            | Bad original "cs\_uri\_query" or "cs\_referer" field parsing. |
| a                               | Bad original "cs\_uri\_query" or "cs\_referer" field parsing. |
| a1f5ea945d8863b612f9488485969e4 | Bad original "cs\_uri\_query" or "cs\_referer" field parsing. |
| action                          | Bad original "cs\_uri\_query" or "cs\_referer" field parsing. |
| ADMINTYPE                       | Bad original "cs\_uri\_query" or "cs\_referer" field parsing. |
| allowedDomain                   | Bad original "cs\_uri\_query" or "cs\_referer" field parsing. |
| ApHost                          | Bad original "cs\_uri\_query" or "cs\_referer" field parsing. |
| aspxerrorpath                   | Bad original "cs\_uri\_query" or "cs\_referer" field parsing. |
| btnSubmit                       | Bad original "cs\_uri\_query" or "cs\_referer" field parsing. |
| BVw                             | Bad original "cs\_uri\_query" or "cs\_referer" field parsing. |
| cache\_timeout                  | Bad original "cs\_uri\_query" or "cs\_referer" field parsing. |
| catid                           | Bad original "cs\_uri\_query" or "cs\_referer" field parsing. |
| cd                              | Bad original "cs\_uri\_query" or "cs\_referer" field parsing. |
| CGIAlias                        | Bad original "cs\_uri\_query" or "cs\_referer" field parsing. |
| cmd                             | Bad original "cs\_uri\_query" or "cs\_referer" field parsing. |
| config                          | Bad original "cs\_uri\_query" or "cs\_referer" field parsing. |
| culture                         | Bad original "cs\_uri\_query" or "cs\_referer" field parsing. |
| debug                           | Bad original "cs\_uri\_query" or "cs\_referer" field parsing. |
| depth                           | Bad original "cs\_uri\_query" or "cs\_referer" field parsing. |
| dest                            | Bad original "cs\_uri\_query" or "cs\_referer" field parsing. |
| do                              | Bad original "cs\_uri\_query" or "cs\_referer" field parsing. |
| dTw                             | Bad original "cs\_uri\_query" or "cs\_referer" field parsing. |
| ei                              | Bad original "cs\_uri\_query" or "cs\_referer" field parsing. |
| eid                             | Bad original "cs\_uri\_query" or "cs\_referer" field parsing. |
| f                               | Bad original "cs\_uri\_query" or "cs\_referer" field parsing. |
| feed                            | Bad original "cs\_uri\_query" or "cs\_referer" field parsing. |
| file                            | Bad original "cs\_uri\_query" or "cs\_referer" field parsing. |
| File                            | Bad original "cs\_uri\_query" or "cs\_referer" field parsing. |
| format                          | Bad original "cs\_uri\_query" or "cs\_referer" field parsing. |
| full                            | Bad original "cs\_uri\_query" or "cs\_referer" field parsing. |
| ge                              | Bad original "cs\_uri\_query" or "cs\_referer" field parsing. |
| GET                             | Bad original "cs\_uri\_query" or "cs\_referer" field parsing. |
| get\_data                       | Bad original "cs\_uri\_query" or "cs\_referer" field parsing. |
| group                           | Bad original "cs\_uri\_query" or "cs\_referer" field parsing. |
| gzip                            | Bad original "cs\_uri\_query" or "cs\_referer" field parsing. |
| highlighterId                   | Bad original "cs\_uri\_query" or "cs\_referer" field parsing. |
| hk                              | Bad original "cs\_uri\_query" or "cs\_referer" field parsing. |
| id                              | Bad original "cs\_uri\_query" or "cs\_referer" field parsing. |
| iN                              | Bad original "cs\_uri\_query" or "cs\_referer" field parsing. |
| item                            | Bad original "cs\_uri\_query" or "cs\_referer" field parsing. |
| Itemid                          | Bad original "cs\_uri\_query" or "cs\_referer" field parsing. |
| jQuery                          | Bad original "cs\_uri\_query" or "cs\_referer" field parsing. |
| layout                          | Bad original "cs\_uri\_query" or "cs\_referer" field parsing. |
| link                            | Bad original "cs\_uri\_query" or "cs\_referer" field parsing. |
| module                          | Bad original "cs\_uri\_query" or "cs\_referer" field parsing. |
| movieName                       | Bad original "cs\_uri\_query" or "cs\_referer" field parsing. |
| n990136                         | Bad original "cs\_uri\_query" or "cs\_referer" field parsing. |
| name                            | Bad original "cs\_uri\_query" or "cs\_referer" field parsing. |
| operator                        | Bad original "cs\_uri\_query" or "cs\_referer" field parsing. |
| option                          | Bad original "cs\_uri\_query" or "cs\_referer" field parsing. |
| ordering                        | Bad original "cs\_uri\_query" or "cs\_referer" field parsing. |
| page                            | Bad original "cs\_uri\_query" or "cs\_referer" field parsing. |
| param                           | Bad original "cs\_uri\_query" or "cs\_referer" field parsing. |
| playerready                     | Bad original "cs\_uri\_query" or "cs\_referer" field parsing. |
| plugin                          | Bad original "cs\_uri\_query" or "cs\_referer" field parsing. |
| Pp                              | Bad original "cs\_uri\_query" or "cs\_referer" field parsing. |
| print                           | Bad original "cs\_uri\_query" or "cs\_referer" field parsing. |
| pZ                              | Bad original "cs\_uri\_query" or "cs\_referer" field parsing. |
| q                               | Bad original "cs\_uri\_query" or "cs\_referer" field parsing. |
| QA                              | Bad original "cs\_uri\_query" or "cs\_referer" field parsing. |
| qH                              | Bad original "cs\_uri\_query" or "cs\_referer" field parsing. |
| rct                             | Bad original "cs\_uri\_query" or "cs\_referer" field parsing. |
| rdoSearch                       | Bad original "cs\_uri\_query" or "cs\_referer" field parsing. |
| redirects                       | Bad original "cs\_uri\_query" or "cs\_referer" field parsing. |
| report                          | Bad original "cs\_uri\_query" or "cs\_referer" field parsing. |
| returnto                        | Bad original "cs\_uri\_query" or "cs\_referer" field parsing. |
| rev                             | Bad original "cs\_uri\_query" or "cs\_referer" field parsing. |
| route                           | Bad original "cs\_uri\_query" or "cs\_referer" field parsing. |
| rurl                            | Bad original "cs\_uri\_query" or "cs\_referer" field parsing. |
| sa                              | Bad original "cs\_uri\_query" or "cs\_referer" field parsing. |
| search                          | Bad original "cs\_uri\_query" or "cs\_referer" field parsing. |
| searchphrase                    | Bad original "cs\_uri\_query" or "cs\_referer" field parsing. |
| searchword                      | Bad original "cs\_uri\_query" or "cs\_referer" field parsing. |
| section                         | Bad original "cs\_uri\_query" or "cs\_referer" field parsing. |
| sections                        | Bad original "cs\_uri\_query" or "cs\_referer" field parsing. |
| self\_a                         | Bad original "cs\_uri\_query" or "cs\_referer" field parsing. |
| selfor                          | Bad original "cs\_uri\_query" or "cs\_referer" field parsing. |
| sig2                            | Bad original "cs\_uri\_query" or "cs\_referer" field parsing. |
| skip                            | Bad original "cs\_uri\_query" or "cs\_referer" field parsing. |
| sl                              | Bad original "cs\_uri\_query" or "cs\_referer" field parsing. |
| src                             | Bad original "cs\_uri\_query" or "cs\_referer" field parsing. |
| tab                             | Bad original "cs\_uri\_query" or "cs\_referer" field parsing. |
| task                            | Bad original "cs\_uri\_query" or "cs\_referer" field parsing. |
| template                        | Bad original "cs\_uri\_query" or "cs\_referer" field parsing. |
| test                            | Bad original "cs\_uri\_query" or "cs\_referer" field parsing. |
| time                            | Bad original "cs\_uri\_query" or "cs\_referer" field parsing. |
| title                           | Bad original "cs\_uri\_query" or "cs\_referer" field parsing. |
| tl                              | Bad original "cs\_uri\_query" or "cs\_referer" field parsing. |
| tmpl                            | Bad original "cs\_uri\_query" or "cs\_referer" field parsing. |
| txtSearchname                   | Bad original "cs\_uri\_query" or "cs\_referer" field parsing. |
| type                            | Bad original "cs\_uri\_query" or "cs\_referer" field parsing. |
| u                               | Bad original "cs\_uri\_query" or "cs\_referer" field parsing. |
| uploadifyID                     | Bad original "cs\_uri\_query" or "cs\_referer" field parsing. |
| url                             | Bad original "cs\_uri\_query" or "cs\_referer" field parsing. |
| usg                             | Bad original "cs\_uri\_query" or "cs\_referer" field parsing. |
| ved                             | Bad original "cs\_uri\_query" or "cs\_referer" field parsing. |
| view                            | Bad original "cs\_uri\_query" or "cs\_referer" field parsing. |
| window\_x                       | Bad original "cs\_uri\_query" or "cs\_referer" field parsing. |
| wNO                             | Bad original "cs\_uri\_query" or "cs\_referer" field parsing. |
| wvstest                         | Bad original "cs\_uri\_query" or "cs\_referer" field parsing. |
| wVz                             | Bad original "cs\_uri\_query" or "cs\_referer" field parsing. |
| xC                              | Bad original "cs\_uri\_query" or "cs\_referer" field parsing. |
| xmlDataPath                     | Bad original "cs\_uri\_query" or "cs\_referer" field parsing. |
| xml\_stylesheet                 | Bad original "cs\_uri\_query" or "cs\_referer" field parsing. |
| yqQA5                           | Bad original "cs\_uri\_query" or "cs\_referer" field parsing. |

IIS jq delete command :

```
jq -c 'del(._si, ._cd, ._bkt, ._indextime, ._serial, ._eventtype_color, ._sourcetype, ._subsecond, .a, .a1f5ea945d8863b612f9488485969e4, .action, .ADMINTYPE, .allowedDomain, .ApHost, .app, .aspxerrorpath, .btnSubmit, .BVw, .cache_timeout, .catid, .cd, .CGIAlias, .cmd, .config, .culture, .date, .date_hour, .date_mday, .date_minute, .date_month, .date_second, .date_wday, .date_year, .dvc, .debug, .depth, .dest, .do, .dTw, .ei, .eid, .f, .feed, .file, .File, .format, .full, .ge, .GET, .get_data, .group, .gzip, .highlighterId, .hk, .index, .linecount, .id, .iN, .item, .Itemid, .jQuery, ._kv, .layout, .link, .module, .movieName, .n990136, .name, .operator, .option, .ordering, .page, .param, .playerready, .plugin, .Pp, .print, .pZ, .q, .QA, .qH, .rct, .rdoSearch, .redirects, .report, .returnto, .rev, .route, .rurl, .sa, .search, .searchphrase, .searchword, .section, .sections, .self_a, .selfor, .sig2, .skip, .sl, .source, .sourcetype, .splunk_server, .src, .tab, .tag, .["tag::eventtype"], .task, .template, .test, .time, .title, .tl, .tmpl, .txtSearchname, .type, .u, .uploadifyID, .url, .usg, .ved, .view, .window_x, .wNO, .wvstest, .wVz, .xC, .xmlDataPath, .xml_stylesheet, .yqQA5)' botsv1.iis.json > botesv1.iis.json
```

## Nessus removed fields

### Nessus:Scan

| Field Name | Reason                                  |
| ---------- | --------------------------------------- |
| dest       | Duplicate of "host-ip" field.           |
| dest\_host | Duplicate of "hostname" field.          |
| dest\_ip   | Duplicate of "host-ip" field.           |
| eventtype  | Event will be tagged/typed by Logstash. |
| product    | Will be set with Logsash.               |
| signature  | Duplicate of "plugin\_name" field.      |
| vendor     | Will be set with Logsash.               |

nessus:scan jq delete command :

```
jq -c 'del(._si, ._cd, ._bkt, ._indextime, ._serial, ._eventtype_color, ._sourcetype, ._subsecond, .app, .date, .date_hour, .date_mday, .date_minute, .date_month, .date_second, .date_wday, .date_year, .dest, .dest_host, .dest_ip, .dvc, .eventtype, .index, .linecount, .punct, .product, .source, .sourcetype, .splunk_server, .tag, .["tag::eventtype"], .timeendpos, .timestamp, .timestartpos, .vendor)' botsv1.nessus-scan.json > botesv1.nessus-scan.json
```

## Stream removed fields

### Stream:DHCP

| Field Name      | Reason |
| --------------- | ------ |
| action          |        |
| dest            |        |
| lease\_duration |        |
| signature       |        |
| src             |        |

stream:dhcp jq delete command :

```
jq -c 'del(._si, ._cd, ._bkt, ._indextime, ._serial, ._eventtype_color, ._sourcetype, ._subsecond, .action, .app, .date, .date_hour, .date_mday, .date_minute, .date_month, .date_second, .date_wday, .date_year, .dest, .dvc, .index, .lease_duration, .linecount, .punct, .signature, .source, .sourcetype, .src, .splunk_server, .tag, .["tag::eventtype"], .timeendpos, .timestamp, .timestartpos)' botsv1.stream-dhcp.json > botesv1.stream-dhcp.json
```

### Stream:DNS

| Field Name       | Reason |
| ---------------- | ------ |
| answer           |        |
| dest             |        |
| message\_type{}  |        |
| query{}          |        |
| query\_type{}    |        |
| reply\_code{}    |        |
| response\_time{} |        |
| src              |        |

stream:dns jq delete command :

```
jq -c 'del(._si, ._cd, ._bkt, ._indextime, ._serial, ._eventtype_color, ._sourcetype, ._subsecond, .app, .answer, .date, .date_hour, .date_mday, .date_minute, .date_month, .date_second, .date_wday, .date_year, .dest, .dvc, .index, .linecount, .["message_type{}"], .["query{}"], .["query_type{}"], .["reply_code{}"], .["response_time{}"], .source, .sourcetype, .src, .splunk_server, .tag, .["tag::eventtype"], .timeendpos, .timestamp, .timestartpos)' botsv1.stream-dns.json > botesv1.stream-dns.json
```

### Stream:HTTP

| Field Name              | Reason |
| ----------------------- | ------ |
| action                  |        |
| capture\_hostname       |        |
| client\_rtt             |        |
| client\_rtt\_packets    |        |
| client\_rtt\_sum        |        |
| data\_center\_time      |        |
| dest                    |        |
| duplicate\_packets\_in  |        |
| duplicate\_packets\_out |        |
| duration                |        |
| form\_data              |        |
| host                    |        |
| reply\_time             |        |
| request\_ack\_time      |        |
| request\_time           |        |
| response\_ack\_time     |        |
| response\_time          |        |
| server\_rtt             |        |
| server\_rtt\_packets    |        |
| server\_rtt\_sum        |        |
| src                     |        |

stream:http jq delete command :

```
jq -c 'del(._si, ._cd, ._bkt, ._indextime, ._serial, ._eventtype_color, ._sourcetype, ._subsecond, .action, .app, .capture_hostname, .client_rtt, .client_rtt_packets, .client_rtt_sum, .date, .date_hour, .date_mday, .date_minute, .date_month, .date_second, .date_wday, .date_year, .data_center_time, .dest, .duplicate_packets_in, .duplicate_packets_out, .duration, .dvc, .form_data, .host, .index, .linecount, .punct, .reply_time, .request_ack_time, .request_time, .response_ack_time, .response_time, .server_rtt, .server_rtt_packets, .server_rtt_sum, .source, .sourcetype, .splunk_server, .src, .tag, .["tag::eventtype"], .timeendpos, .timestamp, .timestartpos)' botsv1.stream-http.json > botesv1.stream-http.json
```

### Stream:ICMP

| Field Name | Reason |
| ---------- | ------ |
| dest       |        |
| duration   |        |
| host       |        |
| src        |        |

stream:icmp jq delete command :

```
jq -c 'del(._si, ._cd, ._bkt, ._indextime, ._serial, ._eventtype_color, ._sourcetype, ._subsecond, .app, .date, .date_hour, .date_mday, .date_minute, .date_month, .date_second, .date_wday, .date_year, .dest, .duration, .dvc, .host, .index, .linecount, .punct, .source, .sourcetype, .splunk_server, .src, .tag, .["tag::eventtype"], .timeendpos, .timestamp, .timestartpos)' botsv1.stream-icmp.json > botesv1.stream-icmp.json
```

### Stream:IP

| Field Name | Reason |
| ---------- | ------ |
| dest       |        |
| host       |        |
| src        |        |

stream:ip jq delete command :

```
jq -c 'del(._si, ._cd, ._bkt, ._indextime, ._serial, ._eventtype_color, ._sourcetype, ._subsecond, .app, .date, .date_hour, .date_mday, .date_minute, .date_month, .date_second, .date_wday, .date_year, .dest, .dvc, .host, .index, .linecount, .punct, .source, .sourcetype, .splunk_server, .src, .tag, .["tag::eventtype"], .timeendpos, .timestamp, .timestartpos)' botsv1.stream-ip.json > botesv1.stream-ip.json
```

### Stream:LDAP

| Field Name | Reason |
| ---------- | ------ |
| dest       |        |
| host       |        |
| src        |        |

stream:ldap jq delete command :

```
jq -c 'del(._si, ._cd, ._bkt, ._indextime, ._serial, ._eventtype_color, ._sourcetype, ._subsecond, .app, .date, .date_hour, .date_mday, .date_minute, .date_month, .date_second, .date_wday, .date_year, .dest, .dvc, .host, .index, .linecount, .punct, .source, .sourcetype, .splunk_server, .src, .tag, .["tag::eventtype"], .timeendpos, .timestamp, .timestartpos)' botsv1.stream-ldap.json > botesv1.stream-ldap.json
```

### Stream:MAPI

| Field Name     | Reason |
| -------------- | ------ |
| dest           |        |
| duration       |        |
| host           |        |
| reply\_time    |        |
| request\_time  |        |
| response\_time |        |
| src            |        |

stream:mapi jq delete command :

```
jq -c 'del(._si, ._cd, ._bkt, ._indextime, ._serial, ._eventtype_color, ._sourcetype, ._subsecond, .app, .date, .date_hour, .date_mday, .date_minute, .date_month, .date_second, .date_wday, .date_year, .dest, .duration, .dvc, .host, .index, .linecount, .punct, .reply_time, .request_time, .response_time, .source, .sourcetype, .splunk_server, .src, .tag, .["tag::eventtype"], .timeendpos, .timestamp, .timestartpos)' botsv1.stream-mapi.json > botesv1.stream-mapi.json
```

### Stream:SIP

| Field Name | Reason |
| ---------- | ------ |
| dest       |        |
| duration   |        |
| host       |        |
| src        |        |
| uri        |        |

stream:sip jq delete command :

```
jq -c 'del(._si, ._cd, ._bkt, ._indextime, ._serial, ._eventtype_color, ._sourcetype, ._subsecond, .app, .date, .date_hour, .date_mday, .date_minute, .date_month, .date_second, .date_wday, .date_year, .dest, .duration, .dvc, .host, .index, .linecount, .punct, .source, .sourcetype, .splunk_server, .src, .tag, .["tag::eventtype"], .timeendpos, .timestamp, .timestartpos, .uri)' botsv1.stream-sip.json > botesv1.stream-sip.json
```

### Stream:SMB

| Field Name | Reason |
| ---------- | ------ |
| dest       |        |
| duration   |        |
| host       |        |
| src        |        |

stream:smb jq delete command :

```
jq -c 'del(._si, ._cd, ._bkt, ._indextime, ._serial, ._eventtype_color, ._sourcetype, ._subsecond, .app, .date, .date_hour, .date_mday, .date_minute, .date_month, .date_second, .date_wday, .date_year, .dest, .duration, .dvc, .host, .index, .linecount, .punct, .source, .sourcetype, .splunk_server, .src, .tag, .["tag::eventtype"], .timeendpos, .timestamp, .timestartpos)' botsv1.stream-smb.json > botesv1.stream-smb.json
```

### Stream:SNMP

| Field Name | Reason |
| ---------- | ------ |
| dest       |        |
| duration   |        |
| host       |        |
| src        |        |

stream:snmp jq delete command :

```
jq -c 'del(._si, ._cd, ._bkt, ._indextime, ._serial, ._eventtype_color, ._sourcetype, ._subsecond, .app, .date, .date_hour, .date_mday, .date_minute, .date_month, .date_second, .date_wday, .date_year, .dest, .duration, .dvc, .host, .index, .linecount, .punct, .source, .sourcetype, .splunk_server, .src, .tag, .["tag::eventtype"], .timeendpos, .timestamp, .timestartpos)' botsv1.stream-snmp.json > botesv1.stream-snmp.json
```

### Stream:TCP

| Field Name              | Reason |
| ----------------------- | ------ |
| client\_rtt             |        |
| client\_rtt\_packets    |        |
| client\_rtt\_sum        |        |
| connection              |        |
| dest                    |        |
| duplicate\_packets\_in  |        |
| duplicate\_packets\_out |        |
| duration                |        |
| server\_rtt             |        |
| server\_rtt\_packets    |        |
| server\_rtt\_sum        |        |
| src                     |        |
| ssl\_end\_time          |        |
| ssl\_hash               |        |
| ssl\_serialnumber       |        |
| ssl\_session\_id        |        |
| ssl\_start\_time        |        |

stream:tcp jq delete command :

```
jq -c 'del(._si, ._cd, ._bkt, ._indextime, ._serial, ._eventtype_color, ._sourcetype, ._subsecond, .app, .client_rtt, .client_rtt_packets, .client_rtt_sum, .connection, .date, .date_hour, .date_mday, .date_minute, .date_month, .date_second, .date_wday, .date_year, .dest, .duplicate_packets_in, .duplicate_packets_out, .duration, .dvc, .host, .index, .linecount, .punct, .server_rtt, .server_rtt_packets, .server_rtt_sum, .source, .sourcetype, .splunk_server, .src, .ssl_end_time, .ssl_hash, .ssl_serialnumber, .ssl_session_id, .ssl_start_time, .tag, .["tag::eventtype"], .timeendpos, .timestamp, .timestartpos)' botsv1.stream-tcp.json > botesv1.stream-tcp.json
```

## Suricata removed fields

### Suricata

| Field Name                                    | Reason |
| --------------------------------------------- | ------ |
| action                                        |        |
| alert\_gid                                    |        |
| alert\_rev                                    |        |
| bytes                                         |        |
| bytes\_in                                     |        |
| bytes\_out                                    |        |
| capture\_kernel\_drops                        |        |
| category                                      |        |
| decoder\_avg\_pkt\_size                       |        |
| decoder\_bytes                                |        |
| decoder\_erspan                               |        |
| decoder\_ethernet                             |        |
| decoder\_gre                                  |        |
| decoder\_icmpv4                               |        |
| decoer\_icmpv6                                |        |
| decoder\_invalid                              |        |
| decoder\_ipraw\_invalid\_ip\_version          |        |
| decoder\_ipv4                                 |        |
| decoder\_ipv4\_in\_ipv6                       |        |
| decoder\_ipv6                                 |        |
| decoder\_ipv6\_in\_ipv6                       |        |
| decoder\_ltnull\_pkt\_too\_small              |        |
| decoder\_ltnull\_unspported\_type             |        |
| decoder\_max\_pkt\_size                       |        |
| decoder\_mpls                                 |        |
| decoder\_null                                 |        |
| decoder\_pkts                                 |        |
| decoder\_ppp                                  |        |
| decoder\_pppoe                                |        |
| decoder\_raw                                  |        |
| decoder\_sctp                                 |        |
| decoder\_ssl                                  |        |
| decoder\_tcp                                  |        |
| decoder\_teredo                               |        |
| decoder\_udp                                  |        |
| decoder\_vlan                                 |        |
| decoder\_vlan\_qinq                           |        |
| defrag\_ipv4\_fragments                       |        |
| defrag\_ipv4\_reassembled                     |        |
| defrag\_ipv4\_timeouts                        |        |
| defrag\_ipv6\_fragments                       |        |
| defrag\_ipv6\_reassembled                     |        |
| defrag\_max\_frag\_hits                       |        |
| defrag\_ipv6\_timeouts                        |        |
| dest                                          |        |
| detect\_alert                                 |        |
| dns\_memcap\_global                           |        |
| dns\_memcap\_state                            |        |
| dns\_memuse                                   |        |
| duration                                      |        |
| dvc                                           |        |
| endtime                                       |        |
| filename                                      |        |
| file\_size                                    |        |
| file\_state                                   |        |
| file\_stored                                  |        |
| file\_tx\_id                                  |        |
| flow\_emerg\_mode\_entered                    |        |
| flow\_emerg\_mode\_over                       |        |
| flow\_memcap                                  |        |
| flow\_memuse                                  |        |
| flow\_mgr\_closed\_pruned                     |        |
| flow\_mgr\_est\_pruned                        |        |
| flow\_mgr\_new\_pruned                        |        |
| flow\_spare                                   |        |
| flow\_tcp\_reuse                              |        |
| http\_content\_type                           |        |
| http\_memcap                                  |        |
| http\_memuse                                  |        |
| http\_method                                  |        |
| http\_protocol                                |        |
| http\_referrer                                |        |
| http\_user\_agent                             |        |
| message\_type                                 |        |
| packets\_in                                   |        |
| packets\_out                                  |        |
| query                                         |        |
| reason                                        |        |
| reply\_code                                   |        |
| severity                                      |        |
| severity\_id                                  |        |
| signature                                     |        |
| src                                           |        |
| ssh\_client\_software                         |        |
| ssh\_client\_version                          |        |
| ssh\_server\_software                         |        |
| ssh\_server\_version                          |        |
| ssl\_issuer\_common\_name                     |        |
| ssl\_publickey                                |        |
| ssl\_server\_name\_indication                 |        |
| ssl\_subject\_common\_name                    |        |
| ssl\_version                                  |        |
| starttime                                     |        |
| state                                         |        |
| status                                        |        |
| stream\_3whs\_ack\_in\_wrong\_dir             |        |
| stream\_3whs\_async\_wrong\_seq               |        |
| stream\_3whs\_right\_seq\_wrong\_ack\_evasion |        |
| suricata\_signature\_id                       |        |
| tcp\_ack                                      |        |
| tcp\_cwr                                      |        |
| tcp\_ecn                                      |        |
| tcp\_fin                                      |        |
| tcp\_flag\_hex                                |        |
| tcp\_flag\_hex\_to\_client                    |        |
| tcp\_flag\_hex\_to\_server                    |        |
| tcp\_invalid\_checksum                        |        |
| tcp\_memuse                                   |        |
| tcp\_no\_flow                                 |        |
| tcp\_pseudo                                   |        |
| tcp\_pseudo\_failed                           |        |
| tcp\_psh                                      |        |
| tcp\_reassembly\_gap                          |        |
| tcp\_reassembly\_memuse                       |        |
| tcp\_rst                                      |        |
| tcp\_segment\_memcap\_drop                    |        |
| tcp\_sessions                                 |        |
| tcp\_ssn\_memcap\_drop                        |        |
| tcp\_state                                    |        |
| tcp\_stream\_depth\_reached                   |        |
| tcp\_syn                                      |        |
| tcp\_synack                                   |        |
| transaction\_id                               |        |
| transport                                     |        |
| ttl                                           |        |
| tx\_id                                        |        |
| uptime                                        |        |
| url                                           |        |
| vendor\_gid                                   |        |
| vendor\_rev                                   |        |
| vendor\_sid                                   |        |

suricata jq delete command :

```
jq -c 'del(._si, ._cd, ._bkt, ._indextime, ._serial, ._eventtype_color, ._sourcetype, ._subsecond, .action, .alert_gid, .alert_rev, .app, .bytes, .bytes_in, .bytes_out, .capture_kernel_drops, .capture_kernel_packets, .category, .date, .date_hour, .date_mday, .date_minute, .date_month, .date_second, .date_wday, .date_year, .decoder_avg_pkt_size, .decoder_bytes, .decoder_erspan, .decoder_ethernet, .decoder_gre, .decoder_icmpv4, .decoer_icmpv6, .decoder_invalid, .decoder_ipraw_invalid_ip_version, .decoder_ipv4, .decoder_ipv4_in_ipv6, .decoder_ipv6, .decoder_ipv6_in_ipv6, .decoder_ltnull_pkt_too_small, .decoder_ltnull_unspported_type, .decoder_max_pkt_size, .decoder_mpls, .decoder_null, .decoder_pkts, .decoder_ppp, .decoder_pppoe, .decoder_raw, .decoder_sctp, .decoder_ssl, .decoder_tcp, .decoder_teredo, .decoder_udp, .decoder_vlan, .decoder_vlan_qinq, .defrag_ipv4_fragments, .defrag_ipv4_reassembled, .defrag_ipv4_timeouts, .defrag_ipv6_fragments, .defrag_ipv6_reassembled, .defrag_max_frag_hits, .dfrag_ipv6_timeouts, .dest, .detect_alert, .dns_memcap_global, .dns_memcap_state, .dns_memuse, .duration, .dvc, .endtime, .filename, .file_size, .file_state, .file_stored, .file_tx_id, .flow_emerg_mode_entered, .flow_emerg_mode_over, .flow_memcap, .flow_memuse, .flow_mgr_closed_pruned, .flow_mgr_est_pruned, .flow_mgr_new_pruned, .flow_spare, .flow_tcp_reuse, .http_content_type, .http_memcap, .http_memuse, .http_method, .http_protocol, .http_referrer, .http_user_agent, .message_type, .packets_in, .packets_out, .query, .reason, .reply_code, .index, .linecount, .punct, .severity, .severity_id, .signature, .source, .sourcetype, .splunk_server, .src, .ssh_client_software, .ssh_client_version, .ssh_server_software, .ssh_server_version, .ssl_issuer_common_name, .ssl_publickey, .ssl_server_name_indication, .ssl_subject_common_name, .ssl_version, .starttime, .state, .status, .stream_3whs_ack_in_wrong_dir, .stream_3whs_async_wrong_seq, .stream_3whs_right_seq_wrong_ack_evasion, .suricata_signature_id, .tcp_ack, .tcp_cwr, .tcp_ecn, .tcp_fin, .tcp_flag_hex, .tcp_flag_hex_to_client, .tcp_flag_hex_to_server, .tcp_invalid_checksum, .tcp_memuse, .tcp_no_flow, .tcp_pseudo, .tcp_pseudo_failed, .tcp_psh, .tcp_reassembly_gap, .tcp_reassembly_memuse, .tcp_rst, .tcp_segment_memcap_drop, .tcp_sessions, .tcp_ssn_memcap_drop, .tcp_state, .tcp_stream_depth_reached, .tcp_syn, .tcp_synack, .tag, .["tag::eventtype"], .timeendpos, .timestamp, .timestartpos, .transaction_id, .transport, .ttl, .tx_id, .uptime, .url, .vendor_gid, .vendor_rev, .vendor_sid)' botsv1.suricata.json > botesv1.suricata.json
```

## WinEvent removed fields

### WinEventLog:Application

| Field Name                  | Reason                                                                 |
| --------------------------- | ---------------------------------------------------------------------- |
| \_pre\_msg                  | Raw data already contain in \_raw field.                               |
| Context                     | Bad original "Message" field parsing.                                  |
| dest                        | Duplicate of "ComputerName" field.                                     |
| Dirty\_Shutdown             | Bad original "Message" field parsing.                                  |
| event\_id                   | Duplicate of "RecordNumber" field.                                     |
| eventtype                   | Collision with "EventType" field. Can be set with Logstash if needed.  |
| host                        | Duplicate of "dvc\_nt\_host" field.                                    |
| id                          | Duplicate of "RecordNumber" field.                                     |
| Logon                       | One log. Not usefull in this Windows Event Application Log.            |
| Message                     | Raw data already contain in \_raw field.                               |
| severity                    | Duplicate of "Type" field. And not corresponding to log severity.      |
| severity\_id                | Duplicate of "EventType" field. And not corresponding to log severity. |
| signature\_id               | Duplicate of "EventCode" field.                                        |
| The\_specified\_object\_... | Bad original "Message" field parsing.                                  |

wineventlog:application jq delete command :

```
jq -c 'del(._si, ._cd, ._bkt, ._indextime, ._pre_msg, ._serial, ._eventtype_color, ._sourcetype, ._subsecond, .app, .Context, .date, .date_hour, .date_mday, .date_minute, .date_month, .date_second, .date_wday, .date_year, .dest, .Dirty_Shutdown, .dvc, .event_id, .eventtype, .host, .id, .index, .linecount, .Message, .Logon, .punct, .severity, .severity_id, .signature_id, .source, .sourcetype, .splunk_server, .tag, .["tag::eventtype"], .The_specified_object_cannot_be_found__Specify_the_name_of_an_existing_object____HRESULT, .timeendpos, .timestamp, .timestartpos)' botsv1.WinEventLog-Application.json > botesv1.WinEventLog-Application.json
```

### WinEventLog:Security

| Field Name                                 | Reason                                                                 |
| ------------------------------------------ | ---------------------------------------------------------------------- |
| \_pre\_msg                                 | Raw data already contain in \_raw field.                               |
| ACCESS\_SYS\_SEC                           | Bad parsing of "Access Reasons" field.                                 |
| action                                     | Duplicate of "Keywords" field.                                         |
| AppendData\_\_or\_AddSubdirectory\_or\_... | Bad parsing of "Access Reasons" field.                                 |
| body                                       | Raw data already contain in \_raw field.                               |
| DELETE                                     | Bad parsing of "Access Reasons" field.                                 |
| dest                                       | Duplicate of "ComputerName" field.                                     |
| dest\_nt\_domain                           | Duplicate of "Account\_Domain" field.                                  |
| dest\_nt\_host                             | Duplicate of "ComputerName" field.                                     |
| event\_id                                  | Duplicate of "RecordNumber" field.                                     |
| eventtype                                  | Collision with "EventType" field. Can be set with Logstash if needed.  |
| host                                       | Duplicate of "dvc\_nt\_host" field.                                    |
| id                                         | Duplicate of "RecordNumber" field.                                     |
| member\_dn                                 | Duplicate of "Account\_Name" field.                                    |
| member\_id                                 | Duplicate of "Security\_ID" field.                                     |
| member\_nt\_domain                         | Duplicate of "Account\_Domain" field.                                  |
| msad\_action                               | Duplicate of "subject" field.                                          |
| Message                                    | Raw data already contain in \_raw field.                               |
| name                                       | Collision with "Name" field. Duplicate of "subject" field.             |
| object                                     | Will be set with Logsash.                                              |
| privilege                                  | Splunk generated field. Can be obtain with Logstash if needed.         |
| privilege\_id                              | Duplicate of "Privileges" field.                                       |
| product                                    | Will be set with Logsash.                                              |
| ReadAttributes                             | Bad parsing of "Accesses" field.                                       |
| READ\_CONTROL                              | Bad parsing of "Accesses" field.                                       |
| ReadEA                                     | Bad parsing of "Accesses" field.                                       |
| ReadData\_\_or\_ListDirectory*\_*     | Duplicate of "Access\_Reasons" field.                                  |
| session\_id                                | Collision with "Session\_ID" field. Duplicate of "Logon\_ID" field.    |
| severity                                   | Duplicate of "Type" field. And not corresponding to log severity.      |
| severity\_id                               | Duplicate of "EventType" field. And not corresponding to log severity. |
| signature\_id                              | Duplicate of "EventCode" field.                                        |
| src                                        | Duplicate of "ComputerName" field.                                     |
| src\_ip                                    | Duplicate of "Source\_Network\_Address" field.                         |
| src\_nt\_host                              | Duplicate of "ComputerName" field.                                     |
| src\_port                                  | Duplicate of "Source\_Port" field.                                     |
| src\_nt\_domain                            | Duplicate of "Account\_Domain" field.                                  |
| src\_user                                  | Duplicate of "Account\_Name" field.                                    |
| status                                     | Duplicate of "Keywords" field.                                         |
| SYNCHRONIZE                                | Bad parsing of "Access Reasons" field.                                 |
| tag::app                                   | Tags will be set with Logstash.                                        |
| tag::privilege\_id                         | Tags will be set with Logstash.                                        |
| tag::Token\_Elevation\_Type\_id            | Tags will be set with Logstash.                                        |
| user                                       | Duplicate of "Account\_Name" field.                                    |
| user\_group                                | Duplicate of "Group\_Name" field.                                      |
| vendor                                     | Tags will be set with Logstash.                                        |
| vendor\_privilege                          | Duplicate of "Privileges" field.                                       |
| WriteAttributes                            | Bad parsing of "Accesses" field.                                       |
| WriteData\_\_or\_AddFile\_                 | Bad parsing of "Accesses" field.                                       |
| WriteEA                                    | Bad parsing of "Accesses" field.                                       |

wineventlog:security jq delete command :

```
jq -c 'del(._si, ._cd, ._bkt, ._indextime, ._pre_msg, ._serial, ._eventtype_color, ._sourcetype, ._subsecond, .ACCESS_SYS_SEC, .action, .AppendData__or_AddSubdirectory_or_CreatePipeInstance_, .app, .body, .date, .date_hour, .date_mday, .date_minute, .date_month, .date_second, .date_wday, .date_year, .DELETE, .dest, .dest_nt_domain, .dest_nt_host, .dvc, .event_id, .eventtype, .id, .index, .linecount, .host, .member_dn, .member_id, .member_nt_domain, .msad_action, .Message, .name, .object, .privilege, .privilege_id, .product, .punct, .ReadData__or_ListDirectory_, .ReadAttributes, .READ_CONTROL, .ReadEA, .session_id, .severity, .severity_id, .signature, .signature_id, .source, .sourcetype, .splunk_server, .src, .src_ip, .src_nt_domain, .src_nt_host, .src_port, .src_user, .status, .SYNCHRONIZE, .tag, .["tag::app"], .["tag::eventtype"], .["tag::privilege_id"], .["tag::Token_Elevation_Type_id"], .timeendpos, .timestamp, .timestartpos, .user, .user_group, .vendor, .vendor_privilege, .WriteAttributes, .WriteData__or_AddFile_, .WriteEA)' botsv1.WinEventLog-Security.json > botesv1.WinEventLog-Security.json
```

### WinEventLog:System

| Field Name     | Reason                                                                       |
| -------------- | ---------------------------------------------------------------------------- |
| \_pre\_msg     | Raw data already contain in \_raw field.                                     |
| body           | Raw data already contain in \_raw field.                                     |
| dest           | Duplicate of "ComputerName" field.                                           |
| dvc\_nt\_host  | Duplicate of "Host\_Name" field.                                             |
| event\_id      | Duplicate of "RecordNumber" field.                                           |
| eventtype      | Collision with "EventType" field. Can be set with Logstash if needed.        |
| host           | Duplicate of "Host\_Name" field.                                             |
| id             | Duplicate of "RecordNumber" field.                                           |
| Message        | Raw data already contain in \_raw field.                                     |
| package\_title | Bad original "Message" field parsing. Can be obtain with Logstash if needed. |
| product        | Will be set with Logsash.                                                    |
| severity       | Duplicate of "Type" field. And not corresponding to log severity.            |
| severity\_id   | Duplicate of "EventType" field. And not corresponding to log severity.       |
| signature\_id  | Duplicate of "EventCode" field.                                              |
| src            | Duplicate of "ComputerName" field.                                           |
| status         | Duplicate of "Keywords" field.                                               |
| user           | Collision with "User" field. Duplicate of "User" field.                      |
| vendor         | Tags will be set with Logstash.                                              |

wineventlog:system jq delete command :

```
jq -c 'del(._si, ._cd, ._bkt, ._indextime, ._pre_msg, ._serial, ._eventtype_color, ._sourcetype, ._subsecond, .app, .body, .date, .date_hour, .date_mday, .date_minute, .date_month, .date_second, .date_wday, .date_year, .dest, .dvc, .dvc_nt_host, .event_id, .eventtype, .index, .linecount, .host, .id, .Message, .package_title, .punct, .product, .severity, .severity_id, .signature_id, .source, .sourcetype, .splunk_server, .src, .tag, .["tag::eventtype"], .timeendpos, .timestamp, .timestartpos, .user, .vendor)' botsv1.WinEventLog-System.json > botesv1.WinEventLog-System.json
```

### WinRegistry

| Field Name            | Reason |
| --------------------- | ------ |
| \_kv                  |        |
| action                |        |
| dest                  |        |
| LinkId                |        |
| msg                   |        |
| object                |        |
| object\_category      |        |
| object\_path          |        |
| registry\_path        |        |
| registry\_value\_data |        |
| registry\_value\_type |        |
| user                  |        |
| user\_type            |        |
| vendor\_action        |        |
| vendor\_status        |        |

winregistry jq delete command :

```
jq -c 'del(._si, ._cd, ._bkt, ._indextime, ._kv, ._serial, ._eventtype_color, ._sourcetype, ._subsecond, .action, .dest, .dvc, .index, .linecount, .LinkId, .msg, .object, .object_category, .object_path, .punct, .registry_path, .registry_value_data, .registry_value_type, .source, .sourcetype, .splunk_server, .tag, .["tag::eventtype"], .user, .user_type, .vendor_action, .vendor_status)' botsv1.winregistry.json > botesv1.winregistry.json
```

### XmlWinEventLog:Microsoft Windows Sysmon Operational

| Field Name          | Reason                                                                               |
| ------------------- | ------------------------------------------------------------------------------------ |
| action              | Duplicate of "Keywords" field.                                                       |
| cmdline             | Duplicate of "CommandLine" field.                                                    |
| dest                | Duplicate of "Computer" field.                                                       |
| dest\_host          | Duplicate of "DestinationHostname" field.                                            |
| dest\_ip            | Duplicate of "DestinationIp" field.                                                  |
| dest\_port          | Duplicate of "DestinationPort" field.                                                |
| direction           | Information derivated from logs but not accurate.                                    |
| eventtype           | Duplicate of "EventChannel" field.                                                   |
| file\_create\_time  | Duplicate of "CreationUtcTime" field.                                                |
| file\_path          | Duplicate of "TargetFilename" field.                                                 |
| hashes              | Collision with "Hashes" field. Duplicate of "Hashes" field.                          |
| host                | Duplicate of "dvc\_nt\_host" field.                                                  |
| object\_category    | Not usefull derivated field. Can be obtain with Logstash if needed.                  |
| parent\_process     | Duplicate of "ParentImage" field.                                                    |
| parent\_process\_id | Duplicate of "ParentProcessId" field.                                                |
| process\_id         | Duplicate of "ProcessId" field.                                                      |
| protocol            | Collision with "Protocol" field. Duplicate of "(Source\|Destination)PortName" field. |
| session\_id         | Duplicate of "ProcessGuid" field.                                                    |
| signature           | Collision with "Signature" field. Duplicate of "EventDescription" field.             |
| signature\_id       | Duplicate of "EventCode" field.                                                      |
| src                 | Duplicate of "Computer" field.                                                       |
| src\_host           | Duplicate of "SourceHostname" field.                                                 |
| src\_ip             | Duplicate of "SourceIp" field.                                                       |
| src\_port           | Duplicate of "SourcePort" field.                                                     |
| transport           | Duplicate of "Protocol" field.                                                       |
| user                | Collision with "User" field. Duplicate of "User" field.                              |
| vendor\_product     | Tags will be set with Logstash.                                                      |

xmlwineventlog: Microsoft Windows Sysmon Operational jq delete command :

```
jq -c 'del(._si, ._cd, ._bkt, ._indextime, ._serial, ._eventtype_color, ._sourcetype, ._subsecond, .action, .app, .cmdline, .date, .date_hour, .date_mday, .date_minute, .date_month, .date_second, .date_wday, .date_year, .dest, .dest_host, .dest_ip, .dest_port, .direction, .dvc, .eventtype, .file_create_time, .file_path, .hashes, .host, .index, .linecount, .object_category, .parent_process, .parent_process_id, .process_id, .protocol, .punct, .session_id, .signature, .signature_id, .source, .sourcetype, .splunk_server, .src, .src_host, .src_ip, .src_port,  .tag, .["tag::eventtype"], .timeendpos, .timestamp, .timestartpos, .transport, .user, .vendor_product)' botsv1.XmlWinEventLog-Microsoft-Windows-Sysmon-Operational.json > botesv1.XmlWinEventLog-Microsoft-Windows-Sysmon-Operational.json
```