Data Sanitization

This page contains information about sanitization process. Tables of which fields have been deleted and the reason are also in this page.

Datas have to be cleaned before ingestion, some fields are deleted for this purpose. These fields were specific to Splunk, duplicated, reproductible with Logstash if needed or not usefull.

Note that no lines containing data have been deleted. The line number of the BOTES dataset is therefore the same as that of the BOTS dataset.

This section contains lists referencing global fields deleted for all Datasets/Source types and specfic fields deleted for each Dataset/Sourcetype.

Datasets sanitization

Step 1

Each BOTS JSON data are nested in "result". First step is to bring data up to the first level, for more convinience. Data initially at the first level will not be kept because they are not useful :

sed -i -r 's/\{\"preview\"\:false\,\"offset\"\:[0-9]+\,\"result\"\://g' $JSONfile
sed -i -r 's/\{"preview":false,"lastrow":true\}//g' $JSONfile
sed -i -r 's/\}$//g' $JSONfile

or to modify all file in same time :

for file in ./*.json; do sed -i -r 's/\{\"preview\"\:false\,\"offset\"\:[0-9]+\,\"result\"\://g' "$file"; sed -i -r 's/\{"preview":false,"lastrow":true\}//g' $file; sed -i -r 's/\}$//g' $file; done

Step 2

Second step is to list unique fields for each JSON file. The goal is to sort and then delete duplicated or useless fields, to finally establish a list of all unique fields and match them to an ECS field.

Sort unique fields by launching the following command for each JSON file :

jq 'keys' botsv1.$dataset.json | sort -u

Reminder : jq need to be installed for this and the following steps. Check prerequisites page if it's not already done.

Result from commands on each JSON files are put in separate table to be sorted and cleaned on next step.

Step 3

Some values can be found twice or three times in different original fields. In this case, fields created/derivated from Splunk parsing are deleted and fieldsname from the original logs are kept.

Example :

{... "dest":"8.8.8.8",
"dest_ip":"8.8.8.8","dest_mac":"08:5B:0E:93:92:AF","dest_port":"53", ...
"_raw": ... \"dest_ip\":\"8.8.8.8\",\"dest_mac\":\"08:5B:0E:93:92:AF\" ...}

"dest" and dest_ip" fields contain the same value. The raw log indicate that the "dest_ip" is the true original field, so in this case "dest_ip" will be kept and "dest" deleted.

Some values are deleted because of bad parsing or because it's easier to set them directly at Logstash level (tags, vendor, etc.).

Example :

{... "cs_uri_query":"searchword=&n990136=v920580&ordering=newest&searchphrase=all&areas[0]=categories",
... "host":"we1149srv","n990136":"v920580","ordering":"newest", ...}

In this example, the field "cs_uri_query" is not correctly parsed because of "=" in the value and field "n990136" is created when it shouldn't be.

Tables clean-up actions :

• Fields that are deleted are removed from table(s) created on the previous step.

• Fields that are deleted are add in jq delete list (Lists are at the end of each Datasets table).

• All kept fields are then matched with an ECS field and the corresponding type, fieldset, level and description.

• If no ECS Fieldset can be used to match an original field, a new fieldset is created or a new field in an existing ECS fieldset, with the 'botes' prefix, is created.

Refer to section BOTES Fields for Excel and CSV files containing cleaned Datastes fields.

Step 4

For each JSON files, delete choosen fields with jq and lists established in previous step.

Example :

jq -c 'del(._si, ._cd, ._bkt, ._indextime, ._serial, ._eventtype_color, ._sourcetype, 
._subsecond, .app, .date, .date_hour, .date_mday, .date_minute, .date_month, 
.date_second, .date_wday, .date_year, .dest, .dvc, .host, .index, .linecount, .punct, 
.source, .sourcetype, .splunk_server, .src, .tag, .["tag::eventtype"], .timeendpos, 
.timestamp, .timestartpos)' botsv1.stream-ip.json > botesv1.stream-ip.json

Step 5

Rename "_time" and "_raw" fields to avoid issues during Logstash ingestion :

sed -i -r 's/\"\_time\"/\"time\"/g' $botesv1.*.json
sed -i -r 's/\"\_raw\"/\"message\"/g' $botesv1.*.json

Step 6

For fields with type "Long", remove quote on the JSON files or they will be interpreted as Strings by Elasticsearch.

sed -r -i 's/"(datapackets_in|missing_packets_in|crscore|bytes_out|rcvdbyte|flow.bytes_toclient|bytes_in|tranport|rcvdpkt|flow.pkts_toclient|packets_in|dstport|dest_port|ports{}.port|DestinationPort|lease|ip_lease_time|total|used|response_time|dns.ttl|ttl{}|sn|RecordNumber|RecordID|fileinfo.size|filesize|filesize{}|refused|http.length|http_content_length|sc_substatus|time_taken|http.status|sc_status|status|status{}|checksum|icmp_code|code|sequence|id|icmp_type|type|folder_id|hostcount|host_id|object_id|count|sid|severity_index|user_permissions|vuln_index|fragment_count|tos|version|duration|flow.age|bytes|packets|pid|data_packets_out|missing_packets_out|s_port|request_call_id|search_attributes|sentbyte|flow.bytes_toserver|flow.pkts_toserver|vendor_transport|sentpkt|packets_out|srcport|src_port|Port|Source_Port|SourcePort|stats.capture.kernel_drops|stats.capture.kernel_packets|stats.decoder.avg_pkt_size|stats.decoder.bytes|stats.decoder.erspan|stats.decoder.ethernet|stats.decoder.gre|stats.decoder.icmpv4|stats.decoder.icmpv6|stats.decoder.invalid|stats.decoder.ipraw.invalid_ip_version|stats.decoder.ipv4|stats.decoder.ipv4_in_ipv6|stats.decoder.ipv6|stats.decoder.ipv6_in_ipv6|stats.decoder.ltnull.pkt_too_small|stats.decoder.ltnull.unsupported_type|stats.decoder.max_pkt_size|stats.decoder.mpls|stats.decoder.null|stats.decoder.pkts|stats.decoder.ppp|stats.decoder.pppoe|stats.decoder.raw|stats.decoder.sctp|stats.decoder.sll|stats.decoder.tcp|stats.decoder.teredo|stats.decoder.udp|stats.decoder.vlan|stats.decoder.vlan_qinq|stats.defrag.ipv4.fragments|stats.defrag.ipv4.reassembled|stats.defrag.ipv4.timeouts|stats.defrag.ipv6.fragments|stats.defrag.ipv6.reassembled|stats.defrag.ipv6.timeouts|stats.defrag.max_frag_hits|stats.detect.alert|stats.dns.memcap_global|stats.dns.memcap_state|stats.dns.memuse|stats.flow.emerg_mode_entered|stats.flow.emerg_mode_over|stats.flow.memcap|stats.flow.memuse|stats.flow.spare|stats.flow.tcp_reuse|stats.flow_mgr.closed_pruned|stats.flow_mgr.est_pruned|stats.flow_mgr.new_pruned|stats.http.memcap|stats.http.memuse|stats.stream.3whs_ack_in_wrong_dir|stats.stream.3whs_async_wrong_seq|stats.stream.3whs_right_seq_wrong_ack_evasion|stats.tcp.invalid_checksum|stats.tcp.memuse|stats.tcp.no_flow|stats.tcp.pseudo|stats.tcp.pseudo_failed|stats.tcp.reassembly_gap|stats.tcp.reassembly_memuse|stats.tcp.rst|stats.tcp.segment_memcap_drop|stats.tcp.sessions|stats.tcp.ssn_memcap_drop|stats.tcp.stream_depth_reached|stats.tcp.syn|stats.tcp.synack|stats.uptime|countav|countapp|countips|policyid|alert.gid|alert.rev|countweb|ssl_cipher_id|ssl_publickey_bit_len|Revived_Cache|Saved_Cache|Logon_Type|Nominal_Frequency__MHz|Key_Length|OpCode|Opcode|Maximum_performance_percentage|Minimum_performance_percentage|Restricted_SID_Count|Security_Error|Session_ID|Version|Task|TerminalSessionId|Minimum_throttle_percentage|Token_Elevation_Type_id|New_State|Number_of_Elements|Report_Status|sc_win32_status)":"([+-]?[0-9]+([.][0-9]+)?)"/"\1":\2/g' botesv1.*.json

Step 7

Modifications specific to Data sourcetype are needed too.

Stream-DNS

sed -r -i 's/"(host_addr\{\})":"(([0-9]{1,3}.){3}([0-9]{1,3}))"/"\1":["\2"]/g' botesv1.stream-dns.json

Global removed fields

Reason	Field Name
_si	Splunk specific.
_cd	Splunk specific. (Event address in index)
_bkt	Splunk specific. (Bucket ID)
_serial	Splunk specific.
_eventtype_color	Splunk specific.
_sourcetype	Event sourcetype be added by Logstash.
_subsecond	Already in contained in "_time" field. Can be obtain with Logstash if needed.
app	Inaccurate value or can be obtain with Logstash if needed.
date	Already in contained in "_time" field. Can be obtain with Logstash if needed.
date_hour	Splunk default datetime fields.
date_mday	Splunk default datetime fields.
date_minute	Splunk default datetime fields.
date_month	Splunk default datetime fields.
date_second	Splunk default datetime fields.
date_wday	Splunk default datetime fields.
date_year	Splunk default datetime fields.
dvc	Duplicated value.
index	Splunk specific. (Index name)
linecount	Splunk specific. (Current log line count)
punct	Splunk specific. (Punctuation pattern for an event)
source	Event source will be added by Logstash.
sourcetype	Event sourcetype be added by Logstash.
splunk_server	Splunk specific. (Splunk server name)
tag	Event will be tagged by Logstash.
tag::eventtype	Event will be tagged by Logstash.
time	Already in contained in "_time" field. Can be obtain with Logstash if needed.
timeendpos	Confusing with other fields.
timestamp	Duplicate of "time" field.
timestartpos	Confusing with other fields.

Fgt removed fields

fgt_event

Field Name	Reason
action
category
command
cpu
cpu_load_percent
dest
disk
disklograte
duration
fams_pause
fazlograte
limit
mem
object_category
product_version
session_id
setuprate
totalsession
src
vd
vendor_eventtype
vendor_status

fgt_event jq delete command :

jq -c 'del(._si, ._cd, ._bkt, ._indextime, ._serial, ._eventtype_color, ._sourcetype, .action, .category, .command, .cpu, .cpu_load_percent, .date, .date_hour, .date_mday, .date_minute, .date_month, .date_second, .date_wday, .date_year, .dest, .disk, .disklograte, .duration, .dvc, .fams_pause, .fazlograte, .index, .linecount, .limit, .mem, .object_category, .product_version, .session_id, .source, .sourcetype, .splunk_server, .src, .setuprate, .totalsession, .tag, .["tag::eventtype"], .time, .timeendpos, .timestartpos, .vd, .vendor_eventtype, .vendor_status)' botsv1.fgt_event.json > botesv1.fgt_event.json

fgt_traffic

Field Name	Reason
action
bytes_in
bytes_out
category
dest
dest_interface
dest_port
dest_translated_ip
dest_translated_port
dstcountry
ftnt_action
packets_in
packets_out
product_version
rule
rule_id
session_id
srccountry
src_interface
src_ip
src_port
src_mac
src_translated_ip
src
vd
vendor_eventtype

fgt_traffic jq delete command :

jq -c 'del(._si, ._cd, ._bkt, ._indextime, ._serial, ._eventtype_color, ._sourcetype, .action, .bytes_in, .bytes_out, .category, .date, .date_hour, .date_mday, .date_minute, .date_month, .date_second, .date_wday, .date_year, .dest, .dest_interface, .dest_port, .dest_translated_ip, .dest_translated_port, .dstcountry, .dvc, .ftnt_action, .index, .linecount, .product_version, .packets_in, .packets_out, .rule, .rule_id, .srccountry, .src_interface, .src_ip, .src_port, .src_mac, .src_translated_ip, .session_id, .source, .sourcetype, .splunk_server, .src, .tag, .["tag::eventtype"], .time, .timeendpos, .timestartpos, .vd, .vendor_eventtype)' botsv1.fgt_traffic.json > botesv1.fgt_traffic.json

fgt_utm

Field Name	Reason
action
agent
bytes_in
bytes_out
category
dest
dest_interface
dest_port
file_name
ftnt_action
http_method
product_version
session_id
signature
site
src
src_interface
src_port
srccountry
status
vd
vendor_eventtype

fgt_utm jq delete command :

jq -c 'del(._si, ._cd, ._bkt, ._indextime, ._serial, ._eventtype_color, ._sourcetype, .action, .agent, .bytes_in, .bytes_out, .category, .date, .date_hour, .date_mday, .date_minute, .date_month, .date_second, .date_wday, .date_year, .dest, .dest_interface, .dest_port, .dvc, .file_name, .ftnt_action, .http_method, .index, .linecount, .product_version, .session_id, .source, .sourcetype, .splunk_server, .signature, .site, .src_interface, .src_port, .srccountry, .status, .src, .tag, .["tag::eventtype"], .time, .timeendpos, .timestartpos, .vd, .vendor_eventtype)' botsv1.fgt_utm.json > botesv1.fgt_utm.json

IIS removed fields

IIS

Field Name	Reason
_kv	Bad original "cs_uri_query" or "cs_referer" field parsing.
a	Bad original "cs_uri_query" or "cs_referer" field parsing.
a1f5ea945d8863b612f9488485969e4	Bad original "cs_uri_query" or "cs_referer" field parsing.
action	Bad original "cs_uri_query" or "cs_referer" field parsing.
ADMINTYPE	Bad original "cs_uri_query" or "cs_referer" field parsing.
allowedDomain	Bad original "cs_uri_query" or "cs_referer" field parsing.
ApHost	Bad original "cs_uri_query" or "cs_referer" field parsing.
aspxerrorpath	Bad original "cs_uri_query" or "cs_referer" field parsing.
btnSubmit	Bad original "cs_uri_query" or "cs_referer" field parsing.
BVw	Bad original "cs_uri_query" or "cs_referer" field parsing.
cache_timeout	Bad original "cs_uri_query" or "cs_referer" field parsing.
catid	Bad original "cs_uri_query" or "cs_referer" field parsing.
cd	Bad original "cs_uri_query" or "cs_referer" field parsing.
CGIAlias	Bad original "cs_uri_query" or "cs_referer" field parsing.
cmd	Bad original "cs_uri_query" or "cs_referer" field parsing.
config	Bad original "cs_uri_query" or "cs_referer" field parsing.
culture	Bad original "cs_uri_query" or "cs_referer" field parsing.
debug	Bad original "cs_uri_query" or "cs_referer" field parsing.
depth	Bad original "cs_uri_query" or "cs_referer" field parsing.
dest	Bad original "cs_uri_query" or "cs_referer" field parsing.
do	Bad original "cs_uri_query" or "cs_referer" field parsing.
dTw	Bad original "cs_uri_query" or "cs_referer" field parsing.
ei	Bad original "cs_uri_query" or "cs_referer" field parsing.
eid	Bad original "cs_uri_query" or "cs_referer" field parsing.
f	Bad original "cs_uri_query" or "cs_referer" field parsing.
feed	Bad original "cs_uri_query" or "cs_referer" field parsing.
file	Bad original "cs_uri_query" or "cs_referer" field parsing.
File	Bad original "cs_uri_query" or "cs_referer" field parsing.
format	Bad original "cs_uri_query" or "cs_referer" field parsing.
full	Bad original "cs_uri_query" or "cs_referer" field parsing.
ge	Bad original "cs_uri_query" or "cs_referer" field parsing.
GET	Bad original "cs_uri_query" or "cs_referer" field parsing.
get_data	Bad original "cs_uri_query" or "cs_referer" field parsing.
group	Bad original "cs_uri_query" or "cs_referer" field parsing.
gzip	Bad original "cs_uri_query" or "cs_referer" field parsing.
highlighterId	Bad original "cs_uri_query" or "cs_referer" field parsing.
hk	Bad original "cs_uri_query" or "cs_referer" field parsing.
id	Bad original "cs_uri_query" or "cs_referer" field parsing.
iN	Bad original "cs_uri_query" or "cs_referer" field parsing.
item	Bad original "cs_uri_query" or "cs_referer" field parsing.
Itemid	Bad original "cs_uri_query" or "cs_referer" field parsing.
jQuery	Bad original "cs_uri_query" or "cs_referer" field parsing.
layout	Bad original "cs_uri_query" or "cs_referer" field parsing.
link	Bad original "cs_uri_query" or "cs_referer" field parsing.
module	Bad original "cs_uri_query" or "cs_referer" field parsing.
movieName	Bad original "cs_uri_query" or "cs_referer" field parsing.
n990136	Bad original "cs_uri_query" or "cs_referer" field parsing.
name	Bad original "cs_uri_query" or "cs_referer" field parsing.
operator	Bad original "cs_uri_query" or "cs_referer" field parsing.
option	Bad original "cs_uri_query" or "cs_referer" field parsing.
ordering	Bad original "cs_uri_query" or "cs_referer" field parsing.
page	Bad original "cs_uri_query" or "cs_referer" field parsing.
param	Bad original "cs_uri_query" or "cs_referer" field parsing.
playerready	Bad original "cs_uri_query" or "cs_referer" field parsing.
plugin	Bad original "cs_uri_query" or "cs_referer" field parsing.
Pp	Bad original "cs_uri_query" or "cs_referer" field parsing.
print	Bad original "cs_uri_query" or "cs_referer" field parsing.
pZ	Bad original "cs_uri_query" or "cs_referer" field parsing.
q	Bad original "cs_uri_query" or "cs_referer" field parsing.
QA	Bad original "cs_uri_query" or "cs_referer" field parsing.
qH	Bad original "cs_uri_query" or "cs_referer" field parsing.
rct	Bad original "cs_uri_query" or "cs_referer" field parsing.
rdoSearch	Bad original "cs_uri_query" or "cs_referer" field parsing.
redirects	Bad original "cs_uri_query" or "cs_referer" field parsing.
report	Bad original "cs_uri_query" or "cs_referer" field parsing.
returnto	Bad original "cs_uri_query" or "cs_referer" field parsing.
rev	Bad original "cs_uri_query" or "cs_referer" field parsing.
route	Bad original "cs_uri_query" or "cs_referer" field parsing.
rurl	Bad original "cs_uri_query" or "cs_referer" field parsing.
sa	Bad original "cs_uri_query" or "cs_referer" field parsing.
search	Bad original "cs_uri_query" or "cs_referer" field parsing.
searchphrase	Bad original "cs_uri_query" or "cs_referer" field parsing.
searchword	Bad original "cs_uri_query" or "cs_referer" field parsing.
section	Bad original "cs_uri_query" or "cs_referer" field parsing.
sections	Bad original "cs_uri_query" or "cs_referer" field parsing.
self_a	Bad original "cs_uri_query" or "cs_referer" field parsing.
selfor	Bad original "cs_uri_query" or "cs_referer" field parsing.
sig2	Bad original "cs_uri_query" or "cs_referer" field parsing.
skip	Bad original "cs_uri_query" or "cs_referer" field parsing.
sl	Bad original "cs_uri_query" or "cs_referer" field parsing.
src	Bad original "cs_uri_query" or "cs_referer" field parsing.
tab	Bad original "cs_uri_query" or "cs_referer" field parsing.
task	Bad original "cs_uri_query" or "cs_referer" field parsing.
template	Bad original "cs_uri_query" or "cs_referer" field parsing.
test	Bad original "cs_uri_query" or "cs_referer" field parsing.
time	Bad original "cs_uri_query" or "cs_referer" field parsing.
title	Bad original "cs_uri_query" or "cs_referer" field parsing.
tl	Bad original "cs_uri_query" or "cs_referer" field parsing.
tmpl	Bad original "cs_uri_query" or "cs_referer" field parsing.
txtSearchname	Bad original "cs_uri_query" or "cs_referer" field parsing.
type	Bad original "cs_uri_query" or "cs_referer" field parsing.
u	Bad original "cs_uri_query" or "cs_referer" field parsing.
uploadifyID	Bad original "cs_uri_query" or "cs_referer" field parsing.
url	Bad original "cs_uri_query" or "cs_referer" field parsing.
usg	Bad original "cs_uri_query" or "cs_referer" field parsing.
ved	Bad original "cs_uri_query" or "cs_referer" field parsing.
view	Bad original "cs_uri_query" or "cs_referer" field parsing.
window_x	Bad original "cs_uri_query" or "cs_referer" field parsing.
wNO	Bad original "cs_uri_query" or "cs_referer" field parsing.
wvstest	Bad original "cs_uri_query" or "cs_referer" field parsing.
wVz	Bad original "cs_uri_query" or "cs_referer" field parsing.
xC	Bad original "cs_uri_query" or "cs_referer" field parsing.
xmlDataPath	Bad original "cs_uri_query" or "cs_referer" field parsing.
xml_stylesheet	Bad original "cs_uri_query" or "cs_referer" field parsing.
yqQA5	Bad original "cs_uri_query" or "cs_referer" field parsing.

IIS jq delete command :

jq -c 'del(._si, ._cd, ._bkt, ._indextime, ._serial, ._eventtype_color, ._sourcetype, ._subsecond, .a, .a1f5ea945d8863b612f9488485969e4, .action, .ADMINTYPE, .allowedDomain, .ApHost, .app, .aspxerrorpath, .btnSubmit, .BVw, .cache_timeout, .catid, .cd, .CGIAlias, .cmd, .config, .culture, .date, .date_hour, .date_mday, .date_minute, .date_month, .date_second, .date_wday, .date_year, .dvc, .debug, .depth, .dest, .do, .dTw, .ei, .eid, .f, .feed, .file, .File, .format, .full, .ge, .GET, .get_data, .group, .gzip, .highlighterId, .hk, .index, .linecount, .id, .iN, .item, .Itemid, .jQuery, ._kv, .layout, .link, .module, .movieName, .n990136, .name, .operator, .option, .ordering, .page, .param, .playerready, .plugin, .Pp, .print, .pZ, .q, .QA, .qH, .rct, .rdoSearch, .redirects, .report, .returnto, .rev, .route, .rurl, .sa, .search, .searchphrase, .searchword, .section, .sections, .self_a, .selfor, .sig2, .skip, .sl, .source, .sourcetype, .splunk_server, .src, .tab, .tag, .["tag::eventtype"], .task, .template, .test, .time, .title, .tl, .tmpl, .txtSearchname, .type, .u, .uploadifyID, .url, .usg, .ved, .view, .window_x, .wNO, .wvstest, .wVz, .xC, .xmlDataPath, .xml_stylesheet, .yqQA5)' botsv1.iis.json > botesv1.iis.json

Nessus removed fields

Nessus:Scan

Field Name	Reason
dest	Duplicate of "host-ip" field.
dest_host	Duplicate of "hostname" field.
dest_ip	Duplicate of "host-ip" field.
eventtype	Event will be tagged/typed by Logstash.
product	Will be set with Logsash.
signature	Duplicate of "plugin_name" field.
vendor	Will be set with Logsash.

nessus:scan jq delete command :

jq -c 'del(._si, ._cd, ._bkt, ._indextime, ._serial, ._eventtype_color, ._sourcetype, ._subsecond, .app, .date, .date_hour, .date_mday, .date_minute, .date_month, .date_second, .date_wday, .date_year, .dest, .dest_host, .dest_ip, .dvc, .eventtype, .index, .linecount, .punct, .product, .source, .sourcetype, .splunk_server, .tag, .["tag::eventtype"], .timeendpos, .timestamp, .timestartpos, .vendor)' botsv1.nessus-scan.json > botesv1.nessus-scan.json

Stream removed fields

Stream:DHCP

Field Name	Reason
action
dest
lease_duration
signature
src

stream:dhcp jq delete command :

jq -c 'del(._si, ._cd, ._bkt, ._indextime, ._serial, ._eventtype_color, ._sourcetype, ._subsecond, .action, .app, .date, .date_hour, .date_mday, .date_minute, .date_month, .date_second, .date_wday, .date_year, .dest, .dvc, .index, .lease_duration, .linecount, .punct, .signature, .source, .sourcetype, .src, .splunk_server, .tag, .["tag::eventtype"], .timeendpos, .timestamp, .timestartpos)' botsv1.stream-dhcp.json > botesv1.stream-dhcp.json

Stream:DNS

Field Name	Reason
answer
dest
message_type{}
query{}
query_type{}
reply_code{}
response_time{}
src

stream:dns jq delete command :

jq -c 'del(._si, ._cd, ._bkt, ._indextime, ._serial, ._eventtype_color, ._sourcetype, ._subsecond, .app, .answer, .date, .date_hour, .date_mday, .date_minute, .date_month, .date_second, .date_wday, .date_year, .dest, .dvc, .index, .linecount, .["message_type{}"], .["query{}"], .["query_type{}"], .["reply_code{}"], .["response_time{}"], .source, .sourcetype, .src, .splunk_server, .tag, .["tag::eventtype"], .timeendpos, .timestamp, .timestartpos)' botsv1.stream-dns.json > botesv1.stream-dns.json

Stream:HTTP

Field Name	Reason
action
capture_hostname
client_rtt
client_rtt_packets
client_rtt_sum
data_center_time
dest
duplicate_packets_in
duplicate_packets_out
duration
form_data
host
reply_time
request_ack_time
request_time
response_ack_time
response_time
server_rtt
server_rtt_packets
server_rtt_sum
src

stream:http jq delete command :

jq -c 'del(._si, ._cd, ._bkt, ._indextime, ._serial, ._eventtype_color, ._sourcetype, ._subsecond, .action, .app, .capture_hostname, .client_rtt, .client_rtt_packets, .client_rtt_sum, .date, .date_hour, .date_mday, .date_minute, .date_month, .date_second, .date_wday, .date_year, .data_center_time, .dest, .duplicate_packets_in, .duplicate_packets_out, .duration, .dvc, .form_data, .host, .index, .linecount, .punct, .reply_time, .request_ack_time, .request_time, .response_ack_time, .response_time, .server_rtt, .server_rtt_packets, .server_rtt_sum, .source, .sourcetype, .splunk_server, .src, .tag, .["tag::eventtype"], .timeendpos, .timestamp, .timestartpos)' botsv1.stream-http.json > botesv1.stream-http.json

Stream:ICMP

Field Name	Reason
dest
duration
host
src

stream:icmp jq delete command :

jq -c 'del(._si, ._cd, ._bkt, ._indextime, ._serial, ._eventtype_color, ._sourcetype, ._subsecond, .app, .date, .date_hour, .date_mday, .date_minute, .date_month, .date_second, .date_wday, .date_year, .dest, .duration, .dvc, .host, .index, .linecount, .punct, .source, .sourcetype, .splunk_server, .src, .tag, .["tag::eventtype"], .timeendpos, .timestamp, .timestartpos)' botsv1.stream-icmp.json > botesv1.stream-icmp.json

Stream:IP

Field Name	Reason
dest
host
src

stream:ip jq delete command :

jq -c 'del(._si, ._cd, ._bkt, ._indextime, ._serial, ._eventtype_color, ._sourcetype, ._subsecond, .app, .date, .date_hour, .date_mday, .date_minute, .date_month, .date_second, .date_wday, .date_year, .dest, .dvc, .host, .index, .linecount, .punct, .source, .sourcetype, .splunk_server, .src, .tag, .["tag::eventtype"], .timeendpos, .timestamp, .timestartpos)' botsv1.stream-ip.json > botesv1.stream-ip.json

Stream:LDAP

Field Name	Reason
dest
host
src

stream:ldap jq delete command :

jq -c 'del(._si, ._cd, ._bkt, ._indextime, ._serial, ._eventtype_color, ._sourcetype, ._subsecond, .app, .date, .date_hour, .date_mday, .date_minute, .date_month, .date_second, .date_wday, .date_year, .dest, .dvc, .host, .index, .linecount, .punct, .source, .sourcetype, .splunk_server, .src, .tag, .["tag::eventtype"], .timeendpos, .timestamp, .timestartpos)' botsv1.stream-ldap.json > botesv1.stream-ldap.json

Stream:MAPI

Field Name	Reason
dest
duration
host
reply_time
request_time
response_time
src

stream:mapi jq delete command :

jq -c 'del(._si, ._cd, ._bkt, ._indextime, ._serial, ._eventtype_color, ._sourcetype, ._subsecond, .app, .date, .date_hour, .date_mday, .date_minute, .date_month, .date_second, .date_wday, .date_year, .dest, .duration, .dvc, .host, .index, .linecount, .punct, .reply_time, .request_time, .response_time, .source, .sourcetype, .splunk_server, .src, .tag, .["tag::eventtype"], .timeendpos, .timestamp, .timestartpos)' botsv1.stream-mapi.json > botesv1.stream-mapi.json

Stream:SIP

Field Name	Reason
dest
duration
host
src
uri

stream:sip jq delete command :

jq -c 'del(._si, ._cd, ._bkt, ._indextime, ._serial, ._eventtype_color, ._sourcetype, ._subsecond, .app, .date, .date_hour, .date_mday, .date_minute, .date_month, .date_second, .date_wday, .date_year, .dest, .duration, .dvc, .host, .index, .linecount, .punct, .source, .sourcetype, .splunk_server, .src, .tag, .["tag::eventtype"], .timeendpos, .timestamp, .timestartpos, .uri)' botsv1.stream-sip.json > botesv1.stream-sip.json

Stream:SMB

Field Name	Reason
dest
duration
host
src

stream:smb jq delete command :

jq -c 'del(._si, ._cd, ._bkt, ._indextime, ._serial, ._eventtype_color, ._sourcetype, ._subsecond, .app, .date, .date_hour, .date_mday, .date_minute, .date_month, .date_second, .date_wday, .date_year, .dest, .duration, .dvc, .host, .index, .linecount, .punct, .source, .sourcetype, .splunk_server, .src, .tag, .["tag::eventtype"], .timeendpos, .timestamp, .timestartpos)' botsv1.stream-smb.json > botesv1.stream-smb.json

Stream:SNMP

Field Name	Reason
dest
duration
host
src

stream:snmp jq delete command :

jq -c 'del(._si, ._cd, ._bkt, ._indextime, ._serial, ._eventtype_color, ._sourcetype, ._subsecond, .app, .date, .date_hour, .date_mday, .date_minute, .date_month, .date_second, .date_wday, .date_year, .dest, .duration, .dvc, .host, .index, .linecount, .punct, .source, .sourcetype, .splunk_server, .src, .tag, .["tag::eventtype"], .timeendpos, .timestamp, .timestartpos)' botsv1.stream-snmp.json > botesv1.stream-snmp.json

Stream:TCP

Field Name	Reason
client_rtt
client_rtt_packets
client_rtt_sum
connection
dest
duplicate_packets_in
duplicate_packets_out
duration
server_rtt
server_rtt_packets
server_rtt_sum
src
ssl_end_time
ssl_hash
ssl_serialnumber
ssl_session_id
ssl_start_time

stream:tcp jq delete command :

jq -c 'del(._si, ._cd, ._bkt, ._indextime, ._serial, ._eventtype_color, ._sourcetype, ._subsecond, .app, .client_rtt, .client_rtt_packets, .client_rtt_sum, .connection, .date, .date_hour, .date_mday, .date_minute, .date_month, .date_second, .date_wday, .date_year, .dest, .duplicate_packets_in, .duplicate_packets_out, .duration, .dvc, .host, .index, .linecount, .punct, .server_rtt, .server_rtt_packets, .server_rtt_sum, .source, .sourcetype, .splunk_server, .src, .ssl_end_time, .ssl_hash, .ssl_serialnumber, .ssl_session_id, .ssl_start_time, .tag, .["tag::eventtype"], .timeendpos, .timestamp, .timestartpos)' botsv1.stream-tcp.json > botesv1.stream-tcp.json

Suricata removed fields

Suricata

Field Name	Reason
action
alert_gid
alert_rev
bytes
bytes_in
bytes_out
capture_kernel_drops
category
decoder_avg_pkt_size
decoder_bytes
decoder_erspan
decoder_ethernet
decoder_gre
decoder_icmpv4
decoer_icmpv6
decoder_invalid
decoder_ipraw_invalid_ip_version
decoder_ipv4
decoder_ipv4_in_ipv6
decoder_ipv6
decoder_ipv6_in_ipv6
decoder_ltnull_pkt_too_small
decoder_ltnull_unspported_type
decoder_max_pkt_size
decoder_mpls
decoder_null
decoder_pkts
decoder_ppp
decoder_pppoe
decoder_raw
decoder_sctp
decoder_ssl
decoder_tcp
decoder_teredo
decoder_udp
decoder_vlan
decoder_vlan_qinq
defrag_ipv4_fragments
defrag_ipv4_reassembled
defrag_ipv4_timeouts
defrag_ipv6_fragments
defrag_ipv6_reassembled
defrag_max_frag_hits
defrag_ipv6_timeouts
dest
detect_alert
dns_memcap_global
dns_memcap_state
dns_memuse
duration
dvc
endtime
filename
file_size
file_state
file_stored
file_tx_id
flow_emerg_mode_entered
flow_emerg_mode_over
flow_memcap
flow_memuse
flow_mgr_closed_pruned
flow_mgr_est_pruned
flow_mgr_new_pruned
flow_spare
flow_tcp_reuse
http_content_type
http_memcap
http_memuse
http_method
http_protocol
http_referrer
http_user_agent
message_type
packets_in
packets_out
query
reason
reply_code
severity
severity_id
signature
src
ssh_client_software
ssh_client_version
ssh_server_software
ssh_server_version
ssl_issuer_common_name
ssl_publickey
ssl_server_name_indication
ssl_subject_common_name
ssl_version
starttime
state
status
stream_3whs_ack_in_wrong_dir
stream_3whs_async_wrong_seq
stream_3whs_right_seq_wrong_ack_evasion
suricata_signature_id
tcp_ack
tcp_cwr
tcp_ecn
tcp_fin
tcp_flag_hex
tcp_flag_hex_to_client
tcp_flag_hex_to_server
tcp_invalid_checksum
tcp_memuse
tcp_no_flow
tcp_pseudo
tcp_pseudo_failed
tcp_psh
tcp_reassembly_gap
tcp_reassembly_memuse
tcp_rst
tcp_segment_memcap_drop
tcp_sessions
tcp_ssn_memcap_drop
tcp_state
tcp_stream_depth_reached
tcp_syn
tcp_synack
transaction_id
transport
ttl
tx_id
uptime
url
vendor_gid
vendor_rev
vendor_sid

suricata jq delete command :

jq -c 'del(._si, ._cd, ._bkt, ._indextime, ._serial, ._eventtype_color, ._sourcetype, ._subsecond, .action, .alert_gid, .alert_rev, .app, .bytes, .bytes_in, .bytes_out, .capture_kernel_drops, .capture_kernel_packets, .category, .date, .date_hour, .date_mday, .date_minute, .date_month, .date_second, .date_wday, .date_year, .decoder_avg_pkt_size, .decoder_bytes, .decoder_erspan, .decoder_ethernet, .decoder_gre, .decoder_icmpv4, .decoer_icmpv6, .decoder_invalid, .decoder_ipraw_invalid_ip_version, .decoder_ipv4, .decoder_ipv4_in_ipv6, .decoder_ipv6, .decoder_ipv6_in_ipv6, .decoder_ltnull_pkt_too_small, .decoder_ltnull_unspported_type, .decoder_max_pkt_size, .decoder_mpls, .decoder_null, .decoder_pkts, .decoder_ppp, .decoder_pppoe, .decoder_raw, .decoder_sctp, .decoder_ssl, .decoder_tcp, .decoder_teredo, .decoder_udp, .decoder_vlan, .decoder_vlan_qinq, .defrag_ipv4_fragments, .defrag_ipv4_reassembled, .defrag_ipv4_timeouts, .defrag_ipv6_fragments, .defrag_ipv6_reassembled, .defrag_max_frag_hits, .dfrag_ipv6_timeouts, .dest, .detect_alert, .dns_memcap_global, .dns_memcap_state, .dns_memuse, .duration, .dvc, .endtime, .filename, .file_size, .file_state, .file_stored, .file_tx_id, .flow_emerg_mode_entered, .flow_emerg_mode_over, .flow_memcap, .flow_memuse, .flow_mgr_closed_pruned, .flow_mgr_est_pruned, .flow_mgr_new_pruned, .flow_spare, .flow_tcp_reuse, .http_content_type, .http_memcap, .http_memuse, .http_method, .http_protocol, .http_referrer, .http_user_agent, .message_type, .packets_in, .packets_out, .query, .reason, .reply_code, .index, .linecount, .punct, .severity, .severity_id, .signature, .source, .sourcetype, .splunk_server, .src, .ssh_client_software, .ssh_client_version, .ssh_server_software, .ssh_server_version, .ssl_issuer_common_name, .ssl_publickey, .ssl_server_name_indication, .ssl_subject_common_name, .ssl_version, .starttime, .state, .status, .stream_3whs_ack_in_wrong_dir, .stream_3whs_async_wrong_seq, .stream_3whs_right_seq_wrong_ack_evasion, .suricata_signature_id, .tcp_ack, .tcp_cwr, .tcp_ecn, .tcp_fin, .tcp_flag_hex, .tcp_flag_hex_to_client, .tcp_flag_hex_to_server, .tcp_invalid_checksum, .tcp_memuse, .tcp_no_flow, .tcp_pseudo, .tcp_pseudo_failed, .tcp_psh, .tcp_reassembly_gap, .tcp_reassembly_memuse, .tcp_rst, .tcp_segment_memcap_drop, .tcp_sessions, .tcp_ssn_memcap_drop, .tcp_state, .tcp_stream_depth_reached, .tcp_syn, .tcp_synack, .tag, .["tag::eventtype"], .timeendpos, .timestamp, .timestartpos, .transaction_id, .transport, .ttl, .tx_id, .uptime, .url, .vendor_gid, .vendor_rev, .vendor_sid)' botsv1.suricata.json > botesv1.suricata.json

WinEvent removed fields

WinEventLog:Application

Field Name	Reason
_pre_msg	Raw data already contain in _raw field.
Context	Bad original "Message" field parsing.
dest	Duplicate of "ComputerName" field.
Dirty_Shutdown	Bad original "Message" field parsing.
event_id	Duplicate of "RecordNumber" field.
eventtype	Collision with "EventType" field. Can be set with Logstash if needed.
host	Duplicate of "dvc_nt_host" field.
id	Duplicate of "RecordNumber" field.
Logon	One log. Not usefull in this Windows Event Application Log.
Message	Raw data already contain in _raw field.
severity	Duplicate of "Type" field. And not corresponding to log severity.
severity_id	Duplicate of "EventType" field. And not corresponding to log severity.
signature_id	Duplicate of "EventCode" field.
The_specified_object_...	Bad original "Message" field parsing.

wineventlog:application jq delete command :

jq -c 'del(._si, ._cd, ._bkt, ._indextime, ._pre_msg, ._serial, ._eventtype_color, ._sourcetype, ._subsecond, .app, .Context, .date, .date_hour, .date_mday, .date_minute, .date_month, .date_second, .date_wday, .date_year, .dest, .Dirty_Shutdown, .dvc, .event_id, .eventtype, .host, .id, .index, .linecount, .Message, .Logon, .punct, .severity, .severity_id, .signature_id, .source, .sourcetype, .splunk_server, .tag, .["tag::eventtype"], .The_specified_object_cannot_be_found__Specify_the_name_of_an_existing_object____HRESULT, .timeendpos, .timestamp, .timestartpos)' botsv1.WinEventLog-Application.json > botesv1.WinEventLog-Application.json

WinEventLog:Security

Field Name	Reason
_pre_msg	Raw data already contain in _raw field.
ACCESS_SYS_SEC	Bad parsing of "Access Reasons" field.
action	Duplicate of "Keywords" field.
AppendData__or_AddSubdirectory_or_...	Bad parsing of "Access Reasons" field.
body	Raw data already contain in _raw field.
DELETE	Bad parsing of "Access Reasons" field.
dest	Duplicate of "ComputerName" field.
dest_nt_domain	Duplicate of "Account_Domain" field.
dest_nt_host	Duplicate of "ComputerName" field.
event_id	Duplicate of "RecordNumber" field.
eventtype	Collision with "EventType" field. Can be set with Logstash if needed.
host	Duplicate of "dvc_nt_host" field.
id	Duplicate of "RecordNumber" field.
member_dn	Duplicate of "Account_Name" field.
member_id	Duplicate of "Security_ID" field.
member_nt_domain	Duplicate of "Account_Domain" field.
msad_action	Duplicate of "subject" field.
Message	Raw data already contain in _raw field.
name	Collision with "Name" field. Duplicate of "subject" field.
object	Will be set with Logsash.
privilege	Splunk generated field. Can be obtain with Logstash if needed.
privilege_id	Duplicate of "Privileges" field.
product	Will be set with Logsash.
ReadAttributes	Bad parsing of "Accesses" field.
READ_CONTROL	Bad parsing of "Accesses" field.
ReadEA	Bad parsing of "Accesses" field.
ReadData__or_ListDirectory_	Duplicate of "Access_Reasons" field.
session_id	Collision with "Session_ID" field. Duplicate of "Logon_ID" field.
severity	Duplicate of "Type" field. And not corresponding to log severity.
severity_id	Duplicate of "EventType" field. And not corresponding to log severity.
signature_id	Duplicate of "EventCode" field.
src	Duplicate of "ComputerName" field.
src_ip	Duplicate of "Source_Network_Address" field.
src_nt_host	Duplicate of "ComputerName" field.
src_port	Duplicate of "Source_Port" field.
src_nt_domain	Duplicate of "Account_Domain" field.
src_user	Duplicate of "Account_Name" field.
status	Duplicate of "Keywords" field.
SYNCHRONIZE	Bad parsing of "Access Reasons" field.
tag::app	Tags will be set with Logstash.
tag::privilege_id	Tags will be set with Logstash.
tag::Token_Elevation_Type_id	Tags will be set with Logstash.
user	Duplicate of "Account_Name" field.
user_group	Duplicate of "Group_Name" field.
vendor	Tags will be set with Logstash.
vendor_privilege	Duplicate of "Privileges" field.
WriteAttributes	Bad parsing of "Accesses" field.
WriteData__or_AddFile_	Bad parsing of "Accesses" field.
WriteEA	Bad parsing of "Accesses" field.

wineventlog:security jq delete command :

jq -c 'del(._si, ._cd, ._bkt, ._indextime, ._pre_msg, ._serial, ._eventtype_color, ._sourcetype, ._subsecond, .ACCESS_SYS_SEC, .action, .AppendData__or_AddSubdirectory_or_CreatePipeInstance_, .app, .body, .date, .date_hour, .date_mday, .date_minute, .date_month, .date_second, .date_wday, .date_year, .DELETE, .dest, .dest_nt_domain, .dest_nt_host, .dvc, .event_id, .eventtype, .id, .index, .linecount, .host, .member_dn, .member_id, .member_nt_domain, .msad_action, .Message, .name, .object, .privilege, .privilege_id, .product, .punct, .ReadData__or_ListDirectory_, .ReadAttributes, .READ_CONTROL, .ReadEA, .session_id, .severity, .severity_id, .signature, .signature_id, .source, .sourcetype, .splunk_server, .src, .src_ip, .src_nt_domain, .src_nt_host, .src_port, .src_user, .status, .SYNCHRONIZE, .tag, .["tag::app"], .["tag::eventtype"], .["tag::privilege_id"], .["tag::Token_Elevation_Type_id"], .timeendpos, .timestamp, .timestartpos, .user, .user_group, .vendor, .vendor_privilege, .WriteAttributes, .WriteData__or_AddFile_, .WriteEA)' botsv1.WinEventLog-Security.json > botesv1.WinEventLog-Security.json

WinEventLog:System

Field Name	Reason
_pre_msg	Raw data already contain in _raw field.
body	Raw data already contain in _raw field.
dest	Duplicate of "ComputerName" field.
dvc_nt_host	Duplicate of "Host_Name" field.
event_id	Duplicate of "RecordNumber" field.
eventtype	Collision with "EventType" field. Can be set with Logstash if needed.
host	Duplicate of "Host_Name" field.
id	Duplicate of "RecordNumber" field.
Message	Raw data already contain in _raw field.
package_title	Bad original "Message" field parsing. Can be obtain with Logstash if needed.
product	Will be set with Logsash.
severity	Duplicate of "Type" field. And not corresponding to log severity.
severity_id	Duplicate of "EventType" field. And not corresponding to log severity.
signature_id	Duplicate of "EventCode" field.
src	Duplicate of "ComputerName" field.
status	Duplicate of "Keywords" field.
user	Collision with "User" field. Duplicate of "User" field.
vendor	Tags will be set with Logstash.

wineventlog:system jq delete command :

jq -c 'del(._si, ._cd, ._bkt, ._indextime, ._pre_msg, ._serial, ._eventtype_color, ._sourcetype, ._subsecond, .app, .body, .date, .date_hour, .date_mday, .date_minute, .date_month, .date_second, .date_wday, .date_year, .dest, .dvc, .dvc_nt_host, .event_id, .eventtype, .index, .linecount, .host, .id, .Message, .package_title, .punct, .product, .severity, .severity_id, .signature_id, .source, .sourcetype, .splunk_server, .src, .tag, .["tag::eventtype"], .timeendpos, .timestamp, .timestartpos, .user, .vendor)' botsv1.WinEventLog-System.json > botesv1.WinEventLog-System.json

WinRegistry

Field Name	Reason
_kv
action
dest
LinkId
msg
object
object_category
object_path
registry_path
registry_value_data
registry_value_type
user
user_type
vendor_action
vendor_status

winregistry jq delete command :

jq -c 'del(._si, ._cd, ._bkt, ._indextime, ._kv, ._serial, ._eventtype_color, ._sourcetype, ._subsecond, .action, .dest, .dvc, .index, .linecount, .LinkId, .msg, .object, .object_category, .object_path, .punct, .registry_path, .registry_value_data, .registry_value_type, .source, .sourcetype, .splunk_server, .tag, .["tag::eventtype"], .user, .user_type, .vendor_action, .vendor_status)' botsv1.winregistry.json > botesv1.winregistry.json

XmlWinEventLog:Microsoft Windows Sysmon Operational

Field Name	Reason
action	Duplicate of "Keywords" field.
cmdline	Duplicate of "CommandLine" field.
dest	Duplicate of "Computer" field.
dest_host	Duplicate of "DestinationHostname" field.
dest_ip	Duplicate of "DestinationIp" field.
dest_port	Duplicate of "DestinationPort" field.
direction	Information derivated from logs but not accurate.
eventtype	Duplicate of "EventChannel" field.
file_create_time	Duplicate of "CreationUtcTime" field.
file_path	Duplicate of "TargetFilename" field.
hashes	Collision with "Hashes" field. Duplicate of "Hashes" field.
host	Duplicate of "dvc_nt_host" field.
object_category	Not usefull derivated field. Can be obtain with Logstash if needed.
parent_process	Duplicate of "ParentImage" field.
parent_process_id	Duplicate of "ParentProcessId" field.
process_id	Duplicate of "ProcessId" field.
protocol	Collision with "Protocol" field. Duplicate of "(Source|Destination)PortName" field.
session_id	Duplicate of "ProcessGuid" field.
signature	Collision with "Signature" field. Duplicate of "EventDescription" field.
signature_id	Duplicate of "EventCode" field.
src	Duplicate of "Computer" field.
src_host	Duplicate of "SourceHostname" field.
src_ip	Duplicate of "SourceIp" field.
src_port	Duplicate of "SourcePort" field.
transport	Duplicate of "Protocol" field.
user	Collision with "User" field. Duplicate of "User" field.
vendor_product	Tags will be set with Logstash.

xmlwineventlog: Microsoft Windows Sysmon Operational jq delete command :

jq -c 'del(._si, ._cd, ._bkt, ._indextime, ._serial, ._eventtype_color, ._sourcetype, ._subsecond, .action, .app, .cmdline, .date, .date_hour, .date_mday, .date_minute, .date_month, .date_second, .date_wday, .date_year, .dest, .dest_host, .dest_ip, .dest_port, .direction, .dvc, .eventtype, .file_create_time, .file_path, .hashes, .host, .index, .linecount, .object_category, .parent_process, .parent_process_id, .process_id, .protocol, .punct, .session_id, .signature, .signature_id, .source, .sourcetype, .splunk_server, .src, .src_host, .src_ip, .src_port,  .tag, .["tag::eventtype"], .timeendpos, .timestamp, .timestartpos, .transport, .user, .vendor_product)' botsv1.XmlWinEventLog-Microsoft-Windows-Sysmon-Operational.json > botesv1.XmlWinEventLog-Microsoft-Windows-Sysmon-Operational.json