Data Sanitization

This page contains information about sanitization process. Tables of which fields have been deleted and the reason are also in this page.

Datas have to be cleaned before ingestion, some fields are deleted for this purpose. These fields were specific to Splunk, duplicated, reproductible with Logstash if needed or not usefull.

Note that no lines containing data have been deleted. The line number of the BOTES dataset is therefore the same as that of the BOTS dataset.

This section contains lists referencing global fields deleted for all Datasets/Source types and specfic fields deleted for each Dataset/Sourcetype.

Datasets sanitization

Step 1

Each BOTS JSON data are nested in "result". First step is to bring data up to the first level, for more convinience. Data initially at the first level will not be kept because they are not useful :

sed -i -r 's/\{\"preview\"\:false\,\"offset\"\:[0-9]+\,\"result\"\://g' $JSONfile
sed -i -r 's/\{"preview":false,"lastrow":true\}//g' $JSONfile
sed -i -r 's/\}$//g' $JSONfile

or to modify all file in same time :

for file in ./*.json; do sed -i -r 's/\{\"preview\"\:false\,\"offset\"\:[0-9]+\,\"result\"\://g' "$file"; sed -i -r 's/\{"preview":false,"lastrow":true\}//g' $file; sed -i -r 's/\}$//g' $file; done

Step 2

Second step is to list unique fields for each JSON file. The goal is to sort and then delete duplicated or useless fields, to finally establish a list of all unique fields and match them to an ECS field.

Sort unique fields by launching the following command for each JSON file :

jq 'keys' botsv1.$dataset.json | sort -u

Reminder : jq need to be installed for this and the following steps. Check prerequisites page if it's not already done.

Result from commands on each JSON files are put in separate table to be sorted and cleaned on next step.

Step 3

Some values can be found twice or three times in different original fields. In this case, fields created/derivated from Splunk parsing are deleted and fieldsname from the original logs are kept.

Example :

{... "dest":"8.8.8.8",
"dest_ip":"8.8.8.8","dest_mac":"08:5B:0E:93:92:AF","dest_port":"53", ...
"_raw": ... \"dest_ip\":\"8.8.8.8\",\"dest_mac\":\"08:5B:0E:93:92:AF\" ...}

"dest" and dest_ip" fields contain the same value. The raw log indicate that the "dest_ip" is the true original field, so in this case "dest_ip" will be kept and "dest" deleted.

Some values are deleted because of bad parsing or because it's easier to set them directly at Logstash level (tags, vendor, etc.).

Example :

{... "cs_uri_query":"searchword=&n990136=v920580&ordering=newest&searchphrase=all&areas[0]=categories",
... "host":"we1149srv","n990136":"v920580","ordering":"newest", ...}

In this example, the field "cs_uri_query" is not correctly parsed because of "=" in the value and field "n990136" is created when it shouldn't be.

Tables clean-up actions :

Fields that are deleted are removed from table(s) created on the previous step.
Fields that are deleted are add in jq delete list (Lists are at the end of each Datasets table).
All kept fields are then matched with an ECS field and the corresponding type, fieldset, level and description.
If no ECS Fieldset can be used to match an original field, a new fieldset is created or a new field in an existing ECS fieldset, with the 'botes' prefix, is created.

Refer to section BOTES Fields for Excel and CSV files containing cleaned Datastes fields.

Step 4

For each JSON files, delete choosen fields with jq and lists established in previous step.

Example :

jq -c 'del(._si, ._cd, ._bkt, ._indextime, ._serial, ._eventtype_color, ._sourcetype, 
._subsecond, .app, .date, .date_hour, .date_mday, .date_minute, .date_month, 
.date_second, .date_wday, .date_year, .dest, .dvc, .host, .index, .linecount, .punct, 
.source, .sourcetype, .splunk_server, .src, .tag, .["tag::eventtype"], .timeendpos, 
.timestamp, .timestartpos)' botsv1.stream-ip.json > botesv1.stream-ip.json

Step 5

Rename "_time" and "_raw" fields to avoid issues during Logstash ingestion :

sed -i -r 's/\"\_time\"/\"time\"/g' $botesv1.*.json
sed -i -r 's/\"\_raw\"/\"message\"/g' $botesv1.*.json

Step 6

For fields with type "Long", remove quote on the JSON files or they will be interpreted as Strings by Elasticsearch.

sed -r -i 's/"(datapackets_in|missing_packets_in|crscore|bytes_out|rcvdbyte|flow.bytes_toclient|bytes_in|tranport|rcvdpkt|flow.pkts_toclient|packets_in|dstport|dest_port|ports{}.port|DestinationPort|lease|ip_lease_time|total|used|response_time|dns.ttl|ttl{}|sn|RecordNumber|RecordID|fileinfo.size|filesize|filesize{}|refused|http.length|http_content_length|sc_substatus|time_taken|http.status|sc_status|status|status{}|checksum|icmp_code|code|sequence|id|icmp_type|type|folder_id|hostcount|host_id|object_id|count|sid|severity_index|user_permissions|vuln_index|fragment_count|tos|version|duration|flow.age|bytes|packets|pid|data_packets_out|missing_packets_out|s_port|request_call_id|search_attributes|sentbyte|flow.bytes_toserver|flow.pkts_toserver|vendor_transport|sentpkt|packets_out|srcport|src_port|Port|Source_Port|SourcePort|stats.capture.kernel_drops|stats.capture.kernel_packets|stats.decoder.avg_pkt_size|stats.decoder.bytes|stats.decoder.erspan|stats.decoder.ethernet|stats.decoder.gre|stats.decoder.icmpv4|stats.decoder.icmpv6|stats.decoder.invalid|stats.decoder.ipraw.invalid_ip_version|stats.decoder.ipv4|stats.decoder.ipv4_in_ipv6|stats.decoder.ipv6|stats.decoder.ipv6_in_ipv6|stats.decoder.ltnull.pkt_too_small|stats.decoder.ltnull.unsupported_type|stats.decoder.max_pkt_size|stats.decoder.mpls|stats.decoder.null|stats.decoder.pkts|stats.decoder.ppp|stats.decoder.pppoe|stats.decoder.raw|stats.decoder.sctp|stats.decoder.sll|stats.decoder.tcp|stats.decoder.teredo|stats.decoder.udp|stats.decoder.vlan|stats.decoder.vlan_qinq|stats.defrag.ipv4.fragments|stats.defrag.ipv4.reassembled|stats.defrag.ipv4.timeouts|stats.defrag.ipv6.fragments|stats.defrag.ipv6.reassembled|stats.defrag.ipv6.timeouts|stats.defrag.max_frag_hits|stats.detect.alert|stats.dns.memcap_global|stats.dns.memcap_state|stats.dns.memuse|stats.flow.emerg_mode_entered|stats.flow.emerg_mode_over|stats.flow.memcap|stats.flow.memuse|stats.flow.spare|stats.flow.tcp_reuse|stats.flow_mgr.closed_pruned|stats.flow_mgr.est_pruned|stats.flow_mgr.new_pruned|stats.http.memcap|stats.http.memuse|stats.stream.3whs_ack_in_wrong_dir|stats.stream.3whs_async_wrong_seq|stats.stream.3whs_right_seq_wrong_ack_evasion|stats.tcp.invalid_checksum|stats.tcp.memuse|stats.tcp.no_flow|stats.tcp.pseudo|stats.tcp.pseudo_failed|stats.tcp.reassembly_gap|stats.tcp.reassembly_memuse|stats.tcp.rst|stats.tcp.segment_memcap_drop|stats.tcp.sessions|stats.tcp.ssn_memcap_drop|stats.tcp.stream_depth_reached|stats.tcp.syn|stats.tcp.synack|stats.uptime|countav|countapp|countips|policyid|alert.gid|alert.rev|countweb|ssl_cipher_id|ssl_publickey_bit_len|Revived_Cache|Saved_Cache|Logon_Type|Nominal_Frequency__MHz|Key_Length|OpCode|Opcode|Maximum_performance_percentage|Minimum_performance_percentage|Restricted_SID_Count|Security_Error|Session_ID|Version|Task|TerminalSessionId|Minimum_throttle_percentage|Token_Elevation_Type_id|New_State|Number_of_Elements|Report_Status|sc_win32_status)":"([+-]?[0-9]+([.][0-9]+)?)"/"\1":\2/g' botesv1.*.json

Step 7

Modifications specific to Data sourcetype are needed too.

Stream-DNS

sed -r -i 's/"(host_addr\{\})":"(([0-9]{1,3}.){3}([0-9]{1,3}))"/"\1":["\2"]/g' botesv1.stream-dns.json

Global removed fields

Reason

Field Name

_si

Splunk specific.

_cd

Splunk specific. (Event address in index)

_bkt

Splunk specific. (Bucket ID)

_serial

Splunk specific.

_eventtype_color

Splunk specific.

_sourcetype

Event sourcetype be added by Logstash.

_subsecond

Already in contained in "_time" field. Can be obtain with Logstash if needed.

app

Inaccurate value or can be obtain with Logstash if needed.

date

Already in contained in "_time" field. Can be obtain with Logstash if needed.

date_hour

Splunk default datetime fields.

date_mday

Splunk default datetime fields.

date_minute

Splunk default datetime fields.

date_month

Splunk default datetime fields.

date_second

Splunk default datetime fields.

date_wday

Splunk default datetime fields.

date_year

Splunk default datetime fields.

dvc

Duplicated value.

index

Splunk specific. (Index name)

linecount

Splunk specific. (Current log line count)

punct

Splunk specific. (Punctuation pattern for an event)

source

Event source will be added by Logstash.

sourcetype

Event sourcetype be added by Logstash.

splunk_server

Splunk specific. (Splunk server name)

tag

Event will be tagged by Logstash.

tag::eventtype

Event will be tagged by Logstash.

time

Already in contained in "_time" field. Can be obtain with Logstash if needed.

timeendpos

Confusing with other fields.

timestamp

Duplicate of "time" field.

timestartpos

Confusing with other fields.

Fgt removed fields

fgt_event

Field Name

Reason

action

category

command

cpu

cpu_load_percent

dest

disk

disklograte

duration

fams_pause

fazlograte

limit

mem

object_category

product_version

session_id

setuprate

totalsession

src

vendor_eventtype

vendor_status

fgt_event jq delete command :

jq -c 'del(._si, ._cd, ._bkt, ._indextime, ._serial, ._eventtype_color, ._sourcetype, .action, .category, .command, .cpu, .cpu_load_percent, .date, .date_hour, .date_mday, .date_minute, .date_month, .date_second, .date_wday, .date_year, .dest, .disk, .disklograte, .duration, .dvc, .fams_pause, .fazlograte, .index, .linecount, .limit, .mem, .object_category, .product_version, .session_id, .source, .sourcetype, .splunk_server, .src, .setuprate, .totalsession, .tag, .["tag::eventtype"], .time, .timeendpos, .timestartpos, .vd, .vendor_eventtype, .vendor_status)' botsv1.fgt_event.json > botesv1.fgt_event.json

fgt_traffic

Field Name

Reason

action

bytes_in

bytes_out

category

dest

dest_interface

dest_port

dest_translated_ip

dest_translated_port

dstcountry

ftnt_action

packets_in

packets_out

product_version

rule

rule_id

session_id

srccountry

src_interface

src_ip

src_port

src_mac

src_translated_ip

src

vendor_eventtype

fgt_traffic jq delete command :

jq -c 'del(._si, ._cd, ._bkt, ._indextime, ._serial, ._eventtype_color, ._sourcetype, .action, .bytes_in, .bytes_out, .category, .date, .date_hour, .date_mday, .date_minute, .date_month, .date_second, .date_wday, .date_year, .dest, .dest_interface, .dest_port, .dest_translated_ip, .dest_translated_port, .dstcountry, .dvc, .ftnt_action, .index, .linecount, .product_version, .packets_in, .packets_out, .rule, .rule_id, .srccountry, .src_interface, .src_ip, .src_port, .src_mac, .src_translated_ip, .session_id, .source, .sourcetype, .splunk_server, .src, .tag, .["tag::eventtype"], .time, .timeendpos, .timestartpos, .vd, .vendor_eventtype)' botsv1.fgt_traffic.json > botesv1.fgt_traffic.json

fgt_utm

Field Name

Reason

action

agent

bytes_in

bytes_out

category

dest

dest_interface

dest_port

file_name

ftnt_action

http_method

product_version

session_id

signature

site

src

src_interface

src_port

srccountry

status

vendor_eventtype

fgt_utm jq delete command :

jq -c 'del(._si, ._cd, ._bkt, ._indextime, ._serial, ._eventtype_color, ._sourcetype, .action, .agent, .bytes_in, .bytes_out, .category, .date, .date_hour, .date_mday, .date_minute, .date_month, .date_second, .date_wday, .date_year, .dest, .dest_interface, .dest_port, .dvc, .file_name, .ftnt_action, .http_method, .index, .linecount, .product_version, .session_id, .source, .sourcetype, .splunk_server, .signature, .site, .src_interface, .src_port, .srccountry, .status, .src, .tag, .["tag::eventtype"], .time, .timeendpos, .timestartpos, .vd, .vendor_eventtype)' botsv1.fgt_utm.json > botesv1.fgt_utm.json

IIS removed fields