Data Enrichment in KHIKA
About Enrichment
KHIKA supports "live" enrichment of the raw logs based on predefined criteria. This means, you can add additional fields in the raw log message when certain pre-defined condition is met. Take example of firewall logs that has information about the source ip, destination ip, source port and destination port. Most of the times, the firewall message does not have information about the user or hostname from which the traffic is originated. Also, it does not have information about the reputation of the external IP address. One can import simple CSV files as simple databases in KHIKA, identify the primary key in each CSV database and start correlating it with in the incoming streams of messages by comparing the value of a particular field in the live message with the primary key in the defined database. When a match is found, KHIKA will enrich the message by adding additional key-value pairs from CSV record to the original raw message. Example: Firewall log looks something like this
date=01JAN2019 time=10:10:32IST src_ip=192.168.1.10 src_port=55331 dst_ip=80.53.57.146 dst_port=80,
date=01JAN2019 time=10:10:33IST src_ip=192.168.1.11 src_port=34512 dst_ip=209.33.5.1 dst_port=80
date=01JAN2019 time=10:10:33IST src_ip=192.168.2.9 src_port=553245 dst_ip=53.23.2.1 dst_port=22
date=01JAN2019 time=10:10:34IST src_ip=192.168.1.10 src_port=55331 dst_ip=80.53.57.146 dst_port=80
date=01JAN2019 time=10:10:35IST src_ip=192.168.1.12 src_port=44331 dst_ip=163.2.4.1 dst_port=53
I wish to check the reputation of each external IP address at run time by looking at the IP address reputation database (which is a CSV file) and enrich the original message with entries in the reputation database. The IP reputation database looks something like this.
The primary key of the CSV database is malicious_ip. I want to compare the value of each dst_ip from firewall log with the primary key of the reputation database and enrich the raw log with malicious_ip, malicious_source, malicious_count, malicious_confidence, country and city only when the match is found.
Enrichment of logs in KHIKA
You can do this in KHIKA using KHIKA's enrichment functionality. Below section explains step-by-step guide for performing this task. Login to KHIKA using admin credentials of the customer. Go to Configure --> Enrichment Rules
Click on "Manage Lookup Database" button
Click "Add Lookup" to add a new lookup database. (You may select existing lookup database as KHIKA ships a lot of useful lookups out of box.)
Select "Choose file" to browse to exact location of the CSVV file to be uploaded into KHIKA. Note that the CSV file must be present on your machine being used for the browser.
Select the file and press "Upload"
Select Primary Key as PRIMARY KEY : malicious_ip and click OK
malicious_ip_lookup database will get created. You can see it listed by clicking "manage lookup database" and examine it by clicking the info icon next to it
It will show all the information about the created lookup database, such as lookup name, primary key, columns etc.
After creating the lookup database, it is the time to start using it by referring to it in runtime. We have to define an "Enrichment Rule" in KHIKA to do this
Click "Add Rule" from the "Enrichment Rules" tab (in Configure option)
An Enrichment Rule has a name and it has to be written on a KHIKA data source (i.e. Event Class). In this example, we will create this rule for Fortigate Firewall.
Give Rule Name as " Fortigate_firewall_Reputation_check ", Select Fortigate Firewall as Event Class, Set Enrichment Type as "Refer", order number as "1" and to "2. Database Lookup" tab
In the "Database Lookup" tab, select the Database as "malicious_ip_lookup". Select reequired columns for the enrichment.
Click Submit.
This will create the enrichment rule which will :
- Load the malicious_ip file into KHIKA
- Start comparing the values of srcip and dstip from incoming messages with the malicous_ip values in lookup database (note the primary key of the lookup database was set to malicious_ip)
- When a match is found with either of dstp or scrip the raw message will be enriched with malicious_ip malicious_source malicious_count, malicious_confidence
You can examine your incoming data by going to the discover tab and searching by simple query "malicious_ip:*"
This was an overview of what is enrichment and how it works in KHIKA. For advanced use cases and more on enrichment you can refer Build and Refer Enrichment feature in KHIKA
The next section is about hardening monitoring, analysis and compliance by KHIKA
