Define your own enrichment

Introduction

Enrichment, as the word suggests, can be used to add context to the streaming data. At its simplest or basic level, you can enrich the data at run time by referring to an external CSV file. Some of the examples could be as under

• A csv file can contain information about the inventory (such as name of computer, location, owner, service tag etc) with name of computer as the primary key. You can use this information to enrich the windows AD logs to add more context to the login information.
• If you have a CSV database of IP addresses with bad reputation where IP address is the primary key and country, city etc are the other columns, you can very well correlate this information to streaming firewall logs to enrich any communication with bad IPs because your firewall logs will have source and destination IP addresses which can be used for the lookup.

There could be several more examples of using static CSV based enrichment. You can change these CSV file dynamically and KHIKA will consume it immediately in real-time.

More advanced and real cool things about enrichment is KHIKA's ability to build the CSV database from a streaming data source and being able to use it in other data source for enrichment. Using this ability, you can literally correlate or stitch the logs from different data sources at run time, provided they have a field in common. Some of the examples could be as under

• We can build IP and username database at run-time using AD logs with IP address as the primary key. Further, this database can be referred in Linux logins where AD user can be enriched as Linux logs would have IP address of login workstation, but not the AD username. (Linux usernames are different from AD user names)
• We can extract session ID, IP address from Web logs with session ID as primary key and use it to enrich the IP address in application logs which had session ID but not IP of the client.

Lets us walk through a examples, starting with simple enrichment using static CSV files.

Example of CSV Based Enrichment

Please refer to section Data Enrichment in KHIKA for understanding how CSV based enrichment works in KHIKA.

Example of Building and Referring Dynamic Enrichment

You can build any number of primary-key based tables from any streaming KHIKA data source by selecting any key as the primary key. This key can have values associated with it from any any fields present in the records. This database can then be referred by any streaming KHIKA data source to match the key to any of the selected fields and values from the database can be used to enrich the message when a match is found, in real-time.

In the example below we'll build a database using Windows AD login logs. Primary key will be the IP Address of the workstation from where the login is happening and value would be username (WindowsUser). This way, we build the workstationIP to WindowsUser database. Consequently, this information will be referred in the Linux logs, wherein we will use the IP address from login message and match it with database to fetch the windows user which will be enriched in the Linux login message. This give us the AD user doing actual login in Linux which is a useful log correlation.

1. Login to KHIKA GUI as a customer (you must be admin of the customer)
2. Click "Configure" from side menu and click "Enrichment Rules"

<<<image>>

1. Click "Manage Lookup Database". We will first create the schema of the database

<<image>>

1. Click "Add Lookup"

<<image>>

1. Now on your local computer create create a simple csv file with header (and no values as values will be added dynamically from the streaming data). Let us add two columns separated by comma as shown :- tl_win_ip,tl_win_user. Save the file with name "IP_to_User_Mapping_Lookup.csv" and close

1. On KHIKA Web GUI, in Upload the file with name "IP_to_User_Mapping_Lookupc.csv"

<<image>>

1. The uploaded file should be visible in

1. Numbered list item