Difference between revisions of "Define your own enrichment"
Jump to navigation
Jump to search
(→Introduction) |
|||
| Line 1: | Line 1: | ||
=== Introduction === | === Introduction === | ||
| − | Enrichment, as the word suggests, can be used to add context to the streaming data. At its '''simplest or basic level''', you can enrich the data at run time by referring to external | + | Enrichment, as the word suggests, can be used to add context to the streaming data. At its '''simplest or basic level''', you can enrich the data at run time by referring to an external CSV file. Some of the examples could be as under |
| − | * A csv file can contain information about the inventory (such as name of computer, location, owner, service tag) with name of computer as primary key. You can use this information to enrich the windows AD logs to add more context to the login information | + | * A csv file can contain information about the inventory (such as name of computer, location, owner, service tag etc) with name of computer as the primary key. You can use this information to enrich the windows AD logs to add more context to the login information. |
| − | * If you have a CSV database of | + | * If you have a CSV database of IP addresses with bad reputation where IP address is the primary key and country, city etc are the other columns, you can very well correlate this information to streaming firewall logs to enrich any communication with bad IPs because your firewall logs will have source and destination IP addresses which can be used for the lookup. |
| − | There could be several more examples | + | There could be several more examples of using static CSV based enrichment. You can change these CSV file dynamically and KHIKA will consume it immediately in real-time. |
More '''advanced and real cool''' things about enrichment is KHIKA's ability to build the CSV database from a streaming data source and being able to use it in other data source for enrichment. Using this ability, you can literally ''correlate or stitch'' the logs from different data sources at run time, provided they have a field in common. Some of the examples could be as under | More '''advanced and real cool''' things about enrichment is KHIKA's ability to build the CSV database from a streaming data source and being able to use it in other data source for enrichment. Using this ability, you can literally ''correlate or stitch'' the logs from different data sources at run time, provided they have a field in common. Some of the examples could be as under | ||
| − | * We can build IP and username database at | + | * We can build IP and username database at run-time using AD logs with IP address as the primary key. Further, this database can be referred in Linux logins where AD user can be enriched as Linux logs would have IP address of login workstation, but not the AD username. (Linux usernames are different from AD user names) |
* We can extract session ID, IP address from Web logs with session ID as primary key and use it to enrich the IP address in application logs which had session ID but not IP of the client. | * We can extract session ID, IP address from Web logs with session ID as primary key and use it to enrich the IP address in application logs which had session ID but not IP of the client. | ||
| − | Lets us walk through examples, starting with simple enrichment using static CSV files. | + | Lets us walk through a examples, starting with simple enrichment using static CSV files. |
| + | |||
| + | === Example of CSV Based Enrichment === | ||
| + | Please refer to section [[Data Enrichment in KHIKA|Data Enrichment in KHIKA]] for understanding how CSV based enrichment works in KHIKA. | ||
| + | |||
| + | === Example of Building Dynamic and Referring Dynamic Enrichment === | ||
Revision as of 11:08, 11 June 2019
Introduction
Enrichment, as the word suggests, can be used to add context to the streaming data. At its simplest or basic level, you can enrich the data at run time by referring to an external CSV file. Some of the examples could be as under
- A csv file can contain information about the inventory (such as name of computer, location, owner, service tag etc) with name of computer as the primary key. You can use this information to enrich the windows AD logs to add more context to the login information.
- If you have a CSV database of IP addresses with bad reputation where IP address is the primary key and country, city etc are the other columns, you can very well correlate this information to streaming firewall logs to enrich any communication with bad IPs because your firewall logs will have source and destination IP addresses which can be used for the lookup.
There could be several more examples of using static CSV based enrichment. You can change these CSV file dynamically and KHIKA will consume it immediately in real-time.
More advanced and real cool things about enrichment is KHIKA's ability to build the CSV database from a streaming data source and being able to use it in other data source for enrichment. Using this ability, you can literally correlate or stitch the logs from different data sources at run time, provided they have a field in common. Some of the examples could be as under
- We can build IP and username database at run-time using AD logs with IP address as the primary key. Further, this database can be referred in Linux logins where AD user can be enriched as Linux logs would have IP address of login workstation, but not the AD username. (Linux usernames are different from AD user names)
- We can extract session ID, IP address from Web logs with session ID as primary key and use it to enrich the IP address in application logs which had session ID but not IP of the client.
Lets us walk through a examples, starting with simple enrichment using static CSV files.
Example of CSV Based Enrichment
Please refer to section Data Enrichment in KHIKA for understanding how CSV based enrichment works in KHIKA.
