Writing advanced adapters

After understanding Khika Data Format and going through the initial exercise of Writing you own KHIKA Data Adapters , it is the time to create a production level KHIKA Adapter. A few points to note here before we begin writing our own Adapter:

• Adapters are scripts that execute on KHIKA Data Aggregator
• Adapters can be written in any programming language (our favorite is python 2.7)
• Adapters are scheduled processes and KHIKA Data Aggregator is responsible for scheduling them to run at a periodic interval (typically 1 minute to 5 minutes)
• The Adapter scripts
  • read the raw log messages one-by-one (from source such as files, queues, APIs, Databases etc),
  • parse the log messages,
  • convert it in Khika Data Format
  • Write the output to stdout
• KHIKA Data Aggregator pipes the output of the Aggregator script and send it to KHIKA over a SSL connection

With these concepts in mind, let proceed with an example of a production ready KHIKA Data Adapter. Login to your KHIKA Data Aggregator node (default username/password is khika/khika123). We will study a syslog based adapter that processes the messages received from a PaloAlto Firewall. Open file TLHook_Adaptor_PaloAlto.py from directory /opt/KHIKA/Apps/Adapters/PaloAltoFW.

Check first few lines of this file where we import some important python modules

     1 #!/bin/env python
     2 import os, sys
     3 import socket
     4 import csv,StringIO
     5 import logging #The logging libraries
     6 import re #python inbuilt regular expressions library
     7 import time
     8 from time import strptime #Useful time format conversion functions
     9 from datetime import datetime
    10 import random
    11 import calendar
    12 from ipaddress import IPv4Network,IPv4Address
    13 import pdb

Note that we have imported logging and some time libraries. We have also imported python "regular expression" library Go the bottom of this file and locate function "__main__". This is start of execution of the code.