Monitoring a local file using OSSEC Integration

In addition to event logs or syslogs, a lot of local files are created by various applications. These files reside on the hosts/servers where applications run and contain wealth of information, valuable for both security and operational intelligence. The application logs are useful for debugging or capturing run time errors/exception or even business opportunities in production production environments. It is imperative to monitor local files for gaining actionable insights, real time alerting, correlations and forensic debugging.

KHIKA integrates closely with OSSEC to monitor application logs in real time. This section explains how to use OSSEC to monitor the application logs. We begin with broad level steps and then dive deep into each step so that we explain the methodology and the intricate details associated with it. At a broad level you perform following steps

Install an OSSEC Agent on the end node.

OSSEC Agent provides the simplest way to monitor a local file on any computer in real time. You must install the OSSEC agent where the file is being created. Please refer the appropriate section sections on Linux or Windows for installing the OSSEC Agent.

Configure the OSSEC Agent to monitor the local file

Once the OSSEC agent is installed, you need to locate the file you want to monitor. In this example, we will monitor apache access logs created in the directory /path/of/apache/access/log and name of the file is access.log

1. Login to OSSEC Agent node and open file ossec.conf. On Windows this file is located in C:\Program Files (x86)\ossec-agent directory . On Linux, you will find it at /vat/ossec/ossec-agent/etc directory. (if you have installed the agent in the default location)
2. Locate section <localfile>. Note that ossec.conf uses XML formatting and hence you have to be careful enough not to disturb the other tags. Navigate just above "Rootcheck - Policy monitor config" tag and insert following section

 <localfile>
   <location>/path/of/apache/access/log/access.log</location>
   <log_format>syslog</log_format>
 </localfile>

Note that we have added <log_format>syslog<log_format> as the apache access logs are single line messages and OSSEC will treat them same as syslog messages.

We save the file and restart OSSEC Agent. This will start apache access logs being received on the OSSEC Server side (i.e. on Data Aggregator Node).

Login to KHIKA Data Aggregator and navigate to directory /opt/ossec/logs/archives/2019/May/ (needless to say, enter current year and month). Here you will find a multiple directories with names of workspaces prepended to it. Enter appropriate directory and check the current file. You should see live logs coming from our newly added apache server

Parse the file using extensible KHIKA Adapter for OSSEC

As OSSEC Server on your KHIKA Data Aggregator starts receiving live logs, you will observe that all the logs are being gathered in the same file. If you have already configured OSSEC Adapter via KHIHKA App for Linux or KHIKA App for Windows, you will need to tell this Adapter to parse the Apache access logs using the appropriate parser logic. This is where extensible KHIKA Adapter for OSSEC comes handy.

KHIKA Adapter for OSSEC is modular and can be extended by adding an appropriate python to it. To understand how to add a new python a function to parse a new type of logs, let us first understand the structure of this adapter.

1. Login to KHIKA Data Aggregator and go to directory /opt/KHIKA/Apps/Adapters/OssecArchiveLog
2. You will find a file filename_parser_mapping.csv. Open this file and check the header and a couple of first lines

  Parser_ID,File_Name,Parser_Name,Invoke_Parser,Headers,"Agent Source Name"
  1,"(\/var\/log\/audit\/audit.log)","unix_audit","CallProcessUnixAuditLogLine",,
  2,"(\/project\/job\/remove_NetScreen_)(.*)(log)","juniper_netscreen","CallProcessJuniperNetscreenLogLine",,
  3,"(\/project\/job\/remove_fortigate.log)","varnish_apache_access_logs","CallProcessVarnishApacheAccessLogLine",,