KHIKA App for Linux

Introduction

Linux servers run most critical business applications. Monitoring of Linux servers is important, from both security and operational standpoint. With KHIKA App for Linux servers, you can :

• Monitor syslogs to identify live attack vectors such as brute force attempts, weak ssh ciphers, communication with bad IPs and many others.
• Use Actionable dashboards showing gaps in secured configuration and hardening of the severs
• Do File integrity monitoring
• Monitor Superuser activities

We explain below, steps to configure and interpret the output of KHIKA App for Linux Server. The key parts to get here are :

1. Install the KHIKA App for Linux
2. Get data from your Linux Server into KHIKA Aggregator

How to Install the KHIKA App for Linux?

This section assumes that you have already configured KHIKA Data Aggregator in your environment. If not, please read Configure KHIKA Aggregator and perform the pre-requisite steps. Here, we explain how to pick and install the KHIKA application for Linux Servers. Installing the KHIKA App for Linux shall put together and activate the adapter (parser) that can handle Linux data format, the dashboards and the alert rules preconfigured.

Go to “Applications” tab in the “Configure” menu

Check whether the appropriate Workspace is selected

Note: Application is always loaded in a Workspace. Read the section on KHIKA Workspaces to know more.

Select your KHIKA aggregator name in the Node dropdown

This is to ensure that we are collecting data from the desired source and into the correct workspace which is ready with the configured application and components.

Click on the “+” button

A pop up appears.

User can now select the contents of the application required

For example, on the dropdown for “Reports”, click to expand it. List of all reports can be seen. User can individually select the reports required by checking on the checkbox next to each. Alternatively, check on “Select All” option to get all of them.

Similarly you can select contents from Alerts and Dashboards.

KHIKA Alerts

KHIKA Reports

KHIKA Dashboards

Click “OK”

to proceed with the installation of the selected Application.

After successful installation, following status should be displayed :

This simple procedure to install a KHIKA App, automatically configures the Adapter (required for parsing the data from raw syslogs), calculated KHIKA reports on raw data, Visualizations, Dashboards and Alerts – all in one click.

How to get your Linux data into KHIKA ?

KHIKA recommends, popular open source OSSEC integration to monitor the Linux servers. There are 2 components in OSSEC Integration with KHIKA.  OSSEC Agent – Installed on each Linux server which we wish to monitor  OSSEC Server – Present on KHIKA Data Aggregator (which you must install before) The OSSEC agent and server communicate with each other using a unique key for encryption. The main steps to start getting data from a Linux server are  Install Ossec agent on the Linux server  Add the Linux server details in KHIKA  Extract a unique key for this device from KHIKA  Insert this key in the Ossec agent (ie. on your Linux server to be monitored)  Reload Configuration  Verify data collection Each of this can be a hyperlink to below sections. Each of these steps is explained in detail in the further sections.

1. Installing OSSEC Agent for Linux Download OSSEC agent for Linux from here. Copy the downloaded installer on your Linux server that you wish to monitor using KHIKA and run the installer with "root" credentials on the Server. Please Note : It is extremely important to install the OSSEC agent with "root" privileges as this agent reads the /var/log/security, /var/log/messages and some other important files. In order to read it successfully the ossec-agent process must be installed with "root" privileges. You will have to run following command as "root" user to install the Ossec Agent :- Remove / rename ossec directory if already exists on the agent. ie. our Linux server. mv /opt/ossec /opt/ossec_bak Go to the location where you have copied the Ossec agent installer mentioned above. Extract it using the following command tar –zxvf ossec_TL_Agent.tar.gz Then go to that directory using the cd command. You shall see a script by the name install.sh Then Run following command. "sudo ./install.sh" (you need not do sudo if you have already logged in as root)

Now, add KHIKA Data Aggregator IP address (OSSEC server IP address) to point the OSSEC agent to the OSSEC server.

NOTE :- You will have to repeat these steps on each of the Linux Servers that you wish to monitor using KHIKA. 2. Adding the device in the Adaptor Go to Adapter tab in the “Configure” menu. Next to our “linux_ossec_adapter”, click on the “Manage Devices” icon.

Pop up appears for device details

Click on “Add / Modify Device” tab. Another pop up appears for device details.

Enter the expected device name. Also, in the field for IP address, enter “any”. Please note : Always enter the IP Address as “any”. This is a safe and sure option to establish a connection with the server where we are suggesting ossec agent to use “any” of its configured IPs to be used to connect with the OSSEC Server. The device may have multiple NIC cards/IP addresses and unless we are sure of what IP will be used for connection, the connect will fail. Hence, it is safe to “any” as interface or IP address. Select appropriate time zone of this device. In the “Node” field dropdown, select the name of the Aggregator or local data collector for this device. Click on Submit. We get a success message and the linux server is added successfully to this adaptor.

3. Extract key from KHIKA OSSEC Server Now the expected Linux server is added in the required adapter. To see this device entry, click on “Manage Devices” icon next to the adapter.

A pop up with device details of the adaptor appears. Select “List of Devices” tab.

Click on the “Get OSSEC Key” icon next to this device.

This is the unique key for this device created by the OSSEC server. Paste this key in the Ossec agent which is installed on this Linux server. 4. Insert unique OSSEC key in OSSEC Agent on the Linux Server Perform following simple steps on the Linux Agent  Login as "root" on the agent server  Please note OSSEC Server listens on UDP port 1514 and the firewall between the ossec agent and ossec server must be open for UDP protocol and 1514 port.  In the OSSEC Agent installation directory, run manage-agent script from sudo /opt/ossec/bin/manage_agents  You'll be presented with these options

Select "i" to import the key (which we created in above section, on the Ossec server)  Copy and paste the key generated on the server  Restart the agent using command /opt/ossec/bin/ossec-control restart  Repeat these steps for all the servers to be monitored.  Finally, go to Workspace tab and click on “Apply Configuration” icon. 

5. Reload Configuration Login into the KHIKA portal. Go to Configure  Select workspace, eg. LINUX_SERVERS  Go to Node Tab  Click Reload Config

This step restarts OSSEC Server. Wait for a few minutes for server to restart.

6. Verifying OSSEC data collection

Once the device is added successfully, we can check the data for this device on Discover screen. Go to “Discover” from the main menu. Select the appropriate index for the same. Raw (khika formatted) data of all your Linux servers added is seen here.

To see the data for our newly added device, enter search string in lower case – tl_src_host : name_of_the_device_added_in_lower_case and click on the search icon.

3. How to check the output of KHIKA Linux App ? Dashboards Linux Login Activity This dashboard focusses on the login activity in the Linux servers in your system (which are added into KHIKA). Details like which user logged in how many times into which server, authentication information, login methods like SSH etc. is shown in an analytical fashion. Elements in the Dashboard are explained below : Visualization Description Logon Authentication pie chart Contribution of success and failure authentication events IP address wise Users bar graph X axis : all the IP addresses from where users have logged into the Linux server(s) Y axis : Usernames and count of events Time trend Trend of login events over time. Useful to identify unusual spikes at a glance.

X axis : date & time Y axis : count of events Summary Table Detailed data with timestamp and count Suggestion for useful interaction with this dashboard could be : Click on “failed” login section in the Logon Authentication pie chart. This gets selected and a filter for “failed” is applied across the rest of the dashboard. The next bar shall show then the statistics of users who failed to login into the server and a high number of count on the y axis, may need attention. Details of “failed” login can be seen in the summary table. How to remove this filter is explained here Linux File Integrity Critical files in your system are monitored for any change / edit and real time alerts are fired if any such incident, as well as displayed, on this monitoring dashboard. Elements in the Dashboard are explained below : Visualization Description Contribution of Filename pie chart Contribution of file names, as per the modification events happening on it. Server wise contribution of Files Modified X axis : on or more linux servers Y axis : stacked in each bar (server name) the names of files and count of events occurred for each file. Time trend Trend of login events over time. Useful to identify unusual spikes at a glance.

X axis : date & time Y axis : count of events Summary Table Detailed data with timestamp and count

A suggestion for useful interaction with this dashboard could be : Examine the time trend, for higher number of events. If there has been any software update / patch activity at the time and still high number of events, click on that point in time on the time trend. Rest of the dashboard also gets filtered and we can isolate – the server on which this happened and the file names.

Linux Server Hardening Server Hardening is the process of enhancing server security through a variety of means which results in a more secure server operating environment. This is due to the advanced security measures that are put in place during the server hardening process. KHIKA checks each server against out-of-box server hardening policies to ensure your servers are securely configured. It helps you to pinpoint and tune the exact details on hosts for better security posture. The server hardening policies against which the servers are checked can be seen here. Elements in the Dashboard are explained below : Visualization Description Contribution of status pie chart Failed or Passed compliance status Server wise Hardening Status X axis : Linux servers added into KHIKA Y Axis : stacked within each bar (server) the count of failed / passed events for various rules / policies Policy wise status X axis : Policy names Y axis : stacked with each bar (policy) count of failed or passed servers for that policy Time trend Trend of login events over time. Useful to identify unusual spikes at a glance.

X axis : date & time Y axis : count of events Summary Table Detailed data with timestamp and count

Some suggestions for useful interaction with this dashboard could be : 1. Click on “Failed” in the “Contribution of Status” pie chart. The rest of the dashboard gets filtered and shows only Failed events. Enables having an easier look at the servers / policies which failed more often 2. Click on a particular server in the bar “Server Wise Hardening Status”. Also click on the “Failed” in the above pie. This isolates the actionable inputs that you need to tune the server in question.

Linux Sudo Commands This report summarises which sudo commands were fired by whom, and from which host. The sudo commands and super user details. Elements in the Dashboard are explained below : Visualization Description Contribution of Users pie Names and contribution of the Super users according to the commands they have fired Contribution of Commands Names and contribution of commands which were fired Contribution of Path The path names used Hostname wise User X axis : all the hostnames from where users have logged into the Linux server(s) Y axis : Usernames and count of events Time trend Trend of login events over time. Useful to identify unusual spikes at a glance.

X axis : date & time Y axis : count of events Summary Table Detailed data with timestamp and count Some suggestions for useful interaction with this dashboard could be : 1. Click on a particular username in the “Contribution of Users” pie. You can monitor all the activities of this super user. 2. Alternately, click on a particular hostname from the bar graph “Hostname wise User” to check all the Super users logging into and firing commands on this host.

Linux User Activity This report focusses on the user activity on Linux servers. Which actions users have taken, programs used etc. Names and contribution of commands which were firedhi Elements in the Dashboard are explained below : Visualization Description Contribution of Users pie Names and contribution of the users according to the activities / events Contribution of Programs pie Names and contribution of commands / actions which were fired Activities on Group bar graph X axis : Activities carried out Y axis : Stacked within each bar (ie. for each activity) the group names and count of events Time trend Trend of login events over time. Useful to identify unusual spikes at a glance.

X axis : date & time Y axis : count of events Summary Table Detailed data with timestamp and count Some suggestions for useful interaction with this dashboard could be : 1. Drill down on the name of activity by clicking on any one bar on the “Activities on Group” – “new user” addition. Thus you can investigate how many new users were added on this group and other details about it can be seen in the summary table. 2. Click on any sensitive program name from the pie “Contribution of Programs” and this will isolate the users, hostnames who executed it and their machine names.

Linux Root Check Details This report focusses on the access given to critical files. Details of who has access, who is the owner and the hostname etc. can be found.

Elements in the Dashboard are explained below : Visualization Description Contribution of Permissions Names and contribution of the permissions given Hostname wise Filename X axis : Hostnames Y axis : Files stacked in one bar (host) and the count. Time trend Trend of login events over time. Useful to identify unusual spikes at a glance.

X axis : date & time Y axis : count of events Summary Table Detailed data with timestamp and count A suggestion for useful interaction with this dashboard could be : Click on and select a particular permission in the first pie. The rest of the dashboard reflects all the files and users with this permission. Alerts Alerts are generated when certain ciritical behaviour is observed in the system – real time and notified on the Alerts Dashboard in KHIKA as well as can be received in email to relevant stakeholders. The details of KHIKA Alerts are mentioned here Click on “Alert Dashboard” on left menu. Certain alerts for linux are pre-canned and shipped with KHIKA, keeping in mind the requirements of the users. They are mentioned in the table below : Alerts Description Authentication failure by unauthorised user / Attempt of Brute Force attack Critical Files' Integrity Change Alert Linux Audit configuration changed Successful Brute Force Attack Possibililty of SSH attack Attempt of Brute Force Attack Linux Server is in promiscuous mode New user password not changed File System Full Abnormal termination of process observed User account created and deleted within 1 hour Multiple syscall failure Attempt of Brute force attack Concurrent multiple logins or password share / leak Alteration in IP tables rule -possibility of unauthorised access Linux fatal or critical device error

Visualization	Description
Logon Authentication pie chart	Contribution of success and failure authentication events
IP address wise Users bar graph	X axis : all the IP addresses from where users have logged into the Linux server(s), Y axis : Usernames and count of events
Time trend	Trend of login events over time. Useful to identify unusual spikes at a glance. X axis : date & time, Y axis : count of events
Summary Table	Detailed data with timestamp and count