Data Archival in KHIKA

Overview

The purpose of this section is to provide KHIKA SIEM Users and Administrators, an understanding of the complete life cycle of data stored in KHIKA. In KHIKA, time series log data from data sources is segregated into one or more workspaces such that data from a distinct data source is typically stored on its own, in each workspace’s index. On receiving log data, KHIKA identifies the workspace associated with the data source and stores the data in its corresponding day’s index. In other words, the data received today will be stored in today’s data index while the data received tomorrow will be stored in tomorrow’s data index.

Since log data combined over a period of time tends to becomes large (> few TBs) in size, in-order to maintain optimal KHIKA application performance as well as to ensure prudent use of IT infrastructure and resources, KHIKA data storage is categorized into two types viz.

• Online storage – the data that is readily searchable via KHIKA UI is stored in online storage. The setting/parameter that controls online data retention period is called “TIME-TO-LIVE” or TTL and TTL is a workspace level setting and can be configured as per customer requirements. The default TTL or online data retention period for the workspace is 90 days.
• Offline storage - The older data i.e. data beyond the TTL period is archived or moved from online storage to offline storage.

Checking Data Archival details

Go to Configure from the left pane and select Workspace tab.

KHIKA Data Archival procedure automatically moves data in this workspace, only when it is 91 days old, to the Offline storage. Newer data in the workspace is not moved until 90 days.

Please note : If the online storage disk utilisation reaches 80%, ie. If it is 80% full, then, oldest day data shall be moved to the Offline storage even if it is not 91 days old yet.

To review the Data archival status for a workspace, go to Configure from the main KHIKA menu and select Workspace tab.

Select the required workspace from the dropdown and click on Archival status icon for it. A pop up appears asking for from and to dates for duration of archival report. Select dates and you can get the archival status report as follows:

Offline storage and Archival

If you have implemented KHIKA on premise

You have to provide storage, for data to move out from online storage to offline storage after retention period expires.

As explained in above section, there is a "Retention Period" value, in days for every workspace. The oldest day's data in KHIKA for this workspace, which is one day older than the retention period value, has to be moved to another secondary storage called the "Offline Storage". For example, if retention period in your LINUX workspace say, is 30 days. Lets say you have data beginning from 1st of March in this workspace. On 31st March, the data of 1st March has crossed its retention period. A snapshot is taken of elastic data index for 1st March and it is copied to the offline storage.

Once data moves to Offline storage, it is not searchable on the Discover screen