Skip to content

How to Configure Audit Log Storage in Elasticsearch#

5.5

By default, TheHive stores audit logs in Apache Cassandra via JanusGraph. However, if your organization generates a large volume of audit logs, you can switch to Elasticsearch.

This topic provides step-by-step instructions for configuring TheHive's audit log storage in Elasticsearch for existing instances, including the option to migrate historical logs from JanusGraph.

For new installations, refer to the Step-by-Step Guide for instructions.

Reasons to consider Elasticsearch

Elasticsearch is better suited for managing large volumes of audit logs. It enhances performance by efficiently handling data, reducing latency, and offering advanced search capabilities. If your organization generates a significant amount of audit logs, migrating to Elasticsearch can improve both data management and retrieval.

No rollback possible

Since TheHive can only use one audit storage system at a time, when you switch to Elasticsearch, the audit logs stored in JanusGraph become unavailable. TheHive doesn't support transferring audit logs back from Elasticsearch to JanusGraph.

Audit logs visibility

With Elasticsearch, audit logs retain the visibility they had at the time of creation, regardless of the current visibility of the case. This means that even if you set restricted visibility for a case, audit logs remain visible to all users who originally had access.

Prerequisites#

Use Elasticsearch 7.17 or later#

Make sure you're using Elasticsearch version 7.17 or later to guarantee compatibility with TheHive's audit log storage. OpenSearch isn't supported for audit logs.

Back up Elasticsearch indices#

Regularly back up your Elasticsearch indices to ensure you can recover audit logs in the event of an incident. This is critical for maintaining the integrity and availability of your data.

Step 1: Activate audit log storage in Elasticsearch#

To activate audit log storage in Elasticsearch, update your TheHive configuration file with the following settings:

audit {
  storage = elasticsearch
}

(Optional) Step 2: Configure index template and Index Lifecycle Management (ILM)#

Configure the default index template and Index Lifecycle Management (ILM) for thehive-audits using the following settings:

  • index-name
  • health-request-timeout
  • create.ext.number_of_replicas
  • create.ext.number_of_shards
  • request-timeout
  • hot.rollover.maxAge
  • hot.rollover.maxSize
  • delete.minAge

Index name

The index name shouldn't be the same as the one used in JanusGraph. If you don’t configure a specific name, TheHive uses the JanusGraph index name appended with the suffix -audits.

Index template#

The index template ensures consistent storage and indexing of audit logs.

Default index template:

{
  "index": {
    "lifecycle": {
      "name": "thehive-audits"
    },
    "number_of_shards": "1",
    "number_of_replicas": "1"
  }
}

ILM#

TheHive automatically generates an Index Lifecycle Management (ILM) policy based on your configuration settings. ILM manages the storage, rollover, and deletion of audit logs over time to optimize long-term storage.

Default generated ILM:

{
  "policy": {
    "phases": {
      "hot": {
        "min_age": "0ms",
        "actions": {
          "set_priority": {
            "priority": 100
          }
        }
      }
    }
  }
}

Example

To create a new index every day or when it reaches 1 GB, and retain it for 7 days, use the following configuration:

  • Index template:
audit {
    storage = elasticsearch
    elasticsearch {
        hot.rollover {
            maxAge = 1d
            maxSize = 1g
        }
        delete.minAge = 7d
    }
}
  • The generated ILM:
{
    "policy": {
        "phases": {
            "hot": {
                "min_age": "0ms",
                "actions": {
                    "set_priority": {
                        "priority": 100
                    },
                    "rollover": {
                        "max_primary_shard_size": "1gb",
                        "max_age": "1d"
                    }
                }
            },
            "delete": {
                "min_age": "7d",
                "actions": {
                    "delete": {
                        "delete_searchable_snapshot": true
                    }
                }
            }
        }
    }
}

API calls changes

After migrating to Elasticsearch, the following Query API calls no longer work with audit logs stored in Elasticsearch:

  • listAudit
  • listAuditFromObject
  • getAudit

Instead, use the following API endpoints:

  • /api/v1/audit/list
  • /api/v1/audit/count
  • /api/v1/audit/$ID

These API calls are currently available but not yet officially documented and may change in future releases.

(Optional) Step 3: Migrate historical audit logs to Elasticsearch#

TheHive provides a script to assist with migrating historical audit logs from JanusGraph to Elasticsearch. Run the script only if you need to transfer historical audit logs to Elasticsearch.

JanusGraph audit log deletion

By default, when migrating audit logs from JanusGraph to Elasticsearch, the logs are deleted from JanusGraph. If you wish to keep the audit logs in JanusGraph while migrating to Elasticsearch, you can prevent their deletion by using a specific option.

Run the migrateAudits script:

  • If you installed TheHive on a server:
./migrateAudits --audit-from-date <date> -c </etc/thehive/application.conf>
  • If you deployed TheHive with a Docker image:
docker run --network host --rm -ti -v </path/to/your/application.conf>:/etc/thehive/application.conf:ro -v </path/to/your/logback.xml>:/etc/thehive/logback.xml:ro strangebee/thehive:5.5.0-1-SNAPSHOT  migrateAudits -Dlogback.configurationFile=/etc/thehive/logback.xml -- --audit-from-date <date> -c /etc/thehive/application.conf

Available options are:

-v, --version: Displays the version of the migration tool.
-h, --help: Displays help information.
-c, --config <file> : Specifies the global configuration file for TheHive.
-y, --transaction-pagesize <value>: Defines the page size for each transaction during the migration.
-n, --no-deletion: Prevents the deletion of audit logs from JanusGraph after they are migrated to Elasticsearch.
--audit-from-date <date>: Migrates only the audit logs created from the specified date. The date format is yyyyMMdd[HH[mm[ss]]] or yyyy/MM/dd HH:mm:ss.
--audit-until-date <date>: Migrates only the audits created until the specified date. The date format is yyyyMMdd[HH[mm[ss]]] or yyyy/MM/dd HH:mm:ss.

Next steps