Skip to content

Configure Audit Log Storage in Elasticsearch#

5.5

By default, TheHive stores audit logs in Apache Cassandra via JanusGraph. However, if your organization generates a large volume of audit logs, you can switch to Elasticsearch.

This topic provides step-by-step instructions for configuring TheHive audit log storage in Elasticsearch for existing instances, including the option to migrate historical logs from JanusGraph.

For new installations, refer to the installation guide for instructions.

Reasons to consider Elasticsearch

Elasticsearch is better suited for managing large volumes of audit logs. It enhances performance by efficiently handling data, reducing latency, and offering advanced search capabilities. If your organization generates a significant amount of audit logs, migrating to Elasticsearch can improve both data management and retrieval.

No rollback possible

Since TheHive can only use one audit storage system at a time, when you switch to Elasticsearch, the audit logs stored in JanusGraph become unavailable. TheHive doesn't support transferring audit logs back from Elasticsearch to JanusGraph.

Audit logs visibility

With Elasticsearch, audit logs retain the visibility they had at the time of creation, regardless of the current visibility of the case. This means that even if you set restricted visibility for a case, audit logs remain visible to all users who originally had access.

Prerequisites#

Use Elasticsearch 7.17 or later#

Make sure you're using Elasticsearch version 7.17 or later to guarantee compatibility with TheHive audit log storage. OpenSearch isn't supported for audit logs.

Back up Elasticsearch indices#

Regularly back up your Elasticsearch indices to ensure you can recover audit logs in the event of an incident. This is critical for maintaining the integrity and availability of your data.

Step 1: Activate audit log storage in Elasticsearch#

  1. Open the /etc/thehive/application.conf file.

  2. In the application.conf file, activate audit log storage in Elasticsearch by adding the following lines:

    audit {
  storage = elasticsearch
}

  3. Save your modifications in the application.conf file.

(Optional) Step 2: Configure index template and Index Lifecycle Management (ILM)#

Adjust the application.conf file to configure the index template and Index Lifecycle Management (ILM) policy for the thehive-audits index. These settings control how Elasticsearch stores, rolls over, and deletes audit logs.

Supported configuration keys:

  • index-name: Name of the Elasticsearch index for audit logs (default: thehive-audits)
  • health-request-timeout: Timeout for checking cluster health
  • create.ext.number_of_replicas: Number of index replicas
  • create.ext.number_of_shards: Number of index shards
  • request-timeout: Timeout for Elasticsearch requests
  • hot.rollover.maxAge: Maximum age of an index before rollover
  • hot.rollover.maxSize: Maximum size of an index before rollover
  • delete.minAge: Minimum age before an index is deleted

Example

To create a new index every day or when it reaches 1 GB, and retain it for 7 days, use the following configuration:

audit {
    storage = elasticsearch
    elasticsearch {
        hot.rollover {
            maxAge = 1d
            maxSize = 1g
        }
        delete.minAge = 7d
    }
}

Index name conflict

The index name shouldn't be the same as the one used in JanusGraph. If you don’t configure a specific name, TheHive uses the JanusGraph index name appended with the suffix -audits.

Based on the configuration above, TheHive automatically generates the index template and ILM policy when it first writes audit data to Elasticsearch.

Index template#

Default generated index template:

{
  "index": {
    "lifecycle": {
      "name": "thehive-audits"
    },
    "number_of_shards": "1",
    "number_of_replicas": "1"
  }
}
ILM#

Default generated ILM:

{
  "policy": {
    "phases": {
      "hot": {
        "min_age": "0ms",
        "actions": {
          "set_priority": {
            "priority": 100
          }
        }
      }
    }
  }
}

ILM generated from the example configuration

Based on the example configuration, TheHive will generate the following ILM policy:

{
  "policy": {
      "phases": {
          "hot": {
              "min_age": "0ms",
              "actions": {
                  "set_priority": {
                      "priority": 100
                  },
                  "rollover": {
                      "max_primary_shard_size": "1gb",
                      "max_age": "1d"
                  }
              }
          },
          "delete": {
              "min_age": "7d",
              "actions": {
                  "delete": {
                      "delete_searchable_snapshot": true
                  }
              }
          }
      }
  }
}

API calls changes

After migrating to Elasticsearch, the following Query API calls no longer work with audit logs stored in Elasticsearch:

  • listAudit
  • listAuditFromObject
  • getAudit

Instead, use the following API endpoints:

  • /api/v1/audit/list
  • /api/v1/audit/count
  • /api/v1/audit/$ID

These API calls are currently available but not yet officially documented and may change in future releases.

(Optional) Step 3: Migrate historical audit logs to Elasticsearch#

TheHive provides a script to assist with migrating historical audit logs from JanusGraph to Elasticsearch. Run the script only if you need to transfer historical audit logs to Elasticsearch.

JanusGraph audit log deletion

By default, when migrating audit logs from JanusGraph to Elasticsearch, the logs are deleted from JanusGraph. If you wish to keep the audit logs in JanusGraph while migrating to Elasticsearch, you can prevent their deletion by using a specific option.

Run the migrateAudits script:

  • If you installed TheHive on a server:
./migrateAudits --audit-from-date <date> -c </etc/thehive/application.conf>
  • If you deployed TheHive with a Docker image:
docker run --network host --rm -ti -v </path/to/your/application.conf>:/etc/thehive/application.conf:ro -v </path/to/your/logback.xml>:/etc/thehive/logback.xml:ro strangebee/thehive:5.5.8-1-SNAPSHOT  migrateAudits -Dlogback.configurationFile=/etc/thehive/logback.xml -- --audit-from-date <date> -c /etc/thehive/application.conf

Available options are:

-v, --version: Displays the version of the migration tool.
-h, --help: Displays help information.
-c, --config <file> : Specifies the global configuration file for TheHive.
-y, --transaction-pagesize <value>: Defines the page size for each transaction during the migration.
-n, --no-deletion: Prevents the deletion of audit logs from JanusGraph after they are migrated to Elasticsearch.
--audit-from-date <date>: Migrates only the audit logs created from the specified date. The date format is yyyyMMdd[HH[mm[ss]]] or yyyy/MM/dd HH:mm:ss.
--audit-until-date <date>: Migrates only the audits created until the specified date. The date format is yyyyMMdd[HH[mm[ss]]] or yyyy/MM/dd HH:mm:ss.

Next steps