Skip to content

Migration from TheHive 3.x#

TheHive 5.x is delivered with a tool to migrate your data from TheHive 3.x. stored in Elasticsearch.

Supported versions#

The migration tool supports migrating data from both TheHive 3.4.x and 3.5.x.

Migrating from Elasticsearch version
TheHive 3.4.x v6.x
TheHive 3.5.x v7.x

How it works#

All packages of TheHive v5.x distributed come with the migration program which can be used to import data from TheHive 3.4.0+. By default, it is installed in /opt/thehive/bin/migrate.

Pre-requisite#

In order to migrate the data:

  • TheHive v5.x must be installed on the system running the migration tool;

  • TheHive must be configured ; in particular database, index, and file storage ;

  • The service thehive must be stopped (service thehive stop) on the target server.

This tools must also have access to Elasticsearch database (http://ES:9200) used by TheHive 3, and the configuration file of TheHive 3.x instance.

Configuration of TheHive#

Warning

In TheHive, users are identified by their email addresses. Thus, a domain will be appended to usernames in order to migrate users from TheHive 3.

TheHive v5.x comes with a default domain named thehive.local. Starting the migration without explicitly specifying a domain name will result in migrating all users with a username formatted like user@thehive.local.

Change the default domain name used to import existing users in the configuration file of TheHive (/etc/thehive/application.conf) ; add or update the setting named auth.defaultUserDomain:

auth.defaultUserDomain: "mydomain.com"

This way, the domain mydomain.com will be appended to user accounts imported from TheHive 3.4+ (user@mydomain.com).

Run the migration#

Prepare, install and configure your new instance of TheHive v5.x by following the associated guides. In order to obtain the best migration performance, it is recommended to not start TheHive 5 before running the migration tool. Thus, the migration will be done without index. At the end of the migration, the index will be added.

Once TheHive 5 configuration file (/etc/thehive/application.conf) is correctly filled the migrate command ca be executed.

Warning

The configuration file must not contain cluster configuration (akka.cluster). The migration tool works on single node only.

Info

This recommended to run this program as the user in charge of running TheHive service ( thehive if you are installing the application with DEB or RPM package)

The program comes with a large set of options:

# /opt/thehive/bin/migrate --help
TheHive migration tool 
Usage: migrate [options]

  -v, --version
  -h, --help
  -l, --logger-config <file>
                           logback configuration file
  -c, --config <file>      global configuration file
  -i, --input <file>       TheHive 3 configuration file
  -o, --output <file>      TheHive 5 configuration file
  -d, --drop-database      Drop TheHive 5 database before migration
  -r, --resume             Resume migration (or migrate on existing database)
  -m, --main-organisation <organisation>
  -u, --es-uri http://ip1:port,ip2:port
                           TheHive3 ElasticSearch URI
  -x, --es-index <index>   TheHive3 ElasticSearch index name
  -x, --es-index-version <index>
                           TheHive3 ElasticSearch index name version number (default: autodetect)
  -a, --es-keepalive <duration>
                           TheHive3 ElasticSearch keepalive
  -p, --es-pagesize <value>
                           TheHive3 ElasticSearch page size
  -s, --es-single-type <bool>
                           Elasticsearch single type
  -y, --transaction-pagesize <value>
                           page size for each transaction
  -t, --thread-count <value>
                           number of threads
  -k, --integrity-checks   run integrity checks after the migration
  --max-case-age <duration>
                           migrate only cases whose age is less than <duration>
  --min-case-age <duration>
                           migrate only cases whose age is greater than <duration>
  --case-from-date <date>  migrate only cases created from <date>
  --case-until-date <date>
                           migrate only cases created until <date>
  --case-from-number <number>
                           migrate only cases from this case number
  --case-until-number <number>
                           migrate only cases until this case number
  --max-alert-age <duration>
                           migrate only alerts whose age is less than <duration>
  --min-alert-age <duration>
                           migrate only alerts whose age is greater than <duration>
  --alert-from-date <date>
                           migrate only alerts created from <date>
  --alert-until-date <date>
                           migrate only alerts created until <date>
  --include-alert-types <type>,<type>...
                           migrate only alerts with this types
  --exclude-alert-types <type>,<type>...
                           don't migrate alerts with this types
  --include-alert-sources <source>,<source>...
                           migrate only alerts with this sources
  --exclude-alert-sources <source>,<source>...
                           don't migrate alerts with this sources
  --max-audit-age <duration>
                           migrate only audits whose age is less than <duration>
  --min-audit-age <duration>
                           migrate only audits whose age is greater than <duration>
  --audit-from-date <date>
                           migrate only audits created from <date>
  --audit-until-date <date>
                           migrate only audits created until <date>
  --include-audit-actions <value>
                           migration only audits with this action (Update, Creation, Delete)
  --exclude-audit-actions <value>
                           don't migration audits with this action (Update, Creation, Delete)
  --include-audit-objectTypes <value>
                           migration only audits with this objectType (case, case_artifact, case_task, ...)
  --exclude-audit-objectTypes <value>
                           don't migration audits with this objectType (case, case_artifact, case_task, ...)
  --case-number-shift <value>
                           transpose case number by adding this value
Accepted date formats are "yyyyMMdd[HH[mm[ss]]]" and "MMdd"
The Format for duration is: <length> <unit>.
Accepted units are:
  DAY:         d, day
  HOUR:        h, hr, hour
  MINUTE:      m, min, minute
  SECOND:      s, sec, second
  MILLISECOND: ms, milli, millisecond

Most of these options are filters you can apply to the program. For example, you could decide to import only some of the Cases/Alerts from your old instance:

  • Import Cases/Alerts not older than X days/hours,
  • Import Cases/Alerts with the ID number,
  • Import part of Audit trail
  • ...

Taking the assumption that you are migrating a database hosted in a remote server, with TheHive 3, a basic command line to migrate data to a new instance will be like:

/opt/thehive/bin/migrate \
  --output /etc/thehive/application.conf \
  --main-organisation myOrganisation \
  --es-uri http://ELASTICSEARCH_IP_ADDRESS:9200 \
  --es-index the_hive

with:

Option Description
--output specifies the configuration file of TheHive 5 (the one that has previously been configured with at least, the database and file storage)
--main-organisation specifies the Organisation named myOrganisation to create during the migration. The tool will then create Users, Cases and Alerts from TheHive3 under that organisation;
--es-uri specifies the URL of the Elasticsearch server. If using authentication on Elasticsearch, --input option with a configuration file for TheHive3 is required
--es-index specifies the index used in Elasticsearch.

Info

The migration process can be very long, from several hours to several days, depending on the volume of data to migrate. We highly recommend to not start the application during the migration.

Continue an incomplete migration#

If the migration has been stopped, or only part of the data has been migrated, it is possible to continue the migration by running the tool again with the parameter --resume. This parameter prevents creating data if it already exists.

Merge several TheHive 3 data into one TheHive 5#

The migration tool can be run several times to migrate different TheHive 3 data into one TheHive 5 instance. The name of the target organisation can be changed for each execution. In order to prevent case number collision (if a case with the same number already exists, the new case won't be created), you can shift the case number with the parameter --case-number-shift.

Using authentication on Cassandra#

if you are using a dedicated account on Cassandra to access TheHive 4 data, this user must have permissions to create keyspaces on the database:

GRANT CREATE on ALL KEYSPACES to username;

Migration logs#

The migration tool generates some logs during the process. By default, every 10 sec. a log is generated with information regarding the situation of the migration:

[info] o.t.t.m.Migrate - [Migrate cases and alerts] CaseTemplate/Task:32 Organisation:1/1 Case/Task:160/201 Case:31/52 Job:103/138 ObservableType:3/17 Alert:25/235 Audit:3207/2986 CaseTemplate:6/6 Alert/Observable:700(52ms) Case/Observable:1325/1665 User:9/9 CustomField:13/13 Case/Task/Log:20/27

Numbers of Observables, Cases and others are estimations and not a definite value as computing these numbers can be very tedious.

Files from MISP imported with TheHive 2.13 and earlier

It is important to notice that migrating Cases/Alerts containing MISP event that were imported with TheHive 2.13 (Sept 2017) or older will cause observable files not being imported in TheHive 5.

Indeed, until this version, TheHive referenced the file to the AttributeId in MISP and was not automatically downloaded. It then could generate a log like this:

[warn] o.t.t.m.t.Input - Pre 2.13 file observables are ignored in MISP alert ffa3a8503ab0cd4f99fc6937a8e9b827

Starting TheHive#

Once the migration process is successfully completed, TheHive can be started.

Warning

During the first start data are indexed and service is not available ; this can take some time. Do not stop or restart the service at this time.