Migration from Elasticsearch 6.8.2 to ES 7.x#
⚠️ IMPORTANT NOTE
- This migration process is intended for single node of Elasticsearch database
- The current version of this document is provided for testing purpose ONLY!
- This guide has been written and tested to migrate data from ES 6.8.2 to ES 7.8.1, and Cortex 3.0.1 to Cortex 3.1.0 only!
- This guide starts with Elasticsearch version 6.8.2 up and running, indexes and data. To test this guide, we recommend using a backup of you production server. (see Backup and Restore page for more information)
- This guide is illustrated with Cortex index. The process is identical for Cortex, you just have to adjust index names.
Prerequisite#
The software jq
is required to manipulate JSON and create new indexes. More information at https://stedolan.github.io/jq/.
Identify if your index should be reindexed#
You can easily identify if indexes should be reindexed or not. On the index named cortex_4
run the following command:
curl -s http://127.0.0.1:9200/cortex_4?human | jq '.cortex_4.settings.index.version.created'
if the output is similar to "5xxxxxx"
then reindexing is required, you should follow this guide.
If it is "6xxxxxx"
then the index can be read by Elasticsearch 7.8.x. Upgrade Elasticsearch, and Cortex 3.1.0.
Migration guide#
Current status#
Current context is: - Elasticsearch 6.8.2 - Cortex 3.0.1
All up and running.
Start by identifying indices on you Elasticsearch instance.
curl http://localhost:9200/_cat/indices\?v
The output should look like this:
health status index uuid pri rep docs.count docs.deleted store.size pri.store.size
green open cortex_4 Y5rDTO23RBC_n6pjFP0-Qw 5 0 8531 8 13mb 13mb
The index name is cortex_4
. Record this somewhere.
Stop services#
Before starting updating the database, lets stop applications:
sudo service cortex stop
Create a new index#
The First operation lies in creating a new index named new_cortex_4
with settings from current index cortex_4
(ensure to keep index version, needed for future upgrade).
curl -XPUT 'http://localhost:9200/new_cortex_4' \
-H 'Content-Type: application/json' \
-d "$(curl http://localhost:9200/cortex_4 |\
jq '.cortex_4 |
del(.settings.index.provided_name,
.settings.index.creation_date,
.settings.index.uuid,
.settings.index.version,
.settings.index.mapping.single_type,
.mappings.doc._all)'
)"
Check the new index is well created:
curl -XGET http://localhost:9200/_cat/indices\?v
The output should look like this:
health status index uuid pri rep docs.count docs.deleted store.size pri.store.size
green open new_cortex_4 wRX6rhzXTuW_F2wLNxqVyg 5 0 0 0 1.1kb 1.1kb
green open cortex_4 Y5rDTO23RBC_n6pjFP0-Qw 5 0 8531 8 13mb 13mb
Proceed to reindex#
Next operation lies in running the reindex command in the newly created index:
curl -XPOST -H 'Content-Type: application/json' http://localhost:9200/_reindex -d '{
"conflicts": "proceed",
"source": {
"index": "cortex_4"
},
"dest": {
"index": "new_cortex_4"
}
}'
After a moment, you should get a similar output:
{
"took": 5119,
"timed_out": false,
"total": 5889,
"updated": 0,
"created": 5889,
"deleted": 0,
"batches": 6,
"version_conflicts": 0,
"noops": 0,
"retries": {
"bulk": 0,
"search": 0
},
"throttled_millis": 0,
"requests_per_second": -1.0,
"throttled_until_millis": 0,
"failures": []
}
Ensure new index has been created#
Run the following command, and ensure the new index is like the current one (size can vary):
curl -XGET http://localhost:9200/_cat/indices\?v
The output should look like this:
health status index uuid pri rep docs.count docs.deleted store.size pri.store.size
green open new_cortex_4 wRX6rhzXTuW_F2wLNxqVyg 5 0 8531 0 12.6mb 12.6mb
green open cortex_4 Y5rDTO23RBC_n6pjFP0-Qw 5 0 8531 8 13mb 13mb
Delete old indices#
This is the thrilling part.
Now the new index new_cortex_4
is created and similar to cortex_4
, older indexes should be completely deleted from the database. To delete index named cortex_4
, run the following command:
curl -XDELETE http://localhost:9200/cortex_4
Run the same command for older indexes if exist (cortex_3, cortex_2....). Elasticsearch 7.x cannot run with index created with Elasticsearch 5.x.
Create an alias#
Before stopping Elasticsearch service, let’s create an alias to keep index names in the future.
curl -XPOST -H 'Content-Type: application/json' 'http://localhost:9200/_aliases' -d '{
"actions": [
{
"add": {
"index": "new_cortex_4",
"alias": "cortex_4"
}
}
]
}'
Doing so will allow Cortex 3.1.0 to find the index without updating the configuration file.
Check the alias has been well created by running the following command
curl -XGET http://localhost:9200/_alias?pretty
The output should look like:
{
"new_cortex_4" : {
"aliases" : {
"cortex_4" : { }
}
}
}
Stop Elasticsearch version 6.8.2#
sudo service elasticsearch stop
Update Elasticsearch#
Update the configuration of Elastisearch. Configuration file should look like this:
[..]
http.host: 127.0.0.1
discovery.type: single-node
cluster.name: hive
script.allowed_types: inline
thread_pool.search.queue_size: 100000
thread_pool.write.queue_size: 10000
Now, upgrade Elasticsearch to version 7.x following the documentation for your operating system, and ensure the service start successfully.
Install or update to Cortex 3.1.0#
-
Download the package along with its SHA256 checksum and signature files.
Destination path
Replace
/path/to/
with the full local directory path where you want to save the downloaded files.-
Using Wget:
wget -O /path/to/<file_name>.deb https://cortex.download.strangebee.com/<major.minor_version>/deb/<file_name>.deb wget -O /path/to/<file_name>.deb.sha256 https://cortex.download.strangebee.com/<major.minor_version>/sha256/<file_name>.deb.sha256 wget -O /path/to/<file_name>.deb.asc https://cortex.download.strangebee.com/<major.minor_version>/asc/<file_name>.deb.asc
Example:
wget -O /tmp/cortex_3.2.1-2_all.deb https://cortex.download.strangebee.com/3.2/deb/cortex_3.2.1-2_all.deb wget -O /tmp/cortex_3.2.1-2_all.deb.sha256 https://cortex.download.strangebee.com/3.2/sha256/cortex_3.2.1-2_all.deb.sha256 wget -O /tmp/cortex_3.2.1-2_all.deb.asc https://cortex.download.strangebee.com/3.2/asc/cortex_3.2.1-2_all.deb.asc
-
Using cURL:
curl -o /path/to/<file_name>.deb https://cortex.download.strangebee.com/<major.minor_version>/deb/<file_name>.deb curl -o /path/to/<file_name>.deb.sha256 https://cortex.download.strangebee.com/<major.minor_version>/sha256/<file_name>.deb.sha256 curl -o /path/to/<file_name>.deb.asc https://cortex.download.strangebee.com/<major.minor_version>/asc/<file_name>.deb.asc
Example:
curl -o /tmp/cortex_3.2.1-2_all.deb https://cortex.download.strangebee.com/3.2/deb/cortex_3.2.1-2_all.deb curl -o /tmp/cortex_3.2.1-2_all.deb.sha256 https://cortex.download.strangebee.com/3.2/sha256/cortex_3.2.1-2_all.deb.sha256 curl -o /tmp/cortex_3.2.1-2_all.deb.asc https://cortex.download.strangebee.com/3.2/asc/cortex_3.2.1-2_all.deb.asc
-
-
Verify the integrity of the downloaded package.
-
Check the SHA256 checksum by comparing it with the provided value.
a. Generate the SHA256 checksum of your downloaded package.
sha256sum /path/to/<file_name>.deb
b. Compare the output hash with the official SHA256 value listed in the .sha256 file.
c. If both hashes match exactly, the file integrity is verified. If not, the file may be corrupted or tampered with—don't proceed with installation, and contact the StrangeBee Security Team.
-
Verify the GPG signature using the public key.
a. Download the public key at keys.download.strangebee.com using Wget or cURL.
wget -O /path/to/strangebee.gpg https://keys.download.strangebee.com/latest/gpg/strangebee.gpg
curl -o /path/to/strangebee.gpg https://keys.download.strangebee.com/latest/gpg/strangebee.gpg
b. Import the key into your GPG keyring.
gpg --import /path/to/strangebee.gpg
c. Verify the downloaded package signature.
gpg --verify /path/to/<file_name>.deb.asc /path/to/<file_name>.deb
d. You should see a message stating indicating that the signature is valid and the package is authentic. If you see warnings or errors, don't install the package as its integrity or authenticity can't be confirmed. Report the issue to the StrangeBee Security Team.
-
-
Install the package.
-
Using
apt-get
to manage dependencies automatically:sudo apt-get install /path/to/<file_name>.deb
-
Using
dpkg
:sudo dpkg -i /path/to/<file_name>.deb
Missing dependencies
While using
dpkg
, you might encounter warnings about missing dependencies during installation. To resolve this, run the commands described in the Required packages section of the installation guide.
-
-
When switching from the previous repository-based installation, remove the old repository to avoid stale configurations.
Why you should remove the old repository
Keeping the old repository configuration can cause your system to download outdated or conflicting packages during updates, potentially breaking your installation or causing unexpected behavior.
sudo rm /etc/apt/sources.list.d/strangebee.list sudo apt-get update
-
Download the package along with its SHA256 checksum and signature files.
Destination path
Replace
/path/to/
with the full local directory path where you want to save the downloaded files.-
Using Wget:
wget -O /path/to/<file_name>.rpm https://cortex.download.strangebee.com/<major.minor_version>/rpm/<file_name>.rpm wget -O /path/to/<file_name>.rpm.sha256 https://cortex.download.strangebee.com/<major.minor_version>/sha256/<file_name>.rpm.sha256 wget -O /path/to/<file_name>.rpm.asc https://cortex.download.strangebee.com/<major.minor_version>/asc/<file_name>.rpm.asc
Example:
wget -O /tmp/cortex-3.2.1-2.noarch.rpm https://cortex.download.strangebee.com/3.2/rpm/cortex-3.2.1-2.noarch.rpm wget -O /tmp/cortex-3.2.1-2.noarch.rpm.sha256 https://cortex.download.strangebee.com/3.2/sha256/cortex-3.2.1-2.noarch.rpm.sha256 wget -O /tmp/cortex-3.2.1-2.noarch.rpm.asc https://cortex.download.strangebee.com/3.2/asc/cortex-3.2.1-2.noarch.rpm.asc
-
Using cURL:
curl -o /path/to/<file_name>.rpm https://cortex.download.strangebee.com/<major.minor_version>/rpm/<file_name>.rpm curl -o /path/to/<file_name>.rpm.sha256 https://cortex.download.strangebee.com/<major.minor_version>/sha256/<file_name>.rpm.sha256 curl -o /path/to/<file_name>.rpm.asc https://cortex.download.strangebee.com/<major.minor_version>/asc/<file_name>.rpm.asc
Example:
curl -o /tmp/cortex-3.2.1-2.noarch.rpm https://cortex.download.strangebee.com/3.2/rpm/cortex-3.2.1-2.noarch.rpm curl -o /tmp/cortex-3.2.1-2.noarch.rpm.sha256 https://cortex.download.strangebee.com/3.2/sha256/cortex-3.2.1-2.noarch.rpm.sha256 curl -o /tmp/cortex-3.2.1-2.noarch.rpm.asc https://cortex.download.strangebee.com/3.2/asc/cortex-3.2.1-2.noarch.rpm.asc
-
-
Verify the integrity of the downloaded package.
-
Check the SHA256 checksum by comparing it with the provided value.
a. Generate the SHA256 checksum of your downloaded package.
sha256sum /path/to/<file_name>.rpm
b. Compare the output hash with the official SHA256 value listed in the .sha256 file.
c. If both hashes match exactly, the file integrity is verified. If not, the file may be corrupted or tampered with—don't proceed with installation, and contact the StrangeBee Security Team.
-
Verify the GPG signature using the public key.
a. Download the public key at keys.download.strangebee.com using Wget or cURL.
wget -O /path/to/strangebee.gpg https://keys.download.strangebee.com/latest/gpg/strangebee.gpg
curl -o /path/to/strangebee.gpg https://keys.download.strangebee.com/latest/gpg/strangebee.gpg
b. Import the key into your GPG keyring.
gpg --import /path/to/strangebee.gpg
c. Verify the downloaded package signature.
gpg --verify /path/to/<file_name>.rpm.asc /path/to/<file_name>.rpm
d. You should see a message stating indicating that the signature is valid and the package is authentic. If you see warnings or errors, don't install the package as its integrity or authenticity can't be confirmed. Report the issue to the StrangeBee Security Team.
-
-
Install the package.
-
Using
yum
to manage dependencies automatically:sudo yum install /path/to/<file_name>.rpm
-
Using
dnf
to manage dependencies automatically:sudo dnf install /path/to/<file_name>.rpm
-
Using
rpm
:sudo rpm -ivh /path/to/<file_name>.rpm
Missing dependencies
While using
rpm
, you might encounter warnings about missing dependencies during installation. To resolve this, run the commands described in the Required packages section of the installation guide.
-
-
When switching from the previous repository-based installation, deactivate the old repository to avoid stale configurations.
Why you should deactivate the old repository
Keeping the old repository configuration can cause your system to download outdated or conflicting packages during updates, potentially breaking your installation or causing unexpected behavior.
a. Deactivate the repository.
```bash sudo yum-config-manager --disable strangebee ```
b. Optional: Remove the repository file.
```bash sudo rm /etc/yum.repos.d/strangebee.repo sudo yum clean all ```
Docker images are also provided on Docker Hub.
docker pull thehiveproject/cortex:3.1.0-1
⚠️ Starting from this version, Docker images don't contain analyzers anymore. Analyzers__/__Responders and Cortex have different life-cycles, their update including their dependencies should not be correlated to Cortex update.
It is recommended to use Docker version of analyzers : this can be done by binding Docker service docket inside Cortex container (run with -v /var/run/docker.sock:/var/run/docker.sock
).
Update database#
Connect to TheHive (and Cortex), the maintenance page should ask to update.
Once updated, ensure a new index named cortex_5
has been created.
curl -XGET http://localhost:9200/_cat/indices\?v
The output should look like this:
health status index uuid pri rep docs.count docs.deleted store.size pri.store.size
green open new_cortex_4 GV-3Y8QjTjWw0F-p2sjW6Q 5 0 30977 0 26mb 26mb
yellow open cortex_5 Nz0vCKqhRK2xkx1t_WF-0g 5 1 30977 0 26.1mb 26.1mb