Summary

The Beacon v2 Reference Implementation is a software that must be installed locally in a Linux server/workstation.

The software consists of several components that are distributed in two different GitHub repositories, one for data ingestion tools and another for the API. The software also needs external dependencies to work. In this regard, we provide several alternatives for download and installation.

Before starting, it will be necessary to install:

• Mongoimport
• Docker + Docker-compose

Containerized (Recommended)

Method 1: All containers in one

Setting up all the containers, beacon2-ri-tools and beacon2-ri-api together using the beacon2-ri-api repository.

Clone beacon2-ri-api GitHub repository.

git clone https://github.com/EGA-archive/beacon2-ri-api.git

From now on, the commands should be executed from the deploy directory.

cd beacon2-ri-api/deploy

Light up all the containers.

docker network create my-app-network
docker-compose up -d --build

Deploying external tools in beacon2-ri-tools container

This step will inject the external tools and DBs into the image and modify the configuration files. It will also run a test to check that the installation was successful. Note that running deploy_external_tools.sh will take some time (and disk space!!!).

Go inside beacon2-ri-tools container.

docker exec -it deploy_beacon-ri-tools_1 bash
bash /usr/share/beacon-ri/beacon2-ri-tools/BEACON/bin/deploy_external_tools.sh

With mongo-express it is possible to see the contents of the data base at http://localhost:8081.

For more installation options, please follow the instructions provided in the README.

Method 2: Independent containers

Download the beacon2-ri-tools container and the beacon2-ri-api container independently.

Data ingestion tools (beacon2-ri-tools)

Containerized options to install the Data ingestion tools repository.

Option 1: Build the container from Docker Hub

A set of Docker images can be obtained from Docker Hub. Please see the documentation there.

Option 2: Build the container from the Dockerfile in the GitHub repository

Download the Dockerfile from Github by typing:

wget https://raw.githubusercontent.com/EGA-archive/beacon2-ri-tools/main/Dockerfile

Then, execute the following commands to build the container (~1.1G):

docker build -t crg/beacon2_ri:latest .

Then:

docker run -tid --name beacon2-ri-tools crg/beacon2_ri:latest # run the image detached
docker exec -ti beacon2-ri-tools bash # connect to the container interactively

Deploy external tools needed

This step will inject the external tools and DBs into the image and modify the configuration files. It will also run a test to check that the installation was succesful. Note that running deploy_external_tools.sh will take some time (and disk space!!!).

bash beacon2-ri-tools/BEACON/bin/deploy_external_tools.sh

To see more information regarding the Data ingestion tools repository, please follow the instructions provided in the README.

Beacon v2 REST API (beacon2-ri-api)

Containerized options to install the Data ingestion tools repository.

Clone beacon2-ri-api GitHub repository.

git clone https://github.com/EGA-archive/beacon2-ri-api.git

All the commands should be executed from the deploy directory.

cd deploy

Light up only the containers needed:

docker network create my-app-network
docker-compose up -d --build training-ui db mongo-express beacon permissions idp idp-db

With mongo-express we can see the contents of the database at http://localhost:8081.

For more installation options, please follow the instructions provided in the README.

Now you should have the two things downloaded, installed and set up: the beacon2-ri-tools and the beacon2-ri-api. Together (Method 1) or independently (Method 2)

You can start transforming your data to BFF and loading it to the database following the Data beaconization tutorial.

Non containerized (data ingestion tools)

We will download the data ingestion tools from this repository. Please follow the instructions provided in the README.

The data ingestion tools need external software to function:

• BCFtools (version 1.15.1)
• SnpEff + databases (version 5.0)
• MongoDB

We will download BCFtools, SnpEff and MongoDB utilities from a public ftp server (ftp://xfer13.crg.eu) located at CRG. We will use wget to get the five parts (~65G total). Each part should take around 20 min to download:

wget ftp://FTPuser:FTPusersPassword@xfer13.crg.eu:221/beacon2_data.md5
wget ftp://FTPuser:FTPusersPassword@xfer13.crg.eu:221/beacon2_data.part1
wget ftp://FTPuser:FTPusersPassword@xfer13.crg.eu:221/beacon2_data.part2
wget ftp://FTPuser:FTPusersPassword@xfer13.crg.eu:221/beacon2_data.part3
wget ftp://FTPuser:FTPusersPassword@xfer13.crg.eu:221/beacon2_data.part4
wget ftp://FTPuser:FTPusersPassword@xfer13.crg.eu:221/beacon2_data.part5

Once you have downloaded the 5 parts and the checksum file (*.md5) please check that they are complete:

md5sum beacon2_data.part? > my_beacon2_data.md5
diff my_beacon2_data.md5 beacon2_data.md5

Now join the 5 parts by typing (note that momentarily we'll be using ~128G):

cat beacon2_data.part? > beacon2_data.tar.gz 
rm beacon2_data.part?

OK, everything ready to untar the file:

tar -xvf beacon2_data.tar.gz
cd snpeff/v5.0 ; ln -s GRCh38.99 hg38 # In case the symbolic link does not exist already

NB: Feel free now to erase beacon2_data.tar.gz if needed.

Great! now we recommend moving the directories to your favourite location and keep the path. We will be using the paths to set up some variables for SnpEff and for beacon configuration files.

For SnpEff:

Use the path where you have left the databases and use it to change data.dir variable in snpEff.config file (located in SnpEff installation folder). For instance, in my case:

#data.dir = ./data/

data.dir = /media/mrueda/4TB/Databases/snpeff/v5.0/

And finally....

For Beacon:

Open the file config.yaml (inside beacon_X.X.X dir installation) and change the paths to the files/exes according to your new locations. Note that SnpSift is part of SnpEff main distribution.

You can start transforming your data to BFF and loading it to the database following the Data beaconization tutorial.

That's it!

MongoDB installation

We're going to install it by using docker-compose. You need to have docker and docker-compose installed. docker-compose enables defining and running multi-container Docker applications.

First you need to create a file named docker-compose.yml with these contents:

version: '3.1'

# networks:
#   beacon-priv:
#   idp-priv:
#   pub:

services:

  ###########################################
  # MongoDB Database
  ###########################################

  mongo:
    image: mongo
    hostname: mongo
    ports:
      - 27017:27017
    environment:
      MONGO_INITDB_ROOT_USERNAME: root
      MONGO_INITDB_ROOT_PASSWORD: example
    # networks:
    #   - beacon-priv

  mongo-express:
    image: mongo-express
    restart: always
    ports:
      - 8081:8081
    environment:
      ME_CONFIG_MONGODB_ADMINUSERNAME: root
      ME_CONFIG_MONGODB_ADMINPASSWORD: example
      ME_CONFIG_MONGODB_URL: mongodb://root:example@mongo:27017/
    # networks:
    #   - beacon-priv

And then run:

docker-compose up -d

Once complete you should have two Docker processes: mongo and mongo-express. You can check this by typing:

docker ps -a

Mongo Express is an open source lightweight web-based administrative interface deployed to manage MongoDB databases interactively. You can access mongo-express at http://localhost:8081.

References