Tutorial (Data "beaconization")

In this tutorial it is expected to already have the Data ingestion tools and Beacon V2 REST API downloaded, installed and set up following the instructions in Download & Installation.

Let's start by the simplest case. Imagine that you have:

• Metadata (including phenotypic data) in your system, labelled according to your internal nomenclature.
• A VCF file.

Previous steps

Connect to the container

The beacon container is running in detached mode or in the background. To connect, you should invoke from your terminal:

docker exec -ti beacon2-ri-tools bash

Then:

docker cp your_data.vcf.gz beacon2-ri-tools:/your_data.vcf.gz

To see all the running container names, execute the next command:

docker ps -a

Now you should be inside the beacon2-ri-tools container to start the Data beaconization.

STEP 1

First, we are going to convert your metadata (sequencing methodology, bioinformatics tools, phenotypic data, etc.) to the format of the Beacon v2 Models. As input, we will be using this XLSX template.

The first thing you need to do is manually map/convert your metadata to match the syntax of the provided XLSX file, ensuring you fill it out accurately.

Once you have completed filling out the Excel file, you can proceed to validate it. At this stage, it’s common to have uncertainties about your mapping to Beacon v2 syntax. Luckily, the B2RI utility bff-validator is there to assist you with this task. The validator ensures that all values in the XLSX file conform to the specifications outlined in the Beacon v2 Models default schemas. Technically, this process is referred to as validating data against JSON Schemas.

./utils/bff_validator/bff-validator -i your_xlsx_file.xlsx --out-dir my_bff_dir

When you run it, it's highly probable that you'll encounter errors or warnings related to your data. The bff-validator tool is designed to detect these errors and provide explanations for the validation failures. Please ensure to address all the issues within the XLSX file. Feel free to run the script as many times as needed. :-)

At some point, you won't encounter any validation errors. By then, the script should have generated 6 text files, which we refer to as the Beacon Friendly Format. These files are in JSON format (specifically, JSON arrays) and will be utilized later in STEP 3.

Congratulations! Now you can go to STEP 2.

STEP 2

Now that you have processed the metadata, it's time to process the VCF file.

The VCF file has to be gzipped (or bgzipped). What we are going to do it's to annotate it (or re-annotate it if your file already has annotations) with SnpEff and SnpSift and transform the format so that it becames the 7th BFF file (i.e., genomicVariationsVcf.json).

./beacon vcf   -n 1    -i input.vcf.gz -p param_file
     |      |      |         |              |
     exe    mode   #cores   <vcf>           parameters file (optional)

Here we are using beacon script in mode vcf. This mode is one of the three available [vcf|mongodb|full].

The parameters file is optional if you want to use the default value (hg19) but it is needed if you want to change them. Note that you must provide the reference genome (unless you're using hg19 which is the default one) that was used to create your VCF. See all the script options here.

The param_file should look something like this: genome:hs37d5g

If something is wrong with the input files, the script will complain and provide possible solutions.

Once completed, you will end up with a dir like this one beacon_XXXXXXX/vcf. Inside, you will find genomicVariationsVcf.json.gz, the 7th BFF file.

Now that you have the 7 JSON files it's time to go to the STEP 3.

STEP 3

The objective of this step is to load (a.k.a. ingest) the 7 JSON files into MongoDB. Once loaded in MongoDB, they are named collections.

For doing this we will use again beacon script, but this time in mode mongodb.

Let's assume that we have the 6 files from STEP 1 in the directory my_bff_dir and the file from STEP 2 at beacon_XXXXXXX.

We will add these values to a new parameters (YAML) file:

---
bff:
  metadatadir: my_bff_dir
  # You can change the name of the JSON files
  runs: runs.json
  cohorts: cohorts.json
  biosamples: biosamples.json
  individuals: individuals.json
  analyses: analyses.json
  datasets: datasets.json
  # Note that genomicVariationsVcf is not affected by <metadatadir>
  genomicVariationsVcf: beacon_XXXXXXX/vcf/genomicVariationsVcf.json.gz

Finally, you execute this command

./beacon mongodb -p param_file
     |      |      |
     exe    mode   paramaters file.

If everything goes well, all your data should be loaded into an instance of MongoDB.

Congratulations! You can now go to the STEP 4.

STEP 4

You can create the necessary indexes running the following Python script.

docker exec beacon python beacon/reindex.py

Fetch the ontologies and extract the filtering terms

This step consists of analyzing all the collections of the Mongo database for first extracting the ontology OBO files and then filling the filtering terms endpoint with the information of the data loaded in the database.

You can automatically fetch the ontologies and extract the filtering terms running the following script:

docker exec beacon python beacon/db/extract_filtering_terms.py

Get descendant and semantic similarity terms

If you have the ontologies loaded and the filtering terms extracted, you can automatically get their descendant and semantic similarity terms running the following script:

docker exec beacon python beacon/db/get_descendants.py

STEP 5

Start making queries with the API.

Cheers!

Manu