Tutorial (Data "beaconization")

Let's start by the simplest case. Imagine that you have:

• Metadata (including phenotypic data) in your system, labeled according to your internal nomenclature.
• A VCF file.

STEP 1

First of all, we are going to convert your metadata (sequencing methodology, bioinformatics tools, phenotypic data, etc.) to the format of the Beacon v2 Models. As input we will be using this XLSX template.

The first thing that needs to be done is to map/convert your metadata so that it follows the syntax of the provided XLSX file.

Once you have filled the Excel file then you can proceed to validate it. At this stage, it's normal that you have doubts regarding your mapping to Beacon v2 syntax. Fortunately, B2RI's utility bff-validator will help you complete this task. The validator checks that all values in the XLSX match the specifications present in the Beacon v2 Models default schemas. In technical terms, it's called validating data against JSON Schemas.

./utils/bff_validator/bff-validator -i your_xlsx_file.xlsx --out-dir my_bff_dir

When you run it, it's very likely that you'll find have errors/warnings on your data. The script will catch them and explain the cause of the error. Please address all the issues at the XLSX. You can run the script as many times you want :-)

At some point you won't get any errors. By now the script should have created 6 text files (what we call the Beacon Friendly Format). The files are in JSON format (JSON arrays) and will be used later (STEP 3).

Congratulations! Now you can go to STEP 2.

STEP 2

Now that you have processed the metadata, it's time to process the VCF file.

The VCF file has to be gzipped (or bgzipped). What we are going to do it's to annotate it (or re-annotate it if your file already has annotations) with SnpEff and SnpSift and transform the format so that it becames the 7th BFF file (i.e., genomicVariationsVcf.json).

./beacon vcf   -n 1    -i input.vcf.gz -p param_file
     |      |      |         |              |
     exe    mode   #cores   <vcf>           parameters file (optional)

Here we are using beacon script in mode vcf. This mode is one of the three available [vcf|mongodb|full].

The parameters file is needed if you want to change the default values. Note that you must provide the reference genome (unless you're using hg19 which is the default one) that was used to create your VCF. See all the script options here.

If something is wrong with the input files, the script will complain and provide possible solutions.

Once completed, you will end up with a dir like this one beacon_XXXXXXX/vcf. Inside, you will find genomicVariationsVcf.json.gz, the 7th BFF file.

Now that you have the 7 JSON files it's time to go to the STEP 3.

STEP 3

The objective of this step is to load (a.k.a. ingest) the 7 JSON files into MongoDB. Once loaded in MongoDB, they are named collections.

For doing this we will use again beacon script, but this time in mode mongodb.

Let's assume that we have the 6 files from STEP 1 in the directory my_bff_dir and the file from STEP 2 at beacon_XXXXXXX.

We will add these values to the parameters file:

---
bff:
  metadatadir: my_bff_dir
  # You can change the name of the JSON files
  runs: runs.json
  cohorts: cohorts.json
  biosamples: biosamples.json
  individuals: individuals.json
  analyses: analyses.json
  datasets: datasets.json
  # Note that genomicVariationsVcf is not affected by <metadatadir>
  genomicVariationsVcf: beacon_XXXXXXX/vcf/genomicVariationsVcf.json.gz

Finally you execute this command

./beacon mongodb -p param_file
     |      |      |
     exe    mode   paramaters file.

If everything goes well, all your data should be loaded into an instance of MongoDB.

Congratulations! You can now go to the STEP 4.

STEP 4

Start making queries with the API.

Cheers!

Manu