BIDS to NIDM-E.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# BIDS dataset to a JSON-LD NIDM-Experiment dataset"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "(work under conda environment)\n",
    "This document illustrates an example showing how to go from a BIDS dataset to a JSON-LD NIDM-experiment dataset.\n",
    "We pick our bids dataset from DataLad where the hierarchy of folders follows the bids standard. For this example, we import the ABIDE DataLad dataset from CMU_a site : http://datasets.datalad.org/?dir=/abide/RawDataBIDS/CMU_a "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "conda install -c conda-forge git-annex\n",
    "pip install datalad \n",
    "datalad install ///abide/RawDataBIDS/CMU_a"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We use the BIDSMRI2NIDM.py file from PyNIDM/nidm/experiment/tools.\n",
    "This program will convert a BIDS MRI dataset to a NIDM-Experiment RDF document. It will parse phenotype information and simply store variables/values and link to the associated json data dictionary file.\n",
    "Argument used: \n",
    "- -d [root directory of BIDS dataset]\n",
    "- -jsonld [If flag set, output is json-ld not TURTLE]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "python ~/PyNIDM/nidm/experiment/tools/BIDSMRI2NIDM.py -d ~/bids_dataset/CMU_a/"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The Datalad hosted abide dataset doesn’t have participants.tsv files and doesn’t have a dataset_description.json file so it is really not even valid BIDS. The participants.tsv file is optional but we must have a datasaet_description.json file. \n",
    "One has to create some basic dataset_description.json files for each abide and adhd200 site so one could run bidsmri2nidm on it and add it in the github repo.\n",
    "For instance see https://bids-specification.readthedocs.io/en/stable/03-modality-agnostic-files.html to put the required fields in the dataset_description.json file. \n",
    "Below our dataset_description.json file:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "{\n",
    "\t\"Name\": \"ABIDE dataset CMU_a Site\",\n",
    "\t\"BIDSVersion\": \"1.0.1\",\n",
    "\t\"License\": \"CC BY-SA 4.0\"\n",
    "\t\n",
    "}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "One must also parse other json files available at the root of the bids directory: T1w.json and taskxxx.json files. \n",
    "if key from T1w.json file or taskxxx.json file is mapped to term in BIDS_Constants.py then add to NIDM object. \n",
    "See function bidsmri2project(directory, args):\n",
    "- for dataset_description.json file "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def bidsmri2project(directory, args):\n",
    "    #Parse dataset_description.json file in BIDS directory\n",
    "    if (os.path.isdir(os.path.join(directory))):\n",
    "        try:\n",
    "            with open(os.path.join(directory,'dataset_description.json')) as data_file:\n",
    "                dataset = json.load(data_file)\n",
    "        except OSError:\n",
    "            logging.critical(\"Cannot find dataset_description.json file which is required in the BIDS spec\")\n",
    "            exit(\"-1\")\n",
    "    else:\n",
    "        logging.critical(\"Error: BIDS directory %s does not exist!\" %os.path.join(directory))\n",
    "        exit(\"-1\")\n",
    "\n",
    "    #create project / nidm-exp doc\n",
    "    project = Project()\n",
    "\n",
    "    #add various attributes if they exist in BIDS dataset\n",
    "    for key in dataset:\n",
    "        #if key from dataset_description file is mapped to term in BIDS_Constants.py then add to NIDM object\n",
    "        if key in BIDS_Constants.dataset_description:\n",
    "            if type(dataset[key]) is list:\n",
    "                project.add_attributes({BIDS_Constants.dataset_description[key]:\"\".join(dataset[key])})\n",
    "            else:\n",
    "                project.add_attributes({BIDS_Constants.dataset_description[key]:dataset[key]})\n",
    "        #add absolute location of BIDS directory on disk for later finding of files which are stored relatively in NIDM document\n",
    "        project.add_attributes({Constants.PROV['Location']:directory})\n",
    "\n",
    "    #get BIDS layout\n",
    "    bids_layout = BIDSLayout(directory) "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- For T1w.json file "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "\n",
    "            if file_tpl.entities['datatype']=='anat':\n",
    "                #do something with anatomicals\n",
    "                acq_obj = MRObject(acq)\n",
    "                #add image contrast type\n",
    "                if file_tpl.entities['suffix'] in BIDS_Constants.scans:\n",
    "                    acq_obj.add_attributes({Constants.NIDM_IMAGE_CONTRAST_TYPE:BIDS_Constants.scans[file_tpl.entities['suffix']]})\n",
    "                else:\n",
    "                    logging.info(\"WARNING: No matching image contrast type found in BIDS_Constants.py for %s\" % file_tpl.entities['suffix'])\n",
    "\n",
    "                #add image usage type\n",
    "                if file_tpl.entities['datatype'] in BIDS_Constants.scans:\n",
    "                    acq_obj.add_attributes({Constants.NIDM_IMAGE_USAGE_TYPE:BIDS_Constants.scans[file_tpl.entities['datatype']]})\n",
    "                else:\n",
    "                    logging.info(\"WARNING: No matching image usage type found in BIDS_Constants.py for %s\" % file_tpl.entities['datatype'])\n",
    "                #add file link\n",
    "                #make relative link to\n",
    "                acq_obj.add_attributes({Constants.NIDM_FILENAME:getRelPathToBIDS(join(file_tpl.dirname,file_tpl.filename), directory)})\n",
    "\n",
    "                #add sha512 sum\n",
    "                if isfile(join(directory,file_tpl.dirname,file_tpl.filename)):\n",
    "                    acq_obj.add_attributes({Constants.CRYPTO_SHA512:getsha512(join(directory,file_tpl.dirname,file_tpl.filename))})\n",
    "                else:\n",
    "                    logging.info(\"WARNINGL file %s doesn't exist! No SHA512 sum stored in NIDM files...\" %join(directory,file_tpl.dirname,file_tpl.filename))\n",
    "                #get associated JSON file if exists\n",
    "                #There is T1w.json file with information \n",
    "                json_data = (bids_layout.get(suffix=file_tpl.entities['suffix'],subject=subject_id))[0].metadata\n",
    "                if len(json_data.info)>0:\n",
    "                    for key in json_data.info.items():\n",
    "                        if key in BIDS_Constants.json_keys:\n",
    "                            if type(json_data.info[key]) is list:\n",
    "                                acq_obj.add_attributes({BIDS_Constants.json_keys[key.replace(\" \", \"_\")]:''.join(str(e) for e in json_data.info[key])})\n",
    "                            else:\n",
    "                                acq_obj.add_attributes({BIDS_Constants.json_keys[key.replace(\" \", \"_\")]:json_data.info[key]})\n",
    "                   \n",
    "                #Parse T1w.json file in BIDS directory to add the attributes contained inside\n",
    "                if (os.path.isdir(os.path.join(directory))):\n",
    "                    try:\n",
    "                        with open(os.path.join(directory,'T1w.json')) as data_file:\n",
    "                            dataset = json.load(data_file)\n",
    "                    except OSError:\n",
    "                        logging.critical(\"Cannot find T1w.json file which is required in the BIDS spec\")\n",
    "                        exit(\"-1\")\n",
    "                else:\n",
    "                    logging.critical(\"Error: BIDS directory %s does not exist!\" %os.path.join(directory))\n",
    "                    exit(\"-1\")\n",
    "\n",
    "                #add various attributes if they exist in BIDS dataset\n",
    "                for key in dataset:\n",
    "                    #if key from T1w.json file is mapped to term in BIDS_Constants.py then add to NIDM object\n",
    "                    if key in BIDS_Constants.json_keys:\n",
    "                        if type(dataset[key]) is list:\n",
    "                            acq_obj.add_attributes({BIDS_Constants.json_keys[key]:\"\".join(dataset[key])})\n",
    "                        else:\n",
    "                            acq_obj.add_attributes({BIDS_Constants.json_keys[key]:dataset[key]})     "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- for taskxxx.json file "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "                                                          \n",
    "            elif file_tpl.entities['datatype'] == 'func':\n",
    "                #do something with functionals\n",
    "                acq_obj = MRObject(acq)\n",
    "                #add image contrast type\n",
    "                if file_tpl.entities['suffix'] in BIDS_Constants.scans:\n",
    "                    acq_obj.add_attributes({Constants.NIDM_IMAGE_CONTRAST_TYPE:BIDS_Constants.scans[file_tpl.entities['suffix']]})\n",
    "                else:\n",
    "                    logging.info(\"WARNING: No matching image contrast type found in BIDS_Constants.py for %s\" % file_tpl.entities['suffix'])\n",
    "\n",
    "                #add image usage type\n",
    "                if file_tpl.entities['datatype'] in BIDS_Constants.scans:\n",
    "                    acq_obj.add_attributes({Constants.NIDM_IMAGE_USAGE_TYPE:BIDS_Constants.scans[file_tpl.entities['datatype']]})\n",
    "                else:\n",
    "                    logging.info(\"WARNING: No matching image usage type found in BIDS_Constants.py for %s\" % file_tpl.entities['datatype'])\n",
    "                #make relative link to\n",
    "                acq_obj.add_attributes({Constants.NIDM_FILENAME:getRelPathToBIDS(join(file_tpl.dirname,file_tpl.filename), directory)})\n",
    "                 #add sha512 sum\n",
    "                if isfile(join(directory,file_tpl.dirname,file_tpl.filename)):\n",
    "                    acq_obj.add_attributes({Constants.CRYPTO_SHA512:getsha512(join(directory,file_tpl.dirname,file_tpl.filename))})\n",
    "                else:\n",
    "                    logging.info(\"WARNINGL file %s doesn't exist! No SHA512 sum stored in NIDM files...\" %join(directory,file_tpl.dirname,file_tpl.filename))\n",
    "\n",
    "                if 'run' in file_tpl.entities:\n",
    "                    acq_obj.add_attributes({BIDS_Constants.json_keys[\"run\"]:file_tpl.entities['run']})\n",
    "\n",
    "                #get associated JSON file if exists\n",
    "                json_data = (bids_layout.get(suffix=file_tpl.entities['suffix'],subject=subject_id))[0].metadata\n",
    "\n",
    "                if len(json_data.info)>0:\n",
    "                    for key in json_data.info.items():\n",
    "                        if key in BIDS_Constants.json_keys:\n",
    "                            if type(json_data.info[key]) is list:\n",
    "                                acq_obj.add_attributes({BIDS_Constants.json_keys[key.replace(\" \", \"_\")]:''.join(str(e) for e in json_data.info[key])})\n",
    "                            else:\n",
    "                                acq_obj.add_attributes({BIDS_Constants.json_keys[key.replace(\" \", \"_\")]:json_data.info[key]})\n",
    "                #get associated events TSV file\n",
    "                if 'run' in file_tpl.entities:\n",
    "                    events_file = bids_layout.get(subject=subject_id, extensions=['.tsv'],modality=file_tpl.entities['datatype'],task=file_tpl.entities['task'],run=file_tpl.entities['run'])\n",
    "                else:\n",
    "                    events_file = bids_layout.get(subject=subject_id, extensions=['.tsv'],modality=file_tpl.entities['datatype'],task=file_tpl.entities['task'])\n",
    "                #if there is an events file then this is task-based so create an acquisition object for the task file and link\n",
    "                if events_file:\n",
    "                    #for now create acquisition object and link it to the associated scan\n",
    "                    events_obj = AcquisitionObject(acq)\n",
    "                    #add prov type, task name as prov:label, and link to filename of events file\n",
    "\n",
    "                    events_obj.add_attributes({PROV_TYPE:Constants.NIDM_MRI_BOLD_EVENTS,BIDS_Constants.json_keys[\"TaskName\"]: json_data[\"TaskName\"], Constants.NIDM_FILENAME:getRelPathToBIDS(events_file[0].filename, directory)})\n",
    "                    #link it to appropriate MR acquisition entity\n",
    "                    events_obj.wasAttributedTo(acq_obj)\n",
    "                    \n",
    "                #Parse task-rest_bold.json file in BIDS directory to add the attributes contained inside\n",
    "                if (os.path.isdir(os.path.join(directory))):\n",
    "                    try:\n",
    "                        with open(os.path.join(directory,'task-rest_bold.json')) as data_file:\n",
    "                            dataset = json.load(data_file)\n",
    "                    except OSError:\n",
    "                        logging.critical(\"Cannot find task-rest_bold.json file which is required in the BIDS spec\")\n",
    "                        exit(\"-1\")\n",
    "                else:\n",
    "                    logging.critical(\"Error: BIDS directory %s does not exist!\" %os.path.join(directory))\n",
    "                    exit(\"-1\")\n",
    "\n",
    "                #add various attributes if they exist in BIDS dataset\n",
    "                for key in dataset:\n",
    "                    #if key from task-rest_bold.json file is mapped to term in BIDS_Constants.py then add to NIDM object\n",
    "                    if key in BIDS_Constants.json_keys:\n",
    "                        if type(dataset[key]) is list:\n",
    "                            acq_obj.add_attributes({BIDS_Constants.json_keys[key]:\",\".join(map(str,dataset[key]))})\n",
    "                        else:\n",
    "                            acq_obj.add_attributes({BIDS_Constants.json_keys[key]:dataset[key]}) "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The idea behind is to create acquisition objects for each scan for each subject using the information available in the bids directory. \n",
    "Below, an extract of the nidme.json document created with the acquisition informations for one subject: "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    " {\n",
    "            \"@id\": \"nidm:8edfb514-b761-11e9-8f81-e1efbef5a165\",\n",
    "            \"@type\": [\n",
    "                \"Entity\",\n",
    "                \"AcquisitionObject\"\n",
    "            ],\n",
    "            \"crypto:sha512\": \"840affc90aeda40601a48f8d5cd3421fb73e0cbecc3111c4a3194aef74a716b732e7649b8b36d55f0b5559a88531ac10acd00bf27a95d52b59da78a88b7eeb8f\",\n",
    "            \"dicom:AcquisitionMatrix\": \"256x256\",\n",
    "            \"dicom:EchoTime\": 0.00248,\n",
    "            \"dicom:FlipAngle\": {\n",
    "                \"@type\": \"xsd:int\",\n",
    "                \"@value\": \"8\"\n",
    "            },\n",
    "            \"dicom:InversionTime\": 1.1,\n",
    "            \"dicom:MagneticFieldStrength\": {\n",
    "                \"@type\": \"xsd:int\",\n",
    "                \"@value\": \"3\"\n",
    "            },\n",
    "            \"dicom:PixelBandwidth\": 170.0,\n",
    "            \"dicom:RepetitionTime\": 1.87,\n",
    "            \"dicom:ScanningSequence\": \"MPRAGE\",\n",
    "            \"nidm:PhaseEncodingDirection\": \"j-\",\n",
    "            \"hadAcquisitionModality\": {\n",
    "                \"@id\": \"nidm:MagneticResonanceImaging\"\n",
    "            },\n",
    "            \"hadImageContrastType\": {\n",
    "                \"@id\": \"nidm:T1Weighted\"\n",
    "            },\n",
    "            \"HadImageUsageType\": {\n",
    "                \"@id\": \"nidm:Anatomical\"\n",
    "            },\n",
    "            \"filename\": \"/udd/nperez/sub-0050656/anat/sub-0050656_T1w.nii.gz\",\n",
    "            \"prov:wasGeneratedBy\": {\n",
    "                \"@id\": \"nidm:8edfacae-b761-11e9-8f81-e1efbef5a165\"\n",
    "            }\n",
    "        },"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "However, some limitations appear: \n",
    "- we only extract the data that map to BIDS_Constants.py and not all bids attributes are referenced in this document \n",
    "- for bval and bvec files, what to do with those?"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}