Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Set up docker for genenetwork #29

Open
wants to merge 30 commits into
base: master
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from 24 commits
Commits
Show all changes
30 commits
Select commit Hold shift + click to select a range
f64227a
update package.json to get it to install
npklein Jun 11, 2019
5a67a7a
Merge branch 'master' of https://github.com/molgenis/molgenis-app-gen…
npklein Jul 18, 2019
a97b231
add docker files
npklein Jul 25, 2019
3d6dc5d
fixes to make page open, api not working yet
npklein Jul 25, 2019
c21fde7
need host: redis otherwise will try to connect to localhost
npklein Jul 26, 2019
a4b3adf
add elasticsearch
npklein Jul 26, 2019
c0fc4f7
domain should point to nothing
npklein Jul 26, 2019
4cb5a38
update lcoal for metabrain
npklein Jul 26, 2019
500b544
add scripts for keeping
npklein Jul 26, 2019
14d8eaf
remove min to work locally
npklein Jul 26, 2019
7f7f934
session also hast o use redis as host
npklein Jul 26, 2019
01bf6d4
elasticsearch is on the network elasticsearch, not localhost
npklein Jul 27, 2019
0c3faab
change network to elasticsearch
npklein Jul 27, 2019
26a3484
update docker-compose for connecting elasticsearch to webapp container
npklein Jul 27, 2019
f905fb6
fix typo
npklein Jul 27, 2019
112f739
networks was not necesarry
npklein Jul 28, 2019
3d2472e
separate Dockerfile for prediction server and pwas server
npklein Jul 28, 2019
b3a1a47
fill elasticsearch with entrypoint script
npklein Jul 29, 2019
72bc6a5
remove extra CMD lines and add entrypoint script
npklein Jul 29, 2019
59d5fe5
add fake gene pred. scores
npklein Jul 29, 2019
62ec3fd
remove networks, add predictionserver and pwasserver
npklein Jul 29, 2019
cd169d7
connect to redis container and use local node path
npklein Jul 29, 2019
ee9f746
upgrade package (not sure if necesasrry)
npklein Jul 29, 2019
e450091
configs have to be in same directory as pwas and prediction server Do…
npklein Jul 29, 2019
720b580
update
npklein Jul 30, 2019
551f94a
add min back
npklein Jul 30, 2019
8013ee7
add domain back
npklein Jul 30, 2019
44460b2
change back to genenetwork locations
npklein Jul 30, 2019
acec6fa
change back to genenetwork
npklein Jul 30, 2019
a281665
chagne to genenetwork
npklein Jul 30, 2019
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions .dockerignore
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
node_modules
npm-debug.log
30 changes: 30 additions & 0 deletions Dockerfile
Original file line number Diff line number Diff line change
@@ -0,0 +1,30 @@
FROM node:11.9.0

# Set the working directory to /app
WORKDIR /app

# Copy the current directory contents into the container at /app
COPY package*.json entrypoint.sh ./

# Install any needed packages specified in requirements.txt
RUN npm install
# If you are building your code for production
# RUN npm ci --only=production

#

RUN npm build
RUN npm i natives

# Bundle app source
COPY . .

# Make port 80 available to the world outside this container
EXPOSE 80
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

do not use expose in the dockerfile only specify it in the docker-compose file


# Fill elasticsearch with entrypoint.sh script
RUN chmod +x entrypoint.sh # if not already executable
ENTRYPOINT ["/app/entrypoint.sh"]

# Run app.py when the container launches
CMD ["npm", "start"]
7 changes: 6 additions & 1 deletion api/controllers/requestHandlers/handlePathwayAnalysis.js
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,12 @@ var dbutil = require('../utils/dbutil')
var genedesc = require('../utils/genedesc')
var quicksortobj = require('../utils/quicksortobj')

var Queue = kue.createQueue() // this does not create a new queue, it's just redis access
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

need to test if this would break anything

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should I change it back? Without this change docker won't work

Copy link
Collaborator

@royoelen royoelen Jul 30, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This breaks running locally, will investigate how to make it work in both instances

var Queue = kue.createQueue({
redis:{
host: 'redis'
}
}) // this does not create a new queue, it's just redis access

var lastAnalysisRequest = null
var lastAnalysisResult = null

Expand Down
2 changes: 1 addition & 1 deletion config/domain.js
Original file line number Diff line number Diff line change
@@ -1,3 +1,3 @@
module.exports = {
domain: 'https://www.genenetwork.nl'
npklein marked this conversation as resolved.
Show resolved Hide resolved
domain: ''
};
38 changes: 17 additions & 21 deletions config/local.js
Original file line number Diff line number Diff line change
Expand Up @@ -6,38 +6,34 @@

module.exports = {

genesToTranscripts: '/data/genenetwork/files/genesToTranscripts.txt',
genesToTranscripts: '/data/metabrainnetwork/files/genesToTranscripts.txt',
npklein marked this conversation as resolved.
Show resolved Hide resolved
transcriptMappingFile: 'xxx',
mim2gene: '/data/genenetwork/files/new/mim2gene.txt',
mim2gene: '/data/metabrainnetwork/files/new/mim2gene.txt',

geneDescFile: '/data/genenetwork/files/new/ENSGToGeneNameHGNCBiotypeChromosomeStartStopStrandAndDescriptionV83FilteredNonChromosomesRemovedDuplicateTranscriptsRemoved.txt',
genePredScoreFile: '/data/genenetwork/files/new/skewnessSummary.txt',
geneDBPath: '/data/genenetwork/level/new/dbgenes_uint16be',
pathwayDBPath: '/data/genenetwork/level/new/dbexternal_uint16be',
geneDescFile: '/data/metabrainnetwork/files/new/ENSGToGeneNameHGNCBiotypeChromosomeStartStopStrandAndDescriptionV83FilteredNonChromosomesRemovedDuplicateTranscriptsRemoved.txt',
genePredScoreFile: '/data/metabrainnetwork/files/new/skewnessSummary.txt',
geneDBPath: '/data/metabrainnetwork/level/new/dbgenes_uint16be',
pathwayDBPath: '/data/metabrainnetwork/level/new/dbexternal_uint16be',

transcriptDBpath: '/data/genenetwork/level/new/transcriptdb',
transcriptBarsDBpath: '/data/genenetwork/level/new/transcriptbars',
tissuecorrelationDBPath: '/data/genenetwork/level/new/tissuedb',
transcriptDBpath: '/data/metabrainnetwork/level/new/transcriptdb',
transcriptBarsDBpath: '/data/metabrainnetwork/level/new/transcriptbars',
tissuecorrelationDBPath: '/data/metabrainnetwork/level/new/tissuedb',

// geneDescFile: '/data/genenetwork/files/ENSGToGeneNameHGNCBiotypeChromosomeStartStopStrandAndDescriptionV75.txt.filtered.txt',
// geneDBPath: '/data/genenetwork/level/dbgenes_uint16be',
// pathwayDBPath: '/data/genenetwork/level/dbexternal_uint16be',

celltypeDBPath: '/data/genenetwork/level/new/celltypedb',
correlationDBPath: '/data/genenetwork/level/new/dbpccorrelationzscores_uint16be_genescompsstdnorm',
hpocorrelationDB: '/data/genenetwork/level/new/hpocorrelationdb',
celltypeDBPath: '/data/metabrainnetwork/level/new/celltypedb',
correlationDBPath: '/data/metabrainnetwork/level/new/dbpccorrelationzscores_uint16be_genescompsstdnorm',
hpocorrelationDB: '/data/metabrainnetwork/level/new/hpocorrelationdb',
//correlationDBPath: '/srv/molgenis/dbpccorrelationzscores_uint16be',
requestDBPath: '/data/genenetwork/level/new/dbreq',
networkShortURLDBPath: '/data/genenetwork/level/new/dbnetworkurls',
requestDBPath: '/data/metabrainnetwork/level/new/dbreq',
networkShortURLDBPath: '/data/metabrainnetwork/level/new/dbnetworkurls',

svgUploadDir: '/data/genenetwork/uploads/svg/',
genelistUploadDir: '/data/genenetwork/uploads/genelist',
svgUploadDir: '/data/metabrainnetwork/uploads/svg/',
genelistUploadDir: '/data/metabrainnetwork/uploads/genelist',

networkFontFamily: 'Geogrotesque Lg',
networkFontFile: 'assets/fonts/Geogtq-Lg.svg',

useElastic: true,
elasticHost: 'localhost:9200',
elasticHost: 'elasticsearch:9200',
elasticLogLevel: 'debug',

pubmine: {
Expand Down
100 changes: 100 additions & 0 deletions config/localDocker.js
Original file line number Diff line number Diff line change
@@ -0,0 +1,100 @@
/**
* Local environment settings
* In a production environment, you probably want to leave this file out
* entirely and leave all your settings in env/production.js
*/

module.exports = {

genesToTranscripts: '/app/data/metabrainnetwork/files/genesToTranscripts.txt',
transcriptMappingFile: 'xxx',
mim2gene: '/app/data/metabrainnetwork/files/new/mim2gene.txt',

geneDescFile: '/app/data/metabrainnetwork/files/new/ENSGToGeneNameHGNCBiotypeChromosomeStartStopStrandAndDescriptionV83FilteredNonChromosomesRemovedDuplicateTranscriptsRemoved.txt',
genePredScoreFile: '/app/data/metabrainnetwork/files/new/skewnessSummary.txt',
geneDBPath: '/app/data/metabrainnetwork/level/new/dbgenes_uint16be',
pathwayDBPath: '/app/data/metabrainnetwork/level/new/dbexternal_uint16be',

transcriptDBpath: '/app/data/metabrainnetwork/level/new/transcriptdb',
transcriptBarsDBpath: '/app/data/metabrainnetwork/level/new/transcriptbars',
tissuecorrelationDBPath: '/app/data/metabrainnetwork/level/new/tissuedb',

celltypeDBPath: '/app/data/metabrainnetwork/level/new/celltypedb',
correlationDBPath: '/app/data/metabrainnetwork/level/new/dbpccorrelationzscores_uint16be_genescompsstdnorm',
hpocorrelationDB: '/app/data/metabrainnetwork/level/new/hpocorrelationdb',
//correlationDBPath: '/srv/molgenis/dbpccorrelationzscores_uint16be',
requestDBPath: '/app/data/metabrainnetwork/level/new/dbreq',
networkShortURLDBPath: '/app/data/metabrainnetwork/level/new/dbnetworkurls',

svgUploadDir: '/app/data/metabrainnetwork/uploads/svg/',
genelistUploadDir: '/app/data/metabrainnetwork/uploads/genelist',

networkFontFamily: 'Geogrotesque Lg',
networkFontFile: 'assets/fonts/Geogtq-Lg.svg',

useElastic: true,
elasticHost: 'elasticsearch:9200',
elasticLogLevel: 'debug',

pubmine: {
journalFile: '/app/data/pubmine/GeneticsJournalsImpactFactorAboveThree.txt',
startYear: 2000,
stopYear: 2014,
impactFactorBins: [3, 5, 10],
numTopJournals: 5
},

/***************************************************************************
* Your SSL certificate and key, if you want to be able to serve HTTP *
* responses over https:// and/or use websockets over the wss:// protocol *
* (recommended for HTTP, strongly encouraged for WebSockets) *
* *
* In this example, we'll assume you created a folder in your project, *
* `config/ssl` and dumped your certificate/key files there: *
***************************************************************************/

// ssl: {
// ca: require('fs').readFileSync(__dirname + './ssl/my_apps_ssl_gd_bundle.crt'),
// key: require('fs').readFileSync(__dirname + './ssl/my_apps_ssl.key'),
// cert: require('fs').readFileSync(__dirname + './ssl/my_apps_ssl.crt')
// },

/***************************************************************************
* The `port` setting determines which TCP port your app will be *
* deployed on. *
* *
* Ports are a transport-layer concept designed to allow many different *
* networking applications run at the same time on a single computer. *
* More about ports: *
* http://en.wikipedia.org/wiki/Port_(computer_networking) *
* *
* By default, if it's set, Sails uses the `PORT` environment variable. *
* Otherwise it falls back to port 1337. *
* *
* In env/production.js, you'll probably want to change this setting *
* to 80 (http://) or 443 (https://) if you have an SSL certificate *
***************************************************************************/

port: process.env.PORT || 1337,

/***************************************************************************
* The runtime "environment" of your Sails app is either typically *
* 'development' or 'production'. *
* *
* In development, your Sails app will go out of its way to help you *
* (for instance you will receive more descriptive error and *
* debugging output) *
* *
* In production, Sails configures itself (and its dependencies) to *
* optimize performance. You should always put your app in production mode *
* before you deploy it to a server. This helps ensure that your Sails *
* app remains stable, performant, and scalable. *
* *
* By default, Sails sets its environment using the `NODE_ENV` environment *
* variable. If NODE_ENV is not set, Sails will run in the *
* 'development' environment. *
***************************************************************************/

environment: process.env.NODE_ENV || 'development',
hookTimeout: 120000 // 2 minutes
};
104 changes: 104 additions & 0 deletions config/localGeneNetwork.js
Original file line number Diff line number Diff line change
@@ -0,0 +1,104 @@
/**
* Local environment settings
* In a production environment, you probably want to leave this file out
* entirely and leave all your settings in env/production.js
*/

module.exports = {

genesToTranscripts: '/data/genenetwork/files/genesToTranscripts.txt',
transcriptMappingFile: 'xxx',
mim2gene: '/data/genenetwork/files/new/mim2gene.txt',

geneDescFile: '/data/genenetwork/files/new/ENSGToGeneNameHGNCBiotypeChromosomeStartStopStrandAndDescriptionV83FilteredNonChromosomesRemovedDuplicateTranscriptsRemoved.txt',
genePredScoreFile: '/data/genenetwork/files/new/skewnessSummary.txt',
geneDBPath: '/data/genenetwork/level/new/dbgenes_uint16be',
pathwayDBPath: '/data/genenetwork/level/new/dbexternal_uint16be',

transcriptDBpath: '/data/genenetwork/level/new/transcriptdb',
transcriptBarsDBpath: '/data/genenetwork/level/new/transcriptbars',
tissuecorrelationDBPath: '/data/genenetwork/level/new/tissuedb',

// geneDescFile: '/data/genenetwork/files/ENSGToGeneNameHGNCBiotypeChromosomeStartStopStrandAndDescriptionV75.txt.filtered.txt',
// geneDBPath: '/data/genenetwork/level/dbgenes_uint16be',
// pathwayDBPath: '/data/genenetwork/level/dbexternal_uint16be',

celltypeDBPath: '/data/genenetwork/level/new/celltypedb',
correlationDBPath: '/data/genenetwork/level/new/dbpccorrelationzscores_uint16be_genescompsstdnorm',
hpocorrelationDB: '/data/genenetwork/level/new/hpocorrelationdb',
//correlationDBPath: '/srv/molgenis/dbpccorrelationzscores_uint16be',
requestDBPath: '/data/genenetwork/level/new/dbreq',
networkShortURLDBPath: '/data/genenetwork/level/new/dbnetworkurls',

svgUploadDir: '/data/genenetwork/uploads/svg/',
genelistUploadDir: '/data/genenetwork/uploads/genelist',

networkFontFamily: 'Geogrotesque Lg',
networkFontFile: 'assets/fonts/Geogtq-Lg.svg',

useElastic: true,
elasticHost: 'elastichost',
elasticLogLevel: 'debug',

pubmine: {
journalFile: '/data/pubmine/GeneticsJournalsImpactFactorAboveThree.txt',
startYear: 2000,
stopYear: 2014,
impactFactorBins: [3, 5, 10],
numTopJournals: 5
},

/***************************************************************************
* Your SSL certificate and key, if you want to be able to serve HTTP *
* responses over https:// and/or use websockets over the wss:// protocol *
* (recommended for HTTP, strongly encouraged for WebSockets) *
* *
* In this example, we'll assume you created a folder in your project, *
* `config/ssl` and dumped your certificate/key files there: *
***************************************************************************/

// ssl: {
// ca: require('fs').readFileSync(__dirname + './ssl/my_apps_ssl_gd_bundle.crt'),
// key: require('fs').readFileSync(__dirname + './ssl/my_apps_ssl.key'),
// cert: require('fs').readFileSync(__dirname + './ssl/my_apps_ssl.crt')
// },

/***************************************************************************
* The `port` setting determines which TCP port your app will be *
* deployed on. *
* *
* Ports are a transport-layer concept designed to allow many different *
* networking applications run at the same time on a single computer. *
* More about ports: *
* http://en.wikipedia.org/wiki/Port_(computer_networking) *
* *
* By default, if it's set, Sails uses the `PORT` environment variable. *
* Otherwise it falls back to port 1337. *
* *
* In env/production.js, you'll probably want to change this setting *
* to 80 (http://) or 443 (https://) if you have an SSL certificate *
***************************************************************************/

port: process.env.PORT || 1337,

/***************************************************************************
* The runtime "environment" of your Sails app is either typically *
* 'development' or 'production'. *
* *
* In development, your Sails app will go out of its way to help you *
* (for instance you will receive more descriptive error and *
* debugging output) *
* *
* In production, Sails configures itself (and its dependencies) to *
* optimize performance. You should always put your app in production mode *
* before you deploy it to a server. This helps ensure that your Sails *
* app remains stable, performant, and scalable. *
* *
* By default, Sails sets its environment using the `NODE_ENV` environment *
* variable. If NODE_ENV is not set, Sails will run in the *
* 'development' environment. *
***************************************************************************/

environment: process.env.NODE_ENV || 'development',
hookTimeout: 120000 // 2 minutes
};
Loading