Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

déploiement de l'application sur le server shiny de Migale #18

Open
mandiayba opened this issue Dec 15, 2020 · 19 comments
Open

déploiement de l'application sur le server shiny de Migale #18

mandiayba opened this issue Dec 15, 2020 · 19 comments
Assignees

Comments

@mandiayba
Copy link
Member

Déploiement de l'application onto-relation-vizualisation sur le server shiny hébergé par Migale. Une application shiny à déployer sur le server shiny de Migale doit être dans une image docker et avoir un script de lancement selon. Le dockerfile et le script de lancement seront localisés dans le projet des apps shiny https://forgemia.inra.fr/migale/shinyproxy-docker.

Étapes

  1. création du script de lancement de l'application
  2. création de la recette docker permettant de construire l'image de l'application (Dockerfile). La recette doit (1) installer les dépendances nécessaires pour l'application, (2) créer un dossier dédié à l'application (/opt/ontoRelationVisualizer) où le code de l'application sera copié, (3) déposer le script de lancement de l'application (run.R) dans le dossier de l'application (4) ajouter une petite doc d'utilisation
  3. création et déploiement de l'image docker de l'application dans la VM du server shiny à partir du projet des apps shiny
@mandiayba mandiayba self-assigned this Dec 15, 2020
@mandiayba
Copy link
Member Author

Le HOME de l'application dans le serveur shiny sera /opt/ontoRelationVisualizer

Le contenu du script de lancement inspiré de la doc de l'appli :

$ cat run.R 
library(shiny)
options(shiny.port = 3838)
options(shiny.host = "0.0.0.0")
runApp(appDir = "/opt/ontoRelationVisualizer/inrae/", launch.browser = FALSE)

@mandiayba
Copy link
Member Author

mandiayba commented Dec 15, 2020

La modification de l'emplacement du dossier data/ sera utile.

Propose de mettre le dossier data/ sous le HOME de l'application.

/opt/ontoRelationVisualizer/data

Il faudra ainsi éditer les lignes de codes concernées pour cette modification

@mandiayba
Copy link
Member Author

mandiayba commented Dec 15, 2020

We have the run script and data folder containing the ontologies

The dockerfile :

$ cat Dockerfile 
#1.__ base image 
FROM migale/centos-shiny-migale:latest

#2.__ dependences of the shiny app
RUN mkdir /usr/share/doc/R-3.6.0/html/
RUN R -e "install.packages(c('shiny', 'rmarkdown', 'DT', 'future', 'httr', 'jsonlite', 'promises', 'shiny', 'shinyjs', 'shinyWidgets', 'tidyverse', 'visNetwork', 'ontologyIndex', 'devtools', 'sankeyD3'), repos='https://cloud.r-project.org')"
RUN R -e "install.packages('dplyr', repos='https://cloud.r-project.org')"
RUN R -q -e "remotes::install_github('fbreitwieser/sankeyD3', INSTALL_opts=c('--no-help', '--no-html'))"

#3.__ home directory of the app 
RUN mkdir -p /opt/ontoRelationVisualizer

#4.__ source code of the app
RUN git clone https://github.com/Bibliome/onto-relation-vizualisation.git /opt/ontoRelationVisualizer

#5.__ trigger of the app 
COPY run.R /opt/ontoRelationVisualizer/run.R
COPY data /opt/ontoRelationVisualizer/data

#6.__ metadata of the app 
LABEL software="onto-relation-vizualisation"
LABEL software.version="v0.0.0"
LABEL about.summary="Interface for the synthetic visualization of relational data with ontologies (Florilege)."
LABEL about.home="https://github.com/Bibliome/onto-relation-vizualisation"
LABEL about.documentation="https://github.com/Bibliome/onto-relation-vizualisation"
LABEL extra.identifiers="ontoRelationVisualizer"
LABEL about.tags="data, analysis, 16S, interactive"
LABEL about.license_file="https://github.com/Bibliome/onto-relation-vizualisation/blob/main/LICENSE"
LABEL about.license="GNU GENERAL PUBLIC LICENSE"
LABEL author="Mouhamadou Ba, [email protected]"

WORKDIR /opt/ontoRelationVisualizer

@mandiayba
Copy link
Member Author

mandiayba commented Dec 17, 2020

La modification de l'emplacement du dossier data/ sera utile.

Propose de mettre le dossier data/ sous le HOME de l'application.

/opt/ontoRelationVisualizer/data

Il faudra ainsi éditer les lignes de codes concernées pour cette modification

Revoir globalement les chemins relatifs dans le code qui causent des erreurs.
Les chemins semblent résolus au niveau du serveur à partir du HOME de l'application qui aura le contenu suivant :

$ls  /opt/ontoRelationVisualizer/
INRAE-onto-visualization.Rproj
LICENSE
README.md
build.sh
data/
docker
inrae/
run.R
run.sh

Les modifications vont se faire dans la branche deploy-2-migale-server crée pour le deploiement

@mandiayba
Copy link
Member Author

Problème de chemins relatifs résolu avec les modifications dans la branche deploy-2-migale-server

@mandiayba
Copy link
Member Author

L'application apparaît dans la liste des apps

image

mais l'application met un temps long à charger...

image

... avant de répondre par une erreur 502

image

Nous avons cependant au moins pu afficher la page d'accueil en local

image

@mandiayba
Copy link
Member Author

mandiayba commented Dec 18, 2020

on a modifié container-wait-time: 120000 et ça passe https://shiny.migale.inrae.fr/app/ontoRelationVizualizer

image

@mandiayba
Copy link
Member Author

mandiayba commented Dec 21, 2020

Application déployée et fonctionne sur l'url https://shiny.migale.inrae.fr/app/ontoRelationVizualizer

Une lenteur constatée, elle est apparemment due au démarrage du conteneur docker à chaque nouvelle utilisation de l'application : le conteneur est arrêté après un moment d'inactivé et un nouveau conteneur est crée au lancement de l'appli.

@mandiayba
Copy link
Member Author

mandiayba commented Jan 29, 2021

Solutions tentées sans succès pour résoudre le problème de lenteur de lancement de l'app (qui abouti à une erreur 502)

  • image docker de l'app allégée en montant le dossier des ontologies au lancement et en supprimant les libs utilisés
  • container-wait-time augmenté jusqu'à 300000 ms

Cela ne résout pas le problème, les logs affichent ceci avec le container qui est activé au bout du 49e essai.

avec container-wait-time : 120000

2021-01-29 14:50:24.202  INFO 1 --- [  XNIO-2 task-6] c.s.docker.client.DefaultDockerClient    : Starting container with Id: f2a0dab12993bd0e4522cce30c61d249292bccc5a7909674677b70b17b372dff
2021-01-29 14:50:26.658  WARN 1 --- [  XNIO-2 task-6] e.o.shinyproxy.ShinyProxyTestStrategy    : Container unresponsive, trying again (2/60): http://f2a0dab12993:3838
...
2021-01-29 14:51:58.737  WARN 1 --- [  XNIO-2 task-6] e.o.shinyproxy.ShinyProxyTestStrategy    : Container unresponsive, trying again (48/60): http://f2a0dab12993:3838
2021-01-29 14:52:00.739  WARN 1 --- [  XNIO-2 task-6] e.o.shinyproxy.ShinyProxyTestStrategy    : Container unresponsive, trying again (49/60): http://f2a0dab12993:3838
2021-01-29 14:52:02.998  INFO 1 --- [  XNIO-2 task-6] e.o.containerproxy.service.ProxyService  : Proxy activated [user: cV6g8Jc7jlVt9OhymFHOpcxdwHwl5PDz81-hfOYA] [spec: ontoRelationVisualizer] [id: f843c3cf-36bd-4544-92eb-206522ec0ddb]

avec container-wait-time : 300000

2021-01-29 17:24:00.842  INFO 1 --- [  XNIO-2 task-3] c.s.docker.client.DefaultDockerClient    : Starting container with Id: 8f5b82297fe13e7324984c87fd2bd821f7e4eadbb6e8c94a8389e5cb1ac56efc
2021-01-29 17:24:03.552  WARN 1 --- [  XNIO-2 task-3] e.o.shinyproxy.ShinyProxyTestStrategy    : Container unresponsive, trying again (2/150): http://8f5b82297fe1:3838
...
2021-01-29 17:25:37.655  WARN 1 --- [  XNIO-2 task-3] e.o.shinyproxy.ShinyProxyTestStrategy    : Container unresponsive, trying again (49/150): http://8f5b82297fe1:3838
2021-01-29 17:25:39.929  INFO 1 --- [  XNIO-2 task-3] e.o.containerproxy.service.ProxyService  : Proxy activated [user: cV6g8Jc7jlVt9OhymFHOpcxdwHwl5PDz81-hfOYA] [spec: ontoRelationVisualizer] [id: 12482c51-140d-4839-b716-9ab2769c2b16]
2021-01-29 17:26:52.558  INFO 1 --- [tiveProxyKiller] e.o.c.service.HeartbeatService           : Releasing inactive proxy [user: cV6g8Jc7jlVt9OhymFHOpcxdwHwl5PDz81-hfOYA] [spec: ontoRelationVisualizer] [id: 12482c51-140d-4839-b716-9ab2769c2b16] [silence: 72649ms]

avec container-wait-time : 3000000

2021-01-29 17:44:20.327  INFO 1 --- [  XNIO-2 task-3] c.s.docker.client.DefaultDockerClient    : Starting container with Id: 08c6b20cc23dcb76df5e0045ef21f9abd92f1f52fc46abc2302381d816edcbf6
2021-01-29 17:44:22.995  WARN 1 --- [  XNIO-2 task-3] e.o.shinyproxy.ShinyProxyTestStrategy    : Container unresponsive, trying again (2/1500): http://08c6b20cc23d:3838
...
2021-01-29 17:45:53.088  WARN 1 --- [  XNIO-2 task-3] e.o.shinyproxy.ShinyProxyTestStrategy    : Container unresponsive, trying again (47/1500): http://08c6b20cc23d:3838
2021-01-29 17:45:55.340  INFO 1 --- [  XNIO-2 task-3] e.o.containerproxy.service.ProxyService  : Proxy activated [user: cV6g8Jc7jlVt9OhymFHOpcxdwHwl5PDz81-hfOYA] [spec: ontoRelationVisualizer] [id: 0ae8938b-f4c8-44c7-a752-85a981ce424a]

avec container-wait-time : 300000000

2021-01-29 17:57:12.025  INFO 1 --- [  XNIO-2 task-3] c.s.docker.client.DefaultDockerClient    : Starting container with Id: 51a927de9dfc939d66d3e92f3befdc7c11fd4c801cb5e79b7d41b26bc0185e78
2021-01-29 17:57:14.656  WARN 1 --- [  XNIO-2 task-3] e.o.shinyproxy.ShinyProxyTestStrategy    : Container unresponsive, trying again (2/150000): http://51a927de9dfc:3838
...
2021-01-29 17:58:44.746  WARN 1 --- [  XNIO-2 task-3] e.o.shinyproxy.ShinyProxyTestStrategy    : Container unresponsive, trying again (47/150000): http://51a927de9dfc:3838
2021-01-29 17:58:47.009  INFO 1 --- [  XNIO-2 task-3] e.o.containerproxy.service.ProxyService  : Proxy activated [user: cV6g8Jc7jlVt9OhymFHOpcxdwHwl5PDz81-hfOYA] [spec: ontoRelationVisualizer] [id: a594b460-2b2b-4dfc-bf1b-2796f6201e08]

L'application fonctionne cependant ensuite si on attend un peu et qu'on rafraîchit la page après l'erreur 502

@mandiayba
Copy link
Member Author

reprise des tests sur un server shiny dev sur migale. Aucune solution trouvée pour la lenteur de orv

Comparaison de temps de lancement entre OntoRelationVisualizer et Easy16S

  • orv
 ontoRelationVisualizer]# docker run -v /opt/shinyproxy-docker/ontoRelationVisualizer/data:/opt/ontoRelationVisualizer/data -it -p 3838:3838 migale/ontorelationvisualizer:latest

[root@a6d652a27023 ontoRelationVisualizer]# time Rscript --verbose run.R
running
  '/usr/lib64/R/bin/R --slave --no-restore --file=run.R'

Failed to create bus connection: No such file or directory
Warning in system("timedatectl", intern = TRUE) :
  running command 'timedatectl' had status 1

Listening on http://0.0.0.0:3838
^C

Execution halted

real	1m37.827s
user	1m34.093s
sys	0m1.730s
  • Easy16S
[root@shinyproxy-dev shinyproxy-docker]# docker run -it -p 3838:3838 migale/easy16S:latest
docker: invalid reference format: repository name must be lowercase.
See 'docker run --help'.
[root@shinyproxy-dev shinyproxy-docker]# docker run -it -p 3838:3838 migale/easy16s
[root@9cf2b816ec60 easy16S]# time Rscript --verbose run.R 
running
  '/usr/lib64/R/bin/R --slave --no-restore --file=run.R'

Warning in read.dcf(con) :
  invalid input found on input connection './easy16S//DESCRIPTION'

Listening on http://0.0.0.0:3838
^C

Execution halted

real	0m4.554s
user	0m0.430s
sys	0m0.134s

@mandiayba
Copy link
Member Author

Investigation poussée sur le temps de lancement

Pour une raison que j'ignore le garbage collector prend beaucoup de temps avant le lancement de orv, le lancement se fait trop tardivement comparé à d'autres outils

[root@e00ad1e0100d ontoRelationVisualizer]# R --verbose -e "shiny::runApp('/opt/ontoRelationVisualizer', host = '0.0.0.0', port = 3838)"
'verbose' and 'quietly' are both true; being verbose then ..
now dyn.load("/usr/lib64/R/library/methods/libs/methods.so") ...

R version 3.6.0 (2019-04-26) -- "Planting of a Tree"
Copyright (C) 2019 The R Foundation for Statistical Computing
Platform: x86_64-redhat-linux-gnu (64-bit)

R is free software and comes with ABSOLUTELY NO WARRANTY.
You are welcome to redistribute it under certain conditions.
Type 'license()' or 'licence()' for distribution details.

R is a collaborative project with many contributors.
Type 'contributors()' for more information and
'citation()' on how to cite R or R packages in publications.

Type 'demo()' for some demos, 'help()' for on-line help, or
'help.start()' for an HTML browser interface to help.
Type 'q()' to quit R.

'verbose' and 'quietly' are both true; being verbose then ..
'verbose' and 'quietly' are both true; being verbose then ..
now dyn.load("/usr/lib64/R/library/utils/libs/utils.so") ...
Garbage collection 1 = 0+0+1 (level 2) ... 
12.1 Mbytes of cons cells used (35%)
3.1 Mbytes of vectors used (5%)
'verbose' and 'quietly' are both true; being verbose then ..
now dyn.load("/usr/lib64/R/library/grDevices/libs/grDevices.so") ...
'verbose' and 'quietly' are both true; being verbose then ..
now dyn.load("/usr/lib64/R/library/graphics/libs/graphics.so") ...
'verbose' and 'quietly' are both true; being verbose then ..
now dyn.load("/usr/lib64/R/library/stats/libs/stats.so") ...
 ending setup_Rmainloop(): R_Interactive = 0 {main.c}
 >R_ReplConsole(): before "for(;;)" {main.c}
> shiny::runApp('/opt/ontoRelationVisualizer', host = '0.0.0.0', port = 3838)
now dyn.load("/usr/lib64/R/library/digest/libs/digest.so") ...
now dyn.load("/usr/lib64/R/library/rlang/libs/rlang.so") ...
now dyn.load("/usr/lib64/R/library/htmltools/libs/htmltools.so") ...
now dyn.load("/usr/lib64/R/library/Rcpp/libs/Rcpp.so") ...
Garbage collection 2 = 1+0+1 (level 0) ... 
17.4 Mbytes of cons cells used (51%)
4.2 Mbytes of vectors used (6%)
Garbage collection 3 = 2+0+1 (level 0) ... 
20.8 Mbytes of cons cells used (60%)
5.1 Mbytes of vectors used (8%)
now dyn.load("/usr/lib64/R/library/later/libs/later.so") ...
now dyn.load("/usr/lib64/R/library/magrittr/libs/magrittr.so") ...
now dyn.load("/usr/lib64/R/library/promises/libs/promises.so") ...
now dyn.load("/usr/lib64/R/library/httpuv/libs/httpuv.so") ...
now dyn.load("/usr/lib64/R/library/mime/libs/mime.so") ...
now dyn.load("/usr/lib64/R/library/ellipsis/libs/ellipsis.so") ...
now dyn.load("/usr/lib64/R/library/fastmap/libs/fastmap.so") ...
Garbage collection 4 = 3+0+1 (level 0) ... 
21.4 Mbytes of cons cells used (62%)
5.3 Mbytes of vectors used (8%)
Loading required package: shiny
now dyn.load("/usr/lib64/R/library/tools/libs/tools.so") ...
Garbage collection 5 = 4+0+1 (level 0) ... 
23.8 Mbytes of cons cells used (69%)
5.8 Mbytes of vectors used (9%)
now dyn.load("/usr/lib64/R/library/cachem/libs/cachem.so") ...
Garbage collection 6 = 5+0+1 (level 0) ... 
26.0 Mbytes of cons cells used (76%)
7.2 Mbytes of vectors used (11%)
Garbage collection 7 = 6+0+1 (level 0) ... 
27.2 Mbytes of cons cells used (79%)
7.5 Mbytes of vectors used (12%)
Garbage collection 8 = 7+0+1 (level 0) ... 
27.8 Mbytes of cons cells used (81%)
7.6 Mbytes of vectors used (12%)
Garbage collection 9 = 7+1+1 (level 1) ... 
26.7 Mbytes of cons cells used (78%)
7.1 Mbytes of vectors used (11%)
Garbage collection 10 = 8+1+1 (level 0) ... 
27.2 Mbytes of cons cells used (79%)
7.2 Mbytes of vectors used (11%)
Garbage collection 11 = 9+1+1 (level 0) ... 
30.6 Mbytes of cons cells used (89%)
8.3 Mbytes of vectors used (13%)
Garbage collection 12 = 9+2+1 (level 1) ... 
30.7 Mbytes of cons cells used (89%)
8.3 Mbytes of vectors used (13%)
Garbage collection 13 = 9+2+2 (level 2) ... 
30.9 Mbytes of cons cells used (43%)
8.3 Mbytes of vectors used (13%)
Garbage collection 14 = 10+2+2 (level 0) ... 
34.0 Mbytes of cons cells used (48%)
9.0 Mbytes of vectors used (14%)
Garbage collection 15 = 11+2+2 (level 0) ... 
35.3 Mbytes of cons cells used (50%)
9.2 Mbytes of vectors used (14%)
Garbage collection 16 = 12+2+2 (level 0) ... 
37.9 Mbytes of cons cells used (53%)
10.0 Mbytes of vectors used (16%)
Failed to create bus connection: No such file or directory
Warning in system("timedatectl", intern = TRUE) :
  running command 'timedatectl' had status 1
Garbage collection 17 = 13+2+2 (level 0) ... 
43.9 Mbytes of cons cells used (62%)
11.2 Mbytes of vectors used (17%)
Garbage collection 18 = 14+2+2 (level 0) ... 
46.6 Mbytes of cons cells used (66%)
11.8 Mbytes of vectors used (18%)
Garbage collection 19 = 15+2+2 (level 0) ... 
47.3 Mbytes of cons cells used (67%)
12.0 Mbytes of vectors used (19%)
Garbage collection 20 = 16+2+2 (level 0) ... 
48.8 Mbytes of cons cells used (69%)
12.4 Mbytes of vectors used (19%)
Garbage collection 21 = 17+2+2 (level 0) ... 
50.5 Mbytes of cons cells used (71%)
12.8 Mbytes of vectors used (20%)
Garbage collection 22 = 18+2+2 (level 0) ... 
51.6 Mbytes of cons cells used (73%)
13.1 Mbytes of vectors used (20%)
'envir' chosen:<environment: R_GlobalEnv>
encoding = "native.enc" chosen
--> parsed 3 expressions; now eval(.)ing them:

>>>> eval(expression_nr. 1 )
		 =================

> invert <- function(list, simplify = T) {
+     new_list <- names(list)
+     quote <- paste0("\"", as.character(list), "\"")
+     unquote <- as.cha .... [TRUNCATED] 
curr.fun: symbol <-
 .. after 'expression(invert <- function(list, simplify = T) {''    new_list <- names(list)''    quote <- paste0("\"", as.character(list), "\"")''    unquote <- as.character(list)''    names(new_list) <- if (simplify) {''        unquote''    }''    else {''        quote''    }''    return(new_list)''})'

>>>> eval(expression_nr. 2 )
		 =================

> is.valid <- function(x) {
+     require(shiny)
+     is.null(need(x, message = FALSE))
+ }
curr.fun: symbol <-
 .. after 'expression(is.valid <- function(x) {''    require(shiny)''    is.null(need(x, message = FALSE))''})'

>>>> eval(expression_nr. 3 )
		 =================

> listToReactive <- function(list) {
+     require(shiny)
+     do.call("reactiveValues", list)
+ }
curr.fun: symbol <-
 .. after 'expression(listToReactive <- function(list) {''    require(shiny)''    do.call("reactiveValues", list)''})'
Garbage collection 23 = 19+2+2 (level 0) ... 
55.4 Mbytes of cons cells used (78%)
14.8 Mbytes of vectors used (23%)
Garbage collection 24 = 20+2+2 (level 0) ... 
55.7 Mbytes of cons cells used (78%)
14.9 Mbytes of vectors used (23%)
Garbage collection 25 = 20+3+2 (level 1) ... 
55.6 Mbytes of cons cells used (78%)
14.8 Mbytes of vectors used (23%)
Garbage collection 26 = 21+3+2 (level 0) ... 
56.1 Mbytes of cons cells used (79%)
15.2 Mbytes of vectors used (24%)
Garbage collection 27 = 22+3+2 (level 0) ... 
56.8 Mbytes of cons cells used (80%)
15.6 Mbytes of vectors used (24%)
Garbage collection 28 = 23+3+2 (level 0) ... 
57.5 Mbytes of cons cells used (81%)
16.0 Mbytes of vectors used (25%)
Garbage collection 29 = 23+4+2 (level 1) ... 
56.7 Mbytes of cons cells used (80%)
15.8 Mbytes of vectors used (25%)
Garbage collection 30 = 24+4+2 (level 0) ... 
57.1 Mbytes of cons cells used (80%)
16.5 Mbytes of vectors used (26%)
Garbage collection 31 = 24+5+2 (level 1) ... 
62.6 Mbytes of cons cells used (88%)
22.5 Mbytes of vectors used (35%)
Garbage collection 32 = 24+5+3 (level 2) ... 
66.5 Mbytes of cons cells used (46%)
30.9 Mbytes of vectors used (48%)
Garbage collection 33 = 24+6+3 (level 1) ... 
88.6 Mbytes of cons cells used (61%)
54.2 Mbytes of vectors used (85%)
Garbage collection 34 = 24+6+4 (level 2) ... 
97.5 Mbytes of cons cells used (67%)
60.0 Mbytes of vectors used (48%)
Garbage collection 35 = 24+6+5 (level 2) ... 
136.9 Mbytes of cons cells used (46%)
105.8 Mbytes of vectors used (70%)
Garbage collection 36 = 24+6+6 (level 2) ... 
141.7 Mbytes of cons cells used (48%)
109.8 Mbytes of vectors used (49%)
Garbage collection 37 = 25+6+6 (level 0) ... 
152.7 Mbytes of cons cells used (52%)
154.0 Mbytes of vectors used (69%)
Garbage collection 38 = 25+6+7 (level 2) ... 
152.7 Mbytes of cons cells used (52%)
175.4 Mbytes of vectors used (65%)
Garbage collection 39 = 26+6+7 (level 0) ... 
152.7 Mbytes of cons cells used (52%)
207.0 Mbytes of vectors used (77%)
Garbage collection 40 = 26+6+8 (level 2) ... 
152.7 Mbytes of cons cells used (52%)
203.5 Mbytes of vectors used (63%)
Garbage collection 41 = 27+6+8 (level 0) ... 
152.7 Mbytes of cons cells used (52%)
263.2 Mbytes of vectors used (82%)
Garbage collection 42 = 27+7+8 (level 1) ... 
152.7 Mbytes of cons cells used (52%)
266.7 Mbytes of vectors used (83%)
Garbage collection 43 = 27+7+9 (level 2) ... 
268.3 Mbytes of cons cells used (47%)
276.4 Mbytes of vectors used (71%)
Garbage collection 44 = 27+7+10 (level 2) ... 
571.0 Mbytes of cons cells used (46%)
311.0 Mbytes of vectors used (67%)
Garbage collection 45 = 28+7+10 (level 0) ... 
890.5 Mbytes of cons cells used (71%)
368.6 Mbytes of vectors used (79%)
Garbage collection 46 = 29+7+10 (level 0) ... 
890.5 Mbytes of cons cells used (71%)
368.6 Mbytes of vectors used (79%)
Garbage collection 47 = 30+7+10 (level 0) ... 
890.5 Mbytes of cons cells used (71%)
389.7 Mbytes of vectors used (84%)
Garbage collection 48 = 30+8+10 (level 1) ... 
890.5 Mbytes of cons cells used (71%)
368.6 Mbytes of vectors used (79%)
Garbage collection 49 = 31+8+10 (level 0) ... 
890.5 Mbytes of cons cells used (71%)
368.6 Mbytes of vectors used (79%)
Garbage collection 50 = 32+8+10 (level 0) ... 
892.1 Mbytes of cons cells used (71%)
390.5 Mbytes of vectors used (84%)
Garbage collection 51 = 32+9+10 (level 1) ... 
899.2 Mbytes of cons cells used (72%)
394.2 Mbytes of vectors used (85%)
Garbage collection 52 = 32+9+11 (level 2) ... 
906.2 Mbytes of cons cells used (51%)
376.7 Mbytes of vectors used (67%)
Garbage collection 53 = 33+9+11 (level 0) ... 
923.5 Mbytes of cons cells used (52%)
401.7 Mbytes of vectors used (72%)
Garbage collection 54 = 34+9+11 (level 0) ... 
940.3 Mbytes of cons cells used (53%)
410.4 Mbytes of vectors used (73%)
Garbage collection 55 = 35+9+11 (level 0) ... 
957.0 Mbytes of cons cells used (54%)
419.1 Mbytes of vectors used (75%)
Garbage collection 56 = 35+10+11 (level 1) ... 
973.3 Mbytes of cons cells used (55%)
427.9 Mbytes of vectors used (77%)
Garbage collection 57 = 36+10+11 (level 0) ... 
989.3 Mbytes of cons cells used (56%)
436.5 Mbytes of vectors used (78%)
Garbage collection 58 = 37+10+11 (level 0) ... 
1004.9 Mbytes of cons cells used (57%)
444.9 Mbytes of vectors used (80%)
Garbage collection 59 = 38+10+11 (level 0) ... 
1020.1 Mbytes of cons cells used (58%)
453.0 Mbytes of vectors used (81%)
Garbage collection 60 = 38+11+11 (level 1) ... 
1035.1 Mbytes of cons cells used (58%)
461.0 Mbytes of vectors used (83%)
Garbage collection 61 = 38+11+12 (level 2) ... 
1049.7 Mbytes of cons cells used (59%)
460.9 Mbytes of vectors used (69%)
Garbage collection 62 = 39+11+12 (level 0) ... 
1064.1 Mbytes of cons cells used (60%)
468.6 Mbytes of vectors used (70%)
Garbage collection 63 = 40+11+12 (level 0) ... 
1078.1 Mbytes of cons cells used (61%)
476.2 Mbytes of vectors used (71%)
Garbage collection 64 = 41+11+12 (level 0) ... 
1092.0 Mbytes of cons cells used (62%)
484.5 Mbytes of vectors used (72%)
Garbage collection 65 = 42+11+12 (level 0) ... 
1105.6 Mbytes of cons cells used (62%)
492.8 Mbytes of vectors used (73%)
Garbage collection 66 = 43+11+12 (level 0) ... 
1118.9 Mbytes of cons cells used (63%)
501.0 Mbytes of vectors used (75%)
Garbage collection 67 = 44+11+12 (level 0) ... 
1131.9 Mbytes of cons cells used (64%)
509.0 Mbytes of vectors used (76%)
Garbage collection 68 = 45+11+12 (level 0) ... 
1221.2 Mbytes of cons cells used (69%)
565.9 Mbytes of vectors used (84%)
Garbage collection 69 = 45+11+13 (level 2) ... 
406.2 Mbytes of cons cells used (29%)
407.3 Mbytes of vectors used (61%)
Garbage collection 70 = 46+11+13 (level 0) ... 
546.0 Mbytes of cons cells used (38%)
487.0 Mbytes of vectors used (73%)
Garbage collection 71 = 47+11+13 (level 0) ... 
656.6 Mbytes of cons cells used (46%)
580.5 Mbytes of vectors used (86%)
Garbage collection 72 = 47+12+13 (level 1) ... 
521.9 Mbytes of cons cells used (37%)
480.4 Mbytes of vectors used (72%)
Garbage collection 73 = 48+12+13 (level 0) ... 
537.1 Mbytes of cons cells used (38%)
482.6 Mbytes of vectors used (72%)
Garbage collection 74 = 49+12+13 (level 0) ... 
545.9 Mbytes of cons cells used (38%)
519.0 Mbytes of vectors used (77%)
Garbage collection 75 = 50+12+13 (level 0) ... 
545.9 Mbytes of cons cells used (38%)
582.2 Mbytes of vectors used (87%)
Garbage collection 76 = 50+13+13 (level 1) ... 
545.9 Mbytes of cons cells used (38%)
536.5 Mbytes of vectors used (80%)
Garbage collection 77 = 50+13+14 (level 2) ... 
545.9 Mbytes of cons cells used (38%)
535.6 Mbytes of vectors used (66%)
Garbage collection 78 = 51+13+14 (level 0) ... 
597.3 Mbytes of cons cells used (42%)
596.1 Mbytes of vectors used (74%)
Garbage collection 79 = 52+13+14 (level 0) ... 
597.3 Mbytes of cons cells used (42%)
644.3 Mbytes of vectors used (80%)
Garbage collection 80 = 52+14+14 (level 1) ... 
632.0 Mbytes of cons cells used (45%)
574.8 Mbytes of vectors used (71%)
Garbage collection 81 = 53+14+14 (level 0) ... 
695.6 Mbytes of cons cells used (49%)
636.3 Mbytes of vectors used (79%)
Garbage collection 82 = 54+14+14 (level 0) ... 
695.6 Mbytes of cons cells used (49%)
701.6 Mbytes of vectors used (87%)
Garbage collection 83 = 54+15+14 (level 1) ... 
690.4 Mbytes of cons cells used (49%)
611.3 Mbytes of vectors used (76%)
Garbage collection 84 = 55+15+14 (level 0) ... 
734.9 Mbytes of cons cells used (52%)
628.3 Mbytes of vectors used (78%)
Garbage collection 85 = 55+16+14 (level 1) ... 
648.6 Mbytes of cons cells used (46%)
609.4 Mbytes of vectors used (75%)
Garbage collection 86 = 56+16+14 (level 0) ... 
697.8 Mbytes of cons cells used (49%)
643.1 Mbytes of vectors used (80%)
Garbage collection 87 = 57+16+14 (level 0) ... 
706.2 Mbytes of cons cells used (50%)
653.7 Mbytes of vectors used (81%)
Garbage collection 88 = 57+16+15 (level 2) ... 
558.9 Mbytes of cons cells used (39%)
571.3 Mbytes of vectors used (59%)
Garbage collection 89 = 58+16+15 (level 0) ... 
576.4 Mbytes of cons cells used (41%)
591.6 Mbytes of vectors used (61%)
Garbage collection 90 = 59+16+15 (level 0) ... 
595.9 Mbytes of cons cells used (42%)
616.0 Mbytes of vectors used (63%)
Garbage collection 91 = 60+16+15 (level 0) ... 
607.2 Mbytes of cons cells used (43%)
634.4 Mbytes of vectors used (65%)
Garbage collection 92 = 61+16+15 (level 0) ... 
622.3 Mbytes of cons cells used (44%)
654.5 Mbytes of vectors used (67%)

Listening on http://0.0.0.0:3838

@mandiayba
Copy link
Member Author

Garbage collector plus court avant lancement pour easy16S par exemple

[root@5d679f6ea33d easy16S]# R --verbose -e "shiny::runApp('/opt/easy16S/easy16S', host = '0.0.0.0', port = 3838)"
'verbose' and 'quietly' are both true; being verbose then ..
now dyn.load("/usr/lib64/R/library/methods/libs/methods.so") ...

R version 3.6.0 (2019-04-26) -- "Planting of a Tree"
Copyright (C) 2019 The R Foundation for Statistical Computing
Platform: x86_64-redhat-linux-gnu (64-bit)

R is free software and comes with ABSOLUTELY NO WARRANTY.
You are welcome to redistribute it under certain conditions.
Type 'license()' or 'licence()' for distribution details.

R is a collaborative project with many contributors.
Type 'contributors()' for more information and
'citation()' on how to cite R or R packages in publications.

Type 'demo()' for some demos, 'help()' for on-line help, or
'help.start()' for an HTML browser interface to help.
Type 'q()' to quit R.

'verbose' and 'quietly' are both true; being verbose then ..
'verbose' and 'quietly' are both true; being verbose then ..
now dyn.load("/usr/lib64/R/library/utils/libs/utils.so") ...
Garbage collection 1 = 0+0+1 (level 2) ... 
12.1 Mbytes of cons cells used (35%)
3.1 Mbytes of vectors used (5%)
'verbose' and 'quietly' are both true; being verbose then ..
now dyn.load("/usr/lib64/R/library/grDevices/libs/grDevices.so") ...
'verbose' and 'quietly' are both true; being verbose then ..
now dyn.load("/usr/lib64/R/library/graphics/libs/graphics.so") ...
'verbose' and 'quietly' are both true; being verbose then ..
now dyn.load("/usr/lib64/R/library/stats/libs/stats.so") ...
 ending setup_Rmainloop(): R_Interactive = 0 {main.c}
 >R_ReplConsole(): before "for(;;)" {main.c}
> shiny::runApp('/opt/easy16S/easy16S', host = '0.0.0.0', port = 3838)
now dyn.load("/usr/lib64/R/library/digest/libs/digest.so") ...
now dyn.load("/usr/lib64/R/library/rlang/libs/rlang.so") ...
now dyn.load("/usr/lib64/R/library/htmltools/libs/htmltools.so") ...
now dyn.load("/usr/lib64/R/library/Rcpp/libs/Rcpp.so") ...
Garbage collection 2 = 1+0+1 (level 0) ... 
17.4 Mbytes of cons cells used (51%)
4.2 Mbytes of vectors used (6%)
Garbage collection 3 = 2+0+1 (level 0) ... 
20.8 Mbytes of cons cells used (60%)
5.1 Mbytes of vectors used (8%)
now dyn.load("/usr/lib64/R/library/later/libs/later.so") ...
now dyn.load("/usr/lib64/R/library/magrittr/libs/magrittr.so") ...
now dyn.load("/usr/lib64/R/library/promises/libs/promises.so") ...
now dyn.load("/usr/lib64/R/library/httpuv/libs/httpuv.so") ...
now dyn.load("/usr/lib64/R/library/mime/libs/mime.so") ...
now dyn.load("/usr/lib64/R/library/ellipsis/libs/ellipsis.so") ...
now dyn.load("/usr/lib64/R/library/fastmap/libs/fastmap.so") ...
Garbage collection 4 = 3+0+1 (level 0) ... 
21.4 Mbytes of cons cells used (62%)
5.3 Mbytes of vectors used (8%)
Loading required package: shiny
now dyn.load("/usr/lib64/R/library/tools/libs/tools.so") ...
Garbage collection 5 = 4+0+1 (level 0) ... 
23.8 Mbytes of cons cells used (69%)
5.8 Mbytes of vectors used (9%)
now dyn.load("/usr/lib64/R/library/cachem/libs/cachem.so") ...
Warning in read.dcf(con) :
  invalid input found on input connection '/opt/easy16S/easy16S/DESCRIPTION'
Warning in loadSupport(appDir, renv = sharedEnv, globalrenv = globalenv()) :
  Loading R/ subdirectory for Shiny application, but this directory appears to contain an R package. Sourcing files in R/ may cause unexpected behavior.

Listening on http://0.0.0.0:3838
Garbage collection 6 = 5+0+1 (level 0) ... 
30.6 Mbytes of cons cells used (89%)
8.8 Mbytes of vectors used (14%)

@mandiayba
Copy link
Member Author

Le cas de MALDIquantTypeR

[root@2f1e3482e36c MALDIquantTypeR]# R --verbose -e "shiny::runApp('/opt/MALDIquantTypeR/MALDIquantTypeR', host = '0.0.0.0', port = 3838)"
'verbose' and 'quietly' are both true; being verbose then ..
now dyn.load("/usr/lib64/R/library/methods/libs/methods.so") ...

R version 3.6.0 (2019-04-26) -- "Planting of a Tree"
Copyright (C) 2019 The R Foundation for Statistical Computing
Platform: x86_64-redhat-linux-gnu (64-bit)

R is free software and comes with ABSOLUTELY NO WARRANTY.
You are welcome to redistribute it under certain conditions.
Type 'license()' or 'licence()' for distribution details.

R is a collaborative project with many contributors.
Type 'contributors()' for more information and
'citation()' on how to cite R or R packages in publications.

Type 'demo()' for some demos, 'help()' for on-line help, or
'help.start()' for an HTML browser interface to help.
Type 'q()' to quit R.

'verbose' and 'quietly' are both true; being verbose then ..
'verbose' and 'quietly' are both true; being verbose then ..
now dyn.load("/usr/lib64/R/library/utils/libs/utils.so") ...
Garbage collection 1 = 0+0+1 (level 2) ... 
12.1 Mbytes of cons cells used (35%)
3.1 Mbytes of vectors used (5%)
'verbose' and 'quietly' are both true; being verbose then ..
now dyn.load("/usr/lib64/R/library/grDevices/libs/grDevices.so") ...
'verbose' and 'quietly' are both true; being verbose then ..
now dyn.load("/usr/lib64/R/library/graphics/libs/graphics.so") ...
'verbose' and 'quietly' are both true; being verbose then ..
now dyn.load("/usr/lib64/R/library/stats/libs/stats.so") ...
 ending setup_Rmainloop(): R_Interactive = 0 {main.c}
 >R_ReplConsole(): before "for(;;)" {main.c}
> shiny::runApp('/opt/MALDIquantTypeR/MALDIquantTypeR', host = '0.0.0.0', port = 3838)
now dyn.load("/usr/lib64/R/library/digest/libs/digest.so") ...
now dyn.load("/usr/lib64/R/library/rlang/libs/rlang.so") ...
now dyn.load("/usr/lib64/R/library/htmltools/libs/htmltools.so") ...
now dyn.load("/usr/lib64/R/library/Rcpp/libs/Rcpp.so") ...
Garbage collection 2 = 1+0+1 (level 0) ... 
17.4 Mbytes of cons cells used (51%)
4.2 Mbytes of vectors used (6%)
Garbage collection 3 = 2+0+1 (level 0) ... 
20.7 Mbytes of cons cells used (60%)
5.1 Mbytes of vectors used (8%)
now dyn.load("/usr/lib64/R/library/later/libs/later.so") ...
now dyn.load("/usr/lib64/R/library/magrittr/libs/magrittr.so") ...
now dyn.load("/usr/lib64/R/library/promises/libs/promises.so") ...
now dyn.load("/usr/lib64/R/library/httpuv/libs/httpuv.so") ...
now dyn.load("/usr/lib64/R/library/mime/libs/mime.so") ...
now dyn.load("/usr/lib64/R/library/fastmap/libs/fastmap.so") ...
Garbage collection 4 = 3+0+1 (level 0) ... 
21.4 Mbytes of cons cells used (62%)
5.3 Mbytes of vectors used (8%)
now dyn.load("/usr/lib64/R/library/tools/libs/tools.so") ...
Loading required package: shiny

Listening on http://0.0.0.0:3838
Garbage collection 5 = 4+0+1 (level 0) ... 
27.6 Mbytes of cons cells used (80%)
7.0 Mbytes of vectors used (11%)
Garbage collection 6 = 4+1+1 (level 1) ... 
31.1 Mbytes of cons cells used (90%)
7.7 Mbytes of vectors used (12%)

@mandiayba
Copy link
Member Author

mandiayba commented Apr 2, 2021

La lenteur est causée par les instructions dans le global.R ==> il faut trouver un moyen de charger les données après l’affichage de la page d’accueil pour que l’application fonctionne sur le server shiny de Migale

test sur un hello world !!!

[root@3eb21322f482 ontoRelationVisualizer]# cat ui.R        
############################# LIBRARIES ################################


library(shiny)
library(shinyjs)
library(shinyWidgets)
library(tidyverse)


################################# UI ###################################

shinyUI(bootstrapPage(
     theme = "styles.css",
     "Hello, world!!!"
)

[root@3eb21322f482 ontoRelationVisualizer]# rm -rf global.R 

[root@3eb21322f482 ontoRelationVisualizer]# ls
INRAE-onto-visualization.Rproj  LICENSE  README.md  build.sh  conf.ini  data  docker  help-ui.R  run.R  run.sh  scripts  server.R  ui.R  www



[root@3eb21322f482 ontoRelationVisualizer]# R --verbose -e "shiny::runApp('/opt/ontoRelationVisualizer', host = '0.0.0.0', port = 3838)"
'verbose' and 'quietly' are both true; being verbose then ..
now dyn.load("/usr/lib64/R/library/methods/libs/methods.so") ...

R version 3.6.0 (2019-04-26) -- "Planting of a Tree"
Copyright (C) 2019 The R Foundation for Statistical Computing
Platform: x86_64-redhat-linux-gnu (64-bit)

R is free software and comes with ABSOLUTELY NO WARRANTY.
You are welcome to redistribute it under certain conditions.
Type 'license()' or 'licence()' for distribution details.

R is a collaborative project with many contributors.
Type 'contributors()' for more information and
'citation()' on how to cite R or R packages in publications.

Type 'demo()' for some demos, 'help()' for on-line help, or
'help.start()' for an HTML browser interface to help.
Type 'q()' to quit R.

'verbose' and 'quietly' are both true; being verbose then ..
'verbose' and 'quietly' are both true; being verbose then ..
now dyn.load("/usr/lib64/R/library/utils/libs/utils.so") ...
Garbage collection 1 = 0+0+1 (level 2) ... 
12.1 Mbytes of cons cells used (35%)
3.1 Mbytes of vectors used (5%)
'verbose' and 'quietly' are both true; being verbose then ..
now dyn.load("/usr/lib64/R/library/grDevices/libs/grDevices.so") ...
'verbose' and 'quietly' are both true; being verbose then ..
now dyn.load("/usr/lib64/R/library/graphics/libs/graphics.so") ...
'verbose' and 'quietly' are both true; being verbose then ..
now dyn.load("/usr/lib64/R/library/stats/libs/stats.so") ...
 ending setup_Rmainloop(): R_Interactive = 0 {main.c}
 >R_ReplConsole(): before "for(;;)" {main.c}
> shiny::runApp('/opt/ontoRelationVisualizer', host = '0.0.0.0', port = 3838)
now dyn.load("/usr/lib64/R/library/digest/libs/digest.so") ...
now dyn.load("/usr/lib64/R/library/rlang/libs/rlang.so") ...
now dyn.load("/usr/lib64/R/library/htmltools/libs/htmltools.so") ...
now dyn.load("/usr/lib64/R/library/Rcpp/libs/Rcpp.so") ...
Garbage collection 2 = 1+0+1 (level 0) ... 
17.4 Mbytes of cons cells used (51%)
4.2 Mbytes of vectors used (6%)
Garbage collection 3 = 2+0+1 (level 0) ... 
20.8 Mbytes of cons cells used (60%)
5.1 Mbytes of vectors used (8%)
now dyn.load("/usr/lib64/R/library/later/libs/later.so") ...
now dyn.load("/usr/lib64/R/library/magrittr/libs/magrittr.so") ...
now dyn.load("/usr/lib64/R/library/promises/libs/promises.so") ...
now dyn.load("/usr/lib64/R/library/httpuv/libs/httpuv.so") ...
now dyn.load("/usr/lib64/R/library/mime/libs/mime.so") ...
now dyn.load("/usr/lib64/R/library/ellipsis/libs/ellipsis.so") ...
now dyn.load("/usr/lib64/R/library/fastmap/libs/fastmap.so") ...
Garbage collection 4 = 3+0+1 (level 0) ... 
21.4 Mbytes of cons cells used (62%)
5.3 Mbytes of vectors used (8%)
Loading required package: shiny
now dyn.load("/usr/lib64/R/library/tools/libs/tools.so") ...
Garbage collection 5 = 4+0+1 (level 0) ... 
23.8 Mbytes of cons cells used (69%)
5.8 Mbytes of vectors used (9%)
now dyn.load("/usr/lib64/R/library/cachem/libs/cachem.so") ...

Listening on http://0.0.0.0:3838
Garbage collection 6 = 5+0+1 (level 0) ... 
30.6 Mbytes of cons cells used (89%)
8.8 Mbytes of vectors used (14%)

@letailli
Copy link
Contributor

letailli commented Apr 3, 2021

Les données des ontologies chargées au début de global.R font environ 72 Mo. Et ce, même après les transformations / réductions que nous avions réalisées sur les fichiers bruts (qui étaient autour de 600 Mo).

  habitat = get_ontology("../../data/BioNLP-OST+EnovFood-Habitat.obo"),
  phenotype = get_ontology("../../data/BioNLP-OST+EnovFood-Phenotype.obo"),
  use = get_ontology("../../data/Use_V2.obo"),
  taxon = get_ontology("../../data/microorganisms.obo")

On peut regarder à quel moment il est préférable de les charger mais il faut éviter que cela se fasse au moment où l'utilisateur en a besoin car cela provoquerait de grosses lenteurs (ce que nous avions lors des 1e tests).
Etes-vous toujours sur une architecture qui vise à arrêter/redémarrer régulièrement le service ? Je crains que cela soit problématique si on veut éviter de répéter ces chargements longs.

Je suis dispo pour en discuter si besoin.

@mandiayba
Copy link
Member Author

Merci @letailli pour la précision.

Nous voulons installer sur le ShinyProxy de Migale car cela permet de ne pas avoir de limitation sur le nombre d'utilisateurs travaillant en même temps (chaque utilisateur aura son conteneur indépendant) et cela permet de regrouper les applis shiny au même endroit mais effectivement si on n'a pas de solution pour le temps de chargement on peut essayer de déployer autrement.

Je ne connais pas bien R et shiny mais j'ai essayé de regarder le coté traitement asynchrone avec future et promises. Apparemment c'est possible d'exécuter le chargement en background et laisser une interface d'accueil s'afficher et éviter ainsi l'erreur 502. Il semble également possible de changer une fois les données et les rendre accessibles à toutes les sessions (ce que vous faites déja avec global.R). A voir si c'est simple à mettre en place...

En attendant je vais déployer sur une VM en dehors du shinyproxy. La lenteur au démarrage restera mais au moins on aura l'appli qui tourne.

@letailli
Copy link
Contributor

letailli commented Apr 3, 2021

Merci @mandiayba

Je pense qu'il faut éviter de charger ces données pour chaque utilisateur. Cela me semble à la fois lourd pour le serveur et oblige à un temps de chargement pour chaque utilisateur.

As-tu plus d'info sur la possibilité de changer une fois les données et les rendre accessibles à toutes les sessions dans l'archi Migale ?
On se prévoir un point la semaine prochaine pour en parler.

@mandiayba
Copy link
Member Author

mandiayba commented Apr 3, 2021 via email

@mandiayba
Copy link
Member Author

mandiayba commented Apr 3, 2021 via email

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants