Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

something went wrong #9

Open
MathieuGrosso opened this issue Jun 30, 2023 · 5 comments
Open

something went wrong #9

MathieuGrosso opened this issue Jun 30, 2023 · 5 comments

Comments

@MathieuGrosso
Copy link

MathieuGrosso commented Jun 30, 2023 •
edited
Loading

I have followed all the tutorial, was able to log on the mlflow server but then i get this message "something went wrong" on the uI interface.
Do you know why ?
In logs i have an error of engine, the password is seen as the port. I have followed the same path as you provide for engine but it does not seem to work.
@mkarpicz
Copy link
Contributor

mkarpicz commented Jul 2, 2023 via email

@basia-c
Copy link

basia-c commented Jul 17, 2023

Hello!
Looks like your MlFlow is not able to connect to the database, there may be a few different reasons:

  1. API is not activated, the following APIs need to be activated (it may take up to 24h):
  1. Database is down (it needs to be up and running)
  2. Service account used by CloudRun does not have the right privileges - it needs to have access to the secrets created for this project (roles/secretmanager.secretAccessor) and to the database (roles/cloudsql.client)
  3. If it still does not work at this point try updating requirements.txt file as follows :

google-cloud-storage==2.9.0 google-cloud-secret-manager==2.16.1 mlflow==2.4.1 click==8.1.3 pg8000==1.29.3 wsgi-basic-auth==1.1.0 gunicorn==20.1.0 psycopg2-binary==2.9.6

@BioGeek
Copy link

BioGeek commented Sep 28, 2023 •
edited
Loading

I had the same issue.

The tutorial writes that the mlflow_database_url Cloud SQL connection string must be in SQLAlchemy-format and gives as example:

postgresql+pg8000://<dbuser>:<dbpass>@/<dbname>?unix_sock=/cloudsql/tc-global-dlabs:europe-west3:mlfow/.s.PGSQL.5432

After much head scratching I found this stackoverflow post that shows you need to use psycopg2 instead:

postgresql+psycopg2://<username>:<password>@<dbname>?host=/cloudsql/<my-project>:<us-central1>:<dbinstance>

Some other places where I deviated from the tutorial:

  • Container Registry is deprecated, so I used Artifactory registry
  • The get_secret.py script is not needed. You can expose secrets as environment variables in Cloud Run. So I modified the entry-point.sh script.

Also, it is unclear why I needed to download the client-cert.pem, client-key.pem and server-ca.pem files because the tutorial doesn't seem to use them anywhere.

@mkarpicz
Copy link
Contributor

mkarpicz commented Sep 29, 2023 via email

@mz-dlabs
Copy link
Contributor

mz-dlabs commented Nov 2, 2023

Hello @BioGeek,

I've updated repository documentation with changes related to Artifact Registry etc.

If MLflow cannot connect to the database, try one of the following:

  1. Make sure that in Cloud Run configuration, in Container(s), Volumes, Networking, Security accordion, Cloud SQL connections section, you added your database instance
  2. Try using Cloud SQL without SSL
  3. For the database URL, I use a string like the following - make sure yours is similar
postgresql+pg8000://username:password@/database_name?unix_sock=/cloudsql/project_name:database_region:database_name/.s.PGSQL.5432

Let us know if you have successfully launched the application

Regards
Maciej

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
5 participants
@BioGeek @mkarpicz @basia-c @MathieuGrosso @mz-dlabs