[Draft] Native image #174

agilob · 2020-08-27T17:51:46Z

This is a draft and WIP as the native image is quite problematic with so many dependencies. I found ehcache and logstash being hard to use without a few native-image generator flags.

To try it out:

./mvnw -Pprod package -DskipTests=true && bash docker-build.sh
./gradlew  -Pprod clean assemble && bash docker-build.sh

Closes #115

metrics issue

``` 16:35:30.183 [main] ERROR io.micronaut.runtime.Micronaut - Error starting Micronaut server: Bean definition [javax.sql.DataSource] could not be loaded: Error instantiating bean of type [javax.sql.DataSource]: Class must be instance of com.codahale.metrics.MetricRegistry or io.micrometer.core.instrument.MeterRegistry io.micronaut.context.exceptions.BeanInstantiationException: Bean definition [javax.sql.DataSource] could not be loaded: Error instantiating bean of type [javax.sql.DataSource]: Class must be instance of com.codahale.metrics.MetricRegistry or io.micrometer.core.instrument.MeterRegistry at io.micronaut.context.DefaultBeanContext.initializeContext(DefaultBeanContext.java:1540) at io.micronaut.context.DefaultApplicationContext.initializeContext(DefaultApplicationContext.java:220) at io.micronaut.context.DefaultBeanContext.readAllBeanDefinitionClasses(DefaultBeanContext.java:2768) at io.micronaut.context.DefaultBeanContext.start(DefaultBeanContext.java:228) at io.micronaut.context.DefaultApplicationContext.start(DefaultApplicationContext.java:166) at io.micronaut.runtime.Micronaut.start(Micronaut.java:64) at com.mycompany.myapp.Mhip2App.main(Mhip2App.java:59) Caused by: io.micronaut.context.exceptions.BeanInstantiationException: Error instantiating bean of type [javax.sql.DataSource]: Class must be instance of com.codahale.metrics.MetricRegistry or io.micrometer.core.instrument.MeterRegistry at io.micronaut.context.DefaultBeanContext.doCreateBean(DefaultBeanContext.java:1918) at io.micronaut.context.DefaultBeanContext.createAndRegisterSingletonInternal(DefaultBeanContext.java:2635) at io.micronaut.context.DefaultBeanContext.createAndRegisterSingleton(DefaultBeanContext.java:2621) at io.micronaut.context.DefaultBeanContext.loadContextScopeBean(DefaultBeanContext.java:2163) at io.micronaut.context.DefaultBeanContext.initializeContext(DefaultBeanContext.java:1538) ... 6 common frames omitted Caused by: java.lang.IllegalArgumentException: Class must be instance of com.codahale.metrics.MetricRegistry or io.micrometer.core.instrument.MeterRegistry at com.zaxxer.hikari.HikariConfig.setMetricRegistry(HikariConfig.java:667) at com.zaxxer.hikari.HikariDataSource.setMetricRegistry(HikariDataSource.java:237) at io.micronaut.configuration.jdbc.hikari.DatasourceFactory.addMeterRegistry(DatasourceFactory.java:79) at io.micronaut.configuration.jdbc.hikari.DatasourceFactory.dataSource(DatasourceFactory.java:67) at io.micronaut.configuration.jdbc.hikari.$DatasourceFactory$DataSource0Definition.build(Unknown Source) at io.micronaut.context.BeanDefinitionDelegate.build(BeanDefinitionDelegate.java:143) at io.micronaut.context.DefaultBeanContext.doCreateBean(DefaultBeanContext.java:1889) ... 10 common frames omitted ```

metrics aren't supported (upstream issue)
- Metrics to not allow app to start, micrometer needs to be removed from dependencies before building .jar
setting env variables (from docker-compose)
attempt to remove --report-unsupported-elements-at-runtime
endpoints are protected fine
Resources aren't included / do not load

atomfrede · 2020-08-27T18:16:54Z

Should we really make it either native or not? I am thinking of micronaut launch which creates an additional configuration to package it as a native image either via docker or Graal directly. Having this would be good in order not to move too far away from micronaut launch/CLI.

agilob · 2020-08-27T18:33:39Z

@atomfrede Yes, I agree that would be best. That's in fact how I started working on mn-native image a couple of days ago, but... I run into the same issue as above, so I tried, maybe different approach could change results, but it's the same. So at the end I will revert that and add dockerfile with build-docker.sh like mn generates, but I need to fix that hibernate error first. Posting this for visibility and feedback like yours.

atomfrede · 2020-08-28T05:41:53Z

Sounds good. I just manage to create a default mn application without jpa for now with native image support but didn't had much time to have closer look.

agilob · 2020-08-28T20:55:31Z

@JasonTypesCodes could you please advice on the liquibase error? It seems to be coming from micronaut dependency

JasonTypesCodes · 2020-08-28T21:53:15Z

@agilob I'll ask around. I don't have experience building native images with GraalVM myself, but I think we can work through it :)

atomfrede · 2020-08-29T06:07:28Z

Are these related? https://liquibase.jira.com/browse/CORE-3470

agilob · 2020-08-29T07:34:12Z

@atomfrede might be, but liquibase seems to be working with quarkus native-image

JasonTypesCodes · 2020-08-30T23:04:00Z

@agilob It seems like that class is getting loaded dynamically. This may be resolved by providing some reflection configurations.

There is some guidance on doing that in this guide:

https://guides.micronaut.io/micronaut-security-session/guide/index.html#graalreflectjson

agilob · 2020-09-01T20:05:44Z

This is a possible solution, haven't looked much into it yet, but I found one problem: reflect-config.json could be added directly in dependency jar, like hibernate and others do it, it would be more future proof than generated by us for every project. It also seems to reflect file must be regenerated (or combined with preexisting) for each created @introspected model.

ilopmar · 2020-09-03T09:43:33Z

@agilob Micronaut generates reflect-config,json on the fly and it will include all the classes annotated with @Instrospected. Another way to add classes to the GraalVM configuration is using @TypeHint. For example, the section in the guide that Jason linked is not necessary anymore because of this https://github.com/micronaut-projects/micronaut-views/blob/123d516bc4701ebe90703e84e5d3c982c7c76bfa/views-velocity/src/main/java/io/micronaut/views/velocity/VelocityGraalSupport.java.

generators/server/templates/Dockerfile.ejs

generators/server/templates/native-image.properties.ejs

generators/server/templates/pom.xml.ejs

agilob · 2020-09-03T17:31:12Z

I've made some progress today:

Updated svm and graal to 20.2.0 as per review.
Resources are loaded to native-image compilation process by extensions
resources and reflection for native-image configs are precompiled and copied to META-INF-<=appNamesomething=> directory to be picked up by native-image

I've only tried so far with postgresql without cache, I know ehcache didnt work due to graal-reflection issues.

All resources from native-image load in sub-millisecond

Still can't login due to liquibase issue, that I'll be working on next.
Metrics dependencies have to be removed before you attempt compiling native image, there seems to be micronaut-micrometer-hikari conflict after compilation(?)

You can try it with:
MICRONAUT_ENVIRONMENTS=prod ./gradlew assemble && bash ./docker-build.sh

agilob · 2020-09-03T17:36:02Z

Reflection config is currently provided for example purposes so it contains psql and jwt dependencies. Depending on configuration, or number of options, might have to leave generation of this file to end users https://guides.micronaut.io/micronaut-security-session/guide/index.html#graalreflectjson

I would be a lot of maintenance less for us and safer for users.

ilopmar · 2020-09-04T08:45:05Z

@agilob Please keep in mind that I don't know how JHispter works so maybe what I'm going to say doesn't make sense:

In this commit 15ae0af you have added:

resources.json.ejs: This file declares some static resources (html, js, png,...). I think this file is not really necessary because during compilation Micronaut will generate that file with everything included in the src/main/resources directory. Don't know how the i18n.messages bundle works but probably that's the only thing you need to declare.
refect.json.ejs: Pretty much everything declared in that file is not necessary, specially all the entries for io.micronaut.

agilob · 2020-09-04T09:12:33Z

resources.json.ejs: This file declares some static resources (html, js, png,...). I think this file is not really necessary because during compilation Micronaut will generate that file with everything included in the src/main/resources directory. Don't know how the i18n.messages bundle works but probably that's the only thing you need to declare.

webpack frontend is generated from src/main/webapp and generated to build/www:

task webpackBuildDev(type: NpmTask, dependsOn: 'npmInstall') {
    inputs.dir("src/main/webapp/")
    inputs.files(fileTree('src/main/webapp/'))
    outputs.dir("build/www/")
    outputs.file("build/www/app/main.bundle.js")
    args = ["run", "webpack:build"]
}

This part is absolutely necessary for frontend to be linked in native-image file, but not for liquibase (which lives in src/main/resources)

refect.json.ejs: Pretty much everything declared in that file is not necessary, specially all the entries for io.micronaut.

I'll be likely reducing this file, but so far it is required for liquibase to compile with native-image, but still doesn't fix starup problem.

agilob · 2020-09-15T21:38:06Z

Fixed liquibase startup issue, now postgres doesn't connect :D one step at a time

Caused by: io.micronaut.context.exceptions.BeanInstantiationException: Error instantiating bean of type [javax.sql.DataSource]: Failed to initialize pool: The connection attempt failed.
        at io.micronaut.context.DefaultBeanContext.doCreateBean(DefaultBeanContext.java:1918)
        at io.micronaut.context.DefaultBeanContext.createAndRegisterSingletonInternal(DefaultBeanContext.java:2635)
        at io.micronaut.context.DefaultBeanContext.createAndRegisterSingleton(DefaultBeanContext.java:2621)
        at io.micronaut.context.DefaultBeanContext.loadContextScopeBean(DefaultBeanContext.java:2163)
        at io.micronaut.context.DefaultBeanContext.initializeContext(DefaultBeanContext.java:1538)
        ... 6 common frames omitted
Caused by: com.zaxxer.hikari.pool.HikariPool$PoolInitializationException: Failed to initialize pool: The connection attempt failed.
        at com.zaxxer.hikari.pool.HikariPool.throwPoolInitializationException(HikariPool.java:595)
        at com.zaxxer.hikari.pool.HikariPool.checkFailFast(HikariPool.java:581)
        at com.zaxxer.hikari.pool.HikariPool.<init>(HikariPool.java:115)
        at com.zaxxer.hikari.HikariDataSource.<init>(HikariDataSource.java:81)
        at io.micronaut.configuration.jdbc.hikari.HikariUrlDataSource.<init>(HikariUrlDataSource.java:35)
        at io.micronaut.configuration.jdbc.hikari.DatasourceFactory.dataSource(DatasourceFactory.java:66)
        at io.micronaut.configuration.jdbc.hikari.$DatasourceFactory$DataSource0Definition.build(Unknown Source)
        at io.micronaut.context.BeanDefinitionDelegate.build(BeanDefinitionDelegate.java:143)
        at io.micronaut.context.DefaultBeanContext.doCreateBean(DefaultBeanContext.java:1889)
        ... 10 common frames omitted
Caused by: org.postgresql.util.PSQLException: The connection attempt failed.
        at org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:297)
        at org.postgresql.core.ConnectionFactory.openConnection(ConnectionFactory.java:49)
        at org.postgresql.jdbc.PgConnection.<init>(PgConnection.java:217)
        at org.postgresql.Driver.makeConnection(Driver.java:458)
        at org.postgresql.Driver.connect(Driver.java:260)
        at com.zaxxer.hikari.util.DriverDataSource.getConnection(DriverDataSource.java:138)
        at com.zaxxer.hikari.pool.PoolBase.newConnection(PoolBase.java:358)
        at com.zaxxer.hikari.pool.PoolBase.newPoolEntry(PoolBase.java:206)
        at com.zaxxer.hikari.pool.HikariPool.createPoolEntry(HikariPool.java:477)
        at com.zaxxer.hikari.pool.HikariPool.checkFailFast(HikariPool.java:560)
        ... 17 common frames omitted
Caused by: java.net.SocketException: Connection reset
        at java.net.SocketInputStream.read(SocketInputStream.java:186)
        at java.net.SocketInputStream.read(SocketInputStream.java:140)
        at org.postgresql.core.VisibleBufferedInputStream.readMore(VisibleBufferedInputStream.java:161)
        at org.postgresql.core.VisibleBufferedInputStream.ensureBytes(VisibleBufferedInputStream.java:128)
        at org.postgresql.core.VisibleBufferedInputStream.ensureBytes(VisibleBufferedInputStream.java:113)
        at org.postgresql.core.VisibleBufferedInputStream.read(VisibleBufferedInputStream.java:73)
        at org.postgresql.core.PGStream.receiveChar(PGStream.java:370)
        at org.postgresql.core.v3.ConnectionFactoryImpl.enableSSL(ConnectionFactoryImpl.java:416)
        at org.postgresql.core.v3.ConnectionFactoryImpl.tryConnect(ConnectionFactoryImpl.java:140)
        at org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:197)
        ... 26 common frames omitted

atomfrede · 2020-10-21T07:17:35Z

@agilob Whats the current status of this ticket? Can I help testing something? We could mark this feature as experimental in case there are cases/configurations it doesn't work yet. Would be cool to have it in the 1.0 release (e.g. the micronaut gradle plugin).

agilob · 2020-10-21T07:22:59Z

I look at this every weekend. Last time I migrated config from .properties file to gradle to use the micronaut gradle plugin. It currently is unable to create a hibernate session on start. I have example projects where Session works, but was unable to successfully migrate that config to mhipster.

JasonTypesCodes · 2020-11-24T22:20:42Z

@agilob If you are still having issues with micronaut-projects/micronaut-core#4155 , you may have better luck with Micronaut 2.2.0

…ut into native-image

ilopmar · 2020-12-04T09:12:01Z

I forgot to mention that we included GraalVM support for Liquibase in micronaut-liquibase 3.0.0. It also includes the upgrade to Liquibase 4.2.0.
This 3.0.0 version is included by default in Micronaut 2.2.0.

You may need to adjust the GraalVM configuration because now everything it's included in Micronaut.

…ut into native-image

atomfrede · 2021-03-09T07:34:59Z

@agilob Can you try to fix the conflicts? Maybe we can integrate it into the 1.0.0 release as preview or just document known issues/combinations not working.

agilob · 2021-03-09T08:03:41Z

I'll check this over the weekend @atomfrede

…ut into native-image

agilob · 2021-03-13T09:25:05Z

Still can't get micronaut-liquibase to run initial migration... no idea about it. Any tips @ilopmar? hibernate connects to PSQL database, datasource bean is created, liquibase doesn't start, no columns are created

ilopmar · 2021-03-16T08:59:20Z

@agilob This is the GraalVM test application we use for Micronaut, Liquibase, GraalVM and Postgres: https://github.com/micronaut-graal-tests/micronaut-liquibase-graal/tree/2.4.x_postgres
Maybe you are missing something in the configuration?

agilob · 2021-03-16T15:38:08Z

Brilliant, found out that adding:

        '-H:+StaticExecutableWithDynamicLibC',

causes startup exception:

15:38:13.803 [io-executor-thread-1] ERROR i.m.l.LiquibaseMigrationRunner - Migration failed! Liquibase encountered an exception.
liquibase.exception.ChangeLogParseException: liquibase.exception.SetupException: liquibase.exception.UnexpectedLiquibaseException: java.lang.NoSuchMethodException: liquibase.change.core.LoadDataColumnConfig.<init>()
        at liquibase.parser.core.xml.AbstractChangeLogParser.parse(AbstractChangeLogParser.java:25)
        at liquibase.Liquibase.getDatabaseChangeLog(Liquibase.java:337)
        at liquibase.Liquibase.lambda$update$1(Liquibase.java:229)
        at liquibase.Scope.lambda$child$0(Scope.java:160)
        at liquibase.Scope.child(Scope.java:169)
        at liquibase.Scope.child(Scope.java:159)
        at liquibase.Scope.child(Scope.java:138)
        at liquibase.Liquibase.runInScope(Liquibase.java:2277)
        at liquibase.Liquibase.update(Liquibase.java:215)
        at liquibase.Liquibase.update(Liquibase.java:201)
        at io.micronaut.liquibase.LiquibaseMigrationRunner.performUpdate(LiquibaseMigrationRunner.java:190)
        at io.micronaut.liquibase.LiquibaseMigrationRunner.migrate(LiquibaseMigrationRunner.java:147)
        at io.micronaut.liquibase.LiquibaseMigrationRunner.migrateAsync(LiquibaseMigrationRunner.java:123)
        at io.micronaut.liquibase.$LiquibaseMigrationRunnerDefinition$Intercepted.$$access0(Unknown Source)
        at io.micronaut.liquibase.$LiquibaseMigrationRunnerDefinition$Intercepted$$proxy0.invokeInternal(Unknown Source)
        at io.micronaut.context.AbstractExecutableMethod.invoke(AbstractExecutableMethod.java:146)
        at io.micronaut.aop.chain.MethodInterceptorChain.proceed(MethodInterceptorChain.java:87)
        at io.micronaut.scheduling.async.AsyncInterceptor.lambda$intercept$3(AsyncInterceptor.java:103)
        at io.micrometer.core.instrument.composite.CompositeTimer.record(CompositeTimer.java:79)
        at io.micrometer.core.instrument.Timer.lambda$wrap$0(Timer.java:157)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
        at java.util.concurrent.FutureTask.run(FutureTask.java:264)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.lang.Thread.run(Thread.java:834)
        at com.oracle.svm.core.thread.JavaThreads.threadStartRoutine(JavaThreads.java:519)
        at com.oracle.svm.core.posix.thread.PosixJavaThreads.pthreadStartRoutine(PosixJavaThreads.java:192)
Caused by: liquibase.exception.SetupException: liquibase.exception.UnexpectedLiquibaseException: java.lang.NoSuchMethodException: liquibase.change.core.LoadDataColumnConfig.<init>()
        at liquibase.changelog.DatabaseChangeLog.handleChildNode(DatabaseChangeLog.java:373)
        at liquibase.changelog.DatabaseChangeLog.load(DatabaseChangeLog.java:320)
        at liquibase.parser.core.xml.AbstractChangeLogParser.parse(AbstractChangeLogParser.java:23)
        ... 26 common frames omitted

I've got something to work with now, thanks @ilopmar

ilopmar · 2021-03-16T15:41:34Z

That flag is to generated "mostly" static images by default. We've changed our plugins to not doing that by default because that's the same default as in GraalVM, I mean, generate dynamic native images by default.
The flag is because we wanted to test the generation of those mostly native images and test them in our CI. The application should work without that flag.

mraible · 2021-08-04T19:24:53Z

Any updates on this?

atomfrede · 2021-09-06T09:01:48Z

I will try to have a look at this, but time is rare at the moment, would at least like to get gradle enterprise in the main generator, which is pending way too long.

agilob · 2021-09-06T10:17:18Z

Ignore changes in this PR, most likely this needs to be started from new, as there are too many merge conflicts with this branch.

mraible · 2023-08-17T03:33:44Z

Closing based on comments from @agilob.

agilob added 3 commits August 27, 2020 18:31

Add prompt to use native image

3e35e44

Add maven config for native image

fc511fa

Fix package.classname generation

4fa8784

agilob added 5 commits August 28, 2020 10:32

Generate docker native-image compilation MN-way

3fd9bd5

Add micronaut native-image building files

bd7da41

Re-add jib-maven-plugin

4880a50

Remove redundant native-image properties

52a90e3

Move gradle micronaut-registry dependency next to micrometer

184b753

ilopmar reviewed Sep 3, 2020

View reviewed changes

generators/server/templates/Dockerfile.ejs Outdated Show resolved Hide resolved

ilopmar reviewed Sep 3, 2020

View reviewed changes

generators/server/templates/Dockerfile.ejs Outdated Show resolved Hide resolved

generators/server/templates/native-image.properties.ejs Outdated Show resolved Hide resolved

generators/server/templates/pom.xml.ejs Outdated Show resolved Hide resolved

agilob added 4 commits September 3, 2020 11:06

Bump graal vm to 20.2.0

38d26dc

Remove --no-server

0347e66

Add reflection and resource config

a9bd031

Add reflect and resources files

15ae0af

Remove micronaut reflection tags, fix liquibase startup issue

86d0b63

agilob added 2 commits October 17, 2020 16:23

Bump micronaut gradle plugin

a9c5ae5

Remove /home/app from native-image resource location

85d97ba

atomfrede changed the base branch from master to main October 20, 2020 07:43

agilob added 4 commits December 2, 2020 14:53

Merge branch 'main' of github.com:jhipster/generator-jhipster-microna…

306aee6

…ut into native-image

Bump oracle graalvm image to 20.3.0-java11

2e6bac0

Fix generation of docker.gradle.ejs for location of native image files

12dcd13

Enable --report-unsupported-elements-at-runtime

3a19057

agilob added 2 commits January 19, 2021 21:08

Add more native-image compilation flags, solves hikaru connection issue

0fdf128

Merge branch 'main' of github.com:jhipster/generator-jhipster-microna…

7baeebb

…ut into native-image

agilob added 4 commits March 13, 2021 08:14

Merge branch 'main' of github.com:jhipster/generator-jhipster-microna…

a956d70

…ut into native-image

Bump org.graalvm.nativeimage:svm to 21.0.0.2

6204d6e

Update location of graal-ce docker image

bbf3941

Remove liquibase reflect configuration

7686b9a

Bump micronaut liquibase to 3.0.2

2f59100

kevintanhongann mentioned this pull request Aug 10, 2022

Upgrade micronaut 2 to 3 #267

Merged

mraible closed this Aug 17, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Draft] Native image #174

[Draft] Native image #174

agilob commented Aug 27, 2020 •

edited

Loading

atomfrede commented Aug 27, 2020

agilob commented Aug 27, 2020

atomfrede commented Aug 28, 2020

agilob commented Aug 28, 2020

JasonTypesCodes commented Aug 28, 2020

atomfrede commented Aug 29, 2020

agilob commented Aug 29, 2020

JasonTypesCodes commented Aug 30, 2020 •

edited

Loading

agilob commented Sep 1, 2020

ilopmar commented Sep 3, 2020

agilob commented Sep 3, 2020

agilob commented Sep 3, 2020 •

edited

Loading

ilopmar commented Sep 4, 2020

agilob commented Sep 4, 2020 •

edited

Loading

agilob commented Sep 15, 2020

atomfrede commented Oct 21, 2020

agilob commented Oct 21, 2020

JasonTypesCodes commented Nov 24, 2020

ilopmar commented Dec 4, 2020

atomfrede commented Mar 9, 2021

agilob commented Mar 9, 2021

agilob commented Mar 13, 2021

ilopmar commented Mar 16, 2021

agilob commented Mar 16, 2021

ilopmar commented Mar 16, 2021

mraible commented Aug 4, 2021

atomfrede commented Sep 6, 2021

agilob commented Sep 6, 2021

mraible commented Aug 17, 2023

[Draft] Native image #174

[Draft] Native image #174

Conversation

agilob commented Aug 27, 2020 • edited Loading

atomfrede commented Aug 27, 2020

agilob commented Aug 27, 2020

atomfrede commented Aug 28, 2020

agilob commented Aug 28, 2020

JasonTypesCodes commented Aug 28, 2020

atomfrede commented Aug 29, 2020

agilob commented Aug 29, 2020

JasonTypesCodes commented Aug 30, 2020 • edited Loading

agilob commented Sep 1, 2020

ilopmar commented Sep 3, 2020

agilob commented Sep 3, 2020

agilob commented Sep 3, 2020 • edited Loading

ilopmar commented Sep 4, 2020

agilob commented Sep 4, 2020 • edited Loading

agilob commented Sep 15, 2020

atomfrede commented Oct 21, 2020

agilob commented Oct 21, 2020

JasonTypesCodes commented Nov 24, 2020

ilopmar commented Dec 4, 2020

atomfrede commented Mar 9, 2021

agilob commented Mar 9, 2021

agilob commented Mar 13, 2021

ilopmar commented Mar 16, 2021

agilob commented Mar 16, 2021

ilopmar commented Mar 16, 2021

mraible commented Aug 4, 2021

atomfrede commented Sep 6, 2021

agilob commented Sep 6, 2021

mraible commented Aug 17, 2023

agilob commented Aug 27, 2020 •

edited

Loading

JasonTypesCodes commented Aug 30, 2020 •

edited

Loading

agilob commented Sep 3, 2020 •

edited

Loading

agilob commented Sep 4, 2020 •

edited

Loading