Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ContentRepository::setUp is too slow #4762

Closed
mhsdesign opened this issue Nov 17, 2023 · 1 comment
Closed

ContentRepository::setUp is too slow #4762

mhsdesign opened this issue Nov 17, 2023 · 1 comment

Comments

@mhsdesign
Copy link
Member

mhsdesign commented Nov 17, 2023

Setting up a cr will take some time. This is due to our implementation of creating the database tables of the projections:

$schemaDiff = (new Comparator())->compare($schemaManager->createSchema(), $schema);
// toSaveSql to not drop other tables found in the database
foreach ($schemaDiff->toSaveSql($connection->getDatabasePlatform()) as $statement) {
    $connection->executeStatement($statement);
}

In each projection setup we will call $schemaManager->createSchema() twice. Once for the main tables and once for the checkpoint lock tables. This is super expensive as it has to go through all tables and their columns and indexes uncached – even though we are only interested in one or two tables per projection.

Additionally this pattern is deprecated with doctrine/dbal 3+

@bwaidelich fiddled a bit with the whole schema diffing, and found that changing

$schema = $schemaManager->createSchema();

to

$schema = new Schema([$schemaManager->listTableDetails($tableName)], [], $schemaManager->createSchemaConfig());

reduced the runtime of a single instance from 0.08 s to 0.0012 s.
This seems like a very quick win.. And that code might even safely be runtime-cached because it only affects the given table(s)


Currently this affects also our testing execution time. By hacking around and making assumptions we can cleverly cache it but it would be great to not have to go through the trouble of #4750


Extracted from earlier discussions:

Another alternative to speed up the setup would naively be to somehow cache the createSchema calls and dont optimize the $cr->setUp(). That would have a high performance gain, but it comes with a downside as @bwaidelich said:

There is most certainly a lot of potential for speed improvements, but we should be careful not to let them degrade the system architecture.
E.g. I would suggest not to cache the createSchema call even though it might be tempting because that can lead to caching issues (you might have heard of them).

Related: #4388

@mhsdesign
Copy link
Member Author

Wow @bwaidelich thanks a lot!

#4846 really improves the performance:

before before with hack after
1.460s 0.370s 0.360s

with hack i meant caching the AbstractSchemaManager::createSchema

The speedup of the behavioural tests #4750 is still useful as it speeds up the tests by a factor of two, but its not as essential as before :D

@github-project-automation github-project-automation bot moved this from Todo to Done ✅ in Neos 9.0 Release Board Jan 22, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
Archived in project
Development

No branches or pull requests

1 participant