Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add a hard limit to the transaction size #2101

Open
balegas opened this issue Dec 5, 2024 · 1 comment
Open

Add a hard limit to the transaction size #2101

balegas opened this issue Dec 5, 2024 · 1 comment

Comments

@balegas
Copy link
Contributor

balegas commented Dec 5, 2024

Electric loads entire transactions from Postgres logical replication into memory before starting to write to shape logs. For very large transactions this might exhaust memory, crashing the server.

While we don't have a way to handle arbitrarily large transactions, we shall set a limit to the amount of memory a transaction can use and throw an erros.

Handling errors is not trivial, since Electric can't continue consuming the replication stream. Dropping all shapes and recreating the replication slot might be a way of getting out of that state.

Exhausting memory shouldn't happen very often and does not affect initial sync of shapes

@KyleAMathews
Copy link
Contributor

Nice — yeah if we can't handle it gracefully then we really have no alternative but to dropping the transaction and killing all shapes and starting over.

We could probably set the limit based on the amount of memory of the server? If someone has trouble with big transactions then the fix is just getting a bigger box.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants