New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Assessing the risk of duplicated entries for BEST_EFFORT reliabilityLevel #252

Open

jcblancomartinez opened this issue Jan 4, 2024 · 1 comment

jcblancomartinez commented Jan 4, 2024

Hi,

From here:

Implements the BEST_EFFORT write strategy for Single Instance. All executors insert into a user specified table directly. Write to table is not transactional and may results in duplicates in executor restart scenarios.

I'm not sure I understand when we could end up with duplicates when using BEST_EFFORT reliabilityLevel.

For each Spark's dataframe partition we call savePartition and savePartition only commits the transaction after SQLServerBulkCopy.writeToServer has succeded.

Are we worried that executor restarts after committing the transaction?
In which cases could that happen? When an SQLException gets thrown when committing the transation? Are we thinking of a connection reset that causes Spark to handle an SQLException even though the transaction has been committed in Sql Server? Are there other situations where this could happen?

Thanks.

Author

jcblancomartinez commented Jan 4, 2024 •

edited

Loading

@shivsood @luxu1-ms could you please help here?

Thanks.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment