Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[feat] Sort SQL result tuples for fair comparison during evaluation #159 #162

Conversation

lucaordronneau
Copy link

Description

This feature was created to provide a better comparison when evaluating the Bird dataset. With the use of generative models for SQL generation, the SQL clauses in the SELECT statement are not always in the same order (see issue #159)

Example

Ground Truth: SELECT price, user FROM <table>
Prediction: SELECT user, price FROM <table>

In the evaluation of the Bird dataset, this case is not taken into account. This improvement integrates a sorting mechanism for the tuples resulting from the SQL query execution to ensure consistent evaluation.

Changes:

  • Added sorting of tuples in the SQL query results for accurate comparison.
  • Updated evaluation logic to handle the sorted clauses.

@CLAassistant
Copy link

CLAassistant commented Aug 14, 2024

CLA assistant check
All committers have signed the CLA.

@lucaordronneau lucaordronneau force-pushed the feat/fair-evaluation-comparison branch from aabdf16 to 1743ec4 Compare August 19, 2024 13:31
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants