Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add config helper (WIP) and missed constants in existing dataloaders #605

Merged
merged 30 commits into from
Jun 19, 2024
Merged
Changes from 1 commit
Commits
Show all changes
30 commits
Select commit Hold shift + click to select a range
b23bec1
Add config helper (WIP) and missed constants in existing dataloaders
holylovenia Apr 4, 2024
a3911c2
Clarify method names
holylovenia Apr 4, 2024
5858e56
Fix some error-triggering parts
holylovenia Apr 12, 2024
ad8bb9b
Modify package setup
holylovenia Apr 12, 2024
0ecc98d
Fix bug
holylovenia Apr 17, 2024
3f8c571
Add the latest version
holylovenia Apr 21, 2024
648bfcc
Merge branch 'master' of https://github.com/SEACrowd/seacrowd-datahub…
holylovenia Apr 30, 2024
a5541fa
Remove task for source-only dataset
holylovenia Apr 30, 2024
6468361
Change train to test split
holylovenia Apr 30, 2024
a667cc3
Include all languages in the data
holylovenia Apr 30, 2024
1e7026d
Change train to test
holylovenia Apr 30, 2024
1467309
Change train to test
holylovenia Apr 30, 2024
700c177
Remove numbering from options and answer
holylovenia Apr 30, 2024
7d0ab71
Add the newest version
holylovenia Apr 30, 2024
f8e1212
remove main entry of module for dataloaders (#662)
sabilmakbar May 2, 2024
9bd7bf6
Fix tgl --> fil for AYA dataset
holylovenia May 8, 2024
c1ec65d
Change train -> test and add eng as the MT lang pair
holylovenia May 10, 2024
ef89194
Change incorrect name
holylovenia May 11, 2024
970afbc
Change incorrect name
holylovenia May 11, 2024
636ebfa
Change subset id to '*_{lang}_eng_*' or '*_eng_{lang}_*'
holylovenia May 12, 2024
4902542
Merge branch 'master' of https://github.com/SEACrowd/seacrowd-datahub…
holylovenia May 12, 2024
b3ad89c
Fix paracotta_id's download issue
holylovenia May 12, 2024
83901a6
Merge branch 'master' of https://github.com/SEACrowd/seacrowd-datahub…
holylovenia May 13, 2024
09051b2
Normalize subset names and enable eng_{lang} pairings
holylovenia May 13, 2024
dba6e62
Merge branch 'master' of https://github.com/SEACrowd/seacrowd-datahub…
holylovenia Jun 19, 2024
0be9162
Fix load_* methods
holylovenia Jun 19, 2024
0e590b6
Fix available_* methods
holylovenia Jun 19, 2024
2e7509b
Change _SEACROWD_VERSION to reflect the date of last update
holylovenia Jun 19, 2024
4ac5ffa
Add SEACrowd benchmark config list
holylovenia Jun 19, 2024
9db6d22
Update seacrowd to 0.1.0
holylovenia Jun 19, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Clarify method names
holylovenia committed Apr 4, 2024
commit a3911c2744addd325d596522e8e2d693c367d79b
4 changes: 2 additions & 2 deletions seacrowd/config_helper.py
Original file line number Diff line number Diff line change
@@ -849,7 +849,7 @@ def load_datasets(self, dataset_names, schema='seacrowd'):
)
}

def load_config_name(self, config_name, schema='seacrowd'):
def load_dataset_by_config_name(self, config_name, schema='seacrowd'):
try:
for helper in sorted(self.filtered(
lambda x: (
@@ -861,7 +861,7 @@ def load_config_name(self, config_name, schema='seacrowd'):
except:
raise ValueError(f"Couldn't find dataset with config.name=`{config_name}` and schema=`{schema}`")

def load_config_names(self, config_names, schema='seacrowd'):
def load_datasets_by_config_names(self, config_names, schema='seacrowd'):
return {
helper.config.name: helper.load_dataset()
for helper in self.filtered(