Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Support Transformers >= 4.22 #42

Closed
umitbuyuksahin opened this issue Sep 19, 2022 · 1 comment · Fixed by #104
Closed

Support Transformers >= 4.22 #42

umitbuyuksahin opened this issue Sep 19, 2022 · 1 comment · Fixed by #104
Assignees
@tkilias @MarleneKress79789
Labels
feature New feature or request

Comments

@umitbuyuksahin
Copy link
Contributor

umitbuyuksahin commented Sep 19, 2022 •
edited by tkilias
Loading

Background

  • The directory structure where pre-trained models are downloaded has been changed with transformers 4.22
  • While the models files directly are downloaded into the cache directory in older versions, with version 4.22 transformers uses complex directory structure.
  • We need to upload this directory to bucket as we downloaded.
  • In order to complete this issue, the ticket Upload directory to bucket bucketfs-python#1 must be done.

Acceptance Criteria

  • switched to upload_directory method in bucketfs-utils
  • update toml file with `transformers="^4.22"'
    • pay attention that newer versions install nvidia modules which increases SLC size ~2GB
@umitbuyuksahin umitbuyuksahin added feature New feature or request source:exasol labels Sep 19, 2022
@tkilias tkilias self-assigned this Jul 25, 2023
@tkilias tkilias mentioned this issue Jul 27, 2023
Merged
@MarleneKress79789 MarleneKress79789 self-assigned this Aug 10, 2023
@redcatbear
Copy link
Contributor

Needs another review from Marlene.

tkilias added a commit that referenced this issue Aug 25, 2023
@tkilias @MarleneKress79789
#42: Update transformers to 4.31 and adapt the model uploader (#104)
0b61631 
* Updated to transformers 4.31.0 and torch 2.0.1+cpu.
* Adapt bucketfs_operations.py to the new model format
* Used tar to upload model directory
* Fix upload_model.py by unifying in model and tokenizer path to a single local-model-path. Because we are using a tar archive of the cache directory as the model file which then gets extracted from the BucketFS, we required that the Model and Tokenizer are in the same cache directory under the same name.

Co-authored-by: Marlene Kreß <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@tkilias tkilias

@MarleneKress79789 MarleneKress79789

Labels
feature New feature or request
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

#42: Update transformers to 4.31 and adapt the model uploader exasol/transformers-extension
4 participants
@redcatbear @umitbuyuksahin @tkilias @MarleneKress79789