unify how to freeze some parameters for coca pre-training #526
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
facebook-github-bot
added
the
CLA Signed
|
This pull request was exported from Phabricator. Differential Revision: D54559503 |
zhangtemplar
added a commit
to zhangtemplar/multimodal
that referenced
this pull request
Mar 14, 2024
…search#526) Summary: 1. we already have support of freezing vision encoder; as experiment goes, we want to experiment to freeze other part of coca, e.g., text decoder. This diff provides unified way of freezing/unfreezing modules, the same way as we are doing for linear probe or finetune. 2. add configuration of using MLP instead of attention pooler for vision adapter; 3. for output projection in text decoder, change bias=False to True. In many other places, e.g., LP head, ember's output module and LLAVA, they are using bias=True (which is default value in Linear). Differential Revision: D54559503 Privacy Context Container: 303860477774201
zhangtemplar
force-pushed
the
export-D54559503
branch
from
f4a1103
to
3953609
Compare
|
This pull request was exported from Phabricator. Differential Revision: D54559503 |
zhangtemplar
added a commit
to zhangtemplar/multimodal
that referenced
this pull request
Mar 14, 2024
…search#526) Summary: 1. we already have support of freezing vision encoder; as experiment goes, we want to experiment to freeze other part of coca, e.g., text decoder. This diff provides unified way of freezing/unfreezing modules, the same way as we are doing for linear probe or finetune. 2. add configuration of using MLP instead of attention pooler for vision adapter; 3. for output projection in text decoder, change bias=False to True. In many other places, e.g., LP head, ember's output module and LLAVA, they are using bias=True (which is default value in Linear). Differential Revision: D54559503 Privacy Context Container: 303860477774201
zhangtemplar
force-pushed
the
export-D54559503
branch
from
3953609
to
44b179b
Compare
|
This pull request was exported from Phabricator. Differential Revision: D54559503 |
Codecov ReportAll modified and coverable lines are covered by tests ✅
Additional details and impacted files@@ Coverage Diff @@
## main #526 +/- ##
=======================================
Coverage 75.61% 75.62%
=======================================
Files 234 234
Lines 16122 16126 +4
=======================================
+ Hits 12191 12195 +4
Misses 3931 3931 ☔ View full report in Codecov by Sentry. |
zhangtemplar
added a commit
to zhangtemplar/multimodal
that referenced
this pull request
Mar 20, 2024
…search#526) Summary: 1. we already have support of freezing vision encoder; as experiment goes, we want to experiment to freeze other part of coca, e.g., text decoder. This diff provides unified way of freezing/unfreezing modules, the same way as we are doing for linear probe or finetune. 2. add configuration of using MLP instead of attention pooler for vision adapter; 3. for output projection in text decoder, change bias=False to True. In many other places, e.g., LP head, ember's output module and LLAVA, they are using bias=True (which is default value in Linear). Differential Revision: D54559503 Privacy Context Container: 303860477774201
zhangtemplar
force-pushed
the
export-D54559503
branch
from
44b179b
to
da89229
Compare
|
This pull request was exported from Phabricator. Differential Revision: D54559503 |
zhangtemplar
added a commit
to zhangtemplar/multimodal
that referenced
this pull request
Mar 21, 2024
…search#526) Summary: 1. we already have support of freezing vision encoder; as experiment goes, we want to experiment to freeze other part of coca, e.g., text decoder. This diff provides unified way of freezing/unfreezing modules, the same way as we are doing for linear probe or finetune. 2. add configuration of using MLP instead of attention pooler for vision adapter; 3. for output projection in text decoder, change bias=False to True. In many other places, e.g., LP head, ember's output module and LLAVA, they are using bias=True (which is default value in Linear). Differential Revision: D54559503 Privacy Context Container: 303860477774201
zhangtemplar
force-pushed
the
export-D54559503
branch
from
da89229
to
88933e9
Compare
|
This pull request was exported from Phabricator. Differential Revision: D54559503 |
zhangtemplar
added a commit
to zhangtemplar/multimodal
that referenced
this pull request
Mar 29, 2024
…search#526) Summary: 1. we already have support of freezing vision encoder; as experiment goes, we want to experiment to freeze other part of coca, e.g., text decoder. This diff provides unified way of freezing/unfreezing modules, the same way as we are doing for linear probe or finetune. 2. add configuration of using MLP instead of attention pooler for vision adapter; 3. for output projection in text decoder, change bias=False to True. In many other places, e.g., LP head, ember's output module and LLAVA, they are using bias=True (which is default value in Linear). Differential Revision: D54559503 Privacy Context Container: 303860477774201
zhangtemplar
force-pushed
the
export-D54559503
branch
from
88933e9
to
abc1037
Compare
|
This pull request was exported from Phabricator. Differential Revision: D54559503 |
zhangtemplar
force-pushed
the
export-D54559503
branch
from
abc1037
to
dbeed97
Compare
|
This pull request was exported from Phabricator. Differential Revision: D54559503 |
zhangtemplar
added a commit
to zhangtemplar/multimodal
that referenced
this pull request
Apr 8, 2024
Summary: 1. for output projection in text decoder, change bias=False to True. In many other places, e.g., LP head, ember's output module and LLAVA, they are using bias=True (which is default value in Linear). 2. add configuration of using MLP instead of attention pooler for vision adapter; Differential Revision: D55897450 Privacy Context Container: 303860477774201
facebook-github-bot
pushed a commit
that referenced
this pull request
Apr 25, 2024
Summary: Pull Request resolved: #527 Pull Request resolved: #526 1. for output projection in text decoder, change bias=False to True. In many other places, e.g., LP head, ember's output module and LLAVA, they are using bias=True (which is default value in Linear). 2. add configuration of using MLP instead of attention pooler for vision adapter; Reviewed By: Bellaktris Differential Revision: D55897450 Privacy Context Container: 303860477774201 fbshipit-source-id: 8e012b0c3d37566364f216dbfa8aec389142afe1
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary:
Differential Revision:
D54559503
Privacy Context Container: 303860477774201