Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

InternVL2 num frame #467

Closed
wants to merge 64 commits into from
Closed

InternVL2 num frame #467

wants to merge 64 commits into from

Conversation

pufanyi
Copy link
Collaborator

@pufanyi pufanyi commented Dec 20, 2024

Before you open a pull-request, please check if a similar issue already exists or has been closed before.

When you open a pull-request, please be sure to include the following

  • A descriptive title: [xxx] XXXX
  • A detailed description

If you meet the lint warnings, you can use following scripts to reformat code.

pip install pre-commit
pre-commit install
pre-commit run --all-files

Thank you for your contributions!

kcz358 and others added 30 commits October 24, 2024 16:34
@kcz358 @KairuiHu
[Fix] Fix extra calling in qwen_vl_api, use tempfile for tmp (#312)
b229412 
* Fix extra calling in qwen_vl_api, and use tempfile to replace storing tmp files

* Add cache mode for qwen_vl_api
@KairuiHu
add two tasks
2e0535a
@KairuiHu
correct parse mcq function to handle period for mmmu mcq response
c5f7c27
@KairuiHu
remove incorrect parse in videosearch
f19ea1d
@KairuiHu
optimize submission format for combined_milestone
039cf48
@pufanyi @KairuiHu
redefine read_video_pyav
26639f1
@pufanyi @KairuiHu
Optimize parsing of multiple-choice responses in videosearch
c3f3b3e
@pufanyi @KairuiHu
Optimize loading logic for video frames
255bb28
@pufanyi @KairuiHu
Optimize video frame loading and encoding
cdf602a
@pufanyi @KairuiHu
lint
7aa68db
@pufanyi @KairuiHu
lint
07eded7
@KairuiHu
add transcript for videosearch
484613d
@pufanyi @KairuiHu
lint
834fffe
@pufanyi @KairuiHu
[Bug] Fix the checking error in gpt4v generation (#274)
ff3b260 
* fix

* Fix error handling in GPT4V class
@KairuiHu
optimize videosearch parse script, add perception and understanding u…
5681934 
…tils, and add audio to accept transcript as input
@KairuiHu
video avg time for videoperception task
f5a0550
@KairuiHu
add videoperception task
60f0f4e
@KairuiHu
add two tracks for videoperception
b48e7f9
@KairuiHu
debug application video file path
f6156fe
@pufanyi @KairuiHu
Add video_mmmu task configuration
2a11a97
@pufanyi @KairuiHu
lint
3fe2ed7
@KairuiHu
augment options to 10 for application
aaab008
@KairuiHu
remove extra print from utils
5ecb093
@pufanyi
Add videommmu task configuration for audio_gemini
f8f7157
@pufanyi
Refactor code in samplers.py and audio_gemini_utils.py
e76927e 
Simplify the code in samplers.py by removing unnecessary conditions and improving readability. In audio_gemini_utils.py, remove the unused import and replace print statements with logger. Also, refactor the aggregate_results function to handle different subtasks.
@pufanyi
Refactor code in samplers.py and audio_gemini_utils.py
c8c8b13
@KairuiHu
Merge pull request #347 from EvolvingLMMs-Lab/pufanyi/video_mmmu_audio
2462443 
Pufanyi/video mmmu audio
@pufanyi
Merge remote-tracking branch 'origin/main' into kr/video
1ed5e7d
@pufanyi
fix qwen and vila
423244a
@pufanyi
remove debug logs
d6b75b3
pufanyi and others added 27 commits November 2, 2024 17:32
@pufanyi
lint
d1d4a83
@KairuiHu
Merge pull request #390 from EvolvingLMMs-Lab/pufanyi/claude_video_lo…
65cd722 
…ader

Claude video loader
@pufanyi
[Fix] fix some bugs in InternVL2 and Claude (#399)
a550ec1 
* fix claude

* fix internvl2 shard model

* remove debug output
@pufanyi
llava onevision, force include last frame
6384577
@pufanyi
fix llama template
055e140
@pufanyi
revert aria, as aria is already exists in #391
dc8b00a
@pufanyi
Merge commit 'bcbdc493d729e830f4775d1a1af4c1d7d8e449f2' into pufanyi/…
03a5823 
…llama
@pufanyi
lint
3d00a35
@pufanyi
fix aria device map
31d5de4
@KairuiHu
Merge pull request #407 from EvolvingLMMs-Lab/pufanyi/llama
8895a38 
Fix LLaMA, Oryx, merge main branch to add aria
@KairuiHu
adjust torch_dtype and add additional videoperception experiment
b0db512
@KairuiHu
lint
008d347
@KairuiHu
reformat samplers.py to pass lint
7e72485
@KairuiHu
reformat samplers.py to pass lint
2fd1c02
@KairuiHu
Apply black formatting to samplers.py
78cf156
@pufanyi
fix
82a2001
@pufanyi
fix
ab96465
@KairuiHu
fix python 3.10 issue in pyav load video
1637365
@KairuiHu
add annotation scripts for videoperception dataset
1345d43
@KairuiHu
add videofinal utils functinos
a5bcb73
@KairuiHu
enable transcripts evaluation all three tracks
c148532
@KairuiHu
modify utils to exclude samples with API Error in API model responses
9017e49
@KairuiHu
debug file path error
70f0826
@pufanyi
refactor video loading method to use read_video_pyav_pil for improved…
7f7a10c 
… frame handling
@pufanyi
lint
bbacbbd
@KairuiHu
Merge pull request #466 from EvolvingLMMs-Lab/pufanyi/update_aria
32b0753 
[Fix] Update Aria from extraction
@pufanyi
update load_video function parameters for improved flexibility in fra…
c61b430 
…me and segment handling
@pufanyi pufanyi requested a review from KairuiHu December 20, 2024 07:09
@pufanyi pufanyi closed this Dec 20, 2024
@pufanyi pufanyi deleted the pufanyi/internvl2-numframe branch December 20, 2024 07:12
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@KairuiHu KairuiHu Awaiting requested review from KairuiHu

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@pufanyi @kcz358 @KairuiHu