Add DETA #20983

NielsRogge · 2023-01-03T13:59:53Z

What does this PR do?

This PR adds DETA. DETA is a slight change to Deformable DETR by using traditional IoU-based assignment as opposed to the Hungarian matching used in the original DETR, and incorporating NMS (non-maximum suppression) in the postprocessing.

Note: this model has a torchvision dependency for NMS.

To do:

transfer checkpoints

NielsRogge · 2023-01-04T10:46:05Z

cc @alaradirik this PR is in a ready state, except for 2 things:

whether or not we leverage torchvision's batched_nms => the CI is currently failing because this library is not installed. Will also ask for @sgugger and @LysandreJik's opinion here
the post_process_object_detection method might require an in-depth look

sgugger · 2023-01-04T13:55:56Z

There is no problem with the model requiring torchvision to be installed. We have many models with specific dependencies, some of which you ported yourself ;-).
Just protect the import between if is_torchvision_available() and have a the first line in the init of the models be a require_backends(["torchvision"]).

alaradirik

Thanks for adding this! I added a few comments but it looks good to me overall.

Could you add an nms_threshold argument to the object detection post processing? I can do a follow up PR to add it to all other post processing methods. Or it can be left as it is and I can add it shortly and update the object detection pipeline as well

src/transformers/models/deta/test.py

src/transformers/models/deta/modeling_deta.py

alaradirik · 2023-01-13T11:20:59Z

src/transformers/models/deta/modeling_deta.py

+                    is_level_ordered = (
+                        level_ids[keep_inds][None]
+                        == torch.arange(len(spatial_shapes), device=level_ids.device)[:, None]
+                    )  # LS


Could we make the comment more descriptive?

Pinging the authors here, @xingyizhou @jozhang97, could you clarify what LS means here?

tests/models/deta/test_modeling_deta.py

src/transformers/models/deta/image_processing_deta.py

alaradirik · 2023-01-15T15:24:44Z

src/transformers/models/deta/image_processing_deta.py

+        Args:
+            outputs ([`DetrObjectDetectionOutput`]):
+                Raw outputs of the model.
+            threshold (`float`, *optional*):


Could we add a nms_threshold argument and set it to 0.7 by default? We can leave DETA out of object detection mapping for now and I can do a followup PR shortly to add NMS support to all post_process_object_detection methods.

We can't just add NMS support to all post_process_object_detection methods I'm afraid, since for that one uses Torchvision's NMS op which is written in C and much faster than plain python. Also our existing models don't need NMS, DETA is actually the first one that needs it

NMS is a quite common to post-processing method for object detection and useful for models with noisy bounding box proposals. Given two bounding boxes with high prediction scores, they might be detecting the same object instance and NMS eliminates the redundant boxes.

I think it'd be nice to have an option to perform NMS but I'm fine with not adding it as well.
CC'ing @amyeroberts and @sgugger

since for that one uses Torchvision's NMS op which is written in C and much faster than plain python

@NielsRogge I'm not sure I completely follow - is the issue here speed or the use of torchvision?

In general, I agree with @alaradirik, NMS is common enough that it's something I think we want to support, even if it's only for our PyTorch models at the moment.

Ok we can add NMS as an option, this way the API of all post_process_object_detection methods will be the same. I assume we'll want to leverage torchvision for this

Yes, it makes sense to leverage torchvision for this

HuggingFaceDocBuilderDev · 2023-01-18T14:29:54Z

The documentation is not available anymore as the PR was closed or merged.

sgugger

Thanks for adding this new model. Left a couple of comments.

setup.py

src/transformers/__init__.py

src/transformers/models/deta/__init__.py

src/transformers/models/deta/configuration_deta.py

src/transformers/models/deta/image_processing_deta.py

src/transformers/models/deta/modeling_deta.py

src/transformers/utils/dummy_torchvision_objects.py

NielsRogge · 2023-01-25T14:06:59Z

@sgugger I've addressed all comments, except for adding support for the custom kernel.

Could we perhaps add support for the custom kernel for the 3 models (Mask2Former, OneFormer and DETA) in a separate PR?

sgugger · 2023-01-25T15:36:15Z

In this case, remove the code trying to load the custom kernels in the modeling file and we can add it back in the PR that will deal with custom kernels.

NielsRogge · 2023-01-26T12:24:11Z

@sgugger ok, feel free to approve :)

sgugger

Thanks for all your work on this!

src/transformers/models/deta/modeling_deta.py

NielsRogge · 2023-01-31T09:43:06Z

Failing test is unrelated/flaky, merging.

@ydshieh

* First draft * Add initial draft of conversion script * Convert all weights * Fix config * Add image processor * Fix DetaImageProcessor * Run make fix copies * Remove timm dependency * Fix dummy objects * Improve loss function * Remove conv_encoder attribute * Update conversion scripts * Improve postprocessing + docs * Fix copied from statements * Add tests * Improve postprocessing * Improve postprocessing * Update READMEs * More improvements * Fix rebase * Add is_torchvision_available * Add torchvision dependency * Fix typo and README * Fix bug * Add copied from * Fix style * Apply suggestions * Fix thanks to @ydshieh * Fix another dependency check * Simplify image processor * Add scipy * Improve code * Add threshold argument * Fix bug * Set default threshold * Improve integration test * Add another integration test * Update setup.py * Address review * Improve deformable attention function * Improve copied from * Use relative imports * Address review * Replace assertions * Address review * Update dummies * Remove dummies * Address comments, update READMEs * Remove custom kernel code * Add image processor tests * Add requires_backends * Add minor comment * Update scripts * Update organization name * Fix defaults, add doc tests * Add id2label for object 365 * Fix tests * Update task guide

@ydshieh

* First draft * Add initial draft of conversion script * Convert all weights * Fix config * Add image processor * Fix DetaImageProcessor * Run make fix copies * Remove timm dependency * Fix dummy objects * Improve loss function * Remove conv_encoder attribute * Update conversion scripts * Improve postprocessing + docs * Fix copied from statements * Add tests * Improve postprocessing * Improve postprocessing * Update READMEs * More improvements * Fix rebase * Add is_torchvision_available * Add torchvision dependency * Fix typo and README * Fix bug * Add copied from * Fix style * Apply suggestions * Fix thanks to @ydshieh * Fix another dependency check * Simplify image processor * Add scipy * Improve code * Add threshold argument * Fix bug * Set default threshold * Improve integration test * Add another integration test * Update setup.py * Address review * Improve deformable attention function * Improve copied from * Use relative imports * Address review * Replace assertions * Address review * Update dummies * Remove dummies * Address comments, update READMEs * Remove custom kernel code * Add image processor tests * Add requires_backends * Add minor comment * Update scripts * Update organization name * Fix defaults, add doc tests * Add id2label for object 365 * Fix tests * Update task guide

NielsRogge force-pushed the add_deta branch from 0212921 to 478c7b4 Compare January 3, 2023 14:04

NielsRogge force-pushed the add_deta branch from e2c0cf7 to 2572d50 Compare January 5, 2023 15:52

alaradirik reviewed Jan 16, 2023

View reviewed changes

NielsRogge force-pushed the add_deta branch from 8c000f3 to 47a90b1 Compare January 16, 2023 20:48

alaradirik approved these changes Jan 19, 2023

View reviewed changes

NielsRogge requested a review from sgugger January 19, 2023 14:22

sgugger reviewed Jan 19, 2023

View reviewed changes

NielsRogge force-pushed the add_deta branch 2 times, most recently from d64a408 to ef7ff2b Compare January 24, 2023 16:13

NielsRogge force-pushed the add_deta branch from d3475d2 to a8f5952 Compare January 26, 2023 11:12

sgugger approved these changes Jan 26, 2023

View reviewed changes

src/transformers/models/deta/modeling_deta.py Show resolved Hide resolved

NielsRogge force-pushed the add_deta branch 2 times, most recently from 7d97248 to 2d8d824 Compare January 30, 2023 21:00

NielsRogge added 11 commits January 31, 2023 08:42

First draft

8ef72d4

Add initial draft of conversion script

756a053

Convert all weights

da2cc18

Fix config

771584e

Add image processor

6d0a402

Fix DetaImageProcessor

7da311b

Run make fix copies

137c36f

Remove timm dependency

3673174

Fix dummy objects

460f629

Improve loss function

fce672e

Remove conv_encoder attribute

201add7

NielsRogge added 25 commits January 31, 2023 08:42

Fix bug

433672a

Set default threshold

6761600

Improve integration test

afafd18

Add another integration test

32ac2a0

Update setup.py

21cd670

Address review

92e54e6

Improve deformable attention function

8652aa4

Improve copied from

e042fd2

Use relative imports

6a60910

Address review

2d2cb82

Replace assertions

1045e8d

Address review

6fca8d4

Update dummies

624bc41

Remove dummies

cc89371

Address comments, update READMEs

80bd3c0

Remove custom kernel code

5d855ce

Add image processor tests

7adaa78

Add requires_backends

e7ae8df

Add minor comment

0267a59

Update scripts

44f1b61

Update organization name

d47eab7

Fix defaults, add doc tests

84e4a2b

Add id2label for object 365

6e17822

Fix tests

f9d2cce

Update task guide

2377623

NielsRogge force-pushed the add_deta branch from 2d8d824 to 2377623 Compare January 31, 2023 08:46

NielsRogge merged commit 5451f88 into huggingface:main Jan 31, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add DETA #20983

Add DETA #20983

NielsRogge commented Jan 3, 2023 •

edited

Loading

NielsRogge commented Jan 4, 2023 •

edited

Loading

sgugger commented Jan 4, 2023

alaradirik left a comment

alaradirik Jan 13, 2023

NielsRogge Jan 17, 2023

alaradirik Jan 15, 2023

NielsRogge Jan 16, 2023

alaradirik Jan 17, 2023

amyeroberts Jan 17, 2023

NielsRogge Jan 18, 2023

alaradirik Jan 19, 2023

HuggingFaceDocBuilderDev commented Jan 18, 2023 •

edited

Loading

sgugger left a comment

NielsRogge commented Jan 25, 2023

sgugger commented Jan 25, 2023

NielsRogge commented Jan 26, 2023

sgugger left a comment

NielsRogge commented Jan 31, 2023

Add DETA #20983

Add DETA #20983

Conversation

NielsRogge commented Jan 3, 2023 • edited Loading

What does this PR do?

NielsRogge commented Jan 4, 2023 • edited Loading

sgugger commented Jan 4, 2023

alaradirik left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

HuggingFaceDocBuilderDev commented Jan 18, 2023 • edited Loading

sgugger left a comment

Choose a reason for hiding this comment

NielsRogge commented Jan 25, 2023

sgugger commented Jan 25, 2023

NielsRogge commented Jan 26, 2023

sgugger left a comment

Choose a reason for hiding this comment

NielsRogge commented Jan 31, 2023

NielsRogge commented Jan 3, 2023 •

edited

Loading

NielsRogge commented Jan 4, 2023 •

edited

Loading

HuggingFaceDocBuilderDev commented Jan 18, 2023 •

edited

Loading