Sam2.1 #4
Closed
Sam2.1 #4
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
sam2/build_sam.py:81:14: TOR102 [*] `torch.load` without `weights_only` parameter is unsafe. Explicitly set `weights_only` to False only if you trust the data you load and full pickle functionality is needed, otherwise set `weights_only=True`. Found with https://github.com/pytorch-labs/torchfix/
Configure a workflow for format checking
…all available kernels if Flash Attention fails (#155) In this PR, we make it optional to build the SAM 2 CUDA extension, in observation that many users encounter difficulties with the CUDA compilation step. 1. During installation, we catch build errors and print a warning message. We also allow explicitly turning off the CUDA extension building with `SAM2_BUILD_CUDA=0`. 2. At runtime, we catch CUDA kernel errors from connected components and print a warning on skipping the post processing step. We also fall back to the all available kernels if the Flash Attention kernel fails.
This PR suggests a way to resolve the error of `unsupported Microsoft Visual Studio version!` in INSTALL.md. Adding `-allow-unsupported-compiler` argument for the `nvcc` worked. Editing [setup.py](https://github.com/facebookresearch/segment-anything-2/blob/main/setup.py) is required to add the `-allow-unsupported-compiler` argument for `nvcc`. ```python def get_extensions(): srcs = ["sam2/csrc/connected_components.cu"] compile_args = { "cxx": [], "nvcc": [ "-DCUDA_HAS_FP16=1", "-D__CUDA_NO_HALF_OPERATORS__", "-D__CUDA_NO_HALF_CONVERSIONS__", "-D__CUDA_NO_HALF2_OPERATORS__", "-allow-unsupported-compiler" # Add this argument ], } ext_modules = [CUDAExtension("sam2._C", srcs, extra_compile_args=compile_args)] return ext_modules ```
This is referring to facebookresearch/sam2#137 , which in itself refers to a common problem during installation, mentioned on facebookresearch/sam2#19 . Some users may encounter significant trouble installing the project, running into the error `OSError: CUDA_HOME environment variable is not set. Please set it to your CUDA install root.`. Simply adding the `--no-build-isolation` flag to the pip install, e.g. `pip install --no-build-isolation -e .`, usually solves this problem. However, this fix is not mentioned anywhere within the readmes or installation troubleshooting docs. This PR adds this recommendation into the INSTALL.md file under the "My installation failed with `CUDA_HOME environment variable is not set` " section, ensuring that more users are aware of this potential fix. Examples of users experiencing related difficulties when installing: facebookresearch/sam2#19 facebookresearch/sam2#41 facebookresearch/sam2#99 facebookresearch/sam2#133
Not used head_dim = dim_out // num_heads self.scale = head_dim**-0.5 F.scaled_dot_product_attention takes care of this automatically.
Integrate with Hugging Face
This PR adds an example to provide box prompt in SAM 2 as inputs to the `add_new_points_or_box` API (renamed from`add_new_points`, which is kept for backward compatibility). If `box` is provided, we add it as the first two points with labels 2 and 3, along with the user-provided points (consistent with how SAM 2 is trained). The video predictor notebook `notebooks/video_predictor_example.ipynb` is updated to include segmenting from box prompt as an example.
…s` package during installation (#176)
ufmt formatting fixed.
remove unused attributes from hieradet.py
…edictor Fix HF image predictor
…rious typos (#218) (close #217, #66, #67, #69, #91, #126, #127, #145)
…m2` (#341) This PR update repo links after we renamed the repo from `segment-anything-2` to `sam2`. It also changes `NAME` in setup.py to `SAM-2` (which is already the named used in pip setup since python packages don't allow whitespace)
Use `weights_only` for loading
This small PR adds GPU specification in `docker-compose.yaml` for the SAM 2 interactive webdemo, following https://docs.docker.com/compose/how-tos/gpu-support/#example-of-a-compose-file-for-running-a-service-with-access-to-1-gpu-device. It fixes a GPU access error as reported in facebookresearch/sam2#354. (close facebookresearch/sam2#354)
… parent directory of the sam2 repo (#359) If the user has "sam2/sam2" in their path, they are likey importing the repo itself as "sam2" rather than importing the "sam2" python package (i.e. "sam2/sam2" directory). This typically happens because the user is running Python from the parent directory that contains the sam2 repo they cloned. In general, the user should not run Python from the parent dir when the repo is cloned into (same is true for e.g. Numpy repo that contains names like `numpy/numpy` where the module and the repo have the same name), as the user encountered in facebookresearch/sam2#346. (close facebookresearch/sam2#346)
Summary:
The Strawberry GraphQL library recently disabled multipart requests by default. This resulted in a video upload request returning "Unsupported content type" instead of uploading the video, processing it, and returning the video path.
This issue was raised in #361. A forward fix is to add `multipart_uploads_enabled=True` to the endpoint view.
Test Plan:
Tested locally with cURL and upload succeeds
*Request*
```
curl http://localhost:7263/graphql \
-F operations='{ "query": "mutation($file: Upload!){ uploadVideo(file: $file) { path } }", "variables": { "file": null } }' \
-F map='{ "file": ["variables.file"] }' \
-F [email protected]
```
*Response*
```
{"data": {"uploadVideo": {"path": "uploads/<HASH>.mp4"}}}
```
[sam2][demo][1/x] Fix file upload
Summary:
The Strawberry GraphQL library recently disabled multipart requests by default. This resulted in a video upload request returning "Unsupported content type" instead of uploading the video, processing it, and returning the video path.
This issue was raised in #361. A forward fix is to add `multipart_uploads_enabled=True` to the endpoint view.
Test Plan:
Tested locally with cURL and upload succeeds
*Request*
```
curl http://localhost:7263/graphql \
-F operations='{ "query": "mutation($file: Upload!){ uploadVideo(file: $file) { path } }", "variables": { "file": null } }' \
-F map='{ "file": ["variables.file"] }' \
-F [email protected]
```
*Response*
```
{"data": {"uploadVideo": {"path": "uploads/<HASH>.mp4"}}}
```
…edup and a new SAM2VideoPredictor to better handle multi-object tracking (#486) This PR provides new features and updates for SAM 2: - We now support `torch.compile` of the entire SAM 2 model on videos, which can be turned on by setting `vos_optimized=True` in `build_sam2_video_predictor` (it uses the new `SAM2VideoPredictorVOS` predictor class in `sam2/sam2_video_predictor.py`). * Compared to the previous setting (which only compiles the image encoder backbone), the new full model compilation gives a major speedup in inference FPS. * In the VOS prediction script `tools/vos_inference.py`, you can specify this option in `tools/vos_inference.py` via the `--use_vos_optimized_video_predictor` flag. * Note that turning on this flag might introduce a small variance in the predictions due to numerical differences caused by `torch.compile` of the full model. * **PyTorch 2.5.1 is the minimum version for full support of this feature**. (Earlier PyTorch versions might run into compilation errors in some cases.) Therefore, we have updated the minimum PyTorch version to 2.5.1 accordingly in the installation scripts. - We also update the implementation of the `SAM2VideoPredictor` class for the SAM 2 video prediction in `sam2/sam2_video_predictor.py`, which allows for independent per-object inference. Specifically, in the new `SAM2VideoPredictor`: * Now **we handle the inference of each object independently** (as if we are opening a separate session for each object) while sharing their backbone features. * This change allows us to relax the assumption of prompting for multi-object tracking. Previously (due to the batching behavior in inference), if a video frame receives clicks for only a subset of objects, the rest of the (non-prompted) objects are assumed to be non-existent in this frame (i.e., in such frames, the user is telling SAM 2 that the rest of the objects don't appear). Now, if a frame receives clicks for only a subset of objects, we do not make any assumptions about the remaining (non-prompted) objects (i.e., now each object is handled independently and is not affected by how other objects are prompted). As a result, **we allow adding new objects after tracking starts** after this change (which was previously a restriction on usage). * We believe that the new version is a more natural inference behavior and therefore switched to it as the default behavior. The previous implementation of `SAM2VideoPredictor` is backed up to in `sam2/sam2_video_predictor_legacy.py`. All the VOS inference results using `tools/vos_inference.py` should remain the same after this change to the `SAM2VideoPredictor` class.
…eoPredictor` (#490) This PR adds a pathc for the case of `offload_state_to_cpu=True` where `pred_masks` might have been offload to CPU device (close facebookresearch/sam2#489)
…r in MPS (#495) In this PR, we remove `.pin_memory()` in `obj_pos` of `SAM2Base` to resolve and error in MPS. Investigations show that `.pin_memory()` causes an error of `Attempted to set the storage of a tensor on device "cpu" to a storage on different device "mps:0"`, as originally reported in facebookresearch/sam2#487. (close facebookresearch/sam2#487)
# Conflicts: # sam2/modeling/sam2_base.py # sam2/sam2_video_predictor.py # setup.py
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
No description provided.