Skip to content

Commit

Permalink
docs: more formatting and pruning
Browse files Browse the repository at this point in the history
  • Loading branch information
amitkparekh authored Jul 3, 2024
1 parent 60b8ea4 commit f37019a
Showing 1 changed file with 1 addition and 12 deletions.
13 changes: 1 addition & 12 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -18,8 +18,6 @@

</div>

---

Evaluating the generalisation capabilities of multimodal models based solely on their performance on out-of-distribution data fails to capture their true robustness. This work introduces a comprehensive evaluation framework that systematically examines the role of instructions and inputs in the generalisation abilities of such models, considering architectural design, input perturbations across language and vision modalities, and increased task complexity. The proposed framework uncovers the resilience of multimodal models to extreme instruction perturbations and their vulnerability to observational changes, raising concerns about overfitting to spurious correlations. By employing this evaluation framework on current Transformer-based multimodal models for robotic manipulation tasks, we uncover limitations and suggest future advancements should focus on architectural and training innovations that better integrate multimodal inputs, enhancing a model's generalisation prowess by prioritising sensitivity to input content over incidental correlations.

<br/>
Expand All @@ -30,7 +28,7 @@ Evaluating the generalisation capabilities of multimodal models based solely on
![Table of perturbations from the paper](docs/PERT%20Table.png)

<div align='center'>
<small><i>Our evaluation framework. Each perturbation affects the instruction or observation inputs, which can be linguistic, visual, or a combination of both. The plausibility of a perturbation relates to a model's expected performance. Sensitivity to unreasonable conditions (:heavy_multiplication_x:) indicates that a model should not perform the task successfully given the perturbation, while plausible perturbations (:heavy_check_mark:) suggest that it should still perform successfully.</i></small>
<small><i>Our evaluation framework. Each perturbation affects the instruction or observation inputs, which can be linguistic, visual, or a combination of both. The plausibility of a perturbation relates to a model's expected performance.</i></small>
</div>


Expand All @@ -48,15 +46,6 @@ Additionally, I've tried to work in a constrained, clean, and robust manner. I h
> This project is codenamed `cogelot` so that's what the library is called to prevent needing to rewrite everything.

### Limitations on further development

> [!WARNING]
> This is a research project, and so the scope of it is limited to the paper it is attached to.
It is unlikely that I will maintaining this repository to ensure it continues to work with the latest dependencies outside those pinned. That said, I've tried to be lax in my pinning of deps so that building on this project will not be overly restrictive.




### Model Architectures and Checkpoints

Expand Down

0 comments on commit f37019a

Please sign in to comment.