Documentation for verify() #960

vkovinicTT · 2024-12-24T17:57:54Z

Added documentation for the verify() function and VerifyConfig class. This includes an overview of verification steps, examples of usage, and details on configurable checkers such as AutomaticValueChecker, AllCloseValueChecker and FullValueChecker. Additionally, provided information for enabling/disabling verification features through the VerifyConfig.

pilkicTT

thanks for adding the docs!

ps. please update the PR title

pilkicTT · 2024-12-24T18:30:16Z

docs/src/SUMMARY.md

@@ -7,6 +7,7 @@
 # Project setup
 - [Building](./build.md)
 - [Testing](./test.md)
+- [Verification](./verification.md)


imo this should be in the dev notes section, something like Verification in tests

pilkicTT · 2024-12-24T18:30:51Z

docs/src/verification.md

+def test_add():
+
+	class Add(nn.Module):
+		def __init__(self):


fix indent (4 spaces)

for some reason github renders it to 8 spaces, even though it's 4 spaces for me locally.

you should use spaces, not tabs

pilkicTT · 2024-12-24T18:32:45Z

docs/src/verification.md

+<br/>
+<br/>
+
+<img src="imgs/checkers/checkers.svg" alt="Checkers" style="width: 1200%;">


let's see what others say, but to me the picture doesn't add much value and makes the doc harder to edit/update

@nvukobratTT what do you think?

I'm okay with using images and they can be really useful as part of documentaion. However, it's important that those can be edit/updated over time.

That said, I would opt out for using ExcallyDraw to create this and similar diagrams, and next to the image also store exported ED file which we can load and edit in the future.

pilkicTT · 2024-12-24T18:34:14Z

docs/src/verification.md

+| Feature                           | Enabled (default) |
+|-----------------------------------|:-----------------:|
+| Verification as a method          | `True`            |
+| Output size check                 | `True`            |


what is the output size?

I've changed table a bit, hopefully it's clearer now.

what is the output length? 😄 is it the number of output tensors?

hahah yes (I will put number of output tensors instead)

nvukobratTT · 2024-12-27T12:24:26Z

docs/src/verification.md

+
+## General Overview
+
+When comparing our `compiled_model` with the `framework_model` (e.g., `PyTorch`), we aim to verify whether the output from the `compiled_model` is sufficiently similar to the output from the `framework_model` (where required degree of similarity is configurable).


For the introduction section, let's use common language instead of code-specific names (e.g. instead of compiled_model, let's use compiled model).

Using code-specific names is beneficial below examples. In other case, I would more prefer to have common language :))

nvukobratTT · 2024-12-27T12:25:15Z

docs/src/verification.md

+
+## General Overview
+
+When comparing our `compiled_model` with the `framework_model` (e.g., `PyTorch`), we aim to verify whether the output from the `compiled_model` is sufficiently similar to the output from the `framework_model` (where required degree of similarity is configurable).


Let's also mention what the framework model is. E.g. model that is run on host/cpu.

nvukobratTT · 2024-12-27T12:25:50Z

docs/src/verification.md

+2. Run a forward pass through the framework model
+3. Compile the framework model using `Forge`
+4. Run a forward pass through the compiled model
+5. Compare the outputs.


Nit picking, but let's put . on every point, or remove it :))

nvukobratTT · 2024-12-27T12:27:17Z

docs/src/verification.md

+|-----------------------------------|---------------------|:-----------------:|
+| Verification as a method          | `enabled`           | `True`            |
+| Number of output tensors check    | `verify_size`       | `True`            |
+| Output type check                 | `verify_dtype`      | `True`            |


Let's be more precise. E.g. output data format/type check

nvukobratTT · 2024-12-27T12:29:40Z

docs/src/verification.md

+For more information about `VerifyConfig` you can check `forge/forge/verify/config.py`.
+
+**Example of usage**
+```python


Maybe we can keep following examples a bit shorter. E.g. now that we have full example above, we can do something like this:

... fw_out = framework_model(*inputs) co_out = compiled_model(*inputs) verify(inputs, framework_model, compiled_model, VerifyConfig(verify_dtype=False))

nvukobratTT

LGTM!

vkovinicTT requested review from nvukobratTT, pilkicTT and dgolubovicTT as code owners December 24, 2024 17:57

vkovinicTT force-pushed the vkovinic/verify_doc branch from eac11ce to bab1353 Compare December 24, 2024 17:58

pilkicTT requested changes Dec 24, 2024

View reviewed changes

vkovinicTT changed the title ~~Vkovinic/verify doc~~ Documentation for verify() Dec 24, 2024

vkovinicTT force-pushed the vkovinic/verify_doc branch 2 times, most recently from e8e5897 to 6dd3150 Compare December 25, 2024 11:26

vkovinicTT requested a review from pilkicTT December 25, 2024 11:32

vkovinicTT force-pushed the vkovinic/verify_doc branch 5 times, most recently from 0301a80 to 4966a68 Compare December 26, 2024 10:57

nvukobratTT reviewed Dec 27, 2024

View reviewed changes

vkovinicTT added 4 commits December 27, 2024 13:28

verification docs

69d614b

pr comments

403735e

replaces tab with spaces and changed description in table

3ea2f6a

additional pr comments

6296c29

vkovinicTT force-pushed the vkovinic/verify_doc branch from 4966a68 to 6296c29 Compare December 27, 2024 14:38

vkovinicTT requested a review from nvukobratTT December 27, 2024 14:38

nvukobratTT approved these changes Dec 27, 2024

View reviewed changes

vkovinicTT self-assigned this Dec 27, 2024

vkovinicTT added the documentation Improvements or additions to documentation label Dec 27, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Documentation for verify() #960

Documentation for verify() #960

vkovinicTT commented Dec 24, 2024

pilkicTT left a comment

pilkicTT Dec 24, 2024

pilkicTT Dec 24, 2024

vkovinicTT Dec 25, 2024

pilkicTT Dec 26, 2024

pilkicTT Dec 24, 2024

vkovinicTT Dec 25, 2024

nvukobratTT Dec 27, 2024

pilkicTT Dec 24, 2024

vkovinicTT Dec 25, 2024

pilkicTT Dec 26, 2024

vkovinicTT Dec 26, 2024

nvukobratTT Dec 27, 2024

nvukobratTT Dec 27, 2024

nvukobratTT Dec 27, 2024

nvukobratTT Dec 27, 2024

nvukobratTT Dec 27, 2024

nvukobratTT left a comment


		## General Overview

		When comparing our `compiled_model` with the `framework_model` (e.g., `PyTorch`), we aim to verify whether the output from the `compiled_model` is sufficiently similar to the output from the `framework_model` (where required degree of similarity is configurable).

Documentation for verify() #960

Are you sure you want to change the base?

Documentation for verify() #960

Conversation

vkovinicTT commented Dec 24, 2024

pilkicTT left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

nvukobratTT left a comment

Choose a reason for hiding this comment