[FA-2] Revert suggestion that broke FA2 fine-tuning with quantized models

[younesbelkada]

What does this PR do?

Reverts: #26846 (comment)

This leads to an error because getattr will try to evaluate the third argument. for quantized models weight attribute do not exist leading to an error making the fine-tuning not possible

To repro on a more local scope:

>>> import torch
>>> t = torch.randn(1)
>>> getattr(t, 'device', t.non_existent_attribute)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
AttributeError: 'Tensor' object has no attribute 'non_existent_attribute'

cc @ArthurZucker

[LysandreJik]

Ah, nice catch! Thanks @younesbelkada