[bug] Constevaluating trainable nn.Parameter #929
Comments
Merged
pmarkovicTT
added a commit
that referenced
this issue
Dec 18, 2024
Add test to make sure LoRA layer can be implemented and used in a full
training pipeline.
Problem encountered
Hitting `KeyError: 'gradient_lora1.b.consteval_graph.output'` when
implementing LoRA layer with nn.Parameters.
```
class LoraLayer(nn.Module):
def __init__(self, input_size, output_size, rank=8, alpha=4, dtype=torch.float32):
super(LoraLayer, self).__init__()
self.a = nn.Parameter(torch.empty(input_size, rank, dtype=dtype), requires_grad=True)
self.b = nn.Parameter(torch.zeros(rank, output_size, dtype=dtype), requires_grad=True)
self.alpha = alpha / rank
nn.init.normal_(self.a, mean=0, std=1)
def forward(self, x):
return self.alpha * (x @ self.a @ self.b)
```
Raised issue: #929
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Description:
Hitting KeyError:
'gradient_lora1.b.consteval_graph.output'when implementing LoRA layer with nn.Parameters.Repro branch:
https://github.com/tenstorrent/tt-forge-fe/tree/pmarkovic/parameter-consteval-issue
The text was updated successfully, but these errors were encountered: