-
Notifications
You must be signed in to change notification settings - Fork 1.8k
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
fix fp16 overflow in totalsegmentator <3
- Loading branch information
1 parent
de48541
commit d78f17f
Showing
1 changed file
with
2 additions
and
2 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
d78f17f
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We were just checking a similar issue and the same was happening with us at fp32... so good fix :-)
d78f17f
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This should not happen in fp32 as the value range is much higher. Are you certain this was what caused it? We only ran into the problem because we use fp16 here
d78f17f
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It was only in a few instances but it still happened. I changed it from 1000 to 1 exactly where 1000 was changed to 10 and we stopped having this issue, so we are fairly certain that this is what caused it. In any case it is fixed so that's the most important
d78f17f
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Glad this fixed it, then :-) Maybe we need to add an explicit inf check at some point in the pipeline to avoid future problems o.O
d78f17f
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think maybe something like
would be an easy fix for it?
A more overbearing question I had here - why is there a scaling factor in the gaussian? Is the point avoiding underflow? If so then maybe the gaussian could be 'adapted' to
and this would avoid the earlier solution which is slightly more intensive