You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
if datasets["dropped_samples"] is not None:
# log model performance on the dropped samples
if verbose:
print("Evaluating on samples dropped from the training set:")
stats = classifier.evaluate(datasets["dropped_samples"], verbose=verbose)
if verbose:
print(stats)
if not kwargs.get("test", False):
for param, value in zip(param_names, stats):
wandb.run.summary[f"dropped_samples_{param}"] = value
p, r = (
wandb.run.summary["dropped_samples_precision"],
wandb.run.summary["dropped_samples_recall"],
)
wandb.run.summary["dropped_samples_f1"] = 2 * p * r / (p + r)
Since the dropped samples only contain labels from the underrepresented class, the precision p and recall r will be sometimes be zero, leading to a divide by zero error and a 'failed' run in wandb. A quick solution is to remove the f1 score from the metrics computed for the dropped samples.
The text was updated successfully, but these errors were encountered:
The
Dataset
class inscope/utils.py
allows for samples to be dropped to balance positive and negative examples for a class. In the training process, the code attempts to evaluate an f1 score for the dropped samples (full code below):https://github.com/ZwickyTransientFacility/scope/blob/ac22f211a188eb1736e35eb9c6c552a902735fb5/scope.py#L665
Since the dropped samples only contain labels from the underrepresented class, the precision
p
and recallr
will be sometimes be zero, leading to a divide by zero error and a 'failed' run in wandb. A quick solution is to remove the f1 score from the metrics computed for the dropped samples.The text was updated successfully, but these errors were encountered: