如何判断一个样本是unknow？

[liyangfan0]

作者你好！请教一个问题，在测试的时候论文是将前75个有着最大energy score的query作为unknow，但如果是测试一张query，如何判断它是unknow呢？很多论文是采用阈值来判断unknow，而这里直接用前75个作为unknow，实际上引入了先验信息，这种测试方式我很疑惑到底合不合理。

[jiajingchen113322]

作者你好！请教一个问题，在测试的时候论文是将前75个有着最大energy score的query作为unknow，但如果是测试一张query，如何判断它是unknow呢？很多论文是采用阈值来判断unknow，而这里直接用前75个作为unknow，实际上引入了先验信息，这种测试方式我很疑惑到底合不合理。

I have the same question for the author. During the evaluation, author used the first 75 samples as unknown, that means author make use of the prior information that half the query is unknown and the rest are known. However, in the real application, we don't know how many unknown and known class objects are in the query.

[liyangfan0]

作者你好！请教一个问题，在测试的时候论文是将前75个有着最大energy score的query作为unknow，但如果是测试一张query，如何判断它是unknow呢？很多论文是采用阈值来判断unknow，而这里直接用前75个作为unknow，实际上引入了先验信息，这种测试方式我很疑惑到底合不合理。

I have the same question for the author. During the evaluation, author used the first 75 samples as unknown, that means author make use of the prior information that half the query is unknown and the rest are known. However, in the real application, we don't know how many unknown and known class objects are in the query.

I agree with you. I have given up following this paper because I think it is unreasonable and unrealistic.

[LoveMiki]

作者你好！请教一个问题，在测试的时候论文是将前75个有着最大energy score的query作为unknow，但如果是测试一张query，如何判断它是unknow呢？很多论文是采用阈值来判断unknow，而这里直接用前75个作为unknow，实际上引入了先验信息，这种测试方式我很疑惑到底合不合理。

I have the same question for the author. During the evaluation, author used the first 75 samples as unknown, that means author make use of the prior information that half the query is unknown and the rest are known. However, in the real application, we don't know how many unknown and known class objects are in the query.

Agreed. I also found that the author used the meta-trained model of FEAT to further meta-train GEL, which is not reasonable and unfair to other methods.

[00why00]

Sorry for the late response. I'll answer your questions one by one.

1. How to judge unknown samples during the evaluation
In order to ensure the fairness of the comparison, our judgment method is consistent with SnaTCHer, TANE, RFDNet et al., that is, TPR and FPR are calculated by the open label of each sample to calculate the AUROC.

Ours:

Glocal/model/utils/train_utils.py

Lines 70 to 80 in db161db

	def calc_auroc(known_scores, unknown_scores):
	y_true = np.array([0] * len(known_scores) + [1] * len(unknown_scores))
	y_score = np.concatenate([known_scores, unknown_scores])
	y_pred = np.where(y_score >= np.sort(y_score)[75], 1, 0)
	fpr, tpr, thresholds = roc_curve(y_true, y_score)
	fpr95 = float(interpolate.interp1d(tpr, fpr)(0.95))
	auc_pr = average_precision_score(y_true, y_score)
	auc_score = roc_auc_score(y_true, y_score)
	f_score = f1_score(y_true, y_pred, average="binary")
	return auc_score, fpr95, auc_pr, f_score

SnaTCHer:
https://github.com/MinkiJ/SnaTCHer/blob/335c42469f0a7ad72c5c3480c8effc8c293823e0/model/trainer/fsl_trainer_SnaTCHerF.py#L17-L23

TANE:
https://github.com/shiyuanh/TANE/blob/fe089d477ba9387dd2d008d8872cdc84f8ad77ed/trainer/FSEval.py#L53-L73

RFDNet:
https://github.com/shule-deng/RFDNet/blob/bc03309447b9013415563fdc6f9a938a0443d525/train.py#L206-L209

2. Why use the meta-trained model of FEAT to further meta-train GEL
For a fair comparison, we trained the model in the same way as our baseline SnaTCHer, i.e. use the meta-trained model of FEAT to further meta-train our model.