Skip to content

Commit

Permalink
smooth term weight (#3510)
Browse files Browse the repository at this point in the history
### What problem does this PR solve?

#3499

### Type of change

- [x] Performance Improvement
  • Loading branch information
KevinHuSh authored Nov 20, 2024
1 parent 36e75b3 commit e16b7c5
Showing 1 changed file with 2 additions and 2 deletions.
4 changes: 2 additions & 2 deletions rag/nlp/term_weight.py
Original file line number Diff line number Diff line change
Expand Up @@ -228,7 +228,7 @@ def idf(s, N): return math.log10(10 + ((N - s + 0.5) / (s + 0.5)))
idf2 = np.array([idf(df(t), 1000000000) for t in tks])
wts = (0.3 * idf1 + 0.7 * idf2) * \
np.array([ner(t) * postag(t) for t in tks])
wts = [math.exp(s) for s in wts]
wts = [math.pow(s, 2) for s in wts]
tw = list(zip(tks, wts))
else:
for tk in tks:
Expand All @@ -237,7 +237,7 @@ def idf(s, N): return math.log10(10 + ((N - s + 0.5) / (s + 0.5)))
idf2 = np.array([idf(df(t), 1000000000) for t in tt])
wts = (0.3 * idf1 + 0.7 * idf2) * \
np.array([ner(t) * postag(t) for t in tt])
wts = [math.exp(s) for s in wts]
wts = [math.pow(s, 2) for s in wts]
tw.extend(zip(tt, wts))

S = np.sum([s for _, s in tw])
Expand Down

0 comments on commit e16b7c5

Please sign in to comment.