Algorithm logic flow example calculation

[365sma11]

Would this be an accurate flow for the calculations?
If the hierarchy is 1. Attribute b)trait
In the example below I have walked through the calculation for 1 trait.

AC= Attribute count // the number of attributes in a collection
TC= Trait count // the number of traits within an Attribute
STC= Sum of all Trait counts
TO= Trait occurance // the number of times the trait occurs
TK= Total number of Tokens
NA= Normalization of attributes
P1= Probability of trait occurance
P2= P1 normalized
RS=Rarity Score
OR1=OpenRarity trait score
OR2=OpenRarity normalized trait score

SAMPLE CALCULATION

The WilderWorld collection Wilder Beasts has 11 attributes and 3333 NFT’s. It’s Eye Accessory attribute has 13 traits. The Ski Goggles trait occurs 145 times. Following calculates the Rarity Score for this trait.

NORMALIZATION

AC= 11 // there are 11 attributes
STC=114 //the sum of all the traits in all the attributes
NA= STC/AC
NA= 114/11=10.364

P1=1/(TO/TK)
P1=1/(145/3333) // this trait occurs 145 times in this collection of 3333 nfts
P1=22.986

If P1 was to be normalized
TC= 13 // The number of traits in this attribute
P2= P1 x (NA/TC)
P2= 22.986 X (10.364/13)
P2=18.325

Popular rarity calcs for this trait would have the RS=P1 or if normalized RS=P2. The RS for each individual trait that the NFT held would be added together to get the final RS.

OpenRarity Calc

OR1= LOG2 (P1)
OR1= 4.523

OR2=LOG2 (P2)
OR2=4.192

Rarity score for this trait
RS= OR1/OR2
RS= 4.523/4.192
RS= 1.0790

In OpenRarity calc, the RS for each individual trait that the NFT held would be added together to get the final RS.

[365sma11]

@impreso would this be accurate?

[impreso]

Almost:
OR2 normalization factor called information entropy https://en.wikipedia.org/wiki/Entropy_(information_theory) and this is normalization factor for the whole collection. Each NFT consist ofTraits and Null Traits in the OpenRarity model.
Probability of Trait computed as trait_count/total_supply and Null Trait probability ( basically probability if trait is missing is total_supply-sum(trait_count_for_all_values)/total_supply. The sum of Trait and NullTrait log gives you the final Information Content measure (information in bits).

[365sma11]

2 Questtions

1. If I programically added a trait "NULL" to any nft that was missing the attribute, I would not need to use any other calculation other than my trait_count/total supply because the missing trait is now accounted for in my dataset. Correct?
2. If I am understanding you correct you are normalizing the score of the probability of the combination of the traits in one NFT. The calculation then becomes:
   OR2= OR1 x (Sum of all the OR1/the total NFT's (3333 in this case))
   Correct?

[impreso]

For 1. I don't think we are adding any trait new traits here as we don't believe in meta-traits :) , just accounting for the probability of the event ( trait not happening). you could. But overall i think you got the idea.
For 2. I'm not following TBH. We are not really normalizing any traits. So really you just need to $\sum Log_2(Ptrait_i)$ and divide it by entropy of the whole collection. https://openrarity.gitbook.io/developers/fundamentals/methodology

[365sma11]

#2 Yes I think we are on the same page,
Rarity Score (RS) for each NFT= I(x)/ E[I(x)] or OR1/OR2

1.Process, find OR1 for all NFT's first
OR1= SUM of LOG2(P1) for all traits in the NFT

2.For E[I(x)] (which labeled OR2 above)
"is the “expected value”, which is a weighted average of the information of all the NFTs in the collection ... We include it because it normalizes the scores for collections"

OR2= Sum of all OR1 in the collection/ TOTAL NFT'S (3333 in our example)
= weighted average of the information of all the NFTs in the collection

3.RS for each NFT can now be calculated= OR1/OR2

Do I have this correct?
I very much appreciate this, thanks!

[impreso]

OR2 formula (or entropy) is the following - $-\sum P_{attr_i}*log2(P_{attr_i})$ , we sum over all information content of each trait weighted by it's probability. OR2 factor is just for normalizing purposes.

[erzich]

Normalizing as you call it totally changes the calculations for what is actually more rare. You have to think about this logically. If a trait is less common, it is more rare. Anything that modifies that reduces the accuracy of this thing. Therefore you have sabotaged your own algorithm.

[impreso]

Normalization with entropy is constant factor and will never change the rarity rank within single collection.

[erzich]

Adjusting for the number of traits in a category to inflate or deflate actual rarity of a trait completely skews the math and places totally ruins the algorithm. You don’t need to do that, and it reduces the transparency. Non math geniuses should be able to asses what is more rare, but this adjustment to “normalize” totally skews everything.

[bmm-brobbins]

Hey erzich,

I'm happy to report that the whole "normalization" thing does NOT affect rarity scoring or token ranking within a collection at all.

Because I agree with your comment, I wanted to see what, if any, difference it makes when the normalization is included or not.

So I spent a few hours implementing the openRarity algo, from scratch, in a different language. I am able to get a match with Opensea's published rankings for the five collections I tested without implementing any normalization at all.

What the openRarity project calls "normalization" is only applicable when comparing the total information content between different collections. I don't know the use-case for that. (I'd be interested if anyone cares to explain.)

It's just unfortunate that the project uses the words "trait" and "normalization" in the same sentence or even paragraph, because "trait normalization" is an overloaded term used by the rarity sites, and even among those it means different things to different sites.

Adjusting for the number of traits in a category to inflate or deflate actual rarity of a trait completely skews the math and places totally ruins the algorithm. You don’t need to do that, and it reduces the transparency. Non math geniuses should be able to asses what is more rare, but this adjustment to “normalize” totally skews everything.