Evaluation metric bits-per-byte

[jxiw]

Hi there,

Megabyte paper uses bits-per-byte in Table 2 as their evaluation metric. It seems it has difference compared with byte level perplexity, since their number in arXiv and Code is < 1. So it should not be perplexity. This repo uses the cross-entropy loss and can easily calculate the byte level perplexity. May I ask how to compute bits-per-byte metric?

Thanks a lot.

[eegli]

BPB can be computed from cross entropy loss (which is the model output) as:

def cc_to_bpb(cc_loss: float):
    return cc_loss * math.log2(math.e)