BLEU score

Bilingual Evaluation Understudy (BLEU) is a metric for evaluating Machine translation, with value in range 0 to 1. The higher the BLEU score, the closer the computer-generated text is to the human-translated reference text. It uses a weighted sum of precisions of n-grams $p_n$ at its core along with a brevity penalty (BP) to penalize translations that are too short. The BLEU formula is:

$$\text{BLEU}=\text{BP}\cdot \exp \left(\sum _{n=1}^N{w}_n\log (p_n)\right)$$

Calculation steps:

• Choose the n-gram order: N=4 typically
• Count matching n-grams: For each n-gram size, calculate how many matching n-grams exist between the machine-generated text and the reference translations. Be sure to account for repeated phrases
• Calculate the precision of n-grams in BLEU i.e. $p_n$ values
• Do the weighted sum of log probs, followed by exponentiation, which is very similar to geometric mean average precision: $$\begin{array}{rl}& \exp \left(\sum _{n=1}^N{w}_n\log p_n\right)\\ & =\prod _{n=1}^N{p}_n^{w_n}\\ & =(p_1{)}^{\frac{1}{1}}\cdot (p_2{)}^{\frac{1}{2}}\cdot (p_3{)}^{\frac{1}{3}}\cdot (p_4{)}^{\frac{1}{4}}\end{array}$$
• Apply BP to get the final BLEU score

Related

• encoder-decoder architecture