cross-entropy loss formula

Topics

• loss functions
• cross-entropy

$$\begin{array}{rl}{\mathcal{L}}_{\text{CE}}& =-\sum _{i=1}^N\sum _{c=1}^C{y}_{i,c}\log ({\hat{y}}_{i,c})\\ \text{where:}& \\ {\mathcal{L}}_{\text{CE}}& =\text{Cross-Entropy Loss}\\ N& =\text{Number of samples}\\ C& =\text{Number of classes}\\ y_{i,c}& =\text{True probability for class c of the i-th sample}\\ {\hat{y}}_{i,c}& =\text{Predicted probability for class c of the i-th sample}\end{array}$$

The outer summation over the N samples (or mini-batch) can either be sum (like above) or average. This is governed by the param reduction in PyTorch. This formula can be used for multi-class classification as well as multi-label classification, but in Pytorch, nn.CrossEntropyLoss is used for multi-class since it applies softmax internally, while nn.BCEWithLogitsLoss is used for mult-label since it applies sigmoid internally.

Related