why choose nn.Embedding over nn.Linear

Topics

• pytorch-internals
• word embeddings

nn.Embedding and nn.Linear essentially work the same but there are few reasons to prefer nn.Embedding for lookup:

• Implementation is efficient
• For nn.Linear, lot of wasteful multiplications by 0
• Memory acess pattern: For nn.Linear, entire weight matrix is loaded in memory during forward pass
• During backpopagation, entire weight matrix is updated (most updates are 0 though)question

Related