token routing

Topics

• mixture of experts

In MoEs, a learned gating network (G) is used to generate routing scores for all experts, followed by:

1. selecting the experts (Top-K or noisy Top-K)
2. normalization of scores for those selected experts ⎯ softmax commonly used
3. combine their outputs using these normalized weights

Warning

Steps 1. and 2. aren’t exactly followed as is. Implementations and their order vary.

The last step is mathematically represented as,

$$y=\sum _{i=1}^{n}G(x{)}_i{E}_i(x)$$

G is typically a simple network with weights $W_g$. One can choose to selec

$$\begin{array}{rl}G(x)& =\mathrm{Softmax}(x\cdot W_g)\\ G(x)& =\mathrm{Softmax}(\mathrm{KeepTopK}(H(x))\end{array}$$

Noisy Top-k Gating introduces some (tunable) noise and then keeps the top k values.

Question

Why not just select the top expert?
The initial conjecture was that routing to more than one expert was needed to have the gate learn how to route to different experts, so at least two experts had to be picked.

Related