fixed-length context vector bottleneck

Topics

• encoder-decoder architecture
• neural information retrieval

Traditional encoder-decoder architecture for seq2seq modeling compresses entire input sequence $X=(x_1,...,x_{T_x})$ into single fixed-length context vector $c$.

Encoder processes input sequence, produces final hidden state $h_{T_x}$. This $h_{T_x}$ serves as context vector $c$:
$c=h_{T_x}=\text{Encoder}(x_1,...,x_{T_x})$
Decoder uses fixed vector $c$ as initial state or conditioning input at each time step to generate output $Y=(y_1,...,y_{T_y})$.

Decoder equations:

$s_t=f(s_{t-1},y_{t-1},c)$
$y_t=g(s_t,c)$

$s_t$ is decoder hidden state at time $t$. $f$ is decoder RNN cell. $g$ is output layer.

Problems with fixed-length context vector:

• Information Loss: Mapping variable-length input to fixed vector limits information retention. Capacity of $c$ limited, especially for long inputs $T_x$
• Difficulty with Long Sequences: Performance degrades as input length $T_x$ grows. Fixed $c$ struggles to encode all relevant information for long output sequence generation. Decoder relies on single compressed representation for all output steps
• Inability to Focus: Fixed context vector provides global input summary. Does not allow decoder to focus on specific input parts most relevant for current output generation. All input parts treated equally during compression

The attention mechanism addresses this bottleneck. Introduced in models like bahdanau attention, luong attention and later “widespread” via the transformer. Allows decoder to compute context vector $c_t$ at each decoding step $t$.

$c_t$ is weighted sum of encoder hidden states $h_1,...,h_{T_x}$:

$c_t={\sum }_{i=1}^{T_x}{\alpha }_{ti}{h}_i$

Weights ${\alpha }_{ti}$ are dynamically computed based on relevance of each encoder state $h_i$ to current decoder hidden state $s_{t-1}$. This allows decoder to focus on most pertinent input information for each output token, mitigating information bottleneck.