auto-regressive property

Topics

• sequence modelling

The auto-regressive property refers to a model’s ability to make predictions based on its own previous outputs or historical values. More specifically, an auto-regressive model predicts the current value based on past values in a sequence.

$$P(x_1,x_2,...,x_n)=P(x_1)\times P(x_3\mid x_1,x_2)\times \dots \times P(x_n\mid x_1,...,x_{n_{-}1})$$

Input: The cat sits on

Step 1: The cat sits on → the 
Step 2: The cat sits on the → mat

Common Applications

• Language Models (like GPT)
• Time Series Prediction
• Speech Generation
• Music Generation

This property is well suited for generative tasks, but can be slow (one output at a time) and suffer from error accumulation.

Related

• teacher forcing