perception classification rule

Topics

• binary classification
• classical machine learning

Perceptron is a simple binary classification algorithm. The decision rule can be illustrated with an example:

Give a person, we want to predict whether to approve credit or not. The person has some features related to his/her transaction history, credit score etc, given by $\mathbf{x}$ which is a d-dimensional feature vector. We have a weight vector $\mathbf{w}$ and some threshold $\gamma$, using which we can define some decision rule as:

$$\begin{array}{rl}\hat{y}& =\{\begin{array}{ll}1& \text{if }\mathbf{w}\cdot \mathbf{x}\ge \gamma \\ -1& \text{otherwise}\end{array}\\ & =\text{sign}(\sum _{i=1}^d{w}_i{x}_i-\gamma )\end{array}$$

where $\hat{y}=1$ implies approving credit for the person and $\hat{y}=-1$ implies rejection. Note that different choices of $\gamma$ and $\mathbf{w}$ yield different hypothesis which collectively form a hypothesis set.

Note

One can see that this is the same as weights $\mathbf{w}$ and biases $\gamma$ in neural net linear layers.

If we set $w_0=-\gamma$ and $x_0=1$ , we can simply the expression above as:

$$\begin{array}{rl}& =\text{sign}\left(\sum _{i=0}^d{w}_i{x}_i\right)\\ & =\text{sign}({\mathbf{w}}^{\top }\mathbf{x})\end{array}$$

which is basically the sign of the dot product between $\mathbf{w}$ and $\mathbf{x}$. Our goal would then be to find good values for $\mathbf{w}$ such that we get good (or even perfect) classification. That’s where we need to come up with a learning algorithm.

Related