Recurrent Neural Networks

In the Neural Networks Lessons, we learned how to use Neural Networks to make predictions based on some input fixed size$x$. Sometimes are inputs are sequential and vary in length, we often call this time series data but the index doesn’t have to be time, it just means the variables have a fixed order: $$x_1,x_2,\dots ,x_{t-1},x_t$$

and we want to predict the next time step $x_{t+1}$, which can be a classification or a regression task as before.

We have two goals.

1. We want the model $x_{t+1}$ to be dependent on previous timesteps $x_t$
2. We want the model to work with input sequences of any length

We efficiently achieve this with an RNN Cell, depicted below. The cell’s output $y_t$ depends on the previous timesteps activation output $a_{t-1}$ and the current input $x_t$. It is basically two neurons in one.