Learn practical skills, build real-world projects, and advance your career

Building your Recurrent Neural Network - Step by Step

Welcome to Course 5's first assignment! In this assignment, you will implement key components of a Recurrent Neural Network in numpy.

Recurrent Neural Networks (RNN) are very effective for Natural Language Processing and other sequence tasks because they have "memory". They can read inputs xtx^{\langle t \rangle} (such as words) one at a time, and remember some information/context through the hidden layer activations that get passed from one time-step to the next. This allows a unidirectional RNN to take information from the past to process later inputs. A bidirectional RNN can take context from both the past and the future.

Notation:

  • Superscript [l][l] denotes an object associated with the lthl^{th} layer.

  • Superscript (i)(i) denotes an object associated with the ithi^{th} example.

  • Superscript t\langle t \rangle denotes an object at the ttht^{th} time-step.

  • Subscript ii denotes the ithi^{th} entry of a vector.

Example:

  • a5(2)[3]<4>a^{(2)[3]<4>}_5 denotes the activation of the 2nd training example (2), 3rd layer [3], 4th time step <4>, and 5th entry in the vector.

Pre-requisites

  • We assume that you are already familiar with numpy.
  • To refresh your knowledge of numpy, you can review course 1 of this specialization "Neural Networks and Deep Learning".

Be careful when modifying the starter code

  • When working on graded functions, please remember to only modify the code that is between the
#### START CODE HERE

and

#### END CODE HERE
  • In particular, Be careful to not modify the first line of graded routines. These start with:
# GRADED FUNCTION: routine_name
  • The automatic grader (autograder) needs these to locate the function.
  • Even a change in spacing will cause issues with the autograder.
  • It will return 'failed' if these are modified or missing."

Updates for 3b

If you were working on the notebook before this update...
  • The current notebook is version "3b".
  • You can find your original work saved in the notebook with the previous version name ("v3a")
  • To view the file directory, go to the menu "File->Open", and this will open a new tab that shows the file directory.
List of updates
  • rnn_cell_backward
    • fixed error in equations
    • harmonize rnn backward diagram with rnn_forward diagram and fixed Wax multiple (changed from at to xt).
    • clarified dba batch as summing 'm' examples
    • aligned equations
  • lstm_cell_backward
    • aligned equations
  • lstm_forward
    • fixed typo, Wb to bf
  • lstm_cell_forward
    • changed c_next_tmp.shape to a_next_tmp.shape in test case
    • clarified dbxx batch as summing 'm' examples

Let's first import all the packages that you will need during this assignment.

import numpy as np
from rnn_utils import *

1 - Forward propagation for the basic Recurrent Neural Network

Later this week, you will generate music using an RNN. The basic RNN that you will implement has the structure below. In this example, Tx=TyT_x = T_y.