Assignment 3, Part 1: BERT Loss Model

Welcome to the part 1 of testing the models for this week's assignment. We will perform decoding using the BERT Loss model. In this notebook we'll use an input, mask (hide) random word(s) in it and see how well we get the "Target" answer(s).

IMPORTANT

• As you cannot save the changes you make to this colab, you have to make a copy of this notebook in your own drive and run that. You can do so by going to File -> Save a copy in Drive. Close this colab and open the copy which you have made in your own drive.

• Go to this google drive folder named NLP C4 W3 Colabs & Data. In the folder, next to its name use the drop down menu to select "Add shortcut to Drive" -> "My Drive" and then press ADD SHORTCUT. This should add a shortcut to the folder NLP C4 W3 Colabs & Data within your own google drive. Please make sure this happens, as you'll be reading the data for this notebook from this folder.

• Make sure your runtime is GPU (not CPU or TPU). And if it is an option, make sure you are using Python 3. You can select these settings by going to Runtime -> Change runtime type -> Select the above mentioned settings and then press SAVE

Note: Restarting the runtime maybe required.

Colab will tell you if the restarting is necessary -- you can do this from the:

Runtime > Restart Runtime

option in the dropdown.

Outline

• Part 0: Downloading and loading dependencies
• Part 1: Mounting your drive for data accessibility
• Part 2: Getting things ready
• Part 3: Part 3: BERT Loss
  • 3.1 Decoding

Part 0: Downloading and loading dependencies

Uncomment the code cell below and run it to download some dependencies that you will need. You need to download them once every time you open the colab. You can ignore the kfac error.