Learn practical skills, build real-world projects, and advance your career

course-project-worn-out-tires-resnet

Training an image classifier from scratch to over 90% accuracy in less than 5 minutes on a single GPU

Part 6 of "PyTorch: Zero to GANs"

This post is the sixth in a series of tutorials on building deep learning models with PyTorch, an open source neural networks library. Check out the full series:

  1. PyTorch Basics: Tensors & Gradients
  2. Linear Regression & Gradient Descent
  3. Image Classfication using Logistic Regression
  4. Training Deep Neural Networks on a GPU
  5. Image Classification using Convolutional Neural Networks
  6. Data Augmentation, Regularization and ResNets
  7. Generating Images using Generative Adverserial Networks

This notebook is an extension to the tutorial Image Classification using CNNs in PyTorch, where we trained a deep convolutional neural network to classify images from the CIFAR10 dataset with around 75% accuracy. Here are some images from the dataset:

cifar10

In this tutorial, we'll use the following techniques to achieve over 90% accuracy in less than 5 minutes:

  • Data normalization
  • Data augmentation
  • Residual connections
  • Batch normalization
  • Learning rate scheduling
  • Weight Decay
  • Gradient clipping
  • Adam optimizer

System Setup

This notebook is hosted on Jovian.ml, a platform for sharing data science projects. If you want to follow along and run the code as you read, you can choose the "Run on Kaggle" option from the "Run" dropdown above. Remember select "GPU" as the accelerator and turn on internet from the sidebar within the Kaggle notebook.

Otherwise, to run the code on your machine, you can clone the notebook, install the required dependencies using conda, and start Jupyter by running the following commands:

pip install jovian --upgrade               # Install the jovian library 
jovian clone aakashns/05b-cifar10-resnet   # Download notebook & dependencies
cd 05b-cifar10-resnet                      # Enter the created directory 
conda create -n 05b-cifar10-resnet         # Create virtual env
conda activate 05b-cifar10-resnet          # Activate virtual env
conda install jupyter                      # Install Jupyter
jupyter notebook                           # Start Jupyter

For a more detailed explanation of the above steps, check out the System setup section in the first notebook. Before you start executing the code below, you may want to clear the cell outputs by selecting "Kernel > Restart and Clear Output" from the Jupyter notebook menu bar, to avoid confusion.

We begin by importing the required modules & libraries.

pip install jovian --upgrade 
WARNING: Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<pip._vendor.urllib3.connection.VerifiedHTTPSConnection object at 0x7f749cfe7790>: Failed to establish a new connection: [Errno -3] Temporary failure in name resolution')': /simple/jovian/ WARNING: Retrying (Retry(total=3, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<pip._vendor.urllib3.connection.VerifiedHTTPSConnection object at 0x7f749cf6ed90>: Failed to establish a new connection: [Errno -3] Temporary failure in name resolution')': /simple/jovian/ WARNING: Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<pip._vendor.urllib3.connection.VerifiedHTTPSConnection object at 0x7f749cf6eb50>: Failed to establish a new connection: [Errno -3] Temporary failure in name resolution')': /simple/jovian/ WARNING: Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<pip._vendor.urllib3.connection.VerifiedHTTPSConnection object at 0x7f749cf6ea50>: Failed to establish a new connection: [Errno -3] Temporary failure in name resolution')': /simple/jovian/ Collecting jovian Downloading jovian-0.2.16-py2.py3-none-any.whl (63 kB) |████████████████████████████████| 63 kB 1.1 MB/s eta 0:00:011 Requirement already satisfied, skipping upgrade: pyyaml in /opt/conda/lib/python3.7/site-packages (from jovian) (5.3.1) Requirement already satisfied, skipping upgrade: click in /opt/conda/lib/python3.7/site-packages (from jovian) (7.1.1) Requirement already satisfied, skipping upgrade: requests in /opt/conda/lib/python3.7/site-packages (from jovian) (2.23.0) Collecting uuid Downloading uuid-1.30.tar.gz (5.8 kB) Requirement already satisfied, skipping upgrade: certifi>=2017.4.17 in /opt/conda/lib/python3.7/site-packages (from requests->jovian) (2020.4.5.2) Requirement already satisfied, skipping upgrade: idna<3,>=2.5 in /opt/conda/lib/python3.7/site-packages (from requests->jovian) (2.9) Requirement already satisfied, skipping upgrade: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in /opt/conda/lib/python3.7/site-packages (from requests->jovian) (1.24.3) Requirement already satisfied, skipping upgrade: chardet<4,>=3.0.2 in /opt/conda/lib/python3.7/site-packages (from requests->jovian) (3.0.4) Building wheels for collected packages: uuid Building wheel for uuid (setup.py) ... done Created wheel for uuid: filename=uuid-1.30-py3-none-any.whl size=6500 sha256=bf84d56369870929116c87d0b7715bfdd831fe87e9d397ee4c765d8a46c32627 Stored in directory: /root/.cache/pip/wheels/2a/ea/87/dd57f1ecb4f0752f3e1dbf958ebf8b36d920d190425bcdc24d Successfully built uuid Installing collected packages: uuid, jovian Successfully installed jovian-0.2.16 uuid-1.30 Note: you may need to restart the kernel to use updated packages.
# Uncomment and run the commands below if imports fail
# !conda install numpy pandas pytorch torchvision cpuonly -c pytorch -y
# !pip install matplotlib --upgrade --quiet
import os
import torch
import torchvision
import tarfile
import torch.nn as nn
import numpy as np
import torch.nn.functional as F
from torchvision.datasets.utils import download_url
from torchvision.datasets import ImageFolder
from torch.utils.data import DataLoader
import torchvision.transforms as tt
from torch.utils.data import random_split
from torchvision.utils import make_grid
import matplotlib.pyplot as plt
%matplotlib inline

from fastai.vision import Path, download_images
from torch.utils.data import random_split, Dataset