This guide outlines the procedure required to set up JupyterHub on Amazon Web Services (AWS), including setting up HTTPS, setting up a dynamic DNS service, and installing a number of Jupyter extensions.

In the context of this guide, Jupyter extensions were a requirement as the notebook server was to be used in a teaching environment, however this is a general guide and does not exclude other use cases.

The following will be performed step by step (and can be considered requirements):

• HTTPS: all connections to the server must be secure (users will be logging in and out, users may enter personal information, etc.)
• Notebook Extensions: In this use case the Jupyter notebooks are intended to be used in a teaching environment. Therefore, it is a requirement that some Jupyter extensions be installed at a system level. In other words, the extensions should be available to all users, both current and new!
• Cloud based: to avoid the overhead of needing to administer physical hardware, a cloud provider is used to host the service. Using a cloud based server means it can easily be scaled up to handle larger numbers of users.
• Static URL: the server should be accessible via a URL and not via an IP address or similar. Most cloud providers give access to your servers using non-static IP addresses, hence a dynamic DNS service will be used (preferably free)
• Cheap: the server should be as cheap as possible, therefore a number of free resources are used- such as a free certificate provider (Let's Encrypt) and a free dynamic DNS provider (DuckDNS).

The guide is split into the following sections and subsections:

• Creating the Server
  • Creating an AWS Instance
  • Installing JupyterHub
  • Let's Encrypt
• Managing Users
  • Batch Creation of Users
• Add Ons

We begin by creating an Amazon Web Services (AWS) EC2 instance which will be our JupyterHub web server. An AWS EC2 instance is basically a virtual machine.

To create a JupyterHub server we will first create an Amazon EC2 instance (a virtual machine) and then install JupyterHub on this machine. These instructions pertain to Amazon Web Services (AWS) but most cloud hosting providers will work similarly.

This guide presumes the use of Ubuntu (specifically 14.04, however later versions should also work).

In the AWS dashboard, create a Ubuntu Server 14.04 LTS EC2 instance. You can begin with a t2.micro instance (which is free tier eligible) and work up from there depending on your requirements. Attach as much hard disk space as required. The hard disk space can also be expanded at a later date.

Once an instance has been created, you will be prompted to download the private key file (a .PEM file). Do this and store is somewhere safe. This will be used to log in to the server you just created instead of using a password. It cannot be created again so do not lose it.

Once the instance has been created, start it and you will get a public IP address and public DNS, which you can use to connect to your new instance via SSH. The DNS will look something like ec2-203-0-113-55.eu-central-1.compute.amazonaws.com which corresponds to the public IP of 203.0.113.55 (note that 203.0.113.0/24 is a reserved IP address range used for documentation purposes and will never resolve to anything).

To log in to the server, type the following into the command line (this assumes the you saved the private key as JupyterHub.pem:

ssh -i JupyterHub.pem  ubuntu@ec2-203-0-113-55.eu-central-1.compute.amazonaws.com

Do not forget the ubuntu@ before the DNS or IP. This is specific to Ubuntu and the instructions provided by AWS do not point this out.

Once you have logged in, you can start installing JupyterHub and its dependencies, which we will do in the next section.

JupyterHub requires Python 3. In many cases Python 2 is the default version that is pre-installed on many systems.

To set up a Python 3 environment:

1. Log in to your AWS instance using SSH, e.g. ssh -i JupyterHub.pem ubuntu@203.0.113.55
2. Install Python 3: sudo apt install python3. You can check if Python 3 is installed by using which python3
3. Install the pip3 package manager: sudo apt-get install python3-pip
4. Install IPython and Jupyter (ensure you use pip3 to install so that we get the Python 3 version): pip3 install ipython and pip3 install jupyter

Node.js is required for JupyterHub to work. Install it using

sudo apt-get install npm nodejs-legacy

The package nodejs-legacy is required as this installs the node executable and this is required for JupyterHub to function.

Once the prerequisites have been installed, we can install JupyterHub and the proxy server:

1. Log in to your AWS instance using SSH, e.g. ssh -i JupyterHub.pem ubuntu@203.0.113.55
2. Proxy server: npm install -g configurable-http-proxy
3. JupyterHub server: python3 -m pip install jupyterhub

You can now run JupyterHub to start the server:

jupyterhub

Your server should now be accessible under the public DNS address:

http://ec2-203-0-113-55.eu-central-1.compute.amazonaws.com

Now create a configuration file for JupyterHub in /etc/jupyterhub/

sudo jupyterhub --generate-config -f /etc/jupyterhub/jupyterhub_config.py

Later we will edit this to point to our HTTPS certificates, for example.

By default JupyterHub uses PAM for authentication. We will use PAM also in this guide. Scripts are provided in the Managing Users section for creating and deleting local users in bulk.

Running JupyterHub as a service means it will start when the AWS instance is started. Also it means you can restart, stop, or check the status of the server more easily.

First, create a file named jupyterhub with these contents https://gist.github.com/c70d6bad0652c62214ebc85b0d953113 and copy this file to /etc/init.d/

Make the file executable:

sudo chmod +x /etc/init.d/jupyterhub

And then run:

sudo update-rc.d jupyterhub defaults

To start and stop JupyterHub:

# Start JupyterHub:
$ sudo service jupyterhub start
# Stop JupyterHub:
$ sudo service jupyterhub stop

Each time you start your AWS instance and your JupyterHub server, you will get a new IP address and therefore a new domain/URL (for example http://ec2-203-0-113-55.eu-central-1.compute.amazonaws.com).

To overcome this we will use a dynamic DNS service to map your new IP to a static domain name. This is done automatically each time the IP address changes. The end result is that you will be able to access your server at a domain such as http://subdomain.duckdns.org.

For enabling HTTPS, a static domain is required. Hence if you require HTTPS you must also obtain a static domain.

We will use DuckDNS for this guide (https://www.duckdns.org/). DuckDNS is very easy to setup and does not require any software to be installed (except a very short shell script). It works using cron and therefore does not use any resources.

Follow the guide here https://www.duckdns.org/install.jsp to use DuckDNS.

Once you have a domain, we can create a certificate and enable HTTPS on the server.

An alternative to using dynamic DNS is to purchase a custom domain, and update the DNS servers with your server's IP address each time it changes.

Some service providers, such as Namecheap (see https://www.namecheap.com/) provide free dynamic DNS, which can be updated using a simple script, such as the following:

# -*- coding: utf-8 -*-
from urllib.request import urlopen

urlopen("https://dynamicdns.park-your-domain.com/update?host=%s&domain=%s&password=%s" % ("@", "example.com", "your-random-password"))

This can be set up to run periodically, or at startup using cron.

First, save the code above to a file and make the script executable:

chmod +x dynamic.py

And edit your crontab:

crontab -e

And add the following line to your crontab file:

*/5 * * * * python3 /home/ubuntu/namecheap.py

This will make it run every 5 minutes.

AWS instances are given a new IP address each time they are started, however they do not change while they are running. Therefore it is sufficient to run the script at start up, and this can also be done with cron. Again, after running crontab -e, add the following line to your cron file:

@reboot /usr/local/bin/namecheap

In this case, the namecheap.py script has been renamed to namecheap and placed in the /usr/local/bin/ directory.

The service Let's Encrypt provides free certificates that enable HTTPS on web servers. Here we describe how to enable HTTPS on your JupyterHub server using Let's Encrypt.

Also, a HTTPS certificate requires that the domain remains static. AWS and most other cloud providers do not give you a permanent DNS for free. Therefore, we combine the use of a dyanmic DNS and Let's Encrypt to provide a HTTPS certificate for a static domain.

Let's Encrypt certificates are valid for 90 days after they are generated. They must be renewed every 90 days or they become invalid. This can be done automatically using cron jobs or manually. Here we describe how to update the certificate manually.

First, ensure your AWS instance is accessible from any IP. This is required as Let's Encrypt must be able to connect to your machine to validate your domain (it does this by creating a temporary server of its own). Allowing access from all IPs can be temporarily allowed in the security settings for your instance.

The procedure for renewing your certificate is as follows:

1. Open HTTPS port 443 on EC2 to the public (certbot creates a temporary server to authenticate the domain)
2. Stop JupyterHub: sudo service jupyterhub stop
3. Run sudo certbot renew --dry-run --preferred-challenges http (this must succeed before continuing to the next step)
4. Run sudo certbot renew --preferred-challenges http
5. Start JupyterHub: sudo service jupyterhub start
6. Close access to IPs that you do not require

To check your certificate's expiration date, you can use a browser to view the server's certificate.

Note: Due to the deprecation of TLS-SNI-01 method of domain validation by Let's Encrypt, you need to use HTTP using the --preferred-challenges http parameter above. This may require that port 80 be opened during validation.

See https://community.letsencrypt.org/t/february-13-2019-end-of-life-for-all-tls-sni-01-validation-support/74209 for more information.

AWS allows you to define security rules (via a Security Group) that block or allow certain IP addresses or ranges. For example, you may want to block all IPs outside of your company or school campus. This means that any incoming connections simply cannot connect to your JupyterHub server or log in via SSH into the virtual machine.

Add-ons and extenions need to be handled slightly differently for a JupyterHub environment compared to a standard, locally run Jupyter notebook server. Here we will explain how this is done.

Here we desribe how to install ipywidgets.

Here we will describe how to install nbgrader to automatically grade students.

JupyterHub supports PAM user authentication, meaning users can log in to JupyterHub using their Linux credentials. Creating Linux users can be batch processed using commands such as useradd or newusers.

This section makes available a number of scripts that should facilitate the creation of users in batches. The scripts create local Linux accounts and users can log in using these same credentials into JupyterHub.

The Linux command newusers can be used to create users in bulk, however it crashes with larger files. The following script passes a list of users line by line to the command:

# Create batch users

import subprocess
import sys

try:
    f = open("./user-list.txt", "r")
except:
    print("File not found or unreadable.")
    sys.ext(1)

for line in f:
    subprocess.call("echo %s | newusers" % (line.strip()), shell=True)

f.close()

The following script can be used to delete users in bulk, using a list of users in the format described above:

import subprocess
import sys

try:
    f = open("./user-list.txt", "r")
except:
    print("File not found or unreadable.")
    sys.exit(1)

for line in f:
    user_name = str.split(line, ":")[0]
    subprocess.call("userdel -rf %s" % (user_name), shell=True)
    print("Removed %s." % (user_name))

f.close()

In this case, the Linux command userdel is used to delete each user within the user-list.txt file.

The Linux command rsync can be used to copy files to each user, while

import sys
import subprocess

try:
  f = open("./users-list.txt", "r")
except:
  print("File not found or unreadable.")
  sys.exit(1)

for line in f:
  # Get username
  user = str.split(line, ":")[0]
  # Call rsync for each user
  subprocess.call("rsync -rogv --delete --exclude=\".*\" --exclude=\".*/\" --chown=%s:student /home/ubuntu/notebooks/ /home/%s/" % (user, user), shell=True)

f.close()

These scripts all presume users are listed in the standard UNIX format:

s0000001:seminar::1002:John Smith,,,:/home/s0000001:/bin/bash
s0000002:seminar::1002:Joe Bloggs,,,:/home/s0000002:/bin/bash

Each line is in the form <Username>:<Password>:<UID>:<GID>:<User Info>:<Home Dir>:<Default Shell> where fields such as <UID> can be left blank (as the command useradd auto assigns this). The <User Info> field is comma separated, and includes the first name, address, room number, and so on. Most of these fields are often blank.

Therefore, the remainder of the scripts here expect that your users-list.txt is in this format.

Creating this list will depend on how you get your user data, but it can be written using a loop such as the following:

for index, student in students.iterrows():
    user_name = "s" + student["MATRICULATION_NUMBER"]
    f.write("%s:seminar::1002:%s %s,,,:/home/%s:/bin/bash\n" % (user_name, student["FIRST_NAME"], student["FAMILY_NAME"], user_name))

The following script can be used to archive the notebooks that were completed by a list of students (again found in users-list.txt):

import os
import sys
import subprocess

try:
    f = open("./users-list.txt", "r")
except:
    print("File not found or unreadable.")
    sys.exit(1)

for line in f:
    # Get username
    user = str.split(line, ":")[0]

    # Files to copy. Comment as appropriate.
    files_to_copy = []

    files_to_copy.append("Notebook1.ipynb")
    files_to_copy.append("Notebook2.ipynb")
    # And so on

    for file_to_copy in files_to_copy:
        # Call subprocess and perform the cp command, e.g.
        # cp /home/s0731621/Notebook1.ipynb /home/ubuntu/archived-notebooks/Notebook1-s0731621.ipynb
        subprocess.call("cp /home/%s/%s /home/ubuntu/archived-notebooks/%s-%s.ipynb" % (user, file_to_copy, file_to_copy, user), shell=True)

f.close()    

You can embed video and audio in Jupyter notebooks, and while there is no support within Markdown for embedding video, most HTML code will render within Jupyter cells. To embed video, the HTML 5 <video/> tag can be used:

To embded video in a Jupyter cell:

<video controls src="./videos/media1.mp4" />

to embed audio in a Jupyter cell:

<audio controls src="./audio-clips/podcast-media.mp3" />

Again, the controls attribute can toggled to enable or disable playback controls. Other options for both tags include loop and autoplay, among others.

Widgets allow you to add interactive elements to Jupyter notebooks. Here the installion of ipywidgets for all users in a multi-user JupyterHub server will be described. Also, a number of scripts for making simple interactive elements will be provided.