Tom Busby

Setting Up a ZNC IRC Bouncer to Use Tor

2019-08-04T23:47:00+00:00

Motivation

I’m at least partially writing this as an aid to my own memory. It took me a little while to muddle through and get everything set up. I recently had to do part of the process again and struggled to remember. Writing this blog post seems like a good way to help my future self and others looking to do the same thing.

I’m going to assume that you want your bouncer to connect to Freenode. The steps should be fairly similar for other IRC servers.

ZNC

Why use a bouncer at all? Couldn’t I just directly connect to the IRC server?

Yes, you could. The disadvantage is that you will not receive messages sent while you were away. For some channels you may not want to miss a single message. For others, you may just want the last 50 messages so that you have the context of the conversation that you just entered.

Another advantage of a bouncer is that we can also set up “client buffers”. These are separate buffers for the different clients that you may connect from. This ensures that if I connect from my desktop’s IRC client, then my laptop, the messages will be replayed correctly for both.

This is similar to the problem that IMAP email solves. Emails are synced accross devices these days. In the 90s, we used POP3. This pulled emails from the server; clearing the buffer in the process. This meant that (in the standard configuration), if you had two devices, your emails would be split accross them. If an email was pulled to your laptop then it wouldn’t be pulled to your desktop later.

Tor

When using IRC, we tend to leak quite a lot of information about the hostname we’re connecting from. This is somewhat undesirable if you’re connecting directly or self-hosting an IRC bouncer, since it may make you a target for people probing for vulnerabilities. Generally speaking, revealing your hostname and/or IP address publicly when engaging in an online community is best avoided if possible.

Setting Up Your Freenode Account

Firstly, we need to set up a Freenode account. We will be directly connecting to Freenode from the IRC client (without using a bouncer) for this step. The ultra-paranoid may want to connect to a VPN (or perhaps take their laptop to a coffee shop) before doing this step. This is to prevent your true origin IP ever being leaked.

We will be using Weechat as our IRC client since it’s free/open-source and cross-platform.

Note: If you’re using a different IRC client, this tutorial will still be fairly applicable to you but you’ll need to look up how to achieve the same things with your client. This page may be of assistance. If you use Textual, this page helped me get set up.

To install on OSX (with brew):

$ brew install weechat

To install on a Debian-based Linux (such as Ubuntu):

$ sudo apt update && sudo apt install -y weechat

For other OSes, either install from source or look up weechat in your system’s package manager.

Next launch weechat by entering weechat at the terminal.

The program will launch and you should see something like this:

You’ll notice that you have a prompt at the bottom of the terminal window. We’re going to use that to set up our connection with freenode:

/server add freenode-direct chat.freenode.net/6697 -ssl
/set irc.server.freenode-direct.nicks "zncblogpost,_zncblogpost"

Note: if you don’t want to manually connect each time time you run weechat, you can set the -autoconnect flag when adding the server config. For the purposes of this tutorial, I recommend leaving autoconnect off for any connections we create until everything is set up.

After executing these two commands you should see similar output to the following:

You can specify multiple comma-separated names. These will be tried in order and your nick will be the first name in the list that isn’t already in use.

We connect via the following command:

/connect freenode-direct

You should see a successful connect operation signified by a lot of text appearing in the buffer welcoming you to freenode.

At this point there’s nothing stopping you from joining a channel and chatting. You’ll notice though that many channels redirect you to some other channel:

/join #ubuntu

If you do this, you’ll see you’re actually redirected to #ubuntu-unregged. The reason why is explained to you upon joining the channel:

You can follow the instructions here (use Ctrl+N and Ctrl+P to navigate forward and back through your channel buffers to freenode to see the output of executing the /msg nickserv help register command) but I’ll summarise it here so that isn’t necessary.

Registration essentially marks a given nick as yours. This is analagous to signing up for an account with a given username on other services. We do this in the following way (your password will show up in the prompt as asterixes):

/msg NickServ REGISTER qwerty123 tom+zncblogpost@busby.ninja

Note: I changed this password prior to making this blog post public :P

When you check your inbox you’ll find an email from freenode with further instructions:

In order to complete your account registration, you must type the following
command on IRC:

/msg NickServ VERIFY REGISTER zncblogpost mzubwubihvmn

After running this you should see something like the following in your freenode buffer:

If you now try to join #ubuntu you’ll see that it connects normally and you can chat freely:

However, currently, you’ll need to run /msg NickServ identify <password> every time you connect. We can prevent the need to do this with SASL.

To start with, we’ll set this up using the PLAIN method which takes your username/password but later on we’ll be authenticating with an SSL certificate. Instructions for this section from this page.

/set irc.server.freenode-direct.sasl_mechanism PLAIN
/set irc.server.freenode-direct.sasl_username zncblogpost
/set irc.server.freenode-direct.sasl_password qwerty123
/save

If we /quit and then open up weechat again we’ll see that when we connect (/connect freenode-direct), it will authenticate with the freenode network automatically:

...
19:51:06 freenode-direct  -- | SASL authentication successful
...

We have now set up our connection with freenode and registered our nick. If we didn’t care about using Tor or a bouncer we could stop here.

Installing Your ZNC Bouncer

I will assume that you already have SSH access to a suitable server. Getting an externally-accessible server set up is outside the scope of this tutorial. There are many other tutorials which cover this for the various hosting providers. I’ll be using a fresh Amazon AWS instance in the following section and will loosely follow instructions from this page.

Firstly, let’s install znc:

$ sudo apt update && sudo apt install -y znc

Next item on the agenda is to install Tor:

Note: Further details about this process here. I’m using Ubuntu 18.04 LTS. For other OSs, see the linked page for instructions.

$ sudo su -
⌗ printf "deb https://deb.torproject.org/torproject.org bionic main\ndeb-src https://deb.torproject.org/torproject.org bionic main\n" > /etc/apt/sources.list.d/tor.list
⌗ curl https://deb.torproject.org/torproject.org/A3C4F0F979CAA22CDBA8F512EE8CBC9E886DDD89.asc | gpg --import
⌗ gpg --export A3C4F0F979CAA22CDBA8F512EE8CBC9E886DDD89 | apt-key add -
⌗ logout
$ cat /etc/apt/sources.list.d/tor.list
deb https://deb.torproject.org/torproject.org bionic main
deb-src https://deb.torproject.org/torproject.org bionic main
$ sudo apt update && sudo apt install -y tor deb.torproject.org-keyring

Next we need to install proxy chains and bind the freenode onion address to a local IP

$ sudo apt install -y proxychains
$ printf "mapaddress 10.99.99.90 freenodeok2gncmy.onion\n" | sudo tee -a /etc/tor/torrc
$ sudo systemctl daemon-reload
$ sudo systemctl restart tor

Now that Tor is installed we can look into getting ZNC running and connecting to freenode via Tor. We want ZNC to run as a system daemon so we’ll be following the instuctions on this page. If your server OS does not use systemd, consult the page I just mentioned. Your instructions will differ slightly.

$ sudo useradd --create-home -d /var/lib/znc --system --shell /sbin/nologin --comment "Account to run ZNC daemon" --user-group znc

We now need to create the skeleton config for the server:

$ sudo -u znc /usr/bin/znc --datadir=/var/lib/znc --makeconf

Note: if you want, you can enable SSL. However, the cert won’t be signed by a CA so you’ll receive a warning about this from your browser when you navigate to the page. See this footnote for notes about securing the connection to the web admin.

Now create the systemd unit:

/etc/systemd/system/znc.service

[Unit]
Description=ZNC, an advanced IRC bouncer
After=network-online.target

[Service]
ExecStart=/usr/bin/znc -f --datadir=/var/lib/znc
User=znc

[Install]
WantedBy=multi-user.target

It’s time to start the service.

$ sudo systemctl enable znc.service
Created symlink /etc/systemd/system/multi-user.target.wants/znc.service → /etc/systemd/system/znc.service.
$ sudo systemctl start znc.service

Now we’ve done that we can go to znc’s web interface at <host IP/DNS Name>:<Port we chose above>. For me this is http://52.16.112.138:8000/.

Note: Those of you following along using an AWS instance, don’t forget to allow incoming connections on any exposed ports via the security group.

Next we need to modify our systemd unit so that ZNC is loaded via proxychains:

/etc/systemd/system/znc.service

[Unit]
Description=ZNC, an advanced IRC bouncer
After=network-online.target

[Service]
ExecStart=/usr/bin/proxychains /usr/bin/znc -f --datadir=/var/lib/znc
User=znc

[Install]
WantedBy=multi-user.target

Now restart znc:

$ sudo systemctl daemon-reload
$ sudo systemctl restart znc.service

Configuring Your ZNC Bouncer to Connect to Freenode

After this we’ll set up freenode in ZNC. Go to “Your settings” and then click “Add” in the Networks section:

Note: the IP is the mapped local IP address that we set up in the torrc file.

At this point I also set up the DNS so that our server is available at blogpost.busby.ninja.

Now we need to follow the instructions (taken from this page) to connect to our ZNC bouncer from weechat.

On our server, we need to get the SSL cert fingerprint:

$ sudo -u znc cat /var/lib/znc/znc.pem | openssl x509 -sha512 -fingerprint -noout | tr -d ':' | tr 'A-Z' 'a-z' | cut -d = -f 2
162f9c92f80b048efb1ec06d4d04edab9fb7c399c36a2879a23994197970c8fbcefbe264401ea626b600b01e04bf99a4b483cdfd16ca4246de7958bad54bda11

While inside weechat (replacing the relevant details with your own):

/server add freenode-znc blogpost.busby.ninja/6697 -ssl -username=tom/freenode -password=same_as_webadmin_password123
/set irc.server.freenode-znc.ssl_fingerprint 162f9c92f80b048efb1ec06d4d04edab9fb7c399c36a2879a23994197970c8fbcefbe264401ea626b600b01e04bf99a4b483cdfd16ca4246de7958bad54bda11
/save
/connect freenode-znc

You should see something like this in the buffer:

A short while later, you’ll notice another buffer pop up in the left hand bar called *status. Use Ctrl+N to navigate to that buffer.

You’ll see a lot of information about an SSL cert ending with something like this:

This is a message from ZNC asking us to approve the SSL cert that freenode is presenting to ZNC as it attempts to connect on our behalf.

By default, weechat won’t transmit commands it doesn’t recognise, so we need to whitelist /znc:

/alias add znc /quote znc

We will now follow the instructions and trust this fingerprint:

/znc AddTrustedServerFingerprint e9:db:e1:0c:79:66:a8:da:0c:84:18:bc:81:59:6c:87:b0:25:7f:b4:80:2d:94:25:d6:40:2c:dc:96:03:51:c8

The SASL access only error relates to what is mentioned in the freenode tor blogpost which is that, when connecting to freenode via Tor, you must authenticate via a more secure means than PLAIN. We’ll set this up next. For now, let’s disconnect from ZNC:

/disconnect -all

To get set up with certificate-based SASL authentication, we’ll be following instructions from this page and this page.

On the server execute the following commands (replacing zncblogpost with your own nick):

$ openssl req -nodes -newkey rsa:4096 -keyout user.pem -x509 -days 3650 -out user.pem -subj "/CN=zncblogpost"
...
writing new private key to 'user.pem'
...
$ openssl x509 -sha1 -noout -fingerprint -in user.pem | sed -e 's/^.*=//;s/://g;y/ABCDEF/abcdef/'
c745ff09c1c5e11fef6fd8d9249124ce49a80194

Now that we’ve acquired a certificate and calculated its fingerprint, let’s move it to where ZNC cert module can find it (tom and freenode will be replaced with your ZNC username and ZNC network name):

$ sudo chown znc:znc user.pem
$ sudo mv user.pem /var/lib/znc/users/tom/networks/freenode/moddata/cert/

Next, we need to inform freenode’s NickServ of our certificate’s fingerprint. Because we don’t yet have a working ZNC connection, we’re going to reuse our old freenode-direct connection in weechat:

/connect freenode-direct

Once connected we’ll inform NickServ of our fingerprint:

/msg NickServ cert add c745ff09c1c5e11fef6fd8d9249124ce49a80194

We can double check that things worked correctly:

/query NickServ cert list

That’s all we need to do here, so let’s disconnect and reconnect to znc:

/disconnect -all
/connect freenode-znc

Note: you can navigate between open buffers with Ctrl+P/Ctrl+N and /close the old unused ones.

Let’s check in with the cert module to make sure it can see our certificate:

/query *cert info

If you have a different output than above then your user.pem file is not in the right directory or maybe has the wrong permissions.

Next we need to tell ZNC’s sasl module to use our certificate:

/query *sasl Mechanism
/query *sasl Mechanism EXTERNAL

We can now navigate to our *status buffer and watch it connect:

15:25:16 *status | Connected!

And now, the part we’ve all been waiting for… Let’s actually connect to a channel:

/join #ubuntu

Congrats, you’re up and running. The next section will deal with some recommended configuration and how to sync with multiple devices via ClientBuffer.

Futher Configuration

Enabling Colours for ZNC Playback

To get this working, we’ll be using an unofficial script called zncplayback.py. This is hosted here if you’d like to read the source. All that’s required in order to install it is the following:

/script install zncplayback.py

Auto-sort Channel Buffers by Name

When you are a member of many channels, keeping track of them can be a pain. It is possible to manually sort them, but it’s much easier to let a script do it for you. we’ll be using an unofficial script called autosort.py. This is hosted here if you’d like to read the source.

/script install autosort.py

Hide Join/Leave Messages

As per this blog post, removing all the annoying join-message spam is very simple:

/set irc.look.smart_filter on
/filter add irc_smart * irc_smart_filter *

Before and after:

Enable the Mouse

I have mixed feelings about whether this is a good idea. If you’re interested in being able to use the mouse then I recommend reading the manual. Using the mouse lets you click on channel names and scroll in buffers with the mouse wheel. I suspect that just learning the keyboard shortcuts well, and setting up your own, is more productive long term.

Setting Up the ClientBuffer Module

If you’d like to connect to ZNC from mutiple devices, this section is necessary. If you only ever intend to connect from one device, it is not.

As described in the introduction, if you use the same ZNC playback buffer for both devices, the replayed messages will be split across the devices. The ClientBuffer module solves this problem by having ZNC maintain a separate playback buffers for each.

We will be following instructions on this page to install the module and this page to configure the module. The source code for the module is hosted here.

Firstly, it’s nessesary to acquire the znc-buildmod tool:

$ sudo apt install -y znc-dev
$ znc-buildmod
[ ** ] USAGE: /usr/bin/znc-buildmod <file.cpp> [file.cpp ... ]

Next we have to get hold of the ClientBuffer source:

Note: If you don’t have git on the server and don’t want to install it, you can get a zip from the repository’s “Releases” page.

$ cd ~
$ git clone https://github.com/CyberShadow/znc-clientbuffer.git
$ cd znc-clientbuffer
$ znc-buildmod clientbuffer.cpp
Building "clientbuffer.so" for ZNC 1.6.6... [ ok ]
$ sudo chown znc:znc clientbuffer.so
$ sudo mv clientbuffer.so /usr/lib/znc/
$ sudo systemctl restart znc

Note: I don’t believe the service restart is necessary, but better safe than sorry.

With the module built and installed, we need to enable the module in our Network’s settings:

Log in to the web admin and navigate to the “Your settings” page. Next, scroll down to the networks section and edit your network. Enable the module from its list of available modules:

Now that the module is enabled, it’s time to add a client:

/query *clientbuffer Help
/query *clientbuffer ListClients
/query *clientbuffer AddClient mac
/query *clientbuffer ListClients

In order to tell ZNC which client we want to use, we need to include it in our usename upon connection:

/disconnect -all
/set irc.server.freenode-znc.username "tom@mac/freenode"
/connect freenode-znc
/query *clientbuffer ListClients

For any other devices, all we need to do is create a new client (using AddClient) like we did above and include that with our ZNC username upon connection. That way messages will be correctly replayed on both devices. Don’t use the same client name for multiple connections or you’ll run back into the “messages split across devices” issue.

Conclusion and Clean-up

If you installed ClientBuffer, I recommend leaving your working dir where you have access to it. You may need to recompile the module in future.

Removing our direct connection that we used to get the cerificate set up is likely desirable at this point:

/server del freenode-direct

We may also want to set up our ZNC-based connection to autoconnect when weechat is opened:

/set irc.server.freenode-znc.autoconnect on

Is the connection between weechat and ZNC secure?

I’ve heard talk of people making ZNC itself also a tor hidden service. I personally don’t see the point. A tor hidden service is meant to protect the client and server from revealing their true source IP to one-another (and also to hide who is communicating with whom from global observers - Hello Five Eyes!). In this case, we are both client and server so we’d be protecting ourselves from ourselves.

It’s sufficient to encrypt the traffic between our IRC client and ZNC. We may not have a certificate signed by a certificate authority, but given that ZNC generated the certificate itself and we added the fingerprint to weechat’s config, we can be pretty sure we aren’t under a man-in-the-middle attack while communicating with our bouncer. If you don’t trust ZNC’s autogenerated cert, you can replace it with your own without much issue. All a global observer sees is encrypted traffic between one IP and another.

Currently though, we’re serving the web admin over plaintext HTTP. That isn’t ideal. If you would like to encrypt that traffic, you have a couple of options:

The first is to just enable SSL for the web admin port (via the web admin). If your’re using the web admin to do this, you may have to create a new HTTP port temporarily. This is because the web admin won’t allow you to alter the port you’re currently connected via. Next time you navigate to the page via HTTPS, you’ll get a warning from your browser. You can check that the fingerprint is what you expect and tell the browser to ignore the warning.

The second option involves using Let’s Encrypt to get a valid, signed certificate from a trusted authority. If you’re interested in that approach, you should check out my other blog post on the subject or read about signed SSL certificates on the ZNC wiki.

If we use the method I wrote about, we can also replace our generated certificate for the IRC connection with the new Let’s Encrypt certificate too. In this instance, you should unset ssl_fingerprint in your weechat server config. When using a signed certificate, you can rely on the Public Key Infrastructure to validate it. The frequently-renewing nature of Let’s Encrypt certificates means that not removing the fingerprint will cause you problems when the certificate is renewed (because the fingerprint will no longer match). I may write this up in future.

Cypherpunks Mailing List Archive - Archive of posts to the list during its 90s heyday. Soon to be expanded to include the 2000-2016 currently-unarchived “dark ages” of the list.

2018-07-11T21:42:00+00:00

cryptoanarchy.wiki - An encyclopedia of cypherpunk thought, people and events.

2018-07-11T21:30:00+00:00

Understanding Cryptography by Christof Paar and Jan Pelzl - All Problems and Solutions

2017-12-10T00:50:00+00:00

Useful links

Official Website for “Understanding Cryptography” - crypto-textbook.com
University Lecture Series Covering the Material from the Book - taught by Christof Paar
Problem sets from the book:
- Chapter 1
- Chapter 2
- Chapter 3
- Chapter 4
- Chapter 5
- Chapter 6
- Chapter 7
- Chapter 8
- Chapter 9
- Chapter 10
- Chapter 11
- Chapter 12
- Chapter 13
Solutions for Odd-Numbered Questions

Introduction

During my self-study on the topic of cryptography, I’ve found that the textbook “Understanding Cryptography” by Christof Paar and Jan Pelzl, and the accompanying YouTube lectures, are the most accessible introductory material I have found. The book contains a great many exercises related to the material.

There is a solution manual freely available from the website called Solutions for Odd-Numbered Questions, however the even numbered questions are unavailable. I have contacted the authors, but licensing restrictions prevent them providing the full manual to anyone except instructors in educational institutions. It does not appear that anyone has leaked the manual to the internet either.

As such, I have decided to create a comprehensive solution set to all problems in the book.

List of Questions and Answers

Chapter 1 - Introduction to Cryptography and Data Security

Ex 1.1 - Breaking a Substitution Cipher by Frequency Analysis
Ex 1.2 - Breaking a Caesar/Shift Cipher by Frequency Analysis
Ex 1.3 - Calculating the Length of Brute-Force Attacks against AES
Ex 1.4 - Password Length and Key Sizes in Bits
Ex 1.5 - Multiplication in Finite Sets
Ex 1.6 - Division in Finite Sets
Ex 1.7 - Computing Addition and Multiplication Tables in Finite Sets
Ex 1.8 - Computing Multiplicative Inverses of 5 in Several Finite Sets
Ex 1.9 - Computing Large Exponents in Finite Sets
Ex 1.10 - Computing Euler's Phi Function
Ex 1.11 - Decrypting with the Affine Cipher
Ex 1.12 - Decrypting with the Extended German-Alphabet Affine Cipher
Ex 1.13 - Breaking the Affine Cipher with a Chosen Plaintext Attack
Ex 1.14 - Proving that Double Encryption with the Affine Cipher is Equivalent to Single Encryption

Chapter 2 - Stream Ciphers

Chapter 3 - The Data Encryption Standard (DES) and Alternatives

Chapter 4 - The Advanced Encryption Standard (AES)

Ex 4.1 - The AES Development Process and How that Differed from DES
Ex 4.2 - Computing Addition and Multiplication Tables in Galois Prime Fields
Ex 4.3 - Computing Addition and Multiplication Tables in Galois Extension Fields
Ex 4.4 - Performing Addition and Reduction in GF(2⁴)
Ex 4.5 - Performing Multiplication and Reduction in GF(2⁴)
Ex 4.6 - Calculating a Division in GF(2⁸) and Reducing via the AES Irreducible Polynomial
Ex 4.7 - Finding Multiplicative Inverses in GF(2⁴)
Ex 4.8 - Finding Irreducible Polynomials in GF(2)
Ex 4.9 - Calculating the Output of the First Round of AES Encryption with All 1s
Ex 4.10 - Calculating the State After One Round for AES
Ex 4.11 - Deriving Equations for the Constant Multiplications Done in the MixColumn GF Computations
Ex 4.12 - Calculating the Number of XOR Gates Required for AES Diffusion
Ex 4.13 - Checking Multiplicative Inverses in GF(2⁸)
Ex 4.14 - Calculating the AES S-box for Certain Values Using the Affine Mapping
Ex 4.15 - Derive RC Constants in AES Key Schedule
Ex 4.16 - Calculating the Length of Brute-Force Attacks against AES

Chapter 5 - More about Block Ciphers

Let’s Encrypt Auto-Renewal for Nginx Reverse Proxies

2016-05-27T11:03:00+00:00

Motivation

If you are familiar with using Nginx as a reverse proxy and have already used Let’s Encrypt, skip to “Provisioning a Server“.

2016-06-11 - Improved the nginx config based on a suggestion from /u/nikomo

Nginx

A very useful feature of nginx is that you can host multiple services on the same host and the same IP. For example, you could have a Node.js application running on localhost:8080, a Jenkins CI service running on localhost:8088 and any other combination of web services written in various languages.

These local ports are not exposed to the pubic internet. Instead, nginx listens on port 80/443 and proxies the relevent service based on what domain name is being requested. For example, a request to jenkins.example.com would be proxied to localhost:8080 in our earlier example.

In order to secure our connection with the client, HTTPS connections can be set up in the nginx configuration. The local services can safely serve plaintext HTTP since all connections are local.

Let’s Encrypt

Let’s Encrypt is a new certificate authority which aims to provide free, automated SSL certificates. All the major browsers have now added Let’s Encrypt to their default list of trusted providers. This means that site operators who wanted to serve HTTPS but didn’t want to pay for a certificate now can.

This is actually very exciting for ordinary internet users’ privacy and security!

Certbot

Let’s Encrypt provide an automated CLI tool to obtain certificates called Certbot. Certbot proves to the Let’s Encrypt certificate authority that you own the domain by simply listning on port 80/443 and having the certificate authority make a request.

If you’re trying to acquire a certificate for newdomain.example.com then Let’s Encrypt will just resolve that domain name and make a request. If they receive the response they’re expecting then this proves that whoever is running certbot does control that domain.

There are currently three modes you can use to obtain your certificate:

Apache - Certbot is able to plug-in directly to the Apache web server and allow Let’s Encrypt to make their request without any down-time for what’s normally hosted there. This is an ideal solution for Apache users.
Webroot - If you’re not hosting your site with Apache, but have write access to the directory which serves as the web-root for the service, Certbot can drop some files in here and they’ll be served up when Let’s Encrypt comes knocking just as normal. Again, no interruption in service.
Standalone - Certbot will directly listen on the required port. This obviously requires some down-time for whatever service is normally listening on that port.

Unfortunately, a lot of the time when we’re using nginx as a reverse proxy, it (at first) appears that we need to use Standalone. For example, if you’re proxying a service that’s running in a Docker container, then you don’t have access to the web-root.

To make things worse, every time you need to renew the certificate for that one service, you need to stop nginx (severing the link to all other services too) so that Certbot can listen on those ports.

Fortunately, I’m going to show you a way around this. There is an easy way to set up your services behind an nginx reverse proxy and still get the benefits of automated certificate renewal.

Provisioning a server

For this example, I’m starting out by spinning up a fresh Ubuntu instance on Amazon EC2 (t2.micro). I’ve assigned an Elastic IP and, to begin with, allowed all incoming traffic via its security group. I’ve also pointed ssltest.busby.ninja at the instance. I’ll be including all installation commands so this tutorial can be followed exactly.

Setting up the services

For this example, I’m going to use a Rancher server as my example service. I’m doing this precisely because it runs in a Docker container, so I can’t use Certbot’s Apache mode, and I don’t have access to the service’s web-root directory.

The first thing we need to do after SSHing into the instance is install nginx and docker:

sudo apt-get update && sudo apt-get install nginx -y
wget -qO- https://get.docker.com/ | sh
# The next command is optional, it allows you to run docker without sudo
# Replace ubuntu with whatever your user is called
sudo usermod -aG docker ubuntu

After this, double-check that you see the default nginx page when you navigate to the domain. If this works correctly, then the next thing to do is start up our rancher server. Retrieving and running the server can be done with a single docker command:

# If you want to listen on another port, change only the first 8080
cid=$(sudo docker run -d --restart=always -p 8080:8080 rancher/server)

It may take a little while for rancher to be downloaded and run. Once the previous command has terminated, you can watch Rancher’s progress as it boots up using the following command:

sudo docker logs -f $cid

When you see Connection established in the logs, it should be ready and listening. Assuming you didn’t change it, you can test that by calling port 8080. In my case, that means navigating to http://ssltest.busby.ninja:8080.

Basic nginx configuration

Now that we have nginx and rancher running, we can configure it as a basic reverse proxy that supports HTTP (but not HTTPS yet). You’ll want to use sudo su - to login as root, then navigate to /etc/nginx/sites-available.

Below are the two configuration files which you need to create in this directory. The first ensures that if you enter the server’s IP into the browser, or nginx doesn’t recognise the domain in the HTTP host, then it will refuse to serve any content. The second is the configuration for our reverse proxy to rancher.

/etc/nginx/sites-available/default

server {
    deny  all;
}

/etc/nginx/sites-available/rancher

upstream rancher {
    # change this if you used a port other than 8080
    server 127.0.0.1:8080 fail_timeout=0;
}

server {
    listen 80;
    listen [::]:80;

    server_name ssltest.busby.ninja; # replace this with your domain

    location / {
        # Some of these might not be required for other services
        # proxy_pass is the most important
        proxy_set_header Host $host;
        proxy_set_header X-Forwarded-Proto $scheme;
        proxy_set_header X-Forwarded-Port $server_port;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_pass http://rancher;
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection "upgrade";
        proxy_read_timeout 900s;
    }
}

Next you need to symlink these into /etc/nginx/sites-enabled and restart the server.

cd ../sites-enabled
ln -sf ../sites-available/default
ln -sf ../sites-available/rancher
sudo service nginx restart

If restarting nginx fails, then the logs are at /var/log/nginx/error.log. If it succeeds, you should now be able to navigate to your domain (in my case http://ssltest.busby.ninja) and see rancher load as if you’d accessed it directly. If you see the nginx default page again, try refreshing, it may be the browser cache. You should also try accessing the server directly by IP (in my case http://52.51.228.42). If all has gone well you’ll see a 403 Forbidden message.

Setting up a web-root

As discussed in the introduction, we will be using the Certbot tool that Let’s Encrypt provide.

While we could use the standalone mode, this would involve shutting down nginx while it runs and disrupting service to rancher.
We can’t use the apache mode since we’re using nginx and rancher.
We don’t have access to rancher’s web-root so we also can’t use the webroot mode…

…or can we.

This is one of the ways in which nginx is really very cool. It’s possible for us to configure a separate web-root in our /etc/nginx/sites-available/rancher file. When Let’s Encrypt make the request, it expects to find the proof file located within the /.well-known subdirectory of the web-root. This is a relatively new IETF standard to prevent things like favicon.ico and other things cluttering the web-root.

As such we can recognise paths beginning with /.well-known as SSL proof requests and serve it up correctly. A previous version of this blog post used a solution that required checking the file system to see if the path existed for every request. If no file was found in the web-root it was proxied. This avoids that overhead: we only do web-root file system checks for requests beginning with /.well-known.

Firstly let’s create the webroot (I’m assuming you’re still root here):

mkdir -p /var/www/ssl-proof/rancher/.well-known
# Create file we'll use later to test 
echo "nginx is awesome!" > /var/www/ssl-proof/rancher/.well-known/test.html

Next, we need to modify our nginx config:

/etc/nginx/sites-available/rancher

upstream rancher {
    # change this if you used a port other than 8080
    server 127.0.0.1:8080 fail_timeout=0;
}

server {
    listen 80;
    listen [::]:80;

    server_name ssltest.busby.ninja; # replace this with your domain

    # Here we define the web-root for our SSL proof
    location /.well-known {
        # Note that a request for /.well-known/test.html will
        # look for /var/www/ssl-proof/rancher/.well-known/test.html
        # and not /var/www/ssl-proof/rancher/test.html
        root /var/www/ssl-proof/rancher/;
    }

    location / {
        proxy_set_header Host $host;
        proxy_set_header X-Forwarded-Proto $scheme;
        proxy_set_header X-Forwarded-Port $server_port;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_pass http://rancher;
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection "upgrade";
        proxy_read_timeout 900s;
    }
}

Restart nginx and access /.well-known/test.html (http://ssltest.busby.ninja/.well-known/test.html for me). You should see a message correctly stating how awesome nginx is. You should also check that other paths are still redirecting to rancher. If you’re a clean-freak like me you’ll probably want to delete test.html after confirming that everything is working. You can also log out of root for now.

Acquiring an SSL certificate

Next, we’ll need to get Certbot installed. Many of you might want to run it in its own user, and to get the latest version via GitHub. I’m going to keep things simple and just directly download a stable version to the ubuntu user’s home directory (and it also ensures hopefully the tutorial will still work in the medium-term future).

Important note: If you’re following along on a low power instance like the EC2 t2.micro that I’m using, the certbot install my cause errors. You just need to stop the rancher service while you do this segment since it’s a bit of a memory hog. I realise that this defeats the purpose of using webroot instead of standalone but this is intended as a demonstration. On a more powerful machine, or with less demanding services, you won’t have this issue. The commands to do this are below:

# Look for rancher, the container ID is on the left
# You can use ps -a to see stopped containers
sudo docker ps
sudo docker stop <container id>
# To restart it later:
sudo docker start <container id>

In the latest version of ubuntu you can actually install certbot via apt-get:

sudo apt-get install letsencrypt -y

To make this tutorial a little more universal though, I won’t do that. I’m going to get a stable version from https://certbot.eff.org.

# <Log out of root if necessary>
cd ~
wget https://dl.eff.org/certbot-auto
chmod a+x certbot-auto
# Run certbot once with no arguments
# This will install its dependencies
./certbot-auto

Now we get to the important part: actually requesting the certificate!

# Change the webroot and the domain to your own values if needed
./certbot-auto certonly --webroot \
    -w /var/www/ssl-proof/rancher/ \
    -d ssltest.busby.ninja

You’ll be prompted for an email, and to accept some terms and conditions. When you’ve done that, you’ll see something like the following:

IMPORTANT NOTES:
 - Congratulations! Your certificate and chain have been saved at
   /etc/letsencrypt/live/ssltest.busby.ninja/fullchain.pem. Your cert
   will expire on 2016-08-24. To obtain a new version of the
   certificate in the future, simply run Certbot again.
 - If you lose your account credentials, you can recover through
   e-mails sent to tom@busby.ninja.
 - Your account credentials have been saved in your Certbot
   configuration directory at /etc/letsencrypt. You should make a
   secure backup of this folder now. This configuration directory will
   also contain certificates and private keys obtained by Certbot so
   making regular backups of this folder is ideal.
 - If you like Certbot, please consider supporting our work by:

   Donating to ISRG / Let's Encrypt:   https://letsencrypt.org/donate
   Donating to EFF:                    https://eff.org/donate-le

Configuring nginx to use the SSL certificate

Users affected by the out-of-memory errors mentioned earlier will want to restart rancher now.

Again, it’s time to modify the rancher nginx config:

/etc/nginx/sites-available/rancher

upstream rancher {
    # change this if you used a port other than 8080
    server 127.0.0.1:8080 fail_timeout=0;
}

# This block 301 redirects HTTP requests to HTTPS
server {
    listen 80;
    listen [::]:80;

    server_name ssltest.busby.ninja; # replace this with your domain
    return 301 https://$server_name$request_uri;
}

server {
    listen 443 ssl;
    listen [::]:443;

    server_name ssltest.busby.ninja; # replace this with your domain

    root /var/www/rancher-certbot-webroot;

    # The public and private parts of the certificate are linked here
    ssl_certificate /etc/letsencrypt/live/ssltest.busby.ninja/fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/ssltest.busby.ninja/privkey.pem;

    location /.well-known {
        root /var/www/ssl-proof/rancher/;
    }

    location / {
        proxy_set_header Host $host;
        proxy_set_header X-Forwarded-Proto $scheme;
        proxy_set_header X-Forwarded-Port $server_port;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_pass http://rancher;
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection "upgrade";
        proxy_read_timeout 900s;
    }
}

This is quite a basic, naive way of setting up SSL but I’ll include instructions for increasing the security with extra configuration options in a future post.

Once you’ve restarted nginx then you should confirm by navigating to your domain name. You should be redirected to HTTPS and served up rancher with a nice green padlock showing in the address bar.

At this point, you’re basically done. You have your reverse proxy set up with a valid SSL cert (at least until the certificate expires) and you can repeat this process to proxy as many other services as you like.

Automated renewal and revoking certificates

In order to renew your certificates, you simply run the following:

# You can add --dry-run to test without changes
./certbot-auto renew

If you have any certificates that are within the renewal window (usually 30 days before expiry, Let’s Encrypt certs typically expire after three months) then these will be automatically renewed. Given that we left the web-root infrastructure in place, this can be done without any interruption to the service.

It’s a good idea to set up a cron job to run periodically. The consensus seems to be that you should run renew once or twice a day.

If, at this point, you’ve followed the tutorial and understood it then you should revoke your certificate before destroying your instance or test environment. This is accomplished with the following command:

#Change the path to reflect your domain name
./certbot-auto revoke --cert-path \
    /etc/letsencrypt/live/ssltest.busby.ninja/fullchain.pem

As a final note, I encourage you to have a read of Let’s Encrypt’s own documentation. There are things I haven’t covered (such as acquiring one certificate for multiple domains) and the things they cover, they do so in more detail than me. If, like me, you find ngnix to be a very powerful, flexible piece of software then you can find their documentation here. I haven’t scratched the surface of what it’s capable of.

Thank you to everyone who read this. (Constructive) feedback and corrections are always welcome.