POSTS

VPN for remote development (and edge compute)

So in an earlier post I mentioned outlined how I use a central dev machine with remote VSCode to keep one checkout of my code and develop on it, no matter what device I’m using, while maintaining a good developer experience (low latency). There’s a couple of issues that I needed to overcome while developing this approach.

  1. When developing it is common to fire up a development server that you can hit with a browser to see how the final product will behave. These dev servers commonly have poor authentication and security, primarily because they’re not intended to be exposed on the internet. If I’m coding on a remote machine, how can I hit it with my browser but maintain security?
  2. I do a lot of work with edge compute devices (IoT, AWS Greengrass, Video cameras etc…). If I’m at home, I still need to be able to SSH into the jetson nano I’m currently using as my deployment node to debug issues, look at logs, etc… If Its just sitting there on my device network at work, how can I get access to it?

The answer to these questions is to set up a special development VPN that all of my devices tie into. I set up a special IP range that is reserved for all of my devices that is routed over the VPN, but all other normal traffic goes out via the device’s normal internet connection. This gives me just like normal connectivity on my edge device, but the ability to get to them. This blog post will outline how I have set up my system.

Setting up the VPN

First up, I needed to choose my VPN software. I needed something that would work on all of my devices (iPad, Mac OS laptop, Linux Laptop, Linux edge devices, in the future my device lab router). OpenVPN seems to be the obvious choice here. Its widely supported and free.

We need a central “place” where we’re going to host our VPN. When I first started thinking about this, I tried using AWS' Client VPN endpoint feature. This allowed me to set up a VPN that would accept multiple connections easily, but it has a fatal flaw - It uses NAT to mangle the packets going back and forward between your client devices and the AWS VPC that is hosting them. This makes it impossible to route packets from one edge device to another, which is a fundamental requirement of my solution.

Instead I chose to host my own VPN on my development server (running on AWS EC2). This server is already running my VSCode instance and is my development server, to its going to be the hub of all my operations. It makes sense to run OpenVPN there. Installing OpenVPN is as simple as running an apt command (assuming you’re running Ubuntu)

sudo apt install openvpn

Then we need to perform the following steps:

  1. Set up a Certificate Authority (CA) to manage security for all our devices
  2. Configure the VPN.
  3. Alter the security group to allow VPN traffic into the EC2 host
  4. Create client configuration for each of my devices
  5. (optional) Set up a DNS domain to make my hosts easy to address

Setting up a Certificate Authority

The VPN needs a way to authenticate its clients (and itself). You could use user/password pairs, but certificate based authentication will be more secure. You will need to create a Certificate Authority, a server certificate to identify the server, and then one certificate for each end device that will be connecting.

Most people use EasyRSA to set up their CA. This makes it really easy to host a CA. It won’t be the considered world’s best practice for certificate management (you need Safes and air gapped computers to do that stuff), but it will be sufficient for most use cases.

As I work for AWS, I chose to go a slightly different route. AWS provides a private CA service that will manage the CA for me and keep it secure. This is perfect for my needs, but be aware that if you were to go down this route there is an ongoing cost associated with the keeping the root CA secure that will price it out of most hobby user’s budget.

No matter what route you take, all you need to do is end up with the following certificates:

  1. The CA certificate (can be public)
  2. The Server certificate file (can be public) and private key (must be kept secret)
  3. A certificate (can be public) and private key (must be kept private, only put it on the edge device) for each device that will connect. Technically, you could use the same certificate for each device, but its easy to create one certificate for each, and gives you better repeatability for your IP addresses.

Configuring the VPN.

Here’s the openvpn configuration that I placed in /etc/openvpn/server.conf:

local SUBSTITUTE_YOUR_BIND_IP_HERE
port 443
proto udp
dev tun
ca server/ca.crt
cert server/server.crt
key server/server.key
dh server/dh.pem
auth SHA512
tls-crypt server/tc.key
topology subnet
server 10.8.0.0 255.255.255.0
ifconfig-pool-persist ipp.txt
keepalive 10 120
cipher AES-256-CBC
user nobody
group nogroup
persist-key
persist-tun
status openvpn-status.log
verb 3
crl-verify server/crl.pem
explicit-exit-notify
push "route 172.31.0.0 255.255.0.0"

You’ll need to modify the file to suit your use cases. There’s a couple of call outs I’d like to make

  1. The first line is used to specify what interface openvpn will bind to. Put in the IP address of your server here. In my case, this is an EC2 host running inside a VPC, so I used the internal VPC IP address - Your use case might be different
  2. This VPN uses UDP - This is recommended by OpenVPN and comes with one great advantage - If your end device moves from one network to another (say you go to a coffee shop), the VPN tunnel will stay up when your device changes IP address. If you use TCP the VPN will need to re-join.
  3. The various CA certs, keys and paths will need to be changed to your server files. There’s also some additional crypto files like dh and tls-crypt that you can specify to create a slightly more secure environment. Check the OpenVPN documentation for details.
  4. The port you choose to listen on (see line 2) doesn’t really matter. The default is 1194. I chose 443 here because some work firewalls block unknown ports. Be aware that in this case, this is a UDP port so even though it looks like its for HTTPS, this is separate to any web servers that might be running.
  5. The last line of the file pushes an additional route - This is telling the VPN to send a route to all clients to route traffic for 172.31.0.0/16 over the VPN. This IP range is the IP range for my AWS VPC. It allows my client devices to access any hosts that I have set up in my AWS VPC as if I was there (assuming my VPN server does IP forwarding, and I’ve set up the VPC routing table correctly). IF you’re not spinning up additional resources in your AWS VPC, this line is not needed.

you can now run the following commands to run your server

sudo systemctl start openvpn@server

The VPN should auto-start when your host boots up (assuming you’ve configured it correctly. It does by default)

Configuring security groups

NOTE: This is only needed if you are hosting on AWS.

Our server is now listening on port 443, but by default AWS blocks all inbound traffic to its hosts. You’ll need to add a rule to your EC2 host’s security group. Here’s what I did to enable it.

  1. Sign in to the AWS console.
  2. Navigate to EC2, and find your host in the list
  3. In the details pane, there should be an entry for the host’s security group. click on the name of the security group to bring up its detail.
  4. Select the Inbound rules in the detail pane, then click the Edit button.
  5. Add a rule with the following settings
    1. Custom UDP rule
    2. Port 443 (Or whatever you put in your config)
    3. Source 0.0.0.0/0 - This is everywhere on the internet. If you have known IPs that you’ll be accessing the VPN from, you can be more restrictive here.
    4. Add in a suitable description - I chose ‘VPN’
  6. Save. You’re VPN is now open to the world, but protected by certificates so in theory nobody but you can get in. You kept all the secret keys secret, right?

Client configuration

For each device, create a file that looks like this:

client
dev tun
proto udp
remote HOSTNAME_OF_YOUR_SERVER 443
resolv-retry infinite
nobind
persist-key
persist-tun
remote-cert-tls server
auth SHA512
cipher AES-256-CBC
ignore-unknown-option block-outside-dns
verb 3
<ca>
-----BEGIN CERTIFICATE-----
Content of CA certificate file
-----END CERTIFICATE-----
</ca>
<cert>
-----BEGIN CERTIFICATE-----
Content of device certificate file
-----END CERTIFICATE-----
</cert>
<key>
-----BEGIN RSA PRIVATE KEY-----
Content of device Private Key file
-----END RSA PRIVATE KEY-----
</key>
<tls-crypt>
-----BEGIN OpenVPN Static key V1-----
Content of tls key
-----END OpenVPN Static key V1-----
</tls-crypt>

There’s a bunch of substitutions you’ll need to do here. This version of the config embeds the contents of your CA file, the TLS crypt params, and the device’s certificate and private key directly in the config file. You’ll also need to change the remote host name to the name or IP of your server. I’ve done it this way so that only one file needs to be present on the target device, instead of lots. This is especially useful when adding OpenVPN profiles to an iOS device.

Once all this is set up, you should be able to start the VPN on your client device and connect to your server (or other devices). How you do this will vary based upon the device you’re using. In my circumstance, I used the following:

  1. Linux: apt-get install openvpn, configuration goes in /etc/openvpn
  2. Mac: Tunnelblik
  3. iOS: OpenVPN connect

Each operating system will have its own clients. A simple google will find them for you.

DNS settings for each device.

One nice feature of openVPN is that it can cache which IP address is used for each client, based on the common name in the certificate. If you look at /etc/openvpn/ipp.txt (you’ll need root access to get to it), you’ll see something like:

common.name.net,10.8.0.3
common2.name.net,10.8.0.4
common3.name.net,10.8.0.5
common4.name.net,10.8.0.6

One entry per line, specifying the common name of the client and the IP address in your VPN range that it maps to. While this file is kept around, any connection from a client with that certificate will be given that IP address. This is (one of the reasons) why its a good idea to have a separate certificate for each device.

To make it so that I don’t have to remember IP addresses, I have mapped each of these values to a DNS domain that I have set up for my devices. I originally thought that setting up a Route53 internal domain would be a good idea, and free, but this would mean I would need to use Route53’s internal DNS servers for all DNS traffic when connected to the VPN. My employer has its own internal DNS servers, so they’d clash (You can really only have one DNS server at any given time). Instead, I chose to publish these IP addresses on public DNS so that I can get at them through whatever DNS server I’m using at the time. Does this leak information publicly? Yes. Is it critical information? Not really.

Wrap up

So there you have it. Once this is all set up I now have a bunch of devices that are all permanently connected to each other, and the AWS VPC that is tying this all together. I can SSH/mosh into any of them no matter what network they’re currently sitting on. This also provides an additional layer of security for any other services that you might be running in your VPC or dev hosts. Thats useful when you’re doing development work, or have a particularly sensitive service that you don’t want to expose to the world just yet.

One note on reliability - This isn’t a robust VPN. If the development host crashes all of the devices will become unresponsive. I haven’t put in any form of redundancy. Other options exist for providing this if it is a requirement. For the moment, this is good enough for development.

In the future, I’m going to set up the router in my device lab to have one of these tunnels as well, so that all of the devices in my device lab are on the VPN network, even if they don’t have their own individual VPN connection. This will require some routing changes, but this is all within the capabilities of OpenvPN.

If you have any questions, feel free to get in touch via any of the means up top.