Set up UniFi Controller on Google Cloud Platform

Google offers one free virtual machine on Google Cloud Platform. This script will make setting up a UniFi Controller on GCP a breeze and it includes all the goodies. No command line required, everything is done in the GCP Console and it takes 15 minutes total and that includes transferring your current sites to the cloud. You should try this!

The script I have created will set up an UniFi Controller on GCP with these extras:

  • Automatically acquire and install a Let’s Encrypt certificate to relieve you from HTTPS certificate warnings. For a certificate you need to have a domain name for your controller, dynamic or static.
  • Dynamic DNS support to help you set up the domain name.
  • You don’t need to type https://your.server.dns:8443 every time. You just type the DNS name of your controller and it will be redirected to HTTPS port 8443 automatically.
  • Backup files will be copied to a Google Storage bucket for offline storage.
  • Fail2Ban protects your controller from attackers trying to brute force the password: the IP address of the offender will be denied access for an hour after every three failed attempts.
  • The free tier GCP micro instance comes with only 600MB of memory. The script will create a swap file to accommodate the controller.
  • The underlying Linux system and UniFi Controller will be automatically updated to the latest stable releases.
  • If the MongoDB database supporting UniFi Controller needs repairing, a reboot will do the trick. There is a support script to run the repair commands as needed before the controller starts.
  • HAVEGEd will store entropy for encryption. Without sufficient entropy available, the boot time could be 30 minutes or more.

None of this will cost a dime to try out. At least you can watch this 10 minute video to see the whole procedure including transferring your current controller:

(There are subtitles available under the YouTube gear icon)

1. Preliminaries

Take a backup of your current controller in Settings > Maintenance. This is always a good idea. You may also want first to improve your password. In Google Cloud your controller can be accessed by anyone who guesses your username and password. You don’t want that. Fail2Ban will lock the IP address for an hour after three failed attempts, but it won’t help if your password is guessable.

A proper SSL certificate requires a DNS name that points to your controller. You can register your own domain or use one of the free ones that dynamic DNS providers offer. If you want to use a dDNS service, you need to set it up first. There are links to some providers at the end. If you control your DNS yourself you won’t need dDNS.

Next you have to create an account on Google Cloud Platform. You will need a credit card for this, but the micro instance that Google offers for free is sufficient for a small UniFi Controller and the free 5GB on Cloud Storage is plenty for the backups. Google does not guarantee this offer will run forever, but it has run for years now and they will certainly let you know if this changes. The current price for a micro instance is $4.28 per month, if you need to run another for example.

First of all, use a good password or better yet, set up two factor authentication for your Google account. If someone breaks into your account the intruder could create a huge bill in just a few hours. You can also set a spending limit in Billing > Budgets & Alerts. Create a new budget for the whole billing account and set a budget. $50 or even $10 should be enough, if you ever want to do some testing on an extra controller for a few hours. The budget feature won’t protect you from an intruder breaking into your account, however. The first thing the intruder would do is to remove the limit.

If you set up budget alerts (50%, 90% and 100% by default) you need to nominate yourself to a billing administrator to receive the alert emails. This is done in IAM & Admin > IAM section. Locate your identity in the list and click in the Roles column. Check Billing > Project Billing Manager to add this role to your identity.

2. Create a Storage Bucket (optional)

UniFi Controller will create backups automatically. The default is once a month, but you can change this in UniFi Controller > Settings > Auto Backup. I run backups once a week and keep the last 26 or a half year’s worth. These backups are stored on the computer the controller runs on. If this computer is lost for some reason, so are the backups. You should store them somewhere else and a Google Storage bucket is the perfect place. If you create a bucket the script will sync the backups to the bucket daily.

In GCP Console switch to Storage > Browser and create a new bucket. The bucket name has to be globally unique, so all the obvious ones are already in use. Use something like petri-office-unifi-backup. For the free offer select Regional as the Storage class and choose a Regional Location in the U.S. I am in Europe and East Coast is marginally closer, so I use it.

3. Set Up the Virtual Network

This is something you will need to do only once. All your future UniFi controllers will use the same network settings.

Switch to VPC Network > Firewall Rules. Create a new firewall rule and name it allow-unifi for example, the name isn’t important. In the Target tags field enter unifi. You can use any name for the tag, but you will need to remember it for the next step. In the Source IP range type 0.0.0.0/0 to denote any address. In the Protocols and ports section select Specified protocols and ports and copy this into the field:

tcp:8443;tcp:8080;udp:3478;tcp:8880;tcp:8843;tcp:6789;tcp:443;tcp:80

Switch to  VPC Networks > External IP Addresses and click on Reserve Static Address. You need to give a name for the address, choose the region where your virtual machine is going to run and click Reserve. I suggest you use the same region as the Storage bucket. You can’t attach the address yet, since you haven’t yet created the virtual machine. (Google will charge for the static addresses while they are not in use. The plan is to run the controller non-stop so this is not an issue. Just remember to release the address if you don’t need it any longer.) If you manage your DNS yourself, add an A record for this IP at this point.

4. Create the Virtual Machine

In Compute Engine > VM Instances create a new instance. You can name it anything you like. Select a Zone in the U.S. for the free offer. You need to place the VM in a zone that is in the same region as the static address. The Machine type should be micro for the free instance. By default the Boot disk is 10GB which suffices, but it is tricky to expand afterwards. The free offer is 30GB but I prefer to set the boot disk to 15GB in case I need to run another VM. To change the size, click on Change and set the size. The script is designed to run on Debian Linux so don’t change that. Click Select to save.

If you created a bucket for the backups, you need to allow the virtual machine to write to Google Storage. The default permission is Read Only. In Access Scopes choose Set access for each API. In the list that appears change Storage to Read Write.

No need to select the HTTP and HTTPS options in the Firewall section. You created your own firewall rules already.

Click on Management, disks, networking, SSH keys to open advanced options. On the appearing Management tab add the following Metadata key/value pairs. The first one is mandatory: the script that does the magic. All the others are optional and the order doesn’t matter. There is no way to do any error checking or report script failures, some feature will just not work if the metadata is erroneous. You can come back and change them if necessary. The changes will take effect after next reboot. See the Maintenance section at the end for how to reboot.

Key Value Purpose
startup-script-url gs://petri-unifi/startup.sh Required!
ddns-url http://your.dyn.dns/update.php?key=xxxxx Helps you access the Controller
timezone Europe/Helsinki Lets you adjust the reboot hour
dns-name your.server.dns Required for HTTPS certificate
bucket your-bucket-name-here Required for offline backups

Click on the Networking tab and add unifi to the Network Tags field. This tag ties the firewall rules you created earlier to this virtual machine instance. This step is easy to forget, but missing it will prevent access to your controller! Click on Default under Network Interfaces and choose the static IP address you created earlier as the External IP.

Click Create. Go get coffee. Even if the virtual machine appears to be running, the script is not done yet. It usually takes less than five minutes, but give it ten.

5. Set Up the Controller

Connect to your new controller. On the first screen of the UniFi wizard click restore from a previous backup and upload the latest backup. Wait for the restore to complete. Log on to the new controller using your old username and password. In Settings > Controller add the DNS name or the IP address of the new controller to Controller Hostname/IP field and check Override inform host with controller hostname/IP. Confirm the change. Click Apply Settings.

Connect to your old controller. Do the same change in Settings > Controller and add the DNS name or the IP address to Controller Hostname/IP field and check Override inform host with controller hostname/IP. This setting is for the current devices that are already associated with the current controller. Now they will start to contact the new controller in Google Cloud.

Connect to your new controller again. Check the devices page to see whether the devices come online eventually. If they don’t, you may have to force reprovisions them on the old controller. In that case go to the Devices page on the old controller, select each device in turn and on the Config tab click Provision under Manage Device.

How to Set Up a New Network with a GCP Controller

If you are starting from scratch you won’t have any controller backup to transfer to the GCP based UniFi Controller. In that case you have three alternatives:

  1. Install a temporary controller on a local computer, set up the network and transfer the controller to GCP as outlined above.
  2. If you want to do a pure cloud based installation, you’ll need to tell the devices the address of the controller. If you have an UniFi Security Gateway connect its WAN interface to the Internet and your laptop directly to the LAN interface. Connect to the USG with a browser (https://192.168.1.1) and configure the inform URL to http://your.server.dns:8080/inform. Once you have adopted the USG in the controller other UniFi devices will appear in the controller as you start to connect them.
  3. If you have some other router than an USG, you can either set your DNS server to resolve name unifi to the IP address of your cloud controller or set the DHCP option 43 in your DHCP server. There are examples of both in Ubiquiti’s L3 Adoption Guide. The last resort is to SSH to each device in turn and use command
    set-inform http://your.server.dns:8080/inform.

Maintenance and troubleshooting

If your devices won’t switch to the new controller the reason is usually either in the firewall or DNS. Is your VPC firewall rule correct and does the network tag of the virtual machine match the tag on the rule? Is your DNS name correct both in the metadata and in both controllers’ Override Inform Host fields and does the DNS name point to the correct IP address? In either case the devices will try to connect for several minutes before reconnecting to the old controller. Sometimes it takes hours for DNS to settle, depending on the time-to-live settings. In that case let it sit overnight and force provision the devices next morning.

Let’s Encrypt limits the number of certificates issued per domain to five per week (at the time). This usually isn’t an issue, but if you use the free domains offered by dDNS providers, the most popular domains will have used this week’s five certificates already. In this case the script will try to acquire a certificate once an hour until it succeeds. You will see certificate warnings at first but the address bar should eventually turn green. You can also anticipate this and choose a less popular domain (or register your own).

Don’t set up two controllers to back up to the same bucket. They will delete each other’s backups. You can either create a new bucket or create folders inside the bucket. In the metadata set bucket to your-bucket/foldername.

Google blocks sending email directly to SMTP port 25 from within GCP because they don’t want all spammers on board. If you want to receive email notifications you need to set up authenticated SMTP on port 587 or 465 in UniFi’s Settings > Controller > Mail Server. You can use your personal email account or even use Google’s email engine if you can prove you are not a spammer.

The Cloud Console will typically start displaying a recommendation to increase performance by upgrading to a bigger virtual machine (at cost), even though the CPU utilisation graph typically stays below 15%. Apparently the updates the devices send cause very short spikes, because occasionally the devices report that the controller is busy. This may cause some inaccuracies in the reports, I am not certain. If you can live with that, just dismiss the suggestion. In my experience the micro instance can serve at least the same 30 devices a Cloud Key can. You can also use this script on any virtual machine type you like. When you select the machine type you see the cost per month. You get a better price if you can commit to use the resource for 1 or 3 years.

If the controller appears to be malfunctioning the first remedy is to reboot it. A reboot is also the way to re-run the script if you made changes to the metadata. The safe way to reboot is to Stop and then Start the virtual machine. This won’t change the IP address since you have reserved a static address. Don’t use the Reset button. A Reset will immediately restart the virtual machine, which may damage the UniFi database and/or the Linux filesystem (if you are really unlucky).

The underlying Linux system and UniFi Controller will be automatically updated to the latest stable releases. If an update requires a server restart or a reboot it will occur after 04AM UTC. Set the timezone metadata to adjust the reboot to 04AM local time. A reboot will make a captive wireless guest portal inaccessible for a couple of minutes so you don’t want it to happen during peak hours.

Automatically updating the system is a risk. For example there could be an update to Java or MongoDB that is incompatible with the installed UniFi controller. There is no way you could prevent the installation of the update. However, I decided that NOT updating the system is even bigger risk. I don’t trust the intended audience (you) to log on regularly over SSH and run apt-get. In any case, if the risk realizes I trust that it will be transient. The incompatibility will affect a lot of other users as well and there will be another update to solve the issue. This other update will be also automatically installed . If you are familiar with Linux you can also log on the VM and edit /etc/apt/apt.conf.d/51unattended-upgrades-unifi to your requirements.

The automatic update won’t do major Linux version upgrades. Every couple of years it is wise to upgrade the underlying operating system. The Cloud Way is to create a new UniFi controller:

  1. Take a backup of the old controller.
  2. Create a new bucket (or folder).
  3. Remove the static IP address from the old virtual machine (VPC Network > External IP Addresses and Change to None)
  4. Create a new virtual machine as outlined above and restore the backup
  5. Delete the old virtual machine
  6. Done! Since the IP address is the same there is no need to update any settings and the turnover is immediate.

If your controller starts to act up and rebooting doesn’t help, just create a new one. Just like upgrading the underlying Linux, create a new virtual machine. If you cannot connect to the old controller to retrieve the backup, use the latest auto backup in the storage bucket. That’s what they are for. Don’t fall in love with your virtual machine, you should treat them like pencils: very useful but replaceable.

If you are concerned about the security of the script you better read it through first (the link is at the end). Be aware that the script is in my bucket so I can update it as necessary. I could also make malicious changes that would take effect on your server after next boot (after a system update for instance). If you are cautious you should remove the whole startup-script-url row from the metadata as soon as you are happy with the other metadata settings. The script doesn’t do anything once the system is configured. If you ever need to reconfigure something you can just add the startup-script-url back, Stop and Start the VM to run the script and then remove the row again.

Google currently offers for free one micro instance with 30GB of disk space and 5GB of bucket storage in the U.S. regions. Both come with monthly 1GB of free egress (i.e. outbound) data transfer with China and Australia excluded. Egress data is used when you download backups to your computer and when your devices download upgrades from the controller. The static IP address is free as long as it is attached to a running virtual machine. You will be charged if you exceed these limits, but typically the charges are minuscule. For example when you upgrade your Linux you will run another micro instance for a while. The budget feature should protect you from nasty surprises.

Links

Some dynamic DNS providers and how their single line update URLs look like:

  • Afraid.org
    http://freedns.afraid.org/dynamic/update.php?xxxdynamicTokenxxx
  • DNSPark
    https://username:passsword@control.dnspark.com/api/dynamic/update.php?hostname=home.example.com
  • DuckDNS
    https://www.duckdns.org/update?domains={YOURVALUE}&token={YOURVALUE}
  • DynDNS
    https://{user}:{updater client key}@members.dyndns.org/v3/update?hostname={hostname}
  • EasyDNS
    https://username:dynamictoken@api.cp.easydns.com/dyn/generic.php?hostname=example.com
  • NameCheap
    https://dynamicdns.park-your-domain.com/update?host=[host]&domain=[domain_name]&password=[ddns_password]
  • Sitelutions
    https://dnsup.sitelutions.com/dnsup?id=990331&user=myemail@mydomain.com&pass=SecretPass
  • ZoneEdit
    https://<username>:<password>@dynamic.zoneedit.com/auth/dynamic.html?host=<hostname>

 

14 thoughts on “Set up UniFi Controller on Google Cloud Platform”

  1. The biggest update so far didn’t go cleanly. Upgrading UniFi controller 5.6 to 5.7.20 killed the MongoDB server disabling the controller. Starting and stopping the virtual machine recovers it. I can’t control the package contents and instructions. I can only hope for the best. Unattended upgrading is always a gamble.

  2. Hi Petri, thank you very much for all the work you shared with us.
    I follow your procedure and the video but controller is not working.
    How can I check if it’s been downloading and rouning?
    Can I force via SSH script execution so I can see if any error appears?

    Thank you again.

    1. You can SSH into the virtual machine from the Google Cloud Console. Click the SSH button and a pop-up window will appear (unless your pop-up blocker gets into way). No password is required, you are already authenticated in the console. The SSH session is useful if you want to see the logs in your VM.

      Another option is to click on the name of the virtual machine in the VM Instances list. On the VM Instance Details page you’ll see a small link for “Serial Port 1 (Console)”. That will show you all the startup messages. The screen doesn’t update automatically if you want to watch it in real time. You need to click the Refresh for an update. The lines containing “startup-script:” are produced from my script. Do you see any errors?

      1. Thanks for your Reply Prtri:
        In “Serial Port 1 (Console)” there are 6 lines with startup-script:
        Apr 15 17:45:03 unifi-controller startup-script: INFO Starting startup scripts.
        Apr 15 17:45:03 unifi-controller startup-script: INFO Found startup-script-url in metadata.
        Apr 15 17:45:03 unifi-controller startup-script: INFO Downloading url from gs://petri-unifi/startup.sh to /startup-Vm93gn/tmp792gVc using gsutil.
        Apr 15 17:45:08 unifi-controller startup-script: WARNING Could not download gs://petri-unifi/startup.sh using gsutil. Command ‘[‘gsutil’, ‘cp’, u’gs://petri-unifi/startup.sh’, ‘/startup-Vm93gn/tmp792gVc’]’ returned non-zero exit status 1.
        Apr 15 17:45:08 unifi-controller startup-script: INFO No startup scripts found in metadata.
        Apr 15 17:45:08 unifi-controller startup-script: INFO Finished running startup scripts.

        I don’t explain myselft correctly in my last comment, I wanted to know how to execute your script manually 😉
        I found how to download it and executed with sudo, it seem my VM is not accesible from itself:
        Script logrotate set up
        MongoDB logrotate set up
        IPv6 disabled
        ERROR: Address 35.190.181.102 has not changed.
        Dynamic DNS accessed
        592 megabytes of memory detected
        Swap file created
        Localtime set to Europe/Madrid
        Unifi added to APT sources
        System upgraded
        Haveged installed
        CertBot installed
        Extracting templates from packages: 100%
        Unifi installed
        Synchronizing state of mongodb.service with SysV service script with /lib/systemd/systemd-sysv-install.
        Executing: /lib/systemd/systemd-sysv-install disable mongodb
        Lighttpd installed
        Fail2Ban installed
        Unattended upgrades set up
        Created symlink /etc/systemd/system/multi-user.target.wants/unifidb-repair.service → /etc/systemd/system/unifidb-re
        pair.service.
        Unifi DB autorepair set up
        Backups to bdlsolucinoes-backup set up
        Saving debug log to /var/log/letsencrypt/letsencrypt.log
        Plugins selected: Authenticator standalone, Installer None
        Registering without email!
        Obtaining a new certificate
        Performing the following challenges:
        http-01 challenge for mysub.domain.com
        Waiting for verification…
        Cleaning up challenges
        Failed authorization procedure. mysub.domain.com (http-01): urn:acme:error:connection :: The server could n
        ot connect to the client to verify the domain :: Fetching http://mysub.domain.com/.well-known/acme-challeng
        e/rKudGtFidPxCAHNxR__YzQ3qb8PohwWs38yaS6voXcg: Timeout

        I pinged “mysub.domain.com” from VM and my PC, always reply public GCP IP configured as you explained.

        From my PC http://mysub.domain.com or /.well-known/acme-challeng shows ERR_CONNECTION_TIMED_OUT

        Maybe webserver is not working?
        Thanks.

        1. The first part of your post indicates that the VM couldn’t load the startup.sh script at all. That’s odd. I don’t have any other GCP accounts to try on right now. I’ll look into this.

          The second part where you run the script manually indicates that Let’s Encrypt service cannot connect to your domain from their data center. Does the dns-name metadata resolve to 35.190.181.102 (check for typos)? Is port 80 included in the firewall rule and the same tag applied to the VM? Testing tools are dig, nslookup, getent or host for the former and fetch or curl for the latter question.

          1. Hi Petri: your script works great! It’s awasome.

            Connection to my instance was failing becouse I forgot /0 in the 0.0.0.0 on firewall source… 😀

            About startup-script: I found this in serial log:
            ——
            Apr 18 15:25:43 unifi-controller google_metadata_script_runner[683]: AccessDeniedException: 403 Access Not Configured. Please go to the Google Cloud Platform Console (https://cloud.google.com/console#/project) for your project, select APIs and Auth and enable the Google Cloud Storage JSON API.
            ——

            I finally fixed it.

            I want to ask you something: is it as difficult as I read to set a port mapping in GCP?
            My current ISP blocks 8080 so I’m running CloudKey Unifi at TCP/8082

            To easily migrate all my deviced, as you wrote, I want to change controller hostname. So I need to redirect GCP-public-IP:8082 -> VM:8080

            I read lots of commands to enable this port mapping. Do you know a simple way?

            Thank you again! An incredible job.

  3. You sir, are the man!

    Awesome work, seems you’ve thought of everything.

    This must be the only GCP+Unifi tutorial to exist, and it’s fairly easy to follow, it was an absolute treasure of a find, especially given that GCP just happens to be the most difficult to figure out..

    This is killer, it works perfectly, many thanks for this, much appreciated!

    1. Thanks! If you did set up your system recently, it proves that the script download problem in the previous comment by Limbo is either new or random occurence.

  4. Hi! Petri:
    I followed your article, I found that certbot doesn’t work for my cloud controller on GCP.

    Message of Serial Port 1 (Console) contain “startup-script:” looked like error info to me:
    Apr 21 20:33:13 unifi-controller startup-script: INFO startup-script-url: debconf: unable to initialize frontend: Dialog
    Apr 21 20:33:13 unifi-controller startup-script: INFO startup-script-url: debconf: (TERM is not set, so the dialog frontend is not usable.)
    Apr 21 20:33:13 unifi-controller startup-script: INFO startup-script-url: debconf: falling back to frontend: Readline
    Apr 21 20:33:13 unifi-controller startup-script: INFO startup-script-url: debconf: unable to initialize frontend: Readline
    Apr 21 20:33:13 unifi-controller startup-script: INFO startup-script-url: debconf: (Can’t locate Term/ReadLine.pm in @INC (you may need to install the Term::ReadLine module) (@INC contains: /etc/perl /usr/local/lib/x86_64-linux-gnu/perl/5.24.1 /usr/local/share/perl/5.24.1 /usr/lib/x86_64-linux-gnu/perl5/5.24 /usr/share/perl5 /usr/lib/x86_64-linux-gnu/perl/5.24 /usr/share/perl/5.24 /usr/local/lib/site_perl /usr/lib/x86_64-linux-gnu/perl-base .) at /usr/share/perl5/Debconf/FrontEnd/Readline.pm line 7, line 2.)
    Apr 21 20:33:13 unifi-controller startup-script: INFO startup-script-url: debconf: falling back to frontend: Teletype
    Apr 21 20:33:13 unifi-controller startup-script: INFO startup-script-url: dpkg-preconfigure: unable to re-open stdin:
    Apr 22 04:38:04 unifi-controller startup-script: INFO startup-script-url: /usr/local/sbin/certbotrun.sh: 7: [: =: argument expected

    Then I managed to use acme.sh which is an alternative to certbot to apply let’s encrypt SSL certificate:
    https://www.naschenweng.info/2017/01/06/securing-ubiquiti-unifi-cloud-key-encrypt-automatic-dns-01-challenge/

    The certificate then installed successfully.

    Is there anything I need to do with the error info besides certbot?
    Thanks.

    1. Each and every apt-get install will create all those debconf and dpkg-preconfigure errors to the log. I haven’t found a way to suppress them without losing valuable debug info. Apt-get is just trying to communicate with the user and doesn’t find a controlling terminal.
      The last line is significant, though, about the certbotrun.sh. It means that for some reason the script couldn’t retrieve the external IP address of the VM. Does your VM have one? I couldn’t reproduce this. Anyways, I have updated the script (currently at 1.0.3) so that it will output a more useful error message instead.
      I haven’t looked into acme.sh. You may have to import the certificate manually to the Java keystore after every renewal. Or you can edit /etc/letsencrypt/renewal-hooks/deploy/unifi to work with acme.sh.

      1. Hi~ Petri,
        I’ve reserved a static external IP address for my cloud Controller VM on GCP as your tutorial steps.
        I am fine with acme.sh, just created a crontab to perform the auto renewal.
        Thanks for your reply!

  5. How do I use Let’s Encrypt with this ?

    How do I see what is in the gs script ?

    Can this setup be done for UNMS ?

    1. 1) Create a DNS name and point it to your VM instance. Add the DNS name to the VM metadata with key dns-name as instructed. Everything else is automated.
      2) At the end of the article in section Links there is a link “The startup script on Google Storage”
      3) My script won’t do it, but you could write a similar setup for UNMS. You can also read through the script and do the steps manually to apply them to an UNMS instance.

Leave a Reply

Your email address will not be published. Required fields are marked *