GCP Unifi Controller startup script explained

This is a walkthrough of the Google Cloud Platform virtual machine startup script to launch full featured Unifi Controllers on demand.

See how-to instructions and a video in another article.

This script was written for Google Cloud Platform and Debian. It should be fairly straightforward to port to any other cloud platform. They have a similar way of accessing virtual machine metadata. The only other GCP specific function is gsutil rsync in the backup script.

Porting to other Linux distributions or Unices would take some more effort. If the destination system doesn’t use systemd then the services and timers should be rewritten. The script also uses apt-get to install packages, but that is fairly easy to edit. Ubiquiti only provides a dpkg package for the controller, though.

I have tried to write the script in open style without shortcuts. A lot of the ifs could be streamlined, but I value legibility. The resulting system is also easy to reconfigure, if you are familiar with Unix command line. The list of files is at the end of the article.

# Set up logging for unattended scripts and UniFi's MongoDB log
# Variables $LOG and $MONGOLOG are used later on in the script.
if [ ! -f /etc/logrotate.d/gcp-unifi.conf ]; then
	cat > /etc/logrotate.d/gcp-unifi.conf <<_EOF
$LOG {
	rotate 4
	echo "Script logrotate set up"
if [ ! -f /etc/logrotate.d/unifi-mongod.conf ]; then
	cat > /etc/logrotate.d/unifi-mongod.conf <<_EOF
	rotate 10
	echo "MongoDB logrotate set up"

Although most users won’t ever log on to the virtual machine, I still find it worthwhile to write a log. This log will only contain unattended script actions like database repairs or certificate updates every 3 months or so. Not much volume, but it is still good practice to set up logrotate. Four months will cover the last certificate update. For some reason Ubiquiti doesn’t set up logrotate for their MongoDB instance so we’ll cover that as well.

# Turn off IPv6 for now
if [ ! -f /etc/sysctl.d/20-disableIPv6.conf ]; then
	echo "net.ipv6.conf.all.disable_ipv6=1" > /etc/sysctl.d/20-disableIPv6.conf
	sysctl --system > /dev/null
	echo "IPv6 disabled"

I found out that occasionally the UniFi controller will try to bind to IPv6 only. GCP doesn’t route IPv6 to the virtual machines at this time, so we can just as well turn it off.

# Update DynDNS as early in the script as possible
ddns=$(curl -fs -H "Metadata-Flavor: Google" "http://metadata.google.internal/computeMetadata/v1/instance/attributes/ddns-url")
if [ ${ddns} ]; then
	curl -fs ${ddns}
	echo "Dynamic DNS accessed"

If the user supplied a dynamic DNS service URL we’ll call it as early as possible so the DNS has a couple of minutes to settle before we’ll need it for Let’s Encrypt certificate at the end of the script.

# Create a swap file for small memory instances and increase /run
if [ ! -f /swapfile ]; then
	memory=$(free -m | grep "^Mem:" | tr -s " " | cut -d " " -f 2)
	echo "${memory} megabytes of memory detected"
	if [ -z ${memory} ] || [ "0${memory}" -lt "2048" ]; then
		fallocate -l 2G /swapfile
		chmod 600 /swapfile
		mkswap /swapfile >/dev/null
		swapon /swapfile
		echo '/swapfile none swap sw 0 0' >> /etc/fstab
		echo 'tmpfs /run tmpfs rw,nodev,nosuid,size=400M 0 0' >> /etc/fstab
		mount -o remount,rw,nodev,nosuid,size=400M tmpfs /run
		echo "Swap file created"

If the memory of the virtual machine is less than 2GB (or we can’t find out) we’ll create a 2GB swap file and register it on /etc/fstab. The size of default tmpfs mounted at /run also depends on the amount of memory. On a small machine /run may be too small to create backups so we’ll set it to 400M. Tmpfs will be swapped out as needed, so it won’t consume real memory.

# Set the time zone
tz=$(curl -fs -H "Metadata-Flavor: Google" "http://metadata.google.internal/computeMetadata/v1/instance/attributes/timezone")
if [ ${tz} ] && [ -f /usr/share/zoneinfo/${tz} ]; then
	apt-get -qq install -y dbus >/dev/null
	if ! systemctl start dbus; then
		echo "Trying to start dbus"
		sleep 15
		systemctl start dbus
	if timedatectl set-timezone $tz; then echo "Localtime set to ${tz}"; fi
	systemctl reload-or-restart rsyslog

If the user supplied a timezone reference we’ll check whether it exists on Debian. To set the timezone properly we need DBus. It is finicky and often won’t start on first try. We retry right away and if needed a third time 15 seconds later. RSysLog is already running but it needs to be made aware of the new local time.

# Add Unifi to APT sources
if [ ! -f /etc/apt/trusted.gpg.d/unifi-repo.gpg ]; then
	echo "deb http://www.ubnt.com/downloads/unifi/debian stable ubiquiti" > /etc/apt/sources.list.d/unifi.list
	curl -Lfs -o /etc/apt/trusted.gpg.d/unifi-repo.gpg https://dl.ubnt.com/unifi/unifi-repo.gpg
	echo "Unifi added to APT sources";

If the file doesn’t exist we’ll add the Ubiquiti repo and its GPG key to APT sources.

# Add backports if it doesn't exist
release=$(lsb_release -a 2>/dev/null | grep "^Codename:" | cut -f 2)
if [ ${release} ] && [ ! -f /etc/apt/sources.list.d/backports.list ]; then
	cat > /etc/apt/sources.list.d/backports.list <<_EOF
deb http://deb.debian.org/debian/ ${release}-backports main
deb-src http://deb.debian.org/debian/ ${release}-backports main
	echo "Backports (${release}) added to APT sources"

CertBot should be installed from the backports repo. The repo definition needs the codename for the release, so we need to figure it out first. Google provided images should contain this by default, but we’ll check for it just in case.

# Install stuff
if [ ! -f /usr/share/misc/apt-upgraded ]; then
	apt-get -qq update -y >/dev/null
	apt-get -qq upgrade -y >/dev/null
	touch /usr/share/misc/apt-upgraded
	echo "System upgraded"

Update the APT package list and upgrade the system to the latest on the first run only. We’ll leave a flag file in /var/share/misc so the next boots won’t be delayed. Unattended Upgrades will keep the system up to date on a daily basis.

haveged=$(dpkg-query -W --showformat='${Status}\n' haveged 2>/dev/null)
if [ "x${haveged}" != "xinstall ok installed" ]; then 
	if apt-get -qq install -y haveged >/dev/null; then
		echo "Haveged installed"

Haveged provides entropy for the random number generator needed for encryption. Cloud based virtual machines don’t have access to entropy sources like ordinary computers do, which may delay booting 30 minutes or more.

certbot=$(dpkg-query -W --showformat='${Status}\n' certbot 2>/dev/null)
if [ "x${certbot}" != "xinstall ok installed" ]; then
if (apt-get -qq install -y -t ${release}-backports certbot >/dev/null) || (apt-get -qq install -y certbot >/dev/null); then
		echo "CertBot installed"

CertBot is the tool to acquire and renew Let’s Encrypt certificates. According to EFF docs it should be installed from backports, but this occasionally gives errors. We try the backports first and then the default repos if it fails.

unifi=$(dpkg-query -W --showformat='${Status}\n' unifi 2>/dev/null)
if [ "x${unifi}" != "xinstall ok installed" ]; then
	if apt-get -qq install -y unifi >/dev/null; then
		echo "Unifi installed"
	systemctl stop mongodb
	systemctl disable mongodb

UniFi Controller is the beef.

UniFi Controller will install and enable MongoDB, even though the service is never used. UniFi Controller will start its own MongoDB instance instead. By stopping and disabling the stock MongoDB we’ll save some memory and processor cycles.

httpd=$(dpkg-query -W --showformat='${Status}\n' lighttpd 2>/dev/null)
if [ "x${httpd}" != "xinstall ok installed" ]; then
	if apt-get -qq install -y lighttpd >/dev/null; then
		cat > /etc/lighttpd/conf-enabled/10-unifi-redirect.conf <<_EOF
 \$HTTP["scheme"] == "http" {
	\$HTTP["host"] =~ ".*" { 
		url.redirect = (".*" => "https://%0:8443")
		systemctl reload-or-restart lighttpd
		echo "Lighttpd installed"

Install Lighttpd and configure it to redirect plain http requests to https on port 8443. Lighttpd needs to be reloaded after the configuration change.

f2b=$(dpkg-query -W --showformat='${Status}\n' fail2ban 2>/dev/null)
if [ "x${f2b}" != "xinstall ok installed" ]; then 
	if apt-get -qq install -y fail2ban >/dev/null; then
			echo "Fail2Ban installed"
	if [ ! -f /etc/fail2ban/filter.d/unifi-controller.conf ]; then
		cat > /etc/fail2ban/filter.d/unifi-controller.conf <<_EOF
failregex = ^.* Failed .* login for .* from \s*$
		cat > /etc/fail2ban/jail.d/unifi-controller.conf <<_EOF 
filter = unifi-controller 
port = 8443 
logpath = /var/log/unifi/server.log 

# The .local file will be installed in any case cat > /etc/fail2ban/jail.d/unifi-controller.local <<_EOF
enabled  = true
maxretry = 3
bantime  = 3600
findtime = 3600
	systemctl reload-or-restart fail2ban

Fail2Ban will protect the controller from brute-force login attacks. We need three files for it: the filter contains the regex, the jail file contains the generic settings and the local jail file for local setup. The two latter could be combined but I have submitted the first two upstream to Fail2Ban maintainers. If they are accepted they will eventually be installed automatically. Then we only need to create the local jail file.

# Set up unattended upgrades after 04:00 with automatic reboots
if [ ! -f /etc/apt/apt.conf.d/51unattended-upgrades-unifi ]; then
	cat > /etc/apt/apt.conf.d/51unattended-upgrades-unifi <<_EOF
Acquire::AllowReleaseInfoChanges "true";
Unattended-Upgrade::Origins-Pattern {
Unattended-Upgrade::Remove-Unused-Dependencies "true";
Unattended-Upgrade::Automatic-Reboot "true";
	cat > /etc/systemd/system/timers.target.wants/apt-daily-upgrade.timer <<_EOF
Description=Daily apt upgrade and clean activities
	systemctl daemon-reload
	systemctl reload-or-restart unattended-upgrades
	echo "Unattended upgrades set up"

By Google default only the security updates are installed automatically. This file will add all Debian stable updates and Ubiquiti updates to the automatic update schedule. If the update requires a restart or reboot, it will occur after 04:00 local time. This is the reason for setting the timezone.

# Set up automatic repair for broken MongoDB on boot
if [ ! -f /usr/local/sbin/unifidb-repair.sh ]; then
	cat > /usr/local/sbin/unifidb-repair.sh <<_EOF
#! /bin/sh
if ! pgrep mongod; then
	if [ -f /var/lib/unifi/db/mongod.lock ] \
	|| [ -f /var/lib/unifi/db/WiredTiger.lock ] \
	|| [ -f /var/run/unifi/db.needsRepair ] \
	|| [ -f /var/run/unifi/launcher.looping ]; then
		if [ -f /var/lib/unifi/db/mongod.lock ]; then rm -f /var/lib/unifi/db/mongod.lock; fi
		if [ -f /var/lib/unifi/db/WiredTiger.lock ]; then rm -f /var/lib/unifi/db/WiredTiger.lock; fi
		if [ -f /var/run/unifi/db.needsRepair ]; then rm -f /var/run/unifi/db.needsRepair; fi
		if [ -f /var/run/unifi/launcher.looping ]; then rm -f /var/run/unifi/launcher.looping; fi
		echo >> $LOG
		echo "Repairing Unifi DB on \$(date)" >> $LOG
		su -c "/usr/bin/mongod --repair --dbpath /var/lib/unifi/db --logappend --logpath ${MONGOLOG} 2>>$LOG" unifi
	echo "MongoDB is running. Exiting..."
	exit 1
exit 0
	chmod a+x /usr/local/sbin/unifidb-repair.sh

	cat > /etc/systemd/system/unifidb-repair.service <<_EOF
Description=Repair UniFi MongoDB database at boot
Before=unifi.service mongodb.service
	systemctl enable unifidb-repair.service
	echo "Unifi DB autorepair set up"

This section will create a script to in /usr/local/sbin to repair MongoDB if necessary. Then it will create a system.d unit to launch the script at every boot before Unifi and MongoDB are started.

In the script the first condition checks if MongoDB is running. The script should be run by systemd at boot before UniFi is started, but the check is there just in case someone runs this as a command. The script is executable by anyone and /usr/local/sbin is in root’s path so this is a possibility.

If the MongoDB is shut down improperly it will leave lock files, which indicate a repair operation is needed. UniFi controller may add flag files that the startup is looping or the database is broken. If any of these files is found, they are all removed and MongoDB repair is attempted.

At last we’ll create the unit file for a service and enable it so this will be run automatically at boot.

# Set up daily backup to a bucket after 01:00
bucket=$(curl -fs -H "Metadata-Flavor: Google" "http://metadata.google.internal/computeMetadata/v1/instance/attributes/bucket")
if [ ${bucket} ]; then
	cat > /etc/systemd/system/unifi-backup.service <<_EOF
[Unit] Description=Daily backup to ${bucket}
service After=network-online.target
[Service] Type=oneshot
ExecStart=/usr/bin/gsutil rsync -r -d /var/lib/unifi/backup gs://$bucket

	cat > /etc/systemd/system/unifi-backup.timer <<_EOF
Description=Daily backup to ${bucket} timer
	systemctl daemon-reload
	systemctl start unifi-backup.timer
	echo "Backups to ${bucket} set up"

If the user has specified a bucket we’ll create two systemd unit files. One for a timer to run once a day and a service to actually execute the rsync command.

The unit files are rewritten at every boot in case the user has changed the destination bucket. One could argue that there is no point in enabling the service, since it is also started at every boot. Enabling does very little harm and perhaps one day the script isn’t loaded for some error.

# Set up Let's Encrypt
dnsname=$(curl -fs -H "Metadata-Flavor: Google" "http://metadata.google.internal/computeMetadata/v1/instance/attributes/dns-name")
if [ -z ${dnsname} ]; then exit 0; fi

Setting up Let’s Encrypt reliably turned out to be messy. At first we check whether the user supplied a DNS name and if not then we are done. We also set some file paths for shorthand.

if [ ! -f $caroot ]; then
	cat > $caroot <<_EOF

This cross-signed certificate is required for the import process. You should check it matches the original (courtesy of Steve Jenkins).

Next we’ll write a deploy hook that CertBot will automatically run every time it has renewed a certificate. We need to massage and then import the new certificate to Java keystore for UniFi.

if [ ! -d /etc/letsencrypt/renewal-hooks/deploy ]; then
	mkdir -p /etc/letsencrypt/renewal-hooks/deploy
cat > /etc/letsencrypt/renewal-hooks/deploy/unifi <<_EOF 
#! /bin/sh
if [ -e $privkey ] && [ -e $pubcrt ] && [ -e $chain ]; then 
	echo >> $LOG
	echo "Importing new certificate on \$(date)" >> $LOG
	if ! openssl pkcs12 -export \\
	-in $pubcrt \\
	-inkey $privkey \\
	-CAfile $chain \\
	-out \${p12} -passout pass:aircontrolenterprise \\
	-caname root -name unifi >/dev/null ; then
		echo "OpenSSL export failed" >> $LOG
		exit 1
	if ! keytool -delete -alias unifi \\
	-keystore /var/lib/unifi/keystore \\
	-deststorepass aircontrolenterprise >/dev/null ; then
		echo "KeyTool delete failed" >> $LOG
	if ! keytool -importkeystore \\
	-srckeystore \${p12} -srcstoretype PKCS12 \\
	-srcstorepass aircontrolenterprise \\
	-destkeystore /var/lib/unifi/keystore \\
	-deststorepass aircontrolenterprise \\
	-destkeypass aircontrolenterprise \\
	-alias unifi -trustcacerts >/dev/null; then
		echo "KeyTool import failed" >> $LOG
		exit 2
	systemctl stop unifi
	if ! java -jar /usr/lib/unifi/lib/ace.jar import_cert \\
	$pubcrt $chain $caroot >/dev/null; then
		echo "Java import_cert failed" >> $LOG
		systemctl start unifi
		exit 3
	systemctl start unifi
	rm -f \${p12}
	echo "Success" >> $LOG
	echo "Certificate files missing" >> $LOG
	exit 4
chmod a+x /etc/letsencrypt/renewal-hooks/deploy/unifi

There are many steps where this may fail. The exit codes and the log will hint what went wrong. It is still important to restart the UniFi controller to be of any service. The only error which won’t cause premature exit is keytool delete. If it fails, then most of the time the keytool import will also fail, causing exit.

Next we’ll create a script in /usr/local/sbin to run the CertBot. This way we can use it in a timer if we don’t succeed the first time:

cat > /usr/local/sbin/certbotrun.sh <<_EOF 
#! /bin/sh 
extIP=\$(curl -fs -H "Metadata-Flavor: Google" "http://metadata.google.internal/computeMetadata/v1/instance/network-interfaces/0/access-configs/0/external-ip") 
dnsIP=\$(getent hosts ${dnsname} | cut -d " " -f 1) 
echo >> $LOG
echo "CertBot run on \$(date)" >> $LOG
if [ \${extIP} = \${dnsIP} ]; then
	if [ ! -d /etc/letsencrypt/live/${dnsname} ]; then
		systemctl stop lighttpd
		if certbot certonly -d $dnsname --standalone --agree-tos --register-unsafely-without-email >> $LOG; then
			echo "Received certificate for ${dnsname}" >> $LOG
		systemctl start lighttpd
	if /etc/letsencrypt/renewal-hooks/deploy/unifi; then
		systemctl stop certbotrun.timer
		echo "Certificate installed for ${dnsname}" >> $LOG
	echo "No action because ${dnsname} doesn't resolve to ${extIP}" >> $LOG
chmod a+x /usr/local/sbin/certbotrun.sh

First we need to figure out the external IP of the VM and the IP address of the user-supplied DNS name. If they match and we don’t yet have a certificate for the domain, we’ll try. We need to shutdown Lighttpd for the time because CertBot may need port 80 as well. If we succeed we need to run the deploy hook manually, since it is run automatically only for renewals. In case this was a timer job, we’ll stop the timer after success.

If we don’t succeed on the first try, we’ll run the CertBot once an hour as a timer job. This requires two unit files and a systemd reload:

cat > /etc/systemd/system/certbotrun.timer <<_EOF 
[Unit] Description=Run CertBot hourly until success 
[Timer] OnCalendar=hourly 

systemctl daemon-reload cat > /etc/systemd/system/certbotrun.service <<_EOF
Description=Run CertBot hourly until success

This short part will set the Let’s Encrypt machinery rolling:

if [ ! -d /etc/letsencrypt/live/${dnsname} ]; then
	if ! /usr/local/sbin/certbotrun.sh; then
		echo "Installing hourly CertBot run"
		systemctl start certbotrun.timer

If there is no certificate yet, we’ll run the script manually. If it fails, we’ll start the timer.

List of files

These files are created or modified by the script. You can edit these to your needs once the system is up. In the order of appearance:

  • /etc/logrotate.d/gcp-unifi.conf
  • /etc/logrotate.d/unifi-mongod.conf
  • /etc/sysctl.d/20-disableIPv6.conf
  • /etc/fstab
  • /etc/apt/sources.list.d/unifi.list
  • /etc/apt/sources.list.d/backports.list
  • /etc/lighttpd/conf-enabled/10-unifi-redirect.conf
  • /etc/fail2ban/filter.d/unifi-controller.conf
  • /etc/fail2ban/jail.d/unifi-controller.conf
  • /etc/fail2ban/jail.d/unifi-controller.local
  • /etc/apt/apt.conf.d/51unattended-upgrades-unifi
  • /etc/systemd/system/timers.target.wants/apt-daily-upgrade.timer
  • /usr/local/sbin/unifidb-repair.sh
  • /etc/systemd/system/unifidb-repair.service
  • /etc/systemd/system/unifi-backup.service (*)
  • /etc/systemd/system/unifi-backup.timer (*)
  • # /var/lib/unifi/system.properties (*)
  • /etc/letsencrypt/renewal-hooks/deploy/unifi (*)
  • /usr/local/sbin/certbotrun.sh (*)
  • /etc/systemd/system/certbotrun.timer
  • /etc/systemd/system/certbotrun.service

The starred files are rewritten at every boot if bucket or dnsname metadata fields are defined. If you want to edit them you need to remove the startup-script-url from the metadata.
System.properties file will be edited in place for xms and xmx (once Ubiquiti fixes a bug). You can edit other lines as required.


6 thoughts on “GCP Unifi Controller startup script explained”

  1. Could you please put a link to this script to download. I would like to use it and I’m having a hard time setting the script up with it broken up in chunks. I’m getting strange errors. Greatly appreciated!!

    1. Good point. The link was already on the actual how-to page. You should take a look there, but I added the link to the end of the article.

  2. I got an error on this line: “systemctl reload-or-restart unattended-upgrades”

    Failed to reload-or-restart unattended-upgrades.service: Unit unattended-upgrades.service not found.

    When I changed it to: “systemctl reload-or-restart apt-daily-upgrade”

    No error was printed.

    1. I wonder if the unattended-upgrades was installed in the first place. Could you please log in via the SSH button in the Google console and type “apt list –installed” without the quotes. Does it appear in the list?

      I’ll need to test. The packages change once in a while and I haven’t spun up a new VM for a few months. Thanks for the heads up!

    2. I have spun up a couple of test servers but I couldn’t reproduce this. Are you running your controller on Debian Stretch as instructed?

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.