Category: Networking

See how-to instructions and a video in another article.

This script was written for Google Cloud Platform and Debian. It should be fairly straightforward to port to any other cloud platform. They have a similar way of accessing virtual machine metadata. The only other GCP specific function is gsutil rsync in the backup script.

Porting to other Linux distributions or Unices would take some more effort. If the destination system doesn’t use systemd then the services and timers should be rewritten. The script also uses apt-get to install packages, but that is fairly easy to edit. Ubiquiti only provides a dpkg package for the controller, though.

I have tried to write the script in open style without shortcuts. A lot of the ifs could be streamlined, but I value legibility. The resulting system is also easy to reconfigure, if you are familiar with Unix command line. The list of files is at the end of the article.

#
# Set up logging for unattended scripts and UniFi's MongoDB log
# Variables $LOG and $MONGOLOG are used later on in the script.
#
LOG="/var/log/unifi/gcp-unifi.log"
if [ ! -f /etc/logrotate.d/gcp-unifi.conf ]; then
	cat > /etc/logrotate.d/gcp-unifi.conf <<_EOF
$LOG {
	monthly
	rotate 4
	compress
}
_EOF
	echo "Script logrotate set up"
fi
MONGOLOG="/usr/lib/unifi/logs/mongod.log"
if [ ! -f /etc/logrotate.d/unifi-mongod.conf ]; then
	cat > /etc/logrotate.d/unifi-mongod.conf <<_EOF
$MONGOLOG {
	weekly
	rotate 10
	copytruncate
	delaycompress
	compress
	notifempty
	missingok
}
_EOF
	echo "MongoDB logrotate set up"
fi

Although most users won’t ever log on to the virtual machine, I still find it worthwhile to write a log. This log will only contain unattended script actions like database repairs or certificate updates every 3 months or so. Not much volume, but it is still good practice to set up logrotate. Four months will cover the last certificate update. For some reason Ubiquiti doesn’t set up logrotate for their MongoDB instance so we’ll cover that as well.

#
# Turn off IPv6 for now
#
if [ ! -f /etc/sysctl.d/20-disableIPv6.conf ]; then
	echo "net.ipv6.conf.all.disable_ipv6=1" > /etc/sysctl.d/20-disableIPv6.conf
	sysctl --system > /dev/null
	echo "IPv6 disabled"
fi

I found out that occasionally the UniFi controller will try to bind to IPv6 only. GCP doesn’t route IPv6 to the virtual machines at this time, so we can just as well turn it off.

#
# Update DynDNS as early in the script as possible
#
ddns=$(curl -fs -H "Metadata-Flavor: Google" "http://metadata.google.internal/computeMetadata/v1/instance/attributes/ddns-url")
if [ ${ddns} ]; then
	curl -fs ${ddns}
	echo "Dynamic DNS accessed"
fi

If the user supplied a dynamic DNS service URL we’ll call it as early as possible so the DNS has a couple of minutes to settle before we’ll need it for Let’s Encrypt certificate at the end of the script.

#
# Create a swap file for small memory instances and increase /run
#
if [ ! -f /swapfile ]; then
	memory=$(free -m | grep "^Mem:" | tr -s " " | cut -d " " -f 2)
	echo "${memory} megabytes of memory detected"
	if [ -z ${memory} ] || [ "0${memory}" -lt "2048" ]; then
		fallocate -l 2G /swapfile
		chmod 600 /swapfile
		mkswap /swapfile >/dev/null
		swapon /swapfile
		echo '/swapfile none swap sw 0 0' >> /etc/fstab
		echo 'tmpfs /run tmpfs rw,nodev,nosuid,size=400M 0 0' >> /etc/fstab
		mount -o remount,rw,nodev,nosuid,size=400M tmpfs /run
		echo "Swap file created"
	fi
fi

If the memory of the virtual machine is less than 2GB (or we can’t find out) we’ll create a 2GB swap file and register it on /etc/fstab. The size of default tmpfs mounted at /run also depends on the amount of memory. On a small machine /run may be too small to create backups so we’ll set it to 400M. Tmpfs will be swapped out as needed, so it won’t consume real memory.

#
# Set the time zone
#
tz=$(curl -fs -H "Metadata-Flavor: Google" "http://metadata.google.internal/computeMetadata/v1/instance/attributes/timezone")
if [ ${tz} ] && [ -f /usr/share/zoneinfo/${tz} ]; then
	apt-get -qq install -y dbus >/dev/null
	if ! systemctl start dbus; then
		echo "Trying to start dbus"
		sleep 15
		systemctl start dbus
	fi
	if timedatectl set-timezone $tz; then echo "Localtime set to ${tz}"; fi
	systemctl reload-or-restart rsyslog
fi

If the user supplied a timezone reference we’ll check whether it exists on Debian. To set the timezone properly we need DBus. It is finicky and often won’t start on first try. We retry right away and if needed a third time 15 seconds later. RSysLog is already running but it needs to be made aware of the new local time.

#
# Add Unifi to APT sources
#
if [ ! -f /etc/apt/trusted.gpg.d/unifi-repo.gpg ]; then
	echo "deb http://www.ubnt.com/downloads/unifi/debian stable ubiquiti" > /etc/apt/sources.list.d/unifi.list
	curl -Lfs -o /etc/apt/trusted.gpg.d/unifi-repo.gpg https://dl.ubnt.com/unifi/unifi-repo.gpg
	echo "Unifi added to APT sources";
fi

If the file doesn’t exist we’ll add the Ubiquiti repo and its GPG key to APT sources.

#
# Add backports if it doesn't exist
#
release=$(lsb_release -a 2>/dev/null | grep "^Codename:" | cut -f 2)
if [ ${release} ] && [ ! -f /etc/apt/sources.list.d/backports.list ]; then
	cat > /etc/apt/sources.list.d/backports.list <<_EOF
deb http://deb.debian.org/debian/ ${release}-backports main
deb-src http://deb.debian.org/debian/ ${release}-backports main
_EOF
	echo "Backports (${release}) added to APT sources"
fi

CertBot should be installed from the backports repo. The repo definition needs the codename for the release, so we need to figure it out first. Google provided images should contain this by default, but we’ll check for it just in case.

#
# Install stuff
#
if [ ! -f /usr/share/misc/apt-upgraded ]; then
	apt-get -qq update -y >/dev/null
	apt-get -qq upgrade -y >/dev/null
	touch /usr/share/misc/apt-upgraded
	echo "System upgraded"
fi

Update the APT package list and upgrade the system to the latest on the first run only. We’ll leave a flag file in /var/share/misc so the next boots won’t be delayed. Unattended Upgrades will keep the system up to date on a daily basis.

haveged=$(dpkg-query -W --showformat='${Status}\n' haveged 2>/dev/null)
if [ "x${haveged}" != "xinstall ok installed" ]; then 
	if apt-get -qq install -y haveged >/dev/null; then
		echo "Haveged installed"
	fi
fi

Haveged provides entropy for the random number generator needed for encryption. Cloud based virtual machines don’t have access to entropy sources like ordinary computers do, which may delay booting 30 minutes or more.

certbot=$(dpkg-query -W --showformat='${Status}\n' certbot 2>/dev/null)
if [ "x${certbot}" != "xinstall ok installed" ]; then
if (apt-get -qq install -y -t ${release}-backports certbot >/dev/null) || (apt-get -qq install -y certbot >/dev/null); then
		echo "CertBot installed"
	fi
fi

CertBot is the tool to acquire and renew Let’s Encrypt certificates. According to EFF docs it should be installed from backports, but this occasionally gives errors. We try the backports first and then the default repos if it fails.

unifi=$(dpkg-query -W --showformat='${Status}\n' unifi 2>/dev/null)
if [ "x${unifi}" != "xinstall ok installed" ]; then
	if apt-get -qq install -y unifi >/dev/null; then
		echo "Unifi installed"
	fi
	systemctl stop mongodb
	systemctl disable mongodb
fi

UniFi Controller is the beef.

UniFi Controller will install and enable MongoDB, even though the service is never used. UniFi Controller will start its own MongoDB instance instead. By stopping and disabling the stock MongoDB we’ll save some memory and processor cycles.

httpd=$(dpkg-query -W --showformat='${Status}\n' lighttpd 2>/dev/null)
if [ "x${httpd}" != "xinstall ok installed" ]; then
	if apt-get -qq install -y lighttpd >/dev/null; then
		cat > /etc/lighttpd/conf-enabled/10-unifi-redirect.conf <<_EOF
 \$HTTP["scheme"] == "http" {
	\$HTTP["host"] =~ ".*" { 
		url.redirect = (".*" => "https://%0:8443")
	}
}
_EOF
		systemctl reload-or-restart lighttpd
		echo "Lighttpd installed"
	fi
fi

Install Lighttpd and configure it to redirect plain http requests to https on port 8443. Lighttpd needs to be reloaded after the configuration change.

f2b=$(dpkg-query -W --showformat='${Status}\n' fail2ban 2>/dev/null)
if [ "x${f2b}" != "xinstall ok installed" ]; then 
	if apt-get -qq install -y fail2ban >/dev/null; then
			echo "Fail2Ban installed"
	fi
	if [ ! -f /etc/fail2ban/filter.d/unifi-controller.conf ]; then
		cat > /etc/fail2ban/filter.d/unifi-controller.conf <<_EOF
[Definition]
failregex = ^.* Failed .* login for .* from \s*$
_EOF
		cat > /etc/fail2ban/jail.d/unifi-controller.conf <<_EOF 
[unifi-controller] 
filter = unifi-controller 
port = 8443 
logpath = /var/log/unifi/server.log 
_EOF 
	fi 

# The .local file will be installed in any case cat > /etc/fail2ban/jail.d/unifi-controller.local <<_EOF
[unifi-controller]
enabled  = true
maxretry = 3
bantime  = 3600
findtime = 3600
_EOF
	systemctl reload-or-restart fail2ban
fi

Fail2Ban will protect the controller from brute-force login attacks. We need three files for it: the filter contains the regex, the jail file contains the generic settings and the local jail file for local setup. The two latter could be combined but I have submitted the first two upstream to Fail2Ban maintainers. If they are accepted they will eventually be installed automatically. Then we only need to create the local jail file.

#
# Set up unattended upgrades after 04:00 with automatic reboots
#
if [ ! -f /etc/apt/apt.conf.d/51unattended-upgrades-unifi ]; then
	cat > /etc/apt/apt.conf.d/51unattended-upgrades-unifi <<_EOF
Acquire::AllowReleaseInfoChanges "true";
Unattended-Upgrade::Origins-Pattern {
	"o=Debian,a=stable";
	"c=ubiquiti";
};
Unattended-Upgrade::Remove-Unused-Dependencies "true";
Unattended-Upgrade::Automatic-Reboot "true";
_EOF
	cat > /etc/systemd/system/timers.target.wants/apt-daily-upgrade.timer <<_EOF
[Unit]
Description=Daily apt upgrade and clean activities
After=apt-daily.timer
[Timer]
OnCalendar=4:00
RandomizedDelaySec=30m
Persistent=true
[Install]
WantedBy=timers.target
_EOF
	systemctl daemon-reload
	systemctl reload-or-restart unattended-upgrades
	echo "Unattended upgrades set up"
fi

By Google default only the security updates are installed automatically. This file will add all Debian stable updates and Ubiquiti updates to the automatic update schedule. If the update requires a restart or reboot, it will occur after 04:00 local time. This is the reason for setting the timezone.

#
# Set up automatic repair for broken MongoDB on boot
#
if [ ! -f /usr/local/sbin/unifidb-repair.sh ]; then
	cat > /usr/local/sbin/unifidb-repair.sh <<_EOF
#! /bin/sh
if ! pgrep mongod; then
	if [ -f /var/lib/unifi/db/mongod.lock ] \
	|| [ -f /var/lib/unifi/db/WiredTiger.lock ] \
	|| [ -f /var/run/unifi/db.needsRepair ] \
	|| [ -f /var/run/unifi/launcher.looping ]; then
		if [ -f /var/lib/unifi/db/mongod.lock ]; then rm -f /var/lib/unifi/db/mongod.lock; fi
		if [ -f /var/lib/unifi/db/WiredTiger.lock ]; then rm -f /var/lib/unifi/db/WiredTiger.lock; fi
		if [ -f /var/run/unifi/db.needsRepair ]; then rm -f /var/run/unifi/db.needsRepair; fi
		if [ -f /var/run/unifi/launcher.looping ]; then rm -f /var/run/unifi/launcher.looping; fi
		echo >> $LOG
		echo "Repairing Unifi DB on \$(date)" >> $LOG
		su -c "/usr/bin/mongod --repair --dbpath /var/lib/unifi/db --logappend --logpath ${MONGOLOG} 2>>$LOG" unifi
	fi
else
	echo "MongoDB is running. Exiting..."
	exit 1
fi
exit 0
_EOF
	chmod a+x /usr/local/sbin/unifidb-repair.sh

	cat > /etc/systemd/system/unifidb-repair.service <<_EOF
[Unit]
Description=Repair UniFi MongoDB database at boot
Before=unifi.service mongodb.service
After=network-online.target
Wants=network-online.target
[Service]
Type=oneshot
ExecStart=/usr/local/sbin/unifidb-repair.sh
[Install]
WantedBy=multi-user.target
_EOF
	systemctl enable unifidb-repair.service
	echo "Unifi DB autorepair set up"
fi

This section will create a script to in /usr/local/sbin to repair MongoDB if necessary. Then it will create a system.d unit to launch the script at every boot before Unifi and MongoDB are started.

In the script the first condition checks if MongoDB is running. The script should be run by systemd at boot before UniFi is started, but the check is there just in case someone runs this as a command. The script is executable by anyone and /usr/local/sbin is in root’s path so this is a possibility.

If the MongoDB is shut down improperly it will leave lock files, which indicate a repair operation is needed. UniFi controller may add flag files that the startup is looping or the database is broken. If any of these files is found, they are all removed and MongoDB repair is attempted.

At last we’ll create the unit file for a service and enable it so this will be run automatically at boot.

#
# Set up daily backup to a bucket after 01:00
#
bucket=$(curl -fs -H "Metadata-Flavor: Google" "http://metadata.google.internal/computeMetadata/v1/instance/attributes/bucket")
if [ ${bucket} ]; then
	cat > /etc/systemd/system/unifi-backup.service <<_EOF
[Unit] Description=Daily backup to ${bucket}
service After=network-online.target
Wants=network-online.target
[Service] Type=oneshot
ExecStart=/usr/bin/gsutil rsync -r -d /var/lib/unifi/backup gs://$bucket
_EOF

	cat > /etc/systemd/system/unifi-backup.timer <<_EOF
[Unit]
Description=Daily backup to ${bucket} timer
[Timer]
OnCalendar=1:00
RandomizedDelaySec=30m
[Install]
WantedBy=timers.target
_EOF
	systemctl daemon-reload
	systemctl start unifi-backup.timer
	echo "Backups to ${bucket} set up"
fi

If the user has specified a bucket we’ll create two systemd unit files. One for a timer to run once a day and a service to actually execute the rsync command.

The unit files are rewritten at every boot in case the user has changed the destination bucket. One could argue that there is no point in enabling the service, since it is also started at every boot. Enabling does very little harm and perhaps one day the script isn’t loaded for some error.

#
# Set up Let's Encrypt
#
dnsname=$(curl -fs -H "Metadata-Flavor: Google" "http://metadata.google.internal/computeMetadata/v1/instance/attributes/dns-name")
if [ -z ${dnsname} ]; then exit 0; fi
privkey=/etc/letsencrypt/live/${dnsname}/privkey.pem
pubcrt=/etc/letsencrypt/live/${dnsname}/cert.pem
chain=/etc/letsencrypt/live/${dnsname}/chain.pem
caroot=/usr/share/misc/ca_root.pem

Setting up Let’s Encrypt reliably turned out to be messy. At first we check whether the user supplied a DNS name and if not then we are done. We also set some file paths for shorthand.

if [ ! -f $caroot ]; then
	cat > $caroot <<_EOF
-----BEGIN CERTIFICATE-----
MIIDSjCCAjKgAwIBAgIQRK+wgNajJ7qJMDmGLvhAazANBgkqhkiG9w0BAQUFADA/
MSQwIgYDVQQKExtEaWdpdGFsIFNpZ25hdHVyZSBUcnVzdCBDby4xFzAVBgNVBAMT
DkRTVCBSb290IENBIFgzMB4XDTAwMDkzMDIxMTIxOVoXDTIxMDkzMDE0MDExNVow
PzEkMCIGA1UEChMbRGlnaXRhbCBTaWduYXR1cmUgVHJ1c3QgQ28uMRcwFQYDVQQD
Ew5EU1QgUm9vdCBDQSBYMzCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEB
AN+v6ZdQCINXtMxiZfaQguzH0yxrMMpb7NnDfcdAwRgUi+DoM3ZJKuM/IUmTrE4O
rz5Iy2Xu/NMhD2XSKtkyj4zl93ewEnu1lcCJo6m67XMuegwGMoOifooUMM0RoOEq
OLl5CjH9UL2AZd+3UWODyOKIYepLYYHsUmu5ouJLGiifSKOeDNoJjj4XLh7dIN9b
xiqKqy69cK3FCxolkHRyxXtqqzTWMIn/5WgTe1QLyNau7Fqckh49ZLOMxt+/yUFw
7BZy1SbsOFU5Q9D8/RhcQPGX69Wam40dutolucbY38EVAjqr2m7xPi71XAicPNaD
aeQQmxkqtilX4+U9m5/wAl0CAwEAAaNCMEAwDwYDVR0TAQH/BAUwAwEB/zAOBgNV
HQ8BAf8EBAMCAQYwHQYDVR0OBBYEFMSnsaR7LHH62+FLkHX/xBVghYkQMA0GCSqG
SIb3DQEBBQUAA4IBAQCjGiybFwBcqR7uKGY3Or+Dxz9LwwmglSBd49lZRNI+DT69
ikugdB/OEIKcdBodfpga3csTS7MgROSR6cz8faXbauX+5v3gTt23ADq1cEmv8uXr
AvHRAosZy5Q6XkjEGB5YGV8eAlrwDPGxrancWYaLbumR9YbK+rlmM6pZW87ipxZz
R8srzJmwN0jP41ZL9c8PDHIyh8bwRLtTcm1D9SZImlJnt1ir/md2cXjbDaJWFBM5
JDGFoqgCWjBH4d1QB7wCCZAA62RjYJsWvIjJEubSfZGL+T0yjWW06XyxV3bqxbYo
Ob8VZRzI9neWagqNdwvYkQsEjgfbKbYK7p2CNTUQ
-----END CERTIFICATE-----
_EOF
fi

This cross-signed certificate is required for the import process. You should check it matches the original (courtesy of Steve Jenkins).

Next we’ll write a deploy hook that CertBot will automatically run every time it has renewed a certificate. We need to massage and then import the new certificate to Java keystore for UniFi.

if [ ! -d /etc/letsencrypt/renewal-hooks/deploy ]; then
	mkdir -p /etc/letsencrypt/renewal-hooks/deploy
fi
cat > /etc/letsencrypt/renewal-hooks/deploy/unifi <<_EOF 
#! /bin/sh
if [ -e $privkey ] && [ -e $pubcrt ] && [ -e $chain ]; then 
	echo >> $LOG
	echo "Importing new certificate on \$(date)" >> $LOG
	p12=\$(mktemp)
	
	if ! openssl pkcs12 -export \\
	-in $pubcrt \\
	-inkey $privkey \\
	-CAfile $chain \\
	-out \${p12} -passout pass:aircontrolenterprise \\
	-caname root -name unifi >/dev/null ; then
		echo "OpenSSL export failed" >> $LOG
		exit 1
	fi
	
	if ! keytool -delete -alias unifi \\
	-keystore /var/lib/unifi/keystore \\
	-deststorepass aircontrolenterprise >/dev/null ; then
		echo "KeyTool delete failed" >> $LOG
	fi
	
	if ! keytool -importkeystore \\
	-srckeystore \${p12} -srcstoretype PKCS12 \\
	-srcstorepass aircontrolenterprise \\
	-destkeystore /var/lib/unifi/keystore \\
	-deststorepass aircontrolenterprise \\
	-destkeypass aircontrolenterprise \\
	-alias unifi -trustcacerts >/dev/null; then
		echo "KeyTool import failed" >> $LOG
		exit 2
	fi
	
	systemctl stop unifi
	if ! java -jar /usr/lib/unifi/lib/ace.jar import_cert \\
	$pubcrt $chain $caroot >/dev/null; then
		echo "Java import_cert failed" >> $LOG
		systemctl start unifi
		exit 3
	fi
	systemctl start unifi
	rm -f \${p12}
	echo "Success" >> $LOG
else
	echo "Certificate files missing" >> $LOG
	exit 4
fi
_EOF
chmod a+x /etc/letsencrypt/renewal-hooks/deploy/unifi

There are many steps where this may fail. The exit codes and the log will hint what went wrong. It is still important to restart the UniFi controller to be of any service. The only error which won’t cause premature exit is keytool delete. If it fails, then most of the time the keytool import will also fail, causing exit.

Next we’ll create a script in /usr/local/sbin to run the CertBot. This way we can use it in a timer if we don’t succeed the first time:

cat > /usr/local/sbin/certbotrun.sh <<_EOF 
#! /bin/sh 
extIP=\$(curl -fs -H "Metadata-Flavor: Google" "http://metadata.google.internal/computeMetadata/v1/instance/network-interfaces/0/access-configs/0/external-ip") 
dnsIP=\$(getent hosts ${dnsname} | cut -d " " -f 1) 
echo >> $LOG
echo "CertBot run on \$(date)" >> $LOG
if [ \${extIP} = \${dnsIP} ]; then
	if [ ! -d /etc/letsencrypt/live/${dnsname} ]; then
		systemctl stop lighttpd
		if certbot certonly -d $dnsname --standalone --agree-tos --register-unsafely-without-email >> $LOG; then
			echo "Received certificate for ${dnsname}" >> $LOG
		fi
		systemctl start lighttpd
	fi
	if /etc/letsencrypt/renewal-hooks/deploy/unifi; then
		systemctl stop certbotrun.timer
		echo "Certificate installed for ${dnsname}" >> $LOG
	fi
else
	echo "No action because ${dnsname} doesn't resolve to ${extIP}" >> $LOG
fi
_EOF
chmod a+x /usr/local/sbin/certbotrun.sh

First we need to figure out the external IP of the VM and the IP address of the user-supplied DNS name. If they match and we don’t yet have a certificate for the domain, we’ll try. We need to shutdown Lighttpd for the time because CertBot may need port 80 as well. If we succeed we need to run the deploy hook manually, since it is run automatically only for renewals. In case this was a timer job, we’ll stop the timer after success.

If we don’t succeed on the first try, we’ll run the CertBot once an hour as a timer job. This requires two unit files and a systemd reload:

cat > /etc/systemd/system/certbotrun.timer <<_EOF 
[Unit] Description=Run CertBot hourly until success 
[Timer] OnCalendar=hourly 
RandomizedDelaySec=15m 
[Install] 
WantedBy=timers.target 
_EOF 

systemctl daemon-reload cat > /etc/systemd/system/certbotrun.service <<_EOF
[Unit]
Description=Run CertBot hourly until success
After=network-online.target
Wants=network-online.target
[Service]
Type=oneshot
ExecStart=/usr/local/sbin/certbotrun.sh
_EOF

This short part will set the Let’s Encrypt machinery rolling:

if [ ! -d /etc/letsencrypt/live/${dnsname} ]; then
	if ! /usr/local/sbin/certbotrun.sh; then
		echo "Installing hourly CertBot run"
		systemctl start certbotrun.timer
	fi
fi

If there is no certificate yet, we’ll run the script manually. If it fails, we’ll start the timer.

List of files

These files are created or modified by the script. You can edit these to your needs once the system is up. In the order of appearance:

• /etc/logrotate.d/gcp-unifi.conf
• /etc/logrotate.d/unifi-mongod.conf
• /etc/sysctl.d/20-disableIPv6.conf
• /etc/fstab
• /etc/apt/sources.list.d/unifi.list
• /etc/apt/sources.list.d/backports.list
• /etc/lighttpd/conf-enabled/10-unifi-redirect.conf
• /etc/fail2ban/filter.d/unifi-controller.conf
• /etc/fail2ban/jail.d/unifi-controller.conf
• /etc/fail2ban/jail.d/unifi-controller.local
• /etc/apt/apt.conf.d/51unattended-upgrades-unifi
• /etc/systemd/system/timers.target.wants/apt-daily-upgrade.timer
• /usr/local/sbin/unifidb-repair.sh
• /etc/systemd/system/unifidb-repair.service
• /etc/systemd/system/unifi-backup.service (*)
• /etc/systemd/system/unifi-backup.timer (*)
• # /var/lib/unifi/system.properties (*)
• /etc/letsencrypt/renewal-hooks/deploy/unifi (*)
• /usr/local/sbin/certbotrun.sh (*)
• /etc/systemd/system/certbotrun.timer
• /etc/systemd/system/certbotrun.service

The starred files are rewritten at every boot if bucket or dnsname metadata fields are defined. If you want to edit them you need to remove the startup-script-url from the metadata.
System.properties file will be edited in place for xms and xmx (once Ubiquiti fixes a bug). You can edit other lines as required.

Links

• How to use this script (with video)
• See version history on GitHub
• Download the current version

[2023/02] If you are impatient and want to upgrade your controller to the latest version, see Jonathan’s comment below. If you want to upgrade your Debian as well, see Derek’s comment. (Use the search in your browser!) I am still waiting for Ubiquiti to come up with an automatic upgrade without the command line.

[2022/12] UniFi Network 7.3 will REQUIRE Java 11. I don’t yet know how smooth the transition will be. I will do my best.

[Note 2021/12] If you are getting a certificate error then see Fix GCP UniFi Controller certificate. It is finally solved [with help from Djo and Derek]! Basically you just need to stop and start your VM.

[Note 2021/09] If you are still being charged for egress traffic, check your network tier. Apparently Premium is on the free offer and Standard is not! [Thanks, gman!]

[Note 2020/01] If Google has charged you for egress traffic to some remote areas of the globe, you can stop Lighttpd as a fix. Log in to the VM and type sudo systemctl stop lighttpd and sudo systemctl disable lighttpd. That means you will need to use the full https://my.ctrl.dns:8443 URL, but your browser should already know it by now.

The script I have created will set up an UniFi Controller on GCP with these extras:

• Automatically acquire and install a Let’s Encrypt certificate to relieve you from HTTPS certificate warnings. For a certificate you need to have a domain name for your controller, dynamic or static.
• Dynamic DNS support to help you set up the domain name.
• You don’t need to type https://your.server.dns:8443 every time. You just type the DNS name of your controller and it will be redirected to HTTPS port 8443 automatically.
• Backup files will be copied to a Google Storage bucket for offline storage.
• Fail2Ban protects your controller from attackers trying to brute force the password: the IP address of the offender will be denied access for an hour after every three failed attempts.
• The free tier GCP micro instance comes with only 600MB of memory. The script will create a swap file to accommodate the controller.
• The underlying Linux system and UniFi Controller will be automatically updated to the latest stable releases.
• If the MongoDB database supporting UniFi Controller needs repairing, a reboot will do the trick. There is a support script to run the repair commands as needed before the controller starts.
• HAVEGEd will store entropy for encryption. Without sufficient entropy available rebooting the server may take 30 minutes or more.

None of this will cost a dime to try out. With a new account you will get credit for the first year. At least you can watch this 10 minute video to see the whole procedure including transferring your current controller:

(There are subtitles available under the YouTube gear icon)

1. Preliminaries

Take a backup of your current controller in Settings > Maintenance. This is always a good idea. You may also want first to improve your password. In Google Cloud your controller can be accessed by anyone who guesses your username and password. You don’t want that. Fail2Ban will lock the IP address for an hour after three failed attempts, but it won’t help if your password is guessable.

A proper SSL certificate requires a DNS name that points to your controller. You can register your own domain or use one of the free ones that dynamic DNS providers offer. If you want to use a dDNS service, you need to set it up first. There are links to some providers at the end. If you control your DNS yourself you won’t need dDNS.

Next you have to create an account on Google Cloud Platform. You will need a credit card for this, but the micro instance that Google offers for free is sufficient for a small UniFi Controller and the free 5GB on Cloud Storage is plenty for the backups. Google does not guarantee this offer will run forever, but it has run for years now and they will certainly let you know if this changes. The current price for a micro instance is $4.28 per month, if you need to run another for example.

First of all, use a good password or better yet, set up two factor authentication for your Google account. If someone breaks into your account the intruder could create a huge bill in just a few hours. You can also set a spending limit in Billing > Budgets & Alerts. Create a new budget for the whole billing account and set a budget. $50 or even $10 should be enough, if you ever want to do some testing on an extra controller for a few hours. The budget feature won’t protect you from an intruder breaking into your account, however. The first thing the intruder would do is to remove the limit.

If you set up budget alerts (50%, 90% and 100% by default) you need to nominate yourself to a billing administrator to receive the alert emails. This is done in IAM & Admin > IAM section. Locate your identity in the list and click in the Roles column. Check Billing > Project Billing Manager to add this role to your identity.

[UPDATE:] Some readers have reported that they needed to enable the storage API in APIs & Services > Dashboard > Enable APIs & Services > search for Google Cloud Storage JSON API > click Google Cloud Storage JSON API tile > click Enable button. My account had this enabled by default, but my account is older and perhaps the default has been changed. [Thank you Limbo and Chrysilis!]

2. Create a Storage Bucket (optional)

UniFi Controller will create backups automatically. The default is once a month, but you can change this in UniFi Controller > Settings > Auto Backup. I run backups once a week and keep the last 26 or a half year’s worth. These backups are stored on the computer the controller runs on. If this computer is lost for some reason, so are the backups. You should store them somewhere else and a Google Storage bucket is the perfect place. If you create a bucket the script will sync the backups to the bucket daily.

In GCP Console switch to Storage > Cloud Storage > Browser and create a new bucket. The bucket name has to be globally unique, so all the obvious ones are already in use. Use something like petri-office-unifi-backup. For the free offer select Region as the Location type and choose a Location in the U.S. I am in Europe and East Coast is marginally closer, so I use it.

3. Set Up the Virtual Network

This is something you will need to do only once. All your future UniFi controllers will use the same network settings.

Switch to Networking > VPC Network > Firewall. Create a new firewall rule and name it allow-unifi for example, the name isn’t important. In the Target tags field enter unifi. You can use any name for the tag, but you will need to remember it for the next step. In the Source IPv4 ranges type 0.0.0.0/0 to denote any address. In the Protocols and ports section select Specified protocols and ports and copy these into the TCP and UDP fields:

tcp: 8443,8080,8880,8843,6789,443,80
udp: 3478

Switch to  VPC Networks > External IP Addresses and click on Reserve Static Address. You need to give the address a name, choose the region where your virtual machine is going to run and click Reserve. I suggest you use the same region as the Storage bucket. You can’t attach the address yet, since you haven’t yet created the virtual machine. (Google will charge for the static addresses while they are not in use. The plan is to run the controller non-stop so this is not an issue. Just remember to release the address if you don’t need it any longer.) If you manage your DNS yourself, add an A record for this IP at this point.

4. Create the Virtual Machine

In Compute > Compute Engine > VM Instances create a new instance. You can name it anything you like. Choose between Regions us-west1, us-central1 or us-east1 for the free offer. You need to place the VM in a Zone that is in the same region as the static address. The Machine type should be E2 series e2-micro for the free instance.

Click on Change for the Boot disk.  The default operating system is Debian  GNU/Linux 10 (Buster), but UniFi Controller is not supported on Buster. Change the version to Debian GNU/Linux 9 (Stretch) [Kudos to James!]. The default size is 10GB which suffices, but it is tricky to expand afterwards. The free offer is 30GB but I prefer to set the boot disk to 15GB in case I need to run another VM. To change the size, click on Change and set the size. Also change the Boot disk type to Standard persistent disk (not Balanced) to stay on the free tier [Thanks Steve!]  and then you are ready to click Select to save.

If you created a bucket for the backups, you need to allow the virtual machine to write to Google Storage. The default permission is Read Only. In Identity and API access choose Set access for each API. In the list that appears change Storage to Read Write.

No need to select the HTTP and HTTPS options in the Firewall section. You created your own firewall rules already.

Click on Networking, disks, security, management, sole-tenancy to open advanced options. In the Networking section add unifi to the Network Tags field. This tag ties the firewall rules you created earlier to this virtual machine instance. This step is easy to forget, but missing it will prevent access to your controller! Leave the hostname empty. Click on Default under Network Interfaces and choose the static IP address you created earlier as the External IP. Set Network Service Tier to Premium [apparently Premium is on the Free Tier. Thanks, gman!].

In the Management section add the following Metadata key/value pairs. The first one is mandatory: the script that does the magic. All the others are optional and the order doesn’t matter. There is no way to do any error checking or report script failures, some feature will just not work if the metadata is erroneous. You can come back and change them if necessary. The changes will take effect after next reboot. See the Maintenance section at the end for how to reboot.

Key	Value	Purpose
startup-script-url	gs://petri-unifi/startup.sh	Required!
ddns-url	http://your.dyn.dns/update.php?key=xxxxx	Helps you access the Controller
timezone	Europe/Helsinki	Lets you adjust the reboot hour
dns-name	your.server.dns	Required for HTTPS certificate
bucket	your-bucket-name-here	Required for offline backups

Some users have reported problems with the gs: style URL for the script. You can try https://storage.googleapis.com/petri-unifi/startup.sh instead [Thank you Miczita]

Click Create. Go get coffee. Even if the virtual machine appears to be running, the script is not done yet. It used to run in minutes, but give it a half an hour these days.

5. Set Up the Controller

Connect to your new controller. On the first screen of the UniFi wizard click restore from a previous backup and upload the latest backup. Wait for the restore to complete. Log on to the new controller using your old username and password. In Settings > Controller add the DNS name or the IP address of the new controller to Controller Hostname/IP field and check Override inform host with controller hostname/IP. Confirm the change. Click Apply Settings.

Connect to your old controller. Do the same change in Settings > Controller and add the DNS name or the IP address to Controller Hostname/IP field and check Override inform host with controller hostname/IP. This setting is for the current devices that are already associated with the current controller. Now they will start to contact the new controller in Google Cloud.

Connect to your new controller again. Check the devices page to see whether the devices come online eventually. If they don’t, you may have to force reprovisions them on the old controller. In that case go to the Devices page on the old controller, select each device in turn and on the Config tab click Provision under Manage Device.

[UPDATE:] The controller doesn’t log failed login attempts by default any more. Logging is required for Fail2Ban protection. You need to change the logging level of the controller: Settings > Maintenance > Services > Log level and change Mgmt to More and click Apply. [Thank you Gabe!]

How to Set Up a New Network with a GCP Controller

If you are starting from scratch you won’t have any controller backup to transfer to the GCP based UniFi Controller. In that case you have three alternatives:

1. Install a temporary controller on a local computer, set up the network and transfer the controller to GCP as outlined above.
2. If you want to do a pure cloud based installation, you’ll need to tell the devices the address of the controller. If you have an UniFi Security Gateway, connect its WAN interface to the Internet and your laptop directly to the LAN interface. Connect to the USG with a browser (https://192.168.1.1) and configure the inform URL to http://your.server.dns:8080/inform. Once you have adopted the USG in the controller other UniFi devices will appear in the controller as you start to connect them.
3. If you have some other router than an USG, you can either set your DNS server to resolve name unifi to the IP address of your cloud controller or set the DHCP option 43 in your DHCP server. There are examples of both in Ubiquiti’s L3 Adoption Guide. The last resort is to SSH to each device in turn and use command
   set-inform http://your.server.dns:8080/inform.

Maintenance and troubleshooting

If your devices won’t switch to the new controller the reason is usually either in the firewall or DNS. Is your VPC firewall rule correct and does the network tag of the virtual machine match the tag on the rule? Is your DNS name correct both in the metadata and in both controllers’ Override Inform Host fields and does the DNS name point to the correct IP address? In either case the devices will try to connect for several minutes before reconnecting to the old controller. Sometimes it takes hours for DNS to settle, depending on the time-to-live settings. In that case let it sit overnight and force provision the devices next morning.

Let’s Encrypt limits the number of certificates issued per domain to five per week (at the time). This usually isn’t an issue, but if you use the free domains offered by dDNS providers, the most popular domains will have used this week’s five certificates already. In this case the script will try to acquire a certificate once an hour until it succeeds. You will see certificate warnings at first but the address bar should eventually turn green. You can also anticipate this and choose a less popular domain (or register your own).

Don’t set up two controllers to back up to the same bucket. They will delete each other’s backups. You can either create a new bucket or create folders inside the bucket. In the metadata set bucket to your-bucket/foldername.

Google blocks sending email directly to SMTP port 25 from within GCP because they don’t want all spammers on board. If you want to receive email notifications you need to set up authenticated SMTP on port 587 or 465 in UniFi’s Settings > Controller > Mail Server. You can use your personal email account or even use Google’s email engine if you can prove you are not a spammer.

The Cloud Console will typically start displaying a recommendation to increase performance by upgrading to a bigger virtual machine (at cost), even though the CPU utilisation graph typically stays below 15%. Apparently the updates the devices send cause very short spikes, because occasionally the devices report that the controller is busy. This may cause some inaccuracies in the reports, I am not certain. If you can live with that, just dismiss the suggestion. In my experience the micro instance can serve at least the same 30 devices a Cloud Key can. You can also use this script on any virtual machine type you like. When you select the machine type you see the cost per month. You get a better price if you can commit to use the resource for 1 or 3 years.

If the controller appears to be malfunctioning the first remedy is to reboot it. A reboot is also the way to re-run the script if you made changes to the metadata. The safe way to reboot is to Stop and then Start the virtual machine. This won’t change the IP address since you have reserved a static address. Don’t use the Reset button. A Reset will immediately restart the virtual machine, which may damage the UniFi database and/or the Linux filesystem (if you are really unlucky).

The underlying Linux system and UniFi Controller will be automatically updated to the latest stable releases. If an update requires a server restart or a reboot it will occur after 04AM UTC. Set the timezone metadata to adjust the reboot to 04AM local time. A reboot will make a captive wireless guest portal inaccessible for a couple of minutes so you don’t want it to happen during peak hours.

Automatically updating the system is a risk. For example there could be an update to Java or MongoDB that is incompatible with the installed UniFi controller. There is no way you could prevent the installation of the update. However, I decided that NOT updating the system is even bigger risk. I don’t trust the intended audience (you) to log on regularly over SSH and run apt-get. In any case, if the risk realizes I trust that it will be transient. The incompatibility will affect a lot of other users as well and there will be another update to solve the issue. This other update will be also automatically installed. If you are familiar with Linux you can also log on the VM and edit /etc/apt/apt.conf.d/51unattended-upgrades-unifi to your requirements.

The automatic update won’t do major Linux version upgrades. Every couple of years it is wise to upgrade the underlying operating system. The Cloud Way is to create a new UniFi controller:

1. Take a backup of the old controller.
2. Create a new bucket (or folder).
3. Remove the static IP address from the old virtual machine (VPC Network > External IP Addresses and Change to None)
4. Create a new virtual machine as outlined above and restore the backup
5. Delete the old virtual machine
6. Done! Since the IP address is the same there is no need to update any settings and the switchover is immediate.

If your controller starts to act up and rebooting doesn’t help, just create a new one. Just like upgrading the underlying Linux, create a new virtual machine. If you cannot connect to the old controller to retrieve the backup, use the latest auto backup in the storage bucket. That’s what they are for. Don’t fall in love with your virtual machine, you should treat them like pencils: very useful but replaceable.

If you are concerned about the security of the script you better read it through first (the link is at the end). Be aware that the script is in my bucket so I can update it as necessary. I could also make malicious changes that would take effect on your server after next boot (after a system update for instance). If you are cautious you should remove the whole startup-script-url row from the metadata as soon as you are happy with the other metadata settings. The script doesn’t do anything once the system is configured. If you ever need to reconfigure something you can just add the startup-script-url back, Stop and Start the VM to run the script and then remove the row again.

Google currently offers for free one micro instance with 30GB of disk space and 5GB of bucket storage in the U.S. regions. Both come with monthly 1GB of free egress (i.e. outbound) data transfer with China and Australia excluded. Egress data is used when you download backups to your computer and when your devices download upgrades from the controller. The static IP address is free as long as it is attached to a running virtual machine. You will be charged if you exceed these limits, but typically the charges are minuscule. For example when you upgrade your Linux you will run another micro instance for a while. The budget feature should protect you from nasty surprises.

Links

• Google Cloud Platform Free Offer
• Debian time zones (1/2018)
• The startup script on Google Storage
• Explanation and tech details of the startup script
• My UniFi project on GitHub
• UniFi – Device Adoption Methods

Some dynamic DNS providers and how their single line update URLs look like:

• Afraid.org
  http://freedns.afraid.org/dynamic/update.php?xxxdynamicTokenxxx
• DNSPark
  https://username:passsword@control.dnspark.com/api/dynamic/update.php?hostname=home.example.com
• DuckDNS
  https://www.duckdns.org/update?domains={YOURVALUE}&token={YOURVALUE}
• DynDNS
  https://{user}:{updater client key}@members.dyndns.org/v3/update?hostname={hostname}
• Dynu [thank you Moses!]
  https://api.dynu.com/nic/update?hostname=HOSTNAME&alias=ALIAS&username=USERNAME&password=PASSWORD
• EasyDNS
  https://username:dynamictoken@api.cp.easydns.com/dyn/generic.php?hostname=example.com
• NameCheap
  https://dynamicdns.park-your-domain.com/update?host=[host]&domain=[domain_name]&password=[ddns_password]
• Sitelutions
  https://dnsup.sitelutions.com/dnsup?id=990331&user=myemail@mydomain.com&pass=SecretPass
• ZoneEdit
  https://<username>:<password>@dynamic.zoneedit.com/auth/dynamic.html?host=<hostname>