GCP Unifi Controller startup script explained

This is a walkthrough of the Google Cloud Platform virtual machine startup script to launch full featured Unifi Controllers on demand.

See how-to instructions and a video in another article.

This script was written for Google Cloud Platform and Debian. It should be fairly straightforward to port to any other cloud platform. They have a similar way of accessing virtual machine metadata. The only other GCP specific function is gsutil rsync in the backup script.

Porting to other Linux distributions or Unices would take some more effort. If the destination system doesn’t use systemd then the services and timers should be rewritten. The script also uses apt-get to install packages, but that is fairly easy to edit. Ubiquiti only provides a dpkg package for the controller, though.

I have tried to write the script in open style without shortcuts. A lot of the ifs could be streamlined, but I value legibility. The resulting system is also easy to reconfigure, if you are familiar with Unix command line. The list of files is at the end of the article.

#
# Set up logging for unattended scripts and UniFi's MongoDB log
# Variables $LOG and $MONGOLOG are used later on in the script.
#
LOG="/var/log/unifi/gcp-unifi.log"
if [ ! -f /etc/logrotate.d/gcp-unifi.conf ]; then
	cat > /etc/logrotate.d/gcp-unifi.conf <<_EOF
$LOG {
	monthly
	rotate 4
	compress
}
_EOF
	echo "Script logrotate set up"
fi
MONGOLOG="/usr/lib/unifi/logs/mongod.log"
if [ ! -f /etc/logrotate.d/unifi-mongod.conf ]; then
	cat > /etc/logrotate.d/unifi-mongod.conf <<_EOF
$MONGOLOG {
	weekly
	rotate 10
	copytruncate
	delaycompress
	compress
	notifempty
	missingok
}
_EOF
	echo "MongoDB logrotate set up"
fi

Although most users won’t ever log on to the virtual machine, I still find it worthwhile to write a log. This log will only contain unattended script actions like database repairs or certificate updates every 3 months or so. Not much volume, but it is still good practice to set up logrotate. Four months will cover the last certificate update. For some reason Ubiquiti doesn’t set up logrotate for their MongoDB instance so we’ll cover that as well.

#
# Turn off IPv6 for now
#
if [ ! -f /etc/sysctl.d/20-disableIPv6.conf ]; then
	echo "net.ipv6.conf.all.disable_ipv6=1" > /etc/sysctl.d/20-disableIPv6.conf
	sysctl --system > /dev/null
	echo "IPv6 disabled"
fi

I found out that occasionally the UniFi controller will try to bind to IPv6 only. GCP doesn’t route IPv6 to the virtual machines at this time, so we can just as well turn it off.

#
# Update DynDNS as early in the script as possible
#
ddns=$(curl -fs -H "Metadata-Flavor: Google" "http://metadata.google.internal/computeMetadata/v1/instance/attributes/ddns-url")
if [ ${ddns} ]; then
	curl -fs ${ddns}
	echo "Dynamic DNS accessed"
fi

If the user supplied a dynamic DNS service URL we’ll call it as early as possible so the DNS has a couple of minutes to settle before we’ll need it for Let’s Encrypt certificate at the end of the script.

#
# Create a swap file for small memory instances and increase /run
#
if [ ! -f /swapfile ]; then
	memory=$(free -m | grep "^Mem:" | tr -s " " | cut -d " " -f 2)
	echo "${memory} megabytes of memory detected"
	if [ -z ${memory} ] || [ "0${memory}" -lt "2048" ]; then
		fallocate -l 2G /swapfile
		chmod 600 /swapfile
		mkswap /swapfile >/dev/null
		swapon /swapfile
		echo '/swapfile none swap sw 0 0' >> /etc/fstab
		echo 'tmpfs /run tmpfs rw,nodev,nosuid,size=400M 0 0' >> /etc/fstab
		mount -o remount,rw,nodev,nosuid,size=400M tmpfs /run
		echo "Swap file created"
	fi
fi

If the memory of the virtual machine is less than 2GB (or we can’t find out) we’ll create a 2GB swap file and register it on /etc/fstab. The size of default tmpfs mounted at /run also depends on the amount of memory. On a small machine /run may be too small to create backups so we’ll set it to 400M. Tmpfs will be swapped out as needed, so it won’t consume real memory.

#
# Set the time zone
#
tz=$(curl -fs -H "Metadata-Flavor: Google" "http://metadata.google.internal/computeMetadata/v1/instance/attributes/timezone")
if [ ${tz} ] && [ -f /usr/share/zoneinfo/${tz} ]; then
	apt-get -qq install -y dbus >/dev/null
	if ! systemctl start dbus; then
		echo "Trying to start dbus"
		sleep 15
		systemctl start dbus
	fi
	if timedatectl set-timezone $tz; then echo "Localtime set to ${tz}"; fi
	systemctl reload-or-restart rsyslog
fi

If the user supplied a timezone reference we’ll check whether it exists on Debian. To set the timezone properly we need DBus. It is finicky and often won’t start on first try. We retry right away and if needed a third time 15 seconds later. RSysLog is already running but it needs to be made aware of the new local time.

#
# Add Unifi to APT sources
#
if [ ! -f /etc/apt/trusted.gpg.d/unifi-repo.gpg ]; then
	echo "deb http://www.ubnt.com/downloads/unifi/debian stable ubiquiti" > /etc/apt/sources.list.d/unifi.list
	curl -Lfs -o /etc/apt/trusted.gpg.d/unifi-repo.gpg https://dl.ubnt.com/unifi/unifi-repo.gpg
	echo "Unifi added to APT sources";
fi

If the file doesn’t exist we’ll add the Ubiquiti repo and its GPG key to APT sources.

#
# Add backports if it doesn't exist
#
release=$(lsb_release -a 2>/dev/null | grep "^Codename:" | cut -f 2)
if [ ${release} ] && [ ! -f /etc/apt/sources.list.d/backports.list ]; then
	cat > /etc/apt/sources.list.d/backports.list <<_EOF
deb http://deb.debian.org/debian/ ${release}-backports main
deb-src http://deb.debian.org/debian/ ${release}-backports main
_EOF
	echo "Backports (${release}) added to APT sources"
fi

CertBot should be installed from the backports repo. The repo definition needs the codename for the release, so we need to figure it out first. Google provided images should contain this by default, but we’ll check for it just in case.

#
# Install stuff
#
if [ ! -f /usr/share/misc/apt-upgraded ]; then
	apt-get -qq update -y >/dev/null
	apt-get -qq upgrade -y >/dev/null
	touch /usr/share/misc/apt-upgraded
	echo "System upgraded"
fi

Update the APT package list and upgrade the system to the latest on the first run only. We’ll leave a flag file in /var/share/misc so the next boots won’t be delayed. Unattended Upgrades will keep the system up to date on a daily basis.

haveged=$(dpkg-query -W --showformat='${Status}\n' haveged 2>/dev/null)
if [ "x${haveged}" != "xinstall ok installed" ]; then 
	if apt-get -qq install -y haveged >/dev/null; then
		echo "Haveged installed"
	fi
fi

Haveged provides entropy for the random number generator needed for encryption. Cloud based virtual machines don’t have access to entropy sources like ordinary computers do, which may delay booting 30 minutes or more.

certbot=$(dpkg-query -W --showformat='${Status}\n' certbot 2>/dev/null)
if [ "x${certbot}" != "xinstall ok installed" ]; then
if (apt-get -qq install -y -t ${release}-backports certbot >/dev/null) || (apt-get -qq install -y certbot >/dev/null); then
		echo "CertBot installed"
	fi
fi

CertBot is the tool to acquire and renew Let’s Encrypt certificates. According to EFF docs it should be installed from backports, but this occasionally gives errors. We try the backports first and then the default repos if it fails.

unifi=$(dpkg-query -W --showformat='${Status}\n' unifi 2>/dev/null)
if [ "x${unifi}" != "xinstall ok installed" ]; then
	if apt-get -qq install -y unifi >/dev/null; then
		echo "Unifi installed"
	fi
	systemctl stop mongodb
	systemctl disable mongodb
fi

UniFi Controller is the beef.

UniFi Controller will install and enable MongoDB, even though the service is never used. UniFi Controller will start its own MongoDB instance instead. By stopping and disabling the stock MongoDB we’ll save some memory and processor cycles.

httpd=$(dpkg-query -W --showformat='${Status}\n' lighttpd 2>/dev/null)
if [ "x${httpd}" != "xinstall ok installed" ]; then
	if apt-get -qq install -y lighttpd >/dev/null; then
		cat > /etc/lighttpd/conf-enabled/10-unifi-redirect.conf <<_EOF
 \$HTTP["scheme"] == "http" {
	\$HTTP["host"] =~ ".*" { 
		url.redirect = (".*" => "https://%0:8443")
	}
}
_EOF
		systemctl reload-or-restart lighttpd
		echo "Lighttpd installed"
	fi
fi

Install Lighttpd and configure it to redirect plain http requests to https on port 8443. Lighttpd needs to be reloaded after the configuration change.

f2b=$(dpkg-query -W --showformat='${Status}\n' fail2ban 2>/dev/null)
if [ "x${f2b}" != "xinstall ok installed" ]; then 
	if apt-get -qq install -y fail2ban >/dev/null; then
			echo "Fail2Ban installed"
	fi
	if [ ! -f /etc/fail2ban/filter.d/unifi-controller.conf ]; then
		cat > /etc/fail2ban/filter.d/unifi-controller.conf <<_EOF
[Definition]
failregex = ^.* Failed .* login for .* from \s*$
_EOF
		cat > /etc/fail2ban/jail.d/unifi-controller.conf <<_EOF 
[unifi-controller] 
filter = unifi-controller 
port = 8443 
logpath = /var/log/unifi/server.log 
_EOF 
	fi 

# The .local file will be installed in any case cat > /etc/fail2ban/jail.d/unifi-controller.local <<_EOF
[unifi-controller]
enabled  = true
maxretry = 3
bantime  = 3600
findtime = 3600
_EOF
	systemctl reload-or-restart fail2ban
fi

Fail2Ban will protect the controller from brute-force login attacks. We need three files for it: the filter contains the regex, the jail file contains the generic settings and the local jail file for local setup. The two latter could be combined but I have submitted the first two upstream to Fail2Ban maintainers. If they are accepted they will eventually be installed automatically. Then we only need to create the local jail file.

#
# Set up unattended upgrades after 04:00 with automatic reboots
#
if [ ! -f /etc/apt/apt.conf.d/51unattended-upgrades-unifi ]; then
	cat > /etc/apt/apt.conf.d/51unattended-upgrades-unifi <<_EOF
Acquire::AllowReleaseInfoChanges "true";
Unattended-Upgrade::Origins-Pattern {
	"o=Debian,a=stable";
	"c=ubiquiti";
};
Unattended-Upgrade::Remove-Unused-Dependencies "true";
Unattended-Upgrade::Automatic-Reboot "true";
_EOF
	cat > /etc/systemd/system/timers.target.wants/apt-daily-upgrade.timer <<_EOF
[Unit]
Description=Daily apt upgrade and clean activities
After=apt-daily.timer
[Timer]
OnCalendar=4:00
RandomizedDelaySec=30m
Persistent=true
[Install]
WantedBy=timers.target
_EOF
	systemctl daemon-reload
	systemctl reload-or-restart unattended-upgrades
	echo "Unattended upgrades set up"
fi

By Google default only the security updates are installed automatically. This file will add all Debian stable updates and Ubiquiti updates to the automatic update schedule. If the update requires a restart or reboot, it will occur after 04:00 local time. This is the reason for setting the timezone.

#
# Set up automatic repair for broken MongoDB on boot
#
if [ ! -f /usr/local/sbin/unifidb-repair.sh ]; then
	cat > /usr/local/sbin/unifidb-repair.sh <<_EOF
#! /bin/sh
if ! pgrep mongod; then
	if [ -f /var/lib/unifi/db/mongod.lock ] \
	|| [ -f /var/lib/unifi/db/WiredTiger.lock ] \
	|| [ -f /var/run/unifi/db.needsRepair ] \
	|| [ -f /var/run/unifi/launcher.looping ]; then
		if [ -f /var/lib/unifi/db/mongod.lock ]; then rm -f /var/lib/unifi/db/mongod.lock; fi
		if [ -f /var/lib/unifi/db/WiredTiger.lock ]; then rm -f /var/lib/unifi/db/WiredTiger.lock; fi
		if [ -f /var/run/unifi/db.needsRepair ]; then rm -f /var/run/unifi/db.needsRepair; fi
		if [ -f /var/run/unifi/launcher.looping ]; then rm -f /var/run/unifi/launcher.looping; fi
		echo >> $LOG
		echo "Repairing Unifi DB on \$(date)" >> $LOG
		su -c "/usr/bin/mongod --repair --dbpath /var/lib/unifi/db --logappend --logpath ${MONGOLOG} 2>>$LOG" unifi
	fi
else
	echo "MongoDB is running. Exiting..."
	exit 1
fi
exit 0
_EOF
	chmod a+x /usr/local/sbin/unifidb-repair.sh

	cat > /etc/systemd/system/unifidb-repair.service <<_EOF
[Unit]
Description=Repair UniFi MongoDB database at boot
Before=unifi.service mongodb.service
After=network-online.target
Wants=network-online.target
[Service]
Type=oneshot
ExecStart=/usr/local/sbin/unifidb-repair.sh
[Install]
WantedBy=multi-user.target
_EOF
	systemctl enable unifidb-repair.service
	echo "Unifi DB autorepair set up"
fi

This section will create a script to in /usr/local/sbin to repair MongoDB if necessary. Then it will create a system.d unit to launch the script at every boot before Unifi and MongoDB are started.

In the script the first condition checks if MongoDB is running. The script should be run by systemd at boot before UniFi is started, but the check is there just in case someone runs this as a command. The script is executable by anyone and /usr/local/sbin is in root’s path so this is a possibility.

If the MongoDB is shut down improperly it will leave lock files, which indicate a repair operation is needed. UniFi controller may add flag files that the startup is looping or the database is broken. If any of these files is found, they are all removed and MongoDB repair is attempted.

At last we’ll create the unit file for a service and enable it so this will be run automatically at boot.

#
# Set up daily backup to a bucket after 01:00
#
bucket=$(curl -fs -H "Metadata-Flavor: Google" "http://metadata.google.internal/computeMetadata/v1/instance/attributes/bucket")
if [ ${bucket} ]; then
	cat > /etc/systemd/system/unifi-backup.service <<_EOF
[Unit] Description=Daily backup to ${bucket}
service After=network-online.target
Wants=network-online.target
[Service] Type=oneshot
ExecStart=/usr/bin/gsutil rsync -r -d /var/lib/unifi/backup gs://$bucket
_EOF

	cat > /etc/systemd/system/unifi-backup.timer <<_EOF
[Unit]
Description=Daily backup to ${bucket} timer
[Timer]
OnCalendar=1:00
RandomizedDelaySec=30m
[Install]
WantedBy=timers.target
_EOF
	systemctl daemon-reload
	systemctl start unifi-backup.timer
	echo "Backups to ${bucket} set up"
fi

If the user has specified a bucket we’ll create two systemd unit files. One for a timer to run once a day and a service to actually execute the rsync command.

The unit files are rewritten at every boot in case the user has changed the destination bucket. One could argue that there is no point in enabling the service, since it is also started at every boot. Enabling does very little harm and perhaps one day the script isn’t loaded for some error.

#
# Set up Let's Encrypt
#
dnsname=$(curl -fs -H "Metadata-Flavor: Google" "http://metadata.google.internal/computeMetadata/v1/instance/attributes/dns-name")
if [ -z ${dnsname} ]; then exit 0; fi
privkey=/etc/letsencrypt/live/${dnsname}/privkey.pem
pubcrt=/etc/letsencrypt/live/${dnsname}/cert.pem
chain=/etc/letsencrypt/live/${dnsname}/chain.pem
caroot=/usr/share/misc/ca_root.pem

Setting up Let’s Encrypt reliably turned out to be messy. At first we check whether the user supplied a DNS name and if not then we are done. We also set some file paths for shorthand.

if [ ! -f $caroot ]; then
	cat > $caroot <<_EOF
-----BEGIN CERTIFICATE-----
MIIDSjCCAjKgAwIBAgIQRK+wgNajJ7qJMDmGLvhAazANBgkqhkiG9w0BAQUFADA/
MSQwIgYDVQQKExtEaWdpdGFsIFNpZ25hdHVyZSBUcnVzdCBDby4xFzAVBgNVBAMT
DkRTVCBSb290IENBIFgzMB4XDTAwMDkzMDIxMTIxOVoXDTIxMDkzMDE0MDExNVow
PzEkMCIGA1UEChMbRGlnaXRhbCBTaWduYXR1cmUgVHJ1c3QgQ28uMRcwFQYDVQQD
Ew5EU1QgUm9vdCBDQSBYMzCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEB
AN+v6ZdQCINXtMxiZfaQguzH0yxrMMpb7NnDfcdAwRgUi+DoM3ZJKuM/IUmTrE4O
rz5Iy2Xu/NMhD2XSKtkyj4zl93ewEnu1lcCJo6m67XMuegwGMoOifooUMM0RoOEq
OLl5CjH9UL2AZd+3UWODyOKIYepLYYHsUmu5ouJLGiifSKOeDNoJjj4XLh7dIN9b
xiqKqy69cK3FCxolkHRyxXtqqzTWMIn/5WgTe1QLyNau7Fqckh49ZLOMxt+/yUFw
7BZy1SbsOFU5Q9D8/RhcQPGX69Wam40dutolucbY38EVAjqr2m7xPi71XAicPNaD
aeQQmxkqtilX4+U9m5/wAl0CAwEAAaNCMEAwDwYDVR0TAQH/BAUwAwEB/zAOBgNV
HQ8BAf8EBAMCAQYwHQYDVR0OBBYEFMSnsaR7LHH62+FLkHX/xBVghYkQMA0GCSqG
SIb3DQEBBQUAA4IBAQCjGiybFwBcqR7uKGY3Or+Dxz9LwwmglSBd49lZRNI+DT69
ikugdB/OEIKcdBodfpga3csTS7MgROSR6cz8faXbauX+5v3gTt23ADq1cEmv8uXr
AvHRAosZy5Q6XkjEGB5YGV8eAlrwDPGxrancWYaLbumR9YbK+rlmM6pZW87ipxZz
R8srzJmwN0jP41ZL9c8PDHIyh8bwRLtTcm1D9SZImlJnt1ir/md2cXjbDaJWFBM5
JDGFoqgCWjBH4d1QB7wCCZAA62RjYJsWvIjJEubSfZGL+T0yjWW06XyxV3bqxbYo
Ob8VZRzI9neWagqNdwvYkQsEjgfbKbYK7p2CNTUQ
-----END CERTIFICATE-----
_EOF
fi

This cross-signed certificate is required for the import process. You should check it matches the original (courtesy of Steve Jenkins).

Next we’ll write a deploy hook that CertBot will automatically run every time it has renewed a certificate. We need to massage and then import the new certificate to Java keystore for UniFi.

if [ ! -d /etc/letsencrypt/renewal-hooks/deploy ]; then
	mkdir -p /etc/letsencrypt/renewal-hooks/deploy
fi
cat > /etc/letsencrypt/renewal-hooks/deploy/unifi <<_EOF 
#! /bin/sh
if [ -e $privkey ] && [ -e $pubcrt ] && [ -e $chain ]; then 
	echo >> $LOG
	echo "Importing new certificate on \$(date)" >> $LOG
	p12=\$(mktemp)
	
	if ! openssl pkcs12 -export \\
	-in $pubcrt \\
	-inkey $privkey \\
	-CAfile $chain \\
	-out \${p12} -passout pass:aircontrolenterprise \\
	-caname root -name unifi >/dev/null ; then
		echo "OpenSSL export failed" >> $LOG
		exit 1
	fi
	
	if ! keytool -delete -alias unifi \\
	-keystore /var/lib/unifi/keystore \\
	-deststorepass aircontrolenterprise >/dev/null ; then
		echo "KeyTool delete failed" >> $LOG
	fi
	
	if ! keytool -importkeystore \\
	-srckeystore \${p12} -srcstoretype PKCS12 \\
	-srcstorepass aircontrolenterprise \\
	-destkeystore /var/lib/unifi/keystore \\
	-deststorepass aircontrolenterprise \\
	-destkeypass aircontrolenterprise \\
	-alias unifi -trustcacerts >/dev/null; then
		echo "KeyTool import failed" >> $LOG
		exit 2
	fi
	
	systemctl stop unifi
	if ! java -jar /usr/lib/unifi/lib/ace.jar import_cert \\
	$pubcrt $chain $caroot >/dev/null; then
		echo "Java import_cert failed" >> $LOG
		systemctl start unifi
		exit 3
	fi
	systemctl start unifi
	rm -f \${p12}
	echo "Success" >> $LOG
else
	echo "Certificate files missing" >> $LOG
	exit 4
fi
_EOF
chmod a+x /etc/letsencrypt/renewal-hooks/deploy/unifi

There are many steps where this may fail. The exit codes and the log will hint what went wrong. It is still important to restart the UniFi controller to be of any service. The only error which won’t cause premature exit is keytool delete. If it fails, then most of the time the keytool import will also fail, causing exit.

Next we’ll create a script in /usr/local/sbin to run the CertBot. This way we can use it in a timer if we don’t succeed the first time:

cat > /usr/local/sbin/certbotrun.sh <<_EOF 
#! /bin/sh 
extIP=\$(curl -fs -H "Metadata-Flavor: Google" "http://metadata.google.internal/computeMetadata/v1/instance/network-interfaces/0/access-configs/0/external-ip") 
dnsIP=\$(getent hosts ${dnsname} | cut -d " " -f 1) 
echo >> $LOG
echo "CertBot run on \$(date)" >> $LOG
if [ \${extIP} = \${dnsIP} ]; then
	if [ ! -d /etc/letsencrypt/live/${dnsname} ]; then
		systemctl stop lighttpd
		if certbot certonly -d $dnsname --standalone --agree-tos --register-unsafely-without-email >> $LOG; then
			echo "Received certificate for ${dnsname}" >> $LOG
		fi
		systemctl start lighttpd
	fi
	if /etc/letsencrypt/renewal-hooks/deploy/unifi; then
		systemctl stop certbotrun.timer
		echo "Certificate installed for ${dnsname}" >> $LOG
	fi
else
	echo "No action because ${dnsname} doesn't resolve to ${extIP}" >> $LOG
fi
_EOF
chmod a+x /usr/local/sbin/certbotrun.sh

First we need to figure out the external IP of the VM and the IP address of the user-supplied DNS name. If they match and we don’t yet have a certificate for the domain, we’ll try. We need to shutdown Lighttpd for the time because CertBot may need port 80 as well. If we succeed we need to run the deploy hook manually, since it is run automatically only for renewals. In case this was a timer job, we’ll stop the timer after success.

If we don’t succeed on the first try, we’ll run the CertBot once an hour as a timer job. This requires two unit files and a systemd reload:

cat > /etc/systemd/system/certbotrun.timer <<_EOF 
[Unit] Description=Run CertBot hourly until success 
[Timer] OnCalendar=hourly 
RandomizedDelaySec=15m 
[Install] 
WantedBy=timers.target 
_EOF 

systemctl daemon-reload cat > /etc/systemd/system/certbotrun.service <<_EOF
[Unit]
Description=Run CertBot hourly until success
After=network-online.target
Wants=network-online.target
[Service]
Type=oneshot
ExecStart=/usr/local/sbin/certbotrun.sh
_EOF

This short part will set the Let’s Encrypt machinery rolling:

if [ ! -d /etc/letsencrypt/live/${dnsname} ]; then
	if ! /usr/local/sbin/certbotrun.sh; then
		echo "Installing hourly CertBot run"
		systemctl start certbotrun.timer
	fi
fi

If there is no certificate yet, we’ll run the script manually. If it fails, we’ll start the timer.

List of files

These files are created or modified by the script. You can edit these to your needs once the system is up. In the order of appearance:

  • /etc/logrotate.d/gcp-unifi.conf
  • /etc/logrotate.d/unifi-mongod.conf
  • /etc/sysctl.d/20-disableIPv6.conf
  • /etc/fstab
  • /etc/apt/sources.list.d/unifi.list
  • /etc/apt/sources.list.d/backports.list
  • /etc/lighttpd/conf-enabled/10-unifi-redirect.conf
  • /etc/fail2ban/filter.d/unifi-controller.conf
  • /etc/fail2ban/jail.d/unifi-controller.conf
  • /etc/fail2ban/jail.d/unifi-controller.local
  • /etc/apt/apt.conf.d/51unattended-upgrades-unifi
  • /etc/systemd/system/timers.target.wants/apt-daily-upgrade.timer
  • /usr/local/sbin/unifidb-repair.sh
  • /etc/systemd/system/unifidb-repair.service
  • /etc/systemd/system/unifi-backup.service (*)
  • /etc/systemd/system/unifi-backup.timer (*)
  • # /var/lib/unifi/system.properties (*)
  • /etc/letsencrypt/renewal-hooks/deploy/unifi (*)
  • /usr/local/sbin/certbotrun.sh (*)
  • /etc/systemd/system/certbotrun.timer
  • /etc/systemd/system/certbotrun.service

The starred files are rewritten at every boot if bucket or dnsname metadata fields are defined. If you want to edit them you need to remove the startup-script-url from the metadata.
System.properties file will be edited in place for xms and xmx (once Ubiquiti fixes a bug). You can edit other lines as required.

Links

32 thoughts on “GCP Unifi Controller startup script explained”

  1. Could you please put a link to this script to download. I would like to use it and I’m having a hard time setting the script up with it broken up in chunks. I’m getting strange errors. Greatly appreciated!!

    1. Good point. The link was already on the actual how-to page. You should take a look there, but I added the link to the end of the article.

  2. I got an error on this line: “systemctl reload-or-restart unattended-upgrades”

    Failed to reload-or-restart unattended-upgrades.service: Unit unattended-upgrades.service not found.

    When I changed it to: “systemctl reload-or-restart apt-daily-upgrade”

    No error was printed.

    1. I wonder if the unattended-upgrades was installed in the first place. Could you please log in via the SSH button in the Google console and type “apt list –installed” without the quotes. Does it appear in the list?

      I’ll need to test. The packages change once in a while and I haven’t spun up a new VM for a few months. Thanks for the heads up!

    2. I have spun up a couple of test servers but I couldn’t reproduce this. Are you running your controller on Debian Stretch as instructed?

  3. Just found your script. Thanks, this is awesome. Some changes I made
    This line:
    apt-get -qq install -y unifi
    Changed to:
    apt-get -qq install -y –allow-unauthenticated unifi

    For some reason these directories were not created which caused issues, I added these lines near the top
    if [ ! -d /var/log/unifi ]; then
    mkdir -p /var/log/unifi
    echo “Created /var/log/unifi”
    fi
    if [ ! -d /usr/lib/unifi/logs ]; then
    mkdir -p /usr/lib/unifi/logs
    echo “Created /usr/lib/unifi/logs”
    fi

    1. Thanks! I’ll do some testing and probably add your improvements to the script.

  4. Hi again,
    I enabled StackDriver logging so you can view the system logs from the Cloud Console mobile app or the Logging menu item inside GCP.
    https://cloud.google.com/logging/docs/agent/

    Here is what I added to the script (keep in mind I’m a bash noob):
    if [ ! -f /usr/lib/logging-agent/install-logging-agent.sh ]; then
    echo “Installing StackDriver agent”
    if [ ! -d /usr/lib/logging-agent ]; then
    mkdir -p /usr/lib/logging-agent
    echo “Created /usr/lib/logging-agent”
    fi
    curl -Lfs -o /usr/lib/logging-agent/install-logging-agent.sh https://dl.google.com/cloudagents/install-logging-agent.sh
    bash /usr/lib/logging-agent/install-logging-agent.sh
    echo “StackDriver agent installed”
    fi

    No need to configure the /var/log/messages file that unifi uses as this file is already reported by default.
    I look forward to your updated script. Thanks.

  5. Great script and instructions. I was able to install on my Google Cloud. On the same VM I would like to install UNMS with instructions form this page:

    https://help.ubnt.com/hc/en-us/articles/115012196527

    I run this command:

    curl -fsSL https://unms.com/install > /tmp/unms_inst.sh && sudo bash /tmp/unms_inst.sh

    It complains of needing at least 1GB of RAM. I stopped the VM and increased the RAM to 1GB.

    After running the command again it continues for a good while but then stops with this message:

    Status: Downloaded newer image for ubnt/unms-nginx:0.12.2
    Building docker images.
    fluentd uses an image, skipping
    postgres uses an image, skipping
    redis uses an image, skipping
    unms uses an image, skipping
    rabbitmq uses an image, skipping
    nginx uses an image, skipping
    Checking available ports
    Creating docker-compose.yml
    Deploying templates
    Writing config file
    no crontab for unms
    Deleting old firmwares from /home/unms/data/firmwares/unms/*
    Starting docker containers.
    Creating network “unms_internal” with the default driver
    Creating network “unms_public” with the default driver
    Creating unms-fluentd
    Creating unms-rabbitmq
    Creating unms-redis
    Creating unms-postgres
    Creating unms
    Creating unms-nginx
    ERROR: for nginx Cannot start service nginx: driver failed programming external connectivit
    y on endpoint unms-nginx (735ee03658a75f1e494cafe47af542df3f83366bc5ce0114408a9e709cb00394):
    Error starting userland proxy: listen tcp 0.0.0.0:80: bind: address already in use
    ERROR: Encountered errors while bringing up the project.
    Failed to start docker containers

    What can I do to fix this error?

    1. That is the Lighttpd reserving port 80. You can remove Lighttpd with commands sudo systemctl stop lighttpd and sudo systemctl disable lighttpd. The first one will stop the currently running Lighttpd and the second one will prevent Lighttpd from starting after next reboot. Without Ligttpd you’ll need to use :8443 to access your UniFi controller.

      My script was intended for users without Linux or GCP skills to automate setting up their UniFi controller. You can use it as a starting point for your own customized version, but I cannot support all possible variations. You are on your own after you diverge. It isn’t necessarily bad, it is a standard Debian system and you can get support for it as it is.

  6. Got everything setup and running fine and migrated from my existing Cloud Key to it fine.
    But now I can’t directly access the url @ 8443 at all. The redirect from 80 to 8443 works but it never connects. Running netstat -l on the VM shows it’s listening on 8443 but I don’t get the login screen.

    1. Do you mean you first could connect to 8443 when you restored the backup? Then it stopped? In that case the problem isn’t VPC firewall restrictions that came first to my mind. I would suggest you delete the VM and create a new one with the same IP. Troubleshooting VMs is not worthwhile. If you do want to take a look then the controller log is at /var/log/unifi/server.log. Please let me know how it went.

  7. So I borked networking on my GCP controller and SSH times out, here’s what the GCP serial console prints:
    Jan 27 13:56:38 unifi rc.local[644]: Warning: Overwriting existing alias unifi in destination keystore
    [ 11.642732] rc.local[644]: Warning:
    Jan 27 13:56:38 unifi rc.local[644]: Warning:
    [ 11.643488] rc.local[644]: The JKS keystore uses a proprietary format. It is recommended to migrate to PKCS12 which is an industry standard format using “keytool -importkeystore -srckeystore /var/lib/unifi/keystore -destkeystore /var/lib/unifi/keystore -deststoretype pkcs12”.
    Jan 27 13:56:38 unifi rc.local[644]: The JKS keystore uses a proprietary format. It is recommended to migrate to PKCS12 which is an industry standard format using “keytool -importkeystore -srckeystore /var/lib/unifi/keystore -destkeystore /var/lib/unifi/keystore -deststoretype pkcs12”.
    Jan 27 13:56:38 unifi systemd[1]: Stopped unifi.
    [ OK ] Stopped unifi.
    Jan 27 13:56:38 unifi systemd[1]: Starting unifi…
    Starting unifi…
    Jan 27 13:56:38 unifi unifi.init[866]: Starting Ubiquiti UniFi Controller: unifi (already running) failed!
    [ OK ] Started unifi.
    Jan 27 13:56:38 unifi systemd[1]: Started unifi.
    [ OK ] Started /etc/rc.local.
    [ OK ] Started Serial Getty on ttyS0.
    [ OK ] Started Getty on tty1.
    [ OK ] Reached target Login Prompts.
    Jan 27 13:56:38 unifi systemd[1]: Started /etc/rc.local.
    Jan 27 13:56:38 unifi systemd[1]: Started Serial Getty on ttyS0.
    Jan 27 13:56:38 unifi systemd[1]: Started Getty on tty1.
    Jan 27 13:56:38 unifi systemd[1]: Reached target Login Prompts.
    Jan 27 13:56:38 unifi unifi.init[732]: Starting Ubiquiti UniFi Controller: unifi
    Jan 27 13:56:43 unifi instance-setup: ERROR GET request error retrieving metadata. .

    Debian GNU/Linux 9 unifi ttyS0

    unifi login:

    Since I did not set a password, I am unable to login via the serial console. Is there any way I could add a password for the default user via Metadata/GC Console? Any other way I could log back in and fix it?

    1. I don’t know what you mean by borked networking. There is something really odd going on it is definitely not solely a networking issue.

      You should be able log in via the SSH button. GCP will create a temp account and transfer the keys to it. However, I don’t recommend troubleshooting it. My recommendation is to start over. Delete the current instance, go through your VPC settings once more and create a new controller. It will take 15 minutes instead of hours spent resurrecting this. You do have the backup safe, don’t you.

    2. In my case, the VM lost internet connection, my problem was get the config files from SSD disk. What I’ve done?
      1º – I created a new VM with own disk,
      2º – I edit this new VM and attached the old SSD disk as additional disk (not boot).
      3º – I accessed the new VM using SSH and mount the old disk partition using the command: /dev/sdb1 /mnt (in my case was sdb1)
      4º – I opened the folder /mnt and I get what I needed

      1. Well done! However, those steps require Unix/Linux skills that most people lack.

  8. Hey Petri, first of all thanks for putting this together, really appreciate it. That said I’m having a couple of issues getting my APs to inform the controller running in the GC. It appears the controller is not listening on http://server:8080/inform therfore the APs can’t find the controller.

    I have disabled the lighttpd as advised above and also edited the unifi system.properties file. However after issuing a `systemctl start unifi ` command the system.properties file is over-written with the original contents and my changes are lost.

    Would you have any idea what is happening?

    Thanks again, Andy!

    1. Either the controller is not listening or there is a firewall blocking the connection. The former is easy to verify with sudo netstat -tulpn
      Why did you disable lighttpd? (Disabling it won’t hurt anything, though)
      Why and what did you change in system.properties?
      I wrote the script for users without Linux skills so they wouldn’t ever need to log in. I can’t support all possible configurations. I may be able to give you some hints, but mostly you are on your own.

  9. Petri,
    Thank you for great work. Your guide perfectly worked for me in 2 hours migration from on-premise UniFi Controller to GCP. I am an average Linux user.
    My first attempt was 100% following your guide and let the machines do the rest. After long reboot nothing happened, maybe your script could not run.
    2nd attempt is to read through this post and copy paste lines from startup.sh to terminal console. That monkey imitation helped me to understand the whole process of installation.
    3nd attempt was downloading your startup script, chmod +x and fired it. A few little corrections and everything runs smoothly.
    Just a few more commands to change device settings and now I can bury my old Windows Server box with Unifi Controller required restarting and mongod –repair every 6 hours or so.
    I’d like to suggest you should add some lines about java prerequisites unifi does not install it as default required dependencies.
    Thank you,

  10. This is really great. It worked right away. The upload of the backup to restore took quite a while on my slow connection. I wonder if there is a clever way to grab it from a bucket or something..

    Thanks for contributing this code and all of the time you give!

    1. No. The way it works is that your browser sends the backup to the controller. In theory, you could mount the bucket on your PC and upload from there, but in that case your browser would download and upload simultaneously, slowing the process even more.

  11. Petri,

    Thank you for your hard work on this script and tutorial. I have having a problem when restoring my backup file to the new VM. It says the backup is from a newer version than the Controller. My backup is from version 5.11.39, released 19 Aug 2019. How can I upgrade the VM to run that version controller?

    Many thanks and Cheers!
    Matt

    1. The repo is still at 5.10. Usually the release is moved to the repo in a week. I guess this time there has been too many complaints and I guess they are working on a quick fix release.
      You can open an SSH session, download the package file and install it manually to get going. The automatic updating will continue from there when there will be a later release in the repo.

  12. Perhaps look at adding rng-tools5 in addition to haveged.

    rng-tools5 includes a version of rngd that will take advantage of the RDRAND / RDSEED CPU instruction available to GCP VMs to improve the quality of the entropy pool, as compared to just haveged.

  13. Hello! Thank you for the excellent guide. I’ve followed it and all appears to be working except one thing. I’m still running Unifi Controller 5.14.23. It appears that people who have upgraded to the latest version, 6.0.43, have had some very serious issues. I’ve been studying your startup.sh file all night and cannot figure out how to remove the auto-update. Any chance you could post another file without the auto-update setting? I’m new to this but do know how to do updates via ssh. Thank you kindly, Mike

    1. Remove line 214: “c=ubiquiti”;

      The script will install the latest version by default. If you don’t want the controller installed at all then comment out lines 137-139. Then you can install the version of your choice.

  14. That’s exactly what I was looking for! Thank you SO much for the fast response. I’m grateful to have found your work and that you still check on it. Thanks for documenting it in such detail.

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.