A regular question on the go-nuts mailing list, in the #go-nuts IRC channel and on StackOverflow seems to be: how do I run my Go application in the background? Developers eventually reach the stage where they need to deploy something, keep it running, log it and manage crashes. So where to start?
There’s a huge number of options here, but we’ll look at a stable, popular and cross-distro approach called Supervisor. Supervisor is a process management tool that handles restarting, recovering and managing logs, without requiring anything from your application (i.e. no PID files!).
We’re going to assume a basic understanding of the Linux command line, which in this case is understanding how to use a text-editor like vim, emacs or even nano, and the importance of not running your application as root—which I will re-emphasise throughout this article! We’re also going to assume you’re on an Ubuntu 14.04/Debian 7 system (or newer), but I’ve included a section for those on RHEL-based systems.
I should also head off any questions about daemonizing (i.e. the Unix meaning of daemonize) Go applications due to interactions with threaded applications and most systems (aka Issue #227).
Note: I’m well aware of the “built in” options like Upstart (Debian/Ubuntu) and systemd (CentOS/RHEL/Fedora/Arch). I’d even originally wrote this article so that it provided examples for all three options, but it wasn’t opinionated enough and was therefore confusing for newcomers (at whom this article is aimed at).
For what it’s worth, Upstart leans on start-stop-daemon too much for my liking (if you want it to work across versions), and although I really like systemd’s configuration language, my primary systems are running Debian/Ubuntu LTS so it’s not a viable option (until next year!). Supervisor’s cross-platform nature, well documented configuration options and extra features (log rotation, email notification) make it well suited to running production applications (or even just simple side-projects).
I’ve been using Supervisor for a long while now, and I’m a big fan of it’s centralised approach: it will monitor your process, restart it when it crashes, redirect stout to a log file and rotate that all within a single configuration.
There’s no need to write a separate logrotated config, and there’s even a decent web-interface (that you should only expose over authenticated HTTPS!) included. The project itself has been around 2004 and is well maintained.
Anyway, let’s install it. The below will assume Ubuntu 14.04, which has a recent (>= 3.0) version of Supervisor. If you’re running an older version of Ubuntu, or an OS that doesn’t package a recent version of Supervisor, it may be worth installing it via
pip and writing your own Upstart/systemd service file.
$ sudo apt-get install supervisor
Now, we also want our application user to be able to invoke
supervisorctl (the management interface) as necessary, so we’ll need to create a
supervisor group, make our user a member of that group and modify Supervisor’s configuration file to give the supervisor group the correct permissions on the socket.
$ sudo addgroup --system supervisor # i.e. 'sudo adduser deploy supervisor' $ sudo adduser <yourappuser> supervisor $ logout # Log back in and confirm which should now list 'supervisor': $ groups
That’s the group taken care of. Let’s modify the Supervisor configuration file to take this into account:
[unix_http_server] file=/var/run/supervisor.sock chmod=0770 # ensure our group has read/write privs chown=root:supervisor # add our group [supervisord] logfile=/var/log/supervisor/supervisord.log pidfile=/var/run/supervisord.pid childlogdir=/var/log/supervisor [rpcinterface:supervisor] supervisor.rpcinterface_factory = supervisor.rpcinterface:make_main_rpcinterface [supervisorctl] serverurl=unix:///var/run/supervisor.sock [include] files = /etc/supervisor/conf.d/*.conf # default location on Ubuntu
And now we’ll restart Supervisor:
$ sudo service supervisor restart
If it doesn’t restart, check the log with the below:
$ sudo tail /var/log/supervisor/supervisord.log
Typos are the usual culprit here. Otherwise, with the core configuration out of the way, let’s create a configuration for our Go app.
Supervisor is infinitely configurable, but we’ll aim to keep things simple. Note that you will need to modify the configuration below to suit your application: I’ve commented the lines you’ll need to change.
Create a configuration file at the default (Ubuntu) includes directory:
# where 'mygoapp' is the name of your application $ sudo vim /etc/supervisor/conf.d/mygoapp.conf
… and pull in the below:
[program:yourapp] command=/home/yourappuser/bin/yourapp # the location of your app autostart=true autorestart=true startretries=10 user=yourappuser # the user your app should run as (i.e. *not* root!) directory=/srv/www/yourapp.com/ # where your application runs from environment=APP_SETTINGS="/srv/www/yourapp.com/prod.toml" # environmental variables redirect_stderr=true stdout_logfile=/var/log/supervisor/yourapp.log # the name of the log file. stdout_logfile_maxbytes=50MB stdout_logfile_backups=10
Let’s step through it:
useris who we want the application to run as. I typically create a “deploy” user for this purpose. We should never run an Internet-facing application as root, so this is arguably the most important line of our configuration.
logfile_backupshandle log rotation for us. This saves us having to learn another configuration language and keeps our configuration in one place. If your application generates a lot of logs (say, HTTP request logs) then it may be worth pushing maxbytes up a little.
autostartruns our program when supervisord starts (on system boot)
autorestart=truewill restart our application regardless of the exit code.
startretrieswill attempt to restart our application if it crashes.
environmentdefines the environmetal variables to pass to the application. In this case, we tell it where the settings file is (a TOML config file, in my case).
redirect_stderrwill re-direct error output to our log file. You can keep a separate error log if your application generates significant amounts of log data (i.e. HTTP requests) via stdout.
Now, let’s reload Supervisor so it picks up our app’s config file, and check that it’s running as expected:
$ supervisorctl reload $ supervisorctl status yourapp
We should see a “running/started” message and our application should be ready to go. If not, check the logs in
/var/log/supervisor/supervisord.log or run
supervisorctl tail yourapp to show our application logs. A quick Google for the error message will go a long way if you get stuck.
If you’re running CentOS 7 or Fedora 20, the directory layout is a little different than Ubuntu’s (rather, Ubuntu has a non-standard location), so keep that in mind. Specifically:
- The default configuration file lives at
- The includes directory lives at
Otherwise, Supervisor is much the same: you’ll need to install it, create a system group, add your user to the group, and then update the config file and restart the service using
sudo systemctl restart supervisord.
Pretty easy, huh? If you’re using a configuration management tool (i.e. Ansible, Salt, et. al) for your production machines, then it’s easy to automate this completely, and I definitely recommend doing so. Being able to recreate your production environment like-for-like after a failure (or moving hosts, or just for testing) is a Big Deal and worth the time investment.
It’s also easy to see from this guide how easy it is to add more Go applications to Supervisor’s stable: add a new configuration file, reload Supervisor, and off you go. You can choose how aggressive restarts need to be, log rotations and environmental variables on a per-application basis, which is always useful.Posted on 01 August 2014