request / response

a blog about the web, Go, and building things

(written by Matt Silverlock)


HTTP Request Contexts & Go

•••

Alternatively titled map[string]interface.

Request contexts, for those new to the terminology, are typically a way to pass data alongside a HTTP request as it is processed by composable handlers (or middleware). This data could be a user ID, a CSRF token, whether a user is logged in or not—something typically derived from logic that you don’t want to repeat over-and-over again in every handler. If you’ve ever used Django, the request context is synonymous with the request.META dictionary.

As an example:


func CSRFMiddleware(http.Handler) http.Handler {
    return func(w http.ResponseWriter, r *http.Request) {
        maskedToken, err := csrf.GenerateNewToken(r)
        if err != nil {
			http.Error(w, "No good!", http.StatusInternalServerError)
			return
        }

        // How do we pass the maskedToken from here...
    }
}

func MyHandler(w http.ResponseWriter, r *http.Request) {
    // ... to here, without relying on the overhead of a session store,
    // and without either handler being explicitly tied to the other?
    // What about a CSRF token? Or an auth-key from a request header?
    // We certainly don't want to re-write that logic in every handler!
}

There’s three ways that Go’s web libraries/frameworks have attacked the problem of request contexts:

  1. A global map, with *http.Request as the key, mutexes to synchronise writes, and middleware to cleanup old requests (gorilla/context)

  2. A strictly per-request map by creating custom handler types (goji)
  3. Structs, and creating middleware as methods with pointer receivers or passing the struct to your handlers (gocraft/web).

So how do these approaches differ, what are the benefits, and what are the downsides?

Global Context Map

gorilla/context’s approach is the simplest, and the easiest to plug into an existing architecture.

Gorilla actually uses a map[interface{}]interface{}, which means you need to (and should) create types for your keys. The benefit is that you can use any types that support equality as a key; the downside is that you need to implement your keys in advance if you want to avoid any run-time issues with key types.

You also often want to create setters for the types you store in the context map, to avoid littering your handlers with the same type assertions.


import (
    "net/http"
    "github.com/gorilla/context"
)

type contextKey int

// Define keys that support equality.
const csrfKey contextKey = 0
const userKey contextKey = 1

var ErrCSRFTokenNotPresent = errors.New("CSRF token not present in the request context.")

// We'll need a helper function like this for every key:type
// combination we store in our context map else we repeat this
// in every middleware/handler that needs to access the value.
func GetCSRFToken(r *http.Request) (string, error) {
	val, ok := context.GetOk(r, csrfKey)
	if !ok {
		return "", ErrCSRFTokenNotPresent
	}

	token, ok := val.(string)
	if !ok {
		return "", ErrCSRFTokenNotPresent
	}

	return token, nil
}

// A bare-bones example
func CSRFMiddleware(h http.Handler) http.Handler {
    return func(w http.ResponseWriter, r *http.Request) {
        token, err := GetCSRFToken(r)
        if err != nil {
            http.Error(w, "No good!", http.StatusInternalServerError)
            return
        }

        // The map is global, so we just call the Set function
        context.Set(r, csrfKey, token)

        h.ServeHTTP(w, r)
    }
}

func ShowSignupForm(w http.ResponseWriter, r *http.Request) {
	// We'll use our helper function so we don't have to type assert
	// the result in every handler that triggers/handles a POST request.
	csrfToken, err := GetCSRFToken(r)
	if err != nil {
		http.Error(w, "No good!", http.StatusInternalServerError)
		return
	}

	// We can access this token in every handler we wrap with our
	// middleware. No need to set/get from a session multiple times per
	// request (which is slow!)
	fmt.Fprintf(w, "Our token is %v", csrfToken)
}


func main() {
	r := http.NewServeMux()
	r.Handle("/signup", CSRFMiddleware(http.HandlerFunc(ShowSignupForm)))
	// Critical that we call context.ClearHandler here, else
    // we leave old requests in the map.
	http.ListenAndServe("localhost:8000", context.ClearHandler(r))
}

Full Example

The plusses? It’s flexible, loosely coupled, and easy for third party packages to use. You can tie it into almost any net/http application since all you need is access to http.Request—the rest relies on the global map.

The downsides? The global map and its mutexes may result in contention at high loads, and you need to call context.Clear() at the end of every request (i.e. on each handler). Forget to do that (or wrap your top-level server handler) and you’ll open yourself up to a memory leak where old requests remain in the map. If you’re writing middleware that uses gorilla/context, then you need to make sure your package user imports context calls context.ClearHandler on their handlers/router.

Per Request map[string]interface

As another take, Goji provides a request context as part of an (optional) handler type that embeds Go’s usual http.Handler. Because it’s tied to Goji’s (fast) router implementation, it no longer needs to be a global map and avoids the need for mutexes.

Goji provides a web.HandlerFunc type that extends the default http.HandlerFunc with a request context: func(c web.C, w http.ResponseWriter, r *http.Request).


var ErrTypeNotPresent = errors.New("Expected type not present in the request context.")

// A little simpler: we just need this for every *type* we store.
func GetContextString(c web.C, key string) (string, error) {
	val, ok := c.Env[key].(string)
	if !ok {
		return "", ErrTypeNotPresent
	}

	return val, nil
}

// A bare-bones example
func CSRFMiddleware(c *web.C, h http.Handler) http.Handler {
	fn := func(w http.ResponseWriter, r *http.Request) {
		maskedToken, err := GenerateToken(r)
		if err != nil {
			http.Error(w, "No good!", http.StatusInternalServerError)
			return
		}

		// Goji only allocates a map when you ask for it.
        if c.Env == nil {
			c.Env = make(map[string]interface{})
		}

		// Not a global - a reference to the context map
		// is passed to our handlers explicitly.
		c.Env["csrf_token"] = maskedToken

		h.ServeHTTP(w, r)
	}

	return http.HandlerFunc(fn)
}

// Goji's web.HandlerFunc type is an extension of net/http's
// http.HandlerFunc, except it also passes in a request
// context (aka web.C.Env)
func ShowSignupForm(c web.C, w http.ResponseWriter, r *http.Request) {
	// We'll use our helper function so we don't have to type assert
	// the result in every handler.
	csrfToken, err := GetContextString(c, "csrf_token")
	if err != nil {
		http.Error(w, "No good!", http.StatusInternalServerError)
		return
	}

	// We can access this token in every handler we wrap with our
	// middleware. No need to set/get from a session multiple times per
	// request (which is slow!)
	fmt.Fprintf(w, "Our token is %v", csrfToken)
}

Full Example

The biggest immediate gain is the performance improvement, since Goji only allocates a map when you ask it to: there’s no global map with locks. Note that for many applications, your database or template rendering will be the bottleneck (by far), so the “real” impact is likely pretty small, but it’s a sensible touch.

Most useful is that you retain the ability to write modular middleware that doesn’t need further information about your application: if you want to use the request context, you can do so, but for anything else it’s just http.Handler. The downside is that you still need to type assert anything you retrieve from the context, although like gorilla/context we can simplify this by writing helper functions. A map[string]interface{} also restricts us to string keys: simpler for most (myself included), but potentially less flexible for some.

Context Structs

A third approach is to initialise a struct per-request and define our middleware/handler as methods on the struct. The big plus here is type-safety: we explicitly define the fields of our request context, and so we know the type (unless we do something naive like setting a field to interface{}).

Of course, what you gain in type safety you lose in flexibility. You can’t create “modular” middleware that uses the popular func(http.Handler) http.Handler pattern, because that middleware can’t know what your request context struct looks like. It could provide it’s own struct that you embed into yours, but that still doesn’t solve re-use: not ideal. Still, it’s a good approach: no need to type assert things out of interface{}.


import (
	"fmt"
	"log"
	"net/http"

	"github.com/gocraft/web"
)

type Context struct {
	CSRFToken string
	User      string
}

// Our middleware *and* handlers must be defined as methods on our context struct,
// or accept the type as their first argument. This ties handlers/middlewares to our
// particular application structure/design.
func (c *Context) CSRFMiddleware(w web.ResponseWriter, r *web.Request, next web.NextMiddlewareFunc) {
	token, err := GenerateToken(r)
	if err != nil {
		http.Error(w, "No good!", http.StatusInternalServerError)
		return
	}

	c.CSRFToken = token
	next(w, r)
}

func (c *Context) ShowSignupForm(w web.ResponseWriter, r *web.Request) {
	// No need to type assert it: we know the type.
	// We can just use the value directly.
	fmt.Fprintf(w, "Our token is %v", c.CSRFToken)
}

func main() {
	router := web.New(Context{}).Middleware((*Context).CSRFMiddleware)
	router.Get("/signup", (*Context).ShowSignupForm)

	err := http.ListenAndServe(":8000", router)
	if err != nil {
		log.Fatal(err)
	}
}

Full Example

The plus here is obvious: no type assertions! We have a struct with concrete types that we initialise on every request and pass to our middleware/handlers. But the downside is that we can no longer “plug and play” middleware from the community, because it’s not defined on our own context struct.

We could anonymously embed their type into ours, but that starts to become pretty messy and doesn’t help if their fields share the same names as our own. The real solution is to fork and modify the code to accept your struct, at the cost of time/effort. gocraft/web also wraps the ResponseWriter interface/Request struct with its own types, which ties things a little more closely to the framework itself.

How Else?

One suggestion would be to provide a Context field on Go’s http.Request struct, but actually implementing it in a “sane” way that suits the common use case is easier said than done.

The field would likely end up being a map[string]interface{} (or with interface{} as the key). This means that we either need to initialise the map for users—which won’t be useful on all of those requests where you don’t need to use the request context. Or require package users to check that the map is initialised before using it, which can be a big “gotcha” for newbies who will wonder (at first) why their application panics on some requests but not others.

I don’t think these are huge barriers unto themselves, but Go’s strong preference for being clear and understandable—at the cost of a little verbosity now and then—is potentially at odds with this approach. I also don’t believe that having options in the form of third-party packages/frameworks is a Bad Thing either: you choose the approach that best fits your idioms or requirements.

Wrap

So which approach should you choose for your own projects? It’s going to depend on your use-case (as always). Writing a standalone package and want to provide a request context that the package user can easily access? gorilla/context is probably going to be a good fit (just document the need to call ClearHandler!). Writing something from scratch, or have a net/http app you want to extend easily? Goji is easy to drop in. Starting from nothing? gocraft/web’s “inclusive” approach might fit.

Personally, I like Goji’s approach: I don’t mind writing a couple of helpers to type-assert things I commonly store in a request context (CSRF tokens, usernames, etc), and I get to avoid the global map. It’s also easy for me to write middleware that others can plug into their applications (and for me to use theirs). But those are my use cases, so do your research first!


Custom Handlers and Avoiding Globals in Go Web Applications

•••

Note: I’ve also published a second article about custom handlers that builds on the foundations here. Might be worth reading after you finish this!

Go’s net/http package is extremely flexible, thanks to the fact that it centres on the http.Handler interface. Building around an interface gives you the option of both extending the included implementation and keeping it compatible with other packages out in the wild. Given that the default implementation is pretty simple, we’ll look at how we can build our own handler type (to remove error handling repetition), and how to extend it so we can explicitly pass a “context” containing our database pool, template map, a custom logger and so on, letting us remove any reliance on global variables.

Creating Our Custom Handler Type

net/http provides a basic HandlerFunc type that is just func(w http.ResponseWriter, r *http.Request). It’s easy to understand, pervasive, and covers most simple use-cases. But for anything more than that, there’s two immediate “issues”: a) we can’t pass any additional parameters to http.HandlerFunc, and b) we have to repeat a lot of error handling code in each handler. If you’re new to Go it may not seem immediately obvious how to resolve this but still retain compatibility with other HTTP packages, but thankfully it’s an easy problem to solve.

We create our own handler type that satisfies http.Handler (read: it has a ServeHTTP(http.ResponseWriter, *http.Request) method), which allows it to remain compatible with net/http, generic HTTP middleware packages like nosurf, and routers/frameworks like gorilla/mux or Goji.

First, let’s highlight the problem:

func myHandler(w http.ResponseWriter, r *http.Request) {
    session, err := store.Get(r, "myapp")
    if err != nil {
        http.Error(w, http.StatusText(http.StatusInternalServerError), http.StatusInternalServerError)
        return // Forget to return, and the handler will continue on
    }

    id := // get id from URL param; strconv.AtoI it; making sure to return on those errors too...
    post := Post{ID: id}
    exists, err := db.GetPost(&post)
    if err != nil {
        http.Error(w, http.StatusText(http.StatusInternalServerError), http.StatusInternalServerError)
        return // Repeating ourselves again 
    }

    if !exists {
        http.Error(w, http.StatusText(http.StatusNotFound), http.StatusNotFound)
        return // ... and again.
    }

    err = renderTemplate(w, "post.tmpl", post)
    if err != nil {
        // Yep, here too...
    }
}

Things are not only verbose (we have to do this in every handler), but we’re at the risk of a subtle and hard-to-catch bug. If we don’t explicitly return when we encounter an error—such as a serious database error or when a password comparison fails—our handler will continue. At best this might mean we render an empty struct to our template and confuse the user. At worst, this might mean we write a HTTP 401 (Not Authorised) response and then continue to do things that (potentially) only a logged in user should see or be able to do.

Thankfully, we can fix this pretty easily by creating a handler type that returns an explicit error:


type appHandler func(http.ResponseWriter, *http.Request) (int, error)

// Our appHandler type will now satisify http.Handler 
func (fn appHandler) ServeHTTP(w http.ResponseWriter, r *http.Request) {
    if status, err := fn(w, r); err != nil {
        // We could also log our errors centrally:
        // i.e. log.Printf("HTTP %d: %v", err)
        switch status {
        // We can have cases as granular as we like, if we wanted to
        // return custom errors for specific status codes.
        case http.StatusNotFound:
            notFound(w, r)
        case http.StatusInternalServerError:
            http.Error(w, http.StatusText(http.StatusInternalServerError), http.StatusInternalServerError)
        default:
            // Catch any other errors we haven't explicitly handled
            http.Error(w, http.StatusText(http.StatusInternalServerError), http.StatusInternalServerError)
        }
}

func myHandler(w http.ResponseWriter, r *http.Request) (int, error) {
    session, err := store.Get(r, "myapp")
    if err != nil {
        // Much better!
        return http.StatusInternalServerError, err
    }

    post := Post{ID: id}
    exists, err := db.GetPost(&post)
    if err != nil {
        return http.StatusInternalServerError, err
    }

    // We can shortcut this: since renderTemplate returns `error`,
    // our ServeHTTP method will return a HTTP 500 instead and won't 
    // attempt to write a broken template out with a HTTP 200 status.
    // (see the postscript for how renderTemplate is implemented)
    // If it doesn't return an error, things will go as planned.
    return http.StatusOK, renderTemplate(w, "post.tmpl", data)
}

func main() {
	// Cast myHandler to an appHandler
	http.Handle("/", appHandler(myHandler))
	http.ListenAndServe(":8000", nil)
}

This is, of course, nothing new: Andrew Gerrand highlighted a similar approach on the Go blog back in 2011. Our implementation is just an adaptation with a little extra error handling. I prefer to return (int, error) as I find it more idiomatic than returning a concrete type, but you could certainly create your own error type if you wished (but let’s just keep it simple for now).

Extending Our Custom Handler Further

A quick aside: global variables get a lot of hate: you don’t control what can modify them, it can be tricky to track their state, and they may not be suitable for concurrent access. Still, used correctly they can be convenient, and plenty of Go docs & projects lean on them (e.g. here & here). database/sql’s *sql.DB type can be safely used as a global as it represents a pool and is protected by mutexes, maps (i.e. template maps) can be read from (but not written to, of course) concurrently, and session stores take a similar approach to database/sql.

After being inspired by @benbjohnson’s article from last week on structuring Go applications and a debate with a fellow Gopher on Reddit (who takes a similar approach), I figured I’d take a look at my codebase (which has a few globals of the above types) and refactor it to explicitly pass a context struct to my handlers. Most of it was smooth sailing, but there’s a couple of potential pitfalls you can run into if you want your context instance to be available in more than just the handlers themselves.

Here’s the actual global variables I had before:

var (
    decoder   *schema.Decoder
    bufpool   *bpool.Bufferpool
    templates map[string]*template.Template
    db        *sqlx.DB
    store     *redistore.RediStore
    mandrill  *gochimp.MandrillAPI
    twitter   *anaconda.TwitterApi
    log       *log.Logger
    conf      *config // app-wide configuration: hostname, ports, etc.
)

So, given the custom handler type we created above, how can we turn this list of global variables into a context we can pass to our handlers and our ServeHTTP method—which may want to access our template map to render “pretty” errors or our custom logger—and still keep everything compatible with http.Handler?

package main

import (
	"fmt"
	"log"
	"net/http"

	"html/template"

	"github.com/gorilla/sessions"
	"github.com/jmoiron/sqlx"
	"github.com/zenazn/goji/graceful"
	"github.com/zenazn/goji/web"
)

// appContext contains our local context; our database pool, session store, template
// registry and anything else our handlers need to access. We'll create an instance of it
// in our main() function and then explicitly pass a reference to it for our handlers to access.
type appContext struct {
	db        *sqlx.DB
	store     *sessions.CookieStore
	templates map[string]*template.Template
    decoder *schema.Decoder
    // ... and the rest of our globals.
}

// We've turned our original appHandler into a struct with two fields:
// - A function type similar to our original handler type (but that now takes an *appContext)
// - An embedded field of type *appContext
type appHandler struct {
	*appContext
	H func(*appContext, http.ResponseWriter, *http.Request) (int, error)
}

// Our ServeHTTP method is mostly the same, and also has the ability to
// access our *appContext's fields (templates, loggers, etc.) as well.
func (ah appHandler) ServeHTTP(w http.ResponseWriter, r *http.Request) {
    // Updated to pass ah.appContext as a parameter to our handler type.
    status, err := ah.H(ah.appContext, w, r)
    if err != nil {
        log.Printf("HTTP %d: %q", status, err)
        switch status {
        case http.StatusNotFound:
            http.NotFound(w, r)
            // And if we wanted a friendlier error page, we can
            // now leverage our context instance - e.g.
            // err := ah.renderTemplate(w, "http_404.tmpl", nil)
        case http.StatusInternalServerError:
            http.Error(w, http.StatusText(status), status)
        default:
            http.Error(w, http.StatusText(status), status)
        }
    }
}

func main() {
    // These are 'nil' for our example, but we'd either assign
    // the values as below or use a constructor function like
    // (NewAppContext(conf config) *appContext) that initialises
    // it for us based on our application's configuration file.
	context := &appContext{db: nil, store: nil} // Simplified for this example

	r := web.New()
	// We pass an instance to our context pointer, and our handler.
    r.Get("/", appHandler{context, IndexHandler})

	graceful.ListenAndServe(":8000", r)
}

func IndexHandler(a *appContext, w http.ResponseWriter, r *http.Request) (int, error) {
    // Our handlers now have access to the members of our context struct.
    // e.g. we can call methods on our DB type via err := a.db.GetPosts()
	fmt.Fprintf(w, "IndexHandler: db is %q and store is %q", a.db, a.store)
	return 200, nil
}

Everything still remains very readable: we lean on the type system and existing interfaces, and if we just want to use a regular http.HandlerFunc, we can do that too. Our handlers are still wrappable by anything that takes (and spits out) a http.Handler, and if we wanted to ditch Goji and use gorilla/mux or even just net/http, we don’t have to change our handler at all. Just make sure that your context’s fields are safe for concurrent access. Putting a map in there that requests write to would not be, for example: you’ll need a mutex from the sync package for that.

Other than that, it just works. We’ve reduced repetition around our error handling, we’ve removed our reliance on globals and our code is still readable.

Addendum

  • Worth reading is Justinas’ great article on errors in Go: read the section on implementing a custom httpError.
  • Writing some HTTP middleware for your Go application? Align with func(http.Handler) http.Handler and you’ll end up with something portable. The only “exception” to this rule is when you need to pass state between handlers (i.e. a CSRF token), which is when you’ll need to tie yourself to a request context (like Goji’s web.C, or gorilla/context). Plenty of middleware doesn’t need to do that however.
  • This is how you would catch errors before rendering templates (in short: use a buffer pool).
  • There’s a compilable version of the final example that you can leave comments on.

Approximating html/template Inheritance

•••

Go’s html/template package is fairly minimal compared to templating packages associated with other languages (Jinja, Mustache, even Django’s templates), although it makes up for this with security and great docs.

There are however a few “tricks” to using it: specifically when it comes to approximating template inheritance (aka “nesting”). Being able to specify a base layout (or layouts), stake out your blocks and then fill those blocks with template snippets isn’t immediately clear. So how do we do this?

First, we define base.tmpl:


{{ define "base" }}
<html>
<head>
    {{ template "title" . }}
</head>
<body>
    {{ template "scripts" . }}
    {{ template "sidebar" . }}
    {{ template "content" . }}
<footer>
    ...
</footer>
</body>
</html>
{{ end }}
// We define empty blocks for optional content so we don't have to define a block in child templates that don't need them
{{ define "scripts" }}{{ end }}
{{ define "sidebar" }}{{ end }}

And index.tmpl, which effectively extends our base template.



{{ define "title"}}<title>Index Page</title>{{ end }}
// Notice the lack of the script block - we don't need it here.
{{ define "sidebar" }}
    // We have a two part sidebar that changes depending on the page
    {{ template "sidebar_index" }} 
    {{ template "sidebar_base" }}
{{ end }}
{{ define "content" }}
    {{ template "listings_table" . }}
{{ end }}


Note that we don’t need to define all blocks in the base layout: we’ve “cheated” a little by defining them alongside our base template. The trick is ensure that the {{ define }} blocks in the base template are empty. If you define two blocks and both have content, the application will panic when it attempts to parse the template files (on startup, most likely). There’s no “default” content we can fall back on. It’s not a a deal-breaker, but it’s worth remembering when writing these out.

In our Go application, we create a map of templates by parsing the base template, any necessary snippets, and the template that extends our base template. This is best done at appication start-up (and panics are okay here) so we can fail early. A web application with broken templates is probably not much of a web application.

It’s also critical that we ensure any look-ups on map keys (template names) that don’t exist are caught (using the comma-ok idiom): otherwise it’s a run-time panic.


import (
    "fmt"
    "html/template"
    "net/http"
    "path/filepath"
)

var templates map[string]*template.Template

// Load templates on program initialisation
func init() {
	if templates == nil {
		templates = make(map[string]*template.Template)
	}

	templatesDir := config.Templates.Path

	layouts, err := filepath.Glob(templatesDir + "layouts/*.tmpl")
	if err != nil {
		log.Fatal(err)
	}

	includes, err := filepath.Glob(templatesDir + "includes/*.tmpl")
	if err != nil {
		log.Fatal(err)
	}

    // Generate our templates map from our layouts/ and includes/ directories
	for _, layout := range layouts {
		files := append(includes, layout)
		templates[filepath.Base(layout)] = template.Must(template.ParseFiles(files...))
	}

}

// renderTemplate is a wrapper around template.ExecuteTemplate.
func renderTemplate(w http.ResponseWriter, name string, data map[string]interface{}) error {
	// Ensure the template exists in the map.
	tmpl, ok := templates[name]
	if !ok {
		return fmt.Errorf("The template %s does not exist.", name)
	}

	w.Header().Set("Content-Type", "text/html; charset=utf-8")
	return tmpl.ExecuteTemplate(w, "base.tmpl", data)
}

We create our templates from a set of template snippets and the base layout (just the one, in our case). We can fill in our {{ template "script" }} block as needed, and we can mix and match our sidebar content as well. If your pages are alike, you can generate this map with a range clause by using a slice of the template names as the keys.

Error Handling

Slightly tangential to this, there’s the common problem of dealing with the error returned from template.ExecuteTemplate. If we pass the writer to an error handler, it’s too late: we’ve already written (partially) to the response and we’ll end up with a mess in the user’s browser. It’ll be part of the page before it hit the error, and then the error page’s content. The solution here is to write to a bytes.Buffer to catch any errors during the template rendering, and then write out the contents of the buffer to the http.ResponseWriter.

Although you can create your own buffer per-request, using a pool (https://github.com/oxtoacart/bpool) reduces allocations and garbage. I benchmarked and profiled a bare approach (as above; write out directly), a 10K fixed buffer per-request (big enough for most of my responses), and a pool of buffers. The pooled approach was the fastest, at 32k req/s vs. 26k req/s (fixed buffer) and 29k req/s (no buffer). Latency was no worse than the bare approach either, which is a huge plus.


import (
    "fmt"
    "html/template"
    "net/http"

    "github.com/oxtoacart/bpool"
)

var bufpool *bpool.BufferPool

// renderTemplate is a wrapper around template.ExecuteTemplate.
// It writes into a bytes.Buffer before writing to the http.ResponseWriter to catch
// any errors resulting from populating the template.
func renderTemplate(w http.ResponseWriter, name string, data map[string]interface{}) error {
	// Ensure the template exists in the map.
	tmpl, ok := templates[name]
	if !ok {
		return fmt.Errorf("The template %s does not exist.", name)
	}

	// Create a buffer to temporarily write to and check if any errors were encounted.
	buf := bufpool.Get()
	defer bufpool.Put(buf)
    
	err := tmpl.ExecuteTemplate(buf, "base.tmpl", data)
	if err != nil {
		return err
	}

	// Set the header and write the buffer to the http.ResponseWriter
	w.Header().Set("Content-Type", "text/html; charset=utf-8")
	buf.WriteTo(w)
	return nil
}

func init() {
	...
	bufpool = bpool.NewBufferPool(64)
	...
}

We can catch that returned error in our handler and return a HTTP 500 instead. The best part is that it also makes testing our handlers easier. If you try to take over the http.ResponseWriter with your error handler, you’ve already sent a HTTP 200 status header, making it much harder to test where things are broken. By writing to a temporary buffer first, we ensure that don’t set headers until we’re sure the template will render correctly; making testing much simpler.

And that’s about it. We have composable templates, we deal with our errors before writing out, and it’s still fast.

Postscript

  • This post was triggered after I asked the question on the /r/golang sub-reddit, which is what prompted me to look at re-using buffers via a pool.
  • Credit goes to this answer on SO for the clever map[string]*template.Template approach, and a thanks to @jonathanbingram for the great “optional blocks” trick.
  • I highly suggest reading Jan Newmarch’s html/template tutorial, which covers {{ with }}, {{ range . }} and template.Funcs comprehensively.

httpauth - Basic Auth Middleware For Go

•••

httpauth is a HTTP Basic Authentication middleware for Go.

I originally designed it for the Goji micro-framework, but it’s compatible with vanilla net/http. We can thank Go’s http.Handler interface for that, but I’d recommend Alice to minimise the function wrapping if you’re particularly framework adverse.

package main

import(
    "net/http"
    "github.com/goji/httpauth"
    "github.com/zenazn/goji"
)

func main() {

    goji.Use(httpauth.SimpleBasicAuth("dave", "password"))
    goji.Use(SomeOtherMiddleware)
    // myHandler requires HTTP Basic Auth to access
    goji.Get("/thing", myHandler)
    
    goji.Serve()
}

As always, note that HTTP Basic Auth credentials are sent over the wire in plain-text, so serve your application over HTTPS (TLS) using Go’s built-in ListenAndServeTLS or nginx up front.

Full examples are in the README, and I’m open to any pull requests.


Generating Secure Random Numbers Using crypto/rand

•••

You’re writing an application and you need to generate some session keys, CSRF tokens, and HMACs. For all of these activities, you need a crytographically secure pseudo-random number generator (CSPRNG). Or, in other words, a source of random bytes that is unpredictable and without bias. Specifically, you want to mitigate the chance that an attacker can predict future tokens based on previous tokens.

So what does this mean? No math.Rand, no time.UnixNano; in fact, it means that you only (ever) use the system CSPRNG that your operating system provides. This means using /dev/urandom/ and Windows’ CryptGenRandom API.

Go’s crypto/rand package, thankfully, abstracts these implementation details away to minimise the risk of getting it wrong:


import(
	"crypto/rand"
	"encoding/base64"
)

// GenerateRandomBytes returns securely generated random bytes. 
// It will return an error if the system's secure random
// number generator fails to function correctly, in which
// case the caller should not continue.
func GenerateRandomBytes(n int) ([]byte, error) {
	b := make([]byte, n)
	_, err := rand.Read(b)
    // Note that err == nil only if we read len(b) bytes.
	if err != nil {
		return nil, err
	}

	return b, nil
}

// GenerateRandomString returns a URL-safe, base64 encoded
// securely generated random string.
// It will return an error if the system's secure random
// number generator fails to function correctly, in which
// case the caller should not continue.
func GenerateRandomString(s int) (string, error) {
	b, err := GenerateRandomBytes(s)
	return base64.URLEncoding.EncodeToString(b), err
}

// Example: this will give us a 44 byte, base64 encoded output
token, err := GenerateRandomString(32)
if err != nil {
	// Serve an appropriately vague error to the
    // user, but log the details internally.
}

We have two functions here:

  • GenerateRandomBytes is useful when we need the raw bytes for another cryptographic function, such as HMAC keys.
  • GenerateRandomString wraps this and generates keys we can use for session IDs, CSRF tokens and the like. We base64 URL encode the output in order to provide secure strings that we can use in file-names, templates, HTTP headers and to minimise encoding overhead compared to hex encoding.

For CSRF tokens and session [cookie] IDs, 32 bytes (256 bits) is more than sufficient and common in a number of large web frameworks. If you’re generating HMAC keys, you’ll need to size it based on which HMAC algorithm you are using, and the same goes for AES.

Note that, critically, we always ensure that we check for the extremely rare case our operating systems CSPRNG might fail us with an error. And if it’s failing, we want to fail too, because that means there’s something seriously wrong with our operating system. Checking and handling errors is also idiomatic Go, and good software practice in general. In this case however it’s definitely worth emphasising the need to both log the error and stop processing.

If you are interested in the mechanics behind CSPRNGs I’d recommend reading Schneier’s Crytography Engineering, and if you’re interested in web security in general, Adam Langley’s blog is worth following—noting that Adam contributes to much of Go’s crytographic code.



© 2017 Matt Silverlock | His other site | Code snippets are MIT licensed | Built with Jekyll