Paul Smith

Interview: Mike Migurski

2012-03-30T03:45:00Z

Mike Migurski is partner and Director of Technology at Stamen design studio in San Francisco. He and his Stamen colleagues recently unveiled a new site, maps.stamen.com, that included three custom-rendered maps using data from OpenStreetMap, including a remarkable one that, despite being generated by algorithm from precise vector data, resembled a hand-made watercolor painting.

Paul Smith

You’ve blogged about the process of creating the Terrain map tiles, and the code for the Toner tiles is open-sourced and on GitHub, but the Watercolor tiles seemed to come out of the blue. Can you talk about what inspired them, how you and your Stamen colleagues went about designing them, and what uses you imagined for them?

Mike Migurski

The main use we imagined for them was that they would look really, really good, and unlike anything else being done with online maps. The process started with Eric Rodenbeck and Zach Watson kicking around some ideas on simulating the appearance of watercolor painting algorithmically, and expanded to include Geraldine Sarmiento’s hand-done textures.

We saw an amazing reaction to Aaron Straup Cope’s work on Prettymaps last year, and more generally a lot of interest in map-based art, so it seemed natural to try something that was algorithmic and at the same time had the appearance of being made by hand. I figured people would swallow their tongues in surprise when they saw these, especially when we switched from image search results for painted textures to pictures we were making ourselves for the project.

PS

Making maps these days is often about mechanizing the conversion of vector data, like from OSM, into raster images, which has meant perfecting our ability to layer and style points, lines, and polygons in very precise ways. I think what captivated people about the Watercolor tiles is that they seemed to color outside the lines, so to speak, in a way that didn’t seem possible with current tools. What tools did you use to generate the tiles, and how did you engineer them in a way to achieve this effect? Did you have to write new software to aid you?

MM

We did write new software, yes. We’ve been exploring ways to make OSM data more usable by cartographers over the past few years, and that’s largely been expressed in code designed to make large data sets easier for cartographers (and ourselves) to deal with:

Cascadenik was the first widespread application of CSS-like style rules in this area, since developed and expanded by Development Seed in Carto.
TileStache is a tile-generation server built to deploy custom visualization and rendering code.

I’ve been talking about this stuff for a few years, and the release of maps.stamen.com is intended to be a showcase that helps shift expectations of what good, online cartography should look like. It hasn’t been long since using a solid anti-aliased vector-rendering library like Mapnik was enough to make a strong showing against commercial map providers, but now we’re starting to establish new ideas around texture, color and labeling that require a layer of additional work beyond Mapnik.

Watercolor itself is a custom Provider to TileStache, feeding on simple Mapnik output and adding techniques like gaussian blur and Perlin noise as well as high-quality scans of actual water color paintings to achieve its effect. The development of watercolor had a number of stages, beginning with Zach’s experiments in noise and texture, Eric and George Oates’s encouragement, Geraldine’s addition of color and texture, and proceeding to my work with Jeff Easter and Nathaniel Kelso to improve performance until the code was good enough for public launch. The visual appearance is ridiculously CPU-hungry compared to your typical vector cartography, so we’ve got a few virtual servers on the moon doing the live calculation of new maps in new places.

Without diving too deeply into the actual watercolor code, we owe a lot to Kai Krause’s Algorithmic Painting ideas from almost 20 years ago.

PS

I see that Stamen is releasing these tiles under a Creative Commons license as part of a Knight Foundation-funded project centered around data visualization and cities. City governments are historically big users of enterprise GIS software like ArcGIS; are there any lessons here for government or citizens, or does the work, specifically the Watercolor tiles, speak for themselves as art?

MM

I think the latter: watercolor is basically art. Of course there’s an undeniable political component to the OpenStreetMap data it’s made of. In the U.S. OSM Foundation we’re interested in ways to get government at various levels to participate custodians of OSM, and I think open data and enterprise GIS software can cooperate. Esri puts a lot of effort into OpenStreetMap-related efforts, and I’m not expecting to see their involvement in city governments wane.

PS

Can you expand a bit on the interplay of vector data, raster data, and image filters to produce the tiles? What sort of preliminary work did you need to do to get the components parts to fall into place? Do you see patterns emerging and opportunities for abstraction for future artist map tools?

MM

I probably can’t explain as well as Zach just did.

As a bystander to this part of the process, it seemed like the actual simulation technique took shape in just a few days, based on Zach’s existing experience with image manipulation in Python. The real work came in the knob-twiddling, or as we call it internally, “spring tuning” (based on our experience with the Digg Swarm project from 2007). Here, for example, is one of Zach’s own parameter views that he used to tune noise thresholds for the ground texture.

I think the pattern that’s emerging for me is the raw labor-intensivity of this kind of work, the parameter-tweaking in a space of possible outcomes that results in something that looks right and feels exciting. Once the basic structure of noise, blur and threshold is in place, all you can really do is watch carefully as you repeatedly try combinations until something clicks.

PS

We’re usually talking about algorithms and stylesheets when we talk about web maps. Traditional cartographers often exercise artistic license over data streams as well—for example, manually but subtlety tweaking the curve of a road so that it reflects a shared or colloquial understanding of its location rather than it’s literal location. And then there are the more abstract but functional examples like subway system maps, whose stops and lines are not intended to be scale representations of their real-world counterparts. Do you see it possible for automated cartography to produce maps like these? What techniques would we need to develop (for example, a “hints” file ala typography for manually overriding certain points)?

MM

I’ve not yet seen an attempt at automating this kind of cartography which has resulted in a satisfying outcome, but it’s still the subject of many PhD theses, so maybe it’s just too early. I suspect that we’ll end up seeing is a companion project to OSM where human make decisions about how things should be shown and contribute those to a free and open data source. Everything is still so manual in this world, and the subject of most maps doesn’t move around all that much, so you can really apply a human eye to get it right. Even with the watercolors, we had to do a lot of manual work to ensure that the 1024×1024 watercolor texture blended cleanly and the various road sizes looked correct at each zoom level.

PS

Regarding deployment. One of the challenges of producing raster map tiles for the web is the amount of storage and CPU time it takes to generate them; I notice that Stamen’s tiles are available down to zoom 18, which for a worldwide set means there are millions and millions of individual PNG files. To a degree CPU time can be amortized over the life of the project if you’re using a tile server to dynamically generate and cache tiles as users first request them, but even with commodity storage like S3, you’re talking about hundreds of GB or more. Are there knock-on challenges this presents for deployment and maintenance? Is there a sustainability plan for maps.stamen.com with regard to storage and bandwidth costs, or is Stamen as a company just going to eat those costs to provide this resource?

MM

We’re using a mix of physical and virtual machines for each of the tile sets we just released, blending the strengths and weaknesses of each. The CPU-intensive rendering of watercolor maps is done on Amazon EC2 where we can invoke extra machines as necessary, but the PostGIS, Mapnik and cache storage parts are all living on an actual server in a colocation facility. We decided last year to invest in physical hardware to take advantage of the high random-access speed of solid state disks, which make it possible to serve the entire OpenStreetMap planet database without incurring the overhead of Amazon’s terribly slow I/O speeds.

Fortunately, the back-end of this project is used to drive a lot of our other work, so we’re folding the cost into a series of different engagements that all use different components of the map tiles.

PS

We appear to be reaching a tipping point where rolling your own custom map stack seems not only practical but desirable for many applications. What is your reaction to the embrace of OSM by prominent technology companies, and the emergence of designer-friendly tools like TileMill? Do we have everything we need for most designers and developers to create the map experiences they want to provide? What tools would you’d like to see built; what data sets made available?

MM

Honestly, I’m completely thrilled. With this year’s high-profile addition of Foursquare and Apple to the OpenStreetMap community, I’m looking forward to seeing what new artists and designers decide to do with maps—I can’t wait for the U.S. State Of The Map conference this autumn.

I think there are two issues that current tools don’t address well enough, and I’m excited to be working on them both: medium-scale data for counties and towns, and more options for bitmap filtering and output. I want Photoshop and Illustrator in the sky, essentially, and tools like TileMill help expose places where we need to be doing more with data before rendering and more with pixels after rendering.

PS

Elaborate on what you mean by “medium-scale data”, and how it would improve map-making.

MM

My colleague Nathaniel Kelso runs a project called Natural Earth Data, which offers global vector data at three different scales, optimized for rendering images of large regions, countries and continents, up to about zoom=9 if you think in terms of web slippy maps. OpenStreetMap meanwhile offers small-scale data down to the level of individual carriageways on major streets. There’s a gap between these two data sets where OSM is too detailed and Natural Earth is not detailed enough, so on many renderings you get bizarre selections of town names to render, doubled-up street names, or no global context.

The Terrain layer on maps.stamen.com is a vehicle for exploring a few avenues through this problem: feature generalization for route shields and large street names using the Skeletron library, simulated annealing for smarter label placement with Dymo, and cross-blending of raster data sets for ground cover and hill shading. We’re doing a lot of work to make OSM data better for rendering, and releasing all the component parts as software for processing data sets.

PS

Do you see Mapnik as the appropriate place to grow the bitmap filtering and output functions?

MM

As far as Mapnik’s role in all this, I think it’s the single best place to do vector rendering, but I’m looking elsewhere for filtering and output. I prefer to use tools that are specialized for individual tasks, so we use the pixel output of Mapnik as a source for Python-based pixel manipulation code, often implemented in libraries like NumPy that offer rapid manipulation of bitmap arrays.

PS

It sounds like you’re hinting at a new program for this kind of manipulation. You and Stamen have contributed a great deal of open-source map making software over the years; will the code that created Watercolors be released?

MM

I hope I’m not sounding too coy; there’s not any secret piece of software running the show, just a set of well-known techniques in a new arrangement. I’ve released everything I’ve ever started, but my role in Watercolor was more about taking something that already worked and making it work a little better so more people could see it.

PS

Most web maps are a pre-baked set of raster images assembled in the viewport of the browser—do you see this fundamental arrangement changing in the foreseeable future? With WebGL and HTML5 Canvas, are we ready to composite maps client-side with servers pushing vectors data over the wire? What tradeoffs are there here? Do you think you could do Watercolors entirely in the browser anytime soon?

MM

I don’t think we’re far off, though there are a few impediments still in the way. Much of the core functionality of Watercolor’s bitmap effects can be implemented entirely on a GPU in a WebGL fragment shader, so there’s no reason that we couldn’t build it that way. As far as shipping vectors, I’d love to see it happen. It’s actually not unrelated to the work we’re doing with differently-scaled data: you want your underlying data to be at the “right” level of complexity, and that generally means modifying it in some way, by dropping extra points, scrunching narrow polygons into lines, making small things disappear, and lumping groups of things together into single blobs. If we can figure out a better way to simplify on the fly, then complete client-side rendering could be a reality. Of course, Google has already done a lot of this themselves, but it’s different when the open source community does it and shares the results and the research.

PS

Simplification of lines and polygons is a common need for mapping projects, not only for display but also for interaction—clients can only handle so many points during the refresh cycle. It can be a big challenge for workflows to simplify vectors, and also to preserve topology, the relationship between shapes—our model is of disconnected points, lines, and polygons. That’s one reason I’m excited about topology as a type coming in PostGIS 2.0. What simplification techniques should we be using and exploring, both on the algorithm side as well as the workflow side?

MM

I spent about two years (off and on) researching line generalization. Skeletron went through three iterations, starting with a port of the straight skeleton technique from Tom Kelly’s description, to a collaboration with Schuyler Erle binding to the CGAL library from Python, finally settling on an application of the Voronoi diagram first published by Esri in 1996. All the way through, I kept thinking that there just had to be an easier way to make simple lines out of complex ones, and why wasn’t I finding code to do it for me? The simplification techniques we should explore are all known and published, but exist largely as plugins for systems like ArcGIS, instead of chainable tools in the Unix style. PostGIS 2.0 is going to help in a big way, and I hope that some of that effort migrates out to tools for managing workflows around flat files.

If all goes well, it should start to make sense to ship vectors over the wire and render them on the client.

It’s an interesting question to me whether client-side rendering really something we want to aim for, though. Bitmap tiles have an equilibrium of performance, size and design that I don’t think will be disturbed any time soon. I’m learning what I can about GPUs to be ready when the day comes, but in the meantime most of my focus is on developing workflows for data. OSM extracts are one aspect of this; simplifying OSM data and standing up fresh new rendering databases from source data is another. The scaffolding that makes life easiest is a combination of bias toward flat files and Postel’s Law: don’t screw around with “seamless” servers, publish data flat using old file formats, and only invent brand new things when they’re needed. Spherical Mercator slippy map tiles are fast becoming one of those well-understood old file formats, so this is what we’ve aimed for with Watercolor; otherwise, how could we have a section on maps.stamen.com showing how to use the imagery in your own applications?

PS

That’s interesting—so are you generally fine with the way Spherical Mercator has colonized the web map world? It seems to have been a decision purely about tradeoffs: what projection “works” for world-wide maps all the way down to zoom level 22, or however detailed Google Maps gets these days. But clearly choice of projection is a never-ending debate. Is Spherical Mercator an acceptable enough constraint, from your perspective, within which to do the kind of artistic expression Stamen does, and maintain interoperability?

MM

Oh yeah, I’m absolutely fine with it. It’s one of those “assume a spherical chicken” engineering solutions that actually leads to so much follow-on innovation that in retrospect I’m glad no one was letting the cartographers drive at Google in 2005. Projections only matter when you’re looking at large areas, and that’s really not the case when you’re searching for driving directions or checking out the neighborhood where you spent elementary school. It’s interesting to me, though, that the typical fifty states map you see on every link bait infographic out there is based on a conic projection—it’s about what looks right for a genre of mapmaking, and in the case of slippy maps the Spherical Mercator is clearly the obvious choice.

—@paulsmith

Going to work for Democrats

2011-09-04T16:00:00Z

I have some exciting news: in a couple of weeks, I will be the new deputy director of technology at the Democratic National Committee. I’ll be helping to create software that helps get Democrats elected. It’s a great opportunity for me to do what I love for an organization that I passionately support. I hope to help make the technology department there top-notch in terms of software engineering practices, bringing what I’ve learned from having helped develop, deploy, and grow EveryBlock. And I’ll have some projects of my own, which will hopefully advance the state of campaign software somewhat. It’s especially exciting, too, to be able to collaborate with friends in Chicago working on that one campaign.

I have been volunteering on political campaigns—federal, state, and local races—for years, and have often lamented the state of campaign software. It’s partly understandable, because campaigns tend to be all-hands-on-deck, hair-on-fire affairs, where it’s hard to justify long-range planning and software development, even if it might make the lives of your staff, organizers and volunteers easier, since your organization may not even exist for more than a few months. And campaigns rarely have in-house software engineers, so opportunities to capture and encode knowledge in the form of software, and explore new technologies, are missed.

Obama For America gets this—that’s why they’ve hired like a start-up for this campaign cycle, recognizing that great software is a competitive advantage and no longer an afterthought you contract out for. And the DNC gets it, too, and that’s why I’m excited to join them. The chance to help re-elect this president, restore Democratic majorities in Congress, and also to help down-ballot Democrats across the country in this and future campaign cycles is one I couldn’t pass up.

I’ll be commuting to the DNC’s offices in Washington, D.C. from Baltimore on a regular basis, though I’ll still be working from home a couple of days each week, so that I won’t too miss much of this kind of stuff.

It’s a bittersweet development, because I’ll be leaving EveryBlock, which I helped found 4 years ago. With the success of the recent relaunch, though, I feel now is as good a time as any to step away. The site is in great hands, and the response from users to the new version has been enthusiastic. It couldn’t have a better home than msnbc.com, who have provided great guidance and resources. I’m thrilled with the success we’ve had and for how far it’s come, and I’m confident that it will continue to be the best place on the internet to help make your block a better place.

I am particularly grateful for having worked with my great EveryBlock colleagues. I’m humbled by them and their talents and work ethic. It was a privilege to learn from them and improve my craft, however modestly, by their examples.

For now, I’m focusing on winding down at EveryBlock, and getting prepared for a new commute (I’ll try to hack it with the MARC train and a bike) and a campaign season now fully engaged. It’s a thrilling opportunity, and I hope to make the most of it.

—@paulsmith

Maxine

2011-08-27T20:10:00Z

One month ago today, Maxine Mills Smith arrived in this world. Her mom gave birth to her at Mercy Medical Center in Baltimore, Maryland, at six in the morning. She was, and continues to be, a long and strong gal. She enjoys being carried about and looking up at the changing scenery. Like her father, she zonks out to the motion of a vehicle ride, be it stroller or car. She has healthy lungs, likes to exercise them, and is a vocal chirper. She has been respecting her mom and dad with 5 and 6-hour stretches of sleep at night, but likes to mix it up from time to time and throw her mom some napless curveballs. When she is awake, she is bright and alert. Given the choice, she’d rather have some light assistance and try to stand and monster march (I said she was strong) than squirm around on her belly like a beetle. Her binky is a frequent companion, and she is a champion eater of mom’s milk. The doctors and nurses at the pediatrician’s office think she is doing fine, and give her an extra wink. We think she’s is doing fine, too. Family love Maxine. Welcome home, my daughter.

Parsing and formatting date/time in Go

2011-05-20T02:00:00Z

Go takes an interesting approach to parsing strings to time objects, and formatting time objects as strings. Instead of using codes like most languages to represent component parts of a date/time string representation—like %Y for a 4-digit year like “2011” or %b for an abbreviated month name like “Feb”—Go uses a mnemonic device: there is a standard time, which is:

Mon Jan 2 15:04:05 MST 2006  (MST is GMT-0700)

Or put another way:

01/02 03:04:05PM '06 -0700

Instead of having to remember or lookup the traditional formatting codes for functions like strftime, you just count one-two-three-four and each place in the standard time corresponds to a component of a date/time object (the Time type in Go): one for day of the month, two for the month, three for the hour (in 12-hour time), four for the minutes, etc.

The way you put this into action is by putting together the parts of the standard time in a layout string that matches the format of either the string representation you want to parse into a Time object or the opposite direction, when you want to generate a string representation from an Time object.

Parsing:

package main

import (
    "fmt"
    "time"
)

func main() {
    value  := "Thu, 05/19/11, 10:47PM"
    // Writing down the way the standard time would look like formatted our way
    layout := "Mon, 01/02/06, 03:04PM"
    t, _ := time.Parse(layout, value)
    fmt.Println(t)
}

// => "Thu May 19 22:47:00 +0000 2011"

Formatting:

package main

import (
    "fmt"
    "time"
)

func main() {
    t := time.SecondsToLocalTime(1305861602)
    t.ZoneOffset = -4*60*60
    fmt.Println(t.Format("2006-01-02 15:04:05 -0700"))
}

// => "2011-05-20 03:20:02 -0400"

There are predefined constants in the time package for common formats such as RFCs 822 and 3339.

The use of a mnemonic over obscure formatting codes I think reflects the pragmatism of Go’s developers and their focus on creating a language that makes its users more productive. While it is a break with tradition to abandon strftime-style formatting, they probably recognized that most developers, no matter how experienced, reach for man strftime or similar documentation more often than not (hell, I have a mug with the codes printed on it), and each time they do, it is an expensive context switch, in terms of lost focus. Writing the standard time down the way yours looks may be quirky, but it's easy to recall, and it also happens to match the shape of your time string, syntatically.

It’s fascinating to see a language with the pedigree of Go—from the minds of long-time C and Unix developers—toss aside certain old and venerable ways of doing things. But it’s consistent with a language that’s relatively small (in the good way of being comprehensible and coherent), focused on efficiency, and careful in what it includes.

A partial list of metaphors, visual and otherwise, anti-skeuomorphism zealots have to tackle to reach utopia

2011-04-05T18:48:00Z

Motivated by irrational hatred and overstated claims of the continuing utility of a 3.5" floppy disk icon to mean “save”:

Handset icon, meaning “phone,” on every iPhone, Android
email
Website (implies a physical place)
Wrench or cog icon, meaning “settings” or “preferences”
Shopping cart icon
Clock with analog hands, meaning “time” or time-centric application
Incandescent light bulb, several meanings, including energy settings
Bound, paper book, meaning “book”

Of course there are many, many anachronistic interface elements and metaphors in the world, and we get by okay. Mainly, this is because new generations don’t suddenly appear next to us and start using our computers without any foreknowledge of the metaphorical items. They spend time learning with the old fogeys for whom information was sometimes stored inside square pieces of plastic and metal. This overlap is necessary in general because knowledge about tools is not encoded in our genetic material. All understanding of use is part of a multi-layered strata of culture, experience, and relationships.

The real problem anti-floppy-disk people have is explaining a harm, specifically, a harm that matters. Often when we think of user interface improvements that matter, we think of examples like improving medical devices to reduce user error, changes that literally save people’s lives. Or tweaks to software that improve user efficiency and productivity, saving money. It’s hard to conceive of what might be improved by finding a better metaphor for “save” other than some designers’ personal sensibilities.

More Redis internals: Tracing a GET & SET

2011-03-10T19:48:00Z

In my previous article, I took a superficial look at how Redis starts up and prepares itself to process commands. In this article, I'll follow a GET and a SET command as they move from client through the server and back. The GET will be for a key that doesn't exist, and the SET will set that key. Then I'll look quickly at a subsequent GET and how it differs.

As before, I'm exploring Redis with the source code open in my editor and indexed with a tags file, and the Redis server running under gdb in another terminal.

Caveat: this article was written against the codebase of Redis 2.2.1. With respect to my previous article, for a list of what has changed in Redis since I wrote it, see this comment on HN.

Edit: I made two minor changes based on feedback—Redis keys are not Redis objects, they are sds strings; and you don’t have to hack the Makefile to compile without optimizations.

GET

Let's execute the command get users:1234 on Redis.

Preparing

If you inspect certain variables under gdb, you might not get what you want. Instead you see a message like "." This is because the compiler has optimized the machine code in such a fashion that the portion of memory you wanted to look at that was never used, at least for the variable under inspection. To make stepping through the code and inspecting a little easier, we make sure that Redis is not compiled with optimizations. You can do this by either building Redis with the following invocation:

make noopt

or by setting an environment variable:

OPTIMIZATION= make

Sending the command from the client

If we look at the repl loop of redis-cli, we see that it uses the linenoise library to split the arguments of the command. It dispatches on the first argument. The loop checks for client commands that aren't processed as a command by the Redis server, like exit/quit, clear (to clear the screen), and connect (which is a way to connect to a specified Redis server on host port, instead of the default host of 127.0.0.1 and port of 6379.

Any other argument is considered the name of a command to send to the server. The remaining arguments are the arugments for that command.

We're trying to get the on-the-wire representation of the get users:1234 command. redis-cli uses a redisContext struct to encapsulate the state of the connection to the server, as well as the output buffer where the command in the form of the Redis protocol is written for sending to the server. We know from reading the source of hiredis (the Redis C client library that the redis-cli program is built on) that the obuf field is where the raw command is stored:

# hiredis.h:107
/* Context for a connection to Redis */
typedef struct redisContext {
    int fd;
    int flags;
    char *obuf; /* Write buffer */
    int err; /* Error flags, 0 when there is no error */
    char *errstr; /* String representation of error when applicable */

    /* Function set for reply buildup and reply reader */
    redisReplyObjectFunctions *fn;
    void *reader;
} redisContext;

If we set a breakpoint on cliReadReply, we can inspect the output buffer and see exactly how the command looks as a bytestring bound for the server.

client $ gdb src/redis-cli
(gdb) b cliReadReply
(gdb) run
Starting program: /home/paul/src/redis-2.2.0-RC2/src/redis-cli 
Reading symbols for shared libraries +. done
redis> get users:1234

Breakpoint 1, cliReadReply (output_raw_strings=0) at redis-cli.c:421
421         if (redisGetReply(context,&_reply) != REDIS_OK) {
(gdb) p context->obuf 
$1 = 0x100102428 "*2\r\n$3\r\nget\r\n$10\r\nusers:1234\r\n"

We see that the Redis command get users:1234 as entered in our client is translated to *2\r\n$3\r\nget\r\n$10\r\nusers:1234\r\n. Any Redis client is going to convert our command expressed in its respective syntax to the same on-the-wire format. So in Python:

>>> redis_client.get('users:1234')

will send the same *2\r\n$3\r\nget\r\n$10\r\nusers:1234\r\n bytestring to the server.

Let's print that bytestring to screen and render those `\r\n's as line feeds so we can see an expanded view and get a better look at the protocol.

*2
$3
get
$10
users:1234

The first bit is *2, which tells us that the arity of the command, including the command name, is 2. That is, the server should expect two arguments to follow.

The next bit is $3, which means that the length of the first argument in bytes is 3. The argument itself follows, our command name, get.

The next bit after that is $10, so the length in bytes of the second argument is 10. Our one and only argument to the command is next, users:1234, the key we are trying to get.

Receiving the command on the server

If you remember from the last article, readQueryFromClient is a good place to set a breakpoint in your debugger on the server side for inspecting an inbound client command.

server $ gdb src/redis-server
(gdb) b readQueryFromClient
Breakpoint 1 at 0x100011445: file networking.c, line 882.
(gdb) run redis.conf
Starting program: /home/paul/src/redis-2.2.0-RC2/src/redis-server redis.conf
Reading symbols for shared libraries +. done
[63700] 01 Mar 11:04:40 * Server started, Redis version 2.2.1
[63700] 01 Mar 11:04:40 * The server is now ready to accept connections on port 6379

Now back in the terminal with the client running in the debugger, continue to send the command to the server, which will stop at the breakpoint we set.

# client
(gdb) c
Continuing.

# server
Breakpoint 1, readQueryFromClient (el=0x100200000, fd=5, privdata=0x100804e00, mask=1) at networking.c:801
801     void readQueryFromClient(aeEventLoop *el, int fd, void *privdata, int mask) {

Let's step to the following line:

# src/networking.c:808
nread = read(fd, buf, REDIS_IOBUF_LEN);

If we step to here in our debugger, we see that the server read 30 bytes. If you count the number of bytes in our Redis protocol-encoded command, *2\r\n$3\r\nget\r\n$10\r\nusers:1234\r\n, you'll see it's 30. Just for good measure, let's look at the 30 bytes beginning at the memory location pointed to by buf:

(gdb) p nread
30
(gdb) x/30cb buf
0x7fff5fbfeaf0: 42 '*'  50 '2'  13 '\r' 10 '\n' 36 '$'  51 '3'  13 '\r' 10 '\n'
0x7fff5fbfeaf8: 71 'G'  69 'E'  84 'T'  13 '\r' 10 '\n' 36 '$'  49 '1'  48 '0'
0x7fff5fbfeb00: 13 '\r' 10 '\n' 117 'u' 115 's' 101 'e' 114 'r' 115 's' 58 ':'
0x7fff5fbfeb08: 49 '1'  50 '2'  51 '3'  52 '4'  13 '\r' 10 '\n'

And we can see our whole command bytestring is there, byte by byte.

The server has now read in the entirety of our command in one step. (Because we had a relatively short command, one that fits inside a kernel buffer, and we are the only client on a loopback network device, this is the case, but it need not be. Since Redis is event-driven, this function, readQueryFromClient, is called whenever there are bytes from buffers to be read. If our command was particularly long, or there was a lot of network contention, the command may take more than one I/O event before it is fully read. For this reason, Redis builds up a buffer per client and appends bytes to it on each call to this function. It only proceeds with processing the command when it has been fully read. But we don't need to consider this in our simple example, so we will proceed.)

We're going to elide the processing of the input buffer. This is the point where the server takes the Redis protocol-encoded bytestring of our request and unpacks it into arguments on the client struct object. If you are interested in the details of that parsing, examine the function processMultibulkBuffer in networking.c. All we are interested in at this point is that the argc member of the client object is the number of command arguments (counting the command name itself) and argv is a pointer to the list of arguments.

The bit of code we care at this point is processCommand. The first thing the server does is look up the command in its command table (see "Setting up command table" in the previous article, but note that this lookup is now O(1), see the HN thread linked above). Assuming the command is found (which our get will be), the server will double-check that the arity of the command as defined in the command table matches the number of arguments received from the client (c->argc).

# redis.c:998
cmd = lookupCommand(c->argv[0]->ptr);
if (!cmd) {
    addReplyErrorFormat(c,"unknown command '%s'",
        (char*)c->argv[0]->ptr);
    return REDIS_OK;
} else if ((cmd->arity > 0 && cmd->arity != c->argc) ||
           (c->argc < -cmd->arity)) {
    addReplyErrorFormat(c,"wrong number of arguments for '%s' command",
        cmd->name);
    return REDIS_OK;
}

Skip down to the end of processCommand. Because our humble get is not a "multi" command like mget, mset, etc., it doesn't require queue-like processing of the underlying multiple commands, so we go right to call, which is where our command is dispatched.

# redis.c:953
void call(redisClient *c, struct redisCommand *cmd) {
    long long dirty;

    dirty = server.dirty;
    cmd->proc(c);
    dirty = server.dirty-dirty;

    if (server.appendonly && dirty)
        feedAppendOnlyFile(cmd,c->db->id,c->argv,c->argc);
    if ((dirty || cmd->flags & REDIS_CMD_FORCE_REPLICATION) &&
        listLength(server.slaves))
        replicationFeedSlaves(server.slaves,c->db->id,c->argv,c->argc);
    if (listLength(server.monitors))
        replicationFeedMonitors(server.monitors,c->db->id,c->argv,c->argc);
    server.stat_numcommands++;
}

Let's focus on line 952, cmd->proc(c);. This is Redis's dynamic dispatching of command function calling. Redis makes this clean and simple by encapsulating commands and giving all the actual underlying command functions the same function signature, taking our client object, which carries the payload of our command's arguments. So we're interested in looking into the details of the Redis command object and the actual function that will handle our get.

# redis.h:504
struct redisCommand {
    char *name;
    redisCommandProc *proc;
    int arity;
    int flags;
    /* Use a function to determine which keys need to be loaded
     * in the background prior to executing this command. Takes precedence
     * over vm_firstkey and others, ignored when NULL */
    redisVmPreloadProc *vm_preload_proc;
    /* What keys should be loaded in background when calling this command? */
    int vm_firstkey; /* The first argument that's a key (0 = no keys) */
    int vm_lastkey;  /* THe last argument that's a key */
    int vm_keystep;  /* The step between first and last key */
};

If we pop up to the top of redis.c, we see the definition of the Redis command table, and our get is the first entry.

# redis.c:71
struct redisCommand readonlyCommandTable[] = {
    {"get",getCommand,2,0,NULL,1,1,1},

getCommand is the function that does the actual work for our command. It's a thin wrapper for getGenericCommand.

# t_string.c:62
int getGenericCommand(redisClient *c) {
    robj *o;

    if ((o = lookupKeyReadOrReply(c,c->argv[1],shared.nullbulk)) == NULL)
        return REDIS_OK;

    if (o->type != REDIS_STRING) {
        addReply(c,shared.wrongtypeerr);
        return REDIS_ERR;
    } else {
        addReplyBulk(c,o);
        return REDIS_OK;
    }
}

void getCommand(redisClient *c) {
    getGenericCommand(c);
}

The arguments to lookupKeyReadOrReply are the client object, the key users:1234 we're trying to look up, and an object, shared.nullbulk that will be the default reply to the client if the key is not found.

# db.c:58
robj *lookupKeyReadOrReply(redisClient *c, robj *key, robj *reply) {
    robj *o = lookupKeyRead(c->db, key);
    if (!o) addReply(c,reply);
    return o;
}

lookupKeyRead is a thin wrapper for lookupKey that handles removing keys that have been set to expire.

Now we get to the heart of the get command -- looking up the key in the database.

# db.c:9
robj *lookupKey(redisDb *db, robj *key) {
    dictEntry *de = dictFind(db->dict,key->ptr);
    if (de) {
        robj *val = dictGetEntryVal(de);

        /* Update the access time for the aging algorithm.
         * Don't do it if we have a saving child, as this will trigger
         * a copy on write madness. */
        if (server.bgsavechildpid == -1 && server.bgrewritechildpid == -1)
            val->lru = server.lruclock;

        if (server.vm_enabled) {
            if (val->storage == REDIS_VM_MEMORY ||
                val->storage == REDIS_VM_SWAPPING)
            {
                /* If we were swapping the object out, cancel the operation */
                if (val->storage == REDIS_VM_SWAPPING)
                    vmCancelThreadedIOJob(val);
            } else {
                int notify = (val->storage == REDIS_VM_LOADING);

                /* Our value was swapped on disk. Bring it at home. */
                redisAssert(val->type == REDIS_VMPOINTER);
                val = vmLoadObject(val);
                dictGetEntryVal(de) = val;

                /* Clients blocked by the VM subsystem may be waiting for
                 * this key... */
                if (notify) handleClientsBlockedOnSwappedKey(db,key);
            }
        }
        server.stat_keyspace_hits++;
        return val;
    } else {
        server.stat_keyspace_misses++;
        return NULL;
    }
}

Redis uses its own hash table implementation to store keys and their values in memory. Inside the db object, the field dict is a pointer to the hash value for the current Redis database (remember there can be up to 16 separate databases in a single Redis server instance, indexed by number).

First, Redis calls dictFind with the database's hash table and a pointer to the key's bytestring. dictFind looks up the hash of the key in the table, using a standard algorithm that should be familiar to anyone who's implemented a hash table (check out dict.c starting at line 391 if you're interested, the table is an array with linked lists for colliding hashes).

If the key is found in the table, dictFind returns a pointer to the entry in the table. Otherwise, it returns NULL. Back in lookupKey, if the entry is not null, Redis retrieves the value (i.e., the Redis object our key references) from the hash table via dictGetEntryVal and takes care of a bit of bookkeeping for expiry and VM, if the key was found, and stats in either case (hits and misses). If the entry was NULL, then lookupKey also returns NULL; we'll see how this is handled by Redis for a reply to the client when the key is not found, which is the case for us at this stage.

With the value of lookupKey, we'll go back up the stack to our callers. Back to lookupKeyReadOrReply, we look at line 60:

# t_string.c:60
        if (!o) addReply(c,reply);

Since we got NULL from lookupKey this time, we call addReply. The value of reply here comes from the call in getGenericCommand, and it is shared.nullbulk. This field in the global struct object shared is initialize thusly:

# redis.c:712
shared.nullbulk = createObject(REDIS_STRING,sdsnew("$-1\r\n"));

We can see that it is a Redis string object who's on-the-wire value is $-1\r\n, meaning a length of -1, Redis's way of indicating null to a client, according to the protocol.

addReply builds the reply to the client. It does this by first setting up a write event on the main event loop listener with _installWriteEvent. This makes sure that the reply is written out to the client connection when there are bytes present in the buffer. Next, Redis adds the reply to the client's buffer. If the reply object were an non-string value like an integer, or a list, or a set, Redis would first decode it to a bytestring that can be serialized to, for example, on a socket. Redis string objects are encoded "raw," or as-is. The nullbulk object is technically a string object, so no decoding is necessary in our case. In any case, the reply bytestring is copied to the client's reply buffer with _addReplyToBuffer, which for all intents and purposes completes the execution of our get command on the server.

The client will read the on-the-wire reply of $-1\r\n and know that it is a string reply of length -1, and therefore is the null (or "nil," in the context of redis-cli) object, and to convert that into the appropriate object for the language of the client. Back to our redis-cli client patiently waiting for a reply from our breakpointed server, which we continue from, that looks like:

(nil)
redis>

SET

The set command proceeds much the same way as the get, up to the point of command dispatching on the server.

# client
redis> set users:1234 "Paul Smith"

# server
Breakpoint 1, readQueryFromClient (el=0x100400000, fd=6, privdata=0x100805e00, mask=1) at networking.c:801
801     void readQueryFromClient(aeEventLoop *el, int fd, void *privdata, int mask) {
(gdb) n
802         redisClient *c = (redisClient*) privdata;
(gdb) n
808         nread = read(fd, buf, REDIS_IOBUF_LEN);
(gdb) n
809         if (nread == -1) {
(gdb) print nread
$1 = 47
(gdb) x/47cb buf
0x7fff5fbfeba0: 42 '*'  51 '3'  13 '\r' 10 '\n' 36 '$'  51 '3'  13 '\r' 10 '\n'
0x7fff5fbfeba8: 115 's' 101 'e' 116 't' 13 '\r' 10 '\n' 36 '$'  49 '1'  48 '0'
0x7fff5fbfebb0: 13 '\r' 10 '\n' 117 'u' 115 's' 101 'e' 114 'r' 115 's' 58 ':'
0x7fff5fbfebb8: 49 '1'  50 '2'  51 '3'  52 '4'  13 '\r' 10 '\n' 36 '$'  49 '1'
0x7fff5fbfebc0: 48 '0'  13 '\r' 10 '\n' 80 'P'  97 'a'  117 'u' 108 'l' 32 ' '
0x7fff5fbfebc8: 83 'S'  109 'm' 105 'i' 116 't' 104 'h' 13 '\r' 10 '\n'
(gdb) print (char *)buf
$2 = 0x7fff5fbfeba0 "*3\r\n$3\r\nset\r\n$10\r\nusers:1234\r\n$10\r\nPaul Smith\r\n"

This time, our protocol-encoded bytestring is 47 bytes long, owing to the extra argument "Paul Smith" and the length tag it requires. Also notice the leading *3 indicates there are three arguments: set, users:1234, Paul Smith.

Let's skip ahead now to the point in call in redis.c, after the command has been looked-up in the command table and the server is about ready to call the underlying proc function with the client object argument. The set command, in the form of a redisCommand struct, looks like this:

# redis.c:73
{"set",setCommand,3,REDIS_CMD_DENYOOM,NULL,0,0,0},

Notice that the arity of the command is 3, which includes the leading command name, plus key and value, and matches what we expect from the client. The set command has a flag that get did not: the constant REDIS_CMD_DENYOOM means that, in out-of-memory situations where Redis can't allocate any more memory, the execution of the command should be denied. (The absence of this flag means that Redis can continue to serve client "read" requests like get even when the server can no longer write any new data.)

I set a breakpoint on setCommand and let the server continue running until that point:

# server
(gdb) b setCommand
Breakpoint 2 at 0x10001a6e2: file t_string.c, line 48.
(gdb) c
Continuing.
Breakpoint 2, setCommand (c=0x100805e00) at t_string.c:48
48          c->argv[2] = tryObjectEncoding(c->argv[2]);

Incidentally, you can inspect the values of the client's command arguments at any time, with a simple gdb invocation. The arguments are of type robj, which has a field ptr that is a pointer to the actual value in memory. Since in our set case these are strings, we can inspect them by typecasting to char * like so:

(gdb) p (char *)c->argv[0]->ptr
$10 = 0x10032ae78 "set"
(gdb) p (char *)c->argv[1]->ptr
$11 = 0x10032b068 "users:1234"
(gdb) p (char *)c->argv[2]->ptr
$12 = 0x10032b098 "Paul Smith"

The first thing the server does in setCommand is encode the value being set with tryObjectEncoding. It will try to create an efficient encoding if the bytestring can be interpreted as an integer, for example. This can save space especially in the case where many numbers are being stored. Additionally, Redis will try to reuse shared integers as values instead of allocating resources for new ones -- see the previous article for more on the creation and use of shared integers.

Once the value being set has been encoded, setGenericCommand is called (set shares setGenericCommand with the setnx and setex commands). From here, dbAdd is called, with the client, key, and value as arguments. dbAdd will only add the value to the database's hash table if the key does not already exist. In our case, since the key users:1234 does not exist, the value of dictFind is null, and the function proceeds to add the value with dictAdd.

dictAdd takes the dictionary hash table, key, and value as arguments. It uses _dictKeyIndex to find the index of a free slot in the hash table for our new entry. See the implementation in dict.c and dict.h for the details of key hashing, and the structure of the dictionary and its component hash tables (each Redis dictionary contains two hash tables in order to provide incremental rehashing as the dictionary grows). dictAdd allocates memory for the new entry and stores it in the new index.

The server returns back up the stack from dictAdd and dbAdd to setGenericCommand, where it increments the reference count on our new value. Redis uses reference counting in order to free memory used by values that have been deleted or have expired. It then "touches" the key so that if any clients are watching the key, the next exec command will fail. It also increments the server's dirty flag, which it uses to determine when to write out the dump file to disk. Finally, it writes out the reply to the client, which is a shared object, shared.ok. This is special Redis string object in the protocol that consists of the bytestring "+OK\r\n". Clients will typically convert this into the equivalent "true" value for their language.

GET redux

Our key is now set, so we can try the get users:1234 command again and see how it differs for a found key.

# db.c:9
robj *lookupKey(redisDb *db, robj *key) {
    dictEntry *de = dictFind(db->dict,key->ptr);
    if (de) {
        robj *val = dictGetEntryVal(de);

        // ... skipping the lru & vm parts ... 

        server.stat_keyspace_hits++;
        return val;

The point where a get on an existing key and a get on a non-existent key differ is line 11, where the de entry in the database's dictionary is found. dictGetEntryVal is a simple macro for accessing the field in the de struct that carries the value associated with the key. Redis updates its statistics to indicate a key hit and returns the value object.

Again, as with the key miss from above (remember the null value is a Redis object, too), the value is decoded into the Redis bytestring protocol. This is the response to the client, and we have concluded our GET/SET/GET dance.

The All-Blue-Collar League

2011-01-24T00:22:00Z

In honor of the all-union Super Bowl, I present the All-Blue-Collar League, a collection of U.S. professional sports franchises named after local unions, industry.

In solidarity:

NFL
Green Bay Packers
Pittsburgh Steelers
Dallas Cowboys
MLB
Houston Astros
Milwaukee Brewers
Seattle Mariners
Texas Rangers
NBA
Detroit Pistons
NHL
Edmonton Oilers

Thanks to @kostuch, @ikesmith, @joshandrews, @tcarmody, @lmsahistory, and @janieporche for suggestions.

Gold is the same as paper

2011-01-14T19:38:00Z

We won’t return to a gold standard because changing to a fiat currency was one of the great innovations of modern economics. You aren’t literally and figuratively weighed down by gold, and that allows central banks and governments to set policy that allows them to rebound from crises, spur growth, and lift people out of poverty and short lifespans. As Liaquat Ahamed points out, the gold standard is also a sure way to get yourself in a deflationary cycle.

But, wait! gold bugs will say. That paper isn’t worth the paper it’s printed on. Gold has real value, that’s why it was the world’s currency for many years. The problem with this is that money is a turtles-all-the-way-down concept. Gold is just as much a pointer to value as paper is, except it has many more downsides: it’s heavier, more cumbersome to exchange, and has a fixed supply.

Gold isn’t valuable in and of itself. You can’t live in gold. You can’t eat gold. You can’t clothe yourself in gold. A gold coin can’t paint your house, or build you a website. You could (hypothetically) use it to put your kids through college, so it that sense it’s valuable to you, but only in its exchange, because on the other end, the trustees and professors and janitors at the college need to live somewhere, eat something, wear something, and have websites built. And again they exchange it for those goods and services. So really the only thing that’s intrinsic to this little economy I’ve just described is the fact that you, the college trustees, the service providers, et al., all agree on the value or rate of exchange. It’s like an algebraic equation where constants cancel out — gold is the constant, so we can abstract it away. It may as well be “neighborhood fun bucks,” so long as everyone agrees on the rate of exchange for goods and services.

And there you have it. Economies have nothing to do with the particulars of their currencies, and everything to do with people and institutions trusting that, in exchange for some amount, I will receive something of perceived equal value. Trust, and perception. Currencies are just a convenient means by which we mitigate that exchange, an abstraction, a stand-in for a hypothetical exchange, ultimately, an exchange built on trust.

Interestingly, the only inherent value gold has is in its use as jewelry. Which, aside from any aesthetic concerns, couldn’t be more appropriate for our discussion, because people wear gold jewelry as a signifier of wealth. A pointer, a marker, an abstraction for the wealth itself, which, again, is just the potential for exchange.

It is this essential trust network at the core of economics that explains why there is so much noise about gold. It reflects a lack of trust on the part of gold advocates in our central banking and governmental institutions, which they feel pursue policies that aren’t in their interest, or are in the interest of undeserving populations. The switch to a gold standard, therefore, has nothing to do with inflation or any other “economic” concern, and everything to do with pursuing a political objective. The market of exchange in this little political economy is for ideas about reprioritizing our institutions. Which is a legitimate topic of national debate. But reverting to the gold standard, as a practical economic matter, would be bonkers.

Taking Go for a spin

2011-01-12T12:00:00Z

I was struck by the high-level abstractions Go provides for writing networked clients and servers, even by the standards of conventionally high-level languages like Python. Given Go’s identification as a systems programming language, this is initially surprising, because such languages tend to be low-level for performance reasons (Go does expose a lot of low-level interfaces through various packages like os and syscall), but of course there would be good support for these things because Go is a modern language designed by folks with many years of experience writing networked servers.

Specifically, the net package provides the Listener and Conn types that hide many of the details of setting up socket connections. A simple net.Listen("tcp", ":1234") is enough to get the equivalent of a listener, or server, socket. The bufio package provides buffered read methods, simplifying common tasks like reading lines from a socket.

Let’s dive in and take a look at one way to implement a classic echo server. I tried to use Go’s language features for concurrent communication, goroutines and channels, at crucial points. Where a different implementation in different language might fork a new process or spawn a new thread to handle a new client connection, a goroutine was started and the details of the connection communicated with a channel.

package main

import (
    "net"
    "bufio"
    "strconv"
    "fmt"
)

const PORT = 3540

All Go programs must have a package main. We declare a number of packages to be imported, net and bufio already mentioned, strconv for its conversion function Itoa() that converts an integer to a string, and fmt for printing strings to the console. Our echo server will listen on port 3540.

func main() {
    server, err := net.Listen("tcp", ":" + strconv.Itoa(PORT))
    if server == nil {
        panic("couldn't start listening: " + err.String())
    }
    conns := clientConns(server)
    for {
        go handleConn(<-conns)
    }
}

Every Go program must have a function main() in its package main. We start by declaring and initializing a new listener for our echo server. We choose the more generic net.Listen() over net.TCPListen(), because we can conveniently parameterize the type of the listener with a string to the first argument of Listen. (It is generally preferred in Go to deal with the most generic type, or interface, especially when writing function and method signatures, and either allow a specific type to be inferred by the compiler or specify it when declaring/initializing a variable.) ":" + strconv.Itoa(PORT) is a string concentation expression, and the port constant we defined earlier is converted to a string. (Strictly, a numeric constant is not an integer of a particular type, but the string conversion function works here because the compiler converts the constant into the concrete type to match the function signature.)

The idiom for Go when handling multiple return values where the last has an error type is to check the main object for nil, and then panic() with the stringified error object. This is generally preferred over printing to stderr and using os.Exit(), because it gives callers higher up the stack the chance to recover(), sort of a raise/catch exception flow.

Using Go’s compact syntax for simultaneous declaration and initialization, we set conns to the value of the function call clientConns(server). This is the channel we’ll use for getting new client connections.

The equivalent of a infinite loop like while True: or for (;;) in Go is for { ... }. Each time through the loop, we start a goroutine, calling handleConn() with the value of the receive operation on our client connections channel. The unary receive operator <- blocks until a value is available on the channel, in our case, a new client having connected.

func clientConns(listener net.Listener) chan net.Conn {
    ch := make(chan net.Conn)
    i := 0
    go func() {
        for {
            client, err := listener.Accept()
            if client == nil {
                fmt.Printf("couldn't accept: " + err.String())
                continue
            }
            i++
            fmt.Printf("%d: %v <-> %v\n", i, client.LocalAddr(),
                client.RemoteAddr())
            ch <- client
        }
    }()
    return ch
}

We create a new channel of type net.Conn, which corresponds to the type that we’ll get from calling Accept() on our listener connection object. We start off a new, anonymous goroutine which runs in an infinite loop, constantly accepting new connections. listener.Accept() blocks as long as there are no new clients to deal with, but since we’re running inside a goroutine and "detached" from the main program flow, other already-connected clients can continue to be handled without interruption (this is where a fork() or a new thread would happen in a typical server). Instead of panic()ing here if there is an error connecting with the client, we simply note it on the console of the server and move on. We also keep track of the number of clients we’ve seen with the counter i. fmt.Printf() works like you’d expect, though the %v format is not like anything in C’s printf() — it prints a value in a default format and works for any type, somewhat like repr() in Python.

The binary <- communication operator is used to send the client, of type net.Conn to the channel we created at the top. Go has lexical scope (and is garbage-collected), so ch is available inside the anonymous goroutine and after our function returns (because the infinite Accept()ing for-loop keeps the goroutine alive).

func handleConn(client net.Conn) {
    b := bufio.NewReader(client)
    for {
        line, err := b.ReadBytes('\n')
        if err != nil { // EOF, or worse
            break
        }
        client.Write(line)
    }
}

As we saw back in main(), handleConn() is invoked as a goroutine with each new client connection (the connection being received from the other side of the channel we just created). bufio.NewReader() wraps the client object with a nicer interface for reading lines of bytes from. If we decided to use the Read() method of the net.Conn object, our code would be more complex, having to check for the substring "\n" and testing for EOF. Instead, we can treat this as a line-oriented protocol, and simply get a line and write it back to the client, as long as there are lines to be read.

Here’s the whole thing.

The pattern of getting new connections from a channel and starting a goroutine to handle them is conceptually clean, and also happens to be a straightforward way to write a multiplexing, concurrent server.

Use Oplop for your web site passwords

2010-12-18T05:52:52Z

The Gawker breach made mainstream what’s been known for a while: we don’t use good passwords, and we use the same password over and over again on multiple sites, exposing us to a potential chain reaction of account intrusions if it is uncovered.

One way to address this problem is to use a password generator—the idea is that all you have to remember is a single, good, strong password, and the password generator will spit out a random string of characters to be used as a password on a particular site. This generated password is hard for an attacker to guess, and hard for them to “crack” with password-attacking software. You’re not supposed to remember or even care what the actual password is: you just remember the master password. And if one of the sites you have an account with is compromised, it may suck for you with respect to your sensitive information on that site, but because the generated password is unique to the site, you’re safe from having it used against you on other sites where you have accounts.

The password generator I use is called Oplop. It’s just a web site, so you visit it in your browser; there’s nothing to install. There are two fields to enter—a nickname, and the master password. The nickname is any name you want to identify the site or service for which you need a password. After you enter a nickname and your master password, you click the “Create” button, and an unguessable password is generated for you in the field below the button. Enter that password into the site that’s asking you for a password, and you’re all done. The next time you visit the site and need to enter your password, you repeat the process exactly to get the same unique password that’s just for that site.

There are a few tips and tricks to using Oplop successfully. The nickname is just a label that you remember to refer to the site you need a password for, but if it is different by even one character, an entirely different password will be generated. So it’s important to be consistent, and ideally use a simple convention so that you enter the label the same way every time. I’ve chosen a convention of using all-lowercase and no non-alphanumeric characters, and the domain name of the site, minus the top-level domain (the .com or .org, etc.). For example, if I’m on amazon.com, I’ll use “amazon” (without the quotes) for the nickname. If I was visiting a site with the domain name “Cat-Pics-123.net”, I’d use “catpics123” for the nickname. Whatever your convention for nicknames, it should be simple and you should use it consistently.

Your master password should be a good one—fairly long and hard to guess—since it is the key to the kingdom. Security researchers often praise the idea of “passphrase,” a sentence instead of a single word. These can often be easier to remember and are just as strong if not stronger than shorter, gobbledygook passwords, even if the passphrase is composed of simple words. A four or five word sentence, separated with spaces, should suffice, although longer passphrases are even better. In any case, use care when entering your master password into Oplop. By default, it doesn’t do any checking, so, like in the case of the nickname, a single character’s difference means a whole different generated password. You can optionally click on the checkbox “Validate master password?”, which adds an additional password field for you to re-enter your master password, providing a simple check that might catch a typo in the first field.

The final tip is to cut-and-paste the generated password out of its field instead of copy-and-paste. It’s not so much a tip per se because Oplop recommends that you do this right on the page, but it’s important enough to reiterate. Doing it this way prevents someone from snooping over your shoulder, or if you are careless and leave the Oplop page open untentionally on a shared computer.

You might have stopped short at the notion of a web-based password generator. Isn’t it insecure to trust a web site to generate a password for you? The answer is, you are only actually interacting with your browser—there is no network activity taking place when you generate a password with Oplop. That is because the entire Oplop program is just HTML and JavaScript loaded in your browser, and it is your browser on your local computer that takes the nickname and master password, performs the cryptographic magic on them, and outputs the generated password. In fact, Oplop works offline just as well. You can test it by visiting the site and bookmarking it on your iPhone or Android, deleting the browser window, putting the phone in airplane mode, then loading Oplop from the bookmark. It works just the same as if you were connected to the internet.

The browser-centric nature of Oplop gets to another key concept about it—there’s nothing to install. In fact, it is essentially already installed on any internet-enabled device with a JavaScript-enabled browser. There’s no central database of account names and generated passwords, like you might expect, and which password generators like 1Password and Password Gorilla are built around. These programs work similarly to Oplop in the end result, but their central database presents a user experience problem—it needs to be shared among the computers you want to use the program on. This can make it difficult to retrieve a password if you are on your friend’s laptop or a PC at an internet café. Not only do you have to install the password generator software itself, if you are allowed to by policy of the computer’s owner or manager, you have to synchronize the central database with the computer. By simply eschewing the notion of storing account names and passwords, Oplop provides a universally-accessible password management tool. Those unique generated passwords are just ephemera, the results of combining the secret you keep in your head (your master password), the site nicknames, and a cryptographic function in your computer. Even the notion of the “master password” is turned on its head—it’s not unlocking some database of passwords, it’s combined with the nickname to make a cryptographic hash, the bits of which make up the final generated password. You could start using a different master password at any time, which essentially would invalidate any extant passwords generated with the old one (you still would have to manually go in and update each password on each site, though).

There are other approaches to securing your passwords, including writing them down. In fact, if you have trouble remembering strong passwords or passphrases of sufficient length, you might consider writing down your Oplop master password on a slip of paper you keep in your wallet.

Oplop is not perfect—it outputs a fixed style of generated passwords that may not meet the password criteria for various sites and services, which require certain lengths, or a certain mix of characters. It also doesn't address the need of strong, crypotgraphically-secure storage of ad-hoc information such as credit card numbers and other sensitive account information, which are often stored alongside passwords and account names in more traditional central database-style password generator programs. But this is by design and not a failing of Oplop. I continue to use Password Gorilla for this scenario, especially for protecting financial information. Also, there are certain threats Oplop can’t protect against, And it’s worth understanding what they are before you start using it.

Oplop addresses the proliferation of passwords across multiple web site accounts, is simple to use, and works everywhere there’s a browser. So stop using “letmein” for all your web site accounts—you’d be much better off plugging that in to Oplop as your master password. But don’t. Use a better one.