Mike Migurski is partner
and Director of Technology at Stamen design studio in San Francisco. He
and his Stamen colleagues recently unveiled a new site, maps.stamen.com, that included
three custom-rendered maps using data from OpenStreetMap, including a
remarkable one that, despite being generated by algorithm from
precise vector data, resembled a hand-made watercolor
painting.
Paul Smith
You’ve blogged about the process of creating the Terrain
map tiles, and the code for the Toner tiles is open-sourced and on
GitHub, but the Watercolor tiles seemed to come out of the
blue. Can you talk about what inspired them, how you and your
Stamen colleagues went about designing them, and what uses
you imagined for them?
Mike Migurski
The main use we imagined for them was that they would look
really, really good, and unlike anything else being done with
online maps. The process started with Eric Rodenbeck and Zach
Watson kicking around some ideas on simulating the appearance
of watercolor painting algorithmically, and expanded to
include
Geraldine Sarmiento’s hand-done textures.
We saw an amazing reaction to
Aaron Straup Cope’s work on Prettymaps last year, and
more generally a lot of interest in map-based art, so it
seemed natural to try something that was algorithmic and at
the same time had the appearance of being made by hand. I
figured people would swallow their tongues in surprise when
they saw these, especially when we switched from image search
results for painted textures to pictures we were making
ourselves for the project.
PS
Making maps these days is often about mechanizing the
conversion of vector data, like from OSM, into raster images, which has
meant perfecting our ability to layer and style points,
lines, and polygons in very precise ways. I think what
captivated people about the Watercolor tiles is that they
seemed to color outside the lines, so to speak, in a way that
didn’t seem possible with current tools. What tools did you
use to generate the tiles, and how did you engineer them in a
way to achieve this effect? Did you have to write new
software to aid you?
MM
We did write new software, yes. We’ve been exploring ways
to make OSM data more
usable by cartographers over the past few years, and that’s
largely been expressed in code designed to make large data
sets easier for cartographers (and ourselves) to deal
with:
Cascadenik
was the first widespread application of CSS-like style
rules in this area, since developed and expanded by
Development Seed in Carto.
TileStache is a
tile-generation server built to deploy custom
visualization and rendering code.
I’ve been talking about this stuff for a few years, and
the release of maps.stamen.com is intended to
be a showcase that helps shift expectations of what good,
online cartography should look like. It hasn’t been long
since using a solid anti-aliased vector-rendering library
like Mapnik was enough to make a strong showing against
commercial map providers, but now we’re starting to establish
new ideas around texture, color and labeling that require a
layer of additional work beyond Mapnik.
Watercolor itself is a custom Provider to TileStache,
feeding on simple Mapnik output and adding techniques like
gaussian blur and Perlin noise as well as high-quality scans
of actual water color paintings to achieve its effect. The
development of watercolor had a number of stages, beginning
with Zach’s experiments in noise and texture, Eric and George
Oates’s encouragement, Geraldine’s addition of color and
texture, and proceeding to my work with Jeff Easter and
Nathaniel Kelso to improve performance until the code was
good enough for public launch. The visual appearance is
ridiculously CPU-hungry compared to your typical vector
cartography, so we’ve got a few virtual servers on the moon
doing the live calculation of new maps in new places.
Without diving too deeply into the actual watercolor code,
we owe a lot to Kai Krause’s
Algorithmic Painting ideas from almost 20 years ago.
PS
I see that Stamen is releasing these tiles under a
Creative Commons license as part of a Knight
Foundation-funded project centered around data visualization
and cities. City governments are historically big users of
enterprise GIS software like ArcGIS; are there any lessons
here for government or citizens, or does the work,
specifically the Watercolor tiles, speak for themselves as
art?
MM
I think the latter: watercolor is basically art. Of course
there’s an undeniable political component to the
OpenStreetMap data it’s made of. In the U.S. OSM Foundation we’re interested in
ways to get government at various levels to participate
custodians of OSM, and I
think open data and enterprise GIS software can cooperate.
Esri puts a lot of effort into OpenStreetMap-related efforts,
and I’m not expecting to see their involvement in city
governments wane.
PS
Can you expand a bit on the interplay of vector data,
raster data, and image filters to produce the tiles? What
sort of preliminary work did you need to do to get the
components parts to fall into place? Do you see patterns
emerging and opportunities for abstraction for future artist
map tools?
As a bystander to this part of the process, it seemed like
the actual simulation technique took shape in just a few
days, based on Zach’s existing experience with image
manipulation in Python. The real work came in the
knob-twiddling, or as we call it internally, “spring tuning”
(based on our experience with the Digg Swarm project from
2007). Here,
for example, is one of Zach’s own parameter views that he
used to tune noise thresholds for the ground texture.
I think the pattern that’s emerging for me is the raw
labor-intensivity of this kind of work, the
parameter-tweaking in a space of possible outcomes that
results in something that looks right and feels exciting.
Once the basic structure of noise, blur and threshold is in
place, all you can really do is watch carefully as you
repeatedly try combinations until something clicks.
PS
We’re usually talking about algorithms and stylesheets
when we talk about web maps. Traditional cartographers often
exercise artistic license over data streams as well—for
example, manually but subtlety tweaking the curve of a road
so that it reflects a shared or colloquial understanding of
its location rather than it’s literal location. And then
there are the more abstract but functional examples like
subway system maps, whose stops and lines are not intended to
be scale representations of their real-world counterparts. Do
you see it possible for automated cartography to produce maps
like these? What techniques would we need to develop (for
example, a “hints” file ala typography for manually
overriding certain points)?
MM
I’ve not yet seen an attempt at automating this kind of
cartography which has resulted in a satisfying outcome, but
it’s still the subject of many PhD theses, so maybe it’s just
too early. I suspect that we’ll end up seeing is a companion
project to OSM where human
make decisions about how things should be shown and
contribute those to a free and open data source. Everything
is still so manual in this world, and the subject of most
maps doesn’t move around all that much, so you can really
apply a human eye to get it right. Even with the watercolors,
we had to do a lot of manual work to ensure that the
1024×1024 watercolor texture blended cleanly and the various
road sizes looked correct at each zoom level.
PS
Regarding deployment. One of the challenges of producing
raster map tiles for the web is the amount of storage and CPU
time it takes to generate them; I notice that Stamen’s tiles
are available down to zoom 18, which for a worldwide set
means there are millions and millions of individual PNG
files. To a degree CPU time can be amortized over the life of
the project if you’re using a tile server to dynamically
generate and cache tiles as users first request them, but
even with commodity storage like S3, you’re talking about
hundreds of GB or more. Are there knock-on challenges this
presents for deployment and maintenance? Is there a
sustainability plan for maps.stamen.com with regard to
storage and bandwidth costs, or is Stamen as a company just
going to eat those costs to provide this resource?
MM
We’re using a mix of physical and virtual machines for
each of the tile sets we just released, blending the
strengths and weaknesses of each. The CPU-intensive rendering
of watercolor maps is done on Amazon EC2 where we can invoke
extra machines as necessary, but the PostGIS, Mapnik and
cache storage parts are all living on an actual server in a
colocation facility. We decided last year to invest in
physical hardware to take advantage of the high random-access
speed of solid state disks, which make it possible to serve
the entire OpenStreetMap planet database without incurring
the overhead of Amazon’s terribly slow I/O speeds.
Fortunately, the back-end of this project is used to drive
a lot of our other work, so we’re folding the cost into a
series of different engagements that all use different
components of the map tiles.
PS
We appear to be reaching a tipping point where rolling
your own custom map stack seems not only practical but
desirable for many applications. What is your reaction to the
embrace of OSM by
prominent technology companies, and the emergence of
designer-friendly tools like TileMill? Do we have everything
we need for most designers and developers to create the map
experiences they want to provide? What tools would you’d like
to see built; what data sets made available?
MM
Honestly, I’m completely thrilled. With this year’s
high-profile addition of Foursquare and Apple to the
OpenStreetMap community, I’m looking forward to seeing what
new artists and designers decide to do with maps—I can’t
wait for the U.S. State Of The Map conference this
autumn.
I think there are two issues that current tools don’t
address well enough, and I’m excited to be working on them
both: medium-scale data for counties and towns, and more
options for bitmap filtering and output. I want Photoshop and
Illustrator in the sky, essentially, and tools like TileMill
help expose places where we need to be doing more with data
before rendering and more with pixels after rendering.
PS
Elaborate on what you mean by “medium-scale data”, and how
it would improve map-making.
MM
My colleague Nathaniel Kelso runs a project called
Natural
Earth Data, which offers global vector data at three
different scales, optimized for rendering images of large
regions, countries and continents, up to about zoom=9 if you
think in terms of web slippy maps. OpenStreetMap meanwhile
offers small-scale data down to the level of individual
carriageways on major streets. There’s a gap between these
two data sets where OSM is
too detailed and Natural Earth is not detailed enough, so on
many renderings you get bizarre selections of town names to
render, doubled-up street names, or no global context.
The Terrain
layer on maps.stamen.com is a vehicle for exploring a few
avenues through this problem: feature generalization for
route shields and large street names using the Skeletron library,
simulated annealing for smarter label placement with Dymo, and cross-blending of
raster data sets for ground cover and hill shading. We’re
doing a lot of work to make OSM data better for rendering, and
releasing all the component parts as software for processing
data sets.
PS
Do you see Mapnik as the appropriate place to grow the
bitmap filtering and output functions?
MM
As far as Mapnik’s role in all this, I think it’s the
single best place to do vector rendering, but I’m looking
elsewhere for filtering and output. I prefer to use tools
that are specialized for individual tasks, so we use the
pixel output of Mapnik as a source for Python-based pixel
manipulation code, often implemented in libraries like NumPy
that offer rapid manipulation of bitmap arrays.
PS
It sounds like you’re hinting at a new program for this
kind of manipulation. You and Stamen have contributed a great
deal of open-source map making software over the years; will
the code that created Watercolors be released?
MM
I hope I’m not sounding too coy; there’s not any secret
piece of software running the show, just a set of well-known
techniques in a new arrangement. I’ve released everything
I’ve ever started, but my role in Watercolor was more about
taking something that already worked and making it work a
little better so more people could see it.
PS
Most web maps are a pre-baked set of raster images
assembled in the viewport of the browser—do you see this
fundamental arrangement changing in the foreseeable future?
With WebGL and HTML5 Canvas, are we ready to composite maps
client-side with servers pushing vectors data over the wire?
What tradeoffs are there here? Do you think you could do
Watercolors entirely in the browser anytime soon?
MM
I don’t think we’re far off, though there are a few
impediments still in the way. Much of the core functionality
of Watercolor’s bitmap effects can be implemented entirely on
a GPU in a WebGL fragment shader, so there’s no reason that
we couldn’t build it that way. As far as shipping vectors,
I’d love to see it happen. It’s actually not unrelated to the
work we’re doing with differently-scaled data: you want your
underlying data to be at the “right” level of complexity, and
that generally means modifying it in some way, by dropping
extra points, scrunching narrow polygons into lines, making
small things disappear, and lumping groups of things together
into single blobs. If we can figure out a better way to
simplify on the fly, then complete client-side rendering
could be a reality. Of course, Google has already done a lot
of this themselves, but it’s different when the open source
community does it and shares the results and the
research.
PS
Simplification of lines and polygons is a common need for
mapping projects, not only for display but also for
interaction—clients can only handle so many points during the
refresh cycle. It can be a big challenge for workflows to
simplify vectors, and also to preserve topology, the
relationship between shapes—our model is of disconnected
points, lines, and polygons. That’s one reason I’m excited
about topology as a type coming in PostGIS 2.0. What
simplification techniques should we be using and exploring,
both on the algorithm side as well as the workflow side?
MM
I spent about two years (off and on) researching line
generalization. Skeletron went through three iterations,
starting with a port of the straight skeleton technique from
Tom Kelly’s
description, to a collaboration with Schuyler Erle
binding to the CGAL library
from Python, finally settling on an application of the
Voronoi diagram first
published by Esri in 1996. All the way through, I kept
thinking that there just had to be an easier way to make
simple lines out of complex ones, and why wasn’t I finding
code to do it for me? The simplification techniques we should
explore are all known and published, but exist largely as
plugins for systems like ArcGIS, instead of chainable tools
in the Unix style. PostGIS 2.0 is going to help in a big way,
and I hope that some of that effort migrates out to tools for
managing workflows around flat files.
If all goes well, it should start to make sense to ship
vectors over the wire and render them on the client.
It’s an interesting question to me whether client-side
rendering really something we want to aim for, though. Bitmap
tiles have an equilibrium of performance, size and design
that I don’t think will be disturbed any time soon. I’m
learning what I can about GPUs to be ready when the day
comes, but in the meantime most of my focus is on developing
workflows for data. OSM
extracts are one aspect of this; simplifying OSM data and standing up fresh new
rendering databases from source data is another. The
scaffolding that makes life easiest is a combination of bias
toward flat files and Postel’s Law: don’t screw around with
“seamless” servers, publish data flat using old file formats,
and only invent brand new things when they’re needed.
Spherical Mercator slippy map tiles are fast becoming one of
those well-understood old file formats, so this is what we’ve
aimed for with Watercolor; otherwise, how could we have a
section on maps.stamen.com showing how to use the imagery in
your own applications?
PS
That’s interesting—so are you generally fine with the way
Spherical Mercator has colonized the web map world? It seems
to have been a decision purely about tradeoffs: what
projection “works” for world-wide maps all the way down to
zoom level 22, or however detailed Google Maps gets these
days. But clearly choice of projection is a never-ending
debate. Is Spherical Mercator an acceptable enough
constraint, from your perspective, within which to do the
kind of artistic expression Stamen does, and maintain
interoperability?
MM
Oh yeah, I’m absolutely fine with it. It’s one of those
“assume a spherical chicken” engineering solutions that
actually leads to so much follow-on innovation that in
retrospect I’m glad no one was letting the cartographers
drive at Google in 2005. Projections only matter when you’re
looking at large areas, and that’s really not the case when
you’re searching for driving directions or checking out the
neighborhood where you spent elementary school. It’s
interesting to me, though, that the typical fifty states map
you see on every link bait infographic out there is based on
a conic projection—it’s about what looks right for a genre of
mapmaking, and in the case of slippy maps the Spherical
Mercator is clearly the obvious choice.
I have some exciting news: in a couple of weeks, I will be the new deputy
director of technology at the Democratic National Committee. I’ll
be helping to create software that helps get Democrats elected. It’s
a great opportunity for me to do what I love for an organization that I
passionately support. I hope to help make the technology department there
top-notch in terms of software engineering practices, bringing what I’ve
learned from having helped develop, deploy, and grow EveryBlock. And
I’ll have some projects of my own, which will hopefully advance the
state of campaign software somewhat. It’s especially exciting, too, to
be able to collaborate with friendsinChicagoworkingonthat one campaign.
I have been volunteering on political campaigns—federal, state, and
local races—for years, and have often lamented the state of campaign
software. It’s partly understandable, because campaigns tend to be
all-hands-on-deck, hair-on-fire affairs, where it’s hard to justify
long-range planning and software development, even if it might make
the lives of your staff, organizers and volunteers easier, since your
organization may not even exist for more than a few months. And campaigns
rarely have in-house software engineers, so opportunities to capture and
encode knowledge in the form of software, and explore new technologies,
are missed.
Obama For America gets this—that’s why they’ve hired like a start-up
for this campaign cycle, recognizing that great software is a competitive
advantage and no longer an afterthought you contract out for. And the DNC
gets it, too, and that’s why I’m excited to join them. The chance to
help re-elect this president, restore Democratic majorities in Congress,
and also to help down-ballot Democrats across the country in this and
future campaign cycles is one I couldn’t pass up.
I’ll be commuting to the DNC’s offices in Washington, D.C. from Baltimore
on a regular basis, though I’ll still be working from home a couple of days
each week, so that I won’t too miss much of this kind of stuff.
It’s a bittersweet development, because I’ll be leaving EveryBlock,
which I helped found 4 years ago. With the success of the recent
relaunch, though, I feel now is as good a time as any to step
away. The site is in great hands, and the response from users to the
new version has been enthusiastic. It couldn’t have a better home than
msnbc.com, who have provided great guidance and resources. I’m
thrilled with the success we’ve had and for how far it’s come, and
I’m confident that it will continue to be the best place on the internet
to help make your block a better place.
I am particularly grateful for having worked with my great EveryBlock
colleagues. I’m humbled by them and their talents and work ethic. It
was a privilege to learn from them and improve my craft, however modestly,
by their examples.
For now, I’m focusing on winding down at EveryBlock, and getting prepared
for a new commute (I’ll try to hack it with the MARC train and a bike)
and a campaign season now fully engaged. It’s a thrilling opportunity,
and I hope to make the most of it.
One month ago today, Maxine Mills Smith arrived in this world. Her mom gave
birth to her at Mercy Medical Center in Baltimore, Maryland, at six in the
morning. She was, and continues to be, a long and strong gal. She enjoys
being carried about and looking up at the changing scenery. Like her father,
she zonks out to the motion of a vehicle ride, be it stroller or car. She
has healthy lungs, likes to exercise them, and is a vocal chirper. She
has been respecting her mom and dad with 5 and 6-hour stretches of sleep
at night, but likes to mix it up from time to time and throw her mom some
napless curveballs. When she is awake, she is bright and alert. Given the
choice, she’d rather have some light assistance and try to stand and
monster march (I said she was strong) than squirm around on her belly like
a beetle. Her binky is a frequent companion, and she is a champion eater
of mom’s milk. The doctors and nurses at the pediatrician’s office
think she is doing fine, and give her an extra wink. We think she’s is
doing fine, too. Family love Maxine. Welcome home, my daughter.
Go takes an interesting approach to parsing strings to time objects,
and formatting time objects as strings. Instead of using codes like
most languages to represent component parts of a date/time string
representation—like %Y for a 4-digit year like “2011” or %b
for an abbreviated month name like “Feb”—Go uses a mnemonic device:
there is a standard time, which is:
Mon Jan 2 15:04:05 MST 2006 (MST is GMT-0700)
Or put another way:
01/02 03:04:05PM '06 -0700
Instead of having to remember or lookup the traditional formatting codes for
functions like strftime, you just count one-two-three-four and each place
in the standard time corresponds to a component of a date/time object (the
Time type in Go): one for day of the month, two for the month, three for
the hour (in 12-hour time), four for the minutes, etc.
The way you put this into action is by putting together the parts of
the standard time in a layout string that matches the format of either
the string representation you want to parse into a Time object or the
opposite direction, when you want to generate a string representation from
an Time object.
Parsing:
package main
import (
"fmt"
"time"
)
func main() {
value := "Thu, 05/19/11, 10:47PM"
// Writing down the way the standard time would look like formatted our way
layout := "Mon, 01/02/06, 03:04PM"
t, _ := time.Parse(layout, value)
fmt.Println(t)
}
// => "Thu May 19 22:47:00 +0000 2011"
There are predefined constants in the time package for common formats
such as RFCs 822 and 3339.
The use of a mnemonic over obscure formatting codes I think reflects the
pragmatism of Go’s developers and their focus on creating a language
that makes its users more productive. While it is a break with tradition
to abandon strftime-style formatting, they probably recognized that most
developers, no matter how experienced, reach for man strftime or similar
documentation more often than not (hell, I have a mug with the codes printed
on it), and each time they do, it is an expensive context switch, in terms
of lost focus. Writing the standard time down the way yours looks may be
quirky, but it's easy to recall, and it also happens to match the shape of
your time string, syntatically.
It’s fascinating to see a language with the pedigree of Go—from the
minds of long-time C and Unix developers—toss aside certain old and
venerable ways of doing things. But it’s consistent with a language
that’s relatively small (in the good way of being comprehensible and
coherent), focused on efficiency, and careful in what it includes.
Motivated by irrational hatred and overstated claims of the continuing
utility of a 3.5" floppy disk icon to mean “save”:
Handset icon, meaning “phone,” on every iPhone, Android
email
Website (implies a physical place)
Wrench or cog icon, meaning “settings” or “preferences”
Shopping cart icon
Clock with analog hands, meaning “time” or time-centric application
Incandescent light bulb, several meanings, including energy settings
Bound, paper book, meaning “book”
Of course there are many, many anachronistic interface elements and metaphors
in the world, and we get by okay. Mainly,
this is because new generations don’t suddenly appear next to us and
start using our computers without any foreknowledge of the metaphorical
items. They spend time learning with the old fogeys for whom information was
sometimes stored inside square pieces of plastic and metal. This overlap is
necessary in general because knowledge about tools is not encoded in our
genetic material. All understanding of use is part of a multi-layered
strata of culture, experience, and relationships.
The real problem anti-floppy-disk people have is explaining a harm,
specifically, a harm that matters. Often when we think of user interface
improvements that matter, we think of examples like improving medical devices
to reduce user error, changes that literally save people’s lives. Or
tweaks to software that improve user efficiency and productivity, saving
money. It’s hard to conceive of what might be improved by finding a better
metaphor for “save” other than some designers’ personal sensibilities.
In my previous article, I took a
superficial look at how Redis starts up and prepares itself to process
commands. In this article, I'll follow a GET and a SET command as
they move from client through the server and back. The GET will be for
a key that doesn't exist, and the SET will set that key. Then I'll look
quickly at a subsequent GET and how it differs.
As before, I'm exploring Redis with the source code open in my editor and
indexed with a tags file, and the Redis server running under gdb in
another terminal.
Caveat: this article was written against the codebase of Redis 2.2.1. With
respect to my previous article, for a list of what has changed in Redis since
I wrote it, see this comment on
HN.
Edit: I made two minor changes based on feedback—Redis keys are not Redis
objects, they are sds strings; and you don’t have to hack the Makefile to
compile without optimizations.
GET
Let's execute the command get users:1234 on Redis.
Preparing
If you inspect certain variables under gdb, you might not get what you want.
Instead you see a message like "." This is because the compiler has optimized the machine code
in such a fashion that the portion of memory you wanted to look at that was
never used, at least for the variable under inspection. To make stepping
through the code and inspecting a little easier, we make sure that Redis is
not compiled with optimizations. You can do this by either building Redis with
the following invocation:
make noopt
or by setting an environment variable:
OPTIMIZATION= make
Sending the command from the client
If we look at the repl loop of redis-cli, we see that it uses the
linenoise library to split the arguments of the command. It dispatches on
the first argument. The loop checks for client commands that aren't processed
as a command by the Redis server, like exit/quit, clear (to clear the screen),
and connect (which is a way to connect to a specified Redis server on host
port, instead of the default host of 127.0.0.1 and port of 6379.
Any other argument is considered the name of a command to send to the server.
The remaining arguments are the arugments for that command.
We're trying to get the on-the-wire representation of the get users:1234 command.
redis-cli uses a redisContext struct to encapsulate the state of the
connection to the server, as well as the output buffer where the command in
the form of the Redis protocol is written for sending to the server. We know
from reading the source of hiredis (the Redis C client library that the redis-cli
program is built on) that the obuf field is where the raw command is stored:
# hiredis.h:107
/* Context for a connection to Redis */
typedef struct redisContext {
int fd;
int flags;
char *obuf; /* Write buffer */
int err; /* Error flags, 0 when there is no error */
char *errstr; /* String representation of error when applicable */
/* Function set for reply buildup and reply reader */
redisReplyObjectFunctions *fn;
void *reader;
} redisContext;
If we set a breakpoint on cliReadReply, we can inspect the output buffer and
see exactly how the command looks as a bytestring bound for the server.
client $ gdb src/redis-cli
(gdb) b cliReadReply
(gdb) run
Starting program: /home/paul/src/redis-2.2.0-RC2/src/redis-cli
Reading symbols for shared libraries +. done
redis> get users:1234
Breakpoint 1, cliReadReply (output_raw_strings=0) at redis-cli.c:421
421 if (redisGetReply(context,&_reply) != REDIS_OK) {
(gdb) p context->obuf
$1 = 0x100102428 "*2\r\n$3\r\nget\r\n$10\r\nusers:1234\r\n"
We see that the Redis command get users:1234 as entered in our client is
translated to *2\r\n$3\r\nget\r\n$10\r\nusers:1234\r\n. Any Redis client is
going to convert our command expressed in its respective syntax to the same
on-the-wire format. So in Python:
>>> redis_client.get('users:1234')
will send the same *2\r\n$3\r\nget\r\n$10\r\nusers:1234\r\n bytestring to
the server.
Let's print that bytestring to screen and render those `\r\n's as line feeds
so we can see an expanded view and get a better look at the protocol.
*2
$3
get
$10
users:1234
The first bit is *2, which tells us that the arity of the command,
including the command name, is 2. That is, the server should expect two
arguments to follow.
The next bit is $3, which means that the length of the first argument in
bytes is 3. The argument itself follows, our command name, get.
The next bit after that is $10, so the length in bytes of the second
argument is 10. Our one and only argument to the command is next,
users:1234, the key we are trying to get.
Receiving the command on the server
If you remember from the last article, readQueryFromClient is a good place
to set a breakpoint in your debugger on the server side for inspecting an
inbound client command.
server $ gdb src/redis-server
(gdb) b readQueryFromClient
Breakpoint 1 at 0x100011445: file networking.c, line 882.
(gdb) run redis.conf
Starting program: /home/paul/src/redis-2.2.0-RC2/src/redis-server redis.conf
Reading symbols for shared libraries +. done
[63700] 01 Mar 11:04:40 * Server started, Redis version 2.2.1
[63700] 01 Mar 11:04:40 * The server is now ready to accept connections on port 6379
Now back in the terminal with the client running in the debugger, continue to
send the command to the server, which will stop at the breakpoint we set.
# client
(gdb) c
Continuing.
# server
Breakpoint 1, readQueryFromClient (el=0x100200000, fd=5, privdata=0x100804e00, mask=1) at networking.c:801
801 void readQueryFromClient(aeEventLoop *el, int fd, void *privdata, int mask) {
If we step to here in our debugger, we see that the server read 30 bytes. If
you count the number of bytes in our Redis protocol-encoded command,
*2\r\n$3\r\nget\r\n$10\r\nusers:1234\r\n, you'll see it's 30. Just for good
measure, let's look at the 30 bytes beginning at the memory location pointed
to by buf:
And we can see our whole command bytestring is there, byte by byte.
The server has now read in the entirety of our command in one step. (Because
we had a relatively short command, one that fits inside a kernel buffer, and
we are the only client on a loopback network device, this is the case, but it
need not be. Since Redis is event-driven, this function,
readQueryFromClient, is called whenever there are bytes from buffers to be
read. If our command was particularly long, or there was a lot of network
contention, the command may take more than one I/O event before it is fully
read. For this reason, Redis builds up a buffer per client and appends bytes
to it on each call to this function. It only proceeds with processing the
command when it has been fully read. But we don't need to consider this in our
simple example, so we will proceed.)
We're going to elide the processing of the input buffer. This is the point
where the server takes the Redis protocol-encoded bytestring of our request
and unpacks it into arguments on the client struct object. If you are
interested in the details of that parsing, examine the function
processMultibulkBuffer in networking.c. All we are interested in at this
point is that the argc member of the client object is the number of command
arguments (counting the command name itself) and argv is a pointer to the
list of arguments.
The bit of code we care at this point is processCommand. The first thing the
server does is look up the command in its command table (see "Setting up
command table" in the previous article, but note that this lookup is now O(1),
see the HN thread linked above). Assuming the command is found (which our
get will be), the server will double-check that the arity of the command as
defined in the command table matches the number of arguments received from the
client (c->argc).
# redis.c:998
cmd = lookupCommand(c->argv[0]->ptr);
if (!cmd) {
addReplyErrorFormat(c,"unknown command '%s'",
(char*)c->argv[0]->ptr);
return REDIS_OK;
} else if ((cmd->arity > 0 && cmd->arity != c->argc) ||
(c->argc < -cmd->arity)) {
addReplyErrorFormat(c,"wrong number of arguments for '%s' command",
cmd->name);
return REDIS_OK;
}
Skip down to the end of processCommand. Because our humble get is not a
"multi" command like mget, mset, etc., it doesn't require queue-like
processing of the underlying multiple commands, so we go right to call,
which is where our command is dispatched.
# redis.c:953
void call(redisClient *c, struct redisCommand *cmd) {
long long dirty;
dirty = server.dirty;
cmd->proc(c);
dirty = server.dirty-dirty;
if (server.appendonly && dirty)
feedAppendOnlyFile(cmd,c->db->id,c->argv,c->argc);
if ((dirty || cmd->flags & REDIS_CMD_FORCE_REPLICATION) &&
listLength(server.slaves))
replicationFeedSlaves(server.slaves,c->db->id,c->argv,c->argc);
if (listLength(server.monitors))
replicationFeedMonitors(server.monitors,c->db->id,c->argv,c->argc);
server.stat_numcommands++;
}
Let's focus on line 952, cmd->proc(c);. This is Redis's dynamic dispatching
of command function calling. Redis makes this clean and simple by
encapsulating commands and giving all the actual underlying command functions
the same function signature, taking our client object, which carries the
payload of our command's arguments. So we're interested in looking into the
details of the Redis command object and the actual function that will handle
our get.
# redis.h:504
struct redisCommand {
char *name;
redisCommandProc *proc;
int arity;
int flags;
/* Use a function to determine which keys need to be loaded
* in the background prior to executing this command. Takes precedence
* over vm_firstkey and others, ignored when NULL */
redisVmPreloadProc *vm_preload_proc;
/* What keys should be loaded in background when calling this command? */
int vm_firstkey; /* The first argument that's a key (0 = no keys) */
int vm_lastkey; /* THe last argument that's a key */
int vm_keystep; /* The step between first and last key */
};
If we pop up to the top of redis.c, we see the definition of the Redis command
table, and our get is the first entry.
The arguments to lookupKeyReadOrReply are the client object, the key
users:1234 we're trying to look up, and an object, shared.nullbulk that
will be the default reply to the client if the key is not found.
lookupKeyRead is a thin wrapper for lookupKey that handles removing keys
that have been set to expire.
Now we get to the heart of the get command -- looking up the key in the
database.
# db.c:9
robj *lookupKey(redisDb *db, robj *key) {
dictEntry *de = dictFind(db->dict,key->ptr);
if (de) {
robj *val = dictGetEntryVal(de);
/* Update the access time for the aging algorithm.
* Don't do it if we have a saving child, as this will trigger
* a copy on write madness. */
if (server.bgsavechildpid == -1 && server.bgrewritechildpid == -1)
val->lru = server.lruclock;
if (server.vm_enabled) {
if (val->storage == REDIS_VM_MEMORY ||
val->storage == REDIS_VM_SWAPPING)
{
/* If we were swapping the object out, cancel the operation */
if (val->storage == REDIS_VM_SWAPPING)
vmCancelThreadedIOJob(val);
} else {
int notify = (val->storage == REDIS_VM_LOADING);
/* Our value was swapped on disk. Bring it at home. */
redisAssert(val->type == REDIS_VMPOINTER);
val = vmLoadObject(val);
dictGetEntryVal(de) = val;
/* Clients blocked by the VM subsystem may be waiting for
* this key... */
if (notify) handleClientsBlockedOnSwappedKey(db,key);
}
}
server.stat_keyspace_hits++;
return val;
} else {
server.stat_keyspace_misses++;
return NULL;
}
}
Redis uses its own hash table implementation to store keys and their values in
memory. Inside the db object, the field dict is a pointer to the hash
value for the current Redis database (remember there can be up to 16 separate
databases in a single Redis server instance, indexed by number).
First, Redis calls dictFind with the database's hash table and a pointer
to the key's bytestring. dictFind looks up the hash of the key in the
table, using a standard algorithm that should be familiar to anyone who's
implemented a hash table (check out dict.c starting at line 391 if you're
interested, the table is an array with linked lists for colliding hashes).
If the key is found in the table, dictFind returns a pointer to the entry in
the table. Otherwise, it returns NULL. Back in lookupKey, if the entry is
not null, Redis retrieves the value (i.e., the Redis object our key
references) from the hash table via dictGetEntryVal and takes care of a bit
of bookkeeping for expiry and VM, if the key was found, and stats in either
case (hits and misses). If the entry was NULL, then lookupKey also returns
NULL; we'll see how this is handled by Redis for a reply to the client when
the key is not found, which is the case for us at this stage.
With the value of lookupKey, we'll go back up the stack to our callers. Back
to lookupKeyReadOrReply, we look at line 60:
# t_string.c:60
if (!o) addReply(c,reply);
Since we got NULL from lookupKey this time, we call addReply. The value
of reply here comes from the call in getGenericCommand, and it is
shared.nullbulk. This field in the global struct object shared is
initialize thusly:
We can see that it is a Redis string object who's on-the-wire value is
$-1\r\n, meaning a length of -1, Redis's way of indicating null to a
client, according to the protocol.
addReply builds the reply to the client. It does this by first setting up a
write event on the main event loop listener with _installWriteEvent. This
makes sure that the reply is written out to the client connection when there
are bytes present in the buffer. Next, Redis adds the reply to the client's
buffer. If the reply object were an non-string value like an integer, or a
list, or a set, Redis would first decode it to a bytestring that can be
serialized to, for example, on a socket. Redis string objects are encoded
"raw," or as-is. The nullbulk object is technically a string object, so no
decoding is necessary in our case. In any case, the reply bytestring is copied
to the client's reply buffer with _addReplyToBuffer, which for all intents
and purposes completes the execution of our get command on the server.
The client will read the on-the-wire reply of $-1\r\n and know that it is a
string reply of length -1, and therefore is the null (or "nil," in the context
of redis-cli) object, and to convert that into the appropriate object for
the language of the client. Back to our redis-cli client patiently waiting
for a reply from our breakpointed server, which we continue from, that looks
like:
(nil)
redis>
SET
The set command proceeds much the same way as the get, up to the point of
command dispatching on the server.
This time, our protocol-encoded bytestring is 47 bytes long, owing to the
extra argument "Paul Smith" and the length tag it requires. Also notice the
leading *3 indicates there are three arguments: set, users:1234, Paul Smith.
Let's skip ahead now to the point in call in redis.c, after the command has
been looked-up in the command table and the server is about ready to call the
underlying proc function with the client object argument. The set command,
in the form of a redisCommand struct, looks like this:
Notice that the arity of the command is 3, which includes the leading command
name, plus key and value, and matches what we expect from the client. The
set command has a flag that get did not: the constant REDIS_CMD_DENYOOM
means that, in out-of-memory situations where Redis can't allocate any more
memory, the execution of the command should be denied. (The absence of this
flag means that Redis can continue to serve client "read" requests like get
even when the server can no longer write any new data.)
I set a breakpoint on setCommand and let the server continue running until
that point:
# server
(gdb) b setCommand
Breakpoint 2 at 0x10001a6e2: file t_string.c, line 48.
(gdb) c
Continuing.
Breakpoint 2, setCommand (c=0x100805e00) at t_string.c:48
48 c->argv[2] = tryObjectEncoding(c->argv[2]);
Incidentally, you can inspect the values of the client's command arguments at
any time, with a simple gdb invocation. The arguments are of type robj,
which has a field ptr that is a pointer to the actual value in memory. Since
in our set case these are strings, we can inspect them by typecasting to
char * like so:
(gdb) p (char *)c->argv[0]->ptr
$10 = 0x10032ae78 "set"
(gdb) p (char *)c->argv[1]->ptr
$11 = 0x10032b068 "users:1234"
(gdb) p (char *)c->argv[2]->ptr
$12 = 0x10032b098 "Paul Smith"
The first thing the server does in setCommand is encode the value being set
with tryObjectEncoding. It will try to create an efficient encoding if the
bytestring can be interpreted as an integer, for example. This can save space
especially in the case where many numbers are being stored. Additionally,
Redis will try to reuse shared integers as values instead of allocating
resources for new ones -- see the previous article for more on the creation
and use of shared integers.
Once the value being set has been encoded, setGenericCommand is called
(set shares setGenericCommand with the setnx and setex commands). From
here, dbAdd is called, with the client, key, and value as arguments. dbAdd
will only add the value to the database's hash table if the key does not
already exist. In our case, since the key users:1234 does not exist, the
value of dictFind is null, and the function proceeds to add the value with
dictAdd.
dictAdd takes the dictionary hash table, key, and value as arguments. It
uses _dictKeyIndex to find the index of a free slot in the hash table for
our new entry. See the implementation in dict.c and dict.h for the details
of key hashing, and the structure of the dictionary and its component hash
tables (each Redis dictionary contains two hash tables in order to provide
incremental rehashing as the dictionary grows). dictAdd allocates memory for
the new entry and stores it in the new index.
The server returns back up the stack from dictAdd and dbAdd to
setGenericCommand, where it increments the reference count on our new value.
Redis uses reference counting in order to free memory used by values that have
been deleted or have expired. It then "touches" the key so that if any clients
are watching the key, the next exec command will fail. It also increments
the server's dirty flag, which it uses to determine when to write out the
dump file to disk. Finally, it writes out the reply to the client, which is a
shared object, shared.ok. This is special Redis string object in the
protocol that consists of the bytestring "+OK\r\n". Clients will typically
convert this into the equivalent "true" value for their language.
GET redux
Our key is now set, so we can try the get users:1234 command again and see
how it differs for a found key.
# db.c:9
robj *lookupKey(redisDb *db, robj *key) {
dictEntry *de = dictFind(db->dict,key->ptr);
if (de) {
robj *val = dictGetEntryVal(de);
// ... skipping the lru & vm parts ...
server.stat_keyspace_hits++;
return val;
The point where a get on an existing key and a get on a non-existent key
differ is line 11, where the de entry in the database's dictionary is found.
dictGetEntryVal is a simple macro for accessing the field in the de struct
that carries the value associated with the key. Redis updates its statistics
to indicate a key hit and returns the value object.
Again, as with the key miss from above (remember the null value is a Redis
object, too), the value is decoded into the Redis bytestring protocol. This is
the response to the client, and we have concluded our GET/SET/GET dance.
In honor of the all-union Super Bowl, I present the
All-Blue-Collar League, a collection of U.S. professional sports franchises
named after local unions, industry.
We won’t return to a gold standard because changing to a fiat currency was
one of the great innovations of modern economics. You aren’t literally
and figuratively weighed down by gold, and that allows central banks and
governments to set policy that allows them to rebound from crises, spur
growth, and lift people out of poverty and short lifespans. As Liaquat
Ahamed points out, the gold standard is also a sure way to get yourself
in a deflationary cycle.
But, wait! gold bugs will say. That paper isn’t worth the paper it’s
printed on. Gold has real value, that’s why it was the world’s
currency for many years. The problem with this is that money is a
turtles-all-the-way-down concept. Gold is just as much a pointer
to value as paper is, except it has many more downsides: it’s heavier,
more cumbersome to exchange, and has a fixed supply.
Gold isn’t valuable in and of itself. You can’t live in gold. You can’t eat
gold. You can’t clothe yourself in gold. A gold coin can’t paint your house,
or build you a website. You could (hypothetically) use it to put your kids
through college, so it that sense it’s valuable to you, but only in its
exchange, because on the other end, the trustees and professors and janitors
at the college need to live somewhere, eat something, wear something,
and have websites built. And again they exchange it for those goods and
services. So really the only thing that’s intrinsic to this little economy
I’ve just described is the fact that you, the college trustees, the service
providers, et al., all agree on the value or rate of exchange. It’s like
an algebraic equation where constants cancel out — gold is the constant,
so we can abstract it away. It may as well be “neighborhood fun bucks,”
so long as everyone agrees on the rate of exchange for goods and services.
And there you have it. Economies have nothing to do with the particulars of
their currencies, and everything to do with people and institutions trusting
that, in exchange for some amount, I will receive something of perceived
equal value. Trust, and perception. Currencies are just a convenient
means by which we mitigate that exchange, an abstraction, a stand-in
for a hypothetical exchange, ultimately, an exchange built on trust.
Interestingly, the only inherent value gold has is in its use as
jewelry. Which, aside from any aesthetic concerns, couldn’t be more
appropriate for our discussion, because people wear gold jewelry as a
signifier of wealth. A pointer, a marker, an abstraction for the wealth
itself, which, again, is just the potential for exchange.
It is this essential trust network at the core of economics that explains
why there is so much noise about gold. It reflects a lack of trust on the
part of gold advocates in our central banking and governmental institutions,
which they feel pursue policies that aren’t in their interest, or are in
the interest of undeserving populations. The switch to a gold standard,
therefore, has nothing to do with inflation or any other “economic”
concern, and everything to do with pursuing a political objective. The
market of exchange in this little political economy is for ideas about
reprioritizing our institutions. Which is a legitimate topic of national
debate. But reverting to the gold standard, as a practical economic matter,
would be bonkers.
I was struck by the high-level abstractions Go
provides for writing networked clients and servers, even by the standards of
conventionally high-level languages like Python. Given Go’s identification
as a systems programming language, this is initially surprising, because
such languages tend to be low-level for performance reasons (Go does
expose a lot of low-level interfaces through various packages like os
and syscall), but of course there would be good support for these
things because Go is a modern language designed by folks with many years
of experience writing networked servers.
Specifically, the net package provides the Listener and Conn types
that hide many of the details of setting up socket connections. A simple
net.Listen("tcp", ":1234") is enough to get the equivalent of a listener,
or server, socket. The bufio package provides buffered read methods,
simplifying common tasks like reading lines from a socket.
Let’s dive in and take a look at one way to implement a classic echo
server. I tried to use Go’s language features for concurrent communication,
goroutines and channels, at crucial points. Where a different implementation
in different language might fork a new process or spawn a new thread to
handle a new client connection, a goroutine was started and the details
of the connection communicated with a channel.
package main
import (
"net"
"bufio"
"strconv"
"fmt"
)
const PORT = 3540
All Go programs must have a package main. We declare a number of packages to
be imported, net and bufio already mentioned, strconv for its conversion
function Itoa() that converts an integer to a string, and fmt for printing
strings to the console. Our echo server will listen on port 3540.
func main() {
server, err := net.Listen("tcp", ":" + strconv.Itoa(PORT))
if server == nil {
panic("couldn't start listening: " + err.String())
}
conns := clientConns(server)
for {
go handleConn(<-conns)
}
}
Every Go program must have a function main() in its package main. We start
by declaring and initializing a new listener for our echo server. We choose
the more generic net.Listen() over net.TCPListen(), because we can
conveniently parameterize the type of the listener with a string to the first
argument of Listen. (It is generally preferred in Go to deal with the most
generic type, or interface, especially when writing function and method
signatures, and either allow a specific type to be inferred by the compiler or
specify it when declaring/initializing a variable.) ":" + strconv.Itoa(PORT)
is a string concentation expression, and the port constant we defined earlier
is converted to a string. (Strictly, a numeric constant is not an integer of a
particular type, but the string conversion function works here because the
compiler converts the constant into the concrete type to match the function
signature.)
The idiom for Go when handling multiple return values where the last has an
error type is to check the main object for nil, and then panic() with the
stringified error object. This is generally preferred over printing to stderr
and using os.Exit(), because it gives callers higher up the stack the chance
to recover(), sort of a raise/catch exception flow.
Using Go’s compact syntax for simultaneous declaration and initialization, we
set conns to the value of the function call clientConns(server). This is
the channel we’ll use for getting new client connections.
The equivalent of a infinite loop like while True: or for (;;) in Go is
for { ... }. Each time through the loop, we start a goroutine, calling
handleConn() with the value of the receive operation on our client connections
channel. The unary receive operator <- blocks until a value is available on
the channel, in our case, a new client having connected.
func clientConns(listener net.Listener) chan net.Conn {
ch := make(chan net.Conn)
i := 0
go func() {
for {
client, err := listener.Accept()
if client == nil {
fmt.Printf("couldn't accept: " + err.String())
continue
}
i++
fmt.Printf("%d: %v <-> %v\n", i, client.LocalAddr(),
client.RemoteAddr())
ch <- client
}
}()
return ch
}
We create a new channel of type net.Conn, which corresponds to the type that
we’ll get from calling Accept() on our listener connection object. We start
off a new, anonymous goroutine which runs in an infinite loop, constantly
accepting new connections. listener.Accept() blocks as long as there are no
new clients to deal with, but since we’re running inside a goroutine and
"detached" from the main program flow, other already-connected clients can
continue to be handled without interruption (this is where a fork() or a new
thread would happen in a typical server). Instead of panic()ing here if
there is an error connecting with the client, we simply note it on the console
of the server and move on. We also keep track of the number of clients we’ve
seen with the counter i. fmt.Printf() works like you’d expect, though the
%v format is not like anything in C’s printf() — it prints a value in a
default format and works for any type, somewhat like repr() in Python.
The binary <- communication operator is used to send the client, of type
net.Conn to the channel we created at the top. Go has lexical scope (and is
garbage-collected), so ch is available inside the anonymous goroutine and
after our function returns (because the infinite Accept()ing for-loop keeps
the goroutine alive).
func handleConn(client net.Conn) {
b := bufio.NewReader(client)
for {
line, err := b.ReadBytes('\n')
if err != nil { // EOF, or worse
break
}
client.Write(line)
}
}
As we saw back in main(), handleConn() is invoked as a goroutine with each
new client connection (the connection being received from the other side of
the channel we just created). bufio.NewReader() wraps the client object with
a nicer interface for reading lines of bytes from. If we decided to use the
Read() method of the net.Conn object, our code would be more complex,
having to check for the substring "\n" and testing for EOF. Instead, we can
treat this as a line-oriented protocol, and simply get a line and write it
back to the client, as long as there are lines to be read.
The pattern of getting new connections from a channel and starting a goroutine
to handle them is conceptually clean, and also happens to be a straightforward
way to write a multiplexing, concurrent server.
The Gawker breach made mainstream what’s been known for a
while: we don’t use good passwords, and we use the same password over
and over again on multiple sites, exposing us to a potential chain
reaction of account intrusions if it is uncovered.
One way to address this problem is to use a password generator—the
idea is that all you have to remember is a single, good, strong
password, and the password generator will spit out a random string of
characters to be used as a password on a particular site. This
generated password is hard for an attacker to guess, and hard for them
to “crack” with password-attacking software. You’re not supposed to
remember or even care what the actual password is: you just remember
the master password. And if one of the sites you have an account with
is compromised, it may suck for you with respect to your sensitive
information on that site, but because the generated password is unique
to the site, you’re safe from having it used against you on other
sites where you have accounts.
The password generator I use is called Oplop. It’s just a web
site, so you visit it in your browser; there’s nothing to
install. There are two fields to enter—a nickname, and the master
password. The nickname is any name you want to identify the site or
service for which you need a password. After you enter a nickname and
your master password, you click the “Create” button, and an
unguessable password is generated for you in the field below the
button. Enter that password into the site that’s asking you for a
password, and you’re all done. The next time you visit the site and
need to enter your password, you repeat the process exactly to get the
same unique password that’s just for that site.
There are a few tips and tricks to using Oplop successfully. The
nickname is just a label that you remember to refer to the site you
need a password for, but if it is different by even one character, an
entirely different password will be generated. So it’s important to be
consistent, and ideally use a simple convention so that you enter the
label the same way every time. I’ve chosen a convention of using
all-lowercase and no non-alphanumeric characters, and the domain name
of the site, minus the top-level domain (the .com or .org, etc.). For
example, if I’m on amazon.com, I’ll use “amazon” (without the quotes)
for the nickname. If I was visiting a site with the domain name
“Cat-Pics-123.net”, I’d use “catpics123” for the nickname. Whatever
your convention for nicknames, it should be simple and you should use
it consistently.
Your master password should be a good one—fairly long and hard to
guess—since it is the key to the kingdom. Security researchers often
praise the idea of “passphrase,” a sentence instead of a single
word. These can often be easier to remember and are just as strong if
not stronger than shorter, gobbledygook passwords, even if the
passphrase is composed of simple words. A four or five word sentence,
separated with spaces, should suffice, although longer passphrases are
even better. In any case, use care when entering your master password
into Oplop. By default, it doesn’t do any checking, so, like in the
case of the nickname, a single character’s difference means a whole
different generated password. You can optionally click on the checkbox
“Validate master password?”, which adds an additional password field
for you to re-enter your master password, providing a simple check
that might catch a typo in the first field.
The final tip is to cut-and-paste the generated password out of its
field instead of copy-and-paste. It’s not so much a tip per se because
Oplop recommends that you do this right on the page, but it’s
important enough to reiterate. Doing it this way prevents someone from
snooping over your shoulder, or if you are careless and leave the
Oplop page open untentionally on a shared computer.
You might have stopped short at the notion of a web-based password
generator. Isn’t it insecure to trust a web site to generate a
password for you? The answer is, you are only actually interacting
with your browser—there is no network activity taking place when you
generate a password with Oplop. That is because the entire Oplop
program is just HTML and JavaScript loaded in your browser, and it is
your browser on your local computer that takes the nickname and master
password, performs the cryptographic magic on them, and outputs the
generated password. In fact, Oplop works offline just as well. You can
test it by visiting the site and bookmarking it on your iPhone or
Android, deleting the browser window, putting the phone in airplane
mode, then loading Oplop from the bookmark. It works just the same as
if you were connected to the internet.
The browser-centric nature of Oplop gets to another key concept about
it—there’s nothing to install. In fact, it is essentially already
installed on any internet-enabled device with a JavaScript-enabled
browser. There’s no central database of account names and generated
passwords, like you might expect, and which password generators like
1Password and Password Gorilla are built around. These programs work
similarly to Oplop in the end result, but their central database
presents a user experience problem—it needs to be shared among the
computers you want to use the program on. This can make it difficult
to retrieve a password if you are on your friend’s laptop or a PC at
an internet café. Not only do you have to install the password
generator software itself, if you are allowed to by policy of the
computer’s owner or manager, you have to synchronize the central
database with the computer. By simply eschewing the notion of storing
account names and passwords, Oplop provides a universally-accessible
password management tool. Those unique generated passwords are just
ephemera, the results of combining the secret you keep in your head
(your master password), the site nicknames, and a cryptographic
function in your computer. Even the notion of the “master password” is
turned on its head—it’s not unlocking some database of passwords, it’s
combined with the nickname to make a cryptographic hash, the
bits of which make up the final generated password. You could start
using a different master password at any time, which essentially would
invalidate any extant passwords generated with the old one (you still
would have to manually go in and update each password on each site,
though).
There are other approaches to securing your passwords, including
writing them down. In fact, if you have trouble
remembering strong passwords or passphrases of sufficient length, you
might consider writing down your Oplop master password on a slip of
paper you keep in your wallet.
Oplop is not perfect—it outputs a fixed style of generated passwords
that may not meet the password criteria for various sites and
services, which require certain lengths, or a certain mix of
characters. It also doesn't address the need of strong,
crypotgraphically-secure storage of ad-hoc information such as credit
card numbers and other sensitive account information, which are often
stored alongside passwords and account names in more traditional
central database-style password generator programs. But this is by
design and not a failing of Oplop. I continue to use Password Gorilla
for this scenario, especially for protecting financial
information. Also, there are certain threats Oplop can’t protect
against, And it’s worth understanding what they are before
you start using it.
Oplop addresses the proliferation of passwords across multiple web
site accounts, is simple to use, and works everywhere there’s a
browser. So stop using “letmein” for all your web site accounts—you’d
be much better off plugging that in to Oplop as your master
password. But don’t. Use a better one.
Motivated by irrational hatred and overstated claims of the continuing
utility of a 3.5" floppy disk icon to mean “save”:
