Meteor Demystified

In which I open the hood of the Meteor framework and see what’s inside.

I recently started a new web project, and several friends suggested I check out Meteor, a new framework for building real-time webapps. Their intro screencast is impressive, and the project generated a lot of excitement when it was first announced. It lets you write real-time apps focusing just on your domain model and views, without all the plumbing—like Rails did for regular websites several years ago.

At the same time, the prospect of using Meteor for a production app would make any battle-scarred devops veteran nervous. The current version as of this writing is 0.6.3, which they call an “early preview”, saying: “Meteor is still under rapid development. Expect major API changes in each release.” (When will it hit 1.0? “More than a month, less than a year.”) The list of projects still has a lot of toy/demo apps; it’s unclear if anyone uses it in production on a major site.

Moreover, Meteor is “magic.” How does it work? What is it really doing? Will you even be able to understand it?

Despite all this, the prospect of rapid development of a real-time app was so seductive that I decided to look into it thoroughly—in effect, to take it apart and see how it works—to get underneath the magic.

What follows is a summary of what I learned from this exercise, which I haven’t seen explained quite this way anywhere else.

 Meteor’s central goal

To understand Meteor you need to understand its central goal, which drives the design.

The core problem Meteor solves is a data synchronization problem. Your webapp is a distributed system, with data on both client and server. The main challenge is to synchronize the data instantly and seamlessly between the two—and more, to synchronize all clients with each other. Ideally it should appear to all users that they are interacting directly with a single, shared store of data.

To solve this, Meteor gives you:

  1. A local copy of the data model, easily accessible via lookup and query interfaces. (This is common to all webapps that take the approach of client-side MVC.)

  2. An open socket to the server to get real-time updates (over WebSockets, long polling, or a similar transport. This is common to all real-time apps.)

  3. A latency-hiding mechanism that makes user updates appear to happen instantly. The method is clever: The core business logic exists on the client as well as on the server, and when the client sends a request, it also runs that logic locally, without waiting for the response—in effect simulating the server and predicting the outcome. (This is possible because JavaScript is the language on the server as well as the client, so the same code can run in both places.)

    This is an optimistic protocol: in the normal case, it hides all the latency of the network request; if there is an error or race condition, the client will get the authoritative response from the server soon enough. However, it doesn’t preclude security: the server is still the authoritative source of data; the client has no extra privileges. And you can share only a subset of the server code with the client, if you want to keep some of it private.

  4. A dependency-tracking mechanism to automatically refresh views when their underlying models change. In most client apps, the view or controller observes its model in order to refresh the view when the model is updated, but usually these dependencies must be wired up by hand. Depending on the app, this can be tedious and error-prone. Meteor provides a subsystem to track these dependencies automatically.

  5. A sophisticated template management subsystem that allows you to refresh or re-render views at any time without disturbing the user. For instance, the state of all inputs is preserved, so if the user has entered text it is not lost on a refresh. This makes it safe to do refreshes frequently and automatically by the mechanism just described.

The first three of these together mean that all interactions are local; the sync to the server, in both directions, is decoupled from user interaction. The last two mean that the view is kept in sync with the model by default, with minimal developer effort.

All together you get the central goal: a distributed app with the look and feel of a local one.

 What you get: The parts list

With that context, let’s open the box and see what’s inside.

Here are the key pieces Meteor comes with. All of this is described in the documentation, which I’ll link to where appropriate:

Read the docs to learn more; they are clear and helpful.

 Concerns

This technical deep dive had a paradoxical effect on me.

On the one hand, I became much more comfortable with Meteor’s “magic” once I understood how it worked. On the other hand, once I understood it, the gap between Meteor and existing web technologies appeared to shrink. After all, plenty of realtime apps are built on Node and MongoDB using REST APIs and a pub/sub package like Socket.IO. At that point, the only parts of Meteor you’re really missing are latency compensation, dependency tracking, and template management. Each is a solvable problem (especially if you don’t have to solve the general case).

In that light, some of my concerns about Meteor came to the fore:

And that brings us to some of what is odd about the design of the Meteor server. In its enthusiasm for a new way of writing apps, Meteor has left behind some of the basic architecture of the web. It fully embraces the single-page app concept, delivering the entire client bundle on each request (which contributes to the latency problem). There is not yet a way to break up an app into pages: the server actually delivers the same response to the client no matter what URL was requested; like a native iOS app, it’s up the client to interpret the URL and display the appropriate view. Meteor does not provide a REST API; it doesn’t even really use HTTP (except to deliver the initial client bundle), sending all requests and responses over a two-way connection in RPC fashion.

Indeed, a Meteor app is not truly a web app; it is an RPC-based client/server app that happens to be deployed over HTTP and run in a browser.

I think this belies a certain philosophical leaning on the part of Meteor’s designers—but it doesn’t need to become a holy war. The Meteor group is not opposed to a REST API or a page model; in fact, these things are on the roadmap for 1.0. They’re just not there yet.

Overall, Meteor is an impressive piece of engineering, but I’m not going to use it for my current project. Instead, I’m going with old, boring technologies that have been around for years and are already used in industrial-strength applications. Not that Meteor won’t be ready for prime time eventually—the team is smart and accomplished, and I wouldn’t bet against them. But in matters of production software, I’m just old enough and just crusty enough to stick with the devil I know.


Thanks to David Crawford and Andrew Miner for commenting on a draft of this post.

Discuss on Hacker News.

 
176
Kudos
 
176
Kudos

Now read this

Immigration is a moral issue

I’m glad to see many of my colleagues in the tech community promoting immigration reform (e.g., FWD.us). They point out that immigrants are often innovators and job-creators, and that immigration helps our economy. This is true and... Continue →