Building A Server-Side Application With Async Functions and Koa 2 — Smashing Magazine

webberoDesign0 Comments


build The build of the app, containing the compiled code from srcThe src directory contains the following:

  • api.js The module that defines the app’s API
  • app.js The module that instantiates and configures the Koa app’s instance
  • config.js The module that provides the configuration of the app to other modulesIf additional files or modules are needed as the app grows, we would put them in a subfolder of the src folder — for example, src/models for application models, src/routes for more modular API definitions, src/gateways for modules that interact with external services, and so on.

NPM Scripts As A Task Runner

After using Gulp and Grunt as task runners, I came to the conclusion that npm scripts are better than separate tools when working on server-side projects. One of the advantages of npm scripts is that they allow you to invoke locally installed modules as if there were globally installed. I use the following scripts, which need to be defined in package.json:

"scripts": {
  "start": "node index.js",
  "watch": "nodemon --exec npm run start",
  "build": "babel src -d build",
  "test": "npm run build; mocha --require 'babel-polyfill' --compilers js:babel-register"
}

The start script simply runs index.js. The watch script runs the start script using the nodemon tool, which automatically restarts the server whenever you change something in the app. Note that nodemon is installed as a local development dependency and does not have to be installed globally. The build script compiles the files in the src folder using Babel and outputs the results to the build folder. The test script runs the build script first and then runs tests using mocha. Mocha requires two modules: babel-polyfill, to provide runtime dependencies of the compiled code, and babel-register, to compile the test files before executing them. Additionally, presets for Babel have to be added to package.json, so that you don’t have to provide them in the command line:

{
  "babel": {
    "presets": [
      "es2015",
      "stage-3"
    ]
  }
}

This preset enable all ECMAScript 2015 features, as well as features that are currently in stage 3. With this installed and configured, we can start developing the app.

The Application’s Code

First, let’s look at index.js:

const port = process.env.PORT || 4000;
const env = process.env.NODE_ENV || 'development';
const src = env === 'production' ? './build/app' : './src/app';

require('babel-polyfill');
if (env === 'development') {
  // for development use babel/register for faster runtime compilation
  require('babel-register');
}

const app = require(src).default;
app.listen(port);

The module reads two environmental variables: PORT and NODE_ENV. NODE_ENV should be either development or production. In development mode, babel-register will be used to compile modules at runtime. babel-register caches the results of the compilation and, thus, reduces the server start time, so that you can iterate faster during development. Because this module is not recommended for production use, the precompiled version from the build directory will be used in production mode. index.js is the only file of the project that will not be compiled by Babel and that must use native module syntax (i.e. CommonJS). Therefore, the app’s instance is located in the default property of the imported app module, which is an ECMAScript 6 module that exports the app’s instance as a default export:

export default app;

This is important to keep in mind if you mix ECMAScript 6 and CommonJS modules. Now to the app.js file itself. This and the other files discussed below are always compiled by Babel in both development and production environments, and the new syntax (including async functions) may be used freely in them:

import Koa from 'koa';
import api from './api';
import config from './config';
import bodyParser from 'koa-bodyparser';
import cors from 'kcors';

const app = new Koa()
  .use(cors())
  .use(async (ctx, next) => {
    ctx.state.collections = config.collections;
    ctx.state.authorizationHeader = `Key ${config.key}`;
    await next();
  })
  .use(bodyParser())
  .use(api.routes())
  .use(api.allowedMethods());

export default app;

Here, we are using ECMAScript 2015’s import syntax to import the required modules. Then, we create a new instance of the Koa application and attach several middleware functions to it using the use method. The last thing we do is export the app so that it can be used by index.js. The second middleware function in the chain is an async function and an arrow function at the same time:

app.use(async (ctx, next) => {
  // Set up the request context
  ctx..state.collections = config.collections;
  ctx..state.authorizationHeader = `Key ${config.key}`;
  await next();
  // The execution will reach here only when
  // the next function returns and finishes all async tasks
  // console.log('Request is done');
})

In Koa 2, the next parameter is an async function that triggers the next middleware function in the list. Just like in Koa 1, you can control whether the current middleware function should do its job before or after the others by putting the call to next either at the beginning of the current function or at the end. Also, you can catch all errors in the downstream middleware functions by wrapping the await next(); statement in a try/catch statement wherever doing so makes sense.

Defining the API

The api.js file is where the core logic of our app resides. Because Koa does not provide routing out of the box, the app has to use the koa-router module:

import KoaRouter from 'koa-router';

const api = KoaRouter();

Here, koa-router provides functions to define middleware functions for specific HTTP methods and paths — for example, the route that stores events in the database:

// Declare a post method and what it does
// :collection is a parameter
api.post('/:collection',
  // First, validate auth key
  validateKey,
  // Then, validate that the provided collection exists
  validateCollection,
  // Handle adding the new item to the collection
  async (ctx, next) => {
    // Use ES6 destructuring to extract the collection param
    const { collection } = ctx.params;
    // Wait until the persistence layer saves the item
    const count = await ctx
      .state
      .collections[collection]
      .add(ctx.request.body);

    // Reply with 201 Created when the item is saved
    ctx.status = 201;
  });

Each method can have several handlers, which run sequentially and have exactly the same signature as the middleware functions defined in the top level of app.js. For example, validateKey and validateCollection are simply async functions that validate the incoming request and return 404 or 401 if the provided event collection does not exist or if the API key is not valid:

const validateCollection = async (ctx, next) => {
  const { collection } = ctx.params;
  if (!(collection in ctx.state.collections)) {
    return ctx.throw(404);
  }
  await next();
}

const validateKey = async (ctx, next) => {
  const { authorization } = ctx.request.headers;
  if (authorization !== ctx.state.authorizationHeader) {
    return ctx.throw(401);
  }
  await next();
}

Note that arrow middleware functions cannot refer to the context of the current request using this (i.e. this is always undefined in the examples thus far). Therefore, request and response objects as well as Koa helpers are available in the context object (ctx). Koa 1 had no separate context object, and this referred to the current request context. After defining other API methods, we finally export the API to be connected to the Koa application in app.js:

export default api;

Persistence Layer

In api.js, we accessed the collections array in the context ctx, which we initialized in app.js. These collection objects are responsible for storing and retrieving data stored in Redis. The Collection class is as follows:

// Use promise-based Redis client
const redis = require('promise-redis')();
const db = redis.createClient();

class Collection {

  // Full source:
  // https://github.com/OrKoN/koa2-example-app/blob/master/src/collection.js

  async count() {
    // We can `await` for promises
    // The await syntax allows us to work with
    // async calls as if they were synchronous
    var count = await db
      .zcount(this.name, '-inf', '+inf');
    return Number(count);
  }

  async add(event) {
    await db
      .zadd(this.name, 1, JSON.stringify(event));

    await this._incrGroups(event);
  }

  async _incrGroups(event) {
    // ES6 for:of syntax allows for easier iteration
    // groupBy is an array that holds possible attributes of the event
    for (let attr of this.groupBy) {
      // We can use await inside loops,
      // thus calling async operations sequentially in the loop
      await db.hincrby(`${this.name}_by_${attr}`, event[attr], 1);
    }
  }
}

export default Collection;

The async/await syntax is really helpful here because it allows us to coordinate several async operations easily — for example, in a loop. But there is one important thing to keep in mind. Let’s look at the _incrGroups method:

async _incrGroups(event) {
  // ES6 for:of syntax allows iterating easier
  for (let attr of this.groupBy) {
    // We can use await inside loops,
    // thus calling async operations sequentially in the loop
    await db.hincrby(`${this.name}_by_${attr}`, event[attr], 1);
  }
}

Here, the keys are incremented sequentially, meaning that the next key will be incremented once the previous incrementation has succeeded. But this kind of job can be done in parallel! With async/await, the task might not be easy to accomplish, but promises can help:

// Start all increments in parallel
// because there is no await inside the map callback here
const promises = this.groupBy.map(attr =>
  db.hincrby(`${this.name}_by_${attr}`, event[attr], 1));
// Wait for all of them to finish
await Promise.all(promises);

The interchangeability of promises and async functions is very helpful.

Testing

The app’s tests are located in the test folder. Let’s look at apiSpec.js, which represents the test specification for the app’s API:

import { expect } from 'chai';
import request from 'supertest';
import app from '../build/app';
import config from '../build/config';

We import expect from chai and supertest. We use the precompiled version of the app and the config to make sure that we are testing exactly the same code that will be running in production. Next, we write the tests for the API, leveraging the async/await syntax to achieve sequential execution of the test steps:

describe('API', () => {
  const inst = app.listen(4000);

  describe('POST /:collection', () => {
    it('should add an event', async () => {
      const page = 'http://google.com';
      const encoded = encodeURIComponent(page);
      const res = await request(inst)
        .post(<code>/pageviews</code>)
        .set({
          Authorization: 'Key ' + config.key
        })
        .send({
          time: 'now',
          referrer: 'a',
          agent: 'b',
          source: 'c',
          medium: 'd',
          campaign: 'e',
          page: page
        })
        .expect(201);
      // here res is available
      // you can use res.headers, res.body etc
      // no callback for <code>expect</code> or <code>end</code> functions is required.
      expect(res.body).to.be.empty;
    });
  });
});

Note that functions passed to it functions are marked with async. This means that await can be used to run asynchronous tasks, including the supertest requests, which return then-able objects and which, therefore, are compatible with what await expects.

Deployment With PM2

Once the app is ready and tests have passed, let’s prepare everything to run the app in production. For this, let’s declare ecosystem.json, which will hold the production configuration of the app:

{
  "apps" : [
      {
        "name"        : "koa2-example-app",
        // The entry point to the compiled version of the app
        "script"      : "index.js",
        // In production, we don't want to watch for changes in files
        "watch"       : false,
        // And we want to merge logs from all instances
        "merge_logs"  : true,
        // We want to timestamp log messages
        "log_date_format": "YYYY-MM-DD HH:mm Z",
        "env": {
          // And include environment variables for the app
          "NODE_ENV": "production",
          "PORT": 4000
        },
        // Start two processes of the app, and balance the load between them
        "instances": 2,
        // Start app as a cluster
        "exec_mode"  : "cluster_mode",
        // Watch failures and auto-restart processes
        "autorestart": true
      }
  ]
}

If your app requires additional servers (for example, a cron job), you can add them to ecosystem.json, and they will be started together with the main server. To start the app on the production server, you can run this:

pm2 start ecosystem.json

And to persist the configuration, you would run this:

pm2 save

PM2 provides some monitoring features (try pm2 list, pm2 info, pm2 monit). For example, PM2 shows how much memory your application is using. This basic Koa application consumes 44 MB per Node.js process.

Conclusion

Babel allows us to build apps using ECMAScript syntax that is not natively available but that includes async/await, which makes writing asynchronous code more enjoyable. The code that uses the async/await syntax is easier to read and maintain. Koa 2 and Babel allow you to start using async functions right now. Nevertheless, Babel brings additional overhead and requires additional configuration and an extra build step. Therefore, waiting until async/await is natively available in Node.js is recommended. Koa 2 should be officially released once this happens. Then, Koa 2 will be a good alternative to Express because it is more modular and simpler and allows for configuration the way you want it. The deployment section of this tutorial might be too simple and unscalable. It leaves open the question of how and when to build and actually deploy the code — you can do this manually (rsync, scp) or set up a continuous integration server for this. Also, the inner architecture of the app is too simple yet is suitable for a demo. Larger and more ambitious apps might require other entities, such as gateways, mappers, repositories and so on, but all of them can leverage async functions. I hope you’ve enjoyed this tutorial. Thanks for reading!



Source link

Leave a Reply

Your email address will not be published. Required fields are marked *