Please, describe the error

Found a bug? Helps us fix it by filling in this form

Web Service Development on Node.js and Express.js Part 1: The Beginning

Maxim Limonov
Web-developer

Node.js is an open-source, cross-platform JavaScript execution environment. It’s based on Google’s V8 JavaScript Engine and is designed to help create web-based, server-side applications. Node was built in 2009 by Ryan Dahl. Today, Node.js is backed by a large community of developers. It’s ideal for building heavy-duty, scalable web-services. In fact, it’s used by some of the internet’s largest companies, including PayPal and LinkedIn.

js-1000-488

Introduction

In a series of articles, we’re going to examine the basics of creating web services on Node.js and Express.js. We’ll demonstrate how they interact with databases like MySQL, MongoDB, and Redis. We’ll show you how to organize the authorization of users on the web-service using logins and passwords, and how to establish social network verification using the Passport module. We’ll also describe how to restrict access to a web service using roles (through the Control List) before discussing how to make the service fail-proof and scalable.

Our web service will be built using REST API that allows for users to pre-order goods. We’re assuming two roles here, including a potential buyer who accesses the system with the intention of purchasing the product, and the seller, who places the product online and establishes the estimated date of availability.

When you logon and activate the first step, the user is asked to specify the phone number they used to receive the SMS code. Then, an SMS code is sent to that number, and the user enters that code in the second step. The phone number and SMS code will both serve as authorization into the system. System details and additional functionality will be described in the implementation section.

n this part, Part 1, we will look at the architecture of Node.js and its ecosystem, completely relying on the package manager npm. Establish a Node via nvm utility then create an mvc application framework using the module express-generator. Create the configuration file and examine the intricacies of the work function require and the difference between module.exports and exports. Then, connect to the database application and finally, run what we’ve got.

Node.js Architecture

Node.js consists of the following components:

V8: JavaScript is the primary open source engine used by Google. It’s written in C++ and is used to execute JavaScript code in the Chrome browser. We’ve also recently received the ability to use Node on the engine ChakraCore made by Microsoft (this is the engine that executes JavaScript code in Microsoft’s Edge browser).

libuv: Written specifically for the Node multiplatform library in C. Libuv is able to handle the non-blocking asynchronous I/O operations. Libuv is based on platform-dependent libraries, including iocp for Windows, epoll for Linux, and kqueue for Mac/BSD. For operations that cannot be done asynchronously at the operating system level, the library uses an internal thread pool (which works with the file system and DNS). The cycle of events (Event Loop) is implemented in this component.

Other C / C++ Based Services: These services include c-ares, crypto (OpenSSL), http-parser and zlib. These interact with the server at a low level and provide important functionality related to the network, encryption, compression, etc.

Modules written in JavaScript – implementing the Node.js API

Banding (Bindings) — serve as a binder layer between the code written in C / C++ and code written in JavaScript.

Instead of the traditional multi-threaded server module – where each connection is allocated a single thread – Node handles all connections in a single thread. This stream is called the Event Loop (which interprets and executes custom JavaScript code from the application). When there’s a request input/output, the Event Loop assigns the task or the operating system when it arrives at the network or the controller thread pool, which then works with the file system or the DNS (the default is 4 flow). Then, it reports regularly on the state of operation. On its completion, it carries out a pre-assigned effect (like a triggered callback-function or an event listening is triggered – this is theevent-oriented Node). The resulting events are arranged in a special queue (event queue) which determines the order of event processing in the Event Loop. Since the events of the cycle do not wait for the result of the I/O operation, the next request is not blocked for the duration of the I/O operation, and the operation is performed asynchronously with respect to the events of the cycle. When used directly «out of the box», Node does not support asynchronous execution of operations that require CPU resources, and such problems block the event loop. To work around this problem, you’ll need third-party modules (webworker-threads, for example).

Having trouble understanding how the Event Loop works? In the article linked here the author gives an interesting analogy between the application of Node.js and a coffee shop. The author demonstrates a simplified version of the Event Loop, which we’ll talk about below (with some added detail).

magine that visitors (events) to the coffee shop are served by only one highly productive and well-trained waiter (the Event Loop). When a large number of visitors come to drink a cup of coffee at the same time, they line up (event queue) and wait until the water serves them. As soon as the waiter takes the order, he immediately gives it to the manager (libuv), which then assigns a barista to complete the order (flow in the thread pool, or platform-dependent mechanism, depending on the task). The Barista uses different ingredients and the coffee machine (low-level C / C++ components) to prepare different kinds of drinks, depending on the preferences of the visitors. Typically, the coffee shop has 4 baristas (thread pool) working at the same time that prepare only certain types of coffee (file I/O, dns, and user tasks that are assigned through uv_queue_work()). If four baristas are missing, then the number can be increased – but this must be done in advance, either at the beginning of the day or before the first purchase of a visitor (the number of threads in the pool can be installed by starting the application variable, or software can be set to the first reference to the pool and the maximum number of threads –128). When the order is sent to the manager, the waiter does not wait for the finished drink. He goes to serve the next visitor in line. As soon as the drink is prepared, it is sent to the end of the queue of visitors. Waiters call out to visitors when their drink in the queue reaches the counter. Sometimes, it happens that a visitor as the counter asks to serve his friend out of turn – like right behind him, or at least as soon as possible (function setImmediate()), and sometimes, sends a friend into the queue (function process.nextTick()).

Installing Node.js and npm package manager

Node.js is a cross-platform product. It can be installed on most modern operating systems. We will use Mac OS X for this example. When you develop applications on Node, sooner or later, there are situations where you’ll need to use different versions. This can occur when a developer is involved in several proejcts with different requirements, or when you create an application that is compatible with several versions of Node. n this case, a utility called nvm (Node Version Manager)makes it easy to install a different version of Node and quickly switch between these versions. Nvm runs on Linux and OS X. Windows users can access the same functionality using nvm-windows.

Before installing nvm, you’ll want to remove Node.js and npm. You can do that following the tutorials here and here. Also, you’ll want to set the system to C++ compiler. For OS X, you can do that using the command line utility Xcode:

xcode-select --install

Next, make sure the file ~/.bash_profile exists. Otherwise, create touch touch ~/.bash_profile. Now, you’re ready to install nvm (replace the version below last):

curl -o- https://raw.githubusercontent.com/creationix/nvm/v0.31.4/install.sh | bash

In this article, out of all the nvm utility features, we will only use the installation of the latest version of Node.

How Does Node.js Number Work?

We should explain the principles behind the numbering of Node versions. Each new version (v.5, v.6) is published every six months. The even version (released in April) s focused on security and stability, and has a long period of support (Long Term Support – LTS – plan, which lasts 18 months). The LTS version does not receive new functions or features through its updates – it only receives bug fixes that affect stability, security, and non-critical performance. Even-numbered versions are suitable for companies with complex code organization, for which frequent updates are a burden. The odd versions(released in October), on the other hand, receive updates frequently. These updates actively implement new features, improve existing APIs, and enhance performance. these versions are not supported for more than eight months and have more of an experimental nature.

At the time of writing, the latest version is v6.3.1. The sixth version comes with significant performance improvements, including in reliability, ease of operations, and safety. Although v.6 has not yet reached the status of LTS, we will work with it for this tutorial. To install the latest version, follow the terminal:

nvm install node

And check the setting

node -v
v.6.3.1

If all is well-established, this command should give you the correct version of Node.js.

Node.js and the Modular Approach

Building Node.js applications requires a modular approach. Due to the popularity of Node, a large community of supporters created a huge number of modules that significantly expand the functionality. To install modules, you’ll need to designate the package manager npm, which comes with Node.js. Verify the version of the manager:

npm -v
3.10.3

n addition to installing the modules, npm monitors their versions – two modules can be dependent on a third, but use different versions. Previously, the npm manager put dependent modules in the module folder that used it. This was a nested structure that led to different folders with identical modules and identical versions. Since the release of npm v.3, linear modules are now stored in a single folder – the only time modules are separated is when there is a conflict between versions.

Our web service will be based on the Express.js module – which is the most popular framework for Node.js. Node does not impose strict limits on the organization of code, which is nice because it gives developers the freedom of choice. We will use the mvc pattern. For the generation application framework mvc, use the package express-generator. This terminal will perform the command:

npm install -g express-generator

As a result, the package manager npm installs the module express-generator. The flag -g s a global setting (i.e. the module is run in the terminal). Note that if the installation for Node nvm was not used, then the global installation of the modules would have to resort to the use of sudo, which is not secure. Next, create the framework mvc-team

express ~/Documents/site/app

The directory ~/Documents/site/app will be the root of our project (all relative paths in the article willbe carried out from the root directory). We turn into the created directory and set as prescribed in the file ./package.json (modules are installed in ./node_modules)

cd ~/Documents/site/app && npm install

And run the application

npm start

This npm will execute the command register in ./package.json

{
    ...
    "scripts": {
        "start": "node ./bin/www"
    },
    ...
}

Now go to the address http://localhost:3000. If you see a welcome sign saying, «Welcome to Express», then congratulations – you are ready for the next step!

Database

Our service will use the MySQL database system to store users, products, pre-orders, and access rights systems. We’ll use Redis to store session data. And we’ll use MongoDB to store sent SMS codes.

MySQL

There’s no need to introduce MySQL. Download the DMG image of MySQL server and run the installer. When the installation completes, you’ll see a dialog box appear telling you about the temporary root password.

image1

Change the temporary password

/usr/local/mysql/bin/mysqladmin -u root -p password
Enter password: 
New password: 
Confirm new password: 
Warning: Since password will be sent to server in plain text, use ssl connection to ensure password safety.

Start and stop the MySQL server through the panel in System Preferences.

image2

We’ll work with the database scheme using the MySQL Workbench.

Redis and MongoDB

Redis is a network-type data storage system that uses a «key-value» type of non-relational database management system. It stores the database in RAM. MongoDB, on the other hand, is a non-relational database used to store JSON objects.

Redis and MongoDB are easily installed through Homebrew, which is a package manager for Mac OS X. Update the package manager and use the install Redis database command:

brew update && brew install redis

Run it as a service

brew services start redis

Install MongoDB command set

brew install mongodb

Create a directory

mkdir ~/Documents/site/data

Set MongoDB to run along this path

mongod --dbpath ~/Documents/site/data

To check the dial in another terminal window

mongo

You should see the following:

image3

f the screenshot server issues a warning, then you can safely ignore it (if your project is in the development stage). If your project is not in the development stage, I recommend reading here to solve the problem.

Application Configuration File

Create a project in the root of the two sub-directories and an empty file:

mkdir -p config/development && touch ./config/development/index.js

This file will be responsible for the application configuration settings. Place the following code in it (and don’t forget to change the specified value in the comments on your own).

// file ./config/development/index.js
var config = {
    db : {
        mysql : {
            host     : 'localhost',
            user     : 'root',
            database : 'appdb',
            password : 'yourPasswordHere'  
        },
        mongo : 'mongodb://localhost/ourProject'
    },
    redis : {
        port : 6379,
        host : '127.0.0.1'
    },
    port : 3000
};

module.exports =  config;

Connect this to the file ./bin/www as follows

// file ./bin/www
    //...
    var config = require('../config/' + (process.env.NODE_ENV || 'development'));
    //...
    //var port = normalizePort(process.env.PORT || '3000');
    var port = normalizePort(process.env.PORT || config.port);
    //...

This will allow us to connect different configuration files depending on the application startup options. For example, if we create a configuration file ./config/production/index.js, then you run the application to apply it by entering the script as follows:

NODE_ENV=production npm start

A Few Words About Require, Module.exports, and Exports

Node.js uses a modular approach to building applications. Each module connects to a separate file (*.js, *.json, *.node or a file with JavaScript code without the extension). The connection of modules corresponds to the function require. For example, if the built-in modules or the modules stored in ./node-modules (or ../node-modules above), then a function of the parameter require indicates the name of the module only. Otherwise, the function is passed to the path (i.e. a line beginning with './' or '/', or '../'). The expansion module can be omitted and just written. The whole algorithm for obtaining the path to the module in the line is described here.

A variable declared in a module in the usual way will not be available through require – it should pass through module.exports, or exports (referencing module.exports, which is used for a shorter recording). The example below illustrates the situation

// file module.js
var a = [1,2];
var b = [3,4];

exports.a = a;

// file uses_module.js
var module = require('./module');

console.log(module.a) // [1,2]
console.log(module.b) // undefined

Using this method, it’s possible to pass any variables, including functions and constructors. A variable can be directly assigned to the project module.exports (but not exports, because it’s only a reference), just as we did with the configuration file above.

When you first connect the module function, require caches it and places it in an object require.cache. When subsequent connections are made using the same module, the object is loaded from the cache. This model implements the Singleton pattern.

Connecting the Database to Node.js

Installing Drivers

For MySQL, we will not use ORM. Instead, the entire burden will fall on the stored procedure that will be called from the repositories of each corresponding object. The standard tool used to make MySQL work with Node is the npm module mysql (there is a faster alternative driver called mysql2, but it is only in the early stages of release). When writing code, we will stick to the style using PROMIS. Since mysql module does not support such a style, use the wrap-module mysql-promise. Let’s look at the project root directory and set it:

cd ~/Documents/site/app && npm install mysql-promise --save

The flag --save tells the package manager to save the file dependency package.json. The driver for Redis works in a similar way:

npm install promise-redis --save

Finally, set the Mongoose – ODM (Object-Document Mapper) for MongoDB

npm install mongoose --save

Mongoose has its own built-in mpromise library that implements the Promise. The library is currently considered to be outdated – so it’s recommended that you replace it with another one (we will replace it with a standard a ES6).

Initialize Database

Create a folder and a file in it

mkdir libs && touch ./libs/dbs.js

With the following contents

// file ./libs/dbs.js
var mysqlPromise = require('mysql-promise')(),
    mongoose = require('mongoose'),
    Redis = require('promise-redis')(),
    config = require('../config/' + (process.env.NODE_ENV || 'development'));

mysqlPromise.configure(config.db.mysql);
var redis = Redis.createClient(config.redis.port, config.redis.host);

mongoose.Promise = Promise;                 // [1]

function checkMySQLConnection(){            // [2]
    return mysqlPromise.query('SELECT 1');
}

function checkRedisReadyState() {           // [3]
    return new Promise((resolve,reject) => {
        redis.once("ready", () => {redis.removeAllListeners('error'); resolve()});
        redis.once('error', e => reject(e));
    })
}

function init() {                           // [4]
    return Promise.all([
        checkMySQLConnection(),
        new Promise((resolve,reject) => {mongoose.connect(config.db.mongo, err =>
            err ? reject(err):resolve())}),
        checkRedisReadyState()
    ]);
}

module.exports = {                          // [5]
    mysql: mysqlPromise,
    redis: redis,
    init:  init
};

Let’s talk about what’s going on here. Initially, the database drivers are connected with a configuration file. Then, this configures the connection to the MySQL database and creates the Redis client base. Here are the points:

  1. Replace the built-in PROMIS library with the standard ES6 library.
  2. The function checkMySQLConnection() checks the connection to the MySQL by executing a simple request. Configuring the connection to the connection pool creates the MySQL database, and the query command takes a connection pool and makes a request, then releases the connection.
  3. The function checkRedisReadyState() hecks the readiness of the server Redis.
  4. The function init() via ES6 method Promise.all parallel runs the MySQL readiness, MongoDB, and Redis. If any of the functions return an error, the performance of the other stops and Promise.all eturns an error, which we will catch in the ./bin/www using the method .catch(). f the error returns, control is passed to the method .then().
  5. And finally, exporting variables.
Change the file ./bin/www

// file ./bin/www

// ...
var config = require('../config/' + (process.env.NODE_ENV || 'development')),
    dbs = require('../libs/dbs');

// ...

// Replace string
// server.listen(port);

dbs.init().then(() => {
  console.log('Database connections are installed successfully');
  server.listen(port);
}).catch(err => {
  console.log(err);
  process.exit(1);
});

// ...

This means that our web service will not start if there are problems with the database connection. Now, create a database in MySQL with the name appdb and run your application. If everything is working properly, then you’ll see a message about a successful connection in the console.

That’s the end of the first part. In the next article, we’ll talk about deal registration and user roles.

Read and comment