Implementing WebHooks and RESTHooks using TypeSript for Node.js and ExpressJS

; Date: Thu Jun 30 2022

Tags: Node.JS »»»» REST API »»»» TypeScript »»»» Express

The WebHooks architecture lets one webservice send event notifications to another. RESTHooks build on them by advertising available events, allowing software to manage event subscriptions without human intervention.

WebHooks are a common architectural pattern for communicating data between web services. WebHooks are used when one piece of web software, one web service in other words, wants to receive notifications or other data from another piece of web software, or another web service. This is about software talking with software, and as software engineers we need flexibility to use this pattern in a way which makes sense for our application. Hence, the definition for both is purposely broad.

Any webhook involves one web service being notified of an event happening in another web service. RESTHooks are a variant of webhooks with a few more capabilities, that we'll discuss in this article.

For a practical example, I'm assuming you are a software engineer, and that you have a GitHub repository. Go to one of your GitHub repositories, and click on the Settings icon. A little ways down you'll see a tab named Webhooks. Click on that and you'll see the following message:

GitHub defines a webhook as a notification sent to an external service that is triggered when certain events happen. The webhook invocation is a POST call on an HTTP URL, and there is a data payload.

What is the difference between WebHooks and RESTHooks?

Webhooks are simply a kind of Web API invoked between one web service, and another web service. One service will be sending events (the sender), and the other will be receiving the events (the receiver).

In the webhook scenario, the sending service must have a mechanism for manually configuring webhook subscriptions. With the GitHub example we just looked at, the manual configuration is performed in the Settings area of the GitHub repository. One could imagine scenarios where webhooks are configured in a YAML file or a database.

RESTHooks build on the webhook pattern with additional REST API methods to programatically manage subscriptions for a client service. Instead of manual event subscription configuration, the RESTHook API supports methods following this pattern:

• GET /path/to/endpoint - Retrieve the events that can be subscribed, and the current subscriptions
• POST /path/to/endpoint - Subscribe to an event described in JSON in the body
• GET /path/to/endpoint/:id - Retrieve data about a specific subscription
• PUT /path/to/endpoint/:id - Modify a subscription
• DELETE /path/to/endpoint/:id - Delete a subscription

To learn details, see: http://resthooks.org/

Note that RESTHooks is not a specification that one must follow precisely. Instead it is an architectural outline, or a collection of patterns, that one can follow. These five endpoints are therefore guidance rather than prescription.

Example implementation of Webhooks and RESTHooks

As a companion to this article is a GitHub repository containing example code. https://github.com/robogeek/webhook-resthook-demo-nodejs

This repository contains a pair of ExpressJS services (hence, running on Node.js), written in TypeScript, that implement both the WebHook and RESTHook models.

One server is designated as the client (receiver) and is configured to receive both kind of hooks, and to subscribe to several RESTHooks. The other server, designated as the service (sender), and implements the sending side of the hook implementation.

Implementing Webhooks in TypeScript, on Node.js and ExpressJS

A Webhook is an HTTP notification sent from one web service to another.

In the normal case, Webhook invocations result from an event in the sending service. For example, a common use for a GitHub Webhook is to be notified when commits are pushed to a repository. It's also typical that Webhooks are sent as a POST request.

To simulate an event delivered from time to time, the sample code uses the toad-scheduler. This package handles invoking tasks at regular intervals, with the interval defined using an ISO8601 duration string.

In the sender, a webhook subscription is registered using this function:

// app.ts
import { register as registerWebhooks } from './webhooks.js';
const uWebHook = new URL(process.env['CLIENTURL']);
uWebHook.pathname = '/hooks/webhook';
registerWebhooks(uWebHook.toString());

// webhooks.ts
export function register(url: string) {
    let count_id1 = 0;
    registerWebHook('ID1', 'PT1M', async (): Promise<void> => {
        const response = await axios({
            method: 'post',
            url: url,
            data: {
                id: 'ID1',
                hello: 'World',
                count: count_id1++
            }
        });
        console.log(`WebHook status ${response.status} count=${count_id1}`);
    });
}

The goal is to deliver an HTTP POST to a given URL containing data related to the event notification. The notification has an ID code, some text, and a counter.

In this case the receiver URL is derived from the CLIENTURL environment variable. Its value becomes the uWebHook variable, which is a URL that will contain the endpoint in the receiver server. The receiver must therefore implement a /hooks/webhook route handler.

The string PT1M is an ISO8601 duration specifier for a 1 minute interval. The scheduler package will therefore ensure the webhook is invoked every minute.

In the receiver service, the receiving side of this is implemented as so:

HookRouter.post('/webhook', (req, res) => {
    console.log(`WebHook `, req.body);
    res.sendStatus(200);
});

The router, HookRouter, is mounted in the Express app object like so:

app.use('/hooks', HookRouter);

Hence, the route handler responds to /hooks/webhook as specified when the subscription was setup.

In a real application, registerWebHook would be invoked based on configuration data. As we saw earlier, GitHub webhooks are configured in the Settings area. In such a case, the webhook configuration would be stored in a database, and when the appropriate conditions occur, the hook notifications are sent, with configuration read from the database.

The model we show here - regular invocations at a fixed interval - would be suitable in a data measurement environment. For example, an electricity meter connected to a solar array could be sampled every minute to gather data about solar energy production.

An important consideration is whether webhook notifications must be reliably sent and received. Should the notification be entered in a queue, and deleted from the queue only after successful delivery? What should your application do if event delivery fails? The answer to those and other questions depend on your application needs.

In any case, we see that sending a webhook uses a library like Axios to send the notification. On the receiving side, we simply need a simple REST endpoint handler function. Additional code is required to enable humans to configure which hooks, their configuration, and the destination URL.

Implementing RESTHooks in TypeScript, on Node.js and ExpressJS

Now that we've seen how webhooks work, let's explore RESTHooks. This is a more comprehensive model than simple webhooks. It has a complete mechanism for one service to discover information services offered by another service, a.k.a. service discovery.

As we said, RESTHooks is a guideline, not a prescription. The minimum elements to RESTHooks are:

• Data storage for subscriptions to event notifications
• Data storage for client applications allowed to subscribe to event notifications
• Listing event types
• Subscribing to an event, modifying a subscription, deleting a subscription
• Sending event notifications using hooks

Your application might have more needs than this.

Remember that the concept is documented at: http://resthooks.org/

There is a Node.js package at https://resthooks.io/ which has a similar name, but its website talks about different things, and it may provide completely different functionality.

Storing configuration data for applications allowed to subscribe to RESTHook services

The RESTHooks model involves a client application querying for events to listen for, and subscribing to anything of interest to the application. Normally there will be a token for authenticating which applications can subscribe to event notifications.

To emulate this, the example code has this type:

export type client = {
    clientID: string;
    name: string;
}

The clientID is the auth token, and it can be anything. For example, we can generate a UUID as an auth token. The token will be used in both the client code, and in the server code. We'll discuss later how we're implementing authentication.

The example code has these functions for managing an array of client data.

function addClient(id: string, name: string): void
function findClient(id: string): client
// Check that the ID string is for a known client application
const isClient = (id: string): boolean
// Look for `authcode` header in incoming request, look for
// client record for the provided token
const clientForReq = (req): client
// Express middleware function to check an incoming request
// for an authcode header that matches a configured client
const reqFromClient = (req, res, next)

Storing client tokens in a real application will require an internal API something like this, along with a database.

This is an example of registering a client authentication token which is a simple UUID.

addClient('8c4fb1ea-f594-11ec-8678-3bde07b709e0', 'TestClient');

The same token must be used in the client service. We can easily generate multiple client tokens to support multiple client services.

The reqFromClient function is Express middleware to verify that an incoming request contains a valid authentication token.

Storing configuration of RESTHook event notification subscriptions

The RESTHooks model also requires storing data about which events to which a client has subscribed.

The example code has this data type for remembering one subscription:

export type subscription = {

    /**
     * Identifier for this event subscription.
     */
    id: string;

    /**
     * The identifier of the client service that made
     * the subcription
     */
    clientID: string;

    /**
     * The event the client subscribed to
     */
    event: string;

    /**
     * The URL the client requested that event notifications be sent
     */
    hookurl: string;
}

The clientID field is the same as for the client object. The id field is an identifier for this subscription. The subscription is for the event named in event, and results in invoking the URL named in hookurl.

This is meant to simulate storing data about hook subscriptions somewhere, such as a database.

export const eventNames = [ 'OPEN', 'CLOSE', 'PEEK' ];

The example code has this array of event names. In a real application you'll use your own method to define event names.

The example code has these functions for managing the set of subscriptions.

function allSubscriptions(): Array<subscription>
function addSubscription(sub: subscription): void
function deleteClientSubscription(client: client, id: string): void
// Fetch subscriptions to named event for a client
function clientSubscription(client: client, nm: string): subscription
// Fetch subscription data for a specific event
// for a client
function clientSubscriptionByID(client: client, id: string): subscription

The purpose of these functions is to search for client subscriptions related to whatever event must be triggered. Once you've found a subscription object, the event notification can be sent.

Sending a RESTHook event notification

Earlier we showed how to send a simple WebHook using Axios. The sender service requires a little more data to deliver RESTHooks, but the implementation is similarly easy.

export async function sendRESTHook(sub: subscription, data) {
    return axios({
        method: 'post',
        url: sub.hookurl,
        data: data
    });
}

Like Webhooks, RESTHooks are delivered with an HTTP POST to a given URL, and with a given data payload. The subscription object should contain any configuration required for delivering the notification.

Client side function to send REST requests to RESTHook service

The receiver application requires a similar function with which to invoke functions in the RESTHook API.

async function invokeRESTCall(method: string, url: string, data) {

    const uRequest = new URL(SVCURL);
    uRequest.pathname = url;
    const subresp = await axios({
        method: method,
        url: uRequest.toString(),
        headers: {
            authcode: '8c4fb1ea-f594-11ec-8678-3bde07b709e0'
        },
        data: data
    });

    console.log(`${method} ${url} result: `, subresp.data);
    return subresp.data;
}

The RESTHook API uses the GET, PUT, and DELETE verbs, and therefore the the first parameter is the HTTP method to use. The url parameter is the path within the server, while the SVCURL variable is the URL for the server. The data object is the payload to send.

Earlier we described the clientID as an authentication key. The client service is to include this key with RESTHook API requests, to authenticate the client with the service. What we chose is overly simplistic, of course, because we simply assign the authentication key to the authcode header. Notice that the authcode value is the same UUID as used earlier. In a production application it's best to use one of the correct authentication methods.

In the service, the reqFromClient middleware function verifies that the authentication code is correct.

Listing available RESTHook events

Now that we have defined functions for communication in both directions between receiver and sender services, we can start looking at implementing the RESTHook methods. We'll start with a method to retrieve the list of available events.

On the sender side, this is handled by the following router function:

// List the things which can be subscribed
HookRouter.get('/subscribe', reqFromClient, (req, res) => {
    const client = clientForReq(req);
    const ret = {
        subscribable: []
    };
    for (const evnm of eventNames) {
        const subscription = clientSubscription(client, evnm);
        ret.subscribable.push({
            name: evnm,
            subscribed: typeof subscription !== 'undefined',
            id: subscription ? subscription.id : undefined
        });
    }
    res.status(200).json(ret);
});

The HookRouter object is registered with Express this way:

app.use('/hooks', HookRouter);

This handler therefore takes care of a GET operation on /hooks/subscribe.

It uses the reqFromClient middleware for authentication, as just discussed. The clientForReq function looks up the client record for the authentication key in the authcode header.

The purpose is to list the available event names. It was thought to be sensible to also include data indicating whether the client service is subscribed to the event. Therefore subscribable is an array of items where name is the event name, subscribed is a boolean saying whether the client is subscribed, and id is the ID string for this subscription (if any).

In the receiver service, this function can be used to help send this request:

async function checkSubscription() {
    return await invokeRESTCall('get', `/hooks/subscribe`, undefined);
}

Because the example receiver service has a scheduling system to regularly execute tasks, we might want to check the subscriptions every minute or so.

const reqSubsTask = new AsyncTask(`Request subscriptions`, async () => {
    const subscriptions = await checkSubscription();
    console.log(subscriptions.subscribable);
});

const reqSubsJob = new SimpleIntervalJob(
            Duration.parse('PT1M'),
            reqSubsTask,
            'req_subs');

scheduler.addSimpleIntervalJob(reqSubsJob);

Again, the ISO8601 duration PT1M means this executes once a minute.

Subscribing to a RESTHook event

Let's now talk about subscribing to an event. The difference between WebHooks and RESTHooks is how we subscribe to events. With WebHooks, subscriptions are established out of band through a manual process. With RESTHooks, the application subscribes on its own.

In the example code, the sender service handles subscription events like so:

// Subscribe to an event
//
// {
//    name: 'EVENT-NAME', hookurl: 'http://host/path/to/endpoint'
// }
HookRouter.post('/subscribe', reqFromClient, (req, res) => {
    const client = clientForReq(req);
    const subscription = clientSubscription(client, req.body.name);
    if (subscription) {
        res.status(404).json({
            message: `Event ${req.body.name} already subscribed by ${client.clientID}`
        });
    } else {
        const subID = uuidv4();
        console.log(`Subscribing ${client.clientID} to event ${req.body.name} with ${subID}`);
        addSubscription(<subscription>{
            id: subID,
            clientID: client.clientID,
            event: req.body.name,
            hookurl: req.body.hookurl
        });
        res.status(200).json({
            message: 'OK'
        });
    }
});

The comment describes the subscription object. It simply contains the event name, and the URL to send event notifications.

Inside the event handler, we check if the client has already subscribed to the event. If so, an error is returned. The error code 404 is probably incorrect since this is not about a non-existent resource.

In the list of HTTP result codes, some likely choices are:

• 400 Bad Request
• 403 Forbidden
• 409 Conflict

In any case, a subscription object is created, and addSubscription is called to store it away.

The receiver side of the example code subscribes to the OPEN and CLOSE events in this way:

export async function register() {
    await subscribeRESTHook('OPEN');
    await subscribeRESTHook('CLOSE');
}

async function subscribeRESTHook(evname: string) {
    const uClientHook = new URL(CLIENTURL);
    uClientHook.pathname = '/hooks/resthook';

    return await invokeRESTCall('post',
                `/hooks/subscribe`, {
                    name: evname,
                    hookurl: uClientHook.toString()
                });
}

The uClientHook variable is used to compute the URL vor the receiver service resthook endpoint. The value in CLIENTURL is the base URL for the service, and /hooks/resthook is the endpoint.

Retrieving data about a RESTHook event subscription

The request, GET /path/to/endpoint/:id, is specified to retrieve data about a subscription. The subscription ID is passed as a path parameter, and data is returned.

In the sender service, this request handler is triggered:

// Get data on a specific subscription
HookRouter.get('/subscribe/:id', reqFromClient, (req, res) => {
    const client = clientForReq(req);
    const subscription = clientSubscriptionByID(client, req.params.id);
    res.status(202).json(subscription);
});

The function clientSubscriptionByID simply looks up the subscription object for the specified receiver service and subscription ID, and returns that object.

In the receiver code, this function enables making this call:

async function checkSubscriptionDescription(id: string) {
    return await invokeRESTCall('get',
            `/hooks/subscribe/${id}`, undefined);
}

Simply call this whenever you like.

Changing a RESTHook event subscription

The request PUT /path/to/endpoint/:id is defined to modify an event subscription. The subscription ID is passed in, and in the request body is data for the modification.

Think about the definition of the subscription object. In a real application there might be quite a few more fields associated with an event subscription. The purpose of this object is to describe the subscription, such as the conditions under which the notification is sent, or the type of data included in the notification.

With our subscription object, the only field which can effectively be changed is the hookurl.

In the sending service, this function handles the request:

// Modify a subscription
HookRouter.put('/subscribe/:id', reqFromClient, (req, res) => {
    const client = clientForReq(req);
    const subscription = clientSubscriptionByID(client, req.params.id);
    if (req.body.hookurl) subscription.hookurl = req.body.hookurl;
    // if (req.body.newname) subscription.event = req.body.newname;
    res.status(202).json({
        message: 'OK'
    });
});

The object sent to describe the modifications can theoretically be used to modify other fields. For example we could change the event name from PEEK to PEEKABOO. But, the only field we consult to see if there's a modification is hookurl.

In the receiver service code, this function enables sending a modification request:

async function changeRESTSubscription(id: string, url: string) {
    const uClientHook = new URL(CLIENTURL);
    uClientHook.pathname = url;

    return await invokeRESTCall('put', `/hooks/subscribe/${id}`, {
        hookurl: uClientHook.toString()
    });
}

As before, uClientHook is used for computing the URL of the new endpoint.

Deleting a RESTHook event subscription

The last subscription change is to unsubscribe from an event notification.

The sender service takes care of it with this handler function:

// Delete a subscription
HookRouter.delete('/subscribe/:id', reqFromClient, (req, res) => {
    const client = clientForReq(req);
    deleteClientSubscription(client, req.params.id);
    res.status(202).json({
        message: 'OK'
    });
});

And in the receiver service we have this function:

async function unsubscribeRESTHook(id: string) {
    return await invokeRESTCall('delete',
                `/hooks/subscribe/${id}`, undefined);
}

This simply sends a DELETE request to the sender service.

Summary

The RESTHooks model is an interesting architectural pattern. While in most cases Webhook subscriptions can be handled manually by editing details via the settings area of a website, there are many instances where dynamically subscribing to events without human intervention makes sense.

Earlier we described the scenario of an electricity meter sampling electricity consumption. The generalized scenario is to have a variety of meters, some measuring light levels, or temperature, or other data of interest. A data collection service could sample all connected meters, notice which measurements are available on each, and dynamically subscribe to them all.