Republishing HTTP Data Over MQTT with Node-Red

mqtt-republish-iconToday most of the data we receive across the Internet is received using the HTTP protocol.

HTTP is the protocol that powers the Web,and is the most common protocol on the Internet.

HTTP was designed for viewing static web pages and uses  a request/response technique for  viewing a web page.



If the web page page changes the client must issue a new request in order to refresh the data.

For most websites this isn’t a problem as the page content is static, but there are a class of sites were the information being displayed is changing, and for which http may not be the best way to deliver it.

Flight Information – The Problem

Note: Not all flight information systems will work this way

Data for Live flight arrivals is displayed on a web page and is updated on a regular basis (every 60 seconds is common).

Due to the nature of http the refresh is usually accomplished automatically by JavaScript code running on the web page.

When the webpage is updated then usually the entire contents of the page is sent to the web browser regardless of the changes.http-data-request

This isn’t a problem with a single client but for 100s or thousands of connected clients this would cause significant server load.

The Solution Using MQTT

The solution using MQTT is to connect a single http client to the web server as before.

The web page is downloaded, processed, and the data for each flight is extracted.

The data is then sent over MQTT with each flight having its own topic, and only changed data is sent to the subscribing client.

An MQTT subscriber could subscribe to the flight that it wanted information for rather than seeing the data for all flights.

republish-http-data-mqtt

This will result in a considerable reduction in website and network traffic.

Node-Red Version

The original version was written in Python but I have also created a node-red flow that does the same thing Here is a video and here is the flow:

Related Tutorials and resources

Click to rate this post!
[Total: 0 Average: 0]

Leave a Reply

Your email address will not be published. Required fields are marked *