Mosquitto

I just discovered that Shane Hanna has written a Go language binding for libmosquitto available at https://bitbucket.org/shanehanna/mosquitto/. Good work Shane! Note that the readme file states:

Doesn’t expose all of libmosquitto, just what I’ve needed so far.

so you shouldn’t necessarily expect everything to work.

As Debian has been in feature freeze since before Mosquitto 1.0 was released, it will be a long time until there is an updated version of Mosquitto in Debian. It is, however, fairly straightforward to do the packaging yourself. Here’s how to do that from the command line.

Download and unpack the mosquitto source tarball:

wget http://mosquitto.org/files/source/mosquitto-1.1.2.tar.gz
tar -zxf mosquitto-1.1.2.tar.gz

Rename the tarball to match Debian requirements:

mv mosquitto-1.1.2.tar.gz mosquitto_1.1.2.orig.tar.gz

The current mosquitto packaging files are available at http://mentors.debian.net/debian/pool/main/m/mosquitto/ – you want the .debian.tar.gz:

wget http://mentors.debian.net/debian/pool/main/m/mosquitto/mosquitto_1.1.2-1.debian.tar.gz

tar -zxf mosquitto_1.1.2-1.debian.tar.gz -C mosquitto-1.1/

The next step is to build the package, but you may find that you need to install some packages first:

sudo apt-get install build-essential python quilt libwrap0-dev libssl-dev devscripts python-setuptools

To build the packages do

cd mosquitto-1.1.2/

debuild

You should now have a list of .deb files in the parent directory which you can install with:

sudo dpkg -i <deb file>

Please leave comments if you find this useful or have any problems.

As part of the ongoing work on mosquitto 0.16, the libmosquitto C client library has been ported to Python. It provides complete MQTTv3.1 support and will eventually remove the need for the current Python wrapper around the C library and will allow it to be used more easily and in more situations.

The interface is largely the same as the existing Python wrapper. The differences are that it uses the current development interface which differs slightly from that in 0.15 (see the Python documentation), not all of the new interface is implemented – there is no threading support and finally some datatypes may be more Python like (e.g. lists in on_subscribe() callback rather than an integer).

The conversion from ~4000 lines C to ~1000 lines Python took just two evenings and is now ready for testing. It is available in the 0.16 branch in the bitbucket repository, or as a single file at http://mosquitto.org/files/python/mosquitto.py

Please give it a try and report any bugs you find using any of the methods on the Support page.

Please note that the new Python module does not currently support Python 3.

Pachube (now Cosm) has recently announced beta support for publishing and receiving data to their service using MQTT. This is great news and something I know that a lot of people have been hoping for. Well done Pachube!

Their documentation is at https://cosm.com/docs/beta/mqtt/ and provides enough information to get going if you’re already familiar with MQTT.

If you aren’t familiar with MQTT, here’s a few examples of how you can use the new service.

First off, I’m going to use the command line MQTT clients I’ve created to publish and receive data. You can get these clients as part of the mosquitto download.

Command Line Examples

Publishing Data

mosquitto_pub -h api.cosm.com -u <your cosm api-key>
              -t /v2/feeds/504.csv -m "0,29"

In this example we’re connecting to host api.cosm.com, using our cosm api-key as the username, publishing to feed /v2/feeds/504 using the csv format and are updating datastream 0 with the value 29. Another way to achieve the same thing would be to do:

mosquitto_pub -h api.cosm.com -u <your cosm api-key>
              -t /v2/feeds/504/datastreams/0.csv -m 29

mosquitto_pub can read data from stdin and publish it, so on Unix type systems the following arrangement is possible:

sensor_read | mosquitto_pub -h api.cosm.com -u <api-key>
                            -t /v2/feeds/504/datastreams/0.csv -l

The “-l” option reads messages from stdin, sending a separate message for each line. This means that our imaginary executable sensor_read that is reading data from a sensor must be printing each reading as a text line.

Retrieving Data

In the MQTT world, retrieving data is done through subscriptions:

mosquitto_sub -h api.cosm.com -u <api-key> -t /v2/feeds/504/datastreams/0.csv

In this example, mosquitto_sub will print a text line containing the csv data for datastream 0 of feed 504 every time it is updated.

Last Will and Testament

The last will and testament or just “will” is a very nice feature of MQTT. When your client connects to the MQTT broker/server, it can give the broker this will, which consists of a topic and a message. If the client is disconnected from the broker unexpectedly, that is to say without sending a disconnect message, then the broker publishes the will message on the will topic.

This provides a very simple mechanism for client connection monitoring. When your client connects it could publish a message “1″ to a topic. If it also set a will to send a message “0″ to the same topic on unexpected disconnect, then it would be possible to determine whether that client was connected by monitoring the topic.

In the context of Cosm, the same approach is possible, but using a trigger to indicate that the client had disconnected.

The mosquitto_sub client provides support for wills as shown in the example below:

mosquitto_sub -h api.cosm.com -u <api-key> -t /v2/feeds/504/datastreams/0.csv
            --will-topic /v2/feeds/12345/datastreams/0.csv -will-payload "0"

In this example, the Cosm broker would publish the value “0″ to datastream 0 of feed 12345  if mosquitto_sub disconnects unexpectedly. This isn’t the most useful example because of the limitations of what mosquitto_sub provides.

Writing Your Own Clients

In practice, to get the full benefit of the advantages that MQTT provides you will probably want to write your own MQTT client to connect to Cosm for your specific application. The mqtt.org software page lists client implementations for lots of different programming languages including the mosquitto client libraries in C/C++, libraries in Java, Python and also device specific implementations for Arduino and other low power devices.

MQTT Beyond Cosm

The Cosm offering is a slightly restricted MQTT offering. “Full” MQTT offers a bit more scope for doing fun things using topic wildcards for example, something that wouldn’t really make sense for Cosm.

There is an overview of MQTT at http://mosquitto.org/man/mqtt-7.html and examples of some applications at http://mosquitto.org/2012/01/do-you-use-mqtt/.

If you’d like to play on an MQTT broker, try looking at http://test.mosquitto.org/.

If you want some help there are mailing lists and irc channels listed on http://mqtt.org/get-involved

I’ve written a tool to help get data from mqtt to pachube. Existing pachube libraries offer good support for updating feeds that have a single datastream or updating all feeds in a datastream, but seem to offer limited support for updating an arbitrary datastream on its own. This can make life difficult when your data is coming in from sensors as individual messages.

mqtt2pachube allows you to choose what mqtt subscriptions to make and then match incoming messages by their topics to a pachube feed and datastream id.

At the moment it is still experimental, but seems to work. It has highlighted a shortcoming in the mosquitto client library, so requires version 0.15.90 (ie. the in-progress work for the next release). There is no Windows support for the moment and no binary packages either. If you are interested in giving it a try, you will have to compile it yourself. If you need help, please get in touch.

There are two examples of feeds created through mqtt2pachube using data from test.mosquitto.org:

• test.mosquitto.org details
• UK energy data – generation source percentage

Thanks to a data feed courtesy of an IBM broker, test.mosquitto.org now publishes information on energy generation and demand in the UK (in the energy/ topic tree). I think this could be used as a great demonstration for coupling MQTT and the web.

The challenge

Create a web based report that takes energy data from the broker over MQTT and displays it in interesting and useful ways. Alternatively, an Android/iPhone app would be ok, but web based is the preferred option.

The rules

There are no rules really. Having said that, I’d be most pleased if the end result was something that other people could learn from. There are bonus points for solutions that work where a web proxy is the only internet access. If you want to use new or unusual technologies that’s fine.

The prize

I’m afraid there is no tangible prize – I hope you’ll be content with your work being shown here and the respect of your peers.

Some suggestions

Google charts is definitely worth looking at for generating the actual graphs. Some examples of what you might show are:

• Pie chart of generation source
• Gauge of current mains frequency
• Historical graph of electricity export amount

I look forward to any and all responses!

Nanode, the popular arduino-with-ethernet board started early in 2011 is ideal for small MQTT based projects but has so far lacked an implementation of MQTT.

Nick O’Leary, the author of the original Arduino MQTT client, has created a Nanode implementation, but it isn’t quite ready for the public.

Nicholas Humfrey has made public some code at https://github.com/njh/NanodeMQTT that he says “still needs some work” but supports publishing QoS 0 messages of up to 127 bytes long and subscribing to topics with QoS 0.

To celebrate the news that the IBM Java MQTT client implementation will be released as open source, I’ve put together a simple Android example based on the MQTT service code written by Dale Lane. I’m a beginner at both Java and Android, so expect it to be a bit rough.

The example displays incoming payload text on a text label. It’s a complete project that you can build and install on your phone with only a few small changes – search for “CHANGE ME” in src/org/mosquitto/android/mqtt/MQTTDemo.java.

To get the project working, assuming you’ve already installed the android sdk, first get the IBM Java library (see http://mqtt.org/software) and put it in <project dir>/lib then do the following:

android update project -p <path to project>
# If the update complains about build.xml - delete it and run again
cd <path to project>
ant debug
sudo adb start-server
ant installd

I’ll not be at all surprised if there are problems in the project due to different sdk or tool versions. Please comment if you find a problem.

The project is available from http://mosquitto.org/files/examples/android-mqtt-example.zip Until the IBM Java implementation is open source please be aware of the licence attached to it.

Thanks to Dale for the core Android MQTT service implementation.

Stephen Nicholas has carried out some power usage analysis of MQTT on Android. Details are at http://stephendnicholas.com/archives/219 and the conclusion is that it doesn’t use much power.

If you’re trying to debug your MQTT connection, you may be interested in something Karl P has written – an MQTT decoder/dissector for Wireshark. It doesn’t have complete protocol support yet, but is a good start.

• http://false.ekta.is/2011/06/mqtt-dissector-decoder-for-wireshark/