Python client module available for testing

As part of the ongoing work on mosquitto 0.16, the libmosquitto C client library has been ported to Python. It provides complete MQTTv3.1 support and will eventually remove the need for the current Python wrapper around the C library and will allow it to be used more easily and in more situations.

The interface is largely the same as the existing Python wrapper. The differences are that it uses the current development interface which differs slightly from that in 0.15 (see the Python documentation), not all of the new interface is implemented - there is no threading support and finally some datatypes may be more Python like (e.g. lists in on_subscribe() callback rather than an integer).

The conversion from ~4000 lines C to ~1000 lines Python took just two evenings and is now ready for testing. It is available in the 0.16 branch in the bitbucket repository, or as a single file at http://mosquitto.org/files/python/mosquitto.py

Please give it a try and report any bugs you find using any of the methods on the Support page.

Please note that the new Python module does not currently support Python 3.


Quick start guide for MQTT with Pachube/Cosm/Xively

Pachube (now Cosm) has recently announced beta support for publishing and receiving data to their service using MQTT. This is great news and something I know that a lot of people have been hoping for. Well done Pachube!

Their documentation is at https://xively.com/dev/docs/api/communicating/mqtts/ and provides enough information to get going if you're already familiar with MQTT.

If you aren't familiar with MQTT, here's a few examples of how you can use the new service.

First off, I'm going to use the command line MQTT clients I've created to publish and receive data. You can get these clients as part of the mosquitto download.

Command Line Examples

Publishing Data

mosquitto_pub -h api.xively.com
              -u <your xively api-key>
              -t /v2/feeds/504.csv
              -m "0,29"

In this example we're connecting to host api.xively.com, using our xively api-key as the username, publishing to feed /v2/feeds/504 using the csv format and are updating datastream 0 with the value 29. Another way to achieve the same thing would be to do:

mosquitto_pub -h api.xively.com
              -u <your xively api-key>
              -t /v2/feeds/504/datastreams/0.csv
              -m 29

mosquitto_pub can read data from stdin and publish it, so on Unix type systems the following arrangement is possible:

sensor_read | mosquitto_pub -h api.xively.com
                            -u <api-key>
                            -t /v2/feeds/504/datastreams/0.csv
                            -l

The -l option reads messages from stdin, sending a separate message for each line. This means that our imaginary executable sensor_read that is reading data from a sensor must be printing each reading as a text line.

Retrieving Data

In the MQTT world, retrieving data is done through subscriptions:

mosquitto_sub -h api.xively.com
              -u <api-key>
              -t /v2/feeds/504/datastreams/0.csv

In this example, mosquitto_sub will print a text line containing the csv data for datastream 0 of feed 504 every time it is updated.

Last Will and Testament

The last will and testament or just "will" is a very nice feature of MQTT. When your client connects to the MQTT broker/server, it can give the broker this will, which consists of a topic and a message. If the client is disconnected from the broker unexpectedly, that is to say without sending a disconnect message, then the broker publishes the will message on the will topic.

This provides a very simple mechanism for client connection monitoring. When your client connects it could publish a message "1" to a topic. If it also set a will to send a message "0" to the same topic on unexpected disconnect, then it would be possible to determine whether that client was connected by monitoring the topic.

In the context of Xively, the same approach is possible, but using a trigger to indicate that the client had disconnected.

The mosquitto_sub client provides support for wills as shown in the example below:

mosquitto_sub -h api.xively.com
              -u <api-key>
              -t /v2/feeds/504/datastreams/0.csv
              --will-topic /v2/feeds/12345/datastreams/0.csv
              --will-payload "0"

In this example, the Xively broker would publish the value "0" to datastream 0 of feed 12345  if mosquitto_sub disconnects unexpectedly. This isn't the most useful example because of the limitations of what mosquitto_sub provides.

Writing Your Own Clients

In practice, to get the full benefit of the advantages that MQTT provides you will probably want to write your own MQTT client to connect to Xively for your specific application. The http://mqtt.org/software page lists client implementations for lots of different programming languages including the mosquitto client libraries in C/C++, libraries in Java, Python and also device specific implementations for Arduino and other low power devices.

MQTT Beyond Xively

The Xively offering is a slightly restricted MQTT offering. "Full" MQTT offers a bit more scope for doing fun things using topic wildcards for example, something that wouldn't really make sense for Xively.

There is an overview of MQTT at mqtt man page and examples of some applications at http://mosquitto.org/2012/01/do-you-use-mqtt/.

If you'd like to play on an MQTT broker, try looking at test.mosquitto.org.

If you want some help there are mailing lists and irc channels listed on http://mqtt.org/get-involved.


Upcoming incompatible library changes

Version 0.16 of the mosquitto client libraries will have some binary incompatible changes to their APIs. This means that it is a good time to make other changes that are incompatible. If you think any part of the interface (see http://mosquitto.org/api/) is crazy or could be improved in any way, please get in touch or add a comment below.


mqtt2pachube

I've written a tool to help get data from mqtt to pachube. Existing pachube libraries offer good support for updating feeds that have a single datastream or updating all feeds in a datastream, but seem to offer limited support for updating an arbitrary datastream on its own. This can make life difficult when your data is coming in from sensors as individual messages.

mqtt2pachube allows you to choose what mqtt subscriptions to make and then match incoming messages by their topics to a pachube feed and datastream id.

At the moment it is still experimental, but seems to work. It has highlighted a shortcoming in the mosquitto client library, so requires version 0.15.90 (ie. the in-progress work for the next release). There is no Windows support for the moment and no binary packages either. If you are interested in giving it a try, you will have to compile it yourself. If you need help, please get in touch.

There are two examples of feeds created through mqtt2pachube using data from test.mosquitto.org


Version 0.15 released

This is a feature and bugfix release.

  • Implement "once" and "lazy" bridge start types.
  • Add support for $SYS/broker/clients/maximum and $SYS/broker/clients/active topics.
  • Add support for $SYS messages/byte per second received/sent topics.
  • Updated mosquitto man page - $SYS hierarchy and signal support were out of date.
  • Auto generated pub/sub client ids now include the hostname.
  • Tool for dumping persistent DB contents is available in src/db_dump. It isn't installed by default.
  • Enforce topic length checks in client library.
  • Add new return type MOSQ_ERR_ERRNO to indicate that the errno variable should be checked for the real error code.
  • Add support for connection_messages config option.
  • mosquitto_sub will now refuse to run if the -c option (disable clean session) is given and no client id is provided.
  • mosquitto_pub now gives more useful error messages on invalid input or other error conditions.
  • Fix Python will_set() true/True typo.
  • Fix messages to topic a/b incorrectly matching on a subscription a if another subscription a/# exists.

Challenge: Web based MQTT graphing

Thanks to a data feed courtesy of an IBM broker, test.mosquitto.org now publishes information on energy generation and demand in the UK (in the energy/ topic tree). I think this could be used as a great demonstration for coupling MQTT and the web.

The challenge

Create a web based report that takes energy data from the broker over MQTT and displays it in interesting and useful ways. Alternatively, an Android/iPhone app would be ok, but web based is the preferred option.

The rules

There are no rules really. Having said that, I'd be most pleased if the end result was something that other people could learn from. There are bonus points for solutions that work where a web proxy is the only internet access. If you want to use new or unusual technologies that's fine.

The prize

I'm afraid there is no tangible prize - I hope you'll be content with your work being shown here and the respect of your peers.

Some suggestions

Google charts is definitely worth looking at for generating the actual graphs. Some examples of what you might show are:

  • Pie chart of generation source
  • Gauge of current mains frequency
  • Historical graph of electricity export amount

I look forward to any and all responses!


Version 0.14.4 released

This is a bugfix release:

  • Fix local bridge notification messages.
  • Fix return values for more internal library calls.
  • Fix incorrect out of memory checks in library and broker.
  • Never time out local bridge connections.

Do you use MQTT?

I saw this in the nanode irc channel:

I've never seen any real world projects with MQTT... it looks good though.

So I'm looking for real world projects that use MQTT. If you've got a project it'd be great if you could mention it in the comments. A short sentence on what it does and how many clients you run on it - really anything you can say. If it's a secret please still comment if you can, just be very very vague. If you've got a blog post describing it, link that instead. I'm interested in everything from a single temperature sensor reporting to a computer up to millions of mobile users.

Thanks!


MQTT on Nanode

Nanode, the popular arduino-with-ethernet board started early in 2011 is ideal for small MQTT based projects but has so far lacked an implementation of MQTT.

Nick O'Leary, the author of the original Arduino MQTT client, has created a Nanode implementation, but it isn't quite ready for the public.

Nicholas Humfrey has made public some code at https://github.com/njh/NanodeMQTT that he says still needs some work but supports publishing QoS 0 messages of up to 127 bytes long and subscribing to topics with QoS 0.