BLE Gateway V4 MQTT and problems getting information msgpack.unpackb

Hello,

I am trying to use the information coming from the AB BLE Gateway V4 via MQTT server into python. After unpacking via msgpack.unpack (payload) the devices array seems to be partially messed up:

Here an example:
{‘ip’: ‘192.168.2.116’, ‘v’: ‘1.1.0.26’, ‘mid’: 1558, ‘devices’: [’\x03\x12;j\x1bE\xcc\xc1\x02\x01\x06\x1a\xffL\x00\x02\x15^\xb2\xae\xdeO{1n#|\xd7 H\x7f\xb4\xb0\x00\x01\x00\x07\xc5’, ‘\x03\xc4S9<\x9d\xe6\xc7\x02\x01\x06\x1b\xffL\x00\x02\x15^\xb2\xae\xdeO{1n#|\xd7 H\x7f\xb4\xb0\x00\td\x0b\xcbd’, ‘\x03\x12;j\x1bE\xcf\xc0\x02\x01\x06\x1a\xffL\x00\x02\x15^\xb2\xae\xdeO{1n#|\xd7 H\x7f\xb4\xb0\x00\x01\x00\x08\xc5’, ‘\x03\xe9D\xf6\x1eBE\xb9\x02\x01\x06\x1a\xffL\x00\x02\x15^\xb2\xae\xdeO{1n#|\xd7 H\x7f\xb4\xb0\x00\td\x05\xc5’, ‘\x03\x12;j\x1bE\xcf\xbf\x02\x01\x06\x1a\xffL\x00\x02\x15^\xb2\xae\xdeO{1n#|\xd7 H\x7f\xb4\xb0\x00\x01\x00\x08\xc5’, ‘\x03\xde\xfb)\6\x0f\xc2\x02\x01\x06\x1b\xffL\x00\x02\x15^\xb2\xae\xdeO{1n#|\xd7 H\x7f\xb4\xb0\x00\tY\t\xcbY’, ‘\x03\x12;j\x1bE\xcc\xc1\x02\x01\x06\x1a\xffL\x00\x02\x15^\xb2\xae\xdeO{1n#|\xd7 H\x7f\xb4\xb0\x00\x01\x00\x07\xc5’, ‘\x03\x12;j\x1bE\xcc\xc1\x02\x01\x06\x1a\xffL\x00\x02\x15^\xb2\xae\xdeO{1n#|\xd7 H\x7f\xb4\xb0\x00\x01\x00\x07\xc5’, ‘\x03\xe9D\xf6\x1eBE\xb8\x02\x01\x06\x1a\xffL\x00\x02\x15^\xb2\xae\xdeO{1n#|\xd7 H\x7f\xb4\xb0\x00\td\x05\xc5’, ‘\x03\xdf\xca\x9e\xcd\xcfi\xb7\x02\x01\x06\x1b\xffL\x00\x02\x15^\xb2\xae\xdeO{1n#|\xd7 H\x7f\xb4\xb0\x00\td\n\xcbd’, ‘\x03\x12;j\x1bE\xcf\xc0\x02\x01\x06\x1a\xffL\x00\x02\x15^\xb2\xae\xdeO{1n#|\xd7 H\x7f\xb4\xb0\x00\x01\x00\x08\xc5’, ‘\x03\xde\xfb)\6\x0f\xc2\x02\x01\x06\x1b\xffL\x00\x02\x15^\xb2\xae\xdeO{1n#|\xd7 H\x7f\xb4\xb0\x00\tY\t\xcbY’, ‘\x03\x12;j\x1bE\xcc\xc0\x02\x01\x06\x1a\xffL\x00\x02\x15^\xb2\xae\xdeO{1n#|\xd7 H\x7f\xb4\xb0\x00\x01\x00\x07\xc5’, ‘\x03\xc4S9<\x9d\xe6\xc8\x02\x01\x06\x1b\xffL\x00\x02\x15^\xb2\xae\xdeO{1n#|\xd7 H\x7f\xb4\xb0\x00\td\x0b\xcbd’, ‘\x03\x12;j\x1bE\xcf\xc1\x02\x01\x06\x1a\xffL\x00\x02\x15^\xb2\xae\xdeO{1n#|\xd7 H\x7f\xb4\xb0\x00\x01\x00\x08\xc5’, ‘\x03\xe9D\xf6\x1eBE\xb5\x02\x01\x06\x1a\xffL\x00\x02\x15^\xb2\xae\xdeO{1n#|\xd7 H\x7f\xb4\xb0\x00\td\x05\xc5’, ‘\x03\x12;j\x1bE\xcf\xb7\x02\x01\x06\x1a\xffL\x00\x02\x15^\xb2\xae\xdeO{1n#|\xd7 H\x7f\xb4\xb0\x00\x01\x00\x08\xc5’], ‘mac’: ‘30AEA490FE14’, ‘time’: 404}

Any idea why this happened and how to correctly unpack the information to not mess up the devices part?

Any help or idea would be great

Ingo

Please check the wiki for data format. The devices array contains raw advertising data in hex format.

Thank you for the quick answer. Yes I know the format, but when using the standard python unpack on the mqtt message the content of the devices part is messed up. For the time being I am using a http server to receive the message which works, but I need eventually get mqtt working. Do you habe any idea why the message content is not received or unpacked properly. For example via web service the content is after unpack something like

03defb295c360fc10201061bff4c0002155eb2aede4f7b316e237cd720487fb4b000095809cb58

but via mqtt it is somehow messed up and not even pure hex after the unpack command:

\x03\x12;j\x1bE\xcc\xc1\x02\x01\x06\x1a\xffL\x00\x02\x15^\xb2\xae\xdeO{1n#|\xd7 H\x7f\xb4\xb0\x00\x01\x00\x07\xc5

so I guess my method of getting the mqtt message and then unpacking in python via:

messagestring = str(msgpack.unpackb(theMessage.payload, use_list=False, raw=True))

is wrong, but I have no idea what to try. I tried it with raw=False already, no significant change. Do you have any hint what to try?

Regards
Ingo

So which unpack message is right?

03defb295c360fc10201061bff4c0002155eb2aede4f7b316e237cd720487fb4b000095809cb58

or

\x03\x12;j\x1bE\xcc\xc1\x02\x01\x06\x1a\xffL\x00\x02\x15^\xb2\xae\xdeO{1n#|\xd7 H\x7f\xb4\xb0\x00\x01\x00\x07\xc5

The first one coming through http is fine, but in the second one coming through MQTT are not only hexbin data e.g. \x03 but elements like \xffL or \x15^ or \xdeO{1n#| … I would expect only \x followed by two hex letters. So something on the way from sniffer -> MQTT -> python -> unpack -> messagestring goes wrong and I don‘t know what happens. Do you have maybe a code example for python to extract the data from the MQTT message coming from the V4 BLE Gateway? The code I use
messagestring = str(msgpack.unpackb(theMessage.payload, use_list=False, raw=True))
Does not do the trick.
Ingo

No. I have only example PHP code for parse data from MQTT. I think the problem is python side. What mean is this parameter? – use_list=False

The parameter use_list=False only changes the output format, if it comes as an array or as one string. It does not change the content, I tried both. Can you please send me the php code example, then I will try it later this week to identify if the problem also appears there. this would tell me if the problem is maybe in the MQTT part and the MQTT Server needs to be changed.

The php SDK is here. Please check the source file examples/example_gateway4_mqtt.php

I have exactly the same question: we are just asking how can we convert the message publish in a format like “x03\x12;j\x1bE\xcc\xc1\x02\x01\x06\x1a” to a readable format

Please help!

Which programming language do you prefer? Please also check the section “Write your own client

I m using python like this :

import time
import paho.mqtt.client as mqtt
import msgpack

def on_subscribe(mosq, obj, mid, granted_qos):
    print("Subscribed: " + str(mid))

def on_message(mosq, obj, msg):
    print("=============================================")
    for d in msgpack.unpackb(msg.payload)[b'devices']:
      print(d)
def on_connect(mosq, obj,flags, rc):
    print("Connected with result code "+str(rc))
    mqttc.subscribe("lolo", 0)
    print("Connected")


mqttc = mqtt.Client()
mqttc.on_connect = on_connect
mqttc.on_subscribe = on_subscribe
mqttc.on_message = on_message
mqttc.connect("mqtt.bconimg.com", 1883, 60)
mqttc.loop_forever()

And this is the result :

=============================================
b'\x00DO\xf2\x1c\xac\xe6\xbb\x02\x01\x1a\n\xffL\x00\x10\x05\x03\x18\xf8D\x00'
b'\x00S\xb2\xd12\xa80\xa7\x02\x01\x1a\t\xff\xc4\x00\x133\x02\x13\x15\x80\x03\x03\xb9\xfe\x08\x1b\x00$\x98\x05N\x1c|'
b'\x00O\x08:\x1d\x0c\xad\xa6\x02\x01\x1a\x1a\xffL\x00\x0c\x0e\x00z0\x1d|\xceF\x985\x00\x12\xd0fM\x10\x05\x0b\x1c\xab\x1d\xef'
b'\x00hdK\x0c\xb4S\xa3\x02\x01\x1a\x0b\xffL\x00\t\x06\x03t\xc0\xa8\n\x05'
b'\x03k\xd5\x04\x13\xf8P\xa7\x1e\xff\x06\x00\x01\t \x02|\x02\xa0\xba\xf5^\xba\x11\x1e!\x9a \xb8\xe9\xa2\x12\x86w`\xd9\xcct\xf3'

How can I convert this data match this : https://wiki.aprbrother.com/en/User_Guide_For_AB_BLE_Gateway_V4.html

Without your help, I just can not use your product

Thanks in advance

Laurent

We’ll check the python example and reply to you soon

Hi Laurent,

I modified your code and get better result

import time
import paho.mqtt.client as mqtt
import msgpack

def on_subscribe(mosq, obj, mid, granted_qos):
    print("Subscribed: " + str(mid))

def on_message(mosq, obj, msg):
    for d in msgpack.unpackb(msg.payload)[b'devices']:
      print("=============================================")
      # adv type
      print "type:", ord(d[0])
      print "mac:{:02X}{:02X}{:02X}{:02X}{:02X}{:02X}".format(ord(d[1]), ord(d[2]), ord(d[3]), ord(d[4]), ord(d[5]), ord(d[6]))
      print "rssi:", ord(d[7]) - 256
      hex_chars = map(hex,map(ord,d))
      del hex_chars[:8]
      print "adv:", hex_chars

def on_connect(mosq, obj,flags, rc):
    print("Connected with result code "+str(rc))
    mqttc.subscribe("hello", 0)
    print("Connected")
    

mqttc = mqtt.Client()
mqttc.on_connect = on_connect
mqttc.on_subscribe = on_subscribe
mqttc.on_message = on_message
mqttc.connect("ab.bconimg.com", 1883, 60)
mqttc.loop_forever()

here’s my parsing result

type: 0
mac:123B6A1B59E5
rssi: -78
adv: ['0x2', '0x1', '0x6', '0x1a', '0xff', '0x4c', '0x0', '0x2', '0x15', '0xb5', '0xb1', '0x82', '0xc7', '0xea', '0xb1', '0x49', '0x88', '0xaa', '0x99', '0xb5', '0xc1', '0x51', '0x70', '0x8', '0xd9', '0x0', '0x1', '0xe5', '0x59', '0xc5']
=============================================
type: 3
mac:2026698D97F1
rssi: -66
adv: ['0x1e', '0xff', '0x6', '0x0', '0x1', '0x9', '0x20', '0x2', '0xa1', '0x22', '0xe0', '0xdd', '0xf0', '0xc9', '0xf7', '0xb6', '0x40', '0xb6', '0xb9', '0x8e', '0x6e', '0x2', '0x19', '0x1a', '0xe9', '0x60', '0xaf', '0xf6', '0x6e', '0xf5', '0x3b']
=============================================
type: 0
mac:123B6A1B97F0
rssi: -76
adv: ['0x2', '0x1', '0x6', '0x1a', '0xff', '0x4c', '0x0', '0x2', '0x15', '0xb5', '0xb1', '0x82', '0xc7', '0xea', '0xb1', '0x49', '0x88', '0xaa', '0x99', '0xb5', '0xc1', '0x51', '0x70', '0x8', '0xd9', '0x0', '0x1', '0xf0', '0x97', '0xc5']

Hope it’s helpful for you

Great! Thanks you very much!

Yes a big thank you from me too

so I am trying this example, and the examples in the public “SDK” and they are not working for me. I’ve added some debug logging in the beacon library and I am getting the following error when it tries to parse the iBeacon frame

=============================================
type: 0
mac:55FB39EAE43C
rssi: -57
adv: ['0x2', '0x1', '0x1a', '0x2', '0xa', '0xc', '0xb', '0xff', '0x4c', '0x0', '0x10', '0x6', '0x1a', '0x1e', '0x2f', '0x7d', '0xef', '0x99']
parsing expected '\x02\x01' but parsed '\x11\x06'
=============================================

my code is as follows

# -*- coding:utf-8 -*-
import time
import paho.mqtt.client as mqtt
import msgpack
from beacontools import parse_packet


def on_subscribe(mosq, obj, mid, granted_qos):
    print("Subscribed: " + str(mid))


def on_message(mosq, obj, msg):
    # print("got payload: %s" % msgpack.unpackb(msg.payload))
    for d in msgpack.unpackb(msg.payload)[b'devices']:
        # parse iBeacon data
        advData = d[8:]

        adv = parse_packet(advData)

        # if adv is None:
        #     continue
        print("=============================================")
        # adv type
        print "type:", ord(d[0])
        print "mac:{:02X}{:02X}{:02X}{:02X}{:02X}{:02X}".format(ord(d[1]), ord(d[2]), ord(d[3]), ord(d[4]), ord(d[5]), ord(d[6]))
        print "rssi:", ord(d[7]) - 256
        hex_chars = map(hex,map(ord,d))
        del hex_chars[:8]
        print "adv:", hex_chars
        print parse_packet(hex_chars)

        # print("=============================================")
        # print "mac:{:02X}{:02X}{:02X}{:02X}{:02X}{:02X}".format(ord(d[1]), ord(d[2]), ord(d[3]), ord(d[4]), ord(d[5]), ord(d[6]))
        # print "rssi: ", ord(d[7]) - 256
        # print("UUID: %s" % adv.uuid)
        # print("Major: %d" % adv.major)
        # print("Minor: %d" % adv.minor)
        # print("TX Power: %d" % adv.tx_power)
        # print("foo")

def on_connect(mosq, obj, flags, rc):
    mqttTopic = "ab/hello"
    print("Connected with result code " + str(rc))
    mqttc.subscribe(mqttTopic, 0)
    print("Connected")


mqttHost = "192.168.100.14"
mqttPort = 1883
mqttc = mqtt.Client()
mqttc.on_connect = on_connect
mqttc.on_subscribe = on_subscribe
mqttc.on_message = on_message
mqttc.username_pw_set(username='local-user', password='xxxxxxx')
mqttc.connect(mqttHost, mqttPort, 60)
mqttc.loop_forever()

the beacontools change I made was as follows in the parse_ibeacon_packet method, where I printed the exception

"""Beacon advertisement parser."""
from construct import ConstructError

from .structs import LTVFrame, IBeaconAdvertisingPacket
from .packet_types import EddystoneUIDFrame, EddystoneURLFrame, EddystoneEncryptedTLMFrame, \
                          EddystoneTLMFrame, EddystoneEIDFrame, IBeaconAdvertisement, \
                          EstimoteTelemetryFrameA, EstimoteTelemetryFrameB
from .const import EDDYSTONE_TLM_UNENCRYPTED, EDDYSTONE_TLM_ENCRYPTED, SERVICE_DATA_TYPE, \
                   EDDYSTONE_UID_FRAME, EDDYSTONE_TLM_FRAME, EDDYSTONE_URL_FRAME, \
                   EDDYSTONE_EID_FRAME, EDDYSTONE_UUID, ESTIMOTE_UUID, ESTIMOTE_TELEMETRY_FRAME, \
                   ESTIMOTE_TELEMETRY_SUBFRAME_A, ESTIMOTE_TELEMETRY_SUBFRAME_B


def parse_packet(packet):

    """Parse a beacon advertisement packet."""
    frame = parse_ltv_packet(packet)
    if frame is None:
        frame = parse_ibeacon_packet(packet)
    return frame

def parse_ltv_packet(packet):
    """Parse a tag-length-value style beacon packet."""
    try:
        frame = LTVFrame.parse(packet)
        for ltv in frame:
            if ltv['type'] == SERVICE_DATA_TYPE:
                data = ltv['value']

                if data["service_identifier"] == EDDYSTONE_UUID:
                    return parse_eddystone_service_data(data)

                elif data["service_identifier"] == ESTIMOTE_UUID:
                    return parse_estimote_service_data(data)

    except ConstructError:
        return None

    return None

def parse_eddystone_service_data(data):
    """Parse Eddystone service data."""
    if data['frame_type'] == EDDYSTONE_UID_FRAME:
        return EddystoneUIDFrame(data['frame'])

    elif data['frame_type'] == EDDYSTONE_TLM_FRAME:
        if data['frame']['tlm_version'] == EDDYSTONE_TLM_ENCRYPTED:
            return EddystoneEncryptedTLMFrame(data['frame']['data'])
        elif data['frame']['tlm_version'] == EDDYSTONE_TLM_UNENCRYPTED:
            return EddystoneTLMFrame(data['frame']['data'])

    elif data['frame_type'] == EDDYSTONE_URL_FRAME:
        return EddystoneURLFrame(data['frame'])

    elif data['frame_type'] == EDDYSTONE_EID_FRAME:
        return EddystoneEIDFrame(data['frame'])
    else:
        return None

def parse_estimote_service_data(data):
    """Parse Estimote service data."""
    if data['frame_type'] & 0xF == ESTIMOTE_TELEMETRY_FRAME:
        protocol_version = (data['frame_type'] & 0xF0) >> 4
        if data['frame']['subframe_type'] == ESTIMOTE_TELEMETRY_SUBFRAME_A:
            return EstimoteTelemetryFrameA(data['frame'], protocol_version)
        elif data['frame']['subframe_type'] == ESTIMOTE_TELEMETRY_SUBFRAME_B:
            return EstimoteTelemetryFrameB(data['frame'], protocol_version)
    return None

def parse_ibeacon_packet(packet):
    """Parse an ibeacon beacon advertisement packet."""
    try:
        pkt = IBeaconAdvertisingPacket.parse(packet)
        return IBeaconAdvertisement(pkt)

    except ConstructError as e:
        print(e)
        return None

The adverting data is not a valid iBeacon profile. Please search iBeacon profile at stackoverflow.

Note: Please create a new topic on this matter, as we try to keep each subject in separate topics here on support forum.

Hello,

while I am starting to use your BLE Gateways for my own RTLS I am struggling with the very first steps. I have already implemented your BLE Gateway in my WIFI (with Config-Tool, Linux). Unfortunately, I can’t test the Gateway, because I only running Linux & Mac OS.
So I tried to test the BLE Gateway with your Python mqtt-client (https://github.com/AprilBrother/ab-ble-gateway-sdk/tree/master/gateway-v4/examples/python/mqtt-client), but I get not signal or parsing result respectively. I seems, that the Gateway is not found.

Do you habe any ideas?

Thank you very much!

Best,
Joern

  • we opened the source code of ble viewer. You can run it with Linux & macOS by node
  • For the python mqtt-client, please try with python2 instead. Could you please post a screenshot about the parsing result?

Thank you for your support.

The python mqtt-client code seems to “run” properly under python3. I get following output with 1 active BLE-Gateway:

Connected with result code 0
Connected
Subscribed: 1

What I don’t get is the parsing information - it seems, that the function

mqttc.on_message = on_message

will not be excecuted. Any ideas?