Verified Commit 7034e36b authored by Robin Obůrka's avatar Robin Obůrka Committed by Martin Prudek

msgpack: Fix compatibilty issues

The messagepack history started in days when unicode, string, and bytes
(or binary data types) were not distingushed in Python. All string data was
packed as string - and also unpacked.

This changed with Python2 which introduced Unicode - marked as u"Unicode string".
(The default string remained unmarked or possibly marked as bytes
"default string"=b"default string".)

Python3 then switched its default string directly to Unicode. ("Unicode
string"=u"Unicode string". Bytes remained marked as b"bytes string")

To solve this, two logical parametres were
introduced to messagepack - one for packer and the other one for unpacker:

* use_bin_type
    - packer
    - default before 1.0: `False`
    - default after 1.0: `True`
    - when set, the packer distinguish between unicode(raw) and bytes(bin)
    - when unset, the packer doesnt distinguish between unicode and bytes and
      packs everything as internal "raw"

* raw
    - unpacker
    - default before 1.0: `True`
    - default after 1.0: `False`
    - when set, the unpacker doesnt respect internal data types and transforms
      everything either to  bytes when `use_bin_type` is set or to
      unicode when it is not set
    - when unset, the unpacker respects internal data types and transforms
      "internal bin" to bytest and "internal raw" to unicode

Default values before 1.0 were set to be compatible with old versions
that threats everything as "raw" (ASCII) string. So everything was
packed as "raw" (nothing as "bin") and everythong unpacked as (Unicode)
string.

Default values after 1.0 ensure that everything is unpacked as the same
type as it was packed.

The stable version is 0.6.0 right now. Here is a compatibility list of
previous versions:

* Version 0.6.0 - compatible
    - needs catching of `Value Error` in `parse_msg`
* Version 0.5.6 - compatible
* Version 0.5.5 - compatible
* Version 0.5.4 - compatible
* Version 0.5.3 - compatible
    - deprecated (uses some deprecated internals)
* Version 0.5.2 - compatible
    - deprecated (uses some deprecated internals)
* Version 0.5.1 - incompatible
    - unknown keyword "raw" in unpack(raw=...)
parent f9d58727
......@@ -16,12 +16,12 @@ def parse_msg(data):
msg_type = str(data[0], encoding="UTF-8")
if not SN_MSG.match(msg_type):
raise InvalidMsgTypeError("Bad message type definition")
payload = msgpack.unpackb(data[1], encoding="UTF-8")
payload = msgpack.unpackb(data[1], raw=False)
except IndexError:
raise InvalidMsgError("Not enough parts in message")
except (TypeError, msgpack.exceptions.UnpackException, UnicodeDecodeError):
except (ValueError, TypeError, msgpack.exceptions.UnpackException, UnicodeDecodeError):
raise InvalidMsgError("Broken message")
return msg_type, payload
......@@ -39,6 +39,6 @@ def encode_msg(msg_type, data):
raise InvalidMsgError("Empty payload parameter")
b = bytes(msg_type, encoding="UTF-8")
msg = msgpack.packb(data)
msg = msgpack.packb(data, use_bin_type=True)
return (b, msg)
......@@ -58,7 +58,7 @@ def good_payload():
@pytest.fixture
def good_msg(good_type, good_payload):
t = bytes(good_type, encoding="UTF-8")
p = msgpack.packb(good_payload, encoding="UTF-8")
p = msgpack.packb(good_payload, use_bin_type=True)
return (t, p)
......@@ -66,8 +66,8 @@ def good_msg(good_type, good_payload):
@pytest.fixture
def good_long(good_type, good_payload):
t = bytes(good_type, encoding="UTF-8")
p = msgpack.packb(good_payload, encoding="UTF-8")
x = msgpack.packb(good_payload, encoding="UTF-8")
p = msgpack.packb(good_payload, use_bin_type=True)
x = msgpack.packb(good_payload, use_bin_type=True)
return (t, p, x)
......
......@@ -14,7 +14,7 @@ def args_from_string(s):
def build_msg(msg_type, payload):
t = bytes(msg_type, encoding="UTF-8")
p = msgpack.packb(payload, encoding="UTF-8")
p = msgpack.packb(payload, use_bin_type=True)
return (t, p)
......
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment