I spend an unreasonable amount of time trying to figure this out. How to pack your data so you can send it across a TCP connection, receive it on the other side, know what data you received and unpack it accordingly. Hence sharing it, hopefully its useful to someone out there ^_^
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 |
# -*- coding: utf-8 -*- ## Copyright 2019 Trevor van Hoof and Jan Pijpers. ## Licensed under the Apache License, Version 2.0 ## Downloaded from https://janpijpers.com or https://gumroad.com/janpijpers ## See the license file attached or on https://www.janpijpers.com/script-licenses/ ''' Name: bytesForNetworkTransfer Description: byte tests for data transmission across a network Simple example on how to create a data packet for sending across a tcp network connection. ''' import sys import struct def create_packet( message, type = 100 ): ''' This packs our string message into a packet that we can send across a network. The type makes it easier to know how to handle the data on the receiving end. For example, type 100 means normal text data but type 200 can mean jpg image data. the way we construct our packet is as follows. [size][type][message] the first part will say how big our message is. the second part will hold the type number the third part will be the message. :param message: a byte string message. :param type: the message type. This can be any number between 0 and 255. :return: ''' if not isinstance(message, basestring): raise TypeError("The message is not a string. ") try: ## Lets make sure we can encode the data as utf8 and back decodedMessage = messageContents.decode("utf8") ## data is now unicode encodedMessage = decodedMessage.encode("utf8") ## data is now back to binary. except UnicodeDecodeError as e: print "This message can not be decoded as UTF8. There are characters in there utf8 doesnt understand." raise e ## We use the sys getsizeof function to find out how much bytes this string takes in memory. ## using len would give us the number of characters, but not actually the number of bytes. messageSize = sys.getsizeof(message) ## Lets start packing this. ## For this we use the struct module. ## https://docs.python.org/2/library/struct.html ## First we write the size. ## >I means unsigned int. so a maximum value of 65,535 ## if you expect bigger messages you should use >Q byteString = struct.pack(">I", messageSize) ## Then we write the type ## >B is an unsigned char. it allows for 256 different types. byteString += struct.pack(">B", type) ## Now we add our contents. byteString += messageContents return byteString def read_packet( incommingMessage ): ''' Here we decode our message. There are a few interesting caveats with network data transmission. In this case I will assume a tcp network connection. That means that "in theory" all the data comes in, in order. However you can not always be sure that you received exactly one full message. So if we do not have enough data, this function returns [None, remaining data] if we have exactly the right amount of data [extracted message, ""] if we have more data left [extracted message, remaining data] Because we know how we wrote our create packet function, we know that the message size is unsigned int( >I ) which is 4 bytes long *see the table in the struct python docs And the same goes for the type which is 1 byte long. so we use that information here. :param incommingMessage: the message as we received it from the network. :return: list [the message content, the remaining data] ''' if len( incommingMessage ) < 4: return [None, incommingMessage] sizeBytes = incommingMessage[:4] unpackedMessageSize = struct.unpack(">I", sizeBytes)[0] typeBytes = incommingMessage[4] unpackedMessageType = struct.unpack(">B", typeBytes)[0] unpackedMessage = incommingMessage[5:unpackedMessageSize] ## Now we could do something with our data and handle it ## according to the unpackedMessageType. ## i.e. ## if unpackedMessageType == 200: ## unpackedMessage = bytes_to_image(unpackedMessage) return [unpackedMessage, incommingMessage[unpackedMessageSize+5:]] ## If we do not specify encoding then python will store this string as byte code. ## using japanese characters so we cover some wierd edge cases. messageContents = "textAnd漢字カタカナ" ## Create the packet ready for sending. networkPacket = create_packet(messageContents) ## Decode it once we have received it. decodedMessage, remainingPacketData = read_packet(networkPacket) print "Original: ", repr(messageContents) print "Decoded : ", repr(decodedMessage) assert(messageContents == decodedMessage) print "All tests passed" |