[Python] Serializing a C struct in Python and sending over a socket

I'm trying to serializing the following C struct

struct packet
{
int id;
unsigned char *ce;
unsigned char *syms;
};


in Python and send it over a socket. The number of elements pointed by ce and syms are known to be N. Currently I'm doing this way. First, I wrap the struct using ctypes to

class Packet(Structure):
_fields_ = [("id", c_int),
("ce", POINTER(c_ubyte)),
("syms", POINTER(c_ubyte))]


Then, I populate an object of the following class using its fill_data function from a ctypes.POINTER(Packet):

class DataLoad:
def __init__(self):
self.id = -1
self.ce = []
self.syms = []

def fill_data(self, pkt_p):
""" pkt_p is POINTER(Packet)
"""
self.id = pkt_p.contents.id
self.ce = []
for i in range(N):
self.ce.append(pkt_p.contents.ce)
self.syms = []
for i in range(N):
self.syms.append(pkt_p.contents.syms)


Finally, I simply use pickle.dumps(DataLoad) to generate a byte stream and send.

This approach works well. However, it seems to be quite slow. One reason I can see is that pickle.dumps bring much overhead. For example, if the C struct is only 1024 bytes, I may have to send almost 4000 bytes for each struct using pickle. Also, packing/populating DataLoad also takes time.

So my question is, do I have other better choices to serialize this C struct in python and send? I would prefer to avoid pickle and populate a separate class instance. Thanks.

Continue reading...