[Python] Grpc not increasing max size of the message: received message larger than max

I am using grpc to send some pretty large messages (the parameters of a machine learning model over the network). The problem is that I am getting the following error when I make a grpc call:

grpc: received message larger than max (261268499 vs. 4194304)


As suggested in other posts I tried to increase the max message size on the channel and the grpc server, but I keep getting the same error. Any idea on how to get this to work?

My code for the server:

maxMsgLength = 1024 * 1024 * 1024
server = grpc.server(futures.ThreadPoolExecutor(max_workers=10),
options=[('grpc.max_message_length', maxMsgLength),
('grpc.max_send_message_length', maxMsgLength),
('grpc.max_receive_message_length', maxMsgLength)])


The client:

maxMsgLength = 1024 * 1024 * 1024
channel = grpc.insecure_channel(ip_port,
options=[('grpc.max_message_length', maxMsgLength),
('grpc.max_send_message_length', maxMsgLength),
('grpc.max_receive_message_length', maxMsgLength)])


Edit:

Not a solution, but maybe gives a little bit more insight into the problem. For some reason, if I set the max message size to 1024 * 1024 * 1024 it ends up defaulting to 4194304 as the error message implies. Not really sure why that happens. But anyways, I tried reducing the max message size to 1024 * 1024 * 200 and it shows the correct max message size in the error message (209715200). It seems like there is a problem where grpc is not setting the max message size properly. Not sure how to get around this though.

The maximum number I can use where the error message shows the proper max value is 2^28. If I put a max message size of 2^29 it defaults to 4194304.

Continue reading...