I used a generator as an argument to data for Content-Type application/json, and that worked fine. The problem now is that I need to do this for multipart/form-data, and it appears to not support a generator. I tried files={'content': my_generator_func()} and got: "a bytes-like object is required, not 'generator'"; so it looks like it would have accepted a file descriptor, but not a generator. I then tried a generator for the data argument, and set the files argument to a form name and value: files={'encoding': (None, 'base64')} [I read you have to use the files argument to get requests to do multipart/form-data] For that I get the error "Streamed bodies and files are mutually exclusive". Just in case, I tried it different ways setting a header for Content-Type multipart/form-data, both with and without it [I had read that setting it yourself can mess up requests.] If I omitted it, it complained it was missing; and if I included it, I got the errors previously specified. The MultipartEncoder helper, while it does accept a file descriptor, does not accept a generator. Is it possible to furnish a generator for multipart/form-data? If I have to, I will dump the stream into a file and pass a descriptor to the files argument; but seems that you would have furnished a way to use a generator since it is supported for Content-Type application/json PS The closest StackOverflow post on this topic is: How to send a "multipart/form-data" with requests in python? PPS I also tried this: https://toolbelt.readthedocs.io/en/latest/uploading-data.html#streaming-data-from-a-generator and the endpoint is not seeing the form data being sent. [I confirmed it works with curl using --form name=value arguments.] I was asked for a code example: def my_generator(): 'when iterated over, this will return "value_fragment1;value_fragment2;"' yield 'value_fragment1;' yield 'value_fragment2;' response=requests.post(url, # tried with+without this header... headers={'Content-Type': 'multipart/form-data'}, data={'field1': (None, 'value1'), 'field2': (None, 'value2') }, # the data argument is done the way the Stack post suggests for multipart... files={'field3', my_generator()} ) # this code generates the generator not expected error... And using the Streaming iterator: from requests_toolbelt.streaming_iterator import StreamingIterator # . # . # . # this is done just to get a length # for the content in this example; # in a real implementation we would calculate this... content='' for fragment in my_generator(): content+=fragment streamer=StreamingIterator(len(content), my_generator()) response=requests.post(url, # tried with+without this header... headers={'Content-Type': 'multipart/form-data'}, data={'field1': (None, 'value1'), 'field2': (None, 'value2') }, # the data argument is done the way the Stack post suggests for multipart... files={'field3', streamer} ) # the endpoint does not see the fields, # while it sees them for a curl --form field1=value1 --form field2=value2 ... ======================================= Tried this and plugged it into the MultipartEncoder: How do I read a tarfile from a generator? It is not as simple as that post thinks; you need to implement seeking functions in your BufferedReader/RawIOBase class: File "/usr/lib/python3/dist-packages/requests_toolbelt/multipart/encoder.py", line 573, in len return total_len(self.fd) - self.fd.tell() ^^^^^^^^^^^^^^ io.UnsupportedOperation: seek I also tried passing a StreamingIterator to a MultipartEncoder class (where you would pass a file descriptor...), but it hangs on the request. Looks like implementation of a generator for multipart/form-data was an oversight by the requests [toolbelt] developers... Continue reading...
