I am attempting to build an API that will run a Scrapy web spider when requested via a websocket message. I would like to forward the logging output to the websocket client so you see what's going on in the - sometimes quite long-running - process. When finished, I will also send the scraped results. As it is possible to run Scrapy in-process, I would like to do exactly that. I found a solution that will stream an external process to a websocket here, but that doesn't seem right if it's possible to run Scrapy inside the server. https://tomforb.es/displaying-a-processes-output-on-a-web-page-with-websockets-and-python There are two ways I can imagine to make this work in Twisted: Somehow using a LogObserver, or defining a LogHandler (probably StreamHandler with StringIO) and then handle the Stream in some way in Twisted with autobahn.websocket classes like WebSocketServerProtocol. Now I am quite stuck and don't know how to connect the ends. Could someone please provide a short example how to stream logging output from twisted logging (avoiding a file if possible) to a websocket client? Continue reading...
