[Python] How can I serve ML models quickly and with a low latency

Assume a user connects via a Websocket connection to a server, which serves a personalized typescript function based on a personalized JSON file

So when a user connects,

• the personalized JSON file is loaded from an S3-lile bucket (around 60-100 MB per user)
• and when he types a Typescript/JavaScript/Python code is executed which returns some string a reply and the JSON-like data structure gets updates
• when the user disconnects the JSON gets persisted back to the S3-like bucket.

In total, you can think about 10,000 users, so 600 GB in total.

It should

• spin up fast for a user,
• should be very scalable given the number of users (such that we do not waste money) and
• have a global latency of a few tens of ms.

Is that possible? If so, what architecture seems to be the most fitting?

Continue reading...