[Python] Python Multiprocessing on Google Cloud Run

I am trying to use Pythons Native Multiprocessing Lib in my flask server. I don't really care about the concurrency aspect of Multiprocessing, I only care about the fact that using this will give the process its own dedicated memory space.

I was running into an issue where if 2 server calls run the worker function normally without any process, then I get an issue with the Node/JS engine and how these 2 different tasks share the same Node/JS process. So I switched to using Multiprocessing to Fork each task, that way it has its own memory and Node process which is what I want. But when I try this Multiprocessing on google cloud run it does not work. How can I fix this?

Is there a way to fork a task in the sense that each api call will get its own memory space, but not execute in the background. I think this background execution is whats causing the issue.

from flask import Flask, request, jsonify
from multiprocessing import Queue
import multiprocessing
from gen import *
import random
import os

app = Flask(__name__)

def calculate_worker():
try:
data = Calculate()

queue.put({
"success": True,
"data": data,
})
except Exception as e:
queue.put({
"success": False,
"error": str(e)
})

@app.route('/recalculate', methods=['GET', 'POST'])
def recalculate():
try:
// ...


# Create a queue for inter-process communication
result_queue = Queue()

# Create the process
process = multiprocessing.Process(
target=calculate_worker,
args=()
)

# Start the process
process.start()

# Wait for the process to complete
process.join(timeout=30)

# Check if process is still alive
if process.is_alive():
print("Calculate process timed out, terminating...")
process.terminate()
process.join()
return jsonify({"error": "Calculate function timed out"}), 500

# Check if process exited with error
if process.exitcode != 0:
return jsonify({"error": f"Calculate process failed with exit code {process.exitcode}"}), 500

# Get the result from the queue
if not result_queue.empty():
result = result_queue.get()

if result["success"]:
// ...

return jsonify(response_data), 200
else:
return jsonify({"error": result["error"]}), 500
else:
return jsonify({"error": "No result from Calculate function"}), 500

except Exception as e:
print("Exception caught in Recalculate: " + str(e))
return jsonify({"error": str(e)}), 500

Continue reading...