[Python] Measuring resource usage by script started via `subprocess.run`

I'm building a service that allows users to execute their Python code and get execution result back (yay, sort of Lambda). My problem is that any given script may be malicious, and, albeit any network is turned off, these or those variations of rm -rf may take place.

As of now, service works as follows:

1. Get script sent by user,
2. Create RO k8s pod for user script execution (with wrapper script shown below) in manager,
3. Start execution of script in pod and wait till the end/timeout,
4. Return exec result (what's in stdout, stderr) + metrics to manager,
5. Process output in manager,
6. Delete pod (avg lifetime of pod is 1..7s),
7. Return processed output as JSON to user,

Allowed execution time for these scripts is limited to 5s / script, so I'm using timeout too. Additional CPU/RAM limitations also take place and are set for the k8s pod.

Based on limitations above, I chose to use subprocess.run with timeout param and got following script (in the very end) that measures params I'm interested in: time elapsed, CPU used, RAM used.

Problem is, I would like not only to execute script itself, but also show resources used more or less precisely. Simplest solution would be to write time.time() before and after subprocess.run + calculate difference, but the problem is that time for execution of subprocess.run itself would be inside => actual script execution time will be X% lower. Same applies to usage of resource lib.

Thus the question: How could I calculate time, memory and CPU resources spent exclusively by script executed in subprocess.run?

Current resource measurement script:

import time, resource, subprocess, sys, json

start = time.time()
timeout = 5 # 5 seconds exec timeout
exit_code = 0
try:
# /scripts/script.py - with actual user code, its resource usage I want to measure
result = subprocess.run(['python', '/scripts/script.py'],
timeout=timeout,
check=True,
capture_output=True,
text=True)
print(result.stdout, end='')
if result.stderr:
print(result.stderr, file=sys.stderr, end='')
except subprocess.TimeoutExpired:
exit_code = 124
except subprocess.CalledProcessError as e:
if e.stdout:
print(e.stdout, end='')
if e.stderr:
print(e.stderr, file=sys.stderr, end='')
exit_code = e.returncode
end = time.time()
elapsed = end - start

usage = resource.getrusage(resource.RUSAGE_CHILDREN)
cpu_time_used = usage.ru_utime + usage.ru_stime
cpu_usage_m = (cpu_time_used / elapsed) * 1000 if elapsed > 0 else 0
memory_used_mi = usage.ru_maxrss / 1024.0

metrics = {{
"execution_time": round(elapsed, 2), # in seconds
"cpu_usage": round(cpu_usage_m, 2), # in m
"memory_usage": round(memory_used_mi, 2), # in Mi
"exit_code": exit_code
}}
print("\\n###METRICS###")
print(json.dumps(metrics))
sys.exit(0)


Additional context:

I'm not a big fan of bash programming with crap like ${14+15} followed by endless wtf-moments, as well as adding extra busybox container and having another pack of wtf-moments, so python-only solution would be way better IMHO. Yes, I do know that measurement error rate for python-only solution will be not mere <= 1%, but in the ballpark of <= 10..15%, and I accept that.

Target OS: Linux

Continue reading...