[Python] Replicate mouse movement in a 3D FPS game using Python?

I'm trying make a neural network based AI for Jedi Academy via behavioral cloning (learning to imitate pros). For the behavioral cloning to be faithful it is crucial for one to be able to reproduce mouse movements. That is, if the data says rotate the character 45 degrees to the left then the AI should be able to reproduce this when presented with the same situation. Here is a video I made years ago that shows what I'm trying to achieve: https://www.youtube.com/watch?v=YEvQAWRixc8 . Sadly I did not save the code for this particular demonstration.

What I'm finding is many different methods of both recording and moving the mouse, but none of them are consistent with one another such that the recorded mouse movement is replayed exactly like in the video.

I tried a lot of of the methods. Here's one where the timing isn't implemented and it jams the mouse in game (another issue I'm trying to understand).


from pynput import mouse
import time
import json
import win32gui
import pydirectinput
import ctypes
import pynput
from ctypes import windll, Structure, c_long, byref


class POINT(Structure):
_fields_ = [("x", c_long), ("y", c_long)]
SendInput = ctypes.windll.user32.SendInput
hwnd = win32gui.FindWindow(None, 'EternalJK')
win32gui.SetForegroundWindow(hwnd)

recorded_data = []
previous_time = None
previous_x, previous_y = None, None

def MousePosition_ctypes():
pos = POINT()
windll.user32.GetCursorPos(byref(pos))
return pos.x, pos.y

print("Recording..")
t0 = time.time()
t = time.time()
mouse_positions = []
while t < t0 + 3:
x, y = MousePosition_ctypes()
mouse_positions.append((x, y, t))
t = time.time()

print("Playing..")


def set_pos(x, y):
x = 1 + int(x * 65536./2048.)
y = 1 + int(y * 65536./1536.)
extra = ctypes.c_ulong(0)
ii_ = pynput._util.win32.INPUT_union()
ii_.mi = pynput._util.win32.MOUSEINPUT(x, y, 0, (0x0001), 0, ctypes.cast(ctypes.pointer(extra), ctypes.c_void_p))
command=pynput._util.win32.INPUT(ctypes.c_ulong(0), ii_)
SendInput(1, ctypes.pointer(command), ctypes.sizeof(command))

t0 = time.time()
cumulative_time = 0
for delta_x, delta_y, time_delta in mouse_positions:
# time.sleep(time_delta)
set_pos(delta_x, delta_y)

Continue reading...