I'm trying to build an SAC system to output variable speed limits. The environment is built with a framework based on METANET simulation model, where you can define roads with lanes, length, ramps, start speed, demands, and take steps for x time, etc. Separate files, so here's the repo: https://github.com/Anushka-N12/variable-speed-limits The files: network.py: Main file with network architecture agent.py: Initialises networks, defines learning logic sim_env.py: Sets up environment, step function runs for x time and outputs new speeds & sends back. train.py: Main file to run. Initialises agent & env, runs episodes - which includes taking actions & running steps. Learning once in x steps. These are how some of the outputs look like right now: Starting: Action chosen: 86.0 at step 0 Step 0: VSL 86.0, Reward -0.84, Action chosen: 84.9 at step 30 Step 50: VSL 84.9, Reward -0.63, Action chosen: 85.1 at step 60 Action chosen: 85.3 at step 90 Step 100: VSL 85.3, Reward -0.45, Action chosen: 84.7 at step 120 Action chosen: 85.5 at step 150 Step 150: VSL 85.5, Reward -0.71, Action chosen: 84.8 at step 180 Step 200: VSL 84.8, Reward -0.44, Action chosen: 84.4 at step 210 Action chosen: 85.5 at step 240 Step 250: VSL 85.5, Reward -0.64, Action chosen: 85.1 at step 270 End of first episode: Action chosen: 84.2 at step 840 Step 850: VSL 84.2, Reward -0.59, Action chosen: 84.4 at step 870 Avg critic grad norm: 0.28736076 Avg value grad norm: 0.38466153 Avg policy/actor grad norm: 0.00025897642 Ep 0: Train -516.5 | Eval -516.5 | VSL 84 Towards end of run: Action chosen: 84.4 at step 750 Step 750: VSL 84.4, Reward -0.48, Action chosen: 84.5 at step 780 Step 800: VSL 84.5, Reward -0.58, Action chosen: 85.4 at step 810 Action chosen: 83.5 at step 840 Step 850: VSL 83.5, Reward -0.59, Action chosen: 84.9 at step 870 Avg critic grad norm: 0.28735206 Avg value grad norm: 0.38460088 Avg policy/actor grad norm: 0.0002637541 Ep 10: Train -516.5 | Eval -516.5 | VSL 85 Basically, model isn't getting updated, plot of learning curve is fully flat. Speed output always around same value. I think maybe the policy gradiant is too small? This is my first reinforcement learning project, so I'm not sure what to do. SO far I tried Adam & SGD, ReLu & leaky relu. Added checks for NaNs. Removed reward scaling since that was resulting in -0.01 always. Please suggest what I can try to fix this. My deadline is near I HAVE to get this to work! PS. To run, folders sacn_cp & plots need to be present in dir. Continue reading...
