I created a simple card game for 2 players with PettingZoo and agents are trained with Ray RLlib (PPO). Game rules are: 2 players, 32 cards (7 to ace) each trick both players play one card on table, higher card takes and that player plays next both players have 10 cards and after each trick they both draw one card, so it's 10 again, until there are no more cards to draw, then the rest of the cards are played without drawing (so, 16 tricks in total for each episode) you have to "follow the suit" (if opponent plays spades, you have too, if you have any) if you can't follow the suit, you can play any card, but then you never win a trick Agent is trained for 10+ hours now and he is playing fine, except one weird detail: He discards high card (aces and kings) in position where he can't win a trick (he didn't follow the suit). I was adjusting rewards and penalties and agent is still doing that unless I assign ridiculously high penalty to stop it. It works, but it doesn't feel right. I am giving him -35 penalty for discarding ace, and he gets only +1 for taking a trick. I have action mask on suit when he can follow the suit, but he can play any card when he can't follow. Why can't he learn that? File is long, so I am giving you observation space, I went through everything else zillion of times, everything debugged and everything seems fine. But it seems like agent can't understand that he can't take Q of diamonds with A of spades. (It was GPT suggestion to use 0-31 for cards and not 2 dimensional array) return spaces.Dict({ "hand": spaces.MultiBinary(32), "opponent_known_cards": spaces.MultiBinary(32), "talon_count": spaces.Discrete(13), "table_card": spaces.Discrete(33), "lead_suit": spaces.Discrete(5), "passed_cards": spaces.MultiBinary(32), "turn": spaces.Discrete(2), "action_mask": spaces.MultiBinary(32), }) Do you think it would be more clear for agent to have cards like 0, 1, 2, 3 suits and 0-7? So (0, 7) is A of spades etc... I'll appreciate all suggestions, because this is driving me nuts. Continue reading...
