[Python] optional Newline at end of file messes up lark grammar parsing

I have a file contents. if the final line has a NL at the end, i get 2 token trees as expected, but of the file doesn't have a final NL, i end up with 3 trees which is unwanted.

How do i get lark to parse this correctly? I am at a loss as to why a NL at end of line affects the grouping at the beginning of line.

Correct Output, unwanted NL:


FILE = """
__ 95 95 36 __ 95 __ 95 __
__ __ 95 36 32 __ __ __ __
"""

Output will be:


Tree(Token('RULE', 'map'), ['__', '95', '95', '36', '__', '95', '__', '95', '__'])
Tree(Token('RULE', 'map'), ['__', '__', '95', '36', '32', '__', '__', '__', '__'])

Incorrect Output, no final NL:


FILE = """
__ 95 95 36 __ 95 __ 95 __
__ __ 95 36 32 __ __ __ __"""

Output will be:


Tree(Token('RULE', 'map'), ['__', '95', '95', '36', '__', '95', '__', '95', '__'])
Tree(Token('RULE', 'map'), ['__'])
Tree(Token('RULE', 'map'), ['__', '95', '36', '32', '__', '__', '__', '__'])

Code:


from lark import Lark, Transformer, Token, Discard

FILE = """
__ 95 95 36 __ 95 __ 95 __
__ __ 95 36 32 __ __ __ __
"""

grammar = """
start : NEWLINE? map+
map : [coord coord*] NEWLINE?
coord : HEX | FILL
HEX : ("A".."F" | DIGIT)+
FILL : "__"

%import common.DIGIT
%import common.NEWLINE
%import common.WS_INLINE
%ignore WS_INLINE
"""

class MyParser(Transformer):

def start(self, nodes: list) -> list:
return nodes

def NEWLINE(self, tree):
return Discard

def coord(self, tokens: list[Token]) -> str:
return tokens[0].value

def parse(text: str):
parser = Lark(grammar, start='start')
tree = parser.parse(text)
[print(node) for node in MyParser().transform(tree)]

parse(FILE)

Continue reading...