[Python] astor, ast, astunparse do not preserve lineno, astmonkey puts \ (backslash) everywhere

Here is the output run on a very simple urls.py from a test Django project:

from django.contrib import admin
from django.urls import path

urlpatterns = [\
path('test_path1/', views.test_view1, name='test_name1'), \
path('test_path2/', views.test_view1, name='test_name1'), path('admin/', admin.site.test1.test2.urls, name='admin')]


See the last few lines. Astmonkey almost gets it right, but put in \ everywhere!

Here's my code:

from ast import (Name, Call, Load, Attribute, Constant, Expr,
NodeTransformer, parse, unparse, dump, keyword, fix_missing_locations)
from base_injector import BaseInjector
from astmonkey import visitors
import astor

class UrlEntry:
def __init__(self, path, view, name):
self._path = path
self._view = view
self._name = name

@property
def path(self):
return self._path

@property
def view(self):
return self._view

@property
def name(self):
return self._name


class DjangoUrlInjector(BaseInjector):
def __init__(self, filename, urls, index=0):
self._filename = filename,
self._urls = urls
self._index = index
super().__init__()

def visit_Assign(self, node):
target = node.targets[0]

if target.id == 'urlpatterns':
patterns = node.value
index = self._index
index %= len(patterns.elts)

# Determine lineno of existing urlpatterns
if len(patterns.elts) == 0:
line_number = node.lineno + 1
else:
line_number = patterns.elts[index].lineno

for url in self._urls:
path_call = Call(func=Name(id='path', ctx=Load()),
args=[Constant(value=url.path),
self._dot_str_to_attrib(url.view)],
keywords=[keyword(arg='name', value=Constant(value=url.name))])
path_call.lineno = line_number
line_number += 1
patterns.elts.insert(index, path_call)
index += 1

# Adjust remaining line number past where we've inserted:
if index < len(patterns.elts[index:]):
for existing in patterns.elts:
existing.lineno = line_number
line_number += 1

node = self.generic_visit(node)
return node



if __name__ == '__main__':
urls_filename = 'test/django_project/django_project/urls.py'

url_injector = DjangoUrlInjector(
urls_filename,
[UrlEntry('test_path1/', 'views.test_view1', 'test_name1'),
UrlEntry('test_path2/', 'views.test_view1', 'test_name1')]
)

import os
os.chdir('../..')

ast_tree = astor.parsefile(urls_filename)

ast_tree = url_injector.visit(ast_tree)
result_code = visitors.to_source(ast_tree)

print(result_code)

`


Why is it so difficult to preserve line numbers and or inject some code. I thought knowing the AST would be beneficial, but it's not. It's as if I'd get further using regexes...

What I've tried is all combinations of ast, astor, astunparse, astmonkey.

What I'm expecting is that lineno's mean something. To the first 3 they don't. Then for the last one, astmonkey, it decides to randomly put in backslashes.

What I'm looking for is a code fix of ast / astor that preserves lineno's. Or a fix of astmonkey that gets rid of backslashes. OR, an alternative library that actually does what it advertises.

Continue reading...