[Python] Is there a way to break down a log file into columns using python

I am working on a school project in which I am supposed to clip unnecessary data from a log file and we were told to use python. First, I am supposed to divide the data of the log into columns and then proceed with everything else.

So this is the solution I came up with, though it's not quite working:

import pandas as pd
import pytz
from datetime import datetime
import re

def parse_str(x):
if x is None:
return '-'
else:
return x[1:-1]

def parse_datetime(x):
try:
dt = datetime.strptime(x[1:-7], '%d/%b/%Y:%H:%M:%S')
dt_tz = int(x[-6:-3])*60+int(x[-3:-1])
return dt.replace(tzinfo=pytz.FixedOffset(dt_tz))
except ValueError:
return datetime.now()

def parse_int(x):
return int(x) if x is not None else 0

data = pd.read_csv(
'Log_jeden_den.log',
sep=r'\s(?=(?:[^"]*"[^"]/")*[^"]*$)(?![^\\[]*\\])',
engine='python',
na_values='-',
header=None,
usecols=['ip', 'request', 'status', 'size', 'referer', 'user_agent'],
names=['ip', 'time', 'request', 'status', 'size', 'referer', 'user_agent'],
converters={'time': parse_datetime,
'request': parse_str,
'status': parse_int,
'size': parse_int,
'referer': parse_str,
'user_agent': parse_str})
print(data.head())


This is what I get:

Basically I need each part of the log to be split into the mentioned columns.

Log file looks like this:



I tried this:

usecols=[0, 3, 4, 5, 6, 7, 8]


But I am getting error:



I know I can just open the file in excel and split it into columns but I really would like to do it "proper" hard way. Is it possible, though? Thanks for any advice in advance.

Continue reading...