[Python] .csv -> .db taking WAY too long

Discussão em 'Python' iniciado por Stack, Abril 28, 2025.

  1. Stack

    Stack Membro Participativo

    My code (python) works, but at the rate it's running it'll take 72 hours to finish. I'm trying to turn a csv file into a database file in hopes that it'll be easier to use and access the data in it (before i'd spend 2 minutes waiting for python to load the csv into a list every time I ran any code using it).

    Is there any way I can make this code run faster?

    from datetime import date

    #works = "Out_Files\works-20210226.csv"
    #tags = "Out_Files\tags-20210226.csv"

    with sqlite3.connect("2025\works-20210226.db") as con:
    cur = con.cursor()
    cmd = "CREATE TABLE works (creation_date INT, language STRING, restricted INT, complete INT, word_count INT, tags STRING)"
    cur.execute(cmd)
    con.commit()
    with open("Out_Files\works-20210226.csv", 'r', encoding='utf-8') as f:
    i = 0
    next = f.readline()
    next = f.readline()
    while next:
    thedate,language,restricted,complete,word_count,tags = next.split(",")
    thedate = [int(each) for each in thedate.split("-")]
    thedate = date(thedate[0], thedate[1], thedate[2]).toordinal()
    restricted = int(restricted == "true")
    complete = int(complete == 'true')
    word_count = int(word_count) if word_count else 0

    cmd = f"INSERT INTO works VALUES ({thedate}, '{language}', {restricted}, {complete}, {word_count}, '{tags}')"
    cur.execute(cmd)
    con.commit()

    next = f.readline()
    i += 1
    print(i)

    cmd = "SELECT * FROM works WHEN word_count = 1836"
    print(cur.execute(cmd))


    I'm currently at i = 733,590 out of 7,269,695

    Continue reading...

Compartilhe esta Página