[Python] tokenize sentence into words python

Discussão em 'Python' iniciado por Stack, Abril 14, 2025.

  1. Stack

    Stack Membro Participativo

    I want to extract information from different sentences, so I'm using nltk to divide each sentence into words. I'm using this code:

    words=[]
    for i in range(len(sentences)):
    words.append(nltk.word_tokenize(sentences))
    words


    It works pretty good, but I want something a little bit different than that. For example, I have this sentence:

    '[\'Jan 31 19:28:14 nginx: 10.0.0.0 - - [31/Jan/2019:19:28:14 +0100] "POST /test/itf/ HTTP/x.x" 404 146 "-" "Mozilla/5.2 [en] (X11, U; OpenVAS-XX 9.2.7)"\']'


    I want "Mozilla/5.2 [en] (X11, U; OpenVAS-XX 9.2.7)" to be one word and not divided into several single words .

    UPDATE: I want something like that:

    [
    'Jan',
    '31',
    '19:28:14',
    'nginx',
    '10.0.0.0',
    '31/Jan/2019:19:28:14',
    '+0100',
    'POST',
    '/test/itf/',
    'HTTP/x.x',
    '404',
    '146',
    'Mozilla/5.2 [en] (X11, U; OpenVAS-XX 9.2.7)']


    Any idea how to make it possible?

    Continue reading...

Compartilhe esta Página