This is my input string: myserver:/etc/fail2ban # cat one_line.txt 12.123.123.12 - - [25/Jul/2017:09:26:40 +0200] "GET /a/path/file.html HTTP/1.1" 200 1450 "http://example.com/a/path/file.html" "Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/59.0.3071.115 Safari/537.36" I tried to evolve a regular expression beginning with ^<HOST>.*$ and ending with ^<HOST> - - \[2.*$, but when it came to the end something strange had happened. The last pattern iteration didn't match anymore. I can force a match when I state it like this ^<HOST> - - \[\D.*$. Häh?. That would mean that a non-digit char follows the opening square bracket! I print the steps I took in order to show you how it evolved. Hope someone can explain me this strange behaviour. Environment Info: myserver:/etc/fail2ban # fail2ban-regex --version fail2ban-regex 0.9.4 myserver:/etc/fail2ban # python --version Python 2.7.17 And now follows the iterative evolving of my filter expression: Iteration 1: Match - as expected myserver:/etc/fail2ban # fail2ban-regex -l heavydebug one_line.txt '^<HOST>.*$' Running tests ============= D: Setting usedns = warn for Filter(None) D: Created Filter(None) Use failregex line : ^<HOST>.*$ Use log file : one_line.txt Use encoding : UTF-8 D: Working on line u'12.123.123.12 - - [25/Jul/2017:09:26:40 +0200] "GET /a/path/file.html HTTP/1.1" 200 1450 "http://example.com/a/path/file.html" "Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/59.0.3071.115 Safari/537.36"' D: Matched time template Day(?P<_sep>[-/])MON(?P=_sep)Year[ :]?24hour:Minute:Second(?:\.Microseconds)?(?: Zone offset)? D: Got time 1500967600.000000 for "u'25/Jul/2017:09:26:40 +0200'" using template Day(?P<_sep>[-/])MON(?P=_sep)Year[ :]?24hour:Minute:Second(?:\.Microseconds)?(?: Zone offset)? D: Looking for failregex match of [(u'12.123.123.12 - - [', u'25/Jul/2017:09:26:40 +0200', u'] "GET /a/path/file.html HTTP/1.1" 200 1450 "http://example.com/a/path/file.html" "Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/59.0.3071.115 Safari/537.36"')] D: Matched FailRegex('^(?:::f{4,6}?(?P<host>[\\w\\-.^_]*\\w).*$') D: Sorting the template list D: Winning template: Day(?P<_sep>[-/])MON(?P=_sep)Year[ :]?24hour:Minute:Second(?:\.Microseconds)?(?: Zone offset)? with 1 hits Results ======= Failregex: 1 total |- #) [# of hits] regular expression | 1) [1] ^<HOST>.*$ `- Ignoreregex: 0 total Date template hits: |- [# of hits] date format | [1] Day(?P<_sep>[-/])MON(?P=_sep)Year[ :]?24hour:Minute:Second(?:\.Microseconds)?(?: Zone offset)? `- Lines: 1 lines, 0 ignored, 1 matched, 0 missed [processed in 0.00 sec] Iteration 2: Match - as expected myserver:/etc/fail2ban # fail2ban-regex -l heavydebug one_line.txt '^<HOST> - - .*$' Running tests ============= D: Setting usedns = warn for Filter(None) D: Created Filter(None) Use failregex line : ^<HOST> - - .*$ Use log file : one_line.txt Use encoding : UTF-8 D: Working on line u'12.123.123.12 - - [25/Jul/2017:09:26:40 +0200] "GET /a/path/file.html HTTP/1.1" 200 1450 "http://example.com/a/path/file.html" "Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/59.0.3071.115 Safari/537.36"' D: Matched time template Day(?P<_sep>[-/])MON(?P=_sep)Year[ :]?24hour:Minute:Second(?:\.Microseconds)?(?: Zone offset)? D: Got time 1500967600.000000 for "u'25/Jul/2017:09:26:40 +0200'" using template Day(?P<_sep>[-/])MON(?P=_sep)Year[ :]?24hour:Minute:Second(?:\.Microseconds)?(?: Zone offset)? D: Looking for failregex match of [(u'12.123.123.12 - - [', u'25/Jul/2017:09:26:40 +0200', u'] "GET /a/path/file.html HTTP/1.1" 200 1450 "http://example.com/a/path/file.html" "Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/59.0.3071.115 Safari/537.36"')] D: Matched FailRegex('^(?:::f{4,6}?(?P<host>[\\w\\-.^_]*\\w) - - .*$') D: Sorting the template list D: Winning template: Day(?P<_sep>[-/])MON(?P=_sep)Year[ :]?24hour:Minute:Second(?:\.Microseconds)?(?: Zone offset)? with 1 hits Results ======= Failregex: 1 total |- #) [# of hits] regular expression | 1) [1] ^<HOST> - - .*$ `- Ignoreregex: 0 total Date template hits: |- [# of hits] date format | [1] Day(?P<_sep>[-/])MON(?P=_sep)Year[ :]?24hour:Minute:Second(?:\.Microseconds)?(?: Zone offset)? `- Lines: 1 lines, 0 ignored, 1 matched, 0 missed [processed in 0.00 sec] Iteration 3: Match - as expected myserver:/etc/fail2ban # fail2ban-regex -l heavydebug one_line.txt '^<HOST> - - \[.*$' Running tests ============= D: Setting usedns = warn for Filter(None) D: Created Filter(None) Use failregex line : ^<HOST> - - \[.*$ Use log file : one_line.txt Use encoding : UTF-8 D: Working on line u'12.123.123.12 - - [25/Jul/2017:09:26:40 +0200] "GET /a/path/file.html HTTP/1.1" 200 1450 "http://example.com/a/path/file.html" "Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/59.0.3071.115 Safari/537.36"' D: Matched time template Day(?P<_sep>[-/])MON(?P=_sep)Year[ :]?24hour:Minute:Second(?:\.Microseconds)?(?: Zone offset)? D: Got time 1500967600.000000 for "u'25/Jul/2017:09:26:40 +0200'" using template Day(?P<_sep>[-/])MON(?P=_sep)Year[ :]?24hour:Minute:Second(?:\.Microseconds)?(?: Zone offset)? D: Looking for failregex match of [(u'12.123.123.12 - - [', u'25/Jul/2017:09:26:40 +0200', u'] "GET /a/path/file.html HTTP/1.1" 200 1450 "http://example.com/a/path/file.html" "Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/59.0.3071.115 Safari/537.36"')] D: Matched FailRegex('^(?:::f{4,6}?(?P<host>[\\w\\-.^_]*\\w) - - \\[.*$') D: Sorting the template list D: Winning template: Day(?P<_sep>[-/])MON(?P=_sep)Year[ :]?24hour:Minute:Second(?:\.Microseconds)?(?: Zone offset)? with 1 hits Results ======= Failregex: 1 total |- #) [# of hits] regular expression | 1) [1] ^<HOST> - - \[.*$ `- Ignoreregex: 0 total Date template hits: |- [# of hits] date format | [1] Day(?P<_sep>[-/])MON(?P=_sep)Year[ :]?24hour:Minute:Second(?:\.Microseconds)?(?: Zone offset)? `- Lines: 1 lines, 0 ignored, 1 matched, 0 missed [processed in 0.00 sec] Itertaion 4.1: Fail - Why ??? myserver:/etc/fail2ban # fail2ban-regex -l heavydebug one_line.txt '^<HOST> - - \[2.*$' Running tests ============= D: Setting usedns = warn for Filter(None) D: Created Filter(None) Use failregex line : ^<HOST> - - \[2.*$ Use log file : one_line.txt Use encoding : UTF-8 D: Working on line u'12.123.123.12 - - [25/Jul/2017:09:26:40 +0200] "GET /a/path/file.html HTTP/1.1" 200 1450 "http://example.com/a/path/file.html" "Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/59.0.3071.115 Safari/537.36"' D: Matched time template Day(?P<_sep>[-/])MON(?P=_sep)Year[ :]?24hour:Minute:Second(?:\.Microseconds)?(?: Zone offset)? D: Got time 1500967600.000000 for "u'25/Jul/2017:09:26:40 +0200'" using template Day(?P<_sep>[-/])MON(?P=_sep)Year[ :]?24hour:Minute:Second(?:\.Microseconds)?(?: Zone offset)? D: Looking for failregex match of [(u'12.123.123.12 - - [', u'25/Jul/2017:09:26:40 +0200', u'] "GET /a/path/file.html HTTP/1.1" 200 1450 "http://example.com/a/path/file.html" "Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/59.0.3071.115 Safari/537.36"')] D: Sorting the template list D: Winning template: Day(?P<_sep>[-/])MON(?P=_sep)Year[ :]?24hour:Minute:Second(?:\.Microseconds)?(?: Zone offset)? with 1 hits Results ======= Failregex: 0 total Ignoreregex: 0 total Date template hits: |- [# of hits] date format | [1] Day(?P<_sep>[-/])MON(?P=_sep)Year[ :]?24hour:Minute:Second(?:\.Microseconds)?(?: Zone offset)? `- Lines: 1 lines, 0 ignored, 0 matched, 1 missed [processed in 0.00 sec] |- Missed line(s): | 12.123.123.12 - - [25/Jul/2017:09:26:40 +0200] "GET /a/path/file.html HTTP/1.1" 200 1450 "http://example.com/a/path/file.html" "Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/59.0.3071.115 Safari/537.36" `- Itertaion 4.2: Fail - Why ??? myserver:/etc/fail2ban # fail2ban-regex -l heavydebug one_line.txt '^<HOST> - - \[\d.*$' Running tests ============= D: Setting usedns = warn for Filter(None) D: Created Filter(None) Use failregex line : ^<HOST> - - \[\d.*$ Use log file : one_line.txt Use encoding : UTF-8 D: Working on line u'12.123.123.12 - - [25/Jul/2017:09:26:40 +0200] "GET /a/path/file.html HTTP/1.1" 200 1450 "http://example.com/a/path/file.html" "Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/59.0.3071.115 Safari/537.36"' D: Matched time template Day(?P<_sep>[-/])MON(?P=_sep)Year[ :]?24hour:Minute:Second(?:\.Microseconds)?(?: Zone offset)? D: Got time 1500967600.000000 for "u'25/Jul/2017:09:26:40 +0200'" using template Day(?P<_sep>[-/])MON(?P=_sep)Year[ :]?24hour:Minute:Second(?:\.Microseconds)?(?: Zone offset)? D: Looking for failregex match of [(u'12.123.123.12 - - [', u'25/Jul/2017:09:26:40 +0200', u'] "GET /a/path/file.html HTTP/1.1" 200 1450 "http://example.com/a/path/file.html" "Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/59.0.3071.115 Safari/537.36"')] D: Sorting the template list D: Winning template: Day(?P<_sep>[-/])MON(?P=_sep)Year[ :]?24hour:Minute:Second(?:\.Microseconds)?(?: Zone offset)? with 1 hits Results ======= Failregex: 0 total Ignoreregex: 0 total Date template hits: |- [# of hits] date format | [1] Day(?P<_sep>[-/])MON(?P=_sep)Year[ :]?24hour:Minute:Second(?:\.Microseconds)?(?: Zone offset)? `- Lines: 1 lines, 0 ignored, 0 matched, 1 missed [processed in 0.00 sec] |- Missed line(s): | 12.123.123.12 - - [25/Jul/2017:09:26:40 +0200] "GET /a/path/file.html HTTP/1.1" 200 1450 "http://example.com/a/path/file.html" "Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/59.0.3071.115 Safari/537.36" `- Itertaion 4.3: Match - Why ??? myserver:/etc/fail2ban # fail2ban-regex -l heavydebug one_line.txt '^<HOST> - - \[\D.*$' Running tests ============= D: Setting usedns = warn for Filter(None) D: Created Filter(None) Use failregex line : ^<HOST> - - \[\D.*$ Use log file : one_line.txt Use encoding : UTF-8 D: Working on line u'12.123.123.12 - - [25/Jul/2017:09:26:40 +0200] "GET /a/path/file.html HTTP/1.1" 200 1450 "http://example.com/a/path/file.html" "Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/59.0.3071.115 Safari/537.36"' D: Matched time template Day(?P<_sep>[-/])MON(?P=_sep)Year[ :]?24hour:Minute:Second(?:\.Microseconds)?(?: Zone offset)? D: Got time 1500967600.000000 for "u'25/Jul/2017:09:26:40 +0200'" using template Day(?P<_sep>[-/])MON(?P=_sep)Year[ :]?24hour:Minute:Second(?:\.Microseconds)?(?: Zone offset)? D: Looking for failregex match of [(u'12.123.123.12 - - [', u'25/Jul/2017:09:26:40 +0200', u'] "GET /a/path/file.html HTTP/1.1" 200 1450 "http://example.com/a/path/file.html" "Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/59.0.3071.115 Safari/537.36"')] D: Matched FailRegex('^(?:::f{4,6}?(?P<host>[\\w\\-.^_]*\\w) - - \\[\\D.*$') D: Sorting the template list D: Winning template: Day(?P<_sep>[-/])MON(?P=_sep)Year[ :]?24hour:Minute:Second(?:\.Microseconds)?(?: Zone offset)? with 1 hits Results ======= Failregex: 1 total |- #) [# of hits] regular expression | 1) [1] ^<HOST> - - \[\D.*$ `- Ignoreregex: 0 total Date template hits: |- [# of hits] date format | [1] Day(?P<_sep>[-/])MON(?P=_sep)Year[ :]?24hour:Minute:Second(?:\.Microseconds)?(?: Zone offset)? `- Lines: 1 lines, 0 ignored, 1 matched, 0 missed [processed in 0.00 sec] Thx in advance. Continue reading...
