I have to analyse 10 years worth of data and 50+ files for each year. I extracted the data from the internet and all I have done is extracted the text with regular expressions. The format for the files differ for each year and I'm not even sure that the pattern is consistent in the files of the individual years. The format for 2003 seems to be title (.*) [header(.*) color number number number string (\w+\s\d+\s\d+\s\d+\s.+) color number number number string color number number number string color number number number string]<==== one block header color number number number string color number number number string color number number number string color number number number string ........ my question is, is there a way to program something in python to identify patterns within the text files of a given year? A kind of pattern recognition, a program that outputs a regular expression that matches for one block of data perhaps. I will be using this data for linear algebra, but I want the data to be easily accessible and organized for other uses. Continue reading...
