I have always wondered if there is a limit to the amount of data that i can store within a CSV file? I have set up my MVP to store data within a CSV file and currently the project grew to a very large scale and still CSV dependent. I'm working on getting someone on the team who would be able to handle database setup and facilitate the data transfer to a more robust method, but the current question is will be running into issues storing +100 MB of data in a CSV file? note that I did my best to optimize the way that I'm reading these files within my python code, which i still don't notice performance issues. Note 2, we are talking about the following scale: for 500 tracked equipment ~10,000 data points per column per day for 8 columns of different data If keep using the same file format of csv will cause me any performance issues Continue reading...
