I want to read a text file with a python script and determine a best guess as to its character-encoding using the chardetng library. I'm using the chardetng.py binding to the chardetng Rust library. I've created the following python script: # Import sys module to allow processing of command-line arguments. import sys # Import chardetng-py detect module. from chardetng_py import detect # Define a function to open file, read file, and analyze character encoding. def detect_encoding (fileName): try: with open (fileName, 'rb') as file: # Read file. file_data = file.read () # Show how many bytes were read. print (file.tell (), "bytes read from file:", fileName) # Analyze file. result = detect (file_data) # Close input file. file.close() return result except FileNotFoundError as e: print (f"Error: File not found: {e}") sys.exit (1) except IOError as e: print (f"Error reading file: {e}") sys.exit (1) def main (): if len (sys.argv) != 2: print ("Usage: python3 charenc2.py <filename>") sys.exit (1) file_name = sys.argv[1] encoding = detect_encoding (file_name) # Show results. print (f"The character encoding is most likely: {encoding}") if __name__ == "__main__": main () This almost always results in 'windows-1252', whereas chardet and chardetect (which came with Ubuntu linux) get it right. I've tried on test files from: https://github.com/stain/encoding-test-files as well as some old subtitle files. Out of the test files above, it only detects KOI8-R. All others are predicted to be windows-1252, even the utf8.txt file. These are small files, so perhaps chardetng needs larger amounts of text than chardet and chardetect? Am I miss-using chardetng-py, or is there perhaps a threshold setting that needs adjustment? Any advice? Thanks Continue reading...
