I am conducting research on p-hacking, which requires accurately extracting tables from published academic papers. I have downloaded a large number of PDF files for this purpose. So far, I have tried several approaches to extract the tables. Unfortunately, Python packages like fitz and camelot are unable to directly extract the tables from these PDFs. As a workaround, I used a YOLO-based layout detection model to identify the locations of each table. While this approach allows me to extract the text content of the tables, I am losing the structural information (e.g., the alignment of rows and columns), and the OCR model I’m using is not always accurate. Could you suggest better methods or tools to improve the extraction process, ensuring both the content and structure of the tables are preserved? P.S. I apologize, but due to copyright restrictions, I am unable to upload the PDF files I am working on. import json import os import fitz def get_tables_loc(layout_json: dict) -> list: pdf_info = layout_json['pdf_info'] layout = {page: pdf_info[page]['tables'] for page in range(len(pdf_info)) if pdf_info[page]['tables']} tables_loc = [] for page in layout.keys(): for table in layout[page]: try: table_body = [block for block in table['blocks'] if block['type'] == 'table_body'] if not table['bbox'] or not table_body: continue tables_loc.append((page, table['bbox'])) except Exception as e: print(e) return tables_loc def extract_tables(path_paper): path_layout = os.path.join(path_paper, "layout.json") path_origin = os.path.join(path_paper, "origin.pdf") with open(path_layout, "r", encoding="utf-8") as f: layout_json = json.load(f) tables_loc = get_tables_loc(layout_json) doc = fitz.open(path_origin) for page, table_loc in tables_loc: rect = fitz.Rect(*table_loc) table_finder = doc[page].find_tables(clip=rect) if table_finder.tables: table_result = table_finder.tables[0].to_pandas() else: table_text = doc[page].get_text("text", clip=rect) Continue reading...
