I need to build a script which reads a file containing news articles. The file is in JSON format and contains, and fields. The text of the header and body fields will not be 100% equal, so I need a tool to determine the probability of news items covering the same news events. Users of the application will provide a probability percentage and the application should then remove all records where the duplicate probability is higher than the percentage provided by the customer. I have used RapidFuzz and previously also SequenceMatcher within Python, but run into performance issues. I roughly need to process about 40K of records within a 30 minute period. Using RapidFuzz, I can easily work with 5K of records, and remove possible duplicates within about 3 minutes on an AWS r6idn.12xlarge instance. When running files with 20K of records, the system hangs inside the following process after the "print". def find_duplicates(records, threshold=0.1, max_workers=None, chunk_size=100): if max_workers is None: max_workers = multiprocessing.cpu_count() log(f"Using {max_workers} worker processes for deduplication.") all_chunks = list(chunked_combinations(records, chunk_size=chunk_size)) potential_duplicates = [] print("[DEBUG] Starting parallel deduplication loop") with ProcessPoolExecutor(max_workers=max_workers) as executor: futures = [executor.submit(process_chunk, chunk) for chunk in all_chunks] for future in tqdm(as_completed(futures), total=len(futures), desc="Parallel Deduplication"): chunk_result = future.result() # ← YOU NEED THIS potential_duplicates.extend(chunk_result) return potential_duplicates As I am worried that RapidFuzz will probably not be able to handle the workload within the required timeframe, I am wondering whether OpenSearch or ElasticSearch might provide functionality to provide duplicate probability scores (as percentage) for strings of text. Hereby also a sample of the records which are read. {"guid": "RSV6443Fa_2504222JR9D9JMtQ/SewCNtSVWekDTiWYxNlu0A7b2AI", "timestamps": [{"name": "recorded", "source": "AMER", "timestamp": "2025-04-22T06:27:19.381Z"}, {"name": "recorded", "source": "APAC", "timestamp": "2025-04-22T06:27:19.434Z"}, {"name": "recorded", "source": "EMEA", "timestamp": "2025-04-22T06:27:19.345Z"}], "data": {"altId": "nRSV6443Fa", "audiences": ["NP:LSEN"], "body": "For best results when printing this announcement, please click on link below:\nhttps://newsfile.refinitiv.com/getnewsfile/v1/story?guid=urn:newsml:reuters.com:20250422:nRSV6443Fa&default-theme=true\n\nRNS Number : 6443F Folio Investments, Inc. 22 April 2025\n\n \n\nFORM 8 (DD)\n\n \n\nPUBLIC DEALING DISCLOSURE BY A PARTY TO AN OFFER OR PERSON ACTING IN CONCERT\n(INCLUDING DEALINGS FOR THE ACCOUNT OF DISCRETIONARY INVESTMENT CLIENTS)\n\nRules 8.1, 8.2 and 8.4 of the Takeover Code (the "Code")\n\n \n\n1. KEY INFORMATION\n\n \n\n (a) Full name of discloser: FOLIO INVESTMENTS, INC.\n (b) Owner or controller of interests and short positions disclosed, if\n different from 1(a):\n\n The naming of nominee or vehicle companies is insufficient. For a\n trust, the trustee(s), settlor and beneficiaries must be named.\n (c) Name of offeror/offeree in relation to whose relevant securities this form QUALCOMM INCORPORATED\n relates:\n\n Use a separate form for each offeror/offeree\n (d) Status of person making the disclosure: Advisor to ALPHAWAVE IP GROUP PLC\n\n e.g. offeror, offeree, person acting in concert with the\n offeror/offeree (specify name of offeror/offeree)\n (e) Date dealing undertaken: 17 April 2025\n (f) In addition to the company in 1(c) above, is the discloser making NO\n disclosures in respect of any other party to the offer?\n\n If it is a cash offer or possible cash offer, state "N/A"\n\n \n\n2. POSITIONS OF THE PERSON MAKING THE DISCLOSURE\n\n \n\nIf there are positions or rights to subscribe to disclose in more than one\nclass of relevant securities of the offeror or offeree named in 1(c), copy\ntable 2(a) or (b) (as appropriate) for each additional class of relevant\nsecurity.\n\n \n\n(a) Interests and short positions in the relevant securities of\nthe offeror or offeree to which the disclosure relates following the dealing\n\n \n\n Class of relevant security: USD 0.0001 common\n\n Interests Short positions\n Number % Number %\n (1) Relevant securities owned and/or controlled: 5 0.00 0 0.00\n (2) Cash-settled derivatives: 0 0.00 0 0.00\n (3) Stock-settled derivatives (including options) and agreements to 0 0.00 0 0.00\n purchase/sell:\n 5 0.00 0 0.00\n\n TOTAL:\n\n \n\n \n\n \n\nAll interests and all short positions should be disclosed.\n\n \n\nDetails of any open stock-settled derivative positions (including traded\noptions), or agreements to purchase or sell relevant securities, should be\ngiven on a Supplemental Form 8 (Open Positions).\n\n \n\nDetails of any securities borrowing and lending positions or financial\ncollateral arrangements should be disclosed on a Supplemental Form 8 (SBL).\n\n \n\n(b) Rights to subscribe for new securities (including directors'\nand other employee options)\n\n \n\n Class of relevant security in relation to which subscription right exists:\n Details, including nature of the rights concerned and relevant percentages:\n\n \n\n \n\n3. DEALINGS BY THE PERSON MAKING THE DISCLOSURE\n\n \n\nWhere there have been dealings in more than one class of relevant securities\nof the offeror or offeree named in 1(c), copy table 3(a), (b), (c) or (d) (as\nappropriate) for each additional class of relevant security dealt in.\n\n \n\nThe currency of all prices and other monetary amounts should be stated.\n\n \n\n(a) Purchases and sales\n\n \n\n(i) Party to an offer or person acting in concert (except for\na principal trader in the same group as a connected advisor)\n\n \n\n Class of relevant security Purchase/sale Number of securities Price per unit\n\n \n\n(ii) Principal trader where the sole reason for the connection\nis that the principal trader is in the same group as a connected advisor\n\n \n\n Class of relevant security Purchases/ sales Total number of securities Highest price per unit paid/received Lowest price per unit paid/received\n\n Common Stock Purchase 1 136.0930 USD 137.8975 USD\n Common Stock Sale 2 136.0859 USD 137.8510 USD\n\n \n\n(b) Cash-settled derivative transactions\n\n \n\n Class of relevant security Product description Nature of dealing Number of reference securities Price per unit\n\n e.g. CFD e.g. opening/closing a long/short position, increasing/reducing a long/short\n position\n\n \n\n(c) Stock-settled derivative transactions (including options)\n\n \n\n(i) Writing, selling, purchasing or varying\n\n \n\n Class of relevant security Product description e.g. call option Writing, purchasing, selling, varying etc. Number of securities to which option relates Exercise price per unit Type Expiry date Option money paid/ received per unit\n\n e.g. American, European etc.\n\n \n\n(ii) Exercise\n\n \n\n Class of relevant security Product description Exercising / exercised against Number of securities Exercise price per unit\n\n e.g. call option\n\n \n\n(d) Other dealings (including subscribing for new securities)\n\n \n\n Class of relevant security Nature of dealing Details Price per unit (if applicable)\n\n e.g. subscription, conversion\n\n \n\n4. OTHER INFORMATION\n\n \n\n(a) Indemnity and other dealing arrangements\n\n \n\n Details of any indemnity or option arrangement, or any agreement or\n understanding, formal or informal, relating to relevant securities which may\n be an inducement to deal or refrain from dealing entered into by the party to\n the offer or person acting in concert making the disclosure and any other\n person:\n\n Irrevocable commitments and letters of intent should not be included. If there\n are no such agreements, arrangements or understandings, state "none"\n\n NONE\n\n \n\n(b) Agreements, arrangements or understandings relating to\noptions or derivatives\n\n \n\n Details of any agreement, arrangement or understanding, formal or informal,\n between the party to the offer or person acting in concert making the\n disclosure and any other person relating to:\n\n (i) the voting rights of any relevant securities under any option; or\n\n (ii) the voting rights or future acquisition or disposal of any relevant\n securities to which any derivative is referenced:\n\n If there are no such agreements, arrangements or understandings, state "none"\n\n NONE\n\n \n\n(c) Attachments\n\n \n\nAre any Supplemental Forms attached?\n\n \n\n Supplemental Form 8 (Open Positions) NO\n Supplemental Form 8 (SBL) NO\n\n \n\n \n\n Date of disclosure: 22 April 2025\n Contact name: Papa Lette and Andrzej Szyszka\n Telephone number: +33(1) 4212 1459 / +48(22) 317 4817\n\n \n\nPublic disclosures under Rule 8 of the Code must be made to a Regulatory\nInformation Service.\n\nThe Panel's Market Surveillance Unit is available for consultation in relation\nto the Code's disclosure requirements on +44 (0)20 7638 0129.\n\n \n\nThe Code can be viewed on the Panel's website at www.thetakeoverpanel.org.uk\n(http://www.thetakeoverpanel.org.uk) .\n\n \n\n \n\nThis information is provided by RNS, the news service of the London Stock Exchange. RNS is approved by the Financial Conduct Authority to act as a Primary Information Provider in the United Kingdom. Terms and conditions relating to the use and distribution of this information may apply. For further information, please contact\nrns@lseg.com (rns@lseg.com)\n or visit\nwww.rns.com (http://www.rns.com/)\n.\n\nRNS may use your IP address to confirm compliance with the terms and conditions, to analyse how you engage with the information contained in this communication, and to share such analysis on an anonymised basis with others as part of our commercial services. For further information about how RNS and the London Stock Exchange use the personal data you provide us, please see our\nPrivacy Policy (https://www.lseg.com/privacy-and-cookie-policy)\n. END DCCPKBBBABKKBQB\n", "firstCreated": "2025-04-22T06:27:19.000Z", "headline": "REG - Folio Investments Qualcomm Inc - Form 8 (DD) - Qualcomm Incorporated", "id": "RSV6443Fa_2504222JR9D9JMtQ/SewCNtSVWekDTiWYxNlu0A7b2AI", "instancesOf": [], "language": "en", "messageType": 2, "mimeType": "text/plain", "provider": "NS:LSE", "pubStatus": "stat:usable", "subjects": ["B:161", "B:162", "B:163", "B:164", "B:1740", "B:278", "B:279", "E:1", "E:J", "E:R", "G:3", "G:4", "G:6J", "G:7J", "G:9", "G:A", "G3", "M:1P2", "M:1QD", "M:2CM", "M:2CN", "M:2CP", "M:2CQ", "M:2CR", "M:2CU", "M:2CV", "M:2CY", "M:2DZ", "M:2ZB", "M:32", "M:3H", "M:Z", "R:QCOM.O", "N2:AMERS", "N2:BACT", "N2:CHIP", "N2:CHIP1", "N2:CMPNY", "N2EAL1", "N2:EMEA", "N2:EUROP", "N2:EUROPE", "N2:FCDCC", "N2:GB", "N2:INDU", "N2:LEN", "N2:MRG", "N2:NAMER", "N2:NEWR", "N2UBL", "N2:REG", "N2:SEMI", "N2:TECH", "N2:TECH08", "N2:TEEQ", "N2:TEEQ08", "N2:TMT", "N2:US", "N2:WEU", "P:4295907706"], "takeSequence": 1, "urgency": 3, "versionCreated": "2025-04-22T06:27:19.000Z"}}, Last comment is that news articles may also be in different languages and needs to be translated before matching occurs. Any support is appreciated. Many thanks - Werner. Continue reading...
