Context: Mining and Analyzing Real Estate Data: I am developing a data mining and analysis software program targeting housing units (i.e., flats) in a European capital. The goal is to monitor rental prices and to also uncover insights from a highly dimensional data set. Basically, the program crawls through a key website and creates a data base where each row represents a flat advertised for rent, with up to 29 variables recorded for each (e.g., Detailed geographic location, surface area, rental cost, amenities, energy score, etc.). So far, I have a DF of about 3,000 units. I frequently re-run the program to gather new units put on the market throughout the course of time. However, the platform I am targeting is an aggregator: many different agencies advertise their properties there. This induces a key issue, the presence of duplicates in the DF (i.e., same unique flat advertised across more than a single published ad). I need to remove them because they obviously distort my distribution. What I have done for this endeavor: 1. Computer-Assisted Human Verification: I created snippets in Python that assist me in A/B testing, where two flats sharing the approximate same Street, Price, and Surface Area are presented to me in a web browser, and the console asks me whether they point to the same and unique flat. I manually / visually inspect the flats and then I let the program know what my assessment is (Yes, No, Yes Unsure, No Unsure, Outdated URL for handling exceptions). The measure of uncertainty is for my own information, I could potentially task the program with contacting the ad posters to verify whether they are duplicates or not. 2. Post-Verification Duplicate Clustering: Based on my manual human verification, another program creates final duplicate clusters. Each cluster contains flats that are linked to each other by analyzing my answers (e.g., If A = B, and C equals A, then A, B, and C belong to the same cluster). 3. Training Data Final Duplicate Labeling: Since all flats have a unique ID across all my sub-DBs, I ultimately form a new DB where each flat has a binary variable 'Duplicate', where 1 means that the flat is a copy of another in the same data set, where 0 means that the flat does not have a copy anywhere in the data set. This makes for 30 variables per flat in the data set. 4. Tracking Similarity Patterns: I produce additional information from the updated data set with 'similarity recipes'. I combine different variables (e.g., Street+Price+Area, Street+Area+Floor, Street+Area+Type, Street+Area+Amenities) and I scan all the units to determine whether each has at least one other unit with same exact matching features (e.g., Same street, price, and area), as well as the number of exact matches across the data set. I do the same but with tolerance thresholds for Price and Surface Area (+-2%, +-5%). I also keep track of the counts of similar ads posted within 14 days of one another, and within 30 days of one another. 5. Random Forest Train/Test: Step 4 returns an enriched file with binary information and count information about all the units (e.g., Does unit A have strict or tolerant exact copy based on the first, second, third, etc., set of cumulative criteria? If so, how many?). I use this enriched information to perform train / test using Random Forest method (sklearn > RandomForestClassifier; permutation_importance; roc_auc_score, etc.). The data set is chronologically sorted, 80% of older ads are used to predict 20% of newer ads. The goal of the model is to determine whether Duplicate = 1 or 0. The results of the initial Random Forest model are: Key Model Metrics: ROC AUC: .996 PR AUC: .989 Top 4 out of Top 20 Features (Based on Mean Impurity Decrease): CNT_street_price_area: .1489 (i.e., number of flats with same street, price, and surface area) SIM_street_price_area: .1416 (i.e., whether a flat has a similar matching equivalent in the set) SIM_street_p2_area: .089 (i.e., whether a flat has a similar matching equivalent in the set with 2% tol. thresh. for price) CNT_street_p2_area: .089 (i.e., number of flats with similar matching equivalent in the set with 2% tol. thresh. for price) Top 4 out of Top 20 Features (Based on Drop in Precision-Recall AUC): CNT_street_price_area: .029 CNT_street_price_a2: .0254 SIM_street_price_area: .0195 CNT_street_p2_area: .0193 Treshold Sweep: threshold TP FP FN TN precision recall f1 positives predicted_positive 0.05 32 3 0 87 0.914286 1.000 0.955224 32 35 0.10 32 3 0 87 0.914286 1.000 0.955224 32 35 0.15 32 2 0 88 0.941176 1.000 0.969697 32 34 0.20 32 2 0 88 0.941176 1.000 0.969697 32 34 0.25 32 2 0 88 0.941176 1.000 0.969697 32 34 0.30 32 2 0 88 0.941176 1.000 0.969697 32 34 0.35 32 2 0 88 0.941176 1.000 0.969697 32 34 0.40 32 2 0 88 0.941176 1.000 0.969697 32 34 0.45 32 2 0 88 0.941176 1.000 0.969697 32 34 0.50 32 2 0 88 0.941176 1.000 0.969697 32 34 Metrics for Best Threshold (15%): threshold TP FP FN TN precision recall f1 positives predicted_positive 0.15 32 2 0 88 0.941176 1.000 0.969697 32 34 At the 0.15 threshold, the model achieves the best balance between precision and recall. It correctly identifies all 32 true duplicates (recall = 1.0) while only misclassifying 2 non-duplicates as duplicates (precision ≈ 94%). This suggests that the model is effective: it does not miss any actual duplicates and produces very few false alarms. The resulting F1 score of ~0.97 confirms that this threshold is the optimal operating point, giving both high accuracy and reliability in duplicate detection. WHERE I GET STUCK: In short, I used myself as a human training agent tasked with teaching a Random Forest model to accurately detect duplicates. I labelled my entire data set with Duplicate = 0 or 1, and then I split this data set (80/20) to test the performance of similarity, volume, and chronological patterns. The model returned a very satisfying performance. Where I get lost is: how do I implement this in my pipelines going forward?. So far, my Random Forest model worked because it was based on a manually labelled data set (after my human verification). This labeling allowed to track similarity patterns and feed the information to my RFM. But how do I transfer this knowledge to future unlabelled batches? In future batches, similarity patterns will be tracked but without actually knowing whether the units are or aren't duplicates. I think the answer may lie in saving this excellent model as a JOBLIB file to drive future predictions. For now, the code only outputs the tables I have partly parsed to this present post; it does not save the model for future re-use. This is not a process I am familiar with. Happy to receive any thoughts / comments / feedback as to how to proceed from where I am currently standing. Continue reading...
