[Python] What is the best preprocessing strategy for a highly inconsistent, multi-site...

I'm working on a project to predict the remaining backup time of batteries in a large network of remote sites using a Random Survival Forest (RSF) model. I have a solid plan for generating my target variables (duration and event_observed) from historical performance logs.

My main challenge is building a robust pipeline to process the highly inconsistent predictor features, which come from daily time-series logs from each site's Battery Management System (BMS).

The Data Challenge: Extreme Inconsistency The dataset contains data from hundreds of sites, and the quality is very inconsistent:

Varying Data Availability: Some columns are populated for one site but completely null for another.

Different Active Hardware: The "active" battery bank is not the same across all sites. Site A might use currentbank1, while Site B uses currentbank2, leaving the other bank columns empty or with zero values.

Logical & Physical Faults: Some sites report data that is logically or physically impossible. For example, a remainingcapacity value that is stuck at 100% even when the battery is clearly discharging, or a site that reports voltage but has a failed current sensor (always null or zero).

My Proposed Strategies I've developed two competing strategies to handle this, but I'm unsure which is more robust or if there's a better way.

Approach A: The "Per-Site, Bottom-Up" Analysis In this approach, I would first perform a deep analysis of each site individually to understand its unique data profile.

For each site's daily data, run a diagnostic script.

Classify the site into an archetype (e.g., HEALTHY_SINGLE_BANK, FAULT_VOLTAGE_ONLY, DEGRADED_DATA).

For HEALTHY sites, run logical consistency checks on their available data (e.g., "does voltage decrease when current is negative?").

The Conundrum: After this process, I'm left with a set of validated sites, but they all have different active columns and data profiles. It's not clear what the best way is to combine these heterogeneous, validated datasets into a single, cohesive training set for one universal model.

Approach B: The "Golden Features, Top-Down" Filter This approach prioritizes a universal feature set from the start.

Define a "golden set" of features that are theoretically essential for the prediction (e.g., active_current, active_voltage, cell_voltage_delta, ambient_temperature).

For each site's daily data, the pipeline would only check the availability and logical consistency of this pre-defined golden set.

If a site's data fails the validation for any of these critical features, all of its data for that day is dropped.

The Conundrum: This seems efficient and guarantees a standardized final dataset. However, I'm concerned it might be too restrictive, causing me to discard a large number of sites that might have partially complete but still useful data.

My Questions: Which of these strategies is considered a better practice for this kind of messy, real-world IoT data?

What are the main pros and cons of each? Is Approach A too complex to scale effectively? Is Approach B too rigid and likely to discard too much potentially valuable data?

Is there a hybrid approach I'm missing that combines the detailed diagnosis of A with the standardization goal of B?

Here is a conceptual example of my data to illustrate the problem:

siteid voltagebank1 currentbank1 voltagebank2 currentbank2 ... Site_A 52.5V -10.1A (null) (null) ... Site_B 52.4V 0.0A 52.4V -12.5A ...

Any advice on industry best practices, potential pitfalls, or alternative strategies would be greatly appreciated. Thank you!

Continue reading...