206 Data Cleaner
Derceto's Aquadapt software runs on-line and in real-time. It is therefore totally dependent on precise and reliable input data to allow it to produce high quality outputs. A key component of an Aquadapt installation is the Data Cleaner which tests, validates and, if necessary, corrects incoming data.
Reservoir levels, pump status and current flow readings are retrieved from the client's SCADA system at regular intervals and saved in the Aquadapt database.
Invariably SCADA systems are geographically widely distributed, with the inherent need to have long distance communication paths between field instruments and the operator. Faulty instruments, communication loss or faults within the SCADA system can lead to incorrect or missing data.
The Data Cleaner takes the SCADA values and using cross-validation techniques detects errors and omissions and then uses predicted values to correct the data set. An operator alarm is raised if the SCADA data remains invalid for an extended period of time.
Data Cleaner performs the following basic steps:
- Confirms reservoir levels are increasing or decreasing as expected
- Confirms pump flows from pump status and flow data
- Calculates last half hour's water demand for each pressure zone
- Checks explicitly for signal spikes, drop-outs and flat lines
- Error Detect and Correct SCADA Values
The Data Cleaner uses rules based on standard hydraulic system behavior to decide which values to keep and which to discard. Typically, a reservoir level that has flat-lined for a certain period of time would be replaced with the value predicted by Aquadapt software. Sudden level changes that cannot be supported by pump flows or water demand are detected and corrected. Where possible data is cross-validated: for example when comparing flow meter values with pump run statuses, inconsistent information can indicate a fault. The most common data errors are a result of instrument fault or short-term communications loss. Instruments can often be set to "fail low" or "fail high". This means that the instrument is programmed to give either a minimum or maximum signal when it detects a sensor or internal problem. These errors are relatively easy to detect and are immediately replaced with Aquadapt software's predicted values from its previous solution. Short-term communications loss tends to produce flat-lines. These are values that initially appear valid but haven't changed, even slightly, over time. This is a common problem and difficult for operators to detect over short time frames. The Data Cleaner can detect flat-lines very quickly and will again substitute predicted values to enable continued operation. Prolonged flat-lines also generate operator alarms. Depending on system conditions Aquadapt software can use cleaned data for many hours following communications loss. When communications are restored the Data Cleaner resynchronizes with the live data.
Water demand calculations require a combination of pump station flows and storage tank levels to be used together to determine pressure zone demand. These calculations are very important as water demand is one of the most unpredictable inputs for the Aquadapt Solver. To improve accuracy the Data Cleaner uses accumulated values from the flow meters rather than instantaneous flow signals. The use of accumulators inside the flow meter itself or the local RTU makes the Data Cleaner relatively resilient to short-term communication loss.
Every SCADA system has its own unique data cleaning requirements. Through numerous implementations the Aquadapt system has built a vast knowledge base for the Data Cleaner enabling it to handle a wide variety of data corruption issues. This experience has produced a reliable and robust method of dealing with incoming data in real-time, making it possible for the Aquadapt Solver to continually make high quality decisions using imperfect information.
