WSL/SLF GitLab Repository

Commit 715a7f9b authored by Mathias Bavay's avatar Mathias Bavay
Browse files

The min and max filters have been implemented. The "how to write a filter"...

The min and max filters have been implemented. The "how to write a filter" documentation page has been updated to relfect the latest changes. The operator << has been implemented for various objects involved in the data processing.
parent 5ae2eea9
......@@ -224,43 +224,45 @@ namespace mio {
* This means that a filter might be executed for the parameter TA and another one for the parameter HNW, so the filter
* algorithm only has to deal with a generic filtering method based on double values.
*
* To implement a new filter two steps are necessary:
* To implement a new filter, the following steps are necessary:
*
* -# Registering the filter within the function FilterAlgorithms::initStaticData(), by simply adding a line that
* resembles:
* -# Implementing the filter, as a derived class of FilterBlock, by creating two files: the header file and its
* implementation file, in the meteofilters subdirectory of the source code.
* The class will contain two public methods: a constructor
* that takes a vector of strings containing the filter arguments and a method
* @code
* filterMap["filtername"]=FilterProperties(true, (unsigned int)1, Date(0.0), &FilterAlgorithms::ImplementedFilter);
* process(const unsigned int& index, const std::vector<MeteoData>& ivec, std::vector<MeteoData>& ovec)
* @endcode
* Where the filtername can be freely chosen (although it has to be unique) and the parameters to FilterProperties
* are (in this order) a boolean stating whether this filter is only a check or whether it changes data,
* an unsigned integer stating how many points are minimally needed for this filter to be executed, a Date object
* marking the minimal period that data supplied to this filter needs to span and finally a function pointer
* pointing towards the actual implementation of the filter.
* -# Implementation of the filter, a static function which always has the following interface and returns a boolean:
* that
* applies the filter to the provided vector of values, for a meteo parameter pointed to by index. This index
* is the MeteoData parameter that this filter shall be run upon (see MeteoData for the enumeration of
* parameters). The constructor must set up properties.for_second_pass to mark if the filter can be applied
* a second time during a second pass, that is after resampling.
* A private method
* @code
* bool FilterAlgorithms::FilterName(const std::vector<MeteoData>& vecM, const std::vector<StationData>& vecS,
* const unsigned int& pos, const Date& date, const std::vector<std::string>& _vecArgs,
* const unsigned int& paramindex,
* std::vector<MeteoData>& vecFilteredM, std::vector<StationData>& vecFilteredS)
* parse_args(std::vector<std::string> vec_args)
* @endcode
* vecM and vecS represent the raw data as it is read through the readMeteoData functions of the MeteoIO plugins.
* the unsigned integer pos is the index of either the elements within vecM and vecS that represents the data
* data for the given Date object called date, or if there is no exact match possible then pos is the index of
* the first tuple with a date greater (that is newer) than the date requested (in this case a resampling will have
* to interpolate the data for the date requested). paramindex is the MeteoData parameter that this filter shall be
* run upon (see MeteoData for the enumeration of parameters - e.g. TA(=0), HNW(=6), ISWR(=1)). The two vectors
* vecFilteredM and vecFilteredS correspond and they hold the filtered data. Typically they hold one or two elements
* depending on whether resampling is required or not. In case resampling is not required they only hold one value
* each representing the exact date requested. Changes on the vecFilteredM and vecFilteredS vectors will be
* propagated back to the BufferedIOHandler and thus to the user. If the filter has to alter data it may do so only
* on these two vectors. Typically filters that perform checks only (e.g. check whether values are within certain
* bounds), need not look at anything but the data in these vectors, whereas filters that have more complicated
* schemes of operation (like accumulation or resampling) might take into account the "raw" data as accessible
* through vecM and vecS. Helper functions that parse the arguments to the filters through a Config obejct
* are available.
*
*
* Here an example implementation of the MaximumFilter, which checks whether a value is greater then an argument
* is also implemented to extract the numerical
* values out of the vector of strings of arguments.
* -# Adding the created implementation file to meteofilters/CMakeLists.txt in a similar way as for the other
* filters
* -# Registering the filter within the function BlockFactory::initStaticData(), by simply adding a line
* similar to (in meteofilters/ProcessingBlocks.cc):
* @code
* availableBlocks.insert("MIN");
* @endcode
* Where the filter key can be freely chosen (although it has to be unique and easy/meanigful to the end user
* who will use it in his configuration file)
* -# Adding the filter in the processing loop, in BlockFactory::getBlock(), by adding three lines similar to:
* @code
* else if (blockname == "MIN_MAX"){
* return new FilterMinMax(vec_args);
* }
* @endcode
* -# Including the filter's header file in meteofilters/ProcessingBlocks.cc
*
* The class FilterMax can be used as an example of implementation of a basic filter that will check whether a
* value is greater than an argument
* supplied to the filter and if so changes the value either to IOUtils::nodata (normal operation) or to the
* maximum value supplied in the argument (soft mode of operation). An example section in the io.ini file supplied
* to the Config could look like this:
......@@ -271,33 +273,14 @@ namespace mio {
* @endcode
* Which has the following interpretation: Apply filter max (max-value-filter) to the parameter TA (air temperature)
* in case that a value is greater than 280 degrees Kelvin change that value to 280.
*
* Now for the actual implementation of the filter:
* A more customized operation could be:
* @code
* bool FilterAlgorithms::MaxValueFilter(const std::vector<MeteoData>& vecM, const std::vector<StationData>& vecS,
* const unsigned int& pos, const Date& date, const std::vector<std::string>& _vecArgs,
* const unsigned int& paramindex,
* std::vector<MeteoData>& vecFilteredM, std::vector<StationData>& vecFilteredS)
* {
* (void)vecM; (void)vecS; (void)pos; (void)date; (void)vecFilteredS;
* //parse arguments and check whether they are valid
* bool isSoft = false;
* std::vector<double> vecArgs;
* parseFilterArguments("max", _vecArgs, 1, 1, isSoft, vecArgs);
*
* //Run actual MaxValue filter over all relevant meteo data
* for(unsigned int ii=0; ii<vecFilteredM.size(); ii++){
* double& value = vecFilteredM[ii].param(paramindex);
* if (value == IOUtils::nodata) continue;
* if (value>vecArgs[0]){
* if (isSoft) value=vecArgs[0];
* else value=IOUtils::nodata;
* }
* }
*
* return true;
* }
* [Filters]
* TA::filter1 = max
* TA::arg1 = soft 280 260
* @endcode
* Which will replace any value greater than 280 Kelvin by 260 Kelvin.
*
*/
/**
......
......@@ -115,9 +115,11 @@ std::ostream& operator<<(std::ostream& os, const MeteoProcessor& data)
{
os << "<MeteoProcessor>\n";
os << data.mi1d;
os << "Processing stacks:\n";
for (map<string, ProcessingStack*>::const_iterator it=data.processing_stack.begin();
it != data.processing_stack.end(); it++){
//os << (*it->second) << endl;
//os << setw(10) << it->first << "::"; //the processing stack already contains it
os << (*it->second);
}
os << "</MeteoProcessor>\n";
return os;
......
......@@ -513,7 +513,7 @@ WARN_LOGFILE =
# directories like "/usr/src/myproject". Separate the files or directories
# with spaces.
INPUT = ./meteoio ./meteoio/plugins ./meteoio/meteolaws
INPUT = ./meteoio ./meteoio/plugins ./meteoio/meteolaws ./meteoio/meteofilters
# This tag can be used to specify the character encoding of the source files
# that doxygen parses. Internally doxygen uses the UTF-8 encoding, which is
......
......@@ -2,6 +2,8 @@
INCLUDE_DIRECTORIES("${PROJECT_SOURCE_DIR}/")
SET(meteofilters_sources
meteofilters/FilterMin.cc
meteofilters/FilterMax.cc
meteofilters/FilterMinMax.cc
meteofilters/FilterMedianAvg.cc
meteofilters/FilterMeanAvg.cc
......
......@@ -39,7 +39,7 @@ bool FilterBlock::is_soft(std::vector<std::string>& vec_args)
}
void FilterBlock::convert_args(const unsigned int& min_nargs, const unsigned int& max_nargs,
const std::vector<std::string>& vec_args, std::vector<double>& dbl_args)
const std::vector<std::string>& vec_args, std::vector<double>& dbl_args)
{
if ((vec_args.size() < min_nargs) || (vec_args.size() > max_nargs))
throw InvalidArgumentException("Wrong number of arguments for filter " + getName(), AT);
......
......@@ -37,12 +37,12 @@ class FilterBlock : public ProcessingBlock {
virtual ~FilterBlock();
virtual void process(const unsigned int& index, const std::vector<MeteoData>& ivec,
std::vector<MeteoData>& ovec) = 0;
std::vector<MeteoData>& ovec) = 0;
static bool is_soft(std::vector<std::string>& vec_args);
void convert_args(const unsigned int& min_nargs, const unsigned int& max_nargs,
const std::vector<std::string>& vec_args, std::vector<double>& dbl_args);
const std::vector<std::string>& vec_args, std::vector<double>& dbl_args);
protected:
FilterBlock(const std::string& filter_name); ///< protected constructor only to be called by children
......
/***********************************************************************************/
/* Copyright 2009 WSL Institute for Snow and Avalanche Research SLF-DAVOS */
/***********************************************************************************/
/* This file is part of MeteoIO.
MeteoIO is free software: you can redistribute it and/or modify
it under the terms of the GNU Lesser General Public License as published by
the Free Software Foundation, either version 3 of the License, or
(at your option) any later version.
MeteoIO is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
GNU Lesser General Public License for more details.
You should have received a copy of the GNU Lesser General Public License
along with MeteoIO. If not, see <http://www.gnu.org/licenses/>.
*/
#include <meteoio/meteofilters/FilterMax.h>
using namespace std;
namespace mio {
FilterMax::FilterMax(const std::vector<std::string>& vec_args) : FilterBlock("MAX") {
parse_args(vec_args);
properties.for_second_pass = true; //for the rest: default values
}
void FilterMax::process(const unsigned int& index, const std::vector<MeteoData>& ivec,
std::vector<MeteoData>& ovec)
{
ovec.clear();
for (unsigned int ii=0; ii<ivec.size(); ii++){
ovec.push_back(ivec[ii]);
double& tmp = ovec[ii].param(index);
if (tmp == IOUtils::nodata) continue; //preserve nodata values
if (tmp > max_val){
if (is_soft){
tmp = max_soft;
} else {
tmp = IOUtils::nodata;
}
}
}
}
void FilterMax::parse_args(std::vector<std::string> vec_args) {
vector<double> filter_args;
if (vec_args.size() > 1){
is_soft = FilterBlock::is_soft(vec_args);
}
FilterBlock::convert_args(1, 2, vec_args, filter_args);
if (filter_args.size() > 2)
throw InvalidArgumentException("Wrong number of arguments for filter " + getName(), AT);
max_val = filter_args[0];
if (filter_args.size() == 2){
max_soft = filter_args[1];
} else {
max_soft = max_val;
}
}
} //end namespace
/***********************************************************************************/
/* Copyright 2009 WSL Institute for Snow and Avalanche Research SLF-DAVOS */
/***********************************************************************************/
/* This file is part of MeteoIO.
MeteoIO is free software: you can redistribute it and/or modify
it under the terms of the GNU Lesser General Public License as published by
the Free Software Foundation, either version 3 of the License, or
(at your option) any later version.
MeteoIO is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
GNU Lesser General Public License for more details.
You should have received a copy of the GNU Lesser General Public License
along with MeteoIO. If not, see <http://www.gnu.org/licenses/>.
*/
#ifndef __FILTERMAX_H__
#define __FILTERMAX_H__
//#define CALL_MEMBER_FN(object,ptrToMember) ((object).*(ptrToMember))
#include <meteoio/meteofilters/FilterBlock.h>
#include <vector>
#include <string>
namespace mio {
/**
* @class FilterMax
* @brief
* @author Thomas Egger - Mathias Bavay
* @date 2011-01-02
*/
class FilterMax : public FilterBlock {
public:
FilterMax(const std::vector<std::string>& vec_args);
virtual void process(const unsigned int& index, const std::vector<MeteoData>& ivec,
std::vector<MeteoData>& ovec);
private:
void parse_args(std::vector<std::string> vec_args);
bool is_soft;
double max_val;
double max_soft;
};
} //end namespace
#endif
/***********************************************************************************/
/* Copyright 2009 WSL Institute for Snow and Avalanche Research SLF-DAVOS */
/***********************************************************************************/
/* This file is part of MeteoIO.
MeteoIO is free software: you can redistribute it and/or modify
it under the terms of the GNU Lesser General Public License as published by
the Free Software Foundation, either version 3 of the License, or
(at your option) any later version.
MeteoIO is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
GNU Lesser General Public License for more details.
You should have received a copy of the GNU Lesser General Public License
along with MeteoIO. If not, see <http://www.gnu.org/licenses/>.
*/
#include <meteoio/meteofilters/FilterMin.h>
using namespace std;
namespace mio {
FilterMin::FilterMin(const std::vector<std::string>& vec_args) : FilterBlock("MIN") {
parse_args(vec_args);
properties.for_second_pass = true; //for the rest: default values
}
void FilterMin::process(const unsigned int& index, const std::vector<MeteoData>& ivec,
std::vector<MeteoData>& ovec)
{
ovec.clear();
for (unsigned int ii=0; ii<ivec.size(); ii++){
ovec.push_back(ivec[ii]);
double& tmp = ovec[ii].param(index);
if (tmp == IOUtils::nodata) continue; //preserve nodata values
if (tmp < min_val){
if (is_soft){
tmp = min_soft;
} else {
tmp = IOUtils::nodata;
}
}
}
}
void FilterMin::parse_args(std::vector<std::string> vec_args) {
vector<double> filter_args;
if (vec_args.size() > 1){
is_soft = FilterBlock::is_soft(vec_args);
}
FilterBlock::convert_args(1, 2, vec_args, filter_args);
if (filter_args.size() > 2)
throw InvalidArgumentException("Wrong number of arguments for filter " + getName(), AT);
min_val = filter_args[0];
if (filter_args.size() == 2){
min_soft = filter_args[1];
} else {
min_soft = min_val;
}
}
} //end namespace
/***********************************************************************************/
/* Copyright 2009 WSL Institute for Snow and Avalanche Research SLF-DAVOS */
/***********************************************************************************/
/* This file is part of MeteoIO.
MeteoIO is free software: you can redistribute it and/or modify
it under the terms of the GNU Lesser General Public License as published by
the Free Software Foundation, either version 3 of the License, or
(at your option) any later version.
MeteoIO is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
GNU Lesser General Public License for more details.
You should have received a copy of the GNU Lesser General Public License
along with MeteoIO. If not, see <http://www.gnu.org/licenses/>.
*/
#ifndef __FILTERMIN_H__
#define __FILTERMIN_H__
//#define CALL_MEMBER_FN(object,ptrToMember) ((object).*(ptrToMember))
#include <meteoio/meteofilters/FilterBlock.h>
#include <vector>
#include <string>
namespace mio {
/**
* @class FilterMin
* @brief
* @author Thomas Egger - Mathias Bavay
* @date 2011-01-02
*/
class FilterMin : public FilterBlock {
public:
FilterMin(const std::vector<std::string>& vec_args);
virtual void process(const unsigned int& index, const std::vector<MeteoData>& ivec,
std::vector<MeteoData>& ovec);
private:
void parse_args(std::vector<std::string> vec_args);
bool is_soft;
double min_val;
double min_soft;
};
} //end namespace
#endif
......@@ -16,6 +16,8 @@
along with MeteoIO. If not, see <http://www.gnu.org/licenses/>.
*/
#include <meteoio/meteofilters/ProcessingBlock.h>
#include <meteoio/meteofilters/FilterMin.h>
#include <meteoio/meteofilters/FilterMax.h>
#include <meteoio/meteofilters/FilterMinMax.h>
#include <meteoio/meteofilters/FilterMeanAvg.h>
#include <meteoio/meteofilters/FilterMedianAvg.h>
......@@ -28,6 +30,8 @@ const bool BlockFactory::__init = BlockFactory::initStaticData();
bool BlockFactory::initStaticData()
{
availableBlocks.insert("MIN");
availableBlocks.insert("MAX");
availableBlocks.insert("MIN_MAX");
availableBlocks.insert("MEAN_AVG");
availableBlocks.insert("MEDIAN_AVG");
......@@ -42,7 +46,12 @@ ProcessingBlock* BlockFactory::getBlock(const std::string& blockname, const std:
if (availableBlocks.find(blockname) == availableBlocks.end())
throw UnknownValueException("The processing block '"+blockname+"' does not exist" , AT);
if (blockname == "MIN_MAX"){
if (blockname == "MIN"){
return new FilterMin(vec_args);
} else if (blockname == "MAX"){
return new FilterMax(vec_args);
} else if (blockname == "MIN_MAX"){
return new FilterMinMax(vec_args);
} else if (blockname == "MEAN_AVG"){
return new FilterMeanAvg(vec_args);
......@@ -67,9 +76,29 @@ std::string ProcessingBlock::getName() const
ProcessingBlock::~ProcessingBlock() {}
std::ostream& operator<<(std::ostream& os, const ProcessingBlock& data)
{
os << "[" << data.block_name << " ";
//os << data.properties;
os << "]";
return os;
}
const ProcessingProperties& ProcessingBlock::getProperties() const
{
return properties;
}
std::ostream& operator<<(std::ostream& os, const ProcessingProperties& data)
{
os << "{-" << data.time_before.getJulianDate()*3600. << " +";
os << data.time_after.getJulianDate()*3600. << " h ; ";
os << "-" << data.points_before << " +" << data.points_after << " pts";
if(data.for_second_pass==true)
os << " ; 2nd pass";
os << "}";
return os;
}
} //end namespace
......@@ -28,7 +28,9 @@ namespace mio {
class ProcessingProperties {
public:
ProcessingProperties() : for_second_pass(false), time_before(0.0), time_after(0.0),
points_before(0), points_after(0) {}
points_before(0), points_after(0) {}
friend std::ostream& operator<<(std::ostream& os, const ProcessingProperties& data);
bool for_second_pass;
......@@ -54,6 +56,7 @@ class ProcessingBlock {
std::string getName() const;
const ProcessingProperties& getProperties() const;
friend std::ostream& operator<<(std::ostream& os, const ProcessingBlock& data);
protected:
std::string block_name;
......
......@@ -149,4 +149,18 @@ void ProcessingStack::process(const std::vector< std::vector<MeteoData> >& ivec,
}
}
std::ostream& operator<<(std::ostream& os, const ProcessingStack& data)
{
//os << "<ProcessingStack>";
os << setw(10) << data.param_name << "::";
for(unsigned int ii=0; ii<data.filter_stack.size(); ii++) {
os << setw(10) << *(data.filter_stack[ii]);
}
//os << "</ProcessingStack>";
os << "\n";
return os;
}
}
......@@ -45,6 +45,8 @@ class ProcessingStack {
void getWindowSize(ProcessingProperties& o_properties);
friend std::ostream& operator<<(std::ostream& os, const ProcessingStack& data);
private:
unsigned int getFiltersForParameter(const Config& cfg, const std::string& parname, std::vector<std::string>& vecFilters);
unsigned int getArgumentsForFilter(const Config& cfg, const std::string& keyname, std::vector<std::string>& vecArguments);
......
Supports Markdown
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment