WSL/SLF GitLab Repository

Commit 9ff65eb0 authored by Mathias Bavay's avatar Mathias Bavay
Browse files

Some documentation bugs have been fixed (multiple links definitions for...

Some documentation bugs have been fixed (multiple links definitions for doxygen). The config.dox file has been updated (in order to remove the warnings for obsolete keys). The parameter exclusion feature has been documented and now supports both a space delimited external file or by keys such as WFJ2::exclude = TA RH HS
parent 6bf3bbee
......@@ -81,7 +81,8 @@ using namespace std;
namespace mio {
/**
* @page plugins Plugins overview
* The data access is handled by a system of plugins. They all offer the same interface, meaning that a plugin can transparently be replaced by another one. Since they might rely on third party libraries for accessing the data, they have been created as plugins, that is they are loaded on demand (and also compiled only if requested at compile time). A plugin can therefore fail to load (for example if it does not exist) at run time.
* The data access is handled by a system of plugins. They all offer the same interface, meaning that a plugin can transparently be replaced by another one. Since they might rely on third party libraries for accessing the data, they have been created as plugins, that is they are only compiled if requested when configuring the compilation with cmake.
* A plugin can therefore fail to run if it has not been compiled.
*
* @section available_categories Data sources categories
* Several data sources categories have been defined that can be provided by a different plugin. Each data source category is defined by a specific key in the configuration file (usually, io.ini):
......@@ -122,7 +123,8 @@ namespace mio {
* <tr><td>\subpage snowpack "SNOWPACK"</td><td>meteo</td><td>original SNOWPACK meteo files</td><td></td></tr>
* </table></center>
*
* @section data_generators Data generators
* @section data_manipulations Data generators & exclusion
* @subsection data_generators Data generators
* It is also possible to duplicate a meteorological parameter as another meteorological parameter. This is done by specifying a COPY key, following the syntax
* new_name::COPY = existing_parameter. For example:
* @code
......@@ -131,6 +133,26 @@ namespace mio {
* This creates a new parameter VW_avg that starts as an exact copy of the raw data of VW, for each station. This newly created parameter is
* then processed as any other meteorological parameter (thus going through filtering, generic processing, spatial interpolations). This only current
* limitation is that the parameter providing the raw data must be defined for all stations (even if filled with nodata, this is good enough).
*
* @subsection data_exclusion Data exclusion
* It is possible to exclude specific parameters from given stations (on a per station basis). This is either done by using the station ID
* followed by "::exclude" as key with a space delimited list of \ref meteoparam "meteorological parameters" to exclude for the station as key.
* Another possibility is to provide a file containing one station ID per line followed by a space delimited list of \ref meteoparam "meteorological parameters"
* to exclude for the station (the path to the file can be a relative path and will be properly resolved).
*
* @code
* WFJ2::exclude = TA RH ;inline declaration of parameters exclusion
* KLO3::exclude = HS HNW
*
* EXCLUDE_FILE = ../input/meteo/excludes.csv ;parameters exclusions defined in a separate file
* @endcode
*
* In the second example (relying on a separate file), the file "../input/meteo/excludes.csv" could look like this:
* @code
* WFJ2 TA RH
* KLO3 HS HNW
* @endcode
*
*/
IOInterface* IOHandler::getPlugin(const std::string& plugin_name) const
......@@ -332,47 +354,65 @@ void IOHandler::checkTimestamps(const std::vector<METEO_SET>& vecVecMeteo) const
void IOHandler::create_exclude_map()
{
excludes_ready = true;
string exclude_file;
cfg.getValue("EXCLUDE_FILE", "Input", exclude_file, IOUtils::nothrow);
excludes_ready = true;
if (exclude_file.empty()) return;
if (!exclude_file.empty()) {
//if this is a relative path, prefix the path with the current path
const std::string prefix = ( IOUtils::isAbsolutePath(exclude_file) )? "" : cfg.getConfigRootDir()+"/";
const std::string path = IOUtils::getPath(prefix+exclude_file, true); //clean & resolve path
const std::string filename = path + "/" + IOUtils::getFilename(exclude_file);
//if this is a relative path, prefix the path with the current path
const std::string prefix = ( IOUtils::isAbsolutePath(exclude_file) )? "" : cfg.getConfigRootDir()+"/";
const std::string path = IOUtils::getPath(prefix+exclude_file, true); //clean & resolve path
const std::string filename = path + "/" + IOUtils::getFilename(exclude_file);
std::ifstream fin; //Input file streams
fin.open(filename.c_str(), std::ifstream::in);
if (fin.fail()) throw FileAccessException(filename, AT);
std::ifstream fin; //Input file streams
fin.open(filename.c_str(), std::ifstream::in);
if (fin.fail()) throw FileAccessException(filename, AT);
try {
const char eoln = IOUtils::getEoln(fin); //get the end of line character for the file
try {
const char eoln = IOUtils::getEoln(fin); //get the end of line character for the file
vector<string> tmpvec;
string line;
vector<string> tmpvec;
string line;
while (!fin.eof()) { //Go through file
getline(fin, line, eoln); //read complete line meta information
IOUtils::stripComments(line);
const size_t ncols = IOUtils::readLineToVec(line, tmpvec, ' ');
while (!fin.eof()) { //Go through file
getline(fin, line, eoln); //read complete line meta information
IOUtils::stripComments(line);
const size_t ncols = IOUtils::readLineToVec(line, tmpvec, ',');
if (ncols > 1) {
for(vector<string>::iterator it = tmpvec.begin()+1; it != tmpvec.end(); ++it) {
IOUtils::toUpper(*it);
}
if (ncols > 1) {
for(vector<string>::iterator it = tmpvec.begin()+1; it != tmpvec.end(); ++it) {
IOUtils::toUpper(*it);
const set<string> tmpset(tmpvec.begin()+1, tmpvec.end());
excluded_params[ tmpvec[0] ] = tmpset;
}
const set<string> tmpset(tmpvec.begin()+1, tmpvec.end());
excluded_params[ tmpvec[0] ] = tmpset;
}
} catch (const std::exception&) {
fin.close();
throw;
}
} catch (const std::exception&) {
fin.close();
throw;
}
fin.close();
vector<string> exclude_keys;
const size_t nrOfStations = cfg.findKeys(exclude_keys, "::EXCLUDE", "Input", true);
for (size_t ii=0; ii<nrOfStations; ++ii) {
const size_t found = exclude_keys[ii].find_first_of(":");
if (found==std::string::npos) continue;
const string station( exclude_keys[ii].substr(0,found) );
std::vector<std::string> vecString;
cfg.getValue(exclude_keys[ii], "Input", vecString);
if (vecString.empty()) throw InvalidArgumentException("Empty value for key \""+exclude_keys[ii]+"\"", AT);
for(vector<string>::iterator it = vecString.begin(); it != vecString.end(); ++it) {
IOUtils::toUpper(*it);
}
const set<string> tmpset(vecString.begin(), vecString.end());
excluded_params[ station ] = tmpset;
}
}
/**
......
This diff is collapsed.
......@@ -37,7 +37,7 @@ namespace mio {
* It requires <A HREF="http://xmlsoft.org/">libxml2</A> to compile and run. It also assumes that the station IDs are unique
* (ie two data sets with the same station ID are considered to belong to the same station).
*
* @section cosmo_partners COSMO Group
* @section cosmoxml_cosmo_partners COSMO Group
* This plugin has been developed primarily for reading XML files produced by COSMO (http://www.cosmo-model.org/) at MeteoSwiss.
* COSMO (COnsortium for Small scale MOdelling) represents a non-hydrostatic limited-area atmospheric model, to be used both for operational and for research applications by the members of the consortium. The Consortium has the following members:
* - Germany, DWD, Deutscher Wetterdienst
......
......@@ -49,7 +49,7 @@ namespace mio {
*
* As a side note, when calling read2DGrid(grid, filename), it will returns the first grid that is found.
*
* @section cosmo_partners COSMO Group
* @section gribio_cosmo_partners COSMO Group
* This plugin has been developed primarily for reading GRIB files produced by COSMO (http://www.cosmo-model.org/) at MeteoSwiss.
* COSMO (COnsortium for Small scale MOdelling) represents a non-hydrostatic limited-area atmospheric model, to be used both for operational and for research applications by the members of the consortium. The Consortium has the following members:
* - Germany, DWD, Deutscher Wetterdienst
......@@ -67,7 +67,7 @@ namespace mio {
* @section gribio_units Units
* As specified by WMO.
*
* @section files_naming Files naming scheme
* @section gribio_files_naming Files naming scheme
* When a grid is read by providing the filename to open, any file name will obviously work. Otherwise, the file names have to follow the pattern:\n
* {GRID2DPREFIX}{ISO8601 numerical UTC date}{GRID2DEXT}\n
* By default, GRID2DPREFIX is empty and GRID2DEXT is ".grb". This means that by default, a grib file containing data for 2013-10-15T12:00 would be:
......
......@@ -63,7 +63,7 @@ namespace mio {
* - STRICTFORMAT: Whether the NetCDF file should be strictly compliant with the CNRM standard; Parameters not present
* in the specification will be omitted; [Input] and [Output] section
*
* @section example Example use
* @section netcdf_example Example use
* @code
* [Input]
* DEM = NETCDF
......@@ -71,7 +71,7 @@ namespace mio {
* DEMVAR = z
* @endcode
*
* @section Compilation
* @section netcdf_compilation Compilation
* In order to compile this plugin, you need libnetcdf (for C). For Linux, please select both the libraries and
* their development files in your package manager.
*/
......
......@@ -30,7 +30,7 @@ using namespace std;
namespace mio {
/**
* @page pngio PNGIO
* @section template_format Format
* @section pngio_format Format
* This plugin write data to the Portable Network Graphics format (see https://secure.wikimedia.org/wikipedia/en/wiki/Portable_Network_Graphics).
* No data read has been implemented, because reading an existing file would require the exact knowlege of the color gradient that has been used
* to create it. When writing grids, various color gradients will be used depending on the parameter that the data represents. Nodata values
......@@ -38,10 +38,10 @@ namespace mio {
* If a grid containing no data (ie: size 0x0) is sent to the plugin, then no file will be written.
* Finally, the naming scheme for meteo grids should be: YYYY-MM-DDTHH.mm_{MeteoGrids::Parameters}.png
*
* @section template_units Units
* @section pngio_units Units
* All units are MKSA except temperatures that are expressed in celcius.
*
* @section template_keywords Keywords
* @section pngio_keywords Keywords
* This plugin uses the following keywords:
* - COORDSYS: input coordinate system (see Coords) specified in the [Output] section
* - COORDPARAM: extra input coordinates parameters (see Coords) specified in the [Output] section
......@@ -71,7 +71,7 @@ namespace mio {
* 5426523.318065105000 (y coordinate of centre of upper left pixel in map units)
* @endcode
*
* @section example Example use
* @section pngio_example Example use
* @code
* GRID2D = PNG
* png_legend = false
......@@ -79,7 +79,7 @@ namespace mio {
* png_max_size = 1366*768
* @endcode
*
* @section Compilation
* @section pngio_compilation
* In order to compile this plugin, you need libpng and zlib. For Linux, please select both the libraries and their development files in your package manager.
*
* For Windows, you can find zlib at http://switch.dl.sourceforge.net/project/gnuwin32/zlib/1.2.3/zlib-1.2.3.exe
......
......@@ -23,16 +23,16 @@ using namespace std;
namespace mio {
/**
* @page smetio SMET
* @section template_format Format
* @section smetio_format Format
* The Station meteo data files is a station centered, ascii file format that has been designed with flexibility and ease of use in mind. Please refer to its <a href="../SMET_specifications.pdf">official format specification</a> for more information (including the list of standard parameters: TA, TSS, TSG, RH, VW, DW, ISWR, OSWR, ILWR, OLWR, PINT, PSUM, HS).
* This plugin can also provide Points Of Interest, given as a SMET file containing either latitude/longitude/altitude or easting/northing/altitude. For the latter, the header must contain the epsg code (see example below).
*
* Non-standard parameters can also be given, such as extra snow temperatures. These parameters will then take the name that has been given in "fields", converted to uppercase. It is usually a good idea to number these parameters, such as TS1, TS2, TS3 for a serie of temperatures at various positions.
*
* @section template_units Units
* @section smetio_units Units
* All units are MKSA, the only exception being the precipitations that are in mm/h. It is however possible to use multipliers and offsets (but they must be specified in the file header).
*
* @section template_keywords Keywords
* @section smetio_keywords Keywords
* This plugin uses the following keywords:
* - STATION#: input filename (in METEOPATH). As many meteofiles as needed may be specified
* - METEOPATH: meteo files directory where to read/write the meteofiles; [Input] and [Output] sections
......
Supports Markdown
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment