GitLab

Finally!! The virtual stations seem to be in proper order! Now the spatial interpolations are only called every vstations_refresh_rate and otherwise the data is temporally resampled.

Finally, the proper handling of virtual stations is taking shape: using two tsmanagers, it is possible to temporally resample the data between spatially interpolated time steps. The strategy to decide which timesteps should be temporally resampled is still buggy, this will have to be improved (currently, it resamples and then spatially interpolates at time when it should resample the spatial interpolations instead).

The gridsmanager and timeseriesmanager can now provide a copy of their Config object. This has been used to fix a bug in the USER and ALS_SCALING spatial interpolation methods that were not properly reading their input files from GRID2DPATH. Moreover, now ALS_SCALING switches to CST when only 1 station provides measurements.

very minor code cleanup, some comments were wrong and have been fixed, a wrong debug message has been fixed (in toString())

Following a bug report from GeoTop, the add_to_cache() method that had been partially removed is now back, under a slightly different name: add_to_points_cache(). Otherwise, white spaces, copyright headings were modified and some pragmas added for VC++ in GSNIO and CosmoXMLIO.

In order to make it more modular and allow more advanced processing (virtual stations, downscaling, multiple io sources) the IOManager is now a proper wrapper: the processing is fully delegated to either a GridsManager or a TimeSeriesManager. These can in turn be reused at some other places (like in the Meteo2DInterpolator) to offer full buffering and processing of "local" time series. The buffering is now done by these two new classes (therefore the BufferedIOHandler class has been removed) and it is planned that dataClasses will be implemented for proper ringbuffers (one for time series, one for grids and one for isolated points) and receive the generic buffering code later on. 

A new fastPow(double, int) has been implemented for positive exponents. The begining of a generic handling of units also appeared in IOUtils. Finally, the ProcessingLevels are now defined in IOUtils instead of IOManager.

better file structure: all the classes purely focused on data storage (such as grids, dates, etc) have been moved into a "dataClasses" directory. All subdirectories are nhow named similarly to the classes and methods: the words after the first one are capitalized.

Small improvements when computing a variogram (in order to be more tolerant to nodata values), new key for virtual stations (to force using the whole DEM instead of a one point dem), a results cache has been created for the virtual stations (mixing virtual and real stations is a really, really bad idea), more flexibility for the Ryan and Liston wind interpolations.

The first algorithm implementing (Winstral, 2002) is now being tested. This seems to work but the mass balance still has to be validated. The IOManager can now be constructed directly with a io.ini file name. Some code clean up has been performed in InterpolationAlgorithms.

A new spatial interpolation algorithm has been implemented: "NONE". This just fills the grid with nodata. Otherwise, an unnecessary check has been removed in Meteo2DInterpolator and some comments editing, white spaces...

IOManager: added the option to clear the point cache through the clear_cache() procedure. Furthermore the point cache is cleared in case push_meteo_data is successful, i.e. when data (filtered or raw) is pushed into the IOManager. 

A new concept has been introduced: virtual stations. These are points where the meteo data will be transparently spatially interpolated and returned to getMeteoData calls. This allows a point model like Snowpack to work on virtual stations without any internal changes. A few new keys have been introduced in order to provide the necessary information: virtual stations location, meteo parameters to interpolate. For example:
VIRTUAL_STATIONS = true
VIRTUAL_PARAMETERS = TA RH HNW HS VW RSWR ILWR
VSTATION1 = 46.8 9.833333
VSTATION2 = 46.7 9.9

A few drawbacks still exist: the time range getMeteoData call is not supported yet. A DEM must be provided (ie DEM key in ini file). Obviously, all interpolated parameters must have been associated with interpolation algorithms. The virtual stations have IDs such as VIR1, VIR2, etc For performance reasons, the interpolation is then performed on a one cell dem, which means that algorithms using dem characteristics would (currently) be messed up. Basically, this is still a work in progress before everything is robust!

Otherwise, some POPC ifdef have been removed and the "iomanager.interpolate(... Grid2DObject)" call has been removed since the "iomanager.getMeteoData(... Grid2DObject)" call should be used instead (for consistency).

Removing an unnecessary vector copying in BufferedIOHandler. A new private method has been added that IOManager can call.

The cmake files for compiling with LibXML++ have been updated since they would not work with more recent Linux (but we should consider dropping LibXML++ and only rely on LibXML2 in order to reduce the dependencies and package finding mess). The special points are now called POI (Points of Interest, like for a gps device) in order to make them more familiar.

The method to clear the buffers in BufferedIOHandler was not doing everything it should. Some methods in BufferedIOHandler and IOManager have been made const. The issues 117 and 272 have been adressed: the IOHandler now checks that timestamps are unique and in increasing order. This check has therefore been removed from SMETIO.

In order to implement an efficient and portable object serialization, it has been decided that all serializable objects would be manually serialized to an std::iostream. Then, the parallelization framework (mpi, popc, etc) can take over this stream and move it to another node. Therefore, the operators "<<" and ">>" are now reserved for serialization (as seems to be standard) and the debug outputs are obtained by printing "object.toString()". This means that all debug code has potentially to be fixed, by replacing "std::cout << object" by "std::cout << object.toString()".

The METEO_TIMSERIES and STATION_TIMESERIES have been renamed in METEO_SET and STATIONS_SET in order to better reflect their content. 

The concept of data generators is introduced. A new section has been created [Generators] to define how to generate data for each parameter when everything else failed. Soon, you will be able to generate ILWR directly by declaring it in your io.ini!

The estimation of the average sampling rate was quite wrong: it was not correct for multiple stations or for partially empty buffers (ie giving an end date way beyond the real end of the data). It was also claiming to return a value in 1/s but instead it was 1/days. This now only takes into account the time interval when there is some data and properly handles multiple stations and returns a value in Hz (or 1/s). The documentation has been expanded. Also some code cleanup (constification).

Exposing the add_to_cache function globally, this way the user can push data into the IOManager that shall be used for a certain date, bypassing resampling and filtering of the IOManager.

A new getMeteo() method has been implemented that transparently performs spatial interpolations and returns a grid for a given parameter. This should be prefered to interpolate(), since it allows the reading plugin to directly provide gridded data (such as coming out of GRIB).
Code cleanup in ResamplingAlgorithms, improved error message in A3DIO. Several keys have been updated/added to the plugins, in order to be more consistent between plugins. Therefore, meteo file extensions are given with METEOEXT, grid2d extensions with GRID2DEXT, grid2d prefix with GRID2DPREFIX (not all plugins support them, though), etc

IOManager: Removed unnecessary consistency check in IOManager::interpolate(), the grid can only have the same size as the DEM anyway. Expanded the arguments of the interpolate() procedure to allow the user to pass a reference to a string, that will hold some info about the interpolation used. 

IOManager: added another interpolate procedure that takes as arguments a Date, a DEMObject, the parameter to interpolate, a vector of Coords objects and a vector of doubles. The Coords represent different points in the DEM. The interpolation is only performed for these points and for each point one double value is stored in a vector. 

Furthermore IOManager now has a member variable called interpolater of type Meteo2DInterpolator. A circular inclusion problem has been solved between IOManager and Meteo2DInterpolator.

libinterpol2D bugfix: The SimpleDEMWindInterpolate algorithm had  a small bug in it, it wasn't interpolating values in the last row or last column of the 2D grid.

The IOManager was only keeping a reference to the Config object given in its constructor. Unfortunately, this means that if the original Config object would go out of scope, the reference would become invalid. Moreover, all internal objects (BufferedIOHandler, Meteo1DInterpolator, etc) were keeping the same reference to this external Config object. However, IOHandler was keeping a copy of the object. This has now been changed: IOManager keeps a copy of the Config object given to its constructor, and all other objects keep a reference to this copy. Therefore, the original Config object can be destroyed without any problem and the internal copy is kept at only one place (within IOManager).

A division by zero (for min=max) has been found and fixed in PGMIO. Moreover, the min and max are written in the metadata (so that the file can be read again and properly rescaled).

The IOManager can now return a copy of its internal Config object. This is convenient for times when one wants to create extra outputs but using a different plugin. A missing const has been added to Config and creating an IOHandler by passing it a config file name is not possible anymore (this was deprecated when we introduced the IOManager). Therefore, the IOHandler now only keeps a reference to a Config object. This means that in the full structure held within IOManager, only one copy of Config is kept, the rest being references.

This addresses the 2D part of issue 168: the ability to specify which meteo parameter and time step to read/write from/to a 2D grid. This should be the prefered reading method in the future, since it leaves to the plugin the task of building the file name, extension, database table, etc This also provides added metadata and more control over the writing process.

A bug was fixed in SMETIO: when given OLWR, it should have computed TSS, but it was not... (using the wrong MeteoData object).

Documentation and error messages improvements. A new method was added to Atmosphere: virtualTemperatureFactor.

Add a setMinBufferRequirements() method to IOManager and BufferedIOHandler. This allows an application to specify its buffering needs. For example, when SNOWPACK runs grass detection, it needs the data up to 48 hours before the first time step to compute some averages. Using this call, SNOWPACK will request at least 48 hours of data in pre-buffering. if buff_before is set to something larger, it will be kept, but if it is set to something smaller, it will be increased.

The popc structure has been reverted to what it was before. The new structure, without any parallel objects was not working. 

The gcc flags used by the autotools have been adapted to follow more closely what cmake generates.

The POPC version has been heavily transformed. Since we would need IOManager has a popc object and this is quite hard (we need a backreference in Meteo2DInterpolator), all parclasses have been removed. Instead, the user will have the responsability to packaged IOManager in his own parclass if needed. This makes the code simpler and skips difficult issues...

A new fill method has been added to the Array2D and Array3D templates that does the reverse of subset (ie: reassemble arrays).

Better error messages have been put in SNIO.

A new Date setter that takes a Date object has been created.

This version does NOT correctly compile with the autotools (it leads to a segfault when running code using MeteoIO), but the cmake compiled library works fine.

The CMakeLists has been adapted so that it properly supports various Visual C++ options. Now, options are set per compiler and per plateform (depending on the scope of the said option). Therefore, choosing a target like "optimized" really produces compilation optimized for speed on g++ as well as Visual C++.

Several type casting issues (warnings) have also been addressed.

When calling Coords::setProj(), if only x/y were known, the lat/long were not calculated (but this was done for lat/long known and x/y missing). This has been fixed.

The slope angle normalization between 0-90 degrees has been done.

A merge method has been implemented for Coords and StationData. This simply replaces nodata fields from one object by the matching field from another object (priority given to the current object or the first object in the static method). This is quite a primitive merge strategy, but I doubt anything better could ever be done...

Added a possibility to push meteo data into the IOManager in case the user does not wish to read meteo data with MeteoIO but wishes to use other functionality provided by the IOManager (e.g. resampling, interpolation, filtering):

iomanager.push_meteo_data(IOManager::filtered, date_start, date_end, vec_meteo);

The popc version has been updated. It is far from perfect (lots of popc specific hacks), it lacks an IOManager parclass, but this is the minimum that we need for Alpine3D...

The ptr_to_function to ptr_to_object cast warning has been turned off (at least for gcc). This does not fix the core issue (that has to be dealt with by the future c++ standard), but at least we don't see it anymore...

The (advanced) configuration of the BufferedIOHandler through io.ini (General section) has been added. It currently introduces two keywords (BUFF_CHUNKS_SIZE and BUFF_CHUNKS) to drive how much data to read at once and how much data to buffer. Currently, their product is directly used in the read, thus not reading anything by chunks smaller than a full buffer.

Fixed some << outputs, two compilation warnings found on Hera as well as introduced.