GitLab

Finally!! The virtual stations seem to be in proper order! Now the spatial interpolations are only called every vstations_refresh_rate and otherwise the data is temporally resampled.

Finally, the proper handling of virtual stations is taking shape: using two tsmanagers, it is possible to temporally resample the data between spatially interpolated time steps. The strategy to decide which timesteps should be temporally resampled is still buggy, this will have to be improved (currently, it resamples and then spatially interpolates at time when it should resample the spatial interpolations instead).

A new option has been added to smet, to skip the plot_xxx headers (default: keep them). The virtual stations were not providing a proper call to only get the metadata, this is now available.

The gridsmanager and timeseriesmanager can now provide a copy of their Config object. This has been used to fix a bug in the USER and ALS_SCALING spatial interpolation methods that were not properly reading their input files from GRID2DPATH. Moreover, now ALS_SCALING switches to CST when only 1 station provides measurements.

Code cleanup and fix for IMIS: now the ENET stations receive the number 1 for the automatic stations instead of 0.

Following a bug report from GeoTop, the add_to_cache() method that had been partially removed is now back, under a slightly different name: add_to_points_cache(). Otherwise, white spaces, copyright headings were modified and some pragmas added for VC++ in GSNIO and CosmoXMLIO.

In order to make it more modular and allow more advanced processing (virtual stations, downscaling, multiple io sources) the IOManager is now a proper wrapper: the processing is fully delegated to either a GridsManager or a TimeSeriesManager. These can in turn be reused at some other places (like in the Meteo2DInterpolator) to offer full buffering and processing of "local" time series. The buffering is now done by these two new classes (therefore the BufferedIOHandler class has been removed) and it is planned that dataClasses will be implemented for proper ringbuffers (one for time series, one for grids and one for isolated points) and receive the generic buffering code later on. 

A new fastPow(double, int) has been implemented for positive exponents. The begining of a generic handling of units also appeared in IOUtils. Finally, the ProcessingLevels are now defined in IOUtils instead of IOManager.

The grids.isEmpty() method has been renamed as empty() in order to follow the STL convention. The ARC grids written with Alpine3D naming were still not properly named, they should now be directly recognized by the Alpine3D grid viewer.

Mostly code cleanup: trying to find more logical names for the buffering code in BufferedIOHandler, more modularity in InterpolationAlgorithms. Since when computing a variogram from time series it is possible to end up with significantly lower values than usual that lead the krigging in a totally wrong direction (should the variogram be normalized? Is it linked with the stationarity hypothesis?), now the empirical variogram is used instead of the full time series.

Removing nodata values seemed like a good idea when doing a sort(), but this created some bugs (sometimes it is still valuable to know that y=nodata after sorting). This is now an optional behavior. The interpolated grids are now buffered in Meteo2DInterpolator. The grid buffering code has been cleaned up and backported to BufferedIOHandler. The spatial interpolations specific points (for virtual stations, etc) has been moved to Meteo2DInterpolator in order to make IOManager smaller and more focused on general data flow handling.

Small improvements when computing a variogram (in order to be more tolerant to nodata values), new key for virtual stations (to force using the whole DEM instead of a one point dem), a results cache has been created for the virtual stations (mixing virtual and real stations is a really, really bad idea), more flexibility for the Ryan and Liston wind interpolations.

Several bugs have been fixed: in the AllSky generator, the cloudiness was still forced to 0.5 (as for debug). The data creators were also called *after* the generators, thus making it impossible to used a created data in a generator. 

The placeholders have been put in place (but commented out) for the cloudiness/tau_cloud/clearness index support in MeteoData. It seems that Kt=clearness index might be the proper way to go, but this is still a messy field. Some comments have been added in the code to clarify the use of such parameters.

Data Creators have been defined: it is now possible to create an extra parameter from a data generator. Simply define it as {param_name}::create = {generators} in the [Generators] section and the parameter will be created from any data generators specified (beside the "create" syntax, everything is currently identical with data generators).

The ClearSky and AllSky generators have been renamed as ClearSky_LW and AllSky_LW for clarity. The debug outputs (.toString()) have been improved. The Config::findKey method is now able to do pattern matching anywhere in the string (if "anywhere=true"). This makes some code simpler (for generators). 

The parameter copy feature has a new syntax: TA2::COPY = TA will copy TA into TA2 (clearer and more consistent). Please update your ini files if necessary!

The first algorithm implementing (Winstral, 2002) is now being tested. This seems to work but the mass balance still has to be validated. The IOManager can now be constructed directly with a io.ini file name. Some code clean up has been performed in InterpolationAlgorithms.

Preparing the way for altitude thresholds on lapse rates: for example, preventing HNW to get crazy at elevations much higher than the highest station. This would be achieved by using max_altitude instead of the true elevation of a high point when doing detrending/retrending. See issue 358 and issue 359.

A new spatial interpolation algorithm has been implemented: "NONE". This just fills the grid with nodata. Otherwise, an unnecessary check has been removed in Meteo2DInterpolator and some comments editing, white spaces...

IOManager: added the option to clear the point cache through the clear_cache() procedure. Furthermore the point cache is cleared in case push_meteo_data is successful, i.e. when data (filtered or raw) is pushed into the IOManager. 

A new concept has been introduced: virtual stations. These are points where the meteo data will be transparently spatially interpolated and returned to getMeteoData calls. This allows a point model like Snowpack to work on virtual stations without any internal changes. A few new keys have been introduced in order to provide the necessary information: virtual stations location, meteo parameters to interpolate. For example:
VIRTUAL_STATIONS = true
VIRTUAL_PARAMETERS = TA RH HNW HS VW RSWR ILWR
VSTATION1 = 46.8 9.833333
VSTATION2 = 46.7 9.9

A few drawbacks still exist: the time range getMeteoData call is not supported yet. A DEM must be provided (ie DEM key in ini file). Obviously, all interpolated parameters must have been associated with interpolation algorithms. The virtual stations have IDs such as VIR1, VIR2, etc For performance reasons, the interpolation is then performed on a one cell dem, which means that algorithms using dem characteristics would (currently) be messed up. Basically, this is still a work in progress before everything is robust!

Otherwise, some POPC ifdef have been removed and the "iomanager.interpolate(... Grid2DObject)" call has been removed since the "iomanager.getMeteoData(... Grid2DObject)" call should be used instead (for consistency).

Removing an unnecessary vector copying in BufferedIOHandler. A new private method has been added that IOManager can call.

The cmake files for compiling with LibXML++ have been updated since they would not work with more recent Linux (but we should consider dropping LibXML++ and only rely on LibXML2 in order to reduce the dependencies and package finding mess). The special points are now called POI (Points of Interest, like for a gps device) in order to make them more familiar.

Fixing some warnings for Visual C++ and also using ostringstream instead of stringstream when appropriate

Now the toString() method of Meteo2DInterpolator works with the new structure and is called by IOManager.toString(). Small cleanup (white spaces, etc) in 2D_interpolations and ResamplingAlgorithms.

The method to clear the buffers in BufferedIOHandler was not doing everything it should. Some methods in BufferedIOHandler and IOManager have been made const. The issues 117 and 272 have been adressed: the IOHandler now checks that timestamps are unique and in increasing order. This check has therefore been removed from SMETIO.

In order to implement an efficient and portable object serialization, it has been decided that all serializable objects would be manually serialized to an std::iostream. Then, the parallelization framework (mpi, popc, etc) can take over this stream and move it to another node. Therefore, the operators "<<" and ">>" are now reserved for serialization (as seems to be standard) and the debug outputs are obtained by printing "object.toString()". This means that all debug code has potentially to be fixed, by replacing "std::cout << object" by "std::cout << object.toString()".

The METEO_TIMSERIES and STATION_TIMESERIES have been renamed in METEO_SET and STATIONS_SET in order to better reflect their content. 

The concept of data generators is introduced. A new section has been created [Generators] to define how to generate data for each parameter when everything else failed. Soon, you will be able to generate ILWR directly by declaring it in your io.ini!

In order to try to be closer to the spirit of the STL, now the empty strings/vectors/lists are tested with .empty(), the first/last elements of vectors accessed with front()/back(), the empty strings are not contructed with string("") but with string(), etc This makes a benchmark based on meteo_reading ~4% faster and this makes the code suprisingly easier to read. 

Some extra constification took place as well as some minor code cleanup (optimizing the comparison between two vectors in order to perform less tests, etc).

The Date class can now properly generate and parse full ISO timestamps, including time zone information. The full specification is now supported by convertString() with a helper method in the Date class. Ouputing an ISO string with its time zone is done by requesting the ISO_TZ format.

Lots of small fixes after running cppcheck on Windows: unused variables, use ++i instead of i++ for complex types (ie mostly iterators for us) for more performance, a few variables that could be passed as const&, use vector.empty() instead of vector.size==0 (for clarity and potentially performances).

On osx too, the exceptions can now produce a message box. Several warnings have been fixed (mostly signed/unisgned issues)

Some better error messages have been added to SMETIO as well as some code cleanup. Some comments have been added to IOManager, and missing popc code added to DEMObject and Date.

After compiling with -Weffc++, it appeared that lots of things could be done in initialization lists. Some of it has been done for this commit (but quite a few remain).

Some extra documentation has been added for the UPS coordinate system. The coordinates test has been expanded. A missing default value has been added to the Suntrajectory.

The smoothing filters now behave differently depending on the centering that has been specified. The IOManager backreference for Meteo2DInterpolator (pointer this given in the initialization list) has been replaced by a setter.

An invalid flag for Visual C++ has been fixed.

Better error messages in IMISIO, better safety (regarding "undef") in Date and some code cleanup/const/reserve business (all benchmarked) for the others. We also now avoid stacking two "/" at the end of the PLUGINPATH if one is already present.