Commits · 18b10ad062ad6a11cd6b4c55862c2fa8f4e5d8c6

And now the code that uses the previously upgraded infrastructure in order to compute spatial interpolations using a trend/residuals approach. The tests work, all results should remain identical to what it was before. Lots of now unnecessary code will be removed from the libinterpol2D in the next few days...

Moving forward with the trend/residuals approach for the spatial interpolations: here are the necessary changes in the supporting infrastructure: basically, better handling of forced lapse rates and fractional lapse rates, proper toString() methods for the fit1D object and components

A broken link has been fixed in GRIBIO, the use of inishell is now properly encouraged in the "how to set up your io.ini" section and some indenting in Array1D

Big cleanup of the tests: now all ini files are based on io.ini in the root folder of the test (relying on IMPORT) and all use the same copy of forcing data and dem. When meteo_reading fails, it now properly write which fields failed (it was always writing "P").

Preparing the way for the new spatial interpolation architecture: the 1D regressions should now offer the necessary features (but this is a first shot at it, it will need to be redone in a more generic way later).

Fixed the license file (issue 309), updated the documentation to include the SIN generator and fixed a wrong commit of the example io.ini

Adding a link in the documentation, moving libsmet to the plugins subdirectory and small fixes in GRIBIO

The IMPORT feature in Config now works properly on Windows. It is also now possible to call IMPORT_BEFORE from within a section (but it still needs to be the first key in a file). 

In order to properly import config files while supporting relative paths, it is necessary to internally expand the paths (as well as the symlinks) to check for multiple inclusions, to properly look for a file located relative to another one (instead of related to the execution directory). The IOUtils::cleanPath function has been expanded to support this kind of processing (with a flag to turn it on of off) and a few functions added (getPath, getFilename). The code for Windows has been written but not tested yet. The Posix implementation should also be tested on osx.

A description of the data format (not the SNOWPACK requirements, only the data format) has been added to the SNIO documentation

The default ilwr model in Atmosphere as been changed to Unsworth. The fixed snow albedo has been set to 0.85 for all filters/generators using it. 

The Config class can now import an ini file from another ini file, this could greatly simplify the management of operational deployments (a station that has a special configuration now can simply import a standard config file and redefine what it needs). This is either done with IMPORT_BEFORE or IMPORT_AFTER in order to choose who has precedence. Multiple (ie recursive) inclusions are supported with some enforced limitations to prevent circular dependencies.

The exceptions were not printed with a line return anymore, messing up the display. A function to extract the integral and the fractional part of a double has been added. The julian day number calculation (ie how many days since the begining of the year) was numericaly instable (leading to wrong values in an unpredictable way). A data generator that can generate sinusoidal values (for daily or yearly periods) has been added. The potential radiation generator can now handle missing TA and/or RH (of course, reverting to a very simplistic clear sky model, assuming an average reduced precipitable water).

Renaming once more the nearest neighbor algorithm: it is now "nearest". That should be both quick and easy.

This adresses issue 173: now the window_size can be defined per meteo parameter and per resampling algorithm. The Interpolations1D section's syntax has been changed in order to accomodate the change and enable specifying multiple resampling algorithms in the future. An exception is thrown when using the old syntax (until we would consider that everybody should have migrated). The nearest_neighbour algorithm key has been renamed to n_neighbor. 

Overall, this commits breaks compatibility with previous ini file but the calculated results remain the same after migrating the config files.

In order to avoid re-parsing the resampling parameters for every point, the resampling algorithms have been restructured as classes inheriting from an interface class alongside an object factory. This should slightly increase the performances of resampling and clears the way (90% of the work being done) for a per-parameter window size (issue 173).

A useful link has been added to the comments of Sun and some comments improved in DataGenerators

The Unsworth parametrization for ILWR has been fixed and now works as data generator. Some comments have been added to Sun.cc.

The graphical message box for exceptions has been turned off by default, because it was interfering with cdash. Packagers are therefore invited to turn it ON for public packages (since this tends to make users read the error messages).

A major bug has been found in the potential short wave solar radiation calculation: the beam values were actually already projected to the horizontal, leading to a double projection to horizontal when requesting the horizontal components. The whole potential radiation code has been reviewed, comparing line by line with Iqbal. A few variables have been renamed in order to better match the paper. And more detailed comments added to explain some of the most mysterious parameters. The potential ISWR data generator has been adjusted and checked. it can use a measured ILWR to correct for cloudiness if available.

The DataGenerators handling of missing meteo fields has been changed: if a generator is defined for a field that is not found in the vecMeteo, it will silently be skipped.

Since some grumpy people seem not to appreciate the graphical message box for exception messages, the ability to turn them off when compiling has been added to cmake. The code for generating such message boxes on Linux purely relying on libX11 is also committed (the boxes are ugly but they don't require more than libX11 and its headers).

We still have to make sure that enabling it by default (that would be the goal) does not make it more difficult to compile meteoio...

The dataGenerators can now process all parameters (ie including the extra parameters) properly. This makes it possible to create a new parameter by using a COPY followed by *0, +nodata filters and then fill the now empty new parameter with a data generator (very useful to compare generated values with a reference measurement). Moreover the processing overhead should have been reduced by looping over the declared generators instead of over the parameters.

The == and != operators have been added to the arrays (easier for debugging). Array4D was not included in MeteoIO.h and has been added.

The meteo reading workflow diagram has been expanded to show where a user can participate and what he can use

Since correcting the Corripio slope algorithm changed the spatial interpolations that were based on slopes, it was necessary to regenerate the reference for the HNW grid

A typo had slipped through the last commit for Array4D. The Corripio slope computation was till wrong, this has finally been fixed and checked. The slope default algorithm selection was not kept through BufferedIOHandler, this has been fixed. Moreover, unecessary copies of grids were being made in BufferedIOHandler and have been removed. The tests that were not passing anymore because of the slope changes have been updated, including improved error reporting in dem_reading.

Removed an unnecessary include and fixed the Corripio slope algorithm that was not properly implemented (following the paper, it was not centered on the cell. It has been rewritten to be centered).

Trying to help the compiler use SIMD instructions for some array operations. Small documentation improvement for IMISIO.

In order to implement an efficient and portable object serialization, it has been decided that all serializable objects would be manually serialized to an std::iostream. Then, the parallelization framework (mpi, popc, etc) can take over this stream and move it to another node. Therefore, the operators "<<" and ">>" are now reserved for serialization (as seems to be standard) and the debug outputs are obtained by printing "object.toString()". This means that all debug code has potentially to be fixed, by replacing "std::cout << object" by "std::cout << object.toString()".

Hoping that the valgrind tests would finally work properly... And small tweak to the IOUtils::seek optimization

Removing some unnecessary pointers (for things like "const double a = vecM[ii](param)") and replacing a vector push_back by a proper allocation in constructor followed by assignments

Some code cleanup and micro-optimizations. The documentation has been improved (configuration of doxygen as well as some duplicate section labels).

Added a missing test for VW==IOUtils::nodata in Hamon undercatch filter. Updated the drawings to reflect the addition of DataGenerator

I will write 100 times "I won't forget to update the tests anymore" (when swapping MeteoData::parameters around). I used the opportunity to imrpove the outputs of the 2D_interpolations test.

Fixed a stupid bug in Unsworth generator (it was always generating data...), improved the documentation of POT_RADIATION and tried to play with a cloudiness factor (not working yet).

A new generator for ISWR or RSWR has been implemented, providing clear sky, potential radiation. Overall, writing the code and its documentation and testing it took less than half an hour!

A buf in some unsworth implementation in Atmosphere has been fixed (check with nodata). The order of the meteodata fields has been changed because this directly impacts the order of the processing of the fields (the goal being that fields that are often checked with a combination of other fields get processed in the end).

The unsworth_dilley ILWR data generator has been implemented with the usage of the last cloudiness during the night as well as the usage of a generic snow albedo (or grass albedo) to convert RSWR to ISWR (if necessary) coupled with a threshold on the snow height.

The METEO_TIMSERIES and STATION_TIMESERIES have been renamed in METEO_SET and STATIONS_SET in order to better reflect their content. 

The concept of data generators is introduced. A new section has been created [Generators] to define how to generate data for each parameter when everything else failed. Soon, you will be able to generate ILWR directly by declaring it in your io.ini!

The soap files for GSN have been re-generated with gsoap 2.8.7 and some explanations on how to do this added in a txt file

Since compiling the meteoio examples is usually problematic for the users, trying to get a proper (but still as simple as possible) CMakeLists.txt. The FindMeteoIO module has been updated to also look for meteoio in "${SRC}/../../lib" so the examples can find the local meteoio.

the parameters mapping has been extended to cover access to the imis stations through GSN. Some units thransforms were not being performs.

Improving the help text (the timer now has at least 1 us resolution and up to the 1 ns, depending on the platform). Fixed a wrong commit for meteo_reading.

A rounding bug has been found in the Date class: when using a Date object in a loop where it gets incremented, it accumulates errors and after ~70000 iterations, the error would get bigger than one second, making date comparisons with a fixed date impossible. This has been solved by always rounding the internal gmt Julian date to the closest second, after each assignment (from the constructors, the setDate() calls or any arithmetic operation). Moreover, the rounding methods have been re-written to be more efficient as well as other roundings (specially when computing the date decomposition from Julian). 

The seek() call performing a binary search has also been simplified, in order to remove unnecessary tests, put some similar tests together and make the code clearer. 

Some constification and better vector usage took place in ResamplingAlgorithms and Meteo1DInterpolator.

Putting the code into shape for major changes in the spatial interpolations as well as pushing the minor version nunmber up

Lots of vectors copies were made using push_back(). This is definitely not efficient! This has been replaced by vec_o = vec_i, we will see which impact it has on performances...

Porting Nander's implementation of Thomas algorithm (solver for tridiagonal matrix) to the matrix class

Still tracking a stupid bug somewhere between meteoio, snowpack and alpine3d... Trying to remove some unecessary stuff

fixing some of the warnings found on Hera (sorry for not noticing before). All the size_t / unsigned int stuff will have to be addressed later

The A3DIO plugin was not properly reading data, this has been fixed. Some warnings have been turned off for osX (signedness and 32/64 bits because of size_t since we will have to redo all our indices to make they consistently rely on size_t)

Trying to get rid of a pointer bug in A3DIO: the ifstream object does not need to be a class member (ie reducing its scope), simplifying some calls...

My previous commit was terrible... it simply made it impossible to define optional keys. This has now been fixed but required a change in the API. So, don't use Config::nothrow but IOUtils::nothrow instead.

The "nothrow" option was not properly forwarded in Config, leading to unnecessary throw/catch. The sun.cc example was set to executable.