The two arguments accumulate resampling algorithm has been removed. The distribute filter should be used instead. The missing files are now committed.

A new processing element has been added: HNW_DISTRIBUTE. This distributes accumulated precipitation over a number of preceeding time steps. This is useful, for example, when one has an hourly file that contains 24 hours sums of precipitation, written as nodata for all time steps and the accumulation on the last time step of the 24 hours period. Since this is implemented as a filter, it is then possible to also add an undercatch correction (and don't forget to re-accumulate anyway in the resampling methods!).

Finally! Handling precipitation redistribution before re-accumulation (ie the 24 hours sums provided in an otherwise hourly data file) now works with non-uniform timesteps. This introduced some runtime performance hit (~10% slower) but better accomodate the needs of our users!

Finally, it is now posible to handle precipitation data provided as 24 hours sums in an hourly data file and resample them as 15 minutes sums. The limitations are: no time offset between the requested sampling period and the measured accumulation period; only constant sampling in the data file (ie no variations in the sampling period in the data file).

Finally, the reaccumulation looks good and seems to work in all conditions. Using a measuring period different from the file sampling rate (ie daily accumulation in a hourly-sampled file) is not supported yet, but this should now come very soon!

The reaccumulation refactoring is moving forward. This commit is not so clean (lots of commented out debugging output, since this is a work in progress) but it "mostly works". The problem that is left is that the dwonsampling produces yearly sums a little bigger than they should (ie we are double-counting somewhere). But it compiles, it runs and it is "almost" correct. The simple upsampling works properly (ie producing 15 minutes values out of hourly data).

Paving the way for very exciting stuff for reaccumulating precipitation. For now, major code cleanup and simplification. Watch out, there might be cases where this does not work anymore!

Fixing some warnings for Visual C++ and also using ostringstream instead of stringstream when appropriate

Finally, another method for handling daily sums of solar radiation has been implemented: when a point is requested, the potential solar radiation is computed for the whole day with a 20 minutes resolution, filling a vector and computing the daily sum. This enables to compute the loss factor. When another point within the same day is requested, the radiation is interpolated from the values stored in the vector and multiplied by the loss factor. Tested on WFJ data, this works very well (of course, days that experience large cloudiness variations are not reproduced well but at least the average flux is correct).

A typo has been fixed in the documentation of Date. 

Now the dates can be rounded to the day (there was a bug preventing it before). A new parametrization has been implemented in SunObject that computes the daily sum of TOA radiation. This is used by a new resampling method that computes subdaily values out of a daily sum of radiation. This is not yet doing such a great job, but this is a start...

A new test has been created to cover basic statistical functions on vectors. Two bugs have been found and fixed when computing the derivative on a vector of data (the right derivative was in fact a left derivative and the case of X containing nodata was not covered).

Documentation fixes in libinterpol2D and ResamplingAlgorithms. The cmake macro for finding MeteoIO now looks first in the home directory of the user on Mac (as is done on Linux).

Now the toString() method of Meteo2DInterpolator works with the new structure and is called by IOManager.toString(). Small cleanup (white spaces, etc) in 2D_interpolations and ResamplingAlgorithms.

Renaming once more the nearest neighbor algorithm: it is now "nearest". That should be both quick and easy.

This adresses issue 173: now the window_size can be defined per meteo parameter and per resampling algorithm. The Interpolations1D section's syntax has been changed in order to accomodate the change and enable specifying multiple resampling algorithms in the future. An exception is thrown when using the old syntax (until we would consider that everybody should have migrated). The nearest_neighbour algorithm key has been renamed to n_neighbor. 

Overall, this commits breaks compatibility with previous ini file but the calculated results remain the same after migrating the config files.

In order to avoid re-parsing the resampling parameters for every point, the resampling algorithms have been restructured as classes inheriting from an interface class alongside an object factory. This should slightly increase the performances of resampling and clears the way (90% of the work being done) for a per-parameter window size (issue 173).

A useful link has been added to the comments of Sun and some comments improved in DataGenerators

Removing some unnecessary pointers (for things like "const double a = vecM[ii](param)") and replacing a vector push_back by a proper allocation in constructor followed by assignments

A rounding bug has been found in the Date class: when using a Date object in a loop where it gets incremented, it accumulates errors and after ~70000 iterations, the error would get bigger than one second, making date comparisons with a fixed date impossible. This has been solved by always rounding the internal gmt Julian date to the closest second, after each assignment (from the constructors, the setDate() calls or any arithmetic operation). Moreover, the rounding methods have been re-written to be more efficient as well as other roundings (specially when computing the date decomposition from Julian). 

The seek() call performing a binary search has also been simplified, in order to remove unnecessary tests, put some similar tests together and make the code clearer. 

Some constification and better vector usage took place in ResamplingAlgorithms and Meteo1DInterpolator.

Some extra documentation has been added for the UPS coordinate system. The coordinates test has been expanded. A missing default value has been added to the Suntrajectory.

An optimization from Yesterday was actually breaking the whole resampling... This has been fixed (thanks to Nander and David for finding this problem!)

A few "consts" have been added. The dates are now considered to be positive which allows comparison operators to be optimized: converting 1 year of hourly data with data_converter got almost 30% faster. Moreover, when looking for resampling window, two extra checks have been added to quickly see if the start and end point are even in the input vector. This reduces data read miss (because we start by reading vecM[0]) and skips browsing through the whole vector if we are in a large data gap. On the same 1 year data set with one large gap for one parameter, the gain is of almost 20%. Overall, these two optimizations lead to almost a doubling of the speed with data_converter... Soon we'll be so fast that we will get the data before it is even measured!

Fixed a division by zero in libsmet and the "none" resampling was broken (it was deleting all data instead of propagating available data). Fixed!

Investigating issue 210, it has been found that the IMIS precipitation had a division by 2 that was not required (halving the precipitation compared to what it should have been). This has been fixed.

Small documentation improvement in ResamplingAlgorithms.

A new option has been added to the accumulate interpolation: "strict". When this is active, if one of the accumulated value is nodata, the whole sum will be set to nodata. Without this option, all values that can be aggregated are aggregated (ie: best effort). When using multiple stations, it might make sense to enable "strict" in order to avoid polluting the spatial interpolation with potentially under-estimated values.

A documentation issue has been fixed in Meteo1DInterpolator.

The re-accumulation should now work properly for down sampling (tested with data at h+40 converted to h+00 and with 10min data converted to hourly).

Fixed some warnings (about unused variables/parameters). Added a new position for ResamplingAlgorithms: begin to describe that a point is at the begining of the data set (so we don't have to look for this condition again in the algorithm).

Bugfix: The new linear resampling always extrapolated at the end of the meteo data vector, this is fixed now.

TODO: verify and comment

Further improvements for the resampling within the Meteo1DInterpolator: no longer is an element inserted into the filtered_cache and then deleted, instead we have a MeteoData reference handed to the Interpolator representing the MeteoData element to be resampled. This interface change needed to be propagated into the different affected classes. Accumulate resampling currently disabled.

The re-accumulation was broken and producing bogus results. Moreover, its handling of nodata was sub-optimal (assuming nodata=zero lead to artificially low measurements for very noisy stations). This has been fixed. Accumulating over periods larger than in the original data set (ie:up sampling) has been validated. Down sampling has NOT been validated yet.

Documentation, default value for WINDOW_SIZE=10 days, small speed optimizations (1% faster!!), code cleanup to prepare the way for re-accumulation work

The resampling has been deeply transformed in order to produce more consistent results. By doubling the data cut window, the resampling can now properly fill a 23 hours gap with a 24 hours window. The resampling now either fills the whole gap or do not fill anything (if the gap is too large). When activating "extrapolate", a gap is now filled by the sides instead of by the middle (which was producing weird results). The drawback is that the cut window had to be doubled, which is quite costly (going from 1 day to 10 days makes it 7.5 times slower, and going to 20 days makes it 15 times slower). However, this more logical and consistent behavior will allow the user to specify the window size that he needs instead of always picking something extremely large to be on the safe side. Finally, the resampling code has been cleaned up so that each algorithm is built along similar lines.

A new getMeteo() method has been implemented that transparently performs spatial interpolations and returns a grid for a given parameter. This should be prefered to interpolate(), since it allows the reading plugin to directly provide gridded data (such as coming out of GRIB).
Code cleanup in ResamplingAlgorithms, improved error message in A3DIO. Several keys have been updated/added to the plugins, in order to be more consistent between plugins. Therefore, meteo file extensions are given with METEOEXT, grid2d extensions with GRID2DEXT, grid2d prefix with GRID2DPREFIX (not all plugins support them, though), etc

A bug was fixed in SMETIO: when given OLWR, it should have computed TSS, but it was not... (using the wrong MeteoData object).

Documentation and error messages improvements. A new method was added to Atmosphere: virtualTemperatureFactor.

Performance boost: Redesign of class MeteoData. The individual public double members like tss, ta, rh have been removed and instead of the rather complicated internal representation with the help of std::map and pointers, the MeteoData object now holds all its important information in one double vector (meteo data) and one string vector (parameter names). Thus the overloaded copy constructor could be slashed radically. This leads to huge performance gains, especially when dealing with huge vectors of MeteoData (which are needed for filtering, resampling, etc).

Furthermore the access to the meteo parameters has been simpleified by overloading the operator(), replacing the old .param(size_t index) and .param(string paramname) functions:

MeteoData md;
double ta = md(MeteoData::TA);

OR

double ta = md("TA");

These changes were propagated into all of MeteoIO.

The OLWR hack for Norway has been implemented in SMETIO (a cleaner, long term solution will still have to be developed). Various types issues have been identified by Visual C++ and should have been fixed.

Some methods have been moved into another class (like the simple linear interpolation between two points that is now in ResamplingAlgorithms), some renammed (like the new weightedMean that replaces the ill-nammed "linearInterpolate"). The regression model based on two segments is now part of libinterpol1D and has been renammed. More regression models have been implemented. 

The "initFit" method of Fit1D has been renammed in "fit" and the setGuess method now turns fit_ready to true, so that a user can force his provided parameters to be used by a given model (do a setGuess() with the parameters, then call f(x) without calling "fit", so no internal fit of the parameters would be performed).