Commits · 6bf1578ef51a6fbfc458af5306d66dc7f1cd84f6

Two new members have been added to Grid3DObject, to store the heights of the 3D levels (ie: level 1 = 10m above ground, etc). This is similar as what is done by ARPS, GRIB, etc A boolean specifies if these are absolute or relative heights (ie: above DEM).

The unventilated temperatures filter and unheated rain gauge filter have been renamed. The "soft" option has been added to Unheated_RainGauge. And the usual documentation fixes...

The smoothing filters now behave differently depending on the centering that has been specified. The IOManager backreference for Meteo2DInterpolator (pointer this given in the initialization list) has been replaced by a setter.

An invalid flag for Visual C++ has been fixed.

Fixed more warnings, found by Code::Blocks. A bug when compiling gsoap files with migw has been fixed by a workaround (adding a macro definition in the CMakeLists.txt).

Better error messages in IMISIO, better safety (regarding "undef") in Date and some code cleanup/const/reserve business (all benchmarked) for the others. We also now avoid stacking two "/" at the end of the PLUGINPATH if one is already present.

oups, fixed an include that had to be removed... By the way, the last changes lead to good compiler optimizations: in optimized mode, on 1 year of data, the speed up is above 30%

The last filters that had not been ported to the new filtering infrastructure have been ported and their documentation updated. All filters now use window specifications (instead of vector of pointers). The misleading filter argument "index" (for parameter type) has been renamed as "param" for all filters. The use of "unsigned int" for looping over vectors has been replaced by "size_t". The old "soft" behavior for the filters has been restored (ie: if not soft, a point that can not be checked by the filter is deleted).

A few more "consts" have been added.

All the filters using windows have been modified so they use the window specifications (start, end) instead of a vector of pointers. The FilterWindAvg was not properly handling nodata, this has been fixed. By the way, when using a reasonnably large buffer size, the new filter window management is actually slightly faster than the old one (1%). 

The handling of "soft" is now different: soft ONLY applies to the window specifications. Before, a soft filter would leave values that could not get a window untouched while non-soft would force them to nodata. Now, all values that can not get a window are left unchanged, while the window centering can change according to "soft".

This is the new implementation of the filters' windowing scheme. Now, specifying 4 points and 6 hours will guarantee that at least 4 points are in the window AND 6 hours, otherwise it returns unsuccessfully. The run time penalty is ~4% compared to the previous implementation. Since it first builds the window specifications (start and end index), it is possible to directly call this window specification, skipping the building of a vector of pointers. For the MeanAvg filter, this makes the run time penalty go down to 3.3% on a 6 hours window (with hourly data).

First correct implementation of the filtering windows. This is not active yet as it only provides window specifications. But it properly fulfills both the number of points criteria and the time span criteria.

Fixed a division by zero in libsmet and the "none" resampling was broken (it was deleting all data instead of propagating available data). Fixed!

The "passive_T" filter had a bug in its implementation, this has been fixed (using the wrong statistical fit). It should still be tested before operational use, though...

Adding a virtual deconstructor to Array2D makes sure that we can inherit from Array2D without any issues!

Investigating issue 210, it has been found that the IMIS precipitation had a division by 2 that was not required (halving the precipitation compared to what it should have been). This has been fixed.

Small documentation improvement in ResamplingAlgorithms.

A new option has been added to the accumulate interpolation: "strict". When this is active, if one of the accumulated value is nodata, the whole sum will be set to nodata. Without this option, all values that can be aggregated are aggregated (ie: best effort). When using multiple stations, it might make sense to enable "strict" in order to avoid polluting the spatial interpolation with potentially under-estimated values.

A documentation issue has been fixed in Meteo1DInterpolator.

The re-accumulation should now work properly for down sampling (tested with data at h+40 converted to h+00 and with 10min data converted to hourly).

Fixed some warnings (about unused variables/parameters). Added a new position for ResamplingAlgorithms: begin to describe that a point is at the begining of the data set (so we don't have to look for this condition again in the algorithm).

New Resampling: cleanup of IOManager.cc: now also the rare case of IOManager::raw | IOManager::resampled is correctly handled.

Bugfix: The new linear resampling always extrapolated at the end of the meteo data vector, this is fixed now.

TODO: verify and comment

Further improvements for the resampling within the Meteo1DInterpolator: no longer is an element inserted into the filtered_cache and then deleted, instead we have a MeteoData reference handed to the Interpolator representing the MeteoData element to be resampled. This interface change needed to be propagated into the different affected classes. Accumulate resampling currently disabled.

Speedup in the resampling of MeteoIO, instead of always copying a resampling window we are now directly resampling on the filtered_cache. This leads to an enormous perfomance gain (slashing runtime for reading meteo data to fifth), because the cutting of the resampling window was quite costly.

The re-accumulation was broken and producing bogus results. Moreover, its handling of nodata was sub-optimal (assuming nodata=zero lead to artificially low measurements for very noisy stations). This has been fixed. Accumulating over periods larger than in the original data set (ie:up sampling) has been validated. Down sampling has NOT been validated yet.

Checked the Passive_T filter on real data, added more flexibility to it (ie: it can compute the albedo using iswr and rswr, compute rswr from iswr and an albedo, etc) and fixed some mistakes. It now gives realistic results but a proper validation (ie: comparing with high quality, ventilated measurements) is still required.

The Passive_T filter has been reworked and renamed as ProcPassiveT. A proper documentation has been written for it.

All the cout have been replaced by cerr in order to avoid messing with the end user's application. Some old commented out code has been removed.

Outputs that were used for warnings (or information) are now done on cerr instead of cout, in order to avoid poluting the outputs of user code. Should we do the same for the exceptions?

A new method, getSolarTime has been added to SunTrajectory.

Exposing the add_to_cache function globally, this way the user can push data into the IOManager that shall be used for a certain date, bypassing resampling and filtering of the IOManager.

Documentation, default value for WINDOW_SIZE=10 days, small speed optimizations (1% faster!!), code cleanup to prepare the way for re-accumulation work

The resampling has been deeply transformed in order to produce more consistent results. By doubling the data cut window, the resampling can now properly fill a 23 hours gap with a 24 hours window. The resampling now either fills the whole gap or do not fill anything (if the gap is too large). When activating "extrapolate", a gap is now filled by the sides instead of by the middle (which was producing weird results). The drawback is that the cut window had to be doubled, which is quite costly (going from 1 day to 10 days makes it 7.5 times slower, and going to 20 days makes it 15 times slower). However, this more logical and consistent behavior will allow the user to specify the window size that he needs instead of always picking something extremely large to be on the safe side. Finally, the resampling code has been cleaned up so that each algorithm is built along similar lines.

A new getMeteo() method has been implemented that transparently performs spatial interpolations and returns a grid for a given parameter. This should be prefered to interpolate(), since it allows the reading plugin to directly provide gridded data (such as coming out of GRIB).
Code cleanup in ResamplingAlgorithms, improved error message in A3DIO. Several keys have been updated/added to the plugins, in order to be more consistent between plugins. Therefore, meteo file extensions are given with METEOEXT, grid2d extensions with GRID2DEXT, grid2d prefix with GRID2DPREFIX (not all plugins support them, though), etc

After auditing the code with Flawfinder version 1.27, a few things have been fixed: IOUtils::getLogName() now truncates the log name to 64 characters maximum (since it was stored as a C++ string, it should not have been a problem but converting back as string.c_str() could have created problems in the user code in case of very large strings (ie: malicious user trying to break the system) in environment variables. In Coords, the epsg code is checked for range before casting to short int, in order to throw an exception at the proper place (and avoid a possibly corrupted return value that would have surprised the end user in the next exception). A more serious issue has been fixed in Coords::parseLatLon() where a malicious coordinate string could have overflowed on the stack.

The plugins general documentation has been much improved (list of data sources categories, table of what is provided by each plugin, better description and extra requirements)