GitLab

Now the ISO year can be returned alongside the ISO week number. The ISO_WEEK print format properly uses it, making its output standard compliant. The issue 466 can now be closed!

A new date format has been added: RFC868. Two MeteoGrids have been renamed: ISW_DIR as ISWR_DIR and ISW_DIFF as ISWR_DIFF while a new grid has been declared (P_SEA). A few new constants have been declared. Some new links into the GRIB documentation as well as support for TAU_CLD. The NetCDF support is much improved, although there are still things left to do (writing grids out has been disabled for the moment, the parsing of the time units is currently not done, there is no support for automatically getting the data out of multiple files in a directory, etc).

A new Date print format has been added (ISO_DAY). The "daily average" resampling now properly works with avg, avg+min or avg+max using the user provided range as a fallback.

It is now possible to write timestamps as ISO_WEEK (this is YYYY-Www-dd). This is now used for the reports generated by DATA_QA in order to ease data aggregation. The GSN plugin has a new key: GSN_TIMEOUT. A bug in error handling was fixed in the GSN plugin.

The Date class can now compute the dow of week as well as the ISO week number. Fixed a documentation typo in FindMySQL.cmake

Finally adressing issue 72 with second resolution in dates. The ISO strings now contain the seconds when written out and by reading them, the seconds can be provided (but this is not mandatory). In ARC, grids are now named with the seconds (except when enabling the Alpine3D naming scheme). The adresses a bug that was found when requesting" raw" processing level from a plugin that constructs its timestamps from a decomposed time: time-1s and time+1s were the same and therefore this could trigger problems in the plugin.

A new Date::getTime() method has been written. A better error message has been implemented for ProcHNWDistribute. The daily_solar temporal interpolation can now handle daily sums provided at midnight (then assuming they belong to the day that just finished) and the documentation has been updated.

better file structure: all the classes purely focused on data storage (such as grids, dates, etc) have been moved into a "dataClasses" directory. All subdirectories are nhow named similarly to the classes and methods: the words after the first one are capitalized.

Since IOUtils was getting bigger and bigger, all the methods dealing with low level file handling have been moved into a new file, FilesUtils. It remains in the IOUtils namespace. One method has been moved to Date. A few includes have been cleaned up.

Finally, another method for handling daily sums of solar radiation has been implemented: when a point is requested, the potential solar radiation is computed for the whole day with a 20 minutes resolution, filling a vector and computing the daily sum. This enables to compute the loss factor. When another point within the same day is requested, the radiation is interpolated from the values stored in the vector and multiplied by the loss factor. Tested on WFJ data, this works very well (of course, days that experience large cloudiness variations are not reproduced well but at least the average flux is correct).

A typo has been fixed in the documentation of Date. 

In order to implement an efficient and portable object serialization, it has been decided that all serializable objects would be manually serialized to an std::iostream. Then, the parallelization framework (mpi, popc, etc) can take over this stream and move it to another node. Therefore, the operators "<<" and ">>" are now reserved for serialization (as seems to be standard) and the debug outputs are obtained by printing "object.toString()". This means that all debug code has potentially to be fixed, by replacing "std::cout << object" by "std::cout << object.toString()".

A rounding bug has been found in the Date class: when using a Date object in a loop where it gets incremented, it accumulates errors and after ~70000 iterations, the error would get bigger than one second, making date comparisons with a fixed date impossible. This has been solved by always rounding the internal gmt Julian date to the closest second, after each assignment (from the constructors, the setDate() calls or any arithmetic operation). Moreover, the rounding methods have been re-written to be more efficient as well as other roundings (specially when computing the date decomposition from Julian). 

The seek() call performing a binary search has also been simplified, in order to remove unnecessary tests, put some similar tests together and make the code clearer. 

Some constification and better vector usage took place in ResamplingAlgorithms and Meteo1DInterpolator.

In order to try to be closer to the spirit of the STL, now the empty strings/vectors/lists are tested with .empty(), the first/last elements of vectors accessed with front()/back(), the empty strings are not contructed with string("") but with string(), etc This makes a benchmark based on meteo_reading ~4% faster and this makes the code suprisingly easier to read. 

Some extra constification took place as well as some minor code cleanup (optimizing the comparison between two vectors in order to perform less tests, etc).

The Date class can now properly generate and parse full ISO timestamps, including time zone information. The full specification is now supported by convertString() with a helper method in the Date class. Ouputing an ISO string with its time zone is done by requesting the ISO_TZ format.

The Date class has been revereted in order to compute year, month, day, etc since this is useful when debugging. The external includes that are used for the plugins are now included as system headers in order to avoid generating warnings for some code that we are not responsible for.

Removing some costly but unnecessary features (ie there is no need to always keep gmt_year, etc up to date when we in any case never use the gmt values!). The gain is ~5% on the speed of data preprocessing in Snowpack.

Some extra documentation has been added for the UPS coordinate system. The coordinates test has been expanded. A missing default value has been added to the Suntrajectory.

New warnings have been tried to detect variable shadowing (one variable declared with the same name as another one), alignement issues, etc. This produces very verbose output (not all warnings are legitimate) but allowed to fix lots of cases of misalignement and shadowing issues. These options have been left in the CMakeLists but NOT enabled (since these warnings are not so easy to interpret).

A cleaner way of implementing the Date optimization has been implemented (relying on a define to switch back to the previous implementation if necessary). The compilation of the GSN plugin is now turned on by default. The version numbers have been pushed to 2.2.0 in order to get ready for the release.

A few "consts" have been added. The dates are now considered to be positive which allows comparison operators to be optimized: converting 1 year of hourly data with data_converter got almost 30% faster. Moreover, when looking for resampling window, two extra checks have been added to quickly see if the start and end point are even in the input vector. This reduces data read miss (because we start by reading vecM[0]) and skips browsing through the whole vector if we are in a large data gap. On the same 1 year data set with one large gap for one parameter, the gain is of almost 20%. Overall, these two optimizations lead to almost a doubling of the speed with data_converter... Soon we'll be so fast that we will get the data before it is even measured!

The POPC compilation is slowly improving... One still needs to force the compiler to popcc, but otherwise it seems to work. Necessary includes have been added when popc is activated. A cmake method for finding popc has been implemented.

Small optimization/clean up in A3DIO for special points reading.

Performance boost: Redesign of class MeteoData. The individual public double members like tss, ta, rh have been removed and instead of the rather complicated internal representation with the help of std::map and pointers, the MeteoData object now holds all its important information in one double vector (meteo data) and one string vector (parameter names). Thus the overloaded copy constructor could be slashed radically. This leads to huge performance gains, especially when dealing with huge vectors of MeteoData (which are needed for filtering, resampling, etc).

Furthermore the access to the meteo parameters has been simpleified by overloading the operator(), replacing the old .param(size_t index) and .param(string paramname) functions:

MeteoData md;
double ta = md(MeteoData::TA);

OR

double ta = md("TA");

These changes were propagated into all of MeteoIO.

Several HACKS have been removed (either the HACK had been fixed but the comment was still there, or it has now been fixed, or after more (careful) consideration, it has been decided that the HACK was not a HACK)

A (more) intelligent handling of buffered grids has been implemented: the grids are stored in the equivalent of a circular buffer of a given size (user defined, or 10 by default). This prevents running out of memory when processing large numbers of grids!

A method for computing quantiles has been implemented in libinterpol1D. A vector of data has to be provided as well as a vector of the desired quantiles (ex: 0.25, 0.5, 0.75) and a vector of values for these quantiles will be returned. So far, the algorithm that is used produces the same results as R in default operations. The possibility of choosing which quantiles algorithms might be added in the future.

The getJulianDayNumber method was just totally wrong... This has been fixed. One can also force a GMT day of year with a flag.

Finally, some documentation has been written/updated.

The POPC version has been heavily transformed. Since we would need IOManager has a popc object and this is quite hard (we need a backreference in Meteo2DInterpolator), all parclasses have been removed. Instead, the user will have the responsability to packaged IOManager in his own parclass if needed. This makes the code simpler and skips difficult issues...

A new fill method has been added to the Array2D and Array3D templates that does the reverse of subset (ie: reassemble arrays).

Better error messages have been put in SNIO.

A new Date setter that takes a Date object has been created.

This version does NOT correctly compile with the autotools (it leads to a segfault when running code using MeteoIO), but the cmake compiled library works fine.

First batch of fixes for issue 68 (variable names starting with "_" invalid for some versions of Visual C++)

More fixes for Visual C++ (but not all done yet) as well as new assignments operators for 1d, 2d, 3d arrays. It is now possible to do something like my2Dgrid.grid2D = 2.; and every cell will be filled with 2.

In SNIO, the METAFILE can now be optional (which means that the metadata remain nodata, the user has therefore to fill them himself). This is used by Snowpack.

After the Visual C++ compilation log from Raphael, some changes have been implemented: lots of warnigs about signed/unsigned fixed, some unreachable code removed, and some methods reimplemented for Windows. We are not there yet, but moving closer...

Fixed a bug when doing dates additions and substractions (the += and -= operators were fine). Fixed the copy constructor so that it can copy an undefined date. Added the support for Matlab dates.

Another commit related to timezones... The Unix date always being in UTC, this has been fixed. The resampling was not using dates in a proper way, this has been fixed. The GSN plugin has also been fixed, it now rounds time to the closest minute (in order to be consistent with our advertised "1 minute time resolution"). A rounding method has been implemented for dates.

The timezone parameter in io.ini is now named TIME_ZONE instead of TZ for clarity. 

Undefined dates should now be fully handled (ie: also in arithmetic).

- Date.*: allow to handle undefined Dates (== 0. for now), add new members undef, setUndef(), isUndef(), and rndJulianDate()
- IOUtils.*: replaced TZ with time_zone and TIME_ZONE
- ImisIO.*: correct typo seperateDrift to separateDrift

The full support for timezones has been brought back. This fixes issue 71. The SMET, SMET, IMIS, BORMA plugins have been validated. Geotop should be validtaed if an input data set is available. A new class will have to be implemented to represent time periods. This is currently only a typedef pointing to the Date class, but a specific class would be welcomed.

GSN has been fixed for some changes on the web service, but does not retrieve the correct timestamps (is this a GSN issue?). The GSN compilation has been improved: the warnings related to GSOAP have been removed. 

A new long wave radiaiton model using cloudiness as input has been added (Omstedt, 1990). 

The (advanced) configuration of the BufferedIOHandler through io.ini (General section) has been added. It currently introduces two keywords (BUFF_CHUNKS_SIZE and BUFF_CHUNKS) to drive how much data to read at once and how much data to buffer. Currently, their product is directly used in the read, thus not reading anything by chunks smaller than a full buffer.

Fixed some << outputs, two compilation warnings found on Hera as well as introduced.

A new group of classes has been created, under the meteolaws subdirectory. This contains various general meteorological laws, such as a sun radiation model, standard atmosphere, etc

The first implementation (not tested yet) of a non-linear least square fit algorithm has been implemented in the libfit1D. 

Several documentation issues have been fixed (obsolete code examples, etc) and classes grouped by modules. This should help the user to quickly locate the classes of interest. 

Added new plugin for WSMDF format. gzipped version of format currently not supported. binary version works, but since the julian date does not fit into a 4 byte (32bit) float, it is unusable. ASCII format works, although unit_multiplier and unit_offset are currently not supported. For location information the easting, northing version is not yet supported.

Added function IOUtils::stripComments(string&) it, deletes everything after # or ; from the string&