v3.6.0

What's Changed

👓 Version Header Added

From this version on, the version of the LDES Server is added as a header to all the HTTP responses

🧹 Linear Caching has been completely cleaned up

Since 3.4.0, linear timebased caching was broken. The decision has been made not te repair this feature, but to completely remove it from the code base

🚣 Kafka Ingestion

An event stream can now be configured in such a way that it can listen to a Kafka topic for ingesting members

🎌 Prefixes added to RDF models is now more efficient

In older versions, prefixes were extracted from the data itself, but this took too long and so from now on, a dozen of well known prefixes are added in the code. In addition, you can add prefixes to the server self by configuring this through the application.yml file. In addition, prefixes for the event stream and view uri's are added to the RDF model as well.

🖨️ Hibernate deep copy functionality ignored

Hibernate was making a deep copy of the tree node entities, but that deep copy would never be used in the code base. Therefore, a Spring JPA projection is now used for fetching the tree nodes.

🛑 Clearer HTTP status response on event source update

Now a 404 http status is returned when trying to update the event source of a non-existing event stream

☑️ SHACL validation now checks more profound on retention policies

When adding a view, the SHACL validation will now check if only known retention policies are present in the list

😮‍💨 Unreachable DCAT dataset code removed

Code was found when adding a dataset through the admin API that could never be reached, and therefore this code has been removed

🩹 Some fragmetation bugs are fixed

Dependencies

• chore(deps): bump webrick from 1.8.1 to 1.8.2 in /docs (#1397)
• chore(deps): bump rexml from 3.2.5 to 3.3.9 in /docs (#1406)
• chore: bump zonky embedded db

Full Changelog: v3.5.1...v3.6.0

v3.5.1

What's Changed

🩹 startup failure

A bug was detected in 3.5.0, when an event stream was created in 3.4.0 and no members were ingested yet, a row in a newly created table in the database was missing, which led to the application crashing on startup. This has been fixed in #1416

Full Changelog: v3.5.0...v3.5.1

v3.5.0

What's Changed

🔧 Configurable Delimiter on State Object ingestion

As previously available in the LDI Version Object creator, It is now possible to configure a custom version delimiter when ingesting state objects. For more details, check out the documentation.

📑 Maintenance Jobs, Compaction + Retention + Deletion combined

As a cleanup effort and to provide more transparency, the compaction, retention and deletion jobs have been merged into a maintenance job. The biggest user impact is the simplified config through the maintenance cron found in the config table.

📚 Modular restructure

Although this has no user impact, the module structure of the server got a revamp.
Allowing for custom images with unique functionalities and a smaller image size.
For more details, check out the project Readme

Change notes:

• fix: broken compaction due to used uuids by @jobulcke in #1373
• chore: restructuring to maintenance by @jobulcke in #1375
• feat: simplify maintenance by @jobulcke in #1374
• chore(deps): bump org.owasp:dependency-check-maven from 9.0.9 to 10.0.4 by @dependabot in #1385
• chore(deps): bump com.google.guava:guava from 33.0.0-jre to 33.3.1-jre by @dependabot in #1387
• chore(deps): bump org.springframework.data:spring-data-commons from 3.2.3 to 3.3.4 by @dependabot in #1388
• fix: flaky test by @jobulcke in #1381
• chore: db cleanup by @jobulcke in #1382
• feat: charset included in headers by @jobulcke in #1380
• fix: broken compaction rework by @jobulcke in #1383
• chore: fix view stats by @rorlic in #1396
• hotfix: dropped member reader perfo by @jobulcke in #1394
• feat: drop unnecessary db check on ingest by @jobulcke in #1399
• feat: modular restructure by @Yalz in #1400
• fix: missing page members by @Yalz in #1401
• chore: bump jena to next major version by @jobulcke in #1402
• feat: relax timebased fragmentation by @jobulcke in #1403
• docs: update project structure in docs by @Yalz in #1404
• Fix/broken image by @rorlic in #1407
• feat: configurable version delimiter by @jobulcke in #1405

Full Changelog: v3.4.0...v3.5.0

v3.4.0

What's Changed

📊 Fragmentation Optimalisation

Under the hood, the fragmentation processes (bucketisation + pagination) went through a big structural change to optimise database interactions.
Additionally, database views and tables have been optimised in function of this.

🚧 Known Issues

Timebased Linear Caching

Whilst preparing for this release, we noticed that the timebased linear caching feature was causing troubles in our current write process to the database. Therefore, it has been disabled for now, but we will try to enable it again as soon as possible

Change Notes

• feat: pagination rework by @Yalz in #1369
• feat: move all bucket db queries to ItemWriter by @jobulcke in #1370

Full Changelog: v3.3.0...v3.4.0

v3.3.0

What's Changed

Fragmentation revamp: Bucketisation & Pagination 📇

Although no breaking changes on the front end have been introduced, you might see in the logs that there is one fragmentation job that will run periodically.
This job will, for each view, first define the buckets in the bucketisation step. After that, the pagination step will assign the bucketised members to pages.
This was done in terms of performance to decouple the ingestion process from fragmentation.
Note that further releases will improve on performance of these processes.

Fragmentation will check every 30 seconds for unprocessed members.
If you wish to use a specific interval, update the ldes-server.fragmentation-cron spring value with a valid spring cron.

Fragmentation metrics 📈

In terms of observability, we also provide an update for all you Prometheus fans!
You will now have 3 metrics to follow up the internal processes of your LDES Server:

• ldes_server_ingested_members_count : Will keep track of the ingested members per collection.
• ldes_server_bucket_members_count : Will keep track of all bucketised members per view.
• ldes_server_pagination_members_count : Will keep track of all paginated members per view.

Skolemization ♻️

As per request of one of our onboarders, we introduced skolemization.
This feature allows the LDES Server to replace blank nodes with skolemized URIs.
For more details, check out the new documentation

Known Issues 🚧

Compaction vs Fragmentation

Whilst preparing for this release, we noticed how the newly created compaction URIs conflict with our pagination.
This issue is being tracked as part of #1365
We advice you to turn off compaction. This can be done by setting the compaction cron to -.

Change Notes

• chore: finalise 3.2.0 by @Yalz in #1332
• doc: update event source documentation by @jeflaenen in #1333
• Use latest cache actions by @rorlic in #1345
• chore: #1346: auto enable Spring Batch table creation + filter out ac… by @Yalz in #1349
• feat: skolemization by @jobulcke in #1351
• chore: improve ci pipeline by @Yalz in #1352
• feat: data model: retention + fragmentation by @Yalz in #1343
• fix: missing skolemization domain shacl uri validation by @jobulcke in #1354
• feat: include expanded metrics + general fragmentation fixes by @Yalz in #1356
• fix compaction by @Yalz in #1358
• feat: run fragmentation jobs per view in parallel by @Yalz in #1359
• fix: broken needs_pagination db view by @jobulcke in #1360
• chore: make fragmentation jobs parallel by @Yalz in #1361
• chore: removed incorrect unique constraint and added some indexes by @Yalz in #1362
• chore: finalise 3.3.0 release by @Yalz in #1363
• docs: skolemization updates by @Yalz in #1367

New Contributors

• @rorlic made their first contribution in #1345

Full Changelog: v3.0.0...v3.3.0

3.2.0-alpha

What's Changed

Resolving memory issue in Pagination

Due to an issue in pagination in version 3.1.0, the memory kept increasing in long runs.
This has now been resolved. Why therefor highly advise users to move to this release as soon as possible.

Under the hood: Database model changes

For those who might already have noticed, a new database model is showing itself between our releases.
This is with the goal of having a more performant solution.

Once all parts of the database are migrated, a full overview of the new database will be provided for those interested

Change Notes

• feat: new data model ingest by @pj-cegeka in #1316
• feat: less db calls for event stream children by @jobulcke in #1320
• chore: retired mongo references cleanup by @jobulcke in #1315
• doc: change of metrics URL by @jeflaenen in #1326
• fix: query optimisation + async events by @Yalz in #1324
• feat: async event handling by @Yalz in #1328
• fix: decrease load on bucketisation/pagination by @Yalz in #1329
• fix: memory impact by @Yalz in #1330
• Fix: restart with closed collection by @pj-cegeka in #1305

Full Changelog: v3.1.0-alpha...3.2.0-alpha

v3.1.0 alpha

What's Changed

Liquibase introduction

From the 3.1.0 release on, the LDES server contains a migration path between each release.
No extra configuration is needed, all needed tables and migrations will be done on the fly.

Fragmentation rework (batching)

The fragmentation got a big rework to focus towards a more performant LDES Server.
The 2 processes (bucketisation and pagination) are now split up:

• Bucketisation: Based on the configured fragmentation strategy and a given LDES member, the member can be assinged to a number of buckets. This will be reflected in the relations in the LDES fragments.
• Pagination: For each of the assigned buckets, the members will be paginated. This happens based on the configured memberCount per page.

These processes will now work with a time interval to process more members at the same time.

Geospatial fragmentation update

Previously, there was an issue in the geospatial fragmentation where only the tiles that contains the points of a shape were taken.
In the below example, A3 and D3 were taken as tiles of a geospatial fragmentation, but the others were ignored.
From now on, also tiles B3, C2 and C3 are taken.

Closing of an LDES

If, as a dataowner, you know no more data will be ingested in an Event Stream, there is now the ability to close the stream (turns the latest page immutable).
This gives the advantage that the LDES Client can stop following the LDES without needlessly polling for updates that will never come.
To close an LDES, please visit the swagger endpoint.

Change notes

• f:eat: split pagination and fragmentation by @pj-cegeka in #1294
• fix: add liquibase profile by @pj-cegeka in #1297
• feat: VSDSPUB-1283: Close LDES by @ferre-vaes in #1296
• fix: adapt ldes closure endpoint by @ferre-vaes in #1304
• feat: calculate intersecting tiles of geo feature by @ferre-vaes in #1309
• feat: fragmentation batching by @Yalz in #1311
• feat: new data model definition for event streams by @jobulcke in #1307
• fix: invalid documentation by @jobulcke in #1312
• fix: code smells by @jobulcke in #1317
• chore: finalise 3.1.0 by @Yalz in #1318

Full Changelog: v3.0.0-alpha...v3.1.0-alpha

v3.0.0

What's Changed

Postgres implementation

To deliver a more performant and transactional experience, the LDES Server moves from its trusted mongodb database
to a PostgreSQL database.
This however is only the first step in a peformance focus and more updates on the database structure might come.

For more info on how to upgrade, check out the configuration guide.

Change notes

• feat: add eventSource to admin api by @pj-cegeka in #1281
• feat: postgres implementation by @Yalz in #1289
• Fix: eventsource delete view postgres by @pj-cegeka in #1290
• chore: finalise 3.0.0 by @Yalz in #1292

Full Changelog: v2.15.0-alpha...v3.0.0-alpha

v2.15.0-alpha: Merge pull request #1282 from Informatievlaanderen/develop

What's Changed

• fix: attach link to Hulp Nodig button (#1277)
• Feat: fetch streaming LDES fragment (#1272)
• fix: invalid swagger ui in docs (#1273)
• feat: cleanup actuator hostname customizer (#1269)
• docs: reduce documentation padding (#1268)

Full Changelog: v2.14.0-alpha...v2.15.0-alpha

v2.14.0-alpha

What's Changed

• fix: cleanup test resource files in de ldes-server-admin module in #1257
• fix: slow retention insert query in #1263
• feat: member model extractor performance increase in #1264
• fix: mongock order failure in #1266

Full Changelog: v2.13.0-alpha...v2.14.0-alpha