AREkit-0.25.0

Release notes

Full Changelog: v0.24.0-rc...v0.25.0-rc

Support Batching for effecting imputing LLM into text processing pipelines

Previosly, the whole text processing pipeline was relying on the sentence / text part.
Now we overcome that liimitation and therefore we can consider multiple sentences, formed in list i.e. batch.
This step is so important for LLM, LM, neural networks, for which batching accelerates the performance.
As the result, overall pipeline launching is expected to perform faster.

• #539 🔥
• #535
• #540
• #541

Sources collections are no longer going to be a part of AREkit ✨
Tha allow us to lightweight 🪶 the overall framework and so that purely focus on data processing techniques

• #537
• Remove requests library dependency 🪶
• Move all the tutorials 📚 to the AREkit-ss project. 🪶

Flexibility and Performance Enhancements

• #539 🔥
• #535
• #537
• nicolay-r/arekit-ss#80 (Issue has been moved to arekit-ss project)
  • #511
• #538
• #542 🔧

Fixed bugs

• 🔧 RowCacheStorageProvider fixed bug with mismatching size of type list and columns list in case of other force collected columns (ad4312c)

Minor Updates

• ❌ Removed OpinionsIO (76b4c1f)
• ❌ Removed suffix -0 in filenames for samples. (76b4c1f)
• ❌ #543
• ❌ #544
• ❌ #547

Minor

• #135 (No longer available)
• Appropriate formatting of unit tests (https://github.com/nicolay-r/ARElight/blob/main/test/test_translation.py)
• 🔧 #137
• 🔧 #138 (No longer available)

Changeset

Implemented enhancements:

• SamplesIO.create_target -- provide this parameter as function [ARElight backlog] #547
• No input support for pipelines Launcher #546
• _get_text is no longer needed #544
• TermsSplitterParser -- is no longer required [ARElight backlog] #543
• Partitioning -- fancy last operations of the SentenceObjectsParserPipelineItem which has no longer application [ARElight backlog] #542
• SentenceObjectsParserPipelineItem -- rename to the ObjectsParserPipelineItem concept #541
• Pipelines -- refactoring core concept, source customization selection for ppl items #539
• Pipelines -- Batching sentences in document parser [ARElight backlog] #535
• Graph-based sampler #495

Closed issues:

• Provide link to the DEMO ARElight as a technical reference documentation #549
• Pipeline.run might be just a concept of launchers, there is no need to combine storage of items with run operation #540
• ➕ SQlite-based readers and storage providers #538
• Sources Movement in AREkit-ss [including the related dependencies] #537

* This Changelog was automatically generated by github_changelog_generator