generated from cerbero90/skeleton
-
-
Notifications
You must be signed in to change notification settings - Fork 2
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
- Loading branch information
Showing
45 changed files
with
496 additions
and
1,915 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -1 +1,2 @@ | ||
github: cerbero90 | ||
ko_fi: cerbero90 |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -4,3 +4,4 @@ vendor | |
phpcs.xml | ||
phpunit.xml | ||
.phpunit.result.cache | ||
.DS_Store |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -1,4 +1,6 @@ | ||
build: | ||
environment: | ||
php: 8.2 | ||
nodes: | ||
analysis: | ||
project_setup: | ||
|
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -2,45 +2,44 @@ | |
|
||
[![Author][ico-author]][link-author] | ||
[![PHP Version][ico-php]][link-php] | ||
[![Laravel Version][ico-laravel]][link-laravel] | ||
[![Octane Compatibility][ico-octane]][link-octane] | ||
[![Build Status][ico-actions]][link-actions] | ||
[![Coverage Status][ico-scrutinizer]][link-scrutinizer] | ||
[![Quality Score][ico-code-quality]][link-code-quality] | ||
[![PHPStan Level][ico-phpstan]][link-phpstan] | ||
[![Latest Version][ico-version]][link-packagist] | ||
[![Software License][ico-license]](LICENSE.md) | ||
[![PSR-7][ico-psr7]][link-psr7] | ||
[![PSR-12][ico-psr12]][link-psr12] | ||
[![PER][ico-per]][link-per] | ||
[![Total Downloads][ico-downloads]][link-downloads] | ||
|
||
Framework agnostic package using asynchronous HTTP requests and generators to load paginated items of JSON APIs into [Laravel lazy collections](https://laravel.com/docs/collections#lazy-collections). | ||
```php | ||
$lazyCollection = LazyCollection::fromJsonPages($source, fn (Config $config) => $config | ||
->dot('data.results') | ||
->pages('total_pages') | ||
->perPage(500, 'page_size') | ||
->chunk(3) | ||
->timeout(15) | ||
->attempts(5) | ||
->backoff(fn (int $attempt) => $attempt ** 2 * 100)); | ||
``` | ||
|
||
Need to load heavy JSON with no pagination? Consider using [Lazy JSON](https://github.com/cerbero90/lazy-json) instead. | ||
Framework-agnostic package to load items from any paginated JSON API into a [Laravel lazy collection](https://laravel.com/docs/collections#lazy-collections) via async HTTP requests. | ||
|
||
Need to read large JSON with no pagination in a memory-efficient way? Consider using [🐼 Lazy JSON](https://github.com/cerbero90/lazy-json) or [🧩 JSON Parser](https://github.com/cerbero90/json-parser) instead. | ||
|
||
## Install | ||
## 📦 Install | ||
|
||
In a Laravel application, all you need to do is requiring the package: | ||
Via Composer: | ||
|
||
``` bash | ||
composer require cerbero/lazy-json-pages | ||
``` | ||
|
||
Otherwise, you also need to register the lazy collection macro: | ||
## 🔮 Usage | ||
|
||
``` php | ||
use Cerbero\LazyJsonPages\Macro; | ||
use Illuminate\Support\LazyCollection; | ||
|
||
LazyCollection::macro('fromJsonPages', new Macro()); | ||
``` | ||
|
||
## Usage | ||
|
||
- [Length-aware paginations](#length-aware-paginations) | ||
- [Cursor and next-page paginations](#cursor-and-next-page-paginations) | ||
- [Fine-tuning the pages fetching process](#fine-tuning-the-pages-fetching-process) | ||
- [Handling errors](#handling-errors) | ||
- [📏 Length-aware paginations](#-length-aware-paginations) | ||
- [↪️ Cursor and next-page paginations](#-cursor-and-next-page-paginations) | ||
- [🛠 Requests fine-tuning](#-requests-fine-tuning) | ||
- [💢 Errors handling](#-errors-handling) | ||
|
||
Loading paginated items of JSON APIs into a lazy collection is possible by calling the collection itself or the included helper: | ||
|
||
|
@@ -109,7 +108,7 @@ lazyJsonPages($source, $path, function (Config $config) { | |
The configuration depends on the type of pagination. Various paginations are supported, including length-aware and cursor paginations. | ||
|
||
|
||
### Length-aware paginations | ||
### 📏 Length-aware paginations | ||
|
||
The term "length-aware" indicates all paginations that show at least one of the following numbers: | ||
- the total number of pages | ||
|
@@ -198,7 +197,7 @@ $config->lastPage('last_page_key')->perPage(500, 'page_size'); | |
``` | ||
|
||
|
||
### Cursor and next-page paginations | ||
### ↪️ Cursor and next-page paginations | ||
|
||
Some APIs show only the number or cursor of the next page in all pages. We can tackle this kind of pagination by indicating the JSON key holding the next page: | ||
|
||
|
@@ -209,7 +208,7 @@ $config->nextPage('next_page_key'); | |
The JSON key may hold a number, a cursor or a URL, Lazy JSON Pages supports all of them. | ||
|
||
|
||
### Fine-tuning the pages fetching process | ||
### 🛠 Requests fine-tuning | ||
|
||
Lazy JSON Pages provides a number of settings to adjust the way HTTP requests are sent to fetch pages. For example pages can be requested in chunks, so that only a few streams are kept in memory at once: | ||
|
||
|
@@ -275,7 +274,7 @@ $items | |
``` | ||
|
||
|
||
### Handling errors | ||
### 💢 Errors handling | ||
|
||
As seen above, we can mitigate potentially faulty HTTP requests with backoffs, timeouts and retries. When we reach the maximum number of attempts and a request keeps failing, an `OutOfAttemptsException` is thrown. | ||
|
||
|
@@ -296,56 +295,51 @@ try { | |
} | ||
``` | ||
|
||
|
||
## Change log | ||
## 📆 Change log | ||
|
||
Please see [CHANGELOG](CHANGELOG.md) for more information on what has changed recently. | ||
|
||
## Testing | ||
## 🧪 Testing | ||
|
||
``` bash | ||
composer test | ||
``` | ||
|
||
## Contributing | ||
## 💞 Contributing | ||
|
||
Please see [CONTRIBUTING](CONTRIBUTING.md) and [CODE_OF_CONDUCT](CODE_OF_CONDUCT.md) for details. | ||
|
||
## Security | ||
## 🧯 Security | ||
|
||
If you discover any security related issues, please email [email protected] instead of using the issue tracker. | ||
|
||
## Credits | ||
## 🏅 Credits | ||
|
||
- [Andrea Marco Sartori][link-author] | ||
- [All Contributors][link-contributors] | ||
|
||
## License | ||
## ⚖️ License | ||
|
||
The MIT License (MIT). Please see [License File](LICENSE.md) for more information. | ||
|
||
[ico-author]: https://img.shields.io/static/v1?label=author&message=cerbero90&color=50ABF1&logo=twitter&style=flat-square | ||
[ico-php]: https://img.shields.io/packagist/php-v/cerbero/lazy-json-pages?color=%234F5B93&logo=php&style=flat-square | ||
[ico-laravel]: https://img.shields.io/static/v1?label=laravel&message=%E2%89%A56.0&color=ff2d20&logo=laravel&style=flat-square | ||
[ico-octane]: https://img.shields.io/static/v1?label=octane&message=compatible&color=ff2d20&logo=laravel&style=flat-square | ||
[ico-version]: https://img.shields.io/packagist/v/cerbero/lazy-json-pages.svg?label=version&style=flat-square | ||
[ico-actions]: https://img.shields.io/github/workflow/status/cerbero90/lazy-json-pages/build?style=flat-square&logo=github | ||
[ico-actions]: https://img.shields.io/github/actions/workflow/status/cerbero90/lazy-json-pages/build.yml?branch=master&style=flat-square&logo=github | ||
[ico-license]: https://img.shields.io/badge/license-MIT-brightgreen.svg?style=flat-square | ||
[ico-psr7]: https://img.shields.io/static/v1?label=compliance&message=PSR-7&color=blue&style=flat-square | ||
[ico-psr12]: https://img.shields.io/static/v1?label=compliance&message=PSR-12&color=blue&style=flat-square | ||
[ico-per]: https://img.shields.io/static/v1?label=compliance&message=PER&color=blue&style=flat-square | ||
[ico-scrutinizer]: https://img.shields.io/scrutinizer/coverage/g/cerbero90/lazy-json-pages.svg?style=flat-square&logo=scrutinizer | ||
[ico-code-quality]: https://img.shields.io/scrutinizer/g/cerbero90/lazy-json-pages.svg?style=flat-square&logo=scrutinizer | ||
[ico-phpstan]: https://img.shields.io/badge/level-max-success?style=flat-square&logo=data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAACAAAAAgCAYAAABzenr0AAAGb0lEQVR42u1Xe1BUZRS/y4Kg8oiR3FCCBUySESZBRCiaBnmEsOzeSzsg+KxYYO9dEEftNRqZjx40FRZkTpqmOz5S2LsXlEZBciatkQnHDGYaGdFy1EpGMHl/p/PdFlt2rk5O+J9n5nA/vtf5ned3lnlISpRhafBlLRLHCtJGVrB/ZBDsaw2lUqzReGAC46DstTYfnSCGUjaaDvgxACo6j3vUenNdImeRXqdnWV5az5rrnzeZznj8J+E5Ftsclhf3s4J4CS/oRx5Bvon8ZU65FGYQxAwcf85a7CeRz+C41THejueydCZ7AAK34nwv3kHP/oUKdOL4K7258fF7Cud427O48RQeGkIGJ77N8fZqlrcfRP4d/x90WQfHXLeBt9dTrSlwl3V65ynWLM1SEA2qbNQckbe4Xmww10Hmy3shid0CMcmlEJtSDsl5VZBdfAgMvI3uuR+moJqN6LaxmpsOBeLCDmTifCB92RcQmbAUJvtqALc5sQr8p86gYBCcFdBq9wOin7NQax6ewlB6rqLZHf23FP10y3lj6uJtEBg2HxiVCtzd3SEwMBCio6Nh9uzZ4O/vLwOZ4OUNM2NyIGPFrvuzBG//lRPs+VQ2k1ki+ePkd84bskz7YFpYgizEz88P8vPzYffu3dDS0gJNTU1QXV0NqampRK1WIwgfiE4qhOyig0rC+pCvK8QUoML7uJVHA5kcQUp3DSpqWjc3d/Dy8oKioiLo6uqCoaEhuHb1KvT09AAhBFpbW4lOpyMyyIBQSCmoUQLQzgniNvz+obB2HS2RwBgE6dOxCyJogmNkP2u1Wrhw4QJ03+iGrR9XEd3CTNBn6eCbo40wPDwMdXV1BF1DVG5qiEtboxSUP6J71+D3NwUAhLOIRQzm7lnnhYUv7QFv/yDZ/Lm5ubK2DVI9iZ8bR8JDtEB57lNzENQN6OjoIGlpabIVZsYaMTO+hrikRRA1JxmSX9hE7/sJtVyF38tKsUCVZxBhz9jI3wGT/QJlADzPAyXrnj0kInzGHQCRMyOg/ed2uHjxIuE4TgYQHq2DLJqumashY+lnsMC4GVC5do6XVuK9l+4SkN8y+GfYeVJn2g++U7QygPT0dBgYGIDvT58mnF5PQcjC83PzSF9fH7S1tZGEhAQZQOT8JaA317oIkM6jS8uVLSDzOQqg23Uh+MlkOf00Gg0cP34c+vv74URzM9n41gby/rvvkc7OThlATU3NCGYJUXt4QaLuTYwBcTSOBmj1RD7D4Tsix4ByOjZRF/zgupDEbgZ3j4ly/qekpND0o5aQ44HS4OAgsVqtI1gTZO01IbG0aP1bknnxCDUvArHi+B0lJSlzglTFYO2udF3Ql9TCrHn5oEIreHp6QlRUFJSUlJCqqipSWVlJ8vLyCGYIFS7HS3zGa87mv4lcjLwLlStlLTKYYUUAlvrlDGcW45wKxXX6aqHZNutM+1oQBHFTewAKkoH4+vqCj48PYAGS5yb5amjNoO+CU2SL53NKpDD0vxHHmOJir7L5xUvZgm0us2R142ScOIyVqYvlpWU4XoHIP8DXL2b+wjdWeXh6U2FjmIIKmbWAYPFRMus62h/geIvjOQYlpuDysQrLL6Ger49HgW8jqvXUhI7UvDb9iaSTDqHtyItiF5Suw5ewF/Nd8VJ6zlhsn06bEhwX4NyfCvuGEeRpTmh4mkG68yDpyuzB9EUcjU5awbAgncPlAeSdAQER0zCndzqVbeXC4qDsMpvGEYBXRnsDx4N3Auf1FCTjTIaVtY/QTmd0I8bBVm1kejEubUfO01vqImn3c49X7qpeqI9inIgtbpxK3YrKfIJCt+OeV2nfUVFR4ca4EkVENyA7gkYcMfB1R5MMmxZ7ez/2KF5SSN1yV+158UPsJT0ZBcI2bRLtIXGoYu5FerOUiJe1OfsL3XEWH43l2KS+iJF9+S4FpcNgsc+j8cT8H4o1bfPg/qkLt50uJ1RzdMsGg0UqwfEN114Pwb1CtWTGg+Y9U5ClK9x7xUWI7BI5VQVp0AVcQ3bZkQhmnEgdHhKyNSZe16crtBIlc7sIb6cRLft2PCgoKGjijBDtjrAQ7a3EdMsxzIRflAFIhPb6mHYmYwX+WBlPQgskhgVryyJCQyNyBLsBQdQ6fgsQhyt6MSOOsWZ7gbH8wETmgRKAijatNL8Ngm0xx4tLcsps0Wzx4al0jXlI40B/A3pa144MDtSgAAAAAElFTkSuQmCC | ||
[ico-downloads]: https://img.shields.io/packagist/dt/cerbero/lazy-json-pages.svg?style=flat-square | ||
|
||
[link-author]: https://twitter.com/cerbero90 | ||
[link-php]: https://www.php.net | ||
[link-laravel]: https://laravel.com | ||
[link-octane]: https://github.com/laravel/octane | ||
[link-packagist]: https://packagist.org/packages/cerbero/lazy-json-pages | ||
[link-actions]: https://github.com/cerbero90/lazy-json-pages/actions?query=workflow%3Abuild | ||
[link-psr7]: https://www.php-fig.org/psr/psr-7/ | ||
[link-psr12]: https://www.php-fig.org/psr/psr-12/ | ||
[link-per]: https://www.php-fig.org/per/coding-style/ | ||
[link-scrutinizer]: https://scrutinizer-ci.com/g/cerbero90/lazy-json-pages/code-structure | ||
[link-code-quality]: https://scrutinizer-ci.com/g/cerbero90/lazy-json-pages | ||
[link-downloads]: https://packagist.org/packages/cerbero/lazy-json-pages | ||
[link-phpstan]: https://phpstan.org/ | ||
[link-contributors]: ../../contributors |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,10 @@ | ||
<?php | ||
|
||
declare(strict_types=1); | ||
|
||
use Cerbero\LazyJsonPages\LazyJsonPages; | ||
use Illuminate\Support\LazyCollection; | ||
|
||
(static function () { | ||
LazyCollection::macro('fromJsonPages', [LazyJsonPages::class, 'from']); | ||
})(); |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.