Code

Implement a basic generic API client#313

Code

Merged

burnash merged 124 commits intomasterdlt-hub/verified-sources:masterfrom enh/api_helperdlt-hub/verified-sources:enh/api_helperCopy head branch name to clipboardMar 22, 2024

+3,365-17

Commits

Commits on Feb 19, 2024

• Implement a basic generic API client; convert some sources to use this client

  burnash

  committed
  7d65701

• Move the paginaton loop into APIClient

  burnash

  committed
  f7af81b

• Factor out common code

  burnash

  committed
  4bfdc13

• Refactor common code

  burnash

  committed
  3ce56c1

• add generic rest source and an example pipeline

  burnash

  committed
  ce784d4

• Restructure the REST client add pagination detector

  burnash

  committed
  8d7d0bd

• fix the paginator detector

  burnash

  committed
  8bde92a

• Accept paginator instance

  burnash

  committed
  4b686a0

• Add Offset paginator

  burnash

  committed
  2e297e2

• Add comments

  burnash

  committed
  f5db3aa

• Factor out resources

  burnash

  committed
  650c3ca

• Add Literal

  burnash

  committed
  e61e304

• Remove the example

  burnash

  committed
  5e63d33

• Add logging

  burnash

  committed
  d55ae29

• Handle depended resources

  burnash

  committed
  01fe721

• Fix the bug with duplication of nested sources

  burnash

  committed
  8138403

• Add an alternative version that uses classes

  burnash

  committed
  ec689fe

• Rearrange config

  burnash

  committed
  afce8d3

• REST API: support all authentication methods (#354)

  Show description for 3fe3af7
  

  willi-mueller

  authored and

  burnash

  committed
  3fe3af7

• Generic API client: include parent fields in child resource (#355)

  

  willi-mueller

  authored and

  burnash

  committed
  b2e7cec

• Resource based config

  burnash

  committed
  0ea0edb

• Receive a custom Session instance

  burnash

  committed
  16c5a7a

• Include data from parent resource in child resource: ported to a new version

  burnash

  committed
  c6015fe

• Rest API: Ends pagination if next page path is not in response.json() (#361)

  

  willi-mueller

  authored and

  burnash

  committed
  d4a3160

Commits on Feb 20, 2024

• Allow specification of SinglePagePaginator and refactors redundancy (#364)

  willi-mueller

  authored
  9f956ba

• Use the resource name as an endpoint path if path is missing

  burnash

  committed
  7063337

Commits on Feb 22, 2024

• [REST Source] renames default_paginator argument to paginator (#367)

  willi-mueller

  authored
  1e1e676

• Remove the legacy version

  burnash

  committed
  884120b

• Add records_key to SinglePagePaginator

  burnash

  committed
  d5eaee1

• [REST Source] completes renaming of default_paginator to paginator (#370)

  Show description for 2f580a3
  willi-mueller

  authored
  2f580a3

Commits on Feb 25, 2024

• Add tests and pagination

  burnash

  committed
  168b11a

Commits on Feb 26, 2024

• Temporary disable paginator type check

  burnash

  committed
  e13cff2

• [REST source] test case for dependent resource (#371)

  willi-mueller

  authored
  f14f539

• Remove comments

  burnash

  committed
  10bd716

Commits on Feb 27, 2024

• Reuse MOCK_BASE_URL for all endpoints

  burnash

  committed
  0b31301

• Rename the config container

  burnash

  committed
  65c6617

• Add tests for valid source configurations

  burnash

  committed
  a554efa

• Add Flask-style paginaton

  burnash

  committed
  e6e6927

• [REST API source] adds function to check connection (#357)

  willi-mueller

  authored
  06a054e

• [REST Source] allow skipping http errors (#365)

  

  willi-mueller

  authored and

  burnash

  committed
  726b204

• added the possibility to pass HTTPBasicAuth objects (#377)

  francescomucio

  authored
  0b99ba5

• Factor out typings

  burnash

  committed
  0631c98

Commits on Feb 28, 2024

• Add response_actions to enable skipping responses by status code or content

  burnash

  committed
  d1d25f3

• Move records extractor out of the paginator class

  burnash

  committed
  76710ea

Commits on Feb 29, 2024

• [REST] Detailed error handler logging (#383)

  willi-mueller

  authored
  f3ea829

• Fixes records detection for header links paginator

  burnash

  committed
  16cb89a

• [REST source] header_links can extract from responses without a records_path (#382)

  Show description for 987fdfa
  willi-mueller

  authored
  987fdfa

Commits on Mar 1, 2024

• [REST source] fixes deprecation warning (#380)

  Show description for 71b4682
  willi-mueller

  authored
  71b4682

• Use update_dict_nested in place of deep_merge

  burnash

  committed
  eeea3a8

• Update the lockfile

  burnash

  committed
  4d01f35

• Add requirements.txt

  burnash

  committed
  d732976

Commits on Mar 2, 2024

• Upgrade dlt version

  burnash

  committed
  ac39f62

Commits on Mar 4, 2024

• Rename records_path to data_selector

  burnash

  committed
  14748a4

• Mutate request objects in paginators

  burnash

  committed
  39af289

• Merge branch 'master' into enh/api_helper

  burnash

  authored
  3d9c87e

• Regenerate lock

  burnash

  committed
  12b3726

Commits on Mar 5, 2024

• Remove request_client param from RESTClient; set raise_for_status to False for the client

  burnash

  committed
  69c1300

• Pass all incremental params from config

  burnash

  committed
  79030f8

• Refactor to argument unpacking

  burnash

  committed
  46e3385

• Add more auth classes

  burnash

  committed
  4b58b7b

Commits on Mar 6, 2024

• Factor out records extractor logic

  burnash

  committed
  13e21b8

• Add tests for detectors

  burnash

  committed
  efd0d80

• Remove UnspecifiedPaginator

  burnash

  committed
  dbe1b65

• [REST CLIENT] alt response extractor (#396)

  Show description for fefd704
  rudolfix

  authored
  fefd704

• makes openapi friendly auth (#397)

  Show description for c535088
  rudolfix

  authored
  c535088

• Bring detect_paginator back

  burnash

  committed
  5a1f3b5

• Fix test case for nested key (next.url); format code

  burnash

  committed
  05e899e

• Revert Notion source

  burnash

  committed
  0c2ddcf

• Revert Personio and Zendesk

  burnash

  committed
  45be0cf

• Remove an unused file

  burnash

  committed
  9ba18a7

• Restore personio settings

  burnash

  committed
  8734678

• Restore personio tests and

  Show description for 739547c
  burnash

  committed
  739547c

• Add type annotations

  burnash

  committed
  4148a69

Commits on Mar 7, 2024

• bumps dlt to 0.4.6

  rudolfix

  committed
  b15f2dd

Commits on Mar 8, 2024

• Type fixes and dlt session check

  burnash

  committed
  01a08cb

• [REST CLIENT] yields data pages with requests context (#399)

  Show description for e2aac86
  

  rudolfix

  and

  burnash

  authored
  e2aac86

Commits on Mar 9, 2024

• Fix paginator_config unpacking and RESTClient typing errors

  burnash

  committed
  3a67ac8

• Fix more typing errors

  burnash

  committed
  639a649

Commits on Mar 10, 2024

• Fix E741

  burnash

  committed
  f29655e

• Fix linting errors

  burnash

  committed
  bd19ee2

• Update the lock file

  burnash

  committed
  bdd6feb

• Extract build_resource_dependency_graph()

  burnash

  committed
  6ce75e0

• Factor out create_resources()

  burnash

  committed
  4bc5340

• Use requests hooks to handle response actions

  burnash

  committed
  d0a22d9

• Derive the response exception from DltException

  burnash

  committed
  86eb1ed

• Resolve poetry.lock conflict

  burnash

  committed
  ed81495

• Fix black check

  burnash

  committed
  3ae0ba5

• Fix lint

  burnash

  committed
  2d336e9

• Make default token expiration configurable

  burnash

  committed
  4a90485

• Add missing http headers

  burnash

  committed
  55a5924

• Refactor paginator creation in RESTClient to use PaginatorFactory

  burnash

  committed
  dad1e96

• Use frozensets

  burnash

  committed
  e39aaf9

• Remove an unused import

  burnash

  committed
  b2715c9

• Update docstrings

  burnash

  committed
  23b22a5

• Accept additional dlt source arguments in rest_api_source()

  burnash

  committed
  e8cfa32

• Add a workaround to pass test_dlt_init

  burnash

  committed
  24ec947

Commits on Mar 12, 2024

• Extend config test with an auth class instance case

  burnash

  committed
  94beb5b

• Remove Any from PaginatorType

  burnash

  committed
  ba46fda

• Upgrade dlt

  burnash

  committed
  82f3357

• Update lock file

  burnash

  committed
  d278643

• Remove commented code

  burnash

  committed
  3141ecc

Commits on Mar 13, 2024

• Refactor configuration setup into a dedicated module

  burnash

  committed
  94352ed

• Move response hooks setup and handling out of RESTClient

  burnash

  committed
  70496ec

• Remove unused imports

  burnash

  committed
  2e3d50a

Commits on Mar 14, 2024

• Fix hooks typing

  burnash

  committed
  b6d3794

Commits on Mar 20, 2024

• Rename args of the OffsetPaginator

  burnash

  committed
  3b7a0b6

• Create a RESTClient per resource

  burnash

  committed
  b971f9b

• Handle both error statuses and response actions

  burnash

  committed
  ebff65c

Commits on Mar 21, 2024

• Initial version of the README.md (#389)

  francescomucio

  authored
  4fb4ce1

• Remove commented code

  burnash

  committed
  59bba2e

• Clean up docstrings

  burnash

  committed
  390c233

• Remove the useless conditional init for items

  burnash

  committed
  5ecc426

• Fix grammar in the README

  burnash

  committed
  6924233

• Remove UnspecifiedPaginator

  burnash

  committed
  ff5ddd8

• Format README

  burnash

  committed
  c24a5aa

• Add handling end_value and end_param

  burnash

  committed
  d63e30f

Commits on Mar 22, 2024

• Use NamedTuple for incremental params

  burnash

  committed
  5cbf863

• Move check_connection to utils

  burnash

  committed
  6b7a891

• Instantiate auth based on type

  burnash

  committed
  6dab451

• Merge branch 'master' into enh/api_helper

  burnash

  authored
  32e8327

• Update lock file

  burnash

  committed
  559f3a2

• Sthor/api helper updates (#400)

  steinitzu

  authored
  cf8e266

• Remove unused imports

  burnash

  committed
  b8e6b5d

• Use jsonpath for next_path; remove create_nested_accessor

  burnash

  committed
  e6d927c