Skip to content

[PERF] Native globbing early stopping #2078

[PERF] Native globbing early stopping

[PERF] Native globbing early stopping #2078

Triggered via pull request October 2, 2023 18:10
Status Success
Total duration 39s
Artifacts

release-drafter.yml

on: pull_request
update_release_draft
7s
update_release_draft
Fit to window
Zoom out
Zoom in

Annotations

2 errors
update_release_draft
Validation Failed: {"resource":"Release","code":"invalid","field":"target_commitish"} { name: 'HttpError', id: '6383843778', status: 422, response: { url: 'https://api.github.com/repos/Eventual-Inc/Daft/releases/122533306', status: 422, headers: { 'access-control-allow-origin': '*', 'access-control-expose-headers': 'ETag, Link, Location, Retry-After, X-GitHub-OTP, X-RateLimit-Limit, X-RateLimit-Remaining, X-RateLimit-Used, X-RateLimit-Resource, X-RateLimit-Reset, X-OAuth-Scopes, X-Accepted-OAuth-Scopes, X-Poll-Interval, X-GitHub-Media-Type, X-GitHub-SSO, X-GitHub-Request-Id, Deprecation, Sunset', connection: 'close', 'content-length': '195', 'content-security-policy': "default-src 'none'", 'content-type': 'application/json; charset=utf-8', date: 'Mon, 02 Oct 2023 18:10:36 GMT', 'referrer-policy': 'origin-when-cross-origin, strict-origin-when-cross-origin', server: 'GitHub.com', 'strict-transport-security': 'max-age=31536000; includeSubdomains; preload', vary: 'Accept-Encoding, Accept, X-Requested-With', 'x-accepted-github-permissions': 'contents=write', 'x-content-type-options': 'nosniff', 'x-frame-options': 'deny', 'x-github-api-version-selected': '2022-11-28', 'x-github-media-type': 'github.v3; format=json', 'x-github-request-id': '0447:32C3:B27056:16A7796:651B079C', 'x-ratelimit-limit': '1000', 'x-ratelimit-remaining': '904', 'x-ratelimit-reset': '1696270487', 'x-ratelimit-resource': 'core', 'x-ratelimit-used': '96', 'x-xss-protection': '0' }, data: { message: 'Validation Failed', errors: [ { resource: 'Release', code: 'invalid', field: 'target_commitish' } ], documentation_url: 'https://docs.github.com/rest/releases/releases#update-a-release' } }, request: { method: 'PATCH', url: 'https://api.github.com/repos/Eventual-Inc/Daft/releases/122533306', headers: { accept: 'application/vnd.github.v3+json', 'user-agent': 'probot/12.2.5 octokit-core.js/3.5.1 Node.js/16.20.1 (linux; x64)', authorization: 'token [REDACTED]', 'content-type': 'application/json; charset=utf-8' }, body: '{"body":"## Changes\\n\\n## ✨ New Features\\n\\n- [FEAT] ls/list\\\\_dir for AzureBlobStorage @xcharleslin (#1408)\\n- [FEAT] Add `.str.split()` API for splitting string columns. @clarkzinzow (#1409)\\n- [FEAT] Add local native filesystem globbing. @clarkzinzow (#1449)\\n- [FEAT] Native listing of http URLs @jaychia (#1405)\\n\\n## 🧰 Maintenance\\n\\n- [CHORE] update s3 connection defaults @samster25 (#1451)\\n","draft":true,"prerelease":false,"name":"v0.1.19","tag_name":"v0.1.19","target_commitish":"refs/pull/1452/merge"}', request: {} }, event: { id: '6383843778', name: 'pull_request', payload: { action: 'edited', changes: { body: { from: 'Adds early stopping capabilities for our native globbing to prevent runaway parallelism when listing adverserial cases (one file per folder)\r\n' + '\r\n' + 'Other changes in this PR:\r\n' + '\r\n' + '1. Adds benchmarks on 10k local files\r\n' + '2. Adds a new `posix` argument to `ls` and `iter_dir`: this controls whether the operation will perform a posix-like list (of just the next level), or an object-store-like prefix list (of all files starting with the provided prefix)\r\n' + '3. `delimiter` is no longer an `Option` argument and is required by all the ObjectSources: it could actually be argued that this should be automatically determined by the ObjectSource itself?\r\n' + '\r\n' + 'Not in scope in this PR:\r\n' + '\r\n' + '1. Prefix listing for GCS/Azure/Local/HTTP are not implemented in this PR, thus the globbing algorithm will actually not work for those cases\r\n' + '2. We may need a different globbing implementation for Local and HTTP give
update_release_draft
HttpError: Validation Failed: {"resource":"Release","code":"invalid","field":"target_commitish"} at /home/runner/work/_actions/release-drafter/release-drafter/v5/dist/index.js:8462:21 at processTicksAndRejections (node:internal/process/task_queues:96:5) at async Job.doExecute (/home/runner/work/_actions/release-drafter/release-drafter/v5/dist/index.js:30793:18) { name: 'AggregateError', event: { id: '6383843778', name: 'pull_request', payload: { action: 'edited', changes: { body: { from: 'Adds early stopping capabilities for our native globbing to prevent runaway parallelism when listing adverserial cases (one file per folder)\r\n' + '\r\n' + 'Other changes in this PR:\r\n' + '\r\n' + '1. Adds benchmarks on 10k local files\r\n' + '2. Adds a new `posix` argument to `ls` and `iter_dir`: this controls whether the operation will perform a posix-like list (of just the next level), or an object-store-like prefix list (of all files starting with the provided prefix)\r\n' + '3. `delimiter` is no longer an `Option` argument and is required by all the ObjectSources: it could actually be argued that this should be automatically determined by the ObjectSource itself?\r\n' + '\r\n' + 'Not in scope in this PR:\r\n' + '\r\n' + '1. Prefix listing for GCS/Azure/Local/HTTP are not implemented in this PR, thus the globbing algorithm will actually not work for those cases\r\n' + '2. We may need a different globbing implementation for Local and HTTP given that they do not support prefix listing\r\n' + '\r\n' + '## How it works\r\n' + '\r\n' + '1. We can think of globbing a path as a tree search.\r\n' + '4. At tree depth `d`, we define the number of nodes at the depth as `fanout`.\r\n' + '5. This PR adds a `fanout_limit` to limit the fanout of our globbing. When we perform a recursive list and see that entering depth `d+1` will increase `fanout` by more than `fanout_limit`, we will instead fall back on "flat listing" which is a listing of all files starting with the current path as their prefix.\r\n' + '\r\n' } }, number: 1452, organization: { avatar_url: 'https://avatars.githubusercontent.com/u/98941975?v=4', description: 'Eventual Computing', events_url: 'https://api.github.com/orgs/Eventual-Inc/events', hooks_url: 'https://api.github.com/orgs/Eventual-Inc/hooks', id: 98941975, issues_url: 'https://api.github.com/orgs/Eventual-Inc/issues', login: 'Eventual-Inc', members_url: 'https://api.github.com/orgs/Eventual-Inc/members{/member}', node_id: 'O_kgDOBeW8Fw', public_members_url: 'https://api.github.com/orgs/Eventual-Inc/public_members{/member}', repos_url: 'https://api.github.com/orgs/Eventual-Inc/repos', url: 'https://api.github.com/orgs/Eventual-Inc' }, pull_request: { _links: { comments: { href: 'https://api.github.com/repos/Eventual-Inc/Daft/issues/1452/comments' }, commits: { href: 'https://api.github.com/repos/Eventual-Inc/Daft/pulls/1452/commits' }, html: { href: 'https://github.com/Eventual-Inc/Daft/pull/1452' }, issue: { href: 'https://api.github.com/repos/Eventual-Inc/Daft/issues/1452' }, review_comment: { href: 'https://api.github.com/repos/Eventual-Inc/Daft/pulls/comments{/number}' }, review_comments: { href: 'https://api.github.com/repos/Eventual-Inc/Daft/pulls/1452/comments' }, self: { href: 'https://api.github.com/repos/Eventual-Inc/Daft/pulls/1452' }, statuses: { href: 'https://api.github.com/repos/Eventual-Inc/Daft/statuses/8902c0396b32110e544ac3d980d4e96fa2215e8d' }