v0.4.0rc1

What's new

Added 🎉

• Introduced the concept of the Workspace, with LocalWorkspace and MemoryWorkspace as initial implementations.
• Added a stub of a webserver that will be able to visualize runs as they happen.
• Added separate classes for LightningTrainingTypePlugin, LightningPrecisionPlugin, LightningClusterEnvironmentPlugin, LightningCheckpointPlugin for compatibility with pytorch-lightning>=1.5.0.

Removed 👋

• Removed old LightningPlugin class
• Removed requirement of the overrides package

Changed ⚠️

• Made it possible to construct a step graph out of Step objects, instead of constructing it out of StepStub objects.
• Removed dataset fingerprinting code, since we can now use Step to make sure things are cached.
• Made steps deterministic by default.
• Brought back MemoryStepCache, so we can run steps without configuring anything.
• W&B torch::TrainCallback logs with step=step+1 now so that training curves in the W&B dashboard
  match up with checkpoints saved locally and are easier to read (e.g. step 10000 instead of 9999).
• filelock >= 3.4 required, parameter poll_intervall to tango.common.file_lock.FileLock.acquire renamed
  to poll_interval.

Fixed ✅

• Fixed bug in FromParams where a parameter to a FromParams class may not be instantiated correctly
  if it's a class with a generic type parameter.

Commits

b2c09e3 fix typo in example (#96)
aba5758 Merge pull request #94 from allenai/dependabot/pip/cached-path-gte-0.3.3-and-lt-1.1.0
4ae6115 Update requirements.txt
d4d0655 Merge branch 'main' into dependabot/pip/cached-path-gte-0.3.3-and-lt-1.1.0
48b0b24 Merge pull request #92 from allenai/dependabot/pip/datasets-gte-1.12-and-lt-1.17
6044f8b Update cached-path requirement from <0.4.0,>=0.3.3 to >=0.3.3,<1.1.0
1ae82aa Merge branch 'main' into dependabot/pip/datasets-gte-1.12-and-lt-1.17
0107672 Merge pull request #91 from allenai/dependabot/pip/sphinx-4.3.1
fa47e54 Merge branch 'main' into dependabot/pip/datasets-gte-1.12-and-lt-1.17
38c0b42 Merge branch 'main' into dependabot/pip/sphinx-4.3.1
7d9d919 Merge pull request #93 from allenai/NoOverrides
dfa461c Removes the dependency on the overrides package
ff18197 Update datasets requirement from <1.16,>=1.12 to >=1.12,<1.17
fc69dd0 Bump sphinx from 4.3.0 to 4.3.1
b5907de Merge pull request #67 from allenai/ResponsibleSteps
267a6e4 clean up config usage
82862ef Merge branch 'main' into ResponsibleSteps
20951ea Bump furo from 2021.11.16 to 2021.11.23 (#89)
8d8670a Optional server
03049fa Handle the log level consistently
c620405 Merge branch 'main' into ResponsibleSteps
8bb00c4 Bump black from 21.11b0 to 21.11b1 (#88)
cd5a70a Fix tests
71cfbd7 Don't cache uncacheable steps
7839fd8 Merge branch 'main' into ResponsibleSteps
f240ac4 update filelock + cached_path, improve release scripts (#87)
967ecb2 Merge branch 'main' into ResponsibleSteps
10634aa Don't show inherited from_params
4b104be Fix test
4e9910d Avoid a naming conflict in computer science
6db2c29 Improved documentation
44f79ec Added blurb
7f68a97 Use enum for step states
19e6de2 Click logging is disabled by default, enabled in the CLI use case
eba4b8e Bump black from 21.10b0 to 21.11b0 (#86)
1d80766 Log the start of a run
6190e2d Merge branch 'main' into ResponsibleSteps
3550c3f Cleaner workspace docs
08d4056 Better StepCache docs
d5ede4e Formatting
bc80bb8 Merge pull request #85 from allenai/dependabot/pip/filelock-gte-3.3-and-lt-3.5
d69bac4 Check whether a run name already exists
cde0f14 Unused import
a80a87f Fix the case where a step's cacheability changes across restarts
de5f248 Improve comment
f13d717 Losely pin petname
5d72923 Merge branch 'main' into ResponsibleSteps
914902b Merge branch 'ResponsibleSteps' of https://github.com/allenai/tango into ResponsibleSteps
2f5a266 Merge pull request #83 from allenai/petew-ResponsibleSteps
b3df88f Update filelock requirement from <3.4,>=3.3 to >=3.3,<3.5
17d28c7 Bump furo from 2021.11.15 to 2021.11.16 (#84)
8f2b48e add failing test case
1fc9860 Merge remote-tracking branch 'origin/main' into ResponsibleSteps
4118912 Merge pull request #82 from allenai/dependabot/pip/furo-2021.11.15
d5eb968 Merge branch 'main' into dependabot/pip/furo-2021.11.15
abac28a Fix the shortcut for running all (many) checks
c4009de Merge branch 'main' into ResponsibleSteps
e364766 Merge pull request #73 from allenai/petew-ResponsibleSteps
8ae0e27 More doctests
6211025 Format docs better
5814724 Fix docs
cb1b853 Merge pull request #62 from allenai/dependabot/pip/pytorch-lightning-gte-1.4.0-and-lt-1.6.0
50caada Update requirements.txt
9c79b97 Makes the docs build
3ecb952 Merge branch 'main' into dependabot/pip/pytorch-lightning-gte-1.4.0-and-lt-1.6.0
55c5fae Bump furo from 2021.11.12 to 2021.11.15
83add7e Important fixes
301347d Fix tests
cfe29de Create workdir when requested
7b67142 Bring back "needed by"
a46b3e8 This wasn't meant to be checked in.
af4cc3c Use click through the logger
ee81446 Merge branch 'ResponsibleSteps' into petew-ResponsibleSteps
7247528 We don't need this TODO right now.
9b8446c Merge branch 'main' into ResponsibleSteps
30f7a13 W&B log as step+1 (#76)
eab2b7f Merge pull request #74 from allenai/Workspaces
d91302b Simplify!
e605399 changelog
9f360da Bring back deterministic step randomness, without breaking random step names
a2c5cd8 fix order of imports
02ad624 Merge branch 'dependabot/pip/pytorch-lightning-gte-1.4.0-and-lt-1.6.0' of https://github.com/allenai/tango into dependabot/pip/pytorch-lightning-gte-1.4.0-and-lt-1.6.0
7f78b0a remove comment
8f8cbeb separating different plugin types
b139181 Merge branch 'main' into dependabot/pip/pytorch-lightning-gte-1.4.0-and-lt-1.6.0
761f873 Merge branch 'ResponsibleSteps' into Workspaces
12a480a Merge branch 'ResponsibleSteps' into petew-ResponsibleSteps
b7d4e88 merge main
63f6029 Merge branch 'main' into ResponsibleSteps
aab58b6 add some conda instructions to CONTRIBUTING.md (#81)
e7e5c5e Fix symlink creation
b547472 Adds a command to keep a server running permanently
596c278 Not sure how this line got lost
06d7681 Fixes and cleanup
3cebc24 fix bug caused by random seed
d145216 fix comment
067436d clean up
a4ce577 Creates and uses the concept of a workspace, so that the server can consume it
110eb07 ci
e811f47 fix tests
460dd87 executor fixes
837a454 handle generic non-FromParams classes
7fed5a6 fix merge conflicts
12ae8e9 Fixing the torch test
0c208e1 Remove stale comment
b44a0a0 Fix doctest
3e4eb42 Makes det_hash consistent across Python versions
f23b56c Executable documentation!
38388ca Formatting
6aaa01e Fix some documentation
57825a0 Fix docs
cb7591d Order imports correctly 🙄
8caea06 Removing unused imports
f7f5d2e Changelog again
05ca4dd Changelog
da16b5a Merge branch 'main' into ResponsibleSteps
8c51b45 Make nested steps work for classes that aren't FromParams
083516b Remember which extra modules we imported
54e7dbc We can't restore the registry like that.
455b756 Refactors the test to fail in new and exciting ways
62cd1d2 Merge branch 'main' into dependabot/pip/pytorch-lightning-gte-1.4.0-and-lt-1.6.0
cd68752 Don't need this comment anymore
76c10f1 Remove fingerprint stuff
a1f1af8 Merge branch 'FixImport' into ResponsibleSteps
ed9ebc9 Fix Import
14bc2ee Merge branch 'main' into ResponsibleSteps
4a4ab58 Merge branch 'main' into dependabot/pip/pytorch-lightning-gte-1.4.0-and-lt-1.6.0
e5ea1af Fix after merge
7b92dd9 Merge remote-tracking branch 'origin/main' into ResponsibleSteps
d066768 We were not actually using this function
df07690 Mypy inspired changes
87ee96f Formatting
3823f98 🤦🏼
d018838 Fix executor test
a7fb4ea Fix executor
0b1f310 Quiet, you
449a70d Fixes circular references
4fc1f34 Actually write the circular reference test
0df3538 Throw the error for the right reason
afb3b61 WithUnresolvedSteps
1be6816 Update pytorch-lightning requirement
3c4cb19 Dicts are iterable, so these have to be swapped
efc84d4 Adds new failing test
1f2bf16 Relative imports don't work
430112f Fix bug in test
c3dd08d Add test that fails
118f8c2 Better name for the test
a6ac3c9 Detect unsatisfiable dependencies
8eb7a48 Bring back parsing everything as a Step first :-/
bc6f886 Type checks
00b2df6 Makes the test pass
a6ba11a Make the code more compatible with the IDE
a611d48 We don't need these anymore
132153d Formatting
d019499 Start fixing hard tests
f4d77c1 Fix trivial tests
fa6de92 Make steps responsible for their own execution
cfe7007 Slightly more readable error message
c0ded32 Typo