v0.0.4

What's Changed

• Add more comparisons by @hinthornw in #95
• Added benchmarks to typerwriter 1, multiverse, relational data, update evaluators by @eyurtsev in #92
• Version 0.0.4 by @eyurtsev in #96

Full Changelog: v0.0.3...v0.0.4

v0.0.3

What's Changed

• Improve Prefix for key hint by @hinthornw in #72
• Wfh/descriptive names by @hinthornw in #79
• Make output optional too by @eyurtsev in #82
• Tool Usage updates by @eyurtsev in #89
• Fix dataset example for multiverse math by @eyurtsev in #90
• Extraction Update by @hinthornw in #91
• Limit Retries in OpenAI by @hinthornw in #93
• 0.0.3 by @hinthornw in #94

Full Changelog: v0.0.2...v0.0.3

v0.0.2

What's Changed

• Add links, update by @hinthornw in #70
• update link by @hinthornw in #71
• Device 0 default by @hinthornw in #73
• Rename intros by @eyurtsev in #75
• Tool usage docs by @eyurtsev in #76
• Add colab links by @eyurtsev in #77
• Update standard agent evalutator by @eyurtsev in #78
• Minor documentation fixes by @eyurtsev in #80
• Version 0.0.2 by @eyurtsev in #81

Full Changelog: v0.0.1...v0.0.2

v0.0.1

What's Changed

• Create extraction playground by @rlancemartin in #2
• Remove mention of default message by @fpingham in #6
• Add Criteria Evals by @hinthornw in #5
• add openai instruct benchmarking by @hwchase17 in #9
• Add other datasets by @hinthornw in #10
• Add LangChain Docs Benchmark Evals by @hinthornw in #12
• Add environment #1 for agents by @eyurtsev in #14
• Version 2 by @eyurtsev in #16
• Reformat entire codebase with ruff by @eyurtsev in #17
• Remove accidental commits by @eyurtsev in #18
• Add langsmith util to clone dataset by @eyurtsev in #19
• Add clone/download dataset, handle web vs. api paths by @eyurtsev in #20
• Scaffold for sphinx docs by @eyurtsev in #21
• Add helper to Makefile by @eyurtsev in #22
• Add datasets notebook by @eyurtsev in #23
• Update description by @eyurtsev in #24
• Add registry object, add eval notebook by @eyurtsev in #25
• Add additional tasks, re-org repo a bit by @eyurtsev in #26
• Add extraction task by @eyurtsev in #29
• Refactor to remove ID from task, move task definitions out of registry.py by @eyurtsev in #30
• Remove ID that doens't exist by @eyurtsev in #31
• Restore tasks namespace, create agent factory by @eyurtsev in #33
• Expand CI to also build sphinx docs by @eyurtsev in #34
• Create doc publish workflow by @eyurtsev in #35
• Fix typo in workflow to publish docs by @eyurtsev in #36
• Update doc build by @eyurtsev in #39
• Update langchain, relax requirements for deps by @eyurtsev in #40
• Add more extraction code by @eyurtsev in #37
• Update notebook for extraction by @eyurtsev in #41
• Update lock file by @eyurtsev in #42
• Bump dependencies for doc building by @eyurtsev in #44
• Add RAG tasks by @hinthornw in #43
• Add filter option by @hinthornw in #45
• Add datasets for tool usage tasks by @eyurtsev in #46
• Add multiverse math notebook, update dataset id by @eyurtsev in #48
• Switch to api by @hinthornw in #50
• Make the example explicit by @hinthornw in #49
• More tool usage updates by @eyurtsev in #51
• Check public datasets exist by @eyurtsev in #54
• Move RAG unit tests by @eyurtsev in #55
• Update notebooks toc by @eyurtsev in #56
• Add Quick Start by @hinthornw in #52
• Update notebooks by @hinthornw in #47
• Another toc update by @eyurtsev in #57
• Change schema for semi-structured by @hinthornw in #58
• Add retrieval intro; Add links by @hinthornw in #59
• Add extraction intro by @hinthornw in #60
• Agent Intros by @hinthornw in #61
• Add docs link by @hinthornw in #62
• Update extraction dataset by @hinthornw in #63
• Wfh/update name by @hinthornw in #64
• Update repo structure in readme by @hinthornw in #65
• Add MIT license by @eyurtsev in #66
• Add badges to README by @eyurtsev in #67
• Relax langchain constraints by @eyurtsev in #68
• Wfh/rerun notebooks by @hinthornw in #69

New Contributors

• @rlancemartin made their first contribution in #2
• @fpingham made their first contribution in #6
• @hwchase17 made their first contribution in #9
• @eyurtsev made their first contribution in #14

Full Changelog: https://github.com/langchain-ai/langchain-benchmarks/commits/v0.0.1