New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

count assert statements in our test-case notebooks #32

Draft

haesleinhuepf wants to merge 1 commit into main from count-asserts

Owner

haesleinhuepf commented Apr 12, 2024

This PR contains:

a new test-case for the benchmark
- I hereby confirm that NO LLM-based technology (such as github copilot) was used while writing this benchmark
new generator-functions allowing to sample from other LLMs
new samples (sample_....jsonl files)
new benchmarking results (..._results.jsonl files)
documentation update
bug fixes

Related github issue (if relevant): closes #0

Short description:

This delivers a way for estimating the number of unit-tests per test-case. We're counting the actual assert statements which test our functions.

How do you think will this influence the benchmark results?

It adds another metric that tells us something about the quantity of the tests.

Why do you think it makes sense to merge this PR?

I'm not sure if we need it. The original HumanEval paper mentions approx 7.7 unit tests per test-case. We have approx 2.5 assert statements per test-case... I'm not sure if these are comparable...


          count assert statements in our test-case notebooks

c4d2332

haesleinhuepf added the question label

Owner Author

haesleinhuepf commented Apr 13, 2024

I'm not 100% sure if this is the way to go. Don't merge this for now.

haesleinhuepf marked this pull request as draft

June 2, 2024 21:33

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels