Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Update Bencher CLI usage #4335

Merged
merged 1 commit into from
Nov 11, 2024
Merged

Update Bencher CLI usage #4335

merged 1 commit into from
Nov 11, 2024

Conversation

epompeii
Copy link
Contributor

@epompeii epompeii commented Nov 5, 2024

This changeset updates the Bencher CLI to the new recommended usage.

@weiznich weiznich requested a review from a team November 5, 2024 15:00
@weiznich weiznich mentioned this pull request Nov 5, 2024
Open
prkbuilds
prkbuilds reviewed Nov 7, 2024
View reviewed changes
Copy link
Member

@prkbuilds prkbuilds left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi @epompeii , Thanks for the PR!

The changes are good overall just a formatting change I've suggested for you to look into.

.github/workflows/track_benches.yml Outdated
core.exportVariable("PR_HEAD", `${githubEvent.pull_request.head.ref}-${githubEvent.pull_request.head.sha.slice(0, 8)}`);
core.exportVariable("PR_ID", `${githubEvent.pull_request.head.ref}/${process.env.BENCHER_TESTBED}/${process.env.BENCHER_ADAPTER}`);
core.exportVariable("PR_NUMBER", githubEvent.number);
core.exportVariable("PR_HEAD", prEvent.pull_request.head.ref);
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The variable prEvent is not present here, we can use the githubEvent instead.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Great catch! Fixed.

@epompeii
Update Bencher CLI usage
3d3493f 
This changeset updates the Bencher CLI to the new recommended usage.

- Update deprecated CLI options: https://bencher.dev/docs/explanation/branch-selection/
- Safer and more succinct tracking from forks: https://bencher.dev/docs/how-to/github-actions/#pull-requests-from-forks
- Threshold creation as a part of `bencher run`: https://bencher.dev/docs/explanation/thresholds/
- Use `percentage` test: https://bencher.dev/docs/explanation/thresholds/#percentage
prkbuilds
prkbuilds approved these changes Nov 8, 2024
View reviewed changes
Copy link
Member

@prkbuilds prkbuilds left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the changes, LGTM

@weiznich weiznich added the run-benchmarks Used to indicate that github actions should run our benchmark suite label Nov 11, 2024
@github-actions GitHub Actions
Copy link

🐰 Bencher Report

Branchbencher_update-3d3493f6
Testbedubuntu-latest-mysql
Click to view all benchmark results
BenchmarkLatencyBenchmark Result
nanoseconds (ns)
(Result Δ%)		Upper Boundary
nanoseconds (ns)
(Limit %)
bench_insert/diesel/1📈 view plot
🚷 view threshold		607,690.00
(-5.03%)		1,151,778.82
(52.76%)
bench_insert/diesel/10📈 view plot
🚷 view threshold		767,450.00
(-0.55%)		839,589.71
(91.41%)
bench_insert/diesel/100📈 view plot
🚷 view threshold		1,609,000.00
(+0.03%)		1,616,447.27
(99.54%)
bench_insert/diesel/25📈 view plot
🚷 view threshold		901,010.00
(-1.08%)		1,067,252.32
(84.42%)
bench_insert/diesel/50📈 view plot
🚷 view threshold		1,183,000.00
(+0.29%)		1,234,386.18
(95.84%)
bench_loading_associations_sequentially/diesel/bench_loading_associations_sequentially📈 view plot
🚷 view threshold		16,565,000.00
(-0.72%)		18,600,792.65
(89.06%)
bench_medium_complex_query/diesel/1📈 view plot
🚷 view threshold		117,100.00
(+0.71%)		129,462.47
(90.45%)
bench_medium_complex_query/diesel/10📈 view plot
🚷 view threshold		153,280.00
(+1.03%)		176,515.49
(86.84%)
bench_medium_complex_query/diesel/100📈 view plot
🚷 view threshold		217,800.00
(+0.39%)		230,311.42
(94.57%)
bench_medium_complex_query/diesel/1000📈 view plot
🚷 view threshold		868,020.00
(+0.04%)		873,605.45
(99.36%)
bench_medium_complex_query/diesel/10000📈 view plot
🚷 view threshold		7,515,500.00
(+0.15%)		7,680,829.45
(97.85%)
bench_medium_complex_query/diesel_boxed/1📈 view plot
🚷 view threshold		120,060.00
(+1.50%)		146,423.34
(82.00%)
bench_medium_complex_query/diesel_boxed/10📈 view plot
🚷 view threshold		151,900.00
(-0.56%)		166,429.31
(91.27%)
bench_medium_complex_query/diesel_boxed/100📈 view plot
🚷 view threshold		220,290.00
(-0.14%)		225,611.78
(97.64%)
bench_medium_complex_query/diesel_boxed/1000📈 view plot
🚷 view threshold		869,160.00
(+0.19%)		893,065.74
(97.32%)
bench_medium_complex_query/diesel_boxed/10000📈 view plot
🚷 view threshold		7,557,800.00
(+0.38%)		7,979,315.62
(94.72%)
bench_medium_complex_query/diesel_queryable_by_name/1📈 view plot
🚷 view threshold		197,810.00
(+0.38%)		208,906.44
(94.69%)
bench_medium_complex_query/diesel_queryable_by_name/10📈 view plot
🚷 view threshold		237,530.00
(+1.00%)		272,681.13
(87.11%)
bench_medium_complex_query/diesel_queryable_by_name/100📈 view plot
🚷 view threshold		308,860.00
(+0.65%)		338,723.56
(91.18%)
bench_medium_complex_query/diesel_queryable_by_name/1000📈 view plot
🚷 view threshold		1,015,800.00
(+0.84%)		1,141,956.79
(88.95%)
bench_medium_complex_query/diesel_queryable_by_name/10000📈 view plot
🚷 view threshold		8,071,900.00
(-0.24%)		8,394,585.81
(96.16%)
bench_trivial_query/diesel/1📈 view plot
🚷 view threshold		65,494.00
(+0.03%)		65,791.89
(99.55%)
bench_trivial_query/diesel/10📈 view plot
🚷 view threshold		72,573.00
(+0.32%)		76,013.64
(95.47%)
bench_trivial_query/diesel/100📈 view plot
🚷 view threshold		136,120.00
(+0.31%)		142,375.71
(95.61%)
bench_trivial_query/diesel/1000📈 view plot
🚷 view threshold		699,760.00
(+0.54%)		756,061.38
(92.55%)
bench_trivial_query/diesel/10000📈 view plot
🚷 view threshold		6,103,100.00
(-0.44%)		6,563,476.35
(92.99%)
bench_trivial_query/diesel_boxed/1📈 view plot
🚷 view threshold		66,185.00
(-2.27%)		92,101.23
(71.86%)
bench_trivial_query/diesel_boxed/10📈 view plot
🚷 view threshold		73,649.00
(-2.28%)		102,640.04
(71.75%)
bench_trivial_query/diesel_boxed/100📈 view plot
🚷 view threshold		137,140.00
(+0.37%)		144,736.22
(94.75%)
bench_trivial_query/diesel_boxed/1000📈 view plot
🚷 view threshold		693,260.00
(-0.27%)		724,852.80
(95.64%)
bench_trivial_query/diesel_boxed/10000📈 view plot
🚷 view threshold		6,130,100.00
(-0.61%)		6,764,490.16
(90.62%)
bench_trivial_query/diesel_queryable_by_name/1📈 view plot
🚷 view threshold		128,020.00
(-0.41%)		136,974.11
(93.46%)
bench_trivial_query/diesel_queryable_by_name/10📈 view plot
🚷 view threshold		134,300.00
(-1.14%)		160,571.02
(83.64%)
bench_trivial_query/diesel_queryable_by_name/100📈 view plot
🚷 view threshold		198,120.00
(-0.79%)		224,813.38
(88.13%)
bench_trivial_query/diesel_queryable_by_name/1000📈 view plot
🚷 view threshold		780,570.00
(+0.32%)		817,434.00
(95.49%)
bench_trivial_query/diesel_queryable_by_name/10000📈 view plot
🚷 view threshold		6,338,700.00
(+0.13%)		6,463,069.45
(98.08%)
🐰 View full continuous benchmarking report in Bencher

@github-actions GitHub Actions
Copy link

🐰 Bencher Report

Branchbencher_update-3d3493f6
Testbedubuntu-latest-postgres
Click to view all benchmark results
BenchmarkLatencyBenchmark Result
nanoseconds (ns)
(Result Δ%)		Upper Boundary
nanoseconds (ns)
(Limit %)
bench_insert/diesel/1📈 view plot
🚷 view threshold		310,640.00
(+12.31%)		817,724.78
(37.99%)
bench_insert/diesel/10📈 view plot
🚷 view threshold		339,320.00
(+4.49%)		556,705.88
(60.95%)
bench_insert/diesel/100📈 view plot
🚷 view threshold		690,440.00
(+3.45%)		1,033,461.37
(66.81%)
bench_insert/diesel/25📈 view plot
🚷 view threshold		411,710.00
(+6.81%)		802,766.27
(51.29%)
bench_insert/diesel/50📈 view plot
🚷 view threshold		475,880.00
(-0.16%)		489,142.22
(97.29%)
bench_loading_associations_sequentially/diesel/bench_loading_associations_sequentially📈 view plot
🚷 view threshold		6,366,300.00
(-4.57%)		11,511,533.63
(55.30%)
bench_medium_complex_query/diesel/1📈 view plot
🚷 view threshold		85,902.00
(+3.45%)		128,582.32
(66.81%)
bench_medium_complex_query/diesel/10📈 view plot
🚷 view threshold		91,308.00
(+2.52%)		124,761.15
(73.19%)
bench_medium_complex_query/diesel/100📈 view plot
🚷 view threshold		163,870.00
(+12.81%)		440,983.01
(37.16%)
bench_medium_complex_query/diesel/1000📈 view plot
🚷 view threshold		446,310.00
(+1.79%)		562,934.29
(79.28%)
bench_medium_complex_query/diesel/10000📈 view plot
🚷 view threshold		3,449,700.00
(+0.34%)		3,623,221.45
(95.21%)
bench_medium_complex_query/diesel_boxed/1📈 view plot
🚷 view threshold		89,305.00
(+3.97%)		140,102.85
(63.74%)
bench_medium_complex_query/diesel_boxed/10📈 view plot
🚷 view threshold		94,945.00
(+3.06%)		136,977.41
(69.31%)
bench_medium_complex_query/diesel_boxed/100📈 view plot
🚷 view threshold		167,620.00
(+14.67%)		487,107.99
(34.41%)
bench_medium_complex_query/diesel_boxed/1000📈 view plot
🚷 view threshold		451,550.00
(+0.70%)		498,318.87
(90.61%)
bench_medium_complex_query/diesel_boxed/10000📈 view plot
🚷 view threshold		3,434,000.00
(-1.19%)		4,131,744.70
(83.11%)
bench_medium_complex_query/diesel_queryable_by_name/1📈 view plot
🚷 view threshold		246,540.00
(+0.18%)		253,019.13
(97.44%)
bench_medium_complex_query/diesel_queryable_by_name/10📈 view plot
🚷 view threshold		254,160.00
(+2.07%)		331,090.32
(76.76%)
bench_medium_complex_query/diesel_queryable_by_name/100📈 view plot
🚷 view threshold		338,580.00
(+5.93%)		620,906.10
(54.53%)
bench_medium_complex_query/diesel_queryable_by_name/1000📈 view plot
🚷 view threshold		646,550.00
(-1.50%)		813,214.68
(79.51%)
bench_medium_complex_query/diesel_queryable_by_name/10000📈 view plot
🚷 view threshold		3,780,800.00
(+0.71%)		4,175,505.44
(90.55%)
bench_trivial_query/diesel/1📈 view plot
🚷 view threshold		66,327.00
(+2.03%)		85,943.12
(77.18%)
bench_trivial_query/diesel/10📈 view plot
🚷 view threshold		69,767.00
(-0.03%)		70,113.34
(99.51%)
bench_trivial_query/diesel/100📈 view plot
🚷 view threshold		101,610.00
(-0.49%)		110,057.27
(92.32%)
bench_trivial_query/diesel/1000📈 view plot
🚷 view threshold		409,800.00
(-0.91%)		473,070.07
(86.63%)
bench_trivial_query/diesel/10000📈 view plot
🚷 view threshold		3,528,500.00
(-0.65%)		3,918,763.99
(90.04%)
bench_trivial_query/diesel_boxed/1📈 view plot
🚷 view threshold		67,272.00
(-2.25%)		93,424.76
(72.01%)
bench_trivial_query/diesel_boxed/10📈 view plot
🚷 view threshold		71,298.00
(-0.75%)		80,454.84
(88.62%)
bench_trivial_query/diesel_boxed/100📈 view plot
🚷 view threshold		103,930.00
(+0.36%)		109,515.45
(94.90%)
bench_trivial_query/diesel_boxed/1000📈 view plot
🚷 view threshold		412,520.00
(-0.02%)		413,618.15
(99.73%)
bench_trivial_query/diesel_boxed/10000📈 view plot
🚷 view threshold		3,580,900.00
(+0.53%)		3,860,172.72
(92.77%)
bench_trivial_query/diesel_queryable_by_name/1📈 view plot
🚷 view threshold		159,320.00
(+0.15%)		162,894.69
(97.81%)
bench_trivial_query/diesel_queryable_by_name/10📈 view plot
🚷 view threshold		158,710.00
(-0.62%)		175,351.13
(90.51%)
bench_trivial_query/diesel_queryable_by_name/100📈 view plot
🚷 view threshold		192,750.00
(-0.21%)		199,676.76
(96.53%)
bench_trivial_query/diesel_queryable_by_name/1000📈 view plot
🚷 view threshold		513,470.00
(-0.72%)		576,486.65
(89.07%)
bench_trivial_query/diesel_queryable_by_name/10000📈 view plot
🚷 view threshold		3,836,100.00
(+1.96%)		4,932,338.50
(77.77%)
🐰 View full continuous benchmarking report in Bencher

@github-actions GitHub Actions
Copy link

🐰 Bencher Report

Branchbencher_update-3d3493f6
Testbedubuntu-latest-sqlite
Click to view all benchmark results
BenchmarkLatencyBenchmark Result
nanoseconds (ns)
(Result Δ%)		Upper Boundary
nanoseconds (ns)
(Limit %)
bench_insert/diesel/1📈 view plot
🚷 view threshold		2,832.70
(+0.79%)		3,162.61
(89.57%)
bench_insert/diesel/10📈 view plot
🚷 view threshold		6,958.70
(-0.19%)		7,179.17
(96.93%)
bench_insert/diesel/100📈 view plot
🚷 view threshold		44,754.00
(+0.90%)		50,711.82
(88.25%)
bench_insert/diesel/25📈 view plot
🚷 view threshold		13,622.00
(+1.28%)		16,176.41
(84.21%)
bench_insert/diesel/50📈 view plot
🚷 view threshold		24,161.00
(+1.01%)		27,758.03
(87.04%)
bench_loading_associations_sequentially/diesel/bench_loading_associations_sequentially📈 view plot
🚷 view threshold		408,290.00
(+0.31%)		426,908.18
(95.64%)
bench_medium_complex_query/diesel/1📈 view plot
🚷 view threshold		3,357.50
(+0.44%)		3,577.94
(93.84%)
bench_medium_complex_query/diesel/10📈 view plot
🚷 view threshold		5,539.30
(+0.08%)		5,604.84
(98.83%)
bench_medium_complex_query/diesel/100📈 view plot
🚷 view threshold		28,978.00
(+0.07%)		29,290.79
(98.93%)
bench_medium_complex_query/diesel/1000📈 view plot
🚷 view threshold		253,880.00
(-0.10%)		258,188.11
(98.33%)
bench_medium_complex_query/diesel/10000📈 view plot
🚷 view threshold		2,490,300.00
(-0.00%)		2,491,989.45
(99.93%)
bench_medium_complex_query/diesel_boxed/1📈 view plot
🚷 view threshold		4,684.10
(+0.16%)		4,792.09
(97.75%)
bench_medium_complex_query/diesel_boxed/10📈 view plot
🚷 view threshold		6,859.70
(-0.09%)		6,965.29
(98.48%)
bench_medium_complex_query/diesel_boxed/100📈 view plot
🚷 view threshold		30,702.00
(+0.55%)		33,219.18
(92.42%)
bench_medium_complex_query/diesel_boxed/1000📈 view plot
🚷 view threshold		256,920.00
(+0.15%)		262,728.87
(97.79%)
bench_medium_complex_query/diesel_boxed/10000📈 view plot
🚷 view threshold		2,500,000.00
(+0.21%)		2,578,941.09
(96.94%)
bench_medium_complex_query/diesel_queryable_by_name/1📈 view plot
🚷 view threshold		18,205.00
(-0.49%)		19,725.51
(92.29%)
bench_medium_complex_query/diesel_queryable_by_name/10📈 view plot
🚷 view threshold		20,795.00
(-0.27%)		21,757.99
(95.57%)
bench_medium_complex_query/diesel_queryable_by_name/100📈 view plot
🚷 view threshold		48,098.00
(-0.40%)		51,324.86
(93.71%)
bench_medium_complex_query/diesel_queryable_by_name/1000📈 view plot
🚷 view threshold		313,250.00
(+0.04%)		314,888.40
(99.48%)
bench_medium_complex_query/diesel_queryable_by_name/10000📈 view plot
🚷 view threshold		2,919,100.00
(+0.20%)		3,005,488.36
(97.13%)
bench_trivial_query/diesel/1📈 view plot
🚷 view threshold		955.21
(+2.38%)		1,286.17
(74.27%)
bench_trivial_query/diesel/10📈 view plot
🚷 view threshold		2,988.70
(+1.03%)		3,443.73
(86.79%)
bench_trivial_query/diesel/100📈 view plot
🚷 view threshold		24,196.00
(+1.12%)		28,172.84
(85.88%)
bench_trivial_query/diesel/1000📈 view plot
🚷 view threshold		225,160.00
(+0.99%)		258,002.47
(87.27%)
bench_trivial_query/diesel/10000📈 view plot
🚷 view threshold		2,217,700.00
(+0.86%)		2,500,696.35
(88.68%)
bench_trivial_query/diesel_boxed/1📈 view plot
🚷 view threshold		1,491.10
(+0.08%)		1,509.72
(98.77%)
bench_trivial_query/diesel_boxed/10📈 view plot
🚷 view threshold		3,542.60
(+0.28%)		3,692.29
(95.95%)
bench_trivial_query/diesel_boxed/100📈 view plot
🚷 view threshold		24,741.00
(+1.41%)		29,857.28
(82.86%)
bench_trivial_query/diesel_boxed/1000📈 view plot
🚷 view threshold		227,400.00
(+0.60%)		247,507.64
(91.88%)
bench_trivial_query/diesel_boxed/10000📈 view plot
🚷 view threshold		2,216,200.00
(+0.43%)		2,356,208.72
(94.06%)
bench_trivial_query/diesel_queryable_by_name/1📈 view plot
🚷 view threshold		5,481.10
(+0.53%)		5,907.83
(92.78%)
bench_trivial_query/diesel_queryable_by_name/10📈 view plot
🚷 view threshold		7,963.90
(+0.12%)		8,106.89
(98.24%)
bench_trivial_query/diesel_queryable_by_name/100📈 view plot
🚷 view threshold		32,197.00
(+0.15%)		32,934.28
(97.76%)
bench_trivial_query/diesel_queryable_by_name/1000📈 view plot
🚷 view threshold		265,560.00
(-0.09%)		269,445.75
(98.56%)
bench_trivial_query/diesel_queryable_by_name/10000📈 view plot
🚷 view threshold		2,563,500.00
(+0.38%)		2,707,232.36
(94.69%)
🐰 View full continuous benchmarking report in Bencher

@weiznich weiznich added this pull request to the merge queue Nov 11, 2024
Merged via the queue into diesel-rs:master with commit 7e55d40 Nov 11, 2024
51 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@prkbuilds prkbuilds prkbuilds approved these changes

Assignees
No one assigned
Labels
run-benchmarks Used to indicate that github actions should run our benchmark suite
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@epompeii @prkbuilds @weiznich