This repository has been archived by the owner on Oct 22, 2021. It is now read-only.
-
Notifications
You must be signed in to change notification settings - Fork 0
benchmark
Peng Hui Jiang edited this page May 16, 2019
·
17 revisions
To summarise the benchmarks we wanted from the call:
- test cluster: test-fdb-us-south-001
- code: https://github.com/cloudant/fdblucene-perf/compare/put-together-using-enwiki?expand=1
- sample data: /srv/rnewson/enwiki-20190201-pages-articles-multistream.xml
- indexing using documents from EnWiki
java -jar target/benchmarks.jar -t 2 -jvmArgs '-Djmh.ignoreLock=true -Ddir=/tmp -Xms2G -Xmx2G' EnWikiIndexingBenchmark
Benchmark Mode Cnt Score Error Units
EnWikiIndexingBenchmark.indexFDB thrpt 3 271.998 ? 324.467 ops/s
EnWikiIndexingBenchmark.indexNIOS thrpt 3 1516.231 ? 2169.895 ops/s
- indexing using documents from EnWiki DEFAULT_LINE_DOCS_FILE = "europarl.lines.txt.gz"
Benchmark (bigDocs) Mode Cnt Score Error Units
IndexingBenchmark.FDBIndexingBenchmark.indexing true thrpt 3 2607.064 ? 1460.962 ops/s
IndexingBenchmark.FDBIndexingBenchmark.indexing:docNum true thrpt 3 2686.327 ? 529.707 ops/s
IndexingBenchmark.FDBIndexingBenchmark.indexing:docLength true thrpt 3 5707318.133 ? 1178704.597 ops/s
IndexingBenchmark.FDBIndexingBenchmark.indexing false thrpt 3 244002.487 ? 88839.433 ops/s
IndexingBenchmark.FDBIndexingBenchmark.indexing:docNum false thrpt 3 246176.659 ? 100329.031 ops/s
IndexingBenchmark.FDBIndexingBenchmark.indexing:docLength false thrpt 3 17050077.472 ? 7032977.312 ops/s
IndexingBenchmark.NIOFSIndexingBenchmark.indexing true thrpt 3 7586.677 ? 1239.510 ops/s
IndexingBenchmark.NIOFSIndexingBenchmark.indexing:docNum true thrpt 3 7601.922 ? 1015.215 ops/s
IndexingBenchmark.NIOFSIndexingBenchmark.indexing:docLength true thrpt 3 16157372.737 ? 2100179.554 ops/s
IndexingBenchmark.NIOFSIndexingBenchmark.indexing false thrpt 3 757174.319 ? 266604.514 ops/s
IndexingBenchmark.NIOFSIndexingBenchmark.indexing:docNum false thrpt 3 757912.638 ? 262555.035 ops/s
IndexingBenchmark.NIOFSIndexingBenchmark.indexing:docLength false thrpt 3 52868778.958 ? 18378399.611 ops/s
- searching using documents from EnWiki
public IndexSearcher(IndexReader r)
Benchmark Mode Cnt Score Error Units
BasicSearchBenchmark.FDBSearchBenchmark.searchFDB thrpt 3 10626.419 ? 853.438 ops/s
BasicSearchBenchmark.NioSearchBenchmark.searchNIOS thrpt 3 19985.663 ? 423.869 ops/s
GroupSearchBenchmark.FDBGroupSearchBenchmark.searchFDBByGroup thrpt 3 1451.925 ? 94.405 ops/s
GroupSearchBenchmark.NioGroupSearchBenchmark.searchNIOSByGroup thrpt 3 1267.278 ? 38.461 ops/s
SortSearchBenchmark.FDBSortSearchBenchmark.searchFDBBySort thrpt 3 3171.941 ? 32.098 ops/s
SortSearchBenchmark.NioSortSearchBenchmark.searchNIOSBySort thrpt 3 2681.572 ? 197.088 ops/s
- searching using documents from EnWiki
public IndexSearcher(IndexReader r, ExecutorService executor)
the search benchmark to compare public IndexSearcher(IndexReader r) which is sequential segment access versus public IndexSearcher(IndexReader r, ExecutorService executor) which is parallel segment access (and for clarity, use Executors.newCachedThreadPool() as the executor).
# Run complete. Total time: 00:28:23
REMEMBER: The numbers below are just data. To gain reusable insights, you need to follow up on
why the numbers are the way they are. Use profilers (see -prof, -lprof), design factorial
experiments, perform baseline and negative tests that provide experimental control, make sure
the benchmarking environment is safe on JVM/OS/HW level, ask for reviews from the domain experts.
Do not assume the numbers tell you what you want them to tell.
Benchmark Mode Cnt Score Error Units
BasicSearchBenchmark.FDBSearchBenchmark.searchFDB thrpt 3 5868.085 ? 568.627 ops/s
BasicSearchBenchmark.NioSearchBenchmark.searchNIOS thrpt 3 7104.216 ? 892.286 ops/s
GroupSearchBenchmark.FDBGroupSearchBenchmark.searchFDBByGroup thrpt 3 1461.581 ? 110.129 ops/s
GroupSearchBenchmark.NioGroupSearchBenchmark.searchNIOSByGroup thrpt 3 1246.521 ? 145.460 ops/s
SortSearchBenchmark.FDBSortSearchBenchmark.searchFDBBySort thrpt 3 3749.377 ? 157.771 ops/s
SortSearchBenchmark.NioSortSearchBenchmark.searchNIOSBySort thrpt 3 3571.791 ? 416.813 ops/s
- the indexing benchmark using batch mode (multiple parallel indexes being written and then merged at the end).
TBD
- the indexing with
Mode.SampleTimein milliseconds.
Benchmark (bigDocs) Mode Cnt Score Error Units
IndexingBenchmark.FDBIndexingBenchmark.indexing true sample 475132 0.001 ? 0.001 s/op
IndexingBenchmark.FDBIndexingBenchmark.indexing:indexing?p0.00 true sample ? 10?? s/op
IndexingBenchmark.FDBIndexingBenchmark.indexing:indexing?p0.50 true sample ? 10?? s/op
IndexingBenchmark.FDBIndexingBenchmark.indexing:indexing?p0.90 true sample ? 10?? s/op
IndexingBenchmark.FDBIndexingBenchmark.indexing:indexing?p0.95 true sample ? 10?? s/op
IndexingBenchmark.FDBIndexingBenchmark.indexing:indexing?p0.99 true sample 0.003 s/op
IndexingBenchmark.FDBIndexingBenchmark.indexing:indexing?p0.999 true sample 0.030 s/op
IndexingBenchmark.FDBIndexingBenchmark.indexing:indexing?p0.9999 true sample 3.068 s/op
IndexingBenchmark.FDBIndexingBenchmark.indexing:indexing?p1.00 true sample 5.134 s/op
IndexingBenchmark.FDBIndexingBenchmark.indexing false sample 11287502 ? 10?? s/op
IndexingBenchmark.FDBIndexingBenchmark.indexing:indexing?p0.00 false sample ? 10?? s/op
IndexingBenchmark.FDBIndexingBenchmark.indexing:indexing?p0.50 false sample ? 10?? s/op
IndexingBenchmark.FDBIndexingBenchmark.indexing:indexing?p0.90 false sample ? 10?? s/op
IndexingBenchmark.FDBIndexingBenchmark.indexing:indexing?p0.95 false sample ? 10?? s/op
IndexingBenchmark.FDBIndexingBenchmark.indexing:indexing?p0.99 false sample ? 10?? s/op
IndexingBenchmark.FDBIndexingBenchmark.indexing:indexing?p0.999 false sample ? 10?? s/op
IndexingBenchmark.FDBIndexingBenchmark.indexing:indexing?p0.9999 false sample 0.001 s/op
IndexingBenchmark.FDBIndexingBenchmark.indexing:indexing?p1.00 false sample 5.369 s/op
IndexingBenchmark.NIOFSIndexingBenchmark.indexing true sample 1383485 ? 10?? s/op
IndexingBenchmark.NIOFSIndexingBenchmark.indexing:indexing?p0.00 true sample ? 10?? s/op
IndexingBenchmark.NIOFSIndexingBenchmark.indexing:indexing?p0.50 true sample ? 10?? s/op
IndexingBenchmark.NIOFSIndexingBenchmark.indexing:indexing?p0.90 true sample ? 10?? s/op
IndexingBenchmark.NIOFSIndexingBenchmark.indexing:indexing?p0.95 true sample ? 10?? s/op
IndexingBenchmark.NIOFSIndexingBenchmark.indexing:indexing?p0.99 true sample 0.003 s/op
IndexingBenchmark.NIOFSIndexingBenchmark.indexing:indexing?p0.999 true sample 0.012 s/op
IndexingBenchmark.NIOFSIndexingBenchmark.indexing:indexing?p0.9999 true sample 0.294 s/op
IndexingBenchmark.NIOFSIndexingBenchmark.indexing:indexing?p1.00 true sample 0.395 s/op
IndexingBenchmark.NIOFSIndexingBenchmark.indexing false sample 8366029 ? 10?? s/op
IndexingBenchmark.NIOFSIndexingBenchmark.indexing:indexing?p0.00 false sample ? 10?? s/op
IndexingBenchmark.NIOFSIndexingBenchmark.indexing:indexing?p0.50 false sample ? 10?? s/op
IndexingBenchmark.NIOFSIndexingBenchmark.indexing:indexing?p0.90 false sample ? 10?? s/op
IndexingBenchmark.NIOFSIndexingBenchmark.indexing:indexing?p0.95 false sample ? 10?? s/op
IndexingBenchmark.NIOFSIndexingBenchmark.indexing:indexing?p0.99 false sample ? 10?? s/op
IndexingBenchmark.NIOFSIndexingBenchmark.indexing:indexing?p0.999 false sample ? 10?? s/op
IndexingBenchmark.NIOFSIndexingBenchmark.indexing:indexing?p0.9999 false sample 0.001 s/op
IndexingBenchmark.NIOFSIndexingBenchmark.indexing:indexing?p1.00 false sample 0.244 s/op
Benchmark Mode Cnt Score Error Units
BasicSearchBenchmark.FDBSearchBenchmark.searchFDB sample 1042326 ? 10?? s/op
BasicSearchBenchmark.FDBSearchBenchmark.searchFDB:searchFDB?p0.00 sample ? 10?? s/op
BasicSearchBenchmark.FDBSearchBenchmark.searchFDB:searchFDB?p0.50 sample ? 10?? s/op
BasicSearchBenchmark.FDBSearchBenchmark.searchFDB:searchFDB?p0.90 sample 0.001 s/op
BasicSearchBenchmark.FDBSearchBenchmark.searchFDB:searchFDB?p0.95 sample 0.001 s/op
BasicSearchBenchmark.FDBSearchBenchmark.searchFDB:searchFDB?p0.99 sample 0.002 s/op
BasicSearchBenchmark.FDBSearchBenchmark.searchFDB:searchFDB?p0.999 sample 0.002 s/op
BasicSearchBenchmark.FDBSearchBenchmark.searchFDB:searchFDB?p0.9999 sample 0.012 s/op
BasicSearchBenchmark.FDBSearchBenchmark.searchFDB:searchFDB?p1.00 sample 0.028 s/op
BasicSearchBenchmark.NioSearchBenchmark.searchNIOS sample 1277962 ? 10?? s/op
BasicSearchBenchmark.NioSearchBenchmark.searchNIOS:searchNIOS?p0.00 sample ? 10?? s/op
BasicSearchBenchmark.NioSearchBenchmark.searchNIOS:searchNIOS?p0.50 sample ? 10?? s/op
BasicSearchBenchmark.NioSearchBenchmark.searchNIOS:searchNIOS?p0.90 sample 0.001 s/op
BasicSearchBenchmark.NioSearchBenchmark.searchNIOS:searchNIOS?p0.95 sample 0.001 s/op
BasicSearchBenchmark.NioSearchBenchmark.searchNIOS:searchNIOS?p0.99 sample 0.002 s/op
BasicSearchBenchmark.NioSearchBenchmark.searchNIOS:searchNIOS?p0.999 sample 0.002 s/op
BasicSearchBenchmark.NioSearchBenchmark.searchNIOS:searchNIOS?p0.9999 sample 0.014 s/op
BasicSearchBenchmark.NioSearchBenchmark.searchNIOS:searchNIOS?p1.00 sample 0.025 s/op
GroupSearchBenchmark.FDBGroupSearchBenchmark.searchFDBByGroup sample 262824 0.001 ? 0.001 s/op
GroupSearchBenchmark.FDBGroupSearchBenchmark.searchFDBByGroup:searchFDBByGroup?p0.00 sample ? 10?? s/op
GroupSearchBenchmark.FDBGroupSearchBenchmark.searchFDBByGroup:searchFDBByGroup?p0.50 sample 0.001 s/op
GroupSearchBenchmark.FDBGroupSearchBenchmark.searchFDBByGroup:searchFDBByGroup?p0.90 sample 0.003 s/op
GroupSearchBenchmark.FDBGroupSearchBenchmark.searchFDBByGroup:searchFDBByGroup?p0.95 sample 0.003 s/op
GroupSearchBenchmark.FDBGroupSearchBenchmark.searchFDBByGroup:searchFDBByGroup?p0.99 sample 0.005 s/op
GroupSearchBenchmark.FDBGroupSearchBenchmark.searchFDBByGroup:searchFDBByGroup?p0.999 sample 0.008 s/op
GroupSearchBenchmark.FDBGroupSearchBenchmark.searchFDBByGroup:searchFDBByGroup?p0.9999 sample 0.013 s/op
GroupSearchBenchmark.FDBGroupSearchBenchmark.searchFDBByGroup:searchFDBByGroup?p1.00 sample 0.023 s/op
GroupSearchBenchmark.NioGroupSearchBenchmark.searchNIOSByGroup sample 209195 0.002 ? 0.001 s/op
GroupSearchBenchmark.NioGroupSearchBenchmark.searchNIOSByGroup:searchNIOSByGroup?p0.00 sample ? 10?? s/op
GroupSearchBenchmark.NioGroupSearchBenchmark.searchNIOSByGroup:searchNIOSByGroup?p0.50 sample 0.002 s/op
GroupSearchBenchmark.NioGroupSearchBenchmark.searchNIOSByGroup:searchNIOSByGroup?p0.90 sample 0.003 s/op
GroupSearchBenchmark.NioGroupSearchBenchmark.searchNIOSByGroup:searchNIOSByGroup?p0.95 sample 0.004 s/op
GroupSearchBenchmark.NioGroupSearchBenchmark.searchNIOSByGroup:searchNIOSByGroup?p0.99 sample 0.005 s/op
GroupSearchBenchmark.NioGroupSearchBenchmark.searchNIOSByGroup:searchNIOSByGroup?p0.999 sample 0.009 s/op
GroupSearchBenchmark.NioGroupSearchBenchmark.searchNIOSByGroup:searchNIOSByGroup?p0.9999 sample 0.017 s/op
GroupSearchBenchmark.NioGroupSearchBenchmark.searchNIOSByGroup:searchNIOSByGroup?p1.00 sample 0.032 s/op
SortSearchBenchmark.FDBSortSearchBenchmark.searchFDBBySort sample 690983 0.001 ? 0.001 s/op
SortSearchBenchmark.FDBSortSearchBenchmark.searchFDBBySort:searchFDBBySort?p0.00 sample ? 10?? s/op
SortSearchBenchmark.FDBSortSearchBenchmark.searchFDBBySort:searchFDBBySort?p0.50 sample ? 10?? s/op
SortSearchBenchmark.FDBSortSearchBenchmark.searchFDBBySort:searchFDBBySort?p0.90 sample 0.001 s/op
SortSearchBenchmark.FDBSortSearchBenchmark.searchFDBBySort:searchFDBBySort?p0.95 sample 0.001 s/op
SortSearchBenchmark.FDBSortSearchBenchmark.searchFDBBySort:searchFDBBySort?p0.99 sample 0.002 s/op
SortSearchBenchmark.FDBSortSearchBenchmark.searchFDBBySort:searchFDBBySort?p0.999 sample 0.003 s/op
SortSearchBenchmark.FDBSortSearchBenchmark.searchFDBBySort:searchFDBBySort?p0.9999 sample 0.012 s/op
SortSearchBenchmark.FDBSortSearchBenchmark.searchFDBBySort:searchFDBBySort?p1.00 sample 0.032 s/op
SortSearchBenchmark.NioSortSearchBenchmark.searchNIOSBySort sample 627134 0.001 ? 0.001 s/op
SortSearchBenchmark.NioSortSearchBenchmark.searchNIOSBySort:searchNIOSBySort?p0.00 sample ? 10?? s/op
SortSearchBenchmark.NioSortSearchBenchmark.searchNIOSBySort:searchNIOSBySort?p0.50 sample ? 10?? s/op
SortSearchBenchmark.NioSortSearchBenchmark.searchNIOSBySort:searchNIOSBySort?p0.90 sample 0.001 s/op
SortSearchBenchmark.NioSortSearchBenchmark.searchNIOSBySort:searchNIOSBySort?p0.95 sample 0.001 s/op
SortSearchBenchmark.NioSortSearchBenchmark.searchNIOSBySort:searchNIOSBySort?p0.99 sample 0.002 s/op
SortSearchBenchmark.NioSortSearchBenchmark.searchNIOSBySort:searchNIOSBySort?p0.999 sample 0.002 s/op
SortSearchBenchmark.NioSortSearchBenchmark.searchNIOSBySort:searchNIOSBySort?p0.9999 sample 0.014 s/op
SortSearchBenchmark.NioSortSearchBenchmark.searchNIOSBySort:searchNIOSBySort?p1.00 sample 0.030 s/op
- page size changes (for searching and indexing).
TBD