-
Notifications
You must be signed in to change notification settings - Fork 1.2k
Performance history
Andres Riancho edited this page Sep 2, 2014
·
9 revisions
This page holds historical performance information captured by collector and analyzed by w3af-performance-analysis.
The raw data is stored in a private S3 bucket named w3af-performance-data. If you're interested in getting that information, please ask Andrés for access.
Performance analysis
====================
Total memory size
0: 32.4MiB
1: 67.3MiB
2: 67.3MiB
Memory usage summary
First:
Total 201138 objects, 516 types, Total size = 32.4MiB (33995558 bytes)
Index Count % Size % Cum Max Kind
0 7500 3 8082336 23 23 196888 dict
1 61171 30 5746127 16 40 12467 str
2 1324 0 3303360 9 50 49488 module
3 37725 18 2647940 7 58 9236 unicode
4 18522 9 2370816 6 65 128 code
5 18943 9 2273160 6 71 120 function
6 2479 1 2241016 6 78 904 type
7 13952 6 1059872 3 81 3184 tuple
8 7509 3 854704 2 84 20192 list
9 524 0 468704 1 85 8424 set
10 1024 0 360448 1 86 352 U32
11 356 0 321824 0 87 904 Packet_metaclass
12 3423 1 301224 0 88 88 weakref
13 8685 4 208440 0 88 24 int
14 165 0 176736 0 89 1120 ByteEnumField
15 376 0 170648 0 89 12584 OrderedDict
16 122 0 133184 0 90 1120 BitField
17 110 0 122320 0 90 1112 Distribution
18 108 0 118272 0 91 1120 ShortEnumField
19 1428 0 114240 0 91 80 wrapper_descriptor
Before last:
Total 287156 objects, 570 types, Total size = 67.3MiB (70519089 bytes)
Index Count % Size % Cum Max Kind
0 60154 20 20661876 29 29 1624232 unicode
1 22196 7 12532448 17 47 196888 dict
2 70726 24 7423770 10 57 120508 str
3 5839 2 6473000 9 66 1112 URL
4 1325 0 3306768 4 71 49488 module
5 18532 6 2372096 3 74 128 code
6 18980 6 2277600 3 78 120 function
7 2484 0 2245536 3 81 904 type
8 18081 6 1841920 2 83 20192 list
9 5768 2 1708864 2 86 1064 QueryString
10 20345 7 1533344 2 88 3184 tuple
11 701 0 978248 1 89 8424 set
12 1024 0 360448 0 90 352 U32
13 356 0 321824 0 90 904 Packet_metaclass
14 3523 1 310024 0 91 88 weakref
15 619 0 305688 0 91 624 frame
16 272 0 292864 0 92 1112 FuzzableRequest
17 731 0 265288 0 92 1112 _Condition
18 9322 3 223728 0 92 24 int
19 1987 0 190752 0 93 96 lxml.etree._LogEntry
Operating System information
Load average: 5.9 6.79 4.25
Network
Bytes sent: 4.8M
Bytes received: 305.0M
Swap memory
% used (best: 0.0): 0.0
Pages per second (out) (best: 0): 0
Pages per second (in) (best: 0): 0
CPU usage by function
Measurement #0 (w3af-10272-2014-02-09-01_16.cpu)
0: cpu_usage.py:56(dump_data)
1: utils.py:34(dump_data_every_thread)
Measurement #1 (w3af-10272-2014-02-09-01_18.cpu)
0: extended_urllib.py:434(ExtendedUrllib._send)
1: extended_urllib.py:252(ExtendedUrllib.GET)
2: plugin.py:217(meth)
3: threadpool.py:42(one_to_many.__call__)
4: web_spider.py:345(web_spider._verify_reference)
5: urllib2.py:386(OpenerDirector.open)
6: pool.py:64(mapstar)
7: utils.py:34(dump_data_every_thread)
8: memory_usage.py:33(inner)
9: memory_usage.py:42(start_memory_profiling)
10: memory_usage.py:53(dump_objects)
11: scanner.py:123(dump_all_objects)
12: pstats.py:451(TupleComp)
13: base_consumer.py:42(crawl_infrastructure._wrapper)
14: url.py:173(URL.__init__)
15: urllib2.py:414(OpenerDirector._open)
16: HTTPResponse.py:127(from_httplib_resp)
17: urllib2.py:374(OpenerDirector._call_chain)
18: web_spider.py:170(web_spider._urls_to_verify_generator)
19: keepalive.py:779(HTTPHandler.http_open)
Measurement #2 (w3af-10272-2014-02-09-01_20.cpu)
0: extended_urllib.py:434(ExtendedUrllib._send)
1: extended_urllib.py:252(ExtendedUrllib.GET)
2: urllib2.py:386(OpenerDirector.open)
3: plugin.py:217(meth)
4: web_spider.py:345(web_spider._verify_reference)
5: threadpool.py:42(one_to_many.__call__)
6: pool.py:64(mapstar)
7: url.py:173(URL.__init__)
8: web_spider.py:170(web_spider._urls_to_verify_generator)
9: HTTPResponse.py:127(from_httplib_resp)
10: urllib2.py:414(OpenerDirector._open)
11: urllib2.py:374(OpenerDirector._call_chain)
12: base_consumer.py:42(crawl_infrastructure._wrapper)
13: utils.py:34(dump_data_every_thread)
14: url.py:47(URL.changed_wrapper)
15: memory_usage.py:33(inner)
16: memory_usage.py:42(start_memory_profiling)
17: memory_usage.py:53(dump_objects)
18: scanner.py:123(dump_all_objects)
19: pstats.py:451(TupleComp)
Measurement #3 (w3af-10272-2014-02-09-01_23.cpu)
0: extended_urllib.py:434(ExtendedUrllib._send)
1: plugin.py:217(meth)
2: extended_urllib.py:252(ExtendedUrllib.GET)
3: urllib2.py:386(OpenerDirector.open)
4: base_consumer.py:42(crawl_infrastructure._wrapper)
5: web_spider.py:345(web_spider._verify_reference)
6: url.py:173(URL.__init__)
7: threadpool.py:42(one_to_many.__call__)
8: web_spider.py:170(web_spider._urls_to_verify_generator)
9: pool.py:64(mapstar)
10: HTTPResponse.py:127(from_httplib_resp)
11: url.py:47(URL.changed_wrapper)
12: urllib2.py:414(OpenerDirector._open)
13: urllib2.py:374(OpenerDirector._call_chain)
14: web_spider.py:270(web_spider._should_output_extracted_url)
15: headers.py:34(Headers.__init__)
16: db.py:67(store_in_cache)
17: kv_container.py:45(QueryString.__init__)
18: url.py:306(URL.set_querystring)
19: parser_cache.py:51(ParserCache.get_document_parser_for)
Measurement #4 (w3af-10272-2014-02-09-01_25.cpu)
0: urllib2.py:386(OpenerDirector.open)
1: extended_urllib.py:434(ExtendedUrllib._send)
2: plugin.py:217(meth)
3: extended_urllib.py:252(ExtendedUrllib.GET)
4: url.py:173(URL.__init__)
5: web_spider.py:170(web_spider._urls_to_verify_generator)
6: base_consumer.py:42(crawl_infrastructure._wrapper)
7: url.py:47(URL.changed_wrapper)
8: web_spider.py:345(web_spider._verify_reference)
9: threadpool.py:42(one_to_many.__call__)
10: HTTPResponse.py:127(from_httplib_resp)
11: pool.py:64(mapstar)
12: web_spider.py:270(web_spider._should_output_extracted_url)
13: pool.py:298(Pool.map_async)
14: parser_cache.py:51(ParserCache.get_document_parser_for)
15: document_parser.py:40(DocumentParser.__init__)
16: html.py:45(HTMLParser.__init__)
17: sgml.py:62(HTMLParser.__init__)
18: sgml.py:137(HTMLParser._parse)
19: headers.py:34(Headers.__init__)
Core status summary
Measurement #0 (w3af-10272-2014-02-09-01_18.core)
Audit queue input speed: 139.36
Requests per minute: 510
Audit queue output speed: 64.43
Crawl queue size: 29
Crawl queue input speed: 139.37
Crawl queue output speed: 135.73
Audit queue size: 81
Requests sent: 1022
Measurement #1 (w3af-10272-2014-02-09-01_20.core)
Audit queue input speed: 137.82
Requests per minute: 412
Audit queue output speed: 47.49
Crawl queue size: 117
Crawl queue input speed: 137.82
Crawl queue output speed: 48.28
Audit queue size: 152
Requests sent: 1650
Measurement #2 (w3af-10272-2014-02-09-01_22.core)
Audit queue input speed: None
Requests per minute: 331
Audit queue output speed: None
Crawl queue size: None
Crawl queue input speed: None
Crawl queue output speed: None
Audit queue size: None
Requests sent: 1990
Measurement #3 (w3af-10272-2014-02-09-01_24.core)
Audit queue input speed: None
Requests per minute: 312
Audit queue output speed: None
Crawl queue size: None
Crawl queue input speed: None
Crawl queue output speed: None
Audit queue size: None
Requests sent: 2507
Performance analysis
====================
Total memory size
0: 32.4MiB
1: 75.2MiB
2: 75.2MiB
Memory usage summary
First:
Total 201092 objects, 516 types, Total size = 32.4MiB (33992123 bytes)
Index Count % Size % Cum Max Kind
0 7500 3 8082336 23 23 196888 dict
1 61154 30 5745388 16 40 12467 str
2 1324 0 3303360 9 50 49488 module
3 37725 18 2647940 7 58 9236 unicode
4 18522 9 2370816 6 65 128 code
5 18943 9 2273160 6 71 120 function
6 2479 1 2241016 6 78 904 type
7 13924 6 1057200 3 81 3184 tuple
8 7509 3 854704 2 84 20192 list
9 524 0 468704 1 85 8424 set
10 1024 0 360448 1 86 352 U32
11 356 0 321824 0 87 904 Packet_metaclass
12 3423 1 301224 0 88 88 weakref
13 8685 4 208440 0 88 24 int
14 165 0 176736 0 89 1120 ByteEnumField
15 376 0 170648 0 89 12584 OrderedDict
16 122 0 133184 0 90 1120 BitField
17 110 0 122320 0 90 1112 Distribution
18 108 0 118272 0 91 1120 ShortEnumField
19 1428 0 114240 0 91 80 wrapper_descriptor
Before last:
Total 292330 objects, 567 types, Total size = 75.2MiB (78810120 bytes)
Index Count % Size % Cum Max Kind
0 61531 21 27484324 34 34 1375644 unicode
1 23218 7 12834736 16 51 196888 dict
2 70993 24 7833113 9 61 343954 str
3 5773 1 6409208 8 69 1112 URL
4 1325 0 3306768 4 73 49488 module
5 18530 6 2371840 3 76 128 code
6 18989 6 2278680 2 79 120 function
7 2484 0 2245536 2 82 904 type
8 18974 6 1892504 2 84 20192 list
9 5724 1 1695072 2 86 1064 QueryString
10 21068 7 1587160 2 88 3184 tuple
11 689 0 935528 1 89 8424 set
12 565 0 620600 0 90 1112 FuzzableRequest
13 1024 0 360448 0 91 352 U32
14 724 0 356656 0 91 624 frame
15 356 0 321824 0 92 904 Packet_metaclass
16 3522 1 309936 0 92 88 weakref
17 628 0 274208 0 92 1064 Headers
18 643 0 231176 0 93 1112 _Condition
19 9262 3 222288 0 93 24 int
Operating System information
Load average: 8.49 8.48 5.02
Network
Bytes sent: 5.1M
Bytes received: 306.6M
Swap memory
% used (best: 0.0): 0.0
Pages per second (out) (best: 0): 0
Pages per second (in) (best: 0): 0
CPU usage by function
Measurement #0 (w3af-10284-2014-02-09-01_16.cpu)
0: cpu_usage.py:56(dump_data)
1: utils.py:34(dump_data_every_thread)
Measurement #1 (w3af-10284-2014-02-09-01_19.cpu)
0: extended_urllib.py:434(ExtendedUrllib._send)
1: extended_urllib.py:252(ExtendedUrllib.GET)
2: plugin.py:217(meth)
3: threadpool.py:42(one_to_many.__call__)
4: web_spider.py:345(web_spider._verify_reference)
5: pool.py:64(mapstar)
6: urllib2.py:386(OpenerDirector.open)
7: utils.py:34(dump_data_every_thread)
8: memory_usage.py:33(inner)
9: memory_usage.py:42(start_memory_profiling)
10: memory_usage.py:53(dump_objects)
11: scanner.py:123(dump_all_objects)
12: pstats.py:451(TupleComp)
13: HTTPResponse.py:127(from_httplib_resp)
14: url.py:173(URL.__init__)
15: urllib2.py:414(OpenerDirector._open)
16: urllib2.py:374(OpenerDirector._call_chain)
17: base_consumer.py:42(crawl_infrastructure._wrapper)
18: headers.py:34(Headers.__init__)
19: url.py:47(URL.changed_wrapper)
Measurement #2 (w3af-10284-2014-02-09-01_21.cpu)
0: web_spider.py:345(web_spider._verify_reference)
1: extended_urllib.py:434(ExtendedUrllib._send)
2: extended_urllib.py:252(ExtendedUrllib.GET)
3: plugin.py:217(meth)
4: threadpool.py:42(one_to_many.__call__)
5: urllib2.py:386(OpenerDirector.open)
6: pool.py:64(mapstar)
7: HTTPResponse.py:127(from_httplib_resp)
8: url.py:173(URL.__init__)
9: base_consumer.py:42(crawl_infrastructure._wrapper)
10: urllib2.py:374(OpenerDirector._call_chain)
11: urllib2.py:414(OpenerDirector._open)
12: headers.py:34(Headers.__init__)
13: web_spider.py:170(web_spider._urls_to_verify_generator)
14: url.py:47(URL.changed_wrapper)
15: cache.py:78(CacheHandler.http_response)
16: db.py:67(store_in_cache)
17: utils.py:34(dump_data_every_thread)
18: pool.py:298(Pool.map_async)
19: memory_usage.py:33(inner)
Measurement #3 (w3af-10284-2014-02-09-01_23.cpu)
0: extended_urllib.py:434(ExtendedUrllib._send)
1: web_spider.py:345(web_spider._verify_reference)
2: plugin.py:217(meth)
3: extended_urllib.py:252(ExtendedUrllib.GET)
4: threadpool.py:42(one_to_many.__call__)
5: urllib2.py:386(OpenerDirector.open)
6: pool.py:64(mapstar)
7: HTTPResponse.py:127(from_httplib_resp)
8: base_consumer.py:42(crawl_infrastructure._wrapper)
9: url.py:173(URL.__init__)
10: web_spider.py:170(web_spider._urls_to_verify_generator)
11: urllib2.py:374(OpenerDirector._call_chain)
12: urllib2.py:414(OpenerDirector._open)
13: headers.py:34(Headers.__init__)
14: url.py:47(URL.changed_wrapper)
15: threadpool.py:54(return_args.__call__)
16: cache.py:78(CacheHandler.http_response)
17: db.py:67(store_in_cache)
18: extended_urllib.py:205(ExtendedUrllib.send_mutant)
19: web_spider.py:270(web_spider._should_output_extracted_url)
Measurement #4 (w3af-10284-2014-02-09-01_25.cpu)
0: extended_urllib.py:434(ExtendedUrllib._send)
1: web_spider.py:345(web_spider._verify_reference)
2: plugin.py:217(meth)
3: extended_urllib.py:252(ExtendedUrllib.GET)
4: threadpool.py:42(one_to_many.__call__)
5: pool.py:64(mapstar)
6: urllib2.py:386(OpenerDirector.open)
7: web_spider.py:170(web_spider._urls_to_verify_generator)
8: base_consumer.py:42(crawl_infrastructure._wrapper)
9: url.py:173(URL.__init__)
10: HTTPResponse.py:127(from_httplib_resp)
11: threadpool.py:54(return_args.__call__)
12: threadpool.py:78(Pool.map_multi_args)
13: pool.py:298(Pool.map_async)
14: extended_urllib.py:205(ExtendedUrllib.send_mutant)
15: url.py:47(URL.changed_wrapper)
16: urllib2.py:374(OpenerDirector._call_chain)
17: urllib2.py:414(OpenerDirector._open)
18: web_spider.py:270(web_spider._should_output_extracted_url)
19: utils.py:34(dump_data_every_thread)
Core status summary
Measurement #0 (w3af-10284-2014-02-09-01_19.core)
Audit queue input speed: 130.39
Requests per minute: 445
Audit queue output speed: 49.05
Crawl queue size: 45
Crawl queue input speed: 130.39
Crawl queue output speed: 180.62
Audit queue size: 95
Requests sent: 892
Measurement #1 (w3af-10284-2014-02-09-01_21.core)
Audit queue input speed: 119.95
Requests per minute: 399
Audit queue output speed: 46.48
Crawl queue size: 0
Crawl queue input speed: 119.95
Crawl queue output speed: 214.42
Audit queue size: 180
Requests sent: 1601
Measurement #2 (w3af-10284-2014-02-09-01_23.core)
Audit queue input speed: 80.02
Requests per minute: 361
Audit queue output speed: 50.5
Crawl queue size: 0
Crawl queue input speed: 80.02
Crawl queue output speed: 116.76
Audit queue size: 252
Requests sent: 2169
Measurement #3 (w3af-10284-2014-02-09-01_25.core)
Audit queue input speed: 66.22
Requests per minute: 349
Audit queue output speed: 62.8
Crawl queue size: 1
Crawl queue input speed: 66.22
Crawl queue output speed: 66.57
Audit queue size: 262
Requests sent: 2802
Increased the run time for w3af to 15min
TODO
TODO