Request Ruby C API make hooking coverage events publicly available

[mschwager]

Ruzzy implements libFuzzer's SanitizerCoverage to achieve coverage-guided fuzzing of Ruby code.

It achieves this via three of SanitizerCoverage's features:

1. Inline 8-bit counters
2. PC-Table
3. Tracing data flow

To implement counters and the pc-table, Ruzzy hooks coverage events similar to Ruby's builtin Coverage module. It's great that this functionality is built into Ruby and we don't have to modify the Ruby bytecode during execution, or use some other heavy-handed means of tracking coverage. However, coverage instrumentation is not currently part of Ruby's public C API. To gather coverage instrumentation, Ruzzy has to perform two hacky actions:

Hook the internal-only RUBY_EVENT_COVERAGE_BRANCH event:

ruzzy/ext/cruzzy/cruzzy.c

Lines 12 to 16 in 5e39944

	// This constant is defined in the Ruby C implementation, but it's internal
	// only. Fortunately the event hooking still respects this constant being
	// passed from an external source. For more information see:
	// https://github.com/ruby/ruby/blob/v3_3_0/vm_core.h#L2182-L2184
	#define RUBY_EVENT_COVERAGE_BRANCH 0x020000

Call the Ruby interface to the Coverage module to enable coverage tracking, rather than a public C API:

ruzzy/ext/cruzzy/cruzzy.c

Lines 195 to 211 in 5e39944

	static void enable_branch_coverage_hooks()
	{
	// Call Coverage.setup(branches: true) to activate branch coverage hooks.
	// Branch coverage hooks will not be activated without this call despite
	// adding the event hooks. I suspect rb_set_coverages must be called
	// first, which initializes some global state that we do not have direct
	// access to. Calling setup initializes coverage state here:
	// https://github.com/ruby/ruby/blob/v3_3_0/ext/coverage/coverage.c#L112-L120
	// If rb_set_coverages is not called, then rb_get_coverages returns a NULL
	// pointer, which appears to effectively disable coverage collection here:
	// https://github.com/ruby/ruby/blob/v3_3_0/iseq.c
	rb_require("coverage");
	VALUE coverage_mod = rb_const_get(rb_cObject, rb_intern("Coverage"));
	VALUE hash_arg = rb_hash_new();
	rb_hash_aset(hash_arg, ID2SYM(rb_intern("branches")), Qtrue);
	rb_funcall(coverage_mod, rb_intern("setup"), 1, hash_arg);
	}

As the comment mentions, we must call Coverage.setup, which calls rb_set_coverages. If this is not called, then rb_get_coverages will return NULL in iseq.c, and coverage tracking will not be enabled.

To simplify Ruzzy's coverage tracking, I would request that both of these pieces of functionality be made a part of Ruby's public C interface. Particularly, the RUBY_EVENT_COVERAGE_BRANCH event, so we can hook coverage events with the standard rb_add_event_hook function. And a means to initialize Ruby's global coverage state, similar to rb_set_coverages, so a C extension can enable and gather coverage information. It would be very helpful if coverage events like lines and branches were added to the TracePoint API, both for Ruby and for C extensions.

An additional nice to have would be extending the public tracearg functionality to include additional coverage information. Currently, tracearg offers information like rb_tracearg_lineno and rb_tracearg_path. It would be helpful if it also provided additional coverage information like coverage.c's column information and a unique identifier for each branch. Currently, Ruzzy has to use (path, lineno) as a unique identifier for a branch because that's what's offered by the public API, but more information would be helpful for uniquely identify branches.

[mschwager]

I think making this functionality public and officially supported would help too: https://github.com/ruby/ruby/blob/v3_3_0/include/ruby/debug.h#L774-L792.

[mschwager]

Process here: https://docs.ruby-lang.org/en/3.3/contributing/reporting_issues_md.html#label-Requesting+features

[mschwager]

Upstream feature request here: https://bugs.ruby-lang.org/issues/20448