Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Type Hole Inference (post merge) #1155

Draft
wants to merge 141 commits into
base: dev
Choose a base branch
from
Draft
Changes from 1 commit
Commits
Show all changes
141 commits
Select commit Hold shift + click to select a range
08ac8a8
update Typ.re to reflect new provenances and matched patterns with co…
RaefM Oct 30, 2022
a87c395
got new notion of matched functions and provenances compiling
RaefM Nov 1, 2022
ec96408
wip; ammend later
RaefM Dec 26, 2022
8c7a0d6
merge with recent changes from haz3l-tests and improved casts
RaefM Dec 26, 2022
e5b4b7a
fix remaining compiler errors after merge
RaefM Dec 26, 2022
09ddeea
adds constraints to Typ.re; doesn't compile yet
RaefM Dec 29, 2022
c4488cd
adds constraint generation to static type checking
RaefM Dec 31, 2022
5ad3ee9
adds all the inference modules; unlinked to UI or log in this commit
RaefM Dec 31, 2022
431c20f
links inference to UI. currently untested. changes existing code in M…
RaefM Jan 3, 2023
94669dd
makes annotation maps mutable. merges mk_map and mk_annotations
RaefM Jan 15, 2023
7dd160d
makes annotations be accumulated in a single source
RaefM Jan 15, 2023
be649cb
fix unsolved annotations bug. adds toggle for turning annotations on …
RaefM Jan 18, 2023
c48b065
adjusts coloring of annotations. removes empty hole svg when annotati…
RaefM Jan 18, 2023
d75fb75
fix bug with things solved as unknown retaining their annotations fro…
RaefM Jan 19, 2023
fece0e9
fix occurs stack overflowing. clear annotations on slide clear
RaefM Jan 25, 2023
e298ce1
moves unsolved annotations to the cursorinspector; replaces code anno…
RaefM Jan 25, 2023
2927ce5
quick fix to compiler error
RaefM Jan 25, 2023
55f3631
quick change to unsolved printing to make it neater
RaefM Jan 26, 2023
b03f2c8
fixed issue where cursor is misdrawn after hole suggestions change
anandrav Jan 26, 2023
ead7b45
fix printing of annotations types to be consistent; fix annotation co…
RaefM Jan 26, 2023
bdb43df
comments for our roadmap for non global inf maps
RaefM Jan 29, 2023
0f9aa93
create annotations in Editor.new_state
anandrav Jan 29, 2023
a2fe31b
removed global annotation_map from Measured, still present in Code.of…
anandrav Jan 29, 2023
493febf
removed global annotation_map from Code.of_grout
anandrav Jan 29, 2023
4c78da2
removed globals from svg_display_settings() (makes grout red)
anandrav Jan 29, 2023
b61229f
no longer using var accumulated_annotations, still using annotations_…
anandrav Jan 29, 2023
1e305f4
fix bug where type annotations don't show when editor first loaded
anandrav Jan 31, 2023
aa9e622
removed type annotations from editor.state.meta
anandrav Jan 31, 2023
677cf8d
remove annotations_enabled global var and refactor names of types
RaefM Feb 2, 2023
2a37937
add accept suggestions on clicking enter when to the right of a solve…
RaefM Feb 2, 2023
a2e767d
small fix related to hole molds
RaefM Feb 3, 2023
9fd5630
remove prov specific typing in annotations (makes acceptance suggesti…
RaefM Feb 9, 2023
715d5cd
merge in dev
RaefM Feb 9, 2023
a4a1f68
add documentation (housed in rei files where possible) and did some c…
RaefM Feb 9, 2023
0d47347
merge more recent dev changes
RaefM Feb 9, 2023
24a96d9
updated UI
anandrav Feb 28, 2023
ea15b86
dotted line around unsolved type hole
anandrav Feb 28, 2023
e2645a1
removed ? from CI and added ! to unsolved hole in editor
anandrav Mar 1, 2023
7d5925a
renames eqclasses to potential type sets
RaefM Mar 1, 2023
981f15a
merge dev
RaefM Mar 19, 2023
afd91e8
fill type hole is atomic action
anandrav Mar 23, 2023
444490b
hole filling on click (not hover yet)
anandrav Mar 24, 2023
0c8f678
makes the buttons for cursorinspector suggestions hoverable to see re…
RaefM Mar 24, 2023
5f81e86
made buttons brighter on hover
RaefM Mar 24, 2023
14cb043
refactor and combine certain functions
RaefM Apr 8, 2023
b4dbdf7
wip
RaefM Apr 8, 2023
c471ad6
refactor inference results code to be more consistent
RaefM May 12, 2023
db487be
CI hexagons looking good- grout suggestions being funky with lines an…
RaefM May 22, 2023
3e32023
Merge branch 'dev' of https://github.com/hazelgrove/hazel into haz3l-…
RaefM May 22, 2023
86ac43f
fixes issues with svg in suggestions but introduces new issue with th…
RaefM Jun 12, 2023
866be93
fix weird positioning issues
RaefM Jun 14, 2023
7e4d53d
makes parenthesization less/un ambiguous (always parenthesizes left c…
RaefM Jun 14, 2023
34fbb14
Merge branch 'dev' of https://github.com/hazelgrove/hazel into haz3l-…
RaefM Jun 14, 2023
c497f61
Fix UI bugs
RaefM Jul 11, 2023
8af2bea
add message to CI on errors
RaefM Jul 12, 2023
64d109e
fix weird hover bug
RaefM Jul 12, 2023
7985faf
add text to consistent suggestions in ci
RaefM Jul 12, 2023
7607246
merged type_provenance cases 'Internal' and 'TypeHole'
anandrav Aug 23, 2023
1afa3ee
rename Anonymous to NoProvenance
anandrav Aug 23, 2023
4119c85
renamed Inference prov case to Matched
anandrav Aug 23, 2023
1db78ad
rewrite pt. 1
anandrav Aug 25, 2023
25f1808
rewrite pt. 2
anandrav Aug 26, 2023
7c431e9
functional again
anandrav Aug 26, 2023
887574b
cleanup
anandrav Aug 26, 2023
e7aee26
must resolve conflicts in Statics and CursorInspector
anandrav Oct 8, 2023
788fd55
attempt to resolve cursorinspector merge errors
disconcision Oct 8, 2023
44a649f
statics still WIP, does not compile
anandrav Oct 8, 2023
1c2f440
resolving remaining merge type errors. starts up fine but UI is utter…
disconcision Oct 10, 2023
3251d22
resolve some css issues; UI now renders okay, but still some issues
disconcision Oct 10, 2023
8afd381
Update Elaborator.re
disconcision Oct 10, 2023
c07e305
add logging
RaefM Nov 15, 2023
6b60d72
setup logging
anandrav Nov 15, 2023
adf926e
consider tuples conflicting if they have different arity
anandrav Nov 15, 2023
30442f0
fixed how unknown types are displayed
anandrav Nov 15, 2023
7484d17
added comments to clarify last change
anandrav Nov 15, 2023
a8bf67f
occurs check
anandrav Nov 15, 2023
3179e5a
Fix a bunch of todos and missing constraints
RaefM Dec 6, 2023
9a78e94
changed grout width back to 1 for inference suggestions
anandrav Dec 11, 2023
bdacad6
added labels for more clarity
anandrav Dec 11, 2023
6de7a0b
merged dev, probably does not compile.
anandrav Dec 11, 2023
9d17cf2
fixed some compiler errors, more to go...
anandrav Dec 11, 2023
51819c7
Merge branch 'dev' of https://github.com/hazelgrove/hazel into thi-me…
RaefM Dec 19, 2023
6a4bdc9
fix build failures after latest merge
RaefM Dec 19, 2023
3571c92
Fix join constraint threading
RaefM Dec 19, 2023
03facf9
Fix some issues with join_constraints and missing constraints for uex…
RaefM Dec 19, 2023
2b3152b
Review and fix issues in upat_to_info_map
RaefM Dec 19, 2023
ab14eeb
Refactor to use new provenances where synswitch is separated out to e…
RaefM Dec 23, 2023
c0b0820
make debug print string more useful
RaefM Dec 23, 2023
6650633
sub in old inference module
RaefM Dec 23, 2023
26c7fa1
Make things compile after substituting in the old inference algos
RaefM Dec 23, 2023
bf04f1c
debug wip
RaefM Dec 24, 2023
301117e
remove init syn; still broken lines
RaefM Dec 25, 2023
d89198a
Fix UI measured-grout view mismatch issues; Add PatternVar provenance…
RaefM Dec 25, 2023
51336d5
get rid of debug logs
RaefM Dec 25, 2023
4efc804
Add logic to also build out indirect suggestions in global_inference_…
RaefM Dec 26, 2023
115b25f
ADTs seem to work; stack overflowing on the examples page for ADTs th…
RaefM Dec 26, 2023
dcb3d72
fix stack overflow issue
RaefM Dec 28, 2023
4049dfe
Fix issue where solved as type alias and its body type val resulted i…
RaefM Dec 28, 2023
36bc5af
Fix CI hover issues
RaefM Dec 28, 2023
3187947
Add logic to suggest results for unsolved exp holes, but the paste ha…
RaefM Dec 31, 2023
92c5a82
Add directional jump; unsolved exphole suggestions working
RaefM Dec 31, 2023
0e757f8
remove debug logs
RaefM Dec 31, 2023
732fff7
Add suggestions for solved/nested inconsistency exp holes too (easily…
RaefM Dec 31, 2023
1ab2c53
fix issue where annotated patterns were also candidates for annot ins…
RaefM Dec 31, 2023
ba8f2a2
rename NonTypeHoleId to NotSuggestableHoleId
RaefM Dec 31, 2023
d0639da
fix issue of ? being pasted in
RaefM Dec 31, 2023
757b07d
autoformatting
RaefM Dec 31, 2023
19fc24e
fix ADT issue
RaefM Jan 5, 2024
32d0d55
Moved acceptance logic into Tab and out of Enter. Should talk to Andr…
RaefM Jan 6, 2024
4916324
fix text alignment in CI
RaefM Jan 6, 2024
051cb41
fix issue where holes become offset when UpdateAssistant operations o…
RaefM Jan 6, 2024
f529616
make exp hole suggestions only pop up on directly constrained unannot…
RaefM Jan 7, 2024
aa06ada
fix sneaky internals causing not-as-good-as-they-could-be solutions
RaefM Jan 12, 2024
ed8a2bf
Added new update settings_action
RaefM Jan 12, 2024
887c67f
switched all logic over to new toggle in nut bar
RaefM Jan 12, 2024
8114313
fix stray debug change
RaefM Jan 12, 2024
7374be6
cleanup prints and todos
RaefM Jan 13, 2024
95ddccd
fix measurement issue after suggestion accept
RaefM Jan 13, 2024
db7ef98
Add inference_enabled to init to fix issue of weird offset holes on s…
RaefM Jan 13, 2024
c988481
fix stack overflow issue for nested occurs check failures
RaefM Jan 13, 2024
d533620
make occurs check failures have special ci text
RaefM Jan 13, 2024
1247487
fix clunky CI types when parens involved
RaefM Jan 13, 2024
5255070
Fix issue where occurs failures were given solved UI. Fix issue where…
RaefM Jan 14, 2024
c7ddca2
Make all CI unknowns be the hexagon
RaefM Jan 14, 2024
6d9b105
remove debug logs again
RaefM Jan 15, 2024
fe27c3a
hackily disable cons suggestion in patterns
disconcision Jan 15, 2024
12c3f12
Make occurs failures involving type holes show conflicts in CI. Shift…
RaefM Jan 18, 2024
7f60039
Simplify CI text; fix bug where sum was being printed as prod
RaefM Jan 18, 2024
6ce7dc7
Add names when ityps are made into sum types based on position
RaefM Jan 18, 2024
c765aac
Fix stack overflow issue when nullary sum occurs
RaefM Jan 18, 2024
e263e23
fix ambiguous parens on sums in arrows
RaefM Jan 18, 2024
11e6aa6
Hackily disable cons suggestion in patterns (#1162)
cyrus- Jan 18, 2024
76c382e
merge updated nut menu
RaefM Jan 22, 2024
db7f533
Merge branch 'thi-old-engine-with-merge' of https://github.com/hazelg…
RaefM Jan 22, 2024
2f43770
finalize merge conflicts
RaefM Jan 22, 2024
5fad881
make it so if statics turns off, infernce does too
RaefM Jan 22, 2024
14f1e19
Fix empty list type paste bug and solutions
RaefM Feb 26, 2024
dc83c58
Fix issue where type aliases weren't always constrainted to self
RaefM Feb 26, 2024
298a51d
whoops- fix usage of wrong type in prev commit
RaefM Feb 26, 2024
d7f07bf
test - rerun GitHub Action
thomasporter522 Mar 29, 2024
55273cb
added constructor names to internal types
thomasporter522 Apr 1, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
links inference to UI. currently untested. changes existing code in M…
…easured.re (offset of grout set based on associated annotation, if any) and in Code.re (text for grout set to annotation, if any)
RaefM committed Jan 3, 2023
commit 431c20f3338fb61828dfb510170d8525bdde7f4c
10 changes: 9 additions & 1 deletion src/haz3lcore/Measured.re
Original file line number Diff line number Diff line change
@@ -284,6 +284,8 @@ let is_indented_map = (seg: Segment.t) => {

let of_segment = (~old: t=empty, ~touched=Touched.empty, seg: Segment.t): t => {
let is_indented = is_indented_map(seg);
let (term, _) = MakeTerm.go(seg);
let annotation_map = Statics.mk_annotations(term);

// recursive across seg's bidelimited containers
let rec go_nested =
@@ -369,7 +371,13 @@ let of_segment = (~old: t=empty, ~touched=Touched.empty, seg: Segment.t): t => {
let map = map |> add_w(w, {origin, last});
(contained_indent, last, map);
| Grout(g) =>
let last = {...origin, col: origin.col + 1};
let annotation_offset =
g.id
|> InferenceResult.get_annotation_of_id(annotation_map)
|> OptUtil.get(() => " ")
|> String.length;

let last = {...origin, col: origin.col + annotation_offset};
let map = map |> add_g(g, {origin, last});
(contained_indent, last, map);
| Tile(t) =>
41 changes: 41 additions & 0 deletions src/haz3lcore/inference/EqClass.re
Original file line number Diff line number Diff line change
@@ -366,3 +366,44 @@ and sort_eq_class_explore = (eq_class: t): t => {
}
};
};

let string_of_btyp = (btyp: base_typ): string => {
btyp |> base_typ_to_ityp |> ITyp.string_of_ityp;
};

let rec string_of_eq_class = (eq_class: t): string =>
switch (eq_class) {
| [] => ""
| [hd, ...tl] =>
let hd_str = string_of_eq_typ(hd);
String.concat("//", [hd_str, string_of_eq_class(tl)]);
}
and string_of_eq_typ = (eq_typ: eq_typ) =>
switch (eq_typ) {
| Base(btyp) => string_of_btyp(btyp)
| Compound(ctor, eq_class_lt, eq_class_rt) =>
let ctor_string =
switch (ctor) {
| CArrow => " -> "
| CProd => " * "
| CSum => " + "
};

String.concat(
"",
[
string_of_eq_class(eq_class_lt),
ctor_string,
"(",
string_of_eq_class(eq_class_rt),
")",
],
);
| Mapped(ctor, eq_class) =>
let (end_text, start_text) =
switch (ctor) {
| CList => ("[", "]")
};

String.concat("", [start_text, string_of_eq_class(eq_class), end_text]);
};
17 changes: 16 additions & 1 deletion src/haz3lcore/inference/ITyp.re
Original file line number Diff line number Diff line change
@@ -42,7 +42,7 @@ let rec typ_to_ityp: Typ.t => t =
| Prod([hd_ty, ...tl_tys]) =>
Prod(typ_to_ityp(hd_ty), typ_to_ityp(Prod(tl_tys)))
| Prod([]) => Unit
| Var(_) => raise(TypeVarUnsupported);
| Var(_) => Unknown(Anonymous);

let rec ityp_to_typ: t => Typ.t =
fun
@@ -73,3 +73,18 @@ let rec contains_hole = (ty: t): bool =>
| Prod(ty1, ty2) => contains_hole(ty1) || contains_hole(ty2)
| _ => false
};

let rec string_of_ityp = (ityp: t): string => {
switch (ityp) {
| Unknown(_) => "?"
| Unit => "Unit"
| Int => "Int"
| Float => "Float"
| Bool => "Bool"
| String => "String"
| List(t) => "[" ++ string_of_ityp(t) ++ "]"
| Arrow(t1, t2) => string_of_ityp(t1) ++ " -> " ++ string_of_ityp(t2)
| Sum(t1, t2) => string_of_ityp(t1) ++ " + " ++ string_of_ityp(t2)
| Prod(t1, t2) => string_of_ityp(t1) ++ " x " ++ string_of_ityp(t2)
};
};
65 changes: 30 additions & 35 deletions src/haz3lcore/inference/InferenceResult.re
Original file line number Diff line number Diff line change
@@ -4,47 +4,42 @@ type status =

type t = (ITyp.t, status);

let status_to_string: status => string =
fun
| Solved(ityp) =>
String.concat(
" ",
["Solved: ", ityp |> ITyp.sexp_of_t |> Sexplib.Sexp.to_string_hum],
)
| Unsolved(eqClass) =>
String.concat(
" ",
[
"Unsolved: ",
eqClass |> EqClass.sexp_of_t |> Sexplib.Sexp.to_string_hum,
],
);
type annotation_map = list((Id.t, string));

let t_to_string = ((ityp, status)) => {
String.concat(
" ",
[
"{For hole",
ityp |> ITyp.sexp_of_t |> Sexplib.Sexp.to_string_hum,
"result is",
status_to_string(status),
"}\n",
],
);
};
let get_annotations = (inference_results: list(t)): annotation_map => {
let status_to_string = (status: status): string => {
switch (status) {
| Solved(ityp) => ITyp.string_of_ityp(ityp)
| Unsolved(eq_class) => EqClass.string_of_eq_class(eq_class)
};
};

let list_of_t_to_string = (statuses: list(t)): string => {
let acc_str = (acc: string, elt: t) => {
String.concat(" ", [acc, "\n", t_to_string(elt)]);
let id_and_annotation_if_type_hole = (result: t): option((Id.t, string)) => {
switch (result) {
| (Unknown(TypeHole(id)), status) =>
Some((id, status_to_string(status)))
| _ => None
};
};
List.fold_left(acc_str, "", statuses);

List.filter_map(id_and_annotation_if_type_hole, inference_results);
};

let print_statuses = (statuses: list(t)): unit => {
let print_t = (t: t) => {
t |> t_to_string |> print_endline;
let get_annotation_of_id =
(annotation_map: annotation_map, id: Id.t): option(string) => {
let get_annotation_if_for_id = ((k, v)) => k == id ? Some(v) : None;

let get_annotation_text =
(possible_annotations: list(string)): option(string) => {
switch (possible_annotations) {
| [] => None
| [hd, ..._tl] => Some(hd)
};
};
List.iter(print_t, statuses);

annotation_map
|> List.filter_map(get_annotation_if_for_id)
|> get_annotation_text;
};

let condense = (eq_class: MutableEqClass.t): status => {
15 changes: 15 additions & 0 deletions src/haz3lcore/statics/Statics.re
Original file line number Diff line number Diff line change
@@ -816,10 +816,25 @@ let mk_map =
e => {
let (_, _, map, _constraints) =
uexp_to_info_map(~ctx=Builtins.ctx(Builtins.Pervasives.builtins), e);

map;
},
);

let mk_annotations =
Core.Memo.general(
~cache_size_bound=1000,
e => {
let (_, _, _info_map, constraints) =
uexp_to_info_map(~ctx=Builtins.ctx(Builtins.Pervasives.builtins), e);

let inference_results = Inference.unify_and_report_status(constraints);
let annotation_map = InferenceResult.get_annotations(inference_results);

annotation_map;
},
);

let get_binding_site = (id: Id.t, statics_map: map): option(Id.t) => {
open OptUtil.Syntax;
let* opt = Id.Map.find_opt(id, statics_map);
5 changes: 4 additions & 1 deletion src/haz3lweb/view/BackpackView.re
Original file line number Diff line number Diff line change
@@ -15,6 +15,8 @@ let backpack_sel_view =
let map = Measured.of_segment(content);
let settings = Model.settings_init;
});
let (term, _) = MakeTerm.go(content);
let annotation_map = Statics.mk_annotations(term);
// TODO(andrew): Maybe use init sort at caret to prime this
div(
~attr=
@@ -32,7 +34,8 @@ let backpack_sel_view =
),
]),
// zwsp necessary for containing box to stretch to contain trailing newline
Text.of_segment(~no_sorts=true, content) @ [text(Unicode.zwsp)],
Text.of_segment(~no_sorts=true, content, annotation_map)
@ [text(Unicode.zwsp)],
);
};

47 changes: 37 additions & 10 deletions src/haz3lweb/view/Code.re
Original file line number Diff line number Diff line change
@@ -30,7 +30,9 @@ let of_delim =
(sort: Sort.t, is_consistent, t: Piece.tile, i: int): list(Node.t) =>
of_delim'((sort, is_consistent, Tile.is_complete(t), t.label, i));

let of_grout = [Node.text(Unicode.nbsp)];
let of_grout = (annotation: option(string)) => [
annotation |> OptUtil.get(() => Unicode.nbsp) |> Node.text,
];

let of_whitespace =
Core.Memo.general(
@@ -56,7 +58,13 @@ module Text = (M: {
}) => {
let m = p => Measured.find_p(p, M.map);
let rec of_segment =
(~no_sorts=false, ~sort=Sort.root, seg: Segment.t): list(Node.t) => {
(
~no_sorts=false,
~sort=Sort.root,
seg: Segment.t,
annotation_map: InferenceResult.annotation_map,
)
: list(Node.t) => {
//note: no_sorts flag is used for backback
let expected_sorts =
no_sorts
@@ -69,17 +77,32 @@ module Text = (M: {
};
seg
|> List.mapi((i, p) => (i, p))
|> List.concat_map(((i, p)) => of_piece(sort_of_p_idx(i), p));
|> List.concat_map(((i, p)) =>
of_piece(sort_of_p_idx(i), p, annotation_map)
);
}
and of_piece = (expected_sort: Sort.t, p: Piece.t): list(Node.t) => {
and of_piece =
(
expected_sort: Sort.t,
p: Piece.t,
annotation_map: InferenceResult.annotation_map,
)
: list(Node.t) => {
switch (p) {
| Tile(t) => of_tile(expected_sort, t)
| Grout(_) => of_grout
| Tile(t) => of_tile(expected_sort, t, annotation_map)
| Grout(g) =>
g.id |> InferenceResult.get_annotation_of_id(annotation_map) |> of_grout
| Whitespace({content, _}) =>
of_whitespace((M.settings.whitespace_icons, m(p).last.col, content))
};
}
and of_tile = (expected_sort: Sort.t, t: Tile.t): list(Node.t) => {
and of_tile =
(
expected_sort: Sort.t,
t: Tile.t,
annotation_map: InferenceResult.annotation_map,
)
: list(Node.t) => {
let children_and_sorts =
List.mapi(
(i, (l, child, r)) =>
@@ -90,7 +113,7 @@ module Text = (M: {
let is_consistent = Sort.consistent(t.mold.out, expected_sort);
Aba.mk(t.shards, children_and_sorts)
|> Aba.join(of_delim(t.mold.out, is_consistent, t), ((seg, sort)) =>
of_segment(~sort, seg)
of_segment(~sort, seg, annotation_map)
)
|> List.concat;
};
@@ -120,9 +143,11 @@ let simple_view = (~unselected, ~map, ~settings: Model.settings): Node.t => {
let map = map;
let settings = settings;
});
let (term, _) = MakeTerm.go(unselected);
let annotation_map = Statics.mk_annotations(term);
div(
~attr=Attr.class_("code"),
[span_c("code-text", Text.of_segment(unselected))],
[span_c("code-text", Text.of_segment(unselected, annotation_map))],
);
};

@@ -140,9 +165,11 @@ let view =
let map = measured;
let settings = settings;
});
let (term, _) = MakeTerm.go(unselected);
let annotation_map = Statics.mk_annotations(term);
let unselected =
TimeUtil.measure_time("Code.view/unselected", settings.benchmark, () =>
Text.of_segment(unselected)
Text.of_segment(unselected, annotation_map)
);
let holes =
TimeUtil.measure_time("Code.view/holes", settings.benchmark, () =>