racket highlighting #; incorrectly by default

[jestarray]

So this is lang racket not highlighting the #; does not highlight the comment brown but in BSL it does

A few responses on the discord say that it #; should be highlighted in brown as a coment because a comment is a comment, and that it's surprising that it doesn't. This shouldn't interfere with the comment box feature right?
I leave this here as a poll as to what it should be by default.

[rocketnia]

Comments have a syntactic and semantic structure to them, too. Within commented-out text, some parts of the text are code, and ideally those parts should be just as pleasant to read and edit as uncommented code. The same is true of code that appears inside a string literal or a quasiquotation. Assuming we want more fancy editor support for code than for plain text, why should we then settle for plain text editing when the code happens to be in a comment or quotation?

The current state of Racket makes this seem a little unusual because #; is the only comment syntax that gives any indication as to what part of the comment is code. But this is even more indication than most languages have, and I think Racket's editor support for it is a step in the right direction, so I'm voting 👎 in approval of it.

I do think it's worth having commented-out code and quoted code be highlighted differently from other code. Some kind of fading or sharpening of the text or its background could be nice, or perhaps some kind of outlining or margin decoration. As almost a rudimentary kind of margin decoration, the #; token could have its own highlighting style that makes it prominent enough to stay on people's minds as they read the code.

[rfindler]

This has long been a discussion on what is the right thing.

With the current design, we can either have it look like a comment and then lose the s-expression navigation operations (and indentation) or behave as it currently does, and that code is here.

If someone were to want to make it both colored like a comment and also support s-expression navigation and indentation, then the code to improve is here.

[jackfirth]

With the current design, we can either have it look like a comment and then lose the s-expression navigation operations (and indentation) or behave as it currently does, and that code is here.

I'm all for having it look like a comment and losing the navigation operations for now, as an immediate improvement. The current behavior just looks like a bug to most people. I myself have assumed for years that it was just something that nobody had gotten around to implementing.

I agree with the sentiments in this thread that the ideal solution lies somewhere between the two extremes we can choose from today, but "users can't distinguish between commented-out code and uncommented code" is such a painful experience that I think we should flip to the other extreme now and approach the ideal from that direction instead. It's easy to fix that bug today without doing much of any implementation work.

[rfindler]

Based on past discussion, this will be unpopular. If someone were to implement a new lexer, however, I would be happy to add an option to use it. Probably implementing the new lexer wouldn't be difficult. Robby

…

On Mon, Sep 20, 2021 at 7:37 PM Jack Firth ***@***.***> wrote: With the current design, we can either have it look like a comment and then lose the s-expression navigation operations (and indentation) or behave as it currently does, and that code is here. I'm all for having it look like a comment and losing the navigation operations for now, as an immediate improvement. The current behavior just looks like a *bug* to most people. I myself have assumed for years that it was just something that nobody had gotten around to implementing. I agree with the sentiments in this thread that the ideal solution lies somewhere between the two extremes we can choose from today, but "users can't distinguish between commented-out code and uncommented code" is such a painful experience that I think we should flip to the other extreme now and approach the ideal from that direction instead. It's easy to fix that bug today without doing much of any implementation work. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#512 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AADBNMF754C3I4CE3JXCTVDUC7HVPANCNFSM5EMZZTFA> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.

[mfelleisen]

I disagree with "let's have it look like a comment". For some years now, I have been using #; to write signatures like this:

#; [Any -> Boolean -> String]
(define ([foo a] b)
   "hello world")

and it's really neat to have them colored [[ and S-expression navigable ]]. All I wish is that the menu-teaching languages respected this convention too.

[sorawee]

@mfelleisen wouldn't a macro that ignores its operands suit your need better? Something like:

(: Any -> Boolean -> String)
...

I agree with you and @rocketnia though -- I prefer that it's not colored like comment. @rocketnia's suggestion of "fading or sharpening of the text or its background" seems like a perfect compromise to me.

[jestarray]

Alright closing, it seems like the consensus is a tie and a bit of a mixed bag. Thanks for the consideration! Arguably vscode, vim, etc don't highlight #; blocks as a comment either so its fair in a way. Someone can re-open this if they plan to make this a config option or something

[rfindler]

I like the idea of finding "another dimension" like fading (underlining? Yuck...) but worry, in the specific case of fading, about color blindness or other ways in which the text becomes hard to read. I do think this is a perennial issue so trying to find a good idea here would be nice.

[mflatt]

The current design's single "category" dimension — used for coloring, matching, and anything else parsing-sensitive, like tabbing — has been a little awkward for a #lang rhombus prototype. It would be nice to generalize the information not only for coloring and matching, but also other information that a lexer might communicate to an editing tool. Since I have this paged in at the moment, I'd be interested to try implementing something if we find a good direction.

Here's one idea:

• generalize the category-symbol result for a coloring lexer to be a symbol or immutable hash table with symbol keys; a symbol would be a shorthand for a table mapping some key ('class?) to the symbol; and

• add a method like classify-position to color:text<%>, where the new method returns the whole table of lexer-provided information.

For coloring, 'comment? could be an extra key that is recognized by the colorer, and so on.

I'm not sure whether the result form a lexer should be constrained to have symbol keys (probably worthwhile), to have symbol or writable values (probably not worthwhile), or maybe to use a function instead of a hash table for generalized results (probably more flexible than needed).

I'd also be willing to adapt the Racket lexer to identify comment regions.

[samth]

Given that we're thinking about this with Rhombus as a potential application, probably we should make the key type something that's easy to write in Rhombus, suggesting that maybe string-or-symbol or keyword-or-symbol is a good idea.

[Metaxal]

On several occasions I've wished that read-syntax was able to read comments as syntax objects, in particular so as to transform a text file and still be able to re-print the comments where they were. This could also be used by other parsers and colorers, which (ideally again) would all use a single Racket/Rhombus reader. (Apologies if this is beside the point.)

[rfindler]

Not sure if this goes beyond reasonable misson creep, but the colorer keeping enough information to print the results after some change seems useful for renaming. More specifically, in indentation sensitive languages if you rename an identifier to another one that's a different length and you use the current algorithm in drracket, you get the wrong results. I'm not sure if retaining enough information to print everything is necessary or if there's something more indentation-specific that can be saved (perhaps information about what this lined up with on a previous line that can be used to calculate indentation adjustments might be general enough?).

For the specific problem that started this thread, it seems like "commentness" should be an extra dimension that might be combined with other coloring directives.

[mflatt]

On symbols and Rhombus: My initial reaction is that making symbols convenient enough is a Rhombus problem, instead of trying to adapt lots of Racket APIs that use symbols (of which there must be many).

On preserving comments and whitespace: FWIW, the read-syntax implementation in the experiment at https://github.com/mflatt/shrubbery-rhombus-0 records comments and whitespace in syntax properties, as well as recording the original text for an "atomic" form as a syntax property. So, a printer can reconstruct the original source from the result of read-syntax, and when code is printed in syntax errors, it uses that original form. If that approach turns out to be a good idea, it could be backported (in a sense) to the S-expression reader. But the read-syntax level doesn't seem so relevant to a colorer, which already reports whitespace and comments as-is.

On renaming: Annotating color-lexer result to say that it should be kept a column relative to some other result is an interesting idea. That needs a certain amount of flexibility in the values of a table, but still seems compatible with a symbol-keyed table.

[jackfirth]

If we're going down the road of designing a more complete solution to this problem, I'd like us to avoid making something DrRacket exclusive. We've got that langserver after all.

[rfindler]

Nothing is DrRacket-specific so far in this discussion as far as I see. What do you see that is?

…

On Tue, Sep 21, 2021 at 2:30 PM Jack Firth ***@***.***> wrote: If we're going down the road of designing a more complete solution to this problem, I'd like to avoid making something DrRacket exclusive. We've got that langserver <https://github.com/jeapostrophe/racket-langserver/> after all. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#512 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AADBNMDXAHTDSOLJP5NY6Z3UDDMN5ANCNFSM5EMZZTFA> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.

[rocketnia]

(The repo this is on is pretty DrRacket-specific. :-p )

I like the idea of finding "another dimension" like fading (underlining? Yuck...) but worry, in the specific case of fading, about color blindness or other ways in which the text becomes hard to read. I do think this is a perennial issue so trying to find a good idea here would be nice.

Yeah, regardless of being sighted, I frequently encounter syntax highlighting styles which make comments virtually invisible, and I'd rather avoid doing that. At any rate, the DrRacket comment coloring option shown above is monochrome compared to the other one, so it "fades out" the would-be highlighting information to the point of being completely absent.

A friend once observed that he wanted some comments to be less prominent than the code and some to be more visible, so I thought about that and suggested "sharpened" alongside "faded." I think for usability, it should ultimately all have good contrast regardless, which is why I mentioned an outline or other decoration of some sort to keep it from depending on text color.

For coloring, 'comment? could be an extra key that is recognized by the colorer, and so on.

I think something slightly more complex than a boolean might pay off well: A numeric depth would make it feasible to distinguish between commented-out code and commented-out commented-out code.

Each level of depth represents a place that an island of richly editable code is adrift in a sea of arbitrary data. The main two ways code gives way to arbitrary syntactic data are comments and quotations (e.g., string or symbol literals, quote, or syntax). The way I imagine this playing out in general, each level could be tagged with the language that the editor is basing its editor functionality on at that level (usually Racket), and each pair of adjacent levels could be tagged with the kind of embedding that's going on between them (commenting-out or quotation).

But as far as lexing text for #lang racket goes, Racket doesn't have a syntax for designating part of a string as editable Racket code, and perhaps the only richly editable language Racket code can embed in a Racket comment is more Racket code using #;. If this remains the case, a numeric comment depth is enough to represent these considerations comprehensively.

[rfindler]

(The repo this is on is pretty DrRacket-specific. :-p )

Ha! Touché. :). The subject here, however, is not actually the code in this repo (as you may see from my first message on this issue).

Also, I should add that, while DrRacket is definitely the OG of Racket IDEs, it is structured in a way that is trying to be friendly to other IDEs for things beyond the raw editing. Racket mode uses the online check syntax, as does the language server (the thing that connects to VS code) and the language server apparently also uses the colorer api that we're talking about here.

---

As for too-subtle distinctions, I had a thought: we should add support to the color schemes that would enable configuration of the way this dual-color-style comments are handled. Then, in the default white-background color scheme we could use alpha blending; this would then let color schemes optimized for other situations use underlining or use nothing or use something that takes over the colors (making the #; comments just turn the entire comment into comment-color).

[mflatt]

The combination of racket/gui#244 (extends the coloring API), racket/syntax-color#7 (implements the comment? attribute for s-exp and at-exp), and #513 (uses the comment?-attribute lexer by default) is a candidate implementation. More is needed at the style% level of editors to get it right, though.

Generalizing the color-lexing protocol is not completely backward-compatible, because there are uses of the protocol besides the one in the framework (such as "racket-langserver"), and those still expect symbols back. The syntax-color changes attempt to reduce the problem by having module-lexer behave as before; it reduces an attribute table produced by a sub-lexer to just the value of the 'type attribute, while module-lexer* is the new one that lets attribute tables through. Similarly, racket-lexer works as before, while racket-lexer* is the new one. But I didn't bother with scribble-lexer, in the hope that it is normally used only via module-lexer.

Between the protocol change and the need for a little more in style% (which will affect the WXME format), this is probably best to finish and merge after the v8.3 branch.

[jackfirth]

Nothing is DrRacket-specific so far in this discussion as far as I see. What do you see that is?

The main thing I want to avoid is dependencies on racket/gui, because that's rather unfriendly to the langserver. It prevents people from using the langserver in editor setups that don't actually have a GUI available, such as ssh-ing to a machine and editing files from the terminal with vim / neovim.

[rfindler]

Okay great. The colorer currently doesn't depend on racket/gui, nothing suggested here would add such a dependency, and I would also have been against such a dependency if it had appeared. Robby On Thu, Sep 23, 2021 at 10:50 PM Jack Firth ***@***.***> wrote:

…

Nothing is DrRacket-specific so far in this discussion as far as I see. What do you see that is? … <#m_-3654624522725576811_> On Tue, Sep 21, 2021 at 2:30 PM Jack Firth *@*.***> wrote: If we're going down the road of designing a more complete solution to this problem, I'd like to avoid making something DrRacket exclusive. We've got that langserver https://github.com/jeapostrophe/racket-langserver/ after all. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#512 (comment) <#512 (comment)>>, or unsubscribe https://github.com/notifications/unsubscribe-auth/AADBNMDXAHTDSOLJP5NY6Z3UDDMN5ANCNFSM5EMZZTFA . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub . The main thing I want to avoid is for the new coloring changes to depend on racket/gui, because that's rather unfriendly <jeapostrophe/racket-langserver#45> to the langserver. It prevents people from using the langserver in editor setups that don't actually have a GUI available, such as ssh-ing to a machine and editing files from the terminal with vim / neovim. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#512 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AADBNMAPGNSJNUSZBY4NNF3UDPYQ5ANCNFSM5EMZZTFA> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.