mdoc processing of scala 2.13.8 worksheet in ST3/LSP-metals fails with exception if output is not parseable by fansi.Str(_)

[axb21]

I have reported a similar issue to the PPrint library here: com-lihaoyi/PPrint#78

I am using scala 2.13.8 on Ubuntu 21.04, writing code in Sublime Text 3 with the LSP and LSP-metals plugins, the latter of which is ultimately responible to the call out to mdoc. As you know metals allows you to make scala worksheets, and will execute the code in those worksheets and show you (pseudo)comments with the result of each line of code.

The gist of the problem is that I am seeing exceptions in the metals log panel, and failure to correctly process worksheets, whenever I include a line of code in the worksheet whose value cannot be correctly handled by pprint.Renderer and in turn fansi.Str(_). Starting somewhere around here:

mdoc/runtime/src/main/scala-2/mdoc/internal/document/Printing.scala

Line 28 in 3078e6f

out.appendAll(chunk)

eventually pprint.Renderer is called, which in turn calls fansi.Str(_) on a string value that is not sanitized. The default behavior of this method is to throw an exception, though the library allows that behavior to be changed. Unfortunately I cannot share the code and data that produce this issue, but below I'll include a typical stack trace. The behavior I observe is: all my code in the worksheet is fine and shows the expected value in a comment after each line of code. However, if I then include a line of code that returns a value (in my case it's iter.next()), those comments with the values disappear, and a stack trace like the one below appears in the metals log. This does not appear to me to be a metals-specific issue, but one either at the mdoc level or below, which is why I chose to report it here and in the PPrint library repository.

A typical stack trace:

java.lang.IllegalArgumentException: Unknown ansi-escape at index XXX inside string cannot be parsed into an fansi.Str
	at fansi.ErrorMode$Throw$.handle(Fansi.scala:419)
	at fansi.ErrorMode$Throw$.handle(Fansi.scala:407)
	at fansi.Str$.apply(Fansi.scala:272)
	at pprint.Renderer.$anonfun$rec$32(Renderer.scala:147)
	at scala.collection.Iterator$$anon$9.next(Iterator.scala:575)
	at scala.collection.Iterator$$anon$6.hasNext(Iterator.scala:471)
	at pprint.Truncated.hasNext(Truncated.scala:80)
	at scala.collection.Iterator$$anon$6.hasNext(Iterator.scala:470)
	at pprint.Truncated.hasNext(Truncated.scala:80)
	at scala.collection.Iterator$$anon$9.hasNext(Iterator.scala:574)
	at scala.collection.Iterator$$anon$6.hasNext(Iterator.scala:470)
	at scala.collection.Iterator$$anon$10.hasNext(Iterator.scala:592)
	at scala.collection.Iterator$$anon$6.hasNext(Iterator.scala:470)
	at scala.collection.mutable.StringBuilder.appendAll(StringBuilder.scala:185)
	at mdoc.internal.document.Printing$.printOneLine(Printing.scala:28)
	at mdoc.internal.worksheets.WorksheetProvider.$anonfun$renderSummary$1(WorksheetProvider.scala:141)
	at mdoc.internal.worksheets.WorksheetProvider.$anonfun$renderSummary$1$adapted(WorksheetProvider.scala:124)
	at scala.collection.IterableOnceOps.foldLeft(IterableOnce.scala:646)
	at scala.collection.IterableOnceOps.foldLeft$(IterableOnce.scala:642)
	at scala.collection.AbstractIterator.foldLeft(Iterator.scala:1288)
	at mdoc.internal.worksheets.WorksheetProvider.renderSummary(WorksheetProvider.scala:124)
	at mdoc.internal.worksheets.WorksheetProvider.renderDecoration(WorksheetProvider.scala:73)
	at mdoc.internal.worksheets.WorksheetProvider.$anonfun$evaluateWorksheet$3(WorksheetProvider.scala:42)
	at scala.collection.immutable.List.map(List.scala:250)
	at mdoc.internal.worksheets.WorksheetProvider.$anonfun$evaluateWorksheet$2(WorksheetProvider.scala:41)
	at scala.collection.Iterator$$anon$10.nextCur(Iterator.scala:585)
	at scala.collection.Iterator$$anon$10.hasNext(Iterator.scala:599)
	at scala.collection.Iterator$$anon$6.hasNext(Iterator.scala:470)
	at scala.collection.Iterator$$anon$9.hasNext(Iterator.scala:574)
	at scala.collection.immutable.List.prependedAll(List.scala:152)
	at scala.collection.IterableOnceOps.toList(IterableOnce.scala:1251)
	at scala.collection.IterableOnceOps.toList$(IterableOnce.scala:1251)
	at scala.collection.AbstractIterator.toList(Iterator.scala:1288)
	at mdoc.internal.worksheets.WorksheetProvider.evaluateWorksheet(WorksheetProvider.scala:48)
	at mdoc.internal.worksheets.Mdoc.evaluateWorksheet(Mdoc.scala:55)
	at scala.meta.internal.worksheets.WorksheetProvider.scala$meta$internal$worksheets$WorksheetProvider$$evaluateWorksheet(WorksheetProvider.scala:363)
	at scala.meta.internal.worksheets.WorksheetProvider$$anon$1.liftedTree1$1(WorksheetProvider.scala:267)
	at scala.meta.internal.worksheets.WorksheetProvider$$anon$1.run(WorksheetProvider.scala:267)

[tgodzik]

Thanks for reporting! Do you include pprint dependency in your project or only use pprint via worksheets? There was an issue recently about having an incompatible pprint version on classpath versus the one in mdoc. This should be fixed in main.

Do you have a small worksheet example that this fails on? We should be able to print any value including iter.next() Looks like the ansi escape might be an issue in pprint, but would be great to know what it is caused by, otherwise might be hard to pinpoint.

[axb21]

Thanks for reporting! Do you include pprint dependency in your project or only use pprint via worksheets? There was an issue recently about having an incompatible pprint version on classpath versus the one in mdoc. This should be fixed in main.

It is only pprint via a worksheet. I haven't tested whether the same issue arises in other cases.

Do you have a small worksheet example that this fails on? We should be able to print any value including iter.next() Looks like the ansi escape might be an issue in pprint, but would be great to know what it is caused by, otherwise might be hard to pinpoint.

I wish I did, but unfortunately it's a bit complicated. I'm using geotools to parse an ESRI shapefile and retrieve some of the data in it. It's very handy to use worksheets to tinker with these things, which I've done successfully a bunch of times before.

I know some of the properties in this one file that is causing me issues are in Hebrew and the character encoding is UTF-8. It should be fine, and GIS software does not complain about this file, but there seems to be something odd in there that is triggering this error. So far I have not been able to isolate a small example. If I ever manage to do that I'll share it here.

[tgodzik]

It would be great if you managed to see exactly what this is failing on, it should be a straightforward fix once we know that.