Add support for direction nodes

[deepakjois]

Issue by deepakjois
Friday May 19, 2017 at 10:52 GMT
Originally opened as deepakjois/luatex-harfbuzz#6

---

[khaledhosny]

What kind of support you had in mind? If memory is serving me about what these whatsits do, I’d ject convert them to paired LRE/RLE and PDF and let the bidi algorithm deal with them transparently. (or LRO/RLO depending on how you see the rule of the related TeX commands)

[deepakjois]

That’s exactly what I had in mind. Right now, explicit directional overrides using LuaTeX direction commands inside a paragraph wont have an effect because the whatsit generated from that will be treated as 0xFFFC. So that needs to be fixed.

…

On 24-May-2017 06:02, "Khaled Hosny" ***@***.***> wrote: What kind of support you had in mind? If memory is serving me about what these whatsits do, I’d ject convert them to paired LRE/RLE and PDF and let the bidi algorithm deal with them transparently. (or LRO/RLO depending on how you see the rule of the related TeX commands) — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#6 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAAU3jwmDAovLLl_Eh6Kg3LM1jgY0V_Gks5r83Q2gaJpZM4Nh8XN> .

[deepakjois]

Ok so I fixed this in 933a40b. However, one strange thing I noticed in the code below

\input hbtex.sty
% Document starts here.
\nopagenumbers
\parindent=0mm
\font\amiri={./fonts/amiri-regular.ttf}
\amiri
My name is {\textdir TRT دیپک}. Oh wait that’s wrong.
\bye

was that the output is using the Arabic full-stop glyph in a run of text which is mainly Latin. See screenshot below:

@khaledhosny, does libraqm do the same too? Is this expected behavior?

[deepakjois]

Another thing, I am currently using LRE and RLE, and this text doesn’t quite work right.

My name is {\textdir TRT deepak}. Oh wait.

Output:

[deepakjois]

Using LRO and RLO instead of LRE and RLE fixed problem in previous comment. Now I’m not sure what’s appropriate.

[deepakjois]

LuaTeX directions are documented in section 2.3 of the docs, and we are clearly not handling all the cases. But then that section is a bit dense. Need more careful reading and testing.

[deepakjois]

ufylayout behavior and regular LuaTeX behavior seems to differ in some places:

% from the docs 
\textdir TRT nur {\textdir TLT run \textdir TRT NUR} nur

\textdir TRT nur {{\textdir TLT run} {\textdir TRT NUR}} nur


\def\ltr{\textdir TLT\relax}
\def\rtl{\textdir TRT\relax}

run {\rtl nur {\ltr run \rtl NUR \ltr run \rtl NUR} nur}

run {\ltr run {\rtl nur \ltr RUN \rtl nur \ltr RUN} run}


\def\ltr{\linedir TLT\relax}
\def\rtl{\linedir TRT\relax}

run {\rtl nur {\ltr run \rtl NUR \ltr run \rtl NUR} nur}

run {\ltr run {\rtl nur \ltr RUN \rtl nur \ltr RUN} run}

Here is a comparision.

• Left is ufylayout
• Right is plain LuaTeX (with luaotfload)

[khaledhosny]

The choice of embedding vs override depends on how you see the use of LuaTeX direction commands. In “vanilla” LuaTeX they act as override codes, but I don’t find this terribly useful since the most common use case where \textdir would be useful is to set the inline direction of a part of the paragraph, and using override means it can’t have any RTL text, not even numbers, which can be surprising for most users.

For \pardir using override does not make much sense even more since one would be using it to set the base direction, but it just inserting the same direction nodes around the whole text (this should be handled in a special way anyway; by detecting it and using it to set the base direction instead of inserting directional control codes).

Another option is to just strip direction nodes away and clearly say they are not supported since their usefulness is questionable. Instead provide commands at TeX level to set base direction, embedded direction and so on.

[khaledhosny]

The Arabic dot is expected since the script itemization heuristics will consider it Arabic not Latin. The script itemization heuristics are not as well developed as the bidi algorithm and probably there is some way for improvement, but I think it is OK to keep things simple for the time being.

[deepakjois]

This needs more research and thinking.

I am inclined to go with treating the dir nodes as RLO and LRO at the moment, just so that they don’t deviate from their original meaning. However, that still leaves us with the problem of dealing with them correctly in our scenario, where we are also doing bidi re-ordering. It might also be okay to ignore the dir nodes in some places (probably not all), because they just dont make sense for our use case.

@khaledhosny, regarding your statement:

but I don’t find this terribly useful since the most common use case where \textdir would be useful is to set the inline direction of a part of the paragraph, and using override means it can’t have any RTL text, not even numbers, which can be surprising for most users.

I don’t fully understand. Could you please give me some example text i can try out.

[khaledhosny]

A naïve user writing some Arabic text in the middle of an English paragraph might want to mark that Arabic text with \textdir TRT to avoid punctuation ending in the wrong side of the text, but if that Arabic portion contains numbers they will also get forced to be RTL which is not that useful, and having to break the number out of the \textdir without messing up the text requires intimate understanding of how bidi works:

I read the book {\textdir TRT حول العالم في ١٠٠٠ يوم!}\par                % wrong
I read the book {\textdir TRT حول العالم في} ١٠٠٠ {\textdir TRT يوم!}\par % aslo wrong
I read the book {\textdir TRT حول العالم في {\textdir TLT ١٠٠٠} يوم!}\par % OK

This IMO makes LuaTeX’s inline directional commands not terribly useful and I don’t mind just disabling them and have a cleaner and clearer interface instead.