Lambda the Ultimate

When we were young we were taught that "ehud" is a 4-byte array (of some kind). Obviously this isn't the case in Haskell, but most programmers coming from C/Pascal/Ada expect this literal to be (a) unambiguous and (b) to contain 4 byte-wide elements (possibly with a nul terminator).

These days things are much more compilcated, what with char sets, bidi etc. - but most languages don't really take this into account. Even if they support some weak form of "wide chars" string literals are still what they used to be.

Sure, educating programmers about unicode etc. is important, but isn't it about time programming language support for these thing be enhanced so that strings would agin be primitive data types, and string literal will work "correctly"?

You can provide library routines, that support multiple options. It's much more challenging to decide on the "best" way to handle issues like this and provide built in language support, with appropriate syntax etc.

Since the string literal is in an RTL language, how should the language do the concatenation?

Of course it should concatenate on the left, as usual. The semantics of + should not depend on how you express the constant (your Hebrew string here). The semantics of the expression should be the same, for example, if you had written "\uXXXX\uXXXX" or hebrewString() or whatever.

I understand the issue in question here, but it is a type issue and not a syntactic issue. If you really want a language-specific (or rather, writing direction-specific) string concatenation operator, then you ought to have a language-specific string type, to tie it together with.

Your point is actually rather interesting for me because it is the most intuitive example I have seen of the commutativity iso: one wants strings in Right-To-Left languages to be isomorphic via commutativity, and not via identity, to ones in Left-To-Right languages, and this cannot be determined just by looking at the type structure. I think I may use this in a paper...

(BTW, Ehud, as is mentioned in one of the pages cited above, almost everyone will misread RTL as "Run-Time Library"... it took me a couple of seconds to figure it out as well. The page suggests using "BiDi" instead, although that doesn't work here.)

Of course. Never meant to say it's a syntactic issue.

Frank:
[T]he issue in question here...is a type issue and not a syntactic issue. If you really want a language-specific...string concatenation operator, then you ought to have a language-specific string type, to tie it together with.

I agree. Frank really echoes my point. My point was that since we want strings with different semantics than arrays, we need a new type. The semantics of strings should obey Unicode+BiDi rules. The semantics of the plus operator should obey Unicode+BiDi rules. In this particular case, I would inspect BiDi rules for user input and implement them in the compiler. Still, if we get a paper out of Frank over this, then it was well worth the post! (That's a compliment, Frank.)

As Ehud says, we tend to think of C idioms as if Moses brought them down the mountain: string == array, byte == char, ordering == LTR == memory footprint, etc. Strings are of the worst problems in C/C++, and that long before Unicode or BiDi arrived. There is enough difficulty getting strings right, and enough importance attached to the data type itself, to justify native string types and operators. I'm afraid that not all language designers believe this statement. Some do, but consider it too much work. That's where ICU can help.

Hm, maybe if there were a distinct syntax for RTL string literals...

One could adopt a trick like the hex data string that I showed. Pick a prefix to designate strict RTL, and perhaps another to designate true BiDi strings containing both LTR and RTL. It's not just directionality, but encoding at issue too, so languages need ways to enter UTF-8 literals and UTF-32 literals, etc.

Actually, string literals in source code are bad. Doesn't that rule follow the prohibition against GOTO? There ought to be support, but perhaps the harder i18n stuff belongs in resource files anyway.

A very nice and moderately detailed article by Richard Gillam drives the point home.

ICU offers a short discussion of its low-level string storage.

Is this really a problem at all? In RTL languages the first character of a word is still the first character in memory. It's just drawn from the right side.

But I'm possibly missing something, as I assume you have some experience with this.

In RTL languages the first character of a word is still the first character in memory.

How do you know how the PL chooses to represent a string? It can represent it in any order it likes.

Anyway, it can't determine the writing direction of a string just by looking at the characters used in it. I think there is an explicit direction-changing "virtual character" in Unicode, but how is it to know the direction at the start of a string? Perhaps the Unicode says the default is LTR, say, but do you really want to force every user to insert an RTL marker at the beginning of a Hebrew string literal?

Hm, maybe if there were a distinct syntax for RTL string literals...

if there were a distinct syntax for RTL string literals...
Like back ticks instead of quotes? :-)
I would agree if there were back quotes in ASCII, but hey, the source of our language must support Unicode not only inside string literals, right?

Alternatively, what's wrong with prefixes or suffixes denoting text direction in the same way as prefixes in C and Java denote base for numeric literals (like 0x34)?

Anyway, it can't determine the writing direction of a string just by looking at the characters used in it.
Unicode defines the directionality of characters, so actually it can. There are lots of details, such as numerals having weak directionality and spaces having none.

It can represent it in any order it likes.
True, but Unicode requires that a string have a first-to-last logical order, so he's right to question whether there's really a problem. In fact, Ehud's original question of how to concatenate an RTL string is not a problem -- since it is the right-hand operand it is appended after the original characters in logical order.

But there is a problem and that is how to display the string. The LTR part of the string comes before the RTL part. When the two parts are displayed together, which part is on the left and which on the right? That's what the directional formatting codes are for.

The display problem could be solved by inheriting the direction from the context the program is running in, which solves not only the order, but also the alignment.

If you output the string in an English environment, the English part is aligned with the left side of the display (or window), and the Hebrew part comes the the right of that. If you output the string in a Hebrew environment, "first" means "to the right". So you get the Latin part aligned with the right side of the display, and the Hebrew part to the left of that.

The intrinsic string type knows its encoding, BiDi issues, size, etc. The representation of such strings on screen is a text editor problem, not a string type problem. Does that answer work for you?