Lambda the Ultimate

Most people implementing Unicode in programming languages treat Unicode codepoints as characters, the same way we treated ASCII codepoints. But changing capitalization, changing a diacritic, etc, changes the length in codepoints of a string. Conceptually these are point mutations - they correspond to changing a single character on a page of text and shouldn't require re-copying the whole page, invalidating the indices of other characters on the page, invalidating references to the page, etc. There are a number of good, efficient, algorithms for doing things with strings which rely on point mutations that don't invalidate other references to the string. And we mostly don't dare attempt to use them any more.

In Unicode - particularly in variable-length encodings of Unicode such as utf-8 - they are length changing mutations, and if we insist on a representation of strings as contiguous arrays of characters they require re-copying the whole string, invalidating references, potentially reallocating, etc. So we've invented "string iterators" and a number of other things to try to abstract the location of characters. We've abandoned numerous GOOD, EFFICIENT algorithms that relied on point mutations which did not invalidate other references in the same string. This was a mistake.

I think string iterators, while useful, are an abstraction at the wrong level. We abstracted string references because we were dealing with our failure to correct our abstraction of string representation. The right answer isn't refusing to touch string references and assuming any mutation invalidates the whole string. The right answer is changing our string representation to preserve valuable point mutation properties that make handling strings more efficient.

So I think that a proper string representation has characters that are a uniform width. Strings ought to use a 32-bit character encoding, with entities of 0x10FFFF or less referring to complete characters expressible as Unicode-1 codepoints and entities of 0x110000 or higher being keys into a dictionary of complete characters expressed as unicode-2 or higher. Capitalization, diacritic change, etc, become point mutations again and references to particular locations within a string cease to be poisonous.

Strings are limited to use no more than 2^31-or-so distinct characters, without restriction as to how many codepoints each character may be or how many instances of each character may appear in the string.

This leaves a question about whether it's worthwhile to have a fixed-width primitive "character" datatype separate from strings. Offhand I'd say it's not. We can use the word "character" to refer to strings of length one, the same way we use "empty string" to refer to strings of length zero.

A primitive (non pointer containing, requiring no GC tracing) character type might be worthwhile but would necessarily be limited in the character repertoire it could contain. If we trace the space requirements of our string representation (64-bit pointer to a buffer of keys, at least one key, another pointer for a dictionary of entities, but dictionary left empty (pointer NUL) if the key is a member of unicode-1) we get 160 bits, which is enough space to store 7 codepoints. Unicode-7 is a good character repertoire, but the principle of least surprise says that if strings are supposed to be sequences of characters, I don't want to have a character type that can't be used to contain any character that can appear in a string. We'd get programmers trying to use it to hold entities from a string and then getting errors because they tried to store a unicode-12 character in it.

Ray, it's hard to tell what you're arguing for and against here.

Unicode does a great job at defining code-points and binary formats for the same.

It does a pretty horrible job at almost everything else.

There are often multiple correct (but different) ways to encode the same exact same grapheme. And there are often many nearly identical graphemes that differ only in their designated Unicode region.

While there are many code-points, often it takes a sequence of 2, 3, 4, 5, 6, or more code-points (each conceptually a 32 bit value) to represent a single grapheme. Even worse, the order of those code-points within the grapheme is potentially variable.

But as long as you're working in only one sub-set of Unicode (e.g. Chinese) and you're not displaying, searching, comparing, or sorting text, Unicode is OK I guess.

Some

Programming Language expressions are visually formatted, and the Unicode Bidi algorithm is not compatible with our visual layout conventions.

While it's reasonable to have a different visual layout, for example indicating nesting level with right margin indentation rather than left, it is not reasonable to have a visual layout whose criterion changes at every line boundary depending on what was displayed the previous line. Right-margin boundaries don't line up indicating that something is at the same nesting level as lines above and below that line up on a left-margin boundary.

Visual layout is also used to separate comments from code, with line comments usually on the right side and code on the left. While again it is reasonable to have a different layout with comments on the left and code on the right, it is not reasonable to switch convention from one line to the next depending on what was displayed on the previous line.

Finally unicode designates most "syntax" characters such as mathematical operators and symbols as being bidi-weak or bidi-neutral, and that means that sequences of them display in either direction depending on the direction of the most recent bidi-strong character displayed. This is reasonable for natural languages because natural languages have redundant syntax that makes it possible (usually) to instantly identify gibberish when the sequence is reversed. But it's poisonous for formal languages, because formal languages have dense syntaxes and reversing a subsequence often makes a perfectly valid meaningful statement whose meaning is wrong.

So we have a vocabulary of strong two-dimensional semantic hints that are disrupted by one-dimensional bidi changes, and we have sequence-based semantics that are often meaningful with incorrect meanings when subsequences are reversed.

Bidi should operate where it is applicable to sequences that are meaningful in human languages. The spelling of identifiers, the contents of string literals, and the contents of comments SHOULD definitely obey the bidi rules.

But multiple possible rendering sequences for syntax and keywords of code and the visual placement of comments are unacceptable sources of visual ambiguity and confusion. These things must not be influenced by the spelling of identifiers, the contents of literal strings, and the contents of comments.

Unicode has recommendations for programming language identifiers, in terms of character derived properties XID_START and XID_CONTINUE. The recommendation is that an identifier should consist of an XID_START character followed by zero or more XID_CONTINUE characters. Because both XID_START and XID_CONTINUE contain characters that are Left-to-right, Right-to-left, and Bidi neutral or weak, this would cause different identifiers to have different effects on the visual arrangement of surrounding code.

I propose modifying the recommendation by requiring an LRM (unicode LEFT-TO_RIGHT MARK) as a prefix for identifiers that do not otherwise start with a left-to-right character and as a suffix for identifiers which do not otherwise end with a left-to-right character. LRM is not otherwise allowed in identifiers.

This requires the LRMs in identifiers in some cases where they are redundant (for example identifiers composed of Bidi-neutral characters) but this form of the rule is preferred because LRM's are invisible except by their effect on rendering, and strings of bidi-neutral and bidi-weak characters that require complex state to render require complex state to place and check. There's no circumstance where we want an identifier to look visually different depending on rendering state in which it's found and no circumstance in which we want the spelling of an identifier to propagate an RTL rendering state to anything outside the identifier, so the simplest rule is best.

I don't expect programmers to consistently insert or visually check for invisible characters in their code, and any allowed variation in the placement of these invisible characters is another potential for visual confusion in the sense of things that look the same and are different. For both reasons handling this requirement must be automated. This rule requires minimal state and simplest code for the compiler/interpreter to check identifier syntax and requires minimal state and simplest code for IDEs to correctly and consistently add the LRMs to identifiers where required.

https://unicode.org/reports/tr31/ -- Unicode Identifier and Pattern Syntax

Unicode isn't really worth getting upset over. Language is messy, and the ways that we have chosen to record it are equally messy. I doubt there exists a perfectly good representation for text.

If we skip unicode for a second and consider the same set of issues (readability, deceptive formatting etc) then they have always existed when programming in an ascii-only language. Programmers are quite used to the problems around 0 and O, or 1l|. Normally they are solved by picking an appropriate typeface to make the differences more obvious.

There is a similar split in responsbility here - the visual presentation of code depends on the editor used to view it. Highlighting ambiguous sequences, choosing explicit rendering and other approaches to solving these problems are more tractable in the context of building an editor than they are in designing a language. The set of rules to encode workarounds to these problems are very complex. The set of cases for an editor to handle are smpler, or can be avoided by providing forms of UI that don't have a good match when parsing a language, e.g. switching from renering unicode as glyphs to hex-codes or character names.

Why try to solve difficult problems, when their difficulty arises from the place that you are trying to solve them, rather than trust in a separation of concerns and allow them to be solved in an easier context?

PS Apologies for doing the classic stackoverflow thing of "don't solve X do Y instead" ... but doing Y is easier :)

I think I found the right answer.

The Unicode bidi algorithm is applied separately to every paragraph. Treating individual tokens separately where there is any application of bidi direction changes is what we want, so making them technically 'paragraphs' would do the right thing.

Paragraphs are separated by codepoints with the bidi class B (Forced Boundary). Most 'B' codepoints also force a line break, which we don't want. But there are three control characters from the ASCII range, 0x1c, ox1d, and 0x1e, that have the 'B' bidi class and do not create line breaks.

These characters are 'File Separator', 'Group Separator', and 'Record Separator' in Ascii, but lack descriptive unicode names.

Testing in three different text rendering systems consistently gives good results if a 'Group Separator' character is inserted at token boundaries - the tokens are rendered in the same visual location regardless of how they or surrounding tokens are spelled, but the characters within the tokens are located according to the bidi rules.

Testing in GCC treats the 'Group Separator' U+1D as whitespace, which is not problematic if it is encountered at a token boundary. It is a nonprinting, nonspacing character, so if encountered elsewhere it *CREATES* a token boundary with no visual clue that a token boundary exists.

For my next trick I will hack an emacs mode and add 'Group Separator Insertion at token boundaries' to the default syntax coloring (or maybe 'electric autoindent' since syntax coloring doesn't actually insert codepoints) behavior.

Ideally it would be 'Group Separator Insertion at the boundaries of any token containing at least one RTL character' meaning no change to the vast majority of existing code. But that might be a couple more tricks down the line.

https://mcilloni.ovh/2023/07/23/unicode-is-hard/

First time I attempted to post this I got an error message that said 'try again in a few minutes.'

So a few minutes later I tried again, and the post went up. And after that there was one copy of the post showing.

But today there are two. I don't know how long the dupe has been showing here but I'm deleting one of them.

In the course of optimizing my tokenizer I discovered a strange thing. Explicitly defining what constitutes a legal identifier isn't technically necessary.

Syntax is predefined. It consists of semantic syntax like operators, symbols, delimiters, data constant values, etc, and asemic syntax like whitespace, comments, bidi separators to prevent any token from affecting the rendering or placement of any other token, etc.

All syntax has the property that it cannot be a substring of a keyword or identifier. If a complete syntactic form is read, there is no possible prefix or postfix of it which changes the fact that a syntactic form begins at its location. The next character is either part of a longer syntactic form, or the first character of a token following the syntactic form.

Keywords are also a predefined set. But keywords have a different property: because they are made of a subset of the characters legal in identifiers, and the tokenizer makes single tokens from sequences of such characters of maximal length, keywords and identifiers can't immediately follow each other in either order. At the very least they must be separated by some asemic syntax like whitespace.

Some of these properties are specific to the language I've defined, but they seem reasonable to me, and reasonably common.

These lead to a peculiar parsing strategy: My parser is now just scanning for syntactic forms. Any nonsyntax characters found between syntactic forms MUST be either exactly one keyword or exactly one identifier. There is no other possibility. So after a syntactic form I track keyword lexing for as long as a keyword could possibly be completed, and if the lexer accepts characters past the end of a keyword or just never completes one before the beginning of the next complete syntax token, then those characters must be an identifier.

The syntax of identifiers is therefore literally any substring found between syntax forms which is not a keyword. This allows some damned peculiar identifiers, including strings that could have been the beginning of a syntax form, strings of emoji, and other abstract symbols.

This is both amusing and interesting, but impractical. I will need to explicitly add back some checks to limit the set of identifiers for three reasons. First, the first version of the language probably won't be the last. I will need to add both syntax and keywords in the future and it would be awkward to convert existing identifiers into either. Second, I need to impose some structure on identifiers to prevent deceptive identifiers that contain invisible characters or otherwise look like something they aren't. Third, seeing an identifier which contains the initial few characters of an incomplete syntax form is confusing and probably shouldn't be allowed. It doesn't happen often because most initial substrings are complete syntax forms after a single character, but when it happens it's still confusing.

Anyway the discovery that taking time to check identifier syntax isn't strictly necessary is interesting and amusing so I thought I'd share it. All that really

must