Lambda the Ultimate

1. The only condtional in the language is pattern matching, so at some point it would get matched against "true"/"false" and anything else would raise a match error.

2. We evaluate code live so the result is immediately visible. On an error we pause the code and drop the user into a debugger at the point where the error happened. That does still force them to follow the flow of "treu"s back to the source by hand. That seems like a problem we need to solve in general, not just for typos. Some form of type inference might help catch that earlier.

3. We diverge drastically from lua (live-coding, purely functional, pattern matching, graphical editor etc). The focus is on making a highly discoverable environment.

if there's no static checking up front to catch my fat-fingers, then i sure hope e.g. after a 10 hour run and then the failure i can just fix the typo from ture to true, and keep on trucking? :-)

Yep, thats the plan.

We're planning to also have structural type inference (ala dialyzer) in the ide to help catch mistakes like that up front.

Actually, there is a problem:

not ("zing" == "true") == "true"

but:

(not "zing" == not "true") == "true"

is an error. So an example of a boolean identity that is lost is:

not (x == y) == z

is not the same as

(not x == not y) == z

so I guess they should be separate types.

Given that the langauge is dynamically typed, that seems like a problem even with separate boolean types:

not ("zing" == true) == true

(not "zing" == not true) is an error

The only way I can see to avoid that is if ("zing" == true) is an error, but then we the user need to check (type("zing") == type(true)) all over the place.

I'm not sure I like either version. Getting equality right is going to be hard.

Getting equality right is going to be hard.

One way is something like:

a == b is defined if the operand are both of the same type, and an error otherwise.

a same? b is true if a == b and is otherwise false.

Where your pattern matching flow operators *implicitly* pick a form of equality, default to same?.

When teaching a novice programmer about equality in your language, initially only teach == and leave same? as an "advanced topic".

The advanced topic is basically polymorphism and the topic would include same? and type predicates string? x, number? x, boolean? x, and table? x.

The idea is that the novice gets pretty far relying just on == and not having to know much about types.

I have a theory about programming language design.. if you're having trouble figuring out an algorithm, and the implementation of it looks complex .. its wrong. Its wrong because the user is going to have exactly the same problem you are. If the actual algorithm is complex, reasoning about the code will be complex.

I have made this kind of mistake a number of times in my own language, I regret now the extra work I had to do implementing something clever which turned out not to be clever at all.

Business logic is NOT simple. It's intrinsically complex to start with. So anyone coding it needs some understanding of programming and just isn't going to have any problem at all with a boolean type.

In fact, since business often involves multiple options and that's quite normal, I'd seriously think about giving them a real sum type. They'll have less problem with it than the average CS trained programmer.

While I hear some ring of truth in what you say, I think people have to be careful about extremes and about what precisely is needed. There's lots of crap in programming languages that most of us don't recognize as crap because we're inured to it. That doesn't make it stop being crap in the absolute universal truth insightful usability sense.

If we are trying to reduce something (business process) down to math (algorithm) then presumably either the algorithm writer has to know math or the system has to know enough math for them to help them write it out in a way that works. I suspect we don't have systems like that, so it means the algorithm writer does have to know math.

If we are trying to reduce it to something else, like a rules system, then the person / system have to know enough / hand-hold enough to get it working.

etc.

So yes it is probably dangerous and silly when people think it can be "simplified" into something too simple (as in they went past the "but no simpler" rule).

But we should still be striving to look for the true essence, the true essentials, and seeking to reduce and eliminate the accidentals.

It isn't so bad to use strings to represent booleans if you use dependent types or similar, i.e. such that you can enforce upstream the expectation that a string is "true" or "false" and nothing else (and thus carries one bit of information). Even for dynamic types, this can work if 'error' is not observable (i.e. it halts the program).

Though, booleans are awful anyway due to boolean blindness. I ended up favoring sum types as a foundation for conditional behavior.

I ended up favoring sum types as a foundation for conditional behavior.

Exactly. Reifying propositions as values means you lose important semantic information. I don't think many languages have taken this route however, so I'm interested to see how it turns out.

Most uses of bools seem to be justified because of insufficient abstraction, ie. you want to pass a membership testing function (forall a.a -> bool) into some polymorphic function, so you need to be able to abstract over propositions-that-aren't-values. In the case of sums, this implies abstracting over sums ala views, or first-class patterns/cases of some sort.

I didn't find the Bob Harper's article very convincing. Ignoring the mumbo jumbo about constructivism, which I found totally unconvincing, I think "just restructure your program so that all of the properties you are interested in come for free" isn't realistic (or good) advice. It's not even possible in many circumstances. What should you do when the boolean you're trying to eliminate implies the presence of data in several places with differing multiplicities? Even when it is possible, I think it's often not a good idea to do because unwrapping constructors everywhere is noisy and it couples the code that uses some data to the logical structure governing when that data is available. The article makes a snide comment about requiring a SAT solver, but in those cases where the boolean can simply be replaced with constructors the reasoning required to verify that the correct booleans hold is equally trivial.

The difficulty with booleans is something I encountered long before I read Bob Harper's article, but he gave me a nice name for it. The real problem is well summarized towards the end:

"we’ve crushed the information we have about x down to one bit, then branched on it, then were forced to recover the information we lost to justify the call to pred" -- Bob Harper

The pattern of losing information then recovering it does not generalize well to computation with substructural types or heterogeneous systems (e.g. GPU/CPU partitioned memory). It is also not very compositional.

unwrapping constructors everywhere is noisy and it couples the code that uses some data to the logical structure governing when that data is available

I don't follow these complaints. Unwrapping constructors does not need to be noisy (cf. typeclasses). The framework (or logical structure) to access information can be decoupled from the function that uses it (cf. lenses). If we avoid booleans, a few programming idioms will change, but that isn't a problem.

What should you do when the boolean you're trying to eliminate implies the presence of data in several places with differing multiplicities?

We should have a very good, structural explanation of how this boolean is obtained or enforced.

In any case, the idea isn't to "eliminate" booleans. The idea is to avoid them in the first place. The difference is significant; in the latter case, your libraries and frameworks and language will be helping you out; in the former, booleans are already deeply embedded and you'll be fighting for every step.

"just restructure your program so that all of the properties you are interested in come for free" isn't realistic (or good) advice. It's not even possible in many circumstances.

Just be interested in the properties you can achieve structurally. ;)

I don't follow these complaints. Unwrapping constructors does not need to be noisy (cf. typeclasses). The framework (or logical structure) to access information can be decoupled from the function that uses it (cf. lenses).

Ok, I admit that I may have been over-generalizing or reading too much into the article or what you were endorsing by linking to it. And I agree there is a real problem, but think that just wrapping everything in constructors is a cure worse than the disease. I'm open to the possibility of other approaches that solve the problem (typeclasses, lenses, etc.). I have my own ideas.

We should have a very good, structural explanation of how this boolean is obtained in the first place.

No, we generally don't, because when we're writing code to use the boolean we might not have even written the code yet that produces it. And even if we have, coupling the two leads to brittleness.

The idea is to avoid [booleans] in the first place.

I agree and I'm in favor of keeping a better handle on them in the first place. Again, my main complaint is with what I understood to be the proposed solution. FYI: I also don't like Option types (or explicit sums) as the replacement for null in most cases.

when we're writing code to use the boolean we might not have even written the code yet that produces it

I don't see the order in which you write code to be relevant. If a particular subprogram doesn't need much detail about how a condition is observed or enforced or explained, you can abstract on those details at the interface to the subprogram. This abstraction is orthogonal to whether those details are preserved. We can also have functions to compose and extend the abstract explanation - indeed, every function on (whatever replaces booleans) may do so.

In any case, booleans should almost never be inputs to interesting functions or subprograms. If they are, you're probably doing modularity wrong; consider instead writing a separate function for each case. If necessary, separate the case-recognition logic.

I also don't like Option types (or explicit sums) as the replacement for null in most cases.

I'm not fond of nominative types, including `Maybe a = Just a | Nothing`. I would favor `type Maybe a = Either () a` (or `() + a`), and to just have everything in terms of explicit binary sums. There are a lot of nice structural operations we can perform using more structural types - e.g. `not :: (a + b) → (b + a)` and `left :: (a → a') → (a + b) → (a' + b)`. Generic programming is much easier when we can decompose and recompose types. Further, we can readily generalize from `Maybe` to `ErrorT e` if we want to treat it as a Monad.

With dependent types, we can presumably leverage dependently typed pairs to model sums. However, I've had quite a lot of difficulty generalizing structural operations to dependent pairs. How can we preserve the type inference relationship when performing structural operations on the pair? `swap :: (a * b) → (b * a)` isn't particularly difficult, but when we start using `assocl :: (a * (b * c)) → ((a * b) * c)` and `first :: (a → a') → (a * b) → (a' * b)`, we must start reasoning about whether functions on `a` are injective so we can recover the relationship between `a'` and `b`.

Further, dependent pairs still have the same fundamental boolean blindness issue of performing logical recognition on `a` then forgetting about it, repeatedly.

I don't know many other alternatives, so I ended up favoring structural sum types.

If a particular subprogram doesn't need much detail about how a condition is observed or enforced or explained, you can abstract on those details for just that subprogram.

I was considering explicit abstraction to be expensive for such a common operation as this. In the early stages of programming, when you're just writing down fragments of code, I think it's helpful to not worry about writing down the precise assumptions or context in which each fragment works. I still think coarse HM-ish typing is useful at this stage, but more precise types for establishing program properties should be optional and usually come later. Forcing early resolution of such logical properties by embedding them into the structure of data is IMO an anti-pattern.

What I have in mind is more like (oft-maligned?) type state. Rather than a type of "Maybe Int", you'd have a type that says "Int, when initialized". This feels like it fits into my language philosophy better, but I haven't hammered out too many of the details yet.

Forcing early resolution of such logical properties by embedding them into the structure of data is IMO an anti-pattern.

You seem to be assuming early resolution of the data structure.

Coupling logical properties to data structure some nice consequences. We can abstract logical properties by abstracting data structure - something we know how to do very well (ADT, OO). We can manipulate and refine logical properties by manipulating and refining data structures - either constructively (e.g. non-empty list as pair of element and list, substructural and modal types) or analytically (i.e. by assertion or contract).

During early stages of development, we can use more generic data structures - lists, matrices, graphs, trees, streams, etc. - and thus enforce fewer logical properties.

I know I'm making assumptions and acknowledge that these are fuzzy judgements I'm making. (The original rant against booleans was similarly fuzzy). The design space for programming languages is quite large and it's quite difficult to even identify what my assumptions are.

During early stages of development, we can use more generic data structures - lists, matrices, graphs, trees, streams, etc. - and thus enforce fewer logical properties.

When you switch from using lists to using pairs to encode non-empty lists, do you have to change all of the code that uses this type?

When you switch from using lists to using pairs to encode non-empty lists, do you have to change all of the code that uses this type?

In general, yes. Fortunately, this sort of global refactoring can be quite straightforward and easy, especially when guided by a type system.

Partial functions (and inference of refinement types) can help ease the transition, make it more incremental (one small subprogram at a time instead of "change all the code"). E.g. Haskell has `head :: [a] → a` and `tail :: [a] → [a]` on a regular list, which diverge if applied to an empty list. My language offers `assert :: (a + b) → b` for similar reasons, except I treat assertions (and non-divergence in general) as a weak objective for static proof (warn if cannot prove or disprove within limited effort).

Of course, best practices for API design still apply for shared libraries or frameworks. One should encapsulate decisions and structures that are likely to change.

"In any case, booleans should almost never be inputs to interesting functions or subprograms. If they are, you're probably doing modularity wrong; consider instead writing a separate function for each case. If necessary, separate the case-recognition logic."

Although this advice seems sound, it is very often not practical because the language fails to provide suitable support.

In the first instance replacing a function with so-called "threaded logic" with two functions will lead to substantial parts of the function being duplicated. So next you will tell me to factor the common parts out into subroutines. Which now has major problems because you have to name them, and you have to pass context to them which means you need a type representing what was previously private state.

If you now take these issues of bifurcation and have 4 or 5 boolean arguments for threading, unthreading the code is utterly intractable.

This problem is manifest in Ocaml pattern matching, where your first crude split into several cases is followed by a second test, which is repeated in several branches. You can certainly lift the common code out of the match expression but then you have lost context and more importantly locality, defeating the whole idea of lexical scoping.

So despite the apparent evils of threaded code, it actually has better compositional properties than the alternatives in many cases. No one likes the spaghetti presented in many Posix functions, lots of flags and stuff, but when you look at the complexity of the operations represented its hard to see a good alternative.

Language can hinder clean factoring, I agree. Local variables, nominative types, and second-class patterns can be major hindrances. As can be poor dataflow optimization.

I've leaned towards concatenative style (even in Haskell and C++) and structural types (templates, pairs, etc.) for a long time. GHC's ViewPatterns extension is also nice for factoring and flattening out some of the case-recognition logics.

I wonder to what extent boolean blindness can be ameliorated by first class patterns.

Suppose that all the primitive boolean-returning functions are actually patterns instead eg


match x with
| (>= 0) -> ...
| _ -> ...



filter(xs, match (>= 0))


Then when you branch on a proposition the information that it encodes is right there. You could of course still choose to return some bool-like value.

When patterns are statically determined they can provide structural type information directly without having to work backwards from bools and the match compiler has more information to work with when ordering tests.

Erlang goes some way towards that ideal but it lacks first-class patterns and so functions like filter still have use boolean atoms.

Patterns help only if they refine the matched values.

In your example, the `(≥ 0)` pattern would generally not help. The code on the RHS, the `...`, would use `x` again but contain a context-dependent assumption that `x ≥ 0` (or `x < 0` on the failed match branch). Because these assumptions are context dependent, it is difficult to refactor or extract this code for reuse.

You might do better with something like GHC Haskell's ViewPatterns extension, which aren't quite first-class but are flexible and subject to abstraction.

A compiler can trivially add local assumptions to the typing environment. They just have to be conservative. For example, in imperative pseudo-code:

int f(dynamic x) {
  // x :: dynamic
  if (x is int) {  // x :: int
    if (x > 0) {  // x :: int, x > 0
      return (x - 1);
    } else {  // x :: int, x ≤ 0
      return (x + 1);
    }
  } else if (x is object) {  // x :: object
    // x :: object
    if (x = null) {  // x :: object, null(x)
      return 0;
    } else {
      return x.toInteger();  // x :: object, nonnull(x)
    }
  } else {  // x :: dynamic (no new information)
    return 0;
  }
}

You often see boring branchy code like this in the wild, making assumptions that could easily be verified by the compiler—for instance, that it’s safe to access .toInteger on x if x has already been tested for nullity.

Of course, with mutatey OOP-land there are some caveats: you have to propagate assumptions through calls (undecidable?) or assume conservatively that because any non-const member function can mutate any mutable member, assumptions can’t be preserved across calls via this.

Because these assumptions are context dependent, it is difficult to refactor or extract this code for reuse. Without effective modularity and decomposition, we cannot readily abstract these deep 'local' patterns. They get repeated at each site.

Verifying such code can be a good thing, of course. Typed Scheme seems to work on a similar design. But it fundamentally doesn't address the boolean blindness problem - i.e. that you are forced to keep all this context (which is not part of the boolean) in order to make sense of the boolean.

If I want to extract an interior block, I may not be able to do so because its contents may rely on assumptions from an enclosing one. Ideally you would have a first-class way of talking about this kind of information, which effectively implies having proof objects.

I do question whether doing it “the right way” is worth the additional cost, though. Having a model that’s easy to demonstrate concretely is important, and I think you would have little trouble explaining something like this to Sam Imperative Programmer:

int f(nullable int x) {
  return (x + 1);  // error: (+) expected int but got nullable int (x)
}

⇓

int f(nullable int x) {
  if (null(x)) { x := 0; }
  return (x + 1);
}

Proof objects can be a bit heavyweight, yes. What I've been pursuing is something lighter weight - just a different way of expressing observations that is hopefully easier for both human and type system to track.

Instead of `x is int` returning a boolean, I might model observeInt :: (Observable x) ⇒ x → (x(not int) + x(int)). (Here 'Observable' means 'x' must accept this sort of introspection; in a conventional dynamic language, all values might be observable. But I'm not fond of universal introspection.) We could then, in the left branch, set the value to 0 then merge. Similarly, instead of `x ≥ 0` returning a boolean, I might model observeGTE :: (Comparable x y) ⇒ (x*y) → ((x*y)(when x<y)+(x*y)(when x≥y)).

By associating type refinements with data structures, the type checking should (hypothetically) be easier to perform, and certainly operations to collapse the sum (losing information, type unions) are much more explicit. But the biggest motivation is that operations on sum types can be composed and decomposed, i.e. unlike if/then/else expressions which are syntactically closed.

That's pretty convincing. Even for comparisons, the result is not really boolean eg
equals: x. x -> x -> "equal" | "not equal"
compare : x. x -> x -> "less than" | "equal" | "greater than"


Performance-wise, we can maybe do something with interning, along similar lines to erlang atoms or clojure keywords.

Getting rid of separate number types make me nervous. For scientific work its quite important to be explicit about precision and I'm not sure how to do that with only a single number type. Perhaps we could default to decimal arithmetic with string-like numbers and only expose other representations as an advanced feature.

Since we're building a structured editor we have a lot more leeway with syntax eg we already have a prototype 'math mode' editor where you see nicely laid out MathML rather than flat text. The idea is that the user is choosing a context to work in at any moment and we only suggest functions that make sense in that context.

FWIW, I started writing a longer response, but it turned into a big long raw braindump which may or may not be of any value, so I'm a little reluctant to post it here. I could bob it to you over email if you want, or post it here if you really want.

As to numeric types, bear in mind that what the user sees and thinks doesn't necessarily need to match what the interpreter sees and thinks. Language design is UI/UX design: the first goal is to meet users' needs and expectations as best as possible, not make life simple for the poor language developer.;) The only real rule is that where you do hide internal complexities to simplify the external interface, make sure those abstractions aren't going to spring leaks all over the user five minutes after they start to use the.

So, as far as having distinct number and string types:

Your runtime could internally implement both Number and String classes and have the parser optimistically treat any token that looks numeric as Number and the rest as String. You can then implement a pair of Number->String and String->Number coercion handlers, so that it doesn't matter if the user passes (String,Number) to the math addition operator or (Number,String) to the text concatenation operator: the runtime will automatically coerce the operands to the required types, (Number,Number) and (String,String) respectively on the operator's behalf. From the user's POV though, you just document it all as being a single 'text' ['scalar', whatever] type.

This is similar to what kiwi does. Since math and performance are not major requirements it doesn't bother having an internal/explicit Number type at all. Relaxed quoting rules means most text never needs quoted anyway, so 123.45 and "123.45" are both exactly the same thing: a text value that just happens to look like it's some sort of number-y data. OTOH, kiwi's 'text' type is actually implemented as a cluster of concrete classes (to which I could always add a NumberText in future if it became of significant benefit):

1. PlainText is a plain vanilla character sequence (i.e. your standard String), e.g. "Hello, World!"
2. RichText is a text value with one or more unevaluated rules attached (think "attributed string", but with behaviors, not styles, attached), e.g. [bold, case (upper) @ "Hello, World!"]
3. ExpandedRichText is a text value with one or more partially-evaluated rules attached, e.g. [bold @ "HELLO WORLD!"]
4. CompositeText is an ordered sequence of one or more text values and/or unevaluated Tags, e.g. "Hello, "{$name}"!"

From the user's POV, this is all just known as 'text', and documented as a single type. The brackets in #2 are just a convenience syntax you can use when you want to attach some rules to your text. (They're not the only way to attach rules to text values, but they're concise and visually pleasing.) The composite text means I never have to talk about text concatenation operators or interpolation: users just mix tags and text as they like and the interpreter treats it exactly as they'd expent when it evaluates it. Their biggest peeve is forgetting that if they want any whitespace inserted around the tag in #4, they have to quote it else kiwi will collapse it down. But testing, tweaking and re-running their code to obtain the perfect output is no biggie; and consider all the other 'conventional' language concepts they don't have to learn in order to get stuff done.

Kiwi's coercion handler system ensures that any rule that requires a number will get a text value that represents a number, or else a "not a valid number" error will be raised. This applies to both native and primitive rules, so if a primitive rule (which is implemented in Python) requires, say, an int or float, the given kiwi value will be coerced to text, its underlying Python str extracted and cast to the appropriate Python type. Constraints can be checked and composite values can be described, e.g. to coerce a given kiwi value to a CMYK color value (i.e. a tuple of 4 numbers, each between 0.0 and 100.0) in Python:

PySimpleTupleType(4, PyFloatType(0, 100))

or in kiwi:

list (number (0, 100), 4, 4)

Optional/default values can be specified, any unevaluated tags will be automatically expanded, thunks can be requested, and so on. And the whole lot's self-documenting, so when you generate the user documentation the above definitions are displayed as:

list of 4 number from 0 to 100

There's even an option to override these autogenerated descriptions, should you want/need to provide better ones yourself.

I've also made some progress on unit type support, meaning that users can write 1.5in or 38.1mm, and supply it to a primitive rule that wants the length in points as a Python float, and the coercion system will happily produce 108.0 for them. No dedicated data types required: users just need to check the manual to remind themselves which unit type names it recognizes (cm, mm, pt, and so on, including any abbreviations, synonyms, etc.), and what they type in should just work.

I can fill in a bunch of history if anyone wants, but the bottom line is this: awk really wanted to be typeless (except that it has arrays, which can be deduced syntactically), but it ended up with three types: string, number and "maybe number". (Posix calls these "string numbers", I think.) All values enter a program either by being typed directly as part of the program text, in which case they are strings or numbers depending on whether they are quoted or not, or by being read from a text file or command line parameter, in which case they are "maybe numbers". (One exception: variables not explicitly initialized start their lives as "maybe numbers"; an empty "maybe number" is either the empty string or the number 0.)

If you use a string or a maybe_number in a place where a number is required (that is, with an arithmetic operator or as the argument to a math function), then an implicit conversion will be made in which the string representation is made into a number by force, if necessary, which means ignoring the part of a string which follows a legal number, even if that means ignoring the entire string. The result of the arithmetic expression is thus a number. If a number is used where a string is required (concatenation operator, for example), then a string conversion will be inserted. Neither of these conversions do not alter the value or type of the object.

A "maybe number" is a number if its entire value conforms to the lexical form of a number, possibly with trailing and leading spaces, or if it is empty (in which case it is the number 0). Otherwise, it is a string. This has no effect on the above conversions, though, because in a string context, a "maybe number"'s string contents are used as a string. (If the value were a number and used in a string context, the standard conversion would convert 1.0 to the string "1", so this fact is visible.)

Comparison operators work with both strings and numbers, so there is an ambiguity. In comparisons, stringness wins: if either operand to the comparison is a string, including a non-numeric "maybe number", the other one is converted to a string (using the standard conversion operator) and the comparison is lexicographic; otherwise, both operands are numbers and the comparison is numeric.

The end result of this rule is that < and > are not order relations, which you can easily demonstrate on any standard awk with the following command:

awk -v a=10 -v b=10b -v c=9 'BEGIN{print a<b, b<c, c<a}'

Here a and c are numeric "maybe numbers" and b is a non-numeric "maybe number"; consequently, the first two comparisons are lexicographic, and are both true, while the third comparison is numeric and is also true.

This has unfortunate consequences for sortation. Some awk implementations have sorting functions which fail completely because they assume that the built-in < operator is actually an ordering. And user-written sort functions almost always suffer from that problem.

So the original decision to combine numbers and strings "for simplicity" eventually leads to very obscure bugs, which are not at all easy to explain to the supposedly naïve users it was supposedly helping.

If you use a string or a maybe_number in a place where a number is required (that is, with an arithmetic operator or as the argument to a math function), then an implicit conversion will be made in which the string representation is made into a number by force, if necessary, which means ignoring the part of a string which follows a legal number, even if that means ignoring the entire string.

Well, that's a big stinky fault in awk's design, obviously. (But hey, that's Unix: the very best and very worst of hacker culture all rolled into one incredibly persistent package. Ask me sometime what I think of untyped streams...)

My current position on this (and expressed though kiwi) is that coercions should only change information's representation, they should never change what it means to the user. If the user wants to alter the data's logical meaning, they will do so themselves by applying the appropriate rules to it. The data's 'type' is merely metadata advising information consumers on how best to interpret it.

Kiwi coercions are non-lossy: if you try to supply "123abc" where a number is required, it will report an error: 'Can't coerce the following text to number: "123abc". Not a valid number.'

Even explicit casts can't mess with meaning. For example,

R> ^^ 123.45
R> whole number
══════════════════════════════════════════════════════════════════════════════
IKI DETECTED AN ERROR:

The 'whole number' type rule can't coerce its input value to whole number.

Can't coerce the following text to whole number:

"123.45"

Not a valid whole number.

══════════════════════════════════════════════════════════════════════════════

Contrast the many languages that will happily alter the data's fundamental meaning when casting to int. It's really no different to your awk example: muddling and muddying the distinction between meaning and representation, and who and what should be responsible for each. But it's such a ubiquitous practice that nobody even thinks to stop and question it any more: it's the way things have always been done done and always will be done and that's just as it should be.

Casts and coercions should deal with how information is represented. Commands and operators should deal with manipulating and modifying existing information to produce new information.

Here a and c are numeric "maybe numbers" and b is a non-numeric "maybe number"; consequently, the first two comparisons are lexicographic, and are both true, while the third comparison is numeric and is also true.

Again, this is awk's incompetent implementation of the principle, not necessarily an inherent flaw in it. awk's by no means the only one to screw up here either: here's a similar demonstration from AppleScript (also weak, dynamic), where the type of the LH operand determines whether to perform a lexicographic or numerical comparison:

"33" < 5
--> true

33 < "5"
--> false


"foo" < 5
--> false

5 < "foo"
-- Error: Can’t make "foo" into type number.

This never happens in kiwi, because comparison rules are not overloaded. If you want to do a numerical comparison, you use:

=
≠
<
>
≤
≥

If you want to do a textual comparison, you use:

is equal
is not equal
is before
is after
is not after
is not before

(The names are long form for easy end-user readability, but power users can alias them to eq, ne, etc. if they prefer less typing.)

While the first set of rules work as you'd expect, performing a straight mathematical comparison of two numbers, the second set are generalized and far more capable than the simple text comparison operators you'll find in most languages. By default, they perform a simple text comparison (or report a coercion error if both values are not text). However, you can also pass a type rule, which will not only coerce both values to the specified type prior to comparing them, but also provides fine-grained control over how each part of the comparison is performed:

R> define type (wibble, group ((text, number, case-sensitive text, list (number))))
R> 
R> ^^ (FOO, 4, bar, (5, 6, 2))
R> 
R> is after ((foo, 2, Car, (5, 3)), wibble ) 
#  "ok"

Not a realistic example of use, of course, but does demonstrate a point. You can throw all sorts of complex/tricky/sneaky requests at kiwi that would trip up less rigorous 'scripting' languages, and it will do the right thing: either output a correct result or report an error and refuse to do it. (Anything else is an implementation bug.)

The sort list rule, which by default sorts everything by case-insensitive alphabetical order, also allows users to specify alternate sorting behaviors just by passing the appropriate type rule:

R> ^^ (Foo, bar, BAZ)
R> sort list
#  (bar, BAZ, Foo)
R>
R> sort list (case-sensitive text)
#  (BAZ, Foo, bar)

R> ^^ (33, 5, 2)
R> sort list
#  (2, 33, 5)
R> 
R> sort list (number)
#  (2, 5, 33)

For reference, here's the actual implementation of the is after rule:

class _Comparison(Handler):
  
  kInput = (PassThru(kAnyOrNullType), '')
  kOutput = (PassThru(kBoolType), '')

  kParams = [
    ('value', kAnyOrNullType, ''),
    ('type', Default(kPyTypeDescType, PlainText('text')), 
             'compare input and value as this type'),
  ]
  
  @processdata
  def apply(self):
    t = self['type'].cmptype()
    self.output = kTrue if self._calc(
        self.input.expand(self.env, t), 
        self['value'].expand(self.env, t)) else kFalse

##

class IsAfter(_Comparison):
  """ Does input come after value? """
  
  def _calc(self, a, b):
    return a>b

All of the heavy lifting is being done by kiwi's type descriptor system. Python's ability to compare lists and tuples as well as numbers and string means the actual comparison operation is a one-liner, but all of the preparation up to that point comes for free: coercing and unpacking the kiwi values as Python lists, tuples, strs and Decimals, normalizing those strings that require case-insensitive comparison, and anything else. The only unusual thing about this rule's implementation is the cmptype call, which asks the given normal typedesc to return a corresponding typedesc that yields Python values specifically for use as comparison and sorting keys.

Kiwi coercions are non-lossy

This suggests you don't lose formatting information, e.g. that you require all number strings to be in one format. Consider:

• "123.45" vs. "123.450" vs. "12345/100"
• "123" vs. "0x7b" vs. "000123"

If your coercions are truly non-lossy (i.e. coercion functions are bijective) then you'll allow only one string representation for each number - a nice property. `(toNum a) = b` iff `a is equal (toString b)`.

OTOH, if your coercions are lossy, I suspect you'll encounter some subtle reasoning issues. You speak of 'meaning vs. representation' but to the extent meaning depends on non-local context, it can be difficult to validate or reason about.

If your coercions are truly non-lossy (i.e. coercion functions are bijective) then you'll allow only one string representation for each number - a nice property. `(toNum a) = b` iff `a is equal (toString b)`.

Probably non-lossy would mean injective rather than bijective. For any x:S, coercing from S to T to S produces either x or an error. Still, the examples you gave show that having a non-lossy coercion probably isn't desirable for a scripting language. If "123.450" and "123.45" aren't both valid reps of 123.45, then what's the point of treating strings as numbers? It's just going to confuse people. So then we have the issues of a lossy coercion.

This suggests you don't lose formatting information

Correct. Within native kiwi code, a number 'cast' doesn't change the data's type, since it's text in, text out. It only checks that the input is a text value that meets certain constrains then outputs it untouched:

iki – an interactive kiwi interpreter (type '^ h' for help) [1]
R>
R> ^^123.45000
R> number
R> ^
#  "123.45000"


R> ^^000123.45
R> number
R> ^
#  "000123.45"

Conversely, if you supply a non-text value or a text value that isn't a recognized number:

R> ^^ (123.45)
R> number
══════════════════════════════════════════════════════════════════════════════
IKI DETECTED AN ERROR:

The 'number' type rule can't coerce its input value to number.

Can't coerce the following list to number:

(123.45)
══════════════════════════════════════════════════════════════════════════════

R> ^^ forty-two
R> number
══════════════════════════════════════════════════════════════════════════════
IKI DETECTED AN ERROR:

The 'number' type rule can't coerce its input value to number.

Can't coerce the following text to number:

"forty-two"

Not a valid number.
══════════════════════════════════════════════════════════════════════════════

Note that 0x7b and 12345/100 are not considered valid 'numbers' either, so will also be rejected. The first would be considered a transcoding task, so the user would need to apply a behavioral rule that knows how to decode hex notation. e.g. There's already a rule that knows how to convert hex to codepoints:

R> ^^0x7b 0x7d
R> unicode
#  "{}"

The second may in future be recognized as a valid number, depending on how I resolve the challenges of representing irrational numbers. And there will at some point be a calc for doing math with standard algebraic notation, which you could also feed it to to get a decimal representation. But for now only decimal (123.45) and scientific (1.2345e2) notations are recognized as-is, which should be sufficient for the markets I'm currently aiming for (print publishing and packaging).

Conversely, it's up to each behavioral rule to decide precisely what a given input value means to it. For example, the = rule is concerned solely with determining whether or not two text values are numerically identical, and couldn't care less about formatting differences: as long as both text values represent valid numbers, it will be able to compare them [2]:

R> ^^123.45000
R> = (000123.45)
#  "ok"

Being a primitive rule, it requires the native kiwi values to be unwrapped as native Python objects, so declares their type as kPyNumberType. Like number, this makes sure the kiwi value is text, but then unwraps the underlying Python string and converts it to the appropriate Python type/class, which in this case is decimal.Decimal. It then performs the relevant Python operation (Decimal('123.45000') == Decimal('000123.45')), then converts the resulting Python Boolean (True) back to a suitable (i.e. non-empty) kiwi value (ok).

For math operations that return another number, it's up to the rule to determine the new value's representation:

R> ^^123.45000
R> + (000123.45)
#  "246.90000"

(In this case, I've just left it to decimal.Decimal, which preserves the decimal places but ignores any leading zeros.)

I've also implemented a format number rule which gives the user fine control over how numbers are formatted for presentation, though that'll probably go away in future in favor of a generic format rule and an extensible mechanism that allows users to define their formatting requirements for numbers, dates, and other stuff.

...

Basically, kiwi should never change existing values' representations: it respects the user's choice to represent it that way. This is very important in the packaging industry, where there's hell to pay if incorrect artwork ends up on the shelf. Kiwi's job is to minimize risk of such errors occurring - or at least ensure the fault is somebody else's, and not its own doing.

Where kiwi produces new values, it will use whatever representation it deems most appropriate to the task/most likely to match the user's expectations/requirements. Where primitive rules unwrap kiwi values as Python objects, that's purely for their own internal use: as long as what they finally output is 'good kiwi', what they do in the privacy of their own implementation is no-one else's business.

--

[1] Notes:

• Much like a Unix command, a kiwi rule takes an input and produces an output, with the output of one rule being 'piped' to the next.
• In iki, R> is the prompt for entering and running rules, and # indicates a displayed value.
• Typing ^ at the start of a line enters iki control mode: e.g. ^^ VALUE sets a new input value, ^ displays the current input value.

[2] Obviously, when you get down to the gory implementation level, there are issues of rounding and precision which I'm still working to fully address. I've already switched kiwi's primitive math rules from using float to decimal.Decimal, which addresses the biggest gotchas while introducing a few of its own, but I'm already thinking how to plug those too.

@rici, @dmbarbour: This is exactly the sort of tyre-kicking I'd love to get myself in another month or so, once I've got the docs done and can properly submit kiwi for LtU denizens' [blood] sport.

OTOH, my posts were only supposed to give the OP something to chew over while he's deciding how to present types in his own end-user language, so we may be getting too deep into mine for his benefit. Might I suggest we take it to a new thread, and perhaps continue next week by which time I can package it well enough for you folks to play with?

(If you want to pick over my lovely Python3 code, the repo's here. Ignore pacu and saki unless you're into Adobe Illustrator automation. Also, basic usage and kiwi overview.)

I'm not going to defend awk, whose warts are quite obvious, although I use it daily. I just thought it was an interesting case study, because I think that every design decision with respect to string/number types was, at the time, logical.

As far as your hints about kiwi, I could probably find more to say but I'm not sure it would be a fruitful discussion. All I'll say is that I'm prepared to bet that if kiwi becomes popular, then one of the most frequently asked/answered questions will be something like "Why does sorting a list not work?" or more vehemently, "Why does 9 come after 10 when I sort a list?" Of course, the kiwistocracy will patiently remind the novices that you have to remember to add " (number)", to at which least some of the novices will ask, "well, why can't it figure that out itself?". But since that is just a hypothetical based on a hypothetical (but anchored in a few decades of watching scripting language development), it doesn't (yet) count as an instance of "What does the end user want/expect/understand/need?"

The shell scripting operator "test" (also written "["), as well as the extension operator "[[" require the user to remember to use "-lt" when they mean "numerically less than" and "<" (or "\<" in the case of "[") only when they mean "lexicographically before". This is a frequent source of bugs, too, but it probably does not create as many questions as the fact that you need a space before and after the open bracket(s). (ba)sh is even more subject to criticism than awk, I suppose, but even so it stands as a test of certain UI "designs". (I'm always hesitant about using "design" and "bash" in the same phrase.)

I think you underestimate business people. Only a hare brained programming type could dream up the idea a business person can't tell the difference between a numerical product code on a box, and a quantity.

Contrarily, business people will be quite confused programmer can't handle the simple concept of kinds and dimensions. Business people have no problem distinguishing 3 metres of cloth from 3 boxes of eggs.

They're already quite used to type systems. Programming is hard for them because we don't know how to represent what to them is quite obvious.

So I think you're going at this backwards. You're not making it simpler reducing the number of types, but making it harder by failing to provide a strong enough type system. *Especially* if you have an IDE that can leverage the type system to present method choices, colourise things, provide tool tips, or whatever. Your real problem should be tracking these types, not throwing them out.

They're already quite used to type systems. Programming is hard for them because we don't know how to represent what to them is quite obvious.

Knowing the nature of things is not the same as understanding and being able to use formal type systems.

Non-programmers do indeed know how to represent unit values: they read, write, and mentally model them as taught in school:

12mm
37.5°C
220lb
C50M100Y0K0
£29.99
€1.000.000,00
etc.

It's up to the software to infer any formal type information that it needs from these natural representations provided by the user. The moment you demand users explicitly type their variables, you've lost them: they don't even understand the concept of 'variable', never mind the how 'type', 'variable', and 'value' relate and interact. For them, if it looks right, it is.

OTOH, from the sofware's POV, until/unless it has sufficient information to know otherwise, such written representations are all just 'text'.

Now, how the system recognizes and represents user data internally is another question: it might well define a complex network of interrelated numerical and unit types to which it can map these written representations for efficiency and robustness. Or it might have a single 'text' type whose values are interpreted at point of consumption and annotated with additional information as the system learns more about what that data means to the user.

However, the OP's just starting to work out his basic UI/UX/HCI design, so it's way too early to be discussing how best to solve abstract internal implementation problems.

I agree with most of your conclusions, but thought I'd refute the following:

Consider what role(s) Booleans actually serve. I can think of three: conditional switching, flag-style arguments, and arguments/return values to comparison/logic operations.

Booleans only have one use: branching (ie. conditional switching). Your other two examples are just the interaction of branching with function abstraction: if we wrap branches in functions, we end up with 'flag-style arguments'; if we branch on a function call, we get 'comparison/logic operations'.

Even if we output booleans from our program, the only thing the receiver can do with them is branch.

This doesn't affect your argument and I wouldn't normally bother with the distinction, but I feel this is the kind of nit-picking that was requested :)

So you'd suggest we never (directly) compare booleans? Branch on them, instead?

I'm not trying to dictate style*, I'm just pointing out that ultimately booleans must either be ignored or branched on.

The existence of flags and logical connectives is a consequence of making code modular using functions, rather than being intrinsic to booleans.

* Except when it comes to redundant branching and double-negatives, which I have to kill on sight:

if ($foo) {
  return TRUE;
}
else {
  return FALSE;
}

if (!$foo) {
  bar();
}
else {
  baz();
}

In some sense, we could say the same of information in general: "ultimately, information must either be ignored or branched on". Ultimately, the only way to 'use' information is to have it influence some decision.

But we can also compute with information, calculate and communicate without observing or branching. And the same is true for booleans. E.g. if you say `a and b` we can combine two booleans without branching. It isn't clear to me that booleans are distinct from other value types in this respect.

if you say `a and b` we can combine two booleans without branching.

How could "and" be implemented without branching? Even if you shift the branch into a lower layer by making "and" a primitive, we're still stuck with a boolean result that's only good for branching on.

It isn't clear to me that booleans are distinct from other value types in this respect.

They're not, at least not from other finite enumerations. That was kind of my point: a language shouldn't give special attention to booleans 'because they're used for flags and (Boolean) logic', since those are just ways booleans are *used*, which also apply to other enumerations.

There are 'legitimate' reasons to special-case the booleans, for example if you only want a single finite-enumeration type and booleans are the simplest.

When breaking with tradition like this, I feel it's important to distinguish between things which are fundamental/unavoidable (ie. branching) and things which are historical/conventional/avoidable (ie. true/false flags, boolean logic). This way, old patterns aren't shoe-horned into the new design when they're not appropriate. For example, true/false flags are avoidable by using has's observation that passing the strings "true" and "false" into a function is a wasted opportunity when we could be passing descriptive strings like "normal" and "reverse".

That was kind of my point: a language shouldn't give special attention to booleans 'because they're used for flags and (Boolean) logic', since those are just ways booleans are *used*, which also apply to other enumerations.

Yep, this was my point too. The difference is: it's those patterns of usage that you need to study and understand, not the language-defined types they're currently using to do it. You design the tools to serve the tasks, not the tasks to fit the tools.

e.g. To illustrate: While the OP worries about how to represent numbers adequately without providing users a dedicated number type, my response would be: "But as scientists, how are they supposed to represent everyday values like 37.8°C and 8000K, 0.25 Ã…, 314.159µm, and 2.0626e5 AU, and so on, using only number type and its literal representation of values?"

Though IANAS, I'd speculate that being able to describe and manipulate such data efficiently and effectively will be of infinitely more interest and value to such users than anything the language can teach them about types.

But don't rely on my assumptions, your assumptions, or anyone else's. The only way to be sure is to go speak to them and learn from them. Get out on the shop floor and communicate with the people doing the actual work: they're your intended users, and the ones you need to please. Learn how they think, what their jobs are, and how they do them. And make sure you check all your own CS preconceptions at the door: while such knowledge will be a big help once you implement your design, it can only hinder (or cripple) the process of discovering that design in the first place.

Booleans only have one use: branching (ie. conditional switching). Your other two examples are just the interaction of branching with function abstraction

And you'd be correct from a purely technical POV: all Booleans ultimately end up consumed in the fiery depths of a JNZ.

OTOH, I'm coming at it from the users' perspective, and what various Boolean values might mean to them as they fly hither and yon before they reach that final destination.

Since my own end-user language designs are behavior-centric, my interest is what goes on at the user interfaces to those behaviors, because as interfaces - that is, function signatures and calls - are the one bit of a program that's always completely concrete and visible to them, that's what they'll hang their understanding on*.

A better phrasing for my question might be:

What explicit and/or implicit meaning[s] might a given Boolean value carry for 1. the user, 2. the function, at the point it meets the function (behavioral) interface?

followed by:

Of these use-cases, which really are best served by Booleans, and which should/might be better/alternately achieved by other means?

Which, as I say, is something that can only be answered from POV of "What does the end user want/expect/understand/need?" Language design is UI/UX/HCI design, so questions like "What makes the language designer's life simple and painless?" or [worse] "What does the hardware need?" may be indicators you're doing it wrong.;)

--

* i.e. Not fleeting magical "objects" that only machines pretend to see; nor endless arcane ontological quest into the true meaning of "type"**.

** (I'll get my coat...)