Lambda the Ultimate

activeTopic The SkyNet Virus: Why it is Unstoppable; How to Stop it
started 5/19/2004; 8:29:59 AM - last post 5/27/2004; 1:29:28 AM
Peter Van Roy - The SkyNet Virus: Why it is Unstoppable; How to Stop it  blueArrow
5/19/2004; 8:29:59 AM (reads: 11682, responses: 20)
The SkyNet Virus: Why it is Unstoppable; How to Stop it
In Terminator 3, the SkyNet AI, exploiting the fundamental failure of computer security, distributes itself globally and becomes invulnerable to destruction. It then destroys the world when it gains control of America's nuclear missiles. While this is a considerably more serious disaster than any wrought so far by cyber-crackers, cyber-terrorists, or cyber-warriors, the flaws that make SkyNet unstoppable are the same flaws that make crackers, terrorists, and warriors possible. The same fix that eliminates crackers can terminate the Terminators.

This presentation starts by examining in detail the fundamental flaw in computer security today--the ludicrously excessive authority granted to even silly programs like Barbie Fashion Designer. We go on to see how the Principle of Least Authority (also known as the Principle of Least Privilege), ubiquitously applied by bundling designation with authorization, can end the madness while simultaneously making the user interface to security simpler than it is today.

This talk touches on a very important topic, one which may make the crucial difference for our personal liberties in a future where almost all of our society's activities will be happening on a digital substrate. The thorough use of the Principle of Least Authority, which the talk explains, can give us security that is both usable and effective.

More technical information on the talk's contents can be found in the paper Paradigm Regained: Abstraction Mechanisms for Access Control.
Posted to general by Peter Van Roy on 5/19/04; 8:32:11 AM

Andris Birkmanis - Re: The SkyNet Virus: Why it is Unstoppable; How to Stop it  blueArrow
5/19/2004; 9:40:00 AM (reads: 728, responses: 2)
Ironically, the presentation is published over https, and the certificate of the site has expired half a year ago. MS IE is suggesting a security problem :-)

But seriously, both erights.org and cypherpunks.to are valuable resources of capability-related information.

If I were to construct a more or less serious language (or even a toy), I would make sure it's capability-tight.

Andrew, a challenge for you - prove formally whether your language enables capability security :-)

andrew cooke - Re: The SkyNet Virus: Why it is Unstoppable; How to Stop it  blueArrow
5/19/2004; 10:12:16 AM (reads: 722, responses: 0)
i'm pretty sure it doesn't (life is short, you know :o)

Dominic Fox - Re: The SkyNet Virus: Why it is Unstoppable; How to Stop it  blueArrow
5/19/2004; 1:28:54 PM (reads: 669, responses: 0)

So which languages do?

Paul Snively - Re: The SkyNet Virus: Why it is Unstoppable; How to Stop it  blueArrow
5/19/2004; 2:23:31 PM (reads: 650, responses: 1)
There's an excellent overview at <http://erights.org/elib/capability/ode/overview.html>. Briefly, a capability-based language:

  1. Makes it impossible to forge references to entities.
  2. Disallows mutable global variables.
  3. Follows object-oriented principles of encapsulation and polymorphism.
  4. Has either structural equality or rights amplification (e.g. Sealer/Unsealer pairs) as a primitive.

This means that a good many languages are almost capability-secure: Java, upon which E is based; Scheme; Smalltalk; Oz... It turns out that an abstract store and lexical scoping get you something like 80% of the way there.

As of this writing, the only languages I know that are consciously, deliberately aiming for capability security are E and Oz.

andrew cooke - Re: The SkyNet Virus: Why it is Unstoppable; How to Stop it  blueArrow
5/19/2004; 2:54:49 PM (reads: 647, responses: 0)
a challenge for you

ok, i looked at this anyway. kragen's intro is nice - http://www.canonical.org/%7Ekragen/3-sec-arch.html - and the emphasis on namespaces, rather than objects, is useful. i think it might be possible. now i must work...

Ethan Aubin - Re: The SkyNet Virus: Why it is Unstoppable; How to Stop it  blueArrow
5/20/2004; 6:58:11 AM (reads: 533, responses: 0)
You also might be interested in FlowCaml. It does not fit in your capabilities requirement list, but its still an attractive approach to security.

From the documentation:


"Flow Caml is an extension of the Objective Caml language with a type system tracing information flow. Its purpose is basically to allow to write real programs and to automatically check that they obey some security policy. In Flow Caml, usual ML types are annotated with security levels chosen in a suitable lattice. Each annotation gives an approximation of the information which the expression that it describes may convey. Because it has full type inference, the system verifies, without requiring source code annotations, that every information flow performed by the analyzed program is legal w.r.t. the security policy specified by the programmer."

Ehud, the site search is broken.

bryan rasmussen - Re: The SkyNet Virus: Why it is Unstoppable; How to Stop it  blueArrow
5/20/2004; 7:48:29 AM (reads: 526, responses: 0)
is ther a realtion between E and Eros: http://www.eros-os.org/project/novelty.html

hmm, yeah it seems that eros-os.org hosts the e-lang list http://www.eros-os.org/pipermail/e-lang/ which just makes me wonder, what is the connection. is eros written in E as would seem to be the most logical assumption? On Edit: answer would seem to be no, although there is a historical relationship not described yet: http://www.erights.org/history/eros.html http://www.erights.org/related.html

Paul Snively - Re: The SkyNet Virus: Why it is Unstoppable; How to Stop it  blueArrow
5/20/2004; 10:03:11 AM (reads: 496, responses: 0)
The capabilities community that I'm familiar with is quite small. Shap (EROS) and Mark Miller, Marc Stiegler, and many others (E) certainly collaborate a great deal, but they have fundamentally different goals in at least one sense: EROS is an OS with a very tiny TCB (Trusted Computing Base, read: "the part that you have no choice but to trust"). E is a programming language that makes it vastly easier to implement capability security than any other language extant (although Oz and, yes, FlowCaml are very exciting). There's historically been a plan to marry the two: to make E the scripting language for EROS. Initial efforts in that direction revolve around writing a simple E interpreter in C++, vs. the rather mature Java implementation that you can download and use today.

Lately, Shap has mothballed EROS for a variety of reasons, some technical, some not. He's currently engaged in discussions with developers of the L4 microkernel who are interested in crafting a secure system also. I think Shap and the L4 community see a great deal of value in starting with L4's extremely well-designed basis and leveraging Shap's experience in developing EROS (hmm, I don't mean to imply that EROS isn't extrenely well-designed too; it is, just along different dimensions than L4). It will be exciting to see what the future holds for capability-secure L4.

In the meantime, E is very nice. Play with CapDesk and the DARPA Browser and Den to see what I mean.

andrew cooke - Re: The SkyNet Virus: Why it is Unstoppable; How to Stop it  blueArrow
5/20/2004; 10:24:46 AM (reads: 501, responses: 0)
ok, i understand what capabilities are, but i'm not sure i've got all the bases covered with implementing them. the "myths" paper helps, but isn't exactly direct, and almost everything seems to assume objects.

is the following a reasonable set of features to support capabilities?

  • some way of expressing capabilities in the languages (object, name, token, whatever) that cannot be faked
  • support for trees of capabilities (when you give a capability to someone else you generally give them a child of your capability)
  • revoking nodes in the tree that you have capabilities for (which revokes all the child copies too)
  • a way of composing capabilities (combining them into a new capability, which must therefore be opaque, revokable, etc)
  • suitable library design (parameterise interfaces with capabilities, not strings, for example).
  • suitable support for saving capabilities (difficult).
  • suitable separation of "systems" - different threads (for example) should not be able to observe each other and so "see" capabilities unless explicitly allowed (this has nasty consequences for some meta-programming).

anything else? is there a reference list somewhere?

[edited to simplify a bit, and refutable -> revokable(!)]

Brian Hurt - Re: The SkyNet Virus: Why it is Unstoppable; How to Stop it  blueArrow
5/20/2004; 11:28:54 AM (reads: 481, responses: 0)
And now for something completely different...

One thing that always bugged me about the ending of that movie. OK, so where was skynet? Distributed across the dorms and office buildings, infecting the computers therein. So it started a nuclear war, which promptly destroyed all the dorms and office buildings, and the computeres therein. Um. All I could think was, given the usual (oxymoronic) state of Military Intelligence, I shouldn't be surprised Military Artificial Intelligence is not that much better off.

Seriously, the number one problem with increasing security is that there is absolutely *no* incentive on the parts of commercial software vendors to do so. Consider the story of two software products- product A is more secure (and robust), and product B ships sooner and has more features. Which product wins in the market place? Hint: how many people here are still running Microsoft? Make all the excuses you want, Microsoft is learning that being an object lesson in bad computer security doesn't hurt them (financially) in the least, and as such there is no reason for them to improve.

Until some form of product liability is instituted, and some software makers are sued out of existance for selling bad software, there will be no improvement in software security or reliability. In fact, I predict there will be ongoing degredation in both.

Such an opinion is generally greeted by shocked and dismayed exclamations of "are your INSANE?" I am resoundly assured that if software companies were held as liable for their products as automakers (which, I comment, doesn't mean you can't make mistakes, just that if you do, you have to fix them for free, and you have to make good faith efforts to make a good product in the first place), no one would be willing to write software. Why the same logic doesn't apply to cars (and every other consumer good imaginable) is never expounded on.

Of course, the computer industry isn't going to be who decides wether software makers are responsible for their products or not- society as a whole (or at least the legal system) is going to be making that call. The only question is when such legal liability will be enforced. Hopefully, the info-diaster that gives rise to that decision will be somewhat less catastrophic than accidentally kicking off global nuclear war.

Brian

Mark Evans - Re: The SkyNet Virus: Why it is Unstoppable; How to Stop it  blueArrow
5/20/2004; 2:15:12 PM (reads: 444, responses: 0)
Trusted Computing Group

Paul Snively - Re: The SkyNet Virus: Why it is Unstoppable; How to Stop it  blueArrow
5/20/2004; 3:55:50 PM (reads: 426, responses: 0)
Brian Hurt: Consider the story of two software products- product A is more secure (and robust), and product B ships sooner and has more features. Which product wins in the market place?

It's precisely this calculus that efforts like EROS and E are attempting to address. Everyone understands that right now, as things stand, developing anything remotely like robust, high-security software is too time-consuming, and therefore too costly. But what if your OS and programming languages made developing robust, secure software tantamount to just using lexical scoping well, or "objects" well? That is, what if the most obvious design of your software was also the most secure? That's the capability security perspective: it addresses the economic question head-on.

Mark Evans: Trusted Computing Group

Run, don't walk, in the opposite direction! "Trusted Computing," as Microsoft and the other members of the "Trusted Computing Group" define the term, has nothing to do with protecting the user of the device in question and everything to do with propping up content industries faced with the challenge of evolving their business models in the digital age and, in the limit, moving to a software-and-content-rental vs. ownership model. This snake oil has nothing whatsoever to do with capability security, as becomes immediately apparent when you read their literature and find all the references to the already-known-useless security infrastructure: VPNs, PKI, "trusted third parties," ad nauseam.

To a first approximation, anyone who talks about "trusted third parties" isn't talking about capability security. The whole point of capability security is to enable what Mark Miller evocatively describes as "patterns of cooperation without vulnerability." If I have to trust a third party to a transaction, by definition I'm more vulnerable than if I don't have to, so a great deal of work has been done to remove such a necessity. The ERights.org site has many excellent documents and references to learn more from.

Paul Snively - Re: The SkyNet Virus: Why it is Unstoppable; How to Stop it  blueArrow
5/20/2004; 4:00:23 PM (reads: 429, responses: 2)
andrew, I think your list is too long, actually. One of the great benefits of capabilities is that they're simple, local, and therefore easy to reason about. If objects aren't an effective launching-off point for your investigation, perhaps Jonathan Rees' A Security Kernel Based on the Lambda Calculus will prove more useful to you.

Paul Snively - Re: The SkyNet Virus: Why it is Unstoppable; How to Stop it  blueArrow
5/20/2004; 4:08:40 PM (reads: 427, responses: 0)
I'm also remiss in failing to point out that Concepts, Techniques, and Models of Computer Programming treats capability security explicitly using a combination of Oz's unforgeable "Name" construct and lexical scoping, e.g. to model Sealer/Unsealer pairs. The book is a world treasure anyway, reunifying Computer Science in the same sense that Artificial Intelligence: A Modern Approach" has reunified Artificial Intelligence, so you owe it to yourself to buy the book even if you never download the Mozart programming system and work through the exercises. You should, of course, do that, too. If you do, many aspects of capability security will be clearer, as will a couple dozen other programming concepts.

Paul Snively - Re: The SkyNet Virus: Why it is Unstoppable; How to Stop it  blueArrow
5/20/2004; 4:20:22 PM (reads: 430, responses: 0)
I'm also remiss in failing to point out that Concepts, Techniques, and Models of Computer Programming treats capability security explicitly using a combination of Oz's unforgeable "Name" construct and lexical scoping, e.g. to model Sealer/Unsealer pairs. The book is a world treasure anyway, reunifying Computer Science in the same sense that Artificial Intelligence: A Modern Approach" has reunified Artificial Intelligence, so you owe it to yourself to buy the book even if you never download the Mozart programming system and work through the exercises. You should, of course, do that, too. If you do, many aspects of capability security will be clearer, as will a couple dozen other programming concepts.

Andris Birkmanis - Re: The SkyNet Virus: Why it is Unstoppable; How to Stop it  blueArrow
5/20/2004; 10:48:30 PM (reads: 387, responses: 0)
Jonathan Rees' A Security Kernel Based on the Lambda Calculus

Wow, thanks, the text is great!

Besides many other useful things it answers the question we discussed with Andrew recently about what is capability-unsafe in Scheme. It shows how to deal with define's and set!'s... One wonders, how did we miss the memo.

andrew cooke - Re: The SkyNet Virus: Why it is Unstoppable; How to Stop it  blueArrow
5/21/2004; 6:24:23 AM (reads: 332, responses: 0)
thanks - i'll check that.

Brian Hurt - Re: The SkyNet Virus: Why it is Unstoppable; How to Stop it  blueArrow
5/24/2004; 2:20:09 PM (reads: 127, responses: 0)
I said: Consider the story of two software products- product A is more secure (and robust), and product B ships sooner and has more features. Which product wins in the market place?

Paul Snively: It's precisely this calculus that efforts like EROS and E are attempting to address. Everyone understands that right now, as things stand, developing anything remotely like robust, high-security software is too time-consuming, and therefore too costly. But what if your OS and programming languages made developing robust, secure software tantamount to just using lexical scoping well, or "objects" well? That is, what if the most obvious design of your software was also the most secure? That's the capability security perspective: it addresses the economic question head-on.

My response:

Personally, I think the situation is worse than you can possibly imagine. I think if the IDE came with a check box button labeled "make this program secure", at least half the programs out there wouldn't click on it. Why should they? What do they get out of it? Nothing- currently, security has no commercial value. You don't lose any sales if you don't have it, you don't gain any sales if you do have it. Therefor, it's not worth the effor to click on a checkbox. Any major changes- like requiring programmers to think differently, change their habits, or even just learn a new programming language- are right out.

What needs to happen is for there to be a commercial value to security. There needs to be some upside, or at least the lack of some downside, to writting secure software. Some reason to not only click the checkbox, but learn the new language and the new way of thinking about software. Of course, we programmers are opposed to this. And especially the businesses and managers that employee us are opposed to this. It's much easier to beat our chests and screech than it is to think.

bryan rasmussen - Re: The SkyNet Virus: Why it is Unstoppable; How to Stop it  blueArrow
5/27/2004; 1:29:28 AM (reads: 35, responses: 0)
'What do they get out of it? Nothing- currently, security has no commercial value.' Security has no commercial value for anyone. One specific area in which I can see this as being important is in hmm, government systems. Specifically ones dealing with defense. This might explain Darpa's investment in these projects. Also, one of the reasons given for many governments moving from Windows to Linux is security of the system. I agree that there is no commercial value in selling a product with added security to everyone, it's a boutique item, in which security is in fact the primary feature.