Some spaghetti are less tangled than others

They say if you can’t explain it to a six year old, you don’t understand it yourself. Supposedly it was Einstein who said that. Though he also said he didn’t say half of what they say he did. Anyway, how would one explain to a six year old delimited continuations implemented in terms of the standard call/cc?

Respectable scholarly expositions typically relate to the underlying CPS-transformed structure to explain the logic. The metacontinuation is a fine concept, but explaining interaction of obscure concepts in terms of an obscure metaconcept has its drawbacks.

I am going to show a two-layered (three if you count the syntactic sugar) implementation of delimited continuations aimed not at affinity with theoretical models, or efficiency, but at making clear the mechanics of what’s really happening. It’s the result of refactoring what originally was very similar to the implementation in the paper.

We start with “stackable labels”, a helper control mechanism with two primitives: label! and jump!. The latter may sound like the good old goto, though as we’ll see it actually has more in common with return.

The label! expression, shown in this example as part of a larger expression

(+ (label! (foo)) (bar))

evaluates its argument (foo) and returns the result, so the whole expression is almost the same as simply

(+ (foo) (bar))

The difference is that as a side effect label! remembers the spot in the evaluation where the result of its invocation was expected. We call this spot a label (a noun; label! is meant as a verb). Note that I say it’s a spot in the evaluation and not in the code. A label! expression may be evaluated multiple times, and the label it creates each time corresponds to that specific evaluation of label!. If this sounds vague, the examples below will clarify the difference. (And if this sounds like continuations, it’s because that’s what labels are below the surface–but labels! and jump! are there to hide and manage the underlying continuations, so we will avoid peeling back this abstraction).

A label created by an evaluation of label! is pushed onto an implicit label stack. A jump! expression such as

(jump! 42)

removes the top label from the stack and jumps to it. Jumping has the effect of the corresponding label! invocation returning with the value passed to jump!. The jump! expression itself never returns.

Here are two examples to build up some intuition about labels and jumps. They use Racket’s non-standard but obvious printf function. First the basic mechanics:

> (printf "got: ~s~n" (label! (+ 3 4)))
got: 7

The result of (+ 3 4) got through to printf as expected, though as a side effect we’ve also saved a label:

> labels

We can now jump to that label:

> (jump! "hello")
got: "hello"

which has the effect of returning from label! again and proceeding with printf, this time with "hello" as the value to print. The jump also consumed the label and the label stack is now empty:

> labels

The second example is just a slight variation on the first, but the interaction of two labels! expressions makes it quite a bit more interesting. It drives home the point about a label marking a spot in the evaluation and not just in the code.

> (printf "first: ~s second: ~s~n" (label! 1) (label! 2))
first: 1 second: 2
> labels
'(#<continuation> #<continuation>)

So far this is very much like the first example, except we’ve evaluated two label! expressions and pushed two labels onto the stack.

> (jump! 3)
first: 1 second: 3
> labels

The last label pushed was the one for (label! 2), so jumping to it proceeded with printf, this time with 3 as the second value to print. At this point the first value was already ready to go, so 1 and 3 were printed. We’ve also consumed one of the labels and are left with the other one which marks the return of the result of (label! 1). This is where it gets more interesting.

> (jump! 4)
first: 4 second: 2
> labels

Now we supplied 4 to the point where the first value to print was expected. However, at that point the second value was not yet available, so (label! 2) was evaluated again, producing 2. It also pushed a new label replacing the one the jump! consumed, so we still have one label on the stack and can jump again:

> (jump! 5)
first: 4 second: 5
> labels

Now the jump emerged from (label! 2) evaluated in the previous step, with 4 supplied by the previous jump ready to go as the first value to print. Now we get 4 and 5 and have finally consumed all the labels.

The implementation of label! and jump! is pretty much a direct transcription of what we said they do. There is a stack of labels (continuations), label! pushes its own continuation onto it, and jump! pops the stack and passes its argument to the popped continuation.

(define labels '())

(define-syntax label!
  (syntax-rules ()
    ((push-label! expr)
       (lambda (k)
         (set! labels (cons k labels))

(define (jump! v)
  (let ((k (car labels)))
    (set! labels (cdr labels))
    (k v)))

The explicit stack of continuations in labels directly corresponds to *meta-continuation* of the scholarly presentation and the structure of nested closures it contains. While less glamorous, the explicit stack is easier to follow and inspect.

Now, armed with stackable labels, the implementation of reset and shift (or rather their core as higher-order functions; syntactic sugar is unimportant) is trivial.

(define (reset/0 body)
  (label! (jump! (body))))

(define (shift/0 body)
    (lambda (k)
      (jump! (body (escaper k))))))

(define (escaper k)
  (lambda (v)
    (label! (k v))))

OK, maybe it’s not completely trivial. But it’s reduced to the bare essentials, and with control transfers abstracted by stackable labels the structure is much easier to explain. Especially if we do that in the following not very precise but very helpful imperative terms.

Stackable labels can be seen as an auxiliary call stack. The state of evaluation saved on it using label! stays available for returning into regardless of the “regular” calls and returns and their stack. The jump! operator is a “special return” for returning into the saved state.

In the beginning, reset/0 enters a “special call” because it’s entire body is wrapped in label!. Evaluation of both reset and shift bodies ends with a “special return”. If a shift never invokes its escape function, as in either of the fragments

(reset (+ 3 4))            ; => 7
(reset (+ 3 (shift k 4)))  ; => 4

the auxiliary stack only has an entry corresponding to reset. So, the bodies of both shift and reset return out of reset.

If a shift does invoke its escaper function, as in

(reset (* 2 (shift k (+ (k 3) (k 4))))  ; => 14

the escaper pushes a new entry onto the auxiliary stack. It then returns its argument from the shift expression by invoking the continuation k captured by shift/0 (not part of the stackable labels scheme). Now the reset body is proceeding with that as the value of the shift expression. When the reset body completes, it “special returns” to the escaper invocation on top of the stack and not out of reset as it would do otherwise.

These are the two base cases. More complex structures, such as

  (display (shift k (k "a") (k "b")))
  (display (shift k (k 1) (k 2))))  ; prints "a12b12"

are simply a combination of these two scenarios.

Could a six year old understand this? Mine couldn’t. But in any case, refactoring continuations-based code builds character and makes one a better person. I think it was Einstein who said that.

The full source with sugar and examples

Elements of Goodthink: Setter Sends

Smalltalkers are in a pretty good position to understand Newspeak code out of the box. Unfamiliar constructs such as nested classes and the related message lookup semantics need some experience to internalize, but they are usually only an obstacle to understanding the higher-level design. Within a method everything should look familiar, except perhaps for setter sends.

A setter send is an expression like

foo:: a doSomethingWith: b

For a Smalltalker, the short explanation would be to say it’s just like

foo := a doSomethingWith: b

Like most short explanations, it’s a useful lie. Typically when you see something like the above, there is indeed a variable or a slot named “foo” somewhere in the scope, and the expression sets the value of that slot. But it doesn’t have to always be the case.

One half of the true story is that the expression is more like

foo: (a doSomethingWith: b)

It is a message send to an implicit receiver, and the message is a plain keyword message “foo:” with a single colon. Doubling the colon makes it get parsed at a lower precedence so we don’t have to parenthesize the argument (without parentheses that would be an implicit receiver send of “foo:doSomethingWith:”).

Remember that there is no assignment operator in Newspeak. To set a slot, you invoke its setter method using an expression like the above. Without this syntax, most of those would need parentheses. We set slots quite often, so that would be a lot of parentheses. This is why the syntax is called a setter send, even though strictly speaking it does not have to invoke a setter of a slot. It’s a regular message send going through the regular message lookup procedure.

The second half of the story has to do with the value returned by the send. The value of

foo: bar

is the value returned by “foo:” The value of

foo:: bar

is “bar”. The value returned by “foo:” is thrown away. So after all, while a setter send is a regular message send as far as message lookup goes, its value is different from a regular message send. This is to recognize the fact that a setter send is usually used in place of an a assignment. Assignments can be chained, and this behavior guarantees that a chain of setter sends

foo:: bar:: baz

passes the same value to “foo:” regardless of now “bar:” is implemented.

To condense all of the above into a true short story,

foo:: a doSomethingWith: b

is just like

[:v | foo: v. v] value: (a doSomethingWith: b)

Hopscotch presentation video from

Gilad gave a talk about Hopscotch last week at 09. In his blog post about it he modestly speaks in detail only about the talks of others, but since his is the first properly recorded demo of Hopscotch available online, I want to specially mention it here. It features the latest and the greatest version of the tools, so it’s even better than what could be seen at Smalltalk Solutions, WASDeTT and ECOOP last year.

Update: A non-Silverlight (WMV) version is available here.

A Taste of Nested Classes, part 4

(Continues from the intermission).

In the previous examples I sometimes referred to the familiar Smalltalk library classes such as OrderedCollection as if it were the most natural thing to do. But what does something like

OrderedCollection with: 'foo'

really mean when found in a Newspeak method? It’s clear that with: is sent to OrderedCollection, but what is OrderedCollection?

OrderedCollection is, of course, a message. It’s sent to an implicit receiver, meaning we first look for a matching method in the lexical scope and send the message to the corresponding outer object if we find it. If not, we send the message to self and do the standard Smalltalk message lookup. It’s reasonable (and correct) to assume that Object and other superclasses don’t define a method named OrderedCollection, and that second lookup is bound to fail. Which means that in order for the above to work, one of the outer classes has to implement OrderedCollection as a method doing the right thing.

Let’s pause here and summarize the important points.

First, there are no variable references in Newspeak. Any expression that begins with foo or Foo begins its evaluation by sending that message to an implicit receiver. This applies to anything that looks like a global variable, an instance variable, and even (at least conceptually) a temporary variable in the traditional Smalltalk understanding. Even pseudo-variables like self or true, as we’ve seen in the intermission, could be implemented as message sends.

Second, because there are no variable references, there can’t be such a thing as the global scope. We can’t just say OrderedCollection anywhere in the code and expect to get one. But if that is so, how can we use library classes at all?

Let’s start by fleshing out a more concrete example.

class AsteroidsGame = ( )
    class Screen = ( 
        | asteroids = OrderedCollection new |
    ) (

Here we want a slot of the nested class to hold an instance of OrderedCollection. Initializer code is no different from method code, so OrderedCollection here is an implicit receiver send. Someone on the outside has to understand that message and nobody does in this example, so it can’t possibly work. Let’s try modifying it like this:

class AsteroidsGame usingOrderedCollection: orderedCollectionClass = (
    | OrderedCollection = orderedCollectionClass |
) (
    class Screen = (
        | asteroids = OrderedCollection new |
    ) (

This is much better because it can actually work. The top-level class is now instantiated with the usingOrderedCollection: message. The creator is expected to pass an OrderedCollection class metaobject to the instance. (For now let’s not worry where the creator gets it). The game instance stores the metaobject in a slot named OrderedCollection. Because defining a slot automatically defines accessors (in this case only a getter because of the “=” syntax which defines read-only slots), the AsteroidsGame instance now understands the OrderedCollection message. The implicit receiver send of OrderedCollection in the Screen class initializer will now be directed to the outer instance of AsteroidsGame and return the OrderedCollection class.

This example isn’t final yet, but it shows something very important. The top-level AsteroidsGame class is actually a module! It holds together the classes nested inside, and acts as a namespace for their names. But most importantly, it controls their implementation dependencies. The only way a class inside a module can use something from the outside is if that something has been explicitly declared as a requirement by passing it into the module and storing it in a slot.

Of course, passing each class that a module needs as a separate argument to its initializer would be incredibly tedious. Here is the final iteration of our example, this time showing how it’s done for real:

class AsteroidsGame usingPlatform: platform = (
    private OrderedCollection = platform Collections OrderedCollection.
    private ReadStream = platform Streams ReadStream.
    private PlatformScreen = platform Graphics Screen. 
) (
    class Screen = ( 
        | asteroids = OrderedCollection new |
    ) (

Instead of passing each dependency separately, we pass in an object we call the platform. A platform is a “supermodule”—an object that holds together a group of modules and makes them available through messages like Collections. The collections module returns the OrderedCollection implementation in response to the OrderedCollection message. The slot initializers in AsteroidsGame are now essentially “import statements,” binding the required classes to names local inside the module.

As an aside, this is reminiscent of the idiom one occasionally sees in Scheme, where values of some names are bound to the same names but local to a closure:

(let ((car car)
      (cdr cdr))

to capture the known “standard” values and ensure they are used regardless of the possible reassignments of the outer variables. The precise motivation is different but the mechanism is similar, especially if the let is expanded into the equivalent function call.

Another change we introduced is the private modifier on the imported slots. This means that the corresponding accessors are only available via messages sent from the same object or from the instances nested inside. (Though this is not enforced in the current Newspeak prototype). That is a good, and I’d even say required, style in this particular case because it insulates the dependencies of this module. Without that anyone could rely on retrieving the OrderedCollection class from an instance of AsteroidsGame, creating an undeclared and uncontrolled “derivative dependency.”

Finally, the last import illustrates how we can locally rename an imported value to avoid a conflict with a name used inside the module.

What about star imports: just grabbing everything from another module to avoid the hassle of listing every dependency we need by hand? That’s exactly the problem with star imports: they allow using a name without declaring that name as a dependency. It’s a misguided convenience that defeats the very modularity mechanism it is a part of. Even in languages that support star imports, good programming practices dictate avoiding them. In Newspeak they are simply not supported.

And finally, how do we actually use modules? If “actually” refers to the current prototype of Newspeak hosted in Squeak, the procedure I am about to describe is not the final idea.

In the prototype, a top-level Newspeak class resides in the Smalltalk system dictionary. Because message sends in Smalltalk and Newspeak mean the same thing, it is possible for Smalltalk code to communicate with Newspeak. We can bootstrap a Newspeak module by writing code like the following in a Smalltalk workspace (or in a menu item implementation method):

module := AsteroidsGame usingPlatform: Platform new.
gameScreen := module Screen new.
gameScreen openWindow.

The first line instantiates the module, the second instantiates the module’s Screen class, the third tells the instance to open a window. Platform is a special class included in the prototype to give the Newspeak world access to what’s available in the surrounding Smalltalk world. When Newspeak code executes

platform Collections OrderedCollection.

the platform’s doesNotUnderstand: method kicks in and retrives the standard Smalltalk OrderedCollection, making sure it is indeed part of the Collections package. Again, this is a temporary bootstrapping hack and not the final idea—though even as a hack it illustrates the power of modularity based on late-bound message sends.

For the final idea, see Gilad’s post about living without global namespaces. Combined with the ability to serialize and deserialize objects (and module instances are objects too), this ability to mix and link different modules has many exciting uses. Here are just a couple of scenarios.

You have an application that relies on the platform version 3.0. However, you’d also want to use another application written back in the times of version 2.1. Imagining we have an object responsible for module management called (say) CentralServices, launching both applications at the same time is as easy as

platform30:: CentralServices platformVersion: '3.0'.
platform21:: CentralServices platformVersion: '2.1'.
(AnApplication usingPlatform: platform30) MainWindow new open.
(AnotherApplication usingPlatform: platform21) MainWindow new open.

We could just as easily try and plug platform30 into AnotherApplication to see if it can in fact run with the latest version of the platform, and we could do that without even shutting down the already running AnotherApplication linked to the older version.

This leads to another scenario which as a tools developer I’m really excited about. Working on tools in a Smalltalk-like system is great because of the instant feedback, but also dangerous because you can break the very tools you need to fix the breakage. An idea has been floating in the Smalltalk world since (at least) mid-90s to separate the developer and the developed images. You’d have your tools running in one image, working remotely on the objects living in another image. The technology is there, and there have been various prototypes and implementations, but strangely none of those gained momentum. In Newspeak the solution is much simpler. Similar to creating a new instance of AnotherApplication module to try it with the new version of the platform while using the original, we can have two tools module instances at the same time, one running the tools we are using and the other representing the tools we are working on.

This shows an interesting difference between the roles images play in Smalltalk and in Newspeak. In Smalltalk an image is a manifestation of a specific platform version and configuration. We say things like “I tried running this in a 3.1 image with Foo loaded.” In Newspeak an image is only an object universe that can host a mix of applications and libraries of any imaginable version and configuration.

I’m feeling lucky

Just a few months ago I never would have thought I’d work on anything other than Newspeak for the foreseeable future. Working with Gilad, Peter, and others was a fantastic experience, and building a programming platform almost from scratch without political and marketing baggage attached was not a job you find every day on But it turns out sometimes the foreseeable future runs out sooner than you expect.

The end of full-time Newspeak work is a sad thing, but I’m not writing this to cry over spilt milk. They say new beginnings bring new opportunities, and this is how I want to look at it. I have just finished my first week working for Google.

A Taste of Nested Classes, intermission

(Continues from part 3).

This post is an intermission because it’s not specifically about nested classes. In fact, it’s not about anything that really exists in Newspeak. Instead, it reviews a few hypothetical examples to show off some of the expressive capabilities of implicit receiver sends.

Most expressions in Newspeak begin with an implicit receiver send, at least conceptually. In something like

foo size


OrderedCollection new

both foo and OrderedCollection really are message sends, for some definition of “really”.

Of course, in some cases we can’t do without real receivers. There are literals, as well as traditional pseudo-variables (“reserved words”), whose interpretation should be hard-coded into the language.

Or should it be?

Starting with literals, imagine that every occurrence of a literal is compiled as an implicit receiver send identifying the type of the literal, with the “primitive” value of the literal as the argument. For example,

'Hello, brave new world!'

would really mean

(literalString: 'Hello, brave new world!')

with literalString: implemented in Object as

literalString: primitiveValue <String> = (

At this point this is only a possibility, but one that is explicitly mentioned in the language spec (5.1). If implemented, this would allow user code to redefine the interpretation of what appears to be literals for greater flexibility when implementing internal DSLs.

As for pseudo-variables, consider this implementation of the class Object:

class Object = (

    self = (
        "empty - return the receiver"

    true = (
        ^True soleInstance

    false = (
        ^False soleInstance

    nil = (
        ^UndefinedObject soleInstance

    thisContext = (
        ^ActivationMirror forSenderContext

With this, most of the reserved words of Smalltalk become regular messages. The expression

^self foo: nil

now really means

^σ self foo: σ nil

with σ denoting a reference to (usually!) the receiver unrepresentable in the language syntax. We use the fact that an empty method effectively reads as


to grab that elusive σ in the self method.

super is more complicated because it doesn’t denote a separate object but rather self with a special mode of message lookup. However, even that can be lifted from language magic into a reflective implementation at the library level:

super = (
    ^SuperDispatcher forReceiver: self

with SuperDispatcher implementing doesNotUndestand: to reflectively look for the method implementing the message starting from the superclass of the implementor of the sender context’s method. (A mouthful, but that really is what it should do).

In a similar way, outer, the new reserved word of Newspeak not found in Smalltalk could be implemented as a message send handled by reflective code at the image level.

The last two examples, super and outer, are even more hypothetical than the other pseudo-variables. While technically possible, a reflective implementation of those is quite expensive compared to the usual “language magic” solution.

Also, this scheme in general leaves the meaning of things like self or nil open to changes by a class for all its subclasses and nested classes. Perhaps there is such a thing as too much flexibility. But even if these ideas are too much or too expensive for the core language, all of the same mechanisms are available to a designer of an internal DSL, and that’s what makes them interesting. For example, the “dispatcher receiver” pattern illustrated with super and outer was conceived in Hopscotch and used there to support NewtonScript-like object hierarchy-bound message dispatch.

(Continues in part 4).

A Taste of Nested Classes, part 3

(Continues from part 2).

After an unplanned hiatus caused by various real life issues including the irritating surprise of becoming unemployed in these interesting times, here is the long-promised third part of the nested classes series.

Class nesting in our earlier example represented the natural nesting relationship of an asteroid object inside the game that contained it. Implicit receiver sends were a convenient mechanism of communicating with the game from inside an asteroid instance “for free”, without the need to maintain an explicit object reference. Today we look at a few examples with implicit sends of a slightly different flavor.

In a typical Smalltalk system, methods in the Object class define the behavior common to all objects. Some of them, such as class or identityHash are implemented to return some interesting information extracted from the receiver. Others, such as isNil are overridden in subclasses to vary their behavior polymorphically, again depending on the receiver object. In both of these cases, the receiver is involved one way or the other.

There is a third kind, methods like halt, error:, or flag: (in Squeak). They don’t extract any information from the receiver. Also, they are not overridden, so the implementation in Object tells the full story of their behavior. We send them to self and expect that they behave the same regardless of what self happens to be. As far as these messages are concerned, the receiver itself is unimportant.

These messages are utilities representing “ambient” functionality—something potentially useful everywhere in the code. We send them to self because it’s the most readily available receiver, and they are implemented in Object so that the sends work regardless of what self happens to be. Object in this case doubles as the container of these utility functions and inheritance makes them globally visible.

This global visibility is what also makes such utilities potentially problematic. Suppose I want to have a logError: utility that writes an entry into a log file, and I want to use it throughout my application. It’s fine to have it in Object only as long as another application doesn’t decide to use the same message but log the error to the system console instead.

Here is an alternative way to implement such utilities in Newspeak.

class AsteroidsGame = (...)
    class Space = (...)
        moveObject: anObject to: position = (
            logError: 'invalid position'

    class Asteroid = GameObject (...)
        respondToHit = (
            isLive ifFalse: [logError: 'invalid asteroid'].

    logError: message = (
        errorStream nextPutAll: message; cr.

We put logError: in the top-level class representing our application. Any method of any nested class can invoke that utility through an implicit receiver send. The superclass of the nested class doesn’t matter—the utility is accessible because the class is nested in the one that provides it. Compared to the Smalltalk example, the utility is now available everywhere in our application and nowhere outside of it.

Here is another thing that’s interesting about the Newspeak alternative. In Smalltalk, logError: would have to be an extension method of Object. Extension methods are officially ungood in Newspeak, and the example shows the goodthink alternative.

In fact, such use of class nesting as a scoping mechanism for utilities solves an entire class of data conversion problems traditionally solved by extensions. Say, we want to convert an integer to a string with its representation as a Roman numeral. A common solution would be to add an asRomanNumeral extension method to Integer to do the conversion. This is convenient because we can write x asRomanNumeral anywhere, but has the disadvantage of the function leaking into the rest of the system. A more controlled alternative of implementing a convertToRomanNumeral: method in one of the application classes has the disadvantage of being easily accessible only in the implementing class and its subclasses. In other places we need to arrange for a reference to an instance that understands the message. In Newspeak, convertToRomanNumeral: can be a method of the application (module) so that it’s easily accessible, but only within the module.

Here is an interesting hypothetical example that continues the same theme. Suppose we implement the following method in a top-level class:

if: condition <Boolean> then: then <Block> else: else <Block> = (
    ^condition ifTrue: then ifFalse: else

Anywhere in the class and its nested classes we can now write conditionals in this more familiar to the general public style:

if: a < b
then: [^a]
else: [error: 'invalid a']

This example is hypothetical because it could only be fast enough on a very sophisticated VM with a good adaptive optimizer. But as far as less common and therefore less time-critical control messages go, they are no different from other utilities. We can define them in the top-level application class and use them throughout the application. For example:

^return: 5 if: #bar notUnderstoodIn: [foo bar]

defined in a top-level class as

return: value <Object> if: selector <Symbol> notUnderstoodIn: block <[]> = (
        on: MessageNotUnderstood
        do: [:ex |
            ex message selector = selector
                ifTrue: [ex return: value]
                ifFalse: [ex pass]]

To sum up, some methods define the kind of ambient functionality that is used throughout an application. For that reason it’s best if it can be accessed implicitly, without arranging for an explicit reference to the object that provides it. In classic Smalltalk, the only mechanism to support that is inheritance, with the disadvantage that the functionality defined this way escapes the defining module. Nested classes and implicit receiver sends in Newspeak provide an alternative mechanism that keeps such ambient operations contained inside the module.

(Continues in intermission).

A Taste of Nested Classes, part 2

(Continues from part 1)

Let’s now look at some message sends. Suppose we design the application so that the message respondToHit is sent to an asteroid when it’s hit by a missile. In Smalltalk, a typical response would look something like

        replace: self with: self fragments;

game here is an instance variable holding a reference to the Game instance the asteroid belongs to, presumably initialized at the time the asteroid was created. How could we port this to Newspeak? Translating everything literally, we would define a slot named “game” in the Asteroid class and port its initialization logic. We would then port all the methods. The original respondToHit would remain unchanged, apart from the overall “packaging”:

respondToHit = (
        replace: self with: self fragments;

But is it all the same as before? Not quite, because just like in Self, Newspeak slots are accessible only through messages. What looks like a good old game variable reference has now become a send of the message game to (an implicit) self.

Let’s hold this thought and backtrack to the end of the previous post. We said that an asteroid always knows the game instance it belongs to, it’s the enclosing object. We don’t need the game slot we ported from the Smalltalk original because it holds onto the same thing!

We can drop it and any of its initialization logic because the language now keeps track of the enclosing game instance for us. Instead suppose we define this method in Asteroid:

game = (
    "Get and return the enclosing object. To avoid getting ahead
    of the presentation, we don't show how it's done yet."

As for respondToHit and any other methods, they are unchanged—a perfect example of the benefits of representation independence. When game is just a message send, nobody cares how it gets its result.

Now, instead of revealing how the method game is implemented, we are going to do something even better. We are going to get rid of it altogether.

Time to talk about implicit receiver sends. I’ve hinted more than once that an implicit receiver is not always self. If it’s not self, what else can it be? Is there any other object that is somehow implicitly “present” at the location where the message is sent? Of course, it’s the enclosing object. (Or objects, if there is more than one layer of nested classes). Think of it this way: the respondToHit method is contained not only in the Asteroid class, but also indirectly in the Game class. So, the code in the method runs not only in the context of the current receiver, an instance of Asteroid, but also in the context of the enclosing instance of Game.

Without further ado I’m going to show respondToHit rewritten to take advantage of implicit receiver sends, and then describe their exact behavior. This will also explain under what specific conditions such a rewrite would be possible.

respondToHit = (
    replace: self with: fragments.

This reads almost like plain English. replace:with: and incrementScore are now sent to the enclosing game via the implicit receiver mechanism. Note that fragments is also changed to have an implicit receiver, though in this case we expect that the receiver is self. How does it happen that in both cases the actual receiver is what we want it to be?

Let’s recap how these potential receivers are related. We have essentially a queue of them: first self, then its enclosing object. (With more than two levels of nested classes, it would be followed by the enclosing object of the enclosing object and so on). If we represent the situation as a picture of the classes of the objects involved and their inheritance, we get this comb-like structure (to make the picture a little more interesting, Asteroid here is subclassed from a hypothetical ScreenObject, even though our original code didn’t say that).

Classes potentially involved in message processing

Because a method with a matching selector might be defined by any class in the comb, it may seem that the most intuitively reasonable lookup strategy is the one adopted by NewtonScript. The policy there was to search the entire comb starting with the receiver and its superclasses, then the receiver’s enclosing object (called parent in NewtonScript) with its superclasses, and so on. In the example we would look first in Asteroid, ScreenObject and Object with the intent of the Asteroid instance being the receiver, then Game and Object with the intent of the Game instance being the receiver. The send would fail if none of the classes in the comb implemented a method to handle the message.

I emphasized “seem” because the strategy in Newspeak is different, for the reason I am outlining in the end of the post. But first, here is how it really works:

  • If one of the classes on the “trunk” of the comb implements a method with a matching selector—in other words, if a matching method is lexically visible at the send site—the message is sent to the instance of that class.
  • Otherwise, the message is sent to self.

Even more informally, we could say that when by “looking around and up,” “around” meaning the same class definition and “up”—the definitions it’s nested in, we can see one with a matching method, the instance of that class is what receives the message. Otherwise, it is sent to self (and results in an MNU if self does not understand it).

So, the lookup in Newspeak differs from NewtonScript-like approach in two important ways. First, it doesn’t traverse the entire comb. The only potential handlers of the message are the methods defined in the trunk classes and those in the superclasses of the class of self. Second, lexical visibility takes priority over inheritance. If a method of Asteroid sends the message redraw to an implicit receiver and a method named redraw is defined in both Game and ScreenObject, the ScreenObject implementation wins in NewtonScript, while the one in Game wins in Newspeak.

This explains how the correct receiver is chosen in our implementation of respondToHit. As long as replace:with: and incrementScore are defined in Game, they are sent to the enclosing game instance as lexically visible. As for fragments, it’s sent to the asteroid no matter whether it’s defined in Asteroid or inherited. If Asteroid defines it, it’s lexically visible and the asteroid receives it according to the lexical visibility rule. If ScreenObject defines it (and assuming Game doesn’t), it’s lexically invisible and the asteroid receives it because the asteroid is self.

What if Game also had a method named fragments? In that case the definition in Game would win, so in order to get the fragments of the asteroid we would need to write the send explicitly as self fragments. What if replace:with: is inherited by Game from the superclass? In that case we also need to make the receiver explicit, because otherwise it would go to self:

game replace: self with: fragments.

This brings us back to the problem of writing a method named game returning the enclosing instance of Game. Now we know enough to come up with a very simple solution: we add the method to the class Game (not Asteroid), defined as

game = (

So, now that we know how it all works—why does it work this way? Why bring lexical scoping of messages into the picture? Is it Rightâ„¢ to give priority to methods of another object over inherited methods of the receiver? Why not just follow the NewtonScript example—it appears simple and reasonable enough.

The thing is that compared to NewtonScript, class nesting in Newspeak is intended to solve a very different problem.

NewtonScript is prototype-based, and its object nesting is motivated by the need to represent the nesting of UI elements and support communication within that structure. The nesting and the associated message processing is essentially a built-in Chain of Responsibility pattern, very common in the UI field. (As an aside, Hopscotch implements NewtonScript-like mechanism for sending a message up the chain of nested UI elements. This is implemented as a simple library facility with no more language magic than doesNotUnderstand:. Implicit receivers further help make this very unobtrusive).

In Newspeak, even though the “free” back pointer from Asteroid to Game in our example was nice to have, having it was not the primary reason for nesting one class in the other. In fact, class nesting is a rather unwieldy mechanism for implementing arbitrary parent-child relationships—imagine having to define a group of classes for the scenario of “buttons and list boxes in a window,” and an entirely different one for “buttons and list boxes in a tab control in a window”!

Newspeak-like class nesting and method lookup are designed to handle relationships that have more to do with the modular structure of code, and with the organization of information sharing within a module. A is nested in B when B is a larger-scale component as far as the system architecture is concerned. Even though such nesting does effectively create a parent-child relationship between Bs and As, there are parent-child relationships that do not justify nesting.

In nested Newspeak classes, a child calls onto the functionality provided by the parent not as a fallback case in chain-of-responsibility processing, but as a way of interacting with its architectural context. The lookup policy supports that and at the same time allows to modularize the context. That is the motivation behind putting lexical visibility before inheritance.

If this sounds somewhat vague, I hope some of the more realistic examples in the next part will help clarify this point.

Continues in part 3.