How to Learn to Stop Worrying and Love Implicit Receivers

I remember a thread on squeak-dev sometime in summer 2007 discussing implicit self as a hypothetical feature for Smalltalk. Back then we were not yet talking about Newspeak, and so I held my peace. But not forever—Andres recently wrote a blog post about the same thing, with a good summary of most of the same points I remember from the thread. This time Newspeak is known out there and explicitly mentioned in the post, so it’s a good chance to offer an alternative view and correct some misconceptions.

Here is the most important one. Quoting from Andres’ post:

I find that the consistency offered by a few keystrokes makes it easier for me to read and understand code faster and more accurately. Therefore, since we read code much more often than we write it, I think that favoring reading speed over typing speed is the right decision to make.

As far Newspeak is concerned, what we have is not “implicit self”, and its purpose is not saving keystrokes. What Newspeak has are implicit receivers. Because of class nesting, a message with an implicit receiver may really be sent to an object different from the “real” self (the receiver of the current message). This feature is very important in supporting the minimalist module system of Newspeak. Thus, an implicit receiver is not simply an omitted self, and inserting “self” into a message send with an implicit receiver is not a behavior-preserving transformation.

Next point:

I’d rather see self than having to assume it by scanning the first token until the first occurrence of $: (or ‘::’) to only then be able to disambiguate between a receiver and a keyword.

In short: I prefer the work of my internal parser to be made easier by the use of prefixes, rather than to have to keep a stack that only goes down in the presence of a suffix.

This aurgmnet is falwed for the smiple raeson that our percpetion dose’nt wrok this way. We do’nt hvae an intarenl parser. What we rellay hvae culod be desrcbeid as a comlepx adpative, preditcive and bakcptaching paettrn recogznier. This is why we can still read the above even though most of the words are messed up. We don’t scan the text linearly one character and one token at a time. Words are pictures, not character arrays. The structure of lines such as:

foobar baz.
foobar: baz.
foobar:: baz: foo.

and the presence or absence of : or :: in each are perceived by an experienced reader in one “processor cycle”. In short, this is a usability issue, and it’s a mistake to see it as an engineering one.

Those left unconvinced should also consider that modern IDEs, Newspeak’s Hospcotch included, do the parsing for you by colorizing the source code.

So it’s by far not a given that implicit receivers reduce readability. On the other hand, there are situations when they improve readability by eliminating noise. A good example are DSLs embedded in Newspeak. So far we have two such languages widely used in the system: Gilad’s parser combinators and my UI combinators in Hopscotch. The feature common to both are definitions written in a declarative style combining smaller things into larger ones. Compare an example of such a definition the way it’s commonly written:

heading: (
    row: {
        image: fileIcon.
        label: fileName.
    })
details:
    [column: folderContents]

with the same definition with explicit receivers:

self heading: (
    self row: {
        self image: self fileIcon.
        self label: self fileName.
    })
details:
    [self column: self folderContents]

The first example has nothing but the structure it defines. It’s important what the expressions say. The fact that they are message sends is an implementation detail. The second example leaks this implementation, and it takes some effort to see what it really says in between all the “self”s.

And the final point. There’s much to be said about the human nature and the tendency to instinctively resist change to something familiar while trying to rationalize that resistance. It takes some time and experimenting to see a change for what it is and get a feel of the new tradeoffs. I don’t know how many people with a considerable Self expertise are out there, but I doubt there are too many. I do know all the experienced Newspeakers. Which is to say, the time for conclusions hasn’t come yet.

Update:

Peter Ahé pointed out offline that I didn’t mention another important property facilitated by implicit receivers in Newspeak as well as in Self: representation independence.

Brazil Example: a Classic Smalltalk Browser

Here is an example of a classic Smalltalk browser implemented in Brazil. While the main point here is to illustrate Brazil API, an important related point I want to make first is that in our system Brazil is not used the way this example shows (and neither do our browsers look like that).

A good analogy to Brazil’s role in our UI is assembly language. One can program in it directly, but most of us these days don’t do that on any significant scale. In the same vein, one can think of Brazil as the UI assembly language. Earlier I described Hopscotch as an application framework, but that was simply to associate it with something familiar to everyone. In reality, Hopscotch is more of a higher-level UI language that hides or recasts in a different form the details of the low-level language it is based on. And just like one usually isn’t concerned with the exact translation of a high-level language program into machine code, a Hopscotch programmer doesn’t work with Brazil API directly.

The picture of the browser.
What it looks like

In the example, we create a window with the usual four list boxes and a text view, and wire them up so that list selection works the way it works in Smalltalk. The screenshot shows the browser open as a native Vista UI, and browsing the Object class from Squeak. The window in the background is the Newspeak workspace with the expression I used to instantiate and open the browser. The part in parentheses has to do with class nesting and the use of nested classes to implement modules. If you haven’t followed one of Gilad’s explanations, feel free to ignore the parenthesised part for the purposes of this example. The remainder,

ClassicBrowser new open

in the end produces the result a Smalltalker would expect—if not in the entirely expected way.

And here is the ClassicBrowser class itself. First, the class definition and the initializer:

class ClassicBrowser = (
"A simple implementation of a read-only Smalltalk browser."
|
	window = Window new.
	categories = ListBox new.
	classes = ListBox new.
	protocols = ListBox new.
	selectors = ListBox new.
	code = TextView new.
|
	assembleUI.
	configureUI.
)

This may look almost like a method with a comment, a list of temporaries in a not-quite-Smalltalk syntax (but note the vertical bars flushed to the left), and two expressions in the body. The comment here is the class comment, and the “temporaries” are actually slot declarations with initializers. Slots mean instance variables with automatically generated accessor methods. All slots of this class are read-only because of the use of the equals sign tying each slot to its initializer. A read-only slot is simply one with a getter method but no setter. The slots hold onto Brazil visuals assembled and configured into a coherent interface by the two methods called from the initializer. Here is how the assembly is done.

assembleUI = (
	| navigationRow |
	window area bounds: (200 @ 200 extent: 600 @ 600).
	window
		title: 'Brazil System Browser';
		content: Column new.

	navigationRow: Row new.

	window content
		add: navigationRow;
		addBlankSize: 3;
		add: code.

	navigationRow area height: 0; elasticity: 4.
	code area height: 0; elasticity: 6.

	navigationRow
		add: categories;
		addBlankSize: 3;
		add: classes;
		addBlankSize: 3;
		add: protocols;
		addBlankSize: 3;
		add: selectors.

	categories area elasticity: 1.
	classes area elasticity: 1.
	protocols area elasticity: 1.
	selectors area elasticity: 1.
)

The interesting point here all those “area” things the method talks to. Areas is a concept introduced in Brazil layout model to avoid the mess common in other UI frameworks. What’s messy is that it’s never clear who is in charge of a widget’s layout, and how to specify that layout in a particular situation. For example, in Morphic all morphs understand the message bounds: that repositions the morph, as well as the more focused messages position: and extent:. There are also specialized ones like vResizing: or cellSpacing:, which may or may not work in a particular setup. Given that some parent morphs such as AlignmentMorph manage the layout of their children, what happens when I place a morph inside one and then send position: to it? And how can I find out which of the plethora of layout messages that Morph understands will actually work in a given context?

The general problem here is that the protocol used to control the layout of a child depends on the context (the context being first of all the parent who manages the layout). Thus, Morph needs to expose the union of all possible layout attributes, only some of which are applicable at any given time.

The solution in Brazil is simple and natural, though I don’t know of other frameworks taking the same stance. A Brazil visual (the framework’s term for a widget) is unaware of any layout attributes, and has no methods to change them. Layout parameters are managed by a separate object called the visual’s area. The area is manufactured at the time a visual is added to the parent, as an instance of a class capturing the layout attributes meaningful for that particular visual in that particular parent. Thus an area represents the capability of positioning a widget inside the current parent. When a visual is a child of CompositeVisual (a free-form container similar to VisualWorks’ CompositePart), its area allows to arbitrarily position the visual by sending the bounds: message. When the same visual is a child of a Row, the area is an instance of a different class whose attributes and behavior are meaningful for a row cell. Arbitrarily moving the child should not be possible in that situation, and it is indeed impossible since this kind of area does not even understand the bounds: message.

Besides excluding what is impossible, areas also nicely manage and document the possible layout strategies. For example, in one situation we may want to force a Button inside a CompositeVisual to have a particular size. In a different situation, we might want to allow the button to resize to match the size of its label. These two layout disciplines are captured by two area classes called Frame and Anchor, both of which are allowed as areas of a button inside a composite. Assigning one or the other to a Button inside a CompositeVisual selects the layout strategy we want to use for the button.

A third benefit of this design feature is that unlike traditional frameworks whose layout strategies are hard-coded, with individual widgets holding onto layout attributes and behavior to control it, Brazil allows for pluggable layout strategies. For example, to extend the framework with a container that positions its children using a linear constraint system, all that’s needed is the container visual itself plus one or more area classes to capture the constraint parameters. None of the existing visuals require any changes to be used as children of the new container.

Back to the example. Now that the widgets are all assembled, the next method configures and initializes them:

configureUI = (
	configureCategoryList.
	configureClassList.
	configureProtocolList.
	configureSelectorList.
	configureCodeView.
)

Everything is straightforward here (again, we are freely using implicit receiver sends to avoid the clutter of “self” everywhere). Here is just one of the individual configuration methods—others are similar:

configureCategoryList = (
	categories 
		list: Smalltalk organization categories;
		displayBlock: [:symbol | symbol asString].
	categories selectionChanged => (self ~ #categorySelected:).
)

Again, nothing unusual, except perhaps for the last line. The part in parentheses is a section, in this case a shorthand for

[:category | self categorySelected: category]

The message => is part of Ducts, Brazil’s change notification framework, a minimalist alternative to Announcements (two classes and under 20 public methods, with all the same capabilities). It ties a change notifier supplied by the selectionChanged method to the section, so that whenever the selection changes the categorySelected: message is sent to the browser.

The categorySelected: method is straightforward again.

categorySelected: selection <Symbol | nil> = (
	selection
		ifNil: [classes list: Array new]
		ifNotNil: 
			[| names |
			names: (Smalltalk organization listAtCategoryNamed: selection).
			classes list: (names asSortedCollection collect: [:each | Smalltalk at: each])]
)

How to Design a Smalltalk UI Framework

It’s common for Smalltalk implementations to run on multiple platforms, and that leads to the problem of designing a framework to allow an application UI to display on all those platforms. The existing solutions (not only in Smalltalk) suffer from various shortcomings. In order to solve this problem better, Brazil from the start was based on a number of principles different from the industry examples. Here is an overview of those principles.

It’s not about widgets. A UI framework is not (just) a widget library. In that sense, Morphic is on the right track (and in the mainstream, so is WPF). Interfaces visualize information, and a good interactive visualization is typically more than a window with a few labels and list boxes plopped onto it. A UI framework needs to do much of what was traditionally expected from a structured graphics (“diagramming”) framework. Of course, buttons, list boxes and other primitive interactive elements are still necessary, but the value of a framework is primarily in composing rather than in implementing them. Which, in fact, it shouldn’t do at all.

When it’s about widgets, it’s about native widgets. Any OS provides about the same core set of primitive interactive elements. They are what the users expect to see in a “real program”, and the OS by definition does a better job implementing them than anyone could ever hope to. Emulated widgets are bound to be a poor replica of the original and are thus an exercise in futility. On the way from Smalltalk-80, one attraction of the emulated approach might have been the ability to customize, extend and roll your own, for the sake of creating more expressive UIs. But when a UI framework is also a graphics framework, expressive UIs can be built by composition rather than by widget hacking—so there really is no good reason not to use what’s already available, even if its implementation is sealed inside the OS.

Cross-platform shouldn’t mean platform-agnostic. So what about the lowest common denominator problem? It’s often assumed that using native widgets in a cross-platform environment means that only the subset of their features common across all platforms can be accessible. The mistake of this view is the assumption that it’s not cross-platform unless the user is shielded from platform differences. It doesn’t have to be that way, and shouldn’t be. Platforms are different, and a properly designed cross-platform framework should embrace and model those differences rather than try and inevitably fail creating an illusion that they do not exist. In practice, such modeling means that a framework object that represents a widget can provide access to the lowest common denominator features, while platform-specific features can be available as a “capability”—a separate object one can fetch from the widget when the capability is available (i.e. when the current platform is the one whose capabilities you ask for).

Qt is not your friend. It’s a common and understandable question—why not use an existing cross-platform framework like Qt or wxWidgets. Everything that has been said above is reason enough, and another important consideration is that a large complex hard-to-debug third party layer shielding you from the OS facilities you want to use sounds like bad news (something I dubbed as “high Space Shuttle factor” in the original implementation study). Which is also related to the last point.

If it’s not written in Smalltalk, it’s broken. There is a tendency (at least in Smalltalk-80 descendants) to do OS interaction through primitives. That is so perhaps because FFI arrived relatively late to ObjectWorks, and never was available for Squeak. Primitives are an arcane enough feature, so they are not discussed in Smalltalk style guides. If they were, the guides might have said something like this: many primitives you see in VisualWorks and Squeak should not have existed. A primitive is supposed to do what cannot be expressed in Smalltalk. If a call out to the OS or a call back from the OS cannot be expressed in Smalltalk, you need an FFI, not a VM plugin. OS interaction is not something that belongs in a primitive—and if fact, a primitive is about the worst place to do that kind of work. It means writing a large body of complex code in a low-level language with a higher probability of making a mistake, and then hiding it away in the VM where it can’t be easily debugged and fixed. Hardly a winning combination.

A True Pluggable Look-and-Feel

In an earlier post I mentioned that Brazil was designed to support cross-platform native UIs, and that I was working on our second platform mapping, Windows (the first one was Morphic). That was in May and for some time now Windows support is fully functional and Brazil applications can, as intended, work in either Morphic or Windows with no explicit provisions for that in the application code. What comes along is dynamic remapping–the ability to move the UI of a running application between Morphic and Windows without shutting down. This is just like switching the look in an emulated environment like VisualWorks–except that nothing is emulated and the widgets are the real deal.

Here is a screencast showing what it’s like.

This feature is not as gimmicky as it may seem. When an image-based environment is given a native UI, dynamic remapping is exactly what should happen when an image starts–and not only when it starts on a platform different from where it was saved. Even on the same platform, recreating native windows and widgets open at the time the snapshot was taken amounts to essentially the same thing.

A taste of implicit receivers

One of the features that distinguish Smalltalk from most other languages is the keyword message syntax, and one of those that distinguish Newspeak from Smalltalk are implicit receivers. They work very nicely together to allow writing very readable code. Here is a drag-and-drop related method I just wrote:

createDropTargetFor: session <DragDropSession> ^<DropTarget> = (
	^if: session source
		isMovingMyItem: [createDropTargetForItemMove]
		otherwise: [createClientSuppliedDropTargetFor: session]
)

What we have here is a method with the selector createDropTargetFor:. The things in angle brackets are type annotations–essentially, optional structured comments telling the reader what the method expects and returns. Unlike Smalltalk, the body of the method is explicitly delimited by “=(” and “)”.

The three message sends in the body have no explicit receivers, which in this case means they are sent to self, self being what any Smalltalker would expect. In the general case, however, because of class nesting there may be a choice of selves available, so the receiver is not necessarily the object whose class contains the method with the send. This is why “implicit receiver” is a more precise term. For details, see Gilad’s On the Interaction of Method Lookup and Scope with Inheritance and Nesting.

Inside the Refactoring Browser

A recent question on vwnc reminded me of the notes on RB architecture and menu extensions I wrote a couple of years ago. Thanks to Travis, I got my hands on those notes and edited them into a semblance of an article. As the introduction says, these notes are becoming obsolete as the browser in VisualWorks changes. However, since I’m not aware of any other RB architecture overview, I hope this article will stay useful for some time. Get it here.

Which is Which in Newspeak UI – Hopscotch

This is the second part of the Newspeak UI overview. Hopscotch is the name of both the application framework and the IDE of Newspeak, built on top of Brazil. Brazil provides “logical widgets” mapped onto the host platform artifacts and takes care of laying them out. Hopscotch adds the concepts of an application, domain objects, and mechanisms to keep the model and the view synchronized.

Hopscotch-the-IDE is based on a web browser-like model of navigation. I say “web browser-like” because the framework could support other application shapes as well, but in the particular case of an IDE, an interactive document viewed in a universal navigator is a model that works exceptionally well for visualizing a complex structure such as application code. The interactive documents are specified in a declarative and highly composable way.

The work on the framework is still in progress. I am interested in achieving a usage experience (the user in this case being a programmer using the framework) reminiscent of functional reactive programming, and that has some interesting algorithmic challenges.

More details are in this paper, and hopefully the video of my Smalltalk Solutions demo will soon be available. The paper has been accepted to WASDeTT, co-located with ECOOP, as a featured talk on July 8 at 12:00. Unfortunately, this coincides with the DLS, but I’ll also show a short demo during Gilad’s Short Introduction to Newspeak on July 10.

On UI Layout

Michael Lucas-Smith posted an interesting summary of UI layout models. I tried to leave the following as a comment, but the blog server swallowed it without posting. Fortunately, I half-expected this and copied the text before submitting. I’m posting it here instead, with some further edits.

Overall, this is a good summary, but there is a very important model it misses.

The model is a variant of constrained-based layout called linear constraints. In that model widget positions are expressed as a system of linear equations. This more formalized approach, unlike “intuitive” models with springs, wires and other pseudo-real life accessories, isn’t fiddly at all. There are algorithms (Cassowary is the most famous) to efficiently solve such constraint systems.

Also, importantly, many other models: fixed, relative, stacked and grid layouts are special cases of linear constraints. Only flow cannot be expressed this way. Thus, the models are not all incompatible. Also note that I’m listing “relative” as an additional model, which is the correct way to describe the VW approach (first proposed, most likely, by Luca Cardelli). (“Fixed” would be the case when the widget’s bounds are expressed as coordinates in some coordinate system, usually relative to the containing widget’s top left corner).

I’m not sure what it means to say that layout should not be part of the widget framework proper, without first defining the proper. One extreme is Windows which provides no automatic layout capabilities at all. That is indeed the pure fixed model. The other are frameworks with container visuals. In those frameworks the specific layout model is indeed dictated by the container. But still, the only difference between the two is that something like Qt has containers that do layout and Windows doesn’t. In neither of them layout is part of widgets (by widgets I mean atomic leaf components such as buttons or list views).

The important question is how hard or easy it is to introduce new containers with different layout models, but it is a matter of the overall framework design. The main complication here are performance constraints. The framework should reasonably quickly respond to incremental layout changes, and reconciling that need with pluggability of containers and their layout policies is where the devil lies. I’d estimate about 70% (if not more) of Brazil design effort has had something to do with layout management.

Which is Which in Newspeak UI – Brazil

Brazil, which is the foundation of Newspeak UI (and is indeed named after the movie), could be best described as a multi-platform logical UI and layout framework. It provides the familiar widgets (“visuals”) such as Labels, Buttons, and ListBoxes, and container visuals for combining widgets and managing their layout. Unlike the UI frameworks of Squeak or VisualWorks, Brazil does not attempt to actually implement the widgets by emulating them. Instead it functions as a “logical UI” layer that builds and drives the equivalent “physical UI” made of native artifacts of the host platform.

The first host platform of Brazil was Morphic–meaning the first “native” Brazil interfaces were in fact emulated. This trick allowed us to quickly build a minimal working implementation and polish the API without getting bogged down with FFI issues and native widget peculiarities early on. Since Morphic works on all OS platforms, so did Brazil from the start, allowing to use it on all platforms we need before we build real native mappings for those platforms.

Brazil also manages the layout of interfaces using a capability-based extensible model. Layout is fully managed by Brazil both on native platforms which usually provide rather weak layout facilities of their own, and in Morphic which provides a rich but baroque one. All that Brazil expects from the host are the abilities to position a widget within the specified rectangle, and to measure its natural (desired) extent.

Work on Brazil began in mid-March 2007 and in June a first real UI was built using it. I am currently working on a native Windows mapping. This work is at an advanced enough stage to be able to open all of the existing Brazil UIs as native in Windows with most functionality in place, without any changes in application code. Brazil windows can also be dynamically remapped, so that what appears as a Morphic window can “jump out” of Squeak and become a native window with native controls, and then go back to Morphic all the while retaining its state.

Unlike prior implementations of native UIs in Smalltalk, Brazil is implemented entirely at the image level. This means that there is not a single VM primitive to support UI operations. The mapping relies on “raw” OS functions accessed via the general-purpose Newspeak FFI to do everything, from Windows class registration to widget management and GDI+ painting.