doop: Immutable classes for TypeScript

Update 2016-03-07 In version 0.9.2 I’ve updated the name of the decorator function to begin with lowercase, to fit with the convention (of course you can rename it to whatever you prefer when you import).

As great as Immutable.js is, especially with a TypeScript declaration included in the package, the Record class leaves me a little disappointed.

In an ordinary class with public properties we’re used to being able to say:

const a = new Animal();
a.hasTail = true;
a.legs = 2;

const tailPrefix = a.hasTail ? "a" : "no";
const desciption = `Has ${a.legs} legs and ${tail} tail.`

That is, each property is a single named feature that can be used to set and get its value. But immutability makes things a little more complicated, because rather than changing a property of an object, we instead create a whole new object that has the same values on all its properties except for the one we want to change. It’s just a convenient version of “clone and update”. This is how it has to be with immutable data. You can’t change an object, but you can easily make a new object that is modified to your requirements.

Why is this hard to achieve in a statically typed way? This thread gives a nice quick background. In a nutshell, you use TypeScript because you want to statically declare the structure of your data. Immutable.js provides a class called Record that lets you define class-like data types, but at runtime rather than compile time. You can overlay TypeScript interface declarations onto them at compile time, but it’s a bit messy. Inheritance is troublesome, and there’s a stubborn set method that takes the property name as a string, so there’s nothing stopping you at compile-time from specifying the wrong property name or the wrong type of value.

The most complex suggestion in that thread is to use code generation to automatically generate a complete statically typed immutable class, from a simpler declaration in a TS-like syntax. This is certainly an option, but seems like a defeat for something so fundamental to programming as declaring the data structures we’re going to use in memory.

Really this kind of class declaration should be second nature. If we’re going to adopt immutable data as an approach, we’re going to be flinging these things around like there’s no tomorrow.

So I wanted to see if something simpler could be done using the built-in metaprogramming capabilities in TypeScript, namely decorators. And it can! And it’s not as ugly as it might be! And there’s a nice hack hiding under it!

How it looks

This is how to declare an immutable class with some properties and one method that reads the properties.

import { doop } from "../doop";

@doop
class Animal {

    @doop
    get hasTail() { return doop<boolean, this>() }

    @doop
    get legs() { return doop<number, this>(); }

    @doop
    get food() { return doop<string, this>(); }

    constructor() {
        this.hasTail(true).legs(2);
    }

    describe() {
        const tail = this.hasTail() ? "a" : "no";
        return `Has ${this.legs()} legs, ${tail} tail and likes to eat ${this.food()}.`;
    }
}

The library doop exposes a single named feature, doop, and you can see it being used in three ways in the above snippet:

• As a class decorator, right above the Animal class: this allows it to “finish off” the class definition when the code is loaded into the JS engine.
• As a property decorator, above each property: this inserts a function that implements both get and set functionality.
• As a helper function, called inside each property getter

Although not visible in that snippet, there’s also a generic interface, Doop, returned by the helper function, and hence supported by each property:

interface Doop<V, O> {
    (): V;
    (newValue: V): O;
}

That’s a function object with two overloads. So to get the value of a property (as you can see happening in the describe method) you call it like a function with no arguments:

if (a.hasTail()) { ...

It’s a little annoying that you can’t just say:

if (a.hasTail) { ...

But that would rule out being able to “set” (make a modified clone) through the same named feature on the object. If the type of hasTail were boolean, we’d be stuck.

There’s a particular pattern you follow to create a property in an immutable class. You have to define it as a getter function (using the get prefix), and return the result of calling doop as a helper function, which is where you get to specify the type of the property. Note: you only need to define a getter; doop provides getting and pseudo-setting (i.e. cloning) via the same property, with static type checking.

See how the constructor is able to call its properties to supply them with initial values. This looks a lot like mutation, doesn’t it? Well, it is. But it’s okay because we’re in the constructor. doop won’t let this happen on properties of a class that has finished being constructed and therefore is in danger of being seen to change (NB. you can leak a reference to your unfinished object out of your constructor by passing this as an argument to some outside function… so don’t do that).

And in the describe method (which is just here as an example, not part of any mandatory pattern) you can see how we retrieve the values by calling properties as if they were methods, this time passing no parameters.

But what’s not demonstrated in this example is “setting” a value in an already-constructed object. It looks like this:

const a = new Animal();
expect(a.legs()).toEqual(2); // jasmine spec-style assertion

// create a modified clone
const b = a.legs(4);
expect(b.legs()).toEqual(4);

// original object is unaffected
expect(a.legs()).toEqual(2);

Inheritance is supported; a derived class can add more properties, and in its constructor (after calling super() it can mutate the base class’s properties. The runtime performance of a derived class should be identical to that of an equivalent class that declares all the properties itself.

One thing to be wary of is adding ordinary instance properties to a doop class. It would be difficult to effectively block this happening, and in any case there may occasionally be a good reason to do it, as long as you understand one basic limitation of it: ordinary instance properties belong to an instance. When you call a property to set its value, you are returned a new instance, and there is no magic that automatically copies or initialises any instance fields. Only the other doop properties will have the same values as in the original instance. Any plain instance fields in the new instance will have the value undefined.

For simplicity’s sake, just make sure in a doop class that all data is stored in doop properties.

Implementation

The implementation of the cloning is basically the one described here so it’s super-fast.

I mentioned there’s a hack involved, and it’s this: I needed a way to generate, from a single declaration in the library user’s code, something that can perform two completely different operations: a simple get and a pseudo-set that returns a new instance. That means I need each property to be an object with two functions. But if I do that literally, then a get would look like this:

// A bit ugly
const v = a.legs.get();
const a2 = a.legs.set(4);

I don’t like the verbosity, for starters. But there’s a worse problem caused by legs being an extra object in the middle. Think about how this works in JS. Inside the get function this would point to legs, which is just some helper object stored in a property defined on the prototype used by all instances of the Animal class. It’s not associated with an instance. It doesn’t know what instance we’re trying to get a value from. I could fix this by creating a duplicate legs object as an instance property on every Animal instance, and then giving it a back-reference to the owning Animal, but that would entirely defeat the whole point of the really fast implementation, which uses a secret array so it can be rapidly cloned, whereas copying object properties is very much slower.

Or I could make legs, as a property getter, allocate a new middle object on the fly and pass through the this reference. So every time you so much as looked at a property, you’d be allocating an object that needs to be garbage collected. Modern GCs are amazing, but still, let’s not invent work for them.

So what if instead of properties, I made the user declare a function with two overloads for getting and setting? That solves the this problem, but greatly increases the boilerplate code overhead. The user would actually have to write two declarations for the overloads (stating the “property” type twice) and a third for the implementation:

// Ugh
@doop
legs(): number;
legs(v: number): this;
legs(v?: number): any { }

The function body would be empty because the doop decorator replaces it with a working version. But it’s just a big splurge of noise so it’s not good enough. And yet it’s the best usage syntax available. Ho hum.

Lateral thinking to the rescue: in TypeScript we can declare the interface to a function with two overloads. Here it is again:

export interface Doop<V, O> {
    (): V;
    (newValue: V): O;
}

Note that O is the type of the object that owns the property, as that’s what the “setter” overload has to return a new instance of.

Using a getter in the actual doop library looks like this:

const l: number = a.legs();

There are at least two possible interpretations of a.legs():

• legs is function that returns the number we want.
• legs is a property backed by a getter function, that returns a function with at least one overload (): number, which when called returns the number we want.

To explain the second one more carefully: the part that says a.legs will actually call the getter function, which returns a second function, so a.legs() would actually make two calls. The returned function would need to be created on-the-fly so it has access to the relevant this, so this is very much like the GC-heavy option I described earlier.

But it’s not possible to tell which it is from the syntax. And that’s quite good. Because if we tell the TypeScript compiler that we’re declaring a getter function that returns a function, it will generate JavaScript such as a.legs(). But at runtime, we can use the simple implementation where legs is just a function. The doop decorator can make that switcheroo, and we get the best of both worlds: a one-liner declaration of a property getter, and a minimal overhead implementation.

Well, it seemed nifty to me when I realised it would work!

So this is what the doop property decorator does: the user has declared a property, and all we care about is its name. All properties are the same at runtime: just a function that can be called to either get or clone-mutate.

doop on GitHub

Is TypeScript really a superset of JavaScript? And does it even matter?

Questions:

• What does it mean for a programming language to be a superset of another programming language?
• What’s a programming language?
• What’s a program?

In this discussion, a program, regardless of language, is a stream of characters.

If you generated a random stream of characters, it might be a valid program in some hypothetical language, just as the arrangement of stars in the night sky as viewed from Earth might happen to spell out an insulting message in some alien language we’ll never know about.

So a programming language is both:

• the rules for deciding whether a given stream of characters is a valid program, from that language’s point of view, and,
• the set of valid programs, because they are streams of characters that conform to those rules.

It’s the second (slightly surprising) formulation we’re interested in here, because it means that when we say “language A is a superset of language B”, we mean that A and B are sets of programs, and set A includes all the programs in set B. This is useful information, because it means all the programs we wrote in language B can immediately used in language A, without us needing to change them.

People get very muddled about this, because they think of the programming language as a set of rules instead of a set of programs, and therefore assume that a superset would include all the rules of the subset language, plus some extra rules. This could make it stricter, rejecting some previously valid programs, or it could make it looser, allowing new syntactic forms. So without knowing the details of the extra rules in question, we wouldn’t know what’s happened.

So the “set of rules” sense is far less useful than the “set of programs” sense, which does actually tell us something about the compatibility between the languages.

The most common statement in introductions and tutorials about TypeScript is that it is a superset of JavaScript. Really? Here’s a valid JavaScript program:

var x = 5;
x = "hello";

Rename it to .ts and compile it with tsc and you’ll get an error message:

Type 'string' is not assignable to type 'number'.

We can fix it though:

var x: any = "hello";
x = 5;

We’ve stopped the compiler from inferring that x is specifically a string variable just because that’s what we initialised it with. Plain JavaScript can be retro-imagined as a version of TypeScript that assumes every variable is of type any.

In any case, one example is sufficient to show that TypeScript is not a superset of JavaScript in the more useful “set of valid programs” sense, and it seems we’ve found one. Except it’s a bit murkier than that.

If you looked in the folder containing your source file right after you tried to compile the “broken” version, you would have found an output .js file that the TypeScript compiler had generated quite happily.

TypeScript makes your source jump over two hurdles:

1. Is it good enough to produce JavaScript output?
2. Does it pass type checks?

If your source clears the first hurdle, you get a runnable JavaScript program as output even if it doesn’t clear the second hurdle. This quirk allows TypeScript to claim to be a superset of JavaScript in the set-of-programs sense.

But I’m not sure it counts for much. Is anyone seriously going to release a product or publish a site when it has type errors in compilation? They wouldn’t be getting any value from TypeScript (over any ES6 transpiler such as Babel). The compiler has a couple of switches that really should be enabled in any serious project:

• --noEmitOnError – require both hurdles to be cleared (the word “error” here refers to type errors).
• --noImplicitAny – when type inference can’t deduce something more specific than any, halt the compilation.

If you’re going to use TypeScript (hint: you are) then use it properly.

And in that case, it is not a superset of JavaScript. And this is a Good Thing. The whole point is that existing JavaScript programs, due to the language’s dynamically typed looseness, very often contain mistakes that would be trapped by TypeScript. The example above, where a variable is reused for different types, might be a mistake but might not (in performance terms, it’s probably a mistake in that it stops modern JS runtimes from optimising code that accesses the variable).

When we want to use JavaScript, unchanged, as part of a TypeScript project, we just leave it as JavaScript and wrap it in d.ts declarations. It’s no big deal. You only change the extension to .ts because you want it to be more rigorously checked, so you know that the types make sense all the way down into the nitty gritty.

The parallel with C++’s relationship to C is striking. In C (specifically, ANSI C prior to the 1999 standard) it was not necessary to declare a function before you called it. In C++ it became mandatory. This – amongst other differences – meant that C++ was never a superset of C.

But this didn’t matter too much, because C++ was a superset of well-written C – to the extent that every C example in the 2nd edition of K&R was valid C++.

So TypeScript, in any meaningful sense, is not a superset of JavaScript, but this is nothing to get hung up over; if it were a superset of JavaScript, it would be considerably less useful.

TypeScript 1.5: Get the decorators in…

Update 2016-03-07: My new library, Doop, is a more practical demonstration of what you can do with decorators.

No great mysteries about what this class does:

class C {
    foo(n: number) {
        return n * 2;
    }
}

It’s a stupid example with a method that returns its only parameter multiplied by two. So this prints 46:

var c = new C();
console.log(c.foo(23));

A feature coming in TS 1.5 that has really caught my eye is decorators. We can decorate the foo method:

class C {
    @log
    foo(n: number) {
        return n * 2;
    }
}

That on its own will not compile, because we need to define log. Here’s my first try:

function log(target: Function, key: string, value: any) {
    return {
        value: function (...args: any[]) {

            var a = args.map(a => JSON.stringify(a)).join();
            var result = value.value.apply(this, args);
            var r = JSON.stringify(result);

            console.log(`Call: ${key}(${a}) => ${r}`);

            return result;
        }
    };
}

Note: previously I used arrow function syntax to declare value, but as pointed out in this answer on Stack Overflow, that interferes with the value of this, which ought to be passed straight through unchanged.

The three parameters, target, key and value, are passed in by the helper code generated by the TypeScript compiler. In this case:

• target will be the prototype of C,
• key will be the name of the method ("foo"),
• value will be a property descriptor, which here will have a value property that is the function that doubles its input.

So my log function returns a new property descriptor that the TypeScript compiler will use as the new definition of foo. This means I can wrap the real function in one that logs information to the console, so here I log the arguments and the return value.

So when the resulting program runs in node, it prints two lines instead of just one:

Call: foo(23) => 46
46

This is really quite neat. The next thing I think is: can I decorate a simple value so that it turns into a property that is an observable that can be listened to for change events? I’m going to find out.

TypeScript: Physical Code Organisation

When I first started reading about TypeScript, I had one main concern: how am I going to make this work with the weird mix of modular code and old-school JS libraries in my existing codebase?

The features of the language itself are very well covered. I found various great introductions (and now there’s the awesome official Handbook), but they all seemed to gloss over certain fundamental questions I had about the physical organisation of code. So I’m going to go through the basics here and try to answer those questions as I go.

Modularity in the Browser

Let’s get the the controversial opinionated bit out of the way. (Spoiler: turns out my opinions on this are irrelevant to TS!)

How should you physically transport your JS into your user’s browser? There are those who suggest you should asynchronously load individual module files on the fly. I am not one of them. Stitch your files together into one big chunk, minify it, let the web server gzip it, let the browser cache it. This means it gets onto the user’s machine in a single request, typically once, like any other binary resource.

The exception would be during the development process: the edit-refresh-debug cycle. Clearly it shouldn’t be minified here. Nor should it be cached by the browser (load the latest version, ya varmint!) And ideally it shouldn’t be one big file, though that’s not as much of an issue as it was a few years ago (even as late as version 9, IE used to crash if you tried to debug large files, and Chrome would get confused about breakpoints).

But I’ve found it pretty straightforward to put a conditional flag in my applications, a ?DEBUG mode, which controls how it serves up the source. In production it’s the fast, small version. In ?DEBUG, it’s the convenient version (separate files).

In neither situation does it need to be anything other than CommonJS. For about four years now I’ve been using CommonJS-style require/exports as my module API in the browser, and it’s the smoothest, best-of-all-worlds experience I could wish for.

So what’s the point of AMD? Apparently “… debugging multiple files that are concatenated into one file [has] practical weaknesses. Those weaknesses may be addressed in browser tooling some day…” In my house they were addressed in the browser in about 2011.

But anyway… deep breath, calms down… it turns out that TypeScript doesn’t care how you do this. It turns us all into Lilliputians arguing over which way up a boiled egg must be eaten.

The kinds of file in TypeScript

In TS, modules and physical files are not necessarily the same thing. If you want to work that way, you can. You can mix and match. So however you ended up with your codebase, TS can probably handle it.

If a TS file just contains things like:

var x = 5;
function f() {
    return x;
}

Then the compiler will output the same thing (exactly the same thing, in that example). You can start to make it modular (in a sense) without splitting into multiple files:

module MyStuff {
    var x = 5;
    export function f() {
        return x;
    }
}

var y = MyStuff.f();

That makes an (or extends an existing) object called MyStuff with one property, f, because I prefixed it with export. Modules can nest. So just as in JavaScript there’s one big global namespace that your source contributes properties to, but you can achieve modularity by using objects to contain related things.

You can at this point roll your own pattern: write lots of separate files in the above style, each file being responsible for wrapping its code in a named module, then pass them to the TS compiler and stitch the result into one file.

Now try using export at the top level in your file:

var x = 5;
export function f() {
    return x;
}

The compiler will complain that you haven’t told it what module system you want to use. You tell it with the flag --module commonjs (or --module amd if you’re crazy). Now it works and does exactly what you’d expect as a user of your chosen module system.

But what does this mean in terms of the static type system of TS and so on? It means that this particular file no longer contributes any properties to the global namespace. By just using the export prefix at the top level, you converted it into what TS calls an external module.

In order to make use of it from another module, you need to require it:

import myModule = require("super-modules/my-module");

(Subsequent versions of TS will add more flexible ways to write this, based on ES6.)

Nagging question that can’t be glossed over: What happens to the string "super-modules/my-module"? How is it interpreted? In the output JS it’s easy: it is just kept exactly as it is. So your module system better understand it. But the compiler also wants to find a TS file at compile time, to provide type information for the myModule variable.

Suppose the importing module is saved at the location:

somewhere/awesome-code/not-so-much/domestic/toaster.ts

The compiler will try these paths, in this order, until one exists:

• somewhere/awesome-code/not-so-much/domestic/super-modules/my-module.ts
• somewhere/awesome-code/not-so-much/super-modules/my-module.ts
• somewhere/awesome-code/super-modules/my-module.ts
• somewhere/super-modules/my-module.ts

i.e. it searches up the tree until it runs out of parent directories. (It will also accept a file with the extension .d.ts, or it can be “tricked” into not searching at all, but we’ll get to that later).

This is a little different to node’s take on CommonJS, where you’d only get that behaviour if your import path started with ./ – otherwise it inserts node_modules in the middle. But this doesn’t matter, as we’ll see.

One advantage of external modules over the first pattern we tried is that it avoids name clashes. Every module decides what name it will use to “mount” modules into its own namespace. Also note that by importing an external module in this way, your module also becomes external. Nothing you declare globally will actually end up as properties of the global object (e.g. window) any more.

So we have two kinds of file: external modules, and what I’m going to call plain files. The latter just pollute the global namespace with whatever you define in them. The compiler classifies all files as plain files unless they make use of import or export at the top level.

How do you call JavaScript from TypeScript?

No need to explain why this is an important question, I guess. The first thing to note is that widely-used JS libraries are packaged in various ways, many of them having longer histories than any popular JS module systems.

What if you’re dealing with something like jQuery and in your own JS you’ve been blithely assuming that $ exists globally? What you’re wishing for is that someone would rewrite jQuery as a plain TS file that says something like:

function $(selector: any) {
    // Um...
}

No use of export, see? It’s a little trickier than that in reality because $ is not just a function; it has properties of its own. Don’t worry – TS has ways to declare that.

Of course, no one can be bothered to rewrite jQuery in TS and fortunately they don’t have to. TypeScript supports ambient declarations, which are prefixed with the keyword declare like this:

declare var x: number;
declare function f(): number; 

These tell the compiler that somehow arrangements will be made such that the global namespace has properties x and f with those particular shapes. Just trust that they’ll be there, Mr Compiler, and don’t ask any questions. In fact the compiler won’t generate any output code for ambient declarations. (If you’re familiar with the old world of C, think header files, prototypes and extern).

Note that I don’t initialise x or provide a body for f, which would not be allowed; as a result the compiler cannot infer their types. To make the declarations be worth a damn, I specify the type number where necessary.

Finally, you can make sure that a file contains only ambient declarations by naming it with the extension .d.ts. That way, you can tell at a glance whether a file emits code. Your linking process (whatever it is) never needs to know about these declaration files. (Again, by analogy to C, these are header files, except the compiler bans them from defining anything. They can only declare.)

(In case you’re panicking at this point, it isn’t necessary to write your own declarations for jQuery, or for many other libraries (whether in the browser or Node). See DefinitelyTyped for tons of already-written ones.)

What if third party code does use a module system such as CommonJS? For example, if you’re using TS in Node and you want to say:

import path = require("path");

You have a couple of options. The first, and least popular as far as I can tell, is to have a file called path.d.ts that you put somewhere so it can be found by the compiler’s searching algorithm. Inside that file you’d have declarations such as:

export declare function join(...path: string[]): string;

The other option is that you have a file called path.d.ts that you put anywhere you like, as long as you give it to the TS compiler to read. In terms of modules it will be a plain file, not an external module. So it can declare anything you want. But somewhere in it, you write a peculiar module declaration:

declare module "path" {
    export function join(...path: string[]): string;
}

Note how the module name is given as a quoted string. This tells the compiler: if anyone tries to import "path", use this module as the imported type structure. It effectively overrides the searching algorithm. This is by far the most popular approach.

Reference comments

In some TS code you’ll see comments at the top of the file like this:

///<reference path="something/blah.d.ts" />

This simply tells the compiler to add that file (specified relative to the containing directory of the current file) to the set of files it is compiling. It’s like a crummy substitute for project files. In some near-future version of TS the compiler will look for a tsconfig.json in the current directory, which will act as a true project file (the superb TypeStrong plugin for the Atom editor already reads and writes the proposed format).

In Visual Studio projects, just adding a .ts file to a project is sufficient to get the compiler to read it. The only reason nowadays to use reference comments is to impose an order in which declarations are read by the compiler, as TypeScript’s approach to overloading depends on the order in which declarations appear.

DefinitelyTyped and tsd

If you install node and then (with appropriate permissions) say:

npm install -g tsd

You’ll get a command-line tool that will find, and optionally download, type definition files for you. e.g.

tsd query knockout

Or if you actually want to download it:

tsd query knockout --action install

This will just write a single file at typings/knockout/knockout.d.ts relative to the current directory. You can also add the option --save:

tsd query knockout --action install --save

That will make it save a file called tsd.json recording the precise versions of what you’ve downloaded. They’re all coming from the same github repository, so they are versioned by changeset.

Migrating

I uhmm-ed and ahhh-ed for a while trying to decide what approach to take with my existing JS code. Should I write type declarations and only write brand new code in TS? Should I convert the most “actively developed” existing JS into TS?

The apparent dilemma stems from the way that .d.ts files let you describe a module without rewriting it, and “rewriting” sounds risky.

But it turned out, in my experience, that this is a false dilemma. The “rewriting” necessary to make a JS file into a TS file is

1. Not that risky, as most of the actually code flow is completely unmodified. You’re mostly just declaring interfaces, and adding types to the variable names wherever they’re introduced.
2. Phenomenally, indescribably worth the effort. By putting the types right in the code, the TS compiler helps you ensure that everything is consistent. Contrast this with the external .d.ts which the compiler has to trust is an accurate description. A .d.ts is like a promise from a politician.

In the end, I decided that the maximum benefit would come from rewriting two kinds of existing JS:

• Anything where we have a lot of churn.
• Anything quite fundamental that lots of other modules depend on, even if it’s not churning all that much.

You may come to a different conclusion, but this is working out great for me so far. Now when someone on the team has to write something new, they do it in TS and they have plenty of existing code in TS to act as their ecosystem.

I think that’s everything. What have I missed?

TypeScript 1.6 – Async functions

Update: have retitled this post based on the roadmap, which excitingly now has generators and async/await slated for 1.6!

I realise that I’m in danger of writing the same blog post about once a year, and I am definitely going to start making notes on my experiences using TypeScript generally, now that I’m using it on an industrial scale (around 40,000 lines converted from JavaScript in the last month or so, and the latest features in 1.4 have taken it to a new level of brilliance).

But the fact is, we’re getting tantalisingly close to my holy grail of convenient async programming and static typing in one marvellous open source package, on JavaScript-enabled platforms. If you get TypeScript’s source:

git clone https://github.com/Microsoft/TypeScript.git
cd TypeScript

And then switch to the prototypeAsync branch:

git checkout prototypeAsync

And do the usual steps to build the compiler:

npm install -g jake
npm install
jake local 

You now have a TypeScript 1.5-ish compiler that you can run with:

node built/local/tsc.js -t ES5 my-code.ts

The -t ES5 flag is important because for the async code generation the compiler otherwise assumes that you’re targeting ES6, which (as of now, in browsers and mainstream node) you probably aren’t.

And then things are very straightforward (assuming you have a promisified API to call):

    async function startup() {

        if (!await fs.exists(metabasePath)) {
            await fs.mkdir(metabasePath);
        }
        if (!await fs.exists(coverartPath)) {
            fs.mkdir(coverartPath);
        }

        console.log("Loading metabase...");
        var metabaseJson: string;
        try {
            metabaseJson = await fs.readFile(metabaseFile, 'utf8');
        } catch (x) {
            console.log("No existing metabase found");
        }

        // and so on...

This corresponds very closely to previous uses of yield (such as this), but without the need to manually wrap the function in a helper that makes a promise out of a generator.

As explained in the ES7 proposal the feature can be described in exactly those terms, and sure enough the TypeScript compiler structures its output as a function that makes a generator, wrapped in a function that turns a generator into a promise.

This of course made me assume that ES6 generator syntax would also be implemented, but it’s not yet. But no matter! As I previously demonstrated with C#, if a generator has been wrapped in a promise, we can wrap it back in a generator.

To keep the example short and sweet, I’m going to skip three details:

• exception handling (which is really no different to returning values)
• passing values into the generator so they are “returned” from the next use of yield (similarly, passing in an Error so it will be thrown out of yield)
• returning a value at the end of the generator.

The first two are just more of the same, but the last one turned out to be technically tricky and I suspect is impossible. It’s a quirky and non-essential feature of ES6 generators anyway.

To start with I need type declarations for Promise and also (for reasons that will become clear) Thenable, so I grabbed es6-promise.d.ts from DefinitelyTyped.

Then we write a function, generator that accepts a function and returns a generator object (albeit a simplified one that only has the next method):

    function generator<TYields>(
        impl: (yield: (val: TYields) => Thenable<void>
    ) => Promise<void>) {

        var started = false,
            yielded: TYields,
            continuation: () => void;

        function start() {
            impl(val => {
                yielded = val;
                return {
                    then(onFulfilled?: () => void) {
                        continuation = onFulfilled;
                        return this;
                    }
                };
            });
        }

        return {
            next(): { value?: TYields; done: boolean } {
                if (!started) {
                    started = true;
                    start();
                } else if (continuation) {
                    var c = continuation;
                    continuation = null;
                    c();
                }
                return !continuation ? { done: true } 
                    : { value: yielded, done: false };
            }
        };
    }

The impl function would be written using async/await, e.g.:

    var g = generator<string>(async (yield) => {

        console.log("Started");

        await yield("first");

        console.log("Continuing");

        for (var n = 0; n < 5; n++) {
            await yield("Number: " + n);
        }

        await yield("last");

        console.log("Done");
    });

Note how it accepts a parameter yield that is itself a function: this serves as the equivalent of the yield keyword, although we have to prefix it with await:

    await yield("first");

And then we can drive the progress of the generator g in the usual way, completely synchronously:

    for (var r; r = g.next(), !r.done;) {    
        console.log("-- " + r.value);
    }

Which prints:

Started
-- first
Continuing
-- Number: 0
-- Number: 1
-- Number: 2
-- Number: 3
-- Number: 4
-- last
Done

So how does this work? Well, firstly (and somewhat ironically) we have to avoid using promises as much as possible. The reason has to do with the terrifying Zalgo. As it says in the Promises/A+ spec, when you cause a promise to be resolved, this does not immediately (synchronously) trigger a call to any functions that have been registered via then. This is important because it ensures that such callbacks are always asynchronous.

But this has nothing to do with generators, which do not inherently have anything to do with asynchronicity. In the above example, we must be able to create the generator and iterate through it to exhaustion, all in a single event loop tick. So if we rely on promises to carry messages back and forth between the code inside and outside the generator, it just ain’t gonna work. Our “driving” loop on the outside is purely synchronous. It doesn’t yield for anyone or anything.

Hence, observe that when generator calls impl:

    impl(val => {
        yielded = val;
        return {
            then(onFulfilled?: () => void) {
                continuation = onFulfilled;
                return this;
            }
        };
    });

it completely ignored the returned promise, and in the implementation of yield (which is that lambda that accepts val) it cooks up a poor man’s pseudo-promise that clearly does not implement Promises/A+. Technically this is known as a mere Thenable. It doesn’t implement proper chaining behaviour (fortunately unnecessary in this context), instead returning itself. The onFulfilled function is just stashed in the continuation variable for later use in next:

    if (!started) {
        started = true;
        start();
    } else if (continuation) {
        var c = continuation;
        continuation = null;
        c();
    }
    return !continuation ? { done: true } 
                         : { value: yielded, done: false };

The first part is trivial: if we haven’t started, then start and remember that we’ve done so. Then we come to the meat of the logic: if this is the second time next has been called, then we’ve started. That means that impl has been called, and it ran until it hit the first occurrence of await yield, i.e.:

    await yield("first");

The TypeScript compiler’s generated code will have received our Thenable, and enlisted on it by calling then, which means we have stashed that callback in continuation. To be sure we only call it once, we “swap” it out of continuation into a temporary variable before we call it:

    var c = continuation;
    continuation = null;
    c();

That (synchronously) executes another chunk of impl until the next await yield, but note that we left continuation set to null. This is important because what if impl runs out of code to execute? We can detect this, because continuation will remain null. And so the last part looks like this:

    return !continuation ? { done: true } 
                         : { value: yielded, done: false };

Why do we have to use this stateful trickery? To reiterate (pun!) the promise returned by impl is meant to signal to us when impl has finished, but it’s just no good to us, because it’s a well-behaved promise, so it wouldn’t execute our callback until the next event loop tick, which is way too late in good old synchronous generators.

But this means we can’t get the final return value (if any) of impl, as the only way to see that from the outside is by enlisting on the returned promise. And that’s why I can’t make that one feature of generators work in this example.

Anyway, hopefully soon this will just be of nerdy historical interest, once generators make it into TypeScript. What might be the stumbling block? Well, TypeScript is all about static typing. In an ES6 generator in plain JavaScript, all names (that can be bound to values) have the same static type, known in TypeScript as any, or in the vernacular as whatever:

    function *g() {
        var x = yield "hello";
        var y = yield 52;
        yield [x, y];
    }

    var i = g();
    var a = i.next().value;
    var b = i.next(61).value;
    var c = i.next("humpty").value;

The runtime types are another matter: as the only kind of assignments here are initialisation, so each variable only ever contains one type of value, we can analyse it and associate a definite type with each:

x: number = 61
y: string = "humpty"
a: string = "hello"
b: number = 52;
c: [number, string] = [61, "humpty"]

But in TypeScript, we want the compiler to track this kind of stuff for us. Could it use type inference to do any good? The two questions to be answered are:

• What is the type of the value accepted by next, which is also the type of the value “returned” by the yield operator inside the generator?
• What is the type of the value returned in the value property of the object returned by next, which is also the type of value accepted by the yield operator?

The compiler could look at the types that the generator passes to yield. It could take the union of those types (string | number | [number, string]) and thus infer the type of the value property of the object returned by next. But the flow of information in the other direction isn’t so easy: the type of value “returned” from yield inside the generator depends on what the driving code passes to next. It’s not possible to tie down the type via inference alone.

There are therefore two possibilities:

• Leave the type as any. This is not great, especially not if (like me) you’re a noImplicitAny adherent. Static typing is the whole point!
• Allow the programmer to fully specify the type signature of yield.

The latter is obviously my preference. Imagine you could write the interface of a generator:

    interface MyGeneratorFunc {
        *(blah: string): Generator<number, string>;
    }

Note the * prefix, which would tell the compiler that we’re describing a generator function, by analogy with function *. And because it’s a generator, the compiler requires us to follow up with the return type Generator, which would be a built-in interface. The two type parameters describe:

• the values passed to yield (and the final return value of the whole generator)
• the values “returned” from yield

Note that the first type covers two kinds of outputs from the generator, but they have to be described by the same type because both are emitted in the value property of the object returned by the generator object’s next method:

function *g() {
    yield "eggs";
    yield "ham";
    return "lunch";
}

var i = g();
i.next() // {value: "eggs", done: false}
i.next() // {value: "ham", done: false}
i.next() // {value: "lunch", done: true} - Note: done === true 

Therefore, if we need them to be different types, we’ll have to use a union type to munge them together. In the most common simple use cases, the second type argument would be void.

This would probably be adequate, but in reality it’s trickier than this. Supposing in a parallel universe this extension was already implemented, but async/await was still the stuff of nightmares, how might we use it to describe the use of generators to achieve asynchrony? It’d be quite tricky. How about:

    interface AsyncFunc {
        *(): Generator<Promise<?>, ?>;
    }

See what I mean? What replaces those question marks? What we’d like to say is that wherever yield occurs inside the generator, it should accept a promise of a T and give back a plain T, where those Ts are the same type for a single use of yield, and yet it can be a different T for each use of yield in the same generator.

The hypothetical declaration above just can’t capture that relation. It’s all getting messy. No wonder they’re doing async/await first. On the other hand, we’re not in that universe, so maybe this stuff does’t matter.

These details aside, given how mature the prototype appears to be, I’m very much hoping that it will be released soon, with or without ordinary generators. It’s solid enough for me to use it for my fun home project, and it’s obviously so much better than any other way of describing complex asynchronous operations that I am even happy to give up IDE integration in order to use it (though I’d be very interested to hear if it’s possible to get it working in Visual Studio or Eclipse).

(But so far I’m sticking to a strict policy of waiting for official releases before unleashing them on my co-workers, so for now my day job remains on 1.4. And so my next post will be about some of the fun I’m having with that.)

Desktop Apps in JavaScript+HTML

Part of the reason I did Eventless (apart from it being fun and explanatory) was so I’d have something almost as convenient as Knockout available to me the next time I had to write a desktop app with a UI. It has now been many years since we could regard web UI development as a hairshirt chore, where we have to make do without the comforts of familiar mature environments and tools. Quite the opposite: whenever I have to write and support a desktop app I curse the fact that I can’t hit F12 and “inspect the DOM” of my UI while it’s running, or immediately debug the code on the tester’s machine when they find a problem.

In the decade-before-last, Microsoft quietly released an IE feature called HTA, or “HTML application”. It’s the neat (but trivially obvious) idea of switching off the usual browser security checks and allowing you to create any ActiveX scriptable objects, so you could write applications with HTML UIs. Neat, but horrible in practise because… well, you had to be there to appreciate the overall shoddiness. And so that’s why they of course had to rush down to the patent office.

But fast forward to the last few years. We can develop complex server or command-line applications in JavaScript using the Node platform and its ecosystem of libraries. We can use a variety of front end languages to fill in the deficiencies of raw JavaScript. The debugging experience inside modern browsers is the best debugging experience anywhere. And so on. It’s well beyond the time for HTAs done right.

It being such an obvious idea, there are a few implementations floating around, but the best I’ve seen is node-webkit. Which is a fine technical name (because it tells you honestly that it’s just node mashed together with webkit), but I think they should pick some cool and memorable “marketing” name for it, because it’s just too brilliant a combination to go without a name of its own. I suggest calling it 长裤. Or failing that, 内裤.

The easiest way to get started with it is to install node, then the nodewebkit package with the -g flag (it’s not a module that extends node; rather, it’s a separate runtime that has its own copy of node embedded). Then you create a package.json with an HTML file as its main:

"main": "index.html"

From that HTML file you can pull in scripts using the script tag in the usual way. But inside those scripts you can use node’s require. Yup.

The sweet combination for me is the beautiful TypeScript, plus the phenomenal Knockout, plus whatever node modules I want to call (along with TypeScript declarations from DefinitelyTyped). This gives me the best of everything: static typing, Chrome-like debugging, the smoothest most scalable/flexible form of two-way binding, the whole works. So I’ll probably never use Eventless in a real project.

I actually started writing a UI for conveniently driving Selenium (which I hopefully will have time to describe soon) in C#/Windows Forms. After getting it all working, I trashed it and switched to node-webkit and it was ridiculous how quickly I was able to get back to the same spot, plus a huge momentum boost from the fun I was having.

(Though admittedly quite a lot of that fun was probably the first flush of joy from using TypeScript.)

(I’m a nerd, did I mention that?)

A way that async/await-style functionality might make it into browsers

Many moons ago (actually it’s 35.7 moons ago) I wrote an excited post about JavaScript generators in Firefox. Sadly they are still only in Firefox, and there’s no sign of them appearing elsewhere.

But one way they could appear elsewhere is via compilers that generate plain JavaScript, and the latest player is TypeScript. Why is this a good fit? Because, with a very thin layer of helper code, generators replicate the functionality of C# 5’s async/await, and if that was a good idea for C# and Silverlight, it’s got to be a good idea for TypeScript. (The downside is that the auto-generated JavaScript would not be very readable, but I can still dream that this will happen…)

My old post is somewhat messy because I was unaware of jQuery.Deferred. If we bring that into the mix, to serve as the JS equivalent of Task, then things get really nice. To wit:

async(function() {
  
  // sleeping inside a loop
  for (var n = 0; n < 10; n++) {
    $('.counter').text('Counting: ' + n);
    yield sleep(500);
  }

  // asynchronous download
  $('.downloaded').text(yield $.get('test.txt'));

  // and some more looping, why not
  for (var n = 0; n < 10; n++) {
    $('.counter').text('Counting: ' + (10 - n));
    yield sleep(500);
  }
});

In other words, by passing a function to something called async, I can use the yield keyword in exactly the same way as C# 5’s await keyword.

The yielded values are just jquery.Deferred objects – or rather, they are objects that contain a function called done, to which a resulting value may be passed (at some later time). So the implementation of sleep is straightforward:

var sleep = function(ms) {
  var d = $.Deferred();
  setTimeout(function() { d.resolve(); }, ms);
  return d;
};

By calling resolve, we trigger any registered done functions. So who is registering? This is what async looks like:

var async = function(gen) {
  var result;
  var step = function() {
    var yielded;

    try {
      yielded = gen.send(result); // run to next yield
    } catch (x) {
      if (x instanceof StopIteration) {
        return;
      }
      throw x;
    }

    yielded.done(function(newResult) {
      result = newResult; // what to return from yield
      step();
    });
  };
  gen = gen(); // start the generator
  step();
};

So async calls the function passed to it to get a generator object. It can then repeatedly call send on that object to pass it values (which will be returned from yield inside the generator). It assumes that the objects that come back from send (which were passed to yield inside the generator) will have a done function, allowing us to register to be notified when an asynchronous operation completes.

Note that async could use some further complication, because it currently doesn’t deal with exceptions (note that the try/catch block above is merely to deal with the strange way that generators indicate when they’ve finished). But generators have full support for communicating exceptions back to the yielding code, so it should all be do-able.

And that’s all there is to it. You can see the example running here:

http://earwicker.com/yieldasync/

… but only in Firefox, of course.

TypeScript reactions

Miscellaneous

Great implementation of modules. I use commonjs on the browser side (yes, the non-asynchronous module pattern), so the default suits me fine.

Love the => syntax of course, and the fact that it fixes the meaning of this within lambdas. No need to manually copy into a that variable.

Type system

… appears to be better than that of C#/CLR. Structural typing is clearly the right way to do it. It makes much more sense than nominal typing. The dumbest example of that in C# is the strange fact that two identically-signatured delegate types are not assignment compatible. Well, in TS, types are compared by their structures, not their names, so that problem goes away. And the same for classes and interfaces. If a class could implement an interface, then it already does. When they add generics, it’s going to be excellent.

I wonder if they’ve given any thought to the problem of evolution of interfaces. Default implementations of methods (this could be implemented by having the compiler insert some checking/substitution code at the call site).

Async, await, async, await, async, await…

The really big missing ingredient is continuations. They’ve just added this to C#. Clearly it is just as important for browser-based apps, if not more important. And consider the fact that server-side JS at the moment is roughly synonymous with node, which is crying out for continuations in the language. From a Twitter conversation with Joe Palmer, it’s clear that the TS team currently places a big emphasis on the clean appearance of the JS output. But I think they’re going to have to twist the arms of the IE teams and get source map support working, and then they can butcher the JS as much as they like without hurting the debugging experience.

The Dreaded C++ Comparison

Theses days a C++ comparison from me is normally an insult (despite speaking ASFAC++B) but in this case, definitely not. C++ beat a lot of other languages simply by inheriting C. And JS is the C of the web, so TS has a good chance of being the C++ of the web, in the sense of becoming equally popular as its parent language, while probably not actually displacing it everywhere.

And at its best, C++ was a collection of extensions to C, a pretty ugly platform to start building on. To do that tastefully was a challenge, and (despite what people say in jest) C++ succeeded in lots of ways. Playing with TS, I see a similarly tastefully chosen set of extensions.

When you consider C++0x concepts, which were going to be (and may yet one day be) structural type definitions layered over the existing duck-typing of C++ templates, the comparison becomes even stronger. TS’s type system has a lot in common with C++0x concepts.

The Competition

A common criticism so far seems to be “Why not use (my favourite language X) as a starting point?” The answer, surely, is that it may be your favourite language but it’s nowhere compared to JS, which is now so widely deployed and used that it makes most other languages look like hobby projects! This criticism is correlated strongly with the idea that JS is a toy language, disgusting, revolting, etc., i.e. irrational emotional garbage. JS has a lot of stupid things in it, yes, we all know about {} + {} and [] + [] and {} + [] and so on, but I’ve now churned out a zillion lines of JS without ever hitting such issues. No language is perfect, but is JS useful? Yes.

In fact, the main competition faced by TS is therefore JS. This is why I think TS needs to do some heavy lifting (such as continuation support) besides a mere layer of sugar and type checking, in order to become a compelling advantage over JS itself.

And then there is CoffeeScript. Syntax just isn’t that important for most people. Semantics are much more important. It’s no good that your code looks really short and pretty if it takes you just as long to figure out what it needs to say. By the addition of static typing, TS holds out the hope of genuinely improving productivity. (With continuations it could asynchronous event-driven programming a lot easier too.)

Oh and there’s Dart. I can’t even begin to understand what Google is thinking. It’s not compatible with, nor discernibly superior in any way, to anything that already exists. It’s just their version of the same old stuff.