Archive for March, 2015

Acquisition Machines

March 29, 2015 4 comments

This is off-topic from my usual technology geek focus, but it was prompted by reading a blog post Never Invent Here: the even-worse sibling of “Not Invented Here” by Michael O. Church, which reminded me of some things I’ve observed.

My experience of Never-Invent-Here has fortunately been brief and infrequent, and of a specific kind, but I know exactly why it happens and how to recognise it and what to do about it. It’s kind of inevitable.

When your exciting, innovation-happy employer has reached a kind of peak, and has a bunch of customers paying a steady revenue stream, and there’s no more inventing to do, they get acquired by what I call an Acquisition Machine (AM). These serve the opposite purpose to a VC, in that they get in at the top floor instead of the ground floor. They come in at the end of the story. They are 100% risk averse.

An AM has a large-ish technical staff, all former inventors from the mature companies they’ve gobbled up (having fired all the other staff). They never intentionally invent anything in-house. They almost never hire anyone new. The tech staff are only there to put out fires in old products for old customers (only the ones covered by maintenance contracts – that existing stream of revenue is the entire focus of the business).

So you may find yourself as one of those technical staff. You basically work in a museum now. The only projects that come up are “consolidation” efforts: making several old products look like one. In fact the management may just be inventing busy-work to keep you distracted during the down time. That’s cool! You can pitch approaches that involve the latest technologies, get skilled up. Play-act at inventing for a while.

Look at the world from the AM’s perspective. They’re terrified, risk-averse owners of shares that are never going to get back to what they were worth in 1999. They have a choice:

a) Bet on something invented in-house by one of these excitement-starved nerds we employ, and go to all the trouble of marketing it… OMG this sounds difficult and it probably won’t work… no way of estimating how much it will cost in total or how much we’ll ever make. Let’s not.

b) Buy an entire product-business-unit, something already proven, with customers already paying for it, something that is already clearly quantified with known cash inputs and outputs, whose outgoing management have already helpfully made a list of who you need to keep on and who you can safely lay off! Ready to plug into the machine and slowly drain down, generating cash to go on the pile, ready for the next acquisition.

Which are they going to go for? They are not in this for the excitement! It’s a slow way to grow, but it is a kind of growth. The target company’s old management are running out of exit routes, so they sell up cheap and so the AM does actually make a profit in the end. It just doesn’t do it by inventing anything. There’s no need.

I’m painting an extreme caricature of course – real companies may be somewhere on a spectrum from innovator to AM. Or they may be made up of parts with different degrees of maturity/stagnation. But the more stagnant it is, the worse it will be for anyone who wants to invent new stuff.

So if you are an inventor, and your employer gets bought, look for the signs that you’re stuck inside an AM. If you are, wait until you find a new opportunity elsewhere to start over, and then get the hell out. You can’t “fix” an AM. There’s nothing to fix. They’re just not the kind of business you want to be in. They have their own reason to exist.

React-ions – Part 2: Flux, The Easy Way

March 20, 2015 10 comments

The second of a two-part series about React:

Catching up on Flux has been an amusing experience. It’s like reading about a dance craze in the 1950s. Instead of “The Twist”, everybody wants to do “The Flux”. People are nervously looking in the mirror as they try out the moves. They write to the newspaper agony aunt, “I tried to Flux, but am I doing it right?” They want a member of the priesthood to bless their efforts with holy jargon, and say “You are one of us, Daddio!”

You’d think someone with a software project would rather ask:

  • Did my app work, in the end?
  • Does it perform okay?
  • Was it easy?
  • How much boilerplate crap did I have to paste in from blog posts?
  • Do I feel comfortable with the complexity or did it get out of control?
  • Do I know how to extend it further without creating a mess?

And based on their own answers to those questions, they should be able to figure out whether an approach was worthwhile for them.

My executive summary of Flux is: it’s a niche approach at best. For a lot of (maybe most) dynamic interactive UI development it’s not the right choice, because it’s error prone and unwieldy without providing significant advantages.

So how did I reach this conclusion?

It’s extraordinary that of the many hundreds of blog posts about Flux, hardly any try to explain it or justify it. They just describe it, without reference to long-established patterns it partly resembles, and without clarifying why it takes the trouble to deviate from those patterns. (Even worse, most explanations give truncated examples of how it might be used which don’t proceed far enough to demonstrate its intended purpose.)

The three primary sources of information I’ve drawn on are:

Derived Data

From the official site:

We originally set out to deal correctly with derived data: for example, we wanted to show an unread count for message threads while another view showed a list of threads, with the unread ones highlighted. This was difficult to handle with MVC – marking a single thread as read would update the thread model, and then also need to update the unread count model. These dependencies and cascading updates often occur in a large MVC application, leading to a tangled weave of data flow and unpredictable results.

Holy hype alarm, Batman! The part about the tangled weave is absolutely not justified by the scenario being described. This is a very familiar situation: some data set B needs to be computed from some other data set A. How did Facebook end up with a tangled weave and unpredictable results? Did they visit the wrong wig shop?

A sensible approach would be to come up with a pure function that accepts A and returns B. If that is an unworkable technique that causes “unpredictable results”, then someone needs let the applied mathematicians know. Then hopefully they can break it gently to the pure mathematicians who will then need to rebuild their entire subject from scratch.

For an example of something that does this right, look no further than a React component, which has state and props, and if either of those changes then the render function is evaluated to generate a complete new virtual DOM tree – relevant portion of the video.

Key lesson: first, try writing a pure function. If the performance is unacceptable (in 99.9% of cases, it’ll be fine) then consider alternatives.

In the video there is an example of how Facebook chat feature got more complex as it evolved. The code you can see growing on the screen is a function that runs every time the user has a new message – it’s effectively a handler for that event. They interpret the event by dishing out modifications to several different parts of the UI that may or may not be interested in what happened, depending on their current state.

They’re right – it was a very complicated way of doing it. They weren’t using pure functions. They should do what React components do: update a single definitive model of data (the state from which everything else can be computed), and then let the other components know that something has changed (no need to be specific), so they can all recompute all their data from scratch as a pure function.

In this case, that means keep all the messages received so far on a list. When a new message arrives, add it to the list and notify anything that needs to update.

It’s a common reaction to think how wasteful and inefficient it is to do that. But the second half of the video (that half that is not about Flux) is devoted almost entirely to dispelling that belief, as the React DOM reconciliation approach assumes that there will be an insignificant cost to recomputing the entire new virtual DOM every time anything changes.

In this case, rather than going from component state to virtual DOM, we’re going from list-of-all-messages to (for example) count-of-unread-messages. The functional-reactive approach here is to scan the array of message objects and count how many have a boolean property called read that is true. On my notebook such an operation is too fast for the JS timer resolution for any realistic number of messages. For a million messages it takes 18 milliseconds.

The Root of All Evil

The problem here, as so often, is premature optimisation, which is the idea that you can achieve “high performance” by doing everything the hard way. It’s simply not true.

The single most important quality software can have is malleability. It must be easy to change without breaking it. This leads to high performance software because the best way to achieve that is to measure the performance with a profiler and make careful, valuable optimisations only to those specific spots that you have found to be genuine bottlenecks. Easy to change means easy to optimise.

If you lean heavily on pure functions this will be a huge help to you as you apply performance tricks, because you can use caching very easily. And a clear distinction between mutable and immutable data is also important because it makes it easy to know when you need to clear the cache. Again, React components demonstrate this perfectly.

The Flux approach

Instead of radically simplifying and using pure functions like React, the aim of Flux is to stick with the difficult, fine-grained state-mutating approach shown in the video, where each time a message arrives, you run a very imperative set of code that mutates this, mutates that, mutates something else, in an attempt to bring them all up to a state that is consistent with what is now known.

In other words, if you want to keep doing it the hard way, Flux might just be the approach for you.

Here’s the simplest diagram:

Those arrows are effectively function calls, but via callbacks. So the thing on the right has registered a callback with the thing on the left, so that left can call right, and the influence of an action ripples through the layers.

  • An action is a plain JS object tagged with string type property (someone loves writing switch statements!) representing a request to update some data. Think of it as an abstraction of a mutating function call on your data model. As well as type it can contain any other parameters needed by the notional mutating function.

  • Having constructed an action, you pass it to the Dispatcher, which is a global singleton(!). The dispatcher has a list of subscriber callbacks, the subscribers are known as “stores”, and they are also global singletons(!!). The dispatcher loops through the stores and passes the action to all of them.

  • Each store’s subscription callback has a switch statement (hello!) so it can handle specific action types. It makes selective mutations to its internal (global singleton) state according to the instructions in the action, and raises its own change event – each store is an event source.

  • A view is a React component that subscribes to one (or maybe more) stores in the traditional way, i.e. as an event sink. Not all components have to do this. They speak of “controller-views” (two buzzwords for the price of one) that are specific React components that take care of subscribing and then use props to pass the information down their subtree in the standard React way.

So actions carry instructions to mutate data, and they make it as far as the store, where they cause data to be mutated. The last arrow is slightly different: it is not an action being passed. It’s just a change event, so it carries no data. To actually get the current data, the subscribing React component must call a public method of the store to which it is subscribing.

The point of all this, from the web site:

Stores have no direct setter methods like setAsRead(), but instead have only a single way of getting new data into their self-contained world — the callback they register with the dispatcher.

Why are stores banned from having their own setter methods? Because otherwise it would be possible to update their contents without also notifying all other stores that need to update in sync. That’s how the “derived data” problem is to be solved. If you only had one data store interested in various actions, there would be no point to any of this. (So please, if you’re thinking of blogging on this topic, remember to include in your example several stores that respond to the same actions so they can make corresponding updates to remain in sync.)

The examples

There are currently two examples in the github repository: TodoMvc and Chat.

TodoMvc has a problem as an example: it only has one store. This means it doesn’t actually have the problem that Flux is intended to solve. If there’s only one store, there’s no need for separate actions that go via a dispatcher to let multiple stores listen in. It could just have a store with ordinary methods that mutate the state of the store and fire the change event.

Chat has three stores, and is based on the Facebook chat scenario covered in the talk, so it’s got potential to be lot more applicable and illuminating.

In Chat, the three stores are:

  • MessageStore – a flat list of all messages across all threads
  • ThreadStore – a list of threads, with only the last message in each thread
  • UnreadThreadStore – an integer: the number of unread messages across all threads

The last one is more than a little ironic: if you look closely at the video, they were originally responding to events, and so decrementing/incrementing the unread message count. But in the Flux Chat example, even though they’re demo-ing a framework that is based on events so it can do exactly that kind of minimal mutation of the existing data, instead they’ve written the example so it recomputes the count by looping through all the messages (at the moment it doesn’t even cache the count).

If you’re going to be recomputing from scratch like that (and why would’t you?) then the strict action-dispatching approach of Flux is not actually going to be serving any purpose. It’s just ceremony. You could just have primary stores that store data; they’d have simple methods you could call to make them (a) mutate their data and (b) fire their own change event. Then you could have secondary stores that recompute their data in response to change events from other stores (both primary and secondary).

The UnreadThreadStore also gives us a demonstration of a dispatcher function called waitFor. This gives stores control over the order in which stores handle actions, essentially by telling the dispatcher to run specific stores’ action handlers synchronously for the current action. The reason a store will do this is because it wants to read data from those other stores, and it needs to do this after the other stores have updated their state.

It would make more sense for it to listen to the other store’s change event. A project called Reflux suggests doing exactly that.

I’ve seen discussions where people claimed that stores listening to other stores’s change events, perhaps through several layers, is the kind of “tangled weave of data flow and unpredictable results” that Flux is trying to avoid. But it’s just not. A tree of chained event handlers is a fine example of clean composition. If the arrangement is immutable (once constructed, a listener cannot be switched to listen to a different event source) then infinite loops are impossible.

The mess Facebook originally experience was not due to chained event handling, but due to disorganised fine-grained mutation of lots of different states in a single event handler.

Through the use of waitFor, Flux effectively does have stores listening to other stores change events. But it’s worse than that, because notice how in the UnreadThreadStore it has to listen for the two actions that it knows will cause the ThreadStore to change its data:

case ActionTypes.CLICK_THREAD:


If someone changes ThreadStore so it responds to a new action by mutating its state, that someone will have to check all the other stores to see whether they also need to respond to the new action, because these other stores may depend on the state of ThreadStore which is now changing more often. If the other stores just listened on ThreadStore‘s change event this would not be necessary. It would be an internal detail of ThreadStore, and the rest of the system would be isolated from that detail. By following the Flux approach, you are encouraged to spread around knowledge of how every store responds to actions. The intention is to maximise “performance”, but to reiterate, React itself does it much more simply: if any state or props of a component changes in any way, the whole component’s virtual DOM is re-computed by a single render function.

And to be clear, there is only a purpose to any of this if you are doing fine-grained mutation of stored state in response to the same actions in multiple stores, which UnreadThreadStore clearly doesn’t do.

So enough about UnreadThreadStore. How about ThreadStore and MessageStore? Again, they are peculiar. If you run the Chat demo site, you’ll notice that there is no page where you can see all the messages regardless of the thread they are in. What you see is a list of all the threads, and the messages in the currently selected thread.

It’s strange therefore that MessageStore maintains a list of all messages across all threads. It’s even stranger that it then has a function that, on demand, filters that list to get a list of just the messages for one thread! Again, this is the right way to do it, but it makes a mockery of the action-dispatch approach.

So ThreadStore is our last hope, and it delivers! It is actually another store that responds to some of the same actions as MessageStore and mutates its own data. Hurrah! But again, something really weird has happened. There’s a function getAllChrono that recomputes, from scratch, every time it is called, a sorted list of all the threads. And that’s the function that the associated React component calls so it can display the list.

Let’s consider some simpler alternatives:

  • One store that stores all the messages in a list. When you want to know the list of threads, scan through all the messages and gather that information on the fly, ditto the count of unread messages. These can be methods on that store.

This would probably be fine, even with absurdly large numbers of messages. But it might be both clearer and more understandable (as well as “faster”, though not meaningfully) this way:

  • One store that stores threads, which each have a list of their own messages. When a new message arrives, find the thread object and add it to that thread’s list of messages. This means you can now update and keep cached aggregate information about a thread. When you need to recompute it, you can scan just the messages for that thread.

But there’s a problem with both of these approaches: they don’t demonstrate Flux at all! They can have ordinary methods that mutate the single “source of truth” data. No need for actions or dispatchers.

It is true that both Todo and Chat are contrived examples – in fact there is a comment to that effect on this issue. And so we can expect there to be some unrealistic usages; they wanted an example that was simple to follow but which exercised all the key APIs.

However, this does mean that the creators of Flux have yet to provide an example that needs Flux. And in the various 3rd party examples I’ve looked at the situation is typically worse, in that most don’t even have multiple stores.

What do I conclude from this? I’m openminded enough that I would still be interested to see an example that really does something that would genuinely be harder to accomplish (and evolve further) without action-dispatching. But I suspect it would be a very niche, unusual application.

Categories: Uncategorized Tags: ,

React-ions – Part 1: Mostly Great

March 14, 2015 3 comments

The first of a two-part series about React:

I’d been planning to leave React well alone until it settled down a lot more. But over the last week I’ve started idly playing with it while travelling and waiting around, and getting more and more into it. It’s been dividing opinions for over a year now – but then, they let just anyone post on the Internet, so it’s full of idiotic opinions, right?

A work in progress

Turns out I’m not that late to the party. React’s version number starts with a zero, which under semantic versioning means “Anything may change at any time. The public API should not be considered stable.” The React team is taking full advantage of this early stage of development. They are not totally ignoring backward compatibility, but they are making trade-offs, e.g. if they can be backward compatible for code that uses JSX, then it’s okay if they break code that doesn’t use JSX. And yet JSX is supposed to be optional… But this is fine. Some parts of the API are necessarily more stable than others. They’re learning as they go, and one thing that’s gradually influencing them is the importance of stating (and controlling) which things need to be immutable. Every version seems to make an advance in that respect.

On the abandonment of external templates

As a heavy user of such templates, no complaints from me on this. In Angular and Knockout we add extra attributes to standard HTML, and the attributes are themselves a kind of embedded DSL in the HTML. The theory is that this means that the view or “presentation layer” is written in a high-level declarative language, so it can be maintained by a non-programmer. In practise this is unworkable. A template with bindings is fragile against modification by a non-programmer. You really have to know what you’re doing before you touch heavily template-ized HTML. It only appears clean and simple in the most unrealistic examples.

An external HTML template may appear superficially to be “separate” from the view model it binds to, but in reality it is intimately connected to it, having a one-to-one dependency between tags and bindings in the HTML and properties in the view model. And this means that the appearance of separation is unhelpful rather than helpful.

So this is all music to my ears. I’ve long thought that technology layers are overused as a way to carve up systems. Accordingly during my first experiments in large-scale JS app development, I rolled my own library that built very formulaic CRUD-like UIs out of what I called “schemas” (these were actually JS arrays). There was no HTML template in this system. Instead there were “types of control”, such as integer, date-time, etc. and you composed them to make a “record editor” that was self-persisting to JSON. It was crude but adequate. I liked that it lent itself to modularity, and let me add new whole capabilities in one vertical slice that cut across several technology layers.

Shortly after that I got enamoured of Knockout, which emphasised having a separate view (HTML+bindings) and view model (JS+observables). But I rapidly realised that what I very often wanted was a way to build UIs out of components, so I wrote my own custom bindings to achieve this, based again around the idea of a “control”, which is a view model with a built-in HTML template. Knockout 3.2 has since added its own support for components. However, it encourages you to register components into a global namespace so they can be referred to by name in HTML templates. This cuts across any module system you’re using to organise your code; your whole app is one big namespace at the component level.

React components don’t have this problem. Everything is JS, and so it can build on JS scoping and modularity. There is no global behind-the-scenes module-ignorant namespace of registered plugins. In your render function you may refer to another component by name, but it’s just the name of a JS variable that has to be in scope, e.g. imported from another module via require.

Seriously, I’m all over this like a weird rash.

Static typing

Types are taking over JS, kick-started by TypeScript, which is growing rapidly in both user base and features, is already solidly mature and effective, and is prompting further research efforts such as Facebook’s Flow and Google’s SoundScript.

This is another area in which React has an advantage by doing everything in JS and not breaking out into external HTML templates. Checking static types inside the binding attributes in an HTML template requires compile-time understanding of how all the kinds of attribute work. Not to mention special tooling to get design-time feedback, auto-completion in the editor. None of this is a problem for React.

Well, almost. The problem is there’s this strange thing called JSX.


Facebook’s own flavour of TypeScript, Flow (also not really ready for production), has built-in support for React’s JSX syntax (also from Facebook). What a fortunate coincidence! I think this is what they call synergy.

There have also been a couple of efforts to graft JSX support into a fork of TypeScript. But is this even necessary?

I find JSX to be a mere gimmick and distraction, with no discernible value. Indeed its existence may harm rather than help React adoption, because it’s so egregiously unjustifiable. Its only purpose is to look eye-catching in code snippets, providing a visual motif for people to mistake for the essence of React.

The story goes like this:

render() {
    return <div className="foo"></div>;

generates (at the moment, anyway):

render() {
    return React.createElement("div", { className: "foo" });

So at first glance JSX appears to be achieving significant boilerplate reduction. The React docs point us to a built-in shorthand for non-JSX users:

render() {
    return React.DOM.div({ className: "progress" });

Better, though still not that short. But if we’re going to be using div and span a lot, we could just import them our namespace:

var div = React.DOM.div,
    span = React.DOM.span;

Now the “verbose” version is:

render() {
    return div({ className: "progress" });

i.e. not at all verbose, almost the same length as the JSX version, with the advantage of being just plain JS.

In any case, these simple examples are misleading. In a realistic example of a component that actually does something useful there will be conditional elements (shown or not depending on this.state) and repeated elements using Array#map, etc. These parts have to be written in JS, and it’s a sensible React principle that there’s no point inventing a second syntax for them.

So often at least half the code in render is not expressible in JSX anyway. I find that staying in one perfectly adequate syntax is actually more helpful than switching back and forth.

And as for succinctness, when you’re rendering to DOM elements it’s quite common to need to throw in some purely structural wrappers that only have a class attribute, which in React has to be written as className. So what if you used factory functions that could optionally take a string and expand it into an object with a className property?

render() {
    return div("foo");

Uh-oh. Way shorter than the JSX version!

So I threw together a library to make this effortless, but as I was using a rough cut of it and finding it super convenient, I naturally wondered: given how handy this is, why doesn’t the React library itself support passing a string instead of a properties object?

I submitted a pull-request to do just that, but they turned it down. I admire their desire to not absorb into the core things that can be added externally, which is a great general principle to adhere to. But I wouldn’t have applied that principle in this case; the simple sweetness of the string-as-className shortcut is undeniable; so much so that now I’ve thought of it, it feels like an accidental omission that the core library doesn’t already support it.

It’s clear that React would be technically stronger without JSX, but it may be weaker from a marketing perspective. JSX is something concrete and weird-looking that people can focus on as the Chemical X in React, even though that is fundamentally misleading. So there’s the classic marketing-vs.-reality tension.

Events, Observables, Dirty checking etc.

A view has to update itself when the data in the view model changes. There are broadly two ways to do this:

  1. Dirty checking
  2. Observables

Angular uses dirty checking: it keeps a snapshot of the model data. After various events likely to coincide with data changes (e.g. button clicks), Angular compares the model data with the snapshot to find out what has changed.

Pretty much everything else uses observables. An observable is the combination of a value and a change event that fires when the value changes. Obviously you have to call a setter function to set the value, so that the change event can be fired. What if the value is a complex object and you tweak a value inside it? That’s no good – you’re bypassing the mechanism that fires the change event. So a good principle to abide by is to only store immutable objects in observables. The whole observable can be mutated, but only by completely replacing its whole value via the setter function.

React is interesting because sort of uses both these ideas, in very limited ways.

On the one hand, it does dirty checking, but not on plain model data; it holds a snapshot of a description of what the state of the DOM should be. This is a fantastic simplification compared with Angular, because React can make minimal updates to the DOM based on a fixed set of rules.

And on the other hand, every component has an associated observable called its state. We know it’s an observable because we have to call setState to change it and the documentation warns us not to mutate it any other way. On the other hand, there’s no public API to subscribe to a change event. The component itself is the only thing that directly subscribes to it.

There are small weaknesses to the React component API. The central one is that the current state is public property of the component class, so the fact that you’re not supposed to modify it directly is not self-documenting: there’s a setState but no getState.

And maybe there shouldn’t be either of them. According to the docs there are situations where you aren’t allowed to update the state. So might be better for each of the component methods to accept parameters providing the current state and – where applicable – a function to update the state. This would make it self-documenting w.r.t. which operations are allowed during a given method.

Tune in next time, wherein I confront the mysteries of Flux! What is it, really? And more to the point, what should it be?

Categories: Uncategorized Tags: