Exploit your successes

Our huge rebuild and redesign of guardian.co.uk was a successful project. Once it was done we could have slumped back, exhausted. But instead we used it as a springboard to start additional exciting (and arguably controversial) work. Primarily I’m thinking of the Open Platform, led by Matt McAlister. As I wrote at the time, getting that off the ground was a huge testament to the work and faith of many people in the company, particularly given that it’s a very intangible thing at launch.

Still, the Open Platform is built on top of the work in that earlier project, and in particular in that people recognised technology success and were prepared to back more.

Sustainable business needs sustainable technology

Much of our technology, particularly on the web side, is developed in-house. On the one hand that can seem expensive, but on the other hand it does mean we have the ability to take in the direction we see fit. A particular example is the cost of scaling guardian.co.uk. It does cost more money to serve more users, but it doesn’t cost as much as it might because we have control of the technology.

Broadly speaking since we own the technology we can steer it in line with business needs much more readily.

Right-levelled decision-making builds trust

Let’s make one thing clear first: “Right-level” is absolutely not an acceptable verb phrase, and I only used it because it fitted onto the slide better. Now, to the substance of this…

Any reasonably sizable projects are approved to go (or not go) by a group of directors and other senior staff, and they do this at a regular meeting on the basis of information provided by the technology team working with many people around the business. This information of course consists of timescales, costs and benefits, but also what other projects are in progress and waiting in the wings.

This may seem obvious, but the consequence is that in the event of heavy cost control (which is what happens when the economy takes a dive) that cost control is exactly in line with project expenditure, because it’s the same people who are making both kind of decisions. It should be said that this is supported by the Agile principle of delivering value, and delivering frequently, because any project can start demonstrating its value early.

Openness reduces costs and provides options

The Guardian has been open not only with the Open Platform, but also with our full content RSS feeds. Applications have been built on the latter simply because they are a known format and, with full content, have huge utility. By having open full content RSS feeds we have a means of others using our content, and if they (or we) choose that option there are no internal integration costs.

Transparency builds trust

We do a lot of internal reporting: what features we release, our current bug count, project costs, and so on. When times get tough people start asking what value you’re adding to the business, and where all that money that’s being spent on your team is actually going. Because of all our reporting it’s relatively easy to open up our books to the consultants with financial questions, and then we can have an intelligent debate about the value of that expenditure. But that’s fine, and something we tend to welcome, because it’s part of the culture that Agile fosters: within the team we debate every week the value of what we do — that’s what Agile planning is all about — and Agile retrospectives are all about seeking improvement.

And finally…

I recognise that saying all this may be a hostage to fortune: the credit crunch took hold only about a year ago, and we’re not out of the woods yet. But for now I can look back on some of the things that have gone well, and if we emerge from this economic downturn with only a few cuts and bruises then it will be in no small part due to honesty and openness creating trust, which in turn leads to healthier decision-making.

The Eigenharp and the Open Platform

The Eigenharp is a new kind of device. It’s a musical instrument with several patents already filed, and incorporates 132 keys with three-colour LEDs and a breath pipe. The keys operate in three dimensions (so you can exploit, say, pressure, pitch bend and filter effects with one finger) and are sensitive to within one micron. So while it’s compatible with MIDI, that’s an incredibly out-of-date protocol compared to what you could do with it.

The Open Platform has been well-documented here and elsewhere: a full content API directly into the Guardian’s content database, plus a store of raw data on which our journalists base some of their stories.

Although they are clearly two different kinds of beasts a remarkable similarity struck me in that they are both not just new products, but really new products. There is no clearly defined place for them in either of their markets, and no predictable success path for either one. The success of each is dependent entirely on other people’s innovation. The Eigenharp’s success is dependent on musicians taking their creativity into new directions — no-one to date has written music for an Eigenharp. The Open Platform’s success is dependent on developers doing innovative things with the new data they have access to.

So how do you launch a product that’s really new? The similar steps I can see are…

1. The controlled buzz

Eigenlabs (the creators of the Eigenharp) and the Guardian (creators of the Open Platform) managed to seed a few select people with their product — musicians and developers respectively — and some small details leaked out. This clearly did a couple of things.

First, it helps you get the product right. I know from my conversations around the Open Platform that what might seem a good idea to one or two people close to the product can provoke strong negative reactions when you float it past an informed outsider. When you’re inside the company you can easily think too much and lose the big picture, so it’s good to get an informed but independent view before you go public.

Second,  creating a buzz protects you from a bit of unforeseen negative reaction — if people are excited about your product and want it to succeed they will forgive some minor mistakes. You can see Eigenlabs achieved this with this low-fi video of two guys playing the James Bond/Moby theme on YouTube. Eigenlabs didn’t create this themselves, but they let “close friends” do it, and you can see the reaction in the comments underneath: “What the hell!! This is awesome”, “where can I buy one of these?”, “dear god this is epic” and so on.

The benefits of this are well explained by Lance Ulanoff at PCMag.com. Here he compares the lacklustre launch of the Motorola Cliq with the hype from the January 2009 announcement (six months before the launch) of the Palm Pre:

Speaking of Palm, it has a good bit in common with Motorola right now. The Pre (and now Pixi) is its hail-Mary pass. If the new phone and WebOS platform fail, Palm will be done. Back in January, however, Palm ran the best product rollout event I have ever attended. It perfectly conveyed the company’s excitement and all that is good about the Pre. I remember tweeting the event with mounting excitement. Today, I tweeted the Motorola event with mounting confusion.

What Palm did in January was give the Pre a good hard shove off the shore. Those waves of excitement produced a good six months of positive press. Eventually, Palm and the Pre hit some rough waters—a shipping delay, the slow delivery of the SDK and tiny list of apps that has yet to grow. Still, that first day set the tone.

Then after the buzz is…

2. Openness at launch

There’s much that doesn’t need to be said about the launch event, but one thing that struck me about both the launch of the Eigenharp and the launch of the Open Platform was a common ethos: the honesty and openness of the presenters, and the openness of the products.

At both events the “real users” (musicians/developers) were on-stage demonstrating their early adoption of the product and open to questions from the audience. These are the kinds of people who most marketing and PR people would lock in cupboards at a press event for fear they arrive without the regulation press-on smile and go off-message. But when a product’s success relies on innovation and creativity from its real users then it’s important that intended audience hears an authentic voice.

Also, there is an openness about both products. Aside from the very concept of opening up content and data, the Open Platform gives a lot of latitude to developers, including (unusually) commercial use. Meanwhile Eigenlabs are open-sourcing their software. Again, because each product’s success relies on people taking it in unexpected directions it’s important for there to be as many opportunities as possible for that to happen. Otherwise its success is stifled.

And then finally there’s…

3. The follow-through

Well, it’s early days for both products. The Open Platform API is still in beta, but you can see some developments now, such as the launch last week of the Applications Gallery. It’s even earlier for the Eigenharp. Company founder John Lambert was asked what well-known musicians he’d like to see using his creation and he said that a number of high profile people do have the instrument but that he cannot name any names yet. Clearly some really interesting things are going to happen some time soon.

If a product really does depend on the creativity of others then you can’t — and mustn’t — be too controlling of what those people do with it. (When I spoke to him John seemed slightly apprehensive about the quality of some of the things people might produce around the Eigenharp; I bet Matt McAlister didn’t anticipate a swearword tracker coming out of the Open Platform.) But if you can’t control what others do, you can at least show some examples of what might be achieved, and that’s what the follow-through is doing with each of these products.

1-2-3, easy as A-B-C

Now if any marketing people have stumbled across this blog post I wouldn’t be surprised if they were horrified by the naivety of these observations — all of this might just be first grade A-B-C of marketing for them. But I find this fascinating. As a tech person it’s very easy for me to focus on my own role and not spend much time wondering what challenges are faced by my colleagues in other departments. Here are two products whose paths to success are unusually dependent on the unknown and whose stories will be well worth watching. By looking at the communication that’s gone on around them the significance of other people’s role becomes much more apparent. Technology success is about much more than successful technology.

It became apparent to me that much digital media advice for non-technical people really needs to come with a very strong health warning. Here’s one taken from a prescription medicine I’ve just retrieved from my bathroom. It needed only very light editing and seems quite appropriate:

REMEMBER this advice was prescribed only for you. Only a digital professional can prescribe it. NEVER give it to someone else to use even if you think their project is similar.

Looking at Ryan’s blog the advice is for “dealing with developers and choosing a platform for your [Knight News Challenge] project”. As a previous winner of the KNC he should know what he’s talking about here. Unfortunately in the process of being prepared for the KNC blog its intent seems to have been overstated. It is not, unfortunately, advice on “how to manage technology decisions in 5 easy steps” — which is a shame, because I and a lot people I know could do with easy answers to hard technology decisions. Nor does it provide guidance on “How to hire developers”, as the intro suggests, unless you’re hiring one or two developers specifically for the KNC.

This may sound flippant (perhaps, admittedly, because it is) but there is a serious point here. I suspect there are going to be many more people following the KNC than there are participating, and I think a lot of those followers will be people in the process of embracing digital media projects for the very first time, and will be looking for guidance. Perhaps they are just starting on digital projects within their current traditional media companies. They will need good guidance so they can make a reasonable success of their early projects and so feel confident about getting more involved. The media industry needs those people to be successful.

But taking the right advice for the wrong project will lead to problems. Here are some instances of Ryan’s advice and where your (or a colleague’s) situation might require something different…

1. Learn a little bit about any one Web framework, standard, or programming language

If you’re leading a 2 or 3 person team then it’s a good idea to understand the technologies your team members are using. But you may find yourself in a slightly different position in a larger team. In that case the level of conversation you need to have with people will be different. If the technology being used is fairly complex then deep-diving into a particular web framework or programming language is going to be far less useful than having, say, a good overview of the technologies involved and why they’re important — if only so that you can ask appropriate questions and deal with the answers appropriately.

2. Choose the people you want to work with and spend an ample amount of time telling them what you want.

I’d never disagree with the “ample amount of time” part — if you don’t dedicate a lot time to your project don’t expect it to go the way you want. But the “telling” part will only get you so far. It will probably work very well if you’ve become an expert in a particular web framework which is the basis of your project, and if you’ve hired someone with less experience there. But if your situation is different your approach should be different. For anything ambitious or complex or imaginative the development will be an evolving two-way conversation, not a one-way monologue in which you’re telling someone what to do.

So, lots of good advice on the KNC blog for participants of the Knight News Challenge. But if you’re sitting on the sidelines you should not mistake the demands of the Challenge as being the same as the demands of whichever project you happen to be working on next. Get advice that’s been prescribed specifically for you.

This information has been provided free of charge under the National Digital Health Service.

Lean software development is a fascinating step on from Agile, but its history is in manufacturing cars and to date I’ve only been reading up on lean manufacturing, not lean development. There are two reasons for this. First, I’d feel much more confident about lean if I know the background and reasons for things — in other words, I want to derive lean software development from first principles. Second, I was rather cautious, if not downright sceptical, that something can be translated from manufacturing to software. So if there’s any translating to be done then I want to do it myself — or at least understand someone else’s translation of some of the processes sufficiently to trust the rest of their translations.

This, then, is some of what I’ve learnt about lean working so far, based on manufacturing and with my own projections onto software development. It comes with the caveats that it’s partial and not informed by 99.9% of the work Mary and Tom Poppendieck, the acknowledged leaders in the field of lean software development. But it is informed by my reading of two fabulous books: The Toyota Way by Jeffrey K. Liker (TTW), and Lean Thinking by James P. Womack and Daniel T. Jones (LT). My interest in cars registers on no scale visible to the human eye, but learning about car production from these authors has been quite eye-opening.

In this post:

• Lean challenges common sense;
• Lean is about producing value;
• Lean is about the value stream;
• Lean is all about eliminating waste;
• Flow;
• Pull;
• Lean promises the seemingly-impossible.

Lean challenges common sense

The “common sense” way of putting a complicated thing together is to divide the manufacturing process into pieces (the engine, the chassis, the interior, the shell), create a specialist team for each area with their specialist tools, then have them work in a long production line. The result is a very long production line with each product moving from one specialist team to another as it’s built up from beginning to end. If you’re going to be really efficient then you might batch up all the products of one type and send them through in one go, then switch to another product of another type and send those through.

Lean says this is wrong. It’s not trying to be deliberately challenging or quirky or iconclastic just for the sake of making a name for its founders. Rather, it says this is wrong because it generates hidden problems. If we look at the very long production line then every one of the products on that line is in an incomplete state and hiding a multitude of problems. And if you were to implement that single-product-batching it would be even worse.

Lean says the most efficient way of doing something is to create it from beginning to end in a single pass. The single long production line transforms into many, much smaller, teams that are each responsible for the entirety of the production of each product.

This is counter-intuitive. Did we learn nothing from Henry Ford? Well, the evidence — and there is plenty of it — says otherwise.

But I’m getting ahead of myself. Let’s start at the beginning…

Lean is about producing value

This is pretty unstartling and almost uninspiring. But it turns out that what constitutes value is often hard to define. First you have to ask “value to whom?” Often in large organisations people are set targets of something which is not the main goal of the organisation, and this means you end up optimising your processes for the wrong thing. For example, the paint team might be targeted on how quickly they can paint a car body before moving onto the next one. But that would be misleading, and might explain why there’s a retouch team further up the line to cover up the paint problems discovered later. A car company is not about producing painted bodies, it’s about producing cars, and focusing a team on painting ensures they miss the big picture.

The Guardian software team produces software, so it’s easy to see that’s where our value lies and we should optimise our processes around software. But on second thoughts I’m not entirely sure… Perhaps value should be measured from the point of view of the end user. The better we provide news and information the greater the value of what we do, and merely producing software is no good if it doesn’t benefit the end user. And when we do produce software it’s of absolutely no value until it’s released and public.

Lean is about the value stream

Once you’ve identified the value you need to map the value stream.

Let’s suppose our value is in producing software that the end user finds useful. The value stream runs from the point of idea inception to the point the user can use it. Mapping the value stream is the painful process of tracking exactly what happens to what and who, and how long it takes. This is painful because you need to be very, very honest with yourself, and map with your eyes wide open. In particular you need to make sure you track all the times something goes back for rework, and all the times the thing just sits there waiting for something to happen to it. It’s also painful because it means managers have to understand first hand what their team have to deal with — anyone who thinks they are above that due to demands of time or priorities is going to find this first step very difficult.

A typical Agile process has a backlog of work cued up before the iteration or sprint, and then hopefully delivered at the end. But there will be large gaps between a story being defined and actual development. If an iteration is three weeks then on average there will be a gap of 1.5 weeks between definition and development starting, and if we insist on any story being up to five days’ effort then a typical optimum scenario is that a single story is at least 60% non-work. This is relevant because…

Lean is all about eliminating waste

The primary purpose of lean is to eliminate waste, and I’ve not put it first in the list of things I know because we first need to define value and then map the value stream. Once we’ve done that we can tackle the waste.

Waste comes in many forms — well, seven forms according to lean lore. In manufacturing the most significant of the seven is unused inventory, which means parts that are sitting around in racks and warehouses waiting to be used. Unused inventory not only takes up unnecessary space, it actually masks problems. Here are some examples:

• the acres of spare parts mask the fact that part production is not in sync with the demand;
• the fact that different parts are overstocked by different levels means that it’s impossible to tell which ones are most and least over-produced;
• if a part is only used long after it is produced then quality problems cannot be captured and addressed in time — by the time the quality flaw is discovered there will be many similarly-flawed parts in circulation.

Inventory in manufacturing seems similar to Agile stories that aren’t being worked on: they’ve been specified but are sitting around waiting to be picked off the warehouse rack. And when they’ve been developed they might be sitting around waiting to be released — after all, until it’s released it’s of no value to the customer.

This is where we return to the example above of a story that’s 60% non-work — 60% of its time is just sitting around waiting. The goal is to compress this down to an ideal 0%, but not just because we want to do things quicker. It’s because having sight of something from beginning to end, and not lose sight of it for a second. It’s because being able to focus on something means information isn’t lost, and everyone’s expertise can be brought to bear on it in one pass. If that were to happen then less would be needed to be written down on the story card because the team wouldn’t have to suffer context switching. They could also apply much greater creativity to their work, because they would see exactly why certain things were and were not being specified and contribute alternative or additional ideas without causing confusion or risking repetition.

This idea of taking something from beginning to end in a single pass is called…

Flow

In manufacturing this is a big deal. It means rather than having your factory floor as a single production line you create small teams (”cells”) which are responsible for the entire production of each unit (car, lawnmower, etc). If you have huge lathes and paint machines and so on it’s a major change to rearrange the factory floor.

Not so difficult in software development, fortunately — we tend to just have desks and computers, though in any large organisation with centralised functions you always need to win the buy-in of other people.

However, flow comes with a new and serious responsibility for those involved: the cell as a whole is responsible for producing the goods, so they must work together to ensure regular and maximal output. Let me make that concrete…

At no time when I’ve been working with cross-functional teams (software developer, client-side developer, QA, etc) has there been the perfect balance of all roles; we could always have done with one more software developer, or an extra 0.2 of a QA, etc. Much of the time the imbalance is negligible (or quietly welcome), but sometimes it’s very noticeable.  And when a cell is working together on a single deliverable (the car, the lawnmower, the software feature) then it’s up to everyone to help each other. It’s no good the client-side developer producing more widgets to test if the QA can’t keep up. They need to ignore the traditional perceived boundaries created by job titles, reach across to others, and work together to regulate the output.

I said lean is about eliminating waste, and flow helps with that in a way hinted at earlier. Flow increases quality by allowing all participants to see the thing put together from being to end without interruption. This reduces hand-off time, reduces information loss, reduces relearning, and increases knowledge and ownership. The number of times an item needs to go back for rework is reduced, and if it does need to go back then the rework that’s needed is clear and therefore quicker.

Pull

Meanwhile, all this work needs to come from somewhere and that’s what “pull” is all about. The principle of lean is to only do what’s needed, and that means only produce something that is a direct response of a specific request, and only when needed.

The distinction is clear in car manufacturing. A car company’s marketing department will devise a special offer on a particular configuration (these seats, those mirrors, any one of the following colour combinations) and the plant will have to manufacture a whole lot of those particular cars ahead of time, but only with a guess (rather than a certainty) about what the demand will be. In the lean world a car is only produced in response to a specific customer’s specific purchase: customer goes into showroom, customer orders car, order triggers build.

That’s how pull works in relation to delivering the product to the customer. But pull also works in relation to building the product inside the factory. The old method is to keep hundreds of each part in store; the lean method is that a part is only provided to the worker when they need it: when they’re running low they signal the need for more, and it’s provided for them. This triggers a chain right back potentially to the supplier of the part, ensuring they are always delivering just enough, no more and no less.

The parallel flawed system in the software world is the product backlog. (We’ll ignore the even worse scenario of waterfall’s detailed planning up front.) Work is prioritised ahead of the sprint and waits to be developed. Requirements can change, even in that gap between prioritisation and development. The consequences aren’t as terrible in the Agile world as in the waterfall world, but it still causes problems: it disrupts the team’s schedule and of course all the effort that went into the planning of the now-deprioritised story is wasted. Even if the requirements remain constant the gap between planning and developing mean knowledge is lost, or conversations need to be repeated, or the requirements turn into mini-waterfall-style requirements specifications.

The lean software alternative is to prioritise the next story only when the team is ready to work on the next story. That means while the team is developing the current feature they don’t have much certainty about what’s coming next. Like the worker in the car plant they have to signal slightly ahead of time that a new story needs to be worked out. Then the Scrum Master/business owner/internal customer needs to get something ready so that when they become available the team can all get together, thrash out the details, and set to work again.

As a manager I’m uncomfortable with this: I can no longer know what’s going into an iteration when the iteration starts. But the definition of value is not “what a manager’s comfortable with”. I will need to find other ways to ensure we can be accountable to the rest of the business, and they will need to be ways which are closely aligned with end user value — and that can only be a good thing.

The combination of flow and pull doesn’t mean the team is only working on one thing at a time. But it does mean that everyone has an equal balance of work at all times. So if Alf’s development work gets passed over to Betty’s testing then Alf and Betty need to make sure that she is expecting to finish her current piece of testing at pretty much the same time as Alf finishes his current piece of development and passes it over to her. Keeping that flow even is really important.

Lean promises the seemingly-impossible

Lean holds out the seemingly-impossible promise of increased productivity and increased quality. But here are some numbers from the literature:

• The Puget Sound Naval Shipyard, time to prepare a repair document. Originally: 97 days. After a lean workshop and taking action: 26 days. (TTW, p.103)
• Lantech wrapping machinery, before and after lean. Production throughput time was 16 weeks, became 14 hours to 5 days. Delivered defects per machine was 8, became 0.8. Employee time per machine was 160 hours, became 80 hours. (LT, p.121)
• Porsche, production of the 911, before and after lean. Initial stamping to final shipping was 6 weeks, became 5 days. Inventory held was 17 days’ worth, became 4.2 days’ worth. Errors from work on the assembly line dropped 55%. (LT p.213)

I do have some concerns about lean software development, but they’re less about lean itself and more about bandwagon-jumping and doing things without really understanding the reasons. Regardless of that, it’s refreshing to find a new way of looking at what seem to be known problems, and making insights which you might not otherwise have found. It’s certainly something I’ll be spending quite a bit more time on.

In this article:

• What is JavaFX?
• The language (good)
• Development support (ugly)
• The user experience (bad)
• Conclusions and (inevitable) comparison with Flash

What is JavaFX?

JavaFX is probably best viewed as Sun’s answer to Flash, just as Silverlight is Microsoft’s response. So it’s a way of developing rich internet applications, or embedding fancy graphical thingies into web pages.

In some sense Sun might not have had any need for JavaFX. For example, take a look at We Feel Fine. This is an astonishingly beautiful and imaginative application, and though it might look like it’s been written as Flash it’s actually a Java applet.

So why did Sun feel the need for adding to Java? According to JavaFX’s creator, Chris Oliver, “its purpose was to explore making GUI programming easier in general”. In other words, the Java language is not really optimised for GUI development. In addition, the language needs a number of libraries. But all of this compiles down to Java bytecode and runs on the JVM, so JavaFX is really just Java with special add-ons to make graphical-oriented applications easier.

The language (good)

The most obvious initial difference with JavaFX is the language that comes with it: JavaFX Script. As implausible as it sounds Sun has combined the best bits of Javascript and Java, and come up with something really rather wonderful. Consider the following code snippet from my application:

This creates a thin ring, which is actually just a circle with an outline and no fill. It starts off being invisible (opacity 0). The variable ring was declared on a previous line and is initialised here. I think this simple piece of code demonstrates a number of really nice things about the language. First, the code Circle { ... };. This is a constructor, which I really like. It feels lightweight, like Javascript, and it’s got Javascript’s named properties. But like Java, JavaFX Script is a statically typed language, so ring is of type Circle and if I forget this later on in my development then the compiler will signal an error — that error won’t make it into production. Second, the most exciting addition: the bind keyword. This means that the formula that follows will change dynamically when any of its component parts changes. So the x co-ordinate of the ring’s centre is bound to bubble.x — the x property of the variable bubble, which is the x co-ordinate of its centre. Similarly for the y co-ordinates. Now I know that the ring will always be centred on the bubble: whenever I update the bubble’s position the ring’s position will change, too. As one of our team’s developers explained when I started telling him about JavaFX binding (in fact he didn’t even let me finish my sentence) it’s just perfect for a GUI-oriented language where you want the view to track the model. Another great feature of the language is its native handling of timelines. For example, here’s a piece of code I use to fade something in or out over a period of half a second:

This constructs a Timeline with a single keyframe (at the half second point), and JavaFX is told that by that point the opacity needs to be at the target opacity. On calling the play() method it will interpolate the object’s opacity value so that it fades in or out accordingly. Needless to say, you can have several keyframes, and interpolate several values simultaneously or in sequence.

You can also ensure specific actions at certain points. Here is some code which fades a message out, changes it, and fades it back in again. This mixes native timeline syntax with more usual JavaFX syntax:

You might also notice from this that functions are treated as a first class object: another very handy feature of the language.

Development support (ugly)

As you might expect from a Java extension JavaFX integrates well with Java: you can use any Java class or object in your JavaFX code. In my case I wrote a small physics engine in Java and then accessed it via my JavaFX code; it was trivial to do.

On the other hand, JavaFX classes and methods can’t be accessed from Java without a lot of jumping through hoops (you have to create an interface in Java, implement it in JavaFX, and then you can reference that interface in your Java code). Also, Java generics aren’t available in JavaFX, which is a bit of loss.

The JavaFX platform itself is evolving. Version 1.0 was released in December 2008. Version 1.1 came out in February 2009 and changed not only the underlying libraries, but also the language, so even your source code needed be revised. Most recently version 1.2 came out in May 2009, with similar changes and similar consequences.

Overall, these changes are a very good thing. Each new version contains not only extra front-end features (charting capabilities appeared in 1.2) but also performance enhancements and changes which ensure greater API consistency and extensibility. It’s interesting to see Sun take a different approach from Java: there the remit has been to preserve backwards compatibility at all costs, because there are so many applications deployed in it, but JavaFX is still in its infancy and breaking backwards compability is a significantly lesser evil than consistency and extensibility. Nevertheless, the feeling is of a platform that’s yet to mature, and my productivity was knocked back by one or two platform bugs whose equivalents just wouldn’t be be dreamt of in, say, Java. Still, JavaFX is clearly actively evolving, and most certainly for the better.

More disappointing is the poor IDE support. Inevitably, support for Sun’s NetBeans is being actively pursued alongside the evolving platform. But my first experience with JavaFX in NetBeans earlier in 2009 found it to be an imperfect environment, and I opted to switch to Eclipse — I decided I could live with more quirks in return for a more familiar working environment. And after some considerable hassle I got it working.

But Eclipse support really is poor: there are a few annoyances documented over on pliq.com, and I found a few more. Overall it meant that even typing code was a bit of a minefield, and that’s without refactoring support.

There is also the inevitable problem with unit testing graphically-oriented applications. I admit I resorted to the old fashioned develop-debug cycle, which was a little disappointing and inevitably frustrating. But I can’t quite conceive of a unit test framework for a platform in which there is so much visually-oriented code. I suspect it would mean a slightly different approach to coding. My brain can’t quite work out what, though.

The user experience (bad)

So developers have problems. Let’s pretend not to worry about them; their entire lives revolve around solving problems so they’ll find ways to solve those. Much more important is the experience of end users, and with Sun boasting that Java runs on “800 million desktops” (isn’t that more than the number of atoms in the universe…?) those people’s experience is paramount.

Well, if you’re running the right environment then the JavaFX experience isn’t at all bad. Unfortunately you’re very unlikely to be running in the right environment…

Sun have done an excellent job in making it simple and reliable to embed your applet into a web page: once you’ve configured your code you only need some very simple and clear JavaScript in your page. This JavaScript takes care of platform versioning, prompts the user for an update if necessary, shows a standard “Loading…” animation, and downloads and runs your code.

In the circumstances this clever; but the circumstances are far from ideal. For example, when did you last update your JVM? Probably not recently enough to have the latest JavaFX release, so your first use of a 1.2 application is going to prompt you for a very large download of the JavaFX 1.2 runtime. You’re also going to need to run a launcher applet from Sun, which then prompts you with a popup requesting you trust software from Sun. Anyone who wants a seamless user experience is going to take one look at JavaFX and run a mile.

Even if you’ve already got the 1.2 runtime and launcher applet the JavaScript embed code itself references a script on Sun’s website. That means your web page is always relying on Sun’s website being up, running and performant in order to load. For code on my own little blog that’s fine — Sun’s uptime will always be more reliable than mine. But if this is to be adopted by bigger players, such as my own employer, the Guardian, then the sysadmins responsible for page delivery time and reliability won’t be happy with another third party dependency.

And if you’re not impressed with that, then I hope you’re not an Apple user. Java support for Macs has long lagged behind that of Windows machines (I recall the Java 5 SDK came out over a year after the Windows release) and this shows again with JavaFX. In theory JavaFX runs on a Mac. In practice it’s a world of pain: it’s slow and rendering features such as antialiasing are dire. Experiments show that if you’re running a very high end Mac then it’s half-way approaching decent, but not actually decent.

As someone who spent a long time developing this on a Windows machine it was very disspiriting to see the fruits of my labour reduced to a syrupy judder when I wanted to demo it to someone with a Mac.

Conclusions and (inevitable) comparison with Flash

If Flash didn’t exist then my view of JavaFX would be different. But Flash does exist, it’s the dominant RIA platform — certainly within web pages — and so any view of JavaFX has to be assessed in that context.

I find it very amusing that Adobe and Sun have eyed up each others’ territory and each wanted a piece of the other’s action. Adobe has wanted to move out of the web page and onto the desktop, so they’ve introduced AIR, while ActionScript has gone all serious, ditching the JavaScript-like version 2 for the Java-like version 3, and encouraging a cavalcade of grown-up design patterns. Meanwhile Sun has left behind that heavyweight baggage of the Java programming language and gone all JavaScripty, with easy constructor syntax, type inference and treating functions as first-class objects. They’ve each adopted the other’s clothes, but changing clothes is only part of what’s needed.

For my money JavaFX Script is the superior language. It’s designed for graphics creation and management, while ActionScript looks like Java before generics. The code I’ve found online to draw shapes in ActionScript brings back nightmares from the early days of writing Java applets: get a graphics object, set some properties, start the filling, do the drawing, stop the filling… it’s unintuitive, uphill work. (What the hell is a “graphics object”, anyway?) With JavaFX you use a constructor like the one shown above.

Mind you, there’s a reason for this difference. Flash development is predicated on the use of some really well-developed proprietary, commercial tools to do the job: Flash Professional for designers and Flex Builder for application developers. Looking for an ActionScript Hello World program produces some entertaining results. Here’s one conversation in which the request for a Hello World app is met with incomprehension: can’t you just drop the text on the stage? asks one responder, who misses the point of a programming tutorial; I suppose you could use the trace function to print the text out, suggests someone else. Here’s an ActionScript Hello World tutorial which includes the instruction “Select the Text Tool…”. There is a purely programmatic tutorial which Andrew Payne created because “The other examples I could find all used Adobe’s paid tools, Flash Designer, etc.” — but that was written only in January 2009.

So Flash is deeply embedded into design houses, which have come to rely on some very mature tools in which programming is secondary, particularly when it comes to asset creation. And while this might make more traditional developers smile it does demonstrate the cultural chasm that seperates Flash from JavaFX. Designers demand a first rate user experience, or else their work won’t sell itself. That means Flash execution doesn’t suffer the popup bombardment of JavaFX. And it means Flash runs just as well on a Mac as on a PC. And it means Flash developers have become obsessed with performance in a way that JavaFX developers can’t even imagine, so there’s an entire mindset which has yet to be addressed. And it means component support in JavaFX is lacking where in Flash it’s commonplace. One example of this is the lack of a JavaFX preloader: with Flash it’s standard to have a progress bar as the application loads; with JavaFX the standard preloader is an animated image that continues to animate even if there’s a background error, and I can’t find any code which shows a progress bar. Another is the ease in which one Flash swf can load another; again, not readily possible in JavaFX.

Designers would also bring a huge benefit to JavaFX beyond setting high bars for user experience: they are the people who will make JavaFX applications look beautiful, and so attract others’ interest, and so start a rolling snowball of adoption. I think my own application is quite pretty, but it was quite different in the early stages. It was only when I studied and tried to emulate the work of the Guardian’s professional designers that it started to come alive: circles were a friendlier shape than rounded rectangles, the palette of guardian.co.uk was infinitely prettier than my original palette of browns, and so on.

Sun is not unaware that this cultural gap needs to be bridged. That will be one reason why they’ve created JavaFX Production Suite. But years of cultural awareness and workflow embedded into design studios’ daily operations will be very, very hard to reverse.

So my own thoughts about JavaFX reflect my own experience with it. My development experience was very positive: it’s a great language and a mediocre development environment with many teething troubles which will no doubt be ironed out over time. I had terrific fun with it, and getting fast results was very rewarding. But beyond the development experience the future seems more difficult. The user experience is very poor, with JavaFX throwing up barriers in front of the non-technical end user where Flash is invisible. And the leap from niche developer interest to general acceptance by design studios is just enormous.

Making JavaFX a real success is huge job for Sun. They’ve got off to an excellent start, with a language and runtime environment that allow for very quick development. But that, alas, is the easy bit compared to what lies ahead. I do wish them the very best of luck.

Over on Mashable, Vadim Lavrusik has a useful round-up of “12 things newspapers should do to survive”. It’s good to see a lot of thinking brought together, and nice that the Guardian’s last Hack Day was noted with approval. But if there’s one thing that rankles me it’s the seemingly glib assertion that newspaper companies should emulate Internet start-ups, and which features at number six on Vadim’s list. He says that “creating a startup-like environment that encourages innovation in the newsroom” is one way forward, and cites three sources.

First, start-up veteran Scott Porad on the problem:

Over an 8 year period, my last startup grew from a startup into a corporate environment with several hundred employees and layers of management.  For the last 5 or 6 years of that I felt like we spent 80% of our time planning and only 20% of our time doing stuff. [...] On the other hand, my current startup is the opposite — we probably spend 5% of our time planning and 95% doing.

Then Mike Briggs on how a newsroom can generate some start-up-like energy. And finally Ryan Sholin of Publish2, who thinks that you should “Make your newspaper function like a start-up.”

Somehow start-ups seem to be the panacea: not only do they make good stories, but it’s so easy to spot the successful ones: for instance Google started in a garage, and Microsoft was started by just a couple of people. And let’s not forget the many, many start-ups from more recent times. You probably remember Kevin Rose’s Pownce.com, which was successfully bought by Six Apart, which loved it to death; or tr.im, which was so successful at making URLs short, they’ve practically made them vanish. And let’s not forget the current media darling, Twitter, which is generating so much profit they’ve run out of positive numbers to express it and have had to start using negative ones. Meg Pickard has a more complete and up-to-date history for those of us with selective memory.

But that’s not to say there’s nothing there; a typical start-up has the kind of drive and energy in its half dozen employees that any corporate CEO would love to have reproduced across their 1,000-plus workforce. I’ve written before on about injecting start-up goodness into the newsroom, so you can go there for that. Mark Briggs does acknowledge all that is easier said than done.

Instead of me explaining why, let me introduce the start-up standard-bearers to the new blog from Simon Waldman. This extract from his first entry is worth quoting at length. Although his examples are older, rather than new, internet companies the principle is the same, being about “the general ways of the online world”:

It has always struck me that there has been a huge amount written about Google, Amazon, Wikipedia and eBay and the general ways of the online world. Some of this is brilliant, and genuinely insightful, some of it is frothy digital euphoria.

There has also been plenty written about what is wrong with newspapers, broadcasters, Britannica, record labels etc, and what they should or could have done; but there have been very few books that I’ve come across that take a systematic look at the what has happened to these businesses – and what they have done that has actually worked, often in the most trying of circumstances.

The point is – businesses that have to deal with the internet are fundamentally different to those that are the products of it. It is great to look at Google; great to admire Amazon, and Wikipedia is as fascinating a social and creative phenomena as you fan find. But if you are running a business that is profoundly structurally challenged, you share very little of their corporate DNA.

Yes, everyone needs to know about their world, but thinking you can just graft on the bits you like from them in a hope that you will ‘get digital’ is no more likely to succeed than putting on a flashing bow tie and hoping everyone thinks you have a sense of humour.

Simon is writing a book for the rest of us, who need to listen to those with ideas about how to save our challenged industry, and who also have the responsibility, along with our colleagues, to actually do the saving. The blog is the online counterpart to that book, which will be called Creative Disruption. As you can see from the extract above, it’s less “What would Google do?” and more “Never mind Google, this is for you.”

In ten or twenty years we’ll be able to look back and see which ideas from who were the wisest. For now I’m going to be a paying a lot of attention to anyone who recognises the fundamental DNA difference that Simon describes, and who deals with it.

First, huge congratulations to Matt McAlister and the whole organising team. Hack Day 1 set the bar pretty high, but the team managed to meet that, if not surpass it.

Second, a terrific change from the last one was that so many people seemed to be sharing skills and ideas  across organisations. People from Tinker.it, the BBC, Inkling, Google, Talis, Omniture, Rabbit, Guardian and elsewhere all mixed it up to produce some really innovative stuff. Added to this, there was a healthy number of people who don’t claim to be programmers creating some very interesting hacks, thanks in part to Simon Willison’s list of Hack Day tools for non-developers.

So among other things we got a tracker of the most controversial articles, a page of search terms that people are using to find guardian.co.uk — live — a social network for dead people, and much, much more.

The photo here shows Matt under the gaze of the Guardian’s inspirational editor CP Scott — or rather a bust of him. I’ve put this photo and others in a Flickr photo pool. If you were there, please feel free to contribute.

I was quite thrilled last night to have solved a really frustrating bug in a piece of open source software. First the background, then the problem and finally, for the record, the solution.

Background

I’ve been messing around programming with with JavaFX and would have liked to have used my favourite IDE, Eclipse.

Unfortunately JavaFX has a bit of a poor pedigree. It’s a me-too product being Sun’s response to Adobe Flash and AIR, just as Silverlight is Microsoft’s response. It also doesn’t seem to have been given the heavy backing and attention within Sun that anyone would need going up against the incumbent RIA platform. I see this because (a) the language changed slightly after the first official release (1.0) to the next “minor” version, which smacks of carelessness (or pressures on dates of public announcements); (b) JavaFX application loading seems to be a variable experience, and certainly not as slick as Flash’s; (c) patchy documentation — for your reading pleasure I quote here Chapter 10 of the JavaFX language reference in its entirety:

Chapter 10. Animation

Table of contents

[To do: write chapter]

Added to that, Oracle has just bought Sun. Who knows what the new owners will think of this project… it’s a long way from their core business of providing large scale middleware solutions. It’s a shame, because JavaFX has a lot going for it, and as far as I can tell stands up well technically to Flash; but it needs investment in giving it a lot of polish if it’s to demonstrate this widely. Certainly there doesn’t seem to be a fully-featured IDE readily available.

Problem

However, a small team of independent developers wrote an open source Eclipse plugin which was released in December 2008, but the last commit was on 13 January 2009. Since that time JavaFX 1.1 was released in February 2009 which doesn’t play nicely with the Eclipse plugin, but no word from the maintainers on any new releases. It seems to be dormant.

The problem I (and many before me) found is bug 206, documented in bugzilla: when writing seemingly innocuous code the incremental compiler reports

incompatible types found   : Integer required: com.sun.javafx.runtime.location.FloatVariable

Which doesn’t read very well but I think means “I found an integer but requred a float”. The error is reported on about one line in five, swamping genuine error messages, and no amount of bracketing, casting or whatever seems to help. It’s clearly frustrated a lot of people and it certainly made me rather disappointed with the whole JavaFX experience.

Anyway, inspired by insights from some of those similarly affected, and after a lot of semi-random hacking and poking I found a solution, which at least works for me.

Solution

The solution is to build the plugin yourself from the source, and use that. The original documentation for this is a little terse (well, I struggled a couple of times) so here’s an augmented version:

How to build the plugins from sources

1. Requirements

• Install the Eclipse IDE for Java EE Developers from http://www.eclipse.org/downloads/.
• Install the DeltaPack from http://update.eclipse.org/downloads/drops/R-3.4-200806172000/index.php#DeltaPack. (I have no idea what this DeltaPack is, but I think it’s used in this instance as libraries against which the plugin sources are compiled. I unzipped the download and dropped the files into my Eclipse application directory as appropriate: features in the  features directory, plugins in the plugins directory.)
• Get the JavaFX SDK from Sun in the form of a zip file. Since it’s currently only available as an executable, you need to download and  install that in the usual way (if you haven’t done so already) and then zip it up into a file. I called my file javafx-sdk1.1.zip (though I’ve actually got version 1.1.1 — using only major and minor seems to be the convention).

2. Get the plugin source files

• On the Unix commandline the incantation is svn co https://kenai.com/svn/eplugin~subversion/trunk eplugin (though I’ve not checked this).
• Alternatively on Windows you can use TortoiseSVN.
• However my Eclipse was set up to use subversion so I imported the source directly into Eclipse:
• Import the files into Eclipse. In Eclipse I did File > New > Project… > SVN > Checkout Projects From SVN. Then use the repository location https://kenai.com/svn/eplugin~subversion/trunk and get the entire contents. I had to accept a few certificates (temporarily) along the way. One time I tried this the new Eclipse project showed lots of Java errors (libraries not found), but they didn’t seem to cause a problem.

3. Amend and then run the build

Running the build that came with the source files didn’t work, and I needed to amend the Ant build.xml file. I had to change the original line

<property name="javafx.sdk.root.dir" value="javafx-sdk1.0"/>

to

<property name="javafx.sdk.root.dir" value="javafx-sdk1.1"/>

That is,  I had to change the javafx.sdk.root.dir variable to match the name of my JavaFX SDK zip file, but without .zip bit.

Now you can run the build from within Eclipse. To do this I did: Right click the build.xml file in the Eclipse package navigator, choose Run As > Ant Build… Go to the Main tab, then the Arguments box and fill in these arguments

-DeclipseDir=<path> -Djavafx.sdk.build.url=<url_to_zip> clean build-windows [build-macosx] [build-linux]

Note that you need one file path and one URL, and you need to name the operating system you’re building for. This took a few minutes to run, but it completed without problems.

If you’re not doing this from inside Eclipse you can use Ant directly.

4. Install the results

The result of this is in a new dist directory. Here you will find a few files, but the one you’re interested in is the .zip file. In my case it was called com.sun.javafx.eclipse.feature-win32-win32-x86-1.0.1.zip. This is an Eclipse plugin. Unzip it and drop the contents into the relevant places in your Eclipse install directory: features in features, plugins in plugins.

I hope you find this goes relatively smoothly.

Why did this happen?

Honestly, you think I know? My guess is that the plugin was built with JavaFX 1.0.x embedded, and those of us with JavaFX 1.1.x get compatibility errors; recompiling from sources ensures your new plugin in built with your current JavaFX SDK. If so, that’s a pain, as it may mean you’ll need to do all this again when the next JavaFX SDK comes out. But, hey, you’ll get a few hours of happy coding done in the meantime.

Mat Wall on The evolving Guardian.co.uk architecture

Mat’s presentation was structured around rebuilding guardian.co.uk over many months and dealing with the scalability issues that were presented at various stages. Some of the many insights were presented in a deliberately provocative way, such as this one:

Developers try to complicate things; architects simplify them.

I’m not entirely sure about the first part, but it’s certainly true that over-engineering is a much more common trait than under-engineering. The second part, though, is very important. It’s something I’ve always known, but never quite in that pithy way. It’s a useful phrase to tuck away for future reference.

And if you think Mat wasn’t feeling too good towards developers, don’t worry, because he also put up a slide saying

Developers are better than architects.

The message behind this is that it’s the developers who are working on the code each day — they know what’s possible and what not, and what’s reasonable and what’s not. So it’s important to trust them when they say they can or can’t do something.

Dirk-Willem van Gulik on Forging ahead – Scaling the BBC into Web/2.0

Dirk-Willem talked about the programme to re-engineer the BBC websites to enable greater scalability and dynamic content. In an organisation the size and complexity of the BBC you need to spend more time thinking about people than the technology. One of his first slides underlined this by presenting the seven layers of the OSI model (physical up to application) and then said

But most people forget there are two layers on top of that: level 8, the organisation, and level 9, its goals and objectives.

From that point on he kept referring to something being “an L8 problem” or “a level 9 issue”. It was a powerful reminder that technology work is about much, much more than technology.

Another great insight was how much they have chosen to use the simplest of standard internet protocols to join various layers and services within their network — even when those layers and services are organisation- and application-specific. This ties back to Mat’s point about an architect’s job being one of simplification.

Phil Wills on Rebuilding guardian.co.uk with DDD

Phil talked about the role domain driven design has played for us. He also pointed out various lessons that there were learnt along the way, such as the importance of value objects, and the fact that “database tables are not the model — I don’t know how many times I can say this.”

But it was only a day or two after his presentation that I was struck by a remarkable thing: Phil referred so many times to specific members of the team, and talked about contributions they had made to the software design process, even though the people he was talking about were generally not software developers. He put their pictures up on the screen, he named them, he told stories about how they had shaped key ideas. This was an important reminder to me that being close to our users and stakeholders on a day-to-day basis is incredibly important to the health of our software.

Walking away

I didn’t get to see as much at QCon as I’d like to have done, although I dare say most people will say that, even if they went to as many sessions as was physically possible. But what I did see was fascinating and thought-provoking, even when it came from people I work with every day.