MaxAndSoftware

What does it cost these days to develop decent enterprise software?

As I progressed thru my career (22 years in US and counting), I became more and more fascinated with just how much money is spent on so called enterprise software. My fascination has progressed thru several stages:

First, I became terrified by the actual licensing costs – “Oracle (or Sun, or, later, Redhat, or …) charges how much for support ???” This pain has eased a bit over the years as smart people have realized that a lot of open-source components cost less in the long run. But even then, the idea of someone dropping a cool mil on ERP implementation that does not work out of the box and forces his company change the business processes and lose even more money still fascinates me.

Second, the cost of keeping all these developers sunk in – “We have entire offices full of developers unable to develop their way out of a paper bag?” In government three-letter agencies and first tier contractors I had the privilege to work at, this was more of a rule than an exception.

As I matured as IT pro and started paying more attention to a bigger picture, the cost of delivering something in the IT has gained more nuances. See my post about outsourcing for example.

But probably the most fascinating angle of this is related to the productivity of individual developers. But more on that later.

After two year period of silence, I am back to blogging. The break was caused by then-employer not liking my blogging activities much. But now I am free to continue.

I pulled my old postings from backup and re-posted them on my new blog here for your reading pleasure. I kept original dates so you too can enjoy comparing the world circa 2006 to “now” as much as I did.

Ever since the customer support phone numbers everywhere have started to be answered by “Dick” and “Jane” speaking proper yet incomprehensible english, the issue of outsourcing has kept the blogsphere abuzz.

For a typical programmer, outsourcing simply means that his/her company can now employ someone with similar skills and ability for something like 7 times cheaper as long as that someone resides in Bangalore.

The screams of hundreds of thousands programmers being pinkslipped created a great disturbance in the force. And while brave consultants still have no problem finding new gigs, their rates went down by as much as $15/hour (or so some claimed).

The interesting thing about outsourcing is the deep misunderstanding on the subject why this is such a good thing to have. If you think outsourcing is good because you spend less per programmer’s hour, you don’t know the meat of the story. If you participate in real-life outsourcing projects, the realization comes quickly that there is more to it than price of Chicken Kiev in Kiev.

First of all, the benefits of hiring programmers for peanuts are greatly exaggerated. For a typical US company, the real issues impacting a project are almost always completely clouded by vendor-induced tool histeria and local political backstabbing. Getting project out of the gate is like 6% coding and 90% getting people across the hall to communicate and act for the greater good. Moving those people across the ocean somehow does not help.

The real benefit of outsourcing is even simpler than paying Indian wages for (supposingly) equal talent. Let’s assume that it is true (and it definitely seems so) that 80% of all the software projects fail. Let’s also assume that you as a savvy manager got 40% of your work outsourced and are claiming that the outsourced work cost you only one/forth of what the onshore work does. It means that 32% of your team’s workload is garanteed to fail AND it costs four times less!

And if it is to fail no matter what, paying less suddenly starts making sense. The communication, legal and other barriers that are often named as the reasons for unsuccessful outsourcing efforts do not matter for a project that is going to fail anyway.

And if by any stroke of luck, you, the manager, managed to direct more “failable” projects to India and kept more promising “real” ones here, all the sudden you are the hero.

And this is why outsourcing is good for management no matter what – it is the art of outsourcing of that IT black hole that sucks out the 80% of the efforts and fails anyway.

It might come as a bit of news to the majority of the current young programming crowd, but the traditional (web one-o, so to speak) web applications have not been considered “legacy” just a short while ago.

If one was to trace back the evolution of the GUI development, one needs to start with VB. As soon as first true mass-produced OS (Windows) started its march across the world’s desktops, it became evident that slapping together GUIs using convenient visual builder tools has become a popular thing. Visual Basic has not pioneered visual GUI builders and code-attached-to-buttons paradigms, but it was the first environment to bring this kind of programming to the mainstream.

Very soon people realized that there is more to GUI building than a GUI builder.This kind of programming requires a special kind of professional, one who can carefully design the screens and control the overall user experience. In addition, any sofisticated “screen” starts living on its own, starts revealing the personality of its creator and of the application beneath. In order to take full advantage of all the wizbang features we programmers put in those user screens, our poor users have to learn each screen’s quirks and tips, understand ways in and out of special, application-specific situations. Back in the client-server times, it was fully understood that before your shipping clerks, your telemarketers or weapon operators can use your brand-spanking-new UI they will have to be trained.

Tools and skillsets matured and complex, user-intensive applications had been built. But then the web came and changed everything.

On one hand, the interface features delivered through the basic HTML were just primitive subsets of the “rich” GUI tools already available to the programmers of the late 80s and early 90s. And the hassle one had to go through to process and validate user input via CGI semantics made building GUIs rediculuosly expensive from time-and-labor point of view.

But on the other hand, the whole simpleness and ugliness of web GUIs made them more accessible and appealing to the general computer-using crowd. Guess what, you don’t need to teach your grandma how to use amazon.com’s “screens”. It is order of magnitude easier to get your new book from Amazon than cancel appointment in Outlook or navigate any other “rich” client.

The roaring 90s were fueled by dotcom dollars and suddenly validated the idea that it is ok to have a poor GUI toolset and spend days developing every simple screen. As always, mediocracy ruled.

There have been attempts to add “richness” to the standard HTML “experience”. Macromedia Flash brings us all the way up to the level VB programmers enjoyed in 1992. And if someday Sun people pull their collective head out of their collective asses and deliver simple to install and thus usable Java on the desktop, it will rule the world. Meanwhile the current meme is the AJAX – the attempt to raise Javascript silliness to the level of application frameworks.

I first used “AJAX” (without even knowing it!) back in 1999 when I delivered an app where user page did not refresh and instead was using a hidden frame to send and receive pieces of comma-separated “XML” to and from the server. It was the solution that particular app called for and just like everything else, I considered “asynchronous Javascript” to be just a tool on my toolbelt. I never thought that this simple concept can be elevated and generalized as it is today.

Today, AJAX can be found in two major flavors. First, there are “experience enhancement” scripts: pages fading away, buttons changing colors and captions, etc. Second, there are the true UI toolkits: smart form fields, typeahead text boxes, self-adjusting dropdowns. The first kind is often annoying and even can scare an unsuspected user when a static page comes alive.

The second kind – AJAX as a king of smart, “rich” UIs, is the trend in question. Amazingly enough, it brings the developer costs even higher. Not only UI remains largely in hand-coded HTML, now there is a giant sub-presentation layer where XML fragments need to be carefully manipulated by Javascript. Despite using glorifyed associative arrays in a way which is not entirely unlike object-oriented programming, Javascript remains a language severally challenged when it comes to developing large, complex code base.

Most importantly, the complex AJAX UIs bring the curse of UI designers back to the web. Just like it was necessary for the users to be trained to use a complex client-server screen of yore, AJAX screens are often too “brainy” for a casual user and as such not suitable for a larger web.

AJAX development is facing this dilemma: AJAX is perfectly fine for developing those million-dollar phone books on a corporate intranet. But so are (and were)VB, Powerbuilder, Oracle Forms, Java(applets and Web Start), etc. But corporate world and its cash-starving internal departments are far less willing to tolerate hand-polishing screen crafting typical for AJAX.

And while there are plenty of money available for “customer-facing” apps hosted by old entrenched dotcoms and flashy newcomers, wide-scale usage of AJAX and ensuing comlexity of the UI will scare away the very customers those apps are facing.

So my predictions are that AJAX will not get traction necessary to change things in the truly disruptive way. And as far as the whole Web 2.0 thing goes, I always believed that the only real, internal purpose of the IntraWeb prolifiration of the 90s and 00s was to get an IP address in every home and on every phone so new, truly interactive and distributed applications may flourish. IPTV, podcasting, Bit Torrent, social networks, etc. are what define Web 2.0 for me and not AJAX.

Anyone involved in developing large, so-called “enterprise” Java software application has seen the unmistakable signs of the software rot.

There are dozens and dozens of “third-party” software libraries that were brought in for a good reason but over time just kept piling up there in the “lib” directory somewhere. How many ways to do XML parsing Java people need? Chances are, if you application is old enough and big enough, you have jibx and jaxb, castor and xstream, plain SAX and (j)DOM all happily sitting there. Maybe the people who brought those critters in are long gone, maybe the code using one of them is so hairy that everyone is afraid to fireup the lightsaber of refactoring, maybe it is just inertia – but any sizable J2EE project just keeps carrying forward the baggage of endless collection of utility libraries.

And what about persistence? A friend of mine told me recently about large complex ecommerce project he was involved in that is now using plain JDBC, several flavors of Hibernate, Spring JDBC, Apache ORB and EJB3 all at the same time, all inside the same EAR. And everyone of those technology pieces comes with its own set of dependencies and quirks.

The related problem is what I call microframeworks. Every programmer who thinks he has outgrown the junior level thinks it is his blood right to put in a microcontainer, a workflow framework or customize Struts beyond recognition. It is not engineering, it is micro-craftsmanship that has not even reached the level of medieval artisans.

And how much the whole process costs? Even with significant downward pressure applied by the recent outsourcing trend, programmers in US are not cheap. There is nothing wrong with that. What is a problem is very low productivity of individual programmer. Between endless meetings and debugging marathons caused by low quality coding, one line of code per head per day is not an unreasonable estimate. As stupid as any LOC-based estimates, it is still eye-opening that you may be paying upward from $400 for every line of code in your application.

And let’s talk about quality. At some point or another during the lifecycle of a typical large project there were unit tests. And unless the doing the unit tests has become natural thing for the team and management has become used to the idea that 2/3 of the code is “wasted”, chances are that the unit test effort has died the premature and unfortunate death. And as a result, the code quality on any mathure, long running project has nose-dived some time ago.

But there are other ways to improve quality, right? Say like code reviews?

Just about any code review I have attended in the last 15 years ended up as a discussion of where to put the curly brace or other similar earth-shuttering problems.

As you add people to the room, the quality of discussion goes down in the reverse proportion of the number of people present squared.

Code reviews, even when they are not a complete waste of time simply do not scale to cover the majority of the code base.

The eXtreme Programming and the like have brought a much needed breath of fresh air in the software development a few years ago. To my amusement, while enjoying a strong support among young and naive, the X methodologies did not catch on much. Maybe because they do work. Maybe because the XP efforts are often quetely subtaged by programmers who are not sharp enough to keep up but are sharp enough to realize that XP project will expose them to the large crowd for what they are and the hours and hours of Slashdot reading in the corner cube will no longer be possible.

Software industry is often compared to construction (and not in a good way). But this comparason only goes as far as acknowledgement of not having a Microsoft-style monopoly on door handles is a good thing. I would go deeper. I think the software industry today is in its infancy. It is where construction was at the time of pyramids. Look how advanced we have become – we have mastered the act of amassing the hordes of potentially outsourceable programmers to roll the J2EE boulders up the slope.

I believe that the software revolution is not far ahead. In the future postings I ‘ll try to identify the positive trends in the software development today and see what we can do with those.