As I progressed thru my career (22 years in US and counting), I became more and more fascinated with just how much money is spent on so called enterprise software. My fascination has progressed thru several stages:

First, I became terrified by the actual licensing costs – “Oracle (or Sun, or, later, Redhat, or …) charges how much for support ???” This pain has eased a bit over the years as smart people have realized that a lot of open-source components cost less in the long run. But even then, the idea of someone dropping a cool mil on ERP implementation that does not work out of the box and forces his company change the business processes and lose even more money still fascinates me.

Second, the cost of keeping all these developers sunk in – “We have entire offices full of developers unable to develop their way out of a paper bag?” In government three-letter agencies and first tier contractors I had the privilege to work at, this was more of a rule than an exception.

As I matured as IT pro and started paying more attention to a bigger picture, the cost of delivering something in the IT has gained more nuances. See my post about outsourcing for example.

But probably the most fascinating angle of this is related to the productivity of individual developers. But more on that later.

I pulled my old postings from backup and re-posted them on my new blog here for your reading pleasure. I kept original dates so you too can enjoy comparing the world circa 2006 to “now” as much as I did.

For a typical programmer, outsourcing simply means that his/her company can now employ someone with similar skills and ability for something like 7 times cheaper as long as that someone resides in Bangalore.

The screams of hundreds of thousands programmers being pinkslipped created a great disturbance in the force. And while brave consultants still have no problem finding new gigs, their rates went down by as much as $15/hour (or so some claimed).

The interesting thing about outsourcing is the deep misunderstanding on the subject why this is such a good thing to have. If you think outsourcing is good because you spend less per programmer’s hour, you don’t know the meat of the story. If you participate in real-life outsourcing projects, the realization comes quickly that there is more to it than price of Chicken Kiev in Kiev.

First of all, the benefits of hiring programmers for peanuts are greatly exaggerated. For a typical US company, the real issues impacting a project are almost always completely clouded by vendor-induced tool histeria and local political backstabbing. Getting project out of the gate is like 6% coding and 90% getting people across the hall to communicate and act for the greater good. Moving those people across the ocean somehow does not help.

The real benefit of outsourcing is even simpler than paying Indian wages for (supposingly) equal talent. Let’s assume that it is true (and it definitely seems so) that 80% of all the software projects fail. Let’s also assume that you as a savvy manager got 40% of your work outsourced and are claiming that the outsourced work cost you only one/forth of what the onshore work does. It means that 32% of your team’s workload is garanteed to fail AND it costs four times less!

And if it is to fail no matter what, paying less suddenly starts making sense. The communication, legal and other barriers that are often named as the reasons for unsuccessful outsourcing efforts do not matter for a project that is going to fail anyway.

And if by any stroke of luck, you, the manager, managed to direct more “failable” projects to India and kept more promising “real” ones here, all the sudden you are the hero.

And this is why outsourcing is good for management no matter what – it is the art of outsourcing of that IT black hole that sucks out the 80% of the efforts and fails anyway.

If one was to trace back the evolution of the GUI development, one needs to start with VB. As soon as first true mass-produced OS (Windows) started its march across the world’s desktops, it became evident that slapping together GUIs using convenient visual builder tools has become a popular thing. Visual Basic has not pioneered visual GUI builders and code-attached-to-buttons paradigms, but it was the first environment to bring this kind of programming to the mainstream.

Very soon people realized that there is more to GUI building than a GUI builder.This kind of programming requires a special kind of professional, one who can carefully design the screens and control the overall user experience. In addition, any sofisticated “screen” starts living on its own, starts revealing the personality of its creator and of the application beneath. In order to take full advantage of all the wizbang features we programmers put in those user screens, our poor users have to learn each screen’s quirks and tips, understand ways in and out of special, application-specific situations. Back in the client-server times, it was fully understood that before your shipping clerks, your telemarketers or weapon operators can use your brand-spanking-new UI they will have to be trained.

Tools and skillsets matured and complex, user-intensive applications had been built. But then the web came and changed everything.

On one hand, the interface features delivered through the basic HTML were just primitive subsets of the “rich” GUI tools already available to the programmers of the late 80s and early 90s. And the hassle one had to go through to process and validate user input via CGI semantics made building GUIs rediculuosly expensive from time-and-labor point of view.

But on the other hand, the whole simpleness and ugliness of web GUIs made them more accessible and appealing to the general computer-using crowd. Guess what, you don’t need to teach your grandma how to use amazon.com’s “screens”. It is order of magnitude easier to get your new book from Amazon than cancel appointment in Outlook or navigate any other “rich” client.

The roaring 90s were fueled by dotcom dollars and suddenly validated the idea that it is ok to have a poor GUI toolset and spend days developing every simple screen. As always, mediocracy ruled.

There have been attempts to add “richness” to the standard HTML “experience”. Macromedia Flash brings us all the way up to the level VB programmers enjoyed in 1992. And if someday Sun people pull their collective head out of their collective asses and deliver simple to install and thus usable Java on the desktop, it will rule the world. Meanwhile the current meme is the AJAX – the attempt to raise Javascript silliness to the level of application frameworks.

I first used “AJAX” (without even knowing it!) back in 1999 when I delivered an app where user page did not refresh and instead was using a hidden frame to send and receive pieces of comma-separated “XML” to and from the server. It was the solution that particular app called for and just like everything else, I considered “asynchronous Javascript” to be just a tool on my toolbelt. I never thought that this simple concept can be elevated and generalized as it is today.

Today, AJAX can be found in two major flavors. First, there are “experience enhancement” scripts: pages fading away, buttons changing colors and captions, etc. Second, there are the true UI toolkits: smart form fields, typeahead text boxes, self-adjusting dropdowns. The first kind is often annoying and even can scare an unsuspected user when a static page comes alive.

The second kind – AJAX as a king of smart, “rich” UIs, is the trend in question. Amazingly enough, it brings the developer costs even higher. Not only UI remains largely in hand-coded HTML, now there is a giant sub-presentation layer where XML fragments need to be carefully manipulated by Javascript. Despite using glorifyed associative arrays in a way which is not entirely unlike object-oriented programming, Javascript remains a language severally challenged when it comes to developing large, complex code base.

Most importantly, the complex AJAX UIs bring the curse of UI designers back to the web. Just like it was necessary for the users to be trained to use a complex client-server screen of yore, AJAX screens are often too “brainy” for a casual user and as such not suitable for a larger web.

AJAX development is facing this dilemma: AJAX is perfectly fine for developing those million-dollar phone books on a corporate intranet. But so are (and were)VB, Powerbuilder, Oracle Forms, Java(applets and Web Start), etc. But corporate world and its cash-starving internal departments are far less willing to tolerate hand-polishing screen crafting typical for AJAX.

And while there are plenty of money available for “customer-facing” apps hosted by old entrenched dotcoms and flashy newcomers, wide-scale usage of AJAX and ensuing comlexity of the UI will scare away the very customers those apps are facing.

So my predictions are that AJAX will not get traction necessary to change things in the truly disruptive way. And as far as the whole Web 2.0 thing goes, I always believed that the only real, internal purpose of the IntraWeb prolifiration of the 90s and 00s was to get an IP address in every home and on every phone so new, truly interactive and distributed applications may flourish. IPTV, podcasting, Bit Torrent, social networks, etc. are what define Web 2.0 for me and not AJAX.

There are dozens and dozens of “third-party” software libraries that were brought in for a good reason but over time just kept piling up there in the “lib” directory somewhere. How many ways to do XML parsing Java people need? Chances are, if you application is old enough and big enough, you have jibx and jaxb, castor and xstream, plain SAX and (j)DOM all happily sitting there. Maybe the people who brought those critters in are long gone, maybe the code using one of them is so hairy that everyone is afraid to fireup the lightsaber of refactoring, maybe it is just inertia – but any sizable J2EE project just keeps carrying forward the baggage of endless collection of utility libraries.

And what about persistence? A friend of mine told me recently about large complex ecommerce project he was involved in that is now using plain JDBC, several flavors of Hibernate, Spring JDBC, Apache ORB and EJB3 all at the same time, all inside the same EAR. And everyone of those technology pieces comes with its own set of dependencies and quirks.

The related problem is what I call microframeworks. Every programmer who thinks he has outgrown the junior level thinks it is his blood right to put in a microcontainer, a workflow framework or customize Struts beyond recognition. It is not engineering, it is micro-craftsmanship that has not even reached the level of medieval artisans.

And how much the whole process costs? Even with significant downward pressure applied by the recent outsourcing trend, programmers in US are not cheap. There is nothing wrong with that. What is a problem is very low productivity of individual programmer. Between endless meetings and debugging marathons caused by low quality coding, one line of code per head per day is not an unreasonable estimate. As stupid as any LOC-based estimates, it is still eye-opening that you may be paying upward from $400 for every line of code in your application.

And let’s talk about quality. At some point or another during the lifecycle of a typical large project there were unit tests. And unless the doing the unit tests has become natural thing for the team and management has become used to the idea that 2/3 of the code is “wasted”, chances are that the unit test effort has died the premature and unfortunate death. And as a result, the code quality on any mathure, long running project has nose-dived some time ago.

But there are other ways to improve quality, right? Say like code reviews?

Just about any code review I have attended in the last 15 years ended up as a discussion of where to put the curly brace or other similar earth-shuttering problems.

As you add people to the room, the quality of discussion goes down in the reverse proportion of the number of people present squared.

Code reviews, even when they are not a complete waste of time simply do not scale to cover the majority of the code base.

The eXtreme Programming and the like have brought a much needed breath of fresh air in the software development a few years ago. To my amusement, while enjoying a strong support among young and naive, the X methodologies did not catch on much. Maybe because they do work. Maybe because the XP efforts are often quetely subtaged by programmers who are not sharp enough to keep up but are sharp enough to realize that XP project will expose them to the large crowd for what they are and the hours and hours of Slashdot reading in the corner cube will no longer be possible.

Software industry is often compared to construction (and not in a good way). But this comparason only goes as far as acknowledgement of not having a Microsoft-style monopoly on door handles is a good thing. I would go deeper. I think the software industry today is in its infancy. It is where construction was at the time of pyramids. Look how advanced we have become – we have mastered the act of amassing the hordes of potentially outsourceable programmers to roll the J2EE boulders up the slope.

I believe that the software revolution is not far ahead. In the future postings I ‘ll try to identify the positive trends in the software development today and see what we can do with those.

Chances are he is going to start talking layers. Hopefully he is smart/experienced enough not to start with Entity EJBs (however, recently rehyped EJB3 beans may resurrect that dogma). Most likely he is going to describe some kind of Data Access Layer (DAO) magic to talk to the database and end up describing some kind of Service Layer or Business Layer “exposed to the clients”.

If such Architect actually had experience building up a performant system, he will assume all layers living in the same JVM. If he is by the book kind of Architect, he will separate “Service Layer” onto its own set of boxes and claim the victory for Service Oriented Architecture.

And how these newly conjured layers are going to talk to each other?

Well, the DAO layer will get busy running database queries and materializing the Java objects out of thin air based on the result sets for selects and mapping and binding values from the given objects to the database for inserts, updates and deletes.

What is coming out of DAO layer (and what is hopefully captured by set of DAO interfaces) is the number of so called Data Transfer Objects (DTOs) which look sort of not unlike your database schema: Account, Customer, Order, etc. and collections thereof.

Service slash Business Layer will take those DTOs and do something smart with them to implement our “business logic” and then make more instances and collections of DTO-style objects and call the DAO layer again to persist the results.

The most prominent caller of the Service Layer is the always mysterious UI Layer. Most likely, it is just JSPs or some kind of templating thing driven by the MVC framework-of-the-day living in the same VM. But, of course, if your buddy Architect is not ready yet to give up the dream of distributed nirvana, UI layer may live separately on the set of “webhead” boxes. In that case, the hefty doze of remoting (RMI, Spring HTTP kind, ..) is thrown in the picture.

In order to be “clean”, UI layer should be able to manipulate objects coming out of the Service Layer. Being a “customer” of the Service Layer, the UI only has access to the set of interfaces exposed by the Service Layer and the objects manipulated by those interfaces. These objects form yet another set of DTOs. For obvious reasons, these DTOs are very different from DTOs coming out of DAO layer and have to be built up and maintained separately. In addition, if application is multi-tier distributed (Service Layer and UI Layer talk to each other over the network), sooner or later your friendly Architect will attempt to battle inevitable performance issues by making “coarsely grained” requests. What it means, the Service Layer will attempt to minimize number of remote calls by piling up more and more stuff on every request, whether it is related to the original purpose of that requests or not. Regardless of perceived performance gain, it will lead to bigger, badder, harder to maintain DTOs in the Service Layer.

As different, layer-centric DTOs blossom in your application, the code handling them thrives as well. Entire “utility” libraries spring up to convert one type of DTOs to another. What makes this even worse is the fact that those converters often have to make semi-business decisions, for example, how to prune the results from over-eager object graph coming through. Overall the DTO transformation efforts start consuming significant portion of your application time and money.

Hibernate and other object relational mapping (ORM) tools bring interesting twist into this equation. Hibernate goes the distance to make your Java objects easily persistent and hide the ugliness of the database programming. Using Hibernate, programmer suddenly gets to simply save or update his objects as naturally as calling methods or setting properties.

So where does Hibernate fit in your friend Architect’s designs? In the DAO layer of course. In a typical J2EE project, ORM is no more than tools of getting stuff from the database and putting it back in.

Significant effort needs to be put in to ensure that Hibernate mappings, objects and queries truly represent the domain model captured in the database. And all this almost immediately goes down the drain as the DAO layer takes the Hibernate objects coming out of the queries and “converts” them to DTOs that can be understood by the Service Layer and up.

Ask your Architect why his Hibernate-powered DAO layer needs DTOs at all instead of simply passing the “hibernatable” objects out. Most likely you are going to get a lot of hand-waving and glossy-eyed looks we typically get while discussing J2EE “patterns”. If you do get a reasonable response, it is probably going to be centered around the idea that we do not want our DAOs to become “dependent” on Hibernate and we want to be able to swap different DAO implementations in and out as needed. The magical word “reuse” will be brought to the altar.

This attitude is unfortunately shared by many architect types but is wrong nevertheless. Hibernate objects are simple Java beans and as such carry no Hibernate dependencies. The real reason is that very few take their time to learn how to properly attach and detach objects to and from the Hibernate session and how to handle lazy initialization of “getChildren” collections. Did I get null while calling getChildren() because there were none or because we already closed the Hibernate session and this particular relation was set to be lazy-inited?

The real problem with DTOs is rooted in the reality that few of the currently developed J2EE applications actually care about the object model. In fact, the whole J2EE mentality has been anit-OOP from the very beginning. (Don’t get me started on EJB designs).

The solution is the embrace the “objectness” of your domain model. In case of Hibernate, do invest time developing mapping and queries. But instead of making Hibernate a poor bastard child of the DAO layer, make Hibernate the cornerstone of your design, use it to fully capture the intricacies of your domain model. There should be no other place you need to go to learn about your Customer model but “Customer” object mapped by Hibernate. Hibernate classes should become the primary, authoritative source of domain knowledge. And having done that, would you still throw all the knowledge away for sake of translating to Hibernate-less DTOs? Of course not.

Hibernate classes are first class object citizens. Use them to pass the data out of DAO layer or of any layer. Do away with the need to perform endless transformations from one set of DTOs to another to another as you logic traverses the application layers.

There is nothing to stop you to use the core Hibernate model classes even as part of the UI layer. Instead of creating yet another object model acting as DTOs for UI (think Struts form objects), just use the core objects directly or with slight massaging applied. So instead of using constructs like

, inject the domain objects into the UI forms:

.

And how to deal with lazy initialization and the danger of pulling too much from the object graph? – one might ask.

If your application is not deployed on multiple network tiers and, instead, the entire stack runs in the same VM (as it should be for maximum scalability), the solution is simple – just let Hibernate do its thing and proxy the calls to the collections as needed. So if inside of your UI, your form feels like looping through some records using forEach, just let it.

If this particular relation is lazy-initialized in Hibernate, the corresponding database call will only be made when Struts (or whatever) gets to execute this loop in JSP. Of course, this will only work if you employ Hibernate “Open Session in View” pattern (or its Spring equivalent).

This is where patterns police sees the red: this call in fact bypasses their precious DAO layer and gets executed much later in the game.

But after all is said and done, it is your call on the tradeoffs between perceived pureness of the design and enormous simplification obtained by getting rid of multilayered DTO structures and support.

If you want to get something approved – ensure you can claim code re-use as a benefit of your project, whether it is true or not.

But when re-use actually works?

IMHO, re-use works very well on the low-level, you know, in the home-grown utility classes:

if(Utils.isEmpty(someStringFromMethodCall())) {

And, it works well on the very high level when very coarsely grained, enterprise-wide services are defined:

ServiceLocator.getPaymentService().chargeCard(...);

What about all the possible re-use opportunities hidden between these two extremes?

Feel free to explore but be advised that at the today’s level of tools and practices, you are more likely to

waste more time and money in the pursuit of the re-use than save from re-using.

This blog is meant to be the place where I will leave my more or less random thoughts on enterprise technologies for the world to ponder.

So here it comes.

Possible areas that are going to have to suffer from my writings include (in no particular order):