Cubicles are the worst invention in the history of business. I’m still trying to think about how/why somebody thought this was a good idea. The only thing I can come up with is a secret society of Janitors, maintenance people and office furniture salespersons have formed some kind of plutocracy that actually control how we work.

Cubes make sense if you just take them in a two-dimensional plane. They are configurable in thousands of ways. You get a desk. You get a chair. You get some drawers and overhead storage. It locks. When you’re not there, you can put everything away and it’s all pretty and clean. And beige, or gray, or some kind of color that a tribe in Peru may call “Llama Vomit”. They were made for another place and time.

I have sat in cubes for the last 15 years except on a few wonderful occasions. They all pretty much looked the same. High walls that caused you to “gopher” when any kind of sound or personal communication was needed. Cramped and created so you face away from your door, so collaborating with anyone or even coming into another’s cube is an unexpected interruption at best, downright startling at worst. If you’ve been sentenced to a cube, how many of you have “rear view mirrors” or your monitor is positioned in such a way as you can see shadows behind you?

Of course, if you’re some kind of “middle manager”, you might have one of the “bigger” cubes that have a “desk” that protrudes through the middle and allows others to sit at it facing you. You get to be the “boss”, because it’s “your” cube, “your” desk, and others are there to seek your patronage. Now think about it. You’re really not going to do any “serious” business and your superiors sitting in their offices are showing that they have given you no “real” power. If you raise your voice above a whisper, you’ll be heard by persons 20 feet away. Nothing important or sensitive can be discussed there, so a “meeting room” must be booked or squatted in. Worthless.

Cubes have fabric walls and fixed furniture. Want to do some design? Where do you put the whiteboards? How do you collaborate? There’s drawers everywhere, and cubes are designed to stick your monitor in the corner (designed when monitors were 17″ wide and 20″ deep, too) and anyone working with you at your desk will be constantly banging their knees in the crappy “guest” chair that he’s borrowed from the cube around the wall.

Then there’s the cube farm. Row upon row of cubes in a giant room. 6 feet high, 50-75 feet wide and 150 feet long. The rows have hastily-printed pieces of paper in fourteen-point type that tells you who “lives” down that row. It’s like some kind of weird Stalinist housing project gone horribly wrong.

But it’s really easy to keep “looking” clean (whether it’s actually “sanitary” is debatable). I’t easy for your boss to walk down the row and quickly see who’s there and if everyone’s working. Everyone has their place. Everyone knows their job. Everyone is pounding away, waiting for 5 o’clock or a meeting to come up on their dilapidated, ancient version of Outlook, or the biannual review.

It’s also a Petri Dish of the germs that kill a company. People don’t collaborate. People have to book meeting rooms just to have any kind of communication, and that communication is delayed until the meeting room can be booked or until the “weekly” standing meeting because nobody actually talks or can collaborate in a free flowing environment. If someone has a question, they must get up, find the target person’s cube and interrupt whatever they are doing, simply because they have no idea if they are busy or not and have wasted time walking many yards across a room just to get a quick answer, and as soon as they “peek in”, the unwanted distraction has occurred. Politics and backstabbing occur in a cubicle environment. People “go back” to their cubes to stew, and only communicate by email or instant messenger. How many times have you read a message or mail and misread the intention of the sender? How often has a sender sent you an email and beat the thing to your cube so he/she can interrupt you anyway, just to ensure that you’ve been “papered”? How many times have you heard gossip wafting over your cube?

Cubes are STUPID-expensive and?inefficient. A single cubicle can cost upwards of $2000. Each cube takes up large amounts of empty space. It is expensive to move, very heavy and incredibly bulky to store. They have all the character of a minivan. An area that has 5 cubicles can support 10 persons sitting around a large, comfortable table with the best chairs, open lighting, a possible view out the window and… wait for it… face-to-face interaction!

What I’ve found out as I’ve sat around a table with 6 colleagues is:

• You can’t lie to anyone when you stare them in the face 8 hours a day, 200 days a year
• Nobody gossips, ever
• There is no disciplinary action — your peers keep you on the straight and narrow — you have to face your colleagues and pull your weight
• If you have a question, you either ask the person you’re working with, or you wait until they have a free moment
• Meetings start and end on time, and most of the time, nobody gets up. We schedule meetings not only by looking at calendars, we actually go around the room and make sure that the time is good for everyone.
• We more often than not eat lunch together at our table
• There’s no heroes, no chumps, no wallflowers and everyone carries their own weight
• If there is a knowledge gap that needs to be filled, the group will find a solution through pair programming, mentoring or pointing out the best places to find the information
• There’s just not a lot of paper, ever
• People TALK instead of EMAIL; the amount of time I’ve spent reading my email has dropped dramatically. I don’t even leave my email program up in plain site, anticipating some communication.
• The room is bigger — with no walls, the air flows freely around the room, light comes in from the windows and shadows are cast by human beings
• There are no “cold cubes” or “hot cubes” because of ceiling vent positioning
• We’re all happy to come to work. Stupid-happy. We all get along. We know things about our families and care about making sure that everyone is lifted to make the team, and the company a better place
• It’s better to “run with the Buffalo” instead of standing in your own poop with like a steer

And the biggest thing I’ve learned? We innovate. We invent. We create great code because we’ve combined our best practices. There’s no real “leader”, no “tail end charlie”. We write code with few bugs.

We get things done.

I’m doing the best work of my career. I have the best working relationships of my career. I have the best bosses in my career, I know exactly what is expected of me, have confidence in my place in my group, and feel empowered and sometimes humbled by the abilities of my colleagues. I feel respected and willingly give respect. I am happy.

The cube has become a Dinosaur Feed Pen. It’s long past it’s sell-by date in the modern corporation. In fact, it might be the single-largest factor in an unhappy and unproductive environment. Let me tell you, the happier your workers are, the better off you are. Would you like happy or sad people to build your car? How about unhappy airline mechanics? Do you want people that write the code, the intellectual capital of your eCommerce or IT department to feel like people are watching them all the time, or that they will be constantly interrupted, or wasting time to go somewhere to “get some air”? Companies face the necessity of cutting budgets and squeezing dimes to make a dollar. Simply sitting around a table with a great wireless connection, a projector, screen, places to put cards and some whiteboard space, will yield huge dividends.

My attempt here is to show how I used TDD in an Django application. We’ll create a basic project with a single application and use unit testing to figure out an algorithm an implement it in the application’s class.

Why Test at All?

I’ve spoken to many groups about Test Driven Development (TDD). Many experienced engineers don’t do it because they haven’t been indoctrinated into the practice early, and they are established in an organization where they maintain and extend code that they might have been writing for months or even years. The code has been fully tested and it is solid. Why write unit tests?

Newer developers don’t know how to start with tests. They write code, then write tests, then get frustrated because they are continuously writing code twice. Corners get cut, and at best you get the “bare minimum” of coverage, which is basically what the boss sees and reviews.

Both of these scenarios are unfortunate, because there are a few things that TDD gives gives all engineers and the organizations that they work for:

• More efficient QA. QA engineers and testers don’t have to test for functionality, and they get better quality code.?What does this mean? It means that they get to do the part of their job that is more meaningful to the company as a whole. If the basic code and functionality are automatically tested, QA can then find the border conditions and those evil underlying non-obvious bugs that can kill an application, and if large enough, kill a company’s profits.
• Better and more cohesive organizations. Test Driven Development builds confidence throughout the entire organization. Engineers working often in geographically dispersed groups need to “trust” each other, their skills and abilities. If everyone in the organization is using TDD, these groups will know that the quality coming from others will work. This not only creates trust amongst engineers, but throughout the organization, from Marketing and Business resources all the way through QA.
• Professionalism. Being a “Pro” doesn’t mean generating thousands of lines of code.?It means generating efficient, simple to maintain, easily configurable code that “just works” every time. It not only has to work, it has to be tested and documented.
• Peer pressure to “do the right thing”. If the senior members and thought leaders of an engineering group are all using TDD, documenting their code and communicating effectively, this will spread throughout the organization and foment a best practices environment.

Test Driven Development with high unit test coverage creates efficient, spirited and profitable software. The balance of this post will explain just how I use TDD to implement a complex Bank Routing Number (ABN) validation class. I am assuming that the reader is familiar with Django, and has a comfortable understanding with basic Django and Python development principles:

Create the Project and Application

My application’s name is

valid_lib

It is destined to be a Service Platform that allows all kinds of complex financial form data to be validated through a ReSTful interface. At the moment I’m not worried about the Service, the Interfaces or even the Business Model. Currently, I want to get all the validation algorithms and calculations in place. The rest of the stuff is pretty easy. At the command line, in my Django development directory, I type:

django-admin.py startproject valid_lib

To start my project. I cd into the valid_lib directory and then type:

django-admin.py startapp finvs

To create my application and it’s directory. At this point I add the valid_lib.finvs to my INSTALLED_APPS in the the settings.py file, and while I’m there I add my database information, even though I really won’t be using it yet. I also add another folder in the finvs directory that I name util. Run a manage.py syncdb against the application’s root and you should be good to go. Everything should now look like fig 1:

fig 1: Initial TDD validation app structure

(I’m writing this on my MacBook Pro and using Textmate, but you should be getting similar results however you’re doing it)

Notice that there is a test.py file in the finvs directory. This is where we’re going to start. Django’s default unit testing fixture looks for this file and will execute tests within it. We’ll open that file up and begin writing our code there.

ABN Validation

ABA (American Banker’s Association) routing numbers (ABNs) are used to identify financial institutions when making transactions. This number is usually required when setting up a bank account for direct deposits, automated payments, etc. Or when making payments by “paperless” check over the phone or online.

These routing numbers include a checksum digit which can be used to help verify if a given number is valid or not.

The algorithm checks the first second and third number and adds them together, then iterates through every three numbers (total of 9) to return a value. If the resulting sum is an even multiple (think modulus here…) of ten (but not zero), the ABA routing number is good. ?We’ll now write a test with a valid ABN to see if we can get things to work correctly. In the tests.py file, write:

"""
This file tests the finvs application
"""
import unittest
from django.test import TestCase

#Here are some bank numbers that we can run tests against...
SHORT_BAD_ABN_NUM = 1000012
LONG_BAD_ABN_NUM = 255073345999
BAD_ABN_NUM = 255073545
GOOD_ABN_NUM = 255073345

class ValidatorsTestCase(unittest.TestCase):

    def setUp(self):
        self.value = CITY_CODE_FRAUD

    def testAbnAlgorythm(self):
        """Tests the algorythm with a known good bank number.
           This test is here to create the algorythm to be used
             in the validator implementation.  When it's right,
             cut and paste it into the validator class.

           The algorithm checks the first second and third number
            and adds them together, then iterates through every
            three numbers (total of 9) to return a value.

           If the resulting sum is an even multiple of ten
            (but not zero), the ABA routing number is good.
           """
        n=0
        bank_str = str(GOOD_ABN_NUM)
        num_length = len(bank_str)
        if (num_length ==9):
            for j in range(0,num_length,3):
                t = int(bank_str[j]) #this is the first digit
                ti = int(bank_str[j + 1]) #this is the second digit
                tii = int(bank_str[j + 2]) #this is the third digit
                n += (t * 3)+ (ti * 7)+ tii #add them up
            #check for zero and modulus of 10
            print((n != 0) & ((n % 10) == 0))  #we'll take this line out when we get it right.
            self.assertTrue((n != 0) & ((n % 10) == 0))

Now, at the application root, run:

python manage.py test finvs.ValidatorsTestCase

You should get something very similar to this:

$ python manage.py test finvs.ValidatorsTestCase
Creating test database...
Creating table auth_permission
Creating table auth_group
Creating table auth_user
Creating table auth_message
Creating table django_content_type
Creating table django_session
Creating table django_site
Installing index for auth.Permission model
Installing index for auth.Message model
True
.
----------------------------------------------------------------------
Ran 1 test in 0.000s

OK
Destroying test database...

The True that you see in the output is your print statement. Notice that the system also creates and destroys a test database. This is extremely cool functionality that can be used with Django’s Fixtures, but we’re not going to use it here for this code set. We haven’t written a line of production code, but our test passes, and if you look at the test, it is the basis for the actual implementation!

Implement It!

We’re really close, but we’ll need to make a few changes before we implement it in our “production” code. Seriously, we’re really close!

In the utils directory, create the file validation_formulas.py This is where we’ll add our ABN validation class. In my test code I created a check to ensure that the number sent in had exactly 9 digits. I did this because I didn’t want to have a try / except block, and to keep it clean. Feel free to add this if needed. The other change is that we actually need to return something — True or False in this case. The test code’s def doesn’t return anything, it just makes an assertion. To validate something, we need to insert a parameter into the method and then return a true/false. We also need to give it a name. In the validation_formulas.py file, you can now insert:

class ValidationFormulas:

    def isBankRoutingNumber(self,bankNumber):
        """Parses the bank routing number for purposes of finding a fraudulent entry.
            the algorithm check the first second and third number and adds them together,
            then iterates through every three numbers (total of 9) to return a value.

           If the resulting sum is an even multiple of ten (but not zero), the ABA
            routing number is good.

           param -- bankNumber the 9 digit bank number;

           return boolean true if number is valid, false if not. false is the default
           """
        n=0
        retbool=False
        bank_str = str(bankNumber)
        num_length = len(bank_str)
        if (num_length ==9):
            for j in range(0,num_length,3):
                t = int(bank_str[j])
                ti = int(bank_str[j + 1])
                tii = int(bank_str[j + 2])
                n += (t * 3)+ (ti * 7)+ tii
            retbool = (n != 0) & ((n % 10) == 0)
        #return the value
        return retbool

Notice the added documentation. Some of it is cut and paste from the Test, the rest is information that you’ll want to convey to anyone that is using this code in the future, because hopefully at some point the company you work for will realize just how brilliant you are and promote you to CTO, where you won’t get to write code anymore. You sure as heck don’t want to leave the poor schlep that is hired to take your place in the dark!

Also notice that I have a default return of False some applications may need a different default return, or this return may be handled in a manager class that registers the default returns in some kind of persistence layer that is more dynamic. Your mileage may vary.

Wrapping it Up

There it is. You’ve written a test first. You’ve used the implementation of the test and created a sensible implementation. If there are other ways of refining your code or cleaning things up, you can always go back and make more changes. But your’e not done. You need to write more tests to actually verify that this algorithm works, that it shows false for bad routing numbers, and that it an spot when the length of the number is incorrect without throwing any index out of bounds exceptions or the like. you can then add more test methods to your ValidatorsTestCase class. I’ve also leveraged the setUp method that is provided with the framework to generate my constants for me. The code will then look like this:

"""
This file tests the finvs application
"""
import unittest
from django.test import TestCase
from valid_lib.finvs.utils.validation_formulas import ValidationFormulas

class ValidatorsTestCase(unittest.TestCase):

    def setUp(self):
        self.SHORT_BAD_ABN_NUM = 1000012
        self.LONG_BAD_ABN_NUM = 255073345999
        self.BAD_ABN_NUM = 255073545
        self.GOOD_ABN_NUM = 255073345

    def testAbnAlgorythm(self):
        """Tests the algorythm with a known good bank number.
           This test is here to create the algorythm to be used
             in the validator implementation.  When it's right,
             cut and paste it into the validator class.

           The algorithm checks the first second and third number
            and adds them together, then iterates through every
            three numbers (total of 9) to return a value.

           If the resulting sum is an even multiple of ten
            (but not zero), the ABA routing number is good.
           """
        n=0
        bank_str = str(self.GOOD_ABN_NUM)
        num_length = len(bank_str)
        if (num_length ==9):
            for j in range(0,num_length,3):
                t = int(bank_str[j])
                ti = int(bank_str[j + 1])
                tii = int(bank_str[j + 2])
                n += (t * 3)+ (ti * 7)+ tii
            self.assertTrue((n != 0) & ((n % 10) == 0))

    def testBankRouter(self):
        """Tests the algorythm in the validator against
              known good and bad bank numbers.
           """
        bank_num = self.SHORT_BAD_ABN_NUM
        formt = ValidationFormulas()
        bool = formt.isBankRoutingNumber(bank_num)
        self.assertFalse(bool)

        bank_num = self.LONG_BAD_ABN_NUM
        formt = ValidationFormulas()
        bool = formt.isBankRoutingNumber(bank_num)
        self.assertFalse(bool)

        bank_num = self.BAD_ABN_NUM
        formt = ValidationFormulas()
        bool = formt.isBankRoutingNumber(bank_num)
        self.assertFalse(bool)

        bank_num = self.GOOD_ABN_NUM
        formt = ValidationFormulas()
        bool = formt.isBankRoutingNumber(bank_num)
        self.assertTrue(bool)

I’ve added an import for the “production” class that i’m testing (ValidationFormulas). Finally, I’ve added a new method that implements new tests against the actual methods in this class to check my work. Running the tests now reveal:

Creating test database...
Creating table auth_permission
Creating table auth_group
Creating table auth_user
Creating table auth_message
Creating table django_content_type
Creating table django_session
Creating table django_site
Installing index for auth.Permission model
Installing index for auth.Message model
..
----------------------------------------------------------------------
Ran 2 tests in 0.000s

OK
Destroying test database...

If someone in the future messes with the “production” code implementation and things get out of whack, It will break here instead of in front of the user. By combining this with a continuous-build environments, a lot of mistakes can be caught extremely fast, sometimes within seconds of a source control check in!

Enjoy.

TDD is almost like riding a bike; you really don’t know how easy it is and how much fun you can have with it until you get up and going. When the training wheels are off, Test Driven Development (TDD) becomes a brand new world full of possibilities.

I have been writing unit tests for years, out of necessity to start, but later out of the desire to lead by positive example. Having come to the Computer Sciences later than most of my colleagues, I decided early on to adopt best practices to not only ensure that I wasn’t breaking builds, but I wanted to build the respect of my colleagues. Now, I really like to make sure that everything I commit to my various development communities are well-tested and as clean as possible.

Test Driven Development practices give me the verification necessary to check code in with confidence.? ?Unit testing builds this sense of security in three different ways:

• It guarantees that what I’ve written works, whether it’s new or integrated into existing platforms and applications.
• It makes me keep my code simple. If I start writing a test and it becomes a dependency-driven, closely-coupled “implementation monster”, I can pretty much guarantee that the code is going to be the same. I’ll try to refactor this and use my tests in this manner as a guage for it’s quality.
• In large environments that have continuous-build systems, there is always the inevitable “break” that stirs up panic and finger-pointing. Code with strong test coverage rarely breaks a build. This develops a team’s confidence in an individual’s engineering skills.

So exactly HOW would I begin development by writing a test BEFORE I write any actual code, and WHY would I even want to?

We will address the “why” aspect first. “Old School” developers and my earliest mentors would create “fake” layers and tiers would test the code they were writing. A “fake layer” is defined as some type of data or code that simulates the functionality of an “expected” production-level deployment. Some development communities call these fake layers “tools” or “simulators”.

These ?fake layers? would be similar to tests, but these guys wouldn’t write tests because they never did in the past (and some still don?t!).? ?”Fake layers” can create all sorts of problems.? ?They tend to get passed around as API or Interface points, and as two different groups develop against these simulators, each makes compromises because they “must” integrate with the “tool”.? ?Testing just gives you a better framework for creating fake layers that you can create your code around.? ?It also creates a path of communication between development groups, negating the need for tools and promoting integration environments.

HOW do you start with a test?

Implementing your first Test Driven Development cycle is simple. Let?s say you will be writing an application that talks to a database that currently exists (you could even start without the database and just an ER diagram). Looking at your requirements, you are probably going to implement CRUD operations against it. You start thinking about what the interface for the Data Access Object is going to look like. ?Hmmmm? ?Let?s see? CREATE, READ, UPDATE, DELETE. So your simple Interface has four methods.

Now start with a test. Create a class called MyDAOTestImpl. MyDAOTestImpl will test the ?Implementation Class for the interface?s methods. When we create a method in the test that tests well, we then create the interface method, then the move the test method, with proper modifications, into the a new ?MyDAOImpl class. It makes you create very smart tests that ensure you are ?CRUD-ing? the data you expect. Coding in this way also creates the implementation to your interface at the simultaneously with the test. Finally, it makes you think about the interface and the implementation, and affects your overall design in a very positive light.

Now let?s take it further. You?ve created all the methods in your Impl class and your Interface methods can be extracted and the actual interface is now written and, for the time being, complete. Now let?s write a test that ?calls the Interface and implements a ?new Implementation Class through it. You can now see how clean your Interface is? –? how easy are your methods to use, how are things being cleaned up, and finally, how are exceptions being caught? Are you satisfied with the results so far? If so, proceed to the next layer!

The next layer may possibly involve a creating a Session Facade to allow for user interaction with the database (more complex implementations would have deeper structures). You may create an Object model that translates the Database relationships into some type of presentation-level object gouping, and write tests to reflect this. It becomes very obvious what objects you need when you start writing a test that outputs expected data a conditioned manner.

Your first cut at it may look like horrible code; it may be some type of monolithic monster, but it will give you clarity to understand how things must be split up. You keep refactoring and writing tests, and sooner rather than later, you are suprisingly code-complete, tight and ready for QA!

Do what works for you.

Even after all this I?m sure that some engineers are completely poo-pooing this way of developing applications. Many say that it’s writing “extra code”.? ?I dispute this.? ?Before writing so many tests, I wrote quite a bit of code and refactored it before it was checked in.? ?Sometimes I’d get caught up in a particularly cruel block of logic that would take hours to get to the bottom of, or worse I’d break somebody else’s logic block in another part of the application and it wouldn’t be found for a few days.? ?This was the worst case.? ?A set of unit tests would have found it in seconds, and it would take minutes to fix.? ?Concurrently, a set of tests to track the expected output of a logic block would have ensured that anything I’m stuck on can be worked through quickly.

Still not convinced?? ?That?s fine if it works for you.? ?TDD works for me.? ? I?ve found that as my tested code base and coverage build up, my productivity jumps dramatically as the project continues because I write less and less code as the project nears completion.

Test Driven Development is all about making everything solid. Using TDD, you?ll be writing one line of code to your previous ten, and when your ?business guy? walks in and asks for a change, you?ll be going home at 5 instead of having your car languish in the parking lot after midnight. You?ll also have more confidence in the quality of your check-ins and builds, and when things go wrong and the inevitable finger-pointing starts, you won?t be a pointee!

If you get 10,000 hits per week with a conversion rate of 5% with a dollar in revenue per conversion, that’s $500 per week you’ll make. If you rely on more users, doubling your income means doubling the number of hits, which means increasing your user base to 20,000. If you want to double your income with the same number of users, then your conversion rate will need to be 10%. Pretty simple, right?

This seems obvious, but there are many hidden factors. It costs a lot of money to double your traffic. You can’t just add more search engine optimization and expect people to show up. Marketing costs can be substantial in increasing the traffic, which affect the value of doubling your user base (don’t forget maintenance, development, scaling, etc.).

The cost of increasing conversion is much less. Provided that reporting, development and consumer experience are fairly stable cost centers, using these two bodies hand in hand to create a process of solid business metrics – to – targeted development will lead to increases in conversion with much less cost devoted specifically to these increases. Further, benefits will be gained with respect to churn as user experience will be more pleasant, i.e. – users will find what they are looking for and find things on the site that are useful to them as well as profitable for us. Once conversion rate is increased, the value of Marketing Campaigns targeted at new users increases exponentially, since users will find the site more useful, return more often, and finally create more revenue with each visit.

Targeted Development with AB Testing

Targeted development through AB Testing will also increase the quality of the user that visits the site, since the site becomes more useful to them, they will look to other services that are tested and offered through it. By just using marketing dollars without tuning the site, you can risk just “sending anyone” to it, and you’ll definitely end up with higher churn rates.

The most important part of getting conversions and sales is quality traffic. If you have 100 hits a day, and a 5% conversion rate you are going to do a whole lot better than the person who gets 10,000 hits a day with an almost nil conversion rate. In addition, it’s going to cost you in the long run to pay for additional bandwidth or cover the garbage traffic.

In the Real World…

This number becomes even more obvious when you start attaching zeros to it. Let’s say that the current site generates revenue per year of $25,000,000, and let’s say that your conversion rate is currently in a realistic range of 3%. If you have a target to increase your revenue by 50% for the next year, or $23.5 million, you will have to increase your continuous user base by much more than 50%. This includes factoring in churn, scaling, bandwidth and marketing costs, and incremental development matched to marketing.

A revenue increase by targeting conversion will cost much less. You’ll need to increase conversion in actuality less than 50%. Here’s why:

• As user-experience is ramped up, more users will “find the page” and stay.
• As marketing money is spent to find new users, the conversion rate increase will have a doubling effect.
• As conversion is ramped up, more users will convert more than once per visit. This is an ideal situation. AB Testing is directed towards this goal. No amount of marketing in the world can move this figure, but an increase in the number of conversions per visit can increase profits due to the fact that any subsequent conversion is basically “free”.

Target Everything and Leave Nothing to Chance

In actuality, it’s not just enough to maximize search unless you can know what the value of a search is. We need to know what the value of a trade with Amazon, Dell, Expedia, etc is as well. You need to know what a user is doing on your site. Is the browser from an Apple OS? Why would we have a “Dell” button on the page then? Is the user over 50? Why would you have ANY marketing on that page targeted to users under 35? Metrics will show the value of everything you may want to do, and ABTesting will expose this faster than any other method.

• The eCommerce enterprise as a whole exists to get people into their “store” to buy something.
• If this were a traditional “brick and mortar” company, the technology division would be the building’s contractors, janitors, errand runners and parking attendants.
• The enterprise doesn’t really care about their code base from a business standpoint, as long as it is “running as usual.”
• Improving on something that is working diverts resources from doing things that make money.
• Engineers in nearly all eCommerce enterprises think their code base sucks.

“The eCommerce enterprise exists to entice users into their “store” to buy something.”
The eCommerce enterprise’s “store” is an application. Most, but not all of the applications involve websites, but other applications as well may generate an exchange of money for goods and services. There are “border conditions” within the scope of this model that may include sales over telephone or even storefronts, but for the purposes of this treatise one will assume that an eCommerce Enterprise is one that “depends upon some type of internet connection to generate all, or nearly all, of its revenue.”
The object of eCommerce is to make money by getting people to “press the submit button” and generate some type of revenue generating transaction. The various channels within the store have responsibilities to find new ways of making money from the existing user base – increasing conversion, advertising revenue, secondary transactions, etc.
Anything that affects revenue generation and growth in a dispositive manner is a proportional threat to the business. Assuming that all changes to a system are correspondingly risky based on their complexity, weight and frequency, it is necessary for any change to have a revenue impact that maps to this profit/threat matrix.
“Change for the better” is defined as any change that furthers the goals or revenue for the company in proportion to the risk involved. Any change falling outside of this envelope cannot be tolerated – all changes must have a clear and well-defined business case for a company to survive. Change for the better can occur with new, tested, functional changes, and also cleaner, efficiency-related changes on the “server-side” that will deliver more resources away from channels like maintenance and QE. The best-of-breed applications are aware of these and are able to accurately predict short-and-long-term revenue impact before anything moves away from the planning stages.
Change in eCommerce is necessary due to outside and internal pressure. As new competition arises, the established eCommerce operation must rise to meet this competition. As resources supporting legacy technologies diminish, one must address how to migrate to more modern, maintainable and mutable platforms.
Is there room for the slickest new thing? Is there room for cool new techie, “under the covers” stuff? Is there room for stuff that really has no direct revenue attachment? What more conservative changes are smarter? All changes are a gamble. This gamble has many variables:

• Opportunity Cost — Is the change being made at the risk of something else that could be done to increase the position of the risk/reward value? Is this the most profitable move possible with the resources available?
• Efficient Competition – Is your competition being smarter? Is this change being made to improve something that, while it is being done, the competition is building and extending applications that will take customers away or put them far into the lead?
• Enterprise Requirements – Is this change to increase some performance metric of an application that already meets the needs of the business?
• Dependency – Is this change dependent upon other deltas? Does it require future development that may or may not have revenue impact?
• Political – Does this change have no sponsorship in one or more of the common stakeholders for the channel? Do some stakeholders resent it? Does everyone have “skin in the game?”

Gambling is a fast way to the poor house if there is no real reward attached to it. Expert Technologists that specialize in eCommerce cannot and will not do anything that has a negative effect on revenue. Often this means with living in a code base that ranges from “less than optimal” to “downright scary”. The amazing thing is, if it is a going, successful concern, this scary code base has somehow survived and made quite a bit of money. This is because it works well enough, is extendable enough, and maintainable enough for the business to reach its goals.
The bad news it makes Engineers sick to their stomachs at best, and often makes them dread coming to work in the morning wondering what changes will trigger some insanely weird output in their IDEs, or worse, a customer-facing disaster.
There is good news here, too; evil, weird and application-crashing changes to this mess of code rarely if ever make it to production. Why do we know this? Because the business, code, staff and processes to create it are still there. Proof of this exists in the fact that the business is growing and flourishing at some level. Big disasters can cause serious and often unrecoverable revenue hits. Any internal or external changes to this stasis that cause enough problems will trigger a situation where too much risk will exist for this code base to continue in its current state. This is the crux of the problem – understanding the right time to make changes, and defining just how these changes should take place.
We will explore ways of ameliorating this problem after we set all of the elements involved on the stage in the following sections.

Technology is no more or less important than any other business stakeholder.

Technology doesn’t drive the eCommerce Enterprise no more than any other department. While they might actually create the application environment for the business to achieve it’s goals, Tremendous amounts of Business Development, Analysis, Marketing, UI Design, Project and Product Management must be involved to truly find success.
Technology is the conduit for the execution of the combined efforts of an eCommerce Enterprise, and often this particular gate in the general process can use its weight to implement changes in the code base that are not necessarily in-line with the Corporate Goals. These changes can have drastic revenue impact.
If one walks a mile in the shoes of the average Engineer in a group, a completely different opinion would emerge!

• If it weren’t for us, all these “marketing and business types” would be surfing Monster for their next gig (of course it might not occur to think that this would go the other way….).
• We hate our code base more than Osama bin Laden, and think that it needs to be overhauled from end-to-end and brought up-to-date because we are literally MINUTES away from crashing.
• Business doesn’t have its priorities straight. We need better requirements and fewer changes mid-way through a project.
• We’re working our butts off here and nobody gives us any credit.
• We’re holding this ship together, etc. etc.

Great Engineering Staff sticks to the fundamentals and gets off the soapbox. They know that coding to a specific implementation is bad. Loosely-coupled code is the way to go. They’ve learned over time that most of the latest technologies are drastic improvements over old legacy stuff, but implementation can be difficult and risky, especially as an “early adopter”. Finally, many of Engineers are used to long development cycles that are isolated, coordinated and last for months using a (mostly) waterfall approach.
There are literally thousands of books, articles, white papers, and other new media showing “best practices” and “industry standard” protocols, formats, processes and anything else that involves getting an idea into deployed bytes of code. The one thing for sure is, they work for someone very well, but the more specific they become the less they will work for a broad spectrum of teams across a matrix of projects and channels. Problems around process and projects of this nature generally point to Engineering Management issues.
Processes and Projects in an eCommerce environment are quite fluid and often hectic. There are constant maintenance deployments (that’s when code is branched, stabilized, QE’d and deployed, often weekly and even more frequently). Business may have several projects running concurrently that will have different delivery periods with dependencies on existing code. This makes dramatic, “fork-lifted” changes to an application nearly impossible (i.e. “Yeah? Well, YOU go tell business that they can’t do anything for three months.”). Re-architecture vs. refactoring as a question surfaces constantly, with an obvious answer on nearly every occasion.

“The enterprise doesn’t really care about their code base from a business standpoint, as long as it is “running as usual.”

They just don’t. If something is working well enough to meet or exceed the goals of the business, nobody outside of engineering cares what’s under the hood. How often has an engineer been faced with the “GUI doesn’t work right” question, when there’s something deep down in the code that actually is causing the problem? Business should have to care about what’s under the hood as long as it’s working. Business only cares that what they want gets done and the goals of the business are met.
Assuming that the engineering staff is a competent, capable and hard-working group, the code that emanates from this group will meet or exceed industry standards and “work” properly for long periods of time. Business cares about the competency of it’s entire staff; they expect that the underlying work is accurate and working properly to support business goals.
Engineering must line up behind this. Fix what can be fixed, but don’t let it get in the way of making money. “Perfect Code” belongs in Publishing, School Projects, Personal ideas and Open Source Committees. It often is missing in these genres, why would one expect it to exist in an operation with tremendous pressure, deadlines and years of legacy development?

“Improving on something that is working diverts resources from doing things that make money.”

“That man will fight us every day and every hour till the end of the war.
— Lt. General James Longstreet speaking of Lt. General Ulysses S. Grant.

The point of the above quote is that pressure in the form of fierce competition will be kept upon any Enterprise offering services and goods, especially if success is achieved. Anytime that this pressure is diverted, one must instantly assume that the competition is either gaining ground, or worse, stretching out a lead. The resources of a company are exactly like the soldiers in an army. They are limited in number and specialty, and they must be brought to bear to inflict the heaviest force in the places where they will be most effective and efficient.
Diversion from the goals of a business towards re-architecture projects is in almost all cases folly. A complex, highly fungible eCommerce application cannot be both maintained and extended in one area and simultaneously re-architected for later deployment. There is absolutely no way to manage change, improvements, maintenance and QE of an existing application while ensuring that the new application will meet functionality requirements that are only determined at time of deployment. The cracks that things will fall through can be chasms, and recovery from this may cost more than the original project.

“All Engineers probably think their enterprise code base sucks.”

/**
* Always returns true. Please excuse multiple returns. That sucks too.
*/
public boolean isCodeSucky(sucks){
if (sucks){
return sucks;
}else{
return !sucks;
}

Engineers are problem solvers. Most are just wired that way. They want the best for their projects and are often if not always forced to make compromises that constraint within the project dictates. In the ServerSide Article, “Does your code suck?” They state a few cardinal gates that your code must pass to be considered professional:

• Your code sucks if it doesn’t work.
• Your code sucks if it isn’t testable.
• Your code sucks if it’s hard to read.
• Your code sucks if it’s not understandable.
• Your code sucks if it has duplication.

Source: ServerSide.com :: Humor :: Why your code sucks.

Author’s note: “I will not be the one to cast the first stone here. Looking through the code bases of the last eCommerce and software companies with and without one’s contributions, I would venture there would be nobody on the mound for the first pitch. It is possible to do your best work in a professional manner, but could it possibly exceed the business requirements of the application being built?”

The “Evolving” Code Base

So what if an application has JSP 1.1, Java 1.3 and runs on Tomcat 4.x? What if it only has twenty percent Unit Test coverage? Woefully incomplete documentation?

• Is it stable?
• Does it perform in a reasonable manner?
• Are all of the business requirements met in a reasonable time frame?

Chances are that the basic kernel of any ongoing eCommerce application is quite old. It’s implemented with proven technologies that allow changes to be made in a reasonable time frame with reasonable resource cost. It’s stable enough to generate the money needed to keep the lights on and the salaries paid.
It’s also a reasonable assumption that the entire bank of developers, specialists and technical staff are able to develop, maintain and scale the application within reasonable expectations. If all of these factors are in place, an “evolved” or ‘Darwinian” code base exists.
“Darwinian” code bases are usually implementation-specific and fairly tightly coupled. Time and other constraints have manifested themselves into fixes, patches and other compromises. These compromises have created the need for other “fixes”, creating a “Big Ball of Mud” that is about as portable as the Eiffel Tower. The thing is, it works, and nobody in the engineering-side of an organization can fully explain why.
Types of evolving code

Just as life on Earth evolves, so does any code. Evolution is driven by need, in the case of life, these needs could be competition from within or without the species, gradual and inevitable change of the environment, and/or possible stronger “shocks” to the system ranging from a few hard drought years to a well-placed comet-strike.
Code bases too, must evolve. From the new release of the OS Kernel that the systems run on, to the ever-changing needs of the consumer and tastes, slow, gradual change is inevitable. Shocks to the system are also part of life – massive system failures, changes in platform requirements and radical new technologies implemented by the competition are all major shocks.
The minor changes described above may be described as incremental. These changes may be “planned” for and “expected”. While the stronger “shocks” are also expected from time-to-time, they are rarely welcome and can be costly and disruptive. These major, “forced” evolution can best be described as adaptive.
Incremental Evolution

Incremental Evolution is just that. Changing the code base of an application through small changes driven by business requirements, maintenance issues and small changes to the languages used. These changes are usually expected; they have incremental revenue impact, risk or time-to-market.
Adaptive Evolution

Code bases undergoing Adaptive Evolutionary changes are the result of shocks to the system. While some of these shocks may be due to disasters within the environment, most major shocks are borne out of externally driven issues – Denial of Service Attacks, Introduction of new frameworks and more efficient/fierce competition for customers are a few examples. These changes are expected but not anticipated; they have high revenue impact, risk and often can take quite a long time to get to market.

How does a “Darwinian Code Base” handle this?

Darwinian Code handles shocks and gradual changes evenly and with results and risks that are both predictable and manageable. While they may not be “best of breed” and actually be less efficient than ideal implementations, the overall abilities of the code base to work within constraints and absorb and adapt to various inputs make it a valuable asset to an enterprise. When risks, revenue, and level of effort are all factored in, the Darwinian Code Base is too tough to rewrite; the rewards for moving away from it are simply not evident.
What makes a code base strong and “to tough to rewrite” is actually quite simple. The code must be very stable, enough so that it has a highly diminished risk of any disasters causing serious revenue spikes. It must be mutable, in that any changes have diminished risk due to excellent processes, extended tribal knowledge, ease of maintenance, or great design. Finally, it must be workable, in that the existing resources of the Enterprise may impart changes or maintenance upon it with relative and straightforward ease.

So Darwinian code is great!

Stable, Mutable and Workable code sounds great. Now let’s look at some challenges. Heroic efforts sometimes have to come to bear to “save” a deployment, and often there is a very real danger of intellectual capital departing the company and taking critical knowledge with them. As newer and better ways of making the application work become available, the existing Big Ball of Mud code base cannot take advantage of them because they are tied to their implementations and tightly coupled to their dependent technologies.
So the “Good Enough” code base can actually become so “good” that it can never get “better.” Another problem becomes turnover, because developers really aren’t all that enthused about working on really old stuff. The Darwinian Code Base is rarely Great, often Good Enough, and usually On the Cusp of Relevance.

The developer vs. the “fighter jock”

Nobody ever joins the Air Force to become a transport pilot. Most think bombers are slow and dumb as well. Everyone wants to fly the latest, greatest, fastest plane that dumps all kinds of cool bombs and missiles on bad guys.
Developers are pretty much the same; once a developer has been doing their job for a period of time, they get bored and want to do the “exciting”, “interesting” and “new” stuff. Everyone is thinking about their career and worries about getting pigeonholed into old technology and not being able to code up the latest and greatest stuff when they get the call.

This is a really big problem with Darwinian code:

Your best talent will leave you because it’s not fun or interesting to extend and maintain five-year-old-crap!

So is there an answer?

Manage well – Change slowly.

Evolution vs. Revolution

With all the engineers in a company breathing down your neck to DO SOMETHING, it’s very convenient them on Revolution. Just like any country where the politics are a mess, it is easy to promise the followers that everything will be perfect when the revolution is over. History really doesn’t support this in the real world. Revolutions are bloody and fraught with risk and hardship, and that’s AFTER the fighting stops. Changing one’s code base from the ground up while trying to maintain a business is nearly impossible.
Remember that the existing code base helped the company arrive at its current point. More often than not is a successful, but possibly critical, juncture. Revolution doesn’t offer the chance of using what is right with the old code. That paradigm requires a complete rebuild and forklift of new code in place, effectively “throwing the baby out with the bath-water.”
An Evolutionary development style stresses refactoring over re-architecture. Find the low-hanging fruit, clean that area up. Find more. Create maintainable interfaces. Like the instructions on the back of a shampoo bottle; “Lather, Rinse, Repeat”. This allows for the application to grow and flourish while gradually making the changes that need to be made. Finishing is less important than the ability to maintain momentum – one may continuously alter your roadmap to take advantage of the tuition you’ve paid along the way. It becomes a “Zen” thing.

The “end” is not important, only the “journey”.

This becomes easy to sell to your staff and will allow you to manage change and growth effectively. Sell the “Zen” thing:

• Create teams to tackle specific problems and convene meetings to discuss the technologies to use. These teams can develop strategies and methods to migrate an application in a positive and maintainable direction, and still keep good people because they are doing the work that keeps them “on the edge”.
• Interfaces are your friend. Well-defined APIs will allow various parts of an application to be componentized. A team can then “divide and conquer” the application, lowering risk, times to market, and allowing for the “good stuff” to stay. Good interfaces also allow various parts of an application to be “black boxed,” which is highly convenient if different groups are involved, since a group doesn’t have to “own” the risk of changes to code that is not currently under a their purveyance.
• Create “battle plans” to anticipate shocks and adapt to changing environments. When languages face major revisions, it is often published years in advance of the actual release, and backward-compatibility is most likely a very obvious, addressable issue that can be tackled with good planning.
• Constantly involve the team in finding “low hanging fruit” to clean up and extend the existing frameworks. Is the documentation complete? When an engineer opens up a file, how hard is it to check the documentation and append/complete it before check in? Can a custom or open source Tag Library replace the existing scripted code? Is there repeated code in the file that can be added to a utility class? These are only examples, but incremental “clean up” is like compound interest; it takes very little development time, boost morale and standards, and finally keeps the existing code base up-to-date for very little risk.
• Evolve into a code base that is continuously upgraded with “service packs”, and business gets a continuously operational application that allows them to implement their goals.

It doesn’t have to be boring, but in the eCommerce world, it is vital to do “the right thing at the right time.” A successful eCommerce enterprise is almost never alone in its space. Once success is identified, copycats abound as the barriers to entry drop and the product is commoditized. One must assume that any deviation from a stated goal is wasted time, as any efficient competition will NOT deviate. Gambling for cool technology over revenue is a sucker bet.
It’s not the perfect solution for engineers that want to create the next killer app, but remember, in eCommerce, “we’re not a technology or software company.”

* Thanks to Rod Morrison and Josh Lucas for peer review and suggestions.
The Darwinian CodeBase — Dowload as Word Doc.

1. The Best is the enemy of the Good.

I’ve never met an Engineer or Manager of Engineers that didn’t see an infrastructure project as vital to the organization’s ongoing health and stability. I’ve been that Manager before. They’ve actually been wrong more than they’ve been right. The reason for this is the lack of a complete picture of the goals of the organization as a whole, the priorities for various Development/Infrastructure (DEV-INF) projects, and the fact that budgeting exists because resources are finite. Sometimes an eCommerce site can limp along for years on a code base that is far from the ideal architecturally, but it may cost far less to maintain and extend it than to upgrade it to the “Next Great Killer Platform” at the cost of revenue-driven projects.

2. The Holistic view of a Budget.

While all projects will have some value, it’s important to prioritize projects in alignment with the overall organization’s strategy and objectives. There is no use in a complete re-architecture of the site when current demands of Marketing, Consumer Experience and other various Business Units cannot be met. The goal in eCommerce is simple. Keep the business growing, keep the site in “five nines”, and refine the user experience to increase conversion. My DEV-INF budgets must play in this ballpark or as a director/manager I’ll lose credibility with Business Units, which ultimately will lead to a breakdown in communications between Engineering and Business, which is a crisis for an eCommerce workplace.

3. Metrics are the key.
Once projects are aligned with Corporate Objectives, it’s vital to show all cost drivers for these expenses. Many times the cost of ongoing maintenance is not factored in, but can be as much as 65% of an overall budget. Same for time budgeted for QA. While the individual Engineer writing code may only write 250 lines that finally make it into production in a three-month long project, how many times he/she had to write it, how much time was eaten up managing and checking this code, and finally what documentation, knowledge transfer and future extendibility these 250 lines will have also affect costs. All these things must be taken into consideration.
Budgeting isn’t just all planning. Once the planning is done, the spending begins. Vital in this aspect is accountability and assessment of performance. Hard, industry-standard metrics exist to gauge the performance of DEV-INF budget items. In eCommerce I’ve found many easily measured values:

• Actual vs. Proposed Development times.
• Actual vs. Proposed Resources.
• Pre-release bugs.
• Post-release bugs.
• Effects upon site stability and performance.
• Met Expectations with respect to the Project’s goals
• Expectations of Business Units met, with satisfied stakeholders
• Problems handled in the appropriate manner
• In-place processes unbroken

4. Approaches

Approaches to DEV-INF budgeting should be aligned with the current culture and expectations of the Business. They must have significant buy-in by various business stakeholders. If business drivers have “skin in the game” as to line-items in a DEV-INF budget, this automatically give credibility to it and also act as a check to make sure that I’ve done my homework. At Lowermybills.com, we have used the Consolidated Projects Approach to budgeting and allocation for DEV-INF projects with outstanding success. In a nutshell:
Consolidated Projects List basic points:

• IT budget planning process starts before the organization’s process.
• IT managers submit summary project data for consolidation into a single spreadsheet
• These projects are then sorted and prioritized by DEV-INF managers before submission to the overall business Prioritization Committee.
• The Prioritization Committee decides how these projects are to be aligned, budgeted and implemented within the entire Organization’s overall Business needs and strategies.

This approach provides Senior Executives with a complete picture of all proposed projects, their costs, and priority ranking; these units will have a “30,000 foot” picture to make a more informed decision. Total project requirements may now be viewed as a single picture to senior management; priorities become a decision factor.
Business stakeholders, and not the DEV-INF group, are seen to drive costs.

If your application receives 10,000 hits per week with a conversion rate of 5%, and a dollar in revenue per conversion, you’ll generate $500 per week. If your strategy is to rely on more users, to double this income you’ll have to increase the user base 100%, or 20,000 persons. Conversely if you want to double your income with the existing number of users, your conversion rate will need to be 10%. The path of least resistance becomes obvious when you compare the level of effort, time and risk to double the user base vs. incrementally increasing conversion.

This seems obvious, but there are many hidden factors. It costs a lot of money to double your traffic. You can’t just add more search engine optimization and expect people to show up. Marketing costs can be substantial in increasing the traffic, which affect the value of doubling your user base (don’t forget maintenance, development, scaling, etc.). Also, if your application depends on stickiness and returning users, you must be extremely satisfied with your existing UI , customer demographic and conversion rate, because all of these will remain constant. Note that there is a limited audience for you application, and eventually you will run up to a point that the “nth customer” will cost more to acquire than the average expected return from this person.

The cost of increasing conversion is much less. With a skilled and coordinated business, reporting and development team, a coordinated effort to create a process of business-metrics–to–targeted-development will lead to increases in conversion with much less cost devoted specifically to these increases. Further, benefits will be gained with respect to user churn as user experience becomes increasingly engaging, i.e. – users will find what they are looking for and find things on the site that are useful to them as well as profitable.

Once conversion rate is increased, the value of Marketing Campaigns targeted at new users increases exponentially, since users will find the application more useful, return more often, and finally create more revenue with each visit.

Targeted Development with A/B Testing

Targeted development through A/B and multivarate Testing will also increase the quality of the user that visits the application, since the it becomes more useful to them, they will look to other services that are tested and offered through it. By just using marketing dollars without tuning the site, you can risk just “sending anyone” to it, and you’ll definitely end up with higher churn rates and eventually run out of customers.

The most important part of increasing conversion and sales is quality traffic. If you have 100 hits a day, and a 5% conversion rate you are going to do a whole lot better than the person who gets 10,000 hits a day with an .5% conversion rate, because your costs are going to be less in equipment, development and bandwith costs, and the impact of your marketing dollars will be much higher.

A more real world use case…

This number becomes even more obvious when you start attaching zeros to it. Let’s say that the current site generates revenue per year of $25,000,000, and let’s say that your conversion rate is currently in a realistic range of 3%. If you have a target to increase your revenue by 50% for the next year, or $23.5 million, you will have to increase your continuous user base by much more than 50%. This includes factoring in churn, scaling, bandwidth and marketing costs, and incremental development matched to marketing.

A revenue increase by targeting conversion will cost much less. You’ll need to increase conversion in actuality less than 50%. Here’s why:

• As user-experience is ramped up, more users will “find the page” and stay.
• As marketing money is spent to find new users, the conversion rate increase will have a doubling effect.
• As conversion is ramped up, more users will convert more than once per visit. This is an ideal situation. AB Testing is directed towards this goal. No amount of marketing in the world can move this figure, but an increase in the number of conversions per visit can increase profits due to the fact that any subsequent conversion is basically “free”.

Target Everything and Leave Nothing to Chance

In actuality, it’s not just enough to maximize search unless you can know what the value of a search is. We need to know what the value of a shared-lead-bounty with Amazon, Dell, Expedia, or whomever you’re dealing with is as well. You need to know what a user is doing on your site. Is the browser from an Apple OS? Why would we have a “Dell” button on the page then? Is the user over 50? Why would you have ANY marketing on that page targeted to users under 35? Metrics will show the value of everything you may want to do, and A/B Testing will expose this faster than any other method.

The key to all this is simply to have excellent metrics, process, vision and stable environments. These variables cannot be changed dramatically while testing for conversion, and they must definitely be in place to do this effectively. The “lift” provided by a test must be “back-tested” if the risk is high. Couple this with new tests, and one can see how fast things can get out of hand if the team dedicated to making these tests is not well coordinated, and the platform unstable.

Finally, reporting at the crossroads to everything. It is vital to ensure that all required metrics are identified and captured, and that the reporting and business metrics teams can move with speed and accuracy. All the tests in the world aren’t worth a hill of beans unless you can determine their sucess, and when the sample for the test is complete to make valid comparisons. Times that are too short will result in inaccurate results, and times where tests are too long will result in too few tests being run in the time allowed, causing less-than-efficient optimization with respect to time/effort.

So it’s not easy, but it’s very valuable. This type of testing can be a godsend to a stagnant site with good traffic, and can also do wonders to an application that is ready to launch with limited audience but “needs tuning” before it is released to the world.