Handling the Big Blue Ball of Mud

Or, what to do when your organically grown Perl | PostgreSQL application gets too big for it's (and your) own good

Fred Moyer

Red Hot Penguin Consulting LLC

Who the heck am I?

• Hacking Perl and PostgreSQL for about 7 years
• Mechanical Engineering and Computer Science background
• Worked for startups as well as big corporations
• I've created balls of mud
• I've worked on balls of mud

What's a Ball of Mud?

• A system that has no distinguishable architecture
• http://en.wikipedia.org/wiki/Big_ball_of_mud
• Common in large, rapidly developed applications
• Most common in startups, but is found in a lot of Perl codebases
• Cost, Quality, Time - pick two
• Cost (scalar(@engineers)) is fixed
• Time is usually fixed (needs to be done right now!)
• So Quality must give
• Systems are large in complexity of business logic, not necessarily large in LoC

The Startup

• Startups are great places for creating balls of mud
• You and your huge development team (usually two to three engineers if you are really lucky) can try out new ideas fast
• In a startup, you have to constantly adapt to rapidly changing business requirements
• If you don't adapt, your competitors will destroy you, or you will run out of money quickly
• Once ideas start sticking, the application starts to generate revenue
• Great for the business

The tools group of an organization

• Usually one experienced Perl / PostgreSQL developer and a number of other technical specialists
• You're the expert in Perl / PostgreSQL
• But you're not the only one writing the code
• When there's a problem though, you're the one that gets called
• You spend a lot of time maintaining code

Evolution of the Ball of Mud

• Development is driven by organizational needs
• Not always enough information or time to make perfect decisions
• Everything has a purpose at some point in the codebase lifetime
• The Perfect is the Enemy of the Good Enough
• When a piece of code is written that gets the job done, it's usually considered 'ready for production'

Evolution of the Ball of Mud

• Perl / PostgreSQL allows product management, a technical specialist, or a programmer with a new idea to write a proof of concept without risking a whole lot
• Rarely do you have time to architect a perfect solution
• One of Perl / PostgreSQL's greatest strengths is that you can generate prototypes very quickly
• Another of Perl / PostgreSQL's greatest strengths is that those prototypes can be optimized to perform very well without needing to be replaced by a C program or proprietary database

Evolution of the Ball of Mud

• So you've got a large codebase, and it's ugly, but it works
• And it's mission critical
• And forward development is expected to continue at a steady pace, or increase
• "Why do we need to do this refactoring thing, everything works just fine?"
• An analogy I use here is to explain that your codebase needs maintenance, just like your car needs regular oil changes, or the airplanes that you fly in need regular maintenance

Evolution of the Ball of Mud

• Transactional DDL is one of PostgreSQL's greatest strengths
• It can be used effectively to solve problems quickly, but sometimes creates long term difficulties
• Rules and views are similar, allowing quick solutions for immediate problems
• Things get sticky though these quick solutions work and then millions of rows of data pours in to the schema

Evolution of the Ball of Mud

• You now have a ball of mud
• The good news is that it generates revenue, usually a lot of revenue
• Or it makes your team much needed by the organization
• The bad news is that it's higher maintenance than Paris Hilton
• The other bad news is that when new developers touch it, they break it
• The other other bad news is that you sometimes break it when you touch it

Here comes Downtime

• How much does downtime cost you?
• You have dozens or hundreds of servers running this code
• You push out a code change, and something goes wrong
• Assuming a 100 person company with operational costs roughly 1 million dollars per month
• So one hour of downtime costs 1,000,000/30/8, about $4,200
• The real cost is much higher though in terms of morale and lasting impact

How to Handle the Ball of Mud

• Follow existing test design patterns
• Don't rearchitect yet
• Big problems are hard to solve, small problems are easy, so look at your system as lots of small problems instead of one big one
• Use a version control system liberally

Frameworks are not silver bullets...

• "Let's use a framework to add some order. It will fix all our problems!"
• Be very careful of this approach, it's a lot more difficult than you think
• Only a good approach when you have really outstanding test coverage, and you are sure that the framework fits your application design patterns

Refactoring is a Winchester pump...

• Refactoring can be your single biggest friend
• It is paying off technical debt
• Management asks "What value do we gain here?"
• The value proposition is not so much what you gain but what you don't lose
• Refactor _away_ from the big ball of mud, not _towards_ it

Schema, schema, everywhere!

• Versioning software well is difficult
• Versioning schema well is more difficult
• Number one question for new developers - "Where's the data?"
• Most popular (wrong) answers:
• "Uhhhh...."
• "Ask the DBA to make you a copy"
• "Just connect your application to my database, don't worry"
• "We can't give you a copy, the disk is full"
• Most popular right answer:
• "Oh, run 'bin/create_schema.pl dbname', then 'bzcat sql/bak/test_latest.sql.bz2 | psql dbname' and you'll be good to go"

Schema, schema, everywhere!

• Hoarding of schema information is not job security!
• Printouts of ERDs are very useful
• Taking care of your schema is the best approach
• find sql -name '*.sql'
• sql/table/foo.sql
• sql/table/foo__bar.sql
• sql/table/bar.sql
• sql/func/forgot_md5.sql
• sql/trigger/md5.sql
• sql/view/foobar.sql
• Writing a program to construct your schema from this is trivial
• An upgrade.sql which contains the differences between versions is nice too

The Inner-Platform Effect

• http://en.wikipedia.org/wiki/Inner-Platform_Effect
• "In the database world, developers are sometimes tempted to bypass the RDBMS, for example by storing everything in one big table with two columns labeled key and value."
• Under abstract rather than over abstract
• You will be criticized for under abstracting by 'know it alls', but don't sweat it
• It's better to deliver something less complicated that works fairly well rather than something that is extra shiny and has a time-delayed monster bug which comes out after the fanfare has faded

The test suite that runs forever

• As the codebase grows fast, you try to keep up with testing
• More tests equals more time to run the test suite before checkins
• At some point, you hit a piece of code that's hard to test so you write a bad test
• Or you have to make a small code change and don't want to run the entire test suite
• Whatever the case, the test suite is a problem now, instead of a solution

The test suite that runs forever

• The core issue here is that the problem is too big to handle
• So you need to break it up into smaller, more manageable pieces
• If only there was a good design pattern to follow here
• But there is, and you use it every day! (http://search.cpan.org/)
• Trying to fix a really big testing suite while continuing forward feature development is usually a losing battle
• The CPAN approach has proven scalable ( more than 10k modules at last count)

MyPAN

• Most large applications contain the following components in some form:
• One or more web based applications (mod_perl, etc.)
• Standalone job processing daemons
• A database abstraction layer
• Libraries which are extensions to C code
• In the early days of the application, it's hard to see the lines between different functional sections because you are focused on delivering features
• Take your next feature request, and extract the functionality from the main codebase into a module

MyPAN

• Use your operating system's packaging system to deploy code releases
• ppm, rpm, ports, ebuilds, etc...
• Check out Ovid on CPAN for rpm generation
• Creating your spec file, ebuild, etc. is part of development
• Most shops use a version control system for deployment
• That works well for a few servers, but not for a few dozen or a few hundred
• No need to push out a new release with 100k lines of code for a change to one job processing daemon

Social Engineering

• Programming teams are complex social dynamic systems
• It's natural for code authors to become emotionally attached to their code
• How do you explain to your coworker who has a Ph.D. in Astrophysics that you had to add 'use strict;' to his code when you fixed a bug the week he was out?
• Better yet, how to you get him to add strictures to his code in the future?

Social Engineering

• Promote ego-less programming practices
• Use Changes and README files to credit authors, but put an email list alias in the actual POD
• Lead by example, not by persuasion
• If someone writes some code that looks dodgy, write a test for it rather than touching their code
• Then when you find a bug, send them a polite email with a well written fix, and an example of the test passing under your change

Social Engineering

• Code reviews don't always work
• Chances are you are the Perl / PostgreSQL expert in the organization, so you have to remain humble
• What's easy for you might be hard for others
• Focus on stabilizing the code base through best practices and you'll win people over

Tools for success

• Version control - use a system that preserves code change history across file moves (CVS is not one of them)
• Persistence testing environment - the worst failures occur going from qa to production, not dev to qa
• Guile
• Regression testing environment - http://sourceforge.net/projects/smolder
• Coffee, beer, cigarettes - convince your coworkers that life doesn't have to be full of software fire drills

Credits

• Those who had to suffer through the balls of mud I created
• Those who created balls of mud which I suffered through and gained wisdom
• http://use.perl.org/~schwern/journal - paste-archives of mini-essays
• The great programmers I've had a chance to work with and who have shown me what writing good code is all about
• #postgresql on Freenode
• The creator of Perl::Critic for introducing me to the term 'ball of mud'

Thank you SFPUG 2007!

• These slides available at http://www.redhotpenguin.com/talks/pg2007/ball_of_mud.html


• Questions?

Shameless Plug

Need mod_perl / Perl / PostgreSQL consulting?




fred@redhotpenguin.com




just another mod_perl hacker