Category Archives: Software Engineering

Development flows, Version Management, Test driven development and other teams supporting the software delevelopment process

Build-in time bombs

I’ve been refactoring and refactoring some old code, and it’s kind of odd what short-cuts (or even time bombs), you’ll find in code, which apparently wasn’t supposed to live on for years.
In a now retired CMS system, we had an issue every new year, when some kind of bug would reset all “schedules” for upcoming stories and content. No-one ever got around to fix it, as the system was soon to be decommissioned – but sadly the bug did survive a few years anyway.

These days, I’m was working in the system, which came to replace the broken system from before. Here’s another odd thing – It was apparently hard coded into the editor in the backend, that stories need to have a “published date” between 1996 and 2011. As there really isn’t any documentation and the original developer has left the company, it’s hard to know why it was made so. While there probably was a reason, it’s lost by now.

 <select name="year" size="1" style="width:55px;" title="Year">
 <?php
   for ($i = 1996; $i < 2011; $i++) {
       printf("<option value=\"%1\$d\"%2\$s>%1\$d</option>",$i,($year == $i) ? " selected" : "");
   }
   ?>
</select>

We fixed it by making the latest “published date” go between 1996 and current year plus one.

While it was an odd thing, I’m surely glad that we from time to time do some sort of maintenance of old code – even while it seems to work perfectly. It’s far from the only bad thing we found, and while most people worry mainly about new code, sometimes you should also remember the old stuff.

The art of the commit (in version management)

Most developers (and certainly professional shops) working with software- or webdevelopment has understood the ideas behind sourcecode version management – no matter if they choose to use cvs, subversion, git or any of the other fine systems available. How developers use these systems, can provide an easy insight into who is the dummy, the average and the great developer.

When to commit

One of the first distinguishing signs is what they commit – is the version management system used as a backup tool or as a version tracking tool. Some seem to think, you should end the day by committing all your sourcecode (no matter which state it’s in). This is really bad sign. Others commit whenever a new file is finished and thought ready to use – which is slightly better. Others again commit once – just before a new release – and has the commit contain each and every change since last release.

In my world a commit should contain a complete new feature across all the files changed to make the new feature. The commit should preferably only address one change (a new functionality or a changed feature).

The commit message

Another way to spot bad habits is in the commit message. Is there a commit message at all? – if not, it’s a bad sign. Almost as bad are meaningless comments, which may come in all sort of varieties – “work in progress”, “bugfix”, “update”, “foobar”. I’m sure they make sense to the developer at the time of committing them, but hardly 6 months later, when browsing the logs and wondering why a certain change was made.

A less common bad commit habit is the “book style” – where the comment either contain a delta of the changed lines (literally or as a description) or the length of the comment is a small story explaining the reason for the change, why it was made the way it was and on and on.

I’d any day prefer seeing a reference to a bug/ feature tracking system (bugzilla, trac or anything like it), and a brief summary of the change in a 140 characters a less.

Tips for developing WebApplications

So it been awhile since I sort of promised some tips on developing web applications fast. This post contains my five generic tips, which may apply to you (but then again – your mileage may vary). I’ve tried to abstract the advice and not stick to PHP development here, though it’s by far the world I know best.

1. Spend time on the data model

Too often a data model is driven by code, not careful thinking, and this often causes problems has the web application develops and expand over time. When developing a site, which needs data stored in a database (as most web applications tend to do), do try to spend some time on the data model – which tables do you need, what fields should the tables have, are the naming logical and consistent, what constraints and limitations does the data model offer.

It’s often far easier to refactor code and functionality than it is changing the database tables and fields, and it can be a tremendous help while maintaining an application if you clearly understands the data its working with.

2. Get the basics working first and fast

It often takes a while to get a complete web application finished. Help motivating yourself by getting the core working first and then add features and frosting once it’s working. By having a working skeleton, I usually find it much easier guess how much work needs to finish the various features than I do guessing the work needed to complete a section (ie. admin interface) on the application.

3. Think performance and security from the beginning

Remember to think of performance and security from the very beginning. Sometimes performance – or security – may cause you to layout the data model in a certain way or structure the application a certain way. While you may be able to add performance and security during refactoring, it’s often much harder than it is keeping both in mind from the very beginning.

4. Don’t fix the future; focus on the current needs

I don’t care about the next version. I really don’t want an infrastructure in place, which can help me build the future versions of the application. I want the features and infrastructure in place for the current version I’m working on – and frankly the features in the next version has a tendency to change before I’m ready to develop them.
Too often future-proofing an application clutters the code and too often the clutter doesn’t ever come to any good use – it’s just noise once we get to the real future features of the application.

5. Naming matters

Do think of what names you use on functions, classes, variables and database tables through out the application – and choose something meaningful. A variable named $x doesn’t hint what it’s used for. Help yourself (and other looking at the code) by hinting the uses of various things by choosing meaningful names.

Your mileage may vary

It’s always fun to read articles with tips and tricks by other developers and see how they figure “best practices” are handled. Most developers do seem to thing they observations and practices are easily adopted by anyone, and should be accepted without any argumentation or reasoning behind the advice.

One of the nice examples of this, was a story called “5 tips and tools to develop php applications fast”, and while it may apply to the web applications developed by the author, it’s one of those where I question who is supposed to be the audience. All 5 tips are often found in many textbooks and other stories on the net, and in my experience they are all wrong (more or less naturally).

1) Use a MVC framework – it’s an industry accepted standard.
MVC as an idea may be accepted, but there are many ways of implementing the MVC – even with the various PHP frameworks. If you know the MVC pattern it may be an idea to look at the frameworks supporting it, but don’t get carried away.

  • If all your old code is non-MVC code and you don’t have any plans to MVC-refactor it, it may cause you maintenance overhead to try a completely different style  on a single project/web application.
  • If you aren’t used to using a (MVC) framework you need to invest time in learning it.

Do keep an eye on the overhead caused by the framework. If you web application has a heavy load and the framework causes 5-10% overhead, it may a bad idea.

2) Ajax Frameworks
Great idea.. but to utilize the power of an ajax framework, you need to learn how to use it. Also Ajax isn’t the holy grail of the internet. Sometimes it’s cleaver to use it, other times it isn’t. By using plain old HTML and reloading the page from time to time, it may be easier to debug the application.

Do also notice, that writing great/professional quality Javascript often requires just as great skills as writing professional PHP.

3) IDEs
An IDE – Integrated Development Environment – can be a great idea if you know your way around it, and use it regularly. An IDE is – like any other tool – something you need to invest time and resources into, if you want something back – and especially if we’re talking a beast like Eclipse-based tools.

4) Database creation/management software
Let me skip comments on database creating management tools. I’ve never really used them and don’t know if they’re a pain or a benefit. I have often seen though, that too easy access to the database, leads to sloppy database design (or at least something, that isn’t thought through).

So far it’s been my experience, that people who knows the command-line clients to databases, often has invested much more time in thinking their data models through and often spend a little more time considering the table layout, the field definitions, indexes and other important aspects, than those who use a nice graphical interface.

5) Object Relational Mapping
Object-Oriented programming isn’t a holy grail, and most people doing OO programming (in my experience) doesn’t genuinely do object-oriented programming – instead they use the OO-facilities in the language to slap functions together in a class.

There’s nothing wrong with procedural programming as such, and trying to force object-orientation into your code – and even abstracting it away through Object-relational mapping, has often lead to slow performance and odd database designs. If you know what you’re doing great, but there’s no free lunch, and trying to utilze ORM without having a pretty solid grip on what’s happening and why, rarely does anything good to your productivity.

I’m quite sure for some developers, using some sort of object-relational mapping might be a gain in programmer productivity, but do keep in mind, that if you’re developing a site with massive traffic, the programmer gains may easily be crushed in added server resource requirements – spending a few minutes extra optimizing and tuning applications, may in some cases save hours and hours of cpu cycles once the solution is deployed.

Getting back to the article, I’m sure the advise is well-meant and thought through by the author, but do always – as with all advise – think it through and see how may apply to your adoptation. In may daily development some of it is quite terrible (as I often work on a site with thousands if not millions of pageviews every week).

I’m wondering what my 5 tips would be for developing PHP applications fast would be. Check back in a few days, and there may be an answer – which may or may not – apply to you.

Code archeology

I’ve been spending quite some time the past days digging around in old code (3+ years old with no or very little changes). It’s quite fun noticing what efforts pay off and which doesn’t when you go back to make chance to old stuff. I’m sure my observations aren’t generally applicable. We’re web developers and are keeping web platform (several pretty big websites on a common codebase) alive year after year.

Proper and sane variable- and function names pays off instantly. Incredibly short or misguiding function and variable names doesn’t. While it may be fun naming functions “doMagic” or likewise, chances that you – or someone else maintaining the code years after will probably not appreciate it at all.

Reasonable commenting on functions and tricky code parts often come to good use, but too much commenting is distracting. Some is good, and less is often more valuable.

Version control logs can be quite valuable, if they’re descriptive for the change made. Too often though, was the comment “misc. bug fixing”, “problem solved” or something which years later was of no use or value at all.

Documentation kept apart from the source code seem to have none or very little value. Not once have I bothered to look in the wiki-documents or any other documentation which was in the code. In the few instances I did, minor changes and bug fixes had changed the code to such an extend (while not updating the documentation) that it was misleading at best.

The only cases where the documentation apart seem to make a positive is database diagrams and descriptions. They do seem to have value.

Code exit strategy – 3 tips

It seems many developers get stuck in the same systems maintaining the same code for years and years. While it may be a common phenomenon there are a few things you can do as a developer to avoid being trapped in your own code forever.

First make things readable.

While your brain may be wired to a system of only using single character variable and function names or naming global variables after your cats, no one else will get the system. Try to make all names as logical as you possible can. Making it easy to read the code – not only as syntax but also in the names used in classes, functions and other objects in your source, improves the likelihood that someone else can figure out what’s going on and inherit the code someday.

Second make the code do what it supposed to.

Sure it may be cool making the system a generic all-purpose Swiss army knife of software, but preparing the code for all sorts of functions with aren’t used and aren’t needed, it only cause confusion of the next developer trying to move your code forward. Objects and class hierarchies may be suitable in some cases, but sometimes it’s more wrapping and declarations than actual code – and you’re probably on a wrong path.

Third provide in-line documentation.

Documentation are like a ghosts – it probably doesn’t exist. Making pages of documentation is probably a neat way to spend time if your on billable hours, but if you’re a regular developer, documentation is – at best a necessary evil. Do try to provide brief relevant comments, when something complicated or “not too obvious” happens in the code. Don’t place small novels in the source, but short precise guidelines.

While the above tips doesn’t guarantee your freedom to move on to new exciting projects, it surely improves the odds dramatically when you code is inheritable and accessible by another developer.

PHP Coding Standards

If you’re part of a larger team developing (and sharing) code, it is a pretty good idea to have some common standards. In the PHP work, such a set of common standards include the PEAR Coding Standards. While it may be a challenge to agree on standards, it can be done – the hard part is following the standards.

One excellent tool to help you check for standards compliance is the PHP CodeSniffer. We’ve had it running for quite awhile and it seems to work as advertised. We’ve made our own (“TDC”) coding standards which is a variation over the PEAR coding standards, and checking if you follow them is as simple as running a simple command line.While it doesn’t enforce following of the coding standards, it does make it a no-brainer to check if you do follow the common standards.

The output from the codesniffer could look something like this:

mahler@mahler$ phpcs --standard=TDC Example.php

FILE: Example.php
--------------------------------------------------------------------------------
FOUND 44 ERROR(S) AND 2 WARNING(S) AFFECTING 38 LINE(S)
--------------------------------------------------------------------------------
[LINE 2] ERROR: "require_once" is a statement, not a function. No parentheses are required.
[LINE 5] ERROR: "require_once" is a statement, not a function. No parentheses are required.
[LINE 7] ERROR: Space before opening parenthesis of function call prohibited
[LINE 9] ERROR: Space before opening parenthesis of function call prohibited
[LINE 20] ERROR: Line indented incorrectly. Expected 0 tabs but found 1.
[LINE 49] ERROR: Line indented incorrectly. Expected 1 tabs but found 2.
[LINE 51] WARNING: Equals sign not aligned correctly. Expected 1 space, but found 2 spaces.
[LINE 59] ERROR: Space after opening parenthesis of function call prohibited
[LINE 61] ERROR: Line indented incorrectly. Expected 2 tabs but found 3.
[LINE 61] ERROR: Expected "while (...) {\n" found "while(...){\n".
[LINE 63] WARNING: Equals sign not aligned with surrounding assignments. Expected 3 spaces, but found 1 space.
[LINE 74] ERROR: Line indented incorrectly. Expected 1 tabs but found 2.
...

Quality of code

Not  all code are created equal. Some pieces of code are more important than others. The code that powers critical parts of a nuclear power plant is hopefully of a much higher standard than the code behind this site, but how do you recognize which quality of code you should aim for when developing websites?

A website may not have one set level for all content – usually the various pieces which make up the site (no matter if it be classes, files or functions – in PHP). On the level you’re looking, let me suggest, for each piece you judge the required (minimum) quality on two simple parameters:

Business/Real world value: What’s the consequence if the code breaks? are you loosing money? or is it “just” an embarrassment”? The higher the stakes the more effort should be put into the code.

Scope of impact: How large a part of your website breaks if the code fails.

While you may not have a precise scientific way of measuring the two, I’m quite certain, that most developers have a pretty good gut feeling on both, and ought to be able to set a suitable quality level using just that.

Is it so or are there better ways to find acceptable quality levels?

How do you know good code?

One of the great challenges of PHP is that it’s so easy to learn, that just about anyone can learn it with not too much of an effort. While this is great for the number of PHP developers, it also seems to have the effect, that there is a huge number of bad examples of code out there. How do you then know good code?
In my book there are a few signs, which you could judge from – and they may even apply broader than just php-code.

First sign: It does the job

The code should be designed for the challenge at hand. Too often developers seem to apply the same style and energy to solve a challenge – no matter how much they differ in complexity and importance. If it’s a simple little function being developed, it shouldn’t require a huge framework.

If it deals with financial or personal data, it should probably utilize transactions and apply logging. Did the developer think of the challenge they set out to solve? – if so it’s a good sign.

Second sign: Well-structed

How does the sourcecode look? Are there mile long lines of code or are they sanely formatted? Is the code broken into functions, classes or another structure – or is it just page after page of sourcecode? – I don’t need to see all code as classes nor as neat function-libraries, but I do like if the developer has made an effort to break an application into some manageable pieces somehow.

Third sign: Reasonable naming scheme and comments

How does the function names, classnames and variable names look? is it random garbage or does it make sense? – I really hate variables named from $a to $z – and I do hate functions named “doSomething” – without specifying it further. I would expect great code to utilize the same naming conventions (CamelCaseing, underscores and so on) across all functions and/or variables.

If strange – as in unnatural/unexpected – things happen in the source code, I would expect a (short) comment explaining what’s going on.

Fourth sign: Security and Contingency

Did the developer think about security? is the code wide open for xss attacks? is input validated? Is “the unexpected” handled gracefully or does the code explode if exposed to simple URL-manipulation? Do you know what a SQL-injection is? If you need data from another source, what happens if it isn’t available? – does the code blow up or does it burn gracefully?

How do you recognize good code?

Checking requirements for PHP Applications

A great strength in PHP is that it’s so accessible, that almost anyone can get access to a PHP site and often will download “applications” for their site such as discussion boards, photo galleries and others. While it may easy to download such applications, it often happens that the people trying to get things to work, doesn’t read the requirements section of the install.txt
or readme.txt file, and thus never get the application running and doesn’t understand what the problem was and how to fix it. This is a brief guide with some tips on how to improve usability in this area.

First things first

Instead of relying on the user to read and check all steps in a text file, you should provide the install/setup file as a PHP-file, which is called through the browser. By doing this, you put the user in a familiar environment, and enables you to use all the power of PHP to guide the user through the requirements and install procedure. Assuming you think that’s a
bright idea, let see what you can do in such a setup-file.

Checking for the correct PHP version

First, if your Application requires a certain version of PHP, you should check if it’s available.

  • There’s a php function called phpversion() which will return a string with the current version number as a string.
  • PHP has a constant defined called PHP_VERSION. It contains the same string as the phpversion function returns.

If your application requires a certain version of PHP, there’s also a function called version_compare(),
which can help you check for a specific php-version or at least a certain version (ie. 4.4.0 or newer).

You use the function like this:

1
2
3
4
5
if (version_compare(php_version(), "4.4.0", "&gt;=")) {
// you're on 4.4.0 or later
} else {
// you're not
}

Checking for available extensions (compiled into PHP)

There’s function called get_loaded_extensions. It returns all the extensions compiled into PHP,
and if you need a certain extension, you can use this, to see if it’s available.

1
2
3
4
5
6
7
$load_ext = get_loaded_extensions();
 
if (in_array("soap", $load_ext)) {
echo "SOAP is available";
} else {
echo "SOAP is NOT available";
}

The extensions also have different version numbers, and if you need a certain version number to get your application to work, we can use the phpversion function from above to do this. The function can take an extension name as parameter, and will return not the PHP version, but the version of the extension instead:

1
2
3
4
5
$load_ext = get_loaded_extensions();
 
if (in_array("soap", $load_ext)) {
echo "soap extension available in version: " . phpverion("soap") . "&lt;br&gt;n";
}

If you do need a specific (or “at least”) version of the extension, you should do the check as in the above example, where at least a certain PHP version was required.

Checking for Zend Platform

If you’re running with Zend Platform enabled, it adds some extra features, which might be neat to use (if available) such as caching or error reporting. Since most PHP environments doesn’t have Zend available, we should check for the availablility before using the functions. One way of doing this is by using the “function_exists” function in PHP. It returns a boolean (true/false) answer.

1
2
3
if (function_exists("monitor_custom_event")) {
monitor_custom_event(........)
}

(the monitor_custom_event()-function is specific to Zend Platform – if available, we can use it.)

Checking for include files availablity

Some libraries are not compiled into PHP but written as PHP scripts themselves. Examples include MagpieRSS, CakePHP, Zend Framework and others. Checking if they are avaible without causing PHP to throw an error or at least warning message is a little tricky.

Here’s one way to do it. Turn off all error reporting, try to include the file, turn on error reporting and then check the list of included files to see if it was included:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
$fileToInclude = 'SomeFile.inc';$initial_setting = error_reporting(0);
include($fileToInclude);
error_reporting($initial_setting);
$files = get_included_files();
$available = false;
$pattern = "/" . $fileToInclude . "$/";
 
foreach ($files as $file) {
if (preg_match($pattern, $file)) { $available = true; };
echo "$file n";
}
 
echo "The file \"$fileToInclude\" is ";
if (!$available) { echo "not "; };
echo "available";

Do be careful before applying this technique. First you should always be very careful
before toying with the error reporting (you don’t fix errors by turning off error reporting),
and second, while this technique works, it’s not lightweight, so it should only be used in
setup and configuration scripts.

Wrapping it up

Using PHP itself, to check if the requirements posed by your application are satisfied,
should be quite easy. Often however PHP applications seem to rely too much on text files to provide important information. In PHP all the tools and handles you need to check the requirements are available, and with HTML-producing PHP, you can easily provide links and guidance t the user, to help them get any missing pieces, to successfully install you application; so do that please.