Mark A. Grant

Searching Code

2007-06-23T21:43:00.000+01:00

Integrated Development Environment's (IDE's) are stuck in the doldrums when it comes to finding the right bit of code for a job. Today's web 2.0 applications lead the way in information retrieval, but these tools are not available in the world of software development.

IDE's today

Current IDE's are based heavily around directory structures (namespaces) for grouping similar code together. Think System.XYZ. They have some simple tools for following references, i.e. go to the definition of a function or find all references to a function, and a rudimentary search engine.

The current state-of-the-art in Web 2.0 Information Retrieval

For large sets of data Search has kicked Directory structures into touch. Search however is not contextual (its dumb and has no understanding of context). This leads to billions of results may of which are irrelevant. Web 2.0 companies have tried to overcome this through tagging and mark-up (the Semantic Web), which has more meaning, but requires effort on the part of the user (i.e. adding that tag). This is being overcome by smart algorithms that personalise your information based on your previous history of activities - think Amazon and "People who read this also read....".

What are we missing??

Viewing related information to the task currently being undertaken - hey, you are changing interface X - here are all the classes, which implement it. Or, you have just named a class Y here are all the other classes with similar names
Better Search - common can we get a "Code Rank" algorithm - here are the most popular functions/classes based on their links (references) - and don't give me "you need to decouple your code more", some algorithms are more fundamental - like adding to a list
Personalised Search - using the historical results from the work you are doing to hone your search into something useful.
Better Collaboration - none of this I'm going to shove IM in a toolbar and let you get on with it. I want to get realtime visualisation of which of my team members are working on which bits of code etc

The Solution

Make the IDE a Web-App running on a server, with integrated backend into the Repository = Solves 4 for Collaboration and also Version Control
Builds all running on the Server-side = Faster Builds
UI update to include contextual results panes - Solves 1
Server-side search engine to cross correlate user's activity - Solves 2 and 3 for personalisation

Of course, this does come with a downside - us developers like to have the fastest, the whizziest, pc beast on the block, and if its running like a webapp we're not going to get the budget - Damn!

Nesting Splitters

2007-05-07T23:12:00.000+01:00

To recap on my previous post - C# provides the "yield" keyword that allows any function that returns an IEnumerable object to provide lazy iteration capability (a Custom Enumerator). This allows us to iterate across an array or list and filter elements, convert them to different types, aggregate them with another list etc.

My last post detailed how to "Split" an IEnumerable into two separate mutually exclusive enumerators. I.e. the equivalent of an if-else statement acting on each element.

At the end of the post I suggested that it would be great to output multiple enumerators i.e. the equivalent of if-then-else or switch type patterns. This may not be required as we can Nest Enumerators!

It goes like this:

inputList => split1 and split2 using our first test/predicate
split1 => split3 and split4 using our second test/predicate on the results of split1

We can then split the results add infinitum - creating a Binary Tree Splitter. At each node in the tree we queue the results of the "lagging" enumerator until it catches up. What is the benefit?

We can split out multiple enumerators and process each of them differently.

Splitting an IEnumerable into Multiple Custom Iterators

2007-05-02T00:49:00.000+01:00

Bill Wagners MSDN column on custom iterators is excellent. It demonstrates clearly how to create a custom iterator so that IEnumerable iterators can iterate through collections in a lazy fashion. That is only processing the elements in the collection that are required.

As soon as it becomes evident that custom iterators take an IEnumerable as an input and output an IEnumerable it also becomes evident that we can "pipeline" custom iterators.

For Example:

Start with a Collection of People
Filter out all people with surname Grant
Filter out all people with first name Mark
Display List of Results on screen

This is a simple pipeline like so: 1 -> 2 -> 3 -> 4

We can also Transform iterators into different types and we can Merge or Aggregate multiple iterators into one be iterating over each in turn. This got me thinking is it possible to split an iterator into mutiple iterators.

By splitting an iterator it allows us to process the different sets of results differently. As an example:

We have an iterator that recurses a file structure. We need to split out two iterators -one for files and one for directories.

Out input iterator is of type IEnumerable and splits into a directoryIterator and fileIterator of type IEnumerable. These can then be Transformed into an IEnumerable and an IEnumerable respectively.

Having asked Bill if he could blog a solution I had some inspiration! Here is a test case for the solution:


using MarkGrant.Collections;

using NUnit.Framework;
using System;
using System.Collections;
using System.Collections.Generic;
using System.Text;

namespace MarkGrantTest
{

    /// <summary>
    /// 
    /// </summary>
    /// <originalAuthor>mark.grant</originalAuthor>
    /// <creationDate>01/05/2007 23:58:19</creationDate>
    [TestFixture]
    public class TestSplitter
    {
        [Test]
        public void Test()
        {
            List<int> list = new List<int>();
            for (int i = 0; i < 20; i++)
            {
                list.Add(i);
            }

            // intialize our splitter
            Splitter<int> splitter = new Splitter<int>(list, delegate(int number) { return number % 3 == 0; });
            IEnumerable<int> factor3 = splitter.GetMatchingElements();
            IEnumerable<int> nonFactor3 = splitter.GetNonMatchingElements();

            foreach (int factor in factor3)
            {
                Assert.IsTrue(factor % 3 == 0);
            }

            foreach (int nonFactor in nonFactor3)
            {
                Assert.IsTrue(nonFactor % 3 == 1  nonFactor % 3 == 2);
            }
        }
    }
}

The test creates a Splitter class that Matches multiples of 3 and non-multiples. The IEnumerables can be called in any order and can overlap. As we don't know which iterator will be called the splitter will need to record the current state of the two iterators - i.e. which elements are left, which iterator they are related too etc. This can be achieved by queuing elements that are requested but belong to the other iterator.

The Splitter class:


using System;
using System.Collections.Generic;
using System.Text;

namespace MarkGrant.Collections
{
    /// <summary>
    /// Split an input enumerable into two mutually exclusive enumerables
    /// </summary>
    /// <typeparam name="T"></typeparam>
    public class Splitter<T>
    {

        /// <summary>
        /// Are we queuing the results from the first enumerator or the second - it
        /// depends on which one is ahead
        /// </summary>
        private bool _queuingFirstIterator;

        /// <summary>
        /// We Queue one Enumerator unless the two are perfectly synchronised
        /// </summary>
        private Queue<T> _unyieldedElements = new Queue<T>();

        /// <summary>
        /// Our filter
        /// </summary>
        private Predicate<T> _predicate;

        /// <summary>
        /// Our input enumerator
        /// </summary>
        private IEnumerator<T> _inputEnumerator;

        /// <summary>
        /// Construct our Splitter with the input enumerator and the filter
        /// </summary>
        /// <param name="inputEnumerable">the input enumerator</param>
        /// <param name="predicate">our filter</param>
        public Splitter(IEnumerable<T> inputEnumerable, Predicate<T> predicate)
        {
            if (inputEnumerable == null)
            {
                throw new ArgumentNullException("inputEnumerable");
            }

            if (predicate == null)
            {
                throw new ArgumentNullException("predicate");
            }
            _inputEnumerator = inputEnumerable.GetEnumerator();
            _predicate = predicate;
        }

        /// <summary>
        /// All elements that match the predicate
        /// </summary>
        /// <returns></returns>
        public IEnumerable<T> GetMatchingElements()
        {
            return GetNextElement(true);
        }

        /// <summary>
        /// All elements that don't match the predicate
        /// </summary>
        /// <returns></returns>
        public IEnumerable<T> GetNonMatchingElements()
        {
            return GetNextElement(false);
        }

        /// <summary>
        /// Returns the next element for our enumerator
        /// </summary>
        /// <param name="isMatching"></param>
        /// <returns></returns>
        private IEnumerable<T> GetNextElement(bool isMatching)
        {
            while (true)
            {
                // if we have an element in the queue for this iterator then
                // yield it otherwise next element in the inputEnumerator is
                // for this enumerable then yield it otherwise add it to the queue
                // break when no more elements remain in either
                if (_queuingFirstIterator == isMatching && _unyieldedElements.Count > 0)
                {
                    yield return _unyieldedElements.Dequeue();
                }
                else if (_inputEnumerator.MoveNext())
                {
                    if (_predicate(_inputEnumerator.Current) == isMatching)
                    {
                        yield return _inputEnumerator.Current;
                    }
                    else
                    {
                        _unyieldedElements.Enqueue(_inputEnumerator.Current);

                        _queuingFirstIterator = !isMatching;
                    }
                }
                else
                {
                    break;
                }
            }
        }

    }
}

The Splitter class is initialized with the input IEnumerable and the Predicate that is used to filter the elements into the Matching and NonMatching itereators. As we enumerate through the input iterator, we keep iterating until we find an element that matches the one we are after, all the elements that we don't want we queue and a flag is set to record which iterator is "lagging" behind and therefore requires its elements to be queued.

We could for example alternate the requesting of results for either iterator. For example when matching factors of 3 starting with the iterator that matches 3, 6, 9 etc and an input list of 1,2,3,4,5,6 we get, where M is the Matching iterator (i.e. anything which is a factor of 3):

M - Queue (1)
Yield (1)
M - Queue (2)
Yield (2)
M - Yield (3)
Yield (4)
M - Queue (5)
Yield (5)
M - Yield (6)

If anyone fancies improving the code to include multiple predicates, please let me know.

Great Web GUI article

2007-04-10T19:23:00.000+01:00

This article by Bret Victor really stands out when it comes to Web Usability. Not only is it well thought out and easy to read, but the main points are very well made. Worth reading by anyone producing software that interacts (or not - see the article) with humans.

Compression and Communication in Software Development

2007-03-25T19:55:00.000+01:00

The process of refactoring software code is something akin to a compression algorithm. As an example let us take a simple lossless image algorithm that takes runs of the same colour and stores the colour and the number of pixels in the run. We can think of this as finding two functions that are the same, but with different names and creating 1 function and pointing the original two callers to the new function. This process is about abstraction.

Fair enough you might say, but we can take this a bit further. Thinking about compression triggered some thoughts about Information theory I learnt at University. Namely, Shannon's Law- so I looked it up on Wikipedia, and here is what it had to say:-

it is a felony offense to discharge firearms randomly into the air, resulting in injury or death.

Err, ok maybe not that Shannon's Law (in Arizona). A little further digging on the right pages reveals Coding theory (no it not about programming!!!!!). Ah, and now I have found the bit of information I am after (and its a 1 - sorry poor joke -it's a Sunday evening!!!)

Entropy of a source is the measure of information. Basically source codes try to reduce the redundancy present in the source, and represent the source with a fewer bits that carry more information.

SOURCE ENCODING

Translated to software development - there is a minimum number of lines of code required for each program, where there is absolutely no redundant code. This can commonly be seen in some coding competitions, where the winner of a challenge solves the problem in the fewest lines of code.

Putting aside readability, quality, comments, maintainability, the meaning of your code or communicating your code with other developers etc, we can surmise the following. When your code is represented in the most concise manner, no two lines or pieces of code should look the same, do the same thing or repeat any other lines of code. This is because any lines of code with the same data are redundant and do not transfer any new information to the compiler.

So here is the problem, in many languages it is IMPOSSIBLE to make code concise and remove redundancy - thereby reducing the entropy of the code. Take for example design patterns, they are seriously REDUNDANT!!!! The mere fact that they exist, they provide the same construct in multiple scenarios really means they should have been written once, not millions of times by millions of developers!!! (Mind you I have now used generics to write a Singleton - ONCE)

C# has just created some "syntactic suger" to reduce the amount of code required to write a property, which meets one of the key tenets of Information Theory. Namely, use short words for the most frequently used words!

So, on the flip side, is:

CHANNEL ENCODING

When communicating information we need to add redundant information to ensure our message is understood - think listening to a fuzzy radio station - sometimes it is possible to understand without hearing every word!

So, in programming terms, we have two key audiences/recipients for our code:-

The compiler
You, the developer, and other developers

We add large amounts of redundancy for comments and in terms of code structure in order to effectively communicate the meaning of our code. At this point, due to tiredness I am starting to forget what I really wanted to say! A quick wrap up and I'll think about it some more!!!

1. We can apply some of the theories of Information theory to Software development

2. We apply Source Encoding as we refactor/abstract our code

3. We apply Channel Encoding to improve the communication of our code

4. Finally, we are slaves to our compilers when it comes to removing redundant lines of code

Currying, Closures ad how to screw your mind in 1 line of code

2007-03-16T20:43:00.000Z

Ok, so we can pass functions as data using delegates. I recently started using delegates as return values for functions, and this can lead to some odd syntax!

So, the 1 line of code:

int answer = Add(5)(3);

Note the second set of brackets - where the hell did they come from!

So here's the add function

public static AddNumbers Add(int x)
{
return delegate(int y) { return x + y;};
}

where AddNumber is a delegate (function pointer) defined as:

public delegate int AddNumbers(int y);

What the hell does this mean?

When we call Add(5) we are doing two things:

1. Setting the variable x within the anonymous function's return statement
2. returning a function, which takes a parameter y

As the return value of Add(5) is a function we can then call the function Add(5)(3); which will return the value 8!

This leads to two important functional concepts:

1. Currying - a function which takes multiple parameters can now be split into multiple functions each taking one parameter i.e. Add(int x, int y) becomes Add(int x) : return AddNumber(int y)
using the stack to store the x variable

2. Closures - the anonymous method can "see" the variables within the Add(int x) method of which it is a part - just like the contents of an if statement would. The complexity here is when the anonymous function is returned from the Add function

We can use currying to intialize parameters. So for example lets say we wanted to do the following sums:

answer = 3 + 5
answer = 3 + 7
answer = 3 + 9

rather than do:
int answer1 = Add(3, 5);
int answer2 = Add(3, 7);
int answer3 = Add(3, 9);

we can initialise the 3 parameter using the Add function, and rewrite our code as:

AddNumbers adder = Add(3);
int answer1 = adder(5);
int answer2 = adder(7);
int answer3 = adder(9);

This becomes really useful when out initialization occurs in a different part of our code to the actual "running" of the code. It could also allow us to create an "immutable" function, where we initialize all of the functions parameters at one point in our code and return a parameterless function.

Applying Control Theory to Software Development

2007-03-16T19:50:00.000Z

Many moons ago I learnt control theory - possibly the hardest final year engineering paper at Cambridge. And it had the following to say:

Every system can be treated as a black box with inputs and outputs - ok no revolution here, but I need a starting point!

It also said:

Observability is a measure for how well internal states of a system can be inferred by knowledge of its external outputs - thank you Wikipedia.

and,

Controllability denotes the ability to move a system around in its entire configuration space using only certain admissible manipulations - Wikipedia's entry is not so helpful a translation would read how much of the output can we control using the inputs

So, Observability is the ability to deduce whats going on inside by whats coming out - a bit like the stool analysis and Controllability is the ability to configure the system via inputs - the equivalent of the atkins diet.

Now in software systems, we can apply the same ideas. Namely, when I write a chunk of code it is a black box system - stuff goes in and stuff comes out!!!!!

In fact several programming techniques are based entierly on this principal, namely, Inversion of Control Design pattern, Test Driven Development and large chunks of Functional Programming concepts.

In fact, my normal gripes with any framework (Java or .Net) typically occur when the code in the framework is not Observable or Controllable enough. In fact code can typically be improved by increasing ones ability to configure it and by providing clear outputs for given inputs!

delegates and interfaces

2007-03-03T18:51:00.000Z

In trying to explain some functional concepts in C# to an OO programmer, it became apparent that there is a quite clear correspondance between an interface and a delegate (function pointer).

The following is a list of comparisons:

An interface provides a level of indirection/abstraction for a class - a delegate provides a level of indirection/abstraction for a function
An interface provides signatures for a bunch of functions/properties - a delegate provides signatures for a function
both can be passed as variables
An interface will only work with an object, whereas a delegate can work with static and instance methods
Finally, delegates can be "invoked" or called asynchronously
An interface quite clearly relates to a class
An class can only implement an interface once, but can have many functions, which match a delegate

In many cases a delegate can provide simpler implementation than an interface, especially when the function is stateless and can therefore be made static therefore reducing side-effects. Its food for thought anyway!

F#

2007-01-25T22:09:00.000Z

Having spent a good few hours looking into functional programming -> and what a revelation it has been. I thought I would scout out F#. F# is a functional language (with object-orientated and imperative coding acceptable as well) based on top of .Net. I am not sure whether adoption of F# will be very high as C# and VB.Net are going functional (more so) in .Net 3.0 with support for lambda expressions and LINQ.

Even more interesting was finding out that the key developers of the language are based at Microsoft, Cambridge, which is just down the road from me. Their research homepage is here and the lead developer Don Syme's blog is here.

Whilst I'm at it here are the sites of some of the guys I know at Microsoft, Cambridge. They are worth looking at for the cool research in Machine learning and vision.

John Winn's homepage
Oli Williams Cambridge University page

Using Code Generation for Dynamic Proxies

2007-01-01T19:42:00.000Z

Having given code generation a hard time in a previous post, there is one area, where I can definitely see Code Generation providing a valuable solution - Dynamic Proxies. Code Generation typically falls short because it is used within the design cycle and is a bit like baking a cake - its a bit hard to add raisins once you've baked it! With Dynamic Proxies however Code Generation is used at runtime and is a whole different kettle of fish.

What are Proxies? You have a class, which you would like to serialize, add security, logging or maybe even thread locking. In each of these scenarios you typically which to apply logic to public properties and methods (although it is sometimes necessary to serialize private fields). A proxy class will encapsulate our class and expose the classes methods itself, typically by implementing the same interface as the class.

An example:

// class stub
public class Person implements IPerson
{
public String Fullname;
}

// proxy stub
public class ProxyPerson implements IPerson
{
public IPerson Person;

public String Fullname
{
// some proxy logic
return Person.FullName;
}
}

The Person class implements the IPerson interface and the proxy class also implements IPerson interface as well as exposing the original Person class.

What is a Dynamic Proxy? Simple a proxy class generated dynamically at Runtime.

How does it work? We specify the class we wish to generate a proxy for. In .Net this is achieved by decorating the class and its methods with Attributes (does Java have attributes?) When we run our code our proxy generator then reads the attributes and uses them to generate the proxy class e.g.

IPerson personProxy = ProxyGenerator.Proxy(new Person());

We can then us the personProxy instead of the original class.

This sort of technique could also allow us to decouple other functionality from a class such as Equals, GetHashCode, Clone etc leaving our core class less cluttered with Utility functions.

The Person example with Attributes:

// class stub
public class Person implements IPerson
{
[RequiredForEquals()]
public String Fullname;
}

All properties marked with a RequiredForEquals Attribute can be picked up and compared. The comparison code would look something like:

Equalizer personEqualizer = new Equalizer(typeof(Person));

personEqualizer.Equals(person1, person2);

Behind the scenes the Equalizer would use Reflection to read the Attribute and Code Generation to generate our Equlas code before the Equals operation is performed.

// class stub
public class Person implements IPerson
{
public String Fullname;
}

But I can do this with Reflection only! Yup, but Reflecting over a bunch of classes, methods and attributes takes time - generating the proxy and caching it can allow the same proxy to be reused over and over again, with a little overhead on the proxy generation.

Performance

Both Reflection and Code Generation are costly operations, and therefore not something to be undertaken lightly, however, for all but the most time critical functionality they can be optimised by creating the equalizer when the program is loaded (by making the equalizer static).

A real life example, to the best of my knowledge this is how the .Net Frameworks XmlSerializer works. By making the serializer static I managed to improve my serialization code by ten times (originally a 3 second operation became a 0.3 second operation - a significant improvement!)

I have been tempted, but not found time/reason to create a library for creating dynamic proxies. IBM has a interesting Java post on this very topic.

Contracts in Object Orientated Code

2006-12-30T15:43:00.000Z

A school of thought, which is widely supported, suggests that when two pieces of code communicate they form a contract. That is if code A ("the client") calls code B ("the supplier") with some data it will expect code B to have certain behaviour, return certain data and handle exceptional/erroneous/odd inputs in a certain way. The idea is called "Design by Contract" (DBC).

It occured to me that DBC could be formalised by extending interfaces into Contracts.

About Design by Contract

In simple terms:

The Supplier expects input data to meet certain conditions - "pre-conditions"
The Client expects the Supplier to behave in certain way - if the Supplier changes or stores data as well as returning output data it is harder to predict what will happen.
The Client expects output data to meet certain conditions - "post-conditions"

Pre-conditions - the Supplier is responsible for checking pre-conditions

Behaviour - if the Supplier code does not alter the state of the system but instead just returns data to the client then the Supplier's code is said to be Side-Effect free - Functional Programming techniques attempt to minimise Side Effects and Imperative Object Orientated techniques attempt to Encapsulate them within a manageable number of lines of code. Behaviour is usually tested by injecting Mock Object's as input data into the Supplier.

Post-Conditions - Client test code is used to create Assertions about the Supplier's code. Test Framework's allow us to create Unit Tests that test different Post-Conditions.

How to extend Object Orientated (OO) Frameworks to support DBC.

Many OO frameworks such as .Net and Java leave it primarily upto the developer to use DBC techniques. As an example in C# a Pre-Condition might be:

if (inputParameter == null)
{
throw new ArgumentNullException("inputParemter");
}

This code simply checks an input parameter to ensure it is not a null reference and throws an exception if it is.

.Net does have an extension called Spec# that adds DBC functionality. Now I don't claim to be an expert on Spec# or any other specific DBC framework, however, it seems to me that DBC would be better implemented by "upgrading" interfaces into Contracts rather than decorating Classes with contract information.

Why extend interfaces to support DBC?

An interface by its nature has public methods, properties etc - it is public and therefore requires a contract
An interface can then have any number of concrete implementations, which all meet the pre and post conditions
An interface can then be a single port of call for all contractual information - function declarations, pre-conditions, post-conditions and exception types
This would negate the need for verbose commenting of functions and reduce the need for a throws keyword in Java
It would reduce the clutter in classes

An Example:

An interface IPerson, which has a single property Name, which cannot be null might look like:

contract CPerson
{
[PreCondition: NotNull(Name)]
string Name
{
get;
set;
}
}

A concrete class would then "agree to"/"support" a contract, including any Pre-conditions and Post-conditions etc.

Any thoughts on this idea are more than welcome!

Top 5 free .Net tools

2006-12-28T21:08:00.000Z

On a daily basis I use some or all of the following tools for .Net development:

.Net Framework - it is perfectly possible to develop .Net applications without Visual Studio, this can be done by downloading the .Net Framework Software Development Kit and using your preferred Integrated Development Environment - Eclipse would be a good choice.
FxCop - an automated rules-based code auditing tool that audits your compiled code. Checks include: naming conventions, multilingual support, null reference checks for public methods and functions, to name a few. Rules can be turned off and on and new rules created.
.Net Reflector - uses reflection to reverse engineer .Net assemblies into managed code (c# or VB.Net). It can also decompile the .Net framework core libraries, which is invaluable way of both understanding the logic behind certain functions or improving your coding techniques (mind you there are times when you would like to rewrite some of the code :-)
NUnit - a port of the popular Java equivalent JUnit. A fast way to undertake Test Driven Development, especially with the next piece of software.
Test Driven .Net - an inspired piece of software that allows developers to run NUnit test cases from within Visual Studio. Literally right clicking a test case (and selecting Run test) to run a test or a test class to run a whole suite of tests! Cool!

Some other free tools I haven't used, but would love to get some feedback on are: NAnt (build manager), NDoc (C# automated documentation), CLR Profiler (performance profiles).

Also of interest may be the Mono Project that is a Linux/Solaris/Mac OS X/Windows/Unix implementation of the Common Language Runtime (CLR). This allows .Net applications to run on non-Microsoft platforms.

Book Review: C# Cookbook by O'Reilly

2006-12-28T20:08:00.000Z

C# Cookbook is well worth purchasing and is aimed at moderate to advanced level C# developers. It gives good coverage of all the major system libraries and programming techniques and includes "recipies" on all the advanced .Net 2.0 functionality.

I typically end up purchasing 3 types of book on any new language I am learning. These include:

Simple Practical Introductions - step-by-step guides that have plenty of examples and cover both the programming language and the development environment (I have managed to skip buying these recently in favour of internet tutorials)
Reference Books - these cover all the core libraries associated with a language including all classes and their public interfaces
Advanced Guides - specialist topics covered completely and in depth

C# Cookbook does not fall neatly into any of these categories. The Cookbook is about as close as any book to being a single source for C#. This is because it gives loads of practical examples (which are well written and not hacks), covers all the major areas and solves advanced problems. You need look no further to delve deeply into threading, generics, anonymous functions, collections and xml. Happy reading!

Installer Driven Development (tm)

2006-12-23T04:02:00.000Z

Ever been in a software development project with limited specification, tight deadlines, grief, pain, and suffering -- a flying albatross that just won't die. Your cure may just be Installer Driven Development (IDD). The concept is simple: focus on your installer package first and your code second. If this sounds entirely insane and counter intuitive then read on (especially if you've worked on software projects that suck).

A typical project scenario - a la Dilbert.

Day 1: Senior Exec has great idea for making money by selling a new software product to customers
Day 2: Senior Exec speaks to a few Senior Exec's he knows working at Customer sites who say they think its great (this is undocumented, unquantified, unqualified and turns into a solid business case by "the Power of Management")
Day 3: Senior Exec tells Middle Manager that customers are demanding Product X the solution to all their problems and that they need it by Y, and by the way this is vital to the company and he could gete promoted yadda, yadda
Day 4: Middle Manager has meeting with Tech team who say, "sure we're great we can deliver X by Y" and so that albatross starts its run up....

Now, days 1 to 3 cannot easily be altered as they are outside your control, but day 4 can.

Revised Day 4: Middle Manager has meeting with Tech team who say, "we are going to use IDD to ensure that we involve the customer, get the specification 100% correct and can deliver a demo that we can install on customer's machines for them to beta in time Z."

This has just saved Middle Manager's arse and he runs off excited to tell Senior Exec.

What do you do next - IDD?

1. Get a good installer program (Macrovision will do).

2. For you to install your demo you will need to know exactly what types of machine the customer uses, what browser, whether their IT department would install it, security, what other software they use etc. Get the Sales team to find out who the customers are and as many other important requirements as possible, point the Sales team at the Marketing team for all Sales "Collateral" add the tasks to the bug tracking system with weekly email reminders.

3. For your installer to look good you will require logos, help documents, links to websites, flash animations and a community website for your Beta customers etc. Get the Marketing team going, point them at the Sales team for all the customer profiles, feedback etc and add the tasks to the Bug tracking system with weekly email reminders.

4. You need to know how many versions of the installer are required (for the Lite version of the software, the Enterprise version etc). Get the Legal department to start producing the license agreement and software contracts, point them in the direction of Middle Manager for all pricing, product roadmap etc, add the tasks to the Bug tracking system with weekly email reminders.

5. Now you can either:

a) Sit back and enjoy the bun fight and write some code or
b) gather as much information from the different departments as possible, undertake all deployment testing and write a minimal Demo program that will be installed successfully from day 1.

6. Finally, ensure you provide regular progress reports to Middle Manager, highlighting the outstanding issues in the Bug tracking system for Sales, Marketing and Legal departments and point out that they have been given weekly reminders of their tasks.

Enjoy!

Visual Studio Templates and Code Snippets are great, but ...

2006-12-22T21:56:00.000Z

Anyone who has developed any Object Orientated code has re-written the same code patterns many times including creating a class (with comments), implementing getters and setters (or properties in .Net) and creating constructors. This is where templates and code snippets come to the rescue.

Visual Studio (Item and Project) Templates - when you create a new code file in Visual Studio you select from one of many default templates such as a Class, Interface or even a Windows Service. These then use code generation to fill in the blanks (such as a class name) and create your code for you. VS 2005 allows users to create their own custom templates, which is Great! I already have a class template, interface template and exception template customised with default comments, namespaces etc (actually the exception class is entirely generated for me). This is already saving time for myself and my team. Next up a Test Case template.

Visual Studio Snippets - these allow you to insert chunks of code into your file and then fill in the blanks, sort of like code-by-numbers. This is also great and I have created a code snippet for a property that cannot be initialised to null (I use this regularly as it is good design-by-contract programming). See example:

private $type$ $field$;

public $type$ $property$
{
get
{
return $field$;
}
set
{
if (value == null)
{
throw new ArgumentNullException("value");
}
$field$ = value;
}
}

Microsoft lovingly wraps a whole load of metadata around my code snippet in xml (the code is embedded into an xml CDATA section). This is also great and saves me time re-writing the same code.

And now for the 'but'... ...BUT this is entirely symptomatic of a broken programming paradigm. And in English - Object Orientated languages such as Java, .Net and C++ are flawed, as is the concept of Code Snippets and Templates.

The simplified logic as a Powerpoint bullet argument (hey, its late and I'm getting tired):

In a good programming language we would only need to write the property code ONCE, this point will not be explained as it is self-evident
We would therefore NOT need to insert and customize the same chunks of code in multiple locations
Which means Code Generation and therefore Templates and Code Snippets are redundant!!!

Templates and Snippets suffer from the code generation problem that once created, if the snippet or template are changed the code is NOT. This means that the programmer must then go through all their code manually finding and replacing the generated code.

A thought experiment - what if we were to utilise code generation NOT when we insert the Snippet into our class file, but when we compile the code and then refer to the Snippet instead. Hmmm, well this would be an improvement in that we would then only write our Snippet once, and wouldn't have to manually update any changes. Our new code could look like:

Snippet Property (type = String, field = myString, Property = MyString);

Or something along these lines. :-) The Snippet of code is now effectively a "Template" for a Property. And why not take it a little further, lets make everything a Snippet and make it so that Snippets can include Snippets and the compiler will generate all our code from Snippets. Oh S***, what about the metadata, that's in XML and our Snippets are in C#, how would we generate those, and hey, haven't we just changed our language into some new Snippet language.

The problem with Code Generation - is that we end up creating meta-languages (which is fine) that have a new syntax etc, because the language that the code is turned into is inadequate. A number of code generation frameworks use XML to store the customisable fields and XSLT to then generate the code (into C# or Java or whatever). Makes you wonder really, so what are the options:

Include a decent templating concept into the core language: Jura is an example
Use functional programming - C# is starting to do this, but the starting blocks of classes and properties and interfaces are hard to then abstract out.

And while i'm at it: any programming paradigm (Object Orientated programming) that requires design patterns is flawed. Mind you my hat goes off to the people who have managed to turn the design patterns into libraries using advanced OO techniques such as generics, reflection and code generation (there is a point to some of these techniques)!!!

To round up, code generation, templates, snippets and design patterns are all good concepts starting from a flawed language, namely an Object Orientated language.

Rant over :-)

P.S. just found this conference on Code Generation down the road from me - I might well attend!

What is programming?

2006-12-10T19:38:00.000Z

A slightly philosophical title, which will no doubt end up with a lot of introspection, reflection and sole-searching (no really, looking at the bottom of our shoes and trying to extract meaning from a piece of chewing gum). Why ask the question? A little thought skittered across my brain and went something like this: "Does writing a function in an Excel spreadsheet consitute programming?"

This thought was closely followed by a number of other thoughts, such as, is writing HTML programming? what about if we use a tool such as Dreamweaver? Hey, even more bizarre - can the creation of a Word document be called programming? I mean after all in Office 2003/2007 saving the file creates XML, and so does Dreamweaver. Let's face it this is clearly absurd, right?

Ok, lets separate the tools we use, the output data we create and our thought process.

Does the tool maketh the programmer? - Nope, this should be obvious, as even a monkey could use some of the Visual tools out there and still have no idea what they are doing.

Does doing maths make you a Mathematician? - Nope again, although we may generate sums and equations it certainly doesn't mean we could qualify ourselves as numerate. But, just maybe this does mean we need to introduce a sliding scale of programming ability, the idea of programmerness or how much of a programmer are you? Let's face it Excel provides many levels of abstraction that allow you to use the sum function on a column of numbers in the same way Java abstracts away from assembly.

Our thought process - so maybe it is purely our ability to comprehend a problem and provide a software based solution that differentiates us as programmers.

In the end, this means that we as programmers can argue as much as we like about the merits of language X over language Y, but using a particular language does not imply that we are better than the complete dorks programming in Perl, which is disgustingly horrible to read and only a no-brainer would even try :-) Moving on, maybe, just maybe, we should be striving, nay, moving mountains to provide the best software we can by understanding the many different ideas in different languages.

And, a note from my wife, who avoids anything technical like the plague - there's an art to it!

Bugbears

2006-12-02T22:10:00.000Z

Nothing gets me (or anyone else I have worked with) more wound up than unhelpful error messages, or even worse error codes! For that matter I get wound up (because I think I can do better) with any error messages at all. Here's the rational, using a day to day example:

RESTAURANT SCENARIO - You walk into a restaurant and site down. The waiter appears and says, "What would you like to order, sir?" To which you respond, "I would like the spaghetti bolognese, with extra parmesan and some garlic." Here are some potential responses when the spaghetti has run out:

No Response - blank stare - a system crash when everything hangs
"No" - the equivalent of an undecipherable error code, only a complete geek would NOT be offended, but instead look it up on the internet or a FAQ
"We have run out of spaghetti" - a typical response found by most software, it tells you the problem in a rather terse manner that leaves you feeling frustrated
"I am very sorry sir, but we have run out of spaghetti" - slightly better technique used by some websites, which leave you with a warmer feeling, but hell what do you do next?
"I am very sorry sir, but we have run out of spaghetti, perhaps you would like to try the lasagne, which is equally delicious." - not only does this describe the problem but gives the customer an option (or two)
"I am very sorry sir, but we have run out of spaghetti, we have more arriving tomorrow, perhaps you would like to try the lasagne, which is equally delicious." - not only does this describe the problem but gives the customer an option (or two) and shows the customer that the problem is being resolved

Of these, responses 5 or 6 are best, but software typically uses 3 or 4 and sometimes 2 (1 is typically outside our control). I have created 6 potential responses, but many more can be generated when pondering the following questions:

Does the user know an error has occured? Does the user know what the error is? Does the user know why the error has occured? Is the response user friendly? Does the user know what options are available? Does the user know how the problem will be resolved? Who is expected to resolve the problem, the user or the software? How is the message communicated? And, Is the user happy with the outcome?

This is one hell of a lot of stuff to think about for a simple error, but without it users will get frustrated, angry and stressed. This is BAD!!!

Simple Steps to reduce stress:

Use good coding practices/conventions to control errors/exceptional behaviour and validate input properly
Think about the a user's experience (be polite)
Give users options
Give users confidence that the error is a one-off (or very rare) and can be resolved
Resolve the error with as little disruption to the user as possible
gather feedback (in a non-intrusive way - you are now asking the user to do something when the software has died)

Some simple examples of where small changes make big differences.

www.petrolprices.com - in their forms they have textboxes that validate your entries immediately and put a bright green tick at the end. This saves unhelpful page refreshes where you then have to scan back through your web page to find the incorrect field entry.

Microsoft Feedback Gathering System - its optional, but at least it empowers users so they thinnk they can do something (if only a little)

Password reminder services - how annoyed would you be if you forgot your password and there was no option to get a reminder email!

Anyhow, without going on forever, think about the user, resolve the problem with as little hassle to the user as is feasible and ensure it doesn't happen again - You Bad Boy!

Technology Time Warp

2006-12-02T01:20:00.000Z

In the last few weeks I have had cause to try out Lisp and Haskell, both of which are pretty old technologies (although Lisp beats Haskell hands down being specified in 1958 as opposed to 1990). I have come to some interesting conclusions (which surprised me, but will not surprise users to these languages). Firstly, that programming language progression is not linear and secondly that I had missed (but been using in XSLT) an important programming paradigm.

Why try Lisp and Haskell? .Net 3.0 have some interesting new features, which step out from the purely Object Orientated view of the world and allow us to do some pretty cool stuff. These are namely Functional Programming (of which Lisp and Haskell are pure examples) and LINQ which provide relational query, set and transformation programming. Anyone who has used a relational database will instantly grasp LINQ, but what is Functional programming?

Functional Programming - Wikipedia has a great description of FP, but in essence a FP function does not store any state information. You can therefore blackbox FP code and with the same set of inputs you will always get the same set of outputs. This has performance and maintainability benefits, as software can scale (see Google's MapReduce function) and will not have side effects (they are also thread safe).

Conclusion - Was it worth trying Lisp and Haskell - in one word, yes. These languages provide great run-time interpreters/compilers and it took all of 3 minutes after download to start writing some pretty cool programs. Mind you trying to take in a new syntax (or two) with entirely different programming paradigms was pretty hard, but the upshot is a new way of thinking about coding especially for those of us brought up on Imperative Object Orientated code.

Some cool references:

Paul Graham on Lisp - a bit of a Lisp zealot, but is a great blog
Technorabble - this guy got here before me!!!

Post Zero

2004-11-04T17:19:00.000Z

My zeroth post on software development, business and programming concepts (a broad range, but should cover all need to know topics for people in IT!) . After 10 years in IT and start-up companies I finally feel like I might have some things to say about getting software off the ground and paid for by customers. In a nutshell its hard, extremely hard!

Wow there cowboy, lets qualify that last comment, rank it, digest it and put it in perspective. Ok, is it harder than a degree from Cambridge University in Engineering, for want of a better example? Or maybe, job interviews? Or presentations? Or say competing in the Olympics?

Ok, so academically it is probably easier than a Cambridge Degree, interviews and presentations are a walk in the park, and the Olympics is likely to be more physically demanding (let's face it typing really does only tax a few minor muscle groups), but ...

... it takes the dedication of an international athelete, the business and technical smarts of a Bill Gates and the tough sales driven attitude of Steve Ballmer to get the business off the floor and making money.

But how do I go soooopeeeer nova like Chad and Stev (Youtube founders) or Larry and Sergey (Google founders)? I've no idea, please refer to one of their biographies for inspiration, or alternatively I'll sell you one.

STOP, of course I have an idea, given that Youtube sold to Google, you need to create a company, raise loads of money, get loads of people using your software and sell it to Youtube, thus extending the chain and increasing Google's e-domination. So, an automated tool to generate good content :-0

What remains to say? Well, maybe a few hints and tips, some juicy reviews of the current state-of-the-art-or-is-it-a-science and some anecdotes taken from the front line. Bye for now, Mark

P.S. To blogger, please can you update your software to update my date when I update my post.

P.P.S Published 23rd November 2006