Back to School - Topdown Design and Stepwise Refinement

Do you remember how we were doing things back in school?
By Andra Net Posted 01 September 2016

Back to School - Topdown Design and Stepwise Refinement

The Problem

A couple of weeks ago I came across the following task: to create a mechanism that will do some retroactive processing on currently existing data. After talking with our Architect, it was clear what was expected from it, so I had green light to start the implementation.

However, going back to my IDE, I didn’t know where to start. My mind was a mess, full of ideas that came and went with the speed of light. So I start pulling out these ideas and laying them down in plain English. I started with the simplest and clearest ones; they looked like this:

> 1. Iterate through all closed tickets.
> 2. Set the new status for each ticket.
> 3. Notify the end of process.

That’s it! Three simple steps were all I need to implement in order to complete my task. Having this starting point, I instantly realized this was the way we used to solve problems back in school. The teacher wouldn’t even care about the actual code, he always wanted to see how we imagined the algorithm, how we solved the given problem step by step, by solving one issue at a time. Back then, the problems we solved were pretty straightforward and it felt such an overhead to have to write all those steps, but now I finally realize how important is to go through all the refinement process.

The Solution

So, what did I do next? Writing down the initial steps was just the first phase, where I defined the skeleton. From here, the refinement process started:

Phase 0
> 1. Iterate through all closed tickets.  // *There are lots of them, how to iterate through?*
> 2. Set the status for each ticket.     // What’s the new status?
> 3. Notify the end of process.         // *How?*
    End.

Refining for me means going through all the steps and identifying the one detail that will bring each step closer to the final point. After identifying this detail, we’re ready for the next phase; as you’ll see, the next phase is just answering the questions I starred in the current one, and that’s how the whole process unfolds.

Phase 1
> 1. While (there are no unprocessed tickets)   // *How to know a ticket is unprocessed?*
>   a. Get a bunch of tickets              // How many?
>   b. For each ticket
>       i. Calculate the new status           // How to calculate it?
>      ii. Save ticket status
End while                    // How to guarantee a non-infinite loop?
> 2. Log the completion of the process.
    End.

My algorithm is beginning to take shape, so I’ll start to focus on just one problem at a time. As you probably may have heard, our brain can’t handle multitasking, although we believe it can, (that’s the reason you’re not supposed to use your phone while driving), so in the next phases I’ll only be addressing one question at a time.

Phase 2
> 1. While(count(last status update date is null AND ticket is closed))
>   a. Get a bunch of tickets         // *How many?*
>   b. For each ticket
>       i. Calculate the new status  // How to calculate it?
>      ii. Save ticket status        // Already implemented
End while                // How to guarantee a non-infinite loop?
> 2. Log the completion of the process.  // Already implemented
    End.

While keep addressing only one issue at a time, I can also refactor the already-solved parts of it, so the algorithm gets closer to the final form. Note that I don’t have any more questions regarding the first, the last and the Save ticket status steps, so I’ll refactor these ones.

Phase 3
> 1. While(count(ticket.LastStatusUpdate == null AND ticket.IsClosed))
>   a. Get first 500 closed and unprocessed tickets
>   b. For each ticket
>       i. Calculate the new status    // How to calculate it?
>      ii. UpdateTicketStatus(ticket)
End while                             // *How to guarantee a non-infinite loop?*
> 2. logger.Warning(“If you’re reading this means the processing of closed tickets
            has come to an end. Well done!”)
    End.

It’s not necessary to refine the steps in descending order. In fact, I refined them based on their difficulty, starting with the easiest questions and postponed the more difficult ones.

Phase 4
> 1. While(count(ticket.LastStatusUpdate == null AND ticket.IsClosed))
>   a. tickets = GetTicketsTop(500, ticket.LastStatusUpdate == null AND ticket.IsClosed)
>   b. For each ticket in tickets
>       i. Calculate the new status             // *How to calculate it?*
>      ii. ticket.LastStatusUpdate = CurrentTime
>     iii. UpdateTicketStatus(ticket)
End while
> 2. logger.Warning(“If you’re reading this means the processing of closed tickets
            has come to an end. Well done!”)
    End.

We’ve finally come to the last unsolved problem: How to determine the new status of a ticket? While thinking of it, I realized there are more than one unknown issue hiding behind this step, so I decided to take it on a new refinement thread by itself. This kind of situation where you have more than one thread of refinement is very common; the more complex the initial problem is, the more chances to end up with multiple refinement processes you get.

The Benefits

No more diagrams

The first and most obvious benefit of using this problem solving approach is the great help it provides when dealing with medium-to-complex tasks; it’s such a clear way to break down the given task and to spot issues and potential bugs even before the first line of code was written. It’s also a productive way of conducting an analysis in a more organized way, replacing those entangled, unreadable, monster diagrams we all end up with after each meeting with a set of clear, plain English phrases.

I also realized that using this approach I was able to finish the task faster than I would have thought, and I was more confident in my code because I already run it multiple times in my head as I did the breakdown. I even catched a potential bug, the one with the infinite loop.

After finishing the implementation you can still make use of your analysis work by including it in your technical documentation book (if your project happens to have one :)). This way, you document your work as you go along, without the need to write tedious documentation after the code was already deployed.

The End

In the end, as with every tool and process, this technique can be adapted to your needs. And the great part is that you can apply it to any problem you face whatsoever, it’s not limited to programming. I even used it while writing this article too!

However, if you don’t feel like writing so many lines of un-runnable code, or if your task is pretty easy, you can still make use of this refinement method directly into your IDE, by writing the code you’re absolutely sure about, and replace the unknowns by comments that describe what you want to accomplish, then refine the comments one at a time. I like to call this the fast-forward alternative of the full process.

So what do you think? Is this all paperwork worth it, or is just a waste of time?

Andra Net
Andra Net
Software Developer