Coding Ninja


In words lies immortality

If vs Switch Performance, To Switch Or Not To Switch

Introduction

While most people would generally consider spending vast amounts of energy arguing over minute details bizarre; for programmers this kind of weirdness with a taint of madness comes with the territory.

If vs Switch Performance

Recently a nerd squabble came up and it was centered around the use of an “if” statement versus a “switch” statement. I found myself intrigued and curious enough that I decided to take a more scientific approach and put real numbers instead of just nerd speculation.

The Pattern

A common pattern in programming is to return a value when presented with another associated value. In this particular case an additional condition was to return a default value when no match existed.

Clearly given the above description the text book answer most probably would be to use a switch statement. Normally I would also use a switch statement given the above pattern. However in our particular case there was only three associated values (with the first one being the default as well).

Me being the coding rebel, the white space bureaucrat and line break conservative I opted in this situation to use two simple inlined “if” statements. The two coding styles are shown below:

Classical Switch

string department;

switch (departmentCode)
{
	case "FD":
		department = "Finance Department";
		break;
	case "HR":
		department = "Human Resource";
		break;
	case "IO":
	default:
		department = "Information Office";
		break;
}

Rebellious If

string department = "Information Office";
	
if (departmentCode == "FD") { department = "Finance Department";   }
if (departmentCode == "HR") { department = "Human Resource";       }

Switch Visual “Flow”

switch visual flow

If Visual “Flow”

if visual flow

Such rebellious coding style did not go down well with the fellow programmers and predictably began the squabble. I prefered my use of inlined “if” statements for no other reason than aesthetics, I assumed that given the context the performance differences would be entirely negligible.

A Scientific Approach

Later in the evening I started wondering about the two cases and what the actual performance between these where. Looking at some stack overflow questions it appeared that the general view seemed to be that switch statement yielded better performances. However as with all things it really depended on context. In some cases the “if” statement was more performant. This got me even more curious. With compiler optimization added to the mix the question became even less clear cut.

I decided to do some actual benchmarks and see what the numbers looked like. At least this would be more objective than just outright speculation about what “might” happen.

Reading about the “if” versus “switch” performances, I found that after a certain point the compiler would optimize a switch statement by using a “jump table”. So I decided to also test an expanded case from our own specific case.

Results 1 (Short Case)

A simple benchmark program was created and the two styles run for 10 million cycles for each code path. The entire test was then repeated for a total of 7 trials. All tests where compiled with the “optimized” checkbox turned on. The final averaged results are shown below:

Results 1

The disassembled IL code produced by the “if” style is shown below:


test:
IL_0000: ldstr "Information Office"
IL_0005: stloc.0 // department
IL_0006: ldarg.1
IL_0007: ldstr "FD"
IL_000C: call System.String.op_Equality
IL_0011: brfalse.s IL_0019
IL_0013: ldstr "Finance Department"
IL_0018: stloc.0 // department
IL_0019: ldarg.1
IL_001A: ldstr "HR"
IL_001F: call System.String.op_Equality
IL_0024: brfalse.s IL_002C
IL_0026: ldstr "Human Resource"
IL_002B: stloc.0 // department
IL_002C: ldloc.0 // department
IL_002D: call System.Console.WriteLine
IL_0032: ret

The disassembled IL code produced by the “switch” style is shown below:


test:
IL_0000: ldarg.1
IL_0001: dup
IL_0002: stloc.1 // CS$0$0000
IL_0003: brfalse.s IL_003E
IL_0005: ldloc.1 // CS$0$0000
IL_0006: ldstr "FD"
IL_000B: call System.String.op_Equality
IL_0010: brtrue.s IL_002E
IL_0012: ldloc.1 // CS$0$0000
IL_0013: ldstr "HR"
IL_0018: call System.String.op_Equality
IL_001D: brtrue.s IL_0036
IL_001F: ldloc.1 // CS$0$0000
IL_0020: ldstr "IO"
IL_0025: call System.String.op_Equality
IL_002A: brtrue.s IL_003E
IL_002C: br.s IL_003E
IL_002E: ldstr "Finance Department"
IL_0033: stloc.0 // department
IL_0034: br.s IL_0044
IL_0036: ldstr "Human Resource"
IL_003B: stloc.0 // department
IL_003C: br.s IL_0044
IL_003E: ldstr "Information Office"
IL_0043: stloc.0 // department
IL_0044: ldloc.0 // department
IL_0045: call System.Console.WriteLine
IL_004A: ret

The switch produces a slightly longer IL code.

Results 2 (Extended Case)

The simple benchmark program was extended and the two new styles run for 10 million cycles for each code path.

Results 2

The disassembled IL code produced by the “if” style is shown below:


test:
IL_0000: ldstr "Information Office"
IL_0005: stloc.0 // department
IL_0006: ldarg.1
IL_0007: ldstr "FD"
IL_000C: call System.String.op_Equality
IL_0011: brfalse.s IL_0019
IL_0013: ldstr "Finance Department"
IL_0018: stloc.0 // department
IL_0019: ldarg.1
IL_001A: ldstr "HR"
IL_001F: call System.String.op_Equality
IL_0024: brfalse.s IL_002C
IL_0026: ldstr "Human Resource"
IL_002B: stloc.0 // department
IL_002C: ldarg.1
IL_002D: ldstr "MI"
IL_0032: call System.String.op_Equality
IL_0037: brfalse.s IL_003F
IL_0039: ldstr "Mangement Information"
IL_003E: stloc.0 // department
IL_003F: ldarg.1
IL_0040: ldstr "SO"
IL_0045: call System.String.op_Equality
IL_004A: brfalse.s IL_0052
IL_004C: ldstr "Security Officer"
IL_0051: stloc.0 // department
IL_0052: ldarg.1
IL_0053: ldstr "HO"
IL_0058: call System.String.op_Equality
IL_005D: brfalse.s IL_0065
IL_005F: ldstr "Head Office"
IL_0064: stloc.0 // department
IL_0065: ldloc.0 // department
IL_0066: call System.Console.WriteLine
IL_006B: ret

The IL code produced is an extenstion of the shorter version.

The disassembled IL code produced by the “switch” style is shown below:


test:
IL_0000: ldarg.1
IL_0001: dup
IL_0002: stloc.1 // CS$0$0000
IL_0003: brfalse IL_00BF
IL_0008: volatile.
IL_000A: ldsfld {6D2970D2-35A8-4EC1-AA32-6B4B8F1B6F55}.$$method0x6000002-1
IL_000F: brtrue.s IL_0066
IL_0011: ldc.i4.6
IL_0012: newobj System.Collections.Generic.Dictionary..ctor
IL_0017: dup
IL_0018: ldstr "FD"
IL_001D: ldc.i4.0
IL_001E: call System.Collections.Generic.Dictionary.Add
IL_0023: dup
IL_0024: ldstr "HR"
IL_0029: ldc.i4.1
IL_002A: call System.Collections.Generic.Dictionary.Add
IL_002F: dup
IL_0030: ldstr "MI"
IL_0035: ldc.i4.2
IL_0036: call System.Collections.Generic.Dictionary.Add
IL_003B: dup
IL_003C: ldstr "SO"
IL_0041: ldc.i4.3
IL_0042: call System.Collections.Generic.Dictionary.Add
IL_0047: dup
IL_0048: ldstr "HO"
IL_004D: ldc.i4.4
IL_004E: call System.Collections.Generic.Dictionary.Add
IL_0053: dup
IL_0054: ldstr "IO"
IL_0059: ldc.i4.5
IL_005A: call System.Collections.Generic.Dictionary.Add
IL_005F: volatile.
IL_0061: stsfld {6D2970D2-35A8-4EC1-AA32-6B4B8F1B6F55}.$$method0x6000002-1
IL_0066: volatile.
IL_0068: ldsfld {6D2970D2-35A8-4EC1-AA32-6B4B8F1B6F55}.$$method0x6000002-1
IL_006D: ldloc.1 // CS$0$0000
IL_006E: ldloca.s 02 // CS$0$0001
IL_0070: call System.Collections.Generic.Dictionary.TryGetValue
IL_0075: brfalse.s IL_00BF
IL_0077: ldloc.2 // CS$0$0001
IL_0078: switch (IL_0097, IL_009F, IL_00A7, IL_00AF, IL_00B7, IL_00BF)
IL_0095: br.s IL_00BF
IL_0097: ldstr "Finance Department"
IL_009C: stloc.0 // department
IL_009D: br.s IL_00C5
IL_009F: ldstr "Human Resource"
IL_00A4: stloc.0 // department
IL_00A5: br.s IL_00C5
IL_00A7: ldstr "Mangement Information"
IL_00AC: stloc.0 // department
IL_00AD: br.s IL_00C5
IL_00AF: ldstr "Security Officer"
IL_00B4: stloc.0 // department
IL_00B5: br.s IL_00C5
IL_00B7: ldstr "Head Office"
IL_00BC: stloc.0 // department
IL_00BD: br.s IL_00C5
IL_00BF: ldstr "Information Office"
IL_00C4: stloc.0 // department
IL_00C5: ldloc.0 // department
IL_00C6: call System.Console.WriteLine
IL_00CB: ret

This longer swith case produces much more IL code, the compiler optimization is done using a .NET dictionary<string, int>.

Conclusion

Ok so what can we conclude from the raw results? we can certainly say that for this particular pattern the data indicates that the “if” statement is indeed faster and produces smaller IL code.

Does this mean it is conclusive? No, because in most practical real world cases such extreme number of cycles will not typically be used. The performance difference will always be practically zero.

Some may argue that in the above case that the “if” suffers from “bad practice” because it instantiates a string variable and in all but the default case overrides its straight away and thus “wasteful”. This line of argument much like the performance red herring in reality makes no difference because a very large number of objects would have to be created for any real performance degradation.

What it ultimately boils down to is programmer preference, arguments based on performance are irrelevant since they are a massive premature optimization that for 99% of the time do not even apply.

“Beautiful code, is in the eye of the beholder”

Until next time beautiful coding!

Quick and Dirty Eval for C#

Introduction

“Eval is evil” is a well known phrase and with good reason. Allowing arbitrary code execution opens the doors for [hacker stereotype] to pwn your beautiful application.

Quick and Dirty Eval for C#

Evil Programmers

In many situations the need to reach out to eval is not even needed, its mostly a lazy way out. The price of eval is very high, you pay in terms of speed and security as well as increased debugging complexity.

“eval is Evil: The eval function is the most misused feature of JavaScript. Avoid it”

 Douglas Crockford in JavaScript: The Good Parts

Of course JavaScript is not the only language to have eval, PHP and many other languages have the ability to evaluate code from strings. The evil uses aside, the ability to evaluate code on the “fly” does open up some cool possibilities such as meta programing.

When it comes to C# however the eval function doesn’t exist (probably a good thing). However I recently needed to visit the dark side, most examples of C# eval are quite verbose, but I managed to put together a fat free version. Its quick and its dirty and you probably shouldn’t use it but in times of desperation…

Quick and Dirty C# eval in 8 lines:

    private static object Eval(string __code)
    {
        var csc =   new CSharpCodeProvider(new Dictionary<string, string>() { { "CompilerVersion", "v3.5" } });
        var p   =   new CompilerParameters(new[] { "mscorlib.dll", "System.Core.dll" }, null, true);
        p.GenerateInMemory = true; p.GenerateExecutable = false;
        CompilerResults r = csc.CompileAssemblyFromSource(p, "using System; class p {public static object c(){"+__code+"}}");
        if (r.Errors.Count > 0) { r.Errors.Cast<CompilerError>().ToList().ForEach(error => Console.WriteLine(error.ErrorText)); return null; }
        System.Reflection.Assembly  a = r.CompiledAssembly;
        MethodInfo                  o = a.CreateInstance("p").GetType().GetMethod("c");
        return                      o.Invoke(o, null);
    }

No We Are Not Dull Weirdos But Willard Foxton is a Troll

After reading Willard Foxton’s Blog piece entitled “The Government wants to teach all children how to code. Here’s why it’s a stupid idea” I had to stop myself and think rationally; naturally the post was full of factual inaccuracies and a negative portrait (conjured up entirely in the imaginations of Willard’s mind) of a large section of the developer community.

Ironically I agree with the title. This is not a new debate and it’s one that I have posted about before as a response to Jeff Attwood “Please Don’t Learn To Code But Give it a Try”.  Sadly after the title, there is very little to agree with. With all the ad-hominem grenades tossed at developers the instinctive reaction would be to hurl abuse right back at him. However that would not be a dignified response. Instead after calm reflection I concluded that a more subtle and straight headed response would be more suited.

Troll Dad

Troll Dad

While everyone is entitled to an opinion, that doesn’t entirely mean it has any real value. When you speak a view in public to a large audience, those words do have an impact and so a certain level of responsibility and liability has to be taken on if you choose to speak on a public platform.

Willard’s words are inflammatory, derogatory and demeaning. Judging from the comments he has certainly got a reaction. I don’t need to reiterate and correct the many inaccuracies as this has been covered in the comments. Naturally some comments have been just as negative towards the author, but that’s hardly surprising. If you go around looking for a fight don’t be amazed when you get hit.

It’s a terrible shame that at the heart of the post and the central point has been completely side lined. Was it really necessary to write such a loaded and baited post? Was the traffic really worth it?

Conclusion

It’s really sad to see someone would exploit a genuine debate for purely selfish interests. In a strange way I can’t fully blame Willard, I assume if he didn’t do this someone else would have. But these kinds of post don’t help the overall conversation, as they are a distraction away from the main discussions.

Ultimately the best thing to do would be to ignore him. He is after all a troll as demonstrated by his post.

“Don’t feed the trolls” -anon

Using Your Smartphone as a Desktop Replacement

After smashing my smartphone screen (a Sony Xperia S), I got a quote for a repair because foolishly I hadn’t insured it. After hearing the quote I politely said no and hung up.

An alternative option would have been to try and repair the screen myself, however I didn’t want to take the risk of further damage, besides life is too short for that kind of hassle.

Since the phone was still functional, bar the cracked screen I decided to carry on using it until I a replacement smartphone was available.

After a long review I settled on a new smartphone which I plan on getting in the next month. This means however I will have an extra functional yet damaged phone (yes I know, 1st world problems). I started thinking about extra uses for this spare phone as it would be a shame to waste good working tech.

A few months back one of my old PC’s blew up (a dead motherboard), and I have not bothered to replace it. It was mainly used for surfing. With access to lots of web enabled devices in our house the priority for replacing it was low. With the whole “Ubuntu Edge” buzz going around I thought it would be interesting to see just how much a current smartphone could replace a desktop.

I’ve used a laptop setup as my main working environment for a while in the past, its harder to go back to a big bulky desktop setup after a while. The thought of using something even more space saving was very nice.

The idea was to use an OTG (on the go) cable to a USB hub and then connect my keyboard and mouse that way. The old PC had a standard VGA monitor, so this meant I needed to get a HDMI to VGA adapter.

Sony xperia s. using OTG cable and USB hub to connect Keyboard and mouse

Sony xperia s. using OTG cable and USB hub to connect Keyboard and mouse

I ordered the relevant cables and adapters from eBay. A few days later when they arrived the problems started. The first issue I ran into was that the phone could not be charged and use USB host at the same time. It seems that the standard practice on handsets are to combine USB host and charging into one.

The second problem I ran into was getting the HDMI to VGA to work. I proved that the converter worked when using two other Android devices (Tablet and an Android Gamepad). However for some reason it simply did not work with my handset. I managed to find a hack to a hidden service menu which allowed me to change the HDMI resolution and frequency (dial *#*#7378423#*#*), but none of the options worked. In the end the adapter fell apart in my hand due to the low quality part of that cheap product.

I decided to go ahead and try and operate the device without the benefit of an external screen. At least I had a working keyboard and mouse. The next series of issues were really related more towards software rather than hardware limitations.

Getting android to recognise a UK English keyboard was tricky. I used an app which seemed to work (except the “@” and ‘“’ keys were mixed around).

Android has three hardware keys (back, home, menu) although the “back” and “menu” mapped to the “escape” and “right menu” key. I never found any key that mapped to the “home” key!

I decided to carry on, tweeting was a breeze. Sending text messages was really nice. Doing basic surfing was cool. I found that even the “alt + tab” function worked as expected. Then I decided to try do some work and write this blog post.

I tried using Google Docs, but Google Docs would not give me the “desktop version” and forced me to use a mobile version, which frankly is useless. I tried using the “request desktop” feature but that didn’t work either.

Frustrated I went ahead and installed “Kingsoft Office” which is an amazing alternative to MS Office for the desktop (cross platform and free). The Android version however after using it, while workable was slow and didn’t support basic keyboard short-cuts. At one point it crashed my smartphone.

In the end I caved in having only written 50% of the post and I had to go back to my desktop top to finish this post.

I can still see the potential however at this point until a full-blown desktop OS is implemented inside a phone (Ubuntu Edge Style) this setup is not yet ready for prime-time.

I think the following tweet summed up the entire experience perfectly:

The tweet pretty much sums it up

The tweet pretty much sums it up

Conclusion

There is great potential behind the convergent concept, I genuinely believe we are moving towards it. Hardware is “good enough” for most tasks. What is lacking is the software. Ubuntu Edge really has the opportunity to change this. If wireless charging, video, keyboard and mouse could be implemented this would be my dream setup. Until then it is still just a dream. For now I’m running and screaming back to my laptop, there is still life in that old dog yet!

Until next time, keep it lean mean and green.

Time To Junk Your Junk Code

With summer in full swing it becomes harder to sit crammed behind a computer wresting with code for the best part of the day. Having some time off from work has given me the opportunity to start on my growing list of “things to do”.

I’m a bit of a tech junkie, and as such have lots of bits and bobs lying around. One item on my “to do” list was to sort this stuff out. I’ve never considered myself a hoarder, however as I pulled out those dusty old boxes of treasures I started reminiscing, ah an old school stick of ram, an old AOL dial up installer, my old embarrassing VB6 book, OK ignore the last one I’m going to burn it.

Looking at most of this junk, I have no idea why one earth I’ve kept onto them this long, I know most of the time its probably because it was out of sight (and so out of “mind”). Sometimes it was out of nostalgia for example a decade old CD photo slideshow that I had made and was so proud of. I hoarded some things like my ps2 to usb adapter because of that feeling that I “might just need it”.

A pile of junk

A pile of junk

The reality of course is that it is just junk, it has no real value and it is of no real use anymore. It just sits there taking up space, valuable space that I could use for other things.

I’ve thrown away three old PC’s, and planning on turning my current smartphone as a desktop replacement for one of the PC that blew up. It was only ever used for surfing anyway. The thought of using such a light weight and space saving approach makes me feel that my current workstation is such a dinosaur. I mean 2013 and we are still using wires and cables?

After a day of of cleaning up junk from my house, I got this amazing feeling of freedom. Like I just removed this massive ball and chain from my foot which I had been dragging along all this time. Things are much more cleaner and organised, only that which was actually useful has been kept (only a few items).

Going back to programming I happened to look at some old corners of one of my codebases. When I read through it, I got the exact same feeling as I did with my real world junk, a lot of it was useless, but remained because of the failed “I might just need it” reasoning. Other than nostalgia it was just dead code, very little of it was working code.

I sprang into action, killing the code with fire like a zombie slayer, taking delight at watching text disappear into the ether forever, I did a few rounds of refactoring and boy did that feel good! It was like finally scratching that annoying itch.

Conclusion

If you get some down time, go looking through your codebase especially the corners you have not visited in a long time, if you see any junk code, well just junk it because it feels damn good :)

Until next time, kill some junk code with fire!

Breaking Out Of a Programmers Block

Your cursor is still flashing at the same point and you have been staring at your screen blankly for over ten minutes. You make any excuse to get away from programming, “I’ll go make one more coffee” you tell yourself. When you start writing code you only delete it to start over again. If you are experiencing this level of procrastination then you may have programmers block.

Writer’s block or mental block is widely accepted in most creative fields. However programmers block does not seem to be as readily accepted. While they appear superficially similar they are very different things.

Generally most people accept that sometimes you can be stuck on a hard problem that you can’t seem to solve. A programmers block is something different. It is not that you are not able to solve the problem. The problem is that you find it very difficult to muster the will to want to program or to solve the solution.

It is difficult to say what the root causes are as not everyone is impacted in the same way. However I’m picking on three causes I have heard about that could trigger a block. The first is what some call “Analysis Paralysis”, the second is loss of confidence and the third is becoming lost in the details.

Analysis paralysis happens when having to consider too many competing solutions to a given problem. All of the solutions are poor with no real winner. A mental block sets in as a result of having to pick a solution from a set of bad ones.

Failures will usually knock your confidence, once your confidence is shaken it can be very hard to motivate yourself again. This can becomes more acute when multiple failures occur one after another. Perhaps a block is a natural defence mechanism from having to face yet another failure?

The more involved and engrossed you become on a particular problem, sometimes you end up losing yourself in the small details. You are unable to see the “bigger” picture. Those small details end up using up all your energy and it can feel like your project has run aground as it stops moving forward. The lack of progress against a looming deadline puts stress and pressure until you get into a mental block.

Stuck in a programmer block.

So if you find yourself in the midst of a programmers block what can you do to try and break out of it?

I would like to say that some magical “one size fits all” solution exists, but the reality is that different things work for different people. With this in mind there is no harm in at least giving the following methods a go.

Start refactoring some old unrelated code. You need a way to break away mentally from what you are trapped inside. Refactoring old code might help “warm” you up to programming again. The main thing is to get writing code again.

Talk to someone about the problem. Sometimes we can create problems that don’t actually exist. Discussions may throw up a different perspective that you have not considered. As an alternative get someone else to program this part of the system. This may feel like a cop out but against deadlines, it is better to lose the battle then the war.

Stop programming and go do anything that is not programming altogether. This may sound counter-intuitive, but forcing your mind against a task that you are struggling with could make things worse. A mental context switch might just be the break that your brain needs.
Conclusion

Next time you find yourself in a programmers block why not give some of the above ideas a go? Anything that might improve the situation should be at least tried. Having said this, one important aspect that should be noted is that programmers block can also be an early sign of burnout.

It would be really interesting to hear about different approaches to breaking out of a mental block. Have you had programmers block and if so how have you dealt with it?

Until next Monday, “stay calm and keep programming.”

The Successful Elements In Finding the Right Candidate

In my experience the process of finding the right candidate can be stressful. It appears that within the IT industry, the cases of failure are higher than the successes. In the absence of any standards the interview process is inconsistent and conducted using what can only be described as “IT Folklore”.

Part of the problem starts with most recruitment agencies. Agencies work in an almost mechanical way. Instead of filtering candidates intelligently they seem to filter using a dumb keyword or “buzzword” match approach.

Despite the agents, the more damaging part of the process is the interviewing itself. Technical abilities should in theory be easy to measure objectively. However in reality this is not the case as the current style of “testing” often produces inconsistent results.

The testing style that many companies use does not seem to be coherent. For some reason the focus and emphasis of many tests appear to obsessed around brain teasers and puzzles.

Although one could argue that a good surgeon is someone who has excellent “hand eye coordination” it is only one metric. It would be absurd to measure the competence of a surgeon by their video game playing skills.

Finding the right candidate

Finding the right candidate

When I was searching for a candidate I wanted to avoid the current style and so decided on a different approach. The first thing I did was decide what were the objectives I was trying to achieve from an interview. Personally I think an interview has two sides, the technical and the non-technical which are equally important.

The non-technical element for me was about getting a “feel” of the personality of an individual. While it is true a short period of time is not sufficient to get the full picture about someone, I think you can at least get a decent impression about them.

Regarding the technical side I boiled it down to three essential things I wanted to find out about the candidate:

  • What have they done?
  • What can they do?
  • Why do they do it that way?

In terms of the first objective I think asking them about their past jobs and experiences was enough to ascertain what they had done. It is hard to lie about technology you have never used because you simply will not know enough to give a thorough answer.

I derived the last two objectives by writing an exam paper. The “examination” was conducted on an offline laptop. The candidate was left alone (except for popping in at fixed times to check on progress). As for the questions on the paper I created four sections:

  • Section 1: Warm up questions: basic computer science questions.
  • Section 2: Open ended question: “How would solve?” type question.
  • Section 3: Programming question: Create function x.
  • Section 4: A set of SQL questions: write a SQL query to do x.

Once this was done picking the right candidate (for us) was simply about weighing the test score along with our impression of the personality. We found the process was painless and we believe it delivered the right candidate.
Conclusion

Programming interviews can be problematic. If it appears that you are constantly not finding the right candidate, you may want to consider looking at how you conduct your interviews. For myself this approach had the successful elements in finding the right candidate.

Until next Monday, “Live long and prosper”

Full Spectrum Engineer Or Why The World Needs Polymaths

I was explaining the mechanics of how a computer works to a friend, when I realised that although I knew the basic concepts, I lacked the exact details. This reminded me once again about a fantastic courseware entitled “The Elements of Computing Systems” by Noam & Shimon. If you have not yet come across this courseware I highly recommend it. In twelve steps the student builds a real computer as the quote states “from NANDs to Tetris”.

I have noticed a trend that seems to be on the rise at the moment. It feels like a new “learn to program in x weeks” startup launches every other week! Although I’m not in a position to judge the successfulness of each of these companies. I can’t help feel that a new IT bubble is forming. As a professional programmer, bubble aside this feels like the “new” VB6 fiasco.

Around 20 years ago Microsoft launched “Visual Basic Studio” and it was designed to enable “Rapid Application Development” (RAD). And in many ways it could be said that it succeeded at doing just that. It’s simple to use “drag and drop” style of programming enabled applications to be made with disgusting ease. It had in effect “lowered the barrier” to development and made it accessible to a much wider audience.

Much to the chagrin of professional developers the IT industry was flooded with what can only be described as “cowboy programmers” (in the UK a cowboy is a negative term). These cowboy developers (many without programming backgrounds) created extremely poor software. In some cases modern developers are still cleaning up that mess from a decade ago. I must emphasise that there were also competent engineers that did use VB6 to great effect (but they were in the minority).

Sadly it appears history is once again repeating itself. The selling point before was “productivity” This time round it is about “everyone should be able to code”. I fear the outcome will be the same. While the idea of teaching programming to everyone is noble, it is perhaps trying the solve the wrong set of problems.

Aristotle, A polymath hero

Aristotle, A polymath hero

As part of the US military doctrine the term “Full Spectrum Dominance” or “Full Spectrum Superiority” to give its official term are defined below:

The cumulative effect of dominance in the air, land, maritime, and space domains and information environment that permits the conduct of joint operations without effective opposition or prohibitive interference.

The general concept is to “cover all bases”. I would like to extend this concept to the engineering world. In the engineering version I like to call it “Full Spectrum Engineer”. There are many discrete disciplines within engineering (electronic, mechanical, computing). In my mind a full spectrum engineer is one who has a good grasp of all of them.

Although in modern times academia and industry has followed a path of specialisation, this has not always been the case. The giants of history and the classical scholars followed a very different path. In 384 BC Aristotle who was a student of Plato was a scholar who covered many domains from poetry to physics. Aristotle and many great figures like him were all polymaths, that is they were all involved in many diverse fields.

If you think about programming as a domain it is a highly specialised field. Getting more people into programing means pushing more people towards perhaps a more specialised domain. I genuinely believe that what the world needs today are more polymaths in a sea of monomaths.

I’m not going to extol the numerous advantages of being a polymath, as I feel that there are certainly enough space for both to coexist. Having said that the balance currently seems biased towards monomaths, this is most probably as a result of modern academia.

Conclusion

The ultimate goal of engineers are to solve problems. In order to solve problems there are various tools that can be used to bring about a solution. Programming is one particular tool, however solutions need not be solved from this singular perspective. In some cases it is impossible that a single tool is sufficient to solve a particular problem. Recently a lot of focus seems to be drawn towards programming. While this is commendable I feel this is a narrow focus, I believe the focus instead should be made as wide as possible. I think Abraham Maslow described the situation of a monomath engineer the best:

“if all you have is a hammer, everything looks like a nail”

Until next Monday. keep safe and keep learning.

Like Fine Wine, Developers Can Become Better With Age

At the beginning of last week, an article appeared on Proggit (that has since disappeared?). entitled The Tech Industry’s Darkest Secret: It’s All About Age, written by Vivek Wadhwa.
Incidentally this is a nice follow on from a post I had written just a few weeks prior Where Do You See Yourself in 10 Years Time?.

The central point of Vivek’s article was well summarised in his introduction:

“The harsh reality is that if you are middle-aged, write computer code for a living, and earn a six-figure salary, you’re headed for the unemployment lines. Your market value declines as you age and it becomes harder and harder to get a job.” Vivek Wadhwa

The argument is also reinforced by Bureau of Labour statistics and Brown and Linden’s findings:

“Although salaries increased dramatically for engineers in their 30’s, these increases slowed after the age of 40. After 50, the mean salary fell by 17% for those with bachelors degrees and by 14% for those with masters degrees and PhDs.“

I am fully aware statistics should always be taken with a huge pinch of salt, and a link to the original analysis would have helped those more curious readers. However, except from minor differences I don’t think the data was a million miles off. Its hard to shrug off the data where the drop does indeed appear to be substantial.

And in addition to the statistics, this thought experiment does highlights the case further:

Why would any company pay a computer programmer with out-of-date skills a salary of say $150,000, when it can hire a fresh graduate — who has no skills — for around $60,000?

Vivek’s parting advice only helps solidify his prior conclusions, my translation of these points are as follows:

  • “Move up the ladder” basically do something else other than development.
  • Start your own business,and probably fail badly.
  • Try and still hip with the kids, become a hipster and buy a Mac.
  • If you want to keep coding, save your money now because you won’t be worth anything soon.
Legacy Systems, live on

Legacy Systems, live on

Is it really that bad?

After reading, I let out a deep sigh; I couldn’t disagree outright – much of what he said sounded reasonable. Once the emotions subsided my logical side kicked in (maybe helped by another caffeine boost from my fifth coffee of the day!)

Is it really that bad? If we start with the statistics we can see that salary appears to decline with age.

The first problem, we can not infer a single metric as being the cause (in this case, age). I suspect the picture is a lot more complicated than what is presented.

Secondly, although salaries do drop which suggests “less demand”, that could also mean “less interest” from Developers.

Upon closer inspection of the thought experiment, the conclusion only holds for some circumstances. If we applied the same question to a different set of circumstances the question no longer holds true:

  • An established company that needs someone with a proven track record to deliver a business critical projects.
  • Legacy systems (eg COBOL) with skills and experience not widely available.

There are probably many other situations that do that also do not hold true for the thought experiment.

Conclusion

In these volatile economic times, job security is a challenge for everyone. These economic pressures also affect Developers and we are certainly not immune from them. Young Developers or “Apprentices” in other fields certainly have appeal from an economic perspective. However it must be remembered that in business cheap labour are not always the only consideration.

Learning new technology and constant learning should be a natural thing and not coerced, in that sense staying “up to date” shouldn’t be an issue.

The technology sector may be different in terms of “newness” because of the pace it moves at, however one thing has to be kept in mind: Today’s technology is tomorrow’s legacy system, and legacy systems have to be maintained long after the technology has died.

Until next time, happy coding!

Building A Concurrent Web Scraper With Python

Another week and during my internet travels I stumbled upon a blog post by Aditya Bhargava titled “Building a concurrent web scraper with haskell”. I’m not a Haskell programmer and my experience of it is extremely limited. Reading the post most of it read like a cryptic magic spell!

Anyway it was still an interesting read and has inspired me to try my hand at writing something similar. So I reached out and and the quickest thing to hand was Python. In Linux just fire up a terminal and drop into your favorite text editor and away you go.

The Process

It is usually a good idea to think and plan out the basics of the solution design. Our target is a web page and our end result should be all images downloaded from it to disk. The target web page contains links to the images we want to scrape. Once we have a list of images instead of downloading them one at a time, the goal will be to download them “concurrently”.

A Concurrent Web Scraper Process

A Concurrent Web Scraper Process

.

The entire process can be broken down into the following distinct four steps:

  • Get contents of page
  • Parse and create a list of image links
  • For each link start a new thread
  • Download a single image

Now that we have a description of each step (or “function”) we can think about each function’s name and its input and output:

  • “from page” url -> html text
  • “all links” html text -> list of links
  • “in parallel” function, list of links -> start thread
  • “download image” link -> file saved to disk

You may have noticed that the output of one function is the input of another. This has been designed so that each function can be fed into the next one. Thus we can (at least in pseudo code) describe the entire program call:

in parallel ( download image, all links ( on page ( url ) ) )

Voila we are all done! Erm OK perhaps not quite, we just need to create the code for each function:

Lets create the code in the order that the functions are called. First start off with the parallel call, this function will simply take a “function” to invoke and a list of links. We can iterate through the list, and call the function each time and pass the link to the function as a parameter.

.


from threading import Thread

def in_parallel(fn, l):
       for i in l:
           Thread(target=fn, args=(i,)).start()

.

To get the html contents for a given url we can use Python’s urllib module:


import urllib

def from_page(u):
       return urllib.urlopen(u).read()

.

Let’s now create the function that downloads each image, we need to provide a filename. We could parse out the link that is passed in but being a lazy programmer I’ve decided to just create a GUID instead :)

.


from uuid import uuid4

def download_image(i):
       print "saving -> "+i
       f=open(str(uuid4())+".jpg",'wb')
       f.write(from_page(i))
       f.close()

.

Now all that is left is really the meat of the program, the task of parsing out the image links. To do this I have decided to use regex (shudder; I know I know, you can’t parse HTML with Regex). Being naughty sometimes is OK, right?

.


import re

def all_links(p):
       links = re.findall(r'href="([^"]+)"',p)
       return [links[i] for i in xrange(0,len(links)) if "http" and "jpg" in links[i]]

.

Finally now that all the functions have been created, we can call it and enjoy the images being downloaded concurrently, and what better web page than reddit’s /r/pics ?

.


in_parallel(download_image, all_links(from_page("http://www.reddit.com/r/pics")))

.

Conclusion

Programming in Python is fun because the code almost flows in the way you imagine it! Naturally this simple example is most certainly not production ready. However the basics are the same. Extending this example would include limiting the number of concurrent threads, error handling and perhaps recursively calling links on the website. Something for the reader to try out :)

The full 23 lines of source code are included below. Until next time have fun Pythoning!

.

from threading import Thread
from uuid import uuid4
import urllib
import re

def in_parallel(fn, l):
       for i in l:
           Thread(target=fn, args=(i,)).start()

def download_image(i):
       print "saving -> "+i
       f=open(str(uuid4())+".jpg",'wb')
       f.write(from_page(i))
       f.close()

def all_links(p):
       links = re.findall(r'href="([^"]+)"',p)
       return [links[i] for i in xrange(0,len(links)) if "http" and "jpg" in links[i]]

def from_page(u):
       return urllib.urlopen(u).read()

in_parallel(download_image, all_links(from_page("http://www.reddit.com/r/pics")))