My writing has moved

This has been a strange year for me for writing in that most of it has gone elsewhere.  I started writing a paid column for an online Amazon Web Services magazine called SearchAWS.com.

If you're interested in AWS you might find these articles interesting:

Setting Up Your First AWS Instance
CloudWatch Terms and Definitions
CloudWatch Alerts and Alarms
The Inner Workings of AWS Autoscaling
Combat a common error with AWS EC2 instances
Your Mission's Not That Critical
Demystifying AWS and Docker

I've also started writing for LinkedIn Pulse News.  These stories are my own observations on the software industry and on being a software developer (hopefully with the occasional insight on the wider world).  If you'd like to check those out you can visit:

Terminating BYOD
Escalating TO-DOs
Taking a collaboration clue from programmers'...
How hard is the paperwork to take time off?
Don't Waste My Time
"And" is a four letter word
When we're as big as Google...
The Moral of the story isn’t balance…
Facebook quizes as advertising

As always I'd love to hear comments from you, either here or on those pages.  If you have topics you'd like covered please let me know that as well.

Thanks.

I am now writing for SearchAWS.com a web magazine for the Amazon Web Services space

I've started a new writing venture, writing "tips" for a new web magazine devoted to the Amazon Web Services space.  These tips are short (600 word) explanations of how to do various AWS tasks or how to understand aspects of Amazon's offerings.

Many of the tips seem to be evolving into two part series: an overview and background tip followed by a cookbook style tip.  I've taken this approach because I find that many aspects of AWS only make sense if you understand the perspective they are coming from.  They also have a fair amount of non-standard terminology that can be a road block until the terms are explained.

So far I've written on AWS registration, creating your first instance, and understanding CloudWatch.  Please check out searchaws.com and let me know if you have suggestions for topics.  Thanks.

Random observations from Amazon's re:Invent conference

I just returned from Amazon's re:Invent developers conference and had a few social observations.

a) the gender breakdown seemed to be about 99% male which is depressing

b) every presentation I saw was given on a MacBook

c) in the audiences I saw people taking notes on iPads and all manner of 7 inch tables (its hard to distinguish iPad Mini from Kindle Fire from generic Android at a distance.

d) I think I saw a couple of PC laptops during the week and I saw zero Surface tablets.

e) I did for the first time see two people wearing Google Glasses and I heard the term Glass-hole!

f) The Venetian hotel has a truly amazing recreation of Saint Marks Square…they basically built a IMAX like curved ceiling over the entire indoor plaza creating a realistic impression of being outdoors.

g) the technology part of the conference (the real point after all) was of course very cool and compelling but thats a post for another day!

New article at PragPub

I have another article published at PragPub.  That used to be a free publication but they've had to start charging (a very small and reasonable fee) for the magazine.  You can get to the paid magazine at www.swaine.com/pragpub.

They are taking a very open approach and allowing authors such as myself to provide free links to our articles.  So,  here is a link to a copy of that article: hAppsVsWebFinal.pdf

As always, comments are welcome.

How to Setup Amazon CloudWatch to Send Alarm Emails

Amazon Web Services are very powerful and some things are harder than you might expect.  Getting an email notification that something is wrong is one of those things.  I originally wrote this up so that I would remember next time I needed it and then decided it might be helpful to others.

You need to set up several things to get this to work.  You need a metric, which is some value that indicates if the system is ok or not.  You need an alarm that fires when that metric indicates failure.  You need an SNS topic to get notified when an alarm fires.  And lastly you need a subscription to that topic which a where you specify a phone number or email to receive the alarm notification.

create an SNS topic from SNS gui (press “Create and Add”)

give the topic a name and a display name

 create a subscription…some place to send the notification

HTTP, HTTPS, email, sms, SQS  and protocol specific “endpoint”

You will receive a confirmation email, text, etc. depending on the protocol you specified asking if you really want to create this subscription.

Take note of the “ARN” for the topic, it will be used as the alarm-action below

Create a Cloud Watch metric named ZkServerCount

"mon-put-data" is used both to put the data and to create the metric itself if it doesn’t already exist.

We use the command line to create the metric because CloudWatch seems to have a problem if you have defined a lot of metrics…and its easy to create a lot of metrics.  Each instance you create automatically gets half a dozen standard metrics and those metrics live for 14 days regardless of how long your instance lives.  If you create thousands of worker instances  you seem to overwhelm the CloudWatch gui.  We’ve reported this as a bug and are working with Amazon about it but in the meantime we use the command line to create metrics.

Setting up to use the command line

download the jars from http://aws.amazon.com/developertools/2534

setup environment vars as described in README.TXT.  You must in particular setup the AWS_CLOUDWATCH_HOME variable

set up credentials as per the readme.

Run this command which will both create the metric and give it its first value of three: mon-put-data –metric-name ZkServerCount –namespace MyNameSpace –timestamp 2013-08-25T00:00:00Z – value 3

Create a Cloud Watch metric alarm based on that metric; this alarm will fire when the metric’s value is less than 3:

bin/mon-put-metric-alarm –alarm-name zk-mon –alarm-description “Alarm upon zookeeper server failure” –metric-name ZkServerCount –namespace MyNameSpace –statistic Average –period 300 –threshold 3 –comparison-operator LessThanThreshold –evaluation-periods 1 –alarm-actions arn:aws:sns:us-east-1:abc123:ZooKeeper_failure< 

Create some code that will periodically run the mon-put-data command from step 2 above.  This looks like a lot of code to put a single value into the system but bear in mind that this includes some one-time setup, and that you can add multiple values at a time.    

public void putMetric() {

  BasicAWSCredentials    credentials = BasicAWSCredentials(ACCESS_KEY, SECRET_KEY);

  AmazonCloudWatchClient acwc = new AmazonCloudWatchClient(credentials);

  Double metricValue = 3.0;

             MetricDatum datum2 = new MetricDatum().

                withMetricName(INSTANCE_COUNT_METRIC_NAME).

                withTimestamp(new Date()).

                withValue(metricValue). 

                withUnit(StandardUnit.Count);

           PutMetricDataRequest putMetricDataRequest = new PutMetricDataRequest().

                withNamespace(FRAGMENT_NAMESPACE).

                withMetricData(datum1, datum2, datum3);

           acwc.putMetricData(putMetricDataRequest);
       }

When you are all done its of course a good idea to trigger the condition manually just to make sure you actually do get that text or email that you really don't want to get for real!

Moving to Agile: estimates and promises

In transitioning a legacy team to agile the issue of estimates vs promises and how engineers and management interprets them is acknowledged as key.  Management often asks engineers for estimates and then treats the answers as promises.   This leads to a whole set of problems which are documented in the agile literature.   I often think that communications difficulties can be helped by reversing a situation so lets try that here.

I wonder if we can help clarify the difference between estimates and promises by asking management for an estimate. “If my team gives you product X by date Y will you give us a specific level bonus?”. Any management I’ve seen will say they can’t promise a specific bonus in the future because there are too many free variables.

To which we say “exactly”.

If instead we say “we believe we have an 80% chance of hitting date Y with feature set Z” might we get them to say “there’s a 75% chance of getting a bonus of D dollars”?  As we got closer and closer to the target release date might we get them to reduce the uncertainly of our bonus level as we reduce the uncertainly of the feature set?

I don’t think most management would actually do this but it might help them see our estimates in a new light.

Is It Coding in Scala or Coding With Courage And Humility That Matters?

Scala is all the rage these days, replacing its legacy cousin Java in the hearts and minds of all the cool kids.  Admitting that you still code in Java is the geek’s equivalent to saying you drive a mini-van and don’t have a twitter handle.  We’ve all heard the proclamations of how much more powerful and succinct Scala is..and I’ll admit to having done some of that myself.

While not dismissing those statements,  because I do in fact find Scala to be a better language,  I think there is another cluster of factors at work here as well.  Those factors are courage and humility.

There is a correlation between Java and large, often distributed teams.   This may be largely in part due to the relative age of the language.  It is established and “mature” to use a kind word.  If you have a 50-100 person development shop with offices in the US as well as in some subset of China, India,  Russia you are very likely to be a Java shop.  If you have maintenance or sustaining teams as well as a development team you are likely to be a Java shop.  If you spend a fair bit of your time on “process” you are likely to be a Java shop.

Notice that I said “likely”.  There are lots of counter examples.  My own company uses Java and we have only three programmers.  None the less, I think the preceeding statements are generally true.

So what?  Well, my assertion is that along with large, process oriented and distributed comes the notion of lowest common denomenator coding.  As the size and geographic distribution of your team grows so does concern about “those other programmers” being able to understand and maintain your code.  That leads us to want to standardize and simplify the code.  We want to make the code clear to that possibly junior coder who may be new to the project, who may never have absorbed the designs, and who many not be familiar with the code base.

This leads us to write code like:

public int calculateTheValue(long someInput, long someOtherInput) {
    long intermediateOne = someInput * getSomethingElse();
   intermediateOne += someOtherInput;
   intermediateOne = someMethod();
     ….
   return intermediateOne;
}

There is nothing wrong with this code and to a newbie its certainly more accessible than:

public int calculateTheValue(long someInput, long someOtherInput) {
     return somethingElse(someInput, someMethod(someOtherInput));
}

The problem is that 10 lines of code versus 3 for every method in your system results in a sea of code where you literally can’t see the forest for the trees.  I can tell what each individual line of code does but I have no idea why because I can never see more than 0.01% of the code on my screen at a time.

Some readers might protest at this point that this is all just formatting and Eclipse or Emacs could in principal convert between these two representations.  To which I say: not so much.

The functional approach is all about the composition of a method from a collection of existing functions.  In this approach it is clear that the new method is “just” using the existing methods.  The new method has no logic per se other than using the output of other…presumably well named and tested functions.

In the more familiar Java approach each method is a new creation created out of whole cloth.  It might do any old thing it wants.   In this case freedom and creativity are to be considered bad things.  Each method must be examined line by line to see what it might be doing.  Lets look at a bit of open source code I’m actually currently debugging:

int to = readTimeout - clientCnxnSocket.getIdleRecv();
int timeToNextPing = (readTimeout / 2)  - clientCnxnSocket.getIdleSend();                       
if (timeToNextPing <= 0) {
            sendPing();
            clientCnxnSocket.updateLastSend();
 } else {
           if (timeToNextPing < to)
                     to = timeToNextPing;
}
clientCnxnSocket .doTransport(to);

After some period of study we can see that this code has two variables related to time outs: “to” and “readTimeOut”.  Based on the results of two “getIdle” calls we might send a ping, and then we mutate “to” in a couple of possible ways and then use it as a parameter to a socket call.  Further investigation reveals that “to” is the length of time the socket method may spend in a blocking “select” call.  Thus, “to” is related to how long we can block before sending another ping.

I’ve spent the last couple of hours trying to track down a bug in this system and the problem is that every single line is ontologically at the same level.  By that I mean that any of them could have or be a side effect, any could do something other than whats expected and the gestalt of what this code fragment intends can only be gleaned by close study.

I’ll assert that the following code does not suffer from those flaws:

if(timeToPing()) sendPing();
clientCnxnSocket(safeTimeToWaitForRead());

And that brings us to the humility side of the equation.  Its ok to write little one line functions that just do the one thing that their name implies.  TimeToPing is not a function you will put on your resume.  You will not proudly show it to your coworkers.   You will not tell your husband/wife/partner about the amazing bit of code you wrote today.  This one line function will sit there quietly, unnoticed…working.

If we have the humility to write simple functions and then have the courage to combine them into composite functions without extraneous scafolding and temporary mutable variables then we have a chance to achieve greatness…even in a legacy language.

To be sure, there are things that are trivial in Scala that simply can not be done in Java.  I know of no way to annotate a method to indicate that it does or doesn’t ever return null.   We recently changed such a method to never return null.   There is no way however to  find all the code that’s now unncessary.  Or assuming we had made the opposite change…to find the code that was not an NPE timebomb. [1]   

In Scala of course if your function might return a Foo but might return nothing you return an Option(Foo)…and function’s callers must deal with the Option(Foo) or they will not compile.  This isn’t fixed in Java 7, nor will it be fixed inJava’s 8, 9 or 10.  Null is just baked into the language.

Java isn’t going to be “replaced” by any of the newer languages.  It will continue to lose market share but will command a large segment of the market for the foreseeable future.  Its also clear that Java will continue to evolve and will over time gain missing features such as lamda expressions and better package structure.  Other things like null and the Generics system are likely to be with us to the bitter end.    For good or bad erasures and generics are part of the language now and forever.  Java 8 sprinkles a bit of syntactic sugar allowing the second repeat of the type to be omitted as in:

HashMap myHashMap = new HashMap<>();

but that’s a fairly trivial improvement in this age of modern type inference languages.

What this means is that engineers working with Java need to do the best they can with a 15 year old language.  Courage and Humility can help with that. 

---

[1] Yes, we could use PMD but that just points out that there is no support for such things in the language itself.

Nothing but better hiring will fix things

Recently in Slashdot there have been a series of posts about the quality level of the software at various companies.  They've had titles like "How can I make my team write better code?"  or "How can we improve our code quality?"  

I also subscribe to numerous blogs and MeetUps and User Groups where titles like "Do 'x' to improve" are fairly standard.

I might just be getting my jaded but I'm coming to the conclusion that there is one and only one thing that can improve the quality of the software you/your team produces.  That one thing is better hiring.  To quote Joel On Software..the only two answers after an interview are "Hell yes!" or "No".

By and large people either care passionately about the code the write ... or they don't.  In over thirty years of doing this I can't recall a case where a "bad" coder read a book, went to a class, attended a conference or had a talk with management and suddenly started caring.

That's a bit depressing because I speak at conference and have published lots of papers!  On the other hand, my papers and talks have been aimed at the subset of our field that already care and are just looking to hone their skills.  Trying to convince someone that unit testing, proper naming, or coherent design was a good thing is just a waste of time.  People either "got it" a long time ago or they're not going to get it.  Sorry.

So, work to get your team to hire better people or go someplace that already does.  I just don't see another option.

New Article in Pragmatic Programmer

Just a quick note to say that I had another article published at the Pragmatic Programmer's:
http://pragprog.com/magazines/2012-12/agile-in-the-small

Picking the next toys just got harder

My VerizonWireless contract is up in three weeks, my Kindle just died and Apple just introduced the iPad Mini so I've got some deciding to do!

I current travel with my 13" MacBook Air, a 2 year old Android phone and often my iPad2 as well.  The MacBook Air is my primary laptop and serves as my coding and general content creation platform.  The Android phone is going into the crusher as soon as possible, to be replaced with an iPhone.  Apple isn't perfect but they support their devices for longer than 20 minutes so it "never again" for Android.

So now we've dealt with phone calls, navigation, quick email checks, quick web browsing, coding and content creation.  What's left?

The iPad has been getting pretty lonely since I got the Air.  They're both big enough to need a carry bag so if I have the iPad I almost certainly have the Air as well.  This leaves time on the train commuting to work as about the only time I'd use the iPad because there's really not room in our crowded train for the Air.  So I use the iPad for games and some reading.

Do I really need either a Kindle or an iPad Mini?  (And when I say "need" I mean it in the geeky sense as in can I develop a rational for it).

The only thing I can think of using an iPad Mini for rather than my existing iPad is reading; lets face it the iPad gets heavy after a while.  But I could get a Kindle Paper White for about a third the cost or a base Kindle for less than a quarter of the cost of the Mini.  I'd rather get the Paper White but they're back ordered for 4-6 weeks so that's not an option.

Put another way: I can get an iPhone and a base Kindle for less than the cost of the Mini.

So, for folks who don't already own an iPad the iPad Mini might be a great idea but I don't see many people getting both.

Finding your voice

In the past six weeks I've attended a week long Writer's Conference at Wesleyan University, a week long Native American and World Flute Conference at University of Wisconsin and changed jobs (going from a 40,000 person company to a 6 person startup).

I like to look for common threads and the thread here is finding one's voice.

Voice is sometimes thought of as the same thing as "style" which may help explain the idea of programmers having a voice but I think it goes far beyond that.

At the writer's conference we did close readings ("inspections") of various works, looking for themes and patterns and voice.  Voice here meant a way of saying things that was engaging, illuminating, understandable and among other things predictable.  We talked about various contracts a writer can implicitly establish with a reader such as mentioning a shotgun in a foreshadowing paragraph implies a later use of that gun in some future scene.

At the flute festival we attended Master Classes on topics such as use of various scales (major, minor, mixolydian), how to add embellishments or ornaments to ones playing and how to tell a story by use of tempo and volume and such.  While the standard orchestral flute has about 20 keys the various native flutes have six or four keys...and the didgeridoo has no keys.  This can be seen as limited but a smaller set of notes to choose from just means one has to use other ways to produce a full and interesting story. Thus voice.

In my new job I am the second programmer on a medium sized system...meaning that the system was entirely written by a single person.  It definitely has style because the person who wrote it has strong feelings about the myriad of design choices he's made along the way.  He's also free about the choices that were made without a strong sense of one choice being clearly better than another.

His voice is of course different from mine.  We have very different backgrounds and life experiences, but the point is that I can see his voice in the code and I quite like it.  I'm sure that over time our two voices will influence each other, but I suspect that they will remain distinct.

In the past I've been a strong advocate of static analysis tools such as CheckStyle. I don't think I'll be lobbying so much for that tool here.  I'm beginning to think that such tools are most applicable for shops where the engineers have not yet found their voice.

Thoughts about e-Books after attending a Writer's Conference

I just returned from attending the 2012 Wesleyan Writer's Conference in Middletown, Conn.  It was a week jam packed with close reading of poetry, thematic explorations of fiction, exploration of dialog in non-fiction and of course speculation on the future of books, e-books and publishing.

A couple of ideas came to me during the week that I'd like to share.  Let me preface by saying that for me reading means reading books or PDFs on my Kindle or via a Kindle Reading app on my my iPad, MacBook Air or HP laptop.  Last year I read 42 books and only two of them were of the dead tree variety.

One of the great features of Kindle reading (and presumably also for Nook reading though I can't say from experience) is the ability to trivial get the dictionary definition of works in the text via a single click.  I like to pride myself on an extensive vocabulary but there are lots of words that I "mostly know what they mean but could not give you a coherent definition".  I make it a point to click on these words and get a real definition.  That's all well and good but we should be able to do better.

Many books with extensive sets of characters contain a Dramatis personae or list of characters at the beginning of the book.  This can be especially important when reading Science Fiction where some authors show how alien their characters are by giving them unpronounceable names.  Imagine being able to click on a name in your book and get the quick description of who that person is?


A further enhancement to this idea would be to generate what we know about the person at that point in the story.


Some people might view these not as enhancements but as crutches that remove the need to immerse oneself in the book.  I would respectively disagree and say they provide a way to become more immersed in the book, but it doesn't matter.  They are mostly just examples of simple things we can do to play with the book reading experience.


Sadly, while Amazon provides an API to create applications like games on the Kindle they do not (as far as I've been able to tell), provide a way to modify/extend the actual book reading experience.  I don't blame them as most "enhancements" would like be misguided but its still a shame that we can't currently play with the idea.

Scala Static Analysis

I am finally getting around to working on my next pet project which is a Static Analysis tool for Scala.

Java has a number of quite good Static Analysis tools such as:
  checkstyle - enforces a large set of user configurable rules about things like
                       max-line-length, max-parameters, max-if-nesting,
                       max Cyclomatic complexity, etc
  findbugs    - looks for common bug patterns

Scala, for all of its power currently lacks such tools.  There is even debate within the community if such tools matter for functional languages.  Still, it seems like a very interesting problem to me so I decided to give it a go.

In talking with people at Boston Scala Days 2012 the consensus was that a compiler plugin was the best way to approach this problem.  So, I started thinking about a two pronged approach:

  1 - develop a compiler plugin to get access to the Abstract Syntax Tree of a program

 2 - start a discussion about what kind of rules or metrics might make sense for a language like Scala.

The plug-in itself splits into at least two parts: the skeleton that hooks itself into the compilation process and the part that accesses the AST itself.

There is a great "how to" article on the compiler plug-in skeleton at http://www.scala-lang.org/node/140  This article shows how to build a trivial plug-in that looks for divide by zero errors.  The guts are in the "apply" method...and that's where things got spooky for me!

      def apply(unit: CompilationUnit) {
        for ( tree @ Apply(Select(rcvr, nme.DIV), List(Literal(Constant(0)))) <- unit.body;
             if rcvr.tpe <:< definitions.IntClass.tpe)
          {
            unit.error(tree.pos, "definitely division by zero")
          }
      }
I know that this method iterates over the unit.body tree looking for items that are division, with a literal zero and an integer.  That doesn't mean however that I fully grok the Tree hierarchy and its various flavors of Apply and Select methods!

I will say I'm getting some very good help on this via a question I asked on StackOverflow:
http://stackoverflow.com/questions/10419101/how-can-i-find-the-statements-in-a-scala-program-from-within-a-compiler-plugin

More as I get smarter about Abstract Syntax Trees!  If anyone wants to help on this project please contact me as I clearly could use help.  I'll put the project up on GitHub soon.

Hard because its hard or because its easy?

Some problems are hard because they're actually hard.
Some problems however are only hard because the answer is so easy that no one tells you about them.

I recently encountered a problem of the latter type while playing around with Scala's Option type.  If you are not familiar with it Option is a type that can either hold a thing or hold nothing.  So, Option[String] is either a String or its nothing (actually None in Scala).  For reasons that you probably either already know about or don't care to know this is very useful when coding in Scala.

You can say things like:
   foo match {
     case None => // handle the case where is is None
     case Some => // handle the case where there is something
}

(Of course we Scala folks like tricky things and so we'd tend to say
  "case _ =>" in the second case, but for now you can ignore that as showing off).

So, imagine you are in the second case and actually have a Something.  How do you get at it?

You might think you'd say foo.Some, or Some(foo) or a variation on that theme.  Failing that you might check the documentation on Option or read up on the topic in one of the several fine books on the topic.  Unless you looked someplace I didn't you won't find too much.  Foo.some doesn't exist and most variations of Some(foo) give truely unexpected answers.  For example, try this:

val bla = 4
val foo : Option[Long] = Some(3L)
println(bla > foo)

This results in the pretty confusing error message:

:10: error: overloaded method value > with alternatives:
  (x: Double)Boolean
  (x: Float)Boolean
  (x: Long)Boolean
  (x: Int)Boolean
  (x: Char)Boolean
  (x: Short)Boolean
  (x: Byte)Boolean
 cannot be applied to (Some[Option[Long]])
              bla > Some(foo)
                  ^

So, how do you access the "some" part of an Option?  The "obvious after the fact" answer is with "get".

You can say println(bla > foo.get)

I bet that most people learning Scala have struggled with this or perhaps are still struggling with it.  However, it seems like such a dead simple case that most people won't admit to having been confused by it.

That's my definition of hard because its "too simple to mention".

Retrospective Velocity - part 3 of 3

While it may take a huge effort to get your organization to move to Scrum, you can as an individual go compute your personal velocity right now.  Any bug tracking system out there will allow you to gather data about the number of bugs assigned to you and/or fixed by you per unit time. If it were a database query you’d do something like “select count(*) from bugs where assignee==”me” group by month(assignedDate)”.  Now, you may object saying “but I do lots of other things besides fix bugs”.  You would  likely be correct, and it still doesn’t matter.  If a third of your time historically gets absorbed by mind numbing meetings what makes you think next month will be any different?

All discussions of Black Swans aside, whatever last month looked like is a good starting point for guessing what next month might look like.  By all means measure each month and look for trends.  Perhaps you and/or your organization is getting better and your velocity is increasing.  Now we have more confirmation of that fact.  Or perhaps in response to missed deadlines you’re attending still more mind-numbing meetings to discuss such gems as why things take too long and so your velocity is decreasing.  Cold comfort perhaps but now you have data to back up that dark realization.

As an individual you might pick from several data sources to establish a velocity.  The selection may depend on your job function (sustaining engineer, development engineer, etc) and/or on the systems you are using for bug tracking, source code management and the like.

You might track the number of bugs fixed by you, the number of source code checkin’s you do, the number of code inspections you participated in or some combination of these and others.  One interesting strategy is to gather the data from as many interesting sources as possible and then graph the results for longest time period that you have data for.  If we assume that your actual theoretical velocity has been relatively constant you can select a data source or sources that results in a flat graph.  Of course if you have switched languages or processes or problem domain your velocity likely has not been constant so your mileage may vary.

Retrospective Velocity - Part 2 of 3

One of the key take away messages from the experts in the field of estimation is not to estimate at all.  To quote Steve McConnell: “ If you can’t count the answer directly, you should count something else and then compute the answer by using some sort of calibration data”.  So, if we can’t count our velocity directly we can count the number of bugs in our last project, how long they stayed open, how many requirements were present and so on.  Our calibration data is that all of those requirements and bugs took place in the time interval of the project.  It’s not especially fine grained but in a waterfall model we are not looking for fine grained data.

Some may argue that bug reports and requirements are not precisely defined and standardized.  What passes as a single requirement for one team might be 3-5 separate requirements for another team.  Some teams have bugs like “it doesn’t work” while others might enter a dozen or more particular bugs to cover the same underlying defect.  Here is the strange thing though: it doesn’t matter.

Just like story points are not standardized, all that matters is that within your team you tend to be consistent.   Whatever your team’s definition of a story point is if you’re doing Scrum is likely to represent about the same quantum of work next month that it represented last month.  In the waterfall world,  the level of granularity you bring to your bug reports is likely to be fairly constant.  The point to keep in mind is that we’re not looking to equate points or bug counts against anything but other points and other bug counts.  So, if your last several projects had 6 requirements, generated 60 bugs and took 6 months you have a velocity of 1 requirement and/or 10 bugs per month.  If you next project arrives with 15 requirements and a deadline of three months from now we can safely conclude that you are in trouble!

Keep in mind the distinction between accuracy and precision.   In the proceeding case we can say with high confidence that you are probably hosed.

Retrospective Velocity - Part 1 of 3

So how long is this going to take?  This is one of the most common questions to ask or be asked in a traditional software development environment, and one of the most difficult to answer.  The agile methodologies, and Scrum in particular address this problem with the notion of velocity.  Whatever your team accomplished in the last time period (sprint) is likely similar to what they’ll be able to accomplish in the next one.  Scrum’s use of burn down charts, standard definitions of “done” and retrospectives allows for the discovery of a team’s average velocity.  One can argue about hours versus story points and many similar details but almost any implementation of Scrum allows for tracking how much work was done per unit time.

Sadly, many projects are still using “traditional” methods (which seems to be one of the new names for waterfall).  One of the drawbacks of this approach is that the time scales are relatively quite large compared to Scrum, which also means there are fewer time periods to measure.  There is simply less data available is you have two six month buckets versus 12 one month buckets.  This is one reason that “traditional” projects do not typically end up producing a team velocity.  This in turns make it substantially challenging to estimate how long the next project will take.

There is however other data that can be mined so as to uncover this more fine grained time measurements we are looking for.  The data in question lives in your requirement tracking system, bug database and source code control system.  While these systems are often, shall we say imprecise, they do contain useful data.

Using QuickSort to Explore Scala Collection Methods

In my continuing effort to play with my new favorite language Scala I decided to look at one of the Standard Algorithms in the field: quick sort. Quick Sort is a divide and conquer algorithm that picks a pivot point in a list and then calls itself to sort the two halves.


StackOverflow.com demonstrated a very simple Scala quick sort shown below. Let’s examine it as a learning vehicle. The complex looking first line defines a function that takes a List of things T where T is any class that supports "Ordered". It also returns such a list.

 
 
 
 
 
 
 
As with almost all recursive algorithms quick sort has two cases: the empty list and the non-empty list. Scala's "list match" sets up those two cases. In the case of the Nil list we return Nil which is how the recursive algorithm completes. Otherwise we match against "cdr::cons". That expression means "first item in the list followed by the rest of the list". Cons and cdr are the standard names that the Lisp language uses for those terms. This may seem like an odd way to view a list but in the functional world it is a very common idiom.


So given a first element and a rest-of-the-list what do we do? We use another of Scala powerful collections function: partition. "cons partition (_ < cdr)" says "create two lists: a list of the items in the list that do match the condition and another list of the items that do not match the condition. These two lists are returned into the anonymous tuple “val (before after)”. This shows how Tuples are first class citizens in Scala, allowing us to return multiple values from a function call.

Searching the web for Java implementations of quick sort shows the following code as quite typical for the partition portion of the algorithm:

I don’t know about you but Listing 2 looks prime for off-by-one errors and is nothing I’d expect to get right the first time. Compare that to: val (before,after) = cons partition (_ < cdr). The beauty of that line is that its close to the English description: create lists of the items before and after the specified item by partitioning the list based on a test. I’m beginning to feel that Scala feels hard not because it is hard but because it’s so different from Java in that its easy!

Lastly we have the recursive part of the function. We call ourselves on the before list and the after list, and build a new list of the results of those two calls plus the cdr value (because it’s not in either list). This implementation works and has the advantage of being a tiny bit of code. The drawback is that it always picks the first item in each list as the pivot point. It is known to be sub-optimal especially in the case where the function is called on an already sorted list.

This next version of the program picks a better pivot point, in this case the middle entry of the list.  We accomplish this by changing the "case" to "theList : List" which just  means any list.  We manually find the pivot point and then perform two for-comprehensions to find the list items larger and smaller than the pivot entry.  This gains performance via the better pivot point but trades that off against having to scan the main list twice for the two for-comprehensions.  Further testing revealed that this implementation also had a defect in that if an item was present in the list more than once it would only appear once in the  sorted output.  Another reason for robust testing!

Our last version (for now) tackles the duplicate issue and is also a bit more efficient. We go back to using partition to generate our two lists, but now we post-process the second list.  We call partition again on the second list (which contains items not-less-than the pivot) into a list of matching items and not matching items.

Summary: This article is not meant to create the best possible Scala implementation of  Quick Sort, but to give you a vehicle for playing with Scala list manipulation functions.

Scala version of Log4JFugue, plus revamped website

Over the recent holidays I decided to create a Scala version of my Log4JFugue open source music project. I was able to reduce the 2500 lines of Java to just over 250 lines of Scala!

Checkout the new www.log4jfugue.org as well as:
Scala sources

I'll be giving a talk on the experience of converting a small Java project to Scala at the March meeting of New England Java User's Group.

new article in December PragProg magazine

I just had an article published in the December Pragmatic Programmer magazine on
a new approach to logging