Correcting the Billion Dollar Mistake

Written by Jevgeni Kabanov on February 1, 2009 – 11:04 pm

Last week I visited Stockholm to speak at the JFokus 2009. The event was quite spectacular, but for me the most interesting part occurred on the evening before the conference. I was sitting at the speaker’s dinner with Rickard Öberg, Kirk Pepperdine, Simon Ritter and a couple of others. For some reason or other I started talking to Simon about the problem that’s recently been on my mind. Perhaps it’s been eating me after legendary Tony Hoare said this:

I call it my billion-dollar mistake. It was the invention of the null reference in 1965. [...] This has led to innumerable errors, vulnerabilities, and system crashes, which have probably caused a billion dollars of pain and damage in the last forty years.

Now since I have a background in functional languages, I know that null references are not necessary. Null value is a member of all types, but some types don’t have or need a natural notion of absence. In fact in Haskell there is no such thing as an “absent” value generic to all types — types either have to declare it explicitly or wrap the underlying value into a Maybe type that has a dedicated Nothing value.

So e.g. the List type has a notion of an empty list encoded into the type:

// Roughly: a list is an empty list
// or a pair of a value and a list
data List a = Nil | Cons a (List a)

While the only way to encode an absent structure is to use Maybe:

// To build a car you need at most one engine
// and zero or more wheels :)
buildCar :: Maybe Engine -> [Wheel]  -> Car

So while I was discussing that with Simon, Kirk banged me on the head to get my attention (just joking, banging me on the head is nowhere near enough to get my attention). Apparently they were discussing more or less the same topic with Rickard, and he had a similar solution for Java. I assumed it was the widely discussed @NotNull annotation, but it was way cooler than that.

Basically in the beginning Rickard started by adding @NotNull support to Qi4J (BTW an amazing piece of engineering, definitely check it out), so that they could automatically insert runtime assertions for the parameters having the @NotNull annotations, something like that:

PLAIN TEXT

JAVA:

class Test {
void transfer(
@NotNull Account from,
Account to,
double amount) {
//…
}
public static void main(String[] args) {
// Succeeds
transfer(
new Account(“Bugs Bunny”), null, 1000000.0);
// Throws NullPointerException
transfer(
null, new Account(“Bugs Bunny”), 1000000.0);
}
}

However soon after that they discovered that they were inserting @NotNull-s everywhere. So they reverted the notion and decided that they will generate the not null assertions for all parameters except the ones marked as @Optional. And this was the point when I thought to myself “Jevgeni, this is exactly what you were looking for!” So I turn to Rickard and I say “Rickard, this is exactly what I was looking for! This is amazing!” In fact this truly is amazing as it’s exactly captures the semantics of the Maybe type that I liked so much in the functional languages.

After that we go into a discussion whether or not it is possible to validate this assertion during compile-time. Opinions were mixed on this one, though I personally am convinced that it shouldn’t be too easy (your opinion on the topic is welcome, also it will make a great master’s thesis topic). Eventually I get an idea and I say “Rickard, I bet I could implement a JavaRebel plugin that would check this at runtime in about half an hour in 50 lines of code”. And of course Rickard goes “No way!”, so I’m challenged. Next day I sit down for half an hour, then for another half an hour (there was no internet!) and voila — I have a working JavaRebel plugin (in less that 50 lines of code) that will make your methods throw an exception if you try to pass a null reference to a parameter not marked as @Optional (the plugin code deserves another post altogether). Of course I have to show this to Rickard (and he goes “No way!”) and we agree to somehow join forces to promote this sane approach (starting with having a single namespace for the @Optional annotation).

So what do we get? The previous example now looks like this:

PLAIN TEXT

JAVA:

@OptionalCheck
class Test {
void transfer(
Account from,
@Optional Account to,
double amount) {
//…
}
public static void main(String[] args) {
// Succeeds
transfer(
new Account(“Bugs Bunny”), null, 1000000.0);
// Throws NullPointerException
transfer(
null, new Account(“Bugs Bunny”), 1000000.0);
}
}

As you can understand if we make all of the code without the @Optional annotation to throw a NullPointerException for nulls we’ll break a lot of existing code. Therefore at the moment you also have to annotate the class where you want to enable such semantics with @OptionalCheck.

That’s pretty much it — you can download the plugin right away and just drop it in the classpath when JavaRebel is enabled (you’ll need a 2.0 milestone as 1.x was missing the necessary APIs). At the moment both annotations are in the org.optionalalliance package, but when it changes all you have to do is Organize Imports, so I won’t sweat the naming too much. Please do let us know what do you think of the approach and feel free to advocate it further :)

Cheers,
Jevgeni Kabanov

P.S. The plugin along with the source is also avalable in our Maven repository:

URL: http://repos.zeroturnaround.com/maven2
Group id: org.zeroturnaround
Artifact id: javarebel-optional-check-plugin

Posted in Featured, creative | 31 Comments »

21 Other Comments

31 Comments to “Correcting the Billion Dollar Mistake”

Stephan Schmidt Says:
February 2nd, 2009 at 9:01 am
Very nice, though I do not like NullPointerException as an exception in this case.

I discuss often in interviews, if you should do this:

if (null == value) throw new NPE()

If a developer sees a NPE thrown in the stack trace, he doesn’t look for a null check but for dereferencing null:

value.method()

and won’t find the offending cause.

IllegalArgumentException would be much clearer than an NPE in this case (It’s an illegal argument, null, which is passed to the method).

Stephan

–
Programming is hard – http://blog.codemonkeyism.com
http://twitter.com/codemonkeyism
Jesse Says:
February 2nd, 2009 at 9:11 am
Fantastic. This is way better than the proposed ?. operator which attempts to sweep nulls under the carpet. It also doesn’t require a language change, so we can get it today rather than waiting for Java 7. Awesome!

http://docs.google.com/View?docid=dfn5297z_19pnsjkfc6
Jevgeni Kabanov Says:
February 2nd, 2009 at 12:30 pm
Thanks, guys!

@Stephan
There are things to be said for both NPE and IAE. I picked NPE for the moment, though it includes an explanatory message including method and parameter name. But it’s very easy to change if the community prefers it otherwise.

@Jesse
Actually these complement each other. Especially if the compiler would check if the method in chain can even return null before allowing you to handle it with .?.
Stephan Schmidt Says:
February 2nd, 2009 at 12:43 pm
Great to hear – Stephan
Attila Szegedi Says:
February 2nd, 2009 at 12:58 pm
I’d tend to agree with Stephan – IllegalArgumentException is the proper exception for rejecting an argument with illegal value on method invocation, including a null value where null is not acceptable. Normally, Java code would never throw a NPE explicitly; it’s JVM that throws it when null is dereferenced.

This of course applies to method invocations; if you’d want to prevent null value assignment through other means (i.e. explicit field assignment) then it’s a good question what would you throw there. OTOH, applying this logic to other assignments is probably futile, as you can guarantee in your code that you aren’t assigning nulls; it’s really the method invocations from outside your code where you need to check.
Toomas Römer Says:
February 2nd, 2009 at 2:36 pm
Kirk has also mentioned this idea in his JFokus post, http://www.kodewerk.com/jfokus_2009.htm
Mattias Says:
February 2nd, 2009 at 3:03 pm
As the Jfokus subtitle says:
When people meet ideas are born
Jevgeni Kabanov Says:
February 2nd, 2009 at 4:24 pm
@Atilla
NPE is all the time thrown by Java code. It is actually an idiom in the Java API to check for null and throw an NPE (see e.g. String source). There are also quite a few lines of Java code that do the same.

The question really is whether you want to handle this kind of situation together with the rest of NPEs and whether there is a chance that the user will confuse it with some kind of a bug (NPEs are strongly associated with bugs for most developers). I tried to get the best of both worlds and throw an NPE with a detailed message, but perhaps you are right and a better idea is to throw an IAE.
Vlad Says:
February 2nd, 2009 at 5:44 pm
Is this a more concise form of achieving:

void transfer(Account from, Account to, double amount) {
if (from == null) {
throw NullPointerException(…);
}
//…
}

Or is there something that I am missing?
Stephen Colebourne Says:
February 2nd, 2009 at 5:46 pm
Jesse, I agree with Jevgeni that the best approach is to have both the ?. operator and non-null variables by default. This is what the Fan programming language does (hhtp://fandev.org), and the result is perfect. (Even with non-null variables, you still use nulls for some situations, and ?. is needed for those)

I didn’t think that this could be achieved in Java, hence I wrote http://docs.google.com/Doc?id=dfn5297z_3c73gwb to mark Java types as non-null using a type modifier. I still don’t like the annotation approach as it is very wordy, but its good to see it in operation.
Jevgeni Kabanov Says:
February 2nd, 2009 at 6:30 pm
@Vlad
Pretty much :) Except that in the second case all of the methods will reject null by default.
Shams Mahmood Says:
February 2nd, 2009 at 6:39 pm
while this seems a good idea i prefer explictly performing checks than relying on annotations. So it would go something like:
void transfer(Account from, Account to, double amount) {
Validator.notNull(from, “From was Null, require non-null value”); // throws some Exception if from is null
// other validation checks

// Actual code for the method
}
Imagist Says:
February 2nd, 2009 at 9:33 pm
NullPointerException? IllegalArgumentException? Why not NullArgumentException (extends IllegalArgumentException)?
Stephan Schmidt Says:
February 2nd, 2009 at 10:33 pm
Fan is out for me because it doesn’t have generics. Why prevent NPEs and have CCE?
Jevgeni Kabanov Says:
February 2nd, 2009 at 10:34 pm
@Stephen
Annotations may be verbose, but making the null-safe to be the default case actually decreases clutter. Adding a null-safe # everywhere will most likely clutter the code even more than a few @Optional annotations. The problem is of course reversing the default semantics and also that by looking at a method you have no idea if it’s null-safe or not.

If the support would be baked into the compiler and the IDE (both not impossible with the magic of JSR-s) then this would be quite viable, as the IDE could feedback the programmer immediately. You could use the same analysis for @NotNull and @Optional and with the local variable annotation support you could do it everywhere without any syntax changes. Plus you could have the runtime validation right away.

BTW, I looked at your spec and isn’t the “#Person p = null;” problem solved by just declaring “#Person p;” and letting the compiler ensure that it’s assigned (this works today as well).
Jevgeni Kabanov Says:
February 2nd, 2009 at 10:36 pm
@Stephan
Well, Haskell is out for me for a lot of reasons, but that doesn’t mean they don’t have a bunch of great ideas in the language. Fan looks interesting too.
Stephen Colebourne Says:
February 2nd, 2009 at 11:42 pm
@Stephan, Fan’s take on generics is far more interesting than “it doesn’t have them”. Fan has List, Map and Function (closure) classes that make full use of generics, and that is where the bulk of usage in Java is today.

CCEs are also different because Fan allows you to convert between types without casts in most circumstances (auto-casting). The point is that Fan shouldn’t be seen as trying to replicate the (pretend) tight constraints of Java typing or the (excessive) constraints of Scala static typing. Instead it focuses on a slightly more dynamic approach – a long way from Groovy/Ruby, but enough to make coding simpler and more enjoyable IMHO.
Isaac Gouy Says:
February 3rd, 2009 at 4:51 am
Examples like these always make that JVM language Nice look good!

void main(String[] args){
// Succeeds
transfer(
new Account(id: “Bugs Bunny”), null, 1000000.0);

// Compiler error
transfer(
null, new Account(id: “Bugs Bunny”), 1000000.0);
}

void transfer(
Account from,
?Account to,
double amount){
//
}

>nicec –sourcepath .. -a t.jar test
nice.lang: parsing
nullaccount: parsing
nullaccount: typechecking

test.nice: line 7, column 5:
Arguments ( ?any, test.Account, nice.lang.double) do not fit:
nice.lang.void transfer(test.Account from,?test.Account to,nice.lang.double amount)
compilation failed with 1 error
Rickard Öberg Says:
February 3rd, 2009 at 4:56 am
@Shams
The reason the @Optional annotation is better is that it not only triggers the check, but is also a form of documentation for the client coder. Usually there is no way to look at the code and know what parameters can be null. With the @Optional this is very clear, both for human (=visual) and non-human (=reflective) clients. Using @Optional instead of @NotNull has been argued for elsewhere, so I won’t repeat that.

We introduced this in Qi4j some time ago, and as some commenters recommend we use IAE instead of NPE, as this makes it easier to write clients. They only have to handle IAE, and since we also have parameter constraint checking (e.g. “String may not be empty”, “int must be higher then 10″) that throws IAE’s with i18n support we figured that it would make sense to include the null-handling with it.
Shams Mahmood Says:
February 3rd, 2009 at 5:24 am
@Rickard
Yes the annotations provide a better form of documentation as long as they appear on the methods and parameters, but in certain scenarios you will have to resort to additional checks inside the code.

e.g. Validator.validateGreaterOrEqual(from.getBalance(), amount, “error.insufficient.balance”);

To use annotations one would have to extract the value into a local variable and prepend the annotation. Once annotations start appearing inside the method then the point of added visual cue is no longer valid.
Hence my preference towards using actual code for validation over annotations.
Rickard Öberg Says:
February 3rd, 2009 at 7:14 am
@Shams
The example you noted is a different cup of tea. Constraints should be used for constraining the value being sent in, not performing domain logic checks. “amount” in your example, would have the constraint “must be over zero”, so @MinValue(1) would be ok as an annotation, but the business rule you showed should throw an application exception, not an IAE. If you start confusing value checking with business rule checking you’re in trouble…
Kirk Says:
February 3rd, 2009 at 10:24 am
@Attila, The contract is documented, don’t send me a null and if you do, I’ll send you back an NPE. Throwing an NPE instead of an IAE is maybe not ideal but it does simplify many things. The only confusion that I can see is when you try to sort out if the NPE belongs to me or the framework. Traditionally we, as developers, don’t throw NPE. The question is, why not? What in our thinking limits us or prevents us from throwing this exception? If the contract is clear, don’t pass me a null, the exception conveys the problem very clearly.

Jevgeni, I’m sure it took you more than 30 minutes ‘cos you disappeared after that conversation and I never saw you again. The only way that happens is if you start coding ;-)

Kirk
Stephan Schmidt Says:
February 3rd, 2009 at 11:27 am
BTW Some time ago I held a presentation about

“Better Strategies for Null Handling in Java”

added to SlideShare 5 minutes ago

http://www.slideshare.net/Stephan.Schmidt/better-strategies-for-null-handling-in-java

Stephan

–
Programming is hard – http://blog.codemonkeyism.com
http://twitter.com/codemonkeyism
Jevgeni Kabanov Says:
February 3rd, 2009 at 1:03 pm
@Stephan
Nice! I see you’re vouching for the same approach on the 7th slide :)
Stephan Schmidt Says:
February 3rd, 2009 at 1:06 pm
Yes :-)
Jevgeni Kabanov Says:
February 3rd, 2009 at 1:07 pm
Apparently JSR 308 already includes compiler inference for @Nullable annotation with non-null default:
http://groups.csail.mit.edu/pag/jsr308/
Jevgeni Kabanov Says:
February 3rd, 2009 at 1:57 pm
@Kirk
Hey, it was you who was late for the talk, not me. Just ’cause you cannot remember the previous evening doesn’t mean it never happened :P
Kirk Says:
February 3rd, 2009 at 3:19 pm
Hey, I was there and I do remember it all :)
Shams Mahmood Says:
February 3rd, 2009 at 6:22 pm
@Rickard
I guess we are moving away from the main topic but I’m enjoying this, hence I persist ;). In the case of transfer() the constraints are what is expected of the arguments before transfer() can expect to execute properly. Just like making from non-nullable is a constraint, having from.balance ge amount can be looked upon as a constraint for the caller (deferring responsibility).
Peter Lawrey Says:
February 4th, 2009 at 12:59 am
I believe JSR-303 suggest @Notnull and @Nullable.
A type which few people consider could be null is enum types. e.g. This can throw an NPE!

switch(myEnum) {
case ENUM1:

IMHO, Variables/fields/parameters should be ‘final’ and not-null by default and only be non-final and/or nullable if explicitly defined so. (Also fields should private by default)
Peter Lawrey Says:
February 4th, 2009 at 1:03 am
If you read the news these days, losing a 25 million a year for forty years does sound as much as it used to. 8-/
I prefer fail-fast approaches and not having null could have been worse than what we might have had!

dow.ngra.de

no buzzwords allowed