Jevgeni Kabanov

The Ultimate Java Puzzler

February 16th, 2009 | by Jevgeni Kabanov |

Why is this particular one the ultimate? Two reasons:

  • It’s at the very core of the Java language, not some obscure piece of API.
  • It melted my brain when I hit it.

UPDATE 2: If you want to test yourself before reading the post take this test. Results are not saved (it’s a paid feature apparently and I just don’t care enough), but you can post them in the comments.

Let’s start by setting up the puzzler environment. We’ll have three classes in two packages. Classes C1 and C2 will be in package p1:

JAVA:
  1. package p1;
  2. public class C1 {
  3.   public int m() {return 1;}
  4. }
  5. public class C2 extends C1 {
  6.   public int m() {return 2;}
  7. }

Class C3 will be in a separate package p2:

JAVA:
  1. package p2;
  2. public class C3 extends p1.C2 {
  3.   public int m() {return 3;}
  4. }

We will also have the test class p1.Main with the following main method:

JAVA:
  1. public static void main(String[] args) {
  2.   C1 c = new p2.C3();
  3.   System.out.println(c.m());
  4. }

Note that we’re calling the method of C1 on an instance of C3. The output for this example is “3″ as you’d expect. Now let’s change the m() visibility in all three classes to default:

JAVA:
  1. public class C1 {
  2.   /*default*/ int m() {return 1;}
  3. }
  4. public class C2 extends C1 {
  5.   /*default*/ int m() {return 2;}
  6. }
  7. public class C3 extends p1.C2 {
  8.   /*default*/ int m() {return 3;}
  9. }

The output will now be “2″!

Why is that? The Main class that invokes the method does not see the m() method in the C3 class, it being in a separate package. As far as it cares the chain ends with C2. But as C2 is in the same package it overrides the m() method in C1. This does not seem too intuitive, but that’s the way it is.

Now let’s try something different, let’s change the modifier of C3.m() back to public. What will that do?

JAVA:
  1. public class C1 {
  2.   /*default*/ int m() {return 1;}
  3. }
  4. public class C2 extends C1 {
  5.   /*default*/ int m() {return 2;}
  6. }
  7. public class C3 extends p1.C2 {
  8.   public int m() {return 3;}
  9. }

Now Main can clearly see the C3.m() method. But amazingly enough output is still “2″!

Apparently C3.m() is not considered to override C2.m() at all. One way to think about it is overriding methods should have access to the super methods (via super.m()). However in this case C3.m() wouldn’t have access to its super method, as it it not visible to it, being in another package. Therefore C3 is considered to be in a completely different invocation chain from C1 and C2. Were we to call C3.m() directly from Main the output would actually be “3″.

Now let’s look at one last example. Protected is an interesting visibility. It behaves like default for members in the same package and like public for subclasses. What will happen if we change all of the visibilities to protected?

JAVA:
  1. public class C1 {
  2.   protected int m() {return 1;}
  3. }
  4. public class C2 extends C1 {
  5.   protected int m() {return 2;}
  6. }
  7. public class C3 extends p1.C2 {
  8.   protected int m() {return 3;}
  9. }

My reasoning goes like this: as Main is not a subclass of any classes protected should behave as default in this case and output should be “2″. However that is not the case. The crucial thing is that C3.m() has access to super.m() and thus the actual output will be “3″.

Personally, when I first encountered this accessibility issue I got thoroughly confused and couldn’t get it until I did all of this examples through. The intuition I got from this is that if and only if you can access super.m() the subclass is a part of the invocation chain.

UPDATE: Apparently even though the whole thing is obvious to anyone, the intuition I came up with was wrong. A mysterious commenter know only as “C” has provided the following example:

JAVA:
  1. public class C1 {
  2.   /*default*/ int m() {return 1;}
  3. }
  4. public class C2 extends C1 {
  5.   /*default*/ int m() {return 2;}
  6. }
  7. public class C3 extends p1.C2 {
  8.   /*default*/ int m() {return 3;}
  9. }
  10. public class C4 extends p2.C3 {
  11.   /*default*/ int m() {return 4;}
  12. }

Note that C4 is in the package p1. If we now change the Main code as follows:

JAVA:
  1. public static void main(String[] args) {
  2.   C1 c = new C4();
  3.   System.out.println(c.m());
  4. }

Then it will output “4″. However super.m() is not accessible from C4 and putting @Override on the C4.m() method will stop the code from compiling. At the same time if we change the main method to:

JAVA:
  1. public static void main(String[] args) {
  2.   p2.C3 c = new C4();
  3.   System.out.println(c.m());
  4. }

The output will be “3″. This means that C4.m() overrides C2.m() and C1.m(), but not C3.m(). This also makes the issue even more confusing, and the amended intuition is that a method in a subclass overrides a method in a superclass if and only if the method in the superclass is accessible from the subclass. Here superclass can be any ancestor, not necessarily the direct parent and the relation has to be transitive.

For the kicker try reading all of this out from the JVM specification that selects the method to be invoked:


Let C be the class of objectref. The actual method to be invoked is selected by the following lookup procedure:

  • If C contains a declaration for an instance method with the same name and descriptor as the resolved method, and the resolved method is accessible from C, then this is the method to be invoked, and the lookup procedure terminates.
  • Otherwise, if C has a superclass, this same lookup procedure is performed recursively using the direct superclass of C; the method to be invoked is the result of the recursive invocation of this lookup procedure.
  • Otherwise, an AbstractMethodError is raised.

  • Jonathan
    ah, I see the issue now. My apologizes d :)
  • Jevgeni Kabanov
    @jonathan

    C4 has to be in p1, not p2.
  • Jonathan
    d,

    What "test case" are you running? Just for the hell of it, I put it into my IDE, and sure enough, C4.m() does not override C2.m() like I stated above.

    You might want to read the link I posted above.

    ----------------------------
    package p1;

    public class C1 {
    int m() {return 1;}
    }

    ----------------------------
    package p1;

    public class C2 extends C1 {
    int m() {return 2;}
    }


    ----------------------------
    package p1;

    import p2.C4;

    public class Main {
    public static void main(String[] args) {
    C1 c = new C4();
    System.out.println(c.m());
    }

    }

    ----------------------------
    package p2;

    import p1.C2;

    public class C3 extends C2 {
    int m() {return 3;}
    }

    ----------------------------
    package p2;

    public class C4 extends C3 {

    int m() {return 4;}
    }
    ----------------------------
    Output:
    2
  • Jevgeni Kabanov
    @d

    I think there's no way to call C2.m() from C4.m(). Granted, JVM will allow such a call via INVOKESPECIAL, but in the compiler super always refers to direct superclass and there is no other way to do that in Java.
  • d
    If you run the test case, it is clear C4.m() does override C2.m(), even though C2.m() is not directly visible in C4.m(). The question is how to invoke the overridden method C2.m() in the overriding method C4.m(). The normal way of doing it by using super does not work in this case.
  • Jonathan
    C4.m() does not override C2.m(). C2.m() is not visible to C4.m();

    Even if everything was public (instead of package private), you cannot access C2.m() from C4.m() (unless of course C3.m() does not exist or class C3 providers some sort of method to get it like C3.getC2m()). Regardless, if you're trying to do anything like this, it's time to rethink your design.
  • d
    Interesting discussions. I have a question. Since C4.m() overrides C2.m(), how can I invoke C2.m() from C4.m()'s implementation? super.m() does not work. If you cast "this" to C2 and then invoke m(), as ((C2)this).m(), you get stack overflow as you are actually invoking yourself. There must be a way of doing it!
  • Jevgeni Kabanov
    @Hans-Peter

    You still have a wrong mental model :) Which method is called does not depend on the caller, but rather on the target class in the call. Compiler resolved each call to the string Class.method(argsTypes)returnType. "Class" is a part of the message and vtable is selected by it. It's actually not so different from C++ if you think about implementation, but getting to the vtable is quite harder.
  • cgino
    this test is a very good demonstration why one shouldnt go crazy with OO.
    keeping things simple and away from over-sophistication ....saves so much.

    do you want even more fun...jump into aspect-oriented progamming.
  • Sathish
    Thanks for the post, Kathy Sierra author of Sun certification exams has a good explanation about scopes as well.
  • Hans-Peter
    Thanks for the post! This was quite baffling to me. It turned out I had a completely wrong mental model.

    I always thought Java did something comparable to the C++ virtual method table - that is, which method is called does not depend on the caller - and the compiler makes sure that this does not break the encapsulation rules. And now it turns out that Java does something completely different. 8-)
  • Maybe the lesson is that package private methods should not be part of any class that is designed for inheritance.
  • Jevgeni - very interesting discussion! I think that some confusion could arise here because of the definition of invokevirtual that you've seen.

    Looking at the clarifications and amendments to the JVM specification at http://java.sun.com/docs/books/jvms/second_edit...
    you can see that it makes a lot more sense when the text "and the resolved method is accessible from C" is replaced with "and M overrides the resolved method" (especially with regards to "question 3" on the test that you've given)

    What do you think?
  • c
    @Jose, if C3.m() is public then C4.m() overrides that and hence you can't reduce the visibility.

    @am, the problem is not that this is not the correct behaviour, but that @Override causes compile error if everything is package private even though C4.m() overrides C1.m() (and C2.m()) - as Jevgeni pointed out a few comments before.
  • Simon
    Interesting. I think the problem isn't that it's complicated - the answer is fairly intuitive once you realize you're dealing with package-level protection. The problem is that it's really easy to simply not notice that fact, and the compiler won't catch it for you, even with maximum warnings turned on.

    It is, however, a good reason to be using a good IDE. Trying this setup in Eclipse, it gives me the helpful warning that "The method M3.doIt() does not override the inherited method from M2 since it is private to a different package".

    Despite what a few are saying, this is a very useful post. It might be simple stuff, but it doesn't mean a reminder isn't a good thing.
  • am
    I think the behavior is correct because C3 should have no effect on C1's package protected behavior. Doing so could be disastrous.

    (of course I rarely use package protected and protected anyways.. I'm mostly a public and private guy)
  • Taimo Peelo
    @c

    "OTOH, @Override doesn’t give you a compiler error when you implement a method declared in an interface even though this shouldn’t count as overriding"

    In Java 5 one actually had to override existing method implementation to get code to compile. In Java 6 it was changed -- there it can even be applied to directly implemented abstract (interface) methods. Java 6 API docs fail to reflect the @Override annotation recent behaviour correctly though (http://blogs.sun.com/ahe/entry/override_snafu).
  • c
    Jevgeni, thanks, but I like the "mysterious commenter know only as “C”" :) Let's keep it that way.

    Please also note, that your last example won't work as C3.m() is not accessible from p1.Main (assuming C3.m() is still package private). This needs to be a new Main class in the p2 package.
  • Tarun Elankath
    Sorry, didn\'t understand what was the puzzler here...seemed rather obvious to me. You can only override public or protected methods from a different package right?
  • Jevgeni Kabanov
    @Jose

    I think because reducing visibility applies to shadowing hierarchy, not overriding one :) Anyone still thinks this issue is trivial?
  • Jevgeni Kabanov
    @c

    There's one more caveat to the intuition, that is explained in the JLS -- it has to be transitive. Otherwise the case where C1.m() is default, C2.m(), C3.m() are protected won't work, as C1.m() is not accessible from C3.m().
  • @c

    C4.m() overrides C1.m() but it doesn’t override C3.m()!

    Why then Eclipse doesn't allow me to reduce the m() visibility of p2.C3 (from public) in p1.C4?
  • Jevgeni Kabanov
    @c

    Thanks! I updated the post, do you mind giving your name for the credit? Sorry that your comments appeared later, but our antispam system held them.
  • Jevgeni Kabanov
    @Jose

    Updated!
  • Interesting indeed. You should rewrite the post to include a C4 and try to explain the final argument ;-)
  • c
    @Override might be broken, though as its definition doesn't restrict you to overriding a method in the direct superclass or overriding the closest method with the same signature/name/return type in the class hierarchy.

    http://java.sun.com/javase/6/docs/api/java/lang...

    OTOH, @Override doesn't give you a compiler error when you implement a method declared in an interface even though this shouldn't count as overriding...
  • c
    C4.m() overrides C1.m() but it doesn't override C3.m()!

    See def:
    http://java.sun.com/docs/books/jls/third_editio...
  • Jevgeni Kabanov
    @c

    I get this now, but this means that the override intuition is basically broken in Java (including the @Override annotation). If you annotate C4.m() with @Override it won't compile, even though it basically does override C2.m(). This means that overrides can jump over classes, which is quite counter-intuitive. I get now that "the bottom-most method that can access the target method will be invoked" is the layman's translation of the spec, but I'd like something applicable generally and not just in the context of the specific call.
  • c
    Jevgeni, The only thing that matters (apart from the correct signature...etc.) is that C4 can see C1.m(). The fact that it can't see C3.m() is only a problem if you want to call super.m() from C4.

    If you add

    package p2;
    public class Main {
    public static void main(final String[] args) {
    C3 c = new p1.C4();
    System.out.println(c.m());
    }
    }

    It will print out 3 because C4 can't see C3.m().
  • Jevgeni Kabanov
    @Jonathan

    This intuition doesn't work for protected methods invoked from the same package, as is shown in the post.
  • It is reasonable, the methods are being sought from the bottom up in the inheritance chain by checking that they are visible from the interface they are invoked. Then, the first class where the JVM looks is C4.
  • SimonB
    What a brilliant question for an interview!! The majority of developers would not get this.
  • Jevgeni Kabanov
    @c

    Wow! That's cool :) But at the same time intuition of being accessible to the caller breaks on protected methods. I'm confused now.
  • c
    Jevgeni, Your way of thinking won\'t work if you add C4 to p1

    package p1;
    public class C4 extends p2.C3 {
    int m() {return 4;}
    }

    change all the methods to package private and modify Main.main to

    public static void main(final String[] args) {
    p1.C1 c = new p1.C4();
    System.out.println(c.m());
    }

    This time m() in C3 is not accessible from C4 (super.m() won\'t compile), yet it will print out 4.
  • Jevgeni Kabanov
    @White Gandalf

    please don't read my blog.
  • White Gandalf
    please make your blog private.
  • I've had to trace these kind of problems before and they used to be an absolute pain when someone would change visibility without using the IDE to refactor. As David and Jonathon said earlier the @Override annotation can be such a help here, I've set eclipse up with a save action to automatically add the annotation. It means I don't forget to and it gets added to legacy code as well.
  • Jevgeni Kabanov
    @David

    Method shadowing is not news to me, it's the nuances that the post is about. Static methods are always shadowed, but AFAIK C++ does not have virtual method shadowing, whereas Java obviously does. Several override chains over same method name in a class hierarchy are also not so common and complicate things.
  • Jevgeni, you just discovered the difference between method overriding and method shadowing. You can not override what you can not see. There is actually a similar problem in C++ if you play with the virtual keyword.
  • @Jevgeni
    No doubt, this is a nasty situation to run into in production code, and it's good that you bring it up in your blog - hopefully people will watch out for this sort of stuff. From the compiler perspective though, the behavior is logical and well-defined. Changing the type of c from C1 to C3 affects the behavior because method resolution is done by the compiler based on the static type of c at the time of call.

    I think it's yet another example of our "object-oriented intuition" and static type rules not playing together nicely...
  • Jevgeni Kabanov
    @Yardena

    Well, I hope that when you actually run into such a situation in real world it will be just as intuitive for you :) Note also that by changing the line in Main to "C3 c = new C3()" you will change the behavior in some of the examples.
  • Well, this is not really a counter-intuitive behavior once you realize that you cannot override what you do not see... So (a) if C3 sees C2#m it overrides it, otherwise it does not. And (b) because of static dispatch, Main binds the invocation only to overriding method, regardless of other matching methods that may be in C3 - just like with the visitor problem.

    I am surprised that you are surprised :-)
  • @Alfred Hodapp

    I think the public embarrassment is yours. This is a very well thoughtful post that actually explains something useful, something that can be confusing.

    Kirk
  • Jonathan
    This article here will clear the confusion:

    http://java.sun.com/docs/books/tutorial/java/ja...

    The best way to use these errors it to use a proper IDE. If you use the @Override annotation you will receive an error immediately: "method does not override or implement a method from a supertype". Even if you do not use the annotation, if you're observant, you will see that netbeans does not mark is as an overridden method. The best way to override methods is to use the IDE to automatically generate the method stub. Since you do this often enough, you should assign a keyboard shortcut if your IDE does not already have a keymap defined (eclipse by default does not).
  • Ken
    Nice article! I found out the same thing last summer at work. Even more tricky is if the super class has a protected member, and you can't access it in the subclass which is in another package. Protected in Java is a little tricker than "protected" in C++.
  • Jevgeni Kabanov
    @Christine

    Well, I actually caught it in a different context, working with someone else's compiled code. But the reason why it's so annoying is because it broke my assumption on how Java executes usual methods.

    I always assumed that there is only one invocation chain, but with packages there can be arbitrary many separate invocation chains. This means that e.g. by changing the visibility of a method from package to protected you may non-locally change behaviour of your application without even realizing it, because you may have merged two chains into one.
  • Christine
    This is a fun one... I know every couple years something like this will pop up and catch even the most experienced developers! You read about package/default scope, make note of the weirdness... then forget because you never use it. Two years later... UGH where is this weird bug coming from? Ooooohhh yeaaaahh....
  • Tool
    @Alfred: I agree with other commenters here; you really are a tool.
  • Jevgeni Kabanov
    @David

    Pretty much what you'd expect, it only compiles if the method really overrides the superclass method (the super.m() visibility intuition applies).
  • David Chuhay
    So, what happens if you add the @Override annotation to your method overrides?
  • Speaker-to-Animals
    Jevgeni, well said.
  • Speaker-to-Animals
    @Alfred: Perhaps you're right, but you do no public service by pillorying the author of this post. The net effect of your pillory will be to discourage people from sharing information others may need, for fear of public ridicule. Who cares if he "should have known better?" He still had sense enough to use a test-driven methodology to work out the language's behaviour, even if he didn't know where to find the information in the first place... and those who didn't know Java's package behaviour still benefit. Your social ignorance is a hundred times more heinous that Jevgeni's ignorance of a technical detail. You're the one who should be embarrassed!

    P.S. Jevgeni: you should really put a better captcha on your site, one less dependent on acuity of colour perception. Also, your form displays backslashes when relaoding due to a captcha failure; you really ought to address that.
  • Jevgeni Kabanov
    @Alfred

    Well, I'm not afraid of public embarrassment, it's almost always a learning experience :)
  • Alfred Hodapp
    If you knew anything about packages before you started this, you could've avoided making this post altogether!

    You could've avoided public embarassment!
blog comments powered by Disqus