Today we had a most interesting encounter with the quirks of our environment and RMI. We use RMI in our new LiveRebel product to connect the management console to the remote agents. Commonly, the agent code is obfuscated by Proguard. However as obfuscation renames the classes it prevents us from doing any debugging, so I made a vanilla build and started the debugger.
BOOM! The particular method I needed to call throws an exception “java.rmi.UnmarshalException: unrecognized method hash: method not supported by remote object”. Googling tells me that this usually means that different classes are deployed on the client and server, so I spend the next half an hour making sure that the JAR on the client and server is exactly the same. A complicating condition is that this particular method returns an object from a third-party library, and to avoid conflicts with the user code we rename the package of that library using JarJar.
As I establish beyond reasonable or unreasonable doubt that the JAR really is the same I start looking into other potential issues. As the exception reports the hash to be missing I start looking into how the hash is calculated for the RMI calls. A fair amount of debugger stepping (a useful tip is that you can put a method breakpoint even if you don’t have the code) reveals that sun.rmi.server.Util.computeMethodHash() calculates the hash using the digest of the method name and signature. Not much room for mistakes there, but nevertheless I put a method breakpoint on UnicastServerRef.dispatch(), which is the last call in the stack trace and dump hashes for all methods in the target class from a static map.
Next I make sure that the hashes are the same on client and server. Decompiling the RMI Stub shows that it includes the precalculated hashes in the call. First check on a different method that I know to work shows that the hashes are the same. But the hash for the problematic method is different in the Stub than in the server map. WTF is wrong?
Here I probably should have verified the next assumptions, but luckily an epiphany interferes. Turns out that the rmic compiler is called before the third-party library is renamed, so the hash is calculated with the wrong packages in the signature. As the JarJar will then rename the signature in the Stub class, it is almost impossible to notice. However the last remaining question is why the hell did it work at all? The hashes should have been different all the time, not just in the vanilla build. A little digging reveals, that as Proguard commonly renames the classes in RMI signatures it will go and fix the RMI hash for you.
EOF