dow.ngra.de » Featured http://dow.ngra.de no buzzwords allowed Wed, 04 Aug 2010 13:19:05 +0000 http://wordpress.org/?v=2.9 en hourly 1 ClassLoaderLocal: How to avoid ClassLoader leaks on application redeploy http://dow.ngra.de/2009/06/15/classloaderlocal-how-to-avoid-classloader-leaks-on-application-redeploy/ http://dow.ngra.de/2009/06/15/classloaderlocal-how-to-avoid-classloader-leaks-on-application-redeploy/#comments Mon, 15 Jun 2009 07:51:17 +0000 Jevgeni Kabanov http://dow.ngra.de/?p=926 “OutOfMemoryError: PermGen” is a very common message to see after a few redeploys. The reason why it’s so common is that it’s amazingly easy to leak a class loader. It’s enough to hold a single outside reference to an object instantiated from a class loaded by the said class loader to prevent that class loader from being GC-d.

In this post I’ll review how we solved this problem in JavaRebel, and share the solution with you. It’s not a magical solution, but it will help alleviate some of the problems introduced in both libraries and applications in Java EE.

The most common way to leak is to register some kind of a callback object and never deregister it. E.g. look at the following code:

PLAIN TEXT
JAVA:
  1. Core.addListener(new MyListener());

If Core is a part of the framework/platform/container then it will hold to MyListener long after the application was redeployed and the class loader left hanging.

Let’s see if we can do anything to solve this. The Core implementation looks something like this:

PLAIN TEXT
JAVA:
  1. public class Core {
  2.   List listeners = new ArrayList();
  3.  
  4.   void addListener(Listener l) {
  5.     listeners.add(l);
  6.   }
  7.  
  8.   void fireListeners() {
  9.     // Exercise for the reader!
  10.   }
  11. }

The problem is that listeners provides a strong reference to the Listener object. What if we replace it by a weak one?

PLAIN TEXT
JAVA:
  1. public class Core {
  2.   List listeners = new ArrayList();
  3.  
  4.   void addListener(Listener l) {
  5.     listeners.add(new WeakReference(l));
  6.   }
  7.  
  8.   void fireListeners() {
  9.     // Exercise for the reader!
  10.   }
  11. }

Unfortunately although this does solve the problem of GC-ing the class loader, it doesn’t really work. The Listener behind the weak reference will be GC-d at first opportunity and after that it’ll no longer receive any callbacks. To illustrate why it’s a problem the code above is basically equivalent to throwing the reference away altogether:

PLAIN TEXT
JAVA:
  1. public class Core {
  2.   List listeners = new ArrayList();
  3.  
  4.   void addListener(Listener l) {
  5.     // Listener is ignored and GC-d
  6.   }
  7. }

Replacing weak reference with a soft one doesn’t improve the situation, just delays the inevitable a bit further. Both are useful for caches, where objects can be recreated at will, but not in this case where we have an externally created object.

So what do we do? What we’d like to do is have the Listener reference to depend on the class loader somehow. Unfortunately, to the best of my knowledge, there isn’t a ready-made solution for that, and there’s no way to achieve it with any combinations of weak references without causing problems.

What we’d like to have is an ability to add a strong reference to the class loader: in other words have it carry a custom property:

PLAIN TEXT
JAVA:
  1. void addListener(Listener l) {
  2.   ClassLoader cl = l.getClass().getClassLoader();
  3.   List lls = (List) cl.getProperty(“CoreListeners”);
  4.   if (lls == null) {
  5.     lls = new ArrayList();
  6.     cl.putProperty(“CoreListeners”, lls);
  7.   }
  8.   lls.add(l);
  9. }

That would work, wouldn’t it? Well, not quite. We also need to save a reference to the class loaders, so that we could later go over all of them. Here the WeakHashMap is useful:

PLAIN TEXT
JAVA:
  1. Map classLoaders = new WeakHashMap();
  2.  
  3. void addListener(Listener l) {
  4.    //…
  5.   classLoaders.put(cl, Boolean.TRUE);
  6. }

There’s not WeakHashSet in Java, so we’re just using a boolean flag as the value.

So this would probably work, but unfortunately class loaders don’t have a getProperty()/putProperty() API. However, it turns out that with a bit of a hack we can simulate it, by generating a unique class per class loader to hold the properties for us. Let’s see how it’s done!

We start with a little boilerplate:

PLAIN TEXT
JAVA:
  1. class ClassLoaderLocalMap {
  2.   private static Method defineMethod;
  3.   private static Method findLoadedClass;
  4.  
  5.   static {
  6.     try {
  7.       defineMethod = ClassLoader.class.getDeclaredMethod(
  8.          “defineClass”,
  9.           new Class[] {
  10.             String.class,
  11.             byte[].class,
  12.             int.class,
  13.             int.class });
  14.       defineMethod.setAccessible(true);
  15.  
  16.       findLoadedClass =
  17.         ClassLoader.class.getDeclaredMethod(
  18.           “findLoadedClass”,
  19.           new Class[] { String.class});
  20.       findLoadedClass.setAccessible(true);   
  21.     }
  22.     catch (NoSuchMethodException e) {
  23.       throw new RuntimeException(e);
  24.     }
  25.   } 
  26. }

This will give us access to ClassLoader protected methods defineClass() and findLoadedClass() later on. Now let’s setup the basic API:

PLAIN TEXT
JAVA:
  1. public static void put(
  2.   ClassLoader cl,
  3.   Object  key,
  4.   Object value) {
  5.   // Synchronizing over ClassLoader is safest
  6.   synchronized (cl) {
  7.     getLocalMap(cl).put(key, value);
  8.   }
  9. }
  10.  
  11. public static Object get(
  13.   Object key) {
  14.   // Synchronizing over ClassLoader is safest
  15.   synchronized (cl) {
  16.     return getLocalMap(cl).get(key);
  17.   }
  18. }

getLocalMap() method should return a map of entries associated with the class loader. How should that work?

Next we introduce a map from class loaders to unique holder class names. We also introduce a nextHolderName() method that generates unique names:

PLAIN TEXT
JAVA:
  1. private static final Map classLoaderToHolderClassName =
  2.   Collections.synchronizedMap(new WeakHashMap())
  3. private static volatile int counter = 1;
  4.  
  5. private static String nextHolderName() {
  6.   return “ClassLoaderLocalMapHolder$$GEN$$” + counter++;
  7. }

Finally we can implement the getLocalMap() method (to save space I removed all exception handling):

PLAIN TEXT
JAVA:
  1. private static Map getLocalMap(ClassLoader cl) {
  2.   String holderClassName =
  3.     (String) classLoaderToHolderClassName.get(cl);
  4.   if (holderClassName == null) {
  5.     holderClassName= nextHolderName();
  6.     classLoaderToHolderClassName.put(
  7.       cl, holderClassName);
  8.   }
  9.  
  10.   Class holderClass =
  11.     (Class) findLoadedClass.invoke(
  12.       cl,
  13.       new Object[] {propertiesClassName});
  14.  
  15.   if (holderClass  == null) {
  16.     byte[] classBytes =
  17.       buildHolderByteCode(holderClassName);
  18.  
  19.     holderClass = (Class) defineMethod.invoke(cl,
  20.         new Object[] {
  21.           holderClassName,
  22.           classBytes,
  23.           new Integer(0),
  24.           new Integer(classBytes.length)});
  25.   }
  26.  
  27.   return (Map) holderClass
  28.     .getDeclaredField(“localMap”).get(null);
  29. }

The last method to implement is buildHolderByteCode. It’s quite trivial and builds the following class renamed to the unique name:

PLAIN TEXT
JAVA:
  1. public class ClassLoaderLocalMapHolder$$GEN$$X {
  2.   public static final Map localMap = new HashMap();
  3. }

The code can be derived using ASMifier with just a little customization, you can look it up in the full source code.

Although we could now easily implement the original example it makes sense to do just a little bit extra effort and introduce a ClassLoaderLocal, with behavior similar to the ThreadLocal:

PLAIN TEXT
JAVA:
  1. public class ClassLoaderLocal {
  2.   private Object key = new Object();
  3.  
  4.   public Object get(ClassLoader cl) {
  5.     if (!ClassLoaderProperties.containsKey(cl, key))
  6.       return null;
  7.     return ClassLoaderProperties.get(cl, key);
  8.   }
  9.  
  10.   public void set(ClassLoader cl, Object value) {
  11.     ClassLoaderProperties.put(cl, key, value);
  12.   }
  13. }

So the original example now becomes:

PLAIN TEXT
JAVA:
  1. Map classLoaders = new WeakHashMap();
  2. ClassLoaderLocal cll = new ClassLoaderLocal();
  3.  
  4. void addListener(Listener l) {
  5.   ClassLoader cl = l.getClass().getClassLoader();
  6.   List lls = (List) cll.get(cl);
  7.   if (lls == null) {
  8.     lls = new ArrayList();
  9.     cll.set(cl, lls);
  10.   }
  11.   lls.add(l);
  12.  
  13.   classLoaders.put(cl, Boolean.TRUE);
  14. }

In this code if any listener comes from a freed class loader, then it will be GC-d from both Core.classLoaders and ClassLoaderProperties.classLoaderToHolderClassName, as both are WeakHashMaps and there are no strong references to the class loaders. The generated ClassLoaderLocalMapHolder$$GEN$$X class will also be GC-d along with the class loader, so we have effectively eliminated a class loader leak without explicit cleanup calls from the user.

I hope this code will be useful for someone. I cannot give any guarantees whether it will work or not and it’s clearly a hack (though a solid hack). Please use it if you actually understand what is happening.

If you see a bug in the code or have a good suggestion, please be sure to comment. There could be a free JavaRebel license in it for you :)

Full source code: ClassLoaderLocalMap.java, ClassLoaderLocal.java.

]]> http://dow.ngra.de/2009/06/15/classloaderlocal-how-to-avoid-classloader-leaks-on-application-redeploy/feed/ 26 The Ultimate Java Puzzler http://dow.ngra.de/2009/02/16/the-ultimate-java-puzzler/ http://dow.ngra.de/2009/02/16/the-ultimate-java-puzzler/#comments Mon, 16 Feb 2009 15:01:08 +0000 Jevgeni Kabanov http://dow.ngra.de/?p=627 Why is this particular one the ultimate? Two reasons:

  • It’s at the very core of the Java language, not some obscure piece of API.
  • It melted my brain when I hit it.

UPDATE 2: If you want to test yourself before reading the post take this test. Results are not saved (it’s a paid feature apparently and I just don’t care enough), but you can post them in the comments.

Let’s start by setting up the puzzler environment. We’ll have three classes in two packages. Classes C1 and C2 will be in package p1:

PLAIN TEXT
JAVA:
  1. package p1;
  2. public class C1 {
  3.   public int m() {return 1;}
  4. }
  5. public class C2 extends C1 {
  6.   public int m() {return 2;}
  7. }

Class C3 will be in a separate package p2:

PLAIN TEXT
JAVA:
  1. package p2;
  2. public class C3 extends p1.C2 {
  3.   public int m() {return 3;}
  4. }

We will also have the test class p1.Main with the following main method:

PLAIN TEXT
JAVA:
  1. public static void main(String[] args) {
  2.   C1 c = new p2.C3();
  3.   System.out.println(c.m());
  4. }

Note that we’re calling the method of C1 on an instance of C3. The output for this example is “3″ as you’d expect. Now let’s change the m() visibility in all three classes to default:

PLAIN TEXT
JAVA:
  1. public class C1 {
  2.   /*default*/ int m() {return 1;}
  3. }
  4. public class C2 extends C1 {
  5.   /*default*/ int m() {return 2;}
  6. }
  7. public class C3 extends p1.C2 {
  8.   /*default*/ int m() {return 3;}
  9. }

The output will now be “2″!

Why is that? The Main class that invokes the method does not see the m() method in the C3 class, it being in a separate package. As far as it cares the chain ends with C2. But as C2 is in the same package it overrides the m() method in C1. This does not seem too intuitive, but that’s the way it is.

Now let’s try something different, let’s change the modifier of C3.m() back to public. What will that do?

PLAIN TEXT
JAVA:
  1. public class C1 {
  2.   /*default*/ int m() {return 1;}
  3. }
  4. public class C2 extends C1 {
  5.   /*default*/ int m() {return 2;}
  6. }
  7. public class C3 extends p1.C2 {
  8.   public int m() {return 3;}
  9. }

Now Main can clearly see the C3.m() method. But amazingly enough output is still “2″!

Apparently C3.m() is not considered to override C2.m() at all. One way to think about it is overriding methods should have access to the super methods (via super.m()). However in this case C3.m() wouldn’t have access to its super method, as it it not visible to it, being in another package. Therefore C3 is considered to be in a completely different invocation chain from C1 and C2. Were we to call C3.m() directly from Main the output would actually be “3″.

Now let’s look at one last example. Protected is an interesting visibility. It behaves like default for members in the same package and like public for subclasses. What will happen if we change all of the visibilities to protected?

PLAIN TEXT
JAVA:
  1. public class C1 {
  2.   protected int m() {return 1;}
  3. }
  4. public class C2 extends C1 {
  5.   protected int m() {return 2;}
  6. }
  7. public class C3 extends p1.C2 {
  8.   protected int m() {return 3;}
  9. }

My reasoning goes like this: as Main is not a subclass of any classes protected should behave as default in this case and output should be “2″. However that is not the case. The crucial thing is that C3.m() has access to super.m() and thus the actual output will be “3″.

Personally, when I first encountered this accessibility issue I got thoroughly confused and couldn’t get it until I did all of this examples through. The intuition I got from this is that if and only if you can access super.m() the subclass is a part of the invocation chain.

UPDATE: Apparently even though the whole thing is obvious to anyone, the intuition I came up with was wrong. A mysterious commenter know only as “C” has provided the following example:

PLAIN TEXT
JAVA:
  1. public class C1 {
  2.   /*default*/ int m() {return 1;}
  3. }
  4. public class C2 extends C1 {
  5.   /*default*/ int m() {return 2;}
  6. }
  7. public class C3 extends p1.C2 {
  8.   /*default*/ int m() {return 3;}
  9. }
  10. public class C4 extends p2.C3 {
  11.   /*default*/ int m() {return 4;}
  12. }

Note that C4 is in the package p1. If we now change the Main code as follows:

PLAIN TEXT
JAVA:
  1. public static void main(String[] args) {
  2.   C1 c = new C4();
  3.   System.out.println(c.m());
  4. }

Then it will output “4″. However super.m() is not accessible from C4 and putting @Override on the C4.m() method will stop the code from compiling. At the same time if we change the main method to:

PLAIN TEXT
JAVA:
  1. public static void main(String[] args) {
  2.   p2.C3 c = new C4();
  3.   System.out.println(c.m());
  4. }

The output will be “3″. This means that C4.m() overrides C2.m() and C1.m(), but not C3.m(). This also makes the issue even more confusing, and the amended intuition is that a method in a subclass overrides a method in a superclass if and only if the method in the superclass is accessible from the subclass. Here superclass can be any ancestor, not necessarily the direct parent and the relation has to be transitive.

For the kicker try reading all of this out from the JVM specification that selects the method to be invoked:


Let C be the class of objectref. The actual method to be invoked is selected by the following lookup procedure:

  • If C contains a declaration for an instance method with the same name and descriptor as the resolved method, and the resolved method is accessible from C, then this is the method to be invoked, and the lookup procedure terminates.
  • Otherwise, if C has a superclass, this same lookup procedure is performed recursively using the direct superclass of C; the method to be invoked is the result of the recursive invocation of this lookup procedure.
  • Otherwise, an AbstractMethodError is raised.

]]> http://dow.ngra.de/2009/02/16/the-ultimate-java-puzzler/feed/ 54 Correcting the Billion Dollar Mistake http://dow.ngra.de/2009/02/01/correcting-the-billion-dollar-mistake/ http://dow.ngra.de/2009/02/01/correcting-the-billion-dollar-mistake/#comments Sun, 01 Feb 2009 20:04:14 +0000 Jevgeni Kabanov http://dow.ngra.de/?p=628 Last week I visited Stockholm to speak at the JFokus 2009. The event was quite spectacular, but for me the most interesting part occurred on the evening before the conference. I was sitting at the speaker’s dinner with Rickard Öberg, Kirk Pepperdine, Simon Ritter and a couple of others. For some reason or other I started talking to Simon about the problem that’s recently been on my mind. Perhaps it’s been eating me after legendary Tony Hoare said this:

I call it my billion-dollar mistake. It was the invention of the null reference in 1965. [...] This has led to innumerable errors, vulnerabilities, and system crashes, which have probably caused a billion dollars of pain and damage in the last forty years.

Now since I have a background in functional languages, I know that null references are not necessary. Null value is a member of all types, but some types don’t have or need a natural notion of absence. In fact in Haskell there is no such thing as an “absent” value generic to all types — types either have to declare it explicitly or wrap the underlying value into a Maybe type that has a dedicated Nothing value.

So e.g. the List type has a notion of an empty list encoded into the type:

// Roughly: a list is an empty list
// or a pair of a value and a list
data List a = Nil | Cons a (List a)

While the only way to encode an absent structure is to use Maybe:

// To build a car you need at most one engine
// and zero or more wheels :)
buildCar :: Maybe Engine -> [Wheel]  -> Car

So while I was discussing that with Simon, Kirk banged me on the head to get my attention (just joking, banging me on the head is nowhere near enough to get my attention). Apparently they were discussing more or less the same topic with Rickard, and he had a similar solution for Java. I assumed it was the widely discussed @NotNull annotation, but it was way cooler than that.

Basically in the beginning Rickard started by adding @NotNull support to Qi4J (BTW an amazing piece of engineering, definitely check it out), so that they could automatically insert runtime assertions for the parameters having the @NotNull annotations, something like that:

PLAIN TEXT
JAVA:
  1. class Test {
  2.   void transfer(
  3.     @NotNull Account from,
  4.     Account to,
  5.     double amount) {
  6.       //…
  7.   }
  8.  
  9.   public static void main(String[] args) {
  10.     // Succeeds
  11.     transfer(
  12.       new Account(“Bugs Bunny”), null, 1000000.0);
  13.     // Throws NullPointerException
  14.     transfer(
  15.       null, new Account(“Bugs Bunny”), 1000000.0);
  16.   }
  17. }

However soon after that they discovered that they were inserting @NotNull-s everywhere. So they reverted the notion and decided that they will generate the not null assertions for all parameters except the ones marked as @Optional. And this was the point when I thought to myself “Jevgeni, this is exactly what you were looking for!” So I turn to Rickard and I say “Rickard, this is exactly what I was looking for! This is amazing!” In fact this truly is amazing as it’s exactly captures the semantics of the Maybe type that I liked so much in the functional languages.

After that we go into a discussion whether or not it is possible to validate this assertion during compile-time. Opinions were mixed on this one, though I personally am convinced that it shouldn’t be too easy (your opinion on the topic is welcome, also it will make a great master’s thesis topic). Eventually I get an idea and I say “Rickard, I bet I could implement a JavaRebel plugin that would check this at runtime in about half an hour in 50 lines of code”. And of course Rickard goes “No way!”, so I’m challenged. Next day I sit down for half an hour, then for another half an hour (there was no internet!) and voila — I have a working JavaRebel plugin (in less that 50 lines of code) that will make your methods throw an exception if you try to pass a null reference to a parameter not marked as @Optional (the plugin code deserves another post altogether). Of course I have to show this to Rickard (and he goes “No way!”) and we agree to somehow join forces to promote this sane approach (starting with having a single namespace for the @Optional annotation).

So what do we get? The previous example now looks like this:

PLAIN TEXT
JAVA:
  1. @OptionalCheck
  2. class Test {
  3.   void transfer(
  4.     Account from,
  5.     @Optional Account to,
  6.     double amount) {
  7.       //…
  8.   }
  9.  
  10.   public static void main(String[] args) {
  11.     // Succeeds
  12.     transfer(
  13.       new Account(“Bugs Bunny”), null, 1000000.0);
  14.     // Throws NullPointerException
  15.     transfer(
  16.       null, new Account(“Bugs Bunny”), 1000000.0);
  17.   }
  18. }

As you can understand if we make all of the code without the @Optional annotation to throw a NullPointerException for nulls we’ll break a lot of existing code. Therefore at the moment you also have to annotate the class where you want to enable such semantics with @OptionalCheck.

That’s pretty much it — you can download the plugin right away and just drop it in the classpath when JavaRebel is enabled (you’ll need a 2.0 milestone as 1.x was missing the necessary APIs). At the moment both annotations are in the org.optionalalliance package, but when it changes all you have to do is Organize Imports, so I won’t sweat the naming too much. Please do let us know what do you think of the approach and feel free to advocate it further :)

Cheers,
Jevgeni Kabanov

P.S. The plugin along with the source is also avalable in our Maven repository:

]]> http://dow.ngra.de/2009/02/01/correcting-the-billion-dollar-mistake/feed/ 31 Announcing Squill: Not Another ORM http://dow.ngra.de/2008/12/09/announcing-squill-not-another-orm/ http://dow.ngra.de/2008/12/09/announcing-squill-not-another-orm/#comments Tue, 09 Dec 2008 13:24:22 +0000 Jevgeni Kabanov http://dow.ngra.de/?p=501 Remember that post about Typesafe DSLs that had a part one and no follow up? Well, meanwhile Juhan Aasaru and yours truly were joined by Michael Hunger of jexp.de and JEQUEL and together we have created the Squill project that came right out of the ideas in the paper we wrote with Rein Raudjärv. The announcement follows, enjoy!

It is with great pleasure that we announce the first release of Squill. Download it now or check out the quickstart guide, the step-by-step tutorial and the Devoxx presentation.

Squill is a slick internal DSL for writing SQL queries in pure Java. It uses the database metadata and generics to catch as many errors as possible during compilation and is almost completely typesafe.

At the same time it is designed to allow everything SQL allows you to do, exactly the way SQL is meant to do it. This means that you’re encouraged to select only the data you need and no hidden queries are generated for you, leaving you in full control of the query performance. Squill supports database-specific extensions, allowing you to both use advanced features and fully tweak your queries.

Squill also has special support for CRUD operations and table relations, adding some sugar over vanilla SQL. A typical Squill query looks like this:

PLAIN TEXT
JAVA:
  1. ComplaintTable c = new ComplaintTable();
  2.    
  3.   for (Tuple2<String, Integer> tuple2:
  4.     squill
  5.       .from(c, c.customer)
  6.       .where(
  7.         gt(c.customer.isActive, 0),
  8.         notNull(c.percentSolved), 
  9.         notNull(c.refoundSum))
  10.       .orderBy(desc(c.customer.id))
  11.       .selectList(
  12.         c.customer.lastName,
  13.         c.percentSolved)) {
  14.     System.out.println(
  15.       “Customer “ + tuple2.v1 + ” has a complaint solved “ + tuple2.v2 + “%”);
  16.   }

Squill is a very young project and you can follow (and help) its development by joining the user or developer mailing lists.

]]> http://dow.ngra.de/2008/12/09/announcing-squill-not-another-orm/feed/ 2 The Performance Cutoff http://dow.ngra.de/2008/11/10/the-performance-cutoff/ http://dow.ngra.de/2008/11/10/the-performance-cutoff/#comments Mon, 10 Nov 2008 09:36:31 +0000 Jevgeni Kabanov http://dow.ngra.de/?p=456 A few days ago I had a small epiphany on a simple yet important issue. I was trying to squeeze those last few percents of performance out of JavaRebel and it came to the point where I started optimizing individual hotspots method-by-method.

One of the most common ways to improve execution time of a specific method is the cutoff. For many complicated enough methods there are some inputs for which you can return immediately and cut off the main execution path. To explain let me use the following example:

PLAIN TEXT
JAVA:
  1. Output doSomething(Input input) {
  2.   // Do something for 200 ms
  3.   return output;
  4. }

This method could do anything as long as the complexity is constant, we only care how much time it takes on average. I could also have taken some algorithm (e.g. I started with matrix multiplication), but then we’d have to bring in complexity estimates and I’d like to keep it simple for now.

Let’s assume that for some inputs we could calculate the output in 50ms instead of 200ms. We introduce a check and the code is now:

PLAIN TEXT
JAVA:
  1. Output doSomething(Input input) {
  2.   if (isSimpleInput(input)) {
  3.     // Do something for 50 ms
  4.     return output;
  5.   }
  6.  
  7.   // Do something for 200 ms
  8.   return output;
  9. }

This is a common way to optimize the method execution time. However the question is if this actually optimized anything? It seems like a stupid question, but let’s estimate the new average execution time.

To do that we need to know two things. The time t it takes to do the isSimpleInput() check and the proportion p of method calls that are “simple”. Let’s assume that t is 20 ms and 10% of the calls are simple. The average execution time can then be calculated using the expected value formula (EV = p * v1 + (1 – p) * v2):
EV = 0.1 * (20 + 50) + (1 – 0.1) * (20 + 200)
= 20 + 0.1 * 50 + 0.9 * 200
= 20 + 5 + 180 = 205

This calculation shows that with this values we have actually increased the average method execution time by 5 ms. Note that we are still winning every time the cutoff occurs (20 + 50 << 200), but we are losing on the average.

The same equation can also be applied to measuring several checks or measuring execution time that is dependent on the input, however it gets increasingly more complicated. For me the main value of this is realizing that every cutoff has an inherent cost which I will now pay in mind.

Before you think this is all just math, the epiphany that came to me was not to add a check, it was to remove one. I realized that a particular check I’m making is rare and expensive enough to directly influence the method execution time and lo and behold, removing it gave a 30% improvement for the benchmark I was using.

]]> http://dow.ngra.de/2008/11/10/the-performance-cutoff/feed/ 0 Script kiddies have awesome tools http://dow.ngra.de/2008/11/04/script-kiddies-have-awesome-tools/ http://dow.ngra.de/2008/11/04/script-kiddies-have-awesome-tools/#comments Tue, 04 Nov 2008 16:05:08 +0000 Toomas Römer http://dow.ngra.de/?p=413 About 10 years ago a friend of mine showed me an exploit. It was written in C and it tried to spawn a shell at a remote host. It seemed pretty cool. I did not understand the code but the mere idea that almost anybody equipped with a script like that could deface a webpage seemed scary.

You did have to compile the c file, have the right devel packages installed and use the correct flags. And then you had to figure out how to use it. A 14yr old could do it.

Today I spent many hours grepping logs, checking the filesystem for new/changed files to figure out how an old Wordpress instance was hacked and what had the hacker done there.

Going through the changed files I stumbled upon a php file which had some code prepended. The script had a very long line that started like this:

PLAIN TEXT
PHP:
  1. eval(gzinflate(base64_decode(‘FJ3HcqPsFkUf……..

Ok, lets check what does the script do. Lets assign the long string to a variable and base64 decode it and inflate the compression.

PLAIN TEXT
PHP:
  1. $script =  base64_decode($script);
  2. $script = gzinflate($script);
  3. echo $script;

The output was not what I expected.

PLAIN TEXT
PHP:
  1. eval(gzinflate(base64_decode(‘FJ3HjqPcGkUf57……..

The strings looked similar and I was already looking for an error in my code. Nope, code is correct. There is a slight change in the string. It seems it was compressed and encoded couple of times. Wow, it means I can have many evals inside evals. Fun!

PLAIN TEXT
PHP:
  1. do {
  2.   // extract the first 28 characters
  3.   // the eval(gzinflate(base64_decode part
  4.   $start =  substr($string, 0, 28);
  5.   // remove the first 30 chars, the eval(gzinflate(base64_decode(‘ part
  6.   $string = substr($string, 30);
  7.   // remove the last )));
  8.   $string = substr($string, 0, strlen($string)-4);
  9.  
  10.   $string =  base64_decode($string);
  11.   $string = gzinflate($string);
  12.   echo “Iteration:”.$i++.“\n“;
  13.   // iterate as long as we get a eval(gzinflat start
  14. } while ($start == “eval(gzinflate(base64_decode”);

After 11 iterations I got the code. Kind of reminded me a challenge that was posted to a mailing list and the question was what was the output of the program. That time it was more difficult: base64 encoded perl, that outputted base64 encoded bytecode, that outputted Java source file with a byte array that was byte code for the class file of the solution.

Anyways the 11 iterations gave me this (shot is made from my home computer).
Wow!

Lets see the functionality that it has to offer:

  • Full blown file manager
  • Quick menu for
    • Finding all suid files
    • Finding all sgid files
    • Finding all htaccess files
    • Finding all writeable folders
    • …
  • Interface for the UNIX tool find
  • Input field for executing commands as webserver user
  • Tools for installing a backdoor
    • Perl/C flavoured programs that are downloaded from a Singapore server
    • Compiled/Interpreted – depending what is available
  • Processes viewer
  • FTP brute force cracker using users from /etc/passwd
  • System info (CPU, Memory, installed binaries, passwd file, configuration files)
  • SQL dump utility
  • Interface for executing PHP code
  • Self removal
  • Adding a password for the script
  • Fancy design!

I’m just amazed. This is way too eazy. So this is how it works:

  • Lets scan the internet for Wordpress installation (automated)
  • Look for vulnerable versions (automated)
  • Exploit (in this case themes were filled with hidden links – semi automated)
  • PROFIT! (automated)

How to avoid being hacked:

  • Keep an eye on your Wordpress installations
  • Subscribe to WordPress release emails/RSS and upgrade when needed
  • Monitor for changed files (for example fcheck)
  • Run Apache in chroot to minimize the available software for the Apache user
  • Any other ideas?

PS. The script is 2500 lines of code, supports Windows and Linux and looks great :)

]]> http://dow.ngra.de/2008/11/04/script-kiddies-have-awesome-tools/feed/ 28 When protected isn’t protected http://dow.ngra.de/2008/11/03/when-protected-isnt-protected/ http://dow.ngra.de/2008/11/03/when-protected-isnt-protected/#comments Mon, 03 Nov 2008 10:41:45 +0000 Jevgeni Kabanov http://dow.ngra.de/?p=360 For all those spec believers out there: what would you think if I told you that all of the JVM implementations consistently violate the spec for at least one particular instance?

Our story begins on a cold and lonely evening when I discover a bug in JavaRebel in conjunction with Xerces XMLEntityManager. The problem was with the following method:

PLAIN TEXT
JAVA:
  1. public String[] getRecognizedProperties() {
  2.   return (String[])(RPROPERTIES.clone());
  3. }

However when I looked at the class bytecode I saw a different picture:

PLAIN TEXT
JAVA:
  1. {
  2.   GETSTATIC XMLEntityManager.RPROPERTIES : String[]
  3.   INVOKEVIRTUAL Object.clone() : Object[]
  4.   CHECKCAST String[]
  5.   ARETURN
  6. }

For those of you who do not spend hours staring at the Java bytecode, it does the following:

  1. Retrieve the RPROPERTIES static field value (which is of type String[]) and put it on the top of the stack.
  2. Call Object.clone() using the top of the stack as target object (in our case the array). The target is popped off the stack and the result is placed on top of the stack instead of it.
  3. Cast the top of the stack to a String[]
  4. Return the top of the stack as method result.

Whooph. Now that is all fine except for one tiny thing. This is not legal. Object.clone() is a protected method and the JVM spec (see 2.7.4, INVOKEVIRTUAL) allows calls to protected method only if the following is true:

  1. The accessing class is a subclass of the target class or is the same class (check, XMLEntityManager is a subclass of Object, so’s everything else).
  2. The class of the target object (the actual runtime class of the object, not the target class we use in the signature) is either a subclass of the accessing class or is the same class.

The second check may sound crazy, but it restricts the calls to protected methods only to the classes inheriting from the target class, as you would expect. It can also only be enforced during runtime and is the cause of the IllegalAccessError you may see once in a while.

And obviously in our case String[] that is the actual type of the target object is in no way a subclass of the XMLEntityManager. This call should have cause an error in the JVM, but for some inexplicable reason it doesn’t. Further tests show that this is only limited (AFAIK) to the Object.clone() call, which is public as far as JVM cares.

A little googling turned up two JVM bugs that were discussing the same issue (bug 1, bug 2). Turns out the Sun javac compiler has been emitting wrong code at least until 1.4.2 and the JVMs have accommodated accordingly right until today.

Does this have any practical importance? Not for most of you, but if you handle Java bytecode in any way be prepared to handle this exception as well. To me it proves that there is more to Java than just the spec.

]]> http://dow.ngra.de/2008/11/03/when-protected-isnt-protected/feed/ 8 When System.currentTimeMillis() is too slow… http://dow.ngra.de/2008/10/27/when-systemcurrenttimemillis-is-too-slow/ http://dow.ngra.de/2008/10/27/when-systemcurrenttimemillis-is-too-slow/#comments Mon, 27 Oct 2008 11:09:14 +0000 Jevgeni Kabanov http://dow.ngra.de/?p=347 At the moment I am working on reducing the performance overhead of JavaRebel. Once I got rid of all the obvious bottlenecks and optimizations I fired up the profiler and started searching and destroying ad-hoc hotspots. After some time I got to the point, when the bottleneck was in System.currentTimeMillis().

This deserves some explanation. Since JavaRebel reloads code after the class changes, it needs to poll the actual class file for changes. The way we do it is quite lazy and the class itself causes the resource to be polled. However often there will be a lot of polling involved (more or less any method call on the class will cause a check), so we need to filter those calls. What we did is to check if some amount of time (e.g. 500 ms) has passed since the last check:

PLAIN TEXT
JAVA:
  1. if (lastCheck + 500 > System.currentTimeMillis())
  2.   return;
  3. lastCheck = System.currentTimeMillis();
  4. // Resource check code

After some optimization everything else got fast enough that this call started to weigh the performance down. System.currentTimeMillis() causes a system call and often a hardware poll, which is comparatively expensive. I needed some way to eliminate it.

After some thought I decided to cache the call. It may sound crazy to cache the current time, but I only needed the resolution of about half a second, so it was OK for me if the time was lagging behind about that much. What I did is put up a separate thread that would cache the current time and update it every half a sec.

PLAIN TEXT
JAVA:
  1. public class TimeCacheThread extends Thread {
  2.   public static volatile long currentTimeMillis =
  3.     System.currentTimeMillis();
  4.  
  5.   public TimeCacheThread() {
  6.     setDaemon(true);
  7.   }
  8.  
  9.   static {
  10.     new TimeCacheThread().start();
  11.   }
  12.  
  13.   public void run() {   
  14.     while (true) {
  15.       currentTimeMillis = System.currentTimeMillis();
  16.      
  17.       try {
  18.         Thread.sleep(500);
  19.       } catch (InterruptedException e) {
  20.         throw new RuntimeException(e);
  21.       }
  22.     }
  23.   }
  24. }

Then the check became:

PLAIN TEXT
JAVA:
  1. if (lastCheck + 500 > TimeCacheThread.currentTimeMillis)
  2.   return;
  3. lastCheck = TimeCacheThread.currentTimeMillis;
  4. // Resource check code

This check is amazingly fast, because instead of doing a system call we are just making a memory read.

[edit] The original code did not have the field declared volatile with the following explanation:

You may also notice the complete lack of any synchronization (or volatile keyword) on the currentTimeMillis static field. Since the type is primitive we can only deal with stale data here and I cared about performance more than I cared about missing a couple of checks.

There is a follow-up post (What do we really know about non-blocking concurrency in Java?) that explains some more of the reasoning behind initially omitting the volatile and then still putting it in. [/edit]

However once I got that far I decided I could do even better. The currentTimeMillis is of type long and we are still doing some arithmetic on it. In fact the only thing we want to know is if a quantum of time has passed since the last check. And we can do that by using a very simple heartbeat counter:

PLAIN TEXT
JAVA:
  1. public class HeartBeatThread extends Thread { 
  2.   public static volatile int counter = 0; 
  3.  
  4.   public HeartBeatThread() {
  5.     setDaemon(true);
  6.   }
  7.  
  8.   static {
  9.     new HeartBeatThread().start();
  10.   }
  11.  
  12.   public void run() {   
  13.     while (true) {     
  14.       try {
  15.         Thread.sleep(500);
  16.       } catch (InterruptedException e) {
  17.         throw new RuntimeException(e);
  18.       }
  19.      
  20.       counter++;
  21.     }
  22.   }
  23. }

Now the checking code becomes just:

PLAIN TEXT
JAVA:
  1. if (counter == HeartBeatThread.counter)
  2.   return;
  3. counter = HeartBeatThread.counter;

The counter field is now of int type and we only do an equals check on a DWORD, which should translate directly to 3 very fast hardware instructions (two memory reads and one conditional jump). Hopefully if you ever find yourself polling something a lot this post will come of help. As for me, it improved JavaRebel performance by 70% on some benchmarks.

]]> http://dow.ngra.de/2008/10/27/when-systemcurrenttimemillis-is-too-slow/feed/ 54 Case study: Is PHP embarrasingly slower than Java? http://dow.ngra.de/2008/08/04/optimizing-ip2c-php-implementation/ http://dow.ngra.de/2008/08/04/optimizing-ip2c-php-implementation/#comments Mon, 04 Aug 2008 08:50:03 +0000 Toomas Römer http://dow.ngra.de/?p=140 IP2C is a small library that provides IP to country resolution. It uses the free ip-to-country database. IP2C takes the database CSV file that is about 4mb and converts it into a ~600kb binary format and provides PHP and Java frontend to query the database.

The library is great, easy to convert an ip to a country and when using the country flags from it’s side project you could spice up your statistics with the country information. This a lot faster than using reverse DNS lookup.

The problem. The PHP implementation is a lot slower. Embarrassingly slower. Without any caching the Java version is able to do ~6000 queries per second. The PHP counterpart can push through ~850 queries. The implementations are the same. The stats provided by the author of the library are 8000 vs 1200. So about the same as my measurements.

I like PHP, I don’t use it that much anymore but I still care when I see such embarrassing numbers. I took the implementation and started profiling it. Spent the night running different tests and trying to optimize.

General outline of the algorithm is as follows. We take the dotted string IP and convert it to an IPv4 Internet network address (e.g. 69.55.232.153 becomes 1161291929). The DB holds sorted ranges of these addresses. A binary search will happen on these addresses and we have a country for the ip. Take a look at the implementation.


Lets see where the vanilla version of IP2C spends its time at. The results are based on 1000 iterations with Xdebug enabled and visualized by KCacheGrind. It processed about 210 IP addresses during this time.

IO part is surprisingly low. The internal fseek, fread constitute to 2% of the execution time. On the other hand the user level fseek which is just a wrapper alone uses 5%. readShort and readInt take 20% of the execution time.

PLAIN TEXT
PHP:
  1. function readShort() {
  2.     $a = unpack(‘n’, fread($this->m_file, 2));
  3.     return $a[1];}
  4.  
  5. function readInt() {
  6.     $a =unpack(‘N’, fread($this->m_file, 4));
  7.     return $a[1];}
  8.  
  9. function seek($offset){
  10.     fseek($this->m_file, $offset);}

Functions calls are expensive. Lets eliminate them. readInt, readShort, fseek are now inlined. Recursion changed to iteration (e.g. 14 000 less function calls). Able to process 400 queries per second compared to the previous 210.

We see that the latest profiling results have twice the number of freads and unpacks than fseeks. It seems that fseek is used to seek out the right position, read two numbers with unpacking them. The implementation confirms that. Luckily we could just read once (2 bytes more) and unpack once (2 unpackings with one invocation).

PLAIN TEXT
PHP:
  1. $a =unpack(‘N’, fread($this->m_file, 4));
  2. $np[‘ip’] = $a[1];
  3.                            
  4. $a =unpack(‘n’, fread($this->m_file, 2));
  5. $np[‘key’] = $a[1];
  6.  
  7. // this can be changed to
  8. $np =unpack(‘Nip/nkey’, fread($this->m_file, 6));

How does this version stack up to the Java version? Lets disable profiling and run 100 000 iterations. Vanilla version processes ~850 IPs, when functions are inlined the number is around 1400. Java version can still do 6000.

Lets try caching. Peeking at the Java implementation shows that Java caching version (whopping 141 242 IPs per second – yup 141k) uses just a byte[] array and makes lookups from there instead of seeking and reading from file. Easy, lets do the same in PHP.

We read everything into a string and instead of fread with access the string elements with the offset. For fseek with just set the offset. We are using 600kb more memory but can increase the throughput to ~2800.

As it seems I’ve just wasted a night, I just should have checked the Computer Language Benchmarks. PHP in the sense of execution speed is uncomparable to Java.

The upside, we can still take the library, eliminate recursions, double unpacks and add caching. A small gain is still a gain.

]]> http://dow.ngra.de/2008/08/04/optimizing-ip2c-php-implementation/feed/ 34 Typesafe DSLs in Java: Part 1 — Typesafe Bytecode http://dow.ngra.de/2008/03/24/typesafe-dsls-in-java-part-1-typesafe-bytecode/ http://dow.ngra.de/2008/03/24/typesafe-dsls-in-java-part-1-typesafe-bytecode/#comments Mon, 24 Mar 2008 08:25:29 +0000 Jevgeni Kabanov http://dow.ngra.de/2008/03/24/typesafe-dsls-in-java-part-1-typesafe-bytecode/ Fluent Interface. However most of them utilize a lot of strings and untyped behavior to make the interface fluent enough. It turns out that using Java 5 and a bag of tricks we can have the compiler to check a lot more. In this post we'll check out how to write Java bytecode using ASM in a typesafe way.]]>

Domain Specific Languages (DSLs) have been brought to Java under the name of Fluent Interface. However most of them utilize a lot of strings and untyped behavior to make the interface fluent enough. It turns out that using Java 5 and a bag of tricks we can have the compiler to check a lot more. In this post we'll check out how to write Java bytecode using ASM in a typesafe way.

Domain Specific Languages

Domain specific language usually refers to a small sublangauge that has very little overhead when expressing domain specific data and behaviour. DSL is a broad term and can refer both to a fully implemented language and a specialized API that looks like a sublanguage, but still written using some general-purpose language. Such DSLs in the latter meaning have been introduced (perhaps independently?) by both the functional and dynamic languages community. Both these communities (and especially functional) took advantage of function composition and operator overloading to build combinator-based languages that look nothing like the original. The functional communities also strongly support the notion of type-safety, therefore the DSLs they create are usually statically typed.

In Java the DSLs are also becoming more popular. The first examples like jMock and Hibernate Criteria were coined as Fluent Interface. However most of these DSLs are not typesafe and will allow wrong statements to be compiled. In this post we introduce some patterns that help making Java DSLs safer.

Java bytecode engineering

Java bytecode is a relatively simple stack-based language. All the code is contained in methods, class structure is pretty much preserved from source (fields and methods, both can be static, constructors and static initializers are turned into methods named "<init>" and "<clinit>" correspondingly). Inside the method we have the variables, referred by an index with 0 being "this", 1 being the first parameter and so on, local variables starting after parameters. We can load and store variables. We also have the stack, where we can push, pop and duplicate values. We have a number of basic operations on the stack (a la add and multiply) as well as method invocation. When invoking the methods parameters are gathered from the stack with last parameter being on top of the stack. Finally we have some flow control, namely conditional (and unconditional) jumps. That pretty much sums it up.

One of the best libraries for working with Java bytecode is ASM. It provides both a lightweight fast visitor-based interface and a more comfortable tree-based object-oriented interface. Unfortunately both of them (and especially visitor-based one) are completely untyped and debugging the wrong bytecode created using them is a huge pain in the neck. 'Nuff to say that Java bytecode verifier will tell you there's a problem, but will not tell you where exactly it is.

Further we will examine in details the bytecode of the following example:

PLAIN TEXT
JAVA:
  1. public class HelloWorld {
  2.   public static void main(String[] args) {
  3.     System.out.println("Hello, World!");
  4.   }
  5. }

If it looks simple to you -- it should. However the Java bytecode is a bit more complex, we have a default constructor that compiler will generate for you and we can do only one operation per line (more correctly per instruction). Next you see the HelloWorld example in Java bytecode (you can get a similar using "javah -c ClassName", although Bytecode outline Eclipse plugin is more comfortable).

PLAIN TEXT
JAVA:
  1. public class HelloWorld {
  2.   // Default constructor
  3.   public <init>()V
  4.     ALOAD 0
  5.     INVOKESPECIAL java/lang/Object.<init>()V
  6.     RETURN
  7.  
  8.   // System.out.println("Hello, World!");
  9.   public static main([Ljava/lang/String;)V
  10.     GETSTATIC java/lang/System.out : Ljava/io/PrintStream;
  11.     LDC "Hello, World!"
  12.     INVOKEVIRTUAL java/io/PrintStream.println(Ljava/lang/String;)V
  13.     RETURN
  14. }

Here's a summary of the instructions in the example:

  • ALOAD loads the local variables to the top of the stack, index 0 is "this"
  • INVOKESPECIAL in this case invokes super method, it also consumes and Object from the stack.
  • GETSTATIC retrieves the value of the static fields and puts it on the stack (in this case a PrintStream)
  • LDC pushes a constant value to the stack
  • INVOKEVIRTUAL invokes a usual (virtual) method, consuming the parameters and returning the result
  • RETURN returns from the method

If we want to write the same code from Java using ASM it would be almost the same:

PLAIN TEXT
JAVA:
  1. ClassWriter cw = new ClassWriter(0);
  2. MethodVisitor mv;
  3.  
  4. cw.visit(V1_6, ACC_PUBLIC + ACC_SUPER, "HelloWorld", null, "java/lang/Object", null);
  5.  
  6. {
  7. mv = cw.visitMethod(ACC_PUBLIC, "<init>", "()V", null, null);
  8. mv.visitCode();
  9. mv.visitVarInsn(ALOAD, 0);
  10. mv.visitMethodInsn(INVOKESPECIAL, "java/lang/Object", "<init>", "()V");
  11. mv.visitInsn(RETURN);
  12. mv.visitMaxs(1, 1);
  13. mv.visitEnd();
  14. }
  15. {
  16. mv = cw.visitMethod(ACC_PUBLIC + ACC_STATIC, "main", "([Ljava/lang/String;)V", null, null);
  17. mv.visitCode();
  18. mv.visitFieldInsn(GETSTATIC, "java/lang/System", "out", "Ljava/io/PrintStream;");
  19. mv.visitLdcInsn("Hello, World!");
  20. mv.visitMethodInsn(INVOKEVIRTUAL, "java/io/PrintStream", "println", "(Ljava/lang/String;)V");
  21. mv.visitInsn(RETURN);
  22. mv.visitMaxs(2, 1);
  23. mv.visitEnd();
  24. }
  25. cw.visitEnd();

Typesafe ASM frontend

Take a look at the mv.visitMethodInsn(INVOKEVIRTUAL, "java/io/PrintStream", "println", "(Ljava/lang/String;)V");. It requires two elements on the stack -- an instance of PrintStream on which the method is called and a String. What we want to do is for compiler to issue an error when the stack in fact does not contain such elements. We propose the following DSL as the basis:

PLAIN TEXT
JAVA:
  1. new ClassBuilder(cw, V1_4, ACC_PUBLIC, "HelloWorld", "java/lang/Object", null)   
  2.     .beginStaticMethod(ACC_PUBLIC | ACC_STATIC, "main", void.class, String[].class)
  3.     .getStatic(System.class, "out", PrintStream.class)
  4.     .push("Hello, World!")
  5.      //Here String.class refers to the type of the first parameter
  6.     .invokeVirtualVoid(INVOKEVIRTUAL, PrintStream.class, "println", String.class)
  7.     .returnVoid()
  8.     .endMethod();

Note that now all the types are written as class literals instead of strings. In Java 5, class literals are generified to Class<C>, where C refers to the actual underlying type. This already allows for less mistakes, since the classes are no longer written as strings. However we want more!

The first thing we want to do is to track the stack size. We wouldn't want to allow to pop() off an empty stack. And we would want to ensure that when you invoke a method all the parameters are there. To do that we introduce a simple idiom:

The type you return from your DSL method should allow exactly those operations that are possible with the current state.

What does it mean in our context? It means that when we build a method we don't have just a MethodBuilder class, but MethodBuilderS0, MethodBuilderS1, MethodBuilderS2 and so on, where the S* refers to the stack size. And each of them has only the methods possible with the current stack size. Now the method for pop() will not even show up in autocompletion if the stack is not large enough.

We can apply the same trick to tracking variables, allowing to add only one variable at a time and providing methods loadVar*/stroreVar*. This way our class becomes MethodBuilderS*V* where S stands for stack size and V stands for variable count.

Of course we want more. We want to actually track the stack and variable types. To do that we introduce the following idiom:

The DSL type should be parametrized by all of the accumulated types as should be it's methods.

What does that mean? Let's return to our example and see what happens if I replace the string "Hello, World!" with an integer 11?

asm-error-1.PNG

As we can see from the screenshot the compiler inferred that the argument invokeVirtualVoid() will receive is an Integer (generics do not work with primitive types), whereas we have stated that it should receive a String as a parameter. Another thing we can see is that invokeVirtualVoid() method belongs to the MethodBuilderS2V1<Self, PrintStream, Integer, String[]> class.

As we know the name MethodBuilderS2V1 refers to a class that tracks two stack slots and one variable. The types are encoded accordingly in <Self, PrintStream, Integer, String[]> -- the two stack types are PrintStream, Integer and the one variable type us String[]. Self is not important for now.

To understand how the tracking is implemented let's see the push() and pop() implementation (the rest are omitted).

PLAIN TEXT
JAVA:
  1. public class MethodBuilderS2V1 <O, S0, S1, V0> {
  2.   public <S> MethodBuilderS3V1<O, S0, S1, S, V0> push(S value) {
  3.     mv.visitLdcInsn(value);
  4.     return new MethodBuilderS3V1<O, S0, S1, S, V0>(cb, mv);
  5.   }
  6.  
  7.   public MethodBuilderS1V1<O, S0, V0> pop() {
  8.     mv.visitInsn(POP);
  9.     return new MethodBuilderS1V1<O, S0, V0>(cb, mv);
  10.   }

So as you can see the class itself is parametrized by all of the accumulated type parameters (which were one by one added by previously invoked DSL methods). We can see that push() returns an instance of MethodBuilderS3V1. That class tracks exactly one more stack slot, which is passed to it as inferred from the argument (S). pop() on the other hand just discards one stack slot and returns an instance of MethodBuilderS1V1.

So what does the invokeVirtualVoid() look like? First of all, it needs to consume two stack variables, therefore we need at least MethodBuilderS2V* class to call it. Therefore all classes with less stack variables will not have this method. Secondly we need to check that the types in the stack are fitting, an must also allow for some leniency due to subtyping:

PLAIN TEXT
JAVA:
  1. public class MethodBuilderS2V1<O, S0, S1, V0> {
  2.     public MethodBuilderS0V1<O, V0> invokeVirtualVoid(int kind, Class<? super S0> owner, String name, Class<? super S1> parameter1) {
  3.     mv.visitMethodInsn(
  4.         kind,
  5.         owner.getName().replace('.', '/'),
  6.         name,
  7.         Type.getMethodDescriptor(Type.VOID_TYPE, new Type[] {Type.getType(parameter1)}));
  8.    
  9.     return new MethodBuilderS0V1<O, V0>(cb, mv);
  10.   }
  11. }

As you can see it indeed consumes two stack values returning MethodBuilderS0V1. If our method would also return a result we would need to take the result type as well, and return a class with stack depth only one less. The expression "? super S0" means that we require the actual parameter type to be a superclass of the stack type.

Problem 1: The world isn't perfect

One of the problems with the DSL we proposed here is that it assumes all the types exist. However it is very often the case that some of the types (most prominently the class currently being created) do not. You can somewhat alleviate the problem by introducing a special placeholder Self type and use it instead of the current class name, but it won't solve the problem of other classes still awaiting construction.

A different (but connected) problem is that you are not always constructing the full method, instead you could be just creating a prelude for a particular method or replacing one instruction with a series of your own. To solve we add this idiom:

We should allow escaping from the rigid typesafe world by having unsafe operations, which issue compiler warnings

This means that instead of missing pop() from a type with no stack slots we should just deprecate it or otherwise issue a warning. This also means that we should have invoke* methods that take strings as parameter types, similarly deprecated.

However, since we know it's not a perfect world we'd like to at least protect ourselves a bit better. Therefore we add another idiom:

Allow users to document their assumptions.

This means that when we need to write just a fragment of bytecode we want to document what stack values and variables will it need. For that we introduce methods assumePush()/assumePop() and assumeVar*(), which do not push any values, but just add the corresponding type variables:

PLAIN TEXT
JAVA:
  1. public class MethodBuilderS2V1 <O, S0, S1, V0> {
  2.   public <S> MethodBuilderS3V1<O, S0, S1, S, V0> assumePush(Class<S> type) {
  3.     return new MethodBuilderS3V1<O, S0, S1, S, V0>(cb, mv);
  4.   }
  5.  
  6.   public MethodBuilderS1V1<O, S0, V0> assumePop() {
  7.     return new MethodBuilderS1V1<O, S0, V0>(cb, mv);
  8.   }
  9.  
  10.   public <V> MethodBuilderS2V2<O, S0, S1, V0, V> assumeVar1(Class<V> type) {
  11.     return new MethodBuilderS2V2<O, S0, S1, V0, V>(cb, mv);
  12.   }

Using them we can document our expectations and let the compiler validate them. The following is an example of how we can use the assumptions to document that we expect PrintStream to be on stack and String[] to be the variable 0.

PLAIN TEXT
JAVA:
  1. private static <O> void genSayHello(MethodBuilderS0V0<O> mb) {
  2.     mb.assumeVar0(String[].class)
  3.     .assumePush(PrintStream.class)
  4.     .loadVar0(String[].class)
  5.     .push(0)
  6.     .arrayLoad(String[].class, Integer.class, String.class)
  7.     .invokeVirtualVoid(INVOKEVIRTUAL, PrintStream.class, "println", String.class);
  8.   }

Problem 2: Control flow and reuse

The next problem is how to mix the DSL with the general-purpose control flow and method calls. The problem here is that (although it is hidden from the user) we return a different type every time. Therefore if we add a conditional operation, we would need to save the type in a variable, which would be clumsy, since it contains a lot of inferred types. It also wouldn't cope with changes, since the inferred types would change every time. To solve this we invoke the next idiom:

Hide control flow in closures.

In fact we introduce two types of closures -- the strongly typed and weakly typed. The weakly typed one loses all accumulated type information and is meant first of all for reusable methods, since they need to take one based type as an argument:

PLAIN TEXT
JAVA:
  1. public interface Closure {
  2.   public <O> void apply(MethodBuilderS0V0<O> mb);
  3. }

The strongly typed closures are generated with the class and look like that:

PLAIN TEXT
JAVA:
  1. public class MethodBuilderS2V1 <O, S0, S1, V0> {
  2.   public MethodBuilderS2V1<O, S0, S1, V0> closure(ClosureS2V1 closure) {
  3.     closure.apply(this);
  4.     return this;
  5.   }
  6.  
  7.   public MethodBuilderS2V1<O, S0, S1, V0> closure(Closure closure) {
  8.     closure.apply(new MethodBuilderS0V0<O>(cb, mv));
  9.     return this;
  10.   }
  11.  
  12.   interface ClosureS2V1 {
  13.     <O, S0, S1, V0> void apply(MethodBuilderS2V1<O, S0, S1, V0> mb);
  14.   }
  15. }

We illustrating using the weakly typed closures to call the method genSayHello() introduced previously:

PLAIN TEXT
JAVA:
  1. .beginStaticMethod(ACC_PUBLIC | ACC_STATIC, "main", void.class, String[].class)
  2.     .getStatic(System.class, "out", PrintStream.class)
  3.     .closure(new Closure() {
  4.       public <O> void apply(MethodBuilderS0V0<O> mb) {
  5.         genSayHello(mb);
  6.       }
  7.     })
  8.  
  9.     .returnVoid()
  10.     .endMethod();

Of course with the introduction of closures in Java 7 this idiom would be much shorter

Problem 3: Productivity and performance

One of the problems with having classes for each stack depth and variable size is that we have a lot of them ((max stack depth + 1) times (max variable count + 1)). And we need to have a reasonably big number as maximum before deprecating the methods and losing track of some stack slots and variables. Writing them by hand would be exteremely unefficient, therefore we propose to use a generator.

The other problem is that having such a great number of classes and instances at runtime could cause performance problems. The way we could solve it is providing also a source-compatible unsafe library that would provide exactly the same interface, but always return "this" and track no types to allow good perfromance. It cannot be made binary compatible, since every method has a different signature (even though inference hides it from the user), but since those signatures are not present in source it would be a matter of recompiling the application for production, after insuring in development that the code is typesafe.

Problem 4: Primitive types and operations

The final problem is one of the most annoying and the one we tackled the least. Since generic types cannot be parametrized by primitive types we have to track boxed types instead of primitives (Integer v/s int). This means that we cannot distinguish among them and they can still cause a runtime error. What's worse we need to somehow indicate to a method that the parameter is a primitive type, but still need to pass to it a boxed one to satisfy the inference. There are several possibilities here, but all of them are relatively ugly.

A similar problem, which is yet to find an elegant solution is that double and long take two stack slots, whereas everything else takes one. And of course some coercions will not work with the current prototype.

If you have any ideas, please let us know.

But is it real?

The code is available in a Google Code repository. The code is just a prototype to show off the typesafe DSLs for our upcoming paper and is yet nowhere ready to even consider its usage.

The generator is written in PHP and driven by a BASH script. It may sound crazy, but it allowed me to prototype the code very quickly.

UPDATE: I also uploaded a downloadable distribution with 10x20 classes pregenerated for your pleasure. Check out src/Test.java for the example and gen/* for the generated classes. The asm-3.0.jar must be in the classpath.

]]> http://dow.ngra.de/2008/03/24/typesafe-dsls-in-java-part-1-typesafe-bytecode/feed/ 6