Scaling

Throughout the years you’ve built multiple lists. Your data has grown. You have a big announce to make and you would like to send everyone a message. You make a campaign, duplicate this to the lists and hit send on all of them. You get couple of angry replies that they received multiple messages. Oh, okay, they were on multiple lists, I’m sure there is a solution for this.

The chimp says that use groups instead of lists. Have a master list with multiple groups and then you won’t have the problem of people receiving multiple emails of the campaign. Okay, lets give it a try. You create the list, you create the groups, you populate the groups. Some people are on a single group, some are on multiple groups. You send out a master campaign and you see that couple of people unsubscribed. Thats cool. Then you find out that they were unsubscribed from all the groups and unsubscription did not even let them choose a group. WTF?

You ask the chimp for help and they say that this is a feature. Choose one, a granular approach or bazooka but not both.

Blocking Any Account at MailChimp

Scenario: you are a happy chimp and sending out campaigns, everything is fun and cool until one day you are unable to log into your account. You get a message that your account has been blocked because you’re most probably a spammer. Wow, how did that happen?

You send an email to their support and find out that they found your address from stopforumspam.com and they block your account automatically. It does not matter how you ended up on that website ( in this case you have not even registered a forum account with this email address for ages ). You send them one more email and ask what provider do they recommend to use now that this bad luck hit you. They kindly reinstate your account. Weeehah!

So if you want to block somebody from MailChimp, get their email address to one of these stop spam sites. For quickest results see stopforumspam.com/add You can be sure that they can’t do anything for 48 hours.

Conclusions

Besides the two scenarios that I’ve mentioned I’ve had many more smaller issues with MailChimp. From not being able to call them, not being able to bulk delete people from large groups, not being able to remove email header images, not seeing email addresses of shared accessees etc.

The lists are dumb and cannot talk to each other without extensive scripts from infra wizards. Likewise, even if you are a dummy yourself, you cannot actually communicate with anyone at MC (they do not use telephone technology at all) except via email or Live Chat. Also, you frequently get signed out every 30 minutes whether you are in the middle of typing an email campaign or just sitting there wondering why you chose MC all those years ago…

Introduction

My current Lenovo T60 laptop is at least 4 years old (nobody at my company knows anymore how old it is, as even support department doesn’t have any records about it) and is waiting for the replacement. It was running Windows XP pretty fine, if one doesn’t count some sporadic hangs and reboots. Around 1 month ago I started to experiment with different operating systems to see what’s happening in modern operating systems world.

At first I tried Ubuntu with it’s smart installer which installs the system somewhere in the existing NTFS file system which is pretty cool. except that laptop hung every other sleep…

Then I managed to install MacOS X on a separate hard drive, but I couldn’t find any hackintosh drivers for my particular wireless card (Intel PRO/Wireless 3945ABG).

Then I was back to Windows, which has Windows 7 as it’s most modern version. Well, it’s considered to be good practice to reinstall windows time after time so I decided to make a backup and do a clean install. For the first time in my more than 15 year experience with MS software I decided to use it’s own user state migration tool rather than manually copy all needed files and settings from ‘Documents and Settings’ folder. I don’t usually keep there much because of spaces in the folder names and all sorts of problems because of that, but still I got there some assorted important documents.

Fail F.A.S.T.

I happily selected ‘Files and Settings Transfer wizard’ (F.A.S.T.) from the Start menu, clicked next-next-next, let windows show me some progress bars and in ended up with a folder with a magic name USMT2.UNC and 2 files inside it: IMG00000001.DAT (with 1.2Gb size) and file called status (with a size of 16 bytes). Somehow I thought that exporting files from windows xp using it’s migration tool and then importing these settings in windows 7 sounds like a pretty normal usage pattern, but how wrong was I!

I copied all my backups to external hard drive, dropped all partitions and installed windows 7. (In the meantime I managed to fry that hard drive with everything on it, but that’s another story.) Now it was time to restore files from the backup!

So I went to Accessories menu to find something called ‘Easy Transfer’. Sounds promising… Except that it doesn’t know how to handle Windows XP FAST archive, that I had. Hmm. That’s weird. Searching on MS KB gave no pointers except that if I wanted to migrate files from an old computer I should have installed Windows 7 on a new machine, then created some magic disk for the old system, transferred files with it and then would import stuff back on Win7. How sick is that!? How the hell I’m supposed to to a clean upgrade if I must have 2 machines to transfer my stuff?

I googled for the magic folder name: USMT2.UNC. This time I found some command-line app called User State Migration Tool, which can be downloaded from MS, but unfortunately its loadstate.exe still couldn’t do anything with that poor folder. Another bit of information was that I should use exactly the same version of windows xp to restore files from such archive. (What’s the point of such transfer?) Anyhow, I installed Windows XP in a VirtualBox and was hoping to run restore procedure and then just copy files manually as I have done tens of times so far. Yeah… FAST said that I have insufficient space on my disk. How’s that – I had around 4Gigs of free space there and this stupid archive is only 1,2Gb?! So, even with the same Windows XP my ‘backup’ was useless.

Further search revealed a lot of complaining people who found themselves with such useless ‘backup’:

• http://www.bleepingcomputer.com/forums/topic1153.html
• microsoft’s own forums

However, this time I found a link to a tool called fastconv.

Here is a page with lengthy description:
http://windowsxp.mvps.org/fast.htm

Ok, let’s give it a try. At the first try fastconv log said to me: 
Error Source store is invalid. [ERROR=13 (Dh)]

Along with the fastconv there was one more utility called rmv2opq.exe and it was said that it

converts a removable media store into a normal compressed store

Whatever it means. Anyway, I ran it. The result was ‘astonishing’ – target folder contained the same .dat file and status file, but this time status file was 12, rather than 16 bytes! What a progress! However, this time, running fastconv.exe on these ‘new’ files gave more information – it started listing a lot of .dat files but crashed in the end. Fortunately, fastconv had a switch to keep temporary files. When I used that, it generated 2 folders: 0000/ and TRANS/.

Folder 0000/ contained 2255 files with names from 001.DAT to 8CF.DAT. When I looked inside these files I could see that these were extracted original files but with wrong names. TRANS/ folder had a binary file DB.DAT. However, when I looked inside I could see both .DAT file names mixed with original file names and all that in unicode, meaning that simple grepping wouldn’t help as unicode uses 2 bytes per symbol so all names had 0×00 between characters and everything was again mixed with binary data. Fiddling with the fastconv didn’t give any results as it kept crashing after generating temporary and database files.

Getting hands dirty

Ok, lets explore what we have. DB.DAT file’s header said: ‘m e m d b   d a t   f i l e   v 9   n o d b g‘. Couldn’t find anything reasonable for neither opening it nor format description to write my own parser.

Then I started looked for something to extract unicode text from binary files. And found an application called BinText from McAfee, that was doing exactly what I needed. From the DB.DAT file it generated me a listing like this: 

At some point I saw such pattern:

00000002CCB6   00000002CCB6      0   0000002E.DAT
00000002CD3A   00000002CD3A      0   arrow_l.cur
00000002CDB8   00000002CDB8      0   arrow_l.cur
00000002D002   00000002D002      0   arrow_m.cur
00000002D056   00000002D056      0   0000002F.DAT
00000002D0DA   00000002D0DA      0   arrow_m.cur
00000002D158   00000002D158      0   arrow_m.cur
00000002D3A2   00000002D3A2      0   arrow_r.cur
00000002D3F6   00000002D3F6      0   00000030.DAT
00000002D47A   00000002D47A      0   arrow_r.cur
00000002D4F8   00000002D4F8      0   arrow_r.cur

Which could be interpreted like this:

<original file name>
<corresponding .dat file>
<original file name in 8.3 format>
<original file name again in some other format>

Following was simple – clean up the log and leave only original file and .dat file:

$> grep -B 1 DAT trans.txt > files.log

Which resulted in a file that looked like this:

00000002F0A2   00000002F0A2      0   beam_r.cur
00000002F0F6   00000002F0F6      0   00000038.DAT
--
00000002F442   00000002F442      0   beam_rl.cur
00000002F496   00000002F496      0   00000039.DAT
--
00000002F7E2   00000002F7E2      0   beam_rm.cur
00000002F836   00000002F836      0   0000003A.DAT
--

Then I wrote a small program to parse this log and rename files (handling of duplicates was important as I couldn’t restore directory structure). Before final renaming I ran it several times with writing logs to spot all problems in the log file.

Ok, I lost the directory structure and some file names are still broken, but at least I got my files back and I can fix these small issues by hand.

Lessons learned? Don’t blindly trust ‘migration’ software and keep good ol’ files handy.

P.S. Post named after the song ‘I fought the law’ by Dead Kennedys.

I still have vivid memories of the time when JBoss 4 started for 26s on my devel machine with only its vanilla apps deployed (JBoss 5 was twice as slow). I then thought that I’ll rant about it but for no apparent reason I did not.

Time has passed and I got some extra motivation to do a startup timings for containers with their default configurations. The motivation came from a colleague of mine, Anton Arhipov who also helped me time the containers and write this post.

Our plan

Take all containers and app servers out there and compare apples to oranges. Startup a AWS Large Instance (4 EC2 compute units, high I/O performance, 7.5GB of memory) and time the startup of vanilla containers with the latest JDK (1.6.0_22 at the time of running the tests). You can check out the configuration of Ubuntu and run your own tests via launching the ami-84fd0bed at Amazon AWS.

WTF, why? Don’t you have anything better to do?

Well, hopefully this will be a start of something bigger. Actually Anton’s idea was to make something similar to Matt Raible’s web framework comparison. Of course for a similar comparison of containers a lot more is needed than just startup time timings. Clustering, session replication, availability of services comes to mind at the moment. See comparison of application servers from Wikipedia.

I’m personally interested in the startup times because there are developers out there who don’t know that the startup times can have an order of magnitudes differences and vendors don’t care too much about developers’ productivity as just pushing out new features.

And we do have better stuff to do but we do need time off from our regular activities.

Results

Let’s start by observing the overall results of the startup times. As we can see on the graph below, the pure servlet containers, Tomcat and Jetty, are way ahead of the full blown containers, which is obvious (remember, we’re comparing apples to oranges here!). Maybe this data doesn’t help you very much, but actually you can make some conclusions out of it: try to implement your application in a way that it doesn’t depend on a heavyweight container with the slow startup time! Obviously, this is not always possible, but it is worth trying.

Next, lets see how the winners compare. Tomcat seems to be a fraction of a second faster than Jetty with Tomcat 7 being the fastest with under a second for the startup time. Comparing Tomcat to Jetty is pretty fair but not very reasonable as both are very fast. The only area where Tomcat really beats Jetty is the shutdown time. Jetty executes a shutdown hook when you try to stop it, and it takes almost as much as it takes to start the container. Tomcat, in opposite shuts down immediately.

The next chart shows the startup times for the heavyweight guys. We’re missing Geronimo in our test, this will be corrected later.

Let’s have a look at the outsider of the shootout – JBoss. We have to mention here is that the measurements we’re done with the default profile. JBoss is a very configurable application server in terms of services it runs, but we measured the default profile just because it seems that it is the most commonly used by developers (our wild guess, do comment if we’re way off).

You can notice a bad tendency for the startup times starting from JBoss 4.2 and growing until JBoss 5.1. We can see an improvement for JBoss 6.0, and there’s a remarkable improvement in startup time for JBoss 7.0 You might wonder what has changed for JBoss 7.0 so that the time improved an order of magnitude? The answer is that JBoss 7.0 is OSGi-enabled and it seems that it is not starting all its services on the startup modular! Again, this is what happens when you compare apples to oranges! To make the results look more real, we’d have to deploy a mid-size application to the container which would enable some of the services on the startup. We’ll do that in our upcoming tests.

Besides the graphs we’ve also have the data in a spreadsheet.

What next?

We see that OSGI containers are really fast to startup. We’re thinking of running the tests with an app deployed so that more modules are started up on startup (read: real life tests). Also we’ve left out commercial containers at the moment. This is due to we wanting to release the Amazon AMIs and not deal with licensing issues. We’ll definitely run tests with commercial containers if there is interested out there (please comment which container would you like to see in the shootout).

Generally I’m a big fan of Play! It is easy to use, easy to setup and fast to develop with. Its like Struts with JRebel but feels more fresh. Okay, back to the rant.

Lets say you want to make sure that each time you run your application, the HSQLDB is initialized into the same location.

Well, you go to application.conf. Find the db.url field, comment it in, change it to

db.url=jdbc:hsqldb:file:my/fav/static/location

Done! 10 minutes you notice that

/private/var/folders/Em/EmtX3-Q+EgOCv9GUy0k8dU+++TI/-Tmp-/Jetty_0_0_0_0_8080_webapp____.7fbo9s/webapp/WEB-INF/application/db/db

is used as the DB location. After grepping, doublechecking, debugging and swearing you find in the internals of play that db.url, db.pass, db.user, db.driver are ignored if db=fs is set. WTF? No warning, no nothing that your configuration gets overwritten, aaargh!

Lets say your deployment is not Play! (they advise you to use play run to run apps written using Play! instead of a classic servlet container). Lets say you target Jetty instead.

Well, you deploy your WAR archive to Jetty and develop as you would before but then you discover that your files are no longer reloaded. You also discover that you get tons of database exceptions. After grepping, doublechecking, debuggin and swearing you find in the internals of play that once you ditch the play run and move to a classic container Play! will presume that you must have been smoking crack when you wrote application.mode=dev in your configuration file. It will just override it and make sure that you run in production mode this time. WTF? No warning, no nothing that your configuration gets overwritten, aaargh.

So next time you’re having problems with Play!, do some grepping, doublechecking, debuggin and swearing and find the location where your configuration options are overwritten by some Java code inside play.jar.

My Eclipse instance starts with a warning popup. Well, most likely it picked up a JRE instead of the usual JDK specified in my environment variables. But hey, I know how to handle property files (ed: seriously?), and they are actually pointing me towards the eclipse.ini.

Clicking the link on the warning popup does not open the file itself though, so I have to manually navigate to the C:\Programs\eclipse folder and find eclipse.ini from there containing:
[code]
-showsplash
org.eclipse.platform
--launcher.XXMaxPermSize
256M
-framework
plugins\org.eclipse.osgi_3.4.2.R34x_v20080826-1230.jar
-vmargs
-Dosgi.requiredJavaVersion=1.5
-Xms40m
-Xmx256m
[/code]

Adding –vm C:\Program Files\Java\jdk1.6.0_14 to the end of the configuration and restarting. Well – nothing has changed. OK, let’s open up Google.

Proposed solution: -vm option has to be specified before the –vmargs option. Still no help, Eclipse will start with the previously mentioned warning.

Second solution – there has to be a line break between –vm and the value specified – BINGO. We are getting somewhere –

Now let’s just change it to:
[code]
-showsplash
org.eclipse.platform
--launcher.XXMaxPermSize
256M
-framework
plugins\org.eclipse.osgi_3.4.2.R34x_v20080826-1230.jar
-vm
C:\Program Files\Java\jdk1.6.0_14\bin\javaw.exe
-vmargs
-Dosgi.requiredJavaVersion=1.5
-Xms40m
-Xmx256m
[/code]

And it just works. But for heaven’s sake – why on earth do I have to figure out

• where is the configuration I have to change
• in which order do I have to specify the parameters
• that the key-value options must be separated with the line break

Considering the last warning dialog – why cannot the whole configuration be validated and initialization failed properly? If this were a tool used in-house or only by small number of freaks, then – let it be. But this is Eclipse we are speaking about – there are millions of people working their asses off on a daily bases with this tool. Can’t imagine the time wasted for problems like this …

This year’s dow.ngra.de hoax, Microsoft acquires ZeroTurnaround for undisclosed amount has amused us even months later. Couple of weeks ago we spotted a Chinese article that listed all the acquisitions of Microsoft during the past year. And we made the list!

Today we found the news from a business systems news & analysis page, FSN.

FSN provides finance professionals around the globe with independent news, vital analysis and deep insight about key developments in the world of business and accounting systems. We debunk the technical jargon of the IT world and present leading-edge issues in a way that is accessible and relevant to business decision makers.

We’re in the article Shifting Sands (EDIT: link is dead now) with players like Salesforce, Digita, Google, Oracle, Microsoft and HP :)

A fact taken from a blog post on 1st of April, not confirmed from either parties and presented in an article with vital analysis and deep insight.

Of course there is an easy fix. Either Microsoft acquires the Java productivity tooling company ZeroTurnaround or FSN notices a referrer from this blog and makes changes. I’ll keep you posted on who wins the race.

How hard can it be to add JVM arguments to the run configuration of an application when running it from an IDE? Lets say I want to add -Dsuperboost=true or just ~/myconf/.myconf to the Java application?

Eclipse has a green button and Run configurations under it, there you can specify Program arguments and VM arguments. Easy.

IntelliJ Idea has a green button and Edit Configurations under it, there you can specify Program parameters and VM parameters. Easy.

Netbeans has a green button. Thats it. Where are the options? When you check project Properties there is Build » Compiling & Packaging but no run. Googling for such a generic topic does not help me much, just aggrevates more.

Finally I realize, everything is ANT based, I’ll just modify the build.xml files used. The toplevel build.xml includes project/nbproject/build-impl.xml, this includes some NetBeans bundled build.xml. Okay, this is not the best place.

I find a file named project.properties with two options in it, javac.compilerargs and javac.source. Jackpot, I’ve found the hidden settings, I just need to figure out the magic names that I could add. Some more googling and trying out different flags from different NetBeans versions I’m still out of luck, I’m unable to guess the names.

Finally I went through NetBeans/harness/build.xml » common.xml » run.xml and the magic secret hidden flags follow:

• run.jdkhome
• run.args.common
• run.args.prepared
• run.args
• run.args.extra

Weeehah! Now I need only to add “-noverify”. No, more like “-J-Xverify:none”, argghh.

It all started out as a cool infrastructure task. Migrating email to Google Apps. Its easy, quick and on top of that there is a 30 day FREE trial. Wow!

I started with the task in the beginning of January. I created an account at Google Apps, paid for the number of users and was off. 10 minutes later I noticed that I should have used a shorter URL for the domain, so I thought I’ll ditch the mail prefix from the domain.

FAQ says I have to create a new account. Bummer. I created a new account, paid for the number of users again, twiddled with settings and email was working! Easy. I told you so! All I had to do now was cancel the premium service for the longer domain version and I would be all done.

From this point on things got hairy. Firstly as I had tons of tabs open in my browser, I managed to cancel the premium service for the shorter domain. Ok, happens. I’ll just upgrade later on again. Then I cancelled the long URL. Two premium accounts downgraded, one deleted and other running in free edition now.

Of course Google Checkout can’t follow this kind of stuff in real time and I had been charged for both of the services. I contacted the seller from Google Checkout and sent them a message to make my fiddling clear.

Nothing happened. A week later as the funds were still charged I contacted them again via Google Checkout. No luck.

Maybe I’ll have to wait the 30 day for it to be paid back because the free period was also 30 days? Sure thing. Waited couple of more weeks, nothing changed.

Things are getting scary. There is no support if you don’t have a premium account and I’ve just cancelled my two premiums. It seems that the only way to get them talking to me is pay them money, but they already owe me money!

I write to their support forum, twice! Not a single answer during the months. Okay, this is weird already.

I’m able to find a link, that says that one can ask for real support if the question is concerned with billing. Weeehaah, I’m sure I’m saved now! Nope. I made my case using their web form and you know what? No answer!

Usually I would take a step back and talk to the payment mediator, either PayPal or Moneybookers but you know what, Google is handling my payment also (via Google Checkout) and I can’t find a way to report to Google that Google stole my money.