I chatted with the early arrived organizer and speakers about Ruby, Java deployments and British comedy sketches (Big Train, Little Britain and Alan Partridge).

The event had 2 presentations. One was about the Aframe, doing quite a niche stuff, they build a system for online video production (editing, transcoding, tagging). The presenter, John Cowie went over the technical hurdles that they have had to cross to get their product out the door. Also he had a positive story to tell that it is possible to shorten the gap between the ops and dev and thanks to that he is abe to take vacations without having interruptions as he used to (the London underground still told me to Mind the Gap).

The next presenter, John Leach talked about the Ceph project. A posix compliant distributed network storage and filesystem (think of HDFS, GFS) that happens to be OSS and is quickly maturing. We also got a fun history of the network storage stack. I was lucky enough to have read Apache Hadoop Goes Realtime at Facebook quite recently and was able to chime in couple of comments at later discussions.

After the presentations there were random chats and we visited a local pub. It was a great no pressure event where you can meet people with similar interest, either technical, comedy wise or board game wise (met a 9kyu Go player). The venue was really comfy and cosy.

PS. It was the second office I’ve seen so far that has a Go set, the first is ZeroTurnaround.

Scaling

Throughout the years you’ve built multiple lists. Your data has grown. You have a big announce to make and you would like to send everyone a message. You make a campaign, duplicate this to the lists and hit send on all of them. You get couple of angry replies that they received multiple messages. Oh, okay, they were on multiple lists, I’m sure there is a solution for this.

The chimp says that use groups instead of lists. Have a master list with multiple groups and then you won’t have the problem of people receiving multiple emails of the campaign. Okay, lets give it a try. You create the list, you create the groups, you populate the groups. Some people are on a single group, some are on multiple groups. You send out a master campaign and you see that couple of people unsubscribed. Thats cool. Then you find out that they were unsubscribed from all the groups and unsubscription did not even let them choose a group. WTF?

You ask the chimp for help and they say that this is a feature. Choose one, a granular approach or bazooka but not both.

Blocking Any Account at MailChimp

Scenario: you are a happy chimp and sending out campaigns, everything is fun and cool until one day you are unable to log into your account. You get a message that your account has been blocked because you’re most probably a spammer. Wow, how did that happen?

You send an email to their support and find out that they found your address from stopforumspam.com and they block your account automatically. It does not matter how you ended up on that website ( in this case you have not even registered a forum account with this email address for ages ). You send them one more email and ask what provider do they recommend to use now that this bad luck hit you. They kindly reinstate your account. Weeehah!

So if you want to block somebody from MailChimp, get their email address to one of these stop spam sites. For quickest results see stopforumspam.com/add You can be sure that they can’t do anything for 48 hours.

Conclusions

Besides the two scenarios that I’ve mentioned I’ve had many more smaller issues with MailChimp. From not being able to call them, not being able to bulk delete people from large groups, not being able to remove email header images, not seeing email addresses of shared accessees etc.

The lists are dumb and cannot talk to each other without extensive scripts from infra wizards. Likewise, even if you are a dummy yourself, you cannot actually communicate with anyone at MC (they do not use telephone technology at all) except via email or Live Chat. Also, you frequently get signed out every 30 minutes whether you are in the middle of typing an email campaign or just sitting there wondering why you chose MC all those years ago…

The library is great, easy to convert an ip to a country and when using the country flags from it’s side project you could spice up your statistics with the country information. This a lot faster than using reverse DNS lookup.

The problem. The PHP implementation is a lot slower. Embarrassingly slower. Without any caching the Java version is able to do ~6000 queries per second. The PHP counterpart can push through ~850 queries. The implementations are the same. The stats provided by the author of the library are 8000 vs 1200. So about the same as my measurements.

I like PHP, I don’t use it that much anymore but I still care when I see such embarrassing numbers. I took the implementation and started profiling it. Spent the night running different tests and trying to optimize.

General outline of the algorithm is as follows. We take the dotted string IP and convert it to an IPv4 Internet network address (e.g. 69.55.232.153 becomes 1161291929). The DB holds sorted ranges of these addresses. A binary search will happen on these addresses and we have a country for the ip. Take a look at the implementation.

IO part is surprisingly low. The internal fseek, fread constitute to 2% of the execution time. On the other hand the user level fseek which is just a wrapper alone uses 5%. readShort and readInt take 20% of the execution time.

function readShort() {
	$a = unpack('n', fread($this->m_file, 2));
	return $a[1];}
 
function readInt() {
	$a =unpack('N', fread($this->m_file, 4));
	return $a[1];}
 
function seek($offset){
	fseek($this->m_file, $offset);}

We see that the latest profiling results have twice the number of freads and unpacks than fseeks. It seems that fseek is used to seek out the right position, read two numbers with unpacking them. The implementation confirms that. Luckily we could just read once (2 bytes more) and unpack once (2 unpackings with one invocation).

$a =unpack('N', fread($this->m_file, 4));
$np['ip'] = $a[1];
 
$a =unpack('n', fread($this->m_file, 2));
$np['key'] = $a[1];
 
// this can be changed to
$np =unpack('Nip/nkey', fread($this->m_file, 6));

How does this version stack up to the Java version? Lets disable profiling and run 100 000 iterations. Vanilla version processes ~850 IPs, when functions are inlined the number is around 1400. Java version can still do 6000.

Lets try caching. Peeking at the Java implementation shows that Java caching version (whopping 141 242 IPs per second – yup 141k) uses just a byte[] array and makes lookups from there instead of seeking and reading from file. Easy, lets do the same in PHP.

We read everything into a string and instead of fread with access the string elements with the offset. For fseek with just set the offset. We are using 600kb more memory but can increase the throughput to ~2800.

As it seems I’ve just wasted a night, I just should have checked the Computer Language Benchmarks. PHP in the sense of execution speed is uncomparable to Java.

The upside, we can still take the library, eliminate recursions, double unpacks and add caching. A small gain is still a gain.

In fact it is now our firm belief that with technologies like that COBOL will make a return and become the language of choice for web development. I mean who needs local variables, recursion, dynamic memory allocation, or structured programming constructs when we have a language that reads like plain English. All the real programmers know, how important is to have code that reads well, and our new blog platform provides twice the scalability of Java on half the hardware to boot.

The new version is a significant reworking of Prism. First of all you no longer have to install a 6.6 Mb application in addition to Firefox. Now you can just download a 500 Kb Firefox extension, which will start Prism as a particular Firefox profile. And you can create the desktop shortcuts to your web site in one click using “Tools -> Convert Website to Application”.

Secondly Prism will finally pick up the favicons that the website is using and use it both as shortcut icon and (drumroll!) the application window icon! Before you had to download the icons manually and it still would use the Prism icon in the taskbar, which made it much harder to distinguish the windows. Having the GMail icon in the taskbar is just what I’ve been waiting for. Now, if only it would change on new mail…

However, no matter the changes, I’m still stuck with a major annoyance — no Firefox shortcuts work. And since half the sites on Internet do not optimize for 1680×1050, my first reaction in Firefox is often Ctrl+, to increase the font size. Well, hopefully they hit it in the next release, you hear that, Mozilla?

P.S. I’m making this post from my nifty dow.ngra.de admin browser app, now where do I find an icon for that?…

There are tons of plugins out there and only a couple support these features. Why? Well, if you look at the Writing a Plugin documentation you see that a plugin can be packaged absolutely in any possible way.

Plugins can be zip, rar, gz, bz2 etc. archives. They can contain any number of subdirectories that the administrator has to copy to certain other folders and the requirement of meta information is quite relaxed. The automation of installation, upgrade, deletion, versioning for all the plugins is impossible with such relaxed rules.

WP plugins (like the others already have — FF extensions, Eclipse plugins, Google Gadgets, WARs …) should have a solid structure so that the management of the plugins could be automated. One click install, update notifications or automatic updates, public repositories and other cool features can be built upon this.

Besides all the other great ideas already posted I’ve submitted one that would try to address these problems from more ground up.

Aptana Jaxer is exactly such a technology. There is nothing new about having a server-side API. There is nothing new about building applications in HTML and JavaScript. The genius part is running Mozilla engine on the server-side and having access to full server resources from the browser via a controlled environment with no extra layers.

As far as I understood the communication between server and client is done by:

• DOM updates are seamlessly propagated from the server to the client
• Function calls can be proxied to call to the server. All marshalling is done automagically

This pretty much means that code runs seamlessly in a mixed, secure environment.

Although the “runat” attribute that controls whether code is run server-side or client-side can be a bit unlogical at first, the setup can allow to build powerful applications with only one (count it, one) technology — HTML and JavaScript. And this is the technology you have to use anyway to even deploy something on the web.

Of course “the one technology” suffers from being dynamically typed and generally known for its quirkiness, but with JavaScript 2.0 support on the way to Mozilla and good IDE support (which Aptana is in a good position to provide) this technology might yet give a fresh meaning to the word web application.

An interesting question I’d like to ask from the Aptana developers is if we can add a third environment to the mix — desktop applications? Having the same API to use on the server, in the browser and (with less restrictions) on the desktop could give Adobe AIR a run for their money.