Wednesday 5 December 2012

Are you relying on hash keys being ordered?

tl;dr - if you rely on Perl 5, and plan to eventually upgrade to the upcoming 5.18, do this *now*:

  1. Install 5.17.6 (you do use perlbrew, right?);
  2. Try your modules and apps in it (you do have tests, right?);
  3. If  anything breaks, it's likely because you're relying on keys(), values() or each() being in some particular order. You really shouldn't, so go sort() your keys or something :)
  4.  If some CPAN module you depend on suddenly fails on 5.17.6, make sure to let the author know;
  5.  Spread the word!

On Hashes & Security


The Perl 5 core team has always put security as one of its top priorities. To put things under perspective, in late 2011, an algorithmic complexity remote denial of service attack (original paper, advisory, slides, video) was found on major language implementations like PHP, Ruby, Python, Java, even JavaScript. It's been fixed in Perl 5 since... 2003.

That was then. What about now?


Still thinking about security, Yves Orton pushed some important changes these past few weeks, changes that are going into perl 5.18.0. Among other things, to quote 7dc8663964, it:

  • Introduces multiple new hash functions to choose from at build time. This includes Murmur-32, SDBM, DJB2, SipHash, SuperFast, and an improved version of the original One-at-a-time.
  • Rips out the old HvREHASH mechanism and replaces it with a per-process random hash seed.

Optimizations aside, the ability to change hash functions easily is important because, if, for whatever reason, the active function is found vulnerable to an attack, you don't have to wait until the Perl Core Team (or your specific vendor/system) releases a fix: just recompile your perl setting another hash function as default.

The important bit, however, is the per-process random hash seed. Until now, perl was using a not-so-great hash seed, one that was set during compilation. All hashes would use this seed, and if a collision attack was detected it would trigger a rehash, where every item in the hash would have its hash value recalculated, with corresponding effects performance and memory. Of course, when too many collisions were found, the rehash would switch to a random seed instead.

Now, after this change, every process is guaranteed to use a random seed.

Hash randomization should make perl even more robust to complexity attacks, and with simpler code. But, as you may have predicted, there's a side effect to it: the order of hash keys changes more often than before.

Sweet! But, what does it mean to my code?


As it is stated in perlsec since version 5.8.1 (that one from 2003), Perl has never guaranteed any ordering of the hash keys, and in fact the ordering has already changed several times throughout its history. The problem, however, is that a lot of developers end up inadvertently relying on hashes being ordered, or rather in some random but constant order, simply because that particular order worked on their machine. Talk about a subtle bug!

This may not be your case, but you should check nonetheless. Andreas König, Father Chrysostomos and the rest of the P5P/CPANTesters gang have gone through the enormous effort of testing several major CPAN distributions for this and letting authors know whenever it failed a test while running on a patched version of perl, but they can only do so much, and there's *your* code to test, too.

You know, code your app runs, code that you haven't checked to CPAN.

Oddly enough, it looks like most of the found issues are on test cases themselves, tests that expect keys() to be in a particular order. Now, keys() is guaranteed only to return items in the same order as values() or each(), and even that is only true for the same process, so make sure you're not shooting yourself on the foot.

LIES! My code is perfect, you're the ones that broke Perl!


Well, not really. Like I said, it's a subtle bug, one that might be hitting your production code right now, but only on some very specific scenarios, and be very hard to reproduce and debug. If you don't trust me, there's a very simple experiment you can run on your system perl:

First, let's create a simple one liner that creates 15 key/value pairs, and print them on the screen:

   > perl -E 'local $,=q[, ]; $hash{$_} = $_ for 1..15; say keys %hash'
     6, 11, 3, 7, 9, 12, 2, 15, 14, 8, 1, 4, 13, 10, 5

You may have gotten a different order (did you?), but you'll probably get that same order no matter how many times you run it:

   > perl -E 'local $,=q[, ]; $hash{$_} = $_ for 1..15; say keys %hash'
     6, 11, 3, 7, 9, 12, 2, 15, 14, 8, 1, 4, 13, 10, 5
   > perl -E 'local $,=q[, ]; $hash{$_} = $_ for 1..15; say keys %hash'
     6, 11, 3, 7, 9, 12, 2, 15, 14, 8, 1, 4, 13, 10, 5
   > perl -E 'local $,=q[, ]; $hash{$_} = $_ for 1..15; say keys %hash'
     6, 11, 3, 7, 9, 12, 2, 15, 14, 8, 1, 4, 13, 10, 5
   > perl -E 'local $,=q[, ]; $hash{$_} = $_ for 1..15; say keys %hash'
     6, 11, 3, 7, 9, 12, 2, 15, 14, 8, 1, 4, 13, 10, 5
   > ...

What happens, however, if your code adds a 16th key and then, realizing its mistake, removes it right afterwards (highlighted code below)? There are still 15 keys, the very same 15 keys as before, so surely they'll be in the same order, right? Right? Wrong:

   > perl -E 'local $,=q[, ]; $hash{$_}=$_ for 1..15;
             $hash{16}=16; delete $hash{16}; say keys %hash'
     11, 7, 2, 1, 13, 6, 3, 9, 12, 14, 15, 8, 4, 10, 5


This can happen anywhere, like when reusing a hash variable:

    sub init { ( 1=>1, 2=>2, 3=>3, 4=>4, 5=>5 ) }

    my %hash = init();
    say "original: " . join ', ' => keys %hash;
    $hash{$_} = $_ for 6..100;

    %hash = init(); # restores original values
    say "original? " . join ', ' => keys %hash;


This is what I get on my good old 5.14.3:

    original: 4, 1, 3, 2, 5
    original? 2, 1, 3, 4, 5


As you can see, it's a real problem and it could be lurking in your code right now. What Yves' patch does is simply expose the issue more explicitly to you. This is a good thing, because, aside from the extra security protection, it will let you spot buggy code much easier. If you try that previous one-liner on 5.17.6, you'll get a different key order every time you run it:

   > perl -E 'local $,=q[, ]; $hash{$_} = $_ for 1..15; say keys %hash'
     1, 5, 15, 12, 6, 4, 10, 9, 3, 13, 7, 14, 11, 2, 8
   > perl -E 'local $,=q[, ]; $hash{$_} = $_ for 1..15; say keys %hash'
     5, 11, 7, 3, 15, 6, 12, 2, 13, 9, 8, 14, 10, 1, 4
   > perl -E 'local $,=q[, ]; $hash{$_} = $_ for 1..15; say keys %hash'
     2, 15, 14, 13, 5, 1, 9, 10, 3, 11, 6, 8, 12, 4, 7
   > perl -E 'local $,=q[, ]; $hash{$_} = $_ for 1..15; say keys %hash'
     8, 2, 14, 10, 1, 9, 4, 3, 6, 15, 5, 13, 7, 12, 11


Uh-oh... looks like my code is broken.


Not to worry, the fix is usually pretty easy! Look for the failing test and see if whatever is being tested calls keys(), values() or each() at some point. You'll likely want to sort() the results or change your code algorithm to something more deterministic.

I don't really have that many tests... What can I do?


Look for calls to keys(), values() or each() in your code, and make sure they are not relying on the order of the elements being returned. It is ok to do something like:

  my @keys   = keys %hash;
  my @values = values %hash;
  say "hash key $keys[3] is $values[3]";

because, as I said before, keys() and values() will always use the same order for the same process, whatever that order is. However, this is not ok:

  if ($keys[0] eq 'some_key') {
     ...
  }

simply because there's no way to guarantee the order of the list returned by keys(). The code above might have worked, however, if you always sorted the returned value, like so:

  my @keys = sort keys %hash;
 

Indirect usage


Sadly, your code is not safe just because you don't use those functions (or have them properly sorted). Sometimes you expect lists of values from external modules, and those lists might be affected by the change. So make sure you look for arrays that are populated by external functions and see if you rely on their order being a particular one. For example, you might have code like:

   my ($name, $age, $rate) = Some::Module->new->get_list( 'some_user' );
   
And, within Some::Module, you'll find the suspect:

  sub get_list {
    my ($self, $username) = @_;
    return values $self->{data}{$username};
  }

Make a failing test for it, push a fix, rinse and repeat :)

I hate this! Switch it back!


This is hardly going to happen. Remember: hash randomization is a good thing! Please take another look at the sections above and try to fix your code. If you need help, ask for it at the usual places, like your local mailing list or IRC - heck, even Facebook has a Perl group!

But if you really really really need the previous behavior, you can simply stick to 5.16, or try compiling perl defining PERL_HASH_FUNC_ONE_AT_A_TIME_OLD to simulate the old algorithm, but the entire rehashing mechanism is gone, so specifying your own PERL_HASH_SEED value is probably as close as you'll get :)

Many thanks to the nice folks at P5P for their continuous effort in keeping us safe!

Wednesday 30 May 2012

Moving modules across perlbrew installations

This short post was triggered by a conversation I had on Twitter with a friend:


He does strike a nerve there. Ever since Gugod's amazing perlbrew came to life, installing your custom perl - or several perls for that matter - is *really* easy. However, a new problem surfaced: updating your installation so your applications work again. This is, of course, per design. After all, perlbrew is supposed to give you completely separate installations, and this includes installed modules.

So, what can you do? Copying your lib directory is out of the question if you have any module that does XS, and chances are you probably do. I'm also not sure if an external local::lib directory would play nice either.

Luckily, there's a quick-and-dirty recipe to migrate your installed modules from one perlbrew installation to another. Say I just upgraded from 5.14.2 to 5.16.0 and want to install the same modules I had before:

    $ perlbrew switch 5.14.2
    $ perl -MExtUtils::Installed -E 'say for ExtUtils::Installed->new->modules' > /tmp/installed.list
    $ perlbrew switch perl-5.16.0
    $ perlbrew install-cpanm
    $ cat /tmp/installed.list | cpanm --interactive

Done!

What I did there was use the core module ExtUtils::Installed to create a list of installed modules in the file "/tmp/installed.list", then feed that list to cpanm. I used "--interactive" because some modules I have (like SDL) ask a few questions during installation, but whatever rocks your boat.

Also worth noticing that depending on the amount of modules you have installed, that last step can take quite a while, so go watch a movie, read a book or something :)

I think this tip is particularly pertinent if you're trying out a Release Candidate (RC) for a future perl release and want to make sure your toolchain builds properly - which is always a good idea.

I'm going to bug gugod for a bit to see if we can come up with something bundled into perlbrew to make this even easier, but for now you can use this :)

Hope it helps! Happy Perl Brewing!

Monday 9 April 2012

Perl QA Hackathon

Last weekend I was in Paris for the Perl QA Hackathon, a free of charge coding workshop for people involved in Quality Assurance, testing, packaging, CPAN, and other related projects.

I had the best time, met a lot of old friends and made several new ones as well. And we got so much done! It's amazing the amount of work you can do just by sitting next to fellow hackers with the same goals. Coding, debugging, design, feature requests, decision-making, if you had questions or needed help or feedback, all you had to do was look around the room and find the authors/maintainers for a quick tête-à-tête.
Schwern and Apeiron looking at some code
The hackathon was responsible for over 1 thousand man-hours of work on the Perl toolchain ecosystem in just 3 days, and none of that would be possible without all the great companies and organizations supporting the event: The City of Science and Industry, Diabolo.com, Dijkmat, DuckDuckGo, Dyn, Freeside Internet Services, Hedera Technology, Jaguar Network, Mongueurs de Perl, Shadowcat Systems Limited, SPLIO, TECLIB’, Weborama, and $foo Magazine. There were also a lot of amazing people who made donations themselves, like Martin Evans, Mark Keating, Prakash Kailasa, Neil Bowers, 加藤 敦 (Ktat), Karen Pauley, Chad Davis, Franck Cuny, 近藤嘉雪, Tomohiro Hosaka, Syohei Yoshida, 牧 大輔 (lestrrat), and Laurent Boivin. Thank you!

Many thanks are also in order to Laurent Boivin (elbeho), Philippe Bruhat (BooK) and the Mongueurs de Perl, who did an incredible job organizing the event and hosting/feeding us. Merci beaucoup! :-)

You probably already read about some of the great things that happened there. I guess it's my turn to share:

I got there thursday night, after just a couple of minutes lost at Gare du Nord trying to find the right Metro track, nothing my poor french (and a lot of pointing at maps/signs) couldn't solve. I shared the hotel room with Leon Timmermans, who arrived just a few minutes after I had settled in. Turns out Leon is not only a great Perl hacker, but also a very nice guy.

I was a little apprehensive about waking up in time - I want to say "jetlag" here, but truth is I'm just not really a morning person. Fortunately, the excitement of having a joint hacking session with several of my Perl heroes was much greater than my will to stay in bed.

After talking to a few people and finding a nice spot to settle, I was ready to start hacking. I've been meaning to try and add CPAN Testers' support to Miyagawa's great cpanminus for quite a while now, and as it turned out so did he! We talked about some of the details and he was kind enough to help me through part of the cpanminus source code, as we discussed the implementation details.

The first draft of the build.log parser was finished on that same day, so all I had to do was send the email, push things to CPAN and move to my next target, right? Wrong!

As it happens, the CPAN Testers is much more than what is exposed to module authors and users. Its amazing reports and statistics are the result of a delicate and intricate mix of modules, protocols and systems that have to work on all sorts of environment and talk to each other all the time. I have the utmost respect and admiration for people like Barbie, Andreas König, David Golden, BinGOs and Ricardo Signes, and this only increased as I dug deeper and deeper on CPAN Testers' internals.


David and Barbie patiently explained to me all the bits and pieces I needed to know, and about how a lot of the things that should be common to all CPAN Tester's clients were currently scattered around, some even with different implementations, and how great it would be if all clients - including my new cpanminus one - could share all that from a since module. They wanted to do this since the QA Hackathon in 2009, but never got around to it.
...so I got a round tuit :-)
If we could pull this off, it would not only mean life getting much easier for the CPAN Testers' clients, but also that we'd finally be able to create a next generation system to take full advantage of the power of Metabase, sending even more information (and in a much more structured manner) to the infrastructure.

CPAN::Testers::Common::Client

This module aims to provide common utilities to CPAN Tester's clients, populating all Metabase::Facts and getting them ready to be sent via Test::Reporter. It even composes the legacy email for you with the data you provide. Code is already on github, and I'll push it to CPAN as soon as it's stable and ready for consumption.

App::cpantesters

As work on CPAN::Testers::Common::Client started to take shape, I was able to plug it into the build.log parser and see it come to life. Code is pretty raw at this point as I want to make sure it's doing its thing correctly before making an actual application out of it - and before sending data to the servers! If you want to help, I encourage you to try it out and tell me whether it worked or blew up. I can be found on irc.perl.org as "garu", or you can just file a bug report (in which case, please attach the unparsable build.log file if you can).

Here's how to do beta-test it:

1. download and install CPAN::Testers::Common::Client from github;
2. download App::cpantesters from github (no installation yet);
3. mkdir /tmp/reporter

After that, just use cpanm to install modules as you normally would. After each attempt, whether it passed or failed, go to App::cpantesters' base directory and do:

   perl -Ilib bin/cpantesters.pl

You should see some debugging info on the terminal, and after it's done you can see the reports it generated in /tmp/reporter. The files are plain text, so just open them in your favourite text editor and make sure everything seems ok!

HC SVNT DRACONES


The modules above are not ready for general public consumption by the time of this writing - which is why they write to disk instead of sending to CPAN Testers. I still need to coordinate with Andreas whether the email itself is parseable, and with him, Barbie, David and BinGOs as to adapt the CPAN Testers toolchain to use CTCC, send/receive the data from new Metabase Facts, and maybe extracting even more "moving parts" into CTCC, and now that the QA Hackathon is over it might take a bit more time. But it was already a great step forward!

All work and no play?

Every night after the hackathon we'd get together for dinner and hung out afterwards. I got a chance to see a 3D printer in action, play with the new Galaxy Tab, talk like a pirate (Arrrr! Thanks Wendy!), drink some Chartreuse, give a whole new meaning to my cleric's "turn undead" ability during a very cool Role Playing session, and travel from Stalingrad to Oberkampf (you had to be there).

I also learned a lot just by listening to some of the conversation that went by, and got a chance to talk to some really smart people about their very cool projects, some which I might even tackle in the near-ish future.

Oh, and yes. There were pictures =)

See you next year in the UK!

Monday 27 February 2012

Perl Oasis 2012

Last month I was able to attend to my very first international (non-brazilian) Perl conference: the Orlando Perl Workshop, also known as Perl Oasis.

"Awesome" doesn't even begin to describe it.



I was really excited about going. Not because it's a great conference (and it is!), but because it would be my first opportunity to meet all the amazing people behind the modules and applications I use. I've been talking to some of these people for years over at irc.perl.org, and it would be great to finally put a face and a body to their names and irc handles.

Chris Prather (perigrin), the conference organizer and a long-time friend/boss, came pick me up in the Orlando Airport with his brother Mark. It was hard recognizing him because he's a bit camera shy, but he found me in no time. He was way taller than I expected :)

I arrived one day earlier at the venue and, when I asked where everybody was, I should have known the answer: the bar.


As I got to the lobby, it was easy spotting mst. He was also much taller than I expected (I see a pattern here - maybe I'm the one that's shorter than expected :P). He was sitting next to Mark Keating, and they made me feel right at home, just as they did online when I worked for Shadowcat.

The conference had incredible talks, but the thing that really impressed me was this overall feeling of "grassroots". There weren't that many people but no one really cared, because it wasn't meant to be a big event. It was meant to be something made by Perl hackers to other Perl hackers, in all levels. A conference where you could sit back and enjoy not only some mind-blowing talks but also a nice conversation with incredibly smart people like Florian Ragwitz, Casey West, David Golden, Steven Little, Cory Watson and so many others.

To illustrate the point a bit further, the conference had a rented suite on the hotel where everybody would hang out after the talks up to late hours. From vodka-embalmed cherries (or whatever the hell that was) to hotel luggage carts filled with beer and huge piles of pizza, the conversation would quickly shift from SVs and AVs to the problems with Skyrim, and from architectural design and video tutorials to rock climbing and William Shatner singing Bohemian Rhapsody.


I've brought home with me a huge level up in my Perl-fu. But, most importantly, I got a chance to hang out with some amazing people up to late hours, made new friends and had the best time ever. I can't wait for the 2013 edition of OPW!

How about you? Do you remember your first international conference?

Tuesday 3 January 2012

The Perls of 2011

Following my 2010 post, I thought I should register some of the great things that happened in the Perl programming world in 2011. Only this time instead of turning it into a timeline, I'm going to place things in topics so you can just concentrate on what really matters :)

Also, I'm only going to write about Perl 5. If you're looking for a Perl 6 retrospective, Moritz Lenz already did a very nice job with that :)

Apologies if I missed something - it's really hard to keep track of all the wonderful projects and conferences going on! Make sure to add them in the comments section below.

So, without further ado, here are the "Perls of 2011".

Some Perl & CPAN stats

According to Ohloh, over the course of 2011 there were nothing short of 5975 commits to the main Perl 5 repository. Wow! For comparison, Ruby had 3153 commits and PHP, 4461. We were a little short from our friends in the Python community, though, which had 6974 commits. Not bad, huh?

The Perl 5 bugtracking system reports a total of 915 tickets created in 2011, and 928 tickets closed in that same period. That was close, but the awesome folks at p5p once again managed to keep the stats positive :)

By the way, can you guess how many perl releases we had in 2011? 1? 3? more? 10? Try 21. Twenty one, including development and stable releases!

One of the coolest ones was, of course, perl 5.14, which brought us Unicode 6.0 support, new regex flags, the sugary "package Foo { }" syntax, improved IPv6 support and, as if these weren't enough, it uses even less memory and CPU than previous releases!

How about CPAN? 16197 distributions uploaded, of which 1873 were brand new!

Security

The Common Vulnerabilities and Exposures had 5 entries tagged "perl" in 2011, of which only 2 were actually regarding perl itself (namely, CVE-2011-0761 and CVE-2011-1487). Again, for comparison, python also had 2 records, ruby had 7, and php had 36.

Perl's commitment to stability and security was showcased in late december, when Alexander “alech” Klink and Julian “zeri” Wälde delivered a very nice talk at the 28th Chaos Communication Congress security conference in Berlin, entitled "Efficient Denial of Service Attacks on Web Application Platforms". Their work builds on top of an attack vector described in perlsec and fixed back in 2003 (the long-since-deprecated 5.8.1), and shows the issue affects almost every other popular language for the web, including Python, Java, PHP, ASP.NET and JavaScript. Ruby fixed their code in 2008 and people should be fine as long as they use CRuby 1.9 and above.

The King is dead. Long live the King!

For the past two years Jesse Vincent (obra) was our beloved Perl 5 Pumpking. Back in october he passed the torch to the incredibly prolific Ricardo Signes (rjbs), who will undoubtedly make us all very proud! A huge thanks is in order to both of them for the remarkable work they did last year - and that they'll undoubtedly keep on doing.

TPF Grants & Donations

Dave Mitchell once again did an astounding work with his "Fixing Perl 5 Core Bugs". Last year alone he worked more than 470 hours, closing 29 tickets. Great job, Dave!

In the middle of the year Nicholas Clark jumped on the wagon with his "Improving Perl 5" grant, approved with praise. So far Nick reported more than 380 hours of work, and tons of code to make Perl development even smoother.

None of this would be possible without the wonderful support from several companies that are proud to use Perl, giving out not only their public praise but also donating infrastructure and money to help further develop the language. This year we saw some incredible support from Booking.com, cPanel, Dijkmat, Liquid Web, Net-A-Porter, the Oslo Perl Mongers, perl-services.de, and the Vienna Perl Mongers.

Google Summer of Code (GSoC)

Every year The Perl Foundation participates on the Google Summer of Code program, and 2011 was no exception. This time, 6 students were accepted and all of them made their mentors proud - 100% success rate! A huge thanks is in order for everyone that participated.

Google Code-in

Another great initiative from Google in 2011 was the Code-in program, aimed at 13-17 year old school/college students with the idea of getting them involved with open source. The project is still running but we can already see some impressive results, such as over 135 completed tasks.

These numbers are great, and show that there's a big niche for Perl in schools and undergrad courses. I had the opportunity to teach Perl to undergrads in a one-week course at the Federal University of Rio de Janeiro and the receptivity was incredible! If you have the chance to give a short free workshop at your local college/university, I highly recommend you do so :)

Conferences! Conferences! Conferences!

All over the world, the vibrant Perl open-source community and their corporate sponsors filled the year 2011 with conferences showing the best modules, tools, techniques and design patterns, talking about a lot of bleeding-edge features and how to make Perl help with innovation and productivity for your enterprise.

January: Orlando Perl Workshop (OPW - a.k.a. The Perl Oasis)
February: Frozen Perl, Bulgarian Perl Workshop
March: Dutch Perl Workshop
April: Toronto Perl Workshop, QA Hackathon
May: São Paulo Perl Workshop
June: Nordic Perl Workshop, French Perl Workshop, YAPC::NA
August: YAPC::Europe
September: Italian Perl Workshop, Portuguese Perl Workshop
October: Ukrainian Perl Workshop, Pittsburgh Perl Workshop, Belgian Perl Workshop, German Perl Workshop, YAPC::Asia
November: YAPC::Brasil, TwinCity Perl Workshop, London Perl Workshop
December: Russian Perl Workshop

Only July had no perl-centric conference. Quite impressive! And we're not even counting general conferences in which we participated such as FOSDEM, FISL, OSDC, OSCON, or even Perl Mongers tech meetings. Speaking of which...

Perl Mongers

The number of active Perl Mongers group just keeps growing and growing. In 2011 alone, 15 new groups were spawned! Good luck to all our friends from AtlanticCity.pm (US), Makati.pm (PH), Bordeaux.pm (FR), HradecKralove.pm (CZ), Goiania.pm (BR), Petropolis.pm (BR), Brno.pm (CZ), Logan.pm (US), Tolyatti.pm (US), LGBT.pm, SouthernOregon.pm (US), Plzen.pm (CZ), Sendai.pm (JP), WestVirginia.pm (US) and Kerman.pm (IR).

With these new additions, our tiny planet hosts 251 active Perl Mongers groups :)

Prominent Perl People in 2011

This year we had some well-deserved White Camel Award winners: Leo Lapworth, Daisuke Maki and Andrew Shitov. Congratulations and thanks for making the Perl world better!

There are several others whom also deserve a huge praise for last year's work, amongst them Mark Keating of Shadowcat/EPO/TPF, who is tirelessly working on quality Perl marketing; Gábor Szabó who started the Perl Weekly mailing list and a series of Video Tutorials; and Thiago Rondon, who did huge things for the OpenData initiative, leading us (and Perl) into projects that added a lot of transparency to the Brazilian government and even resulted in a W3C Brazil OpenData Developer's Manual (of which I'm proud to have been a part of) and in winning the Latin America OpenData hackathon.

A couple of new, fun and downright useful websites were born in 2011, including Perl News by Dave Cross, Github-Meets-CPAN by Johannes Plunien; and PrePan, by Kentaro Kuribayashi. Important to notice that learn.perl.org was relaunched with a beautiful look and a lot of updated content. Great job, everyone!

But there are also the silent workers, the ones that are mostly behind the scenes but whose efforts were paramount for the Perl 5 ecosystem. People like Karen Pauley, president of The Perl Foundation; and Barbie, who worked really hard in keeping the incredible CPAN Testers service up and running.

In fact, a *HUGE* thanks are due to everybody involved in CPAN Testers in 2011: Barbie, Dave Golden, Chris Williams, David Cantrell, Slaven Rezić and just about everyone else. I had the great opportunity to help Dave Golden upgrade the cpan-reporter module to use metabase and I could see how intricate the whole thing is. By the way, did you know that in 2011 CPAN Testers crossed the barrier of over 1 million test reports in a single month? That's incredible!

Speaking of which, what would be of Perl 5 without its core developers? Several of them were already mentioned and praised here, but you can check the full list of contributors in the AUTHORS file, or in perldelta for a more recent list. This year we had the pleasure of seeing commits from Father Chrysostomos, Florian Ragwitz, H.Merijn Brand, Karl Williamson, Claudio Ramirez, Vladimir Timofeev, Nobuhiro Iwamatsu and many, many others. Thanks, guys!

Perl Games

The Perl gaming scene got yet another massive overhaul in 2011. Coming from all the hard work Kartik Thakore, Tobias Leich and everyone else at the SDL Perl project put in 2010, game development in Perl has never been this easy or fun.

The SDL Perl Manual was finished early in the year, and we were having so much fun we threw together the SDL Perl Game Contest in march, resulting in a total of 16 new games written from scratch in just one month!

The year also saw the coming of a Box2D wrapper for Perl, letting us add some fast physics to games and simulations.

But the best was yet to come. Construder, a jaw dropping 3D game created by Perl hacker Robin "elmex" Redeker, features futuristic settings with some nice graphics and an (almost) infinite world for you to build and play with. Make sure to check it out if you haven't already!

A Web of Perl

Perl's most widely adopted web frameworks also kept extremely busy, and 2011 marked some pretty cool releases and announcements:

Catalyst 5.9 was released back in august, incorporating Plack as its default Engine. This change benefits Catalyst significantly by reducing the amount of code inside the framework, getting upstream bug fixes in Plack, and automatically gaining support for any web server which a PSGI compliant handler is written for.

Mojolicious saw a total of 122(!) releases, including its 2.0 one, and now features updated websockets support, documentation enhancements and several asynchronous/non-blocking features. You can check out the official Mojolicious 2011 retrospective for the full monty.

The nice folks of Dancer started working on a massive core rewrite that will help the project tremendously. Among the changes being made, there will be no more globals in the core, 100% object-oriented backend, better scoping for sub-applications, and a better design overall.

Advent Calendars

The tradition remains, and the 2011 Perl Advent Calendars are filled with great content ranging from beginner tips to advanced hacks. The Japanese Perl Community once again delivered several high-quality articles in 9 different tracks, while the Brazilian Perl Community scattered their great articles throughout the two months of the Equinox.

Books

O'Reilly has been busy in 2011, with the updated 6th edition of Learning Perl - the Llama - by Randal Schwartz, brian d foy and Tom Phoenix, and Johan Vromans' handy Perl Pocket Reference. They're also getting ready for the 4th edition of Programming Perl - the Camel - to be released in early 2012. Heck, at O'Reilly they even used Perl to pick their Secret Santa :)

Speaking of upcoming titles, Ovid's Beginning Perl and chromatic's Little Plack Book are ones to keep an eye for. Who knows, maybe I'll get to talk a bit more about them in the 2012 wrap-up? I look forward to it!

Rise of The MetaCPAN

If you were one of the early adopters, you know MetaCPAN was actually born on late 2010 (november 3rd, to be precise). The project had some ambitious goals: provide a free and open sourced alternative search engine to the ever-glorious CPAN. But in 2011 it became more. So much more.

Moritz "Mo" Onken, a student at the Karlsruhe Institute of Technology (KIT), applied for the GSoC and quickly became a hero. Sure, he already had several years of Perl background, but what Clinton Gormley, Olaf Alders and himself achieved in 2011 exceeded all expectations and revolutionized the Perl world.

MetaCPAN is not only a sophisticated and fast CPAN search engine, but also offers a full featured REST API that lets you build on top of it making all sorts of complex search queries for extracting data from and about the CPAN. Another (very) important aspect: an active and vibrant community that actually encourages people to send patches fixing bugs and adding new features.

Featured Perl Modules

Wrapping up the retrospective, I should go about some of the cool new modules that spawned in 2011. This is not meant to be a thorough list, just a small snippet for your viewing pleasure:

Mason 2 - The traditional HTML::Mason distribution received a major overhaul and became just "Mason". The new distribution is being very actively developed and has a much more modern architecture. Check out the changes and give it a try!

dip - this nifty new tool offers dynamic instrumentation like DTrace, using aspects. Marcel Grünauer builds on top of Adam Kennedy's awesome Aspect Oriented Programming (AOP) Perl interface (that also reached 1.0 in 2011) to provide a tool that lets you change application's behavior without actually touching the source code. Very nice!

Lucy - David Wheeler released a nice Perl wrapper for Apache Lucy, a high-performance, modular full-text search engine library that assimilated the KinoSearch codebase and community.

Data::Printer - I don't like to talk about stuff I wrote doing general retrospectives, but I'm going to open an exception here. Data::Printer provides a simple and powerful - not to mention, colorful! - way to view your data structures. It's highly customizable and if you ever used Data::Dumper to view variable contents on the screen, you should give it a try :-)

There were also some visible trends on CPAN last year:

Towards a lighter Moose: everybody loves Moose (which also reached 2.0 in 2011!), but sometimes you just want - or think you just want - the sugary OO syntax, not the full-blown object system. Or maybe your particular environment doesn't let you install it, and you still need some small piece of it to make everything better. In 2011 there were a lot of uploads for Moose and Mouse alternatives, including Moo, Mo, and Mite. This definitely shows an itch that needs some scratching.

On a somewhat related note, Stevan Little is developing a proposal and a functioning prototype for a Meta Object Protocol, or MOP, to be perhaps included in a future version of Perl 5. Comments are welcome!

Sysadmin tools: Matt S. Trout took the time to bolt some modules together and bring us Tak, a multi-host remote control over ssh. But he wasn't the only one, and new sysadmin tools and modules sprouted all over the perlverse. Some fine examples include helm, providing easy server and cluster automation, and the great Rex (or, rather, "(R)?ex"), that lets you manage all your boxes from a central point through the complete process of configuration management and software deployment. Check out rexify.org for a quick glimpse of some of its features!

Simple ORMs: another trend in 2011 was to create simple DBI wrappers providing ORM-ish features, resulting in lightweight frameworks somewhere between DBI and DBIx::Class. Among the new distributions are Teng and DBIx::Sunny.

Dependency Managers: Finally, following Ingy's "only" pragma from back in 2003, a lot of effort has been put into making new and improved dependency managers for Perl applications and distributions. Miyagawa's carton and Gugod's perlrocks deserve particular attention.


That's it for this 2011 Perl retrospective. Hope you guys had as much of a nice time reading it as I had writing it. Let 2012 be the year of the Velociraptor!!