A Celebration

Posted on 11th August 2014

August the 16th is CPAN Day. Its the day that is marked as the first upload to CPAN. CPAN came into existence in July/August 1995, but on August 16th 1995 Andreas König made the first true upload to the archive. And so began a growth that is continuing and growing to this day. Jumping ahead to several weeks ago, Neil Bowers decided to make a big deal about this day. After all we celebrate Perl's birthday, why not celebrate CPAN too.

Neil has been posting several ideas for how you can improve distributions on CPAN, with the aim of making several releases on CPAN Day. With that aim in mind, Neil posted some stats and graphs to show what has happened on CPAN Day in previous years. I did some data mining and came up with a page to help monitor CPAN Day. I sent the link to Neil, who then came up with several suggestions. Trying to create the graphs proved interesting, and thanks to everyone on Twitter who sent me various links to help out.

The page has expanded slightly and includes the neoCPANisms, which Neil has been monitoring. NeoCPANism being the number of new distributions that have never been uploaded to CPAN before. It will be interesting to see how many new distributions get released on CPAN Day, as the biggest day of new release was nearly 2 years ago, with 41 new distributions release on the same day.

The page is now created in real time (well every 5 minutes) so you can see how we're progressing throughout the day. The page is now available at stats.cpantesters.org/uploads.html. You can watch progress for each day now, not just CPAN Day, but let's see if we can reach the suggested target on Saturday :)

File Under: cpan / perl / statistics
1 COMMENT


Sunshine Of Your Love

Posted on 17th July 2014

The survey results for YAPC::NA 2014 are now online.

Even we with lower numbers of attendees this year, 27% of you took the time to respond to the survey. As always, this doesn't necessarily allow us to see the whole picture, but hopefully it is enough of a cross-section of the attendees to help us improve future events. Once again we had a healthy number of respondees for whom this was their first YAPC, many having never attendeed a workshop either.

There was a bit of a mixed reaction throughout the survey. Although having read the feedback from the talk evaluations, there was a lot of positive comments, and several words of encouragement for some of the new speakers, which was great to see. Overall it seems to have been another great conference, although there are areas of communication that many felt could be improved.

I see I'll have to expand the options for the question "What other areas of the Perl Community do you contribute to?", as firstly I would include hacking on Perl core, as part of a Perl project (i.e. a group of great people doing great work to improve Perl), but also to include a new option; I donate to one of the funds managed by TPF or EPO. During the conference I saw a few Twitter posts about contributing to some of the Perl funds, which I think came about following Dan Wright's presentation. It is great that so many have donated, big and small amounts, to the various funds. They all help to improve and promote Perl, and give us good reasons to continue putting together great conferences and workshops every year.

It was great to see any good list of suggestions for topics this year, and I hope that speakers new and old, get some ideas for future talks from them.

Lastly it does seem that the location question, really does depend where the current location is. The higher numbers last year may also indicate that Austin was easier to get to for most people, whereas a more easterly location, such as Florida, may restrict the ability to attend for those on the west coast. It would be interesting to see whether a similar opposite trend would result if the conference was held in Nevada, California, Oregon, Idaho, Utah or Arizona. There must be several Perl Monger groups in those states, so if you're in one, perhaps think about balancing out the number of eatern hosting states ;)

File Under: community / conference / perl / yapc
4 COMMENTS


100 Nights

Posted on 13th July 2014

100 in more ways than one!

100 #1

11 years ago I was eager to be a CPAN Author, execpt I had nothing to release. I tried thinking of modules that I could write, but nothing seemed worth posting. Then I saw a post on a technical forum, and came up with a script to give the result the poster was looking for. Looking at the script I suddenly realised I had my first module. That script was then released as Calendar::List, and I'm pleased to say I still use it today. Although perhaps more importantly, I know of others who use it too.

Since then, I have slowly increased my distributions to CPAN. However, it wasn't until I got involved with CPAN Testers that my contributions increased noticeably. Another jump was when I wrote some WWW::Scraper::ISBN driver plugins for the Birmingham Perl Mongers website to help me manage the book reviews. I later worked for a book publishing company, during which time I added even more. My next big jump was the release of Labyrinth.

In between all of those big groups of releases, there have been several odds and ends to help me climb the CPAN Leaderboard. Earlier this year, with the idea of the Monthly New Distribution Challenge, I noticed I was tantalisingly close to having 100 distributions on CPAN. I remember when Simon Cozens was the first author to achieve that goal, and it was noted as quite an achievement. Since then Adam Kennedy, Ricardo Signes and Steven Haryanto have pushed those limits even further, with Steven having over 300 distributions on CPAN!

My 100th distribution came in the form of an addoption, Template-Plugin-Lingua-EN-Inflect, originally written by the sadly departed Andrew Ford.

100 #2

My 100th distribution came a few days before I managed to complete my target of a 100 consecutive days of CPAN uploads. A run I started accidentally. After the 2014 QA Hackathon, I had several distribution releases planned. However, had I realised what I could be doing, I might have been a bit more vigilant and not missed the day between what now seems to be my false start and the real run. After 9 consecutive days, I figured I might as well try to reach at least a month's worth of releases, and take the top position from ZOFFIX (who had previously uploaded for 27 consecutive days) for the once-a-day CPAN regular releasers.

As it happened, Neil Bowers was on a run that was 1 day behind me, but inspired by my new quest, decided he would continue as my wingman. As I passed the 100 consecutive day mark, Neil announced that he was to end his run soon, and finally bowed out after 111 days of releases. My thanks to Neil for sticking with me, and additionally for giving me several ideas for releases, both as suggestions for package updates and a few ideas for new modules.

I have another quest to make 200 releases to CPAN this year, and with another 20 release currently planned, I'm still continuing on. We'll see if I can make 200, or even 365, consecutive days, but reaching 100 was quite a milestone that I didn't expect to achieve.

100 #3

As part of my 100 consecutive days of CPAN uploads challenge, I also managed to achieve 100 consecutive days of commits to git. I had been monitoring GitHub for this, and was gutted to realise that just after 101 days, I forgot to commit some changes over that particular weekend. However, I'm still quite pleased to have made 101 days. I have a holiday coming up soon, so I may not have been able to keep that statistic up for much longer anyway.

100 #4

As part of updates to the CPAN Testers Statistics site, I looked at some additional statistics regarding CPAN uploads. In particular looking at the number of distributions authors have submitted to CPAN, both over the life of CPAN (aka BackPAN) and currently on CPAN. The result was two new distributions, Acme-CPANAuthors-CPAN-OneHundred and Acme-CPANAuthors-BACKPAN-OneHundred.

When I first released the distributions, I only featured in the second. For my 100th consecutive day, I released the latest Acme-CPANAuthors-CPAN-OneHundred up to that day, and with my newly achieved 100th distribution, was delighted to feature in the lists for both distributions.

File Under: opensource / perl
NO COMMENTS


Time Waits For No One

Posted on 10th May 2014

When I relaunched the CPAN Testers sites back in 2008, I was in a position to be responsible for 3 servers, the CPAN Testers server, the Birmingham Perl Mongers server, and my own server. While managing them wasn't too bad, I did think it would be useful having some sort of monitoring system that could help me keep an eye on them. After talking to a few people, the two key systems most keenly suggested were Nagios and Munin. Most seemed to favour Munin, so I gave it a go. Sure enough it was pretty easy to set up, and I was able to monitor the servers, using my home server to monitor them. However, there was one area of monitoring that wasn't covered. The performance of the websites.

At the time I had around 10-20 sites up and running, and the default plugins didn't provide the sort of monitoring I was looking for. After some searching I found a script written by Nicolas Mendoza. The script not only got me started, but helped to make clear how easy it was to write a Munin plugin. However, the script as was, didn't suit my needs exactly, so had to make several tweaks. I then found myself copying the file around for each website, which seem a bit unnecessary. So I wrote what was to become Munin::Plugin::ApacheRequest. Following the Hubris and DRY principles copying the script around just didn't make sense, and being able to upgrade via a Perl Module on each server, was far easier than updating the 30+ scripts for the sites I now manage.

Although the module still contains the original intention of the script, how it does it has changed. The magic still happens in the script itself.

To start with an example, this is the current script to monitor the CPAN Testers Reports site:

#!/usr/bin/perl -w
use Munin::Plugin::ApacheRequest;
my ($VHOST) = ($0 =~ /_([^_]+)$/);
Munin::Plugin::ApacheRequest::Run($VHOST,1000);

Part of the magic is in the name of the script. This one is 'apache_request_reports'. The script extracts the last section of the name, in this case 'reports', and passes that to Run() as the name of the virtual host. If you wish to name the scripts slightly differently, you only need to amend this line to extract the name of your virtual host as appropriate. If you only have one website you may wish to name the host explicity, but then if you create more it does mean you will need to edit each file, which is what I wanted to avoid. All I do now is copy an existing file to one to represent the new virtual host when I create a new website, and Munin automatically adds it to the list.

Munin::Plugin::ApacheRequest does make some assumptions, one of which is where you locate the log files, and how you name them for each virtual host. On my servers '/var/www/' contains all the virtual hosts (/var/www/reports, in this example), and '/var/www/logs/' contains the logs. I also use a conventional naming system for the logs, so '/var/www/logs/reports-access.log' is the Access Log for the CPAN Testers Reports site. Should you have a different path or naming format for your logs, you can alter the internal variable $ACCESS_LOG_PATTERN to the format you wish. Note that this is a sprintf format, and the first '%s' in the format string is replaced by the virtual host name. If you only have one website, you can change the format string to the specific path and file of the log, and no string interpolation is done.

The log format used is quite significant, and when you describe the LogFormat for your Access Log in the Apache config file, you will need to use an extended format type. The field to show the time taken to execute a request is needed, which is normally set using the %T (seconds) or %D (microseconds) format option (see also Apache Log Formats). For example my logs use the following:

LogFormat "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-agent}i\" %T %v"

The second to last field is our time field. In Munin::Plugin::ApacheRequest, this is stored in the $TIME_FIELD_INDEX variable. By default this is -2, assuming a similar log format as above. If you have a different format, where the execution time is in another position, like $ACCESS_LOG_PATTERN, you can change this in your script before calling Run(). A positive number assumes a column left to right, while a negative number assumes a column right to left.

The last number passed to the Run() method, determines the number of lines read for the access log to describe the average execution time. For high hit rate sites, you may wish this to be a higher number, but as most of my sites are not that frequently visited, 1000 seems to be a reasonable number.

The config statements that are generated for the Munin master monitor are currently hardcoded with values. This will change in a future version. For the example above the config produced reads as:

graph_title reports ave msecs last 1000 requests
graph_args --base 1000
graph_scale no
graph_vlabel Average request time (msec)
graph_category Apache
graph_info This graph shows average request times for the last 1000 requests
images.warning 30000000
images.critical 60000000
total.warning 10000000
total.critical 60000000

The highlighted values are interpolated from the arguments passed to Run(). In a future version I want to be able to allow you to reconfigure the warning and critical values and the graph base value, should you wish to.

I have now been using Munin::Plugin::ApacheRequest and the associated scripts for 6 years now, and it has proved very successful. I have thought about releasing the module to CPAN previously, and have made several attempts to contact Nicolas over the years, but have never had a reply. I know he was working for Opera when he released his script, but have no idea of his whereabouts now. As the script contained no licensing information, I was also unsure what licensing he had intended the code to be. I hope he doesn't mind me having adapted his original script, that I'm now releasing the code under the Artistic License v2.

Although I haven't been able to contact Nicolas, I would like to thank him for releasing his original script. If I hadn't have found it, it is unlikely I would have found a way to write a Munin plugin myself to do Apache website monitoring. With his headstart, I discovered how to write Munic plugins, and can now set up monitor of new websites within a few seconds. Thanks Nicolas.

File Under: opensource / perl / website
1 COMMENT


Counting Out Time

Posted on 20th March 2014

I had an SQL query I wanted to translate into a DBIx::Class statement. I knew there must be a way, but trying to find the answer took some time. As a result I though it worth sharing in the event somebody else might be trying to find a similar answer.

The SQL I was trying to convert was:

SELECT status,count(*) AS mailboxes,
count(distinct username) AS customers
FROM mailbox_password_email GROUP BY status

The result I got running this by hand gave me:

+-----------+-----------+-----------+
| status    | mailboxes | customers |
+-----------+-----------+-----------+
| active    |     92508 |     48791 |
| completed |       201 |       174 |
| inactive  |    116501 |     56843 |
| locked    |    129344 |     61220 |
| pending   |      1004 |       633 |
+-----------+-----------+-----------+

My first attempt was:

my @rows = $schema->resultset('Mailboxes')->search({},
    {
        group_by => 'status',
        distinct => 1,
        '+select' => [
            { count => 'id', -as => 'mailboxes' },
            { count => 'username', -as => 'customers' } ]
    })->all;

Unfortunately this gave me the following error:

DBIx::Class::ResultSet::all(): Useless use of distinct on a grouped 
resultset ('distinct' is ignored when a 'group_by' is present) at 
myscript.pl line 469

So I took the 'distinct  => 1' out and got the following results:

+-----------+-----------+-----------+
| status    | mailboxes | customers |
+-----------+-----------+-----------+
| active    |     92508 |     92508 |
| completed |       201 |       201 |
| inactive  |    116501 |    116501 |
| locked    |    129344 |    129344 |
| pending   |      1004 |      1004 |
+-----------+-----------+-----------+

Which might be distinct for the mailboxes, but is not sadly distinct for customers. So I try:

my @rows = $schema->resultset('Mailboxes')->search({},
{
group_by  => 'status',
      '+select' => [
            { count => 'id', -as => 'mailboxes' },
            { count => 'username', -as => 'customers', distinct  => 1 } ]
    })->all;

and get:

Failed to retrieve mailbox password email totals: 
DBIx::Class::ResultSet::all(): Malformed select argument - too many keys
 in hash: -as,count,distinct at myscript.pl line 469\n

After several attempts at Google, and reading the DBIx::Class::Manual, I finally stumbled on: SELECT COUNT(DISTINCT colname)

My query now looks like:

my @rows = $schema->resultset('Mailboxes')->search({},
    {
group_by  => 'status',
'+select' => [
{ count => 'id', -as => 'mailboxes' },
{ count => { distinct => 'username' }, -as => 'customers' } ] })->all;

And provides the following results:

+-----------+-----------+-----------+
| status    | mailboxes | customers |
+-----------+-----------+-----------+
| active    |     92508 |     48791 |
| completed |       201 |       174 |
| inactive  |    116501 |     56843 |
| locked    |    129344 |     61220 |
| pending   |      1004 |       633 |
+-----------+-----------+-----------+

Exactly what I was after.

DBIx::Class does require some head-scratching at times, but looking at the final statement it now seems obvious, and pretty much maps directly to my original SQL!

Hopefully, this provides a lesson others can find and learn from.

File Under: database / perl
NO COMMENTS


Rendez-Vous 6

Posted on 17th March 2014

My 2014 QA Hackathon

Day One

I arrived the previous day, as did most of us, and we naturally talked about coding projects. Not necessarily about work at the hackathon, but discussion did come around to that too. I talked with Tux at one point, who convinced me that a stand-alone smoker client would be really useful. Once upon a time, we did have this, but with the advent of the more sophisticated smokers, and the move to the Metabase transport layer, the old script never got updated. The following morning Tux sent me a copy of the script he has, so at some point over the next few months I will take a look to see what I do to make it compatible with the modern smokers.

My intention was to release a distribution each day of the Hackthon. Unfortunately this was scuppered on the first day, when trying to add support for the full JSON report from CPAN Testers, when I realised I don't store the full report in the database. In the future when we have MongoDB and replication set up, this will be a non-issue, but for the moment, I now need to store the full report. This now requires a change to the metabase database on the cpanstats server (as opposed to the Metabase server). Over the course of the hackthon I reviewed the changes needed, and updated a lot of the Generator code, as it was an ideal time to remove SQLite references too.

In looking into the code changes, Andreas and I again looked at the updated timestamp used by the various CPAN Testers sites to do statistical analysis, which was also causing us problems. In the Metabase, the CPAN Testers Report fact is the container for all the child facts, such as LegacyReport and TestSummary. When the facts are created by the tester, the 'creation' timestamp is used to reference the time on the tester's own server that the report was generated. This could be better stored as UTC, but that's a problem for another day. However, it does mean the timestamp could be different to the one on the Metabase server. When the Metabase server retrieves the report from the outside world, it updates the 'updated' timestamp across all facts and saves into the SimpleDB instance on the server. Except it doesn't. The 'updated' timestamp is always the same as the 'creation' timestamp. Andreas has been noting this for quite some time, and finally he convinced me, at which point we roped in David Golden to take a look. Reviewing the code, there is nothing wrong that we can see. The 'updated' timestamp should be updated with the current timestamp on the Metabase server, which should also cascade to each child fact. As such you would expect several reports to have a different 'creation' timestamp from that of the 'updated' timestamp, even if only different by a second. Sadly this is going to take more effort/time to debug, as David in particular is working several different aspects of QA here at the hackathon.

Towards the end of the day, I spoke with liz and Tobias (FROGGS) about how CPAN Testers might handle perl6 modules. Currently there is no client available, but there could be in the future. However, due to the way Perl6 modules are to be uploaded to CPAN it is possible that smokers may submit reports for perl6 only modules, as many ignore the path to the distribution. Right now, liz tells me that all perl6 modules are being release under the /perl6/ path inside the authors' directory. This makes things easier for CPAN Testers as we can initially ignore these test reports, as they will not be valid. However, in the longer term it will be interesting to have a CPAN Testers smoker client for Perl6. The main difference would be to record in the metadata that it's a perl6 only distribution, and we *should* be able to carry on as normal, submitting reports to the Metabase, etc. It may require some distributions to have a 'Did you mean the Perl 6 distribution?' link on the website, but for the most part I think we could handle this. It will require further work to define a CPAN Testers Perl 6 Fact, but it will be a nice addition to the family.

Day Two

The morning was spent visiting the Charteuse cellars, and enjoying a tasting session, before heading back to the hacking in the afternoon.

In the afternoon, I started to look at some of the statistics the CPAN Testers Statistic site generated. After some discussions with Neil Bowers, he was interested in the drop-off of report submissions when a distribution was released. I believed this to be fairly consistent, and found that it did indeed last roughly 8 days, with a tail off that could last for months or years. There was an initial blast of tests within the first few hours, thanks to Chris' and Andreas' smokers, but the rest of the more reliable smokers get submitted within those first 8 days. Neil has created some initial graphs, and I'm looking at ways to integrate those with the Reports site. How we display these will likely revolve around a specific selected version, as overlaying versions might be a bit too much ... we'll see.

It also led me to think about what time of day do testers submit reports. So, I'll be looking at creating some graphs to show submissions per month, per day of the week, and per hour of the day. Along with BooK, we discussed further metrics, although they look likely to be used within their CPAN Dashboard project, although some of the data can be provided by CPAN Testers APIs already, so little work need by me :)

Looking through aggregated data, as stored and indexed within the Statistics codebase, it was obvious some had were now incomplete. It seems some of the outages we had in the last few months, prevented the data storage files from being saved. As such, I started off a complete reindex. It meant the Statistics site was out of sync for the following day, but at least it meant we once had again had correct data to produce the graphs we wanted.

There was more work rewriting the Generator to store the report objects. Yves asked why I wasn't using Sereal sometime ago, when I posted about using Data::FlexSerializer, and at the time I didn't have a need to rework the code. However, seeing as I'm rewriting to store the perl object now, rather than just JSON, it does make sense to move to Sereal, so hopefully that will make Yves happy too ;)

Day Three

Continued work on the Generator to remove all SQLite references, and a few further clean ups. Also worked on adding the necessary support to allow perl6 reports to be ignored. At some point in the future we will accept perl6 reports, but following further discussion with Tobias, we'll handle this using the metadata in the report not on the path of the resource.

Salve interviewed me for a future post about CPAN Testers. It'll be interesting to see whether I made sense or not, but hopefully I managed to convey the usefulness and uniqueness of CPAN Testers to Perl and the community. It good opportunity to also thanked Salve for starting the QA Hackathons, as without them CPAN Testers may well have stalled several years ago. Like many other projects, if we had relied on email to handle all the discussions and move the project forward, it would have taken years to get the Metabase working and move away from the old email/NNTP mechanisms.

charsbar updated CPANTS with some altered metrics, and at the same time added selected CSS colours for BooK and Leon, so I asked too. I now have a shade of my own purple on my author page ;) Thanks charsbar.

As Wendy went to lunch, she made the mistake of asking whether we wanted anything. I asked for a Ferrari, but sadly they couldn't find one, so I got a Lambourgini instead. If you don't ask, you don't get .... vroom, vrooom, vroom :) I'll add a picture once I've sorted them out.

At some point during the afternoon, Ricardo told me one of his asks for the hackathon. He wanted to be able to ignore the NA reports in his No Pass RSS feeds. Mulling it over this seemed entirely sensible, and so I fixed it. Ricardo celebrated :)

During a discussion with Neil, he mentioned that Paul Johnson was creating a Devel::Cover service, that he wanted to run like a CPAN Testers service. The idea was to write a system, that could allow distributed testing with testers sending in reports, which could then be accumulated, based on the OS being tested. As the Metabase is already able to handle different buckets, adding another bucket for coverage reports simplifies some of the work. The distributed client can then be moduled on the CPAN Testers means of report contruction, creating a new coverage report fact and use the same transport mechanism to submit to the Metabase. A web service can then poll the Metabase for the new bucket, and create report pages in exactly the same way as CPAN Testers. It'll be interesting to see whether we can use the same (or similar) code to provide this.

Day Four

The morning threw us a curve-ball, as the building wouldn't open up. It was a Sunday and apparently no-one works on a Sunday. Thankfully a few phonecalls to the right people got us in, just in time for lunch. In the meantime as we all were staying in the same hotel, we took over the bar, and borrowed a conference for the morning.

The poor wifi connection, gave us a good opportunity to have further discussions. Neil gathered together several interested parties to discuss author emails. Both PAUSE and CPAN Testers send emails to authors, and there is a plan to send authors a yearly email to advertise improvements to their modules, and let them know about sites and tools that they might not be aware of. However, although many emails get through without a problem, several fail to reach their intended recipient. Typically this is because authors have changed their email address but failed to update the email stored within the PAUSE system. CPAN Testers highlights some of these Missing In Action authors, but it would be better to have an automated system. Also, as Ricardo noted, the envelope of an email is left unchanged when is sent to the develooper network, so bouncebacks come back to the original sender containing the authors' potenmtially secret email address. It would be much better to have a service that monitors bouncebacks, but change the envelope to return to the handling network and can send an appropriate email to the sender. It could then provide an API to enable PAUSE and CPAN Testers, and any future system, to know whether compiling an email was worth the effort. For CPAN Testers there can be a great deal of analysis to prepare the summary emails, so knowing in advance an author email is not going to get through would be very beneficial. Neil is going to write up the ideas, so we can more formally design a system that will work all of PAUSE related systems. CPAN Testers already has the Preferences site to allow authors to manage their summary emails, and also turn off receiving any emails, and it may be worth extending this to PAUSE or other system to provide a subscription handling system.

The rest of the day was mostly spent monitoring the metabase table in the cpanstats database, as the new 'fact' column was added. The new field will store the reports from the parent in Sereal. I was a bit worried about locking the table all day, but no-one seemed to notice. While this was happening, I started back on the original new module I started on the first day of the conference,and had hoped to release. However, it highlighted further problems with the way reports are stored. I'm not sure what is doing it, but the underlying fact.content field in JSON was being stored as a string. In most cases this isn't a problem, however for this module it caused problems trying to encode/decode the JSON. After fixing the Generator code, it means the new module still didn't get finished. Well at least I have something to start my neocpanism.once-a-week.info stint with :)

Wrap Up

I now have several pieces of work to continue with, some for a few months to come, but these 4 days have been extremely productive. Despite playing with the CPAN Testers databases rather than writing code, the discussions have been invaluable. Plus it's always great to catch up with everyone.

This year's QA Hackthon was great, and it wouldn't have been possible without BooK and Laurent organising it, Wendy keeping us eating healthily (and in good supply of proper English tea ... I'll try and remember to bring the PG Tips next time), Booking.com for supplying the venue and all the other sponsors for helping to make the QA Hackathon the great success it was. In no particular order, thanks to Booking.com, SPLIO, Grant Street Group, DYN, Campus Explorer, EVOZON, elasticsearch, Eligo, Mongueurs de Perl, WenZPerl for the Perl6 Community, PROCURA, Made In Love and The Perl Foundation.

Looking forward to 2015 QA Hackathon in Berlin.

File Under: hackathon / perl / qa
NO COMMENTS


History Of Modern (part I)

Posted on 23rd February 2014

Neil Bowers recently unleashed CPAN::ReleaseHistory on the world. Internally the distribution uses the a BACKPAN Index, which records every release to CPAN. I was already interested in this kind of representation, as I wanted to add a similar metric on each Author page of the CPAN Testers Reports website, but hadn't got around to it. Neil then posted about the script included in the distribution, cpan-release-counts in an interesting post; What's your CPAN release history?.

After a quick download, I ran the following for myself:

barbie@kmfdm:~$ cpan-release-counts --char = --width 30 --user barbie
 2003 ( 12) ==
 2004 ( 26) =====
 2005 ( 80) ===============
 2006 (  6) =
 2007 ( 59) ===========
 2008 ( 62) ===========
 2009 (122) =======================
 2010 (148) ============================
 2011 ( 89) =================
 2012 (156) ==============================
 2013 (123) =======================
 2014 ( 11) ==

So my most prolific year was in 2012. I'll have to see if I can change that this year. However, it does give a nice yearly snapshot of my releases.

As it turns out, for CPAN Testers I don't need the BACKPAN index, as I already generate and maintain an 'uploads' table within the 'cpanstats' database. I do need to write the code to add this metric to the Author pages. Thanks to Neil's script though, he has given me a starting point. Being able to see the releases for yourself (or a particular Author) is quite cool, so I may adapt that to make any such matrix more dynamic. It might also be worth adding a more generic metric for all of CPAN to the CPAN Testers Statistics website. Either way, I now have two more things to add to my list of projects for the QA Hackathon next month. Neil will be there too, so I hope he can give me even more ideas, while I'm there ;)

File Under: hackathon / opensource / perl
NO COMMENTS


Even Flow

Posted on 8th December 2013

The following is part of an occasional series of highlighting CPAN modules/distributions and why I use them. This article looks at Data::FlexSerializer.

Many years ago the most popular module for persistent data storage in Perl was Storable. While still used, it's limitations have often cause problems. It's most significant problem was that each version was incompatible with another. Upgrading had to be done carefully. The data store was often unportable, and made making backups problematic. In more recent years JSON has grown to be more acceptable as a data storage format. It benefits from being a compact data structure format, and human readable, and was specifically a reaction to  XML, which requires lots of boilerplate and data tags to form simple data elements. It's one reason why most modern websites use JSON for AJAX calls rather than XML.

Booking.com had a desire to move away from Storable and initially looked to moving to JSON. However, since then they have designed their own data format, Sereal. But more of that later. Firstly they needed some formatting code to read their old Storable data, and translate into JSON. The next stage was to compress the JSON. Although JSON is already a compact data format, it is still plain text. Compressing a single data structure can reduce the storage by as much as half the original data size, which when you're dealing with millions of data items can be considerable. In Booking.com's case they needed to do this with zero downtime, running the conversion on live data as it was being used. The resulting code was to later become the basis for Data::FlexSerializer.

However, for Booking.com they found JSON to be unsuitable for their needs, as they were unable to store Perl data structures they way they wanted to. As such they created a new storage format, which they called Searal. You can read more about the thoughts behind the creation of Sereal on the Booking.com blog. That blog post also looks at the performance and sizes of the different formats, and if you're looking for a suitable serialisation format, Sereal is very definitely worth investigating.

Moving back to my needs, I had become interested in the work Booking.com had done, as within the world of CPAN Testers, we store the reports in JSON format. With over 32 million reports at the time (now over 37 million), the database table had grown to over 500GB. The old server was fast running out of disk space, and before exploring options for increasing storage capacity, I wanted to try and see whether there was an option to reduce the size of the JSON data structures. Data::FlexSerializer was an obvious choice. It could read uncompressed JSON and return compressed JSON in milliseconds.

So how easy was it to convert all 32 million reports? Below is essentially the code that did the work:

  my $serializer = Data::FlexSerializer->new( detect_compression => 1 );

    for my $next ( $options{from} .. $options{to} ) {
        my @rows = $dbx->GetQuery('hash','GetReport',$next);
        return  unless(@rows);

        my ($data,$json);
        eval {
            $json = $serializer->deserialize($rows[0]->{report});
            $data = $serializer->serialize($json);
        };

        next  if($@ || !$data);

        $dbx->DoQuery('UpdateReport',$data,$rows[0]->{id});
    }

Simple, straighforward and got the job done very efficiently. The only downside was the database calls. As the old server was maxed out on I/O, I could only run the script to convert during quiet periods as the CPAN Testers server would become unresponsive. This wasn't a fault of Data::FlexSerializer, but very much a problem with our old server.

Before the conversion script completed, the next step was to add functionality to permanently store reports in a compressed format. This only required 3 extra lines being added to CPAN::Testers::Data::Generator.

  use Data::FlexSerializer;

    $self->{serializer} = Data::FlexSerializer->new( detect_compression => 1 );

    my $data = $self->{serializer}->serialize($json);

The difference has been well worth the move. The compressed version of the table has reclaimed around 250GB. Because MySQL doesn't automatical free the data back to the system, you need to run the optimize command on a table. Unfortunately, for CPAN Testers this wouldn't be practical as it would mean locking the database for far too long. Also with the rapid growth of CPAN Testers (we now receive over 1 million reports a month) it is likely we'll be back up to 500GB in a couple of years anyway. Now that we've moved to a new server, our backend hard disk is 3TB, so has plenty of storage capacity for several years to come.

But I've only scratched the surface of why I think Data::FlexSerializer is so good. Aside from its ability to compress and uncompress, as well as encode and decode, at speed, it is ability to switch between formats is what makes it such a versatile tool to have around. Aside from Storable, JSON and Sereal, you can also create your own serialisation interface, using the add_format method. Below is an example, from the module's own documentation, which implements Data::Dumper as a serialsation format:

    Data::FlexSerializer->add_format(
        data_dumper => {
            serialize   => sub { shift; goto \&Data::Dumper::Dumper },
            deserialize => sub { shift; my $VAR1; eval "$_[0]" },
            detect      => sub { $_[1] =~ /\$[\w]+\s*=/ },
        }
    );
    
    my $flex_to_dd = Data::FlexSerializer->new(
      detect_data_dumper => 1,
      output_format => 'data_dumper',
    );

It's unlikely CPAN Testers will move from JSON to Sereal (or any other format), but if we did, Data::FlexSerializer would be only tool I would need to look to. My thanks to Booking.com for releasing the code, and thanks to the authors; Steffen Mueller, Ævar Arnfjörð Bjarmason, Burak Gürsoy, Elizabeth Matthijsen, Caio Romão Costa Nascimento and Jonas Galhordas Duarte Alves, for creating the code behind the module in the first place.

File Under: database / modules / opensource / perl
2 COMMENTS


The Great Gates of Kiev

Posted on 27th October 2013

I've now uploaded the survey results for YAPC::Europe 2013 and The Pittsburgh Perl Workshop 2013. Both had only a third of attendees respond, which for PPW is still 20 out of 54, and 122 out of 333 for YAPC::Europe.

YAPC::Europe

In previous years we have had higher percentages of response at YAPC::Europe, but that is possibly because I was in attendance and promoted the surveys during lightning talks, and encouraged other speakers to remind people about them. It may also be the fact that there is a newer crowd coming to YAPCs, and the fact we had 44 out of the 122 respondees saying that this was their first YAPC, who have never experienced the surveys. While definitely encouraging to see newer attendees, it would be great to see more of their feedback to help improve the conferences each year. Like YAPC::NA 2013, we have reintroduced the gender question. This time around I didn't get the negative reaction, but this may also be due to the fact I've had more feedback about approaching the subject this time around. Perhaps unsurprisingly, there were rather more male respondees, but I am also very encouraged to see that 8 respondees were female. While its difficult to know the exact numbers at the event, I'd like to think that we have been able to welcome more women to the event, and hopefully will see this number increase in the future.

Looking at the locations where attendees were travelling from to attend YAPC::Europe in Kiev, it is interesting to see a much more diverse spread. Once upon a time the UK was often the highest number, even eclipsing the host country. This year, it seems many more from across the whole of Europe took advantage of the conference. Again I think this is very encouraging. If Perl is to grow and reach newer (and younger) audiences, it needs to be of interest to a large number of people, particular from many different locations. While the UK (particularly London, thanks to Dave Cross) was perhaps the start of European Perl community, YAPC::Europe is now capable of being hosted in just about any major European city and see several hundred people attend. It will be interesting to see if Sofia next year, has a similar evenly spread of locations.

Of those that responded, it does seem that we had more people in the advanced realm. Particularly seeing as we had 56 people respond with more than 10 years experience of Perl. Back when we started the surveys, it would likely have been only a handful of people who attended who could have said that they had been programming Perl for more than 10 years. Thankfully though, it isn't just us old hands, as those only programming in Perl for a few years or less, are still making it worthwhile for speakers to come back each year and promote their projects big and small to a new audience.

One comment in the feedback however, described the Perl community as hermetic. I'm not entirely convinced that's true, but it is quite likely that some find it difficult to introduce themselves and get involved with projects. Having said that, there are plenty of attendees who have only been coming to YAPCs, or been involved with the Perl community, for a short while, who have made an impact, and are now valued contributors. So I guess it may just be down to having the right personality to just get stuck in and introduce yourself. This is one area of the Perl community that Yaakov Sloman is keen to break down barriers for, even perceived ones. We do need more Yaakov's at these events to not just break the ice, but shatter it, so we all see the benefit of getting know each other better.

And talking of getting to know others better, it was a shame I didn't get to meet the 15 CPAN Testers who responded. We have had group photos in the past, and I'd like to do more when I next attend a YAPC, but I think it would also be very worthwhile if the Catalyst, Dancer, Padre and many other projects could find the time to do some group shots while at YAPCs. At YAPC::NA it is a bit of a tradition for all those who contribute to #perl on IRC to have a large group photo, but it's never encouraged others to do the same. Perhaps this is also a way for people to get to know project contributors better, as new attendees will have a better idea of who to look out for, rather than trying to figure out who fits an IRC nick or PAUSEID.

The suggest topics for future talks were quite diverse, and "Web Development Web Frameworks Testing" is definitely an interesting suggestion, particularly as we are seeing more and more web frameworks written in Perl now, and we are after all very well known for our testing culture. One question I'm planning to include next years surveys, also looks at some of these topics and attempts to find out what primary interests people have. Again, this might help guide future speakers towards subjects that are of interest to their target audience.

Pittsburgh Perl Workshop

Workshops, by their very nature, are much smaller events, but with Pittsburgh being the home of the very first YAPC::NA, it is well established to host a workshop, and it would seem attracted some high profile speakers too. Possibly as a consequence, at least one attendee felt some of the talks were a little too advanced for them. At a smaller technical event it is much harder to try and please everyone, and with fewer tracks there often is less diversity. Having said that, I hope that the attendee didn't feel too overwhelmed, and got something out of the event in other talks.

From the feedback it would seem that more knowledgeable Perl developers were in attendance, so understandable that more talks might lean towards more advanced subjects, but as mentioned for YAPCs, speakers shouldn't feel afraid of beginner style introductions or howtos for their project, that could appeal to all levels of interest.

Overall I think the Pittsburgh Perl Workshop went down very well.

What's Next?

I now have to compile the more detailed personal feedback for these and the YAPC::NA organisers, so expect to see some further documentation updates in the near future. In addition, I want to work more on the raw data downloads. While it's interesting to see the data as currently presented, others may have other ideas to interrogate the raw data for further interesting analysis. I also still need to put the current code base on CPAN/GitHub and add the features to integrate with Act better.

The next survey will be for the London Perl Workshop at the end of November. If you are planning a workshop, YAPC or other technical event that you'd to have a survey for, please let me know and I'll set you up. It typically takes me a weekend to set up an instance, so please provide as much advanced warning as possible.

File Under: community / conference / perl / survey / workshop / yapc
NO COMMENTS


Of All The Things We've Made

Posted on 26th August 2013

Several years ago, we frequently updated the Birmingham.pm website with book reviews. To begin with, updating all the book information was rather labourious. Thankfully, on CPAN there was a set of modules that had been written by Andrew Schamp, that provided the framework to search online resources. I then wrote drivers for Amazon, O'Reilly & Associates, Pearson Education and Yahoo!. As the books we were reviewing were technical books, these four sources were able to cover all the books we reviewed.

A few years ago, I started working for a book company. In one project, we needed to evaluate book data, particularly for books where we had no data or very little. Often these were imports or out of stock titles that we could still order, but we were lacking information about. As such I created a number of further drivers, particularly for non-UK online catalogues, to help retrieve this information. I managed to create a collection of 17 drivers, and 1 bundle, all available on CPAN.

Via my CPAN Testers work, I've been promoting the CPAN::Changes Kwalitee Service website. Neil Bowers read one of the posts, and thought it would be good to improve the Changes files in his distributions, by way of QuestHub. I'd not heard of this site before, but after reading Neil's post I joined up, as I had been looking for a suitable way to keep a TODO list of my Perl work for a while. Neil had created a stencil to standardise the Changes file in 5 distributions, but unfortunately, I only had a few distributions of my own to complete. Another stencil emerged to add License and Repository information to 5 CPAN distributions. Again, I'd completed this for most of my distributions, apart from my 18 ISBN distributions, which I'd never got around to creating repositories for.

Then Neil had the idea to look at some of the quality aspects of all the CPAN distributions, and highlight those that might need adoption. As part of his reviews of similar modules over the past few years, he's adopted several modules, and was looking at what others he could help with. The results included 2 of the modules written by Andrew Schamp, which formed part of the ISBN searching framework I used for my ISBN distributions. Seeing as they hadn't been touched in eight years, I suspected that Andrew had moved on to other languages or work. So I contacted him to see whether he was interested in letting me take the modules on and update them.

It turns out that Andrew had written the modules for a college project, and since moving to C and with his programming interests now nothing to do with books, he was happy to hand over the keys to the modules. Over the past week, I have now taken ownership of Andrew's 5 modules, added these and my own 18 ISBN distributions to my local git repository, added all 23 to GitHub, updated the Changes file, and License & Repository info to the 5 new modules and released them all to CPAN. My next task is to update the Repository info in my 18 ISBN distributions and release these to CPAN.

Although I don't work in the book industry anymore, writing these search drivers has been fun. The distributions are perhaps my most frequently releases to CPAN, due to the various websites updating their sites. Now that I have access to the core modules in the framework, I plan to move some of the repeated code across many of the drivers into the core modules. I also plan to merge the three main modules into one distribution. When Andrew originally wrote the modules, it wasn't uncommon to have 1 module per distribution. However, as all three are tightly bound together, it doesn't make much sense to keep them separate. The two drivers Andrew wrote have not worked for several years, as unsurprisingly the websites have changed in the last 8 years. I've already updated one, and will be working on the other soon.

It's nice to realise that a few of my CPAN Testers summary posts inspired Neil, who in turn has inspired me, and has ended up with me helping to keep a small corner of CPAN relevant and up to date again.

If you're a new Perl developer, who wants to take a more active role in CPAN and the Perl community, a great way to start is to look at the stencils on QuestHub, and help to patch and submit pull/RT requests to update distributions. If you feel adventurous, take a look at the possible adoption list, and see whether anything there is something you'd like to fix and bring up to date. You can also look at the failing distributions lists, and see whether the authors would like help with the test suites in their distributions. You can then create your tasks as quests in QuestHub and earn points for your endeavours. Be warned though, it can become addictive :)

There is one more ISBN distribution on the adoption list, and I have now emailed the author. Depending on the response, I may be going through the adoption process all over again :) [Late update, the author came back to me and he's happy for me to take on his distribution too]

File Under: isbn / opensource / perl
NO COMMENTS


Who Knows Where The Time Goes?

Posted on 24th July 2013

YAPC::NA 2013 - The Results Are Out

The YAPC::NA 2013 Conference Survey results are now online.

The number of responses was much lower than in previous years, which is a shame, but may in part be due to one comment I received, saying it was too long. Reviewing the survey, I'd have to agree, and I'll be removing some of the questions for future surveys. Some of the questions had good intentions originally, and did provide an insight to what people got out of the conference. However, there is now a degree of predictability about them, that doesn't warrant their inclusion. Such questions about holidays and speakers you missed really don't add anything any more. The latter has generated some interesting comments over the years, but typically the same names appear each year.

This year was also slightly different, as the organisers asked for a lot of additional questions. Particularly related to the Code of Conduct. I will be forwarding the results of these questions to the TPF in the next day or two. They may choose to make the results public, but for now they won't appear on the YAPC Survey site. Of the other questions they asked, most related specifically to YAPC::NA, and wouldn't be applicable to other conferences and workshops. These too will be reviewed for next year.

Interestingly, VM Brasseur has done some analysis of the survey data, particularly around the age of attendees, and the length of time people have been a Perl programmer. Although the survey includes the former, it doesn't really include the latter. We do ask what level people feel they are at, but it'll be an area I'll be reviewing for future surveys.

As both the surveys and VM's analysis shows, the Perl community (at least those answering the survey) is getting older. I've noticed this too when attending. There are new and younger people attending, but generally the audience has been getting older. In the UK, this was identified in an technical article I read a few years ago (sadly I don't have a link to the source), which highlighted a shift in the late 80s/early 90s away from writing computer games on Spectrums, Dragons and Beebs to just playing consoles. I suspect the age of attendees at other technical conferences are also seeing a shift.

As noted in a previous post, I'm going to be looking at the Conference Survey software over the summer, and hopefully integrate it more with the Act software. I'm hoping this may encourage more to respond. I'll also be reviewing the survey itself, and looking at better and more relevant questions to include. If you have ideas of how to improve the survey, please feel free to drop me an email.

Enjoy :)

File Under: conference / perl / survey / yapc
NO COMMENTS


The Time of the Turning

Posted on 7th May 2013

A few weeks ago I had the pleasure of attending the 6th annual QA Hackathon. The event has become THE event for developers of test modules, projects and toolchain applications to come together to discuss ideas and plan for the future, as well as release some great work while they are there too.

This year Shadowcat, the primary sponsors, took on the organisational duties. The event was originally to be in London, but due to personal circumstances the decision was made to move the location to Lancaster in the North West of England. Personally they made the right choice. The venue itself was the new InfoLab building at Lancaster University. The attendees came from far and wide once again, and it was great to catch-up with friends old and new, and even be introduced to some newer friends.

My plan for the weekend was mainly to look at CPAN Testers. With the servers for the Metabase coming soon, David Golden and myself had hoped to be able to set them up, and start looking at changing the backend code to work with the new Metabase database. Unfortunately, the servers weren't ready for us just yet, so I started looking at other things. For myself, one area of CPAN Testers, particularly the cpanstats database side of things, needed attention. Speed of processing reports.

My first task once settled in, was to look at the way that the reports are consumed from the Metabase. Due to the way SimpleDB has become very unreliable with the results it sends, in order to avoid missing reports the criteria for the date search has been altered slightly to be a little more thorough, and a smaller range is now used to retrieve a set of GUIDs. The results now appear to be a little more complete, although we still appear to be missing some every so often. There is also a tail of log.txt which also helps to catch up with the reports. This work saw a new release of CPAN-Testers-Data-Generator.

A big factor with the slowness of the CPAN Testers server is that it requires a lot of disk I/O, with the database updates being a key factor. The most intensive updates are surrounding the SQLite database that could be downloaded. This also includes creating the Gzip and Bzip2 archives. As only web crawlers seem to be downloading the files, I've suspended the update. This has now freed up a lot of resources and consequently some of the other tasks, particularly the builder has improved.

Next, the builder was the focus of my attention. Previously the builder has been building pages for both authors and distros all at once. Although the author pages are viewed slightly less, they were getting built more frequently, due to the way the requests are pushed into the queue for each report. Initially the logic for building pages was altered, which improved some of the higher requested pages, but the more optimal solution was to split the builder into two, one for authors and one for distros. With the reduction in processing elsewhere, this improved the builder performance considerably. Monitoring the way the author pages are built since the hackathon, has also allowed me to alter when the builder for authors runs. This has then allowed the builder for distros to take a higher priority. With more distro pages than authors, this now gives distro pages more opportunity to be built quicker. Currently reports are being built in less than 24 hours of being submitted. These updates saw a new release of CPAN-Testers-WWW-Reports.

Another release while at the event, related to the QA Hackathon itself, was the main QA Hackathon website. Before the event, BooK had asked if the files that make up the website that the main QA Hackathon uses could be added to GitHub. As such, I packaged up the site into a git repository and released it. If you wish to help contribute to the site, please do.

Although there was a lot of coding work involved in the weekend, one of the bigger uses of time was the Lancaster Consensus organised by David Golden. For a few hours each afternoon, a large group of key toolchain developers, secondary project developers and various interested parties, gathered to discussed various aspects associated with configuration, installation, testing and specification of Perl and CPAN. With so many developers in one room, it wasn't too surprising to have a few opposing views, but with a guiding hand from David, we did achieve a consensus. If you wish to read the outcome, please read David's write-ups of the discussion points. The Consensus meetings were perhaps the greatest achievement of the event. While there might not have been too much immediate coding output from them, the potential to improve Perl and CPAN is considerable. From a CPAN Testers perspective, Post-installation testing, Case insensitive package permissions and Rules for distribution naming were perhaps of most interest. Although it may be some time before Post-installation testing could be hooked into a CPAN Testers smoker, it will be a valuable addition to the testing reports against pre-installed environments.

During the event, I had several discussions with Garu regarding his work on the cpanminus smoker client, and the common smoker client. In the last minutes of the hackathon we were able to push through a very notable report submission. It is exactly this sort of collaborative effort that makes these hackathons worthwhile. I look forward to see everyone again in Lyon.

The QA Hackathons could not be the success they are with the support of all the sponsors. My personal thanks to them for helping to providing accommodation, food and a venue for us all to hack. A big thank you to cPanel, Dijkmat, Dyn, Eligo, Evozon, $foo, Shadowcat Systems Limited, Enlightened Perl Organisation and Mongueurs de Perl.

File Under: hackathon / perl / qa / testing
NO COMMENTS


To Wish Impossible Things

Posted on 4th May 2013

The QA Hackathon website has had a bit of an update today. Primarily a new page and new photos have been added, but plenty of other updates have been included too.

The new page is a review page, to collect various blog and news posts relating to each year's event. Originally I listed all the reviews from previous years in the side panel, but now that we've just had the 6th annual event, the list was looking a little bit too cramped.

With the extra space, I've also been able to include the group shots that were taken at some of the events. Unfortunately there was no group shot taken in Birmingham, and I've not seen any during the 2010 and 2011 events, so if there are any, please let me know. Also if there is one of the Tokyo Satellite event this year I would love to include it on the site.

I've added some write-ups to the last few events in the About page. The biggest change though is likely only visible to those with screen readers, as I've made many changes to links and images to provide more accessibility. Several fixes to layout, spelling and wording have also been included too.

The site, particularly the list of reviews, is still incomplete. If a blog entry is missing that you think should be there, or you spot other items that could do with an update, feel free to email me with details, or fork the repo on GitHub and send me a pull request.

File Under: hackathon / perl / qa / website
NO COMMENTS


Lost In The Echo

Posted on 26th August 2012

I've just released new versions of my use.perl distributions, WWW-UsePerl-Journal and WWW-UsePerl-Journal-Thread. As use.perl became decommisioned at the end of 2010, the distrubutions had been getting a lot of failure reports, as they used screen-scraping to get the content. As such, I had planned to put them out to pasture in BackPAN. That was until recently I discovered that Léon Brocard had not only released WWW-UsePerl-Server, but also provided a complete SQL archive of the use.perl database (see the POD for a link). Then combining the two, he put up a read-only version of the website.

While at YAPC::Europe this last week, I started tinkering, and fixing the URLs, regexes, logic and tests in my two distributions. Both distributions have had functionality removed, as the read-only site doesn't provide all the same features as the old dynamic site. The most obvious is that posting new journal entries is now disabled, but other lesser features not available are searching for comments based on thread id or users based on the user id. The majority of the main features are still there, and those that aren't I've used alternative methods to retrieve them where possible.

Although the distributions and modules are now working again, they're not perhaps as useful as they once were. As such, I will be looking to merge both distributions for a future release, and also providing support to a local database of the full archive from Léon.

Seeing as no-one else seems to have stepped forward and written similar modules for blogs.perl, I'm now thinking it might also be useful to take my use.perl modules and adapt them for blogs.perl. It might be a while before I finish them, but it'll be nice to have the ability to have many of the same features. I also note that blogs.perl.org also now has paging. Yeah \o/ :) This has been a feature that I have been wanting to see on the site since it started, so thanks to the guys for finding some tuits. There was a call at YAPC::Europe for people to help add even more functionality, so I look forward to seeing what delights we have in store next.

File Under: opensource / perl / website
NO COMMENTS


Every Day Is Exactly The Same

Posted on 15th July 2012

Recently Mark Keating of the Enlightened Perl Organisation created a new Google Calendar for Perl community events, particularly for Perl Monger group meetings. As I haven't been updating the other calendars I have access to for some time, it gave me the push needed to clean-up my script, and post all the forthcoming events to the calendars.

I've now updated the Birmingham.pm events page, to display the new calendar, as well as the West Midlands Tech Events calendar.

If you have access to any similar calendars, you can now update them with Perl (if you weren't already), with the aid of my helpful script. Feel free to use and abuse as you wish. Note that you will need to have a login to Google Calendars, and have access to the calendars you are submitting to.

File Under: perl
NO COMMENTS


Parisienne Walkways

Posted on 3rd April 2012

And so to the final part of my notes from the 2012 QA Hackathon.

CPAN Testers Report Status

After asking several times, Andreas thought he finally understood what the dates mean on the Status page for the CPAN Testers Reports. He started watching and making page requests to see whether his requests were actioned. On Day 3 he pointed out that the date went backwards! Once he'd shown me, I understand now why the first date is confusing. And for anyone else who has been confused by it, you can blame Amazon. SimpleDB sucks. It's why the Metabase is moving to another NoSQL DB.

The date references the update date of the report as it entered the Metabase. The last processed report is the last report that was extracted from the Metabase and entered into the cpanstats DB. Unfortunately, SimpleDB has a broken concept of searching. It will return results before the date requested, and regularly return the sorted results in an unsorted order. As such the dates you see on the Status page may go backwards in time! I'm not going to try and fix this, as it will all work as intended with the new system.

Missing Reports

There have been several questions relating to missing reports over the past few years. Sometimes it just needs me to refresh the indices, but in other cases it may be due to the fact that SimpleDB omits reports from a request. Did I mention SimpleDB sucks? In a request to the Metabase, I will ask for all the reports from a given date. The results are limited to 2500, due to Amazon's own restriction. In the returned list it will often omit entries, due to its ignorance of sorting in the search request. I have gone through the Metabase code on several occasions and can verify it does the right thing. SimpleDB just chooses to ignore the complete search request and returns what it *thinks* you want to know.

Ribasushi questioned me about one of his modules that had been released recently, which still had no Cygwin reports listed, even though he sent a few himself. Further investigation revealed that they are indeed missing from the cpanstats DB. Although they did enter the Metabase, they never came out again.

To resolve this I have been revisiting the Generator code to rework the reparse and regenerate code to enable search requests for missing periods, in the hope that this will retrieve most of the missing results. If it doesn't, then I will be asking David to produce a definitive list for me, and I will make specific requests for any missing reports. The Generator code has been updated in GitHub to include all the performance improvements that have been in live for some time too.

Erronously Parsed Reports

Every so often the parsing mechanism fails and stores the wrong data within the cpanstats DB. These days it seems to only affect the platform, OS version and OS name. I'm not quite sure what is happening, as reparsing the report locally again produces the correct results. This uses the same routine to parse the report, so why they occasional fail remains a mystery. However, to combat this, I  now have a script that can run and search periodicly for this erroneous data and attempt to reparse the results. It can then alert me when it can't fix it and I can investigate manually. The have been occasions where the report can't be parsed due to the output being corrupted on the test machine, which unfortunately we can't always resolve. Sometimes there are enough clues within other parts of the report that point to a particular OS, but sometimes we just have to leave it blank.

It seems in putting some of this code live before leaving the hackthon, I accidentally reintroduced a bug. Slaven was quick to spot it and tell me about it, but unfortunately it was too late for me to fix it, as I needed to leave and catch my flight home. It should be fixed by the time you read this though, so all should be back to your regular viewing pleasure :) With the new script I've written, it should hopefully find and fix these errors in the future, as well as alerting me to fix the bug again!

Thanks Again

So that was the 2012 QA Hackathon. The show ended with a group photo, although a few were missing due to their early departures home, but I think we got most of us in. Including Miyagawa, who was taking the picture. The traditional thanks yous and good byes ensued and then Andreas and I headed off to begin our adventure getting the airport! The next hackathon, the 2013 QA Hackathon, will be in London. We'll have the domain pointed to the right place just as soon as Andy gets the website up and running. I look forward to a lot more involvement for next year, as we have been steadily growing in numbers each year. There has already been some significant output, but the event is much more than that. It's a chance to take to people face to face, discuss ideas and plan for the future. Expect more news for CPAN Testers soon.

Once again I would like to thank ShadowCat Systems for getting me here, and for being a great supporter of the QA Hackthons, as well as many other Perl events over the years. Thanks too to Laurent Boivin (elbeho), Philippe Bruhat (BooK) and the French Perl Mongers for making the 2012 QA Hackathon happen. The Hackathon wouldn't have happened without the generosity of corporations and the communities that donate funds. So thank you to ... The City of Science and Industry, Diabolo.com, Dijkmat, DuckDuckGo, Dyn, Freeside Internet Services, Hedera Technology, Jaguar Network, Mongueurs de Perl, Shadowcat Systems Limited, SPLIO, TECLIB", Weborama, and $foo Magazine. We also have several individuals to thank too, who all made personal contributions to the event, so many thanks to Martin Evans, Mark Keating, Prakash Kailasa, Neil Bowers, 加藤 敦 (Ktat), Karen Pauley, Chad Davis, Franck Cuny, 近藤嘉雪, Tomohiro Hosaka, Syohei Yoshida, 牧 大輔 (lestrrat), and Laurent Boivin

Meanwhile, Dan & Ethne would also like to thank Booking.com for their silly putty ;)

File Under: hackathon / opensource / paris / perl / qa / testing
NO COMMENTS


The Reasons Why

Posted on 25th March 2012

For those that follow the conference surveys, you'll be pleased to hear that I have now put the results of both the Israeli Perl Workshop and the German Perl Workshop online. These are the first events this year to take advantage of the surveys, although several more are to come.

This marks the second survey for the German Perl Workshop and notes some small differences, while it was the first for the Israeli Perl Workshop. I hope the future organisers can make use of the results and that they allow me to continue the surveys with these workshops next year, and for the years to come.

Although the Israeli Perl Workshop was in English this year, Gabor and I are hoping to be able to provide the survey in Hebrew next year. The German Perl Workshop marked the first survey not in English last year, and it helped to start building up a language pack, which can be used to plugin to the survey software. I plan to formalise this during the year, so that other events, using languages other than English, can still take advantage of the surveys.

Thanks to all the organisers and the survey participants for taking the time to respond to the questions. It is very much appreciated.

File Under: conference / opensource / perl / survey / workshop
1 COMMENT


London Calling

Posted on 7th February 2012

London Perl Workshop 2011 Survey - Results Now Online

London Perl Workshop 2011 Survey Results

I'm please to say that the survey results from the London Perl Workshop 2011 are now online. Slightly delayed due to Christmas and my new job, but worth the wait I think. This is the first time we've had a survey for the London Perl Workshop, so I was interested to see how the results differed from YAPCs. The attendees for the workshop differ from YAPCs, as although around 40% of attendees are well know within the Perl community and have attended YAPCs, most of the sttendees, like attendees for many other workshops around the world, don't have the resources or availability to attend a 3-5 day conference event. However, a one-day event, and especially a free event, makes a workshop much more accessible.

It was a shame that we only had 27% of attendees responding, but having said that while my personal aim is always to achieve more than 50% response, 27% is still a great response. As I've said previously, anything more than 10% is a good result. However, now we've done one, hopefully we can encourage more to respond this year :)

The demographic responses interestingly followed what we often see for YAPCs. I guess that may be because most of the respondees are YAPC attendees, but we still had several responses from those who have only attended London Perl Workshops, or for whom this was their first major event. The balance of Perl knowledge, although slightly weighted towards the more experience developeres, was also very pleasing to see with several beginners attending the event. Every year we have been looking to encourage newcomers to these events, as well as into the Perl community. After all, those learning Perl now are the Perl community's future. It was also great to see people being nominated or recommended to attend by colleagues and managers. The promotion of these events is obviously having the right effect.

I was intrigued to see that of all the respondees, 65 of them weren't speakers, with 30 willing to consider being a speaker in the future. Again this is something we should be encouraging, as newer speakers often have a different perspective on a subject, and can bring something new and fresh to the event. It was also encouraging that primary motivations for attending are to get together and meet other Perl developers. Events like The London Perl Workshop are a great way to introduce yourself to other developers you may have spoken to online, or are collaborating with on projects. They are a great way to promote your project, or get to know more about other projects.

In response to the question "What aspects of the conference do you feel gave value for money?", I was actually quite surprised to see comments along the lines of "Since it was free, I do not understand what "gave value for money" means." For those attendees who wondered that, how much did it cost you to attend the event? If you think your answer would be "nothing", consider the question beyond the attendance fee. How did you get there, did you walk, get the train, drive? What about your time, what would you have done that Saturday if you hadn't have gone to the event? Just because the event had no attendance fee, doesn't mean it cost you nothing to attend. Also think about what value it has given you in terms of enhancing your knowledge. Did you see talks or meet people that have inspired you, or given you a better understanding of something you were working on. There are lots of ways "value" can be interpreted beyond any monetary value.

It took a lot of people a lot of time and energy to put event like the London Perl Workshop on, not just Mark and his minions, but also the speakers and the sponsors. From their perspective it is good to know their efforts were appreciated. Thanks you to all those who did respond, and of those that didn't, hopefully we can encourage you to contribute your thoughts this year :)

File Under: london / perl / survey / workshop
NO COMMENTS


Do You Remember the First Time?

Posted on 4th October 2011

YAPC::Europe 2011 Survey Results

During August this year, in Riga, Latvia, YAPC::Europe brought together 285 people to learn, discover and discuss Perl. As previous attendees know the YAPC conferences are a perfect opportunity to introduce yourself to the Perl community. YAPCs are now held all around the world and each is very different another. Each has their own charactistics, and they all get better and better thanks to the feedback from attendees old and new, which is why the YAPC Conference Surveys are well placed to concentrate that feedback for future organisers.

For YAPC::Europe 2011, the survey results are now online.

Although the responses where down from previous YAPC::Europe events, we still had over 50%, so thank you to everyone who took the time to respond. Interestingly of those who took the survey, none recorded themselves as coming from Latvia. I suspect this is in part due to the language barrier. As the surveys are in English, those that don't feel quite comfortable with the language might feel less inclined to feedback their thoughts and experiences. I'd like to be able to have the surveys available in different languages, but accumulating some of the responses, particularly the free text ones, may prove difficult. However, this is a goal for the future.

Unsurprisingly these days, we saw a large number of people attending who are regulars either to the YAPCs and Workshops or to the Perl community generally. At the conference itself we did ask how many attendees were at their first YAPC, and it was quite significant. However, we are still seeing roughly the same numbers, so we are not necessarily able to keep those new attendees coming back as regular attendees. In this survey however, no-one stated that they wouldn't attend another event in the future, so hopefully next year we should start seeing more familiar faces.

This year I plan to get the free text feedback sections online, and may well provide these for previous years too. I normally only provide these to the organisers (both current and succeding), but I think everyone could benefit from the thoughts and ideas, whether a YAPC organiser or an organiser of any other technical event.

Many thanks to all those who took the time to respond, both to the Conference Survey and all the Talk Evaluations. Your time is very much appreciated.

File Under: community / conference / opensource / people / perl / survey / yapc
NO COMMENTS


What's the Frequency, Kenneth?

Posted on 8th September 2011

YAPC::NA 2011 Survey Results

During June this year, in Asheville, North Carolina, YAPC::NA assembled 251 people together to learn and discuss Perl, Perl projects and meet Perl people. The YAPC conferences are a perfect opportunity to tell the Perl community of your latest project, or to talk to other Perl developers face to face. YAPCs have now been running for 12 years, and each gets more focused and exposure than the last. In part in this thanks to all the previous organisers who have gone before, offering help and advice where they can. However, the YAPC Conference Surveys also help to provide value feedback to future organisers.

For YAPC::NA 2011, the survey results are now online.

While only 34% of all attendees responded, the feedback has still proved very helpful and provided me with some additional questions for the future. I was recently asked how I thought the YAPCs had changed, and one of the changes I noted, as is hinted at in the feedback, is that many of the talks now focus more on Perl frameworks and applications, rather than specific modules or techniques. In a way it highlights how Perl has grown up. Perl is still a language and tool to get jobs done, but now there are more stable and constructive ways of getting those jobs done.

Many thanks to all those who took the time to respond, both to the Conference Survey and all the Talk Evaluations. Apologies for the delay in getting the results online, but events with CPAN Testers have taken most of my free time over the last 2 weeks :(

File Under: conference / opensource / perl / survey / yapc
NO COMMENTS


Are We the Waiting

Posted on 27th August 2011

With over a week gone since the end of YAPC::Europe 2011, I'm please to see we already have 102 Conference Survey responses and 541 Talk and Course Evaluations submitted. This is once again a fantastic start to the responses and very much appreciated. However, there are still a further 165 who can still submit their Conference Surveys, and everyone still has time to submit feedback to the speakers of the talks and courses they attended.

If you haven't received your keycode email, please contact me and I will resend it. You still have 3 weeks until the close of the surveys, so please try and take some time to complete them. It really does help to improve the conferences for everyone.

For those interested in the results of the YAPC::NA 2011 surveys, although the speaker feedback has been sent out, I had to postpone my work on the Conference Survey due to some CPAN Testers issues and my attendance at YAPC::Europe 2011. As such, I am now preparing these results for the YAPC Conference Surveys website and hope to have an announcement within the next few days.

File Under: conference / perl / survey / yapc
NO COMMENTS


Rearviewmirror

Posted on 19th August 2011

Earlier this week I attended YAPC::Europe 2011. Many thanks to Andrew, Alex and all the others involved with bringing the conference to life, it was well worth all the effort.

During the conference I gave two talks. The first was my main talk, Smoking The Onion - Tales of CPAN Testers, which looked at how authors can use the CPAN Testers websites to improve their distributions, as well some further hints and tips for common mistakes spotted by testers over the years. It also looked at how some of the sites can be used by users to see whether a particular distribution might be suitable for their purposes or not. The talk seemed to go down well, and it seems a few were disappointed to have missed it, after discovering it wasn't my usual update of what has been happening with CPAN Testers. Thankfully, I did video the talk, and I think the organisers also have a copy, so expect to see it on YAPC TV and Presenting Perl at some point in the future.

Photo by Jon Allen

Photo by Jon Allen

My second talk, Perl Jam - How To Organise A Conference (and live to tell the tale), was a lightning talk to help promote my book and the YAPC Conference Surveys. The book is currently a work in progress, and I'd like to get more feedback from anyone who has been an organiser of a YAPC, Workshop or Hackathon, as well as any photos that would help to highlight particular sections of the book. If you think you could help, please take a look at the GitHub repository and send a pull request with any updates you think appropriate.

Congratulations to Frankfurt.pm for winning the chance to host YAPC::Europe 2012. See you next year.

File Under: book / community / conference / opensource / perl / survey / testing / yapc
NO COMMENTS


Points of Authority

Posted on 27th May 2011

Back in February I did a presentation for the Birmingham Perl Mongers, regarding a chunk of code I had been using to test websites. The code was originally based on simple XHTML validation, using the DTD headers found on each page. I then expanded the code to include pattern matching so I could verify key phrases existed in the pages being tested. After the presentation I received several hints and suggestions, which I've now implemented and have set up a GitHub repository.

Since the talk, I have now started to add some WAI compliance testing. I got frustrated with finding online sites that claimed to be able to validate full websites, but either didn't or charged for the service. There are some downloadable applications, but most require you to have Microsoft Windows installed or again charge for the service. As I already had the bulk of the DTD validation code, it seemed a reasonable step to add the WAI compliance code. There is a considerable way to go before I get all the compliance tests that can be automated written into the distribution, but some of the more immediate tests are now there.

As mentioned in my presentation to Birmingham.pm, I still have not decided on a name. Part of the problem being that the front-end wrapper, Test::XHTML, is written using Test::Builder so you can use it within a standard Perl test suite, while the underlying package, Test::XHTML::Valid uses a rather different approach and does provides a wider API than just validating single pages against a DTD specification. Originally, I had considered these two packages should be two separate releases, but now that I've added the WAI test package, I plan to expose more of the functionality of Test::XHTML::Valid within Test::XHTML. If you have namespace suggestions, please let me know, as I'm not sure Test-XHTML is necessarily suitable.

Ultimately I'm hoping this distribution can provide a more complete validation utility for web developers, which will be free to use and will work cross-platform. For those familiar with the Perl test suite structure, they can use it as such, but as it already has a basic stand-alone script to perform the DTD validation checks, it should be usable from the command-line too.

If this sounds interesting to you, please feel free to fork the GitHub repo and try it out. If you have suggestions for fixes and more tests, you are very welcome to send me pull requests. I'd be most interested in anyone who has the time to add more WAI compliance tests and can provide a better reporting structure, particularly when testing complete websites.

File Under: modules / opensource / perl / technology / testing / usability / web
NO COMMENTS


Behind The Lines

Posted on 25th May 2011

Back last year I got a curious email from a fellow London.pm'er asking why I was releasing so many WWW-Scraper-ISBN distributions. The reason was quite simple, to make my life easier! Well okay, that's why I wrote the distributions, but I figured others might find them useful too.

In the UK the book trade is a bit odd, and I dare say the rest of the world suffers from this too. The publishers don't like to give too much information away about their books, and the central body for allocating ISBNs, Nielsen, don't always have all the necessary metadata available. The book trade uses MARC Records to transfer this metadata around, and unfortunately, while there is provision to include much of the metadata, it often isn't included. The obvious things such as the Author, Title and the ISBN itself are usually there, but some of the data relating to the physical attributes (pages, height, width and weight) rarely are.

Originally I wrote the Amazon, Pearson Education, O'Reilly Media and Yahoo! Books distributions to use within Labyrinth, particularly for the Birmingham Perl Mongers website, and our Book Reviews. The plugin mechanism allowed me, when I received a review, to enter the ISBN and prepopulate the metadata fields and links before adding the review itself. The four distributions saved a lot of time, but the initial releases were quite basic.

Jumping forward several years, now needing this extra metadata, I first expanded the original four distributions. However, not all of these online bookstores provided this extra metadata. Picking a variety of books I searched to see what metadata I could retrieve, and came across several sites around the world that included this information to varying degrees. Much of the basic information regarding an ISBN shouldn't change from country to country, so metadata retrieved from Australia or New Zealand is as valid as that from America or the UK. There are aspects that can differ, such as the cover illustration, but the majority of metadata returned should be applicable regardless of location.

There was some interesting discrepancies with the different units of weights and measures used across the sites too. While some stuck to a set of fixed units, others changed depending how big the values were, particularly for grammes and kilogrammes. I settled on grammes for weight and millimetres for height and width, seeing as metric was the most commonly used on the various sites.

It did cross my mind whether to include the prices in the metadata returned, but as prices often fluctuate frequently and are very location dependent, you are probably better to write this side of things yourself for your specific purpose, such as a comparision website. I also left out depth, as only a few sites regularly provided a value for it. I can always save it for a future release anyway.

Hopefully those that work in the book trade, who have been wishing that MARC Records were populated a little more fully than they are currently, can make use of these distributions to help fill in the gaps.

File Under: book / isbn / opensource / perl
NO COMMENTS


Questions & Answers

Posted on 9th May 2011

I mentioned in my last post that I was working on a Survey Plugin for Labyrinth. The plugin is used within the YAPC Conference Survey system, which has now been running for several YAPC events over the last 5 years. I had promised to try and release the complete survey site last year, but with it being a Labyrinth based site setup, I didn't want to release it without releasing Labyrinth first. Now that's done I can concentrate on getting the Survey Plugin and the complete survey system on CPAN.

This year I will be running the YAPC::NA and YAPC::Europe surveys as per usual. However, this year I am delighted to say I have also been asked to handle the survey for the Pittsburgh Perl Workshop too. Hopefully if all goes to plan, this will provide the test bed for many other workshops to provide surveys.

The Conference Surveys themselves started in 2006, and have provided some very interesting feedback for organisers. While event organisers and myself never expect to get 100% response from all attendees, the levels that we do get is absolutely phenomenal. With this kind of success, I would be very interested to see whether the same Survey system can be used by other non-Perl events. There is certainly nothing that prevents a non-Perl (or even a non-tech) event from using the system. Last year I did have a query from a non-Perl event, but the system wasn't ready for a stand-alone release, and I wasn't able to set anything up. However, this year, with a CPAN release coming soon, I am more hopeful that others might be able to use the system.

If you are an organiser for an event where you think a survey would be useful for feedback, please do get in touch. If I cannot host an instance for you, once I get a full release on CPAN, I can provide help and advice for getting your own hosting instance running.

File Under: conference / labyrinth / perl / survey / workshop / yapc
NO COMMENTS


Into The Blue

Posted on 7th May 2011

I haven't been posting recently about the Perl projects I'm currently working on, so over the next few posts I hope to remedy that.

To begin with, one of the major projects I've been involved with for the past 8 years has been CPAN Testers, although you can find out more of my work there on the CPAN Testers Blog. This year I've been releasing the code that runs some of the websites, specifically those that are based on my other major project, Labyrinth. Spearheading these releases have been the CPAN Testers Wiki and CPAN Testers Blog, with further releases for the Reports, Preferences and Admin sites also planned. The releases have taken time to put together mostly because of the major dependency they all have, which is Labyrinth.

Labyrinth is the website management framework I started writing back in 2002. Since then it has grown and become a stable platform on which to build websites. With both the CPAN Testers Wiki and the CPAN Testers Blog, three key plugins for Labyrinth have also been released which hopefully others can make use of.

The Wiki plugin, was intended to be written for the YAPC::Europe 2006 Wiki, but with pressures of organising the conference and setting up the main conference site (which also used Labyrinth), I didn't get it finished in time. Once a CPAN Testers Wiki was mooted, I began finishing off the plugin and integrating into Labyrinth. The plugin has been very stable for the last few years, and as a consequence was the first non-core plugin to be released. It's a fairly basic Wiki plugin, not too many bells and whistles, although there are a couple of Perlish shortcuts, but for the most part you don't need them. The CPAN Testers Wiki codebase release was also the first complete working site for Labyrinth, which was quite a milestone for me.

Following that success, the next release was for the CPAN Testers Blog. Again the underlying plugin, the Blog Plugin, has been stable for a few years, so was fairly quick to package and release, however the secondary plugin, the Event Plugin, has been evolving for quite some time and took a little more time. As I use both these plugin for several other sites, it was a good opportunity to bring together any minor bug fixes and layout changes. Some of these have seen slight modifications to the core Labyrinth codebase and the core set of plugins. In addition it has prompted me to start working on the documentation. It is still a long way from being complete, but at least the current documentation might provide some guidance to other users.

One of my major goals for Labyrinth was for it to be a 'website in a box'. Essentially this means that I wanted anyone to take a pre-packaged Labyrinth base (similar to the Demo site), drop it on a hosting service and be able to run a simple installation script to instantiate the database and configuration. The installation would then also be able to load requested plugins, and amend the database and configuration files appropriately. I haven't got to that stage yet, but it is still a goal.

With this goal in mind, I have read with interest the recent postings regarding the fact that DotCloud are now able to run Perl apps. This is definitely great news, and is exactly the kind of setup I had wanted to make best use of for the 'website in a box' idea. However, with several other frameworks now racing to have the coolest instance, it isn't something I'm going to concentrate on right now for Labyrinth. Plus there is the fact that Labyrinth isn't a PSGI framework, which others have eagerly added to their favourite framework. Labyrinth came from a very different mindset than other now more well known frameworks, and tries to solve some slightly different problems. With just me currently working on Labyrinth, as opposed to the teams of developers working on other frameworks, Labyrinth is never going to be the first choice for many reasons. I shall watch with interest the successes (and lessons learned from any hiccups) of the other frameworks as it is something I would like to get working with Labyrinth. If anyone who has the time and knows PGSI/Plack well enough, and would like to add those capabilities to Labyrinth, please get in touch.

The next notable plugins I'll be working on are the Survey, Music and Gallery Plugins. The former has its own post coming shortly. The next notable CPAN Testers site released planned is the Reports site. With it being considerably more involved, it might take a little longer to package and document, but it will likely be the most complex site release for Labyrinth, which will give anyone interested in the framework a good idea of how it can be used to drive several sites all at once.

File Under: labyrinth / opensource / perl / web / website
NO COMMENTS


Loose Change

Posted on 1st April 2011

Many years ago I wrote a set of scripts and modules that together formed a way for me to access eBay internationally. I frequently bought records from the UK, US, Germany and Australia, so those were the plugins that I focused on, but the intention was to allow more to interface to other eBay sites. I even did a presentation at YAPC::Europe in 2004, called The Perl Auctioneer, which explained my progress.

As part of the currency calculations and conversion, I used the same site that eBay themselves were using, XE.com. As I became more involved in other projects, and my international eBay buying declined, my efforts to finish and release the Perl Auctioneer waned. However, I was still using the currency conversion module, so released it as a stand-alone package. In time this became Finance::Currency::Convert::XE.

Although I have occasionally updated the module, I no longer use it. However, others still do. XE.com themselves are very protective of their data, understandably, and are very resistent to screen scrapers. Even though their own terms of use allow for personal use, and do not explicitly say screen scrapers are prohibited, they do make accessing the data from the command line very difficult. They have very recently upgraded their website with further measures to prevent automated tools scraping their data.

As I no longer use the module, I feel I have two choices. Pass on the distribution to someone else, who does want to invest time and effort on the module, or to abandon the module and distribution and remove it from CPAN. As the module does not currently work with the latest XE.com site, unless someone does come forward I plan to remove the distribution from CPAN by the end of the month.

If you would like to take over the module, please email me (barbie@cpan.org) and let me know your PAUSE ID. I'll then put the wheels in motion to give you maintainer/author permissions.

File Under: modules / opensource / perl
NO COMMENTS


All Around The World

Posted on 14th March 2011

Paul Weller once sang of "a new direction. We want a reaction. Inflate creation." All three could be attributed to why two major events in the Perl event calendar started in 1999, and now happen all around the world today. The two events, The German Perl Workshop and YAPC::NA, both were a new direction for Perl events and specifically a reaction to more commercial events. They both also brought a new creativity to the Perl community.

In 2011 we now have YAPCs, Workshops and Hackathons happening on a monthly basis somewhere in the world. They are still very much organised by members of the Perl Community, and bring together a diverse group of people to each event. They often inspire some to create Perl events themselves. However, that initial enthusiasm is often quickly followed by panic, when the organisers start to figure out what they need to do to make a great event. Which is where a book might help.

I am planning to publish such a book, entitled 'Perl Jam - How to organise a conference ... and live to tell the tale'. The book is a guide for organisers planning to host a large technical event, with the aim of helping organisers think of everything, and prepare themselves for anything they might not have thought of, or forgotten. Organising a conference, workshop or hackathon can be a daunting prospect, but with the help of this book, it might make the experience much more enjoyable, and may even inspire you to do it all again!

'Perl Jam' is being made available for its first public draft via a GitHub repository. This is the third draft, and my thanks go specifically to Jon 'JJ' Allen and David Golden, for their extensive help and feedback so far. Also thanks to chromatic for allowing me to use the framework and scripts he used for his great book Modern Perl.

I welcome any and all comments and suggestions, so if you've ever organised a large event, please take the time to read the draft and see if there is anything not covered that you would have suggested. For any current organisers, please download and share the book with your team and feel free to send me any additional notes you make as you go along. If you are thinking about organising a technical event in the future, are there any questions you would want to know, that haven't been explained in the book?

Everything is up for discussion, including the cover (which is not the finished version), and I'm very interested to hear from anyone who has suitable photos that can be included in the book, as examples or to emphasise sections.

The draft is released under a Creative Commons Attribution-Noncommercial-No Derivative Works 3.0 License. Please feel free to point people to the git repository, but please do not redistribute with any modifications. Forking with Git is fine, but I request that you send me patches (via perljam@missbarbell.co.uk) or pull requests.

The book also has its own website, Perl Jam, which will be the official source of any releases.

File Under: book / community / conference / opensource / perl / yapc
NO COMMENTS


Ultraviolet (Light My Way)

Posted on 28th February 2011

Last week I gave my first technical talk for several months. Despite being a bit rusty, everyone seemed to find the talk interesting. The talk itself was about code I'd written to test XHTML completeness of web pages and further pattern matching of page content. I've been using and developing the testing code over the last few years, having written the initial basic script, xhtml-valid, back in 2008. Over the last 18 months I have revisited the code and rewritten it into a traditional Perl testing structure. The talk looked at the current state of the code and asked for advice on where to take it next.

The code has developed into two packages, Test::XHTML and Test::XHTML::Valid, and as such the talk naturally fell into two parts, looking at each package in more depth. I had originally planned a demo, but unfortunately my laptop wouldn't talk to the projector, so had to rely on slides alone. This didn't seem to matter too much, as the slides conveyed enough of the API to give a decent flavour of what the packages were about.

The final questions I asked originally centred on where I was thinking of heading with the code base, but I also got asked a few questions regarding the technical aspects. My thanks to Colin Newell and Nick Morrott for giving me some ideas and pointers for further expansion of the code. As for my final questions, it was generally agreed that these should appear on CPAN in some form, and as two separate packages, but unfortunately nobody had a suitable name for either.

I plan to work further on the code, both to package them better and to include the suggestions from Colin and Nick, and then I'll see if anyone has some better suggestions for the names. In the meantime, the slides are now online [1] and the 2008 version 1.00 of the code base is also available [2]. I aim to have the current code base online soon, with a GitHub repo to provide ongoing developments for anyone who might be interested.

[1] http://birmingham.pm.org/talks/barbie/text-xhtml
[2] http://barbie.missbarbell.co.uk/page/code

File Under: opensource / perl / testing
NO COMMENTS


Some Heads Are Gonna Roll

Posted on 11th February 2011

Some time ago I wrote Test-YAML-Meta. At the time the name was given as a compliment to Test-YAML-Valid, which validates YAML files in terms of the formatting, rather than the data. Test-YAML-Meta took that a step further and validated the content data for META.yml files included with CPAN distributions against the evolving CPAN META Specification.

With the release of Parse-CPAN-Meta I wrote Test-CPAN-Meta, which dropped the sometimes complex dependency of the more verbose YAML parsers, for the one that was specifically aimed at CPAN META.yml files. With the emergence of JSON, there was a move to encourage authors to release META.json files too. Although considered a subset of the full YAML specification, JSON has a much better defined structure that has more complete parser support. Coinciding with this move was the desire by David Golden to properly define a specification for the CPAN Meta files. It was agreed that v2.0 of the CPAN Meta Specification should use JSON as the default implementation. As a consequence I then released Test-JSON-Meta.

Although the initial naming structure seemed the right the thing at the time, it is becoming clearer that really the names need to be revised. As such I looking to change two of the distributions to better fit the implementations. So in the coming weeks expect to see some updates. The name changes I'm planning are:

Underneath these current namespaces is the Version module that describes the data structures of the various specifications. In the short term these will also move, but will be replaced by a dependency on the main CPAN-Meta distribution in the future. There will be final releases for Test-YAML-Meta and Test-JSON-Meta, which will act as a wrapper distribution to re-point the respective distributions to their new identities.

File Under: modules / perl / qa / testing
NO COMMENTS


Long Live Rock'n'Roll

Posted on 9th February 2011

On the 1st January 2011, I released the first Open Source version of Labyrinth, both to CPAN and GitHub. In additon I also released several plugins and a demo site to highlight some of the basic functionality of the system.

Labyrinth has been in the making since December 2002, although the true beginnings are from about mid-2001. The codebase has evolved over the years as I've developed more and more websites, and got a better understanding exactly what I would want from a Website Management System. Labyrinth had the intention of being a website in a box, and although it's not quite there yet, hopefully once I've released all the plugin code I can put a proper installation tool in place.

Labyrinth now is the backend to several Open Source websites, CPAN Testers using it for the Reports, Blog, Wiki and Preferences sites, as well as some personal, commercial and community projects. As a consequence Labyrinth has become stable enough to look at growing the plugins, rather than the core code. I'm sure there is plenty that could be done with the core code, but for the moment providing a good set of plugins, and some example sites are my next aims.

As mentioned, I see Labyrinth as a Website Management System. While many similar applications and frameworks provide the scaffolding for a Content Management System, Labyrinth extends that by not only providing the ability to manage your content, but also to provide a degree of structure around the functionality of the site, so the management of users and groups, menu options and access, as well as notification mechanisms, enable you to provide more control dynamically.

When writing the fore-runner to Labyrinth, one aspect required was the ability to turn on and off functionality instantly, which meant much of the logic flow was described in the data, not the code. Labyrinth has built on this idea, so that the dispatch tables and general functionality can be controlled by the user via administration screens, and not by uploading new code. When I started looking at this sort of application back in 2001, there was nothing available that could do that. Today there are several frameworks written in Perl that potentially could be tailored to process a website in this way, but all require the developer to design and code the functionality. Labyrinth aims to provide that pre-packaged.

I'm primarily releasing Labyrinth so that I can release all the code that drives the CPAN Testers websites. Giving others the ability to better suggest improvements and contribute. The system allows me the freedom to build websites quickly and easily, with the hardwork being put into the design and CSS layouts. With so many other frameworks available, all of which have bigger development teams and support mechanisms than I can offer, I'm not intending Labyrinth to be a competitor. It might interest some, which is great, but if you prefer to work on other frameworks that's great too. After all it's still Perl ;)

More news of plugins and sites being released coming soon.

File Under: labyrinth / opensource / perl / website
NO COMMENTS


Living By Numbers

Posted on 13th September 2010

Maisha, now with OAuth support.

A project I started back last year is Maisha, a command line client to interface to social micro-blogging networks, such as Twitter. On 31st August this year, Twitter depreciated the Basic Authention method of allowing applications to login users with a simple username and password combination. In its place they now use OAuth. (See also the blog post by Marc Mims - author of Net-Twitter).

On the face of it, OAuth seemed a bit confusing, and even the documentation is devoid of decent diagrams to explain it properly. Once I did get it, it was surprising to discover just how easy the concept and implementation is. For the most part Marc Mims has implemented all the necessary work within Net-Twitter, so Maisha only needed to add the code to provide the right URL for authorisation, and allow the user to enter the PIN# that then allows the application to use the Twitter API.

The big advantage to OAuth is that you don't need to save your password in plain text for an application. Once you enter the authorisation PIN#, the token is then saved, and reused each time you start up Maisha to access your Twitter feed.

As Identi.ca also implements an Open Source version of Twitter, they have also implemented OAuth in their interface. However, there is a slight modification to Net::Twitter needed, so I will wait for Marc to implement that before releasing the next version of Maisha.

So if you have been using Maisha and have been frustrated that you can no longer access Twitter, you now only need to upgrade to App-Maisha-0.14 and all should work again (once you've entered the PIN# of course).

If you are using Maisha, and have any feedback or wishlist suggestions please let me know.

File Under: life / opensource / perl / technology
NO COMMENTS


People Get Ready

Posted on 20th August 2010

The Optimum YAPC Attendance

In my recent post about promoting YAPCs, Gabor picked on something regarding the optimum number of attendees. I think he makes a good point that for a conference like a YAPC, 300-400 attendees is a good number to aim for. Anything more and it can become a logistical nightmare for organisers. It also means that the conferences themselves can become a little more impersonal, when a major aim of YAPCs is to bring people together.

With bigger numbers attending, it creates problems for organisers, not only to accommadate the large numbers, but also the cost. Universities have been ideal in the past, as they are usually quiet out of term time, and can usually accommodate several hundred people for little outlay. However, looking for venues that can accommodate thousands, which typically means professional conference venues, needs special effort to cover the costs. Events like FOSDEM are now so well established that large corporate sponsors are willing to donate without much persuasion, but a dedicated language conference would struggle to get the same kind of support.

YAPC::Asia can cope with 500 attendees, but now regularly sells out because they just cannot accommodate any more in the venue they use. In North America and Europe most of the venues can usually cope with around 400 attendees. In Europe we generally see lower attendances due to travel and accommodation costs for personal attendance being too high for some, as we see a larger number of attendees paying for themselves. As a consequence it is unlikely we are going to see a dramatic increase in numbers unless Perl suddenly finds itself being the language of choice for many business, especially corporates.

I have attended large conferences in the past, and while there is a wide choice of talks and more people to meet, it can be a bit overwhelming. You don't always get the chance to talk to all the people you wanted to, and many that you might have common interests with remain unknown to you. At the YAPCs it's a lot easier to talk to everyone, and you also have a better chance of someone pointing out someone else who you really should talk to. Although there are usually a few people I forget to find and say hello to, on the whole I do get to chat to some new attendees, and occasionally they'll come an introduce themselves to me, which is always a bonus. The smaller conferences just seem more sociable, which gives more of a fun element about them, which in turn makes them feel a bit more inclusive.

I think we still have plenty of room to manoeuvre, as I doubt we'll see many 400+ attended YAPCs for NA or Europe, so there is still lots of promoting worth doing. It all has a side effect of promoting YAPCs, Workshops, Hackathons, Perl and the community in general, not just in NA and Europe, but around the world. If people can't attend a YAPC, then we should be trying to encourage them to find a more local Perl Workshop. Both YAPCs and Perl Workshops are a great way to introduce yourself to the community and for the community to bring the best out in you. Another 100 or so attending YAPCs would be fantastic, and I'm sure the Perl Workshops around the world would love to see another 30-50 people attending too.

But as stated previously, promotion is the key. If you don't tell people how great you thought a YAPC or Perl Workshop was, how will others know that they should be attending the next one?

File Under: conference / opensource / people / perl / yapc
2 COMMENTS


Building the Perfect Beast

Posted on 17th August 2010

YAPC::Europe 2010 - Thoughts Pt 3/3 - Organising A YAPC

When considering whether to host a YAPC, potential organisers often have no idea what they are letting themselves in for. While it can be very rewarding, and a valuable experience, it is hard work. There are plenty of things to go wrong, and keeping on top of them can be quite daunting. However, when you first consider bidding you usually look to what's gone before, and over the past 10 years YAPC events have come on leaps and bounds. This year, YAPC::Europe in Pisa, Italy was no exception.

As mentioned in the previous post, the only real pitfall that Pisa organisers suffered was lack of promotion. The actual event pretty much ran smoothly. There were glitches with the WiFi network, but that seems to happen every year. This year once again, it seems network companies just don't believe us when we tell them that potentially 300+ devices will be wanting to connect all at once to the network. So although you could connect, the network was rather slow at times. Hopefully, future organisers can point to past experiences and impress on service providers that when we say 300+ devices we mean it! It's not just YAPC::Europe, as YAPC::NA has suffered too. Thankfully these problems didn't detract from a great conference.

For many attendees, the primary motivation for attending YAPC are still the talks. You get to see a wide range of subjects, hopefully covering areas of interest that suit all the attendees. However, this is extremely hard. During a few discussions during the event, I commented on the feedback from the YAPC::NA Conference Survey, which featured several comments from attendees, who felt that a beginner track would have been very useful. In retrospect, it might have been even better to have an Introduction To Perl tutorial before the conference, with the beginner track set aside for a variety of introductory topics covering aspects of the language, recommended modules, best practices or useful projects. The tutorial could then cover a lot of ground covering the basics, that would then be enough for beginners to not lose their way in the subject matter of some of the regular talks. Several people have commented that a beginner track, certainly for the first day, would be extremely useful. There have been several suggested approaches to this, but ultimately they are going to be a set of talks that are roughly the same each year.

At times speakers hear complaints that they are repeating talks, but with so many people attending for the first time every year, attendees often welcome having a chance to hear them. So if you do have an introductory talk that you think would benefit from a repeat performance, take in the comments from the talk evaluations and see what you can improve on, and submit it again the following year. I see some speakers benefiting from this to improve their speaking talents and gain more confidence in themselves.

The scheduling this year, from my perspective, was great. I only had 1 minor clash, and 1 clash where I would have liked to have seen all 4 talks. It's unlikely you'll ever get away with not having any clashes, but if you can gauge the subject matter and level of talks well, and don't put potentially overlapping talks together, you can reduce many such conflicts. This year the list of talks was online for a while before a schedule was published. This allowed those that were already registered a chance to highlight talks they were interested in. I don't know if this helped to guide the schedule, but it did seem a good opportunity to see exactly what talks were going to be popular. Having said, you can only rely on it for a short time, as getting the schedule published is really important both for raising the profile of the conference, and to persuade attendees to come to the event. Some conferences publish the schedule several months in advance, which can be hard to do, but does give potential attendees a chance to show their bosses why they should attend. Saying there might be some good and relevant talks rarely works.

This year the organisers made one of the best decisions ever regarding the schedule, and one that got appreciative comments from just about everyone. The talks started at 10am. In the past we have typically started around 9am, with some YAPCs starting as early as 8am. That early in the morning is a killer on the opening speaker. By starting at 10am, pretty much everyone was there every morning ready for talks. It made for a much more awake and attentive audience.

One aspect of the schedule that is down to the attendees to organise are the BOFs. This year, although several were suggested, I didn't see whether any of them happened. The one that looked likely, I would have attended had I been aware of it. To help these there needed to have been a BOF board by the registration table, which attendees can write their own schedule for. Having everything online is not very suitable for those who don't have laptops or cannot get internet connectivity. Plus a BOF Board helps to promote the BOFs to those who haven't heard of them before. Sometimes you just have to fall back to low-tech ;)

Another potential hazard for organisers is not considering the breaks and lunches. If your venue is in the middle of a city, town or very close to a variety of eating establishments, you can pretty much let your attendees fend for themselves during lunch. However, if they need to search for more than 15 minutes, then that can leave very little time for eating before they have to return to the venue. Due to the venue being quite a walk away from any potential eating establishment, it was rather important that they feed the attendees during lunch. As such they laid on a spread that was fantastic. It certainly avoided any unnecessary wandering into town trying to find something, and also meant we all had an hour for lunch where we could mingle and chat. And pretty much that's exactly what we all did. The breaks and lunches were always full of discussion. It gave us a chance to carry on points from talks, catch up with friends and introduce yourself to new people. If nothing else, this year's YAPC::Europe was extremely social.

As the saying goes, keep your attendees well fed, and you'll have a happy audience. That also means considering additional options, and it was good to see that lunch included a selection of vegetarian options too, as more and more attendees these days are vegetarian or vegan. For the breaks (and lunch if appropriate), try and include water, soft drinks, coffee and tea. Note that last one, tea. While much of Europe might prefer coffee, I can guarantee you'll get complaints if you don't provide at least English Breakfast Tea (we have a wider choice in the UK, but in the rest of the world, it always seems to be labelled as that). In Copenhagen they ran out every break time due to the caterers not anticipating the number of tea drinkers. Thankfully, for Pisa the drinks were very well stocked. A decent cup of tea goes a long way to keeping this attendee happy anyway ;)

The venue choice is always a difficult part of organising an event like YAPC, and largely depends on numbers. Over the last few years, several first choices have had to be abandoned because something hasn't worked out. The venue is never going to be perfect, but as long as there is plenty of room and everyone can get somewhere to sit then you've done well. You always need one room to hold everyone, but If you have some smaller rooms for the other tracks, try and avoid scheduling popular speakers or talks in them. Thankfully it doesn't happen often, and sometimes it can't be foreseen. This year Allison Randal did experience a bit of overcrowding in one of her talks, but no-one seemed to mind sitting on the floor or standing to hear.

The auction is always another trouble spot, and in recent years has rarely been necessary, as YAPCs usually make a profit these days. However, raising funds for the next years organisers, TPF or YEF is never a bad thing, as it all ends up helping to fund and promote Perl events. This year the Pisa organisers tried to be a bit different, and had it have worked as intended, then I think it would have gone down well. This year we had 3 tag teams trying to auction off 4 items each. Had it been kept to that, and the time limit of 5 minutes that had been suggested been rigorously imposed, then the auction would have been short and a lot of fun too. Unfortunately the time limits got abandoned, and some of the items led to a few bemused looks on the faces of the audience. If you've never been to a YAPC, then the auction can be a bit intimidating. None of us are as flush as we once were, so can't always afford to bump up the prices to levels we once saw in years gone by. Having said that, I do think we saw the highest price paid for a T-shirt, with Larry winning the the PIMC shirt off Matt Trout's back, thanks to a lot of friends :)

One point that Dave Rolsky made in his observations of the event, was regarding the price of attendance. We've now been running YAPCs for over 10 years and the prices have largely stayed the same in that time. There has been resistance to price increases, but 99 qr/Dollars|Euros|Pounds)/ is *really* cheap compared to other similar events. I do think there needs to be some alternative options, particularly for students, low-waged (or unwaged) and businesses, but a small increase in the standard price would, as Dave highlights, generate a significant amount of revenue. One aspect of the pricing that we've rarely pitched in the right way, has been for businesses wanting to send attendees, whether singularly or en-masse. It was commented to José at YAPC::NA in 2008, by someone that said that they had to pay for themselves, as their boss considered YAPC too cheap and therefore not a real conference. Having a business package that includes 1 or 2 tutorials in addition to the regular conference is one way to give value for money, but still charge a higher price. Lisbon tried this for 2009 and Riga are looking to use it for 2011. I hope it works, as it has the potential to encourage businesses to regard YAPCs as a credible training event for their employees.

Aside from the tower and the Cathedral there wasn't much to see in the town, which is probably a good thing, as it meant the town wasn't overly touristy or expensive. There were lots of choices for food in the evening, although mostly we all headed for the Piazza where we all met for the pre-conference meet-up. If you'd like your attendees to get a good flavour of your city, then it's worth investing time to point out evening social venues where attendees can meet-up. If you don't then the likelihood is they'll all head for the same place every night, as it's the only place they know how to get to.

If you have strong feelings (or even mild ones) about the conference, it would be great if you could take the time enter them into the Conference Survey. All the responses help the organisers of the future get a good idea of what attendees thought about the conference. In addition, please try and complete the talk evaluations, as I know the speakers do appreciate it. I spoke to a few speakers in Pisa who were very pleased to get the feedback, even if it wasn't always complimentary. Following some discussions, next year the talk evaluations will be simplified a little, so they will hopefully be quicker to complete.

As some may be aware I started writing a book last year, about how to organise a YAPC. After some feedback I had intended to make a second draft. Due to other commitments that hasn't happened as yet. Following further feedback from the YAPC::NA organisers and discussions with organisers and attendees of YAPC::Europe, as well as all the feedback from the surveys, I plan to pool those, together with the original feedback, and work on the next draft over the next month. Seeing the success of the git way, I'll be making the text available on Github, so any one can supply patches. My eventual aim is then to publish an ebook, together with a print on demand version, that can be used by organisers of YAPCs and workshops to help them plan and improve Perl events for the future. If you're interested in such a book, keep an eye out for updates in the near future.

Overall I enjoyed YAPC::Europe this year, and came away with several ideas from talks and discussions in the hallway track. My thanks to the Pisa organisers, you did a grand job. Now have a well earned rest. Next year Riga will be our hosts. With Andrew and his crew now having so many workshops and YAPC::Russians behind them, next year should be every bit as successful as this year. Good luck guys.

A final thought from YAPC::Europe in Pisa this year. Josette Garcia noted that 4 people who attended the very first YAPC::Europe were in Pisa. I was one of them, and I think Dave Cross, Nick Clark and Léon Brocard were the others. Of the 4 of us I think Léon and myself are the only ones to have attended every single YAPC::Europe. I wonder who'll break first :)

File Under: conference / opensource / people / perl / yapc
NO COMMENTS


Calling All The Heroes

Posted on 16th August 2010

YAPC::Europe 2010 - Thoughts Pt 2/3 - Promoting A YAPC

This year, YAPC::Europe was reasonably well attended, with roughly 240 people. However, a few weeks prior to the event, the officially registered attendees for YAPC::Europe 2010 was considerably lower. Although every year it seems that many register in the last 2 weeks, there is usually a higher number registered before then. So why did we have such low numbers registering, until just before the conference this year? I'm sure there are several factors involved, but 2 strike me as significant.

The first is the current dates for the event. As mentioned in my previous post, the Perl community attending YAPCs is getting older, and many of us now have young families. August is notoriously bad for anyone with a family, as the school holidays govern a lot of what you're able to do. Those that can take time out to attend the conferences also have to juggle that with family holidays. Employers are often reluctant to have staff away during August, as too easily they can become short-staffed due to others taking holiday. Having said that, the attendances haven't fluctuated that much in recent times, regardless of whether early/mid-August is chosen or late-August/early-September. Although, the exception does seem to be Vienna in 2007 which attracted 340 attendees. As such, when deciding dates for a YAPC, bear in mind that some of your potential attendees may find it difficult to attend, or only be able to decide almost at the last moment.

The second factor was a pitfall that this year's organisers fell into too. Lack of communication. Immediately prior to the conference and during it, there was lots of news and promotion. However, 6 months ago there was largely nothing. Although, we finally had about 240 attendees, it is possible that there could have been many more. Big splashes across the Perl community with significant updates (website launch, call for papers, opening registration and unveiling the schedule) are a great way to make people aware of what is happening and can generate a buzz about the event long before it begins.

This year I noticed that a twitter search for 'yapc' in the weeks before YAPC::Europe, featured mostly posts about YAPC::Brasil, and I'm currently seeing several posts for YAPC::Asia. Last year, José and Alberto kept a constant feed of news, snippets, and talk link posts onto twitter and other social network micro-blogging services, which helped to generate posts from others attending or thinking of attending. This year that potential audience attracted via the marketing efforts, seems to have been lower than in previous years. The results of the Conference Surveys will hopefully give a better picture of this.

In recent times the Perl community has talked about marketing Perl in various ways. However, promoting our own events seems largely left to the organisers. While the organisers can certainly add fuel for the fire, it's the rest of the community that are needed to fan the flames. In the past YAPCs and Workshops have been promoted across various Perl sites, and in various Linux and OpenSource channels, which in turn generated a lot of interest from attendees and sponsors. The latter target audience are just as important as the former. While we want more people to attend the events, the sponsors are the people who fund them to make the happen. But not marketing the events to get maximum exposure likely means there are potential sponsors who either never get to hear of our events, or are turned off by the lack of exposure the event is generating.

Although the events do manage to get sponsors, for the organisers it can often be a very traumatic process getting sponsors involved. Once you've made initial contact, you'll need to persuade them that sponsoring the event is a good way to market their company. If they're able to see photos online of the events (possibly including sponsor branding), or read blog posts that direct people to the conference website (with all the event sponsors listed), it gives potential sponsors a feeling that it may be a worthwhile investment. Some sponsors are strong supporters of OpenSource and want to give back, but a large number are looking to promote their own brand. They're looking to make maximum revenue for a minimum outlay. They want to see that funding events is going to generate further interest and brand recognition to their target audience. Exposure through blogs and other online sources all helps.

As I've implied, much of this exposure is down to the community. If you attended YAPC::Europe (or YAPC::NA or any other Perl event, including Workshops) have you written a blog post about it? Did you tweet about the event before you went, during or even after? Have you posted photos online and tagged them with the event, in a way that others can find them? YAPC::Brasil and YAPC::Asia attendees seem to be doing this rather well, and there is a lot we can learn from them. In the last week, there have been several posts by attendees of YAPC::Europe 2010, but of the 240 people attending, it really is a small percentage. And likewise I saw a similar kind of percentage posting about YAPC::NA this year too. Several years ago use.perl and personal blogs were full of reports of the event. What did you learn at the event, who did you meet, what aspects of Perl are you going to take away with you from the event? There is a lot you can talk about, even if it was to mention one specific talk that you felt deserved comment.

With aggregators, such as Iron Man, Planet Perl and Perlsphere, whether you post via use.perl, Perl Blogs or your own personal site, you can get the message out. Next year, anyone wondering whether attending a YAPC is worthwhile is likely to search for blog posts about it. Are they going to find enough reasons to attend, or persuade their manager that they should attend? I hope so. YAPCs and Workshops are a great way to promote what is happening in Perl, and by talking about them we can keep that interest going long after the event itself.

In Gabor's lightning talk, looking at Perl::Staff and events group, he highlighted the differences in attendances between the conferences. Typically a YAPC::Europe has 200-300 attendees, YAPC::NA has 300-400 and YAPC::Asia has around 500 attendees. However, FOSDEM (5,000), LinuxTag (10,000) and CeBit (400,000) all attract much higher numbers. It's a fair point that we should try and provide a presence at these other OpenSource events, but a dedicated language interest event is unlikely to attain those attendances. The hope though is that we may have a knock-on effect, with people seeing Perl talks and a good Perl presence at those other events, might just take more of an interest in Perl, the community and the various Perl specific events.

I'd be very interested to see attendance figures for other dedicated language conferences, particularly for Europe, as I think Perl is probably about average. The EuroPython guys certainly attract similar numbers to Birmingham. In the past I've done a fair amount of pitching Perl at Linux, OpenSource and Security Conferences in Europe and to Linux User Groups around the UK. Birmingham Perl Mongers undertook 3 "world" tours in 2006, 2007 & 2008 doing exactly that. It was great fun, and we got to meet a lot of great people. If you have a local non-Perl group, such as a LUG, would they be interested in a Perl topic? Are you able to promote Perl, the Perl community or Perl events to them? Sometimes even just attending is enough, as you'll get to talk to plenty of other interesting people. The initial 2006 tour was primarily used to promote YAPC::Europe 2006, which Birmingham Perl Mongers were hosting that year, and it did help to raise the profile of the event, and eventually got sponsors interested too.

One thing that the Pisa organisers did, specifically osfameron, was to broadcast Radio YAPC podcasts (Episodes 0, 1, 2 & 3). Genius. I got to listen to them after each day, but I can imagine many haven't been able to hear until they returned home. It would have been great to have something before the conference too, even just the news updates and some of the highlights to look forward. Interviews with the organisers and any registered attendees would have been great too. It was a nice touch to the event, and it's promotion, to be able to feature interviews with speakers and attendees to get their experiences. I hope future organisers can try something similar too.

There are several people trying to raise the profile of Perl at the moment, but it takes the whole community to support their efforts by blogging, talking beyond our community and promoting events to those who might not have considered treating the conference as part of their training. We have a great community, and one that I'm pleased to be a part of. I want the community and the events to continue for many years to come, and talking about them can only help that. It's why Matt Trout shouted at many of us to blog about Perl and promoted the Iron Man aggregation competition.

The Perl community and events are very healthy at the moment, we just don't seem to be talking about them enough. As the business cards state, we do suck at marketing. If we want to avoid the mistakes of O'Reilly at OSCON last month, and the badly named tags, then promoting YAPCs and your experiences at them, are a good way to show how it can be done right.

In my next post I'll be looking more at the YAPC event itself.

File Under: conference / opensource / people / perl / yapc
NO COMMENTS


Growing Up

Posted on 13th August 2010

YAPC::Europe 2010 - Thoughts Pt 1/3 - Young Blood & The Old Guard

Last week I was in Pisa for YAPC::Europe 2010. Although I was doing a talk about CPAN Testers, my intention was to keep a low profile and observe more. Having run the conference surveys for the past few years, it has been noticeable that the attendance has been changing. While there are new people coming along to YAPCs, the general average age is getting older. Marketing Perl to companies to encourage its use is one thing, but attracting people in general to the language is also important. The fact that for a notable number of attendees this is their first YAPC, probably means we are getting something right.

There were several European Perl Mongers that were noticeably absent this year. While some had posted apologies (mostly due to imminent baby arrivals it would seem!), others perhaps have moved on to other jobs, projects or languages, or their life means that they cannot commit to something like YAPC any more. While we miss them, it is a natural way for the community to evolve. It does give a chance for newcomers to become involved and this year I wanted to see who we are potentially going to see more of.

It seems we have quite a few people who are giving us, the Perl community, a fresh look and I think that the Perl community is rather healthy at the moment thanks to them. At least from a European perspective. YAPCs are an ideal chance for people to meet and discuss projects, which otherwise can take days or weeks via email and even IRC. Those new to projects can better introduce themselves and forge better communication channels with other project members, both during the conference and at the evening social events. I think it was Dave Rolsky who observed that the Europeans seemed more accustomed to putting down laptops and talking, rather than sitting in silence hacking away. There certainly seemed to be lots of discussion in hallways this year at least.

With all the fresh faces around, it's crossed my mind on several occasions, as to who is the old guard these days. There are several I could name who kind of fit the bill, and many of us have been around working on projects for quite a few years. Not necessarily hacking on perl itself, but certainly helping to build the Perl community. We have quite a vibrant community, one that I think is quite inclusive, supportive and appreciative. We have disagreements at times, but it's a community that seems to easily span age and experience barriers and is willing to learn from each other.

Keeping a low profile initially seemed to be working for me, that is right up until the afternoon of the last day. During the day, José had asked if I would help with his lightning talk, but not wanting to be part of any more talks, I respectfully declined. Little did I realise it was just a ruse, so he could say thank you to me for organising and running the YAPC Surveys. So much for not drawing attention to myself! After the Lightning Talks, brian d foy took centre stage to present the White Camel Awards. I was very pleased to see both Paul Fenwick and José Castro receive awards, and in fact was laughing at José as he realised one of the awards was going to him. However, José was almost in hysterics when he saw my reaction when I realised I was also receiving an award.

As I mentioned in my acceptance speech, I've never wanted an award for what I do. I do it because I want to, and because I love being part of this community. I had been asked before whether I would accept a White Camel Award, and I'd said no. Although I don't think the awards themselves are a bad thing, its just that I think others have been more deserving of them. I've been involved in many Perl projects over the years, and have largely hid behind them, as I've always felt the projects themselves are far more important than me. The fact that several people felt I needed to be acknowledged this year, regardless of my reluctance to receive the award, I guess means that sometimes I just have to accept that people would like to say thank you for the work I do. If like José, there was one person I should thank for introducing me to the Perl community, it would be Richard Clamp. It was Richard who gave me my first proper Perl job and persuaded me to go to a London Perl Mongers social.

Which sort of brings me to one of the projects I helped with last year, and I'm very pleased to see continuing this year. Introducing people to the Perl community is one aspect of the Send-A-Newbie programme. Edmund instigated the programme last year, and we managed to bring 3 people to YAPC, giving them a chance to experience the conference and the community. The hope was that they would use and benefit from the experience, and hopefully feel more empowered to contribute to the community. Then maybe be in the future, they might be able to attend future YAPCs. I was delighted to see Alan Haggai Alavi at this year's YAPC, and surprised to see him so soon. I was then even more impressed to hear what he has been doing to promote Perl in India, as this is exactly the kind of enthusiasm the Send-A-Newbie programme can benefit from too. I spoke briefly with Leon Timmermans, who was this year's attendee via the Send-A-Newbie programme, and again it seems we've found another deserving recipient.

With programmes like Send-A-Newbie, the Perl marketing efforts and the community in general, I'm very hopeful that we'll be seeing more young blood in the community in the years to come. However, it still needs some effort from every one of us to ensure that happens. Which brings me to my next post in this short series, which I'll be posting soon.

I've now been in the community for over 10 years, with Birmingham Perl Mongers celebrating their 10th birthday in September. I'm guess that means I'm one of the old guard now, which isn't bad for a C programmer who had a lot to learn all those years ago. I feel I've come a long way in the last 10 years, and it's been a fantastic journey. Perl and the community have changed immensely in those years, and I'm looking forward to seeing how the young blood and fresh faces now, take us in new and interesting directions over the next 10 years and more.

File Under: conference / opensource / people / perl / yapc
NO COMMENTS


All Over The World

Posted on 19th May 2010

Last year I went to 3 conferences, YAPC::NA, YAPC::Europe and LUGRadio Live. All very different in their own way, although all Open Source. Due to other projects, work and fanmily commitments, it has take quite a bit of time to review all the photos. After several months, I finally found some time to whittle them down to the selection I have uploaded here.

The first conference, YAPC::NA, took place in Pittsburgh, PA, USA. The team have been holding the Pittsburgh Perl Workshops for several years now, and by all accounts they had been very well received. With the YAPC set of conferences having started in Pittsburgh, at the Carnegie Mellon University where this conference also took place, the organisers were quite proud to promote a sort of home coming for the event. It was a good conference, though my first talk was somewhat problematic as we couldn't get a laptop to work with the projector. Thankfully my second talk went without a hitch. My thanks to confound for introducing me to 'xrandr', which solved all the problems I had getting Ubuntu talking to the projectors.

I also took the chance to visit the city of Pittsburgh and take a look around, with Abigail and myself taking an amphibious vehicle tour. The city has a lot of interesting places to see, and I'm glad I got to see the Heinz Building, the Stanley Theatre (where Bob Marley played his last ever gig), the venicular railways, and the site of the Three River Stadium (Yes, the car park! I'd seen a documentary about the building of Heinz Field and the demolishing of the Three River Stadium, so was even more intrigued to have a guided tour of the new stadium). If you're ever near the city, I recommend a visit, especially to see the flood levels of 1936 (the waters peaked at a rise of 46 feet above normal!).

The second conference, YAPC::Europe, was in Lisbon, Portugal. The conference itself was packed full of talks, though I think my lightning talk, which I'd been refining over the previous few months, generated the biggest reaction. Not surprising really, as it reminded people just how productive the Perl community was, particularly regarding CPAN.

I had originally thought about hiring a car and travelling along the Vasco da Gama Bridge (at 10.7 miles long, the longest road bridge in Europe), and do the circuit via the monument on the other side of the Tejo river, and back to Lisbon via the 25 de Abril Bridge (Lisbon's other bridge). I didn't in the end, but maybe I can save that for another time. Instead fellow Birmingham.pm'er Brian McCauley and myself walked around the city and took in some of the sights. When we got to the castle we managed to bump into a few other attendees (Paul Johnson, Aaron Crane and R Geffory Avery), who also had taken the advantage to do some sightseeing.

The last conference I attended was LUGRadio Live. For a number of reasons I didn't put forward a talk this year, but suggested JJ should give a talk instead. With the radio show no longer running, the conference had much more of a grassroots feel to it again. There ware some good talks, a couple of famous names, but mostly it felt like it was one big Linux User Group meeting, which to a degree it was, just a bit more global than your regular user group meeting ;) The conference was dubbed 'Back To Basic', but that really only applied to the extravagance. The quality of the conference was first rate. Being in Wolvehampton, just round the corner for me, I didn't take the opportunity to do any sightseeing, not that Wolverhampton is exactly the kind of place to do any sightseeing. As it happens I had taken Dan to the event, who loved it, especially building the lego models with all the other geeks. The following day was OggCamp, and although I would have liked to have attended, I had other commitments so had to pass. I think having the two events side by side though was a great idea, as it gives both events to feed off each other.

This year I'm currently only planning one conference, YAPC::Europe in Pisa, Italy. All being well I may get to see the tower, but as I'll be flying in and out just for the conference, I don't expect to see much more. I'm still undecided whether to submit a talk, as I'm trying to think of a suitable subject. I don't like repeating myself, but my two biggest profile Perl projects I've now covered for a couple of years (CPAN Testers and YAPC Surveys), so we'll see.

More photos to come, as I find time to get through the plethora of photos I've taken over the last year or so.

File Under: community / conference / lisbon / lugradio / opensource / perl / wolverhampton / yapc
NO COMMENTS


Long Time Gone

Posted on 4th May 2010

It has been quite a few months since I last posted here. Quite a few events and projects have happened and held my attention since I last wrote in my blog. And I still have a backlog of photos and videos from last year to get through too!

I did wonder whether anyone might think that after talking about Why The Lucky Stiff in one of my last posts, that I had done the same. Well for those who follow my CPAN Testers work, will know that CPAN Testers 2.0 has been a rather major project that finally got properly underway in December 2009. It's nearing completion, and I'll cover some of the highlights in a future post. Although it's been my most consuming project over the last 6 months or so, it hasn't been my only one. As mentioned in another of my last posts, I'm writing a book about how to host a YAPC. Due to other projects taking a higher priority, this has taken somewhat of a backseat for the time being, but I do plan on getting a second draft together within the next few months. I have looked into self-publishing the book and I'm now planning to have it formerly submitted with an ISBN (the internation book numbers) and supplied via print-on-demand print runs.

Another project that has been ongoing alongside my CPAN Testers work, has been my website management system, Labyrinth. This has been the website application I have been developing since 2002, and although several other Perl web frameworks have now been developed since, to lesser and greater degrees, Labyrinth has had the disadvantage of only having 1 core developer for the past 8 years. It's not an application that will revolutionise web development and deployment, but it has very successfully worked for a number of websites I have developed over the years. After having been relatively stable for the past year or two, I'm now cleaning up the code so I can properly release it as open source. This is mostly so that anyone wishing to contribute to CPAN Testers, or the YAPC Surveys, will then have all the code available to them. If anyone wants to use it and help develop it further, that would be a welcome bonus, but realistically other web frameworks have gained so much mindshare that I'm not expecting Labyrinth to make much of a dent any more. Not that that is a problem, as Labyrinth has made deploying websites so much easier for me, that I'll just be glad to let people help on CPAN Testers and the YAPC Surveys.

Speaking of the YAPC Surveys, YAPC::NA 2010 and YAPC::Europe 2010 are fast approaching. These will be next projects to get up and running. Thankfully the code base just needs a few upgrades to the latest version of Labyrinth, and some work on skinning the CSS to match the respective YAPC sites. All being well this should only take a few days. Then I'll be looking to release this version of the code base for anyone wishing to run similar surveys for themselves. I've already had one interested party contact me regarding a conference in October, so hopefully the code will be suitable, and only the questions need adapting. We shall see.

My other major project this year, also began back in December 2009. As some readers are well aware, I am an ex-roadie. From 1989-1994 I was a drum tech, lighting engineer and driver for Ark, one of the best Black Country bands ever. Not that I'm biased or anything ;) Last year the band got together for some rehearsals and planned a few reunion gigs. With interest gaining, an album was also planned. So this year, the band began recording and booking gigs. As a consequence the Ark Appreciation Pages desperately needed a makeover. I'll write more about what happened next in another post. Ark are back, and Mikey and I are delighted to be able to be involved with the band once again.

That's just a few of the projects that have taken up my time over the last 6-8 months. There are several others that I hope to post about, with family, time and work permitting. Expect to hear a little more from me than you have so far this year.

File Under: ark / book / conference / labyrinth / opensource / perl / website / yapc
NO COMMENTS


Senses Working Overtime

Posted on 9th July 2009

I haven't written here for a while. and I haven't written that much in my other blogs either. I should have, as I keep having ideas for posts that I figure I should get around to writing up, For various reasons my time has been taken up with other things. I've been very busy with some of my Perl projects, specifically CPAN testers and YAPC Conference Surveys, and relating to the latter, I've recently attended YAPC::NA 2009, have been a judge for the Send-A-Newbie programme and am currently judging the bids, along with the rest of the YEF Venue Committee, for YAPC::Europe 2010 from Keiv.pm and Pisa.pm. On top of that I've also been trying to spend more time with my family.

For the past year or so I've been working 15-20 a week on extra curricular Perl projects. Although I've enjoyed doing the work and I'm really proud of what I've managed to achieve, I do need to take time out for my family. I'm very grateful for the support Nicole has given me, and the fact that Dan and Ethne don't seem to mind me getting up before them at the weekend and waiting to have their breakfast, while I just do this next fix :) But they deserve some of my time now. That's not to say I'm disappearing or anything, just that the balance is slightly shifting in their favour.

Since leaving MessageLabs, I now have a job that is a bus ride away in the centre of Birmingham, which now gives me the freedom to work on the bus for at least an hour a day. It's actually quite amazing how much I can get done, when I have to focus on getting something done with only 30 minites between stops. I've missed travelling to work on the bus, as driving up and down the M5 every work day for the past 6 years has felt more and more of a drain. I used to read a lot, but that all stopped once I started driving to work. I'm now planning to start working through my unread bookshelf again :)

The new job is a bit different, as although they are not a web shop, my role is very much focused on web development. As a consequence I'm really enjoying it. They have a series of sites that have been developed over the years, without any real thought to standards and best practices. As a consequence, they now have me to help clean up their code and processes. I hasten to add that I'm not some kind of saviour, as the guys have been working to improve what they have for some time, but I think they do see me as a bit of welcome relief. It's also been nice to be able to bring a lot of the knowledge I've drawn on for the past 10 years or so, and be able to give their sites a fresh lease of life.

In future posts expect a quick run down of my experiences in Pittsburgh for YAPC|10, some thoughts about YAPC::Europe 2009 in Lisbon and several photos from the various gigs I've been to over the past year. Yes I still have lots of photos to sort though. Soon, I promise :)

File Under: family / perl / yapc
NO COMMENTS


April Skies

Posted on 1st May 2009

For those that might not be aware, I got made redundant on 31st March (the day after the QA Hackathon had finished). Thankfully, I start a new job next week, so I've managed to land on my feet. However, this has meant that I've ended up having the whole of April off to do stuff. My plan was to work on some of the Open Source projects that I'm involved with to move them further along to where I wanted them to be. As it turned out two specific projects got my attention over the last 4 weeks, and I thought it worth giving a summary of what has been going on.

YAPC Conference Surveys

Since 2006, I've been running the conference surveys for YAPC::Europe. The results have been quite interesting and hopefully have help organisers improve the conferences each year. For 2009 I had already planned to run the survey for YAPC::Europe in Lisbon, but this year will also see YAPC::NA in Pittsburgh having a survey of their own.

The survey site for Copenhagen in 2008 added the ability to give feedback to Master Classes and talks. The Master Classes feedback was a little more involved, as I was able to get the attendee list, but the talks feedback was quite brief. As such, I wanted to try and expand on this aspect and generally improve the process of running the surveys. Part of this involved contacting Eric and BooK to see if ACT had an API I could use to automate some of the information. I was delighted to get an email back from Eric, who very quickly incorporated an API that I could use, to retrieve the necessary data to keep the survey site for a particular conference up to date, even during the conference.

With the API and updates done, it was time to focus on expanding the surveys and skinning the websites to match that of the now live conference sites. The latter was relatively easy, and only required a few minor edits to the CSS to get them to work with the survey site. The survey site now has 3 types of survey available, though only 2 are visible to anyone not taking a Master Class. Those that have taken one of the YAPC::Europe surveys will be aware I don't use logins, but a key code to access the survey. This has been extended so that it can now be used to access your portion of the survey website. This can now be automatically emailed to attendees before the conference, and during if they pay on the door, and will allow everyone to feedback on talks during the conference. On the last day of the conference the main survey will be put live, so you can then answer questions relating to your conference experience.

I'm hoping the slight change won't be too confusing, and that we'll see some ever greater returns for the main survey. Once it does go live, I'd be delighted to receive feedback on the survey site, so I can improve it for the future.

CPAN Testers Reports

Since taking over the CPAN Testers Reports site in June 2008, I have spent a great deal of time improving it's usability for users. However, it's come at a price. By using more and more Javascript to dynamically change the contents of the core pages, it's meant that I have received a number of complaints that the site doesn't work for those with Javascript disabled or who use a browser that doesn't implement Javascript. For this reason I had decided that I should create a dynamic site and static site. The problem with this is that the current system to create all the files takes several hours for each set of updates (currently about 16 hours per day). I needed a way to drive the site without worrying about how long everything was taking, but also add some form of prioritisation so that the more frequently requested pages would get updated more quickly than those rarely seen.

During April, JJ and I went along to the Milton Keynes Perl Mongers technical meeting. One of the talks was about memcached and it got me thinking as to whether I could use it for the Reports site. Discussing this with JJ on the way home, we threw a few ideas around and settled on a queuing system to decide what needed updating, and to better managed the current databases to add indexes to speed up some of the complex lookups. I was still planning to use caching, but as it turned out memcached wasn't really the right way forward.

The problem with caching is that when there is too much stuff in the cache, the older stuff gets dumped. But what if the oldest item to get dumped is extremely costly on the database, and although it might not get hit very often, it's frequent enough to be worth keeping in the cache permanently. It's possible this could be engineered with memcached if this was for a handful of pages, but for the Reports site it's true for quite a few pages. So I hit on a slightly different concept of caching. As the backend builder process is creating all these static files, part of the process involves grabbing the necessary data to display the basic page, with the reports then being read in via the now static Javascript file for that page. Before dropping all the information and going on to the next in the list, the backend can simply write the data to the database. The dynamic site can then simply grab that data and display the page pretty quickly, saving ALOT of database lookups. Add to the fact that the database tables have been made more accessible to each other, the connection overhead has also been reduced considerably.

The queuing system I've implemented is extremely simple. On grabbing the data from the cache, the dynamic site checks quickly to see if there is a more recent report in existence. If there is, then a entry is added to the queue, with a high weighting to indicate that a website user is actually interested in that data. Behind the scenes the regular update system simply adds an entry in the queue to indicate that a new entry is available, but at a low weighting. The backend builder process then looks to build the entries with the most and highest weightings and builds all the static files, both for the dynamic site and the static site, including all the RSS, YAML and JSON files. It seems to work well on the test system, but the live site will be where it really gets put through its paces.

So you could be forgiven in thinking that's it, the new site is ready to go. Well not quite. Another part of the plan had always been to redesign the website. Leon had designed the site based on the YUI layouts, and while it works for the most part, there are some pages which don't fit well in that style. It also has been pretty much the same kind of style since it was first launched, and I had been feeling for a while that it needed a lick of paint. Following Adam's blog post recently about the state of Perl websites, I decided that following the functional changes, the site would get a redesign. It's not perhaps as revolutionary as some would want, judging from some of the ideas for skins I've seen, but then the site just needs to look professional, not state of the art. I think I've managed that.

The work to fit all the pieces together and ensure all the templates are correct is still ongoing, but I'm hopeful that at some point during May, I'll be able to launch the new look websites on the world.

So that's what I've been up to. I had hoped to work on Maisha, my other CPAN distributions, the YAPC Conference Survey data, the videos from the QA Hackathon among several other things, but alas I've not been able to stop time. These two projects perhaps have the highest importance to the Perl community, so I'm glad I've been able to get on with them and get done what I have. It's unlikely I'll have this kind of time again to concentrate solely on Open Source/Perl for several years, which in some respects is a shame, as it would be so nice to be paid to do this as a day job :) So for now, sit tight, it's coming soon...

File Under: community / conference / opensource / perl / website
NO COMMENTS


Back On Line

Posted on 16th February 2009

After the last few weeks of trying to access Twitter from the command line, I set about writing something that I could expand to micro-blog to any social networking site that supports many of the Twitter API type commands. At the moment it only works with Twitter and Identi.ca, but my plan is to look at creating plugins, or more likely to allow others to create plugins, that can enable the tool to interact with other micro-blogging sites.

After trying to think of a decent name, I finally settled on Maisha. It's a Swahili word meaning "life". You can grab the code from CPAN as App-Maisha.

Currently you'll need to use the standard Perl install toolset to install the application, but ultimately I'd like to have something that you can install just about anywhere without having to go through all the headache of installing dependencies. I'll have a go at doing an .rpm and a .deb package release, and will also try using PAR. It would be nice to have this as a standalone application that just about anyone can use, but for now CPAN will have to do.

My next immediate step is to look at writing something that interfaces to Facebook without requiring a developer key or any such nonsense. It will probably have to involve a bit of screen scraping, unless there is some more official API, but as yet I haven't found it. Everything regards Facebook applications seems to centre around the developer application that can do all sorts of dubious things, but mine is purely for the user to control from their desktop, not a 3rd party website/server. Thus giving them a developer API key assigned to me is wholly inappropriate. It would be nice if they had a restricted User API, which allows you to update your status and look at your friends' statuses, but I think I'll be in the minority wanting it.

File Under: community / internet / opensource / perl / technology
NO COMMENTS


You Know My Name

Posted on 9th February 2009

Having mentioned twitter in a recent post, I thought I would mention a project I decided to look at recently. The original project, twittershell, is not my own, but it appeared to be the closest to what I was after, a command line interface to Twitter, with the bonus of it being written in Perl. As a consequence of the latter, I was able to hack on the code and submit a patch to do a lot of what I wanted. I currently run the patched version, and it runs rather nicely for me. However, there is something missing.

Twitter is no longer the only micro-logging service and as such I've also signed up to identi.ca. With the APIs being pretty much the same, it should theorectically be simple to plugin an identi.ca interface to twittershell. Except it isn't. Unfortunately twittershell is written with only the Twitter API in mind. To intergrate identi.ca and other micro-blogging services, it requires a rewrite. So that's where I'm currently at. The original twittershell project hasn't been touched in over a year, so I'm hoping the orignial author won't be offended by me forking the code to a new project.

However, what do I call the new project? I would rather it not be something that identifies itself with any specific blogging service, as I would like it to have a broader appeal, that encourages others to add plugins should a new service come along. I realise this project will likely have limited appeal, as iPhone and GUI apps seem the in thing, but I want something that I can run via ssh/screen on my home box and not have to worry about watching some app running on the desktop.

One idea I had was to call it 'Mazungumzo' (Swahili for talk or conversation ... an idea stolen from Joomla!), then I thought of 'Maisha' (Swahili for life). I did look up some Welsh words, but doubt anyone would be able to pronounce them ;) I also thought of 'Rambler', but that might have too many connections to someone who goes walking across hill and dale of a weekend. So any good ideas for projects names?

File Under: internet / perl / technology
NO COMMENTS


Guiding Light

Posted on 6th February 2009

In 2006 I, along with 3 others from Birmingham Perl Mongers, organised the 2006 YAPC::Europe Perl Conference. It was thankfully a great success and invigorated several with ideas of things that they could do to join or create communities. Whether that was forming a local Perl Monger user group or starting a code project that would eventually be submitted to the CPAN. However, one person was inspired to go to another YAPC the following year and then submit a talk and speak at the 2008 YAPC::Europe Perl Conference. Had the 2006 conference not been in Birmingham, UK in 2006, Edmund would likely never have gone to a YAPC, and never realised how valuable they are. Not just in terms of the presentations and speakers, but of the communities and projects that are discussed, that he might not otherwise be aware of. And perhaps most importantly, realise just how easy it is to be included into the community and how easy it is for everyone to make a difference.

At the conference dinner in 2008, Edmund was struck by the lack of younger members of the communty in attendance, and started to think about why. For some time I have been trying to understand what we as a community can do to bring new people into the community, and although my perspective has focused on YAPCs, it equally applies to projects and local user groups. However, there is one aspect that I had neglected, that was obvious to Edmund. Funding. Most of those we are trying to encourage to come along to a YAPC are likely to be unwaged or on low wages, and cannot afford the costs of travel and accommodation for 4-6 days.

Last week Edmund launched the Send-A-Newbie website, with the support of the organisers for the 2009 YAPC::Europe Perl Conference to be held in Lisbon, Portugal, together with several members of the Perl community who have voiced approval. It is a great idea, and is a great way to enable students in particular a chance to attend the biggest Perl developer conferences in Europe.

The initaive aims to send at least 6 people, although even if only 1 person is selcted to attend this year, I would consider it a success. As it happens some grant applications have already been received, so it is likely that at least 1 person will attend thanks to the programme. Hopefully more will be approved for grants providing the funding can be obtained.

So how can you help? Well if you have the ability to do so, please consider donating. Mentioned the programme to anyone who you think might be a worthy recipient of a grant, and get them to apply. Mention it at your local user group, and see whether anyone can help with a donation. In order to keep YAPCs and the Perl community healthy we need to encourage potential future stars that attending the conference is a worthwhile oppotunity. If they could benefit from a grant to cover their travel and accommodation costs, then it really is in yours and their interest to do something about it. Applications will be accepted until 1 June 2009, so there is plenty of time yet to promote and apply for grants.

File Under: community / conference / education / opensource / people / perl / yapc
NO COMMENTS


Washing of the Water

Posted on 27th August 2008

A couple of weeks ago I was in Copenhagen for YAPC::Europe, which was a blast. I did my Understanding Malware talk, which seems to have gone down well, and the posters even better!

Before leaving the UK I finally bought a new camera, a Canon EOS 40D. Unfortunately this was my first time using the camera and I was a little disappointed that I wasn't able to get the same quality of photos as my Fuji FinePix 5100. As such don't expect anything too much from these photos. Hopefully over the coming months I'll get used to the camera and improve the picture quality.

For some personal observations of the conference, see my use.perl post about it. I may do a more detailed write-up about the talks I saw and the discussions I had too at some point, but that's it for now. Anyway, enjoy the photos.

File Under: community / conference / copenhagen / opensource / perl / yapc
NO COMMENTS


Icing Sugar

Posted on 11th April 2008

A while ago a friend of mine brought over a bag of sweets to show me, as he knew I'd appreciate the name :)

Unless you're not as impressed, people in the Perl community generally find anything which can be tenuously tied to the Perl programming language amusing. We're like that. Perle Beer always seems to go down well too :)

File Under: humour / perl
NO COMMENTS


Route 66

Posted on 7th March 2008

As mentioned in a previous post, I'm planning to attend YAPC::NA in June. I've now submitted my talk proposals for Understanding Malware and an updated version of How To Be A CPAN Tester. The deadline for proposals is next week, so hopefully I'll hear fairly soon whether they have accept either talk. I'm also planning to host a CPAN Testers BOF for testers, authors and interested parties to meet and discuss issues and/or the future of CPAN testing.

File Under: chicago / conference / opensource / perl / yapc
NO COMMENTS


A Light In The Black

Posted on 5th January 2008

Now that I'm looking to another year of the Birmingham.pm World Tour, with visits to a number of UK LUG and Perl Monger groups, LUGRadio Live (UK not US), the UKUUG Spring Conference in Birmingham, YAPC::NA and YAPC::Europe, as well as possibly a few European Workshops too, I need to start think what I'm going to present. I like the fact I can go to Linux based groups and conferences and talk about a variety of Perl topics, as although I might not be an expert, I know enough to give an introduction in several areas at least. But for more Perl specific technical events, I really need to stick to what I know.

The problem is that I feel I've done enough with CPAN Testing, Phrasebooks and Selenium for the time being, and it does get a bit boring for both me and the audience if I'm repeating myself every year. I may do some update on CPAN Testing, as there are likely to be changes in the coming year, a lot of which is being worked on currently, but what else is there that I could present that would be of interest to somebody?

One talk subject that has crossed my mind has been to do something like 'Labyrinth - A Perl Success Story'. It's been commented a few times that within the Perl community we talk a lot about the possibilities (particularly with frameworks) rather than getting to the finished product. While Labyrinth might not be for everyone, it might possibly be something that works for some, and as a consequence might interest people who have been asking me what it is and why I wrote it. However, although it is related to web and content management it isn't the next Catalyst or the new Jifty. You might be able to draw similarities between them all, but there are also many differences. Labyrinth isn't a framework as such, it's not meant for high-availability websites, and it also doesn't have the large development team knocking out code and fixing bugs that the others have. It's just me. But it might have just enough functionality and usability for someone to pick it up and get a site running how they want it to work, without having to understand the magic internals of frameworks like Catalyst and Jifty. I wouldn't be talking about the internals anyway, as I would prefer to give examples of how I solved problems and interesting asides that led me to learn something new about web design. I'm just not sure enough people would find it that interesting.

Further topics that come from the guts of Labyrinth, and are things that I have been keen to see how other people solve the same problem, are user input validation and content output correction. At the moment Labyrinth handles these within the same codebase, and it works rather well. However, it seems rather the wrong thing to do, to present a talk where the code to do the job isn't on CPAN and is embedded in another system. As a consequence I've been thinking about abstracting the code out of Labyrinth and releasing it separately. It might make for an interesting discussion and may provide people with an reasonable example of how they can use one solution to treat their input and output.

I've also started thinking about doing a short talk along the lines of "My Favourite CPAN Modules". A number of people have done this in the past and at one London.pm meeting several years ago, Leon presented one that got me looking up a few modules I'd not really heard of before. It's probably a talk better aimed at local group technical meetings and maybe a Workshop if appropriate, but I've also been thinking it might be better to actually to structure several talks of this style, but with a theme. So one talk would be "The Web Edition" and feature several modules useful for website development, another "The Test Edition" feature several useful Test modules, and perhaps also "The Mail Edition" with a selection of useful email modules. I've made an attempt at this style of talk before, but got too involved with the mechanics, when really all you need is a quick flavour of what the module can do, with enough references for you to go and find out more yourself.

I still need something more concrete for LUGRadio and the YAPCs, but at least I have some ideas to work with now. If anyone has other suggestions, please let me know.

File Under: community / conference / labyrinth / linux / perl / yapc
NO COMMENTS


Light Of Day

Posted on 25th August 2007

Back last year, I went to LUGRadio Live and was extremely impressed, as most people were, with the plasma screens around the building, particularly with the imagery they were displaying. It turned out that Aq had written it as a quick PHP/HTML hack. It certainly did the job and impressed me so much that I asked if I could use for the YAPC::Europe conference we were hosting in August. Aq was delighted.

The original code was written in PHP, but seeing as I don't do PHP, I rewrote the whole thing in Perl. I simplified some of the HTML and CSS, but essentially it was still the same concept. We lauch the code for YAPC::Europe and again people were suitably impressed.

Since last August I've been meaning to package up the code and release with a proper Open Source licence. I asked Aq whether he minded me using the Artistic License as used with tradional Perl libraries, and he was happy to release it. So here it is ... The Plasma Application.

All being well the guys in Vienna might be using it for YAPC::Europe 2007, but we'll have to wait and see.

File Under: conference / opensource / perl / web
NO COMMENTS


Dreams Never End

Posted on 8th August 2007

After promising a while ago to upload some of my code, I've created a new section on the site. Click the Code tab on the menu at the top and you'll see what I've done.

My first launch is the latest version of my dbdump.pl utility. I use it to backup my databases to remote servers. It supports MySQL and PostgreSQL at the moment, but potentially it could support others. At some point I'll get around to packaging other utilities too. If you find the code useful, please let me know.

File Under: database / opensource / perl
2 COMMENTS


Both Ends Burning

Posted on 13th July 2007

During José's talk, 'The Acme Namespace - 20 minutes, 100 modules', at YAPC::NA in Houston, he mentioned one of the Acme modules that accesses the info for a Playboy Playmate, Acme::Playmate. After he mentioned it, Liz "zrusilla" Cortell noted that she used to work for Playboy and worked on the site that was screen scrapped by the Acme module, informing us that she wrote the backend in Perl too, "so you see it was Perl at both ends". At this point the room erupted, Liz got rather red and I'm sure wished the ground would swallow her up :)

Despite the rather salacious connotation that can be drawn from that remark, it was a phrase that struck me later as being rather more descriptive of the state of Perl. I started to think about the community, business and the way Perl is perceived. Drawing a line with the individual at one end, moving into community through small businesses and onto corporations at the far end, we can see Perl is not only used at both ends, but all the way through. But people still ask isn't Perl dead?

Perl hasn't died, in fact it's probably more vibrant now than it has been for several years. The difference now though is that it isn't flavour of the month. I did a Perl BOF at LUGRadio at the weekend, and it was a subject that got brought up there. Is Perl still be used? It would seem that Perl publicity to the outside world is extremely lacking, as several non-Perl people I've spoken to over the past few months have been surprised to learn that Perl is used pretty much in every major financial institution, in email filtering or network applications, for the Human Genome project (and bioinformatics in general) and pretty much every type of industry you can think of. It isn't dead, it just isn't sticking it's head above the parapet to say "I'm still here".

Last year at YAPC::Europe, Dave Cross talked about speaking in a vacuum. Inside the Perl community we all know that perl is great and gets the job done, but what about the people who are struggling with other languages, or project managers and technical architects who are looking at what skill set they should be using to write their new applications? What about big business that is continually confronted with the marketing of Java from Sun or .Net from Microsoft?

I see Python gaining momentum simply because several in the Linux and Open Source communities started using it to see how good it was, and now with Ubuntu using it pretty much exclusively, it has gained a large foothold with the wider developer community. Ruby has been seen as great for creating flashy websites, but beyond 37 signals, I've not heard of any big name sites that have been created with it. It gets featured at every Open Source conference and developers generally seem to think its really cool, but I'm still waiting to hear of any big take up outside of the cool, hip and trendy set. Maybe that's Perl's problem. It isn't cool, hip and trendy anymore, it's part of the establishment, part of the furniture. Does the job, does it well and without any fuss.

Perl has generated such a great community, that we seem to have forgotten that there are other communities out there, and they've partly forgotten us too. YAPCs are great conferences, but they grew out of the desire to have more affordable conferences for the developers, students and self-employed. Their success has been to the cost of Perl people wanting to go to other Open Source events such as OSCON, and keep Perl presence in the wider developer communities going. As a consequence Perl is almost seen as an add-on for legacy reasons to those conferences.

Looking back at that line I drew at the beginning, although I see Perl in our community, it doesn't feature very much in the wider communities, and as such small businesses don't notice it so much and look to other languages to develop their applications. The individual or hobbyist still uses it, and the corporations would struggle to remove it now, so to the outside world Perl is very much at both ends, but only at both ends. It's lost its focus in the middle ground.

At LUGRadio this year, I kind of felt rather relieved that people who came and spoke to me, knew me for being part of the Perl community. Most of these people are hardcore Linux, C or Python developers and although several know Perl, don't often use it. I've spent a lot of time speaking at Linux User Groups this year, and plan to speak at more later in the year. I've also been invited to speak to the PHP West Midlands User Group, invited to attend PyCon and will be attending GUADEC next week, but it's hard work to try and remind these other communities that Perl is still there. Although the personal touch certainly does help, I can't help but think there needs to be another way to promote Perl. This isn't about success stories (although they do help) or about talking at conferences and user groups (although they are just as important), but about reaching to the other communities and thus small businesses to remind them that Perl is still a viable choice, and that rather than competing for market share, the different languages can work together.

Having spoken to some developers of other languages, I'm amazed that the FUD of all Perl is unreadable, obfuscated and too hard for the beginner to learn properly is still being peddled. Challenging that mentality is a bit of a battle, but I've had to state on several occasions that you can write unreadable, obfuscate and unmaintainable code in any language, and in fact most of the respected Perl community and much of CPAN strives to write readable, clear and maintable code. It seems the Perl code from over 10 years ago and the dodgy scripts of certain archives are still poisoning the well.

Part of the problem (possibly fueled by the above FUD) that we have in the UK is overcoming the fact that several new Open Source initiatives don't even feature Perl when they talk about Open Source languages. If the networks that work between the communities and small business aren't promoting us, then it's going to be a tough slog. I've already written emails to the National Open Centre and tried to get OpenAdvantage to be more inclusive, but there are other similar initiatives, both here in Europe and in the US that need reminding too. Once they're helping to promote Perl, then it might just be something that Universities and Colleges include in the curriculums again. From there small businesses will be able to see that there is a pool of Perl developers they can employ and Perl again becomes a viable choice.

I firmly believe Perl 5 will still be around in 10 years time. Whether its running on Parrot, within Perl 6 or as it is now remains to be seen. I was asked to describe Perl 6 at the weekend and responded with a generalisation of "Perl 6 is to Perl 5 as C++ is to C". C++ took C into another realm, but C is still around. I just hope that the constant confusing information given out about Perl 6 to non-Perl people, isn't the reason why some think Perl 5 is all but dead.

The theme for the 2005 YAPC::Europe in Braga was "Perl Everywhere". I don't think that's true, but I wish it was :)

(this has been cross-posted from my use.perl journal)

File Under: education / opensource / perl / rant / technology
NO COMMENTS


I'm Bad, I'm Nationwide

Posted on 5th July 2007

The 2007 YAPC::NA Organisers

The 2007 YAPC::NA Organisers

Finally got the time to sort through my photos from last week. From over 2,000 photos, I've got them down to just over 700. There are still a few in there that aren't quite as good as I'd like, but then until I can freeze people in time before taking the shot, I'm going to struggle with the current camera. I'm looking at to getting a DSLR at some point, so hopefully I won't get so many blurred pictures then. Still I'm pleased I managed to get quite a selection that I did like.

For those who discover this entry by searching for YAPC::NA, here are all the photos I have online:

I also took some videos of Luke Closs and the Lightning Talks, so once I've converted them I'll get those online too.

Last week was a lot of fun, and I'm glad I got to go. Looking forward to YAPC::Europe now :)

File Under: conference / houston / opensource / perl / photography / space / technology / yapc
NO COMMENTS


The Real Me

Posted on 13th June 2007

I've had to turn down a fantastic opportunity today. One of the LUGRadio presenters isn't able to make the recording of the show tomorrow, and Aq contacted me to see if I'd be up for being a guest presenter. I'm gutted as they've been meaning to get me on the show for a while, and now would have been an ideal opportunity to plug YAPC::NA and YAPC::Europe.

Not sure who isn't able to make it, but as Adam Sweet is now a regular, they can't call on him to be their stand-in guest any more :) Hence why my name cropped up. Hopefully they manage to recruit another member of the WolvesLUG massive, but I'll definitely be up for another chance to stand in. Hopefully, I haven't scotched my golden opportunity.

File Under: linux / lugradio / perl / technology
NO COMMENTS


Rockin' the Paradise

Posted on 25th May 2007

The Sears Tower, Chicago

The Sears Tower, Chicago

In June 2006 I attended and spoke at the YAPC::NA Perl Conference in Chicago. It was great to meet up with several people I'd met in Toronto, but it was also a bit odd discovering that I wasn't as anonymous as I was last year. In Europe more people recognise me as I've been to every YAPC::Europe Perl Conference and have been involved in several other activities where people have got to know me. But apart from my appearance at OSCON in 2000 (when nobody bar the London.pm guys knew me), I wasn't expecting to be so easily recognised. It was good to put a number of names to faces though and it was a very enjoyable conference.

There was one downside though. On the morning of the last day of the conference I got called by work. Seeing as I was part of the CPR team, I was oncall 24/7 regardless of the fact I wasn't in the UK. Thankfully I didn't have to worry about it conflicting with my talk as I'd already done it. However, it did mean I had to disappear for a while as I set up jobs to go and do the task I needed to do. I'm rather glad that firstly I used screen and secondly that I'd got all the jobs running that I needed to. As while getting up to plug in the power cable, my laptop got knocked on the floor. Had I powered down and shut the laptop, it might not have been so bad, however, it was open and running. I managed to truly trash the laptop, as later investigation seemed to indicate the bus connector to both the DVD/CD writer and the hard disk drive had been damaged beyond repair. Despite swapping the HDD to another laptop, it also appears I complete wrecked the connectors to the HDD. There wasn't enough on the drive (apart from the photos I'd taken over the first few days) to warrant paying for it to be scanned professionally, so it got binned.

At first I thought that I'd now lost all my photos from the conference and was a little downhearted. As a consequence I didn't take the camera to the speakers dinner event that Josh and Heather had laid on after the conference, and I didn't bother taking any photos the following day either. Thankfully, JJ gave me a program to recover image files from portable media, and got back pretty much everything bar the first day of the conference, which unfortunately are now lost due to me writing over them with pictures from the last day. The lesson I've learnt is that to use multiple media to record photos and don't write over anything until you've had a chance to back them all up safely.

The conference was great though, and I did enjoy wandering around the city. On the first day Cog and I went up to the Skydeck of The Sears Tower, walked down to see The Water Tower and visited The Apple Store, where we caught quite a cool band, called The Ruse. For the Early Arrivals Dinner, we ended up near Wrigley Field, where the White Sox play, while the conference itself is close to ? where the Cubs play. Didn't get to see a game, but I do plan to revisit the city again. I never even took the time to check out any blues clubs, so I owe it to myself to go back really ;)

This year's conference is in Houston, so I'm hoping I can go and visit the Saturn rocket at The Lyndon B Johnson Space Centre while I'm there.

File Under: chicago / conference / perl / yapc
NO COMMENTS


Mr Roboto

Posted on 31st March 2007

Following my interviews yesterday for other roles within the company, I've been offered the role of QA Automation Analyst. An unusual role in that I'm going to be looking at ways to enable the automation of current test procedures, so we can have complete regression tests running overnight, allowing the testers to concentrate on edge cases and the like.

It's pretty much going to be a similar excise to automating CPAN Testing, so I should be quite suited to it. I've been surprised that there isn't already some automation, as it's been talked about for some time. I guess they've just needed someone like me to be around to take advantage of :)

It will add more validity to me going to conferences, talking about testing, and being involved in testing workshops, etc, so that makes me happier. If I'd stayed in a traditional development role, it might have been more difficult to justify several people going to YAPC::Europe. This way it'll just be me from the QA department.

File Under: job / perl
NO COMMENTS


Some Rights Reserved Unless otherwise expressly stated, all original material of whatever nature created by Barbie and included in the Memories Of A Roadie website and any related pages, including the website's archives, is licensed under a Creative Commons by Attribution Non-Commercial License. If you wish to use material for commercial puposes, please contact me for further assistance regarding commercial licensing.