Memoirs of a Roadie

Let's Go Crazy

Posted on 31st May 2016

Last weekend saw me in Rubgy for the 9th QA Hackathon. This is a Perl event where the key developers for CPAN, PAUSE, MetaCPAN, CPAN Testers, Dist::Zilla, Test2 and many other Perl 5 and Perl 6 projects, get together to discuss problems, future plans and collaborate on code.

Although I was a co-organiser of the event, I really would like to thank my fellow co-organisers; Neil Bowers (NEILB) and JJ Allen (JONALLEN). Without these guys, organising this QA Hackathon would have been tough, as they really did all the hard work. Also many thanks to Wendy for keeping us fed with nibbles, keeping notes and generally making sure we all stayed focused. An event like this needs a team, and they are an awesome team.

My main aim for this event was to meet Doug Bell (PREACTION). Back last summer, the CPAN Testers server had some severe problems, which meant we had to switch to a new physical server. It was at this moment I realised that I couldn"t do this alone any more. Doug stepped up and started to take over the reins, and has done a great job since. However, I"d never met Doug, so this was going to the first opportunity to catch up in person. After only a few moments of saying hello, I knew we had found the right person to take over CPAN Testers. Doug has a lot of great ideas, and is more than capable of taking the project to the next level, which is where I wanted to see it grow to, but knew it needed fresh eyes to take it there. I feel immensely confident that I have left the keys in capable hands, and with the ideas Doug has already shown me, I expect bigger and better things for CPAN Testers' future. Please look after him :)

On the first day Oriol Soriano Vila (UREE) introduced himself to Doug and I. Oriol was suggested to the organisers by his employer, and after explaining what the event was about, Oriol was even more enthusiastic to attend. I'm glad he did, as he is another great asset to both CPAN Testers and Perl. Although we have referred to him as our "intern", Oriol has proved he was just one of the team. He has some great ideas, asked all the right questions and had results by the end of the hackathon too! You can read more on his own blog.

So once we got our introductions out of the way, we started looking at a high priority problem. One that had been reported in two different ways, but was in fact the same problem. The Summary RAG bars and the release database (as used by MetaCPAN). In turns out the problem was straightforward. After the server crash last year, The database scheme used to rebuild the new server was missing a new column in the release summary table, and thus wasn't getting updated. Once that was fixed, it was "simply" a matter of rebuilding the table. Sadly it took the whole weekend to rebuild, but once completed, we were able to start regenerating the release SQLite database. That took a week, but I"m pleased to say all is now updating and available again.

While that was rebuilding, I started to take a look at some other issues. After introducing Oriol to our family of websites, he registered for the wiki, and spotted a problem with the registration process. After some tinkering, I got that working again. I've no idea how long it's been a problem, but apologies to anyone affected.

In the introductions at the beginning of the event, Leo Lapworth (LLAP) mentioned that he was hoping to refine MetaCPAN"s use of Fastly, and was interested in helping anyone else who might be interested in using the service for their project. I got Leo to sit with me for a while and he gave me a good run through of what the service is, what it can do, and why we should use it for CPAN Testers. I didn"t take much convincing, and quickly opened an account and started to move the main family of websites to it. We have since seen a slight drop on hits to the server, but I expect that to improve as the static pages (the individual reports) are cached. Even the dynamic pages can benefit from their caching, as although many will change throughout the day, only a small portion are updated more than once an hour. Once we learn more about Fastly, expect to see some better response times for your page hits.

Talking with Christian Walde (MITHALDU), he wanted to help with the performance of the websites, particularly the Reports website. However, with the rebuilding ongoing, the server wasn't in the best place to really evaluate performance. He did happen to mention that the reports he was getting from the mailer were coming through as garbage. After some investigation, I discovered that the mailer had not been upgraded to use Sereal, which is now our serialiser of choice for the reports stored in the database. With that fixed, together with some further improvements and with all tests running, we put it live and waited. The following morning Christian report he had readable reports coming through again.

One aspect for testing the Reports site, and one that would have restricted Christian to evaluate the performance, is that apart from mine and Doug"s development machines, there is no stable installable full instance of the CPAN Testers Report site, including databases and cron scripts. As such, Doug has been working on providing exactly that. It has been on my TODO list for some time, as some of the bug reports and issue requests would have been quashed much more efficiently had others been able to fire up a working site and be able to send a pull request. You can read more about Doug"s progress on his blog, and hopefully this will encourage more people in the longer term to get involved with CPAN Testers development work.

Throughout the weekend I worked on cleaning up some of the templates on the various websites, ensuring that sponsors were correctly attributed, and fixed several bugs in some of the associated distributions. Not all have been pushed to CPAN, but work is ongoing.

Having finally met, Doug and I went through all the website logins and social media accounts, and made sure he had all the keys. The handover process has been a longish one, but I didn"t want to overwhelm Doug, and wanted him to find his feet first. After this weekend, expect more posts and updates from him rather than me. Please look after him :)

I also joined in some for the discussions regarding the CPAN River and the naming of the QA Hackathon. Neil has written up both admirably, and while I didn"t contribute much, it was good to see a lot of healthy discussion on both subjects. Regarding the naming of the event, I do think it's a shame that the likes of Google have turn the word "Hackathon" into the concept of a competition event, which the QA Hackathon event is definitely not. Ours is about collaboration and planning for the future, with many of the key technical leads for the various toolchain and associated projects within Perl 5 and Perl 6. I don"t have a suitable name to suggest, but I would recommend ensuring the acronym could not be used negatively.

In the coming weeks, I hope to collate all the website tests I run prior to updating the CPAN Testers family websites, and handing over to Doug for his new development environment for CPAN Testers. This will hopefully enable easier access to anyone wanting to help fix problems on the websites and backends in the future. 

In short, my completed tasks during the hackathon were:

• Fixed the registrations for the CPAN Testers Wiki.
• Got CPAN Testers Reports running on the Fastly (http://fastly.com) service, allowing us to caching some of the pages, and reduce the load on the webserver when trying to recreate reasonably static pages. Also means the routing for anyone viewing the site outside of Europe is going to reduce page load times too.
• Fixed some bugs in the Reports Mailer, refreshed the tests and test data, and tidied up the notifications.
• Fixed the Reports Mailer for sending individual reports, due to the DB storage now using Sereal. Note this had no effect on the summary reports.
• Fixed a long running bug with the Summary panel (and release summary table), which turns out has also been affecting MetaCPAN.
• Continued to hand over the final keys to Doug Bell (PREACTION), who is now carrying the torch for CPAN Testers.
• Fixed a few bugs in other distributions, a couple related to CPAN Testers.
• Cleaned up some of the CPAN Testers family website templates.
• Joined discussions for The Perl River, the (re)naming of the QAH and the future of CPAN Testers.

It was a very productive event, and for CPAN Testers, I'm pleased it gave Doug and I a chance to knowledge share, and ensure he has everything he needs to not only keep the project going, but help develop new ideas to solve some of the big data problems that CPAN Testers sometimes throws up. Over the past 6 months or so, I have been taking a back seat, for various reasons, and in the coming months you will hear much less from me regarding CPAN Testers. Occasionally, I may pitch in to discussions to help give some background to decisions that were made, to give some context to why we wrote code a certain way, or designed a DB table the way we did, but this is now Doug's project, he will be the main point of contact now.

During the wrap at the end of the event, where we got to say a little piece about what we achieved, Chris Williams (BINGOS) made announcement to say thank you to me for 10 years of CPAN Testers. After taking on the challenge to grow CPAN Testers, and make it more interesting for people to get involved, I think I've achieved that. The project is well respected throughout the Perl community, and I've had some kind words from people in the wider OpenSource community too, and with over 68 million test reports in the database, I think I can safely say that has been a success. I wish Doug all the best taking it to the next level, and hope he gains as much knowledge and experience (if not more) from the project as I've done. Thanks to everyone who has support the project, me and all those that came before.

The QA Hackathon would not have been possible without the Sponsors. No matter what they have contributed, we owe them all our thanks for enabling the participants the time and ability to work together for the benefit of all. Thank you to FastMail, ActiveState, ZipRecruiter, Strato, SureVoIP, CV-Library, OpusVL, thinkproject!, MongoDB, Infinity, Dreamhost, Campus Explorer, Perl 6, Perl Careers, Evozon, Booking, Eligo, Oetiker+Partner, CAPSiDE, Perl Services, Procura, Constructor.io, Robbie Bow, Ron Savage, Charlie Gonzalez, and Justin Cook.