Memoirs of a Roadie

Let's Go Crazy

Posted on 31st May 2016

Last weekend saw me in Rubgy for the 9th QA Hackathon. This is a Perl event where the key developers for CPAN, PAUSE, MetaCPAN, CPAN Testers, Dist::Zilla, Test2 and many other Perl 5 and Perl 6 projects, get together to discuss problems, future plans and collaborate on code.

Although I was a co-organiser of the event, I really would like to thank my fellow co-organisers; Neil Bowers (NEILB) and JJ Allen (JONALLEN). Without these guys, organising this QA Hackathon would have been tough, as they really did all the hard work. Also many thanks to Wendy for keeping us fed with nibbles, keeping notes and generally making sure we all stayed focused. An event like this needs a team, and they are an awesome team.

My main aim for this event was to meet Doug Bell (PREACTION). Back last summer, the CPAN Testers server had some severe problems, which meant we had to switch to a new physical server. It was at this moment I realised that I couldn"t do this alone any more. Doug stepped up and started to take over the reins, and has done a great job since. However, I"d never met Doug, so this was going to the first opportunity to catch up in person. After only a few moments of saying hello, I knew we had found the right person to take over CPAN Testers. Doug has a lot of great ideas, and is more than capable of taking the project to the next level, which is where I wanted to see it grow to, but knew it needed fresh eyes to take it there. I feel immensely confident that I have left the keys in capable hands, and with the ideas Doug has already shown me, I expect bigger and better things for CPAN Testers' future. Please look after him :)

On the first day Oriol Soriano Vila (UREE) introduced himself to Doug and I. Oriol was suggested to the organisers by his employer, and after explaining what the event was about, Oriol was even more enthusiastic to attend. I'm glad he did, as he is another great asset to both CPAN Testers and Perl. Although we have referred to him as our "intern", Oriol has proved he was just one of the team. He has some great ideas, asked all the right questions and had results by the end of the hackathon too! You can read more on his own blog.

So once we got our introductions out of the way, we started looking at a high priority problem. One that had been reported in two different ways, but was in fact the same problem. The Summary RAG bars and the release database (as used by MetaCPAN). In turns out the problem was straightforward. After the server crash last year, The database scheme used to rebuild the new server was missing a new column in the release summary table, and thus wasn't getting updated. Once that was fixed, it was "simply" a matter of rebuilding the table. Sadly it took the whole weekend to rebuild, but once completed, we were able to start regenerating the release SQLite database. That took a week, but I"m pleased to say all is now updating and available again.

While that was rebuilding, I started to take a look at some other issues. After introducing Oriol to our family of websites, he registered for the wiki, and spotted a problem with the registration process. After some tinkering, I got that working again. I've no idea how long it's been a problem, but apologies to anyone affected.

In the introductions at the beginning of the event, Leo Lapworth (LLAP) mentioned that he was hoping to refine MetaCPAN"s use of Fastly, and was interested in helping anyone else who might be interested in using the service for their project. I got Leo to sit with me for a while and he gave me a good run through of what the service is, what it can do, and why we should use it for CPAN Testers. I didn"t take much convincing, and quickly opened an account and started to move the main family of websites to it. We have since seen a slight drop on hits to the server, but I expect that to improve as the static pages (the individual reports) are cached. Even the dynamic pages can benefit from their caching, as although many will change throughout the day, only a small portion are updated more than once an hour. Once we learn more about Fastly, expect to see some better response times for your page hits.

Talking with Christian Walde (MITHALDU), he wanted to help with the performance of the websites, particularly the Reports website. However, with the rebuilding ongoing, the server wasn't in the best place to really evaluate performance. He did happen to mention that the reports he was getting from the mailer were coming through as garbage. After some investigation, I discovered that the mailer had not been upgraded to use Sereal, which is now our serialiser of choice for the reports stored in the database. With that fixed, together with some further improvements and with all tests running, we put it live and waited. The following morning Christian report he had readable reports coming through again.

One aspect for testing the Reports site, and one that would have restricted Christian to evaluate the performance, is that apart from mine and Doug"s development machines, there is no stable installable full instance of the CPAN Testers Report site, including databases and cron scripts. As such, Doug has been working on providing exactly that. It has been on my TODO list for some time, as some of the bug reports and issue requests would have been quashed much more efficiently had others been able to fire up a working site and be able to send a pull request. You can read more about Doug"s progress on his blog, and hopefully this will encourage more people in the longer term to get involved with CPAN Testers development work.

Throughout the weekend I worked on cleaning up some of the templates on the various websites, ensuring that sponsors were correctly attributed, and fixed several bugs in some of the associated distributions. Not all have been pushed to CPAN, but work is ongoing.

Having finally met, Doug and I went through all the website logins and social media accounts, and made sure he had all the keys. The handover process has been a longish one, but I didn"t want to overwhelm Doug, and wanted him to find his feet first. After this weekend, expect more posts and updates from him rather than me. Please look after him :)

I also joined in some for the discussions regarding the CPAN River and the naming of the QA Hackathon. Neil has written up both admirably, and while I didn"t contribute much, it was good to see a lot of healthy discussion on both subjects. Regarding the naming of the event, I do think it's a shame that the likes of Google have turn the word "Hackathon" into the concept of a competition event, which the QA Hackathon event is definitely not. Ours is about collaboration and planning for the future, with many of the key technical leads for the various toolchain and associated projects within Perl 5 and Perl 6. I don"t have a suitable name to suggest, but I would recommend ensuring the acronym could not be used negatively.

In the coming weeks, I hope to collate all the website tests I run prior to updating the CPAN Testers family websites, and handing over to Doug for his new development environment for CPAN Testers. This will hopefully enable easier access to anyone wanting to help fix problems on the websites and backends in the future. 

In short, my completed tasks during the hackathon were:

• Fixed the registrations for the CPAN Testers Wiki.
• Got CPAN Testers Reports running on the Fastly (http://fastly.com) service, allowing us to caching some of the pages, and reduce the load on the webserver when trying to recreate reasonably static pages. Also means the routing for anyone viewing the site outside of Europe is going to reduce page load times too.
• Fixed some bugs in the Reports Mailer, refreshed the tests and test data, and tidied up the notifications.
• Fixed the Reports Mailer for sending individual reports, due to the DB storage now using Sereal. Note this had no effect on the summary reports.
• Fixed a long running bug with the Summary panel (and release summary table), which turns out has also been affecting MetaCPAN.
• Continued to hand over the final keys to Doug Bell (PREACTION), who is now carrying the torch for CPAN Testers.
• Fixed a few bugs in other distributions, a couple related to CPAN Testers.
• Cleaned up some of the CPAN Testers family website templates.
• Joined discussions for The Perl River, the (re)naming of the QAH and the future of CPAN Testers.

It was a very productive event, and for CPAN Testers, I'm pleased it gave Doug and I a chance to knowledge share, and ensure he has everything he needs to not only keep the project going, but help develop new ideas to solve some of the big data problems that CPAN Testers sometimes throws up. Over the past 6 months or so, I have been taking a back seat, for various reasons, and in the coming months you will hear much less from me regarding CPAN Testers. Occasionally, I may pitch in to discussions to help give some background to decisions that were made, to give some context to why we wrote code a certain way, or designed a DB table the way we did, but this is now Doug's project, he will be the main point of contact now.

During the wrap at the end of the event, where we got to say a little piece about what we achieved, Chris Williams (BINGOS) made announcement to say thank you to me for 10 years of CPAN Testers. After taking on the challenge to grow CPAN Testers, and make it more interesting for people to get involved, I think I've achieved that. The project is well respected throughout the Perl community, and I've had some kind words from people in the wider OpenSource community too, and with over 68 million test reports in the database, I think I can safely say that has been a success. I wish Doug all the best taking it to the next level, and hope he gains as much knowledge and experience (if not more) from the project as I've done. Thanks to everyone who has support the project, me and all those that came before.

The QA Hackathon would not have been possible without the Sponsors. No matter what they have contributed, we owe them all our thanks for enabling the participants the time and ability to work together for the benefit of all. Thank you to FastMail, ActiveState, ZipRecruiter, Strato, SureVoIP, CV-Library, OpusVL, thinkproject!, MongoDB, Infinity, Dreamhost, Campus Explorer, Perl 6, Perl Careers, Evozon, Booking, Eligo, Oetiker+Partner, CAPSiDE, Perl Services, Procura, Constructor.io, Robbie Bow, Ron Savage, Charlie Gonzalez, and Justin Cook.

Rendez-Vous 6

Posted on 17th March 2014

My 2014 QA Hackathon

Day One

I arrived the previous day, as did most of us, and we naturally talked about coding projects. Not necessarily about work at the hackathon, but discussion did come around to that too. I talked with Tux at one point, who convinced me that a stand-alone smoker client would be really useful. Once upon a time, we did have this, but with the advent of the more sophisticated smokers, and the move to the Metabase transport layer, the old script never got updated. The following morning Tux sent me a copy of the script he has, so at some point over the next few months I will take a look to see what I do to make it compatible with the modern smokers.

My intention was to release a distribution each day of the Hackthon. Unfortunately this was scuppered on the first day, when trying to add support for the full JSON report from CPAN Testers, when I realised I don't store the full report in the database. In the future when we have MongoDB and replication set up, this will be a non-issue, but for the moment, I now need to store the full report. This now requires a change to the metabase database on the cpanstats server (as opposed to the Metabase server). Over the course of the hackthon I reviewed the changes needed, and updated a lot of the Generator code, as it was an ideal time to remove SQLite references too.

In looking into the code changes, Andreas and I again looked at the updated timestamp used by the various CPAN Testers sites to do statistical analysis, which was also causing us problems. In the Metabase, the CPAN Testers Report fact is the container for all the child facts, such as LegacyReport and TestSummary. When the facts are created by the tester, the 'creation' timestamp is used to reference the time on the tester's own server that the report was generated. This could be better stored as UTC, but that's a problem for another day. However, it does mean the timestamp could be different to the one on the Metabase server. When the Metabase server retrieves the report from the outside world, it updates the 'updated' timestamp across all facts and saves into the SimpleDB instance on the server. Except it doesn't. The 'updated' timestamp is always the same as the 'creation' timestamp. Andreas has been noting this for quite some time, and finally he convinced me, at which point we roped in David Golden to take a look. Reviewing the code, there is nothing wrong that we can see. The 'updated' timestamp should be updated with the current timestamp on the Metabase server, which should also cascade to each child fact. As such you would expect several reports to have a different 'creation' timestamp from that of the 'updated' timestamp, even if only different by a second. Sadly this is going to take more effort/time to debug, as David in particular is working several different aspects of QA here at the hackathon.

Towards the end of the day, I spoke with liz and Tobias (FROGGS) about how CPAN Testers might handle perl6 modules. Currently there is no client available, but there could be in the future. However, due to the way Perl6 modules are to be uploaded to CPAN it is possible that smokers may submit reports for perl6 only modules, as many ignore the path to the distribution. Right now, liz tells me that all perl6 modules are being release under the /perl6/ path inside the authors' directory. This makes things easier for CPAN Testers as we can initially ignore these test reports, as they will not be valid. However, in the longer term it will be interesting to have a CPAN Testers smoker client for Perl6. The main difference would be to record in the metadata that it's a perl6 only distribution, and we *should* be able to carry on as normal, submitting reports to the Metabase, etc. It may require some distributions to have a 'Did you mean the Perl 6 distribution?' link on the website, but for the most part I think we could handle this. It will require further work to define a CPAN Testers Perl 6 Fact, but it will be a nice addition to the family.

Day Two

The morning was spent visiting the Charteuse cellars, and enjoying a tasting session, before heading back to the hacking in the afternoon.

In the afternoon, I started to look at some of the statistics the CPAN Testers Statistic site generated. After some discussions with Neil Bowers, he was interested in the drop-off of report submissions when a distribution was released. I believed this to be fairly consistent, and found that it did indeed last roughly 8 days, with a tail off that could last for months or years. There was an initial blast of tests within the first few hours, thanks to Chris' and Andreas' smokers, but the rest of the more reliable smokers get submitted within those first 8 days. Neil has created some initial graphs, and I'm looking at ways to integrate those with the Reports site. How we display these will likely revolve around a specific selected version, as overlaying versions might be a bit too much ... we'll see.

It also led me to think about what time of day do testers submit reports. So, I'll be looking at creating some graphs to show submissions per month, per day of the week, and per hour of the day. Along with BooK, we discussed further metrics, although they look likely to be used within their CPAN Dashboard project, although some of the data can be provided by CPAN Testers APIs already, so little work need by me :)

Looking through aggregated data, as stored and indexed within the Statistics codebase, it was obvious some had were now incomplete. It seems some of the outages we had in the last few months, prevented the data storage files from being saved. As such, I started off a complete reindex. It meant the Statistics site was out of sync for the following day, but at least it meant we once had again had correct data to produce the graphs we wanted.

There was more work rewriting the Generator to store the report objects. Yves asked why I wasn't using Sereal sometime ago, when I posted about using Data::FlexSerializer, and at the time I didn't have a need to rework the code. However, seeing as I'm rewriting to store the perl object now, rather than just JSON, it does make sense to move to Sereal, so hopefully that will make Yves happy too ;)

Day Three

Continued work on the Generator to remove all SQLite references, and a few further clean ups. Also worked on adding the necessary support to allow perl6 reports to be ignored. At some point in the future we will accept perl6 reports, but following further discussion with Tobias, we'll handle this using the metadata in the report not on the path of the resource.

Salve interviewed me for a future post about CPAN Testers. It'll be interesting to see whether I made sense or not, but hopefully I managed to convey the usefulness and uniqueness of CPAN Testers to Perl and the community. It good opportunity to also thanked Salve for starting the QA Hackathons, as without them CPAN Testers may well have stalled several years ago. Like many other projects, if we had relied on email to handle all the discussions and move the project forward, it would have taken years to get the Metabase working and move away from the old email/NNTP mechanisms.

charsbar updated CPANTS with some altered metrics, and at the same time added selected CSS colours for BooK and Leon, so I asked too. I now have a shade of my own purple on my author page ;) Thanks charsbar.

As Wendy went to lunch, she made the mistake of asking whether we wanted anything. I asked for a Ferrari, but sadly they couldn't find one, so I got a Lambourgini instead. If you don't ask, you don't get .... vroom, vrooom, vroom :) I'll add a picture once I've sorted them out.

At some point during the afternoon, Ricardo told me one of his asks for the hackathon. He wanted to be able to ignore the NA reports in his No Pass RSS feeds. Mulling it over this seemed entirely sensible, and so I fixed it. Ricardo celebrated :)

During a discussion with Neil, he mentioned that Paul Johnson was creating a Devel::Cover service, that he wanted to run like a CPAN Testers service. The idea was to write a system, that could allow distributed testing with testers sending in reports, which could then be accumulated, based on the OS being tested. As the Metabase is already able to handle different buckets, adding another bucket for coverage reports simplifies some of the work. The distributed client can then be moduled on the CPAN Testers means of report contruction, creating a new coverage report fact and use the same transport mechanism to submit to the Metabase. A web service can then poll the Metabase for the new bucket, and create report pages in exactly the same way as CPAN Testers. It'll be interesting to see whether we can use the same (or similar) code to provide this.

Day Four

The morning threw us a curve-ball, as the building wouldn't open up. It was a Sunday and apparently no-one works on a Sunday. Thankfully a few phonecalls to the right people got us in, just in time for lunch. In the meantime as we all were staying in the same hotel, we took over the bar, and borrowed a conference for the morning.

The poor wifi connection, gave us a good opportunity to have further discussions. Neil gathered together several interested parties to discuss author emails. Both PAUSE and CPAN Testers send emails to authors, and there is a plan to send authors a yearly email to advertise improvements to their modules, and let them know about sites and tools that they might not be aware of. However, although many emails get through without a problem, several fail to reach their intended recipient. Typically this is because authors have changed their email address but failed to update the email stored within the PAUSE system. CPAN Testers highlights some of these Missing In Action authors, but it would be better to have an automated system. Also, as Ricardo noted, the envelope of an email is left unchanged when is sent to the develooper network, so bouncebacks come back to the original sender containing the authors' potenmtially secret email address. It would be much better to have a service that monitors bouncebacks, but change the envelope to return to the handling network and can send an appropriate email to the sender. It could then provide an API to enable PAUSE and CPAN Testers, and any future system, to know whether compiling an email was worth the effort. For CPAN Testers there can be a great deal of analysis to prepare the summary emails, so knowing in advance an author email is not going to get through would be very beneficial. Neil is going to write up the ideas, so we can more formally design a system that will work all of PAUSE related systems. CPAN Testers already has the Preferences site to allow authors to manage their summary emails, and also turn off receiving any emails, and it may be worth extending this to PAUSE or other system to provide a subscription handling system.

The rest of the day was mostly spent monitoring the metabase table in the cpanstats database, as the new 'fact' column was added. The new field will store the reports from the parent in Sereal. I was a bit worried about locking the table all day, but no-one seemed to notice. While this was happening, I started back on the original new module I started on the first day of the conference,and had hoped to release. However, it highlighted further problems with the way reports are stored. I'm not sure what is doing it, but the underlying fact.content field in JSON was being stored as a string. In most cases this isn't a problem, however for this module it caused problems trying to encode/decode the JSON. After fixing the Generator code, it means the new module still didn't get finished. Well at least I have something to start my neocpanism.once-a-week.info stint with :)

Wrap Up

I now have several pieces of work to continue with, some for a few months to come, but these 4 days have been extremely productive. Despite playing with the CPAN Testers databases rather than writing code, the discussions have been invaluable. Plus it's always great to catch up with everyone.

This year's QA Hackthon was great, and it wouldn't have been possible without BooK and Laurent organising it, Wendy keeping us eating healthily (and in good supply of proper English tea ... I'll try and remember to bring the PG Tips next time), Booking.com for supplying the venue and all the other sponsors for helping to make the QA Hackathon the great success it was. In no particular order, thanks to Booking.com, SPLIO, Grant Street Group, DYN, Campus Explorer, EVOZON, elasticsearch, Eligo, Mongueurs de Perl, WenZPerl for the Perl6 Community, PROCURA, Made In Love and The Perl Foundation.

Looking forward to 2015 QA Hackathon in Berlin.

The Time of the Turning

Posted on 7th May 2013

A few weeks ago I had the pleasure of attending the 6th annual QA Hackathon. The event has become THE event for developers of test modules, projects and toolchain applications to come together to discuss ideas and plan for the future, as well as release some great work while they are there too.

This year Shadowcat, the primary sponsors, took on the organisational duties. The event was originally to be in London, but due to personal circumstances the decision was made to move the location to Lancaster in the North West of England. Personally they made the right choice. The venue itself was the new InfoLab building at Lancaster University. The attendees came from far and wide once again, and it was great to catch-up with friends old and new, and even be introduced to some newer friends.

My plan for the weekend was mainly to look at CPAN Testers. With the servers for the Metabase coming soon, David Golden and myself had hoped to be able to set them up, and start looking at changing the backend code to work with the new Metabase database. Unfortunately, the servers weren't ready for us just yet, so I started looking at other things. For myself, one area of CPAN Testers, particularly the cpanstats database side of things, needed attention. Speed of processing reports.

My first task once settled in, was to look at the way that the reports are consumed from the Metabase. Due to the way SimpleDB has become very unreliable with the results it sends, in order to avoid missing reports the criteria for the date search has been altered slightly to be a little more thorough, and a smaller range is now used to retrieve a set of GUIDs. The results now appear to be a little more complete, although we still appear to be missing some every so often. There is also a tail of log.txt which also helps to catch up with the reports. This work saw a new release of CPAN-Testers-Data-Generator.

A big factor with the slowness of the CPAN Testers server is that it requires a lot of disk I/O, with the database updates being a key factor. The most intensive updates are surrounding the SQLite database that could be downloaded. This also includes creating the Gzip and Bzip2 archives. As only web crawlers seem to be downloading the files, I've suspended the update. This has now freed up a lot of resources and consequently some of the other tasks, particularly the builder has improved.

Next, the builder was the focus of my attention. Previously the builder has been building pages for both authors and distros all at once. Although the author pages are viewed slightly less, they were getting built more frequently, due to the way the requests are pushed into the queue for each report. Initially the logic for building pages was altered, which improved some of the higher requested pages, but the more optimal solution was to split the builder into two, one for authors and one for distros. With the reduction in processing elsewhere, this improved the builder performance considerably. Monitoring the way the author pages are built since the hackathon, has also allowed me to alter when the builder for authors runs. This has then allowed the builder for distros to take a higher priority. With more distro pages than authors, this now gives distro pages more opportunity to be built quicker. Currently reports are being built in less than 24 hours of being submitted. These updates saw a new release of CPAN-Testers-WWW-Reports.

Another release while at the event, related to the QA Hackathon itself, was the main QA Hackathon website. Before the event, BooK had asked if the files that make up the website that the main QA Hackathon uses could be added to GitHub. As such, I packaged up the site into a git repository and released it. If you wish to help contribute to the site, please do.

Although there was a lot of coding work involved in the weekend, one of the bigger uses of time was the Lancaster Consensus organised by David Golden. For a few hours each afternoon, a large group of key toolchain developers, secondary project developers and various interested parties, gathered to discussed various aspects associated with configuration, installation, testing and specification of Perl and CPAN. With so many developers in one room, it wasn't too surprising to have a few opposing views, but with a guiding hand from David, we did achieve a consensus. If you wish to read the outcome, please read David's write-ups of the discussion points. The Consensus meetings were perhaps the greatest achievement of the event. While there might not have been too much immediate coding output from them, the potential to improve Perl and CPAN is considerable. From a CPAN Testers perspective, Post-installation testing, Case insensitive package permissions and Rules for distribution naming were perhaps of most interest. Although it may be some time before Post-installation testing could be hooked into a CPAN Testers smoker, it will be a valuable addition to the testing reports against pre-installed environments.

During the event, I had several discussions with Garu regarding his work on the cpanminus smoker client, and the common smoker client. In the last minutes of the hackathon we were able to push through a very notable report submission. It is exactly this sort of collaborative effort that makes these hackathons worthwhile. I look forward to see everyone again in Lyon.

The QA Hackathons could not be the success they are with the support of all the sponsors. My personal thanks to them for helping to providing accommodation, food and a venue for us all to hack. A big thank you to cPanel, Dijkmat, Dyn, Eligo, Evozon, $foo, Shadowcat Systems Limited, Enlightened Perl Organisation and Mongueurs de Perl.

To Wish Impossible Things

Posted on 4th May 2013

The QA Hackathon website has had a bit of an update today. Primarily a new page and new photos have been added, but plenty of other updates have been included too.

The new page is a review page, to collect various blog and news posts relating to each year's event. Originally I listed all the reviews from previous years in the side panel, but now that we've just had the 6th annual event, the list was looking a little bit too cramped.

With the extra space, I've also been able to include the group shots that were taken at some of the events. Unfortunately there was no group shot taken in Birmingham, and I've not seen any during the 2010 and 2011 events, so if there are any, please let me know. Also if there is one of the Tokyo Satellite event this year I would love to include it on the site.

I've added some write-ups to the last few events in the About page. The biggest change though is likely only visible to those with screen readers, as I've made many changes to links and images to provide more accessibility. Several fixes to layout, spelling and wording have also been included too.

The site, particularly the list of reviews, is still incomplete. If a blog entry is missing that you think should be there, or you spot other items that could do with an update, feel free to email me with details, or fork the repo on GitHub and send me a pull request.

Young Parisians

Posted on 10th April 2012

Did I mention I went to Paris to take part in the 2012 QA Hackathon? Did I remember to mention all the cool stuff I got done? Well if you've been hiding for the past few weeks, have a look at the last couple of posts :)

As per usual, while there I took my camera along. However, unlike many previous visits to Paris, I didn't do any sight-seeing. And that includes failing to wander around the venue we were in and discovering the real submarine among other things, that others found while taking a breath of fresh air.

Instead I spent my time hacking away, and only occasionly coming up for air for food, drink and some camera action.

With over 40 people in attendance, it was going to be difficult to capture everyone, but I think I managed it. If I did miss you, my apologies. It was great to meet so many friends old and new, and a real pleasure to finally put faces to names that I've known for a while, but not had the opportunity to meet in person.

So many great things happened in Paris, and I'm really looking forward to see what we can achieve in London for the 2013 QA Hackathon. See you there.

• Photos from the 2012 QA Hackathon in Paris