Have you ever spent your precious time on submitting a bug report for some project, only to get a response that you’re an idiot and you should f⊄∞÷ off?
Right! Well, perhaps consider spending your time on Perl 6 to see that not every free/open-source project is like this.
In the Perl 6 community, there is a very interesting attitude towards bug reports. Is it something that was defined explicitly early on? Or did it just grow organically? This remains to be a Christmas mystery. But the thing is, if it wasn’t for that, I wouldn’t be willing to submit all the bugs that I submitted over the last year (more than 100). You made me like this.
Every time someone submits a bug report, Perl 6 hackers always try to see if there is something that can done better. Yes, sometimes the bug report is invalid. But even if it is, is there any way to improve the situation? Perhaps a warning could be thrown? Well, if so, then we treat the behavior as LTA (Less Than Awesome), and therefore the bug report is actually valid! We just have to tweak it a little bit, meaning that the ticket will now be repurposed to improve or add the error message, not change the behavior of Perl 6.
The concept of LTA behavior is probably one of the key things that keeps us from rejecting features that may seem to do too little good for the amount of effort required to implement them, but in the end become game changers. Another closely related concept that comes to mind is “Torment the implementors on behalf of the users”.
OK, but what if this behavior is well-defined and is actually valid? In this case, it is still probably our fault. Why did the user get into this situation? Maybe the documentation is not good enough? Very often that is the issue, and we acknowledge that. So in a case of a problem with the documentation, we will usually ask you to submit a bug report for the documentation, but very often we will do it ourselves.
Alright, but what if the documentation for this particular case is in place? Well, maybe the thing is not easily searchable? That could be the reason why the user didn’t find it in the first place. Or maybe we lack some links? Maybe the places that should link to this important bit of information are not doing so? In other words, perhaps there are still ways to improve the docs!
But if not, then yes, we will have to write some tests for this particular case (if there are no tests yet) and reject the ticket. This happens sometimes.
The last bit, even if obvious to some, is still worth mentioning. We do not mark tickets resolved without tests. One reason is that we want roast (which is a Perl 6 spec) to be as full as possible. The other reason is that we don’t want regressions to happen (thanks captain obvious!). As the first version of Perl 6 was released one year ago, we are no longer making any changes that would affect the behavior of your code. However, occasional regressions do happen, but we have found an easy way to deal with those!
If you are not on #perl6 channel very often, you might not know that we have a couple of interesting bots. One of them is bisectable. In short, Bisectable performs a more user-friendly version of git bisect, but instead of building Rakudo on each commit, it has done it before you even asked it to! That is, it has over 5500 rakudo builds, one for every commit done in the last year and a half. This turns the time to run
git bisect from minutes to about 10 seconds (Yes, 10 seconds is less than awesome! We are working on speeding it up!). And there are other bots that help us inspect the progress. The most recent one is Statisfiable, here is one of the graphs it can produce.
So if you pop up on #perl6 with a problem that seems to be a regression, we will be able to find the cause in seconds. Fixing the issue will usually take a bit more than that though, but when the problem is significant, it will usually happen in a day or two. Sorry for breaking your code in attempts to make it faster, we will do better next time!
But as you are reading this, perhaps you may be interested in seeing some bug reports? I thought that I’d go through the list of bugs of the last year to show how horribly broken things were, just to motivate the reader to go hunting for bugs. The bad news (oops, good news I mean), it seems that the number of “horrible” bugs is decreasing a bit too fast. Thanks to many Rakudo hackers, things are getting more stable at a very rapid pace.
Anyway, there are still some interesting things I was able to dig up:
RT #128804 – this is one of the examples where we attempt to print something better than “syntax error”, but have a problem in the error message itself. This was fixed, and now the error message says
Cannot convert string to number: malformed base-35 number in 'li⏏zmat' (indicated by ⏏). Can you spot why this error message is Less Than Awesome?
RT #128421 – sometimes we are just wrong for no good reason. Makes you wonder how many other bugs like this are hiding somewhere. Can you find one?
That being said, my favorite bug of all times is RT #127473. Three symbols in the source code causing it to go into an infinite loop printing stuff about QAST nodes. That’s a rather unique issue, don’t you think?
I hope this post gave you a little insight on how we approach bugs, especially if you are not hanging around on #perl6 very often. Is our approach less than awesome? Do you have some ideas for other bots that could help us work with bugs? Leave it in the comments, we would like to know!
4 thoughts on “Day 23 – Everything is either wrong or less than awesome”
I’ve been using Perl 6 since early Pugs day. And I’ve reported over 150 tickets as well since then. Community is AWESOME, every bug got proper attention, explanation or discussion and tests.
But what I find LTA is bug reporting process itself:
1. There is no word “bug” on any of perl6.org main pages. User may not know where to report.
2. There is link in Rakudo repository pointing to rt.perl.org and this site tells you that “Please use perlbug (link here) to report bugs.”, which redirects you to Perl 5 instructions. And text below “We also track bugs for Rakudo Perl 6.” is not helpful because it still does not tell you WHERE bug should be reported.
3. If user has patience to create account and login with BitCard he gets hit with 80 style website which… still does not tell you WHERE bug should be reported.
4. User gets back to Rakudo repository and finds out that bug can be reported by sending email with some mysterious subject line codes.
5. User sends email but with incomplete info (no rakudo version for example). And gets confirmation on email that ticket was created, but in subsequent updated of bug progress that he receives there is no link to bug itself.
6. If user reported bugs from many emails (company and private) it is hard to track progress or report.
How I imagine friendly bug reporting process:
1. I have link “Report bug” on main page that takes me immediately to bug reporting page. Not instructions how to report. Not another service like R/T. But to actual form where I can report bug immediately.
2. On this form there is place to input code. Which gets evaluated and output gets printed below.
3. Now I have question “Here is how it works now, what behavior do you expect”? With options “Different output” (selected by default) and textarea to enter this expected output. And few more options like “More descriptive error message”, “Should not parse”, etc.
4. Additional place for comment and Reply-To email to get notification when bug gets updated (with link to bug in every email).
5. Big “Submit” button.
As for email notifications – they can be improved further. Besides already mentioned link to bug in every email there should be DKIM signatures to get better deliverability, DMARC on RT side to avoid spoofing and ListUnsubscribe header handling to stop tracking bug progress if needed.
Thank you for your comment!
1. I have created a ticket: https://github.com/perl6/perl6.org/issues/61 (feel free to work on it!)
2. Can you clarify where exactly you’ve read this? README in https://github.com/rakudo/rakudo/ actually points to https://github.com/rakudo/rakudo/wiki/rt-introduction which has some information on how to report rakudo-related bugs.
3. Yes, 80 style website is a problem. It has been discussed many times, but the problem is that the cost of moving to a different bug tracker is a bit high. If someone reading this is ready to spend significant amounts of time on it, perhaps it can be done (but note that there are some people who really like RT and will oppose the change if you fail to satisfy their very specific needs). RT actually had a button to create a new ticket, but this button is temporarily gone (due to a spam attack targeting this button exactly). We are working on it.
4. Yes, it is a bit unusual. At the same time, it does not require creating an account in yet another bug tracker (you just send an email to rakudobug at perl.org and that’s it), so it could’ve been worse :)
5. If there is a code example but no version information, we can ask a bot to run it on all rakudo releases. I think this happens so rarely that it is not a problem (after all, we can always ask the user to clarify something). But you are right that it does not include a link to the ticket, not even in the first message. I have created a ticket for this: https://rt.perl.org/Ticket/Display.html?id=130399, but somebody should mail perlbug-admin to get this stuff done. It is also very likely that a volunteer will be required.
6. If you have a Bitcard account, it seems you can link more than one email to your profile (https://bitcard.org). Perhaps you can try it and tell us if it works?
I’ve also created https://github.com/perl6/doc/issues/1077. You did not mention it, by I think it is related.
Seems like your proposal is to create an alternate page for reporting bugs? Yea, we’ve had this idea some time ago. If you like some drama, you can read this: https://github.com/perl6/user-experience/issues/8 (there was actually some progress on it, but as MadcapJake got a bit less active it was abandoned). Another attempt is http://perl6.fail/ (but right now the login form does nothing, perhaps consider working on it?)
There is indeed a lot we can improve in this area.