Author: Mike Saunders

Xisco Fauli, Ilmari Lauhakangas and Mike Saunders from The Document Foundation, the non-profit organisation behind LibreOffice, discuss Quality Assurance (QA) in free and open source software . (This video is also available on PeerTube.)

Transcript

Mike Welcome to episode three of the LibreOffice and Open Source podcast. I’m Mike Saunders from the Document Foundation, the small non-profit organisation behind the LibreOffice project and community.

In the last two episodes, we talked about marketing free and open source software such as LibreOffice – the challenges and the opportunities as well. And then we talked in the second episode about design, UX, UI in LibreOffice and open source software – the challenges of making changes to user interfaces that people have been using for years, but also some of the cool things happening.

But today we’re going to talk about how we test half a million documents in LibreOffice. Yeah. So we sit down and load half a million documents into LibreOffice. File, Open, File Open and check the compatibility. Yeah, we’re very committed. Of course not – we do that automatically. I’m going to talk about that a bit later, because we’re talking in general today about LibreOffice QA – quality assurance – as well.

So how do we make sure that the software stays strong and stable and reliable when we add new features to LibreOffice, how are they tested and integrated before they go out to millions and millions of people as well.

How do we handle bug reports? So somebody submits a bug with the software. What makes a good bug report as well? And how do we maintain compatibility? There are billions and billions of documents out there in mostly Microsoft formats. So how do we make sure that LibreOffice stays compatible with them when things are changing, the formats are changing?

Microsoft’s famous transitional format makes life extra difficult for many of us. So, let’s talk a bit about QA. I’m joined, as mentioned, by two of my colleagues at The Document Foundation. So, again, we are the small 15 people non-profit organisation. But of course we at The Document Foundation are backed up by a community of hundreds of people making LibreOffice as well.

But I’m joined by Xisco Faulí, who is a QA engineer – I think is your title. Xisco is that right?

Xisco Yes, you’re right. Yes.

Mike And you’ve been at TDF for a few years now. I think the same as me.

Xisco More or less. I joined TDF in 2016. So it’s almost 10 years next year. And I joined as a QA engineer and since then I’ve been doing that.

Mike How did you get involved in QA before that? What’s your background leading to this?

Xisco Yeah, that’s a good question. So before that I studied computer science in the university and at the same time there was – and it’s still going – the Google Summer of Code program. So I first joined LibreOffice as a Google summer of Code student and then after that I got a job as a QA engineer in another company, and at that time I also contributed to the project doing QA as a volunteer as many other people do.

Nowadays they spend their time volunteering doing different things in the project and, yeah, I was doing QA – and then at some point the QA offer was published – this job position – so I applied and I got the position.

Mike And our other guest in this episode is Ilmari Lauhakangas – I think—have I pronounced that correctly?

Ilmari Yeah, Ilmari Lauhakangas

Mike Excellent. Yeah. Who does various things. We all do various things at TDF as well. There’s a lot of overlap in what we do and overseeing the community. But yeah, Ilmari, tell us a bit about what you do at TDF and how you got to this point.

Ilmari I was hired by TDF in 2019 to do mentoring and recruitment of volunteers. So I focus on onboarding new people and I do it for basically most of the areas where you can contribute. So in addition to quality assurance, development, design, documentation – so I also do stuff like web development for TDF applications and stuff like that.

And of course, there’s always room to grow at TDF, so it’s nice. And I got my start as a volunteer at around 2014. And back then I was contributing to a couple of open source projects. And I was wondering, how someone who does not know programming that well might help developers and kind of save them time to focus on the stuff that matters.

It hadn’t really occurred to me that testing bugs would be such an impactful thing. So when I somehow ran into the topic in the context of LibreOffice, I kind of really got excited and really put a lot of effort into regularly testing bugs. So that was a nice experience.

And even before that I had done some recruitment for other open source projects. So yeah, I was kind of motivated to see what was possible – like how can we collaborate as a team and how to organise all the information for contributors.

I was kind of impressed even at that time – like over 10 years ago – how well organised the whole thing was. There were these easy tasks for developers mainly, and all the information was laid out quite nicely in the wiki and then there were people helping, chatting, sending emails and it was quite a supportive experience back then.

Mike Yeah, the recruitment, the mentoring that Ilmari mentions is a big thing of what we do as well and in many other free and open source software projects – because it’s nice to have a load of users – we talk about having around 200 million users – but if nobody’s contributing, then what does that mean?

So a big part of what we do as well as doing our work in marketing, design, QA is mentoring people who want to contribute to the project, but often don’t know where to start – and of course we try to make that easier, but LibreOffice is I think 7 million lines of source code has quite a complex build system (although I think that’s been simplified a lot over the years).

So when people come to us and say “I want to contribute, but where do I start I’m daunted?” Then Ilmari often does interviews with them – just a chat to say right where do you want to go, instead of sending people straight to a big wiki page – which is helpful for some people they like that approach, but often it’s nice just to have a chat and say “Right what do you want to do and where can we get started”.

Of course we’re hugely international project as well – I’m in Germany, Ilmari’s in Finland and Xisco’s in Spain – so that’s just a few, but we have contributors in I think the majority of the 190-ish countries in the world.

Ilmari Yeah. And it was nice for me to discover bug testing. I didn’t have any experience any training on it. So everything I know I learned in the context of LibreOffice originally, and it was nice to have such a low threshold for contributing. So you don’t really need to know anything.

Just follow the steps and then you can progress to more challenging investigations and yeah. It’s also quite a pleasant thing to do – like a very calm activity. So usually when your software crashes you get mad, but in case of QA you’re glad that it crashed because you found something interesting or you could reproduce something. So there’s not that much stress in it as an activity. So that was nice as well to discover.

Mike Yeah. And we see as well over the years, lots of contributors in the LibreOffice project who started out on with QA and testing some things – confirming some bugs and then as we’ll talk about in a moment – but they’re moving on to other stuff in the project as well.

So I think it is a good kind of onramp to contributing. But how does the QA team work then? I guess most of the work is centred around the Bugzilla instance we have. Is that right Xisco?

Mike Yeah that’s correct. So Bugzilla is the bug tracker where all the issues are reported, and where developers coordinate and check the issues that they have to fix or they have to work on. So yeah basically the QA activities around Bugzilla – we have to deal with the reports that the users report in Bugzilla and once we deal with those reports we have to talk with the developers and try to find someone to work on those reports.

Mike And, so when a bug comes into Bugzilla it’s not automatically marked as new. Somebody first has to confirm it. Xisco, is that right?

Xisco Yeah, that’s correct. That’s called triage. It’s basically the same that is done in hospitals for instance. When a new patient comes in, we have to do some triaging and see what’s the problem. So we do the same here. When a new report comes in we check the report; we see if they if the information provided is enough to reproduce the issue and then once we try to reproduce. If we succeed to reproduce it, then we try to analyse it and try to have as much information as possible to help developers, to make their life easier. So for them it gets easier to fix those issues.

Mike And on the QA blog which we’ll link to in the notes, you post monthly reports as well Xisco about the number of bugs fixed. I think it’s often like 400 or 500 issues resolved.

Xisco Yeah that’s correct. We have a monthly report where we talk about QA and development activities. It’s a picture of what’s going on. Not everything is mentioned there because we have list of top 10 QA triagers for instance.

So it’s not like we only have 10 people. We have hundreds of contributors, but we only list the most relevant or the most active ones. But yeah, it’s a way to, at least from my point of view, it’s a way to say thank you to all these people contributing in QA – to say, okay, your work is it’s really appreciated and well, at least you’ve been listed here in these metrics and, at least from my side, it’s a way to say thank you to them.

Mike And occasionally on the other blog on the main TDF blog, we’ve posted some stories about a bug report coming in and then a developer jumping in two hours later to look at something and then resolving the issues. I know that’s not always the case.

Some things take a lot more time, but there have been some stories of bugs being resolved in a few hours because everyone’s in the right place at the right time – and those make nice stories as well.

What I also find interesting, as somebody who’s not involved in QA but watches a lot of what you guys are doing, is the ability to pinpoint exactly where a change took place that caused a problem.

So we do a new major release of LibreOffice every six months. About once a month we have a point release with largely bug fixes or security updates.

But I see that when you guys get a report, a bug report comes in, you can analyse, often pinpoint the exact code change that caused this issue, if it was a code change that made it happen. It could just be another general bug. But if a code change somewhere in the past in all the years of development, you can identify the exact build of LibreOffice – even a development snapshot. And I think, Ilmari, this is called bibisecting. I have no idea how to pronounce it…

Ilmari So it comes from binary bisecting. So we can say bibisecting. So we emphasise binary here because we offer repositories which contain thousands of runnable LibreOffice versions. So each code change can be run as a program – as a binary. So this is a very convenient way for contributors to participate. They don’t have to have a development environment. They don’t have to wait how many hours it would take to do the building of each of these – like, I don’t know, 13 tries when you’re testing a specific bug.

So you might have to do the same steps like 13 times in a row. So it can be a bit tedious, but it’s also very powerful way to help developers. And yeah, obviously if we go into practical things, it will take a lot of space on your computer if you want to really have a long history of the different versions. But of course most of the old issues have already been analysed.

So a new contributor can do a lot of good work with just a handful of the newer repositories.

And yeah, it’s really nice to be able to help developers in this way. And it’s also kind of a way to learn more about the code, in a kind of a soft-landing way. So you get closer to the source code. You can read it exactly where the issue is and learn about it that way.

Mike Thousands of builds that must be take up quite a bit of space.

Ilmari Yeah, it does have some kind of compression benefits there. So they don’t exactly take like terabyte per repository. It only takes like seven gigabytes about per repository. So, not that bad.

Mike But it’s still a very practical thing to have. But where do you even start then if you get a bug report and you really don’t know in the last few years which change may have caused this issue? Do you jump between two dates and say, “Well, it could have happened between a change in 2015 to now – and then we’ll go with these binary builds and try and narrow down the time?” Or how do you approach it?

Ilmari That’s a good point because as the history grows longer, the tedious part also gets longer. So a part of it is when you’re an experienced triager you already kind of remember when some feature was introduced. So maybe you remember this is not inherited from like before 2010 or something like that. So you can immediately select some version that is not super old and then go from there. But yeah, it can be like an intuitive process in a way.

But usually you start from a reasonably old version and then check is it is the bug here and then kind of bisect it or divide the whole testing process manually first – and then you rejoice when you find the version where it appeared in. And obviously the bug report itself may contain already a mention that it wasn’t in this version; it was in this version. So then you can see okay I’ll start with this that the reporter pointed out and yeah, so we get a lot of benefit from the user testing.

Like discussed earlier we mostly do analysis of user reports. So we are kind of in a huge debt to our users for providing all of these manual testing experiences, because we don’t really even have time to do manual testing much ourselves. Although that does happen when we test reports we do discover new issues as well.

Mike And I noticed in the last few years one of the priorities has been to keep the number of unconfirmed bug reports down. I think is that still the case, Xisco? Is that one of the priorities and what are some of the other priorities of the QA team at the moment?

Xisco Yeah, the that’s one of the priorities. Sure. But it’s really tough to keep the number of unconfirmed bugs low because we keep receiving new reports every week. I would say around 100 reports every week, and all of them have to be triaged manually. So yeah, it’s a lot of work recurrent work to do.

So yeah we try to keep volunteers that we have and sometimes it goes up and sometimes it goes down, but that’s one of the priorities here. Another priority is to keep the number of regressions as slow as possible. Of course, the more regressions we have, the worse the quality of the software is and it’s very important – every time we have a new major release – of course there’s going to be regressions it’s software, so there’s always issues. So the one priority is to try to fix those regressions as fast as soon as possible. Yeah, that’s also very important. So then when we have a new major release then we can have a fix within one month in the next minor release for that version.

Ilmari Yeah, you mentioned regression. So to clarify kind of you can have an existing feature which worked fine and then there was some code change which broke something in that feature. That’s called a regression. But on the other hand, you can introduce a new feature, but you overlooked something and it doesn’t really work like you intended.

So that’s kind of like a general bug or an implementation issue – however you want to call it. But yeah, the regressions are the ones that are the target of the detective work. So for the bisecting and such. And sometimes it’s kind of unfortunate that we can’t go back in history beyond like 2012 or something like that. Around that time like we don’t get the precise bisecting ability because things get complicated. The build system was not that high quality back then and you can imagine it’s kind of having a time machine or wishing to have

Mike I remember when the LibreOffice project started like you said – you talked about 2012 and 2010 – the LibreOffice project started, and I was working as a Linux journalist at the time and one of the earliest things the LibreOffice project was trying to do was to modernise the build system. And I guess now looking back at it, it’s not just to make life easier for developers and potential developers, but to make it possible to do these tests more easily as well. So, but Xisco, you talked about how a lot of things have to be done manually, like of course triaging new bug reports that come in, but a lot of things can be automated as well. At the beginning of the episode I mentioned this half a million documents which can be automatically tested. So tell us a bit about automatic testing.

Xisco We have a virtual machine that basically takes thousands of documents from the internet and from public repositories like Bugzilla and other bug trackers and some forums, and downloads those documents. And then recurrently it tries to import those documents, export them, reimport them, reimport them again. So basically this is trying to find crashes or asserts in those documents.

This is something that has been done for many years by Caolán, and it’s has improved a lot the quality of the software because before we had this automation, there were many reports from users saying that LibreOffice was crashing. From time to time, we still get reports about crashes when importing or exporting documents, but it has reduced a lot the number of such reports and it helps a lot in that matter.

We also have another kind of automation. We also have unit tests. So basically nowadays it’s kind of common to see when there is a fix for an issue, to also have a unit test for that fix. So this way we make sure that this is going to work forever and in the future we are not introducing a regression for that fix. So we know for sure it’s going to work – and at the moment I don’t know how many we have, but we also have thousand of unit tests that have been written and added to the codebase in the last years.

This has increased a lot since LibreOffice project started. Before that we had some unit tests, but recently we have many of them and it’s helping a lot with the quality of the software.

Ilmari About the automated tests – if we have so many tests, how can we still get bugs? Well, the explanation is you can’t predict the future. So you can’t really write a test for every possible scenario. So, there will always be something due to the complexity of the software. So, the test writing is often reactive. So when it’s not in conjunction with introducing a new feature, if it’s reacting to a regression, then you have this integration test.

So you write a scenario where you kind of simulate what a user would do. You might have some simple document and manipulate it somehow. So it’s not like a very small atomic unit test. It’s a more broad test, but it’s good because it will also catch things that you might not even know about. So yeah, that’s kind of a one way of future proofing, but it can never be.

Xisco And it’s surprising to see how changing one part of the code might impact on another different part of the code. So yeah, you think: oh this change is going to be safe. I’m not going to break anything, and then a few months later someone reports that something completely unrelated gets gets a regression. So yeah, that’s the magic of LibreOffice.

Mike And whenever I’m looking around on Bugzilla, and seeing what new reports are coming in, when people have to submit a bug report or when people choose to submit a bug report, they have to fill in different things – different bits of information about the software. Because I know from handling the LibreOffice social media side, we also get bug reports via social media, but they tend to be: “LibreOffice doesn’t work – why not?” Or “Why did LibreOffice stop working after the latest update?” – with no information which updates

Those are not the most useful bug reports because there’s no standardisation of what information needs to be required, but Xisco, what makes then a good bug report, what does a good bug report that you guys can work with, what does it need to include?

Xisco Considering that LibreOffice runs on different platforms, different environments, it’s really important to provide as much information as possible.

If you have to record your screen to show how to report the issue, then that’s going to be helpful – providing the document because many times a specific problem might happen with a specific document, but not within a general case. So attaching that document is really help helpful as well. Providing the steps to reproduce the issue. So yeah, basically the more information is provided the better.

Mike Yeah, definitely. You see the big difference it makes – a bug report with a huge amount of information versus a generic “Yeah, this doesn’t work”.

So we’ve talked about what the QA project does and some of the different tools as well. Anybody listening to this, or watching it as well, who thinks, “Right, I want to get involved. I want to get some experience in QA. I want to help out with the LibreOffice project.” Ilmari, you talked about this a bit at the beginning, about how you got involved and, but what is a good way for people to help out and get involved in QA? What steps to reproduce should they take?

Ilmari So people navigate things differently. We have this mentoring approach for people who are not that comfortable maybe absorbing large amounts of information, or are not at that stage in their learning that they would be comfortable jumping into a big project like this. But of course, it’s possible to navigate everything yourself. Our wiki documentation is quite clear. It also has quick start steps, so you can just drill down on the basic stuff – and of course we recommend to start with the basics. So get comfortable with the simple confirming.

It can just be, if you think what is the simplest bug report, it can be “The document doesn’t look the same as in some other software”. Then you just confirm it. You open the document and notice – yes, that’s truem and there might be screenshots. So it can be extremely simple to verify these reports.

In my mentoring I meet with the mentees regularly in the chat – in the text chat – and I review their work. So we go through the reports that they commented on, and I give feedback, and after a couple of dozen basic tests, I already introduced this advanced bisecting technique to them. Which kind of sounds ambitious or at least sounded to me when I started doing it, because myself – I took quite a long time to really get into it, actually over a period of two years. It was somehow daunting to me to get into it, but I wanted to refine the documentation and the whole process of getting into it, and now we are getting quite good results.

Depending a little bit on the technical background of the person, we might have them doing bisecting like in a week or two when they were introduced to it. So they immediately can get results from it instead of just scratching their heads for weeks and weeks and trying to figure out what the thing is about. So that’s always kind of impressive to me how quickly they are able to get into it.

Mike Indeed – two weeks from checking how a document looks to doing bibisecting. That’s cool. So yes, thanks guys – I think that’s a good overview of what we’re doing in the QA project and some of the different challenges and opportunities as well. Anything else you’d like to add?

Xisco For someone new interested in contributed in QA, Ilmari is also doing weekly live sessions where – well maybe you can talk a bit about those because I think it’s interesting for someone new to see the way you think when you are triaging issues, or the thinking behind what your actions.

Ilmari That’s a good point – I forgot it. So I indeed do these kind of live streams. Not really live streams. I don’t use Twitch or YouTube or anything. Just have a conference room and people can join and ask questions.

I do it one hour at a time – I don’t prepare for it beforehand. So it’s very realistic. I just look at the list of reports and go through them one by one and explain what I’m seeing and what I’m going to do and stuff like that. So it gives a realistic view of the testing process.

Mike But not on Twitch – so we can’t make live donations while you’re doing it!

Ilmari Yeah, I don’t really understand all that stuff. Like, if we had thousand people in the chat like – chat is going brrrrrr! It doesn’t seem to mesh well with the testing – it would be too distracting.

Mike Quite probably. OK, thanks a lot Xisco and Ilmari – I think that was a good overview of what we’re doing 0 what you guys are doing – and again the hundreds of people in the community. When I look around on Bugzilla, and as we’re recording this we’re doing a Month of LibreOffice campaign, where I collect statistics from Bugzilla for people who are confirming bug reports, contributing, a lot of new volunteers as well.

So it really is a big group effort – and again anybody I think it’s a good way for anybody to get involved and yes, the simplest things, comparing a document, saying “Oh yes, in this bug report the same thing happens to me in this version on this operating system”

So I think it is a really useful, genuinely useful way to jump into the project, help out and make a difference as well to the software. So let’s wrap it up here then episode three of the podcast – we have more to come covering different topics as well in LibreOffice. So, thanks Xisco. Thanks, Ilmari.