Category: Team

The year 2015 brought some challenging and exciting developments regarding the ongoing restructuring of our infrastructure. At the beginning of the year, the migration of our existing virtual machines and bare metal machines was ongoing after an extensive test phase of the new virtualization platform.

This virtualization platform consists of three servers, each with 256GB RAM, 64 CPU cores and quite a lot of hard drive space. One of the machines is meant to be used exclusively by developers for crash testing. These machines are all hosted by manitu in St. Wendel, Germany, and are currently undergoing migration onto our own dedicated 42U rack – including the flexibility to set up a private network between these machines and others that we house there.

After some problems with the software previously chosen for our virtualization platform, much work went into setting up virtual machines where services run isolated from each other, based on plain KVM. This already led to the transition of the hosted blog to one of our own machines, which give us more control over installed plugins, and also provides more flexible control over the WordPress setup that we use.

During the Hackfest at the University of Gran Canaria, work went into making the used Salt States more easy to hack on by people who want to get involved in our infrastructure. This also resulted in a tutorial video on how to create a development environment for our infrastructure.

Monthly infra calls were also set up, taking place every last Wednesday of the month at 17:00 UTC. They resulted in the creation of a weekly maintenance window for server upgrades, reboots and major configuration changes, every Monday between 03:00 and 05:00 UTC.

Operating system upgrades

During the calls the community decided to upgrade the base operating system to Debian 8 over the next few months. This was already carried out on one of our virtualization hosts during the newly set up maintenance window, in order to check for any problems that may occur during the update. During the upgrade, some obstacles were identified and workarounds were set in place to allow smooth upgrades.

We have also invested in hardware from vendor Thomas Krenn which will allow us to set up two additional Windows buildbots with powerful dual CPUs and high speed SSDs, along with two more Linux buildbots with the same specs. These buildbots will also be housed in St. Wendel and connect to our growing intranet there. Two more servers will be used for backup space. We plan to connect all TDF-owned hardware with a VPN, forming a world-wide intranet.

In the second half of the year, more machines were migrated to Debian 8, including the two hypervisors still running Wheezy (Debian 7). Due to the huge success of the new build bots, two more were ordered and now extend the intranet, with a high performance cloud core router from Mikrotik becoming the central connection point of our intranet. The cloud core router also serves as a VPN provider for TDF members at areas with restricted internet access – such as the LibreOffice conference in Aarhus, Denmark in October.

As the number of new servers grew, we decided to migrate our monitoring platform to TKmon, running on a high-availability virtual machine that is separated from the rest of our infrastructure. TKmon integrates with the hardware vendor’s support and notifies them of hardware failures automatically. TKmon is open source software and uses tools such as icinga and pnp4nagios.

To be more flexible with the monitoring notifications, I wrote a tool called TMB that provides a bot for the Telegram chat service and sends notifications to admins. Development happened with PyCharm, a Python IDE.

Our server fleet

The current state of the infrastructure consists of three rented hypervisors, each with four CPUs, 256 GB RAM, eight HDDs and partially SSDs. Additional rented servers include one backup server and one website stand-in host that was needed after the virtualization problems occurred at the beginning of the year, and that will be decommissioned soon. Nine housed servers with Intel SSDs and powerful dual CPUs are only reachable in the intranet, with access to them being controlled by the core router.

On the hypervisors, there are currently 31 VMs, providing services such as AskBot, WordPress, Gerrit, Bugzilla, Jenkins, MozTrap and much more. At Hetzner there are currently four servers: one that contains the Wiki, MirrorBrain and our public mailing lists, one that is for internal services, and two backup hosts – including one that provides storage capacity of over 17TB and is currently being set up.

Much of our documentation and many of our Salt States are published now at https://github.com/tdf/salt-states-base, while the compiled documentation can be found at http://salt-states-base.readthedocs.org/en/latest/. The Salt States are now tested with Travis and the build results are at https://travis-ci.org/tdf/salt-states-base. It is therefore now very easy to contribute to development and improve the documentation. Just fork the repository and create a pull request – then the results will automatically be tested in Travis. If you want to contribute to the infrastructure of our projects, you are invited to join our monthly infra calls, the next taking place on … or introduce yourself to the infra team in #tdf-infra on Freenode.

I’d like to introduce myself as the new Marketing Assistant at The Document Foundation. My name is Mike Saunders, and some LibreOffice followers may have already seen the 5.1 New Feature videos that I made for Calc, Impress and Writer. Others may have come across my work in Linux Voice magazine in the last two years, and other computing publications such as Linux Format before that. (I’ve also written a book about Linux.)

I’ve been using, developing, writing about and advocating open source and Free Software since 1998, when I took the plunge with Red Hat Linux 5.1 from a magazine coverdisc. Around the same time, I discovered StarOffice, the commercial office suite which, of course, eventually became OpenOffice.org and now LibreOffice. I’ve seen the Free Software community grow from a grassroots movement to a major force in the computing world, and along with writing about and promoting FOSS, I’ve tried to give a little bit back with my own Free Software project: MikeOS.

So I’m really excited to be joining The Document Foundation – helping to promote and spread awareness about LibreOffice. Not only is LibreOffice a hugely versatile and useful piece of software, it represents much more: the growing importance of open standards and document formats. It’s clear that companies, communities and local governments are starting to recognise that fully open formats are the future, so it’s great to see adoption of LibreOffice all around the world.

I’ll be working on a part-time basis, helping The Document Foundation with various tasks and projects. We’re going to be at CeBIT in mid-March, we’re working on our Annual Report for 2015, and we have many ideas for presentations, events and social media promotions in the pipeline.

So far I’ve met lots of great people involved in LibreOffice and TDF, and no doubt I’ll be meeting many more – either in person at events, or collaborating on the wiki and mailing lists – over the coming 12 months. I look forward to working with you!

LibreOffice proper

The year 2015 started off with LibreOffice 4.4.0 and 4.3.6 rc1 – only the first of a whopping 42 (of course!) tags that were created and for which a build was uploaded to our mirror network – and ends with the stable releases 4.4.7 and 5.0.4 and with the first RC for LibreOffice 5.1.0 (for those interested: that’s a total of 216GB for the binary builds).

This constant stream of builds allows our volunteer testers to always stay on top, not waste time with testing stuff only to find out a fix has already been checked in weeks ago.

But even the constant stream of full builds sometimes is not recent enough. Daily builds provided by various tinderboxes fill this gap. Not only do they constantly monitor the buildability of all current branches (and nag committers in case our continuous integration platform didn’t catch the problem before it ended up in the code), many also provide one installationset per day. That way users don’t have to fiddle with building LibreOffice themselves, but still can run the current code.

We also changed our release baseline:

• since LibreOffice 4.4 we require Mac OS X 10.8 or later and dropped the 32bit version

• since LibreOffice 5.0 we added 64bit builds for Windows (requiring Vista or later, 32bit build requires Windows XP SP3 or later)

• since LibreOffice 5.1 the new baseline for Linux is CentOS 6 (kernel 2.6.32 or later, glibc 2.12 or later)

Pootle

But the year was not only spent on LibreOffice alone. After having migrated bugzilla to our own infrastructure, it was time to tackle pootle. As mentioned in the last behind-the-scenes, we upgraded our version of pootle to run on TDF-hosted hardware and added master workflow. This allowed translators to start with their work for the 5.0 and subsequently 5.1 much earlier than with the old process (where translation could only begin after the branch was created).

Android

Good news everyone, there’s now a LibreOffice Viewer for Android! While still in its infancy, it is a first step to an editor application that you can use to give your documents the final touch when on the road.

If you own a smartwatch, then you also might be interested in the updated Impress Remote for Android, that now integrates with Android Wearable devices.

Hackfests

Continuing from last year, Hackfest-VMs were provided for the various gatherings that took place in 2015 and will be provided to those happening in 2016. Participants don’t have to roll thumbs waiting until the build is finished on their personal machines, but can instead start digging into the code on a fast virtual machine, and also do some bibisecting to warm up/find an easy entry point.

tl;dr

• nearly one LibreOffice build per week on top of that daily builds from tinderboxes

• Impress Remote for Android got support for Wearables

• LibreOffice Viewer for Android with basic editing capabilities

• Bugzilla was migrated to TDF-Infrastructure

• Pootle server was upgraded and moved to TDF-Infrastructure, with translators now being able to work on master branch

LibreOffice QA has been busy during the last few months!

We had our first BugHunting Session for the upcoming LibreOffice 5.1 release over the Halloween weekend at the end of October. Testing our alpha1 builds, members of the QA Team helped to lead users, developers, and other community members in identifying and documenting problems in our very first binaries available for this release series. Having support from members of QA, L10n, Developers, members of the Design community, and regular LibreOffice users was a great sign for continued cross-team participation. We’re looking forward to seeing a similar mix of contributors during our next BugHunting Session coming up this weekend.

We identified 104 bugs during the Halloween BugHunting session, primarily from initial testing with the alpha1. Of the 74 of bugs remaining open, 80% have been triaged to previous versions, leaving only 15 bugs tied to LibreOffice 5.1. Looking at all 5.1 versions, there are only 212 open bugs pegged to this release series, with only 44 open bugs pegged against 5.1 alpha1.

We’ve been busy with regression-hunting, performing over 550 bibisects this year, and over 200 since June. With one of our chief regression hunters and bibisect sorcerer Matthew Francis taking a well-earned respite from the cauldron of crafting new bibisect repositories and hacking on Python internals, we’re actively recruiting new QA members to help perform these binary searches on all platforms.

In fact, with so many of our developers (and other community members) using Free Software operating systems on their workstations, there’s always a need for contributors who are running MS-Windows or OS X to help us track down OS-specific bugs. This includes not only performing bibisects, but also debugging and getting a backtrace for a crashing bug.

Keeping our UNCONFIRMED bug count steady has been challenging. Currently hovering in the mid-500s, other important QA tasks and processes have commanded our time, and we could benefit from several more QA Team members to help shoulder the daily influx of new, untriaged bug reports. With the focus of our regular BugHunting Sessions on finding and identifying new issues in our upcoming builds, we may find it useful to host similar events that focus wholly on triage and “gardening” of the mass of existing reports in Bugzilla. Maybe we could have something like “Clean Sweep Mondays,” where we tackle the existing clutter for a set period of time, and post some quick stats such as total drop in UNCONFIRMED, # of NEEDINFO cleaned up, etc.

Work continues on our comprehensive Media Support wiki pages, a collection of tests designed to directly verify the level of support for various image, audio, video, (and other) file formats across all of our platforms. Keeping these pages up to date ensures that they continue to be a resource for QA bug triaging as well as for our volunteers helping to answer questions on the Ask LibreOffice site. Lead by QA Team member raal, we now have initial image test results for Android, and are interested in talking with developers who’d like to expand the Android Viewer’s capabilities in this area, as well as testers who would like to expand our battery of test results for the Android OS.

This weekend (Fri, Sat, Sun), we’re holding our BugHunting Session for the 5.1 Beta1 build. For those of you who are interested in kicking the tires and working with a still-rough-around-the-edges piece of software, your help is greatly appreciated. With a large package such as LibreOffice, it’s invaluable to have the technical members of our userbase exercise some of the more esoteric features of LibreOffice, and identify any minor regressions or usability concerns early-on in the release process.

The QA Team looks forward to seeing you in IRC anytime, or stopping by one of our weekly Wednesday meetings. We’re always happy to answer your questions about bug reports, bibisecting, etc., and to helping new contributors get started with the LibreOffice community!

LibOCon in Aarhus, Denmark, has been a really nice meeting especially for the L10n/NLP communities with a large workshop on the day before the opening of the conference. Even the whole afternoon was too short to discuss all the topics we had on the agenda. Around 40 persons from all over the world were attending the meeting. We have shared experiences and feedback, Christian has explained the new workflow with master and how branching is managed, and discussed some Pootle enhancements as well as other enhancements on coordination.

At the end of the meeting we discussed a specific issue with accelerators that are broken in some languages and specifically in languages using complex script layouts. This has led Niklas Johansson to develop a tool that detects duplicate use of accelerator keys in menus. This tool is of great help for all of us, translators, but also for our users and even more when accessibility matters. Thanks so much Niklas for this!

If you missed the conference, you’ll find all the videos of the talks on the TDF channel [1], there has been a whole track about localization, migrations and marketing. Italo has also organized marketing workshops dedicated to NLPs.

As a direct consequence of this meeting, we have opened a NLP channel on IRC, still too quiet in my opinion, don’t hesitate to join us and say hello! on Freenode #libreoffice-NLP, we will be happy to meet you there.

This meeting has also encouraged the Native Language projects to give more feedback on what they are doing locally, so far we have news about events in Korea, Turkey, Albania, India, Singapour, Sri Lanka… ans still feedback from the very active communities in Japan and in Italy where a first hackfest has been organized.

In more details, the Turkish community has participated to the DevFest in Istambul, giving an introduction on LibreOffice developments. I’d like to highlight the work done by Kader Tarlan who has written almost 50 new tests for MozTrap, and is currently working on porting java tests to python, a big big thank you to her!

The L10n community has extended its translations to some part of the tools we are using and has provided translations for Redmine and Ask in a very timely manner. I’m very happy to see the reactivity of our project especially during the translation sprint of the next version, namely the LibreOffice 5.1 release. This release has had a lot of changes in its off line help, it’s a very good thing (thanks a lot to Lera Goncharuk and Olivier Hallot!) but that means also a bunch of work for the localizers. Chandrakant Dhutadmal, representing the Centre for Development of Advanced Computing, has took the lead on several Indian languages in Pootle and will try to revive their localization, that’s a really nice initiative, thanks also to him.

I wish to all a happy holiday period and see you next year!

[1] https://www.youtube.com/channel/UCQAClQkZEm2rkWvU5bvCAXQ