Behind the scenes at TDF: Localization and Native-Language Projects

Sophie Gautier has been a member of the OpenOffice.org project since its beginning, and then a founding member of The Document Foundation and LibreOffice. She is extremely active in the Francophone and international community, and is a staff member of The Document Foundation. She takes care of the French translation of LibreOffice (interface and help), is a member of LibreOffice certification committee and is a leading member of the quality assurance project.

2015 is more than ever a year packed with exciting projects and ideas around LibreOffice and The Document Foundation, so we want to continue our behind-the-scenes series, to share achievements with our community and our generous donours, to whom we’d like to express our sincerest gratitude and thanks for their incredible and wonderful support and their invaluable contributions!

The localization team has been very busy translating for the 4.4.x version, a lot of dialogues have been modified, so thousands of strings were touched, moved and need to be translated and validated again.

The L10N team had an important discussion on the workflow and the current workload due to changes on the sources, whether they are needed or purely cosmetic, which resulted in several decisions. The first is that the teams willing to work on master will have a new Pootle project reflecting the changes done there. It will be merged once a month and the template will be updated in Pootle accordingly. This allows us to check the strings much earlier and revert eventually unneeded changes. The teams wishing to work at the branch levels will still be able to do so.

To be able to reach out to developers more quickly and get a better communication between the teams, I attend and report the L10N activities and needs to the Engineering Steering Committee, by attending the weekly calls. For example, the request to have a mechanism that handles the localization of the templates will be provided for 4.5 and strings will be uploaded on Pootle.

The migration to the new Pootle version is going on. We are closely working with the Pootle team to get this done smoothly and to have the whole set of features the L10N team needs. The Deckard addendum will be the next step.

A cross work between the documentation and translation projects has been brought up by Milos Sramek from the Slovak community. They have developed a whole workflow to translate the user guides and to maintain them. After some tests in different languages, we have decided to use it for the whole project and document it on the wiki. This is handled via the OmegaT Project feature and we use the LibreOffice GitHub repository to manage revisions in the .ODT file, which turned out to be really time saving and reducing errors – even if the first work is important, it allows afterwards to only handle modifications needed by new LibreOffice versions. If you are willing to use this workflow for your own translation projects, even if it’s another LibreOffice writing, don’t hesitate to contact us either on the documentation or the L10N list.

Some new languages added to Pootle during the first quarter are: Guarani, Nahualt, Tigrinya, Pashto and a new contributor who will work on Tatar, which was stalled for the moment. Welcome to all of them, keep up the good work, guys!

It is a bit early, but let’s already talk about what L10N and NLPs wanted to discuss during the LibreOffice Conference. I am very happy to see that we will have a large group representing the teams this year! 🙂 If you are active in the L10N or NLPs groups and wish to attend, don’t hesitate to come back to me via e-mail and have a look also at the conference website. We will have workshops, discussions and presentations sharing our experiences, difficulties, tips & tricks, but the most importan,t we will be altogether in the same room 🙂 If you can’t attend, don’t feel sad, we will try to organize a hangout and an IRC chat as well.

TDF has also been a supporter of the Document Freedom Day, an event that will be followed by several groups all over the world. I’ll report about it next quarter. The Brazilian team has launched the 15th edition of its magazine. The Japanese team is, as always, organizing several events, trainings and mentoring during this quarter. Don’t forget to have a look at our calendar to follow the activities and perhaps meet a team exhibiting near your place.

Behind the scenes at TDF: Infrastructure

With the beginning of 2015, a new year packed with exciting projects and ideas around LibreOffice and The Document Foundation, we today finish our behind-the-scenes series, to share achievements in 2014 with our community and our generous donours, to whom we’d like to express our sincerest gratitude and thanks for their incredible and wonderful support and their invaluable contributions!

I’m Alexander Werner and I am responsible for the infrastructure of The Document Foundation on a contracted basis since March 2014. I have been with the project since its foundation in 2012, and been a longtime supporter of free and open source software. As a volunteer I helped setting up and maintaining our first server and optimizing it to handle the load of the first days.
The infrastructure is one of the most important things The Document Foundation provides for the community. As long as every part is working as expected, it is basically invisible. It is my job to make sure that this is always the case, mostly by orchestrating the different services on our growing number of virtual machines.

When the LibreOffice fork began, we started with only one server where all services were located – mailing lists, both private and public, website, mirror management, wiki and many more. As time went by, this server survived its first slashdot, but soon it became clear that more power was needed. So our infrastructure started growing organically as more and more servers were added. Our admins specialized on different parts of the infrastructure, while the whole configuration was centrally documented in a single ODT file.

It soon became clear that this was not a viable solution – our quest for infra 2.0, as we internally call it, began. The admin team worked under fast escalating load while looking for ways to optimize resource usage, inclusion of new volunteers, configuration documentation and management. Also high availablility of services became increasingly important. In our sparse free time we started creating concepts, tested HA with DRBD, Pacemaker and Heartbeat, evaluated different solutions for centralized documentation and started using tools for centralized configuration management.

It soon became clear that we needed more flexibility for working HA with the solution described above, so as interim solution we started virtualizing services first in paravirtualized guests with LXC and then switched to fully virtualized guests with KVM. For infrastructure documentation I suggested to use the documentation generator Sphinx. The source files for the documentation – human readable RST text files – are located in a git repository, and the online documentation is automatically updated on every push. For configuration management and deployment, I eventually stumbled upon SaltStack.

My daily work consists of working on various small recurring tasks such as helping people with mailing list troubles, adding and removing mirrors in MirrorBrain, installing updates and doing necessary reboots as well as handling unexpected incidents such as the Heartbleed bug.

In spring I started working on our Salt states, made them more reliable and made sure that all user accounts are now managed by Salt. I have setup a new virtualization host with VMs for Gerrit, Jenkins, Bugzilla and Plone. Apart from that I worked on improving the documentation of our services, looking for undocumented and unused services.

I also worked on our AskBot setup. While having set up the initial AskLibO instance, it was decided to contract Evgeny Fadeev, the primary developer of AskBot, to develop additional features needed by our community, which will then be made available upstream again. Despite that, I also did some changes such as enabling the newly-developed multilanguage support, fixed template bugs and administered the list of moderators.

Except for my ongoing work to improve the Salt states and adding more not yet managed servers to our Salt infrastructure, I also continued to concatenate various documentation sources into our centralized repository.

I also worked on a download counter that will be useful to track all our downloads by language, location, version and operating system.

But the most interesting, time consuming and fascinating part of my work was the planning, testing and setup of our new cluster/cloud infrastructure. As it was decided to virtualize all of our services, I looked for a solution that is easy to manage and maintain but provides powerful tools for easily creating highly available services.

After quite some time of evaluating I decided to go for oVirt – a KVM-based virtualization solution that provides a nice out-of-the-box experience, the simplicity of its setup was worlds apart from other solutions. It is also possible to provide fully high available services with only two nodes by having the management engine run as VM on the platform.

During the time of evaluation I also had contact to hardware suppliers and hosters, and after a good offer from manitu we decided to host our new platform on two large, dedicated servers, each with 256 GB RAM and 64 CPU cores. Until the end of the year, over 20 virtual machines were migrated and a third node was ordered that will be used primarily for crash testing and to increase the stability of the platform even more.

If you are interested in learning more about our infrastructure or helping out, consider subscribing to the website mailing list, where infra calls are announced or write a mail to alex@documentfoundation.org

Behind the scenes at TDF: Executive Director

With the beginning of 2015, a new year packed with exciting projects and ideas around LibreOffice and The Document Foundation, we continue our behind-the-scenes series, to share achievements in 2014 with our community and our generous donours, to whom we’d like to express our sincerest gratitude and thanks for their incredible and wonderful support and their invaluable contributions!

I’m Florian, and I live in the very southern part of Bavaria in Germany, 100 km southwest of Munich, near the border to Austria and Switzerland – a beautiful place to be. 😉

Today, I’d like to shed a light on my role as Executive Director, a fulltime position which I have held since last March. With TDF having grown over the past years in terms of contributions, projects, ideas, staff and donations, my role as Executive Director is, in a nutshell, to keep everything together and have “the show running”, working for the Board of Directors and with the Membership Committee,floeff2012_400x400 and all the other wonderful volunteers, contributors and staff members like Sophie, Italo, Christian, Robinson and Alex.

Having worked intensively on the statutes and the initial setup of TDF, the first part of my role is taking care of many administrative, legal and tax bits, removing that burden from the board to give them more time for strategy and other items. This work reaches from working closely with our accountant, tax advisor, payroll provider and legal counsel, to dealing with trademark requests, protection of domain names, checking and payment of invoices, and reviewing contracts and insurances, making proposals for the investment of our capital stock and other funds, and of course answering questions on details of the statutes and other parameters of the German “Stiftung”.

The second part of my role is handling the daily operations, which includes replying to and distribution of all sorts of inquiries and requests, be it formal letters, user inquiries and, together with other TDF spokespeople, a variety of press and media inquiries. The daily operations also involve monitoring the budget, organizing our internal file storage, preparing presentations for the annual board meetings, participating in and proposing agenda items for the biweekly phone conferences, handling our donation mechanisms, bank accounts and donation confirmations, monitoring deadlines and working on relations to our Advisory Board. When a new Board of Directors or Membership Committee is elected, or we have new staff members, I have the honour of introducing them to our internal workflow and our entity.

The most exciting part of my job surely is working with our staff in various projects, and the coordination of tasks, deadlines, priorities. As Executive Director, I am responsible for oversight of all budget items and projects TDF is carrying out. At the end of 2013 we have switched to Redmine for project management – started out of the infrastructure team’s needs, it has become an important tool for task and project handling in some non-development parts of LibreOffice. In addition to that, we are running weekly team calls with our staff members, have slots reserved for weekly one-on-one calls, and meet in IRC regularly to discuss current projects and challenges. Every once in a while we also have in-person meetings, mostly during LibreOffice Conferences and FOSDEM. Being responsible for that part of the foundation also involved handling tender processes, job openings and talking to candidates who apply for a job.

We do have a couple of recurring projects, like gradually improving our AskBot instance or writing our formal annual report, but there’s also individual items, like the certification program, the Android tender and the Bugzilla migration.

Should I find some spare time during the weekends, I also like hacking around on our infrastructure, especially our e-mail system and our Redmine instance, and try to join our admin phone calls and infra in-person meetings when possible. Having been a part of the German community for many years, I also try to regularly organize community meetings and phone conferences.

Last but not least, every once in a while I give a presentation, staff a booth at an event, and sometimes I even make it to a magazine or try to say some wise words in a podcast or a lecture. 😉

Behind the scenes at TDF: Release Engineering

Towards the end of the year, The Document Foundation would like to share achievements in 2014 with our community and our generous donours, to whom we’d like to express our sincerest gratitude and thanks for their incredible and wonderful support and their invaluable contributions!

Hi, I’m Christian Lohmaier and I’m Release Engineer as well as Infrastructure Administrator at The Document Foundation and have been with the project since it’s foundation. Also long before I was active in the predecessor project, where I filed my first bugreport more than 14 years ago (2000-10-23) and later was one of the founding members of the German native-lang project 🙂

cloph_headIn 2014 we redesigned our main user-facing website (www.libreoffice.org) – working with Eleonora Anzini (who created the design) to bring her vision to our CMS (Silverstripe) was one of the first big tasks this year, that already started in 2013 and after lots of tweaking of both the content (managed by Charles-H. Schulz) and the styles/the framework, the new design went live in March.

Release-wise, the year started with 4.1.5 rc1 and 4.2.0 rc3 – only the first in a series of around 50 released builds (betas and release candidates) for QA and curious individuals, not counting the daily/nightly builds provided by our tinderboxes for all platforms.

Maintaining two concurrent release branches at a time, combined with our time-based release schedule ensures a constant flow of bugfixes and updated translations (for the enduser every four to five weeks) with a new major release every six months, but of course also sometimes it is challenging for some parts of the projects.

Especially for each new major (in this context a new x.y.0 release), work starts much earlier, and integrating the translations takes more time than for a bugfix release, on top of that we updated our baseline build configuration a couple of times. From Windows Server 2008R2 with Visual Studio 2010 for 4.1.x and earlier, to Windows Server 2012R2 with Visual Studio 2012 for 4.2.x and 4.4.x to Windows Server 2012R2 with Visual Studio 2013 for master/4.5.x and later, for Linux from CentOS 5 with gcc4.4 for 4.3.x and earlier, to CentOS 5 with gcc4.8/devtools2 for 4.4.x and later and on Mac from 10.6 baseline for 4.3.x and earlier to 10.8/64bit for 4.4.x and master.

As Sophie mentioned in her posting, LibreOffice transitioned from using static dialogs to dynamically created ones (using glade to define them). That, together with other cleanup, did result in many changes in parts of strings that aren’t meant to be translated, but would still result in the string being flagged as new and thus translators would have to retranslate it. To minimize the impact, I used some scripting-foo that can map the old to the new string and thus can reapply translations while automatically updating the non-visible part, hopefully saving the ~120 language teams quite a bit of work.

To reduce translators’ workload before a new major release, we’re going to add a master based translation project in Pootle – this will allow translators to update and polish translations in the whole six months between releases, and not just in the couple of weeks after branch-off. This also allows to sync translations with the master branch, meaning translators can see their translations in a developer build much easier (and thus earlier).

Besides LibreOffice itself, 2014 also the rewrite of the LibreOffice Impress Remote for Android and iOS were released, allowing you to use your smartphone or tablet to navigate between slides and control a virtual laser-pointer.

In the second half of the year, I prepared a Hackfest-VM for the first time (a ready-to-start-hacking VM with prebuilt LibreOffice source tree run on amazon’s EC2) that attendees could use to start coding right away without having to wait 6 hours for their laptops to complete a build. This was later extended to also allow for bibisecting. Unfortunately communication on the location was lacking sometimes, so people didn’t always know how to make use of that goodness, still it’s something that we’ll continue to provide for future hackfests as well (lession learned!).

Also in the second half of the year, the multilang planet was born. It doesn’t only aggregate English blogposts, but blogpostings from contributors in all kinds of languages. If you want to have your blog added, feel free to send a pointer.

The last months of the year were dominated by the migration to a new infrastructure, but you’ll read more about that in another post 🙂

Behind the scenes at TDF: Certification

Towards the end of the year, The Document Foundation would like to share achievements in 2014 with our community and our generous donours, to whom we’d like to express our sincerest gratitude and thanks for their incredible and wonderful support and their invaluable contributions!

italo-speakingItalo Vignoli, who is in charge of the LibreOffice Certification Program, summarizes the efforts undergone so far, the status quo and the plans for 2015:

Certification Project, looking forward to the first peer-to-peer reviews

certinfographicsTDF has announced the Certification Program in early November, with applications open only to TDF Members until April 2015, for Migrations and Trainings. The first peer-to-peer review session will be at FOSDEM in Brussels, on January 31 and February 2, 2015.

Although the website and all documents are in English, peer-to-peer reviews will be managed – as far as possible – in the native language of the candidate.

For people not able to reach Brussels, we will organize peer-to-peer reviews via videoconferencing.

The deadline for applications, to reach the first review session, is January 16, 2015, to allow two full weeks before the actual peer-to-peer review. In fact, we have to look at applications, approve or reject them (or keep them on hold when they are from non TDF Members), and provide a feedback.

We have a dozen applications so far, from TDF Members, from non TDF Members, and from developers.

We will complete the review of the first applications and provide a feedback by December 23, 2014. The process is still in the early stage, so we are getting up to speed and sharing all the decisions with the group to reach a consensus.

In general, we will accept or reject applications from TDF Members, we will postpone to March/April 2015 applications from non TDF Members, and we will forward applications from developers to the ESC.

In fact, Developers Certification is managed by the ESC, and is by invitation (as reiterated on the website, and on Certification Q&A). So, developers should not apply via the website.

Applying for LibreOffice Certification is not difficult, but to make it even easier we have created an infographics to explain it in detail. Following the seven steps, it will be easier to provide all necessary informations.
For any additional information, please write to certification@libreoffice.org.

Behind the scenes at TDF: L10N and NLP

Towards the end of the year, The Document Foundation would like to share achievements in 2014 with our community and our generous donours, to whom we’d like to express our sincerest gratitude and thanks for their incredible and wonderful support and their invaluable contributions!

Sophie Gautier is in charge of our L10N and NLP projects at The Document Foundation, and gives you insight into this key part of our project:

_SDS5526L10n – NLPs status quo

I would like to give a brief overview of the many things that happened or are in the pipe either on the Localization project (L10n) or on the Native Language projects (NLPs). For the latter, it is not always easy to know what is going on locally, but we tend to get more and more feedback from these groups which is really great.

So, on the Localization side, several new languages were added to Pootle these last months, more Indic languages, for example. Translating UI and Help is a very huge task, and when you think it is finished, there is still some work to do for the next version. For example, currently, the teams are in the starting blocks to translate the 4.4.x new features strings and the various enhancements that have been provided by the Design team. Also the change of file format to .ui means that the l10n teams had to translate again all the dialogs. Fortunately, this change is of great help because the dialogs adapt to the length of the strings (no need anymore to count the characters in the word to fit the space), but we are also able to display those dialogs in Glade, allowing to see the strings in context, which is something all localizers are dreaming of! All in all, that makes many new words for both UI and Help projects. And this is not the only translation projects we handle via Pootle, there is also the Website, Impress for Android and iOS, sometimes AskBot projects.

Dedicated to newcomers of the l10n project, two guides have been written; one concerning how to use Pootle, the second one on the structure of .po files (for example, it shows how to distinguish variables or which xml tags are used). There was absolutely no documentation on the structure contents by the past and one had to guess what he had to translate or not. And it is very easy to break a build when tags miss or mismatch, so this brings some relief to the developers too.

Thanks to our Brazilian friends several of the help articles concerning new functions have been completed. We are also working on porting the translation of the help files on the wiki. This is a difficult task because we do not want to complicate the translation task on one hand, but we want to simplify the help maintenance and open it to non-technical contributors on the other hand, which is currently impossible. Some technical issues have still to be resolved, but we are optimistic that we will be able to set it in a near future. As a work in progress also, we hope to push the migration to a new version of Pootle, with an integrated translation memory.

On the Native Language projects side, we are happy to see more and more contributors to the local projects and really good news coming from several of them, like the Italian community or the Japanese and Chinese ones. They not only contribute to their local projects, but you can find those members active in QA, development or documentation.

One of the major tasks handled this year by these teams was the translation and adaptation of the new website design. During the year, the website itself was translated on Pootle and the content by website owners. This gives a uniformed design to all the language sites bringing more quality and a professional look and feel to our project.

One way to measure the growth of activities in these projects is how we all together manage the press releases. Once the text is fixed by the marketing project, the native language projects translate them and send them back to the marketing team for distribution to the press in their countries. We are now able to release in almost 9 languages for each major release. It is also something really exciting to see how the developers, the quality assurance, native language, design and marketing projects interact during the last month before the release. Of course it happens also all time between two or three of them, but the communication has improved between all of them.

Another great thing that has happened recently while in heavy discussion since some times, is the Planet in all languages. It is really impressive to see all those languages mixed in one thread but that you can filter by the language you prefer. There is currently ten languages available covering several blog writers.

Always trying to be as transparent as possible and to bring as much information to the community as we can, the Annual Report due as a TDF official document to the Berlin authorities, has been translated into English and is available to the Native Language projects for their own use, to inform either on the product and the community.

On the local side, TDF has supported several hackfests and numerous events have been organized all over the world by the Native Language Projects. And we are really happy that the Danish team is organizing the next international LibreOffice conference in Aarhus.

To reflect all this effervescence, we have set a Big Thank You page on the wiki, where all L10n and NLPs contributors are invited to add their name. But that’s not all, we have also a world map, detailing the skills of the contributor in addition to his location. And stay tuned, more is coming!