Category: Tenders

Donations to The Document Foundation are used for many purposes, such as organising events, maintaining our infrastructure, and paying a small team to handle QA, marketing, documentation and other tasks. But donations are also used to fund tenders, whereby companies and individuals improve LibreOffice in specific areas and share knowledge with the community.

One such tender was posted in May 2017: “improve image handling in LibreOffice (#201705-01)“. When images are used in LibreOffice documents, the software manages them in a “life-cycle” which includes importing, displaying, modifying, exporting and more. To save memory – especially with large documents – images that are not currently on screen are sometimes moved out of memory and saved onto disk in a technique known as “swapping” or “paging”. The goal of the tender was to improve LibreOffice in these areas, making it more efficient at handling images and modernising the code base.

Collabora was selected to implement the tender; the work is now complete, and it will benefit all users in the upcoming LibreOffice 6.1 (due to be released in early August). Here are some technical notes about what was improved in the source code of LibreOffice, and what was achieved.

Problems with the image life-cycle

Currently, when an image is read from a document, a GraphicObject is created for the image and handled over to the GraphicManager which manages the life-cycle. When this happens we usually get back the string based unique ID of the GraphicObject with which we can always get access the image by creating a new GraphicObject with the unique ID (GraphicManager will look for the image with that unique ID).

Usually the unique ID is the one that is passed on between layers in LibreOffice (for example, from the ODF filter when loaded, to the model, where it is manipulated and then to the OOXML filter when saving) but the unique ID itself is just a “reference” to the image and by itself it doesn’t have any control over when the image can safely be removed and when not. It could happen that in a certain situation we would still have the unique ID referenced somewhere in the model, but the image would already be removed. This is dangerous and needs to be changed.

Usually for this kind of object we use a reference counting technique, where we pass an object around that holds a reference to the object resource. When the object is created, the reference count is increased; when destroyed, the reference count is decreased; when the reference count reaches zero, the resource object is destroyed.

The solution for the life-cycle

So instead of passing around a unique ID, the idea is to use the usual reference counting technique, which is normally used in this situation. The GraphicObject is mainly a wrapper around Graphic (which then holds a pixel-based image, or animated image, or possibly a vector image), and in addition it keeps additional attributes (gamma, crop, transparency etc.). It also has the implementation of swapping-in and out.

On the other hand, Graphic is properly reference-counted already (Graphic objects are reference counting the private ImpGraphic) so the solution to the life-cycle problem is that instead of GraphicObject unique ID, we would just pass along the Graphic object instead, or XGraphic, XBitmap which are just UNO wrappers around Graphic. Potentially we could also pass along the GraphicObject or XGraphicObject (UNO wrapper for the GraphicObject) when we would need to take into account the graphic attributes too. This should make the life-cycle much more manageable.

GraphicObject refactoring

GraphicObject and the implementation of XGraphicObject (UnoGraphicObject) and XGraphic (UnoGraphic) were located in module svtools, which is hierarchically above vcl. This is problematic when creating new instances like in Graphic.GetXGraphic method, which needs to bend backward to make it even work (ugly hack by sending the pointer value as URL string to GraphicProvider). The solution to this is to move all GraphicObject related things to vcl, which surprisingly didn’t cause a lot problem and once done, it looks like a much more “natural” place.

Managing memory used by images

Previously, the memory managing was done on the level of GraphicObjects, where a GraphicManager and Graphic-Cache were responsible to create new instances from uniqueID and manage the memory usage that GraphicObject take. Here’s the hierarchy before refactoring:

This is not possible anymore as we don’t operate with uniqueIDs anymore, but always use Graphic and XGraphic objects (in UNO), so we need to manage the creation of Graphic object or more precisely – ImpGraphic (Graphic objects are just reference-counted objects of ImpGraphic).

So to make this possible GraphicManager and GraphicCache need to be decoupled and removed from GraphicObject and a new manager needs to be introduced between Graphic and ImpGraphic, where the manager controls the creation and accounts for the memory usage:

Graphic swapping and swapping strategy

The new swapping strategy is relatively simple – if a lot of memory is needed by graphic objects in a certain time, we let it use it and don’t try to over-aggressively try to free it. In the past this cased swap-out and swap-in cycle that made the application completely unusable. In the future, external hints when a certain Graphic object can be swapped out may be added, so we can perform swapping more effectively. There are also several other ideas which will increase performance and reduce memory usage that can be implemented now with the new hierarchy where most all of the swapping is contained inside the Graphic itself, but all of this is currently out of the scope of this work.

In conclusion

Thanks to Collabora and Tomaž Vajngerl for their work on this. Although the details are highly technical, the end result is a faster and more robust office suite. If you’re an end user of LibreOffice and your documents include lots of images, you will be able to enjoy the benefits of this work in future releases, starting with LibreOffice 6.1.

LibreOffice is designed with great attention to accessibility, to make the suite convenient and comfortable to use, and to cater to users with special needs. Last year The Document Foundation published a Tender to Implement Accessibility Improvements regarding user interface widgets that are added to the suite but which have accessibility shortcomings.

French company Hypra, which works on accessibility improvements in free and open source software, was awarded the tender. Today they are announcing the tool that they have developed – note that this tool is not targeted at end users of LibreOffice, but rather at developers. Of course, all users who have experienced accessibility issues in LibreOffice can benefit from it. Let’s hear what Hypra has to say…

Making LibreOffice usable by everybody, including disabled people, is a must so that anyone can work on documents for their own uses or their daily job. Accessibility concerns happen to be very diverse, and fixing or even just detecting them can be complex. Some of them can actually be detected automatically, and this is the goal of the “gla11y” tool, which is the result of the tender.

Gla11y (Glade accessibility) is a Python script which takes .ui files (Glade graphical interface description files), and reports the accessibility issues which can be found with static analysis. This includes mostly the missing labelling relations, which happens to be a very frequent accessibility issue. If, for instance, a dialog box contains several GtkEntry widgets to be filled in by the user, and several GtkLabel widgets to describe them, but with no relations between the two series, a blind user would have to remember the different parts, thus considerably reducing usability. This will show up in gla11y warnings as:

     myfile.ui:5 WARNING: 'GtkLabel' 'label1' does not specify what it labels within 'GtkFrame' 'frame1'
     myfile.ui:9 WARNING: 'GtkLabel' 'label2' does not specify what it labels within 'GtkFrame' 'frame1'
     myfile.ui:12 WARNING: 'GtkEntry' 'entry1' has no accessibility label while there are orphan labels within 'GtkFrame' 'frame1'
     myfile.ui:14 WARNING: 'GtkEntry' 'entry2' has no accessibility label while there are orphan labels within 'GtkFrame' 'frame1'
     4 new warnings 

Adding relations between each widget and its label will allow screen readers to tell users exactly when to type what.

Running a basic version of gla11y alongside compilation of LibreOffice was integrated on February 21; It only checked for broken relation links. It turns out that there is currently no such bug, so it didn’t raise any warning, but now we are sure that no such bug will be introduced in the future (as happened in the past).

Gla11y runs in all compilation cases, except when using the old version 2.6 of Python and python-lxml is not available, so it will be running in mostly all development scenarios, thus catching issues as early as possible. It is useful to have the python-lxml library installed, or make sure that it is built automatically by LibreOffice, so that the analysis can be done faster and warnings get displayed better.

An initial version of more advanced analysis, which actually finds some existing issues, was merged on February 28.

As of today, the tool reports a total of 2155 warnings in 434 .ui files (out of the existing 981 .ui files). Of course, we do not want to overwhelm developers with all of these existing issues, so a suppression mechanism has been implemented, so that only warnings for new issues will be emitted; warnings for existing issues will be silenced by rules in suppression files.

In this way, to start with we will prevent the introduction new accessibility issues, while existing issues are progressively fixed. Also, for now only very basic warnings are enabled, and we will enable the complete set progressively. That will allow us to observe how developers react to some warnings, and fix heuristics and documentation on the
wiki for them, before enabling more warnings.

Regarding fixes for existing issues, gla11y reports both labels and widgets which don’t have relations (“orphaned” labels and widgets). Very often, it is very easy to find which relation is missing and remove the warnings (actually, remove the suppression rules) with a few lines of .ui files. Working on this could thus be part of “Easy Hacks” for new contributors.

Fixing warnings raised by gla11y will hopefully greatly improve usability in LibreOffice for disabled people. Of course, there remain many other accessibility issues which can not be detected by static analysis, but gla11y will helps to cover a fair bit of the work, and leave time to concentrate on the more difficult issues.

Thanks to Hypra for their work on this tool! Learn more about how TDF uses its tendering process to improve LibreOffice and share knowledge with the community.