Tenders for preparing LibreOffice for ODF 1.4

Board Discuss
#1

Hi members of the board,

ODF 1.4 is on the way to become a OASIS standard.

Microsoft has already published to support ODF 1.4.

So I think it’s time to do the next steps for LibreOffice to support ODF 1.4.

Some time ago I have collected all the things that need to be done.
Tenders for budget 2024 - The Document Foundation Wiki.
But I’m not able to give a “CostEstimate”.

Fears were expressed on the dev mailing list that it would exclude potential bidders if they entered cost estimates there.
https://lists.freedesktop.org/archives/libreoffice/2024-July/092196.html

So please discuss how we can go forward. Can members of the TDF staff give cost estimates? Or will TDF tender the needed things without cost estimates and leave it to the bidders to name the costs in their bids? What ways do you see to tender the process of bringing LibreOffice to ODF 1.4?

Kind regards,
Regina

1 Like
Update on the 2024 budget
#2

Thanks a lot Regina for bringing that to our attention. We took over a procurement policy from the previous board and it was not yet finished due to open questions.
I’ve just sent an e-mail to the board list this morning and suggested that I’ll follow-up with Carlo Piana to finalize the procurement policy. I’ll keep you updated on the progress.
Cheers - Sophie

#3

Also, I added the topic to our team call today. Italo proposed to have a dedicated meeting with the technical members of the team. We would also love to invite you to discuss it. The document you added to the wiki helps a lot and we will use it as a basis (thanks a lot for this!).

Florian also mentionned that we should put a much higher emphasis on knowledge transfer, to enable others to review things in the future and avoid that this knowledge stay within a few people.

So I’ll setup a poll to find a suitable slot for the meeting, I’ll send you the link to the poll privately.
Cheers - Sophie

#4

Indeed, this is something that becomes more and more important. We need to have knowledge publicly shared, to enable many more people to work on the code, grow their skills and be involved in tenders and projects.

For sure there will be very specialized knowledge were this is much harder to achieve than for other topics, but I suggest we focus on that in the future - that will be beneficial for the community, the ecosystem and the users.

Thanks @sophi for driving this, I am looking forward indeed to have the procurement policy in place soon!

#5

Hi y’all,

TL;DR: there’s no way to brain-dump, or document, ourselves into deep code understanding

Let me take this statement, which I believe nicely captures in a nutshell, what perhaps mutual misunderstandings have been hampering progress since a number of years (by way of talking past each other, and asking for the wrong things).

One of TDF’s core missions is public and professional education, so it is absolutely useful (and in our own self-interest), to make sure all our projects are well-documented, and in particular on-boarding and initial experience is pleasant, non-frustrating, and ultimately rewarding for as many volunteers as possible.

That said, there’s a limit (both in terms of effort required, as well as in terms of fundamental, computer science complexity / expressiveness challenges), how far documentation (as the most scalable form of knowledge-sharing) can go. Here’s why:

  • software engineering (the process of creating large artifacts made of source code, by applying engineering principles) creates precisely one artifact, that holds the authoritative knowledge-base: the source code
  • that’s all. everything else, like documentation, is by necessity auxiliary (and therefore by corollary, incomplete, inaccurate, and frequently out of date) - so there’s no way to document ourselves out of the inherent fact, that deep understanding of the existing code is necessary, for being able to reason about it (if you don’t want to take just my word for it)
  • so in order to understand (develop a mental model for) a large software system, one needs to work with the source code (build it, change it, debug it, learn how it works and behaves under change). That’s how people learn (a craft, a musical instrument, a language etc), and it’s important that one learns from the authoritative source (and not from some superficial, lossy & leaky abstraction, that was somebody braindumping into documentation). Make no mistake though: good, entry-level howtos, as well as accurate inline comments, and module / data structure concept descriptions are really helpful on that path (though no panacea - it’s like just learning the vocabulary, and assuming one would know the language)

Where does that leave us, in terms of improving knowledge-transfer? We can teach the meta-skills (problem-solving, asking questions well, providing pointers & howtos), and we can continue encouraging new and existing developers to enter the project, and subsequently ‘up-level’ them. But ultimately, we already provide the one truly useful ingredient: the source code, available under the MPL2 opensource license.

The fact that for some areas of the code, deep knowledge sits with just a few people, is therefore not due to missing knowledge-transfer - but instead is the natural selection of only very few people having had the time, dedication (and yes, sometimes the decades of a paid job), to develop this deep understanding.

In conclusion: if the project wants more people with deep knowledge in the code base - we need to get more people to work on the code, perhaps fulltime, and then wait for many, many years. We cannot document ourselves out of that fact.

Cheers, Thorsten

2 Likes
#6

just to make it less abstract for new comers (like me) :

  • how many people are we talking about ? not just TDF ? Team — The Document Foundation
  • couldn’t we at least start with a basic list of people x expertise area ? could it be just derived from git commits ?

not to start a debate, but just to share a personal impression, this sounds very optimistic :thinking:
(like how many “volunteers” were u talking about ?)

#7

A quick update:
We have scheduled the call for this week on Wednesday to discuss the first steps. We will report back here as soon as we have news.
Cheers - Sophie

#8

Thanks a lot for this! I am glad to see this making progress indeed. :slight_smile:

#9

Hi Regina,

ODF conformance is crucial for our project, so many thanks for the detailed task list, also for working on the extension of the ODF standard and its LibreOffice support!

As a code contributor, I propose a 2-week and a 4-week cost estimations for the basic and enhanced transition tasks. Also I suggest to add testing ODF 1.4 compliance of Microsoft Office. I can imagine, that it’s not complete, yet, despite the press release.

As a board member, I think, we will need a Cleanup & further improve ODF conformance budget item again, especially if nobody touches this area of code in the near future. Your detailed list is huge help not only for the potential bidders, but also for volunteers. It’s relatively easy to implement the transition based on the history of the code, so I would encourage everyone not to wait for tendering (which will restart much more slowly than I had hoped). This kind of code contribution can also help new LibreOffice programmers gain experience to take on larger tasks.

Unfortunately, not only ODF 1.4, but supporting ODF 1.3 or older is still our important task. Thanks to NLnet Foundation/EU, I’ve just added the ODF 1.0 fo:hyphenation-keep support, also fo:hyphenate text span support to version 24.8. The former (132599 – Option to stop words hyphenating across pages) solved also an interoperability issue with MSO 2013. The latter (106733 – Implement ODF attribute fo:hyphenate to exclude a portion of text from hyphenation) was reported by you 7 years ago, thanks for that, too! (Note: I’ve extended fo:hyphenation-keep with a loext:hyphenation-keep-type, based on the features of W3C standards, and I plan to propose them for ODF standardization. If you don’t mind, I’m going to ask for your help with this. More details: ReleaseNotes/24.8 - The Document Foundation Wiki)

Best regards,
László

1 Like
#10

Does your “basic” mean the task " Basic transition tasks"? If yes, please add your estimate on the Wiki page.
What task(s) do you mean by “enhanced transition tasks”? Perhaps here too, please add the estimates to the Wiki page.

#11

Hi Regina,

I’ve added the cost estimations, also to the other ODF 1.4 tasks. These are approximate values, because even the most experienced can run into problems that they either solve in days or give up on out of necessity related to the dead line. (For example, last time it took me days to write a test about resizing the height of a table row with a mouse, such was the unfamiliarity and complexity of the code base, but in return LibreOffice can’t go wrong without being noticed in this area, and helped my other improvements here (see Diff - 4da6f52f5fd164082fd42fc58fc7d31da567c924^! - core - Gitiles for the first fix with the unit test).

Best regards,
László

P.S. I’ve added also the task to check the interoperability. Likely not only Office 365, but Google Docs, old MSO versions etc. are important here (especially, if the extended format breaks interoperability). I understand there were fundamental problems with ODF support in MSO (that is why OOXML was chosen by NISZ), but at least the new ODF features are worth checking.

1 Like
#12

ODF 1.4 Kick-off meeting

Presents: Regina, Heiko, Johnathan, Michael W, Christian, Ilmari, Olivier, Xisco, Hossein.

Objective: Evaluate activities, effort and cost estimate for the tender (RH)

General

• Items in wiki page are well defined (IL)
• Good to have someone from the Board in this call (IL)
• Mixed but different discussions (Staff work and Tender), need to separate (MW)

Priority

• Transition should be done immediately (RH)

  • Seems easy, only XML schema (HT)
  • Not available for implementation, have other activities (RH)
  • If nobody available then → tender ?(HT)
    • Budget is to be finalized, no time this year (IL)
    • Cost is 1K/day (HT) ~ 100K year (IL)
  • Put ODF validate data in separate task (RH)
    • ODF Validator is easy task (CL). See with M Stahl and Svante (RH)

Microsoft & OASYS

• MS will change soon to ODF 1.4 (RH)
• There are examples of ODF 1.4 from MS? (CL)
• OASYS is providing a time to comments, deadlines in 8 weeks. Confirm with Chair (RH)
• IS there an inside version from MS (CL)?
• you can get it from MSO dev program (RH)
• Has anyone access to MSO inside? (CL)? nobody.
• TDF has access to MSO dev, but nobody is using it (CL)
• Cost is 70 EUR/year (RH)
• Join the Microsoft 365 Insider Program (HN)

Basic transition

• Basic transition task needs to be solved. Need new schema and then do other things.(RH)
• We already have 1.3 extended, shorten work amount (CL)
• Wiki has set of implemented features, in pvt namespace (RH)
• Team can handle implementation of already available, not a huge effort (CL)
• Need to change export filters, switch for ODF versions, (RH), same for import filter (RH)
• Similar change done for 1.3 (XF), are there refs to commits for that time? (XF)
• Need to find commits in git
• M Stahl did most of the work at that time (RH)
• Can some task be easy-hacks (HT)? Announce in Blog, code pointers from Hossein)
• Is easy-hack, can be thru Outreachy (IL)

  • Will take mentor time anyway (MW)
  • Easy hacks: remove table from Spec, easy for students (RH)
  • Easy hack – example: Easter Sunday, code pointers, but basic transition task must be done (RH)

• Recommend to open BZ tickets on tasks.(HT)

  • Please open tickets to evaluate what can be done (HN).

• Example for updating the schema: https://gerrit.libreoffice.org/c/core/+/116734 (CL)
• Example tickets in BZ: 162118 – (ODF-1.4_support) [META] Preparing LibreOffice for ODF 1.4 (HT)
• Budget Tenders for budget 2024 - The Document Foundation Wiki (JC)

Action Items

• AI: Acknowledges with ED on the basic transition wrt to the staff, with CL, XF, HN, MW
• AI: work on basic transition XF, after vacations
• AI: See what MS is announcing for ODF1.4 (open).
• AI: Look into simple task that can be done as E-hack (all), and set difficulty level (HN), Outreachy (IL, JC)
• AI: Ask board for providing tender support (All)

Project meeting adjourned and continues in Team meeting.

–
Olivier Hallot

#13

I have added 2 commit links for ODF 1.2 → ODF 1.3 to the Wiki page.

1 Like
#14

META bug tdf#162118 could be used to track the subtasks and the progress.

1 Like
#15

Florian wrote:

Indeed, this is something that becomes more and more important. We
need to have knowledge publicly shared, to enable many more people
to work on the code, grow their skills and be involved in tenders
and projects.

This brings back some really bad memories of non-coders thinking that in some way developers are deliberately hiding knowledge around the code - which is/was beyond offensive. I would just like to lay out some of the many ways which ecosystem companies who have won tenders previously have gone about doing the exact opposite of that and making it easier to get into the code.

As far as I know, all of this has been done for free, based on good will, love for the project, and a small amount of self-interest. Of course if TDF wants to pay tender applicants to do all of this, that’s fine too - but expensive. If anyone seriously believes that something was missing here in the past - I’d love to hear it; I expect quite the opposite that it would be really hard for TDF to fund the associated massive ongoing effort to improve the code and its understand-ability

Lets start with:

writing good code:

  1. ideally an implementation is maintainable, has sensible function names, variable names, code comments and so on.
  2. ideally it has a sensible structure, and design such that it leaves the rest of the code better off than when the task was started - rather than piling up hacks.
  3. ideally it has unit tests that capture the subtleties of nasty corner cases, and protect us all from future regression.
  4. none of that is capture-able in a ‘tender’ - much of it is taste based, but it gets done. This is by far the most useful way of transferring understanding: the code is the authoritative documentation of itself as Thorsten points out.

mentoring and training

  1. tender winners have typically had the authors of features travel at the ecosystem’s expense to LibreOffice conferences and not only give a talk on the technical details, but also
  2. answer questions in-person and/or encourage others to get involved with the code, and
  3. also sponsoring the conferences to allow more people to attend and learn, as well as
  4. ongoing free maintenance of features, reviewing patches, filing easy-hacks, getting other people trained & involved and so on
  5. did I mention constructing much of the very-beginners material / slideware to help people understand and get involved with the code-base, its basic structure and classes
  6. the ‘Google’ Summer of Code projects have also been substantially mentored by the ecoystem - this is a very signifant cost in paid engineering time that normally far outweighs Google’s financial stipend.

Getting more people in

  1. clearly ecosystem companies have hired a large number of people to work on LibreOffice - the vast majority of paid development staff; but they have also evangelized LibreOffice technology to many other companies to grow the ecosystem
  2. vitally - they have also invested heavily alongside the community in simplifying, and re-factoring the code in many ways:
  3. to make it easier to read and understand and
  4. with far simpler lifecycle semantics to make it easier to get changes right & avoids bugs for newbies and
  5. increased code sharing & reduced LOC count, remove obsolete pieces so there is less mess to read and
  6. invested to move to far more modern C++ constructs and libraries - making it much easier for experienced C++ developers to get involved.

For sure there will be very specialized knowledge were this is much
harder to achieve than for other topics, but I suggest we focus on
that in the future - that will be beneficial for the community, the
ecosystem and the users.

I keep trying to encourage non-developers to become developers; so perhaps it could be useful to have non-developers determine what is beneficial for that. Ultimately all are welcome to code on LibreOffice, there is mentoring and help available to fit people into the team, but in my experience - the main gap here is focus, determination, discipline/technique and many hundreds of hours of hard work to understand and solve a problem.

Previous tenders IIRC had some requirement to blog about the work as it goes along - which seems reasonable in principle, but the tip of the iceberg.

If TDF wants to make a huge difference in learn-ability, it could put aside a very substantial budget to have a book written on the internal core structure of LibreOffice [NB. not just printing out the fairly unhelpful UNO APIs - but actually documenting the code structure and how things work]. Chris Sherlock did some work in this direction - but the process of building that tends to highlight all sorts of annoyances with the API and people get distracted into cleaning it up. So we’d
want more high-level and algorithmic documentation I think. But that is a big project and would need to condense lots of existing ecosystem talks and slides into a text then extend it I think.

My 2 cents,

Michael.

#16

I think we should start looking at and managing development-related tenders differently from the past, as otherwise we will never get to a positive solution, one which allows TDF to tender with confidence and all other parties to bid with confidence, without being blocked by a Conflict of Interest (which is impossible to get rid of completely, knowing the history and the reality of our project).

MY TAKE: the LibreOffice project, although rather large in comparison with other open source projects, is definitely small in terms of potential reach (because the office suite is a commodity, and the focus on commodities is shrinking in front of the challenges created to software development by the cloud), and the software development skills requested are very specific because they are based on C++.

DISCLAIMER: I am not a software developer, and my technical skills are close to zero, so I might be wrong when I make statements about development. For sure, I am not wrong when I talk about the market and the related challenges.

I think that LibreOffice developers in general have done a great job of mentoring new developers, and transferring their knowledge (keeping in mind that knowledge transfer is a function of the receiver, as I can have the best mentor but if I am limited in my understanding I will always miss some bits). In 2010 we forked with 20 developers, and in less than five years we attracted over 1,000 new developers (I was keeping track of the numbers). Most of the mentoring developers have always been employed by ecosystem companies. These are the facts.

The influence of the senior developers, either employed by ecosystem companies or not, goes beyond the boundaries of the ESC or of the development mailing list, because they are seen as a reference by other developers, including the senior ones who are not focused on a specific task. We all know their names and their competencies.

In addition to this, a large group of people has been contributing to the project for 20+ years, and this means that they know each other for such a long time that this has in some cases built a level of trust which goes beyond the reasonable (but is totally understandable for human beings).

On the other hand, the law is asking for a tender management policy (or procurement policy) which is different from the past, because the process used in the past was subject to conflicts of interests as the same subjects were involved in writing the tender and providing an estimate of costs, but then were also bidding for the same tender (although they were excluded from the decision about the assignment of the tender). Unfortunately, because of all the previous considerations, even if excluded from the final decision they could have a - totally indirect - influence on it. This is also a fact, if looked at from the authorities point of view.

All that said, I think that the only reasonable solution is to have the tenders developed at TDF with the contribution of all the people - either developers or non developers - who are competent in that specific area, independently of the fact that they (or the company they work for) will be bidding or not, because the final decision about the tender will be taken outside TDF (by a group of competent people who will get a token for their work, and there are thousands of options to build this group).

At the same time, TDF could also invest in building a book on the internal core structure of LibreOffice, as this IMHO - even if I am not a developer - would be welcomed by many developers, and would be a very good resource for the organizations deploying or migrating to LibreOffice. It could also help some enterprise which are deploying LibreOffice without support to understand that what they are doing is wrong, because they might be running into unexpected issues.

One last thing. I am exhausted of repeating that only by working together as a single organism (not organization) without repeating every time the merits or demerits, because after four years of hour-long emails we all know them, we will be able to find solutions which are acceptable for all the parties and compliant with the law.

We need suggestions, ideas and proposals backed by good and positive intentions. Otherwise, we will continue to look at the past, and in this way we will lose most of the future opportunities.

#17

Hi Italo,

Thanks for your realistic and friendly assessment =)

Sounds sensible to me - that is the Board’s perogative. While the sums TDF can tender in a sustained way are reasonably small compared with what the ecosystem is investing, there is a big backlog of funds for good things I expect. And of course TDF is able to spend on strategic things that can’t be commercially justified which is great:

I agree - we should take advantage of each other’s strengths; in this case TDF’s funding can be applied to longer term strategic things, that help improve the code-base to make it more accessible, and grow the whole project for the long term.

Last I looked there was a rather large, and reasonably sensible chunk of text that came out of some extremely long process in the previous board; no idea what happened to that in the end; it didn’t seem far from something sensible to me.

The problem is the very significant cost in terms of skills for creating and then tech-editing what is a once-in-a-lifetime one-off task. I did some of this work for GNOME, and it was extremely time consuming - but of course, once done - maintenance is normally rather easy. I would be very happy to draw up an outline spec. for this though I don’t believe Collabora would have time to apply to any tender for it; I would suggest budgeting for hundreds of days of work though, and seeing how it is ranked.

Anyhow - all power to those working to get a sensible tender process finalized; what are the next steps there?

All the best,

Michael.

#18

Hi,
A quick update on that, Carlo came back to me and will prepare a new version of the policy as soon as possible so the board can have a discussion on a consolidated text.
Cheers - Sophie

2 Likes
#19

This is great news! Do we have any indication on how much TDF will be able (and is willing) to spend on tenders this year (2024) and next (2025)? Just a very rough idea?

Gerald

PS: Asking for a friend who is keenly interested in seeing
https://wiki.documentfoundation.org/Development/Budget2022#3D_rotation_of_Impress_shapes
https://wiki.documentfoundation.org/Development/Budget2023#Slideshow:_rendering_content_on_top_of_videos
implemented among others. :slight_smile:

#20

With the procurement policy still in the works, I would expect the bigger part of things to take place next year only, for when a new budget is to be made and we hopefully have the procurement policy in place. What concerns this year, the budget from Update on budget 2024 gives some indication - although it has not been decided yet what will be tendered and what not, from what I can tell. Some items could be tendered, I guess, but there’s no decision yet I am aware of.

Impressum (Legal Info) | Datenschutzerklärung (Privacy Policy)   Content on this site is licensed under a Creative Commons Attribution Share Alike 3.0 license.