QA manual test Litmus session on 3.3.3Rc

projects
#1

Hi all,

*please, follow up on the projects list, thanks in advance*

I would like to try to organize a first session of manual tests - release scenarios - on Litmus [1]. Mostly to see how it goes and fine tune what needs to be done to go further. We need to push our QA process further and log what is done in the language community.

The first version will 3.3.3 RC. Yifan (thanks a lot! ) did upload a session for 3.4, but I guess we were a bit distracted by what happened to the OOo code. But we can use this basis for the next session.

Tests already exists in US - DE - FR, it's a first set that needs to be improved and made available for all languages.
So please, join the projects@ list and let us organize a nice and fun QA process for our future versions.
For testing, you don't need technical skills, only about 3/4 hours available (for beginners, less if you have already participate) per tested version and some curiosity for the new features and the bug you will find.
For translation, understanding of English will be enough, the tests and the process are simple and most of the time written by non English speakers.
If you run VCLTesTool, we will also try to centralize the feedback, so don't hesitate to jump in either.

Process and info on Litmus are available on the wiki [2], but don't hesitate to come on the list, I'll help and answer your questions from there, also on irc #libreoffice

[1] https://tcm.documentfoundation.org/
[2] http://wiki.documentfoundation.org/Litmus

Thanks in advance for your help on the organization and your participation.
Kind regards
Sophie

#2

Hi Sophie,

Hi all,

*please, follow up on the projects list, thanks in advance*

I would like to try to organize a first session of manual tests -
release scenarios - on Litmus [1]. Mostly to see how it goes and fine
tune what needs to be done to go further. We need to push our QA process
further and log what is done in the language community.

The first version will 3.3.3 RC. Yifan (thanks a lot! ) did upload a
session for 3.4, but I guess we were a bit distracted by what happened
to the OOo code. But we can use this basis for the next session.

Should a test run 3.3.3 already be visable?

Tests already exists in US - DE - FR, it's a first set that needs to be
improved and made available for all languages.

Would a additional testgroup with those bugs that occured as blockers
for past releases make sense at this early stage of the process? They
should be easily found in the bugzilla.

Another question, would tests against the daily builds of the 3.4 series
go into the libreoffice 3.4 test run or should there just be tests
against beta and/or RC versions?

Regards,
Michael

#3

Hi Michael,

Hi Sophie,

Hi all,

*please, follow up on the projects list, thanks in advance*

I would like to try to organize a first session of manual tests -
release scenarios - on Litmus [1]. Mostly to see how it goes and fine
tune what needs to be done to go further. We need to push our QA process
further and log what is done in the language community.

The first version will 3.3.3 RC. Yifan (thanks a lot! ) did upload a
session for 3.4, but I guess we were a bit distracted by what happened
to the OOo code. But we can use this basis for the next session.

Should a test run 3.3.3 already be visable?

As soon as I, or another admin, have create them :slight_smile: but surely today

Tests already exists in US - DE - FR, it's a first set that needs to be
improved and made available for all languages.

Would a additional testgroup with those bugs that occured as blockers
for past releases make sense at this early stage of the process? They
should be easily found in the bugzilla.

Oh, yes, that's a good idea.

Another question, would tests against the daily builds of the 3.4 series
go into the libreoffice 3.4 test run or should there just be tests
against beta and/or RC versions?

all releases should be tested and have the results log in Litmus, but not everybody will be able to follow the rate. There is a trunk category that Yifan (I think) has open, may be we can put branches here?

Kind regards
Sophie

#4

Hi Sophie, QAlies,

Hi all,

*please, follow up on the projects list, thanks in advance*

I would like to try to organize a first session of manual tests -
release scenarios - on Litmus [1]. Mostly to see how it goes and fine
tune what needs to be done to go further. We need to push our QA process
further and log what is done in the language community.

The first version will 3.3.3 RC. Yifan (thanks a lot! ) did upload a
session for 3.4, but I guess we were a bit distracted by what happened
to the OOo code. But we can use this basis for the next session.

Tests already exists in US - DE - FR, it's a first set that needs to be
improved and made available for all languages.
So please, join the projects@ list and let us organize a nice and fun QA
process for our future versions.
For testing, you don't need technical skills, only about 3/4 hours
available (for beginners, less if you have already participate) per
tested version and some curiosity for the new features and the bug you
will find.
For translation, understanding of English will be enough, the tests and
the process are simple and most of the time written by non English
speakers.
If you run VCLTesTool, we will also try to centralize the feedback, so
don't hesitate to jump in either.

Process and info on Litmus are available on the wiki [2], but don't
hesitate to come on the list, I'll help and answer your questions from
there, also on irc #libreoffice

[1] https://tcm.documentfoundation.org/
[2] http://wiki.documentfoundation.org/Litmus

Thanks in advance for your help on the organization and your participation.

Some things Christian K. mentioned on the German-ml [1]:

1. Should we re-place Litmus on the Wiki-menu?

2. The German tests are different from the English. Is there any problem with it?

3. Obviously the testcases can only be changed by admins although there is an option in the menu. If only admins can do this we should give some more people admin rights especially mindst 2 persons per l10n. And it should be unlighted for the others.

4. Who are the admins now?

[1] http://go.mail-archive.com/vj-3KdyO9dA35DYuDDwzUFsCD6k= (German)

#5

Hi,

Some things Christian K. mentioned on the German-ml [1]:

[...]

2. The German tests are different from the English. Is there any problem
with it?

I do see some advantages in having tests regarding localization in all
the languages.

But I am not sure if there is any risk in the build process, that bugs
in the logic of the program occur just in a subset of languages.
If not the language does not matter and there is no need to translate
und multiply the work with translating all the tests in more than 100
languages. Litmus becomes unusable and getting statistics gets much
harder as you have to aggregate over all the languages.
Also members that do not speak every language will have no chance to get
information out of more than 80% of the tests and in the end when tests
fail you have to use the english only bgzilla.

But I guess the lang-projects insist on having translated tests, so
whatever.

Having even differing tests in all the languages just make the mess so
much bigger.
So even if we want translated tests please use a top down english ->
rest approach.

Regards,
Michael

#6

Hi Klaus-Jürgen,

Hi Sophie, QAlies,

Hi all,

*please, follow up on the projects list, thanks in advance*

I would like to try to organize a first session of manual tests -
release scenarios - on Litmus [1]. Mostly to see how it goes and fine
tune what needs to be done to go further. We need to push our QA process
further and log what is done in the language community.

The first version will 3.3.3 RC. Yifan (thanks a lot! ) did upload a
session for 3.4, but I guess we were a bit distracted by what happened
to the OOo code. But we can use this basis for the next session.

Tests already exists in US - DE - FR, it's a first set that needs to be
improved and made available for all languages.
So please, join the projects@ list and let us organize a nice and fun QA
process for our future versions.
For testing, you don't need technical skills, only about 3/4 hours
available (for beginners, less if you have already participate) per
tested version and some curiosity for the new features and the bug you
will find.
For translation, understanding of English will be enough, the tests and
the process are simple and most of the time written by non English
speakers.
If you run VCLTesTool, we will also try to centralize the feedback, so
don't hesitate to jump in either.

Process and info on Litmus are available on the wiki [2], but don't
hesitate to come on the list, I'll help and answer your questions from
there, also on irc #libreoffice

[1] https://tcm.documentfoundation.org/
[2] http://wiki.documentfoundation.org/Litmus

Thanks in advance for your help on the organization and your
participation.

Some things Christian K. mentioned on the German-ml [1]:

1. Should we re-place Litmus on the Wiki-menu?

Yes, why not

2. The German tests are different from the English. Is there any problem
with it?

Yes, because, the English one should be the master set. Frequently, when we lack of one tester (you know the OOo rule of 2 testers/plateform to validate it) I check the other language to help validate the version. If there is an issue with one test, it's easier to see if it's language dependent or not.
And we want to offer the same level of quality for all the versions, so really we need to ensure the English is the master.

3. Obviously the testcases can only be changed by admins although there
is an option in the menu. If only admins can do this we should give some
more people admin rights especially mindst 2 persons per l10n. And it
should be unlighted for the others.

This will be done on demand. I can also upload tests for the other language who don't have enough resources or don't want to deal with Litmus

4. Who are the admins now?

Volker Heggerman, Jean-Baptiste Faure and me.

Kind regards
Sophie

#7

Hi Michael,

Hi,

Some things Christian K. mentioned on the German-ml [1]:

[...]

2. The German tests are different from the English. Is there any problem
with it?

I do see some advantages in having tests regarding localization in all
the languages.

But I am not sure if there is any risk in the build process, that bugs
in the logic of the program occur just in a subset of languages.

It has already happened by the past, so it's always good to be able to compare.

If not the language does not matter and there is no need to translate
und multiply the work with translating all the tests in more than 100
languages.

It's important to offer an access to everybody. Manual testing requires very low skills, even a beginner can participate. So over the goal of testing, this is also community/team building

Litmus becomes unusable and getting statistics gets much

harder as you have to aggregate over all the languages.

I'm following the next generation Mozilla is working on. I think we won't have more than 15/20 languages at the beginning, so that won't hurt Litmus so much. But we will be able to improve our process and tooling.

Also members that do not speak every language will have no chance to get
information out of more than 80% of the tests and in the end when tests
fail you have to use the english only bgzilla.

Yes, but not everybody need to fill the bugs, a small team of 2/3 members is enough, will testing on all the release is really important.

But I guess the lang-projects insist on having translated tests, so
whatever.

Having even differing tests in all the languages just make the mess so
much bigger.
So even if we want translated tests please use a top down english ->
rest approach.

Yes, that is important. And as I said above, I don't care the mess (even if it has to eat my nights) if every body is able to participate and we build a larger community :slight_smile:

Kind regards
Sophie

#8

Hi Spohie,

I deleted those parts where we agree.

> If not the language does not matter and there is no need to translate
> und multiply the work with translating all the tests in more than 100
> languages.

It's important to offer an access to everybody.

Are there any rough estimations how many people are affected that can
not contribute in this technical process because of the need to read
basic English texts? As I said, bugzilla, easy hacks, LibOCon, this
project list, ...
When I see what is discussed on the german discuss list (reworking of
the download page) where the non-german speaking community is not aware
of and can not take part in the discussion, and probably similar things
going on on other language mailing lists, I believe we already lose so
much by not using the same language that connects us way to often.

> Litmus becomes unusable and getting statistics gets much
> harder as you have to aggregate over all the languages.

I'm following the next generation Mozilla is working on. I think we
won't have more than 15/20 languages at the beginning, so that won't
hurt Litmus so much. But we will be able to improve our process and tooling.

That would easily result in more than 1000 testcases that have some
obscure tagging and categorizing in the testcase summary. If I look into
the mozilla litmus I can just guess why they do not use localized
testcases this way. I believe this is a huge trade-off and not worth it.

Litmus is already complicated enough to use, working around the workflow
to get some localization feature into it that is not cleanly integrated
into the application just makes it worse. I fear that through making it
look more complex we will scare more people away than we reach through
the localizations that first have to written. And by the way all those
users of the languages that are not in your estimated 15/20 languages do
not even have the opportunity.

But I agree to disagree here.

Regards,
Michael

#9

Hi Sophie, *,

Sophie Gautier wrote (05-06-11 22:36)

I would like to try to organize a first session of manual tests -
release scenarios - on Litmus [1]. Mostly to see how it goes and fine
tune what needs to be done to go further. We need to push our QA process
further and log what is done in the language community.
[...]

Thanks a lot for this :slight_smile:
Just created an account and is gives me this welcome
   " Thank you for your contribution to Mozilla QA. " :wink:

I'll translate the wiki-info and poke some people on the Dutch language lists to take a look too.
If we have any questions, we know where to find you :slight_smile:

Cheers,

#10

Hi Cor,

Hi Sophie, *,

Sophie Gautier wrote (05-06-11 22:36)

I would like to try to organize a first session of manual tests -
release scenarios - on Litmus [1]. Mostly to see how it goes and fine
tune what needs to be done to go further. We need to push our QA process
further and log what is done in the language community.
[...]

Thanks a lot for this :slight_smile:

Thanks a lot for jumping in :slight_smile:

Just created an account and is gives me this welcome
" Thank you for your contribution to Mozilla QA. " :wink:

Oh, yes, we are very keen in supporting Mozilla effort, you know :wink:
I'll see if we can correct this, thanks for the feedback

I'll translate the wiki-info and poke some people on the Dutch language
lists to take a look too.
If we have any questions, we know where to find you :slight_smile:

Yes, don't hesitate, and welcome to the Dutch speaking community!

Kind regards
Sophie

#11

Hi Michael,

Hi Spohie,

I deleted those parts where we agree.

If not the language does not matter and there is no need to translate
und multiply the work with translating all the tests in more than 100
languages.

It's important to offer an access to everybody.

Are there any rough estimations how many people are affected that can
not contribute in this technical process because of the need to read
basic English texts? As I said, bugzilla, easy hacks, LibOCon, this
project list, ...

Look at the subscribers on the language mailing lists, quite numerous, this is why we have native language communities for years now, this is their only chance to participate to our project.

When I see what is discussed on the german discuss list (reworking of
the download page) where the non-german speaking community is not aware
of and can not take part in the discussion, and probably similar things
going on on other language mailing lists, I believe we already lose so
much by not using the same language that connects us way to often.

Ha, are you an Esperanto supporter? :wink: I agree with you, but there is nothing else we can do than coping with it the best we can. Offering areas where every body can use their skills, whatever they are and whatever the language he uses, it may requires more energy for some of us, but I believe, it's a great benefit for the project.

Litmus becomes unusable and getting statistics gets much
harder as you have to aggregate over all the languages.

I'm following the next generation Mozilla is working on. I think we
won't have more than 15/20 languages at the beginning, so that won't
hurt Litmus so much. But we will be able to improve our process and tooling.

That would easily result in more than 1000 testcases that have some
obscure tagging and categorizing in the testcase summary. If I look into
the mozilla litmus I can just guess why they do not use localized
testcases this way. I believe this is a huge trade-off and not worth it.

Do you have another suggestion, or another tool in mind. I'm open to everything, I didn't find something better at the moment, but I may not be aware of some process/tools more adapted.

Litmus is already complicated enough to use, working around the workflow
to get some localization feature into it that is not cleanly integrated
into the application just makes it worse. I fear that through making it
look more complex we will scare more people away than we reach through
the localizations that first have to written. And by the way all those
users of the languages that are not in your estimated 15/20 languages do
not even have the opportunity.

But I agree to disagree here.

:slight_smile: As said above, I'm open to any suggestion either on the process or the tool. We are at the beginning so it would be easy to move now, more than later. We have:
- a large community,
- about 100 languages,
- dev versions, betas, RCs,
- several platforms
- manual TCS and VCLTestTool automated tests

All this have to be combined in a fine, easy to use, useful and robust tool, a bit challenging I guess :slight_smile: Sun did a great job on the last TCM release for QUASTe, but unfortunately, it has never been opensourced...

Kind regards
Sophie

Impressum (Legal Info) | Datenschutzerklärung (Privacy Policy)   Content on this site is licensed under a Creative Commons Attribution Share Alike 3.0 license.