On updates and user experiences and so forth

So, the debate about update policies and user experience etc is spilling onto the Planet, with some constructive posts from Máirín and Jon so far. Thought I'd throw in a few cents.

In the end, it doesn't matter hugely to me how this gets settled - I work QA, and as I figure it, our job is to do the best testing job we can in the context of the overall aims and development method of the project. You sometimes see cases where QA people want to dictate the way development happens on the basis of being able to test things properly, but I think that's ass-backward; whoever is leading the project should set the goals and development methods, and QA should work within those. If whoever's in charge is unhappy with the resulting product's quality they might want to tweak the process, but it's not really QA's job to dictate that. So I'm fine with working with whatever process gets decided in the end.

From an entirely personal point of view, though, I have to say I kinda wind up on a slightly different side of the question to Máirín and Jon. Máirín has Caroline Casual User; Jon talks about what Windows and Mac OS do, what "users" want, and stuff like "can feel quite confident doing an update as a matter of best practice, even ten minutes before a big presentation, or even during the presentation if I want to".

I'm kind of sympathetic towards the point of view that that's a perfectly sound goal for an operating system, but it may not necessarily be the best goal for the Fedora operating system to pick. Just because that's the target of Windows and OS X doesn't mean it has to be ours.

For a start, there's already plenty of Linux distributions besides Fedora with those goals. To put it bluntly, there's Ubuntu, and Ubuntu's doing a fairly decent job of being Ubuntu. I've argued before that it's somewhat dangerous for the overall ecosystem for Ubuntu to drive out all competitors, and to an extent I stand by that, but OTOH, I'm not entirely sure that being Ubuntu's token competition is the best thing Fedora can be. For a start, Fedora isn't set up for it. Exactly this kind of debate makes the advantages of Ubuntu's Benevolent Dictator model apparent; this just wouldn't really happen in Ubuntu-land, or if it did, it'd be in a conference room with Mark and ten other people and it'd be over in an hour. The Fedora project is built in a different way; being run by a chain of partly-elected committees is great for openness and transparency and defending community values, but it does lose the benefits of focus and swiftness that come from having one person in charge and defining the project's scope and vision.

I like the idea that's mostly being proposed by the 'radical' side of the debate that the best and most efficient way to use the resources of the Fedora project going forwards is to be something different from what Windows and OS X and Ubuntu are. One of the advantages of the F/OSS world is that we're supposed to be better at co-operation than competition. I think it would be kind of cool to have a world where everyone knows that Fedora is the distribution with the latest bleeding-edge stuff and Ubuntu is what you install on grandpa's computer, or yours if you want to trade off a quiet life for not getting the latest everything all the time.

In fact I could be quite happy if we revised Fedora's process completely. I can see a future where we aim to be a rolling distribution, and put out a point release only when we have to; when I asked people within Fedora a while back why point releases still exist, the only really valid answer was more or less 'because sometimes changes happen that we can't handle with an in-place update'. That's fine, but in that case, there's no real reason besides PR to schedule releases every six months; why not just do a release when some change means we have to do one? When such a change comes along we put out a set of images and give people six months to reinstall or upgrade, pushing security fixes for the previous codebase during that period, and then just declare it dead and say everyone needs to be on the new code now? Most of the objections to this kind of thing are about providing stable platforms and dependable updates and yadda yadda, but I already said, there's no reason Fedora has to be that project. In a lot of ways I think Fedora could be a much more interesting and useful project in the long term if it wasn't.

tl;dr summary: let's let Ubuntu be Ubuntu, and let's us be something different, and that should be the most efficient way for the whole F/OSS / Linux world to use its resources.

Comments

Máirin Duffy wrote on 2010-09-03 02:32:
Hi Adam! I really screwed up. I made Caroline a composite of Fedora's intended target user and the general public outside of that. That was a bad mistake on my part. I tried to correct it with a follow-up post: http://mairin.wordpress.com/2010/09/02/sweet-caroline/ (but it seems I screwed that one up too, I updated the graphic in attempts to try to help) Anyhow, I just noticed you wrote: "it may not necessarily be the best goal for the Fedora operating system to pick. Just because that’s the target of Windows and OS X doesn’t mean it has to be ours." I don't think the actual Caroline I meant to write about (see the 'yes' section) is necessarily as broad a target as OS X and Windows. I am curious if you think the revised Caroline (again, revised to meet at least the letter if not the spirit of The Board's target user definition) is a better target for Fedora. "Exactly this kind of debate makes the advantages of Ubuntu’s Benevolent Dictator model apparent; this just wouldn’t really happen in Ubuntu-land, or if it did, it’d be in a conference room with Mark and ten other people and it’d be over in an hour." Thank you for that. The next time someone has the nerve to tell me I should give up on Fedora and move to Ubuntu I will link them to this blog post for that. While I do disagree with your final recommendation, thanks for the thoughtful post and the different perspective!
[...] On updates and user experiences and so forth In fact I could be quite happy if we revised Fedora’s process completely. I can see a future where we aim to be a rolling distribution, and put out a point release only when we *have* to; when I asked people within Fedora a while back why point releases still exist, the only really valid answer was more or less ‘because sometimes changes happen that we can’t handle with an in-place update’. That’s fine, but in that case, there’s no real reason besides PR to schedule releases every six months; why not just do a release when some change means we *have* to do one? When such a change comes along we put out a set of images and give people six months to reinstall or upgrade, pushing security fixes for the previous codebase during that period, and then just declare it dead and say everyone needs to be on the new code now? Most of the objections to this kind of thing are about providing stable platforms and dependable updates and yadda yadda, but I already said, there’s no reason Fedora has to be that project. In a lot of ways I think Fedora could be a much more interesting and useful project in the long term if it wasn’t. [...]
Aurélien wrote on 2010-09-03 06:33:
Hell yeah. In the beginning of Fedora, there was "Core" with fixed releases and "Extras" with a rolling-release model. This was great: stability for the base OS, freshness for the applications. Of course the border is hard to set, but it's a minor trade-off IMHO. I'd love Fedora to be going the route of the rolling release. This will upset Carolines though, and our designers who would probably rather design for the general public than for geeks. I guess I'm too Pamela-ish, but it's true that not all distribs should be competing with Windows & MacOS.
Alex Hudson wrote on 2010-09-03 07:03:
I couldn't disagree more, to be honest - and in a sense, I think you're defining the problem in terms of itself. Yes, if it's not the role for QA to influence development, then someone higher up the chain needs to take that role. That doesn't mean that the Ubuntu SABDFL method is the only way to go: with a community of developers, QA people are as important as anyone else and it is precisely their role IMHO to say how things should change in order for them to be better able to do their job, and it's for the community as a whole to attempt to balance those forces with everything else happening. I also don't really agree with "Let's us be something different", because we're already "different" - the target Fedora user base is people who are already relatively computer-friendly. Ubuntu are aiming at everyone, and although in practice the people they pick up are generally of a similar type, they definitely have a different aim. I haven't seen anyone articulate a better vision for Fedora, and if we were to say that people should need to be ever-more technical in order to be users then we probably ought to pack up and go home, tbh.
adamw wrote on 2010-09-03 12:49:
Alex: where I think you're not thinking things through sufficiently is that you don't break down the concept of QA's 'job'. The job of the QA team is going to be vastly different depending on exactly the questions that are being asked: what kind of product is it we're trying to build, and who is it we're trying to build it for? There's no such thing as a one-size-fits-all QA 'job', and it shouldn't be the QA team's job/responsibility/power to define what that job should be, as that sort of implies the QA team defining what the product they're testing ought to be, and that's just all wrong. To put it in more concrete terms, do you think Fedora QA should be exactly as strict and unbending as RHEL QA? Should it be as strict, onerous and unbending as NASA QA? There's not just one way to do QA, there's many, and the appropriate process can only be chosen based on the nature of the product being tested, which is not something the QA team should define. I agree that once the scope of the project has been defined at the appropriate level, it's QA's job to yell if the processes put in place don't allow for the appropriate level of QA for whatever that scope turns out to be, but that's a different and much more limited concept.
adamw wrote on 2010-09-03 12:58:
Alex: on the topic of Fedora's target user base - well, that's pretty much what we're arguing about, isn't it? No-one seems to agree entirely on what it is, even with the Board's statement (of which everyone seems to have their own interpretation). In particular, exactly what "Computer-friendly" or "is familiar with computers, but is not necessarily a hacker or developer" mean, and how radically to interpret them, seem to be up for debate. On the one hand I could spin those words in favour of my position - hey, if you're 'computer-friendly' and 'familiar with computers', this kind of shiny stuff is what you want, right? On the other hand, you can spin them conservatively and say, well, just because you're computer-friendly and familiar with computers doesn't mean you want your shit to get broken all the time. Someone's already cogently pointed out that in practice the phrase is almost meaningless, because anyone willing to go as far as installing Fedora must be in some sense 'computer friendly', and no-one buys it pre-loaded or gets it installed for them on a corporate system, really. So I'm not sure the existing 'official' statements on this are really enough to settle the questions.
Colin Walters wrote on 2010-09-03 13:12:
Some people appreciate Fedora because it doesn't ship proprietary software (video drivers) or tie in by default to proprietary web services, for example. Both of those have nothing to do with update frequency. And in general, I vehemently disagree that it makes strategic sense for Fedora the project (and Fedora contributors) to say that Ubuntu is the OS for normal human beings, and for us to design an OS solely for enthusiasts.
adamw wrote on 2010-09-03 13:29:
To be honest, I suspect that's quite a tiny niche, and logically speaking is better covered by a free-r Ubuntu fork than an entirely different project. i.e., if you were starting from a world where Ubuntu existed and Fedora didn't, you would not invent Fedora in order to address Ubuntu's issues with freedom.
Alex Hudson wrote on 2010-09-03 14:00:
You're still defining the question in terms of itself, though. When you talk of NASA's QA, or RHEL QA, we're still talking about teams which take external direction and do not set a standard themselves: if we think that would work in Fedora then fine, but I just don't see the community being that hierarchical, or who this external actor would actually be. In a project where QA are just another set of developers I just don't see how that model can possibly work. Of course it's not for QA alone to define what "good enough" means, but similarly the people involved in QA are the ones most interested in this issue and (I would hope) have the most to say about it. Fedora QA is fundamentally different to any type of commercial QA you would find in most respects; there is no quality metric and no final quality requirement. It's a continuous improvement process and the only people who know how much improvement is actually being seen is the people doing the QA, IMHO. No-one else is measuring this at all.
adamw wrote on 2010-09-03 14:13:
Alex: As I see it, it's for the Board to define what the Fedora Project's mission should be, and what goals Fedora the distro should aim for. It's then QA's job to design a QA process that will ensure those goals are met. It's not our job to decide what the goals are, though. Look at it this way - QA doesn't and shouldn't decide if we should be a stable or a rolling distro, it doesn't decide how long the release periods should be, it doesn't decide alone how long freeze periods should be, or what features should be accepted for a release. It doesn't decide exactly what we want the finished distribution to actually be capable of. These are all questions that are settled entirely outside of QA's domain, or only partially by QA. Yet they all have a huge impact on how the QA process should be designed and what it should aim to achieve. Practical examples - I would define the desktop release criteria completely differently, and design a completely different testing process, for a distribution aimed at hackers and enthusiasts than I would for a distribution aimed at ordinary consumers. But is it QA's call whether the distribution should be aimed at consumers or hackers? No, it isn't, and it shouldn't be. I can only speak for myself, but I don't think working QA really gives me any more authority or insight than anyone else into what 'good enough' should be. You can only answer that question relative to exactly what the product you're testing is supposed to achieve. "Fedora QA is fundamentally different to any type of commercial QA you would find in most respects; there is no quality metric and no final quality requirement." That's a large part of the problem. Right now, what you suggest should be the case is what happens - we more or less make that shit up on the fly. I don't think that's right, though. I think Fedora is very badly in need of leadership to define exactly those things, based on a broader decision about what the ultimate goals of the project ought to be. Having solid quality requirements provided by the legitimate authority of the project would certainly make it easier to define QA processes.
Alex Hudson wrote on 2010-09-03 14:46:
Ah, I think we're talking at cross-purposes. I'm not talking about the goals of the distribution, who it's for, how it should be developed, etc. At the moment, we only have the most basic of metrics. One of those is the release blocker, which is the "level of brokenness we will not tolerate". There don't appear to be any bright-line tests for most of these, but there is a kind of we-know-it-when-we-see-it. And that's about the only thing which influences release. But if we take a step back, is that the job of QA? To influence release? I don't think it has to be, or even if that is the best thing. QA has to be more than a barrier to release, otherwise it's just reacting to whatever's going on in the project and not making things better. I don't see this as a leadership problem. Right now, we have basically no metrics whatsoever. Until you can start measuring things, you cannot see improvement and you cannot set goals against them. I would struggle to see how a better vision for Fedora or better articulated "ultimate goals" would somehow lead to a set of metrics around which the QA processes could be formed.
adamw wrote on 2010-09-03 14:52:
Alex: it sounds like you're not aware of the rather extensive release criteria - https://fedoraproject.org/wiki/Fedora_Release_Criteria - and validation tests - https://fedoraproject.org/wiki/Test_Results:Current_Desktop_Test , https://fedoraproject.org/wiki/Test_Results:Current_Installation_Test
Alex Hudson wrote on 2010-09-03 15:24:
Given I specifically called out the blocker criteria, what makes you think I'm unaware of them? They're not relevant to what I was speaking about.
adamw wrote on 2010-09-03 15:40:
I'm really not sure exactly what you *are* saying, then. I don't know what a 'bright line test' is, but the whole point of the release criteria is so that blocker bugs are *not* we-know-it-when-we-see-it, but clearly defined.
Alex Hudson wrote on 2010-09-03 15:47:
Right, but the release criteria are only one potential source of blocker bugs, and whether or not a criteria has been met isn't a yes/no thing all the time anyway - you just need to look at go/no-go meetings to see the discussions about what should and shouldn't block. But anyway, practically everything I was saying was about changing things *prerelease*, not just treating QA as some kind of release roadblock. So like I said, the release criteria aren't relevant to what I was speaking about.
Máirin Duffy wrote on 2010-09-03 15:58:
@Colin agree 100%
adamw wrote on 2010-09-03 15:59:
"Right, but the release criteria are only one potential source of blocker bugs" That's not the intent. The intent is that bugs can only be blockers if they violate a release criterion, end of story. We've been quite strict this cycle in not accepting any bugs that are 'obviously' blockers but for which we don't have a criterion; instead we've been requiring new criteria to be written for these cases. "and whether or not a criteria has been met isn’t a yes/no thing all the time anyway" practically speaking it's more or less impossible to remove all the subjective judgement, the range of possible cases is just too wide. I'm really not clear on exactly what it is you're suggesting. What kind of metrics is it that you're proposing?
Alex Hudson wrote on 2010-09-03 16:40:
I'm talking about continuous metrics that reflect the actual workings of the software on a day to day basis. That could be crashers, it could be bugfix update frequency, it could be levels of bug reporting - on just a technical basis alone, measuring where work is going on in the distro is a reasonable metric for QA to watch (for example, the release criteria thing we keep getting dragged back to is basically inconsequential if large portions of the distro are updated past those versions soon after release).
adamw wrote on 2010-09-03 17:11:
"(for example, the release criteria thing we keep getting dragged back to is basically inconsequential if large portions of the distro are updated past those versions soon after release)" they're not, actually, because the image gives you a known good baseline. If an update screws things up at least we know you can reinstall from the image and dodge the offending update, or yum downgrade to the package from the release image. More metrics is probably nice, but I'm not sure it's the biggest priority right now; AutoQA probably is.
Alex Hudson wrote on 2010-09-03 17:32:
AutoQA is a continuous metric of exactly the type I talk about, except that it's pretty much a bright-line test rather than a measurement, and thus it's fail/pass rather than something you can improve on relatively. I disagree that there's much value in the release being a known-good baseline to be honest; I'd love to know how many users know how to downgrade (another metric!) but I would bet it's not particularly high. Would also be interesting to know how many people re-install cleanly rather than some kind of upgrade (metrics again...), but even if we limit things to the single case where someone has clean installed Fedora, the "known good baseline" only applies to the stuff on the LiveCD. Once you install anything outside of that, chances are good not long after release that you're not getting the release version. Again it would be another metric, but knowing how long people spend on the release versions of Fedora packages would be interesting. I would venture to suggest they actually spend more time on subsequent updates that the maintainer feels are "tried and tested stable", and the QA effort that goes into that process is likely more important.
About Caroline! : nmarques wrote on 2010-09-04 01:03:
[...] believe that Adam’s approach was very realistic, in fact Fedora has it’s own positioning and we should be faithful to it, [...]
nmarques wrote on 2010-09-04 01:16:
Adam, I would take the opportunity to ask a sincere apology due to some rude hostile behavior in the past. That said... I strongly like you point. Máirín's post and Caroline introduction made it brilliant. In a way Caroline demonstrates that the general audience is aware of 'Lunix' existence. The fact that it doesn't make part of the Fedora's targeted user base makes out of it a fine example why people like me should be ignored sometimes. Consumer satisfaction can't be an obstacle for organizational goals. That said, we don't really need to care much for Caroline, but it's nice she is aware of us and that we can satisfy her needs. Another point, is the fact that 'Cathedral and Bazaar' points into update often and release often. Then you have First foundation, and in a way you can even introduce Features foundation. That said... Fedora remains loyal to the adopted model and to the FOSS philosophy. This is good and one of the reasons why Fedora can be very attractive to people more fond of the FOSS philosophy and contribute to Fedora. Updates, I like to have a battalion of people with far more knowledge than I to decide that for me. They have worked pretty fine (one or two packages with dependency problems, they didn't installed, no harm to the system). I like the update process. It can be tweaked and given more functionality and some cool alternatives that make it more friendly and more objective in some users demands have been suggested. That's pretty cool and if they ever get deployed, I'm sure many will be even more happy. Speaking only for myself... it might be easy to point the finger to QA, but really how many of us go to your 'test days', I know you are one of the people which would love to have more enthusiasts showing up and that you battle for such to happen. I believe in Fedora QA and I think it rocks. But remember, I'm just like Caroline. Also a brilliant post yours.
Davide Repetto wrote on 2010-09-05 07:22:
WHAT??? Fedora as a rolling distribution? Now *THAT* would be cool!!! Also I believe defining the target audience in terms of demographics is a pointless exercise for a general purpose OS. Moreover, Fedora is a community effort and communities need a focused, possibly steady target. Smoky marketing mantras like target audiences based on subjective, aleatory factors as the familiarity with computers or the mental capacity of people is the opposite of that. It provides fog and wind that could only lead to infinite discussions on futile social-behavioral definitions that even if attained perfectly would still be practically impossible to translate into univocal policies for a project. A community can thrive upon a well defined practical project. Not so much around the rather obscure concept of Caroline the waitress, Judie the trucker or Bill the plumber (IMHO). On the other hand, endless discussion can. And forever last. Then again I rather don't think that the target of an OS can be categories of people. It could be maybe some usage patterns, which by the way rarely match any specific demographics. So, as fun as it is to imagine the life of fedorans... not really useful to steer the project.
Jaroslav Reznik wrote on 2010-09-07 16:26:
Why not to combine it? Currently we have one development branch, unusable even for most of developers and two frozen so called semi-stable releases (with current restrictive updates policies). The question is why? Why not to have feature driven devel branches (not a single branch but with GIT I think it's possible to do a lot of stuff on Fedora without breaking everything to everyone), one Rawhide rolling updates Fedora (with merges from -devel branches but in a better shape than todays) and then one real release, once a time, even more conservative. It would be really very nice. Or at least - we should use the strength of two point releases - make Fn more relaxed, Fn-1 strict - then you have nice and "stable" Fn-1 but with still quite a fresh software from former Fn. I don't understand current policy at all even I understand what they want to achieve. Even if you compare to Windows, Mac OS X, iOS, Android - we are now much more strict and less flexible even compared to closed platforms!!!
[...] yes, my idea. This will be a single-issue campaign. As I vaguely pondered back in September, I think it would be of value to Fedora to take a good look at the Fedora release [...]