GNOME Usability: Calum Benson

An interview with Calum Benson, Usability Engineer, 
Sun Microsystems, Dublin, Ireland. 


LWN: Some general questions about you:

Q1. Please give us your name, place of work and title or position.  Also
    provide a brief biography so we know why your thoughts and comments 
    on UI testing are important.

CB: Calum Benson, Usability Engineer, Sun Microsystems, Dublin, Ireland. 
(I'm Scottish, though...)  I've been a user interface designer/developer
for various companies for nearly 8 years, working on things as diverse
as stock trading systems, virtual reality engineering tools and the air
traffic control system at Heathrow Airport!  I've been with Sun's
Desktop Group for nearly a year, working almost exclusively on GNOME.

Q2. How are you related to UI usability testing for GNOME?

CB: I'm part of a team of seven usability people at Sun working on GNOME--
I'm the only one in Ireland (where most of Sun's GNOME developers are
located), five are based in Sun's Menlo Park campus in California, and
there's one in Colorado.  Our usability labs are also in Menlo Park, so
the team there ran the GNOME usability study-- my only role was to help
decide what tasks we should focus on, and to present some of the
preliminary results at GUADEC.


LWN: Some usability questions:

Q3. What is usability - not usability testing, but the term "usability"
    itself?  Define it for us.

CB: Well, the official ISO definition is dry but accurate: "a measure of the
effectiveness, efficiency and satisfaction with which specified users
can achieve specified goals in a particular environment with that
interface".

A less formal definition would be that a usable product is easy to learn
and remember how to use, and helps you do your job quickly and enjoyably
without making mistakes.  Of course, each of those factors is more or
less important depending on the product and the environment in which it
will be used-- with an air traffic control system, for example, it's
less important to be fun to use or easy to learn, and more important
that it prevents you from making mistakes.
 
Q4. What are the goals of usability testing?  What are you trying to prove or
    disprove?

CB: That can really vary quite a lot between tests.  Some studies are purely
comparative, to determine if it's easier to do the same task with
product A than with product B, or if the new version of a product is
easier to use than the previous version.  Sometimes specific usability
goals will have been written into the product requirements, e.g. "the
user should be able to navigate between any two pages on this website
with no more than four clicks", and you'll be testing to see if those
criteria have been met.  And other times you might be testing some
completely new idea, in which case the goal is often just to determine
whether it's worth developing further, or whether it should just be
filed away in that handy round cabinet under your desk that gets emptied
every night!

LWN: GNOME UI usability tests

Q5. Describe the GNOME UI usability tests discussed at GUADEC II.
    Who decided such testing was necessary?
    Who organized the testing?
    Who performed the tests?
    Who tabulated and interpreted the results?

CB: The whole study was pretty much devised, organized and performed by four
members of Sun's usability team in Menlo Park-- Nancy Frishberg, Suzanna
Smith, Andrea Mankoski and Dave Engen.  Our usability lab staff helped
us recruit the participants and set up the equipment, and we got some
valuable technical support from Arlo Rose at Eazel.  Suzanna's been
mostly responsible for putting together the final report and video
clips.

The study saw a dozen people coming through our labs during the week of
March 12th, 2001.  The goal was to develop a baseline understanding of
any major usability issues in GNOME, rather than focusing on any
particular GNOME applications.  We did this by studying professional
users who had significant desktop GUI experience but who had never seen
GNOME before.  We asked them to perform the sort of tasks they might
need to do if they came into work one day to find that their friendly
sysadmin had installed some new desktop software: changing fonts, trying
to find their old files, visiting a favorite web page, and putting a
clock on their desktop.

Q6. What results surprised you most?  For example, did you expect users to
    misunderstand what the "man" command meant?

CB: That maybe wasn't such a huge surprise, a number of our participants
came from Windows and Macintosh backgrounds where there's no such thing
as "man pages", just "help".  Perhaps the biggest surprise was the
number of positive comments we got when we asked people to sum up their
first experience with GNOME, after we'd just watched them all struggle
with it for an hour!
 
Q7. Some results apparently showed that more than one choice for getting
    something done often was more confusing than helpful.  Does this mean users
    tend to *want* to be lead through their desktop?

CB: Providing different ways of doing the same thing is actually a good way
of supporting users with different backgrounds and levels of
experience.  There are a couple of things you need to be careful about,
though.

The first is making it obvious that the different ways of doing the task
all lead you to the same goal-- by using consistent terminology and
imagery, for example.  One example of this I came across yesterday was
with the Sawfish window manager-- if you go to the Sawfish capplet in
GNOME, the controls for changing keybindings are found under
"Shortcuts".  But if you run the sawfish-ui command from a terminal to
do the same thing, in the GUI that appears the relevant heading is
"Bindings".  This is confusing enough, but since the headings are listed
in alphabetical order, the controls don't even appear in the same place
in the two otherwise-identical GUIs.

The other critical thing is to ensure you don't *have* to know about all
the ways of doing something to complete the whole task.  This problem
showed up in the study with respect to panel customization, and
especially fonts-- there's no one place in GNOME 1.4 to change all the
fonts on your desktop, you have to know at least three different places
to go. 

So yes, it's important that the key features on a desktop are well
signposted, especially if you're new to that particular environment. 
But while more advanced features or quicker ways of doing the same thing
may not become apparent until you reach a higher level of competence and
start experimenting and exploring, they still need to be designed to be
as easy to use as possible.

Q8. How should developers pick target groups?  One of the problems with open
    source is that developers "scratch an itch", but that makes a pretty small
    target group.

CB: On the other hand, one of the advantages of open source is that you have
such a large pool of diverse people within easy reach!  The majority of
people on the GNOME mailing lists and IRC are young male developers,
true, but you can always find people out there with a completely
different perspective on your latest application that will help you
improve it.  You don't need fancy labs and one-way mirrors to get
valuable feedback-- any usability testing is almost always better than
no usability testing.

We'd also encourage people to post their user interface ideas,
screenshots, prototypes etc. to usability@gnome.org, or to hang out on
the #usability channel on IRC.  There are people there with a bunch of
user interface design experience who are always happy to give feedback
and suggestions. There's also a GNOME Usability Project webpage:
http://developer.gnome.org/projects/gup

Q9. What, if any, changes are expected for GNOME as a result of this testing?

CB: The reaction to the presentation at GUADEC was very positive, and most
of the main application developers were happy to take our findings on board,
even from those preliminary results.  Some changes were even made to the
default desktop icons quickly enough to be included in Ximian's GNOME
1.4 release, for example, and it's now likely that GNOME 2.0 will only
ship with one clock applet by default!  We're actively engaged with the
GNOME panel and control center maintainers and will be discussing
possible improvements with them, and keyboard navigation around GNOME
applications and the desktop is going to be much better in 2.0.  Hopefully
that's just a small fraction of the usability improvements we can
expect, though.

The testing also sparked a renewed interest in putting together some
user interface style guidelines for GNOME, which most other GUI
platforms have already-- the community are putting together a team to
work on those now.

Q10. What should open source developers take from this kind of testing?  How
     might they approach their GUI designs differently?

CB: The key message is really "we are not our users".  The GNOME desktop
will become much more mainstream over the next couple of years,
especially once companies like Sun and HP start rolling it out, and that
opens it up to a whole new audience-- mechanical engineers, web
designers, financial analysts and the like, not the developers who
currently make up the largest part of the user base.  So think about who
will be using your application, and try to find similar people to test
your designs regularly from day one-- friends, colleagues, relations,
whoever!  And don't be disappointed if they hate your first design, even
the professionals always end up throwing away their first couple of
attempts!

Q11. When are the video clips expected from Sun?

CB: We're concentrating on getting the full usability report out first, with
as many recommendations in it as possible-- that should be available
within the next three or four weeks.  The video clips will follow as
soon as is practical after that, but editing down 24 hours of video is a
big job!  So we're focusing on the report itself first.