Not just software developers. I was working on a medical device and visited doctors' offices to see how and where they stored their medication (e.g. for how could we make our dispenser more prominent without getting in the way?). We also filmed our investigators holding various prototype devices to see how they picked them up, how they were balanced, how they felt in small and large hands etc.
Universally we were told that nobody had ever asked them anything like that before.
I've never understood how people can design and build anything (software, hardware, whatever) without understanding how it's used. Dogfooding is great (and so many places don't even do that!) but you don't really know what you need to build until you sit down with your users and actually watch them work.
It's really difficult to understand how a large complicated appplication should be used if you have no use for it yourself. I spent most of the last twenty years workin g on design software for transformers (the electrical kind not the newly fashionabloe machine learning stuff) but I have never designed a transformer in my life. There is no way of dogfooding such a product the developers have no use for it at all.
So we do our best, observe the users, try to provide shortcuts and hints. But watching a user work is really difficult too because in a large application the users don't all use all of the application all of the time so getting a realistic picture of the whole use of the program is extremely time consuming which means expensive and the client departments just want to use it, they very often aren't interested in optimizing it.
Well, I've never been called to participate in a political survey poll thing, but that doesn't mean it doesn't happen to people. There's a lot of doctor's offices out there. What was your sample size?
Small, a few hundred, but was in a medical speciality so it was a reasonable number. In addition, that pool included all of our clinical investigators, the “thought leaders” you’d think most likely to have been consulted.
It was a bit of a shock for me to figure out that much of my early success in software boiled down to a few things I was doing that nobody else was, and one of those was watching other people work and doing something - anything - more with that information than simply writing it down or complaining about it. Most of those people have been coworkers, not even customers.
Agreed - I credit my entire career to this. I started watching people trying to solve problems, and then would piece together something "silly" that helped them solve it.
Next thing I was doing a big system implementation. Then consulting.
One note - now that I'm removed from day to day and don't have time to program myself I see how easy it is NOT to pay attention to how anyone will use something.
I imagine this is how windows gets bloated with candy crush saga and all the bloatware even on the "pro" versions. What employer wants this stuff on users computers? It just clutters up menues.
I play games all the time and I don't want games preinstalled. And it's not just that it's a casual game like Candy Crush. I don't care if it's the latest Call of Duty or whatever you young whippersnappers play nowadays. My work PC is for work and my gaming PC is for games I choose to download. Even if it came with Minecraft I'd go and download the Java version instead for mod support.
> I imagine this is how windows gets bloated with candy crush saga
Who knows, it's possible the Windows team just watched a lot of people install Candy Crush, and decided it would be a good addition to Minesweeper/Solitaire/Hearts (and what employer wants those on their computers?).
I don't doubt that this kind of invasive data collection can help developers at some point. But as a user I'm glad that most of the software I'm using doesn't track everything I do (or even register me as a distinct user at all). I'm also willing to accept poorer software quality in return (which curiously, isn't the case).
Having worked on user data before, I think the issue is that for complex software, it’s hard to get value/insight from this data. Imagine if you are Microsoft and had every action every Excel user makes with office365. Where would you even start to be able to get value out of this data?
I worked on the design of the first several versions of Excel. For Excel 3.0 we were intensely interested in user behavior. We often compared alternative prototypes of proposed features in the usability lab. We sometimes would invite develpers to watch users strugging with what they thought was great idea...
Also, long before the Internet we asked users permission to send them a special version of Excel (called the Instrumented Version) that would intelligently log actions, but not data to a floppy disk. Users would send the disks back to us for analysis. We used the analysis to understand how users really did things vs what we thought the did. Sometimes we got some surprises. We used this to prioritize features where users were having difficulty that needed special attention. It also indicated frequenty performed actions that would benefit from, for example, being put on the toolbar.
I think our user focus was an important reason why Excel has been so enduring. (Although Office 2003 screwed it up to some extent - I was no longer there.)
Back then at Microsoft the applications group tended to take the lead in UI developement and many of the things we designed made their way into Windows. Some were even picked up by the Mac! Although Steve Jobs would never have admitted it.
It grieves me to see the shoddy state of software design these days. There is no craftsmanship. Visual design take precedence over real usability. Maybe this is because with fast developemnt cycles and the ability to instantly change a web based design, it's easer to throw crap at the wall and keep iterating until it sort of works.
These days user interaction designers have endless great user data on which to craft wonderful UI but it rarely happens. If they happen to hit on something good, usually another team takes over and changes it, often for the worse!
Interesting to hear. One thing I never understood about Excel is why the cells which have formulas don't look different than cells which contain leaf-data. That I think would make it a lot more "user-friendly".
2ndly why not allow users name their columns, why do they have to always be "A", "B", "C", .. ?
> One thing I never understood about Excel is why the cells which have formulas don't look different than cells which contain leaf-data.
There are some styles actually meant for this, but you have to apply them manually.
> 2ndly why not allow users name their columns, why do they have to always be "A", "B", "C", .. ?
Named columns do exist actually! Not of the excel sheet itself - but in tables you can mark within that sheet. You can use the names in equations and everything. You can also name individual cells as well.
By far the worst UI choice in Excel, which has stopped many potential power users from forming, is that "Format as Table" is named like a visual formatting option, is stuck on the ribbon next to visual formatting options, and prominently makes you pick colors and border formatting.
Whereas the reality is that making a table is the on-ramp to many of Excel's more powerful features.
I imagine that some developer or team had to hide such useful features under the name of "table formatting" because they could not get the ideas through in their organization. Don't know but I imagine.
This is good stuff. Excel was (and in many ways still is) extremely well designed UI-wise (thanks for that).
In fact a lot of windows is (up to 7), and MS can be very good at UIs when they try. It's remarkable what an extraordinary amount of work went into making the interface usable without a mouse - for example, selecting something in the taskbar without a mouse:
Press ctrl-esc, press escape, press tab, then press the right arrow to move along the taskbar, the press return when you're on the one you want. It is actually logical too!
But the UI is the product for the user. The amount of lost goodwill (and lost customers if there is an alternative) is real money ie. less-than-max profitability. It baffles me because UI design needn't be very expensive.
Paying attention to how users work and and what they need can also be extremely profitable. The entire idea for Microsoft Office came from our discovering how often they were struggling with differerences in basic UI between say Excel and Word (which were quite different back then). Also how often they wanted to say copy data from Excel to Word or embed an Excel table in a Word document.
Based on this we decided to make the products more compatible in UI and able to work together better. It was a huge challenge because the products were designed by different teams that intensely belived in they way they had chosen to do things. It required Bill Gates to push it to make it happen and even then it was difficult.
Office 97 was the result and Office grew into a product that made billions for Microsoft. This came from our obsession with user experience...
I think we all agree. I should have rather said maximum short term profitability : fancy features and half-baked UI redesigns, rather than carefully improving the efficiency of the existing ones.
> Where would you even start to be able to get value out of this data?
It's an interesting question, but I think there are a few obvious easy places to get value (UX on common operations, most common error modes/corrections, etc.)
In my experience the problem is more that devs apply logging or telemetry where it is easy rather than where it is insightful. So if you didn't design the system up front to capture user behavior with some granularity you end up with a bunch of questionable proxies for it, which are hard/impossible to interpret.
I've also observed the problem where analysts or product managers cherry pick the data that validates their assumptions.
A "funny" example I ran into was product wanted to ship a notification that "test better" (i.e., had a higher interaction rate) in A/B testing. Some developers pointed out that the notification was likely seeing higher click-throughs because it was incredibly vague and users clicked through to figure out what it was about, and this would likely result in user frustration. Well, this insight was ignored and the notification change was shipped.
Our notification opt-out rate spiked a few weeks later.
I propose you start by studying a small batch of users, documenting their behaviors, develop a set of theories, then look at whether the larger pool of data agrees with or disproves your theory.
Try to come up with theories just from data and you end up with the Texas Sharpshooter scenario. We are complaining about this sort of behavior showing up in the scientific community, let's learn from that mistake.
I agree, I think doing a deep dive interview with a small set of users is better than just collecting telemetry. Ask them questions, carry on conversations, notice when they become frustrated trying to do something.
All the users could be using the right-click context menu to copy text not out of preference but instead ignorance - telemetry can tell you what's wrong but you miss out on most of the chance to tell why.
Install an accelerometer in the keyboard. When heavy impacts are detected from someone smashing their face into it repeatedly, pop up a dialog box asking them if there is something wrong and would they like assistance? Extensive feedback will be forthcoming.
A bit more helpfully, MS were asked repeatedly not to fuck with the menus but they replaced it with the ribbon. Ditto that terrible new interface that was nothing like windows 7. They could have allowed the choice of new interface but they did not. MS got plenty of info and chose to discard it.
MS is now in a position where it doesn't need to care.
There’s a lot of interesting history that I think this comment ignores. The office 2007 project was one of their more heavily feedback-driven initiatives; had they been paying attention the same way for Windows 8, it likely would have looked a lot more like Windows 10.
They had really good reasons for using the ribbon, not the least of which was that people weren’t finding the features they needed. That was all driven by telemetry back before it was considered the evilest of evils.
I’m amazed that you’re still bitter after 13 years. Afaict actual user preference is strongly for the ribbon, minus a very vocal minority that were bothered by the change. Office 2007 was in fact not the boon for OpenOffice that was predicted.
edit: tone, realized it came across as not as respectful as I intended
After 13 years there are still things I have trouble finding on the ribbon that were easy with the menu. And the ribbon totally kills the menu item->keyboard shortcut learning path to efficiently activate things you use in your personal workflow. I am bitter about this one too!
That may be a good guess: people who prefer words to pictures and keyboard to mouse would prefer menus to the ribbon. That pattern holds for me anyway. And I would believe that word/keyboard people are in the minority, so a UI study would identify the ribbon as superior.
Your point seemed to be that MS chose not to care. I think they cared but made an informed decision to go down a path that many said was wrong. If anything, I'd say kudos to them for not living in an echo chamber.
> Your point seemed to be that MS chose not to care
Quite. It was not about the merits of the ribbon. Which sort of echoes the article "Most software companies ignore user behavior"
> I think they cared but made an informed decision
they were heavily "informed" the ribbon would upset a lot of their customers, and for good reason. If there is another meaning to 'informed' here, please clarify.
People were not against the ribbon, they just wanted the choice. Dynamically reskinning an app like excel/word is entirely possible, and in fact straightforward if you actually own the original skin, which of course MS do.
"kudos to them" - really? For unnecessarily upsetting people who'd paid good money for office? Whose workflow was broken until they had re-learnt the new interface? Who now have a reason to move to LibreOffice, which still has menus?
Why are you defending MS's obviously stupid business decision?
It wasn't an obviously stupid business decision, it wasn't even stupid. But I can see how expert users like yourself would be upset.
Prior to the ribbon design, Office was extremely hard to use, excluding the most basic features, for novice and intermediate users. In general, endless menus are confusing and/or annoying to novice and intermediate users. The ribbon design enabled novice and intermediate users to find and use more advanced features that only expert users knew about in previous Office versions. MS optimized their product design for novice and intermediate users while still maintaining all the functionality needed by expert users.
From a business perspective this made sense for two reasons:
1. Most customers are either novice or intermediate users.
2. Even though expert users will likely get pissed off that they have to relearn the interface, they will still relearn the interface. Or put another way, designing for expert users should take a backseat because they will spend the time to learn an interface no matter how convoluted it is.
"It wasn't an obviously stupid business decision, it wasn't even stupid." -and- "Prior to the ribbon design, Office was extremely hard to use"
I'm afraid I'm just going to disagree with both of these. I'd buy that menus are somewhat harder to use for novices and the ribbon has some greater discoverability, though I'd like to see usability stats (edit: I did a quick search but found nothing solid), but you are missing my point - despite me repeating it multiple times! - that it was not about ribbon/menu. You just aren't getting it. Will people please stop dislocating the discussion onto what you want me to be talking about.
(stuff about optimizing their product design for novice and intermediate users)
That is a really good point, but it was entirely possible to support both. I wonder how.
(edit: "Even though expert users will likely get pissed off that they have to relearn the interface, they will still relearn the interface" - this is exactly what you do not inflict on your customer base without exceedingly good reason)
exactly, if you were a power user you'd still remember your lotus123 / commands. you wouldn't need either a menu bar nor a ribbon. microsoft kept a lot of the features the power user uses. the menu is not one of them.
IntelliJ IDEA has a plugin that will prompt you with the keys for commands you click. Pretty neat. Not exactly what you're saying (no compound actions). Still neat.
> Where would you even start to be able to get value out of this data?
Much like excel for certain professions, I work on a piece of software that is used all day by people in another profession.
One thing we do is look at which features are used most (and least). We make it easier to find and use the most popular features and we make rarely used features less prominent (and in some rare cases we might remove them altogether).
In our most recent redesign, we elevated some of the most commonly used information to the point where the user doesn't even have to click anything to gain some high level insight (previous versions needed at least 1-2 clicks to get to that same info).
We have about 30 power users who we work with continuously. Some of them are nearby, and we often go look over their shoulder and pick their brains. Some of them are in other regions where we get valuable global feedback that we wouldn't otherwise get from those regions.
They get early access to new features and they get to help shape the product that they rely on for their livelihood and help us decide on what we should be working on next. It's win/win and I would recommend forming this kind of relationship with a subset of your user base if you can.
>One thing we do is look at which features are used most (and least). We make it easier to find and use the most popular features and we make rarely used features less prominent (and in some rare cases we might remove them altogether).
Well, with all due respect, this is not about making a "better" product, it is about making a product "easier to use" and possibly even "more popular" (which not necessarily are the same thing).
>We have about 30 power users who we work with continuously. Some of them are nearby, and we often go look over their shoulder and pick their brains. Some of them are in other regions where we get valuable global feedback that we wouldn't otherwise get from those regions.
They get early access to new features and they get to help shape the product that they rely on for their livelihood and help us decide on what we should be working on next. It's win/win and I would recommend forming this kind of relationship with a subset of your user base if you can.
On the other hand this can be a much more productive approach, the program will "evolve" following the lead of these "power users", until it will become "almost perfect" (and more complex, and less popular because it has so many functions ...)
In other words, both approaches have their merits (and downsides), and IMHO you have a good process in having both, but ultimately it is the "common sense" applied to the choices you make for your product that may make a difference.
>Where would you even start to be able to get value out of this data?
I'd start with the most undo-ne operations. Which specific actions are most often immediately undone by users (that could lead to addition of functions, changing UI, adding visibility of the effects of an action, all sorts).
Then probably look at most searched terms in both Excel help and online, segmented by user experience if possible (to allow improvements for noobs and advanced users, whilst aiming not to be deleterious to either).
I'm not in software development though, curious what answers y'all have?
FWIW as a relatively novice LibreOffice user I probably most often undo auto-fills where a reference variable hasn't altered how I hoped. (I'd maybe fix that with a tailored tooltip - probably actually implemented as an info box - fired on long-hover of the auto-fill handle).
I read about Office 2000's dynamically hidden menu items feature (with the expand button added to the bottom of each sub-menu) somewhere, that the items that _are_ shown initially were determined by collected usage data. So it's probably aggregation that makes the most sense, not digging into each individual user's behavior.
Well, mcDonalds draws maps of where their workers put their feet, and then designs the kitchens to minimize the number of steps the workers needs to take to complete their jobs.
One way to use Excel telemetry could surely be to minimize the number of cognitive steps workers needs to take to complete the tasks they are trying to do.
Another could be to use telemetry to see at which point a smooth flow suddenly takes a longer time - and perhaps split an action into two actions which are faster (in time) to accomplish
>Well, mcDonalds draws maps of where their workers put their feet, and then designs the kitchens to minimize the number of steps the workers needs to take to complete their jobs.
In a completely different field (architecture, public spaces, parks) it has become common (and I find it a particularly smart approach) to create a new garden/public space/park with no "paths" and open it to the public.
And put some benches "here and there" semi-randomly.
Soon the grass will be worn out where people actually walks (i.e. people that needs to cross the park because it needs to go from point A to point B), so there you make the actual paved paths and the same goes for benches, some will have a path in the grass leading to them and some will look like never used, etc.
But when it comes to Excel (or similar software) there is a difference.
In the case of McDonalds personnel are instructed on exactly what they have to do and how exactly do it, i.e. their footsteps are their own optimization of an exactly specified chore.
In the case of the people on public parks there is no need for instructions, as people actually needs to go through the park to get from point A to point B and knows where they love to seat down.
In the case of Excel (or similar software) the issue is IMHO that most often than not the user doesn't actually know what he/she is doing, in the sense that user may (and often does) take a "wrong" or exceedingly convoluted path to get to the actual wanted result (when/if he/she actually gets to the result AND the result is actually correct).
I often deal with people that (most probably in good faith) bluffed their way in "familiarity with Office" on their resumè and never actually learned to work in Excel, the spreadsheet they make are "terrible", with material (and sometimes also conceptual) mistakes that are all fine if you are using Excel to - say - keep your monthly budget under control but that cause (big) issues if you are using it at work to manage a project expenses/production/forecasts, a bid offer or similar.
Feedback/telemetry from these users would "dumbify" the product and probably even make it much worse.
Can we get telemetry that reports on cognitive steps for developers working on code and minimize for that?
I know there is cyclomatic complexity. Mostly you can look at a method and tell "this is a clusterfuck" without needing to break out a specialized tool. I still feel like a lot of devs don't appreciate the simple philosophy of "don't make me think."
I'd say just invite me over for a chat. I've probably spent 10,000 hours in Excel and can tell you more about its shortcomings than a PB of telemetry ever will
You can do so much with this data, I'm surprised if it's not already collected. You can stratify your users based on what kind of commands they do, how their mouse or cursor is moved around the screen, and what functions they are getting out of excel, and see if you can improve their experiences by lowering unneeded friction and building new features.
I thought about adding something about privacy, but that's sort of a different topic for a different time. There are plenty of privacy preserving ways of learning from this data.
Those same companies mostly load their software and websites full of trackers against all reason. They are very very happy to collect that data. They just never use it - management personalities usually override any data findings... if anyone even ever looks at the results.
I like some of the information in this, and unfortunately I missed the conversation when this made it to the front page a couple weeks ago, I just wasn't in the right headspace.
But I disagree with one of the value judgements:
> As you add more and more users, you learn less and less because you will keep seeing the same things again and again.
No! The frequency of errors is your priority list for UX fixes. You may not, as they claim, find more errors with more than 15 people, but more data samples gives you information about distribution and frequency of errors (do people make this mistake once in a while or every single time?).
You might not need any of that data to make a rough priority list, or it might be helpful for negotiating scope. It'll depend a lot on context.
If you go through that site's (very long) history, you'll see their conclusions that quantitative UX research isn't useful for single projects.
For discovering features of the population, yes, quantitative research is great. But for making your software better, you are much better iterating small qualitative tests.
The point of usability testing is primarily to identify sources of roadblocks and confusion. It helps you understand where people can get stuck and why, but not the frequency.
Doing usability testing is both expensive and time consuming. It requires someone to analyze the videos and the testers are typically paid. So it's a very poor choice for creating frequency data.
Once the usability issues have been uncovered, it might make sense to look at the relevant analytics or add better tagging to get that frequency data.
I take some comfort in the fact that as more and more data gets collected about me, Big Sibling will be drowning in so much data that they can't possibly make sense of it all. Eventually an nearly infinite amount of analysis will conclude that I am boring actually as boring as was initially estimated. That said, I'd be pleased with less monitoring of myself.
Reminds me of this talk[1] I've seen a while ago where an ex-NSA executive talks about how the NSA has too much data to handle it efficiently (amongst other things).
I'm working on an app and trying to be conscientious about my user's behavior, but it's proving to be extremely time consuming for the value I get out.
The problem for me is I can take a relatively accurate guess and achieve 90%+ of what the user is looking for. That last 10% takes a ton of time to figure out. For every moment I spend on the 10%, I'm only getting a fractional return compared to simply moving onto another feature.
Maybe in the future, I'll come back and figure things out, but for now it's just not worth my time.
With a few lines of code, you can integrate Segment (or a similar tool) into your software. You can start tracking user behavior and events automatically. Segment can then send this data onto tools like Mixpanel or Amplitude which has built-in user activity flow analytics.
Take a look into some of these tools if you are looking for an easier way to track user behavior.
I would love to know which features of my app are used and by how many user. But this clashes with my requirement to not have any server side service or telemetry.
You can use a tool like Segment to track user activity client side. With a few lines of code in your app / front end you can be sending user behavior and events to Segment via their API. No custom server side service is required.
Might be worth digging into if you are interested in better user activity tracking.
I'd rather not include a third party either. Then I would have to do all the GDPR work to disclose what do they get, how do they store it and so on. Ideally Apple would provide some framework to do basic functionality analytics.
Universally we were told that nobody had ever asked them anything like that before.