Watching the Goldeneye ad during the Nintendo press conference was absolutely shudder-inducing for someone with experience with focus groups shaping executive decisions. Focus groups aren't un-useful -- there is a lot of good information that can come out of them -- but I think that focus groups like the ones seen in this video hold way too much power over executives and marketers, and seeing a game ad based around a focus test just bolsters this view; as though a bunch of group-think teens getting excited about first-see bragging rights will be as exciting to the world as it was to the marketing team. Ugh.
But this video is a handy example of how flawed focus testing is. Notice the open room, with a group of kids within clear view of each other, talking to a focus test moderator about what each individual thinks. Silly. The first reasonable-sounding answer is almost always echoed around the room, each kid in fear of sticking out, clearly influencing each others' opinions. A kid that dares stick out usually endures ribbing and gets much more quiet, despite the moderator's best intentions. The opinions of the loud kid going "yeah-ha-ha" flavors every other kid in there.
The sad thing is that marketing knows the data is flawed, but the point is less whether the data is accurate than it is, in the eyes of those higher on the totem pole, considered "data." When a marketing team's jobs are on the line for a game's success or failure and the only way to keep your job on failure is to have a nice trail of CYA data points that justify each decision, then they understandably weight decisions based on said data, flawed or not.
So yes, a development team can gather useful information from focus groups, but it won't necessarily match the raw data gathered influenced by group-think. Too often, a development team will have its arm twisted to follow raw data before useful information, because the former will keep the higher-ups employed. Developers aren't always right, but the problems with this approach seem rather obvious to me, so forgive me this pent up rant.