I agree, but isn’t this the curse of most experimental research in programming techniques?
Ideally, we’d compare like-against-like using industrial scale applications, implemented by professional practitioners, controlling for everything except what we’re trying to investigate. Finding opportunities to do that in practice is rather harder, because obviously most real software development projects don’t get implemented twice by identical teams making exactly one significant change in their approach.
What I thought was interesting here was that even though it is only a toy program, there were still some patterns to how the reader explored it that might suggest more general trends. As long as we understand the limitations of the experiment and don’t overgeneralise any conclusions, isn’t some data still better than random conjecture?
Yes, but why can't some regular code (from an open sourced project) be used in the experiments? I'm sure there should be a way to do that.
It is just, that this synthetic code is broken. It doesn't read. There is no flow in it. It just looks from the first glance, like a non-interesting, unimportant piece, that doesn't do anything, so there is no point in reading it.
Does that sound convincing enough, that there is a big difference between reading synthetic examples and actual code?
Yes, it does. There was a study of how well people could remember positions of chess pieces on a board. The general public did equally badly for both random arrangements and those that could actually appear in a game, but skilled chess players did well remembering arrangements that could appear in a game, but were no better than the general public on random configurations.
> isn’t this the curse of most experimental research in programming techniques?
Of most experimental research with humans in general, for that matter. It's a big topic of controversy in psychology, because a lot of quantitative psych results are from synthetic tasks in lab settings, which leads to argument over the extent to which those are accurate proxies for real-world behavior.
I’m not sure if this is what you were alluding to there, but I think there is a real problem with experimental research in this kind of field that often the test subjects are students, who tend to have much less skill and experience than working programmers with a few years of practice behind them.
It’s a specific instance of a more general problem: it seems clear that programmers often work differently depending on their familiarity both with programming generally and with the specific domain they’re working in, but we’ve only scratched the surface in identifying exactly how they work differently, and therefore what practical steps we might take to make things better for programmers in different situations.
This is often the problem with a lot of psychology research, where the only easily available subjects are university students who receive credit for participating in the research (or a nominal payment).
Ideally, we’d compare like-against-like using industrial scale applications, implemented by professional practitioners, controlling for everything except what we’re trying to investigate. Finding opportunities to do that in practice is rather harder, because obviously most real software development projects don’t get implemented twice by identical teams making exactly one significant change in their approach.
What I thought was interesting here was that even though it is only a toy program, there were still some patterns to how the reader explored it that might suggest more general trends. As long as we understand the limitations of the experiment and don’t overgeneralise any conclusions, isn’t some data still better than random conjecture?