What Makes Code Hard to Understand? What Makes Code Hard to Understand? A (Quantitative) Cognitive Model of What Makes Code Hard reading & understanding ^ Quantifying Hard-to-Understand Code Complexity Lines of Code (LOC) ← correlated w/ everything Cyclomatic Complexity (CC) ← control flow Halstead Volume/Effort (V/E) ← related to entropy Percent comments (PC) Readability # of identifiers ↓ (variable, function names) Line length, indentation ↓ # of blank lines ↑ Software Readability Ease Score (SRES) Like Flesch score (FRES) Tokens = syllables, statements = words, units = sentences LOC ≤ 20 CC ≤ 15 V ≤ 1000 PC 30-75% The Psychology of Programming Cognitive Dimensions of Notation (Blackwell, 2001) Consistency, hidden dependencies, etc. Rules of Discourse (Soloway, 1984) Variable names should reflect function Don't include code that won't be used Don't do double duty with code in a non-obvious way Plans & Schemas (Pennington, 1987) Textbase vs. situation/domain models Effect of task Read to recall/use/modify Effect of experience Reading strategies, schema use (i for counter) Understanding "Simple" Programs Understanding = predict output Simple = within metric thresholds Effects of Notation Representation, spacing, ordering Landy & Goldstone, 2007 (spacing & order of operations) Expectation Violations Unused code, misleading schemas, priming Detienne, 2001 (distorted recall) Interactions with Experience? The Experiment eyeCode Format Pre-test survey Experience, age, education 10 Python programs Predict printed output Random version Post-test survey Confidence, feedback 45 minutes No highlighting No feedback Participants 162 participants (129 M / 30 F / 3 UR) Bloomington - 29 ppl @ $10 Mechanical Turk - 130 ppl @ $0.75 Other - 3 Tobii TX300 Mechanical Turk Variables Independent Program version (2-3) Python experience (years) Programming experience (years) Dependent Grade (0-10) 10 = perfect, ≥ 7 = correct, 4-6 = CE Response time (14 - 256 sec) Common error (prob.) Correct (prob.) Results counting nospace twospaces The count is 1 Done counting The count is 2 Done counting The count is 3 Done counting The count is 4 Done counting scope diffname samename 4 | 22 all versions Avg. Grade (0-10) 6.95 Avg. RT (sec) 58.0 Effects ↓ CE with Python exp. (OR=0.71, p < .05) whitespace zigzag linedup 0 1 | 5 1 6 | 10 2 11 | 15 zigzag linedup Correct (%) 84.0 88.6 Effects ↓ correct zigzag (OR=0.18, p < .05) ↑ correct zigzag x Python exp (OR=2.60, p < .05) between functions (24|7|830) inline (19|7|660) [8, 7, 9] [1, 0, 8, 1] [8, 9, 0] between (continued) Actual Output [8, 7, 9] [1, 0, 8, 1] [8, 9, 0] Common Error [8, 7, 9] [1, 0, 8, 1]  functions inline Avg. RT (sec) 142.8 151.5 Effects ↑ CE with Python exp. (OR=1.44, p < .05) ↑ RT inline x Python exp. (β=13.2 sec, p < .01) overload multmixed plusmixed strings 12 | 7 | hibye 14 | 9 | streetpenny 53 | 53 | 53 multmixed plusmixed strings Avg. Grade (0-10) 8.93 8.60 8.53 Avg. RT (sec) 37.3 41.6 39.6 Effects ↑ RT plusmixed x programming exp. (β=3.34, p < .01) What Makes Code Hard to Understand? Unconstrained Notation Vertical spacing (grouping hints) Horizontal spacing (facilitates calculation) Mismatched ordering (efficiency) Expectation Violations Recent calculations should follow from previous Unused code should not be present Single use of operator per block Current / Future Work Eye-tracking &Modeling Eye-tracking: Areas of Interest Identifier Keyword Literal Operator Indentation Punctuation Level of Detail (1/2) Raw fixations + manual shift Level of Detail (1/2) Line-based AOI Fixation Times Mr. Bits Modeling with Mr. Bits Mr. Chips retina Modeling with Mr. Bits Mr. Chips retina ACT-R visicon + EMMA Visual attention shift latencies Encoding latency + eye movements Fitts' Law Modeling with Mr. Bits Mr. Chips retina ACT-R visicon + EMMA Glasgow spatial array Modeling with Mr. Bits Mr. Chips retina ACT-R visicon + EMMA Glasgow spatial array ACT-R declarative memory Chunks with spreading activation Decay and latency predictions Identifier prefix? p_x, p_y, etc. Modeling with Mr. Bits Mr. Chips retina ACT-R visicon + EMMA Glasgow spatial array ACT-R declarative memory CECEP behavioral models Modeling with Mr. Bits Mr. Chips retina ACT-R visicon + EMMA Glasgow spatial array ACT-R declarative memory CECEP behavioral models Python CDO Infer semantics from syntax: f(x) Categorize variables: iterators, accumulators, etc. Algorithm recognition Thank you! Questions?