What Makes Code Hard to Understand?

A (Quantitative) Cognitive Model of

What Makes Code Hard
reading & understanding

^

Quantifying Hard-to-Understand Code

LOC ≤ 20
CC ≤ 15
V ≤ 1000
PC 30-75%

The Psychology of Programming


Understanding "Simple" Programs


The Experiment

eyeCode

Format


  1. Pre-test survey
    • Experience, age, education
  2. 10 Python programs
    • Predict printed output
    • Random version
  3. Post-test survey
    • Confidence, feedback

Participants


Tobii TX300

Mechanical Turk

Variables


Results

counting

nospace

twospaces

The count is 1
Done counting
The count is 2
Done counting
The count is 3
Done counting
The count is 4
Done counting

scope

diffname

samename

4 | 22

all versions
Avg. Grade (0-10) 6.95
Avg. RT (sec) 58.0
Effects ↓ CE with Python exp. (OR=0.71, p < .05)

whitespace

zigzag

linedup

0 1  | 5
1 6  | 10
2 11 | 15
zigzag linedup
Correct (%) 84.0 88.6
Effects ↓ correct zigzag (OR=0.18, p < .05)
  ↑ correct zigzag x Python exp (OR=2.60, p < .05)

between

functions (24|7|830)

inline (19|7|660)

[8, 7, 9]
[1, 0, 8, 1]
[8, 9, 0]

between (continued)

Actual Output

[8, 7, 9]
[1, 0, 8, 1]
[8, 9, 0]

Common Error

[8, 7, 9]
[1, 0, 8, 1]
[8]

functions inline
Avg. RT (sec) 142.8 151.5
Effects ↑ CE with Python exp. (OR=1.44, p < .05)
  ↑ RT inline x Python exp. (β=13.2 sec, p < .01)

overload

multmixed

plusmixed

strings

12 | 7  | hibye
14 | 9  | streetpenny
53 | 53 | 53
multmixed plusmixed strings
Avg. Grade (0-10) 8.93 8.60 8.53
Avg. RT (sec) 37.3 41.6 39.6
Effects ↑ RT plusmixed x programming exp. (β=3.34, p < .01)

What Makes Code Hard to Understand?


Current / Future Work

Eye-tracking &
Modeling

Eye-tracking: Areas of Interest


Level of Detail (1/2)



Raw fixations + manual shift

Level of Detail (1/2)


Line-based AOI Fixation Times


Mr. Bits


Modeling with Mr. Bits


Modeling with Mr. Bits


Modeling with Mr. Bits


Modeling with Mr. Bits


Modeling with Mr. Bits


Modeling with Mr. Bits


Thank you!

Questions?