Making It Stick — A Developers’ Implementation Plan (Long version)

During a holiday around Europe with plenty of downtime between destinations, I started churning through the e-book ‘Make it Stick’ by by Peter C. Brown,‎ Henry L. Roediger III and‎ Mark A. McDaniel. As a Java developer who feels they have a lot of catching up to do in the industry, I thought I needed better strategies for learning than the ones I have been taking.

A small disclaimer to the link above, it’s an affiliate link and I get a small bonus for sales — but it won’t cost you any more. I do truly believe in the book though, which I’ll hope you’ll see from the article:

With previous textbook readings I’ve done, I didn’t feel I was retaining the information I needed. More specifically, I was getting overall concepts from books but it only felt like I retained a tiny percentage of what I needed.

I’m a big believer in ‘learning by doing’, rather than just passively reading text, and this is borne out by several sources, including ‘Make it Stick’1 and the much lauded ‘Learning how to Learn’ MOOC. 2Developing ‘experience’ yourself is what matters, not how many times you’ve read the text.

The problem was that the book itself didn’t give extensive implementation plans — quite intentionally as it didn’t want to be prescriptive. But where does that leave us struggling lifelong students of Software development?

I needed a way of proving whether this system could work before investing too much time in it, and I think I found a system that worked.

Simply put, to get the maximum value out of this you need to:

• Decide which material is worth the investment in this approach
• Find a way to space your practice to push yourself on learning concepts
• Set up a reminder system
• Generate ‘real experience’ through flashcards and ‘learning tests’
• Get full value, by creating something you can refer back to whenever required.

Decide which material is worth the investment in this approach

Its important to decide which subjects warrant this investment because that’s exactly what it is. You will definitely get through material slower if you take this approach but you will improve your recall, as I found.

During the experiment I made a decision to concentrate on learning some of the Core Java trails because I felt that you will always build on top of these fundamentals, and can re-use them in a variety of contexts. A chat with a friend who has always been the ‘go-to’ developer at his work convinced me I had the right idea.

Find a way to space your practice to push yourself on learning concepts

In order to actually get set up, you will need a way to test yourself. ‘Retrieval practice’, whereby you actively look to recall what you’ve learned rather than simply reading has been proven to be more effective than other methods for recall3. Flashcards are one such approach.

I used the website www.cram.com to build up a collection of flashcards which I added to every time I came across a concept that I felt would be important to recall. I preferred this to paper flashcards because I could sync across my tablet, PC and phone, and have a set of tests that were ‘good to go’ whenever I was on public transport, a lunch break, or simply had some spare time.

To space my practice, I set reminders on my phone for 1,3,7 and 10 days after learning material. This way, I was making it harder to recall something, but not a time so far apart that I’d have to learn everything again. The goal was to struggle to recall, but not struggle so much that I’d have to re-read everything.

Getting through a trail wasn’t done in ‘one sitting’, but over a series of days. So as I got further along the set of flashcards, and my challenge to recall them, would grow.

Below is a diagram of how I scheduled out spaced repetition and learning tests

Setup a reminder system

I am a massive fan of using Getting Things Done, or GTD for short. For those who don’t know of it, think of it as an awesome way to separate the ‘what’ from the ‘when’. In this particular context, I knew I’d need a 1, 3, 7 and 10 day reminder system for each day of learning.

I use the tool DoIt as I like the features it offers, and the fact it looks good and works well across phone/tablet and PC. I was able to use the daily task reminder and calendar tool to good effect to set my reminders.

Set a daily reminder template

Using doit, I can set a template to remind me, every day at 07:45, to follow the link to https://www.cram.com/dashboard-flashcards#flashcards and complete the specific task in hand. 07:45 for me coincides with when I get public transport to work so was a convenient time for me, and makes me accountable to complete the task.

Setting up a daily reminder to go through my flashcards

Set the specifics

Doit has a calendar view as well, and because I have set up a task to auto-generated daily, it means that I can, at the time I do a reading of a subject, quickly pinpoint the calendar days that are 1,3,7,10 days ahead. In this example, that will be the 19th, 21st, 25th and 28th April.

Saying for arguments sake, I had just started reading up on concurrency, I then alter the task as follows for the 19th, the 19th, 21st, 25th and 28th April. Notice I include a link straight to the set of flashcards to reduce any extra resistance I may have to the task.

Now I have a system that holds me accountable for each new learning subject.

Generate ‘real experience’ through flashcards and ‘learning tests’

Using my reminder system, I was able to challenge myself on the high level concepts — after all you can only fit so much onto a flashcard. But of course technical jobs need much more attention to implementation details, so I needed a way to retain information from that too.

Enter the ‘learning test’ — from the book Clean Code4. For those unfamiliar with this term, it’s an excellent approach for developers from ‘Uncle’ Bob Martin. Simply put, rather than reading the documentation for software, use an available testing framework to **prove** what the documentation says.

This was exactly what I did — so for example when I read through a tutorial that said that a Java SimpleDateFormat worked in a certain way, I created a TestNG test that would prove (or otherwise what I was reading).

Tutorial Material

Learning Test Example

DateFormat and Style

@Test
public void whenShortDateFormatteFormatIsShortThenForwardSlashSeperator() throws Exception{
    DateFormat dateFormat = DateFormat.getDateInstance(DateFormat.SHORT);
    assertThat(dateFormat.format(EPOCH_DATE),is("01/01/70"));
}

Date format and default Locale

@Test
public void whenLocaleUkAndDateFormatIsMediumThenThreeLetterMonthAndDay() throws Exception{
    DateFormat dateFormat = DateFormat.getDateInstance(DateFormat.MEDIUM,Locale.UK);
    assertThat(dateFormat.format(EPOCH_DATE),is("01-Jan-1970"));
}

This was vital. By running through a series of tests, and seeming them fail, I was able to see that my existing understanding of API’s was wrong in some cases. I could see a test fail, find out from the official docs, or someone’s article and have a test I could run, at will to see the impact.

In some cases, these ‘lower level’ findings were important enough for me to create additional flashcards for, to remind me of something I wasn’t expecting.

Get full value, by creating something you can refer back to whenever required

I setup an account on bitbucket, so that I could check in any tests that I wrote. Combined with cram being available online and offline (on my tablet and phone), I had reference material that I could refer to anytime, anywhere.

This was huge bang for buck. To give an example, this month I’d written learning tests against the BigDecimal API. At work, when I was under time pressure to see why a unit test failed, I simply pulled down my learning test repo and found a scenario matching what I needed. As I was scanning the code, I got a refresher on the API ‘for free’!

As the ‘Learning How To Learn’ Course has also mentioned5 , each time you recall information in new scenarios, the information and memories are strengthened and enriched, so you’re getting even value out of this approach by having a digital system you can call up.

Findings

As I went along, I built a table of results, so my progress wasn’t lost to me, a snippet showing 1,3,7,10 day concepts would be:

Sample of test scores for some subjects in March 2017

On the surface these results look pretty inconsistent, but I had made some interesting discoveries:

• I found the first 24 hours vital — I effectively needed to start again any time that I didn’t test myself within 24 hours. The best approach I found was testing the same evening I initially read something and then again in the morning.
• In first 24 hours my familiarity was still with the *actual page/text*. As I shifted to actively learning, over the following days, my familiarity was then with a more overall concept — the page and text, plus my experience.
• Related to the above, I realised there was less importance in the actual grades and their fluctuations than building a set of test material (the Java learning tests). In hindsight I should have captured the number of questions on the set at the time of each set, so I could have factored in that I was always ‘adding questions to the pile’.
• Conversely the very existence of my own learning tests was a far truer test of mastery — I had to have understood the subject, to be able to write an automated assertion for it, rather than passively read the output of a program.
• When you have to fix a problem yourself, especially when the solution comes from other sources, retention felts more enduring. One example that sprang to mind was the use of an annotation that didn’t behave as excepted because it was not retained at runtime. Once I found an answer, and wrote the test to prove it, that concept felt ‘burned in’.
• As I went along, I adapted the flashcard questions into more specific ones — as I got more understanding of a subject, I knew what I’d need to ask ‘future me’ in order to jog my memory. This can be something as simple as realising there are two points to an answer, and changing the question to be ‘name the 2 things you need to do for x …’

Summary

• Its important to decide which subjects warrant this investment because that’s exactly what it is. I’d lean towards this approach for learning the fundamentals of a subject because they are points that are good ‘bang for buck’. You will get the benefits of long term recall for information that will be applied again and again,in a variety of contexts.
• I’m not sure specific implementations of subject matter need to be committed long term — I don’t think the ‘lifespan’ of the is worth slowing you down.
  To give an example, I’d written learning tests about reflection because a Framework like Spring makes extensive use of it, but I wouldn’t write learning tests for developing Spring Web Project at work. In these cases my learning would be from actually developing the project! (And of course, I’d write unit tests for any development I did anyway)
• I was shocked to see how many times an API would surprise me by failing my tests and this nicely demonstrated two points from the book, ‘you learn far more from failing’ and ‘you learn more when you try and generate the answers for yourself’.
• Where you don’t have a concrete use case right now for a fundamental concept, this is where the learning tests really come into its own. You have a set of *proven* reference material with no need to reinterpret documentation each time.

Your feedback

I’d love to hear from you now, in the comments section. Did you like the article? Any pointers for me to improve what I deliver to you? Any success with this approach? Have you hit any stumbling blocks?

It would be great to know what you think.

Resources

Flashcards

As an example of a set of flashcards, I’ve linked to some flashcards I made on reflection, date/times and collections .

Bitbucket project of learning tests

I included a set of resources here that I’ve been using to learn APIs, you’re welcome to use for yourself, but if you re-blog, please link back to this article. Note that some of the tests (collections and concurrency) are still work in progress to get fixed comprehensively — I’ve left them in here deliberately as a reminder that I still haven’t mastered these points.

Citations

1Brown, Peter C, Roedinger III, Henry L, McDaniel, Mark A. “Reflection Is a Form of Practice.” Make it Stick. Cambridge, Massaechusetts, London England: The Belknap Press of Harvard Press, 2014. Pages 26–27. Print.

2https://www.coursera.org/learn/learning-how-to-learn/lecture/BuFzf/illusions-of-competence

3Brown, Peter C, Roedinger III, Henry L, McDaniel, Mark A. “Testing: Dipstick vs Learning Tool.” Make it Stick. Cambridge, Massaechusetts, London England: The Belknap Press of Harvard Press, 2014. Pages 18–19. Print.

4Martin, Robert C, “Exploring Boundaries” Clean Code: a handbook of agile software craftsmanship. Stoughton, Massachusetts. Pearson Education. Pages 116–118. Print.

5Brown, Peter C, Roedinger III, Henry L, McDaniel, Mark A. “The Takeaway” Make it Stick. Cambridge, Massaechusetts, London England: The Belknap Press of Harvard Press, 2014. Pages 43–44. Print.