Depends on the note-taking strategy. From the same paper you cited, section 8.4 Practice testing - Issues for implementation, pages 34-35:

> Another merit of practice testing is that it can be implemented with minimal training. Students can engage in recall-based self-testing in a relatively straightforward fashion. For example, students can self-test via cued recall by creating flashcards (free and low-cost flashcard software is also readily available) or by using the Cornell note-taking system (which involves leaving a blank column when taking notes in class and entering key terms or questions in it shortly after taking notes to use for self-testing when reviewing notes at a later time; for more details, see Pauk & Ross, 2010).

Further explanation: https://lsc.cornell.edu/notes.html#post-1037

The Cornell note-taking system combines moderate and high learning techniques: elaborative interrogation, self-explanation, practice testing, and distributed practice.

If you think about it, creating Anki cards is note-taking in a specific format. Of course you can skip it and use a set prepared by someone else. It would be interesting to test: is creating your own flashcards better for studying than using ready-made ones?

Founder of StudyWand.com here, who received a 15k grant to develop an AI generating flashcard app in 2020 after an earlier prototype.

We've found students more consistently study ready-made cards that are at desirable difficulty (they get about 80% correct) and which are segmented by topic (e.g. semantic grouping of flashcards to tackle "one lesson at a time" like Duolingo). Students would prefer to use pre-made flashcards by other students in their class, then AI flashcards, then create and use their own.

There is limited evidence by Roediger and Karpicke who are the forefathers of retrieval practise that creating cards is also important. Frank Leeming (2002 study Exam-a-day) also showed that motivation when studying is peaked when you ask just a few questions a day, but every working day.

Now one of the vital benefits of retrieval practise with AI over creating your own cards is foresight bias - not mentioned yet in this thread - the fact that particularly in some subjects like Physics, students don't know what they don't know (watch this amazing Veritasium video, it also explains why misconceptions are so handy for learning physics): https://www.youtube.com/watch?v=eVtCO84MDj8 - basically, if you use AI quizzes (or any prepared subject-specific right/wrong system), you learn quickly where your knowledge sits and what to focus on, and reduce your exam stress. If you just sit their making quizzes, firstly you make questions on things you already know, you overestimate how much you can learn, and you consolidate on your existing strengths, and avoid identifying your own knowledge gaps until later on, which is less effective.

--

To quote from my dissertation experiment on background reading for retrieval practise, the end is about foresight bias a little: Retrieval practice – typically, quizzing - is an exceedingly effective studying mechanism (Roediger & Karpicke, 2006; Roediger & Butler 2011; Bae, Therriault & Redifer, 2017, see Binks 2018 for a review), although underutilized relative to recorded merit, with students vastly preferring to read content (Karpicke & Butler, 2009; Toppino and Cohen, 2009). Notably mature students do engage in practice quizzes more than younger students (Tullis & Maddox, 2020). Undertaking a Quiz (Retrieval practice) can enhance test scores significantly, including web-based quizzes (Daniel & Broida, 2017). Roediger & Karpicke (2006) analysed whether students who solely read content would score differently to students who took a practice quiz, one week after a 5-minute learning session. Students retained information to a higher level in memory after a week with the quiz (56% retained), versus without (42%), despite having read the content less (average 3.4 times) than the control, read-only group (14.2 times). Participants subjectively report preference for regular Quizzing (Leeming, 2002) over final exams, when assessed with the quiz results, with 81% and 83% of participants in two intervention classes recommending Leemings “Exam-a-day” procedure for the next semester, which runs against intuition that students might biases against more exams/quizzes (due to Test Anxiety). Retrieval Practice may increase performance via increasing cognitive load which is generally correlated with score outcomes in (multimedia) learning (Muller et al, 2008). Without adequate alternative stimuli, volume of content could influence results, thus differentiated conditions to control for this possible confound are required when exploring retrieval practice effects (as seen in Renkl 2010 and implemented in Methods). Retrieval practice in middle and high school students can reduce Test Anxiety, when operationalised by “nervousness” (Agarwal et al 2014), though presently no research appears to have analysed the influence of retrieval practice on university students’ Test Anxiety. Quizzing can alleviate foresight bias – overestimation of required studying time – in terms of students appropriately assigning a greater, more realistic study time plan (Soderstrom & Bjork, 2014). Despite the underutilization noted by Karpicke and Butler (2009), quizzing is becoming more common in burgeoning eLearning courses, supported by the research (i.e. Johnson & Johnson, 2006; Leeming, 2002; Glass et al. 2008) demonstrating efficacy in real exam performance.

I'm a long term (10 year+) user of Supermemo (and general fan of SRS stuff) and finally got around to checking out StudyWand today. This is the best experience I've had making flashcards and general study material ever. Hands down, nothing I've seen comes close.

It's wild because StudyWand took my sample notes and did everything that I would have done with them if I was going to use them to get a good grade in school. I was expecting some semi-decent cloze generated cards but got much more.

Literally, when I was in college I would take notes in class, and then spend about 20-40 minutes post class doing almost exactly what StudyWand does. The classes that I bothered doing that for, I always got a good grade in, nearly effortlessly. The hardest part was making the notes.

The part of this that I'm actually excited about is that this tool also works with any sort of documentation. For example, I can clean up any reference page from MDN as a PDF and get a usable (like actually well-made flashcards) set of 15-20 flashcards for it. Oh, you also get summaries and multiple essay questions too. The only way this would be better is if it gave you cloze deletions that were actual sample code to fill in the blank with.

I didn't really like your intro so I took a few days longer than I normally would have to look at your software (I normally check out every SRS software I see on HN). This software is insane. The value is so, so, so ridiculous. I half-hearted uploaded one poorly made PDF of a webpage and got flashcards that are comparable to what I would make as a 10+ SRS user. I almost stopped doing the initial reviews halfway through and looked for a way to pay for this.

Outside of Supermemo, this is the only other SRS software that I've seen that's worth my money. The hardest part is going to be convincing all my younger family members to actually use this. I've tried so many times to get people to use Anki (Supermemo won't happen), and they just don't get it. I think StudyWand might be able to bridge that gap. I'm going to try and see.

Whoa, awesome to see a fellow SuperMemo user here! I've been going for 17 years, I absolutely can't imagine life without SuperMemo and I've stopped trying to sell my friends on it. Have you tried integrating Study Wand with SuperMemo at all? I'm always on the lookout for ways to maximize my information intake (Aside from SuperMemo, which already does a nearly perfect job at it).

Nothing automated yet, but I may plan to. I don't regularly add that many new cards to my collection to where it'd be an immediate benefit for me. I may add more with StudyWand since it seems to do a good enough job of creating cards. Incremental reading is the primary way I add new material to my collection, but I don't really fully process articles that much into items these days. Most of my SuperMemo use is using incremental reading to ensure I always have something interesting to read as well as tasklists for planning/ideas. I have a lot of stuff that I like to revisit or review, which I find using SuperMemo great for too.

That's an excellent response. Thank you. I didn't think of the foresight bias and now that I do, it makes a lot of sense.

> Undertaking a Quiz (Retrieval practice) can enhance test scores significantly, including web-based quizzes

The way I understand it, retrieval practice increases test scores where you test information retrieval (quizzes, multiple choice, etc.). Which makes sense because you're practicing a skill that the test evaluates.

The follow-up question: does retrieval practice increase scores when you evaluate understanding of a subject, such as open tests or essays?

Thank you.

I am afraid I don't know and couldn't easily find any research. I did find this post, but it looks like SEO spam a little, and doesn't cite the essay claim: https://www.bookishelf.com/the-importance-of-information-ret...

I'll ask Prof Roediger as we occasionally communicate and will aim to get back to you. However, I wouldn't be surprised if any correlation was not statistically significant.

One more thing. When you mentioned Test Anxiety it reminded me of frequent comments about anxiety during job interviews. I'm wondering if flashcards (or other retrieval practice) could help there too. Perhaps you can spin it into a product for professionals.

What kind of flashcards would help there for you?

[edit: removed a point about variance which wasn't supported by more recent literate]

High stress (or anxiety) happens only periodically... like those job interviews you mention. It varies massively within individual subjects. There are some ways to reduce anxiety, but I wouldn't say anything has been conclusive yet. Here is an excerpt from my Dissertation about Test Anxiety... I found practise quizzes had a non-significant overall effect on Test Anxiety, and reduced only the "Tenseness" subscale of a relatively old measure sadly. The CTAS scale, referenced below, would probably be most useful for the job interview case.

"Test Anxiety is the additional stress felt when you must provide answers. It is prevalent, affecting students from preschool (McDonald, 2001) to taught university level, and across demographics (Beidel et al 1994; Beer 1991; Mwamwenda,1993). Prevalence is estimated at 20% (Ergene 2003) to 25% (Thomas et al, 2018) in current students. Test Anxiety often involves pressure, highlighting a social, non-genetic nature, and high variance is recorded between subjects (e.g. Keith et al 2010 longitudinal study showed low individual variation, but high between subjects variation). Overwhelming Test Anxiety significantly impairs wellbeing (Steinmayr et al., 2016). Interventions and online course design choices can markedly reduce Test Anxiety (Abdous, 2019). Unfortunately, few online learning studies record Test Anxiety, instead recording grade/score differences. Yet, Test Anxiety interventions are efficacious – e.g. "test-wiseness training" (Kalechstein et al., 1988), group counselling (Alkhawaja, 2013) and mindfulness training (Seidi & Ahmad, 2018). Research often classifies students into High and Low groups. There are disproportionately negative outcomes for High Test Anxiety students, whom more recently have been enrolled automatically in targeted interventions (e.g. Psychoeducation Bedel et al., 2020). Test Anxiety is related to Social Anxiety (Rothman, 2004; Sarason and Sarason, 1990). This manifests in a socialevaluative component, which appears in most scales, such as the Test Anxiety Inventory (TAI; Spielberger, 1980). TAI was designed for assessing children (Ludlow & Guida, 1991). Recent scales, including the Cognitive Test Anxiety Scale (CTAS; Cassady & Johnson, 2002), are adapted to university participants. An example CTAS item is "I feel under a lot of pressure to get good grades on tests.". CTAS has similar correlations with academic performance (Chapell et al., 2005) as those utilizing TAI (such as Hunsley, 1985), without the concerns of TAI applicability discussed in Szafranski (2012). Test Anxiety is typically construed as a trait under Latent state-trait theory, despite short term experimental intervention differences being observed (e.g. "looking ahead" immediately prior to testing in Mavilidi et al., 2014; or taking practice quizzes in Agarwal et al., 2014). Test Anxiety should not be confused with other forms of anxiety, such as General or Social Anxiety Disorder. Prevalence of anxiety is rising in college populations (Zagorski, 2018), as is loneliness, with 1 in 5 young people reporting having no close friends and feeling alone(n=2,522, Mental Health Foundation, 2019). Usage of social media is also associated with greater loneliness (Wohn & LaRose, 2014). However, artificially induced status updates have been shown to reduce it (Deters & Mehl, 2012), which aligns with status updates requesting academic support reducing Test Anxiety as found in Deloatch et al (2017) and findings linking higher wellbeing with greater academic outcomes (Public Health England, 2014). Research on social anxiety and social media is less clear, although recently Erliksson et al (2020) correlated greater internet usage and activity with increased social anxiety. Despite wide research on the negative aspects of social media usage (e.g. for university students, see Odacı & Kalkan, 2010) - little attention has been paid to online helping, which is fundamentally social."

According to some people (Justin Sung on YouTube is where I heard it, probably, so no idea if credible) you can influence the forgetting-curve and make Anki more effective by having more context, putting knowledge into relationships with other knowledge, etc. It's a multiplier on raw/rote space repetition.

I have found that by adding simple etymologies I can improve my recall of Latin names for plants. The etymology connects an otherwise totally new word to existing word-concepts.

Really depends on how you’re defining context, since to me there’s concepts like memory chains/ladders, which simply fit sequential memories into a visualization — then there are applications contexts, which actually mirror reality of future use.

Great example of this is topic of memory sports [1], which do use techniques that work, but are likely less useful for actual remembering real world complex information.

[1] https://wikipedia.org/wiki/Memory_sport

I think the benefits are multiplicative here.

As another experienced Anki user, I have found I get much more leverage out of spaced repetition (both in terms of efficiency of memorization, as well as in how useful the information is) when I've first made the knowledge my own and structured it in a way that makes sense to me before creating the cards, rather than just dumping a bunch of pure entropy into the Anki database.

That's not really note-taking per se, but it is ensuring that the stuff I'm trying to memorize isn't pure entropy (which is always more of a challenge to memorize in any event), but rather is part of a larger sense-making structure. The purpose of spaced repetition is to help prevent that structure from decaying; it's not a substitute for having it in the first place.

As others already pointed out, the linked article seems to say the reverse of how you interpreted it (though I would agree the terms are a bit ambiguous).

Note taking as parrotting is distinct from note taking as distillation. The latter has much more chance of getting results with proper spaced repetition, since it will help establish a concrete foundation on which to build more knowledge in an easily retrievable manner. The former may or may not benefit from spaced repetition at all.

I would say if GPT distills successfully, while this isnt as good as self-distillation, it may still be useful.

If not, or worse, actively bullshits, it probably won't be much help.

Note taking with SRS may be greater than the sum of the parts.

Yeah. But hey! You can only recall what you learned in the first place, right?

Could you elaborate on the definition of 'low utility'? Is it only in the context of memorization (retaining), or in the context of general understsnding?

Can't see how summarizing isn't a preprocessing step for learning in general. Spaced repetition is a natural next step. I don't see how they compete