This is misguided, naive, and likely to end in multiple potential failure outcomes that may be unacceptable, both from a profit standpoint; as well as a societal standpoint.
Education is a part of what gets adopted into each and every student's self concept (identity). There is an uncanny valley in communication where any inconsistent distortion of reflected appraisal can lead students down a path of irrational madness, in a way that they themselves cannot perceive. This is well known in certain circles.
AI always has the biases which are programmed into it by the creators of the AI, whether this is explicit or implicit, the fact remains, there are biases.
AI in education will be proven to be harmful because there is no possible way that AI can remain consistent in its interactions at all times. We already know this from hallucinations where experts went and fact checked it and found it had lied, where lawyers were disbarred or censured.
Put plainly, this is a safety critical system involving our children.
Given the level of care to date that's occurred with AI in industry, there is no room for discussion or adoption of these tools, unless the actual unspoken intention is to drive your children crazy; where they self eliminate later as a result.
Rationally, good people don't create a world of intolerable suffering and then magically expect that somehow, someone, will figure out a way to fix everything you've broken but were too blind to see; at the time when knowing would save lives.
For those who enjoy cinema, a perfect analogy of this might be the Day After Tomorrow, where the main scientist turns out to be correct, but was ultimately ignored until devastating losses and risks were nearly insurmountable.
Leadership didn't want to hear the facts when knowing would have saved lives, and so they had to take a triage on the battlefield approach (abandoning half the country). While it is a fictional story, it sufficiently demonstrates what lack of scientific or rational foresight can lead to for those that can't imagine it themselves (people who are blinded like that should never be in positions of influence or power).
“ There is an uncanny valley in communication where any inconsistent distortion of reflected appraisal can lead students down a path of irrational madness, in a way that they themselves cannot perceive. This is well known in certain circles.”
Foundational material comes from 1940s and 1950s wartime records of torture done on individuals during WW2 and Korean Conflict (under Mao).
You can find the raw case studies in the literature:
Robert Lifton "Thought Reform & The psychology of totalism";
John Meerloo "Rape of the Mind" covers the methodology often used along with knowledge at the time that physical coercion is not effective compared to mental coercion.
There are a number of more modern references. This is just to get you started (as an overview), with the least amount of ambiguity.
Torture induces vulnerable mental states which can create lasting changes in combination with a little knowledge of our (humanities) psychology.
I believe Cialdini covers the other lever's of influence in his book on Influence. These can all be used in any perceived communications system. He directly mentions PoW camps in his chapter on Consistency (with a lot of additional references).
Most modern documented cases of torture involve mental coercion, absent most physical coercion. This may take the form of varying and intermittent loud noises, bright lights (meant to disorient and confuse), isolation, sleep deprivation, etc which are intended to induce hypnotic states. There may also be covert hypnosis techniques that can be used as well, not excluding modern pharmacology derivatives based in narco-analysis (which during WW2 were known to not be effective), or drug dependency (not related to the current discussion).
The subject may become psychotic/violent, or withdraw (disassociate) becoming non-responsive, from extended exposure to mental coercion. Only a very rare few may be cursory unaffected (but still be hallowed out somewhat). Largely this is just a function of time and exposure.
In many cases all cohorts will to a degree adopt characteristics of their torturer (through distorted reflected appraisal), or whatever they promote (towards the promise or hope of relief, which is not physical relief).
Additionally, this material also includes the process of breaking down an individuals self-concept. As an example through destructive interference, by forcing the subject into acts that violate their deeply held beliefs, Abu Ghraib comes to mind, an example might be subscribing deeply to jewish traditions, and then being fed meat in your food without your knowledge (until later) where invectives and humiliation are induced. Overall, this subject matter is some of the darkest evil stuff you'd ever want to know, but it is entirely necessary to know in order to recognize and put a stop to it.
Most people don't recognize how evil works in reality, it works through willful blindness, and continues until its stopped, and perception requires you to not be blind to alert to that fact. Those who have blinded themselves are incapable of stopping.
Many of these structures, tactics, and techniques that originate in torture, form the basis for what is used today very commonly in advertising and marketing, as well as subversion and propaganda.
The reality is its fairly simple to break people with certain structures. One such structure is prompting for confession to induce consistency traps, then using that to mold the person to a certain narrative in a circular loop.
The use of the hot potato in the classroom, for open-ended questions (opinion) is one such example where this material has found purchase in K12 education.
Often, these elements are subtle, and are non-alerting; but demonstrably effective. What I've mentioned is just scratching the surface.
Edit: on a side note:
There is an interesting, and somewhat convincing argument to be made in this area related to active shooters. It may be that these related crimes may in some part be linked to the mental coercive spirals that are regularly being induced in our population. Bullying, Harassment and 'Silencing'/Shunning, and Discrimination (of any kind) for example are forms of social (mental) coercion.
While it is not a causal link, nor does it excuse any outcomes (which are unjustifiable, and worthy of condemnation). It is certainly plausible given the background knowledge in this subject area and some basic a priori reasoning.
Inducements through ads, and other isolating tactics may be ephemeral data that is not collected.
Given the risks, it certainly should merit much further scientific investigation and funding than it currently receives. It is well known that tortured people eventually break.
For simple lookup (Author/ISBN):
Robert Lifton, 0393002217
Meerloo, 9781614277873
Cialdini, 9780062937650
The basis for Reflected Appraisal is examined in Mead, 1934; Cooley, 1902; Sullivan, 1947, Felson, 1985. There has been some research more recently suggesting there may be a physiological mechanism for this process in neuronal structures though one must always consider the source for credibility (DOI:10.1016/j.biopsycho.2015.06.017).
Self Concept and Reflected appraisal Theory form a large chunk of the curricula under Intro to Communications courses at the college level, and a few other fields.
Education is a part of what gets adopted into each and every student's self concept (identity). There is an uncanny valley in communication where any inconsistent distortion of reflected appraisal can lead students down a path of irrational madness, in a way that they themselves cannot perceive. This is well known in certain circles.
AI always has the biases which are programmed into it by the creators of the AI, whether this is explicit or implicit, the fact remains, there are biases.
AI in education will be proven to be harmful because there is no possible way that AI can remain consistent in its interactions at all times. We already know this from hallucinations where experts went and fact checked it and found it had lied, where lawyers were disbarred or censured.
Put plainly, this is a safety critical system involving our children.
Given the level of care to date that's occurred with AI in industry, there is no room for discussion or adoption of these tools, unless the actual unspoken intention is to drive your children crazy; where they self eliminate later as a result.
Rationally, good people don't create a world of intolerable suffering and then magically expect that somehow, someone, will figure out a way to fix everything you've broken but were too blind to see; at the time when knowing would save lives.
For those who enjoy cinema, a perfect analogy of this might be the Day After Tomorrow, where the main scientist turns out to be correct, but was ultimately ignored until devastating losses and risks were nearly insurmountable.
Leadership didn't want to hear the facts when knowing would have saved lives, and so they had to take a triage on the battlefield approach (abandoning half the country). While it is a fictional story, it sufficiently demonstrates what lack of scientific or rational foresight can lead to for those that can't imagine it themselves (people who are blinded like that should never be in positions of influence or power).