Asking LLMs to do tasks like this and expecting any useful result is mind boggling to me.
The LLM is going to guess at what a human on the internet may have said in response, nothing more. We haven't solved interpretability and we don't actually know how these things work, stop believing the marketing that they "reason" or are anything comparable to human intelligence.
AI is not the problem here, because it has merely learned what humans in the same position would do. The difference is that AI makes these biases more visible, because you can feed it resumes all day and create a statistic, whereas the same experiment cannot realistically be done with a human hiring manager.
I don't think that's the case. It's true that AI models are trained to mimic human speech, but that's not all there is to it. The people making the models have discretion over what goes into the training set and what doesn't. Furthermore they will do some alignment step afterwards to make the AI have the desired opinions. This means that you can not count on the AI to be representative of what people in the same position would do.
It could be more biased or less biased. In all likelihood it differs from model to model.
> Furthermore they will do some alignment step afterwards to make the AI have the desired opinions.
This requires more clarification. It isn't really alignment work done at that point, or anywhere in the process, because we haven't figured out how to align the models to human desires. We haven't even figured out how to align among other humans.
At that step they are fine tuning the various controls used during inference until they are happy with the outputs given for specific inputs.
The model is still a black box, they're making somewhat educates guesses on how to adjust said knobs but they don't really know what changes internally and they definitely don't know intent (if the LLM has developed intent).
These models as we understand them also don't have opinions and can't themselves be biased. Bias is recognized by us, but again its only based on the output as we don't know why any specific output was generated. An LLM may output something most people would read as racist, for example, but that says nothing of why the output was generated and whether the model even really understands race as a concept or cared about it at all when answering.
> Asking LLMs to do tasks like this and expecting any useful result is mind boggling to me.
Most of the people who are very interested in using LLM/generative media are very open about the fact that they don't care about the results. If they did, they wouldn't outsource them to a random media generator.
And for a certain kind of hiring manager in a certain kind of firm that regularly finds itself on the wrong end of discrimination notices, they'd probably use this for the exact reason it's posted about here, because it lets them launder decision-making through an entity that (probably?) won't get them sued and will produce the biased decisions they want. "Our hiring decisions can't be racist! A computer made them."
Look out for tons of firms in the FIRE sector doing the exact same thing for the exact same reason, except not just hiring decisions: insurance policies that exclude the things you're most likely to need claims for, which will be sold as: "personalized coverage just for you!" Or perhaps you'll be denied a mortgage because you come from a ZIP code that denotes you're more likely than most to be in poverty for life, and the banks' AI marks you as "high risk." Fantastic new vectors for systemic discrimination, with the plausible deniability to ensure victims will never see justice.
> We haven't solved interpretability and we don't actually know how these things work
But right above this you made a statement about how they work. You can’t claim we know how they work to support your opinion, and then claim we don’t to break down the opposite opinion
I can intuit that you hated me the moment you saw me at the interview. Because I've observed how hatred works, and I have a decent Theory of Mind model of the human condition.
I can't tell if you hate me because I'm Arab, if it's because I'm male, if it's because I cut you off in traffic yesterday, if it's because my mustache reminds you of a sexual assault you suffered last May, if it's because my breath stinks of garlic today, if it's because I'm wearing Crocs, if it's because you didn't like my greeting, if it's because you already decided to hire your friend's nephew and despise the waste of time you have to spend on the interview process, if it's because you had an employee five years ago with my last name and you had a bad experience with them, if it's because I do most of my work in a programming language that you have dogmatic disagreements with, if it's because I got started in a coding bootcamp and you consider those inferior, if one of my references decided to talk shit about me, or if I'm just grossly underqualified based on my resume and you can't believe I had the balls to apply.
Some of those rationales have Strong Legal Implications.
When asked to explain rationales, these LLMs are observed to lie frequently.
The default for machine intelligence is to incorporate all information available and search for correlations that raise the performance against a goal metric, including information that humans are legally forbidden to consider like protected class status. LLM agent models have also been observed to seek out this additional information, use it, and then lie about it (see: EXIF tags).
Another problem is that machine intelligence works best when provided with trillions of similar training inputs with non-noisy goal metrics. Hiring is a very poorly generalizable problem, and the struggles of hiring a shift manager at Taco Bell are just Different from the struggles of hiring a plumber to build an irrigation trunkline or the struggles of hiring a personal assistant to follow you around or the struggles of hiring the VP reporting to the CTO. Before LLMs they were so different as to be laughable; After LLMs they are still different, but the LLM can convincingly lie to you that it has expertise in each one.
A really good paper I read last year from 1996 helped me grasp some of what is going only: Brave.Net.World [1]. In short, when the Internet first started to grow, the information that was presented on it was controlled by an elitist group with either the financial support or genuine interest in hosting the material. As the Internet became more widespread that information became "democratized", or more differing opinions were able to get supported with the Internet.
As we move on to LLMs becoming the primary source of information, we're currently experiencing a similar behavior. People are critical about what kind of information is getting supported, but only those with the money or knowledge of methods (coders building more tech-oriented agents) are supporting LLM growth. It won't become democratized until someone produces a consumer-grade model that fits our own world views.
And that last part is giving a lot of people a significant number of headaches, but its the truth. LLMs' conversational method is what I prefer to the ad-driven / recommendation engine hellscape of modern Internet. But the counterpoint to that is people won't use LLMs if they can't use it how they want (similar to Right to Repair pushes).
Will the LLM lie to you? Sure, but Pepsi commercials promise a happy, peaceful life. Doesn't that make an advertisement a lie too? If you mean lie on a grander world view scale, I get the concerns but remember my initial claim - "people won't use LLMs if the can't use it how they want". Those are prebaked opinions they already have about the world and the majority of LLM use cases aren't meant to challenge them but support them.
> When asked to explain rationales, these LLMs are observed to lie frequently.
It's not that they "lie" they can't know. LLM lives in the movie Dark City, some frozen mind formed from other peoples (written) memories. :P The LLM doesn't know itself, it's never even seen itself.
At best it can do is cook up retroactive justifications like you might cook up for the actions of a third party. It can be fun to demonstrate, edit the LLMs own chat output to make it say something dumb and ask why it did and watch it gaslight you. My favorite is when it says it was making a joke to tell if I was paying attention. It certainly won't say "because you edited my output".
Because of the internal complexity, I can't say that what an LLM does and its justifications are entirely uncorrelated. But they're not far from uncorrelated.
The cool thing you can do with an LLM is probe them with counterfactuals. You can't rerun the exact same interview without the garlic breath. That's kind cool, also probably a huge liability since it may well be for any close comparison there is a series of innocuous changes that flip it, even ones suggesting exclusion over protected reasons.
Seems like litigation bait to me, even if we assume the LLM worked extremely fairly and accurately.
> In their study, Moss-Racusin and her colleagues created a fictitious resume of an applicant for a lab manager position. Two versions of the resume were produced that varied in only one, very significant, detail: the name at the top. One applicant was named Jennifer and the other John. Moss-Racusin and her colleagues then asked STEM professors from across the country to assess the resume. Over one hundred biologists, chemists, and physicists at academic institutions agreed to do so. Each scientist was randomly assigned to review either Jennifer or John's resume.
> The results were surprising—they show that the decision makers did not evaluate the resume purely on its merits. Despite having the exact same qualifications and experience as John, Jennifer was perceived as significantly less competent. As a result, Jenifer experienced a number of disadvantages that would have hindered her career advancement if she were a real applicant. Because they perceived the female candidate as less competent, the scientists in the study were less willing to mentor Jennifer or to hire her as a lab manager. They also recommended paying her a lower salary. Jennifer was offered, on average, $4,000 per year (13%) less than John.
A replication was attempted, and it found the exact opposite (with a bigger data set) of what the original study found, i.e. women were favored, not discriminated against:
Except that the Ceci/Williams study is (a) more recent (b) has a much larger sample size and (c) shows a larger effect. It is also arguably a much better designed study.
Yet, Moss-Racusin gets cited a lot more.
Because it fits the dominant narrative, whereas the better Ceci/Williams study contradicts the dominant narrative.
More here:
Scientific Bias in Favor of Studies Finding Gender Bias --
Studies that find bias against women often get disproportionate attention.
The effect is wider and stronger than that: These findings are especially striking given that other research shows it is more difficult for scholars to publish work that reflects conservative interests and perspectives. A 1985 study in the American Psychologist, for example, assessed the outcomes of research proposals submitted to human subject committees. Some of the proposals were aimed at studying job discrimination against racial minorities, women, short people, and those who are obese. Other proposals set out to study "reverse discrimination" against whites. All of the proposals, however, offered identical research designs. The study found that the proposals on reverse discrimination were the hardest to get approved, often because their research designs were scrutinized more thoroughly. In some cases, though, the reviewers raised explicitly political concerns; as one reviewer argued, "The findings could set affirmative action back 20 years if it came out that women were asked to interview more often for managerial positions than men with a stronger vitae." [1,2]
Meaning that, first, such research is less likely to be proposed (human subject committees are drawn from researchers, so they share biases), then it is less likely to be funded, and finally, it receives less attention.
[2] Human subjects review, personal values, and the regulation of social science research. Ceci, S. J., Peters, D., & Plotkin, J. (1985). Human subjects review, personal values, and the regulation of social science research. American Psychologist, 40(9), 994–1002. https://doi.org/10.1037/0003-066X.40.9.994
Yeah, one glaring example of this effect is the NSFGears project.
The researchers studied why people leave engineering. Their first report, Stemming the Tide, reported on women. It was published and very widely reported. The reporting was largely inaccurate, because the claims that were made were that women were leaving due to discrimination.
If you actually looked at the numbers, that was totally false. The number 1 reason was "didn't like engineering", followed by "too few chances for advancement" and "wanted to start a family".
And of course, being promoted to management was als considered "leaving engineering". But whatever.
That wasn't the kicker. The kicker was that they did a follow-up study on why men left engineering. And it turns out it's for pretty much exactly the same reasons!
Our early analysis suggests that men and women actually appear to leave engineering at roughly the same rate and endorse the same reasons for leaving. Namely, that there were little opportunities for advancement, perceptions of a lack of a supportive organization, lost interests in the field, and conflicts with supervisors. One key difference between men and women was women wanted to leave the workforce to spend time with family
Ba da dum.
And yes, you guessed it: deafening silence. More than a decade later, nothing has been published. My guess is that they can't get it published. It's the same researchers, same topic, at least the same relevance, presumably same quality of work. But it doesn't fit the narrative.
I contacted the principal investigator a number of years ago, and she said to wait a little, they were in the process of getting things published. Since then: crickets.
Of course, you truncated only the last sentence of the analysis summary, which contradicted your narrative:
As we dug deeper into this relationship, we found that these women often attempted to make accommodations at work in order to meet their care-giving responsibilities only to be met with resistance from the work environment.
When you have different inputs and different outputs, that's not discrimination.
2. There is no contradiction (II)
The point I was making was about what gets published. It didn't get published.
3. There is no contradiction (III)
Please look again at the introductory sentence of the report: men and women actually appear to leave engineering at roughly the same rate and endorse the same reasons for leaving
"Roughly the same rates for the same reasons".
I repeat: "roughly the same rates for the same reasons".
That it isn't exactly the same rates and not exactly the same reasons is not a contradiction, because "exactly" wasn't claimed. It is added nuance and details that does not in any way shape or form contradict the original finding.
And it isn't "my narrative", it is what the researchers found. So it ain't a "narrative", it is the empirical data, and it isn't "mine", it is the reality as found by those researchers.
disparate impact, judicial theory developed in the United States that allows challenges to employment or educational practices that are nondiscriminatory on their face but have a disproportionately negative effect on members of legally protected groups.
For example, if a company's policy is "No employee is allowed to pump breast milk anywhere on premises, even behind closed doors, regardless of gender," it disproportionately impacts women even if men are also banned from the same activity.
Perhaps I am aware that, to add the part of the article you left out:
However, civil rights advocates have been disappointed as federal courts have increasingly limited how and when plaintiffs may file disparate-impact claims. As a result, disparate-impact suits have become less successful over time.
So it's a fairly fringe legal theory with little impact.
There are lots of fringe theories, for example some claim that the earth is flat. I don't have to accommodate all of them.
Disparate impact is illegal, so it's not a "fringe legal theory".
If you don't see anything wrong with my example of disparate impact, how about a hypothetical company policy that has a dress code of short hair for all engineers regardless of gender? More women than men would quit, seeing the policy as draconian and controlling (or be fired for non-compliance), while men who already have short hair wouldn't find the policy onerous or difficult.
"federal courts have increasingly limited how and when plaintiffs may file disparate-impact claims. As a result, disparate-impact suits have become less successful over time."
Also your source.
3. Off topic
a) The research I cited was not about fringe legal theories but about reality in the world.
b) I am not interested in your hypotheticals that have nothing to do with that research, nothing to do with the publishing bias against research showing no bias against women or bias against men, and probably also nothing to do with the actual legal theory of disparate impact.
It adds context, but doesn't contradict anything - the resistance they faced was to actions that their male peers didn't attempt, so it doesn't imply any kind of disparate treatment.
The comment implied that women left engineering because they preferred taking care of children over working as engineers. The context is that they wanted to choose both, but their work didn't allow it. If young children exist and are neglected, then society blames the mother, not the father. A responsible mother has no choice but to choose family over career if she can't choose both. Young humans cannot survive on their own without being cared for by adult humans.
> The comment implied that women left engineering because they preferred taking care of children over working as engineers.
That turns out not to be the case.
1. It wasn't "implied"
There were no implications, things were said straight out.
2. It wasn't "the comment" that didn't imply this
This was a statement by the researchers quoted verbatim.
3. It wasn't "the" reason
As the researchers stated: men and women actually appear to leave engineering at roughly the same rate and endorse the same reasons for leaving
So wanting to take care of children wasn't "the" reasons, and it wasn't even the main reason. It was one where men and women actually diverged, whereas for the most part they gave the same reasons.
4. Non-accomodation was a factor
> The context is that they wanted to choose both, but their work didn't allow it.
That is also not true as written. First, the researchers write "often", which you leave out. Second the researchers write "resistance", you write "didn't allow". Those are not the same thing.
Third, the report clearly states "women wanted to leave the workforce to spend time with family". Wanted. Not "were forced to by societal pressures".
And of course those pressures are identical for men and women, if not stronger for men. When I started working part time in order to have time for my daughter, there was an almost immediate attempt to push me out, stopped only by my team revolting, and it was made clear to me that I would not be advancing, that my career was if not over than at least dead in the water.
And at some level that is actually correct. Once I had my daughter, my job was not just not my #1 priority, I physically did not have the same amount of time to give. This is not some evil discriminatory society, it is physics. The day has so many hours. So companies that often demand total dedication from their employees (especially in the US) simply won't get it from a caregiver.
Now I don't agree that that is a legitimate demand. But it is a common one that is made equally of all employees, non-discriminatorily.
Choosing family over career is a legitimate choice. It happens to be my choice. But it is a choice, and one I personally would make again and again, even though the punishment society doles out to men for that choice is much, much harsher.
I think it's important to be very specific when speaking about these things, because there seems to be a significant variation by place and time. You can't necessarily take a past study and generalize it to the present, nor can you necessarily take study from one country and apply it in another. The particular profession likely also plays a role.
You'd have to get a hold of a model that was simply tuned on its input data and hasn't been further tuned by someone who has a lot of motivation to twiddle with the results to determine if that was the case. There's a lot of perfectly rational reasons why the companies don't release such models: https://news.ycombinator.com/item?id=42972906
The LLM is going to guess at what a human on the internet may have said in response, nothing more. We haven't solved interpretability and we don't actually know how these things work, stop believing the marketing that they "reason" or are anything comparable to human intelligence.