I agree with you about redlining. I hadn't really thought about that... A recent scandal involving a predictive analytics firm called Northpointe shows how stupidly people use algorithms, and how much they are abused when there is no transparency: https://www.propublica.org/article/machine-bias-risk-assessm...

The scandal is that propublica wrote a story which directly contradicts their own statistical analysis. Their analysis was unable to show statistically significant bias (and had other flaws that might be genuine mistakes - no multiple comparison correction, possible misinterpretation of the model).

https://www.chrisstucchio.com/blog/2016/propublica_is_lying....

In fact, their r-script suggests that the secret algorithm is probably reducing racism relative to whatever secret algorithm human judges would use.

Isn't that explicitly what OpenAI was formed to prevent?

The OpenAI team would be best to comment but I've not seen them note any specific intent[1] to research topics related to fairness, accountability and transparency in ML. I use those words specifically as FAT ML 2015 was part of ICML 2015[2]. While a great start, that area of research doesn't get anywhere near the attention it should given the gravity of the potential findings.

These issues are already in existence. Given ML systems are being used to filter resumes, decide whether someone should get a loan (and how much their interest rate should be), and find matches on dating services, the fact that ML systems may inadvertently feature racial or gender bias is hugely disturbing. This is likely to only get worse over time as more and more systems feature ML components.

As a simple example, word vectors are used by just about every deep learning system and recent research has found gender and racial stereotypes strongly embedded in them[3]. There's very confusing debate as to what this implies for the data, the models, and predictions[4]... :S

father:doctor :: mother:nurse, man:programmer :: woman:housemaker

black_male>assaulted, whilte_male>entitled_to

I wish I was making those examples up but they're straight out of their analysis. They also only focus on gender stereotypes but you can imagine how many similar issues might be hidden away just beneath the surface.

Even if OpenAI indicated they were explicitly interested in that direction, which afaik they haven't, it's still an area that should be of interest to the field broadly. OpenAI is still a small team and I'm certain they'd appreciate the help, especially as they already have a lot on their plate :)

A disaster scenario for me is to have ML systems help reinforce the negative aspects of our society, conveniently hidden in a black box which can never be properly inspected.

[1]: I'm primarily running off what I've seen in the recent past and their technical goals that they recently published at https://openai.com/blog/openai-technical-goals/

[2]: http://www.fatml.org/

[3]: "Quantifying and Reducing Stereotypes in Word Embeddings" - https://arxiv.org/abs/1606.06121

[4]: https://twitter.com/jackclarksf/status/746039805595762688

According to this paper https://arxiv.org/pdf/1606.06565.pdf, OpenAI (John Schulman and Paul Christiano) explicitly indicate interest in researching transparency and strongly support work on fairness.

Thanks for the response, I had misinterpreted what you meant by redlining. I thought you were referring to the unequal distribution of the technology among various populations (access) as opposed to the reinforcement of existing redlines (misuse).

That's where the right to privacy comes in. The input should not be there to begin with.