This is a really good time to be a Independent Scientist (aka Gentleman scientist) in this field because how nascent deep learning and similar techniques are. It requires a lot of trial and error and time/cost investment to bring the AI techniques to the masses.
The FAANGs are trying to hire all the top talent (including Emil who wrote the post) but I believe these independent researchers will be the one finding new opportunities to make AI useful in the real world (like colorizing b&w photos, create website code from mockups).
The biggest challenge I see for these folks is the access to high quality data. There is a reason Google is releasing so many ML models in production compared to smaller companies. Bridging the data gap requires effort from the community to build high quality open source datasets for common applications.
On the other hand, the lack of data for independent researchers may encourage the development of low data techniques which is much more exciting in the long term since humans are able to learn with much less data than required by most machine learning techniques
How it that useful for subsequent learning? The output is random words that doesn't even forms phrases or sentences and has no relation with the image.
Humans can transfer learn across domains because we can draw on an incredible wealth of past experience. We can understand and abstractly reason about the architecture of problem landscapes and map our understanding into new spaces.
That isn't even counting our hardwired animal intelligence.
Humans learn efficiently, but that has nothing to do with having a lifetime of data I believe.
Humans have a lifetime of data, but you can easily parallelize the model such that it takes in more data than a human can in his/her lifetime, and still humans are still the state of the art.
...which fits into a size of less than 700Mb compressed. Some of the most exciting stories I've read recently for machine learning are cases where learning is re-used between different problems. Strip off a few layers, do minimal re-training and it learns a new problem, quickly. In the next decade, I can easily see some unanticipated techniques blowing the lid off this field.
It indeed strikes me as particularly domain-narrow when I hear neuro or ML scientists claim as self-evident that "humans can learn new stuff with just a few examples!.." when the hardware upon which said learning takes place has been exposed to such 'examples' likely trillions of times over billions of years before — encoded as DNA and whatever else runs the 'make' command on us.
The usual corollary (that ML should "therefore" be able to learn with a few examples) may only apply, as I see it, if we somehow encode previous "learning" about the problem in very the structure (architecture, hardware, design) of the model itself.
It's really intuition based on 'natural' evolution, but I think you don't get to train much "intelligence" in 1 generation of being, however complex your being might be (or else humans would be rising exponentially in intelligence every generation by now, and think of what that means to the symmetrical assumption about silicon-based intelligence).
"The usual corollary (that ML should "therefore" be able to learn with a few examples) may only apply, as I see it, if we somehow encode previous "learning" about the problem in very the structure (architecture, hardware, design) of the model itself."
Yes, and they do. They aren't choosing completely arbitrary algorithms when they attempt to solve a ML problem, they are typically using approaches that have already been proven to work well on related problems, or at least are variants of proven approaches.
The question is, how much information is encoded in those algos (to me, low-order logical truths about a few elementary variables, low degree of freedom for the system overall), compared to how much information is encoded in the "algos of the human brain" (and actually the whole body, if we admit that intelligence has little motivation to emerge if there's no signal to process and no action to ever be taken).
I was merely pointing out this outstanding asymmetry, as I see it, and the unfairness of judging our AI progress (or setting goals for it) relatively to anything even remotely close to evolved species, in terms of end-result behavior, emergent high-level observations.
Think of it this way: a tiny neural net (equivalent to the brain of what, not even an insect?) "generationally evolved" enough by us to be able to recognize cats and license numbers and process human speech and suggest songs and whatnot is really not too shabby. I'd call it monumental successs to be able to focus a NN so well on a vertical skill. But that's also low-order low-freedom, in the grander scheme of things, and "focus" (verticality) is just one aspect of intelligence (e.g. the raging battle is for "context" as we speak, horizontality and sequentiality of knowledge; and you can see how the concept of "awareness", even just mechanical, lies behind that). So, many more steps to go. So vastly much more to encode in our models before they're able to take a lesson in one standing and a few examples.
It really took big-big-big data for evolution to do it, anyway, and we're speeding that up thanks to focus in design, and electronics to hasten information processing, but not fundamentally changing the law of neural evolution, it seems.
If you ask me, the next step is to encode structural information in the neuron itself, as a machine or even network thereof, because that's how biology does it (the "dumb" logic gate transistor model is definitely wrong on all accounts, too simplistic). Seems like the next obvious move, architecturally.
Agree with your first statement and disagree with your second; I don’t think the former implies the latter.
I think there’s a lot of room to be clever with encoding domain-specific inductive biases into models/algorithms, such that they can perform fast+robust inference. Exploiting this trade off as a design parameter to be tuned, rather than sitting at one of the two extremes is potentially going to generate a lot of value. And this is highly under-appreciated currently since most people are obsessed with “data”. I’m willing to bet that this will become big in a few years when the current AI hype machine falters, and will serve as a huge competitive advantage.
These types of techniques are already big in certain fields. E.g., in fluid dynamics and heat transfer, "dimensional analysis" is frequently used to simplify and generalize models. Sometimes models can be nearly fully specified up to a constant of proportionality based solely on dimensional considerations. Beyond what is typically seen as "data" the information here is a list of variables involved in the problem and the dimensions of the variables.
As far as I can tell "dimensions" in this sense are a purely human construct. For two variables to have different dimensions, it means that they can not be meaningfully added, e.g., apples and oranges.
This would be a great area, IMHO, for the government to step in and fund an initiative to provide huge, rich datasets for anyone to use for ML research.
wrt the data point, to be fair most research is still coming out of universities where students have access to the same data as anyone else. So from a research perspective it's not a huge deal, much as with compute industry can scale up known techniques while individual researchers do more interesting stuff.
So if I understand correctly, to reformulate in my own words/views:
while the "big data" (datasets) formed and thus owned by big-tech, big-ads, big-brother, etc. may be instrumental to build at-scale solutions for real-world usage (for profit, knowledge, control, whatever actionable goal),
fundamental research itself, as done in universities, can move forward without these datasets: using what's publicly available is enough.
Did I read this right? It would effectively add much needed nuance to the common perception that big data is necessary to train innovative models, that there might be some sort of monopoly on oil (data, the 'fuel' of ML) by a few champions of data collection.
It's not exactly true that research institutions don't have access to the same big datasets as companies. For example, I took a course that involved tracking soccer players using videos provided by a streaming company that specializes in amateur soccer. They promised to give us access to their internal API under an NDA, which they wouldn't have done for just anyone.
On the other hand, they never actually gave our API keys the necessary privileges, so in the end I just reverse-engineered the URL scheme of their streams and scraped them. Many datasets used in academia are just collections of publicly available data (e.g. Wikipedia, images found by googling), optionally annotated for cheap using Amazon Mechanical Turk. Experimenting with that kind of data is also open to independent researchers. You don't need to work at a data-hoarding company if you can get what you need by scraping their website.
yep, you read that right. Source: I am a PhD student at Stanford at the Stanford Vision and Learning lab (http://svl.stanford.edu/) and read a ton of AI papers. The vast majority of papers are done with datasets anyone can just download / request, as far as I've seen.
personally, without affiliation to a university, I have a hard time downloading the datasets through my slow home connection. I live in a city with a university, I explained the situation but they won't let me download a dataset even if I pay, ... only when I enroll. Instead of just selling the shovel, they want to sell me the wheelbarrow too.
I succeeded one time in convincing the guy behind a desk in an internet cafe, so I could bring my HDD and download a dataset in a calmer time of day, and throttled so it wouldn't disturb other customers. This went without any problems for the other customers in the internet cafe. When I asked again a few months later for a new dataset, they no longer wanted me to do so...
There seems to be no download by mail service (and I only get people forwarding me to google cloud products etc, which as a European is so financially out there with automatic balance deductions and non transparent pricing schemes, I would have no qualms using GCP or others if they ran a prepaid alternative for people who refuse to take on risk)
A lot of research data sets are publicly available, but many researchers based at universities have relationships with private companies where they can get access to data or other resources useful for research (e.g. Google has a big room of robotic arms generating data for pick and place tasks).
There is still plenty you can do with a reasonable personal budget, however.
You will have unlimited training data. But its very difficult task even for humans. Its like trying to reverse a hash. Also a lot of information is lost when you store a color digitally.
The problem with Emil's approach to learning is that it restricts his ability to learn anything that he has no intrinsic goal off. That includes areas like pure mathematics, theoretical computer science, finance, economics, literature, etc. Those subjects require a different sort of motivation than a motivation to just achieve a set goal i.e capitalistic motivation
Also, Emil's approach to learning will create a flawed sense of expertise. Look at how the article presents him as if he has a deep-domain expertise which might not be true.
One important thing to consider is to look at the article more like content marketing tactic, that FloydHub is using promote its brand which might not serve well for engineers as it lacks some aspect of truth.
>Those subjects require a different sort of motivation than a motivation
Is that really the case? Apart from the obvious "maybe that's what they're intrinsically interested in" - if you start with a problem, try to solve it, and "pure mathematics" (whatever exactly that is supposed to be, anyway) is required to arrive at a solution, it becomes part of the intrinsic motivation. And if you keep solving problems without it the question that eventually comes up is "is it really useful at that point?"
I do however agree that if you're looking at someone who's qualification is primarily his "portfolio", you do actually need to check whether it includes interesting problems, or at least projects that are related to what you need them to do at your company. But if that is the case, I really don't see a problem.
That's correct. If one is solving a problem and its requires learning of complex topics, then those topics become a part of the individual's intrinsic motivation. However, the problem with his idea is that learning always requires a goal and ones that do not have any have no intrinsic value. My argument here is that learning doesn't have be attached to any goal and the process of learning can purely be a stimulating activity and might not necessarily have any capitalistic value attached to it.
Few ex:
Like, reading literature is a purely fun and mentally draining activity that might/might not have any goal attached to it.
Like travelling, is a purely fun activity and might/might not have any intrinsic goal attached to it.
I started learning Algebra out of the random, without any intrinsic goal. Because, it was purely out of fun. Playing chess is an activity without any intrinsic goal.
For some people it does. Everyone has a different approach to learning and I think outside of the heavily regimented strata of traditional schools it would help everyone if we could celebrate that. Maybe that means not everyone learns for the sake of learning, but then again not everyone has to do the same things.
>might not necessarily have any capitalistic value attached to it.
You seem to use goals and "capitalistic value" interchangeably, I would just like to add that projects in people's portfolios are often not initially created to make money, but simply to satisfy the creator's curiosity or because they thought it would be neat.
> I would just like to add that projects in people's portfolios are often not initially created to make money
We both can agree on this. But that's not how Emil looks at it, his entire point is that projects are better credentials than even degrees.
And since projects have higher value in portfolios, they will implicitly derive capital? One of the reasons why they have capitalistic value.
You and I can both agree, that projects and learning do have fun elements attached to them and might not necessarily be part of a larger goal. However, that's not what his views are!
It's not a zero sum situation. Him taking this approach does not restrict someone else from studying what they want.
Also finance... really? I'm not sure I would agree that is a subject with a less capitalistic motivation.
I don't get a lot of the bitterness here. I mean not everyone here creates their own programming language, or would know how to write a database, yet we are fine using them as tools. Why must we understand all the code and concepts in a neural network to apply it?
Not necessarily bitterness! I'm very happy for what he is doing. I'm looking at it very objectively, the problem is that the way the article presents his story is very flawed and incorrect. It sounds very much like a PR post. To address the Finance point, there are few areas of finance that is purely research driven and requires a lot of expertise of lateral domain like economics, mathematics. And once cannot grok that expertise in a year or so.. That was the point I was making!
> Why must we understand all the code and concepts to apply it?
That is one of the reason why our engineers are so sub-par because we were told to just shut up and write shit. We could have become a force to be reckoned with because of our expertise, because of our ability to solve complex problems. Yet we are living in an industry were there is huge disparity in salary, structure, and principles.
I don't necessarily agree with why one shouldn't understand the concepts, I'm more a guy who says why not?
Because, if you are standing on the shoulders of pioneers and claiming to be improving their work at least do it with compassion and honesty.
Sorry about the rant.. would love to hear alternative opinions!
This was a great read (and great nuggets, like that paper on Intelligence by Chollet).
I wonder:
— Is math a problem for non-academic researchers?
Most papers strike me as requiring a non-trivial knowledge of linear algebra, for instance; and topology sits right behind; the bold seem to take it one up on category theory as we speak, and geometric algebra is quickly gaining traction too. Lots of math, cool math but math nonetheless.
Not that you can't learn these on your own, but how big is the gap in practice, on the job, compared with actual PhDs in ML/math? (how much of a hinderance, a problem it is for the self-taught researcher)
— "Contracting" in the field of AI sounds great but, how exactly? Especially solo: what type of clients and how/where to find them, what type of 'business proposition' as a freelancer do you offer, what's the pricing structure of such gigs?
I mean, I can sell you websites and visuals and stuff, but AI? I know first-hand most SMBs (IME the only real customers for freelancers) are a tough sell: their datasets are tiny and demand scripting skills to sort out (extract business value), not AI, so the value proposition is low for both parties; it's still early adoption so 90% don't even consider spending 1 cent on "AI" unless as a SaaS (they actually don't need to know if it's AI or programming).
I can imagine tons of fantastic research to do with SMBs, as partners or 'interested sponsors' (should they reap benefits on a low investment), but really not much yet in the way of "freelancer products" to market and sell for a living. I'm eagerly anticipating those days, but it's more like 2025-2030 as I see it.
It takes a while to figure out how to read academic papers, but it's largely about learning the notation. In the end, it maps back to the code you write anyway in most cases, so it's just another way of writing stuff you already know.
It's not so much linear algebra you need, since much of that is not relevant to AI. It's really matrix calculus. Which is largely about multiplying things together and adding them up. Terence Parr and I tried to create a "all you need to know" tutorial here: https://explained.ai/matrix-calculus/ .
You certainly don't need topology (unless you happen to be interested in that particular sub-field).
It might be wrong but I tend to see vectors and matrices as two notations for the same mathematical object[1]. So I indeed meant matrices! However I didn't see calculus itself as such a big requirement, as it all felt pretty "linear" to me (regressions etc). Are we talking things that e.g. "Calc 2"[2] should cover?
I feel reassured by your first paragraph. This can be done.
I'll definitely work on your tutorial; I assume it's a good benchmark for math pre-requisites in the field. Thanks a lot for the work, and advice.
> I think this is correct, if you consider college level linear algebra and an intuition for applying it to novel problems to be non-trivial knowledge
Yes, in the context of "a self-taught researcher", I think I intuitively meant anything that precisely requires a degree, typical academic knowledge. E.g. you can become a great business person who won't feel hindered by lack of academic knowledge, you definitely can't do that as a surgeon or lawyer.
I guess I was wondering where math fit in this picture for AI research. (which I should explicitely relate to "#2" in user ineedasername's post, i.e. "AI Research as examining the theoretical frameworks & approaches to ML/DL in a way that may itself lead to shifts in the understanding of ML/DL as a whole and/or develop fundamentally new tools for the purpose of #1 [AI Engineering Research]. What might be termed "basic" or "pure" research.")
> I guess I was wondering where math fit in this picture for AI research.
Well you can't do #1 or #2 without having a level of maths proficiency that most college grads do not have.
FWIW, I don't understand the difference between #1 and #2 above. Most academic/industrial research is incremental (i.e. #1), and a tiiiiny % will have any impact in the way something like XGBoost would (the example he gave in another comment). That doesn't mean that the non-impactful research isn't 'basic'. You could alternatively just call #2 "groundbreaking research" and #1 "non groundbreaking research, but you need mathematics knowledge for both imo.
I see what you mean. It seems possible the distinction is perhaps academic at best, a matter of perception. (I certainly don't have a personal opinion, yet! but point taken, and the continuity you speak of seems more realistic tbh).
To be honest, linear algebra is not that difficult to learn on your own, and plenty of people do. Gilbert Strang's course on OCW has made introductory linear algebra quite accessible.
Things like topology (e.g. TDA, persistent homology, etc.) aren't really mainstream yet, but even then most of it isn't really "hardcore" math in the sense that you can get away with a basic understanding, e.g. what a Vietoris-Rips complex is and why we use it instead of a Cech complex in TDA. Plus most DL research nowadays is pretty (advanced) math-light. That being said, taking the time to understand the math is absolutely worthwhile in my experience.
It should also be noted that a lot of real world ML/AI projects in industry aren't really about brand new algorithms using advanced math, but rather more about applying mostly existing techniques to messy, noisy real world data and taking the time to understand the domain you are applying it to.
I work as a data engineer, and i was interested in learning some of the stuff our data scientists do so i can better communicate with them. Teaching myself some statistics was fine, probability was fine too and quite fun and surprising. Both subjects have plenty of books that allow you to understand the intuition behind the things they do without having to dive deep into the proofs. Linear algebra was and still is a struggle though. I've sampled many books, from Strang's book to Linear Algebra Done Wrong/Right to some books that are used in the local university in CS courses. But they are all the same. It's clear they are all written by mathematicians for math students, probably to be used as a way to teach students how to write proofs at the same time? It's just one page after another of increasingly esoteric calculations and proof after proof after proof. Which is fine if you study math, but bad for me, because i dont want to work out the proof that taking the determinant of an inverted matrix works, i want to know what it means and why one would want to make the effort to do it.
Basically, i want a book like Statistical Rethinking or Blitzstein's Introduction to Probaiblity, but for linear algebra. And i havent been able to find it.
I think you might like my book, the No Bullshit Guide to Linear Algebra. It doesn't focus on the proofs, and instead gives lots of intuition and applications. You can check the reviews on amazon, and here is a link to a PDF with a few sample chapters: https://minireference.com/static/excerpts/noBSguide2LA_previ...
I won't lie to you and tell you linear algebra is "easy" by any means—there are a lot of things to pick up, so it takes some time, but it is totally worth it since LA is like the swiss-army knife of science: lots of features and super useful.
I self-taught myself linear algebra from Strang's book, combined with his lectures (on youtube). I wouldn't say that his approach is proof-heavy at all. Furthermore, the proofs in LA are rather easy and mechanical, compared to other areas of maths (and also, you can just skip them when reading).
Thanks for the pointer! (link[1] for those interested)
> you can get away with a basic understanding
Great news to me!
> taking the time to understand the math is absolutely worthwhile in my experience.
Strongly agree — for any topic, any field. My concerns are practical indeed, and less about the 10-year horizon (well enough to become skilled at anything) than the early stages of that, the best way to propel oneself far/fast enough on year 1, then 2, etc.
> applying mostly existing techniques to messy, noisy real world data and taking the time to understand the domain you are applying it to.
I hear that. I actually do like the sound of that, hence concerns that I was biased.
Linear algebra can be straightforward to learn on your own, if sufficiently motivated. But homology seems altogether different. The math undergrads who see it often have a tough enough time, I can't imagine an autodidact managing.
It's my understanding that dirty datasets that "demand scripting skills to sort out" is pretty common and most data scientists spend 80% of their time "sorting this out".
I'm not going to lie, his life story made me jealous. Extremely jealous. He did all the things I wanted to do (speaking in categories, not exact things) and is free to do more. It seems like in some cultures (mine is South Asian) there is a threshold on exploration time. Usually around the age of 28-30 years old (for some even lower than that, I consider myself one of the most fortunate ones). As I approach that number I feel the invisible hand of expectations and responsibilities crushing my spirit. But I must also remember comparisons on life scales don't really work and nobody can win neither the happiness Olympics nor the misery Olympics.
I'm not going to lie, his life story was written and presented with the precise goal in mind to make people feel this way. I stopped reading after half the article because I found it way too self-promotional.
So don't feel bad about your life just because someone on the internet pretends to have a more interesting one. Those people are usually just attention seekers and for some reason need the outside validation to feel good about their achievements. And remember that not needing that validation can be a strength too!
I don't want to dismiss somebody else's life experience because it looks too fancy. I have nothing against honest self promotion, it's someone's story and I clicked on the article after reading the headline. I have no indication that this person is pretending or faking it. Also, even without all the quirky things, there are plenty of things to be jealous of. Like all the different things he tried his hands at, the freedom to just up and leave a pretty successful thing.
I realize that these articles suffer from the connecting the dots thing, where people make connection in the present which they would never have in the past. But that is besides my point, even if he failed at all those things, I am still jealous he had the chance to try all these things.
May we go on a short tangent about a word and its associated feeling?
I submit to you that you don't sound jealous, because it really doesn't fit the rest of what you express; I suggest that you are maybe "envious¹" in the sense that jealousy means envy + depriving the other of what they have ("it should be me and not them", it's a matter of exclusivity, like being jealous of #1 if you finished second, or jealous of the one dating someone you love, or whoever took your job/offer). You don't sound hostile to them, merely wishing more for yourself (which in itself is a positive feeling?)
I really don't know what the word you meant to use in your own language actually meant (I'm French personally, so English is just our medium translation layer). But I'm curious, culturally you know, about these nuances².
I'd love it if you could just introspect that feeling a bit and share what it really feels like, the complex emotion and what it "touches" in you (does it bring despair, or motivation, or resolve, etc). [if it's too personal, I got email, just ask]
Personally, I can feel "aspiring to" or "inspiration from" people who achieved more than me — I don't want to deprive them of anything, I don't wish they failed, nor do I wish to belittle their accomplishments; however I'd love to eat everything they know, steal like the greatest of artists (the good ones merely copy!), that is at best become friends with these people and let them influence me (the true deeper meaning of that book³, if you ask me). And I know, somehow deep down, that the more I'd be rooting for these friends, helping them go further, the more I'd be moving forward/up as well, taken in by the positive storm.
[1]: envy is “a feeling of discontented or resentful longing aroused by someone else's possessions, qualities, or luck.”
[2]: Part of my (personal) research on human nature, motivations, "what makes us do what we do".
[3]: How to win friends and influence people, by Dale Carnegie
Honestly speaking, as english is not my first language, I confused the two words. I thought envious means to want to deprive others of what they have. I guess then I may have confused some people, I am envious but I think it doesn't detract from my overall point. The word in my language would literally translate as desire but it's more than that, it would mean more like I would like to have that and it feels bad that I don't even have the opportunity to have it.
I would definitely say this lack of even the opportunity to do this makes me feel despair. I can try as hard as I want, but some things are just out of my control, some truths about my life were written even before I was born. I honestly cannot muster up the strength to derive inspiration or motivation from these, because those things are relevant for things which are possible.
Rest assured that jalousy and envy are actually often confused my many people, even in one's own language. ;-)
> The word in my language would literally translate as desire but it's more than that, it would mean more like I would like to have that and it feels bad that I don't even have the opportunity to have it.
Yeah, OK, I get it. That's a very good word (the one in your language). I think it's a rather universal feeling, this "invisible ceiling". Many people feel that for various reasons.
There's a certain school of thought, somewhere between philosophy and spirituality, that speaks of "abundance", and beyond (or perhaps before, on the way) of "inner peace" or "inner happiness". The oldest forms I know are Stoicism (western cultures) and Zen (eastern cultures), and you'll find it nowadays in e.g. Tony Robbins, that kind of field. I think there's truth in it that just works, at least it did for me (took me about 35 years to figure it out though, as it's just totally outside the realm of "education" nowadays¹, unfortunately IMHO).
One mechanism that I've always found to be true, is that from the depth of our biggest despair comes our symmetrical potential for joy, and vice-versa. It takes knowing how good/bad it gets to really feel how worse/better it is, or rather goes.
It's certainly trying on one side, but invaluably rewarding on the other.
[1]: at least in most of the western world, afaik.
I think the limit is mostly set by having kids, and not necessarily age. My wife and I travel frequently (if you consider this exploration time, but I might have misunderstood). And in addition she is now going back to university for a third master after having worked for some time.
We're about the age you describe, late-20s, early-30s. But our friends of around the same age with kids seem to have a lot less of this 'freedom'. Responsibilities take over.
Not saying you can't do those things with kids - but it does seem harder.
Here in Northern Europe you're expected to do most of your traveling in your 20's, but we also have pretty decent vacations - so the solo / friend / backpacking type of traveling gets replaced with more family friendly stuff when you start getting kids.
I have lots of friends in their late 20's / early 30's that still travel the world, many times a year. But they don't have kids, and their travels (outside summer vacation) tend to be shorter, as in long-weekends etc.
It's not just about traveling. It's just one of the components. This guy was a teacher, musician, maker and now DL expert. And it doesn't seem like he is settled on that yet. He is still exploring what he wants to do. Changes his life to match his interests. I lived some of my life like that. Took some non standard paths. But it is getting increasingly hard for me to do this. Not because there aren't opportunities for me, but because it's getting unjustifiable to "society". And I know the usual line about "who cares what others think" in the west. Although it's logical in the west its not so in South Asia. Even if you don't care, your parents will and you will care for your parents. So it's inescapable. By Western standards I should be a totally free person, I am not married, don't have any debt, a high earning job which I'm bored by only 50% of the time. But these same things trap me. Single status leads to public derision in social gatherings (also friends get married and it becomes increasingly hard to be the only single guy in the group), my insistence on not taking on any debt means I live in a small rented apartment which it hard to be accepted in the society around me, a high earning job means I can never just quit because then I will have no social support due to it being a bad decision.
As I write this, my (admittedly limited) understanding of how Western society works makes me think these would not be problems but my biggest assets.
Let's be real here - and no, I'm trying to put this guy down:
He was hardly a qualified teacher. Lots of young people up here go on voluntourism trips to countries in Africa or Asia to teach a couple of months. If you want to try teaching, you can do so locally; There are lots of options.
Musician, sure, it's fun (I've toured in 4 different countries myself playing in band, though 15 years ago now), but it's hard to do full-time.
DL expert would be to push it, there's a long way between being productive and being an expert.
But I'm gonna be frank with you, doing all those kinds of things is possible here because we have a great welfare system. You can work, save aggressively, and do whatever you want.
The unfortunate fact of life here though, is that a lot of entrepreneurs and indie developers (just to pick a few) are funded by welfare checks.
> You can work, save aggressively, and do whatever you want.
If you think I can do that, I have definitely failed in explaining what the problem is. I have almost my post tax yearly salary as savings (apart from the investments and stuff), but due to the reasons explained in my other post I can't do shit.
Not everyone has the privilege of living out a bildungsroman/Jungian heroic journey. You're right, you sound like you have all the assets including the desire, but perhaps it's not the same as it's not occurring during your formative years and without peer support. Thankfully, you seem be self-conscious enough to be living your life for your self, instead of with the aim of peddling a brand.
You know, they (read: recruiters) say that if you don't have a normal background, you should have an interesting one. For some reason, if you don't have the same cookie-cutter background as everyone else, you need to have some amazing and convincing story to tell.
I think it's good that companies are willing to look into non-trad candidates, that may not have found their "calling", so to speak, until their late 20's / 30's or whatever. But it does start to sound contrived when a bunch of 'em have the same type of alternate-route stories, which involves traveling to Africa / India / SE Asia to help out kids, create some startup aimed at climate / poverty / equality / etc. I guess it makes you sound passionate and legit - no-one can say that you wasted your time on chasing those things.
I guess it can help sound passionate, but the list together doesn't sound interesting -- it sounds like an AI read a bunch of minimalism lifestyle blogs and output "interesting_backstory.txt".
Nothing on the quirks list is actually a quirk. They're interesting things he's done that other people wrote books about, received praise for, and then he followed their newer, well-traveled path.
It's not a non-traditional background. He's not a refugee who managed to learn coding. He's not volunteering at a needle exchange clinic. I think that's what's bothering me; he's pretending to be interesting, and taking the room which could be going to someone else.
Thank you for helping me get to why something felt off. Appreciated, internet stranger.
Not enough, since his entire gimmick appears to be 'selling himself' and it's gotten him gainful employment at somewhere that you think would know better
"Many are realizing that education is a zero-sum credential game."
Can this silly meme die already? Maybe it's understandable coming from an economist who values education for no other reason than it's economic effects, but it's strange coming from someone who clearly understands the value of personal development.
My prediction is that whoever comes up with the next forward leap in AI will be someone who at minimum has a firm grasp on the various branches of undergraduate level maths. Naively tinkering with heuristic statistical ML methods like neural nets and hoping that higher level intelligence somehow magically pops out isn't the way forward. We need a more sophisticated approach.
Pragmatically speaking, the majority of machine learning researchers right now are not trying to make a leap in AI, they 're just trying to get in on the money while the current funding frenzy lasts.
That is, for example, why it is possible to find people presumably seriously suggesting to:
3. Flashcard the Deep Learning Book (4-6m)
4. Flashcard ~100 papers in a niche (2m)
As a method to "bootstrap yourself into deep learning research".
I mean, it's clear to me that the language deployed in the article is ostensibly about teaching yourself to do machine learning research when what it's really discussing is how to get hired by one of the companies that are curently paying six-figure salaries for machine learning engineers etc.
Or I'm just old and cynical. Wait, let me find my false teeth so I can chew that over.
This is already being done in places such as the university of Arizona (Chomsky and his former students). The subject is narrower of course (computational linguistics and some neuroscience), but there are taking an approach that is more Galilean in nature, by designing experiments that reduce externalities rather that simply looking at massive amounts of data. I think that's what's going be the most useful, at least in areas that continue to be challenging for the current trends in AI, namely language.
This is logically independent from any claim about the value of formal education. I speak from experience that an undergraduate degree is not necessary in order to gain a firm grasp of undergraduate level math. Happy to elaborate if that is desired.
I'm sure it's possible to learn on your own, but I think most people would benefit from taking a few years of their lives to dedicate to learning surrounded by a community of teachers and like-minded classmates. Learning on your own requires a lot of discipline and dealing with solitude.
The OP is highlighting maths because deep learning in particular makes use of some light calculus and linear algebra, and the OP is probably mixing together AI, machine learning and Deep learning (as is common today, unfortunately, and I can't blame the OP for that, everyone's doing it).
However, there is a lot more to AI than high-school maths and I don't just meean -more maths. I mean knowledge, lore if you like. It's a field with a long history, stretching back to the 1930's even (before it was actually named as "AI" in Dartmouth, in the 1950's). A lot of very capable people have worked on AI for a very long time and have actually advanced their respective sub-fields each with leaps and bounds and it's not very sensible to expect new leaps while being completely clueless of what has been achived before. You can't stand on the shoulders of giants if you don't know that there are giants and that they have shoulders you can stand on.
Unfortunately, most people who enter the field today know nothing of all that, or even that there was an "all that" before 2012 (if they even know what happened in 2012; and to be honest, one wouldn't understand what 2012 means if one doesn't know what came before). So on the one hand they are not capable of making leaps and on the other hand they don't even know what a leap would look like. And probably think that a "leap" is a 10% improvement of the state of the art for a standard classification benchmark.
I agree with you though that what is needed to make leaps in AI is curiosity. Lots and lots of curiosity. Vast amounts of curiosity. Curiosity of the kind that you only find in people who are a bit zbouked in the head. Or just people who have a lot of time in their hands, to study whatever their fancy tells them to.
So- not the kind of person who flashcards The Deep Learning Book, if nothing else because that means the person doesn't have the time to, you know, actually read the damn book well enough to grokk it.
I mean seriously, what the fuck is it with the bloody flashcards?
I found it to take much less discipline actually. I find an unstructured and curiosity-driven approach to learning math to be much more enjoyable and effective than the typical school approach. You are right about the solitude issue, although I’m unsure about whether this approach to learning is an intrinsically lonely pursuit or if there’s a possible society where it’s not.
I know I’m just speaking from my own experience and what works for me doesn’t necessarily work for everybody. But my claim isn't that everyone should do as I did, my claim is that you're wrong that a self-taught ML researcher would necessarily only be able to make superficial contributions because they are bad at math.
It seems correct to me. If I get a degree from MIT, somebody else can't. They have limited spots. He is promoting models of education for signaling employers that are not zero sum.
Education is more than credentials. It's the opportunity to be a part of a community that cares about ideas and make meaningful relationships with peers and mentors. Education done well unquestionably produces economic benefit. The solution to commodified, low-quality education with questionable benefit is standing up for high-quality education. Pretending that education can't be anything more than it's worse forms is pure stupidity.
"Education is more than credentials. It's the opportunity to be a part of a community that cares about ideas and make meaningful relationships with peers and mentors."
That may be the case for small colleges with high tutors-to-student ratios, but it's not the reality in many of the behemoth colleges / universities that feed warm bodies into jobs.
I've seen first hand classes with hundreds and hundreds of students, where everything worked like an assembly line. Standardized tests with zero feedback, mentors were student TAs, a class or two above you, and they had been assigned to tens of students themselves - while correcting hundreds of homework / problem sets on the side.
When you go to school like that, it can quickly feel like you're just another name on a list, with some avg. grade on the side.
And it's only going to get worse with the ever-rising number of enrolled students.
This statement is proportionally more common in fields where it's challenging to sort truth from bullshit.
If it's easy to see that a piece of output (a paper, code library, machine learning model, whatever) or a job candidate is great, then the credentials behind it don't matter much. However if it's challenging to evaluate quality, then people will shift to looking at secondary signals such as credentials, price, etc.
As to why a lot of economists go with the signaling model of education, well, it might just say something about their field and how much they got out of their own educations.
It is pretty strange even from an economist really - they of all people should be able to understand and articulate the difference between signaling value and direct utility value of a given good or service.
Economists have been debating the skills vs signaling value of education, especially since Bryan Caplan released his book The Case Against Education. If you want to get a smattering of opinion on the issue the book's reviews and dicussionsn would be a good starting point.
It could be that he's an exceptionally smart and driven guy, that just happens to pick up things really fast; I've seen those IRL myself, but not in only a year or two.
But yeah, going from 6 months of programming experience with C, to a Deep Learning internship - that sounds a bit far stretched.
Without submitting from a known university or research group? Seems unlikely to me. I have no doubt it can and has been done, but for it to be a regular thing such that one can recommend it in a 'Couch to 5k' style method? No way.
I have some friends in Oxford who are DPhil/Postdocs in highly reputable research departments specializing in ML and if they sometimes struggle to get more than a poster session at the leading conferences, with the addition of well known professors names attached, then there's just no way I can believe Joe Bloggs who just learnt python 12 months ago is able to do the same.
The idea is to choose conferences like InfoQ where application type research is accepted. Like Build-X-using-TensorFlow, it doesn't have to align with standard research which requires formal education.
One also needs to be skeptical while reading such a PR post and not get swayed away by the hero's journey in it.
You're correct, I'm sure it could happen, it is highly unlikely that a non-academic could get published even at workshops for low ranking conferences. In my experience it takes at least a year of guidance before the average PhD can be accepted to any conferences without supervision. If OP can do this alone then he's exceptional.
for a lot of people who end up in research-type jobs, a sense of curiosity is one of their strongest motivators, and they want work that will let them pursue their curiosity. it sounds like you're motivated by something else.
Why I didn't go into academia but GL convincing other people of the value in that. I am sure there are cultural differences here but where I am, the goal of most people who study CS is: leave me alone while I mess about with X (evidence: the local college was doing speech processing/nlp in the 60s, they actively turned down paid work...unsurprisingly, they got left in the dust, professors are now being encouraged into doing commercial work but, of course, most of it is totally nonviable and is just more messing about with complex nonsense that doesn't work).
I think if you look at history this is also evident: the inventions of the late 18th century were a function of necessity, the invention of semis (not just in the US but how Taiwan developed)...this isn't to say academia is pointless but there is just far more going on (I think if you look at some of the East Asian nations that get great academic results, their progress on actual R&D innovation is far less impressive).
I think in these sorts of discussions two concepts with the same name tend to get conflated, so I think it's important to make a distinction between:
1) AI Research as applying/tweaking known ML/DL methods to a novel problem. I would term these something like "AI Engineering Research"
2) AI Research as examining the theoretical frameworks & approaches to ML/DL in a way that may itself lead to shifts in the understanding of ML/DL as a whole and/or develop fundamentally new tools for the purpose of #1. What might be termed "basic" or "pure" research.
I'm not placing one of these above the other in terms of importance. They are both necessary, and they form a virtuous feedback loop between the two that, one without the other, would see the other wither on the vine.
In the example of this particular person, Emil Wallner, he appears to be doing #1, and perhaps doing so in a way that might help inform more of #2.
I'm having trouble differentiating 1 from 2. Some seem obvious. Discovering deep learning is #2, labeling some data, throwing it at an algorithm after tuning a few hyper parameters sounds like #1.
But in my mind there is also a lot of overlap. Mind providing some concrete examples? For instance what is discovering "transfer learning", "pre-training with self-supervised learning", or "building PyTorch"?
Yep! There can be. But if you want concrete examples, I used Xgboost to identify people within a population at risk for an adverse event. This is strictly #1. If I optimized Xgboost code to make it faster, that's also probably firmly #1. If I improved Xgboost with a better understanding of gradient boosting to provide more accurate results, that's probably a firm case of overlap. When Leo Breiman [0] did his work that led to gradient boosting and tools like Xgboost, that was firmly #2.
It's like the difference between, say, applied and pure sciences. One is focused on developing and studying new algorithms, while the other is focused on using algorithms developed by someone else in practical applications.
To put it differently, it's like physics vs engineering. A physicist might develop new structural analysis methods, while the engineer would use those methods to model a bridge.
I understand the separation between physics. But most structural analysis methods are discovered by professors of structural engineering and not physicists(and much of it is empirical).
But I was asking because I was specifically looking for concrete examples in deep learning.
Yep, this is why I talk about the virtuous feedback loop between these two modes. Empirical methods feed theory which feeds empirical methods ad infinitum.
In the field of ML, a concrete example might be the tool Xgboost (#1) and the original work that led to and developed Gradient Boosting itself (#2), of which Xgboost is an implementation, and probably one that has helped refine the underlying theory as well.
> I understand the separation between physics. But most structural analysis methods are discovered by professors of structural engineering and not physicists(and much of it is empirical).
You're confusing the occupation with the role. Just because your job title is professor of structural engineering it doesn't mean that you are not studying "matter, its motion and behavior through space and time, and the related entities of energy and force."
One possible distinction is: does the work reveal anything beyond the solution itself? The work might, for example, give one instance of a class of problems for which the tool is useful (bonus points for a formal statement to that effect). Or improve the tool, or improve understanding of the tool’s strengths and weaknesses
I think this is what we try to capture as “expanding human knowledge”.
IMO the more isolated the result (“technique x gave good results for problem y, the end”), the less like “research” it is. Though plenty such papers get into good conferences every year. A nice story and a little reviewer luck go a long way.
I think the questions asked by researchers in #2 are very different from those by that of #1. The questions mostly surround the why's and how's of AI, i.e, mathematical questions. To take examples from deep learning, #2 might ask about the robustness and generalization of deep neural networks, applying dynamics/ODE theory to certain types of neural networks such as ResNets etc.
#1 might ask about the performance of a deep neural network in approximating a given model in a specific application. Alphafold, on the front page currently, is an example of #1.
I have a decent understanding of the approach and would vote for #1. I’d say almost all applications of ML in physical sciences are #1. In contrast applying methods of statistical physics to understand how deep learning (as in DNN+SDG) works at all is a good example of #2.
I'm hardly the official judge of these things, but I would say it depends on how novel of an approach AlphaFold is to the problem. If it's a more efficient tool for doing the same things as before, I would put it towards the #1 end of the spectrum, unless it has also improved our basic understanding of folding or approaches to exploring the solution space of folded proteins, which would shift it towards #2.
Personally I don't know enough about AlphaFold or the problems of protein folding to be remotely confident in my judgment on it
Neural networks are differentiable regexes that can be trained from examples. In the alpha fold case, which is the case with a lot of bioinformatics actually, is that you don't need to know a lot about the biological domain to be successful in solving "data" problems in the field.
Is this guy actually a researcher in the way most people would think of it? That is, someone who pushes the boundaries of science; who develops new AI techniques or finds the hard boundaries of existing AI techniques; who finds new ways compose multiple AI techniques cohesively; who explores the theoretical foundations of AI.
Or is he someone who uses AI techniques to solve problems (and then wrote a paper about it)?
I can't help but wonder a bit.
For better or worse, the definition of researcher has morphed into a combination of
1. Solves previously unsolved problems
2. Publishes papers sharing those solutions
without regard to the kind/spirit/scope of problems solved.
Since conference publications don’t have the same number constraints as journal papers, and are accepting of application-specific results, this explosion of what is considered “research” is somewhat inevitable. Also, there are a lot of people chasing this given the prestige associated with the title.
From his GH profile looks like he's a competitive applicant for ML engineering positions or perhaps a fellowship/residency/PhD program.
So, a junior researcher at the level of a decent second or third year PhD student. A researcher, maybe someone you'd trust to build a prototype or product, lots of potential, but probably not someone you'd trust to run a research program.
Research needs people at the entire spread of the spectrum - from those making fundamental improvements to underlying theory, all the way to people running the thing to see if it works on actual problems people have (obviously in a robust and verifiable way).
This reeks of survivorship bias to me. I much prefer Andreas Madsen's more sober and self-conscious take on independent research [0].
> I’d spend 1-2 months completing Fast.ai course V3, and spend another 4-5 months completing personal projects or participating in machine learning competitions... After six months, I’d recommend doing an internship. Then you’ll be ready to take a job in industry or do consulting to self-fund your research.
Where are these internships that will hire you based on your completion of Fast.ai (if done in 1-2 months by a beginner I assume it's only part 1) alone, especially in 2020? How many are going to place in a Kaggle competition with just half a year of experience? More importantly, just how many people are privileged/secure enough to put their all into learning, with no sense of security or peer support?
> I started working with Google because I reproduced an ML paper, wrote a blog post about it, and promoted it. Google’s brand department was looking for case studies of their products, TensorFlow in this case. They made a video about my project. Someone at Google saw the video, though my skill set could be useful, and pinged me on Twitter.
So what really mattered was self-promotion, good timing, and luck.
> Tl;dr, I spent a few years planning and embarking on personal development adventures. They were loosely modeled after the Jungian hero’s journey with the influences of Buddhism and Stoicism.
Why does the author have to present his life like one would in a fucking college essay?
> So what really mattered was self-promotion, good timing, and luck.
Yes. He seems like someone who is good at self-promotion and networking. Well, good for him, but I think he underplays the role these have in his success.
> Why does the author have to present his life like one would in a fucking college essay?
I guess that's the self-promotion. And humble-bragging. Like this bit:
"I started working as a teacher in the countryside, but after invoking the spirit of their dead chief, they later annotated me the king of their village."
> Well, good for him, but I think he underplays the role these have in his success.
Exactly. Good for Emil, but it's always frustrating to hear survivorship bias preaching. Even the interviewer starts off by saying:
"By the way, I really love your CV - the quirks section was especially fun to read."
It's even more frustrating when I hear non-POC's talk about their journey to some non-western country (and subsequent conquering of fantastical goals like gaining the approval of locals) or pursuit of some sense of foreign culture. It's almost a given that they have internalized and appropriated the ideas (i.e. Buddhism or even worse post-retreat Buddhism). Good for the author to receive such positive feedback for such signaling, but it makes me sad to know that I might not receive the same.
> Where are these internships that will hire you based on your completion of Fast.ai (if done in 1-2 months by a beginner I assume it's only part 1) alone, especially in 2020?
I don't think the idea is to look for an internship after the course but an additional 4 months of personal projects. After applying state of the art deep learning for 4 months full time you'll have some very cool projects, and you could probably convince some company to take you on as an intern for a certain amount of time.
I should have been more specific and did not mean to exclude the other 4-5 months in his half year estimate. I meant to say no other background/experience to the position that would qualify the candidate for the role other than fast.ai v3 part 1. I love fast.ai and would still recommend it to all. I just think that the chances for getting into internships in that period, given the difficulties/inefficiencies/biases in the hiring process that even OP has mentioned, are slim.
Yeah I don't think it would be easy. I'm not in a data science role, but I do hire for software engineers. And if someone came to me with "Hey I'm trying to be a software engineer. I finished this boot camp 4 months ago, and since then have built these really cool projects. Could I work for you as an intern for the next 6 months to break into the field?" I'd totally say yes.
and truth be told: anders madsen has a full CS degree. So he basically took a 1-2 year private PhD-time, in which he produced 1 paper. Which is not terribly special. As for the person in question: there's money in it and if there are more aspiring people to do the grunt work (data cleaning, hyperparameter tuning) eventually prices will fall. And you definitely don't need a CS degree for that.
> Creating value with your knowledge is evidence of learning. I see learning as a by-product of trying to achieve an intrinsic goal, rather than an isolated activity to become educated.
> Early evidence of practical knowledge often comes from usage metrics on GitHub, or reader metrics from your work blog. Progress in theoretical work starts by having researchers you consider interesting engage with your work.
> Taste has more to do about character development than knowledge. You need taste to form an independent opinion of a field, having the courage to pursue unconventional areas and to not get caught up in self-admiration.
When I study abstract interpretation or lattices, I'm doing so because I find those subjects interesting and beautiful, and studying math relaxes me. I can lie to myself and say that it's improving my problem solving ability and that it's like doing mental yoga and will make me better at my job or some baloney, but that's not why I do it.
I can spend time with a plant in my garden, take a cutting, root it and replant it, and watch it grow, learn the ebbs and flows of its watering needs through the seasons, learn what its seed pods look like, and eventually watch it die through some misstep of my own or otherwise.
And in doing so, I am learning, and building a mental model for this plant and an intuition for it, but I'm not "creating value" in some weird capitalist sense, which I feel always underlies these sorts of opinions about learning and education, and people who self-identify as "makers" in general. It rubs me the wrong way because it encourages a very narrow view of the human experience and what it means to learn and why we should learn.
I think most of us here on this site could do the same in terms of learning and research
Seems to me the difficult part is how to support yourself financially while spending your time doing interesting learning and research, or how to get paid to do it
Maybe the most important detail in the story is "He co-founded a seed investment firm that focuses on education technology" but it is not discussed further
He also studied at 42, which is most likely why he got picked for the internship to begin with. I don't get this self-congratulating BS, guy says he toured the world (good luck doing that with a shitty passport) and was named king of a village in Ghana (right..). I guess people who get lucky have to always go to great lengths to justify and spin that.
They're free to do so of course, but they should not give advice based on it.
Regardless of that, I suppose the bar for being a "researcher" has been stooped so low. According to this guy publishing an ML paper is equivalent to writing a blog post or making a video about "AI".
I personally found this article to be very interesting. I don't know much about AI, but I was fascinated by the discussion of peer to peer educational system. I believe that they will become more prevalent as student loan payments cause debt to so much of our population in order to get an education .
I don't know why people think getting a credential does nothing or that people "copy and paste" the assignments. Sure it may be possible, but what prevents people from copying and pasting public git repos?
Either way, this whole focus on "portfolios are everything and credentials are meaningless" spits in the face of all the work I did to get my university education. And it didn't involve "copying assignments". And you come out with one hell of a portfolio if you take your education seriously.
I mean I don't think self-educated people are without merit. I happen to think they're really important. But I only ever see them rag on higher education, despite them having "never been there".
Just another example of wunderkin super genius knows all because he was able to follow a non-standard path and make it. Glad he was smart enough to become a Google employee. But I question whether he should be giving advice on paths to get there when there's always many paths to a position. And especially after reading his brief comments on how credentials imply you're a liar.
> I don't know why people think getting a credential does nothing
Then actually pay attention to the arguments they're making instead of talking about how offended you are because it goes against your self-interest as a degree holder. It's not as if the people bashing modern education are some kind of elusive minority.
I've got a master's degree and I've always though our education system is stupid, and at least in the U.S. not unlike a giant pyramid scheme given the cost of tuition these days. Absolutely nothing you learn in a college education you can't learn yourself for free on the internet.
> Absolutely nothing you learn in a college education you can't learn yourself for free on the internet.
This is categorically false. Face-to-face time with an expert is incredibly valuable and incredibly expensive outside of an academic setting. In fairness, you have to show some initiative in college to get quality face-to-face time with a professor, but it still takes a lot less motivation than self-studying a complex subject for a nontrivial amount of time.
Which brings me to my second point. There's an enormous amount of free stuff you could learn from. But actually doing it is a completely different matter and the overwhelming majority will fail. For instance, the bulk of a university-level education in pure mathematics is over a century old, and free resources are easy to find. With stackexchange, you can even get expert feedback on your work! Yet most people who try (who are already a very self-selected sample) do not in fact succeed in teaching themselves undergraduate level mathematics. Even Ph.D. students taking a few years off for whatever reason find it highly (but not impossibly) difficult to do any significant amount of self-study for a prolonged period of time. And these are precisely the people who are training to become independent researchers!
> This is categorically false. Face-to-face time...
Just because talking to an expert is valuable doesn't mean you can't learn it for free on the internet. Also most undergrad curriculums are teaching old stuff - not exactly cutting edge knowledge requiring face-to-face one-on-one time with an expert in your field.
It's not as if the alternative to 4 years of undergrad and $100-250k in tuition + living costs is just teaching yourself the same arbitrary curriculum alone in your room for 4 years getting a degree in some random field learning things you never actually use in the real world. One could instead intern or work, and not only potentially learn significantly more relevant and lucrative real-world skills for free, but actually get paid to do it. A business student could instead work directly for entrepreneurs and use that tuition money to start their own ventures.
Most people use very little of anything they learn in school after they graduate. For example I majored in math, and now as a software engineer I don't use any of that. I know some math majors will try to rationalize it by saying they learned "problem solving" skills or whatever but there are a million other more useful things I could've done instead of what I did in school. Everything I learn now as a software engineer I either teach myself or learn on the job. There is no curriculum that could prepare me for what I do now because by the time the curriculum is written, it would be outdated (well perhaps such a curriculum of "fundamentals" could be constructed, but the CS curriculum is not it).
The system is outdated, inefficient, and a downright pyramid scheme scamming the youth into indentured servitude in the U.S. If tuition was reasonable and having a college degree wasn't required for most jobs then I wouldn't be as critical of college.
I don't know what university you went to, but there are such things as bad universities. Just like there's such things as bad Internet courses.
However, the university I went to, could be classified as "No-Name" and I use what I learned in school almost everyday. In fact, I used K-maps to help a senior engineer struggling with a complex logic problem by simplifying it. The CS fundamentals I learn prevent me from writing ugly code and at least give me a sense for what's slow.
I also went to a university with a built in co-op education program where you got credit and paid for being an intern. And let me tell you, most companies treat interns like shit. They sometimes don't even bother having them do anything besides mediocre grunt work. My intern experience was not the greatest and arguably worse than my college experience. Most the time I was left on my own having no idea what to do and spent most of it reading programming books. Whatever "real-world" skills I picked up, like doing actual projects, was moot.
But again, it's mostly relative. So making categorical statements like "universities are useless" and "credentials are for cheaters" doesn't really help and certainly doesn't speak the truth.
CS fundamentals are great, thankfully you don't need to physically go to any university and pay hundreds of thousands of dollars in tuition to learn them.
The alternative isn't just being an intern taking on grunt work, there are many who forego college to work full-time jobs in industry.
I maintain that there still exist things that you can only learn via osmosis. Sometimes, books and MOOCs just will not do.
Also,
> It's not as if the alternative to 4 years of undergrad and $100-250k in tuition + living costs is just teaching yourself the same arbitrary curriculum alone in your room for 4 years getting a degree in some random field learning things you never actually use in the real world. One could instead intern or work, and not only potentially learn significantly more relevant and lucrative real-world skills for free, but actually get paid to do it.
I mean, sure you can not go to school and do different things, and it might even be a good idea, but that's a far cry from the original claim, which was
> Absolutely nothing you learn in a college education you can't learn yourself for free on the internet.
Not everyone is a genius like you. And if you want to sit in your room and do tutorials and MOOC courses all day without speaking to a single soul, and you can figure everything out on your own without guidance, be my guest. I applaud you, because that's something I struggle with and maybe that says more about me than you.
Not only that, please tell me how many people can afford things like a mass spectrometer, fume hood, VNA, and other pieces of equipment that cost upward of 6,000+ dollars when they're in high school. If you want a real STEM education, you need to learn how to use test equipment unless you stick with CS or math, there isn't a lot of options to learn for "free". Even online courses cost money.
Sure there may be many people that get lucky and get into actual positions, but they're few and far between and are a direct result of success bias. The media shows you the thousands or so odd people that make it under extreme circumstances and never once mentions the people who never make it because that isn't "cool".
Ok yes my statement applies more to fields like CS and math and not to fields that require a lot of physical equipment (eg. chemistry).
I'm not claiming that the best way to learn is alone in your room, I don't even believe that to be the case. The best way to learn is by working with people who know more than you. That is generally what happens when you work a full-time job. Sure in school you can learn from professors, but (1) what you learn is often divorced from the reality of professional work (2) the format of listening to lectures, completing busywork, and taking multiple-choice exams on random information could be Googled isn't the most efficient way to learn.
Yes I know it's difficult to find work without a degree because companies unfortunately discriminate against applicants without degrees. I wouldn't be opposed to a law that banned employers from discriminating against non-degree holders unless the job clearly requires someone with the expertise gained from the degree (eg. medicine, not law or marketing).
> Absolutely nothing you learn in a college education you can't learn yourself for free on the internet.
I don't know who "you" is (perhaps you in particular are very gifted) or what you personally learned in college, but on my end in college I specifically took particular classes to learn topics that I had previously tried and failed to learn on my own, so, if it's meant to be generic, I'm pretty confident your claim is false.
I went to a bad No-Name University for undergraduate and then Top Tier University for phd school, so I have an unusually representative view here.
I do agree that, for CS, the no-name university was... bad. Fortunately, I realized this early and did a lot of self-study. I probably learned more reading taocp and going through MIT open courseware courses in the library during the evenings than I learned in my actual undergraduate courses.
The mathematics courses, even at No Name, definitely provided me with a better education than I could have ever gotten on my own. I'm pretty bad at math, it was my worst subject in high school. So I double-majored in it during undergraduate. This dovetails with your advice to use university as a time to learn things you already tried and failed to learn on your own, or which you otherwise know will be difficult to learn on your own.
The CS education that undergraduates get at Top Tier University is far better than what I got, even though I worked through that Top Tier University's online courseware/lecture notes/exercises during undergraduate on my own.
My hot take: college is always worth it, but only if you intentionally invest in "leveling-up" past your previous potential.
That will happen almost by default at Top Tier unless you're a genius (...but you'll pay a lot for it). But not if you're going to university at No Name. So, in that case, students should definitely a) minor or even double-major in something they're not good at, and b) heavily supplement their CS courses during evenings/weekends.
Also, this is all highly specific to very self-motivated learners -- the sort for whom "self-taught" is a reasonable route. I'm one of those people. Over time, I've realized that we're a very small minority. Our perceptions of what others are capable of learning on their own, and prescriptions for how others should learn, are typically quite warped. Most people probably do need something like a 4 year college degree to become a competent programmer.
I only got my degree because everyone tells you to get one and most employers will throw your resume in the garbage without a college degree, though thankfully that's now changing.
Not only mention those fields that require lab work, practical field work, and/or specialized equipment or facilities to learn. This site is far too CS and math specific sometimes.
Do you have any idea where the majority of massive breakthroughs in technology come from? They come from university.
AI started in university before it was even a thing. Autonomous vehicles were a university funded DARPA project. Nearly everything, even this guys research, is an important derivative of this.
Why do you think this guy chose to publish his paper in a academic journal? It's because it's peer reviewed. The Internet is not peer reviewed, it's public reviewed, as in anyone with an opinion can say whatever they want and get a million other opinions accepting or rejecting that opinion with little evidence. That's essentially what the entirety of Hacker News is. Very rarely do I see a post, including my own, that's properly sourced.
Let me ask you, do you think it's stupid that I paid money, like most people, to build a solar power management system? Do you think it's stupid that I built a memory management system, and text message system from bare bones hardware? Do you think it's stupid that I built my own shell?
I did all that in school with equipment that I could only dream of owning with people that spent more time helping me than writing blog posts trying to get famous.
It's funny, I was actually glad this guy got where he wanted. It must be nice to be a genius. And to be honest, I'm probably not as smart as this guy. It's great that he had a lot of drive and achieved greatness. But he doesn't need to imply that I'm some liar because I chose to go to university. I worked incredibly hard to get my degree, spending hours and hours in the lab doing assignments. Hours thinking I was dumb and that I'd never make it. Months trying to find a job.
And all I see is people that did it a non-standard way and then have these warped views of the traditional way despite the decades of people lifted out of poverty because of it. And despite having never even attending a university! I see all these smart people and how they're the only ones that matter. College did a lot for me. And I try and do my best every single day because of it.
Sure, I may not be able to write an ML paper in a year like this guy, but that doesn't mean I'm not going to defend myself when he essentially implies I cheated my way in because I got a degree.
yeah, as someone who had a partial education, I totally get the value of a degree, so whenever someone says "higher education is useless" I read it as "However successful I am now, I wasn't the kind of person who would succeed in school then"
Whenever I hear someone say "higher education is useless" I hear it in reference to the "common wisdom" that higher education increases incomes. In which case they are absolutely correct. Higher education is useless in achieving that. Incomes have held stagnant for many decades, even as more and more of the population attain higher levels of scholastic achievement. Mathematically, incomes cannot not remain stagnant if more money is earned as a result of attaining higher education.
I'm not sure I have heard anyone claim that "higher education is useless" in general. Education is never useless in general.
A contradiction! Incomes have not increased if incomes are stagnant, and the data is quite clear that incomes are stagnant. How do we resolve this?
Perhaps you are confusing increased income with being higher up on the income ladder? It is true that, statistically, those with higher education do find themselves higher up on the income ladder. They are not making more than they did before, when they did not have higher education, however. Incomes are stagnant.
All we're really observing there is the fact that people range from more to less able, from geniuses who seemingly can do anything to those who have crippling disabilities. At one end of the spectrum you have the people who do well in school and also the workplace due to their natural ability, and at the other, those who struggle in everything they do, be it school or the workplace because of their disabilities. And then everyone else somewhere in between. Statistically, the more able will find themselves higher up on the income ladder, and able to go further in school, thanks to being more able.
An increase income range is an increase in income. Most companies do not hire high school students or drop outs for more than minimum wage.
How you can reconcile a higher educated person being higher up the income ladder, but not making more than what they made working minimum wage at a lower education level, eludes me.
What stagnant income means is, they aren't making more relative to their productivity. Meaning salaries have not changed all that much for about 2 - 3 decades through raises despite high productivity.
> An increase income range is an increase in income.
I think I see the issue here. I am talking about the population as a whole, you are talking about an individual. It is true that over time individuals tend to move up the income ladder. And that more capable people move higher up the ladder thanks to being more capable, free of disability that hinders their achievement. However, the steps of the ladder have remained unchanged over decades. Making more than someone else is not the same as making more than you otherwise could have.
If we break that ladder up into percentiles, the person at the top of, say, the 70th percentile made x of number dollars in 1970 and the person at the top of the 70th percentile still makes x number of dollars today, in real dollars. In 1970 that person did not have an education above high school. Today that person does. Despite promises, their income did not increase with increased educational attainment. Incomes are stagnant. There was no advantage to gaining that higher education with respect to income. The person at the top of the 70th percentile, who got there because of the abilities and constraints they were born with, would have ended up there regardless.
> What stagnant income means is, they aren't making more relative to their productivity.
What stagnant income means is that incomes are literally not changing, relative to inflation. On average, if you made $1 last year, you will make $1.02 this year, assuming a 2% inflation rate. A real increase of $0; stagnant. To put it another way, incomes, in nominal dollars, are increasing at the same rate as inflation.
Your link effectively only shows that higher earners earn more than lower earners.
That is not the same as incomes increasing because of higher education. Incomes have held stagnant for a number of decades, even as more and more people attain higher and higher levels of education. If 0% of the population had a higher education or 100% of the population had higher education, your income would remain the same as it is now.
McDonalds isn't going to pay someone flipping burgers a million dollars more over the span of their burger flipping career simply because they attained a physics degree. That is not how the economy works at all. If 100% of the population had a physics degree, someone is going to be left flipping burgers. 100% of the population are not going to be working on solving string theory. The economy cannot function in that manner.
Inflationary dollars are relevant as that is how we measure stagnant incomes. An income that tracks inflation is stagnant. Incomes are stagnant. We are not talking about spending.
Yes, but a science lab isn't going to hire a high schooler with no science degree. And no business will pay a physicist with a degree minimum wage. That's the point. The result of obtaining higher education categorically results in you earning more income. And most people filling minimum jobs are not people with higher education.
Are people with higher education in low wage jobs? Yes. Is this common? No. And the only reason it would be common is if there was no income incentive to gain higher education, which there isn't.
I'm beginning to think you really don't understand what I mean.
> Yes, but a science lab isn't going to hire a high schooler with no science degree
What choice would they have if 0% of the population had a degree? This is ultimately why incomes haven't changed even as more and more people attain higher education. In the past the best and brightest people were hired into those positions as high schoolers. And now the best and brightest still are, except they have higher education now because of the social pressure to attain a higher education.
> And no business will pay a physicist with a degree minimum wage.
They absolutely would. Of course they would. I have no idea where you got the idea they wouldn't?
A person capable of becoming a physicist has little reason to want to work a minimum wage job though. The fact that they can attain a physics degree means that they do not posses the limiting qualities (disabilities, poverty, lack of intelligence, etc.) that leave less able people stuck in minimum wage jobs. People who are burdened with certain disabilities, poverty, lack of intelligence, etc. never had a real chance of completing physics degree. It is not within their ability.
> The result of obtaining higher education categorically results in you earning more income.
No. Those with higher educations, statistically, earn more than those without higher educations, but they are not earning more than people in the past who did not have higher educations. Additionally, this correlation exists because people who failed to attain higher education have qualities that limit their success in both school and the workplace.
> I'm beginning to think you really don't understand what I mean.
No, I fully understand that being able to excel in school is correlated with being able to excel in the workplace. This is obvious. Someone with crippling autism, which did not allow them to graduate from high school, was never going to become CEO of Google. I get it. That does not mean dropping out of high school will cause you to contract crippling autism.
It is well understood that education acts as a filter, leaving the poor performing people, who also perform poorly in the workplace, behind in academic achievement. This is quite different to school causing someone to become a high performer.
This is a great example of how we "collectively" [1] conflate phenomena, skillsets, ... into one topic: machine learning, AI.
1) There is the general phenomena or collective project, where hardware, algorithms and human insights are improved to approach the situation of man-made intelligent machines.
2) There are the people who are designing algorithms, using mathematical intuition and knowledge, analogies with physics, etc... Most people would agree these people are doing optimization / machine learning "proper".
3) There are the people working on improving hardware for machine learning / optimization purpouses, by looking at the most performant algorithms, breaking them down into primitive operations and requirements for hardware, there are also people working on the algorithms themselves and finding computational shortcuts (which can end up in software or hardware, can end up as proprietary knowledge or common knowledge, ...). The distinction between hard and software is somewhat blurry, since hardware designers can optimize or implement a section of software into hardware. A lot of this can still be considered ML "proper".
4) Then there are the people who apply the ML frameworks and their exposed choices and settings to a specific problem domain. Many of them don't need to understand the internals if they don't need state of the art results. Many would nevertheless benefit from understanding the internals, and the requisite math. What I propose is to stop calling their activity as Machine Learning, and instead call it Machine Teaching. They are teachers, and just like elite schools they can choose which specific type of available student they will teach, and they can tweak (or filter from a large family of students) which student they select to teach the task at hand. There are bound to be many advantages of having actual human teachers get involved in machine teaching. These people will not be proficient in designing novel families of students unless they also know the requisite math, and identify those ML papers that are ML "proper" instead of ML "teacher". When trying to find important foundational insights in ML "proper" one is typically overwhelmed by a large surplus of ML "teacher" type papers. These are important datapoints, and necessary to advance human insight into ML "proper", but they are data, not knowledge. There are actual ML "proper" knowledge papers out there that explain why a certain phenomena is such and so, and they get very little attention because they necessarily lag the breakthrough ML datapoint paper, and most ML "teachers" don't have the math background to understand them. So the probability that a given ML "proper" researcher fundamentally improves the state of the art is much higher than the probability that a given ML "teacher" will fundamentally improve the state of the art. At the same time the probability that a given fundamental breakthrough was achieved by an ML "teacher" is higher than the probability that a given fundamental breakthrough was achieved by an ML "proper" researcher:
P ( Teacher | Breakthrough ) > P ( Proper | Breakthrough )
Since most people don't have the broad math / physics / ... knowledge to draw on, the number of ML "teachers" is much higher than ML "proper" researchers.
[1] well, really, some actors have vested interests in conflating those together...
EDIT: just to be clear, I am not complaining about ML Teachers, we need the ML Teachers, and their breakthrough datapoints. What I am complaining about, is conflating both activities of ML Proper and ML Teaching. This makes it harder for the few ML Proper researchers to find each other's insights.
The FAANGs are trying to hire all the top talent (including Emil who wrote the post) but I believe these independent researchers will be the one finding new opportunities to make AI useful in the real world (like colorizing b&w photos, create website code from mockups).
The biggest challenge I see for these folks is the access to high quality data. There is a reason Google is releasing so many ML models in production compared to smaller companies. Bridging the data gap requires effort from the community to build high quality open source datasets for common applications.