This is still retrieval and RAG, just not vector search and indexing. it’s incredibly important to be clear about terms - and this article does not meet the mark.
“There is a difference,” she said, “between immediate recall of facts and an ability to recall a gestalt of knowledge. We can’t retrieve the specifics, but to adapt a phrase of William James’s, there is a wraith of memory. The information you get from a book is stored in networks. We have an extraordinary capacity for storage, and much more is there than you realize. It is in some way working on you even though you aren’t thinking about it.”
IMO, this quote demonstrates a phenomenon I've long suspected to hold with the books that I read. The essence of a book's or an article's content is captured by a few key ideas and phrases (e.g., 'gestalt of knowledge', 'wraith of memory' in this case), but merely knowing these phrases is not enough. You need to read the entire book to have a sensation of the ideas getting fleshed out. The article on 'metrosexuals', the pamphlet on "the third estate", and the book on 'positioning' are other examples I can think of where this phenomenon plays out.
A rich 3-dimensional idea in the author's mind gets transformed into words. The words themselves are just information. You then fight with the words to reconstruct the idea with all its original potency in your mind. It is not necessary for you to recall every little detail of it unless you are an academic or specialist in the field.
Corollary : This is also the reason why people (i.e. Wall Street) who are close to / handle the money earn more money. Drinking straight from the firehose.
Anyone who wants to demystify ML should read: The StatQuest Illustrated Guide to Machine Learning [0] By Josh Starmer.
To this day I haven't found a teacher who could express complex ideas as clearly and concisely as Starmer does. It's written in an almost children's book like format that is very easy to read and understand. He also just published a book on NN that is just as good. Highly recommend even if you are already an expert as it will give you great ways to teach and communicate complex ideas in ML.
If anyone here hasn't read Borges, I'd strongly recommend him. Pretty much everything he wrote was short, <20 pages, and so it's really easy to sit down and read one of his stories over a lunch break. The common recommendation would be to try out Tlön, Uqbar, Orbis Tertius and see if you like it. If so, it's part of Labyrinths, which is (in my opinion) his best collection of short stories. The best edition in English is probably Penguin's Collected Fictions.
Regarding the content of this interview:
>If you compiled an enormous dataset of everything Borges read, and combined it with an exquisitely sensitive record of every sensory experience he ever had, could you create a Borges LLM?
This is my Kantian way of thinking about epistemology, but I don't think that LLMs can create synthetic a priori knowledge. Such knowledge would be necessary to create Borges out of a world without Borges.
In this interview, Simon's view feels much more like the way Hume viewed people as mechanical "bundles of sensations" rather than possessing a transcendent "self". This led to his philosophical skepticism, which was (and still is I guess) a philosophical dead end for a lot of people. I think such epistemological skepticism is accurate when applied to machines, at least until some way of creating synthetic a priori knowledge is established (Kant did so with categories for humans, what would the LLM version of this be?)
I love the last sentence: “…, if you set yourself the goal of crossing an 8-lane freeway blindfolded, it does make sense to focus on doing it as fast as you possibly can.”
For any young programmers: live within your means, invest the difference, become independent, and work on what you enjoy. It’s the best (work related) gift you can give yourself. Skip the self promotion politics unless you enjoy it.
I've brought this up before, but I never got a response and I'm really interested what people think the business case is here. I keep wondering what a buyer would actually value beyond Chrome's userbase. Chrome is just Chromium with Google integrations, similar to how Edge is Chromium with Microsoft integrations.
If a company acquires Chrome, they don't have many choices: re-establish Google integration deals (so the divestiture would be pointless), replace Google integrations with their own (becoming just another Chromium distro in a sea of Chromium distros), or just monetize the existing userbase.
A Chrome acquisition doesn't include unlimited control over Chromium. Chromium is open source, with contributions from many organizations (who retain the copyright to their contributed code). Google can only sell what it exclusively owns: the brand, infrastructure, signing keys, etc. The real force behind Chromium is having a critical mass of engineers all pushing in generally the same direction. You can't necessarily just buy that, especially when you wouldn't own exclusive rights. Any other company is free to poach engineers and fork the project.
Edited to add: if Microsoft sold VS Code to, say, Oracle... don't you think that another company would leap at the opportunity to fork the project? Would the userbase and the thin layer of closed-source Microsoft customizations really be worth that much?
There's this piece of product-management wisdom that I've been thinking about lately, which is that users almost always understand their problems better than you do. If the metrics say things are going well, and your users say everything sucks, your metrics are probably wrong. But the complement to that is that users mostly suck at solutions: you understand the constraints and difficulties of your product better than they do, so they tend to suggest things that are infeasible, overly specific, or prohibitively difficult to build.
When the public gives (or random bloggers give) give a damn about economics, it's a sign the economy isn't working. Of course they don't have useful solutions - they're not economists - but that feels a little beside the point: you don't have to be a plumber to recognize that your house is full of sewage. And since no one can be an expert in everything, life demands the ability to identify and call attention to problems you cannot personally solve.
An article like this is the equivalent of your roommate going "oh, damn, the living room is full of sewage, we better do something about that!" Of course you'll vigorously agree at first, because you're talking about the problem (which is clear to everyone). But then your roommate suggests fixing it by dropping dynamite down the drain - i.e., talking about the solution - and you're a lot more likely to disagree.
And the corollary to that, from 17th century French writer Nicolas Boileau: "Ce que l'on conçoit bien s'énonce clairement, et les mots pour le dire arrivent aisément." - What we understand well, we express clearly, and words to describe it flow easily.
FYI, the modern version of the intro networking book we (UC undergrad computer science) had 20 years ago is Computer Networks 6th ed. by Tanenbaum.
If you want a programming language to really dig into parsing and creating packets and binary data, Erlang (and Elixir), while a completely different programming paradigm daunting to novice users, are unmatched (pun intended) in generating and parsing binary packets with a native and flexible binary data type.
If you want to be able to observe, decode, and inspect real packets on your local network, then you need wireshark and/or tcpdump.
If you want a home lab for simulating various networks with various clients, servers, and network devices either XenServer (free) or VMware ESXi (pirated "free" v.7 or maybe v.8) are good options because they simulate dumb virtual switches, virtual networks, and virtual computers (VMs) in another piece of software that replaces a real computer's (usually a server but rarely a desktop or laptop, but also it can run inside a VM on desktop, laptop, or server using what's called a type-2 hypervisor) operating system with its own (type-1 hypervisor). (VMware pre-Broadcom and Citrix engineering cultures had an unspoken, unwritten "gentleman's agreement" that industry-limited pirating was cool so long as it wasn't unreasonable, and product licensing wouldn't break customers' production and would be limited to nags. Comes from the mouth(s) of (a) certain former product manager(s) at one or more of the above entities. Pirating developer use for self-demoing within the industry was a nonzero, unrecognized sales acquisition channel that was probably important in large customer bases but under-appreciated... most engineers/IT people don't want to have to deal with vendor sales/sales engineering meeting free lunches and sit through vendor demos so see if a product trial will work or just to get an installer and a demo license key.)
PS: I'm wondering if there's retro networking/netadmin homelab community that buys 10 Mbps - 1 GbE Cisco gear and goes through CCIE courseware to see all the old protocols and old problems like spanning tree loops, crossover cables pre auto–MDI-X, and duplex flapping with Intel PRO/1000 in generation 8-9 HP and Dell servers.
It always seemed common sense to me that once a piece of software became stable and profitable you would need a much smaller engineering staff than when it was being built. The opposite seems to happens and orgs 10X their engineer headcount once they start making money.
It makes sense to keep some high performers and a few redundancies to stabilize/modernize it and make small improvements, but it feels like tech got really bloated with these massive corps who were trying to burn as much as they could to keep all the VC money flowing. Take like Uber having a team that built and maintained a chat app just for internal use, and every single big org having a bunch of teams responsible for various "some_dumb_name" that is the "custom X for 'Y'" where smaller teams just use the OS solutions to those problems.
Best thing I ever did for tmux was change the command key prefix to "C-\". With caps lock mapped to control, this makes tmux commands available with almost no hand motion, just a slight reaching out with the pinkies. It also restores access to C-b for "back one character" in emacs and all programs using emacs key bindings, which is hardwired into my fingers at this point. (You're blocking emacs' function "toggle-input-method" - "Enable or disable multilingual text input method for the current buffer". For some users that would probably be an issue, but hasn't been for me.)
Another useful thing is to "bind-key C-\ last-window". That makes C-\ C-\ jump back to the last window you visited, which facilitates creating short-lived windows for specific tasks.
Final small tweak I like is "set -g base-index 1" so that C-\ 1 is the first window, 2 is the next, etc. instead of having the first window way over on 0.
My other tweaks are probably more idiosyncratic to my workflow.
> Their reasoning is that CBD-based businesses are somehow entitled to pre-COVID customer levels
It's more like downtown property prices are based on those levels, and property is leveraged, and if banks collapse do to commercial property prices plummeting, you're in for a bad time.
Also, although downtown is a very small part of the city - in many cities, downtown property taxes make up a relatively large chunk of total property tax revenues.
You either death spiral downtown property prices by keeping taxes steady while values decline, or you increase tax everywhere else to make up the difference.
Either of those options leads to a bad time for politicians.
Spacing is a challenge. And you lose some legibility giving up proportional fonts. I think kerning in proportional fonts makes a big difference, letting your eyes recognize the shape of different letter groupings.
Monospace text is fine if you avoid long-form text, like when it's structured and highlighted in a code editor.
But it sure is pretty! Especially with Unicode charts and ASCII art.
Any time someone who does not understand the practical outworkings of a given topic tries to "teach" it, you end up with a "presentation" - mostly in the form of propositional statements (dogs are mammals; barges are boats), or key-value pairs (in 1492, Columbus sailed the ocean blue; Caracas is the capital of Venezuela)
Propositional statements and key-value pairs are absolutely vital aspects of learning and understanding - but seeing how those ideas, facts, and statements will work themselves out in practical application is absolutely mandatory for proper understanding of the topic
If you never learn how you might use these facts, you are going to have an incredibly difficult time "learning" them
This is the fundamental problem of why "teaching to the test" (or, for that matter, relying too heavily on multiple-choice, true|false, matching, etc type "tests" to evaluate "learning" or "knowledge") is problematic: I can teach pretty much anyone how to pass a multiple-choice test with zero knowledge of the subject material (did it myself years ago when I passed the first-tier amateur radio licensing exam without ever looking at any study guide)
When you "teach to the test" (which is what, fundamentally, exclusive reliance on propositional statements (and drills over them) and key-value pairs are), you create people who may (or may not) end up being good at trivia challenges ... but have no mental framework for connecting all those dots into anything coherent - iow, you create human-based data lakes: it is all sitting there, but no connective lines have been drawn
To get someone to want to learn, you have to show they why it "matters" - you have to get them to develop intrinsic motivation to learn (vs the purely extrinsic "you have to to pass")
To develop that intrinsic motivation, you need to show the 'why' and the 'where' of the 'what'
I did ... the thing is, most college students need a lot of help, especially in earlier classes. On a statistical level it's somewhat the opposite of a population of self-taught programmers, who self-select for being into the gritty details of computer systems, and being able to figure stuff out. You know, in most college tracks, there are problems with lots of students needing help, cheating, struggling, it's kinda the regular state of things, now that most people go to college.
Some college students switch to something else later; some continue to need lots of help with system setup when they start working professionally; those who are really good were largely self-taught either before or between college courses. College/university is still useful for the self-taught programmer, to cover gaps, and for the advanced courses. I had been installing, configuring, fixing, and re-installing windows and linux for 4+ years before university, and I really loved my systems programming courses.
Now that there are millions of programmers out there in the world, working professionally, yeah lots of them are kinda useless ... there's a reason why "average" doesn't have a great connotation.
Previously, I thought certain math topics were "hard" (e.g. category theory) while others were supposed to be "easy" (e.g. Calc I). I beat myself up for struggling with the "easy" topics and believe this precluded me from ever tackling "hard" topics.
I was thirty-something years old when I finally realized math has a well-documented maturity model, just like emotional maturity or financial maturity. This realization inspired me to go back and take a few math classes that I had previously labeled as "too hard," with the mindset that I was progressing my math maturity.
My point is that choosing an "age-appropriate" (in terms of math maturity, not actual calendar age) textbook is important. I also find it extremely helpful to chat with people who are more mathematically mature than I am, in the same way it's helpful to seek advice from an older sibling.
“Until a man is twenty-five, he still thinks, every so often, that under the
right circumstances he could be the baddest motherfucker in the world. If I
moved to a martial-arts monastery in China and studied real hard for ten years.
If my family was wiped out by Colombian drug dealers and I swore myself to
revenge. If I got a fatal disease, had one year to live, devoted it to wiping
out street crime. If I just dropped out and devoted my life to being bad.
Hiro used to feel that way, too, but then he ran into Raven. In a way, this is
liberating. He no longer has to worry about trying to be the baddest
motherfucker in the world. The position is taken. The crowning touch, the one
thing that really puts true world-class badmotherfuckerdom totally out of reach,
of course, is the hydrogen bomb. If it wasn't for the hydrogen bomb, a man
could still aspire. Maybe find Raven's Achilles' heel. Sneak up, get a drop,
slip a mickey, pull a fast one. But Raven's nuclear umbrella kind of puts the
world title out of reach.
Which is okay. Sometimes it's all right just to be a little bad. To know your
limitations. Make do with what you've got.”
Textual instructions (if you don't want to watch the demo video):
The left column is the shapes library. The center area is the canvas/workspace. The right area is an outline view of the shapes in the canvas.
To add a shape to the canvas, drag your mouse from the shape in the shape library to the workspace.
To move a shape, drag inside it, when the whole shape is highlighted red.
To resize and rotate a shape, drag the edge of it, when it is outlined red.
To delete a shape on the canvas, right-click it and choose Delete Shape. (You currently cannot delete shapes directly from the outline, nor can you delete shapes from the shape library.)
To resize a shape non-uniformly or add skew, hold shift while dragging its outline. You can use this to make line segments and rhombuses, among other things.
Click the + button in the left column to start editing a new shape.
Remember that you can add any shape to the canvas by dragging it from the shape library. This also applies to shapes you create yourself.
Click on any shape in the shape library (except for the starting circle and square) to edit it in the canvas.
You can see the thumbnail of what's in the canvas in the current shape in the shape library. You can also see the shapes your shape is composed of in the outline. Everything is updated live.
To pan across the canvas, drag outside of any shape, in the white area. To zoom into or out from the canvas, scroll with the mouse wheel while pointing inside the canvas.
You can drag even the shape you are currently editing from the shape library into the canvas. That is what makes this recursive drawing rather than merely nested drawing. You will see the current shape repeated relative to itself. (You might call the shape inside itself a "recursive shape instance".)
You can move, rotate, and resize a recursive shape instance to change its repetition pattern. For instance, if you resize the recursive shape instance to be smaller, you will see a line of recursive shapes shrinking forever. If you edit one of the repetitions of a shape, it will edit the whole pattern.
You can go back and edit a sub-shape, and every shape including that shape will update live.
> Stripped of anything else, neural networks are compositions of differentiable primitives
I’m a sucker for statements like this. It almost feels philosophical, and makes the whole subject so much more comprehensible in only a single sentence.
I think François Chollet says something similar in his book on deep learning: one shouldn’t fall into the trap of anthropomorphising and mysticising models based on the ‘neural’ name; deep learning is simply the application of sequences of operations that are nonlinear (and hence capable of encoding arbitrary complexity) but nonetheless differentiable and so efficiently optimisable.
All these suggestions are great but I would add two more:
1. find a good text book in the field as narrowly construed as possible, and just browse the table of contents.
This will give you a quick map of the field, and a place to put all the info you gain.
2. Read the management discussion in the 10k of a public version of the company you work at and 2 leading customers, and the transcript from the earnings call.
I always keep a notebook around. But I use it an extremely short-term external cache system to offset my brain.
I jot down stuff I need exclusively for the current day. An issue number. The phrase " Bob about the thing". A doodle I make in order to give my hand something to do while I attend a meeting.
The pages become completely unintelligible very soon, sometimes during the same day. It is useful - taking something out of the head and putting it in paper allows me to spend less time worrying "I must make sure I don't forget that issue, I must email bob about the thing". Once the issue is addressed, or bob is emailed, 80% of the time the task is done, and the notebook has fulfilled its primary function.
Months from now, it will have become complete jibberish, I won't be able to understand any of it. But I can still enjoy the doodles.
This is pretty screwed up, because it kind of trains your brain to be a gambling brain. To crave the instant gratification, the high of risk taking etc.
I think the time has come when android and ios should make a special versions of their online shops for children and teenagers and simply ban those types of games from those shops. This will at least prevent children from having their brains rewired. Of course this will remove the vast majority of games from the online shops, but so be it.
I remember my experience with these types of games. I wasted a lot of time and got a lot of needless frustration but I never once paid those bastards a single cent. There was a game called Candy Crush Saga. It was very popular in its day. Their in game super power currency was something called "lollipop hammers." They made one candy disappear. It seemed a pretty simple game, assuming the game boards (i.e., the candys) were randomly generated, but of course they weren't. They were carefully created so that they present problems that are very difficult to solve under the game's usual rules but were easy and downright delightful to solve using a lollipop hammer.
I would look at the gameboard for a long time and imagine all the great combos I would achieve, if I could only get rid of a single pesky candy, but alas, one had to use a lollipop hammer for that. I actually ended up having a lot of respect for the designers, it was not easy to design a board that would be incredibly difficult unless one candy was removed but if that candy was removed, it would all solve itself in an explosion of combos and extra points.
I was very proud that I solved so many of these boards without using a single lollipop hammer. But my reward would always be a board that is even harder and where the temptation to use the hammer was even greater. Eventually I decided that this is simply torture. I did not mind hard puzzles, I liked solving hard puzzles, but the temptation of the lollipop hammer being always there and me having to constantly use willpower to avoid clicking on those hammers made the whole experience miserable.
So quit and deleted the game, but I can proudly say that I never paid for a single lollipop hammer and I left 8 of the 10 hammers you get for free unused. I hope some random King employee looked at my cancelled account with so many puzzles solved and yet so few lollipop hammers used and felt a little pang of defeat. But I doubt it.
Ugh. Another one of these blog posts. Engineer is a good communicator. Has a blog. Gives talks. Then starts navel gazing: "Hey, other engineers should focus more on what I'm good at." I'll get downvoted for this last one: The blog has a link at top of page <<Get a FREE coaching session with me.>> Cringe. This whole thing reads like an advert for his services.
Financial advisers are exactly the same with their "weekly newsletters" that seem to do nothing more than make you anxious about your financial portfolio and think: "Hey, I need some coaching!"
