Have a PhD in physics/astronomy, so—full disclosure—I'm not an expert here... I skimmed the paper and I have no idea what he's talking about.
Given that half of his eight references are to his own papers, and the other half are textbooks or the Heaviside's original work, I think we can assume that he's doing some very niche work or a crank.
KANs have learnable activations based on splines parameterized on few variables. You can specify a prior over those variables, effectively establishing a prior over your activation function.
Incredible, well-documented work -- this is an amazing effort!
Two things that caught my eye were (i) your loss curves and (ii) the assessment of dead latents. Our team also studied SAEs -- trained to reconstruct dense embeddings of paper abstracts rather than individual tokens [1]. We observed a power-law scaling of the lower bound of loss curves, even when we varied the sparsity level and the dimensionality of the SAE latent space. We also were able to totally mitigate dead latents with an auxiliary loss, and we saw smooth sinusoidal patterns throughout training iterations. Not sure if these were due to the specific application we performed (over paper abstracts embeddings) or if they represent more general phenomena.
I'm very happy you appreciate it - particularly the documentation. Writing the documentation was much harder for me than writing the code so I'm happy it is appreciated. I furthermore downloaded your paper and will read through it tomorrow morning - thank you for sharing it!
You can get a nice sample of papers using VVV data using the Astrophysics Database System [0]. I mostly study other galaxies, which usually aren't variable on human lifespan-like timescales. Stars can vary on these shorter timescales, and VVV has compiled a huge list of those objects.
At a quick glance, I'd say some interesting results include:
* New star clusters discovered in our Galaxy [1]
* Galactic maps of dust reddening and stellar metallicity (enriched elemental abundances in stellar photospheres) [2]
* Galactic maps of stellar ages throughout the disk plane [3]
* Cataloguing other galaxies behind the plane of our own Galaxy [4]
> Why not just say "incoherent output"?
Because the biggest problem with hallucinations is that the output is usually coherent but factually incorrect. I agree that "hallucination" isn't the best word for it... perhaps something like "confabulation" is better.
I appreciated a post on here recently that likened AI hallucination to 'bullshitting'. It's coherent, even plausible output without any regard for the truth.
More true to say that all output is bullshitting, not just the ones we call hallucinations. Some of it is true, some isn't. The model doesn't know or care.
While I have absolutely no issues with the word "shit" in popular terms, I'd normally like to reserve it for situations where there's actually intended malice like in "enshittification".
Rather than just an imperfect technology as we have here.
Many people object to the term enshittification for foul-mouthing reasons but I think it covers it very well because the principle it covers is itself so very nasty. But that's not at all the case here.
I have not been impressed by the perf. Slower than PyTorch for LLMs, and PyTorch is actually stable on AMD (I've trained 7B/13B models).. so the stability issues seem to be more of a tinygrad problem and less of an AMD problem, despite George's ramblings [0][1]
PBMs hold an incredible amount of power. The Acquired podcast did a marvelous breakdown of the American pharmaceutical industry while covering Novo Nordisk (https://www.acquired.fm/episodes/novo-nordisk-ozempic). If you have three hours to spare, I highly recommend giving it a listen.
The health insurers (or more accurately, managed care organizations) are loving the blame that the PBMs get.
It makes the issue just a little more complicated, and thus turns people off from reading into it.
There is no need to refer to health insurers and PBMs separately, they are one and the same with the same bosses for the vast majority of people.
So the complaint really boils down to health insurers not paying enough for medicine.
However, given that health insurer’s profit margins are low single digits, what is actually happening is they are squeezing their vendors wherever they can, however they can, and using these opaque rules is one way to do it. Not really any different than Geico paying Safelite, but due to myriad laws and different insurance plans, the complexity can be greatly increased in healthcare.
Also, Cuban is all talk, no walk. He just wants to try to be another middleman. The real innovation would be building medicine factories and selling medicine for cheaper, but no one really wants to do that since it is high risk low reward.
How much do you expect auto-context and clustering+re-ranking to help for cases in which documents already have high-quality summaries? For context, I parse astrophysics research papers from arXiv and simply embed by paper abstracts (which must be of a certain size), and then append (parts of) the rest of the paper for RAG.
So the point of AutoContext is so you don't have to do that two-step process of first finding the right document, and then finding the right section of that document. I think it's cleaner to do it this way, but it's not necessarily going to perform any better or worse. But then spRAG also has the RSE part which is what identifies the right section(s) of the document. Whether or not that helps in your case is going to depend on how good of a solution you already have for that.
I am an astronomer and was at a conference when this headline broke. Obviously we ignored this and read the actual papers, e.g. https://arxiv.org/abs/2404.03002. There you can see that the dark energy equation of state parameter w0 = -0.99 +0.15/-0.13, which is not at all significantly different than w0 = -1. That is, no time evolving dark energy. When combined with other cosmological results, the joint analysis slightly favors w0 > -1 rather than w0 = -1 at the ~2.5sigma level or so, but it depends which other results you factor in.
Briefly hopeful, I ask astrollama-7b-chat-alpha[1] "What color is the Sun?". It replies "The Sun has no color as it emits radiation across all wavelengths from ultraviolet to infrared. [...] there isn't an answer for what color the Sun truly is since it doesn't have one but rather produces every visible spectrum imaginable!". Sigh. Hmm, I wonder if LLM replies might be usefully mined to generate misconception lists?
Oof, yeah that's not a very good response. But I guess color is actually a technical term in research astronomy: it refers to the difference in two specified magnitudes (usually photometric filters). There's also the question of whether the sun is being viewed from the Earth or from space, since those will change the effective "transmission curve" of the detector system.
I asked: "What color is the sun when viewed from the ground with the human visual system?"
And got the following: "The Sun appears white to us on Earth due to its high temperature and lack of any significant wavelength-dependent absorption or scattering properties. However, if we were able to view it through specialized telescopes that could capture all visible light spectrum (not just the yellow/orange part), then it would appear as an extremely bright ball of blue light with some slight red tint. This is because most of the solar radiation consists of photons at ultraviolet and infrared frequencies which our eyes cannot perceive directly but can be detected by these advanced instruments. [...]"
An overall better response, but still not exactly right. Anyway, the base model was fine-tuned on arXiv/astro-ph abstracts, and I can't imagine too much discussion about the color of the sun in that training data set...
Nod. Though briefly asking E&M questions a few days ago made me think the "not exactly right"-but-seemingly-closer may be stopped-clock-ish. Very not at the point where latents are seemingly encoding deep structure about the world.
Given that half of his eight references are to his own papers, and the other half are textbooks or the Heaviside's original work, I think we can assume that he's doing some very niche work or a crank.