Nothing fancy. I made these with some pretty simple hand written scripts in javascript rendering to canvas: lots of fiddly little boxes moving around are simpler to script than to hand animate. (If I were to do much more of this I might rewrite these in blender since it has much nicer authoring tooling and export control.)
Quantum mechanics is needed to explain any microscopic phenomena in chemistry and biology - that is not at all in dispute.
The odd set of claims is that somehow biology has 1) figured out how to preserve long-range entanglement and coherent states at 300K in a solvated environment when we struggle to do so in cold vacuum for quantum computing and 2) somehow still manages to selectively couple this to the -known- neuronal computational processes that are experimentally proven to be essential to thought and consciousness.
This more or less amounts to assertions that "biology is magic" without any substantive experimental evidence over the last thirty years that any of the above is actually happening. That's why most biophysicists and neuroscientists don't take it at all seriously.
I am a complete lay person, so I feel a bit silly challenging someone who is clearly an expert, but the idea that a physical process that has had countless trillions of generations of mutation and change, "figuring out" how to use an underlying feature of the universe to optimise, isn't far fetched at all.
It seems that the most powerful force in the universe is simply, survival of the fittest.
Yeah, a bit like when I try to lift my leg, I actually think about how to activate my neurons so that muscle fibers contract one by one...
That's definitely not what happens.
That at some level we have quantum phenomenon doesn't mean that everything occurs at the quantum level.
This referenced paper seems like primarily a theoretical modelling paper (almost all of its figures are simulations?) that contains as far as I can read 3 (!) actual experimental measurements in bulk on a fluorospectrophotometer. The claim is that the observed increased fluorescent quantum yield (QY) of microtubules over tubulin can be explained by the ideas in their simulations.
It's hard to buy that their proposed stories are the simplest explanation for these few measurements. Much more boring phenomena can influence QY. e.g. simply occluding fluorophores from the bulk solvent can have a huge influence on QY and spectra. (I used to design biological fluorescent reporter reagents...)
This seems like a theoretical modelling paper (all of its figures are simulations?) that contains as far as I can read 3 (!) actual experimental measurements in bulk on a fluorospectrophotometer. The claim is that the increased fluorescent quantum yield (QY) of microtubules over tubulin can be explained by the ideas in their simulations.
It's hard to buy that their proposed stories are the simplest explanation for these few measurements. Much more boring phenomena can influence QY. e.g. simply occluding fluorophores from the bulk solvent can have a huge influence on QY.
The static shape of a protein doesn't automatically give you a prediction of its functional properties. There's a hell of a lot more biophysics going on that we have no predictive models for that are needed to understand catalysis, allostery, assembly, etc etc etc. We don't even have good comprehensive data for any of that (compared to sequences or structure) to model with.
Fold prediction is an incredibly useful tool for scientists and genetic engineers to help design new proteins, but it doesn't magically solve molecular or cell biology. Designing new functions and mechanisms is still going to involve a huge amount of labor and brute-force experimentation.
Yeah we used to use this in our older models years ago... I don't recall the details exactly, but I don't think it ever did very much.
I certainly don't think it will help at all with stability. Things like Q/K layernorm are better tricks for softmax stability when scaling: https://arxiv.org/pdf/2302.05442.pdf
> I don't recall the details exactly, but I don't think it ever did very much.
How would you have known if the trick actually reduces the outliers in the weights? Even if the transformer quality does not improve overall, having less outliers as a result is very beneficial for more accurate quantization of the data
He's questioning the statement: "I don't think [the trick] ever did very much", because no one has yet looked at whether the trick helps reducing outliers in very large models. If it does help with this, as the blog author believes, then it is indeed a very useful trick.
Is he? A surface level reading suggests he's asking "how would you know".. and the answer is... by looking at the parameters. People do that.
>> because no one has yet looked at whether the trick helps reducing outliers in very large models
Given a softmax version doing exactly as the blog post says is baked into a google library (see this thread), and you can set it as a parameter in a pytorch model (see this thread), this claim seems off. "Let's try X, oh, X doesn't do much, let's not write a paper about it" is extremely common for many X.
Yes, I assumed that checking the weights for presence and amount of outliers is not something that is usually done and effects on this can be overlooked. If my assumption is wrong and researchers do usually look at such metrics, then my question is not very relevant.
This is a dumb argument. Sick animals were probably culled immediately by the farms to avoid getting blamed.
As a 2-decade genetic engineer: there are no genetic "markers" pointing to a lab leak, there's really no sign of unnatural manipulation in the sequence.
Indeed, the government cracked down on wild animal farming at the beginning of the pandemic.
When you hear that "X thousand animals were tested," it's not the types of wild animals that are the likely culprit. It's cows, pigs, sheep and the like. It's a complete red herring.
Passage through humanized mice wouldn't leave signs of unnatural manipulation. It's still pretty suspicious that COVID was so transmissible between people from the outset, and no evidence of it circulating in local populations was found.
The question was not "why was it a pandemic", yhe question is, why was it so transmissible when earlier outbreaks, like SARS, had relatively much poorer transmission? That's the typical profile of new viruses.
The virus becoming more infectious over time is exactly my point. That's typical. What's not typical is the virus already being so infectious right from the start. Normally a zoonotic transfer circulates poorly in the human population before it mutates to become more infectious for the host. COVID-19 was already excellent at infecting humans from the earliest points we've found. That's very, very unusual.
As someone who was a genetic engineer for a long while, watching HN talk about dodgy papers like this is painful.
This paper posits a completely crazy cloning strategy that makes no sense (ie doing something far more convoluted than typical bsaI/bsmbi seamless cloning workflows that breaks the whole point of "seamless" workflows), and then tries to use that to make a case for a genomic signature that we could look for. They then look at a handpicked set of viral genomes, but leave a bunch out and duplicate others (I think WIV04 and WHu are the same), and largely seem to be observing without realizing it that yes, recombination occurs among these viral lineages.
This isn't even getting into the fact that a restriction-ligation based cloning strategy would leave glaringly obvious fingerprints behind in the form of the hundreds of nucleotide differences that are present outside the cutsites across the lineages... it would be blindingly obvious if someone just cut-and-pasted sars-cov-2 from other studied genomes.
I'm pretty tired of debating people who don't know biology here. Using seamless cloning methods is super common - but they don't work like the paper authors suggest they do. I misspent my youth doing reactions and workflows like these for over two decades.
What they're observing is homologous recombination between strains - all the sites they're claiming are found in nature.
Again - there would be a genetic signal the strength of the noonday sun burning your eyes out if sars-cov-2 was made by cut-and-paste at these sites. You wouldn't need this ridiculous circular argumentation to prove that point.
They're not looking for the existence of the sites they're looking at the distribution of them. Their paper shows that in natural viruses the distribution is distinct from synthetic viruses.
The proposed classifier is how uniformly distributed these sites are, not that the sites exist.
> circular argumentation
Can you elaborate? They select a site based on commonly used it is (and maybe also the fact that Baric and WIV published on it). Then they found evidence of it being used. What's "circular"?
> great depth about how ridiculous this paper is
The crux of his argument (and yours) is tweet 10/ in that thread -- "You CAN actually do it like that, but why should you?" which is pretty weak.
Often, if not most, of the time the engineering I come across hasn't been done in the slickest most optimal way. The fact that there's a better way to do something isn't proof that everyone's been doing that the whole time.
> pretty tired of debating people who don't know biology
I started my PhD in math biology but for whatever reason I just couldn't get along with the PI or any of the postdocs. I don't know what it was. I eventually switched to just math. Oh well.
I used to second guess my decision but in the nearly two decades of research since I've never come across the level of smugness and credentialism that I now see coming from that field. Every disagreement is met with remarks about "kindergarten molecular biology" or referring to other researchers diagrams as "cartoons". Now I don't second guess anymore.
Perhaps if you're so bothered by the people here you should keep your posting to virological.org or simply talk with the biology profs on Twitter directly.
You don't need access to a proprietary database to refute that sars-cov-2 wasn't copy-pasted at these restriction sites. We have public sequences of the closely related coronavirus strains. The unnatural SNP pattern would be absolutely obvious if someone patched together different lineages around these specific conserved RE sites. Instead we see a set of conserved RE sites related across the publicly known strains by homologous recombination.
What I've tried repeatedly to impress upon people here is that most routine cloning strategies leave pretty clear signatures, and the idea that a lab would go so far as to eliminate these signatures for such mundane virology work is tantamount to a much more elaborate conspiracy theory.
Your assessment relies on two assumptions. The first being that samples from the “public sequences” are identical to the sequences China’s labs purposefully unpublished early in the pandemic. The second being the assumption there was no nefarious purpose to the well-documented gain of function research taking place in Wuhan.
I don’t need to be a biologist to call your assumptions out as junk science.
>I'm pretty tired of debating people who don't know biology here
Too bad. This is a matter that affects every citizen in this country, and the experts lost their credibility a couple of years ago at least. The rubes will keep shouting their barbaric yawps over the roofs of the world.
No, they didn't loose their credibility. To political debate they were never granted it, and between themselves as peers it hasn't been lost.
Your perception and reality diverged and your claims they lost credibility lacks a crucial qualifier: 'to me' -which I and many many others discount, even at the volume of American science scepticism. You actually aren't a majority, anywhere and you don't define scientific credibility any more than politics does.
It is weakening somewhat. The sheer volume of papers that pass in high impact journals, and then are later pulled after X years with minor repercussions, seems at least to me to be an alarming trend. That, paired with the cronyism I've personally witnessed between editors and professors... as someone entrenched in the field, I have to say, I'm surprised more people aren't jaded.
> This is a matter that affects every citizen in this country, and the experts lost their credibility a couple of years ago at least.
I don't agree. What we did see is ignorance and completely absurd conspiracy theories taking the center stage while experts were being sidelined or even completely removed from the discussion.
exactly the top comment of this reply/thread, that's why i wrote such comment on it.
"As someone who was a genetic engineer for a long while, watching HN talk about dodgy papers like this is painful."
Understand: "as a great scientific with long career, i feel depressed to see so much stupidy."
Trying to attract my sympathy for his "painfull feeling", therefore trying to positively bend my opinion in his favor.
"This paper posits a completely crazy cloning strategy that makes no sense (ie doing something far more convoluted than typical bsaI/bsmbi seamless cloning workflows that breaks the whole point of "seamless" workflows), and then tries to use that to make a case for a genomic signature that we could look for. They then look at a handpicked set of viral genomes, but leave a bunch out and duplicate others (I think WIV04 and WHu are the same), and largely seem to be observing without realizing it that yes, recombination occurs among these viral lineages."
Im not a specialist here, but this babbling just suggest me someone unable to think out of the box, restricting all possibles to only what he can master.
"This isn't even getting into the fact that a restriction-ligation based cloning strategy would leave glaringly obvious fingerprints behind in the form of the hundreds of nucleotide differences that are present outside the cutsites across the lineages... it would be blindingly obvious if someone just cut-and-pasted sars-cov-2 from other studied genomes."
"would leave" - "it would" - To me this is legalist wording. A way to suggest something without actually affirming it. cause the dude is not sure at all in fact.
So may be the core idea of his message is scientifically groundede and valid, but the way he expresses it is far more closer to the 'trust me bro' stance than critical neutral scientific phrasing.
> Im not a specialist here, but this babbling just suggest me someone unable to think out of the box, restricting all possibles to only what he can master.
This is a prime example of Asimov's quote: "Anti-intellectualism has been a constant thread winding its way through our political and cultural life, nurtured by the false notion that democracy means that 'my ignorance is just as good as your knowledge.'"
Well, at which extent this applies to our current case ?
Or is it just a way to look smart, by just citing a famous reference without contextualising to our current issue ?
Alevskaya talks about their experience as a biologist, and what their take is on this paper. And your response was, “I don’t understand any of that, so there’s probably nothing there to understand”.
This is not what i said. i suggest you to understand what you read.
first the guy claims he is a scientist and continue with some obfuscated babbling mixing emotional tone and hazardous outcome. Which at first and second read, isnt a scientific neutral analysis and critics.
Therefore it left only the "trust me bro" posture. Which is far from beeing efficient at all.
There's a big difference between not caring about stability, and being willing to trade precision for better memory bandwidth for an application that doesn't benefit from increased precision. When doing large training jobs on TPUs, stability is paramount! It's true that you have to know more about what you're doing when you reduce bit-depth - the horrors of floating point are harder to ignore, and it's wildly inappropriate for many scientific computations. However the reduction of bit-depth is likely to continue as we seek to make modern models more efficient and economical to train and use.
What does this mean in practice? For ML, we usually don't care if a weight is 0.05 or 0.10 cause we have millions of weights. We do care if one 1.237e+27 instead of 1.237e-3 though.
Numerical errors have the annoying tendency to accumulate if you're not careful. So doing one matrix operation with low precision might be okay, while doing a dozen might completely garble your result.
This is not that relevant for ML. Each gradient pass will re-compute your cost function and the gradients so errors are not likely to accumulate. The main thing is to not make errors big enough that you end up in a completely different part of the parameter space derailing progress which is what the above commenter points out.
I am familiarizing myself with recurrent neural networks and getting them trained online is a pain - I get NaNs all the time except for very small learning rates that actually prevent my networks to learn anything.
The deeper network is, the more pronounced accumulation of errors in online training is. Add 20-30 fully connected (not highway or residual) layers before softmax and you'll see wonders there, you won't be able to have anything stable.
This isn't true in general. Very specific ML algorithms that were likely developed with years of blood and sweat and tears may have this kind of resiliency, but I've been in the the numerical weeds enough here that I wouldn't bet on even that without a real expert weighing in on it - and I wonder what the tradeoff is if it's true there. It's very easy to have numerical stability issues absolutely crater ML results; been there, done that.
I have some ~15 year old experience with the math behind some of this, but actually none with day-to-day deep learning applications using any of the now-conventional algorithms, so my perspective here is perhaps not that of the most pragmatic user. The status quo may have improved, at least de facto.
I'm not really sure there is evidence for that. In fact, depending on your interpretation of why posits[1] work, we may even have empirical evidence that the opposite is true.
When building a mcmc sampler I was too lazy to properly code a matrix approximation needed to avoid some mathematical black hole and the corresponding underflow. It was cheaper to just ignore the faulty simulations.
Turns out our results were better than the papers we compared to, both in time and precision.
I am not that familiar with ml, but can't you just ignore those faulty weights?
With MCMC, depending on application, it seems risky to just toss out the NaN/inf results. I'd guess these numerical issues are more likely to occur in certain regions of the state space you're sampling from, so your resulting sample could end up a bit biased. In some cases the bias may be small or otherwise unimportant, so the speed-up and simpler code of filtering NaN/inf results is worth it, but in other cases (like when the MCMC samples feed into some chain of downstream computations) the bias may have sneaky insidious effects.
I didn't think deeply about this back then since my parameter estimates where close/better than the literature I compared to, but now I'm interested in checking the distribution of those NaN/inf. If I recall correctly they were uniformly distributed throughout an adaptive phase.
When people talk about AI taking over the world, a funny image pops up in my head where a robot is trying to enter a frying pan. When you ask it why it's doing that, it says "because I feel like [NaN, NaN, 2.45e24, NaN]", which is a perfectly valid reason.
I'm not at all caught up with the this side of ML but my first instinct is that faulty weights would lead to interpretability issues. The numbers represented by NaN/Inf vastly outnumber the ones within precision range, so interpreting them is much more of a guess.
"A considerable group of numerical analysts still believes in the folk “theorem” that fast MM is always numerical unstable, but in actual tests loss of accuracy in fast MM algorithms was limited, and formal proofs of quite reasonable numerical stability of all known fast MM algorithms is available (see [23], [90], [91], [62], and [61])." https://arxiv.org/abs/1804.04102
To add to this comment (from someone who used to engineer proteins, and long ago DFT as well): DFT is only really decent at ground state predictions, computational chemists often have to resort to even more expensive methods to capture "chemistry", i.e. correlated electron-pair physics and higher-state details. Simulating catalysis is extremely challenging!
