I have a chronic problem with people taking a partial description of some of my math, and then complaining about all the things that are missing. Well, those things are in the part you didn't see! Ordinary differential equations are solved with provable bounds of arbitrary accuracy in Chapters 19 and 20 of The End of Error, and this is not possible with traditional interval arithmetic.
With unums 2.0, you can specify your own lattice of preferred exact values between 1 and infinity, albeit with recommended guidelines, and then the projective real line is auto-populated with the reciprocals, the negatives, and the open intervals between exact values. There is a continuum between table lookup (low precision) and float-type calculation (high precision), and I am mainly interested in table lookup for low-precision arithmetic. It is surprising how often it is better to use a small number of bits but obtain rigorous bounds (that do not grow exponentially like they do with intervals) than it is to use double-precision or even single-precision IEEE floats and obtain 15 or 7 decimals of answer, none of which decimals are guaranteed correct.
The idea is to work toward software-defined numerical systems that can be optimized for particular applications, for the purpose of saving bits and thereby reducing energy/power requirements while simultaneously improving answer quality and possibly increasing ease-of-use.
With unums 1.0, too many people had hopes of simply replacing their floats with unums and getting an improvement. While that does happen in many cases, in the general case where there are hazards of the dependency problem or the wrapping problem, you need some knowledge of uboxes, and the Julia and C versions tend to gloss over this essential aspect of the computing environment. I think unums 2.0 solve this by making the SORN the fundamental way of representing arguments to arithmetic operators, since they have their "ubox" aspect built into them and there is no need to choose a preferred ULP size.
The bit about creating a custom tabled implementation per application seemed pretty clear, and awesome!, from the slides (or at least I thought so)
Are there any good notes at least on a methodological level for this 2.0 design? It's super fascinating, and colleagues of mine on the Modelling side loved reading the deck. Also would like to see more of the ideas :)
I'm currently working with John on producing a Julia-based prototype of the original unums (work has stalled until I am getting paid again). A lot of the decision choices in this are based on annoying things that came up during the development. For example: subnormal numbers are a horrifying pain in the butt (but there are also really good reasons for having them)
I think there might be something to this "new unum" especially if there can be a shortcut that trims the size of the lookup tables when going to higher precision.
PS if anyone has some interesting math-ey contract work let me know.
Do you know if there's been any thought on extending this to arbitrary algebras? Seems like you've already got the mechanism to define a partition of the space, and the lookup table describing the behavior of an operation with respect to the subsets of that partition, so this might have utility beyond arithmetic.
Huh. I wonder how fast the lookup tables for various operations could be if they were built using some kind of programmable logic. Then you could theoretically just interpret the bit patterns to be whatever kind of number-like object you want, and program the lookup tables accordingly.
Right, now that I think about it, it's only coincidental that the objects in question represent the subsets in a partition of the reals.
It seems what we have here is a finite set A (in arithmetic, A would be a finite collection of sets which constitute a partition of the reals) and two operations: f, g: P(A) × P(A) -> P(A), taking a pair of subsets of A to another subset of A, with the property that f(W ∪ X, Y) = f(W, Y) ∪ f(X, Y), and so on. This can all be represented as a bitmask table of size |A|^3, or half if commutativity is enforced.
We also have the "inverse" which takes X and Z, and returns the set {a ∈ A : f(X, {a}) ∩ Z ≠ Ø}.
If I'm not mistaken, the mechanisms for doing all of this in hardware would come for free, and it represents quite a variety of problem domains beyond arithmetic. Perhaps some arithmetic-specific optimization would be ruled out (e.g. the bit twiddling to get additive or multiplicative inverses).
So, instead of ROM, it would be SRAM... no "reprogrammable logic" required. The difference between ROM bit cell and a RAM bit cell is around ~3x larger, and uses a lot more power wise for read and writes.
The algebraic integers in Q(√5) — namely, numbers of the form aτ + b, where a and be are integers and the golden ratio τ = (1+√5)/2 — are quite interesting. If you make vectors of them, you can make a dodecahedron, iconsahedron, 120 cell, 600 cell, etc.
This is great practically for modeling quasicrystals, as well as things like buckyballs, certain types of viruses, architectural structures, ...
If you want to play with the symmetries, I recommend buying some Zometool construction toys, https://www.zometool.com
I'm not sure what the etiquette wrt. cross posting comments is, but here goes.
On Reddit I divulged into why this doesn't pass my sniff test. This seems to be dressed up interval arithmetic with a worse subset of the number line as the bounds.
Did you read Gustafson’s book? I’m guessing not. Considering Gustafson had a distinguished career in parallel computing, winning top awards from ACM and IEEE, and working as director of an Intel research lab and then as a senior fellow at AMD, he probably knows at least a little bit about the subject.
I recommend reading the book. Even if you ultimately decide that they’re an offbeat set of ideas that won’t manage to hit the mainstream, it has very clear exposition and does a nice job explaining the current floating point system with great diagrams.
My problem with it is mainly that I think he goes off the rails a little bit in the end when he starts throwing standard analysis techniques out the window and advocating that people solve more problems with brute force methods. This could be useful for some problems but I think he oversells it.
Also, unums are incompatible with any of the significant amounts of work that have gone into studying numerical stability, etc., for floating point computations, so it’s hard to judge a priori how many types of scientific computing algorithms might need to change.
I think your assumptions about what is slow or fast are off base though. Most of the time spent in modern computers is moving data around, not performing arithmetic per se. Adopting something like unums would require quite significant architecture changes, but with a machine designed for it most computations shouldn’t necessarily end up slower than with standard floating point. [This is a bit of a guess, since these don’t exist, making them hard to benchmark.]
Indeed I have not read the book. I'm not really inclined to buy it, but I did skim the Amazon preview which seemed not to include any of the interesting sections (aka. chapter 6).
> I think your assumptions about what is slow or fast are off base.
Large lookup tables are unwieldy. If massive lookup tables worked above 8 bit or so, why wouldn't they be used already?
> these computing number systems [...] have a dyadic system wherein every represented number is some multiple of a power of two.
For IEEE floats, sure. For Unum IIs, logarithmic number systems or level index number systems, no. Except in the sense that every real number is a multiple of some power of two.
It's true that Unum IIs include intervals, but I factored that out of the question when I pointed out it's equivalent to just supporting open and half open intervals without them. And then you do indeed have a subset of the reals.
I don't really get why you say you're not representing real numbers. 1 is a real number, and is exactly representable in IEEE floating point. (Technically infinities and NaN mean it's not strictly a subset, but that's pedanticism.)
If we’re dealing with numerical problems, “real numbers” are not directly computable. All we have are approximations and (in the case of unums) imprecise bins.
1 is an integer. It’s easy to make a computer representation for integers, especially integers with some finite bound. How about exp(1) or arccos(-1)... suddenly not precisely representable anymore.
IEEE floats are exact. That there exist real numbers that aren't IEEE floats is why we say IEEE floats are a subset of real numbers, and why operations on IEEE floats have to round to a representable value.
Floats are also a subset of the rational numbers. To be specific, they’re a particular subset of the dyadic rationals with a bounded number of significant figures and a bounded exponent.
I think it’s misleading / counterproductive to talk about IEEE floats as subsets of “real numbers”, because e.g. Math.PI is a clearly different thing than the “real number” π (whatever that might be).
Calling floats “exact” is also IMO problematic, since in practice every computation using floats involves rounding.
(I also think it’s problematic for Gustafson to call the numbers in his system “real numbers”, even if you could in principle build new hardware with the idea to perform arithmetic on any particular (decided before hardware design) small set of real numbers you want.)
I say subset of the real numbers because other number systems don't restrict themselves to rationals. Logarithmic number systems, for example, almost never express rational numbers.
It's true that most floating point computations involve rounding, but there's no contradiction in that. If you think floats are inexact you can get into a lot of confusion about things; you might miss, for example, that `(a + b) / 2` is exact (except on overflow to infinity, or languages that don't enforce floating point correctness) or that Javascript's integers are actually just floats. Considering floats as inexact might lead you to believe they represent ranges of numbers, eg. x ± ½eps, which is confusing. I've seen more damage from this belief than most other myths about floats, at least among experienced programmers.
My point though is that every non-trivial computation involving floats includes some rounding. Typically the inputs are also coming from some inexact source.
The whole point of floating point arithmetic is to be approximate, maintaining only a particular precision because otherwise the number of bits in the answer doubles every time we multiply. If someone cares about exact arithmetic, floats are the wrong tool for the job.
As such, to do pretty much anything with floats you have to keep some idea in mind of the relative accuracy of various values, and the condition number of every operation/algorithm. None of this is explicitly included in the number, you just have to track it manually.
Instead of thinking of a float coming out of some algorithm as an exact value, you should generally think of it as a representative of some implicit range of possible values, a range typically somewhat bigger than ±½eps.
I gave examples of floating computation that doesn't involve rounding. A lot of programs depend on it; Javascript ones in particular. In fact, in a case like `random() -> [0, 1)`, one often relies on the exactness of the bounds.
Rather than thinking of a float as some range of values, you should think of it as a randomized sample inside some (probably unknown) range.
The SORNs / Unum IIs talked about in these new slides are only described for a relatively low precision, for which a lookup table wouldn’t be impossibly large.
It’s not clear whether a pure lookup table approach would scale to 32-bit or higher numbers. The last slide lists as a future research question, “Create 32-bit and 64-bit unums with new approach; table look-up still practical?”
Ergo "multiplication is reasonable" rather than "very fast". 3 cycles doesn't seem like a ton to me (searching suggests it's actually 5, but that's still not a ton for a 64 bit multiply IMO).
Other than mathematical correctness, which was why John started working on unums, one of the coolest things about "Unums 2.0" as shown in this presentation is that all four arithmetic operations (add, subtract, multiply, divide) can take the same time. The 64 bit IEEE FPU my company is developing is able to do add and subtract in a single cycle, a multiply or fused multiply add in 3 cycles, and a divide in 17 cycles... all of which are pretty much the fastest reasonable implementations you can get.
To give you an idea of actual time to do those things, doing an add takes around 300 picoseconds, while the multiply is around 3,000 picoseconds. An order of magnitude increase in time to do an operation is a lot when it comes to hardware complexity (and thus associated area and power cost).
This is more or less my take as well. Beside the issues that you highlighted, the thing that really doesn't pass the sniff test for me is the completely unsupported claim that with SORNs "Uncertainty grows linearly in general". I simply don't see how this can possibly be the case; interval growth in arithmetic is a straightforward consequence of finite precision, independent of what representation is used. The only way you can avoid it is for representations to grow arbitrarily, which introduces its own set of problems.
I am out of the loop, but I have a few questions, from a scientific perspective.
Inputs into system can be either boundary/initial conditions and constants. It seems like here, you essentially have to design the number system to match the problem, is this correct?
How can someone solve odes with this? I can see this solving constraints (which is basically algebra) but how about dynamical systems, especially when you don't quite know the behavior of the system, specifically, the bounds of your solution (which allow you to define your interval boundaries)? For example, what about chaotic systems (which are difficult in normal math too, I know). How about min/max problems (finding the global minimum of a function)?
It sounds like one would have to iterate your number system until you find your solution, which sounds like just offloading the difficulty somewhere else. (For finding the minimum of a function, you iterate as well, essentially). I think it makes sense that this does some problems well, but I need help to buy that it does all problems well.
A chaotic system is by definition also beyond the reach of existing methods. With unums you’ll presumably get numbers that are less and less precise as you step through the computation, whereas with standard floats you get a “precise” answer which you know must be substantially wrong.
I’m not sure to what extent if any unums can help solving ODEs. I haven’t seen any ODE experts examine unums yet or write down their impressions.
>A chaotic system is by definition also beyond the reach of existing methods.
Not quite, it mainly is a system that is sensitive to initial conditions. For that reason, for a "range" of initial conditions wouldn't be helpful for a chaotic system, as you would get wildly different final results over that range for a particular set of conditions (ie., constants in the ODE), unless the range is small...and bam, you have a floating point with a +/- on it and probably an unum with a large number of intervals.
For the weather, for example, you can still tell someone what it will be (with some degree of accuracy) tomorrow, but the weather three weeks from now is unpredictable due to that sensitivity to the initial conditions (ie. now).
Uhh. That article. It says: "Because computers – which are machines of precision and exactness – are often made to deal with unprecise and inexact numbers (such as pi, and irrationals) [...]"
It just bugs me. There is nothing unprecise or inexact about the number pi. Ahh, very interesting topic nonetheless.
If you look at slide 12 of the linked presentation of this thread, you can see that John says that Pi is an exact number... The VR world article tries really hard, but falls flat when it comes to actually describing unums (I've been working with John and unums for a little over a year at this point).
In fact, in many cases, you can do exact arithmetic with (some) irrational numbers - pi among them. Of course, this needs a smarter representation of numbers than "string of base-2 digits".
Does anybody know what is the state of unums? That thing pop up on HN every now and then, but I still can't see it in practice. It is definitely something promising. Is there any development going on to employ this theory in practice?
Its in the same development hell, as all the other great ideas that surface now and then (new programming paradigms etc.).
Its ready to use, but the world refuses to use because the cost of transition from the c/c++ Codebases or in this case the software/hardware ieee float standard is just to high.
If any of these is ever to succeed it will not be by superiority of concept, but by backwards compatibility - as in either being just another extension to the existing ieee floats or by offering proofen automatic port-functionality for a existing codebase.
Its a shame, as many superior innovations basically remain sidelined, but the pre-existing infrastructure is in production and you never change a running system. You would need a easy integrate able hardware option, that allows a simple auto-conversion of a preexisting codebase and which would then run in parallel for quite a while, to even just be considered :(
I'm not sure you need a new unary "/" operation. Just use "1/" as a unary operation and you get the same results. Trying to do "/" as a new unary operation will probably just lead to confusion.
For instance, when I read an equation like this:
Information = /uncertainty
"Information = 1/uncertainty" is just plain clearer.
The point of the unary / is not the programming language syntax per se.
If you want, in essentially every programming context where `-x` is meaningful, you can write `0-x` instead. It’s arguably less confusing.
The point he’s trying to emphasize is that in his proposed number system, every exactly represented number has an exact multiplicative inverse, which can be computed by flipping bits, analogous to computing the additive inverse with a floating point number or signed integer.
I'm not talking about programming syntax, I'm talking about communication. It doesn't matter that `-x` and `0-x` are equivalent; `-x` is a well-known convention and `/x` is not. (When I read '/x' my brain immediately jumps to trying to interpret that as a malformed expression or a directory or something.)
And you don't need to make a point that every number has a multiplicative inverse. That's already established by simply pointing out that these are the projective reals and that 1/0 = ±∞. All non-zero reals already had multiplicative inverses just as they had additive inverses.
Prolly an etiquette violation to crosspost to reddit, but I wrote a little "bootleg" version of unums in C++. There seems to be some inherent problems with the unum2 concept though:
* The smallest non-zero value is 1e-9.
* The biggest non-infinite value is 1e9.
* The first integer that shows roundoff error is 3.
* The integer 100001 loses its last digit. 100001 -> unum -> float = 100000.
With unums 2.0, you can specify your own lattice of preferred exact values between 1 and infinity, albeit with recommended guidelines, and then the projective real line is auto-populated with the reciprocals, the negatives, and the open intervals between exact values. There is a continuum between table lookup (low precision) and float-type calculation (high precision), and I am mainly interested in table lookup for low-precision arithmetic. It is surprising how often it is better to use a small number of bits but obtain rigorous bounds (that do not grow exponentially like they do with intervals) than it is to use double-precision or even single-precision IEEE floats and obtain 15 or 7 decimals of answer, none of which decimals are guaranteed correct.
The idea is to work toward software-defined numerical systems that can be optimized for particular applications, for the purpose of saving bits and thereby reducing energy/power requirements while simultaneously improving answer quality and possibly increasing ease-of-use.
With unums 1.0, too many people had hopes of simply replacing their floats with unums and getting an improvement. While that does happen in many cases, in the general case where there are hazards of the dependency problem or the wrapping problem, you need some knowledge of uboxes, and the Julia and C versions tend to gloss over this essential aspect of the computing environment. I think unums 2.0 solve this by making the SORN the fundamental way of representing arguments to arithmetic operators, since they have their "ubox" aspect built into them and there is no need to choose a preferred ULP size.