That is ABSOLUTELY incorrect. SSDs have enormous amounts of error detection and correction builtin explicitly because errors on the raw medium are so common that without it you would never be able to read correct data from the device.
It has been years since I was familiar enough with the insides of SSDs to tell you exactly what they are doing now, but even ~10-15 years ago it was normal for each raw 2k block to actually be ~2176+ bytes and use at least 128 bytes for LDPC codes. Since then the block sizes have gone up (which reduces the number of bytes you need to achieve equivalent protection) and the lithography has shrunk (which increases the raw error rate).
Where exactly the error correction is implemented (individual dies, SSD controller, etc) and how it is reported can vary depending on the application, but I can say with assurance that there is no chance your OS sees uncorrected bits from your flash dies.
> I can say with assurance that there is no chance your OS sees uncorrected bits from your flash dies.
While true, there is zero promises that what you meant to save and what gets saved are the same things. All the drive mostly promises is that if the drive safely wrote XYZ to the disk and you come back later, you should expect to get XYZ back.
There are lots of weasel words there on purpose. There is generally zero guarantee in reality and drives lie all the time about data being safely written to disk, even if it wasn't actually safely written to disk yet. This means on power failure/interruption the outcome of being able to read XYZ back is 100% unknown. Drive Manufacturers make zero promises here.
On most consumer compute, there is no promises or guarantees that what you wrote on day 1 will be there on day 2+. It mostly works, and the chances are better than even that your data will be mostly safe on day 2+, but there is zero promises or guarantees. We know how to guarantee it, we just don't bother(usually).
You can buy laptops and desktops with ECC RAM and use ZFS(or other checksumming FS), but basically nobody does. I'm not aware of any mobile phones that offer either option.
> While true, there is zero promises that what you meant to save and what gets saved are the same things. All the drive mostly promises is that if the drive safely wrote XYZ to the disk and you come back later, you should expect to get XYZ back.
I'm not really sure what point you're trying to make. It's using ECC, so they should be the same bytes.
There isn't infinite reliability, but nothing has infinite reliability. File checksums don't provide infinite reliability either, because the checksum itself can be corrupted.
You keep talking about promises and guarantees, but there aren't any. All there is are statistical rates of reliability. Even ECC RAM or file checksums don't offer perfect guarantees.
For daily consumer use, the level of ECC built into disks is generally plenty sufficient. It's chosen to be so.
I would disagree that disks alone are good enough for daily consumer use. I see corruption often enough to be annoying with consumer grade hardware without ECC & ZFS. Small images are where people usually notice. They tend to be heavily compressed and small in size means minor changes can be more noticeable. In larger files, corruption tends to not get noticed as much in my experience.
We have 10k+ consumer devices at work and corruption is not exactly common, but it's not rare either. A few cases a year are usually identified at the helpdesk level. It seems to be going down over time, since hardware is getting more reliable, we have a strong replacement program and most people don't store stuff locally anymore. Our shared network drives all live on machines with ECC & ZFS.
We had a cloud provider recently move some VM's to new hardware for us, the ones with ZFS filesystems noticed corruption, the ones with ext4/NTFS/etc filesystems didn't notice any corruption. We made the provider move them all again, the second time around ZFS came up clean. Without ZFS we would have never known, as none of the EXT4/NTFS FS's complained at all. Who knows if all the ext4/NTFS machines were corruption free, it's anyone's guess.
It really is not the same. Amazon was not profitable because it was building out logistics, and then AWS data centers. There were defensible moats around their business that their growth facilitated. Google built out data centers and fiber, again tangible assets they had that competitors did not.
OpenAI's spending is mostly buying compute from other people. In other words, OpenAI's growth is paying for Microsoft's data centers. The only real asset OpenAI is building are their models. While they may be the best models available today it is unclear if that those any durable advantage since everyone in the industry is advancing very quickly, and it is unclear if they can really monetize effectively them without the infrastructure to run them.
OAI moat is the brand and getting people habituated into using it more. The aim is to become synonymous to AI, same as Google is synonymous to online search(though in recent times, I am horrified to see that my non-techy friends now use Facebook to search first and then if not satisfied, goes to Google).
Also, there are immense lobbying everywhere covertly, if you haven’t seen it, look at the prominent news media, AI is now akin to “productivity gain” and “innovation”, and “adapting AI or we fall behind” attitude.
Look at all the hammers falling on different industries about energy waste, climate change while AI’s usage of massive energy is presented as “innovation and next generation of work and productivity gains to be unlocked”.
The target for OAI(and AI wave in general) is normal people, not tech people. And when people say AI, you conveniently see OAI emerge from news, social media, your colleagues, your boss, your friends.
Target did not figure out a teen was pregnant before she did. She knew she was pregnant, which led to changes in her purchasing habits. Target detected that and sent her promotions which disclosed her father who had need been informed.
Why build a new browser in C++ when safer and more modern languages are available?
Ladybird started as a component of the SerenityOS hobby project, which only allows C++. The choice of language was not so much a technical decision, but more one of personal convenience. Andreas was most comfortable with C++ when creating SerenityOS, and now we have almost half a million lines of modern C++ to maintain.
However, now that Ladybird has forked and become its own independent project, all constraints previously imposed by SerenityOS are no longer in effect. We are actively evaluating a number of alternatives and will be adding a mature successor language to the project in the near future. This process is already quite far along, and prototypes exist in multiple languages.
“A randomly generated UID is fused into the SoC at manufacturing time. Starting with A9 SoCs, the UID is generated by the Secure Enclave TRNG during manufacturing and written to the fuses using a software process that runs entirely in the Secure Enclave. This process protects the UID from being visible outside the device during manufacturing and therefore isn’t available for access or storage by Apple or any of its suppliers.“
Sure, and even Apple can't imitate a different server that they made.
They're making new servers though. Take the keys that are used to vouch for the UIDs in actual secure enclaves, and use them to vouch for the UID in your evil simulated "secure" enclave. Your simulated secure enclave doesn't present as any particular real secure enclave, it just presents as a newly made secure enclave that Apple has vouched for as being a secure enclave.
Windows and macOS both use a form of two level name-spacing, which does the same sort of direct binding to a target library for each symbol. Retrofitting that into a binary format is pretty simple, but retrofitting it into an ecosystem that depends on the existing flat namespace look up semantics is not. I think it is pretty clever that the author noticed the static nature of the nix store allows them to statically evaluate the symbol resolutions and get the launch time benefits of two level namespaces.
I do wonder if it might make more sense to rewrite the binaries to use Direct Binding[1]. That is an existing encoding of library targets for symbols in ELF that has been used by Solaris for a number of years.
You can get the same semantics as for direct binding using symbol versioning, but direct binding is faster.
Also, symbol versioning is only really better than direct binding if you end up having multiple versions of the same symbol provided by the same object, but that's relatively hard to use, so it's really only ever used for things like the C library itself. Mind you, that is a very valuable feature when you need it. In Solaris itself when we needed to deal with the various different behaviors of snprintf() there just wasn't a good way to do it, and only symbol versioning with support for multiple versions of a symbol would have helped.
Not really... symbol versioning is a form of namespacing, but it is somewhat orthogonal to this.
Symbol versioning allows you to have multiple symbols with the same name namespaced by version, but you still have no control over what library in the search path they will be found in. So it does not improve the speed of the runtime searching (since they could be in any library an the search path and you still need to search for them in order), and it does not provide the the same binary compatibility support and dylib hijacking protection (since again, any dylibs earlier in the search path could declare a symbol with he same name.
One could use symbol versioning to construct a system where you had the same binary protection guarantees, but it would involve every library declaring a unique version string, and guaranteeing there are no collisions. The obvious way to do that would be to use the file path as the symbol version, at which point you have reinvented mach-o install names, except:
1. You still do not get the runtime speed ups unless you change the dynamic linker behavior to use the version string as the search path, which would require ecosystem wide changes.
2. You can't actually use symbol versioning to do versioned symbols any more, since you overloaded the use of version strings (mach-o binaries end up accomplishing symbol versioning through header tricks with `asmname`, so it is not completely intractable to do even without explicit support).
> You still do not get the runtime speed ups unless you change the dynamic linker behavior to use the version string as the search path, which would require ecosystem wide changes.
Each ELF library declares the symbol versions it provides. The dynamic linker could track which library declares which versions, and cross reference that when it looks symbols up. I though it did, but from empirical testing, it doesn't. But if it did, it would get similar speed improvements, assuming all libraries provide at least one version each (and of course, assuming no overlaps).
> Symbol versioning allows you to have multiple symbols with the same name namespaced by version, but you still have no control over what library in the search path they will be found in.
Yes, but since the convention is to use the SONAME and SOVERSION in the symbol version therefore in practice the symbol version does -when adhering to this convention- help in binding symbols to objects.
Still, because this is an indirect scheme it does not help speed up relocation processing.
As you say, direct binding is better for safety and speed.
Absolutely, I just think "when adhering to this convention" is a high risk. Admittedly I mostly work on macOS so I don't have a nearly as deep of an experience with ELF, but in my experience even when a system looks to be well maintained that you often find surprising numbers of projects being "clever" and breaking conventions as soon as you try to do something that depends on everyone actually globally following the convention.
Not only is there less crawling around looking for symbols, you're no longer in trouble when two libraries export the same symbol.
Especially given libraries are found by name, and symbols by name, where "type information" or "is that actually the library I wanted" are afterthoughts.
> you're no longer in trouble when two libraries export the same symbol.
Whether you use direct binding or symbol versioning, either way you don't have a problem with multiple libraries exporting the same symbol.
By the way, this is the fundamental problem with static linking for C: it's still stuck with 1970s semantics and you can't get the same symbol conflict resolution semantics as with ELF because the static linker-editors do not record dependencies in static link archives.
The key insight is that when you link-edit your libraries and programs you should provide only the direct dependencies, and the linker-editor should then record in its output which one of those provided which symbol. Compare to static linking where only the final edit gets the dependency information and that dependency tree has to get flattened (because it has to fit on a command-line, which is linear in nature).
Conceptually not much has changed since the book was written, but in practice there has been a lot of advancement. For example, ASLR and the increase in the number of libraries has greatly increased the pressure to make relocations efficient, modern architecture including PC relative load/store and branch instructions has greatly reduced the cost of PIC code, and code signing has made mutating program text to apply relocations problematic.
On Darwin we redesigned our fixup format so it can be efficiently applied during page in. That did in include adding a few new load commands to describe the new data, as well as a change in how we store pointers in the data segments, but those are not really properties of mach-o so much as the runtime.
I generally find that a lot of things attribute to the file format are actually more about how it is used rather than what it supports. Back when Mac OS X first shipped people argued about PEF vs mach-o, but what all the arguments all boiled down to was the calling convention (TOC vs GOT), either of which could have been support by either format.
Another example is symbol lookup. Darwin uses two level namespaces (where binds include both the symbol name and the library it is expected to be resolved from), and Linux uses flat namespaces (where binds only include the symbol name which is then searched for in all the available libraries). People often act as though that is a file format difference, but mach-o supports both (though the higher level parts of the Darwin OS stack depend on two level namespaces, the low level parts can work with a flat namespace, which is important since a lot of CLI tools that are primarily developed on Linux depend on that). Likewise, ELF also supports both, Solaris uses two level namespaces (they call it ELF Direct Binding).
There is a bug here... Clearly the author intended to cache the value of nextmalloc to avoid calling dlsym() on every malloc. The correct code should be:
It really isn't unavoidable on macOS. @rpath is designed specifically to handle this sort of thing, and is how all the command line tools distributed with Xcode link to libraries embedded in Xcode and continue working even when you drag Xcode to a new location.
Admittedly supporting that would require updating how all the tools are built and not just defaulting to Whatever Linux Does™, which is probably too much effort to justify in this case, but it is hardly an unsolvable (or even an unsolved) problem.
I have been out of this area for almost a decade now, but I have very fond memories of the nRF5 SDK. When I was evaluating the (then new) Nordic BLE SoC's for future products it was so much nicer than the TI CC2540 we had used in our first BLE device.
It has been years since I was familiar enough with the insides of SSDs to tell you exactly what they are doing now, but even ~10-15 years ago it was normal for each raw 2k block to actually be ~2176+ bytes and use at least 128 bytes for LDPC codes. Since then the block sizes have gone up (which reduces the number of bytes you need to achieve equivalent protection) and the lithography has shrunk (which increases the raw error rate).
Where exactly the error correction is implemented (individual dies, SSD controller, etc) and how it is reported can vary depending on the application, but I can say with assurance that there is no chance your OS sees uncorrected bits from your flash dies.