There's a bunch in here that I think HN readers will find fascinating, but it may help to have a little context going in. At Oxide, we have developed our own switch[0][1] using switching silicon that implements P4[2]. P4 is really interesting -- you can think of it as a much cleaner eBPF -- and our programmable switch is effectively a P4 engine. This is really powerful; if you want to jump to why, listen to Ry expand on it about an hour into the conversation.[3]
This conversation goes into the details about making all of this work. Along the way, we hit on pushing a cycle-accurate simulator to its limits; developing a new P4 compiler that has a Rust target; using that compiler to then develop an accurate hardware-virtualized model of the ASIC; developing a new routing protocol (delay driven multipathing); and how all of this adds to up solving some gnarly problems like microcongestion and flexible topology construction.
While I’m a big fan of your work and have been deeply both entertained and educated by your presentation skills, an unfortunate and concerning observation is that the products that you’ve championed generally failed to see adoption in the mass market, and it may be due to the overtly technical focus, which is not bad in of itself but lengthens the time to market. Perhaps a more pragmatic approach may be to release an iteration of the product before improving on it.
I say this not to detract from your work but because I’d genuinely like to see companies opting into self-hosted hardware as Oxide envisions. The centralization of all internet services into three big cloud vendors does not bode well for society.
While I appreciate the kind words, I don't really agree with your analysis of my career, for whatever it's worth. ;)
In terms of Oxide: I can assure you that we are shipping the minimum viable product -- the challenge is that the minimum viable product is very (very!) large. (And indeed, the projections we made in our initial investor deck were far more accurate than they had any business being!)
Glad to hear that you're cheering us on, and thanks again for the kind words!
Fishworks made a lot of money for Oracle, if that's not success I don't know what is.
PS: I would really love an episode about Sun deciding to go to bed with AT&T and merge and BSD and System V. That seem to cast a long shadow and must have left some scars.
it's ok for Bryan to work on things that he believes have value.
given his skills and experience compared to anyone I've worked with (at least at the time that I worked with them) I would trust what Bryan puts his weight behind over the internet's techno-fetish of the month, any day.
I don’t disagree with any part of your comment — however, it is sad that technologically superior solutions that Bryan has worked on “lost”. As an example, Manta/Triton had an objectively better model, and yet it is sad that we’re all stuck with S3 like storages today.
sometimes "good enough" wins, even when something better is available.
you must always work on things that have value to you, even if those things are unknown or unpopular. not because they are unknown or unpopular, but because they are the right solution for you.
the amount of homogeny that I see today staggers me. people don't seem to be solving their actual problems, but instead are twisting their problems so that the commonly deployed solutions fit.
Oxide seem to be developing solutions for people that have problems which the common solutions can not address, no matter how much you twist them.
If you allow me to query on a different oxide topic: I noticed that you provide a certificate to each HSM in every system that is signed by an Oxide Computing root cert. This is pretty neat because, combined with other security features, it enables clients to validate that the running firmware came from you. Have you considered publishing a certificate transparency log of all certificates generated for hardware root of trust HSMs? This would enable customers to ensure they are not using firmware signed using a compromised root certificate. https://eprint.iacr.org/2022/981
Yes, and we really need to do an entire episode (or several!) on how we're tackling that problem. Right now that team is very, very heads-down (as you can imagine, having this apparatus in place is a blocker to start manufacturing), but count on an episode on this in the not-so-distant future!
As someone deeply involved with the P4 community and contributor to the P4 compiler it is great to see someone being bullish on the language. Have been following your work with interest. :)
I see that you are developing your own compiler. If you have complaints or suggestions please feel free to reach out to p4-design@p4.org. The language design group is meeting monthly and is always interested in hearing grievances of consumers of the language.
As a side-note, we are currently developing tools to ensure that P4 programs are compiled and executed correctly. Tools include packet-test generators, fuzzers, and general-purpose verification frameworks. Because the language is so restricted you can do really powerful validation. A concrete example is Google using P4 as a specification language for their network devices[0]. If that is something that interests you, shoot me a message (fruffy@nyu.edu).
Hey! Thanks for reaching out. Primary author of the Oxide P4 compiler here. I joined the P4 Language Design WG a while back and have attended a few meetings as time and schedules allow. I'll join in on next week's meeting in case folks are interested in follow-up questions/discussions.
I love your podcasts, your talks, and most of all your technical work, but I would really appreciate it if you found the time to get a better microphone for these recordings.
I get that this is something you just hit record on while chatting on a work laptop or something, but the 's' sounds in your audio track especially are so high pitched it's unbearable to listen to for me.
So, I am on what is supposed to be a decent mic on this episode -- so this is hopefully a solvable problem. If it's not too much to ask, could you point me to a minute that's unbearable, and I'll do what I can to solve it?
Might be you're a little too close to it, or something else like that(too much treble in the settings or something?).
I will say that this recording does sound better than a much earlier one I tried a while back. So if you did change your mic at some point, it probably helped quite a bit.
I just listened to the first couple minutes, but generally quite a few of your 's' sounds trigger the nails on chalkboard type reaction(not that bad though, but same type of reaction). Hope this makes sense to you. Im a couple decades younger than you, so could be I'm hearing some high frequencies you couldn't hear.
Glad to hear that this one was better. If you can bear to indicate the exact offset of a grating "s" it would help, but otherwise we'll listen to the first few minutes and do our best; thanks for the feedback!
0:14 of https://www.youtube.com/watch?v=NVZ80_tbkbc is a good example of previous recordings. It is helpful, if you think you have a good mic, to talk past it rather than into it.
What switching silicon did you end up using? Given the speeds and feeds of the switching silicon you used, was it cost-effective compared to using Broadcom, and just overprovisioning the hell out of it?
See my comment elsewhere[0] for the status of Tofino, but in terms of what's next: this silicon is going to hold us for quite a while -- but whatever's next will continue to be P4 based, in which we remain ardent believers.
Cisco's SiliconOne is programmable using P4 as well. I'd be curious to know if it'd work for your use cases. It certainly isn't going anywhere since we're using it in darn near all our products now.
I assume that since you're using P4, you can likely pivot to Mount Evans chips (or the successors in that line) and a dumber switch if you need to, and the fact that Google is buying them too gives the product some legs from Intel's perspective. I really like Intel's networking products, but they have an unfortunate habit of getting canceled.
I will disclaim that I have no idea how Oxide is using P4 for packet processing, but what you are currently doing with P4 on your switch can likely be moved to some rules on a dumber switch/router (eg what hyperscalers use) plus a special packet encapsulation on the NIC using similar P4 rules. That lets you push the intelligence out of the switch and toward the edge of the network. At least, this is the approach that the major hyperscalers are taking, and there doesn't seem to be a good reason to not do this (other than potentially the cost, since you don't have the scale to get a good discount on the NICs).
Yeah, definitely not a switch ASIC. But would love to see something like the E2000 with 1) an open underlying ISA to enable truly open P4 compilers and 2) no compute complex so we can focus the power budget on the programmable data plane.
Oh man this was great. And also hard to hear. This is a wonderfully ambitious project & capable of yielding really interesting results. But alas, Intel is no longer investing in the underlying Tofino network switch chip (which they inherited/extended from Barefoot), which is the primary/only P4 switch-chip offering.
I'd say... there's a non-zero chance this work alone could revive it, show the technologies worth.
There's a ton of value to having an understandable clear networking stack that you can see & work on, and the podcast is totally correct that vendors almost never offer clarity or possibility. Netops crosses fingers, flips a switch, & sees what happens. By contrast, this is a refreshing take on being responsible for your shit, in a way little else does.
But it's still unclear that embedding much more complex computing in your network switch has real value. Having a far more configurable P4 capability under the belt isn't a little step towards ownership, it's a huge step towards software-defining the network pipelines.
This seems unrelated, but I somewhat look at systems fabrics - smaller scale & faster connections - & see a lot of complexity about figuring what to send where. The idea of embedding more intelligence in the connective fabric has huge alure. But it's unclear who or when someone will truly leverage more flexible switching/interconnect to great effect.
Agreed that it's ambitious, but we feel that the wind is at our back in many dimensions, with a tremendous amount of potential that we can now deliver.
As for Tofino 2, I hit on this earlier when Intel first announced it, but fortunately Tofino 2 is still being supported: the team has been very candid and transparent with us (which we very much appreciate!) and Intel has been both formal and explicit about their ongoing support for Tofino 2. We remain -- for all of the reasons we went into in this discussion -- very bullish on P4, and we believe that we will be able to show its unique value in the Oxide rack. A long way of saying: we are absolutely with you on the non-zero chance of showing the technology's worth!
I work on network services, but generally my server's requests start on a user's computer, get to my service, get proxied to some upstream destination elsewhere on the internet, and then the responses flow back. Point is, I work with networks, but generally the latency and throughput is bottlenecked somewhere outside my code. Probably by the user's internet connection, or the server's.
So it was _really_ interesting to listen to this podcast live and think about truly high performance networking. When your requests and responses are all within one rack on your data center, the bandwidth, latency and throughput bottlenecks are MUCH more likely to be your code and hardware. So Oxide has to think about all these high-performance use-cases that I've never once had to stop and think about.
This episode felt like peeking behind the curtain into forbidden knowledge. Great episode.
Glad you enjoyed it! And yes, agreed there is so much here that isn't spoken of very broadly: part of what we have loved about social audio is allowing engineering teams to speak candidly about the work that they're doing -- and we're hoping that others follow suit, as I (selfishly) would love to listen to similar conversation from other engineering teams...
Bryan Cantrill is really skilled at drawing out the technical details into a cohesive story from various speakers. Oxide's On the Metal was great for individual stories, but Bryan's orchestration of stories from multiple actors in a venture like this and the podcast on electromagnetic and safety compliance (https://www.youtube.com/watch?v=NVZ80_tbkbc) make it feel like there's a palpable momentum building up before the open sale of the Oxide rack. It's very exciting to listen in on.
Thank you for the kind words -- and honestly, it's every bit as exciting for me: there is just so much technically to what we're doing, and I love being able to share that, in the team's own voice!
I like them, and I understand the focus on servers, but I'd really, REALLY, love a company to disrupt the desktop/workstation space.
Yesterday I was half-joking about how an almost single-chip workstation could be built around the AMD MI-300 (24 x86 cores, GPU, and 128GB of HBM) and I think we could have some space for products that deliberately misuse these components.
Heck... Every hardware company should have a "tech misuse" department... IBM should do a Z-based desktop and a 2-core POWER chip, for instance.
What Oxide is doing is nowhere close to the workstation or even midrange-compute market where most "servers" are nowadays. Based on the announced specs of their rack-scale systems, they probably have to be priced in the million-dollar range based on the hardware alone.
I guess what they could do is take a single gimlet and break that out into its own server. You would have to replace bus bar with a normal power supply and likely do some other stuff as well.
It's sad how much of this sounded like pure sci-fi to me. And that's not to be critical - I'm in a world where observability means snmp and frequently modbus. So much of this hit home that I'm in the wrong century.
I tried listening to the podcast that I discovered thanks to this post. I tried 2 episodes and both had audio issues where one person was just too quiet to hear what they were saying.
This conversation goes into the details about making all of this work. Along the way, we hit on pushing a cycle-accurate simulator to its limits; developing a new P4 compiler that has a Rust target; using that compiler to then develop an accurate hardware-virtualized model of the ASIC; developing a new routing protocol (delay driven multipathing); and how all of this adds to up solving some gnarly problems like microcongestion and flexible topology construction.
[0] https://share.transistor.fm/s/c80a79d8
[1] https://share.transistor.fm/s/65a10522
[2] https://p4.org/
[3] https://www.youtube.com/watch?v=AkWh2Sms3aw#t=1h4m46s