Vitess for PostgreSQL will probably just be... Vitess.
The concepts behind Vitess are sufficiently general to simply apply them to PostgreSQL now that PostgreSQL has logical replication. In some ways it can be even better due to things like replication slots being a good fit for these sorts of architectures.
There are no plans right now.If the Postgres community (or a Postgres user) would like to take this project up, the best way to proceed would be to do it as a fork of Vitess. Once the implementation has been proven in production, a “merge” project can be planned to bring the fork back into upstream.
You mentioned your backend being a Rails app + serverless functions, what's the benefit of doing both there? What does your video processing infra look like, is that in one of those systems?
I could go pretty deep here, so let me know if I should elaborate on anything.
The backend is a Ruby on Rails application that serves the frontend app's API. This interfaces with the user tables, database, and handles all the "state" of the app.
The serverless stuff has changed over the months, but primarily it handles the stuff I don't want Rails to handle: file uploads, video processing and transcription.
First, huge props to the Mux (https://mux.com) team and product. I can not express how easy it has been to build video (and audio) products. File uploads are handled to AWS/GCP (depending on a few things) and then trigger a serverless callback to Mux.com. Mux was the fastest way we found to turn an arbitrary video file (mp4/mov/etc) into HLS format for quick streaming.
Then once the video is uploaded, we have another serverless callback that sends the video for transcription using Assembly AI (https://assemblyai.com). There are a ton of transcription based services and they vary dramatically in quality, based on the media content. I believe Google/Amazon services were largely built around the need to process phone calls, so unless you may for their "enhanced" models, the quality is surprisingly bad (and surprisingly slow).
I *highly highly* recommend Mux and Assembly AI if you are doing anything video/transcription based work.
To get an immediate update to the end user, we actually process two transcript requests - one that is just the first 60 seconds, and then the remainder of the video. This lets us render a preview transcript in the first 15-20 seconds.
We also have a serverless pipeline for generating the videos, but I won't go into that unless you're interested. In short, a serverless function kicks off a Docker instance running on ECS.
The requests to the serverless apps (mostly Node) have a callback to the Rails app, which then updates the end user state using websockets (which are very easy to use in Rail's ActionCable).
Interested to hear more about your pipeline and infrastructure for processing and delivering video. I'm working with processing short videos at the moment for my current startup, though I didn't use Mux (I figured it was a core competency we needed to develop). It's just a queue using FFMPEG to convert from MP4 to HLS.
I have horror stories about FFMPEG that I wont go into here.
In short, I'm just one person building this - so I'm sticking to what I know best. I want video to "just work" without having to worry about some video format/extension/containers that I have no idea about.
There are a number of video processing services, but Mux really is the best. The API is simple. They have a ton of really nice helper functions, that I use a lot (like timestamped thumbnails, preview gifs, and VTT storyboard generation), which I could easily spend a few days on making, and then countless hours maintaining.
I dont doubt that building video infra is a good idea, but just as I'm not about to train my own speech-to-text model, I'm not going to build out video infra.
At least for me, I'm more worried about the end user experience, and the more I can focus on that, the better the overall product will be.
I'm in the same boat - I'm the one building it, and my focus is on the user experience, but the business model won't tolerate the amount of video on someone else's service. :(
I haven't had FFMPEG nightmares yet, but I've done relatively little with it so far.
Any video apps I should look out for? I'm also pursuing a content creation angle that I've yet to spec out, so I'm always curious as to how others have approached the problem.
Hey! Jon from Mux here. Curious about this comment:
> the business model won't tolerate the amount of video on someone else's service
Does that mean you aren't using S3/EC2 or the like, or is there something about how we've built our cloud platform that doesn't work for your business model? We've designed Mux to be a low-level primitive for video, like Twilio is for SMS, so I'd be interested if we're doing something that makes this harder for you.
Hey Jon! I've looked at Mux (mainly the careers page), and it's a great platform. It would be a great fit technologically, but I'm not sure that my business model (which is tentative admittedly) would cover infra costs for processing, hosting and consuming the amount of video I'm eventually expecting, as I'm running on a shoestring budget at the moment.
Plus, it's a good chance to for me to learn the ins and outs of video. It's not reflective of the quality of your platform, just a choice I've made early in the piece for curiosity's sake.
rhinoceraptor gave such a clear explanation of the process, that I'd honestly pay for an entire set of bio and chem mechanic breakdowns prepared by them.
The comments here are really interesting. What is the equity level where one goes from thinking like an owner to being an employee? At what equity level does one feel the things in this blogpost maybe apply? 5%? 1%? 0.5%? If startup are power-law gambles, then 0.025% can still be a great outcome.
Should a founder mentality be expected amongst early employees at a certain equity level? Why wouldn't someone want to hire someone who had that mindset, and then appropriately compensate someone for that?
Couldn’t his company be worth more than a trillion because it’s...adding far greater value to the ecosystem it exists in? And his share is in the ~100B as he captures the upside through being a founder and taking the early risk?
But he didn't? Yer man Travis was the founder, this dude Dara is just a career executive brought in to turn around the ship. He basically hasn't taken any risks, with respect to Uber at least.
I do feel like HN is having a rehash of a conversation we've had many times this past month. I'll post my favorite comment [1] on the matter of remote salary adjustment, which captures a key market effect we all seem to forget (Bay Area specific):
"Cost of Living" adjustments are a red herring, what they really are is really "competition density".
There are plenty of tech companies paying great salaries in the bay because they have to, otherwise they would just go work for someone else. On the other hand, if you lived in Oklahoma you aren't going to say no to $LOCAL_OFFER+10k just because bay area salaries are $LOCAL_OFFER+90k.
As long as this disparity exists, I forsee bay area salaries and CoL still being high. Until companies move headquarters out of the bay, the trend will continue.
Similarly, in this thread, Consultant32452 states [2] that the real argument is between those who can demand a high salary regardless of geography, and those who can only demand a high salary _because_ of geography.
The mistake is many people in the latter group think they are in the former.
> the real argument is between those who can demand a high salary regardless of geography, and those who can only demand a high salary _because_ of geography
It sounds dramatic expressed that way, but doing so is committing the binary fallacy.
In reality, there is a spectrum of the leverage between employees and employers that depends on many different factors including the employee's knowledge, skills, experience, reputation, trust, institutional knowledge, and even the degree of the employee's willingness to show up regularly in person.
That and the extent to which the employer values any of those.
The covid19 wfh emergency is going to modify that spectrum, depending on the job, cost of living, and the area. But it is not going to get rid of the spectrum altogether, just as it isn't going to eliminate the intrinsic value of cities and metros as factories for creativity, just diminish them a bit for a subset of the employees.
> I do feel like HN is having a rehash of a conversation we've had many times this past month.
It is not only on remote work, same with Google privacy issues, Zoom issues, micro service/monolithic, Electron... Most of the big conversations on HN have been going on for a long time and still do
What's a bit interesting is that as a consultant with an OK rate I always attributed it to being onsite, directly talking with top management (what I do is routine so pay is basically for being able to interact successfully up and down org chart). This meant I felt very GEOGRAPHICALLY tied down.
Now I've not shown up onsite to ANYONE in 3+ months (we went remote early before shutdown). And it's not clear to me that value has diminished, though I need to up my zoom game a bit.
So some people may be discovering they thought they were geo tied / competitive but may be more broadly valuable or able to serve into that market remotely more easily than they thought?
You may be coasting in your past contracts and connections, although consultants are often better equipped to handle this chaos than the orgs themselves (e.g. a huge semiconductor company I consult for had all its labs shut down while my permanent home lab with their setups kept going).
Ouch. This comment is cutting. The truth can be that way sometimes, no?
Probably the only thing I take issue with is your second point. No matter how much companies talk about how much of a "talent shortage" there is, the mythical "overpaid engineer" in an area with high competition density is a lot rarer than they'd appear. To even get and keep a median salary job that lets you move there (and stay there), you're competing with (and often have to lose to) a lot of folks.
Social participation in that world and the learning of what gets rewarded and punished during your formative years helps condition you towards going about solving problems a given way, and alters you as a person. I am sure that I would be different if I spent the early formative years of my career in SV instead of NYC. I'd be solving different problems with different people and moving up career wise through different maneuvers.
So here's my question: will the pivot towards remote work diffuse or further reinforce hub centrality for current tech hubs? To be honest, I'm kind of mentally split on this one. Here's what I could imagine along both sides:
For concentration:
- Lacking differentiation, areas outside of tech hubs will become increasingly commodified (one giant suburb), stratified and divided into castes
- Executives will tend to cluster around hub centers, middle managers around suburbs, ICs around exurbs and further; moving up will require "moving inwards"
- Customer and capital consolidation into cities will continue
- City lifestyle will continue being a status symbol synonymous with affluence, success, clout and social standing
For diffusion:
- Propped up by inflated prices out of line with their underlying assets, hubs will crash economically when people have the option to work there without living there
- Companies seeking to evade the "geography tax" will learn to work remotely to gain an edge in OpEx over competitors that don't and through lower margins gradually beat competitors that don't adapt into submission
- Natural disasters, climate change and pandemics have/will continue to spark continued interest in the "retreat from society"
- The pioneer spirit will come into vogue again, as enterprising contrarians try to rough it on the frontier once more, taking risks on developing fringe territory to capitalize off the nonlinear value they add (this, along with DoD funds/braintrust, was how SV began)
I don't know. It's hard to reason about what the outcome will be at this point. Greater flexibility seems like it would lead to heterogeneity. But as always, there are nonlinear second order effects that we can't think of now that will doubtless seem obvious in retrospect. What do other folks think?
> areas outside of tech hubs will become increasingly commodified (one giant suburb), stratified and divided into castes - Executives will tend to cluster around hub centers, middle managers around suburbs, ICs around exurbs and further; moving up will require "moving inwards"
Isn't this already the general pattern? I see that continuing, but with a very slight diminishing of the mega-hubs and a slight boost to secondary hubs (Austin, Boulder, etc).
> For diffusion: - Propped up by inflated prices out of line with their underlying assets, hubs will crash economically when people have the option to work there without living there
I think any prediction of the loss of the significance of cities is pretty far fetched. Cities have been robust to millenia of changes, including multiple past pandemics that have always disproportionately affected them vs more remote areas.
> [Companies] will learn to work remotely to gain an edge in OpEx over competitors that don't and through lower margins gradually beat competitors that don't adapt into submission
This will only happen in sectors with competition over thin margins and high OpEx as a percentage of expenditures, like running a manufacturing plant. That doesn't really reflect the kind of tech work done in tech hubs, which is CapEx and creativity heavy. Even pure software has to stay abreast of consumer and cultural trends, which are largely form in cities. A lot of software are dynamic cultural products, similar in many ways to movies, music, and TV shows. It could. however, affect things like consumer-facing tech support, but that has largely already taken place - a lot of tech support for major companies happens out of lower cost cities in the US, not the expensive tech hubs.
Excellent points. I'm again reminded of this 2S[1] investment article about the value of gross margins. I find particularly insightful your comment about tech work done in tech hubs which is CapEx, creativity and presumably margin heavy as well (to support the first two). One could say, well isn't it possible for this work to dry up? And yet, on the other hand, the whole point of investing is to perennially keep a portfolio fresh and full of the most cutting edge, margin heavy firms (when adjusted for risk adjusted returns compared to the nominal risk free return rate). Well, theoretically anyways.
> And yet, on the other hand, the whole point of investing is to perennially keep a portfolio fresh and full of the most cutting edge, margin heavy firms (when adjusted for risk adjusted returns compared to the nominal risk free return rate). Well, theoretically anyways.
I understand what you are saying here, but this perspective sees investment as the driver of the creativity of cities, while another perspective that I put more weight on is that the creativity of cities and the culture they create is a byproduct of human social/group psychology, itself a result of the fundamental reality that there is more security in numbers.
Investment, whether private or public, is then just a way of incentivizing the creativity machine of natural human agglomerations to produce ever more novel stuff.
Or put more succinctly: culture leads, investment follows.
It's an argument with a lot of merit. I think there's a chicken and the egg situation, but I think you're right in that the prime mover is probably the culture rather than investment. That goes along with another saying I think I've seen a lot of mileage, which is that politics is downstream from culture. Seems to follow that investment would follow too from there as well.
It's really hard to setup a cohesive engineering culture in the same office, harder to do with a remote office - and nigh impossible with an outsourcing firm with 12 hours time difference.
Not every business needs a cohesive engineering culture - but if your primary product is tech, then it stands to reason that the cost of your developers is a second priority to effective execution at scale.
It’s often just easier to put people in some physical space and hope the social interactions make for some progress and unison, than to actually manage things with follow ups and communicated objectives etc etc.
I hate my loud, noisy open office and only had two people that I would consider friends. Not that I don’t like my coworkers, they are great to work with. I just haven’t hit it off with them on a personal level. But I am really missing the interaction with them. Even though my company is trying with virtual happy hours.
This to me is the real argument: those who can demand a high salary regardless of geography, and those who can only demand a high salary _because_ of geography.
The mistake is many people in the latter group think they are in the former.
Like many, I think I'm in the former group and will allow myself a little Schadenfreude if the latter group gets an "adjustment." Unless it turns out I'm wrong. :-O
