Hacker Newsnew | past | comments | ask | show | jobs | submitlogin
How we scaled Google Meet during Covid-19 (cloud.google.com)
125 points by caution on Aug 6, 2020 | hide | past | favorite | 56 comments


It's crazy how little Google's marketcap benefits from things like this. Zoom's here sitting on 75B (~1/15th Googs) as we speak and Google Meet probably isn't even legitimately used in pricing Google. Wonder what Google's market cap would be if each individual service got the VC future growth pricing.


Not only that, but modern Google Meet is significantly more useful to my day-to-day remote conferencing needs than Zoom. Modern Google Meet's advantages:

* It's nearly instantaneous to use. You can just bookmark any prior meeting ID you created and reuse it indefinitely. You just click the meeting on your bookmark toolbar and join the meeting. Creating a new meeting is also nearly effortless.

* Works right in your web browser. It doesn't default to pushing you to use a native executable.

* IMO, the video and audio both are slightly better than Zoom.

Google Meet or Hangouts from years ago was quite bad, but it has improved nicely.


Also the transcription. Fastest, most accurate transcription I've ever seen.


Yes - having it completely integrated into your Calendar (if you use Google Calendar) is the right way round, if I have a stack of remote meetings I should just need to know the when and the who - the how should take care of itself, as it does with Meet.


And multiple people can share their screen at the same time. And nothing is interfering with note taking by making the window full screen whenever a presentation starts.

I am using zoom rarely and every time I am baffled by it's horrible UX. Why do people use it?


> IMO, the video and audio both are slightly better than Zoom.

Firm disagree.

Three days ago, I presented a video in a google meet stream for 50+ people. For three minutes the stream dropped frames , displayed visual artifacts and lagged behind my narration.

I was on gigabit internet and so were my colleagues.

I never had this problem with zoom, and I would never use google meets for a streaming setup again.


Did you use the newish "Present a Chrome tab" feature to do this?

https://gsuiteupdates.googleblog.com/2020/04/high-quality-vi...

Normal Google Meet Present is optimized for slides/docs which is a very different problem than streaming video.


Well, it’s still a joke that Google Meet will stream desktop / windows at 4 FPS. I would be curious to know whether they really think the feature is good enough.


This is mostly a limitation of Chrome and other web browsers.

When streaming the desktop, the browser doesn't get notified when some region on the screen changes. That means it has to retrieve a bitmap of the screen and 'diff' it with every frame, which is computationally heavy.


FWIW, AirPlay can send a H264 video of my desktop at 60FPS with ~0% CPU impact (hardware accelerated), good enough to play many games on TV.


I believe Airplay has special OS integration - a third party app wouldn't have access to the screen buffer with zero-copy.. A third party app would at a minimum have to copy the data to system memory and back to the GPU, using a lot of CPU time for that alone, before any encoding.


Can it blur/cover backgrounds?


Yes it now has a built in blur background feature. (Pretty aggressive level of blurring in my experience.)


The stock market is hilariously bad at valuation of big businesses.

The most egregious example was when value of all of EMC’s business was worth less than the value of their VMWare ownership stake.

The Zoom story is really cool, and the business is simple. Are they actually worth $75B in a crowded, competitive market? We’ll see!


With Zoom, I think, the money is riding on the brand. My parents, who aren't tech savvy at all, think they are on Zoom even when they are actually using Google Meets. Of course, the Zoom app is installed on their phones and they take "meetings" on it too.


Google organizationally though is probably incapable of capitalizing on Google Meet. They have a bewildering history of creating and discontinuing and morphing products in this space, and their historic lack of focus and vision means you'd have to really be a Google aficionado to keep up with what their current offering even is, much less whether it's any good.

Whereas Zoom? They do one thing, and they do it well (enough). You don't download Zoom Hangouts one year and then switch to Zoom Duo then Zoom Allo and then migrate Zoom Meet. You just use Zoom and it works well enough.

I feel like Zoom has basically eaten Google's lunch here, not by actually being a better offering, but by simply being a straightforward proposition. Perhaps Google can learn some lessons for the future...


To Googlers working on this: thank you! My work life shifted from 5% Meet to 80% Meet and it only got better during the crisis. Never noticed deteriorated performance.


Literally adding Google Meet URLs to Zoom meeting invites in Google Calendar, so half of a meeting's members went to the wrong room is kinda a bad way to scale your product: https://mspoweruser.com/googles-latest-audacious-growth-hack...


There's a setting you can turn on called "Automatically add Google Meet video conferences to events I create"[1].

I tested on a new account and the default was on, which you might argue is bad, but when I tried to create my first calendar event there was a popup that told me that it was on and gave me the option to turn it off.

Plus when you create a event the option to include a Google Meet is right there and it's one click to remove.

[1] https://support.google.com/a/answer/9898950?hl=en


Question is, what's it doing when you use client software that doesn't support the spammy popups? It appears that for a third party client user, it auto-injected URLs the sender didn't expect or even see. So someone creating a Zoom invite had no idea their coworkers might see a Meet link.

Also, Google could/should recognize that a meeting invite with a competitor's room URL in it is somewhere it shouldn't inject its own. Especially since it opens up yet another avenue for an open and shut anticompetitive case against them.


Blacklisting some competitor URL's is discriminatory against smaller competitors or people's on-site solutions at private URLs


The parent didn't say to blacklist the competitor but rather to not inject googles own when a competitors URL was already in the invite.


PEBKAC. Google Calendar integrates with multiple meeting systems. The organizer should have used Zoom's integration to put the Zoom link in the meeting field instead of putting the Zoom link in the location field and adding a Meet link in the meeting field. https://support.zoom.us/hc/en-us/articles/360020187492-Googl...


> Literally adding Google Meet URLs to Zoom meeting invites in Google Calendar, so half of a meeting's members went to the wrong room is kinda a bad way to scale your product

You could argue that, yes. And I won't blame you. But if you ignore Google and Meet for a moment and consider Microsoft (or in an alternate universe - Apple); product wise, this is a good decision. Reduces friction from creating meetings and provides a better experience.

You can disable automated Meet meeting generation though.


The dates on the shown graph significantly predate this new feature (or GMail tab).


I'd love to hear the war stories about the network capacity plan violations that were involved here. There's no way they had 30x headroom in their 90-day capacity forecast. The organizational ability to reach in and shuffle up the network capacity allocations may be even more interesting than the technical items discussed here.


Nah, this is considerably minor scaling problem for Google.

Youtube bandwidth is there. Google meet is just a rounding error. Of course (most likely) there will be dedicated planning going forward, but these type of traffic ramping up is daily operation at Google.

At Google, application developers rarely pay much attention to scalability, that's how the TI org tries to achieve, and they are probably the best AFAIK.


It's clear that the Google where you worked existed in a parallel universe from mine. On my home planet, google netcap required a forecast per-product, per region pair, years in advance, and the product group got charged back for the capacity requested, not for the usage, so nobody was requesting extra.


Hah - except there were overestimates at every level, like where teams were confident of there 50% growth per quarter for the next 2 years, and that never materialised. Know the right people, and that unused capacity can be reallocated to you in a few hours.

Capacity planning done like that is mostly a waste of time...


Yes, planning is always there.

I was saying that this sudden increase in the traffic should not cause major issues. Most likely several SREs have emergency meetings, and get quotas across places. Then it's done.

If you read my post above, you'll notice that I did not mention SREs. I was saying app developers do not care planning.

Of course, some app developers thought they were doing planning, as in some activity to estimate demands, etc.

That, in Google infrastructure engineering, is not considered capacity planning. The typical capacity planning is done by SRE with more procedures that certainly border the most curious developers.


Every web-based video chat I’ve used (including Google Meet and Slack) makes the fans on my decently-specced 2018 MacBook Pro spin up like it’s about to take off. I’ve tried Firefox, Chrome and Safari. Zoom (via the native Mac app) doesn’t do this. Does anyone know why browser-based video is so resource intensive?


Use Safari. Turn on the developer settings, and find the setting to disable VP9 in WebRTC. That will force Meet to stop using VP9 and switch to H264 for which MacBooks (and Intel CPUs in general) have a hardware accelerated decoder and encoder.


Note that H264 is not as efficient as VP9, so you may end up using more bandwidth or have lower quality because of this.


true but this is a favorable tradeoff.

i also recommend forcing mpv->ytdl to h264 for the same reason; fan keeps quiet while watching youtube.


Just tested a Google Meet meeting while on a native Zoom video call. Google Meet used 3x more GPU resources while having 1/5th the video participants.


I think the point of this article is how the Meet team handled the increase usage, not whether if this product is better of worst than competitors.

I bookmarked the article that can be seen as an engineering approach to solve a problem like this.


When did Google meet get better? I used it actively in April had a some really bad customer meetings on it (it seemed to be bandwidth heavy though that is anecdotal). I decided on Zoom and haven’t really looked back. I also was irritated at the Google Meets browser compatibility issues as I mostly use Firefox.


I used Google Meet once, and immediately ended up back at Zoom. I was not particularly impressed.


I found it to be perfectly adequate. Just like Zoom. Neither stood out and I'm perfectly fine with that.

Some of my fellow employees asked for a company wide Zoom license at the beginning of WFH so out of curiosity I looked to see how much Zoom costs. Turns out it costs about the same as all of G Suite which honestly surprised me (in how expensive Zoom is).

https://zoom.us/pricing

https://gsuite.google.com/pricing.html


The in browser version of zoom is really quite bad and meet is miles ahead of it.

That and I’m not installing a proprietary, possibly spyware ridden (excluding spying at the server) that installs weird daemons on my computer.

For family use, I’ve found FaceTime to have the best AV if everyone has a decent connection.


The fact that one of those requires only a modern browser, and doesn't hide the browser access behind a dark pattern in favour of a software download which is a security dumpster fire, causes me to not worry quite so much about ultra-crisp video.


It's gotten better but the video quality is definitely lacking.


Can't say I notice a difference, most platforms just take your webcam data and forward it so I don't get why people report different qualities? Never had to use zoom luckily but between Wire, Jitsi, Hangouts, BigBlueButton, and two others whose names I don't remember right now, it always depended on people's mics, cams, and WiFis (and sometimes available CPU if my crappy work laptop had too much to do in the background to en/decrypt the streams).


A lot of factors but Zoom is superior to Meet in video quality from my experience. Meet looks grainy. Zoom has various native clients.


> Zoom has various native clients.

Yeah that's precisely what I'm afraid of given their track record.


30x growth only? My wild gutted is that at least 1 billion Gmail App installations must have been updated to integrated Google Meet versions. So I expected much higher jump.

Just yesterday, we were asking people in our school WhatsApp Group to install Teams app. As for Google Meet, just update!!!


> 30x growth only

srsly?


To be fair 1 to 30 is 30x growth. It really depend on the original size.


It says they ended up at 100 million daily active users. The fact that they could deploy that capacity in 60 days should terrify their competitors.


I don’t think so, as their competitors already know Google is Google and has virtually unlimited resources. As far as I understand from the article it was mostly a matter of “hey can we have extra capacity”, which shouldn’t surprise any competitors.


The competitors scaled by similar factors from similar baselines. Here’s a comment I wrote in April, collecting the then-current info from the three big ones: https://news.ycombinator.com/item?id=23029151


Google Search had 1.2 billion users back in 2013, now it probably has 3 billion.

Youtube has 2 billion users.

Gmail has 1.5 billion users.

Android/Play Services has 2.5 billion users (well Android has 2.5, Play Services probably has at least 1.5 billion users).

For Google a service with 100 million users is a drop in the bucket :-)


1.5 billion monthly active users sending short text messages is a really different network capacity problem compared to 100 million daily active users sending high-definition video streams to each other.


But it's still small compared to YouTube video content served.

Also, the WebRTC stuff all browser based videoconferencing apps are built on supports peer to peer Comms in most cases, so worst case they could have moved all the users over to that in an overload situation.


Not sure about the comparison with YouTube. YouTube is one to many, and can be heavily cached. Meet is many to many with no caching.


Right, and YouTube doesn't need high priority traffic because it has a deep buffer on the receiver, whereas videoconferencing will be disrupted by even a single dropped frame.




Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: