Here's some detail, as someone that has worked with those machines before (internally and externally).
In the good old days the soft serve and shake machines were taken apart every night and every piece was washed and bleach-sanitized by hand. Then it was reassembled in the morning. This would leave a lot of room for error and possible contamination in the process. And the sight of a shake machine hitting full pressure with a misaligned O-ring in the barrel is a lovely one indeed.
So the newer machines take a different route: they self-pasturize. The machine goes out of service and slowly heats everything up to a bacteria-killing temperature (including the dairy mix inside), then cools everything back down to freezing temperature to serve.
This process takes about four to five hours. But if you set the clock wrong, the machine will go into its cycle at some super-inconvenient time. There are other triggers that can force a self-clean but in general it's not "broken", it's just doing this process at a bad time for you.
I'm curious how this data is being sourced. The last I checked there was an effort to put cloud-based telemetry into every unit and have them report home constantly (both to the equipment maker and to McD), but that was never intended to be public data.
EDIT: Ah, I didn't RTFT. He's using the public ordering site to scrape it (and now the title has changed). But that does mean there's some working connection from the equipment to the McCloud now. That's cool. Or maybe a manager is doing it by hand.
I actually froze my hand inside an ice cream machine like this when I was a teenager working fast food.
I hit a standby button instead of the off switch. (I was too tall to see the off switch and was never trained.)
I spent two weeks in a burn unit while they healed my hand with magic synthetic sin grafts.
Today you can’t even tell which hand got frozen in there.
Back in the day I worked in a dairy company as an IT guy. When the RFID antennas would need a reboot I had to go to the deep freeze storage (-40 Celsius), walk in with a ladder, climb the ladder, switch off and then on (IT crowd style), wait to see the green light bleeping, and then climb down and leave as fast as I could.
The whole thing would take 40-50 seconds (I was counting EVERY time). I couldn't wear gloves, because ladder/switch. I was wearing all other gear though. Touching anything (aluminum ladders get cold very fast at -40)(so are walls, switches, everything).
I did this process a couple of times wearing just jeans and tshirt. In that "less than a minute" my clothes would freeze solid, the wool was hardning and it felt like wearing a plank. Same with jeans.
In other cold related story, I was in a "cold country", out in the open, -25 Celsius, my phone rang, my gloves didn't have the electro-thingie so I couldn't "slide to answer" so I too my glove off, answered, talked for 60 seconds. Then I went to a warm pub, and after 10mins I could bend my fingers again.
I used to walk to school in sometimes very cold weather (-35 Celsius). Once, in order to try preventing the inside of my nose to freeze, I put my scarf over it. The scarf froze very quickly and I ended up with ice around my neck...
Now that I’m thinking about it, I wonder how people will manage masks this year with COVID in cold countries.
TL;DR: don’t put a scarf over your nose in very cold temperatures.
I'm Canadian and we get -35 temperatures for at least a few days every winter. This doesn't make sense to me. Of course you should put a scarf over your nose. At those temperatures , exposed skin freezes very quickly so you need to cover all your skin or you'll get frostbite.
I've never had a problem with a scarf freezing stiff like you describe. They build up ice crystals of course, but they don't interfere with air flow and are easily broken.
The biggest problem I expect from wearing a mask in the winter is that they'll get damp quickly because the cold temperatures, and perhaps having a scarf over top, will prevent moisture from evaporating.
I usually ride my motorcycle through winters in the upper midwest. No, it's not Canada, but at highway speeds in freezing weather it gets cold very quickly. I can't cover my whole face with cloth, or my breath will fog up and freeze over my glasses and visor. I apply vaseline over all of my exposed skin before riding. It does a great job of protecting my skin. I also do this when bicycling through extremely cold weather.
Heh.. many people commented on that. Perhaps you had a "single layer" that made it hard since the droplets/water from your exhale got stuck in the fabric, and together with the atmosphere's humidity created this effect.
In such cold climates I wear an ultra long scarf (2m-2.5m), I wrap my neck/jaw/ears 3-4-5 times (depending how tight) and I tuck it in my jacket. The outside of the scarf freezes, the inside remains warm. I also try to breathe by my nose, less humid.
It's a virus not a bacteria so the proteins don't disintegrate. The probability of transmission outside is small as sensibly people limit the time outside, but the flipside is they stay indoors, in confined spaces together, which is perfect breeding ground for the virus.
>TL;DR: don’t put a scarf over your nose in very cold temperatures.
Weird, as a Canadian who has been through many -35 days, I absolutely use a face covering, whether that's a scarf, neck gaiter, mask...Never had any problem like that.
A manager is doing it by hand. Their POS system has the ability to lock out individual items, and this information is available in real time to the server backing the mobile app (GMA).
The machines that I've seen don't have real-time telemetry, and are not networked.
There was an old effort a long time ago where the machines were on a powerline-networking system (LONWorks) but that is dead and buried. The next-gen effort was ramping up a few years ago and I don't really know where it is now. The predictive analytics company they bought was a big part of it.
Their tech stack is far more bleeding edge than I would have envisioned. I wish they gave an idea of why they chose to use k8s at the edge. I can see how containers are useful for deployment, and the fail-over built into Kubernetes is really nice, but I'm shocked that's the simplest/best way to do it at the edge.
I would have expected some VMWare product, just because it's "enterprisey". Shame on me for judging.
Deployments and monitoring are probably another draw. Same tools server and edge side.
Another project to keep an eye on is AWS's Greengrass which has a bunch of capability overlap, but a bunch of new ideas for edge and fog IoT also. I had a mixed experience with it but at least some ideas bear borrowing.
Not sure if this is the same thing, but I recall reading a case study in 2006 on McD's cloudifying their kitchen equipment. The project was apparently a huge failure and was eventually stopped due to runaway costs.
- You have a system whereby every instance has a 75%-79% uptime.
- Each instance's 21%-25% daily scheduled maintenance window can (generally) be arbitrarily-scheduled by the ops team.
How do you near guarantee that there's always one instance available?
If you said:
"manage a redundant cluster of at least 2 instances with non-overlapping maintenance windows"
Then you were correct!
BONUS QUESTIONS:
Q: What's the max number of ice cream machines found in a McDonald's location?
Q: What's the cluster size of the McDonald's ice cream monitoring system?
Q: How much money does McDonald's have?
Q: Why the fuck?
If you said: "1", "at least 3", "metric fuck-tons", and "I don't know", respectively, then you were correct!
The answer to #4 is probably that the goal isn't to always have ice cream. The goal is to run a profitable business.
Doubling the cost of the ice cream infrastructure to avoid a 25% downtime may not be a profitable venture -- especially if the downtime is scheduled to occur while the store is closed.
Also, you can use data to schedule the maintenance when the probability of someone ordering ice cream is low. That 21-25% downtime results in significantly fewer people wanting ice cream not being able to get it.
In the good old days the soft serve and shake machines were taken apart every night and every piece was washed and bleach-sanitized by hand.
I was a McD manager in my youth.
Not only were the internals fully disassembled and sanitized, they were left to soak in the solution overnight. All that was left behind was the cylindrical plastic chamber, sanitized and left to air dry overnight.
The messiest part was lubricating the O-rings with a petroleum jelly. Failure to remove all that when cleaning seemed like a good vector for microbes.
Heh! I did both of those jobs as well. I still remember the large plastic tray you had to use to organize all the little parts and O-rings.
Did you ever have a clever employee that thought they could use the 5-gallon plastic buckets as a quicker way to dispose of the old fryer shortening? I watched that happen one Sunday morning. That was amazing.
I worked there in the old days when I was in high school (late 90's), and one of the most disgusting stories I have about a job is when I went to refill the orange juice dispenser machine, and a giant swarm of flies were all over and in the top of the machine. In the orange juice, all along the sides of the inside, and flying around like a small swarm when I opened it.
I yelped and stepped back. A manager rushed over to me, slammed the lid shut, and told me to not tell anyone.
The machines still get taken apart and washed by hand, along with some maintenance, every 14 days. So every machine will additionally be off for an afternoon every two weeks.
Are perpetual stews sanitary? I'm probably misjudging, but the idea of anything being out in the open for that long makes me uneasy. I guess it's kept at a simmer, so bacteria can't grow?
A standalone GPS NTP server is $750 retail. Commercial soft serve ice cream machines are at least $4000. Just building the machines with an autonomous GPS clock that doesn't need to be set, would double the revenue but only increase the upfront cost by 18%.
Since they need internet connectivity to phone home status to McCloud, I don't know why they wouldn't just run an NTP client to keep their time synchronized and accurate?
Moreover the $750 you reference must be a commercial product since you can do GPS on a Raspberry Pi for < $50 easily? Which product are you thinking works for this scenario? Think I've seen everything from $300 to $13000 solutions for this over the years.
I doubt these machines are internet-connected. Most likely is that the store manager notices the machine has gone into its cleaning cycle and manually marks the item on the menu as unavailable.
If it was being integrated, the cost would be more like 5 dollars, since the ntp unit could just be an esp8266 chip that connects to the store wifi to sync the time.
If they can't manage to set the time what makes you think they'll connect the Wifi?
(...although in fairness some of the other comments here suggest the things might already have some kind of int[er/ra]net connectivity in which case no extra hardware required)
That sounds like an awful lot of work to not send NTP requests over WiFi. Then again, I'm sure someone will inevitably find a way to make these things spray ice cream everywhere over WiFi if they do. Or worse, mess with the internal temperature or cleaning settings to make them not safe to consume.
It's actually probably best they never touch the public wifi. I don't know why no one has mentioned ethernet yet, though. These machines are static, they just sit there. There's already tons of wiring going everywhere, so I would assume that it should be relatively easy to get a cable drop to the ice cream machine. Then you don't have to worry about any of this wifi security. You could do port level security, but that's probably overkill for an edge network.
I have actually been wondering lately why there isn't more of a market for some of these things that it seems like you could hack together. There has to be something I'm missing. You'll never be a unicorn, because most of these kinds of sensors have a limited scale, but it seems like a good return on investment for a small team.
I don‘t know about 3G, but I just checked for LTE [1] and the timing is broadcasted in SIB16, which is not encrypted, so theoretically anything could pick it up.
WWV/WWVH might not be a great idea, as I've read that there are rumblings about phasing it out.
You used to be able to pick up a time signal almost anywhere in CONUS with an FM receiver. It was encoded in the transmissions of PBS television stations.
This was back in NTSC days. Now that everything is digital, I don't know if it's still true.
The specification was called XDS and was developed by Sony to automatically set the clocks on their VCRs. PBS was a major participant but some other networks included XDS timecodes as well. Unfortunately late in the systems life reliability became poor (the encoder was not integrated with any of the other station equipment and was not being modernized) and I assume they all died out with the digital transition.
A better source today would be the time codes sent out by some FM radio stations with RDS encoders, to allow car radios to set their clocks. Unfortunately not that many radio stations do this and, once again, the time is not always all that reliable since the RDS encoder may not have any synchronization source itself.
The cellular network used to be an excellent source, CDMA cells required GPS time sync for TDMA reasons (well, CDMA reasons, technically speaking...) and broadcast a time code that is directly off of their GPS time source. Unfortunately, while GSM cells (and LTE) do broadcast the time, there is no guarantee made of precise synchronization as they don't broadcast a time code directly from their GPS source (not an expert in this field but I think the GSM/LTE time information comes from the possibly remote controller rather than the local radio hardware). Still, it would probably be good enough for this application.
GPS time sync is actually quite cheap to implement these days but tends not to work in these scenarios since a clear sky view is needed. WWVB is possibly on the way out. SNTP is probably ruled out less by the BOM cost of WiFi hardware and more by the deployment pain of having to get kitchen equipment configured for the corporate WiFi network.
Here's a fun idea: McDonalds presumably centrally controls the in-restaurant audio. Could they encode timestamps into the background music in a way that machines can cheaply recover? You wouldn't need high reliability, just enough for it to work once in a while. The old-ish Nielsen Peoplemeter system would be a model.
When you order a VPS they would’ve asked you then: “Do you want to supersize your order? For $1 extra you can have a whole bucket of RAM instead of a small cup of RAM”
Imagine if Barnes and Noble had done it, and you could drive to one of their stores to quietly sip a drink and peruse the new machines they had provisioned recently.
Since you and the OP are here, I have a random question: were the blizzards ever made with liquid plastic ingredients?
It probably sounds insane, but someone from a plastics company spoke at my high school and said this was the case since it was easier to ship and needed no refrigeration. Now, decades later, I'm wondering if this was just complete nonsense or had an element of truth to it. I've been able to find nothing online.
Not my field, but edible gums (guar gum, or xanthan gum for example) is "plastic" (adj.). Gums are used more and more, AFAICT as a consumer; presumably to allow products to be bulked out with air.
Basically take anything homogeneous, add gum, add air, create a foam. Basically like an Aero bar but with way smaller bubbles.
You just cut a quarter (I'm guessing on this figure) off your ice-cream ingredients by weight and you can now advertise "scoops from the freezer"!
It's genius but just a more complex version of putting all the pizza toppings in the little window, or having the coleslaw tub be much wider at the top, or putting everything in a wide plastic skirt so the box is 20% bigger than it needs to be, ... packaged food sellers are scumbags.
This 'bulking up with air' is most evident if you go to the yogurt or cream cheese section and look for the Yoplait Whips or Philadelphia Whipped containers. It's the same product, just as a foam, and seems to be marketed as a healthier alternative where users choose a satisfactory portion by volume, not by calories or weight.
And while I think the whipped foods are stupid, I don't think that packaged food sellers are necessarily scumbags. They're merely people responding to incentives, just like everyone else. The food market has razor-thin margins and consumers are ill-informed. Neither consumers (who are hard to inform) nor regulators (who are captive to the multinational conglomerates) have been able to push back more strongly than the profit incentive, so the so dark patterns like those you describe are inevitable.
Plenty of dieters do this themselves with protein fluff/"ice cream" (protein shakes with ice and xanthum gum). Overweight people almost by definition eat by volume not by calories (ie they eat more than their body needs for caloric maintenance).
It's not entirely true that regulators haven't plushed back: in the USA, there's a minimum amount of cream required before the vendor can call the product "ice cream".
The products in that category that have been puffed up with air (and don't meet this minimum) get sold as "dairy desserts".
(I don't work on the food industry, so I won't be shocked if I got details wrong.)
Which is exactly why McDonalds menus list "cones" and "sundaes". Not "ice cream cones". It's a frozen dairy dessert that is not legally ice cream. To be fair, McDonald's version is superior to any other fast food frozen dairy desserts that I've had.
The bulking aspect doesn't bother me so much, what does bother me is that it's not as good and the only major ice cream brands left that aren't like this are Haagen-Dazs and Talenti.
> "having the coleslaw tub be much wider at the top, or putting everything in a wide plastic skirt so the box is 20% bigger than it needs to be"
These days the trend seems to be to reduce the packaging, at least here in the UK. "Same size - less packaging!". So it seems like reduced packaging and distribution costs, and perhaps fitting more onto store shelves, outweighs any extra sales from customers who are fooled into thinking the product is bigger than it really is?
I think there are two things, one is a preference with some consumers towards less packaging (given two products about the same cost and one looks like less packaging I'll tend to pick that) and the other is there's been a lot of talk about taxing single use plastics so I've wondered if they've been trying to alter the supply chains before that point.
I'm not a chemist but I've recently come to realize that the meaning of the term in the popular psyche has probably changed over the decades, taking on largely negative connotations, because of its association with a small subset of materials (albeit large by volume), most of which are either toxic, a pollution problem, or both.
AFAIU, in industry "plastics" is just a blanket term for moldable organic materials that chemists are constantly inventing. They're not necessarily sourced from petroleum, nor necessarily toxic. It's a ridiculously huge category of material, and plastics really were and probably always will figure prominently in our future. And I would assume that some traditional, even edible materials have been subsumed into the plastics category given the general scope of the term and the industry itself.
Former chemist, yeah this always bugs me. Laypeople use "plastic" to refer to mostly inexpensive injection molded parts, but it really includes everything from LDPE bags to glass fibre reinforced nylon powertools to Rayon fibers to thermoplastic elastomers.
Maybe you mean McFlurry (Blizzards were Dairy Queen). Basically the same thing, but not sure if whoever said that was specifically talking about Blizzards.
Yeah! Whatever you do, never eat pretzels! They dip them in oven cleaner before baking, then they sprinkle them with those crystals they use for removing ice from roads!
And dihydrogen-monixide figures heavily in their production. Nearly everyone that's ever died had consumed DHMO in the 24 hours leading up to their death!
A solution of about 4% NaOH is generally recommended. Submerge the pretzels for a few seconds just before baking. It does something to the outside layer of dough that results in the brown crust of the pretzel. In addition to the pleasant color, it also affects the flavor. Also, it makes it easier for the coarse salt to stick to the outside of the pretzels.
NaOH immediately causes stains on clothes and wooden cutting boards and countertops. Obviously it's also bad for your skin and especially the eyes, so be careful.
Bagels are usually dipped in lye as well, it's how they get their bagel-y texture on the outside crust.
Sodium carbonate is sometimes used as an alternative to lye for those who don't want to deal with the safety aspect of handling lye, as lye is very caustic. Of course, lye still provides the best results.
Speaking about BK, the liquid they used to make the soft-serve was most certainly refrigerated.
Logically though, refrigeration based transport is not a concern for any fast food company. They use ingredients that absolutely must be transported cold, thus the cost of adding another menu item that must be shipped cold to that list is immaterial (even at scale).
That's actually incorrect, they list two plastics there. It's just that "plastic" isn't something that is actually necessarily toxic, and the media misuse the terminology all the time.
Someone should save the data over time and figure what is wrong with every machine. If this is made public and enough tech people tell the store every day to change example from 2pm to 2am, this will likely result in a better product for everyone.
At first I was gonna ask why does this not run on off hours, but then you mention bad clock, so the next question is, why is the % of bad clock so high, and why isn't the clock also cloud based or set in a way that can go wrong.
Next up, what about redundancy, do stores only have one machine?
McD is conservative when it comes to kitchen technology. With certain exceptions, franchisees determine when they buy new equipment and not the corporation. Most of it is designed for a 7-year capex cycle.
But to answer the question: no, the machines aren't net enabled and do not NNTP their clocks. But that's changing.
But I still wouldn't design my architecture to let a central server decide when a machine should go out of service. Think about what else you can fuck up that way.
Why are they off? Good question. Sometimes it's as simple as a store owner setting the heat cycle for 2pm instead of 2am.
Why are they not redundant? They're expensive, take up a lot of valuable space and a lot more valuable energy, they need to be filled with 2x the product, and the customer demand isn't that high at any given moment to warrant the expense in most stores.
> Next up, what about redundancy, do stores only have one machine?
From my (old) experience working at one, yes. The annoyance of having to maintain a second machine (especially when cleaning and maintenance involved a physical person cleaning things) was not worth keeping extra 9's of uptime.
Don't you mean NTP? We've had this technology for years! If it's connected to the internet, it's a pretty fundamental piece of pretty much any networked OS. You can ensure correct time this way.
If you have a datacenter full of computers, obviously you don't want to run a separate GPS antenna for every system. NTP allows systems to synchronize to GPS over a network.
I've not heard that term used, and its bad. You shouldn't dehumanize people for being "cheap labor". Honestly, that guy flipping burgers is doing more real good for people than any number of highly paid software devs working on VC funded projects that will mostly end up in the trash.
Did you work in Oakbrook? I was on a team that was trying to make a few smart restaurants. We got to about 50 before the project was canned but we were trying to put everything from fryer oil life to what the platens were cooking and HVAC all on a touch screen. It was a lot of fun.
I’ve worked for a few commercial equipment makers. Talked to Oakbrook and Romeoville but never been inside.
Every time McDs changes CEOs I feel bad because that means almost every current project is going to be shitcanned as the new person tries to make their mark. Are you still there now?
In the good old days the soft serve and shake machines were taken apart every night and every piece was washed and bleach-sanitized by hand. Then it was reassembled in the morning. This would leave a lot of room for error and possible contamination in the process. And the sight of a shake machine hitting full pressure with a misaligned O-ring in the barrel is a lovely one indeed.
So the newer machines take a different route: they self-pasturize. The machine goes out of service and slowly heats everything up to a bacteria-killing temperature (including the dairy mix inside), then cools everything back down to freezing temperature to serve.
This process takes about four to five hours. But if you set the clock wrong, the machine will go into its cycle at some super-inconvenient time. There are other triggers that can force a self-clean but in general it's not "broken", it's just doing this process at a bad time for you.
I'm curious how this data is being sourced. The last I checked there was an effort to put cloud-based telemetry into every unit and have them report home constantly (both to the equipment maker and to McD), but that was never intended to be public data.
EDIT: Ah, I didn't RTFT. He's using the public ordering site to scrape it (and now the title has changed). But that does mean there's some working connection from the equipment to the McCloud now. That's cool. Or maybe a manager is doing it by hand.