I wonder if this could be a case of mismatch between what the recommendations system was designed to do and what the business actually needed it to do. Your team evaluated the models based on live KPIs in an A/B testing environment, but did the recommendations team develop the system specifically with those KPIs in mind? Did they ever have access to adequate information to truly solve the problem your team needed solved? And was the same result observed for other uses of their recommendation systems?
> did the recommendations team develop the system specifically with those KPIs in mind?
Yes they did - in fact they had input on defining them and helped in tracking them.
> Did they ever have access to adequate information to truly solve the problem your team needed solved?
They believed so. Their team was also responsible for our company data warehousing so they knew even better than me what data was available. Basically any piece of data that could be available they had access to.
> And was the same result observed for other uses of their recommendation systems?
I did not have first-hand access to the results of their use in other recommendation contexts. As I mentioned in my original post I only had second-hand accounts from other teams that went the same route. They reported similar results to me.
Some ideas seem to attract smart people like moths to a flame.
It seems like everyone who joins my company to shake things up follows the same path of wanting personalized content to acquire new customers.
But in reality we just don't have enough data points on people before they become customers to segment people that way. Even if we could, being able to accurately
Every time I see people go through the motions of attempting to implement this until they eventually give up.
This idea looks like an obvious win and big companies have done them before with success, but is extremely hard to impossible to pull off for our small company.
