We built a search engine that shows you the most engaging stories/topics being shared across Twitter, Facebook, Linkedin, and Google+. We crawled over 15 million articles the past 3 months, retrieved the total number of Facebook likes, tweets, Google+’s etc and built a search index around it.
Here's what our infrastructure looks like:
Rails/Redis: We use the Sidekiq gem as a message queue. We have hundreds of workers that do the crawling, data mining, and number crunching.
ElasticSearch: We built the search index using ElasticSearch, with the data imported from our Postgres database.
AngularJS: Front-end is built using AngularJS and Laravel 4 (PHP), which sends the requests to our Rails server in the backend.
The speed is really impressive, at least for the queries I tried. There are still a lot of queries with no results, and you might want to think of something clever to put there in those cases to keep the user turned onto your brand. When there are results, they seem quite relevant and useful.
The "Show me how it works" tour is great too.
I think you just need to work on your value proposition on the landing page. Don't be afraid to sell something. Put together few paid packages as soon as possible. Don't worry if it's not perfect.
Tried "customer success" for example and the results were not valuable. Tried "startups" and got lots of results, but that might be too broad of a topic.
Technically I'd say this is pretty good, I agree that OP needs to work on value proposition and to sell: that's something for content marketers, but try to dive more into the problem you're solving. Be specific. "Social insights" seems too broad to me.
For example, the problem could be not knowing which content to write, wanting to keep tabs on what your competiton writes about, spending too much time researching all this...
You're already writing about benefits("discover what people are sharing..."), I'd try to connect that with how your app is better than current existing methods used by content marketers to create their content strategy.
Thanks! We really appreciate the feedback. We're constantly working on growing the index and improving the search so expect that to improve over the coming months.
We built a search engine that shows you the most engaging stories/topics being shared across Twitter, Facebook, Linkedin, and Google+. We crawled over 15 million articles the past 3 months, retrieved the total number of Facebook likes, tweets, Google+’s etc and built a search index around it.
Here's what our infrastructure looks like:
Rails/Redis: We use the Sidekiq gem as a message queue. We have hundreds of workers that do the crawling, data mining, and number crunching.
ElasticSearch: We built the search index using ElasticSearch, with the data imported from our Postgres database.
AngularJS: Front-end is built using AngularJS and Laravel 4 (PHP), which sends the requests to our Rails server in the backend.
1) Ability to search for influencers that shared an article related to a specific topic. For instance, I want a way to search for SEO influencers who shared something related to Breaking Bad b/c I'm making an article about how SEO is related to Breaking Bad, etc.
Seems like a pretty smart idea. Good work. I'm impressed by the speed. I like the tour too. I think the UI is very text-heavy right now, it might benefit from swapping some text for icons.
I'd like to see more of a pitch by way of how this app can help me and why I should be using it. Other than that, good job. Hope it does well.
We're two developers, to get to this point Henley built a crawler and search index around the data. This is then available via a private API. I (James) built the web app using Angular and Laravel.
Laravel is currently just used for user auth and as a proxy to the api. Nearly all the front end logic is handled by AngularJS (it's a single page app).
Really good to know in few minutes what to know about a topic from social media perspective.
I will not miss things of these social media I don't read but I have to know what they are talking about, and you search engine aggreator is good for this sum up.
Yeah, quite interesting project. As someone who is interested in marketing and social media, I find value in the results of buzzsumo. I guess thats because posts on these topics tend to get a lot of shares and there are many signals to feed the algorithm.
This is really interesting but I worry about the share of the net you all have crawled/are actively gathering stats on. Perhaps you need a way to request the crawling of a site or keyword?
Please do a hard refresh (ctrl-r) and let me know if this doesn't fix it (email in profile and on site). I had to push some bug fixes recently and you might have some old cache files causing problems.
jwblackwell 48 minutes ago | link [dead]
We built a search engine that shows you the most engaging stories/topics being shared across Twitter, Facebook, Linkedin, and Google+. We crawled over 15 million articles the past 3 months, retrieved the total number of Facebook likes, tweets, Google+’s etc and built a search index around it. Here's what our infrastructure looks like: Rails/Redis: We use the Sidekiq gem as a message queue. We have hundreds of workers that do the crawling, data mining, and number crunching. ElasticSearch: We built the search index using ElasticSearch, with the data imported from our Postgres database. AngularJS: Front-end is built using AngularJS and Laravel 4 (PHP), which sends the requests to our Rails server in the backend.