I'm just curious how you go about doing it and how you actually index the links. Let's say you scrape tbp (or any other site) every X minutes. You see a link-- how do you know if you already saved that link? From there, I'm assuming you just have a simple model for each link which you save and render on the frontend.
I'm used to building pretty simple webapps, so doing something like this is really interesting to me from a learning perspective.
In addition to the technical tasks, how did you go about marketing your site?
Well, I used Ruby on Rails (I'm a Rails dev). Initially I expected to have to scrape TPB, but checked out google and found there was a PB API. I started looking into that, and was going to write a Ruby wrapper for it, but then realized if I was gong to do it that someone else probably already did it. Looked for a few minutes and found a couple of Ruby gems ready to go.
So what I would do was check TPB every hour, and get the links through the API. If I didn't have them in my DB, I added them. If I had them I updated the seeder / tracker info.
Once I have that info, I scan the DB for magnet links that haven't yet been thumbnailed. I use the transmission-daemon to manage the torrent downloads, and keep a handful of them going at all times. Getting that working properly was kind of a pain in the neck.
Once a torrent is downloaded, I use ffmpeg to scan it and make 10 thumbnails of every movie file in it. The torrents are deleted after they're thumbnailed.
Not much marketing, I just post about it here and there in places where people would be interested in hearing about it.
I'm hoping TPB comes back up soon so I can refresh my data. It's way out of date.
I've been toying around with this for a couple hours. Getting the links is relatively straightforward (I'm using kickass torrents). How do you go about adding the magnet links to transmission-daemon? You can setup a watch directory so that torrents will automatically start downloading when you fetch a torrent file, but that's not what's being done here. I'm assuming you're remotely monitoring the downloads true, correct?
As for ffmpeg, is that automated or do you have to do some manual work every time a torrent is finished?
You could use aria2, which is a command line program so no need to deal with a daemon. Also, you could make it download just parts of a torrent (the start of the torrent for example) instead of the whole thing and take the screenshots from there.
I'm sure there are plenty of technical challenges, but i'm more curious about the bureaucratic ones.
Where do you host your site and how do you get paid? Systems like Google AdSense and payment gateways run from anything related to porn like it's the devil.
edit: Could you also talk about SEO and how you market your site?