I don't remember the exact details because it was a long time ago, but what I do remember is
- Limiting memory usage and number of connections for mysql
- Tracking maximum memory size of rails application servers so you didn't run out a memory by running too many of them
- Avoid writing unnecessarily memory intensive code (This is pretty easy in ruby if you know what you're doing)
- Avoiding using gems unless they were worth the memory use
- Configuring the frontend webserver to start dropping connections before it ran out of memory (I'm pretty sure that was just a guess)
- Using the frontend webserver to handle traffic whenever possible (mostly redirects)
- Using IP tables to block traffic before hitting the webserver
- Periodically checking memory use and turning off unnecessary services and cronjobs
I had the entire application running on a 512mb VPS with roughly 70mb to spare. It was a little less spare than I wanted but it worked.
Most of this was just rate limiting with extra steps. At the time rails couldn't use threads, so there was a hard limit on the number of concurrent tasks.
When the site went down it was due to rate limiting and not the server locking up. It was possible to ssh in and make firewall adjustments instead of a forced restart.
The flag is obnoxious (duplicates and okay-but-inferior sources don’t seem to violate the guidelines, and flags are about guideline violations), but ChrisArchitect is right: this link seems to be the same content but much more informative:
Dealing with the steep costs of internet services in my country and being an avid podcast listener myself, me and a couple of friends came up with a solution. We developed a website that works as a mirror for users' selected podcast feeds and converts the episodes into opus format, slashing the mobile data consumption that's a real concern for internet access here: https://easypodcasts.live
Wow I like this idea at least for many conversation based podcasts. I try to download mine at home but data isn't as expensive for the times I don't. Still there are reasons to use this.
I need to look into how the mirroring works, for some time I have wanted a way to subscribe to a podcast with a timeshifted start. Say from the beginning up to the date I start, or a range between 2 dates. For podcasts that have existed for a long time I find it hard to work through the back catalog in any consistent way.
yt-dlp works with many but not all podcast websites. You could download everything with metadata and mtime and then sort by earliest to most recent in a file browser.
I wrote a tool that might help with this if you don't want to download the files immediately but it definitely isn't required if you just download everything and use yt-dlp's download-archive.
pip install xklb
lb tubeadd podcasts.db 'http://www.hellointernet.fm/podcast?format=rss'
lb listen podcast.db # it will keep track of what you played and prioritize playing things that haven't been played before
And later do tubeupdate to check for new episodes:
lb tubeupdate podcasts.db
If the website extractor provides metadata about file creation (eg. YouTube extractor) then you could use this to play the oldest videos first:
lb listen podcast.db -u time_modified
If you want to download later you could do this:
lb dl podcast.db --audio # with the above example it saves to the folder "Generic" in the current working directory since it uses the yt-dlp generic extractor
The mirroring part is for the rss feed specifically. When a podcast is added to the website a feed is generated with the same data as the original feed but only including the processed episodes and the media links pointing to our files. Then you subscribe to that feed instead of the original using your favorite podcasts app.
We have some "hidden" features like subscribing to a tag or merging many podcasts in one feed, but nothing time related.
