Yes and no. You do need to have broad enough experience with programming that you are comfortable with the basic implementation algorithms even if not up to speed on the nuances of them. However, a modern database kernel typically has few or no B-trees or red-black trees in it, using more specialized and purpose-built algorithms instead. Probably one of the most important lessons I learned when I first started poking at database kernels is that the algorithms in the literature were demonstrably not being used in commercial databases because the behavior did not match what you would expect. The algorithms you learn in a CS curriculum are often not really correct for any real system built with a high degree of skill.
I've only ever owned a few CS-related books (e.g. Li and Vitanyi). Trawling search engines for CS papers and references to CS papers with potentially useful information is how I did most of my work. Honestly, Wikipedia is a good starting point to find search terms but you still have to follow threads in the literature that are poorly or non-existently documented on the Internet. There are important areas of literature that still have very limited presence on the searchable web. This is a huge bias I see with many people fresh out of school -- if it is not on the web it doesn't exist.
On your last point, that is the most difficult part. To be frank, most PhDs in computer science do not have the skills to successfully design new algorithms either. Oddly, chemical engineering provided a very good conceptual framework for thinking about this; there are no pure mathematical solutions and you optimize complex systems in part by understanding every knob in a holistic context. One of the things I've noticed about most people with a computer science education is that they make a large number of implicit assumptions about the nature of computational and algorithmic systems that are not required and sometimes not correct, often without realizing it. Most of the algorithm work I've done amounts to violating implicit but foolish assumptions in the orthodox literature that no one had seriously examined in decades.
It is not just computer science, in most fields of endeavor the vast majority of things are done the way they are because that is the way they've always been done as set forth by a smart person a long time ago. The underlying assumptions and implicit constraints are rarely reexamined. I found a lot of upside to studying theoretical knobs that had been essentially frozen in literature since at least the 1980s.
I've only ever owned a few CS-related books (e.g. Li and Vitanyi). Trawling search engines for CS papers and references to CS papers with potentially useful information is how I did most of my work. Honestly, Wikipedia is a good starting point to find search terms but you still have to follow threads in the literature that are poorly or non-existently documented on the Internet. There are important areas of literature that still have very limited presence on the searchable web. This is a huge bias I see with many people fresh out of school -- if it is not on the web it doesn't exist.
On your last point, that is the most difficult part. To be frank, most PhDs in computer science do not have the skills to successfully design new algorithms either. Oddly, chemical engineering provided a very good conceptual framework for thinking about this; there are no pure mathematical solutions and you optimize complex systems in part by understanding every knob in a holistic context. One of the things I've noticed about most people with a computer science education is that they make a large number of implicit assumptions about the nature of computational and algorithmic systems that are not required and sometimes not correct, often without realizing it. Most of the algorithm work I've done amounts to violating implicit but foolish assumptions in the orthodox literature that no one had seriously examined in decades.
It is not just computer science, in most fields of endeavor the vast majority of things are done the way they are because that is the way they've always been done as set forth by a smart person a long time ago. The underlying assumptions and implicit constraints are rarely reexamined. I found a lot of upside to studying theoretical knobs that had been essentially frozen in literature since at least the 1980s.