It's funny how common wisdom in the software industry is "reuse, don't rebuild", but if you look at projects that are actually successful and developers that are highly regarded, they all have a massive case of NIH syndrome. Google and Microsoft both build all their own software. Mozilla completely reimplemented COM for Gecko. John Resig just got criticized recently because TestSwarm duplicates a lot of the functionality of Selenium Grid. Linus Torvalds wrote his own OS instead of hacking MINIX, and then wrote his own VCS rather than use Subversion or CVS. Chuck Moore wrote everything himself, down to the hardware, which he designed with CAD software he wrote himself in Forth, a language he wrote himself.

If you look at it by numbers, this makes perfect sense. The average library is, by definition, average, so a coder who is way above average can undoubtedly write something better given enough attention to the problem. Plus, many of these super-coders got their reputation by writing lots of code in the first place, which you don't do if you just reuse third-party libraries.

Unfortunately, everyone thinks they're above average, which is why we get such a profusion of libraries in the first place. ;-)

Like all other absolute statements, it's wrong, just as the converse ("rebuild, don't reuse") is, and a balance is required.

For context, I do a lot of XMPP work, which is all XML. In the languages I work in, I've gone through all the XMPP libraries that I know of, and found them wanting. I now use some layers I wrote myself directly around Expat. Why? Because nobody but nobody seems to be able to write an XML library that correctly handles namespaces! Which is death when working with XMPP. Yeah, if you hack hard enough you can get around the problem, but especially given the thinness of the usual XMPP libraries and what little they do, it doesn't take much to flip over to negative value.

On the other hand, when I went to write a routine that takes a PNG file of arbitrary size and resizes it to be an avatar, I'd be a damn fool to yell "screw it!" and reimplement a PNG reader.

So, it depends. Factors to consider include the long-term price you'll pay for a sub-optimal library, how close to core it is to what you are doing, whether or not you've ever implemented anything close enough to the library in question that you are really sure you can build something better (sometimes libraries suck because they suck, but sometimes they suck because they solve a hard problem and the "suckage" is actually the hardness of the problem poking out) as it is so easy to end up creating something worse, and, since I deal almost exclusively in open source now, whether you can take most of an existing library and actually bring it up to spec more easily than starting from scratch.

One of my favorite maneuvers, not that I do it often, is to take some rather large library that does what I need but not terribly well, along with a lot of other stuff, and then tear it to shreds in the process of making it do what I want. A coworker of mine recently wanted to do zeroconf autodiscovery, so they started with dhcpcd and started tearing out everything except the use of the dhcp protocol, because it was easier than trying to start their own stuff from scratch. Worked great in less than an hour; I can't imagine what else they could have done in less time, even finding a conventional library, downloading it, and learning how to use it would have taken longer than that.

I do use 3rd-party libraries. I'm a big fan of JQuery, for example - I can't use it in my day job because we literally count bytes when serving JavaScript, but I use it all the time for prototypes and personal projects. But I know that Resig is a top-notch coder, and more importantly, JQuery has had thousands of projects beat on it for a couple of years. If there're corner cases, they've been explored and uncovered, bug reports have been filed, and patches have been made.

Similarly, I use Apache/Lighttpd and MySQL and Django and PIL and BeautifulSoup (though I had some installation issues with PIL, so maybe that's not a great example of a well-tested, well-documented library). But these are pretty well-exercised projects with large userbases: other people have already done the dirty work of discovering all the bugs.

Makes me wonder why anyone would ever use new open source projects, which is essential to getting to the well-tested, well-exercised stage. I think it comes down to need: they'll reach for an imperfect library if the problem is so scary that they don't even want to approach it. So go work on hard problems, and you'll be popular. ;-)

> The average library is, by definition, average

It's not that simple. The average programmer who releases a library is better than the average programmer.

The benefit of using a library written by an above-average 3rd party is often offset if not negated by the above concerns.

I really sympathize with the OP, because I inherited a web based enterprise system that was built using as many open source packages as possible. It appears to be something north of two dozen; some of which I have never heard of. Have you ever tried googling for "barbeque software"? (I found it, barcode scanner.)

I am a big believer in getting your core compancies down cold. I'm in the Java world, so that's a given. If you are serious into business, you must have a good understanding of SQL. I am now going through hell because the previous developer wrote his object hierarchy in java and XML and let Hibernate put his database together. Now I have serious performance problems, along with mysterious bugs.

I see too much of the attitude of adding layers of adding layers of abstraction to allow them to to use the Java mindset. God sakes, people. SQL is way more powerful than Java. Use it. The same thing with JavaScript libraries. Don't pull in a 100K library to get a couple of useful functions. Write them yourselves. Avoid getting caught in a maintenance trap.

While I agree with a lot of the rest, I'm not sure I understand this part... a 19k library (jQuery) that gives you a single DOM interface that fully takes advantage of JavaScript and works in every major browser is what will keep you out of the maintenance trap, not writing/maintaining that interop code yourself.

I've noticed quite a few programmers inside Google that are easily the caliber of Resig or Zawinski, but I'd never heard of them outside of Google because they work on closed-source software and don't go around taking credit for their work.

Rereading it, I think it was mostly right, but partly wrong. The path to massive programming productivity isn't to rebuild everything from zero; it's to reuse only the right things.

What you won't see reused is more high-level functionality. That's where a lot of assumptions have to be made about the program's internal structure, and it's where getting too generic and open-ended makes things very complex very quickly.

I use 3rd party libraries until I hit a tipping point where either the effort to implement is looking to be greater than the effort to simply write the baseline functionality I need from it. The basic idea is to get it to fill in the functionality you need to prove your point and then rewrite it as your needs increase, or if you're like me, are OCD about a consistent code base.

Instead of writing from scratch, I tend to copy and paste whatever I can salvage and build from there, typically refactoring the original code into something new entirely.

Otherwise, I code in my own scripting language and I deploy to my own app server. I'm very productive this way and I'm able to leverage stuff that doesn't suck.

Whoa, I read a bit too fast there and thought for a second that it was true -- Chuck Norris could do anything!

I am a 10x coder, although I prefer when people just refer to me as someone who gets stuff done.

The answer to your question is simple: Yes, we are faster because we overcome these challenges, because we are as much sysadmins as we are developers.

The secret sauce is to realize that the world does not end at the borders of your VM.

If you refuse to acknowledge that your webapp needs an operating system, a webserver, a cache and a database to operate and if you push the lowly "grunt-work" of setting these up to someone else then you'll never be a 10x developer. Because in that case you're depriving yourself of acquiring the magic systems knowledge that leads to all the little "ah, this could be easier"-moments.

Hubris makes me sad. :(

>we are faster

That's unnecessarily inclusive and doesn't convey anything of substance.

>If you refuse to acknowledge that your webapp needs an operating system, a webserver, a cache and a database to operate

>depriving yourself of acquiring the magic systems knowledge

I smell a proud C/C++ programmer.

waves ASM coder over here, we (??) asm coders can write parsing libraries that spank yours seven ways to Sunday.

Does it matter? No. That's not how you "get things done (TM)".

Sorry to hear that, I was merely trying to provide a frame of reference.

I smell a proud C/C++ programmer.

Way to read something into a post.

waves ASM coder over here, we (??) asm coders can write parsing libraries that spank yours seven ways to Sunday.

That's fine and may get you a nice job at a hardware company. I don't see how it's related, though.

Does it matter? No. That's not how you "get things done (TM)".

And where did I say anything to that tune?

The point I was trying to make is the opposite: As long as you consider yourself a programmer, or worse a programmer of a particular language, you're probably not a "10x coder" [who keeps coming up with these idiotic terms anyways]. It doesn't matter whether that language is C, asm, Java or Lisp.

I've done the same but I am not a 10x coder. I'm just a generalist on steroids who happens to take a fancy to asm, security, arch, and abstracted programming languages.

...that means you are a 10x coder but would not like to claim yourself though.

It's just absurd, I'm an over-diversified generalist, I'm better than no one at anything.

However, the 1x coder ignores the problem, or declares it unsolvable. Inevitably, he waits for the 10x coder to fix it. The 10x rule holds.

10x coding is as much about willpower as it is about technical competence and experience.

Haven't we seen that before? It's usually solvable within 5 minutes. And so true, if you want to become a good programmer, never give up trying to solve a problem, and don't wait for your seniors to solve it for you, because you will never develop the skills needed to solve the problem yourself that way.

One rule I use is that when I pick something up, I make sure that I can google help whenever I need it. If Joe-Bob's awesome language or widget set has only 47 people using it, and you have to put up with abuse on an IRC channel or buy 15 books just to get respect from the community -- forget it.

Wise choice of tools also means your cognitive load decreases. You simply don't have to remember so much crap anymore. Problem remembering how to attach events to an element in Javascript? Ten seconds with Google and you've got your answer. There's a certain tipping point that products reach. Be careful using stuff before acceptance grows to that point.

That's _platform-ish_ stuff. Tools, widgets, languages and such. For whatever you're actually doing, consider rolling your own stuff. Most times you only need a small subset of functionality, and if you're a good coder it's as easy to write it as use somebody else's. Plus, you understand what's going on. Many times I've used stuff with open source code only to spend hours plugging through their code trying to figure out what the heck they were trying to do. That hurts. Don't do that if you can help it. Sure it's great to modify OSS, but it's probably just as easy to spend that time learning more about the technical problem domain.

For instance, I'm playing around with the idea of writing another web app. I took a look at YUI -- I really like it. But at the end of the day, so far at least, it does a whole lot of stuff that I don't need, and payload size is important to me. So I'm just going to roll my own stuff. You're always weighing future hassle against productivity. Sometimes, like with learning functional programming or picking up COM or something, the rewards may be much greater than the hassle. But for most stuff, like version control, project planning, configuration, deployment, setup, etc -- you don't have the luxury of taking 3 days to figure some of these tools out. So don't.

Rewriting the same functions yourself in a better and more efficient way teaches you a lot, especially about 1) how to spot good libraries, and 2) when not to write it yourself.

If you look at the code of really effective hackers, the one thing they have in common is how small the code is compared to what it does. Look at web.py, or bret taylor's datastore built on top of mysql: http://bret.appspot.com/entry/how-friendfeed-uses-mysql

Inexperienced troubleshooters often enter the work with pre-concieved notions of how the system works and hold on to them for too long. Always doubt your knowledge.

I think it usually takes 1-2 days with a system until you start to get our bearings. Before that, you are just collecting data, random tidbits of how the system works.

I second the recommendation to use recommended libraries, and I love github for the code network analysis. Back when I did a lot of java development there was often a java library available which solved the problem available on the internet even for a specialized problem, but more often than not it turned out to be a completely shit implementation that needed to be replaced. Of course there are good java libraries, the trick is to know someone you can trust who can recommend libraries.

OTOH for some rails tasks that can be resolved by just adding a plugin the quality of the code doesn't matter much, since it's just a few lines of code configuration. Still a good library is much better than a bad. I wish there was some kind of leaderboard, or official plugin list for rails, it can be very hard to judge plugins for a newbie.

Also, there is something that has to be said for compromise. Get libraries for the language you know best and you will more productive than if you used a high-performance library for a language you don't know. Start small, and worry about scaling or "going big" later.

Experience makes a huge difference - simply because you've seen this problem before, or at least seen this kind of problem before, or maybe you saw the solution to this problem a year ago when you were googling the solution to some other problem and tucked it away in a cavity in your brain.

With experience, you happen to know that there are certain kinds of many-to-many relationships Hibernate can't map in a legacy schema where the foreign key is a subset of a composite primary key. So when you encounter that on your next project you drop right down to SQL, to the wonder and bewilderment of your new colleagues whom you've just converted to an ORM tool because it "abstracts away all that sql stuff".

With experience, you know the kinds of ways certain systems go wrong, the kinds of solutions and workarounds to apply, and most importantly the kinds of things to google for when hunting for the solution.

Even scanning search results pages - to a novice they're homogeneous, but after some time you start ignoring whole classes of results. For example, you're trying to find what some annoying ActiveRecord exception is about, and you realise half of your search results are pointers to subversion log messages mirrored in a thousand places all over the internet. And they don't have the information you're looking for.

As some other commenters observed, you have to push and solve those problems that seem impossible, because the next time around, you'll find the answer a little faster. Keep up the head-banging. It eventually turns into supernatural powers. Just be careful with the wall.

Here are mistakes I see people make that I try to avoid: 1) Making decisions on technology before they have actually tried it. 2) Upgrading to the latest version of products too often. For every problem it fixes... 3) Fail to give developers system administrator access rights. Nothing worse than not being able to install stuff, but it happens all the time.

I've been using the same toolchain for a decade now. I don't have to futz with the configuration: I learned it 10 years ago.

There's a balance there, between keeping on top of technology and focusing on solving problems.

It's been interesting for me doing a little Ruby of late because coming from mostly C++ (with bits of Java and Perl) I find myself fighting with the interpreter a lot. It's not that C++ doesn't have its quirks, it's just that I've known them all for so long that I work around them intuitively.

After having built software in different languages for different targets on different systems a few times, you start recognizing patterns.

One of the strategies I use to become comfortable with new environments is to build a piece of software (usually a prototype) using only the core language, standard library and maybe one or two 'mature' pieces of third-party library code.

This takes a bit of time and I usually end up with a codebase that is a bit crufty (especially when I'm learning the language as I go). However, this usually gets me to the prototype stage with a lot of newly acquired knowledge about performance of the language, best-practices, etc. As well as a lot of testing code.

After that I'll start looking into third-party libraries and start factoring out my own code with third-party libs where appropriate.

For Java, for example, I've found that a lot of the language helper/util code I write can be replaced with code from org.apache.commons.

For interacting with crufty stuff like webservices in all their non-standard glory, I usually build a few classes in Ruby with breakpoints so I can interactively analyze and poke at them. Much, much faster than going through the 'change code'-save-recompile-run cycle with breakpoints in C# and Java.

When dealing with loads of XML config files, I'll quickly do a git init and also keep all the files open in vim buffers. (Vim and Emacs are excellent tools for working your way through XML junk). Then I'll put print (or equivalent) statements in the code so I can run the app from the command line and filter the output through grep or even some custom scripts that'll filter the output for me and provide some basic analysis or correlation points.

The list goes on and on.

You have to realize that apart from the code that operates solely within your codebase (and not even then), you are always dealing with third-party code. The OS, compiler, the VM, the windowing routines, the threading model, etc.

As you go on you have to learn techniques to handle this. And one of the first things you already seem to have learned is that you should never be quick to voluntarily add to the pile of third-party software you're already dealing with.

And that 10x programmer working opposite you who generates mod_rewrite configs like there's no tomorrow? Yeah, that person spent about 3 days getting to the bottom of that engine and testing the hell out of various configs a few years ago. There's not much of a short-cut for these kind of things.

* Not invented here syndrome: 10x coder may code small things himself that many times is faster then learning other stuff.

* Selection of the tools to use: it will use Ruby instead of Java, for example, and for a reason. In general he'll try to figure the fastest path to solve the problem so will try a lot less.