I wrote a UDF for MySQL. It had just shy of 100% coverage. It used a malloc. I could not figure out how to trigger a malloc failure inside of a imported library running on a MySQL instance.

Do I not check for malloc failure (in order to get 100%)?

Or do I develop some sort of instrumented version of MySQL which lets me test malloc failure? That seemed far more overkill than warranted for the small project I worked on, though I certainly know some projects which do that!

I decided to accept 99.5% or so coverage and manual inspection of the failure case.

A core cultural challenge that we have to deal with is that people are very soft about testing, and they don't make their artifacts or platforms testable.

There is a bit of diminishing returns to what I believe, but it also depends on ones dependencies and what one considers 100%.

As an example, I designed a multi-TB file-format and synchronization engine. I made that engine a dependency free library such that all the math, contracts, threading, and scheduling could be 100% tested before integration. This forced the messy integration into the core application which then was E2E tested with hundreds of randomized tests.

The key concept at play was beating all the math into the ground such that it was damn near perfect because a single flaw would lose data.

Yes, I figure it would have been easy to get to 100% if MySQL had been designed so that plugins could be coverage tested.

For example, there might be an "environment" API which abstracts memory management and file I/O. The plugin could use this API instead of direct calls. Then if MySQL had an option to use a specific environment implementation for a plugin, then testers could drop in their own implementation under test.

Getting that sort of support by MySQL is an example of what I think you mean by "cultural challenge."

> Could not figure out how to trigger a malloc failure inside of a imported library

Glibc has (or at least has had) special (unsupported) instrumentation that lets you substitute out the libc malloc at runtime with another function.

You can use this to shim in a version that fails after a specific number of executions in order to inject malloc faults.

E.g. https://github.com/xiph/opus/blob/master/tests/test_opus_api...

I considered using a shim, but couldn't figure out which malloc comes from my UDF and which comes from MySQL, and I was highly doubtful that MySQL would have identical numbers of malloc by the time my code runs.

I believe you can override libc's malloc by passing in LD_PRELOAD, at which point you've got the basic building blocks to test malloc failure if you can find a way to tell your malloc call apart from the others.

Having said that, I don't disagree with your choice, because messing with LD_PRELOAD is difficult, scary, and probably overkill anyway.

Totally agree with you. This really is a concept that you need to keep in mind when you write your code "testability". And i find that when you get this you also get a Buch of other properties for "free". Orthogonality and reusability for example. And really it's a force multiplier and kicks productivity into high gear when you know that your lower level components work as expected and you don't have to go ok a wilf goose hunt when something would be constantly broken. To summarize I find that writing tests early on reduce product velocity but the compound effect of good quality and regression prevention give you higher velocity later on.

I don't think so. I once took on the challenge of writing as many unit tests as possible for a project at work (the project did not have unit tests - but was well covered with other types of tests).

The two key takeaways I got from the effort:

First, I had no idea how coupled my code was until I tried writing many unit tests for it. If it's hard for you to instantiate your class without involving N other libraries/objects, your code is very coupled. No one in my team would have looked at the code and said "It's highly coupled". The real proof was "Can you test this in isolation?" If not, you're strongly coupled.

I had to redesign a lot of bits to succeed, and as another commenter pointed out, in my attempts to do so my code really was a lot better. I discovered good principles of design in doing it.

The second thing I learned, which may appear to disagree with the first: There are always easy ways to write code to be unit testable. Most of those easy ways are bad and reduce the "quality" of your code. Forcing yourself to not redesign just "for the sake of writing tests", while still ensuring you have 100% code coverage, will really force you to think heavily about your code, architecture, failure points, etc.

So a 100% code coverage really doesn't tell you if you have good code. But less than 100% does indicate potential problems.

(I personally did not go for 100%, FYI).

Could you give an example of the second part?

It's both project and language specific. A trivial example is C++ classes with private methods. Some people do a hack that when compiling for testing converts all private methods/attributes to public. This way it's easy to test private methods individually. Please don't do this.

Unit testing C++ is not as easy as in some other languages. The language is fairly rigid. Some people make heavy use of friend classes to assist with testing, but this can be overdone (indeed, many C++ developers are against any use of friend).

I'm not. One of my side projects for example is a programming language where I have 100% code coverage.

From a career perspective, I have seeded new infrastructure services with 100% code coverage. At core, I believe achieving great reliability requires a solid foundation.

I don't claim these services are perfect, but when a bug is discovered then I can usually use the logs to figure out what state the program is in and then test the bug with a unit test. Then, I can protect future engineers from that issue.

Now, don't get me wrong, there are LOADS of silly tests for languages like Java to make code coverage tools happy. Like "new ClassThatOnlyHasStaticMethodsInIt()", but the key is that you can alarm on code coverage less than 100% but once you let the paper cuts build up it is hard to manage.

I'm a big believer in "slow is smooth, and smooth is fast" when it comes to building services for others.

What I often see is people want 100% coverage AND want tests to run fast. And often that leads to using mocks and other techniques. Once you go down that road you end up with brittle test and your developers spend a lot of time writing and repairing tests.

I'm not against testing but feel people waste a lot of time writing tests that are more a liability then an asset. I'm hoping we get better tools soon and people look back and wonder WTF were people thinking.

> " If you can't test your code easily, then your code is awful. End of story."

I don't think this is trolling. Maybe I would have phrased it in a nicer way, but I agree with their math.

Now I don't think you need to get to 100%, but if you can't easily see how you could get to 100%, that's a problem.

How easy is it test a function add(a, b) that just adds a and b and returns the result?

Or how bout testing:

  function divide(a, b, fn) {
    let [_a, _b] = fn(a, b);
    return _a / _b;
  }

I meant that's an awful function. I never, ever, ever have a single letter identifier. So I can't respond yet.

Changed the name from m to divide. It just transforms a and b through fn and divides those values.

Much better, but you still have 3 meaningless identifiers. Need to narrow your names down more before I can comment.

I explained what divide does in my previous comment. Changing the names wouldn't change how it's tested.