This is a post I've wanted to write for awhile. This wasn't the post I had hoped it to be. The original working title was "Writing a library interface is hard. Really hard." The way the post was supposed to go was with me explaining a whole bunch of general problems with writing good programmatic interfaces for libraries and then giving some helpful advice.
Over time I realized that each library has such unique problems that general advice is almost entirely worthless. It involves balancing so many factor and in some cases the answers are obvious (a jpeg library really only has a handful of functions) or prior-art exists (GUI toolkits) that will help you. Other times, you are on your own and it becomes quite a struggle. So instead of my general advice that will save all of mankind, let me explain some of our struggles, and I'll let you figure out the lessons learned for your situation.
Oh, and as if the engineering challenges weren't enough, you needn't forget to add in these supplementary exercises that are among the most tedious in all of programming.
Documentation. It is not fun. We both know it. And yet, you absolutely must have it and if you've done it right, it'll be incredibly useful. The sad truth is that most people won't read it.
Example Programs. Virtually everyone who uses your library will skip straight by the docs and straight to the example programs. These cannot be bad. Try to make at least one short, simple, and hello-worldish. It should be easy to find.
Error reporting. Figure out a way to handle errors in a useful manner and report that information back to the developer. This is probably the second most soul crushing endeavor in all of programming.
Meetings. Did someone mentioning soul crushing? The only way to hammer out a good API is to get a bunch of people in a room and look at interfaces, functionality, paradigms, and conventions, and decide what is good and bad. As a word of warning, these meetings will not be short.
Iterate. You simply cannot get it right the first time. This shouldn't come as a surprise. Iterate early, and iterate often. Work out a prototype H file first and work from there. Get that into a room with other devs and walk through the use cases. That H file is your spec.
This is where it gets difficult. Our SDK typically has been geared towards ease of use, not power. This meant that certain sophisticated things we do with our internal codebase aren't possible to replicate using our SDK. For example, we have a live camera-fed demo that devoted some threads to video processing (face detection & tracking) and real-time display, and some other threads for psuedo-background recognition that updates the onscreen results as they come in. The complicated mix of threads and asynchronous nature of the recognition made this a non-trivial program to write. While possible, our old SDK was fundamentally ill-equipped to replicate it.
If you want to build an easy-to-use library, it's fairly straightforward. You pick a few use cases, do those right, and everyone else be damned. We decided, recently, to expose more functionality to make it capable of all the complicated and nuanced things you might want to do. This resulted in marathon meetings, lots of teeth gnashing, and at least one heated argument.
And you can take it too far. More than once, while at the whiteboard, we'd finally convinced ourselves that this particular set of functionality lets everyone do everything. At that point it became clear, however, that it's all so complicated that you'll need five times the documentation and no particular user has any hope of actually getting it right on their own. So now you've traded feature request emails for "how do I get this to work?" emails. We went back to the literal drawing board.
Eventually we realized that in any one use case, a developer using our SDK would have to create so many things and get so many of them right, that it simply wouldn't be feasible for him to get the system he wanted. At least not as well as we could. We'd thought long and hard about some of these problems and could build solutions that were simply better.
Adding all of these functions while maintaining full flexibility resulted in a dozen functions (instead of our original one). Every function you add adds to the pain. It create complexity, documentation, and work. But we trimmed down to some minimal set that best balanced all these factors and pushed forward.
So in that spirit, I'd like to nominate Qt as the most elegant and well-documented library I've ever used. Our SDK is not there, yet. All we need is about 100 iterations of progressive refinement.