Idiomatic error handling in D - exception

Question

I tried to find resources on what is the standardised and accepted idiomatic D-way of handling errors, but I couldn't find any. If one is reading the official documentation on error handling, then one finds the following very important statements there:
Errors are not part of the normal flow of a program. Errors are exceptional, unusual, and unexpected.
Because errors are unusual, execution of error handling code is not performance critical.
The normal flow of program logic is performance critical.
I call them important, because the reasoning on using exceptions for such, well, exceptional cases is what leads the article towards the conclusion, that errors are special cases after all, and exceptions are the way to go, no matter what the costs are. Again from the same article:
Because errors are unusual, execution of error handling code is not performance critical. Exception handling stack unwinding is a relatively slow process.
In some special cases, where the exceptions may not be handled explicitly, but their presence should effect the state of things anyway, one should use the exception safe scope guards.
My main problem is, the above mentioned solutions and their examples in the documentations are indeed exceptional cases, which are pretty useful when we are hitting for example memory-related problems, but we don't want our program to fail, we want to maintain integrity and recover from those scenarios if possible, but what about other cases?
As we all know errors are not only used for exceptional cases and unexpected scenarios, but they are ways of communicating between callers and callees. For example errors can be used in a sanitiser. Let's say we want to implement a schema validation for an associative array. The type system alone is not capable of defining the constraints of the keys and values, so we create a function to check such objects during run time. So what should happen if the schema fails? Since we are interested on how it fails, the error that happened in it (that is, invalid data found) should also contain the information on what went wrong, therefore the caller would know how to act upon on it. Using exceptions is an expensive abstraction according to the author of the first article. Using the C-style function conventions, where the return values are all used for error states is the wrong way according to the same author in the same article.
So, what is the proper and idiomatic way of handling errors that are not exceptions in D?

Answer 1

Well, the TLDR version is that using exceptions is the idiomatic way to handle error conditions in D, but of course, the details get a bit more complicated than that.
Part of the question is what constitutes an error. The term error is used for many things, and as a result, talking about errors can be quite confusing. Some classes of errors are programmatic errors (and thus the result of bugs in the program), others aren't programmatic errors but are so catastrophic that the program can't continue, and others depend on stuff like user input and can often be recovered from.
For programmatic errors and catastrophic errors, D has the Error class, which derives from Throwable. Two of the commonly used subclasses of Error are AssertError and RangeError - AssertError being the result of a failed assertion, and RangeError being what you get when you try to index an array with an index that's out-of-bounds. Both of those are programmatic errors; they're the result of bugs in your program, and recovering from them makes no sense, because by definition, your program is in an invalid state at that point. An example of an error that's not a bug but which is generally catastrophic enough that your program should be terminated is MemoryError, which is thrown when new fails to allocate memory.
When an Error is thrown, there is no guarantee that any clean-up code will be run (e.g. destructors and scope statements may be skipped, because the assumption is that because your code is in an invalid state, the clean-up code could actually make things worse). The program simply unwinds the stack, prints out the Error's message and stack trace, and terminates your program. So, attempting to catch an Error and have the program continue is almost always a terrible idea, because the program is in an unknown and invalid state. If something is considered an Error, then it is the sort of condition where the error condition is considered unrecoverable, and programs should not be attempting to recover from it.
For the most part, you probably won't do anything explicit with Errors. You'll put assertions in your code to catch bugs when not compiling with -release, but you probably won't be throwing any Errors explicitly. They're mostly the result of D's runtime or assertions in code you're running catching bugs in your program.
The other class that derives from Throwable is Exception. It's used for cases where the problem is not a bug in your program but rather a problem due to user input or the environment (e.g. the XML that the user provided is invalid, or a file that your program attempts to open does not exist). Exceptions provide a way for a function to report that its input was invalid or that it's unable to complete its task due to issues outside its control. The program can then choose to catch that Exception and try to recover from it, or it can let it bubble up to the top and kill the program (though typically, it's more user-friendly to catch them and print out something more user-friendly than a message with a stack trace). Unlike Errors, Exceptions do result in all clean-up code being run. So, it's completely safe to catch them and continue executing.
However, what the program can do in response to the exception and whether it can do more than report to the user that an error occurred and terminate depends on what the exception was and what the program is doing (which is part of why some code subclasses Exception - it provides a way to report what went wrong beyond just an error message and allows the program to respond to it programmatically based on the type of thing that went wrong rather than simply responding to the fact that "something" went wrong). By using exceptions to report when something goes wrong, it allows for code to not directly handle errors unless it's the place in the code that you want to be handling errors, resulting in much cleaner code overall but with the downside that you can sometimes get exceptions being thrown that you weren't expecting if you weren't familiar enough with what could be thrown when. But that also means that errors that are reported don't get missed like they can be with error codes. If you forget to handle an exception, you'll know it when it happens, whereas with something like an error code, it's easy to forget to check it or not realize that you need to, and errors can be missed. So, while unexpected exceptions can be annoying, they help ensure that you catch problems in your program when they occur.
Now, the best time to use assertions vs exceptions can be a bit of an art. For instance, with Design by Contract, you use assertions to check the input to a function, because any code that calls that function with invalid arguments is in violation of the contract and therefore considered buggy, whereas in defensive programming, you don't assume that the input is valid, so the function always checks its input (not just when not compiling with -release), and it throws an Exception on failure. Which approach makes more sense depends on what you're doing and where the input for the function is likely to come from. But it's never appropriate to use assertions to check user input or anything that is outside of the program's control, because bad input is not a bug in the program.
However, while in general, the idiomatic way to handle error cases in D may be to throw an exception, there are times where that really doesn't make sense. For instance, if the error condition is actually extremely likely to occur, throwing an exception is an awfully expensive way to handle it. Exceptions are generally fast enough for cases that aren't happening all the time, but for something that happens frequently - especially in performance-critical code - they can be too expensive. And in such cases, doing something like an error code can make more sense - or doing something like returning a Nullable and having it be null when the function failed to get a result.
In general, exceptions make the most sense when it's reasonable to assume that the function will succeed and/or when it streamlines the code to make it so that it doesn't have to worry about the error condition.
For instance, imagine writing an XML parser that used error codes instead of exceptions. Each function in its implementation would have to be checking whether any function it called succeeded and return whether it itself succeeded, which would not only be error-prone, but it would mean that you'd essentially have error-handling code everywhere throughout the parser. On the other hand, if you use exceptions, then most of the parser doesn't have to care about errors in the XML. Instead of code that encounters invalid XML having to return an error code that the function calling it has to deal with, it can just throw an exception, and whatever code in the call chain is actually a good place to handle the error (probably the code that called the parser in the first place) is then the only code that has to deal with the error. The only error handling code in your program is then code that needs to deal with errors rather than most of your program. The code is much cleaner that way.
Another example where exceptions really clean up code would be a function like std.file.isDir. It returns whether the file name it's given corresponds to a directory and throws a FileException when something goes wrong (e.g. the file doesn't exist, or the user doesn't have permission to access it). For that to work with an error code, you'd be stuck doing something like
int isDir(string filename, ref bool result);
which means that you can't simply put it in a condition like
if(file.isDir)
{
...
}
You'd be stuck with something ugly like
bool result;
immutable error = file.isDir(result);
if(error != 0)
{
...
}
else if(result)
{
...
}
It's true that in many cases, there's a high risk of the file not existing, which would be an argument for using error codes, but std.file.exists makes it possible to easily check for that condition before calling isDir and thus ensure that isDir failing is the uncommon case - or if the code in question is written in a way that it's highly likely that the file exists (e.g. it was gotten from dirEntries), then you don't have to bother checking whether the file exists. Either way, the result is much cleaner and less error-prone than dealing with error codes.
In any case, the most appropriate solution depends on what your code is doing, and there are cases where exceptions really don't make sense, but in general, they are the idiomatic way to deal with errors that are not bugs in your program or catastrophic errors like running out of memory, and Error's are normally the best way to deal with encountering bugs in your program or catastrophic errors. Ultimately though, it is a bit of art to know when and how to use exceptions vs other techniques, and it generally takes experience to have a good feel for it, which is part of why questions about when to use exceptions, assertions, and error codes pop up from time to time.