I want to do this (no particular language):
print(foo.objects.bookdb.books[12].title);
or this:
book = foo.objects.bookdb.book.new();
book.title = 'RPC for Dummies';
book.save();
Where foo actually is a service connected to my program via some IPC, and to access its methods and objects, some layer actually sends and receives messages over the network.
Now, I'm not really looking for an IPC mechanism, as there are plenty to choose from. It's likely not to be XML based, but rather s. th. like Google's protocol buffers, dbus or CORBA. What I'm unsure about is how to structure the application so I can access the IPC just like I would any object.
In other words, how can I have OOP that maps transparently over process boundaries?
Not that this is a design question and I'm still working at a pretty high level of the overall architecture. So I'm pretty agnostic yet about which language this is going to be in. C#, Java and Python are all likely to get used, though.
I think the way to do what you are requesting is to have all object communication regarded as message passing. This is how object methods are handled in ruby and smalltalk, among others.
With message passing (rather than method calling) as your object communication mechanism, then operations such as calling a method that didn't exist when you wrote the code becomes sensible as the object can do something sensible with the message anyway (check for a remote procedure, return a value for a field with the same name from a database, etc, or throw a 'method not found' exception, or anything else you could think of).
It's important to note that for languages that don't use this as a default mechanism, you can do message passing anyway (every object has a 'handleMessage' method) but you won't get the syntax niceties, and you won't be able to get IDE help without some extra effort on your part to get the IDE to parse your handleMessage method to check for valid inputs.
Read up on Java's RMI -- the introductory material shows how you can have a local definition of a remote object.
The trick is to have two classes with identical method signatures. The local version of the class is a facade over some network protocol. The remote version receives requests over the network and does the actual work of the object.
You can define a pair of classes so a client can have
foo= NonLocalFoo( "http://host:port" )
foo.this= "that"
foo.save()
And the server receives set_this() and save() method requests from a client connection. The server side is (generally) non-trivial because you have a bunch of discovery and instance management issues.
You shouldn't do it! It is very important for programmers to see and feel the difference between an IPC/RPC and a local method call in the code. If you make it so, that they don't have to think about it, they won't think about it, and that will lead to very poorly performing code.
Think of:
foreach o, o.isGreen in someList {
o.makeBlue;
}
The programmer assumes that the loops takes a few nanoseconds to complete, instead it takes close to a second if someList happens to be remote.
Related
Say I have a web application with UserController. Client sends a HTTP POST request that is about to be handled by the controller. That however first must parse the provided json to UserDTO. For this reason there exist a UserDTOConverter with a method toDTO(json): User.
Given I value functional programming practices for its benefits of referential transparency and pure function the question is. What is the best approach to deal with a possibly inparsable json? First option would be to throw an exception and have it handled in global error handler. Invalid json means that something went terrible wrong (eg hacker) and this error is unrecoverable, hence the exception is on point (even assuming FP). The second option would be to return Maybe<User> instead of User. Then in the controller we can based on the return type return HTTP success response or failure response. Ultimately both approaches results in the same failure/success response, which one is preferable though?
Another example. Say I have a web application that needs to retrieve some data from remote repository UserRepository. From a UserController the repository is called getUser(userId): User. Again, what is the best way to handle the error of possible non existent user under provided id? Instead of returning User I can again return Maybe<User>. Then in controller this result can be handled by eg returning "204 No Content". Or I could throw an exception. The code stays referentially transparent as again I am letting the exception to bubble all the way up to global error handler (no try catch blocks).
Whereas in the first example I would lean more towards throwing an exception in the latter one I would prefer returning a Maybe. Exceptions result in cleaner code as the codebase is not cluttered with ubiquitous Eithers, Maybes, empty collections, etc. However, returning these kinds of data structure ensure explicitness of the calls, and imo results in better discoverability of the error.
Is there a place for exceptions in functional programming? What is the biggest pitfall of using exceptions over returning Maybes or Eithers? Does it make sense to be throwing exceptions in FP based app? If so is there a rule of thumb for that?
TL;DR
If there are Maybes/Eithers all over the codebase, you generally have a problem with I/O being mixed promiscuously with business logic. This doesn't get any better if you replace them with exceptions (or vice versa).
Mark Seemann has already given a good answer, but I'd like to address one specific bit:
Exceptions result in cleaner code as the codebase is not cluttered with ubiquitous Eithers, Maybes, empty collections, etc.
This isn't necessarily true. Either part.
Problem with Exceptions
The problem with exceptions is that they circumvent the normal control flow, which can make the code difficult to reason about. This seems so obvious as to barely be worthy of mention, until you end up with an error thrown 20 calls deep in a call stack where it isn't clear what triggered the error in the first place: even though the stack trace might point you to the exact line in the code you might have a very hard time figuring out the application state that caused the error to happen. The fact that you can be undisciplined about state transitions in an imperative/procedural program is of course the whole thing that FP is trying to fix.
Maybe, Maybe not: It might be Either one
You shouldn't have ubiquitous Maybes/Eithers all over the codebase, and for the same reason that you shouldn't be throwing exceptions willy-nilly all over the codebase: it complicates the code too much. You should have files that are entry points to the system, and those I/O-concerned files will be full of Maybes/Eithers, but they should then delegate to normal functions that either get lifted or dispatched to through some other mechanism depending on language (you don't specify the language). At the very least languages with option types almost always support first-class functions, you can always use a callback.
It's kind of like testability as a proxy for code quality: if your code is hard to test it probably has structural problems. If your codebase is full of Maybes/Eithers in every file it probably has structural problems.
You're asking about a couple of different scenarios, and I'll try to address each one.
Input
The first question pertains to converting a UserDTO (or, in general, any input) into a stronger representation (User). Such a conversion is usually self-contained (has no external dependencies) so can be implemented as a pure function. The best way to view such a function is as a parser.
Usually, parsers will return Either values (AKA Result), such as Either<Error, User>. The Either monad is, however, short-circuiting, meaning that if there's more than one problem with the input, only the first problem will be reported as an error.
When validating input, you often want to collect and return a list of all problems, so that the client can fix all problems and try again. A monad can't do that, but an applicative functor can. In general, I believe that validation is a solved problem.
Thus, you'll need to model validation as a type that isomomorphic to Either, but has different applicative functor behaviour, and no monad interface. The above links already show some examples, but here's a realistic C# example: An applicative reservation validation example in C#.
Data access
Data access is different, because you'd expect the data to already be valid. Reading from a data store can, however, 'go wrong' for two different reasons:
The data is not there
The data store is unreachable
The first issue (querying for missing data) can happen for various reasons, and it's usually appropriate to plan for that. Thus, a database query for a user should return Maybe<User>, indicating to the client that it should be ready to handle both cases: the user is there, or the user is not there.
The other issue is that the data store may sometimes be unreachable. This can be caused by a network partition, or if the database server is experiencing problems. In such cases, there's usually not much client code can do about it, so I usually don't bother explicitly modelling those scenarios. In other words, I'd let the implementation throw an exception, and the client code would typically not catch it (other than to log it).
In short, only throw exceptions that are unlikely to be handled. Use sum types for expected errors.
I have seen code like below, where exceptions are wrapped in a generic error. What i don't like about this approach is that we need to write a handler to deal with this UnexpectedError, inspect it, extract the exception and log it. Not sure if this is the correct way to do it.
override suspend fun update(
reservation: Reservation,
history: ReservationHistory
): Either<ReservationError, Reservation> {
return Either.catch {
mongoClient.startSession().use { clientSession ->
clientSession.startTransaction()
mongoClient.getDatabase(database)
.getCollection<ReservationDocument>()
.updateOneById(reservation.reservationId.value, MapToReservationDocument.invoke(reservation))
mongoClient.getDatabase(database)
.getCollection<ReservationHistoryDocument>()
.insertOne(MapToReservationHistoryDocument.invoke(history))
reservation
}
}.mapLeft {
UnexpectedError(it)
}
}
When there is some internal exception in a Webflux application, why do I want/need to write code to handle these exceptions? I understand handling issues and returning appropriate ServerResponse bodies when the service client incorrectly invokes a service, or when a non-error-condition (i.e., query returns empty cursor, etc.) occurs.
But, other than generating debug information into a logfile, is there anything to be gained by rolling-your-own exception handling components? This approach makes "more sense" to me in a monolithic application, where one is trying to avoid a scenario where the app "just dies".
But, for a service implementation, especially if there's some incentive not to expose too much about the internal implementation to a client, why wouldn't Spring's default error/exception handling (and "500 Internal Server Error" response/message) be sufficient.
So, after some time and thought (and little, but still helpful-and-appreciated feedback), I guess it boils down to:
(a) - It provides a localized context to "do things", like logging information about the exception/error condition, or categorizing the severity of the exception within-the-context of a server-client interaction.
(b) - It provides a localized context to hide/expose information from a client, based on the nature of the exception/error condition and whether the server is deployed in a production or test environment.
(c) - Being localized, it makes maintenance/modification a bit easier, as the handling of exceptions/errors is not scattered throughout the code.
(a) and (c) are enough to make me believe it's worth the effort.
I need to transfer data (objects) between client and server, and Twisted seems a good way to accomplish this. I've been doing a lot searching but still haven't found any example to understand the basic principle. So any simple code would help.
Thanks!
EDIT
Both client and server are written in python
The data may be large, so I need a fast, reliable transmission ( I've taken a look at producers, is that good?)
Flask is great, but I am using another framework, so the whole networking thing relies on Twisted.
It's hard to tell if your question is more about json, python or twisted, but here's an overview, more can follow once the specifics are known. Perhaps you could add some more info to your question so we can offer more assistance :-)
re Json: Json is just a string with a defined structure. If you are working in python and have an object to send as json, then you need to convert the object to a json string by use of
import json
json.dumps(objectName)
If your client is javascript then instead of json.dumps you might use JSON.stringify(objectname).
If you intend to use javascript for clients then some of the frameworks like jQuery make it very easy.
Pythons json.dumps has a lot of optional arguments, most of which you won't need. You can see the options at https://docs.python.org/2/library/json.html
Python is python, I assume you know how to create and populate objects. Will your client be python or javascript or something else? From a javascript client to a python server you would most likely use Ajax to send requests and get responses.
Twisted allows you to easily create a server that will listen on a given port and, when data arrives, an event will occur that supplies the data received. You can then do whatever you need to with the data. Just be careful about doing blocking things like database inserts since the server may miss some data or otherwise misbehave if you interrupt it's event loop. Twisted can be difficult to learn initially, but it is a very powerful and reliable system that is well proven. One alternative to consider, particularly if your clients are not python, is node.js. In my opinion, node is a little bit easier to grasp initially and there are thousands of add-on modules that let you do almost anything you'd want. I use both twisted and node for different things.
Neither node.js nor twisted are software that you can use to just quickly spin up a server or client without some study and experimentation. To use Twisted or Node.js properly confidently, using all their features and goodness, requires a bit of research and work on your part.
There are excellent frameworks like Flask that can be used to build a server that can react to a number of different Ajax calls from a client - you can have a single server be able to respond to several different kinds of requests instead of having a server for each Ajax type.
This is a small library that serializes an object with all its children to JSON and also parses it back to a fully working object:
https://github.com/Toubs/PyJSONSerialization/
For example, what does it mean in this quote?
Integrating with an external API is almost a guarantee in any modern web app. To effectively test such integration, you need to stub it out. A good stub should be easy to create and consistently up-to-date with actual, current API responses. In this post, we’ll outline a testing strategy using stubs for an external API.
A stub is a controllable replacement for an Existing Dependency (or collaborator)
in the system. By using a stub, you can test your code without
dealing with the dependency directly.
External Dependency - Existing Dependency:
It is an object in your system that your code
under test interacts with and over which you have no control. (Common
examples are filesystems, threads, memory, time, and so on.)
Forexample in below code:
public void Analyze(string filename)
{
if(filename.Length>8)
{
try
{
errorService.LogError("long file entered named:" + filename);
}
catch (Exception e)
{
mailService.SendEMail("admin#hotmail.com", "ErrorOnWebService", "someerror");
}
}
}
You want to test mailService.SendEMail() method, but to do that you need to simulate an Exception in your test method, so you just need to create a Fake Stub errorService object to simulate the result you want, then your test code will be able to test mailService.SendEMail() method. As you see you need to simulate a result which is from an another Dependency which is ErrorService class object (Existing Dependency object).
A stub, in this context, means a mock implementation.
That is, a simple, fake implementation that conforms to the interface and is to be used for testing.
Layman's terms, it's dummy data (or fake data, test data...etc.) that you can use to test or develop your code against until you (or the other party) is ready to present/receive real data. It's a programmer's "Lorem Ipsum".
Employee database not ready? Make up a simple one with Jane Doe, John Doe...etc.
API not ready? Make up a fake one by creating a static .json file containing fake data.
In this context, the word "stub" is used in place of "mock", but for the sake of clarity and precision, the author should have used "mock", because "mock" is a sort of stub, but for testing. To avoid further confusion, we need to define what a stub is.
In the general context, a stub is a piece of program (typically a function or an object) that encapsulates the complexity of invoking another program (usually located on another machine, VM, or process - but not always, it can also be a local object). Because the actual program to invoke is usually not located on the same memory space, invoking it requires many operations such as addressing, performing the actual remote invocation, marshalling/serializing the data/arguments to be passed (and same with the potential result), maybe even dealing with authentication/security, and so on. Note that in some contexts, stubs are also called proxies (such as dynamic proxies in Java).
A mock is a very specific and restrictive kind of stub, because a mock is a replacement of another function or object for testing. In practice we often use mocks as local programs (functions or objects) to replace a remote program in the test environment. In any case, the mock may simulate the actual behaviour of the replaced program in a restricted context.
Most famous kinds of stubs are obviously for distributed programming, when needing to invoke remote procedures (RPC) or remote objects (RMI, CORBA). Most distributed programming frameworks/libraries automate the generation of stubs so that you don't have to write them manually. Stubs can be generated from an interface definition, written with IDL for instance (but you can also use any language to define interfaces).
Typically, in RPC, RMI, CORBA, and so on, one distinguishes client-side stubs, which mostly take care of marshaling/serializing the arguments and performing the remote invocation, and server-side stubs, which mostly take care of unmarshaling/deserializing the arguments and actually execute the remote function/method. Obviously, client stubs are located on the client side, while sever stubs (often called skeletons) are located on the server side.
Writing good efficient and generic stubs becomes quite challenging when dealing with object references. Most distributed object frameworks such as RMI and CORBA deal with distributed objects references, but that's something most programmers avoid in REST environments for instance. Typically, in REST environments, JavaScript programmers make simple stub functions to encapsulate the AJAX invocations (object serialization being supported by JSON.parse and JSON.stringify). The Swagger Codegen project provides an extensive support for automatically generating REST stubs in various languages.
Stub is a function definition that has correct function name, the correct number of parameters and produces dummy result of the correct type.
It helps to write the test and serves as a kind of scaffolding to make it possible to run the examples even before the function design is complete
This phrase is almost certainly an analogy with a phase in house construction — "stubbing out" plumbing. During construction, while the walls are still open, the rough plumbing is put in. This is necessary for the construction to continue. Then, when everything around it is ready enough, one comes back and adds faucets and toilets and the actual end-product stuff. (See for example How to Install a Plumbing Stub-Out.)
When you "stub out" a function in programming, you build enough of it to work around (for testing or for writing other code). Then, you come back later and replace it with the full implementation.
You have also a very good testing frameworks to create such a stub.
One of my preferrable is Mockito There is also EasyMock and others... But Mockito is great you should read it - very elegant and powerfull package
RPC Stubs
Basically, a client-side stub is a procedure that looks to the client as if it were a callable server procedure.
A server-side stub looks to the server as if it's a calling client.
The client program thinks it is calling the server; in fact, it's calling the client stub.
The server program thinks it's called by the client; in fact, it's called by the server stub.
The stubs send messages to each other to make the RPC happen.
Source
"Stubbing-out a function means you'll write only enough to show that the function was called, leaving the details for later when you have more time."
From: SAMS Teach yourself C++, Jesse Liberty and Bradley Jones
Stub is a piece of code which converts the parameters during RPC (Remote procedure call).The parameters of RPC have to be converted because both client and server use different address space. Stub performs this conversion so that server perceive the RPC as a local function call.
A stub can be said as a fake substitute of the original function, which gives output, which is not accessible right now because of reasons:
It is not developed right now
It is not callable from the current environment (maybe testing)
A stub has :
Exact number of parameters
Exact output format (not necessarily correct output)
Why a stub is used?
When function is not accessible in an environment such as testing, or when its implementation is not available.
Example:
let's say you want to test a function in which there's a network call. While testing the code, you cannot wait for a network call's result for your test. so you write a mock output of the network call and proceed with your test.
TestFunction(){
// Some things here
// Some things here
var result = networkCall(param)
// something depending on the result
}
This networkCall gives out lets say a string, so you have to create a function with exact same parameters and it should give string output.
String fakeNetworkCall(int param){
if(param == 1) return "OK";
else return "NOT OK";
}
Now you have written a fake function, use this as replacement in your code
TestFunction(){
// Some things here
// Some things here
var result = fakeNetworkCall(param)
// something depending on the result
}
This fakeNetworkCall is a stub.
Receiving and sending data with JSON is done with simple HTTP requests. Whereas in SOAP, we need to take care of a lot of things. Parsing XML is also, sometimes, hard. Even Facebook uses JSON in Graph API. I still wonder why one should still use SOAP? Is there any reason or area where SOAP is still a better option? (Despite the data format)
Also, in simple client-server apps (like Mobile apps connected with a server), can SOAP give any advantage over JSON?
I will be very thankful if someone can enlist the major/prominent differences between JSON and SOAP considering the information I have provided(If there are any).
I found the following on advantages of SOAP:
There is one big reason everyone sticks with SOAP instead of using JSON. With every JSON setup, you're always coming up with your own data structure for each project. I don't mean how the data is encoded and passed, but how the data formatted format is defined, the data model.
SOAP has an industry-mature way of specifying that data will be in a certain format: e.g. "Cart is a collection of Products and each Product can have these attributes, etc." A well put together WSDL document really has this nailed. See W3C specification: Web Services Description Language
JSON has similar ways of specifying this data structure — a JavaScript class comes to mind as the most common way of doing this — but a JavaScript class isn't really a data structure used for this purpose in any kind of agnostic, well established, widely used way.
In short, SOAP has a way of specifying the data structure in a maturely formatted document (WSDL). JSON doesn't have a standard way of doing this.
If you are creating a client application and your server implementation is done with SOAP then you have to use SOAP in client side.
Also, see: Why use SOAP over JSON and custom data format in an “ENTERPRISE” application? [closed]
Nowadays SOAP is a complete overkill, IMHO. It was nice to use it, nice to learn it, and it is beautiful we can use JSON now.
The only difference between SOAP and REST services (no matter whether using JSON) is that SOAP WS always has it's own WSDL document that could be easily transformed into a self-descriptive documentation while within REST you have to write the documentation for yourself (at least to document the data structures). Here are my cons'&'pros for both:
REST
Pros
lightweight (in all means: no server- nor client-side extensions needed, no big chunks of XML are needed to be transfered here and there)
free choice of the data format - it's up on you to decide whether you can use plain TXT, JSON, XML, or even create you own format of data
most of the current data formats (and even if used XML) ensures that only the really required amount of data is transfered over HTTP while with SOAP for 5 bytes of data you need 1 kB of XML junk (exaggerated, ofc, but you got the point)
Cons
even there are tools that could generate the documentation from docblock comments there is need to write such comments in very descriptive way if one wants to achieve a good documentation as well
SOAP
Pros
has a WSDL that could be generated from even basic docblock comments (in many languages even without them) that works well as a documentation
even there are tools that could work with WSDL to give an enhanced try this request interface (while I do not know about any such tool for REST)
strict data structure
Cons
strict data structure
uses an XML (only!) for data transfers while each request contains a lot of junk and the response contains five times more junk of information
the need for external libraries (for client and/or server, though nowadays there are such libraries already a native part of many languages yet people always tend to use some third-party ones)
To conclude, I do not see a big reason to prefer SOAP over REST (and JSON). Both can do the same, there is a native support for JSON encoding and decoding in almost every popular web programming language and with JSON you have more freedom and the HTTP transfers are cleansed from lot of useless information junk. If I were to build any API now I would use REST with JSON.
I disagree a bit on the trend of JSON I see here. Although JSON is an order maginitude easier, I'd venture to say it's quite limited. For example, SOAP WS is not the last thing. Indeed, between soap client/server you now have enterprise services bus, authentification scheme based on crypto, user management, timestamping requests/replies, etc. For all of this, there're some huge software platforms that provide services around SOAP (well, "web services") and will inject stuff in your XML. So although JSON is probably enough for small projects and an order of magnitude easier there, I think it becomes quite limited if you have decoupled transmission control and content (ie. you develop the content stuff, the actual server, but all the transmission is managed by another team, the authentification by one more team, deployment by yet another team). I don't know if my experience at a big corp is relevant, but I'd say that JSON won't survive there. There are too many constraints on top of the basic need of data representation. So the problem is not JSON RPC itself, the problem is it misses the additional tools to manage the complexity that arises in complex applications (not to say that what you do is not complex, it's just that the software reflects the complexity of the company that produces it)
I think there is a lot of basic misinformation on this thread. SOAP, REST, XML, and JSON concepts seem to be mixed up in the responses.
Here is some clarification -
XML and JSON (an others) are encodings of information.
SOAP is a communications protocol
REST is an (Architecture) style
each is used for something different although you might use more than one of these things together.
Lets start with encoding data structures as XML vs JSON:
Everything JSON currently supports can be done in XML, but not the other way around. JSON will eventually adopt all the features that XML has, but its proponents haven't encountered all of the problems yet, once they get more experience things will be added on to close the gap. for example JSON didn't start out with Schemas and binary formats.
SOAP is a communication protocol for calling an operation. It runs on top of things like, HTTP, SMTP, etc. Aside from many other features, SOAP messages can span multiple "application" layer protocols. i.e. i can sent a SOAP message by HTTP to a service endpoint which then puts it on a message queue for another system. SOAP solves the problem of maintaining authentication, message authenticity, etc. as the requested moved between different parts of a distributed system.
JSON and other data formats canbe sent via SOAP. I work with some systems that sent binary fixed-width encoded objects via SOAP, its not a problem.
The analogy is that - if only the postman is allowed to send you a letter, then it is just HTTP, but if anyone can send you a letter, then you want SOAP. (i.e. message transport security vs message content security)
the 6 REST constraints are architectural style. Interestingly the first several years of REST the examples were in SOAP. (there is no such thing as REST or SOAP they are not opposites)
A "heavyweight bloated, etc.etc." SOA SOAP system might have monoliths with operations like GET, PUT, POST instances of a single entity. SOAP doesn't have those operations predefined, but that is typically how it is used.
Consider that if you built a "REST" service on HTTP alone with an SSL/TLS terminating proxy, then you may have violated the 4th constraint of REST.
So for your software development today, you wouldn't normally interact with any of these directly. Just as if you were written a graphics program you wouldn't directly work with HDMI vs. DisplayPort typically.
The question is do you understand architecturally what your system needs to do and configure it to use the mechanism that does that job. (for example, all the challenges of applying today's microservices to general systems are old problems previously solved by SOAP, CORBA and the old protocols)
I have spent several years writing SOAP web services (with JAX WS). They are not hard to write. And I love the idea of a single endpoint and single HTTP method (POST). For me, REST is too verbose.
But as a data container, JSON is simpler, smaller, more readable, more flexible, looks closer to programming languages.
So, I reinvented the wheel and created my own approach to writing backends for AJAX requests. In comparison:
REST:
get user: method GET https://example.com/users/{id}
update user: method POST https://example.com/users/ (JSON with User object in request body)
RPC:
get user: method GET https://example.com/getUser?id=1
update user: method POST https://example.com/updateUser (JSON with User object in the request body)
My way (the proposed name is JOH - JSON over HTTP):
get user: method POST https://example.com/ (JSON specifies both user ID and class/method responsible for handling request)
update user: method POST https://example.com/ (JSON specifies both user object and class/method responsible for handling request)