I've written my fair share of loops around accept() or select(); fork() or Thread.start().
I'd like to avoid doing it again. I'd prefer not to re-use my own code. And I'd like to take advantage of benefits that a 'serious' framework offers, such as preforking, thread pooling, etc.
What frameworks do you recommend - in any language - that accept socket connections and present the programmer with a stream handle to work with? There are plenty of well known ones for HTTP. I'm looking for something one protocol level lower down.
Two I'm aware of are:
Netty for Java
SocketServer for Python
Twisted for Python is another option.
It's very powerful but has quite a steep learning curve (although maybe not for someone with previous experience of this style of programming).
With Erlang it is very easy to craft HTTP / TCP servers whilst still fully supporting threading (well, in the case of Erlang, there are all processes).
Related
Can anyone explain when to use protocol buffer instead of JSON for micro-services architecture? And vice-versa? Both on synchronous and asynchronous communication.
When to use JSON
You need or want data to be human readable
Data from the service is directly consumed by a web browser
Your server side application is written in JavaScript
You aren’t prepared to tie the data model to a schema
You don’t have the bandwidth to add another tool to your arsenal
The operational burden of running a different kind of network service
is too great
Pros of ProtoBuf
Relatively smaller size
Guarantees type-safety
Prevents schema-violations
Gives you simple accessors
Fast serialization/deserialization
Backward compatibility
While we are at it, have you looked at flatbuffers?
Some of the aspects are covered here google protocol buffers vs json vs XML
Reference:
https://codeclimate.com/blog/choose-protocol-buffers/
https://codeburst.io/json-vs-protocol-buffers-vs-flatbuffers-a4247f8bda6f
I'd use JSON when the consumer is or could possibly be written in a language with built-in native support for JSON (Javascript is an example), a web browser, or where human readability is wanted. Speaking of which, at least for asynchronous calls, many developers enjoy the convenience of examining the contents of the queue directly for debugging and even during the normal course of development. Depending on the tech stack used, it may or may not be worth the trade off to use protobuf just to reduce network load since any performance increase wont buy you much in the async world. And it's not like we need to write a bunch of boiler plate code anymore like we used to with JSON marshalling and unmarshalling in most languages.
I'd use protobuf for everything else... if there are any other use cases left for it with the considerations above. There are advantages you might see, such as performance, network load, the backwards compatibility offered by its versioning scheme, the lovely documentation that magically comes with proto files, and some validation! If for some reason you have a lot of REST or other synchronous calls between microservices, protobuf can be sent over the wire instead of JSON without many trade offs, if any at all, while offering a heap of advantages.
I'm currently trying to develop (an API in multiple programming languages) that can be accessed from (various other programming languages). I've taken a look at Apache Thrift, and it appears that it might be possible to allow seamless foreign function calls between any two languages using Thrift. Is this correct?
Thrift is created to facilitate communication between different processes over the network, not in process FFI. It is probably possible to take some parts of Thrift (like IDL), and adopt it for FFI, but it could be an nontrivial undertaking, and provide suboptimal results.
I have actually been thinking of something similar myself.
There are core concepts to the Thrift specification.
The Transport: This portion is responsible for facilitating data transfer between a client and server.
The Protocol: This portion is responsible for formatting the said data in different ways. It can be a JSON, compressed binary, even raw uncompressed binary.
The Server: This is responsible for putting these things together and managing them.
Thrift allows you to mix these different parts in unique ways to create something suitable to your purpose. Thrift is still very much server-client oriented though.
To develop an API in thrift would mean that you could theoreticallly have plugins in any language. The main software component would launch the sub-process and use STD-IN/OUT as a Transport. This would allow it to make RPC calls regardless of Language.
Thrift's primary goal is to enable efficient and reliable communication across programming languages. but I think HTTP-RPC can also do that, web developer almost everyone knows how to work on http and it is easier to implement HTTP-RPC(json) than Thrift,
Maybe Thrift-RPC is faster, then who can tell me the difference in perfermance between them?
A few reasons other than speed:
Thrift generates the client and server code completely, including the data structures you are passing, so you don't have to deal with anything other than writing the handlers and invoking the client. and everything, including parameters and returns are automatically validated and parsed. so you are getting sanity checks on your data for free.
Thrift is more compact than HTTP, and can easily be extended to support things like encryption, compression, non blocking IO, etc.
Thrift can be set up to use HTTP and JSON pretty easily if you want it (say if your client is somewhere on the internet and needs to pass firewalls)
Thrift supports persistent connections and avoids the continuous TCP and HTTP handshakes that HTTP incurs.
Personally, I use thrift for internal LAN RPC and HTTP when I need connections from outside.
I hope all this makes sense to you. You can read a presentation I gave about thrift here:
http://www.slideshare.net/dvirsky/introduction-to-thrift
It has links to a few other alternatives to thrift.
Here is good resource on performance comparison of different serializers: https://github.com/eishay/jvm-serializers/wiki/
Speaking specifically of Thrift vs JSON: Thrift performance is comparable to the best JSON libraries (jackson, protostuff), and serialized size is somewhat lower.
IMO, strongest thrift advantages are convenient interoperable RPC invocations and convenient handling of binary data.
I am looking to do some work around mysql and node.js and have found a few different modules out there but I cannot get a good bead on their stability/maturity. I know each author puts very hard work into each one, but for the work we're doing I need to know I've got a solid mysql foundation. The modules I've found that look pretty good are:
db-mysql This appears pretty active.
node-mysql This is a pretty pervasive module I've seen so far, it appears to be in a maintenance phase, and seems solid.
node-mysql-native I like the async work being done here, but I'm not sure how well it works yet.
node-mysql-libmysqlclient I'm not sure about this one, but it appears to be active as well.
I don't have many needs that are too far out of the ordinary. I need regular query support, extras would be nice, I just need a good foundation to start from. Any input as to the strengths and weaknesses of these modules would be great. If there is another quality contender I have not found I am not at all against considering another option.
I'm the author of node-mysql-native driver, from my point of view the differences are
no prepared statements support (yet) in node-mysql
according to my benchmarks node-mysql is 10-20% slower than node-mysql-native
node-mysql has much wider adoption, more tests and users. If you need stability, better use it
node-mysql-libmysqlclient is 2 to 3 times faster on fast queries. However, if you have a lot of slow queries and use connection pools it could be even slower than native JS driver because libmysqlclient async calls are based on nodejs thread pool and not on event loop.
update
As of 11/07/2013
(2). no longer valid (mysql-native is a bit slower than node-mysql)
have this alternative to node-mysql, on some benchmarks it's 2-3 times faster, has same API + support for prepared statements, SSL and compression. Also implements simple subset of server side protocol - see for example MySQL -> Postgres proxy.
node-mariasql is also a very good option (if it's ok to use binary addon) - fast, stable, async, prepared statements support, compression and SSL.
I went through a similar search and ended up settling on node-mysql. I like it's simplicity, the fact that it's pure js, and that it's well supported. It was slower in tests that I did than some of the mixed modules (those that used non-js libs), but I did a minor patch that helped considerably with that for my cases:
https://github.com/geochap/node-mysql
I keep hearing that Lisp is a really productive language, and I'm enjoying SICP. Still, I'm missing something useful that would let me replace PHP for server-side database interaction in web applications.
Is there something like PHP's PDO library for Lisp or Arc or Scheme or one of the dialects?
newLisp has support for mysql5 and if you look at the mysql5 function calls, you'll see that it's close to PDO.
Since nobody has mentioned it, you can try Postmodern, which is an interface to PostgreSQL. It aims for a tighter integration with PostgreSQL and so doesn't pretend to portability between databases.
I've put it together with hunchentoot and cl-who and built a pretty nice website.
newLISP - http://www.newlisp.org/ - has support for MySQL, but I haven't used it (newLISP).
If you're happy with SQL as part of your life, CL-SQL provides a mapping into CLOS objects. It appears to be more mature than Elephant.
I'm using it on my own website.
I've had good success with SBCL and CL-SQL. CL-SQL has a object mapping API, but I used the plain SQL API which simply returns lists and this worked well enough. And in the Clojure language, you interact with JDBC through a maps or structs {:col1 "a", :col2 "b"}, so a generated class library doesn't get you any simpler code, the language handles it nicely. In my experience, there is less cruft between lisp and sql than between more static languages and sql.
our Common Lisp ORM solution is http://common-lisp.net/project/cl-perec/
the underlying SQL lib is http://common-lisp.net/project/cl-rdbms/ (fully tested with PostgreSQL, has a toy SQlite backend and a somewhat tested Oracle backend)
we started out using CLSQL, but after some struggle we decided to roll our own.
these libs and PostgreSQL are used in a clustered web application developed for the Hungarian government for planning the budget of the municipalities. it has about 4000 users, 500 at peek time. a little more info is available at http://common-lisp.net/project/cl-dwim/
Cliki is a good resource for Common Lisp libraries:
http://www.cliki.net/database
There is a project named Elephant (http://common-lisp.net/project/elephant/index.html), which is an abstraction for object persistence in CL.
As long as you're switching your Webapp on Lisp, consider using persistence: you now have a constantly running Lisp image that holds everything about your application. I personnally used Elephant for that.
Elephant can use CL-SQL or BDB as it's backend, which means that you can use MySQL if you have one running. I found using SQLite really practical, though.
We use SBCL, UCW, CL-SQL and MySQL as our back-end for Paragent.com. It has worked very well for us. We also have a number of clients using UCW/CL-SQL/MySQL for custom sites we have built them through our consulting arm Bitfauna.