I am looking for DA’s recommendation/best practices regarding writing and deploying DAML code and object (.daml and .dar) in production grade solutions.
Let us take a scenario - Central Authority (CA) operating node may issue a new role under a contract to Participant 1 (P1) by writing a simple DAML code, below are few questions related to the DAML deployment –
a. Assuming DAML code will be written by CA, can we say that only CA needs to have this code and its build on its node and CA will simply execute the contract workflow allowing Party on P1 Node to simply accept/reject the role without having to know the content of the DAML code (business logic and other contract templates) written by CA ?
b. Will the DAML code file (.daml) written by CA Node needs to be transmitted to Participant 1 (P1) node so that P1 can verify and agree with the DAML file code (contract templates, parties and choices) and put the code and its build (.dar) into its node as well?
c. If the answer to above question is yes, how will the integrity of the DAML code be maintained e.g. what if DAML code is changed by P1 or CA at the time of deployment, which may cause conflict later?
The contract model, in the form of a dar file has to be supplied to all nodes that participate in the workflows modeled in that dar file.
A dar file can contain multiple DAML "packages" and each package is identified by its name and a hash.
On ledger, contract types (called template) are fully qualified, including package hash. If you change your templates, the package hash changes and thus the new templates are seen as completely different from the old ones by the ledger.
To change an existing contract model, you have to upgrade existing contracts using a DAML workflow. Of course, all signatories of the existing contracts need to agree to the upgrade workflow. You can only unilaterally upgrade data that you are in full control of. In the crypto currency world, you can consider all miners as signatories. Either they all agree on an upgrade or they hard fork, leading to two slightly different models of the same currency.
This sort of model upgrade process in DAML is described in detail here: https://github.com/digital-asset/ex-upgrade
I am using IBM webSphere MQ 7.5 server as queue manager for my applications.
Already I am receiving data through single queue.
On the other hand there are 3 applications that want to process data.
I have 3 solutions to duplicate/distribute data between them.
Use broker to duplicate 1 to 3 queue - I don't have broker so it is not accessible for me.
Write an application to get from queue and put them in other 3 queues on same machine
Define publish/subscribe definitions to publish input queue to 3 queues on same machine.
I want to know which methods (2 & 3) is preferred and have higher performance and acceptable operational management effort.
Based on the description I would say that going PubSub would achieve the goal; try to thinking in pure PubSub terms rather than thinking about the queues. i.e. You have an application that publishes to a topic with then 3 applications each having their own subscription to get copies of the message.
You then have the flexibility to define durable/nondurable subscriptons for example.
For option # 2, there are (at least) 2 solutions available:
There is an open source application called MMX (Message Multiplexer). It will do exactly what you describe. The only issue is that you will need to manage the application. i.e. If you stop the queue manager then the application will need to be manually restarted.
There is a commercial solution called MQ Message Replication. It is an API Exit that runs within the queue manager and does exactly what you want. Note: There is nothing external to manage as it runs within the queue manager.
I think there is another solution, with MQ only, to define a Namelist which will mirror queue1 to queue2 and queue3
Should be defined like: Source, Destination, QueueManager.
Hope it is useful.
Biruk.
I am trying to find the most light-weight message bus (queue?) that can handle the following:
Producer A subscribes to the bus. The bus is specified via a well known form of identification (like a name, a socket or something).
Consumer B subscribes to the same bus and registers to only a certain kind of messages.
Consumer C subscribes to the same bus and registers to another kind of messages that overlaps with that of B.
Producer A puts a message in the bus such that both B and C are interested in. Both B and C receive the message (not just one of them, but both of them).
A, B, C and the bus are residing in different machines.
You are probably best using a queue (such as ActiveMQ) for this and for the lightweight requirement, you can use MQTT as the underlying protocol which is extremely lightweight.
See:
http://activemq.apache.org/mqtt.html
http://mqtt.org/
For your scenario:
Your producer A will connect to the queue
Your consumer B will connect to the same queue and listens to a topic (for instance /topic/onlyBisInterested). It will also subscribe to /topic/everyoneIsInterested
Your consumer C will connect to the queue and listens to /topic/everyoneIsInterested
Your producer will publish a message to /topic/everyoneIsInterested and both B and C will receive this message.
You can play around with the topics a lot. You can have topic matching rules on a wildcard basis, on an exact basis or any combination thereof. There are also levels of hierarchy (such as /a/b/*) that can be setup which will accept (/a/b/c or /a/b/d or anything else).
You can also transfer your data over SSL and use authentication if required and use message persistence and guaranteed delivery in case one/many of your node(s) go down.
I recommend reading the official documentation and deciding if it's good enough for you. There are plenty of other free or commercial queues available, most/all of which will satisfy your requirements since they are very standard.
We've written a peer-to-peer message bus, Zebus, based on ZeroMq (transport), Cassandra (peer discovery and persistence) and Protobuf (serialisation). It supports routing of messages based on properties of the messages.
It is open source and production tested https://github.com/Abc-Arbitrage/Zebus
Zebus is being actively developed and it is in heavy in-house production use. Currently there is only a .NET language binding.
How can we distribute work via RabbitMQ such that worker pools can subscribe to work messages based on differing (but frequently overlapping) criteria from each other but such that when a message is routed that matches to multiple worker pools, only one worker will pick up the job?
Simplified example:
We have host1 and host2.
Host1 handles jobs of classA and classB; host2 handles jobs of classB and classC.
If we route a job of
classA, only host1 will pick it up; if we route a job of classB,
either host1 or host2 will pick it up (based on their current load /
first available) but never both.
It would seem that we need to use a topic exchange, as our routing criteria is complex and using wildcards gives us the type of flexible matching we want.
However:
If we use the same name for the worker pool queue (say “worker-jobs”) we get the desired work splitting out to arbitrary matching workers, but every worker subscribing to the named queue seems to infect the other workers with each other’s routing criteria as they bind it. I.e. the binding of the routing key seems to be at the central queue name level not on a connection-to-queue basis.
If we use different queue names for each worker pool connection (say “poolA-jobs” and “poolB-jobs”) to the same exchange then we get the desired behavior with the different routing criteria maintained between pools but a job coming in that can match to both poolA and poolB gets routed to both of them (albeit only to one worker in each).
Notes:
I’ve spared you the details of why but suffice to say we have an existing multi-petabyte distributed search application that needs response times < 50ms. We already achieve this with our own custom routing hub but we’d like to replace this with RabbitMQ as its performance is attractive (as is retiring homemade code that overlaps with general purpose community projects) if we can get the sophisticated routing we need.
We use Python
Disco isn’t viable for many reasons, too numerous to go into.
It doesn’t have to be RabbitMQ but the performance needs to be as good. ØMQ looks very interesting and like it might provide both the flexibility and the performance but we’re already using RabbitMQ and after wading through the first half of the colorfully written ØMQ guide I’m still not sure if it will support the routing we need but it does look like we’ll have to pretty much write a broker to do it.
We actually have the luxury of knowing which hosts are capable of serving which jobs, so we can do something like have host1 subscribe to #.host1.# and host2 to #.host2.#. Then when we route a classB message, we can give it a key of host1.host2 to indicate which backends are acceptable for service. This simplifies the routing rules but still doesn’t overcome the problem described.
Some of our partners are telling us that our software needs to interact with an Enterprise Service Bus. After researching this a bit, my instinct is to say that this is just buzz speak for saying that we need to have a platform-indpendent way to pass messages back and forth. I'm just trying to get a feel for what our partners are telling us. Am I correct in dismissing our partners' request as just trying to get our software to be more buzzword-compliant, or are they telling us something we should listen to (even if encoded in buzzspeak)?
Although ESB is based on messaging, it is not "just" messaging and not just a buzzword.
So if you start with plain old async messaging, the early networks tended to be very point-to-point. You had to wire up (i.e. configure through some admin interface) each connection and each pair of destinations and if you dared to move anything around invariably something broke. Because the connection points were wired by hand these networks never achieved high connection density. The incremental cost was too high and did not scale. There was also a lot of access control and policy embedded in the topology. The lack of connection density actually favors this approach to security, even though it inhibits flexibility.
The ESB attempts to address these issues with...
Run-time resolution of destinations/services/resources
Location transparency
Any-to-any connectivity and maximum connection density
Architected for redundancy, horizontal scalability, failover
Policy, access control, rules externalized from topology
Logical messaging network layer implemented atop the physical messaging network layer
Common namespace
So when your customer asks for ESB compatibility, they want things like the above. From an application standpoint, this also implies...
Avoiding message affinities such as requirements to process in strict sequence or to address requests only to specific nodes instead of to a generic network destination
Ability to resolve destinations dynamically at run time (i.e. add another instance of a queue and it automatically starts getting traffic, delete one and traffic routes to the remaining nodes)
Requestor and provider apps decoupled from knowing where each other "lives". Requestor makes one connection, regardless of how many services it might need to call
Authorize by policy rather than by topology
Service provider apps able to recognize and handle dupes (as per JMS spec, see "functional duplicate" due to session handling)
Ability to run multiple active instances of a service provider application
Instrument the service provider applications so you can inquire on the status of the network or perform a test without sending an actual transaction
On the other hand, if your client cannot articulate these things then they may just want to be able to check a box that says "works with the ESB."
I'll try & keep it buzzword free (but a buzz acronym may creep in).
When services/applications/mainframes/etc... want to integrate (so send messages to each other) you can end up with quite a mess. An ESB hides that mess inside of itself (or itselves) so that an organisation can pretend that there isn't a mess and that it has something manageable. It then wraps a whole load of features around this to make this box even more enticing to the senior people in an organisation who'll make the decision to buy such an expensive product. These people typically will want to introduce a large initiative which costs a lot of money to prove that they are 'doing something' and know how to spend large amounts of money. If this is an SOA initiative then vendors various will have told them that an ESB is required to make the vendors vision of what SOA is work (typically once the number of services which they might want passes a trivial number).
So an ESB is:
A vehicle for vendors to make lots of money;
A vehicle for consultants to make lots of money;
A way for senior executives (IT Directors & the like) to show they can spend lots of money;
A box to hide a mess in;
A total PITA for a technical team to work with.
After researching this a bit, my
instinct is to say that this is just
buzz speak for saying that we need to
have a platform-indpendent way to pass
messages back and forth
You are correct, partially because the term ESB is always nice word that fits well with another buzzword, legitimate or not - which is governance (i.e. helps you manage who is accessing your endpoints and reporting metrics - Metrics btw is what all the suits like to see, so that may be a contributor)
Another reason they might want a platform neutral device is so that any services they consume are always exposed as endpoints from a central location, instead of a specific machine resource. The ESB makes the actual physical endpoints of your services irrelevant to them, which they shouldn't care much about anyway, but that enables you to move services around however they will only consume the ESB Endpoint.
Apart from a centralized repository for Discovery, an ESB also makes side by side versioning of services easier. If I had a choice and my company had the budget, we would have purchased IBM's x150 appliance :(
Thirdly, a lot of more advanced buses, like SoftwareAG's product if I recall, is natively able to expose legacy data, like from data sitting on main frames as services without the need for coding via adapters
I don't know if their intent is to leverage all the benefits an ESB provides, or as you said, make it buzzword compliant.
After researching this a bit, my instinct is to say that this is just buzz speak for saying that we need to have a platform-indpendent way to pass messages back and forth
That's about right. Sometimes an ESB will go a little bit further and include additional features like message delivery guarantees, confirmation/acknowledgement messages, and so on. The presence of an ESB also usually explicitly or implicitly creates a new protocol where none previously existed, which is another important consideration. (That is, some sort of standard or interface has to be set regarding the format of the messages.)
Am I correct in dismissing our partners' request as just trying to get our software to be more buzzword-compliant, or are they telling us something we should listen to (even if encoded in buzzspeak)?
You should always listen to your customers, even if it initially sounds silly. It's usually worth at least spending the effort to decide what's going on. Reading between the lines, what your partners probably mean is that they want a way for your service to integrate more easily with their own services and products.
An enterprise service bus handles the messaging between systems in a standard way. This allows you to communicate with the bus in the same exact way across all your platforms and the bus handles the actual translating to individual communication mechanism needed for the specific endpoint. This means you write all your code to talk to the bus using a common messaging scheme and the bus handles taking your common scheme and translating it so the endpoint understands it.
The simplest explanation is to explain what it provides:
For many years companies acquired different platforms and technologies to achieve specific functions in their business from Finance to HR. These systems needed to talk to each other to share data so middleware became the glue that allowed them to connect. Before the business knew it, they were paying for support and maint on each of these systems and the middleware. As needs in the business changed departments decided to create their own custom solutions to address special needs rather than try to make the aging solutions flexible enough to meet their needs. Before they knew it, they were paying to support and maintain the legacy systems, middleware, and custom solutions. With new laws like Sarbanes Oxley, companies need to have better information available for reporting purposes. A single view requires that they capture data from all of the systems. In addition, CIOs are now being pressured to lower costs and increase customer service. One obvious solution is the eliminate redudant systems, expensive support and maint contracts, and high cost legacy solutions which require specialists to support. Moving to a new platform allows for this, but there needs to be a transition. There are no turnkey solutions that can replicate what the business does. To address the needs for moving information around they go with SOA because it allows for information access through a generic entity. If I ask for AllEmployees from the service bus it gets them whether it is from 15 HR systems or 1. When the 15 HR systems becomes 1 system the call and result does not change, just how it was done behind the scenes. The Service Bus concept standardizes the flow of information and allows IT managers to conduct transitions behind the bus with no long term effect on upstream users.