Is CouchbaseQueue API in http://docs.couchbase.com/sdk-api/couchbase-java-client-2.5.5/com/couchbase/client/java/datastructures/collections/CouchbaseQueue.html 1 thread safe? Or can i achieve the similar behavior by Bucket.getAsync().queuePop()?
I want to access this async behavior from multiple JVM and JVM’s don’t know about each other. I do not want to go with Kafka path at this moment. Can anyone give some light as what will be the best way to achieve async queue behavior or refer me to appropriate section?
Thanks
Yes, according to the documentation on Couchbase.
Instead of maintaining in-memory storage, these implementations are backed by JSON documents stored in Couchbase Server. The implementations are thread-safe and suitable for concurrent use.
Related
Suppose you have a Schema that is used in a UI-App (e.g. Vue), a Node.js or Springboot Server and has to validate against Databases (e.g. SQL, mongoDB,...), and maybe some Micro-services running on whatever.
How and where do I manage a this JSON-Schema, so that if I have to change the schema for whatever Reason, that every architectural component can handle the new JSON-Schema(s).
Otherwise I need to update the Schema in up to 10 projects so none is incompatible.
Is it really as simple as having a git project full with just JSON-Schemas or do I need specific loaders for each language/environment?
Are there best practices that I am unaware of?
PS: I don't really think I need the automatically synchronized on runtime, so don't really think I need another Microservice to achieve that.
That being said, if a Microservice is the best way to go, then getting a Microservice it is.
If you keep them in a git project, how do you load them? Clone the project each time the app starts? It may work, but I would go with a more flexible approach that should take too much effort to be done:
Build a JSON schema repository accessible via a REST API
When the app starts, it makes a request to grab the schema (latest, or a specific version)
That way you get an uniform (and scalable) way of playing with the schemas. Even if you think about a hot-reload sometime in the future, you can leverage this approach to do that.
Here is an old project in this direction, you may give it a shot to see if it works (or for some inspiration, at least)
Nowadays, we always think like "send your data to a server, it computes it for you, then send you back the response".
But imagine something else : i want my client to compute the data itself.
The question is : is there something like a universal protocol to send actions rather than data through http ? So that the server can send the action to the client, whatever system it uses. If it does not exist, what are the technical difficulties you can face creating this kind of system ?
I'm talking about "static" actions, like mathematical functions for example.
You're unfortunately going to run into a problem pretty quick because, technically speaking, a universal language is impossible. Systems are going to have different architecture, different languages available, and different storage means. I believe what you intend (correct me if I'm wrong) is a "widespread" protocol. One way or another, you're going to have to drill down based on your personal use-case.
For a widespread example, you could keep a set of JavaScript files with functions server-side, and refer a web client to the one they need to run it by loading a javascript file during some event. Pass the location of the file and the function name, load it using the link above, then call the JavaScript function by name to run it. I could see this being an admitedly somewhat roundabout solution. This also may work in Java due to its built in JavaScript engine, although I haven't tested it.
Beyond that, I am unaware of anything particularly widespread. Most applications limit what they accept as instructions quite strictly to prevent security breaches (Imagine a SQL Injection that can run free on a client's machine). In fact, JavaScript limits itself quite severely, perhaps most notably in regards to local file reading.
Hopefully this helps with your ideas. Let me know in a comment if you have any questions/issues about what I've said.
I have the following setup (that I cannot change) and I'd like some advice from people who have been down that road. I'm not sure if this is the right place to ask, but here goes anyway.
Various JSON messages are placed on a different channels of a JMS queue (Universal Messaging/webMethods).
Before the data can be stored in relational-style DBs it has to be transformed: renamed, arrays flattened and some structures from nested objects extracted.
Data has to be appended to MySQL (as a serving layer for a visualization tool) and Hive (for long-term storage).
We're stuck on Spark 1.4.1 and may move to 1.6.0 in a few months' time. So, structured streaming is not (yet) an option.
At some point the events will be streamed directly to real-time dashboards, so having something in place that is capable of doing that now would be ideal.
Ideally coding is done in Scala (because we already have considerable batch-based repo with Spark and Scala), so the minimal requirement is JVM-based.
I've looked at Spark Streaming but it does not have a JMS adapter and as far as I can tell operating on JSON would be done using a SQLContext instance on the DStream's RDDs. I understand that it's possible to write a custom adapter, but then I'm not sure if Spark is still the best/easiest solution. I've also looked at the doc for Samza and Flink but did not find much for JMS and/or JSON, at least not natively.
Apache Camel seems like it might have a substantial set of connectors but I'm not too familiar with it, and I get the impression it does not do the streaming part, 'just' the bit where you connect to various systems. There's also Akka although I get the impression it's more of a replacement for messaging systems and JMS is set.
There is an almost bewildering amount of available tools and I'm at this point at a loss what to look at or what to look out for. What do you recommend based on your experience that I use to pick up the messages, transform, and insert into Hive and MySQL?
I was reading this article about protobuf and I wondered where to use it in the projects. I read some articles that said google created protobuf to replace XML, but as far as I know in 2008 (the first release) JSON was already there.
I searched more and I found an article that the writer suggested to use it instead of JSON, but I still don't get the idea completely.
So where shall I use it? Any special scenario, or like JSON whenever that I want to transport data? Any other scenarios?
It is useful whenever you want to serialize/deserialize your data. Typical situations include sending your data to someone else over the network, storing it to disk or keeping it in context while performing asynchronous processes.
Here is a brief explanation about the main differences between protocol buffer, json and XML: https://stackoverflow.com/a/14029040/6681872
I'm trying to implement the advice found in this great blog post for batch processing in grails with MySQL. The problem that I'm having is that inclusion of periodic calls to session.clear() in my loop causes org.hibernate.LazyInitializationException's to be thrown. There's a quote down in the comments section of the page:
You’re second point about potentially
causing LIEs is absolutely true. If
you’re doing other things outside of
importing with the current thread,
you’ll want to make sure to reattach
any objects to the session after
you’re doing your clearing.
But how do I do that? Can anyone help me specifically understand how to "reattach any objects to the session after I'm done clearing?
I'm also interested in parallelizing the database insertion process so that I can take advantage of having a multi core processor. Can anyone provide advice on how to do that in Grails?
Grails has a few methods to help with this (they leverage hibernate under the covers).
If you know an object is detached, you can use the attach method to reconnect it.
If you've made changes to the object while it was detatched, you can use merge.
If for whatever reason, you're not sure if an object is attached to the session, you can use the link text method to find out if it is or isn't.
It might also be worth reviewing the Hibernate documentation on Session.