Recently I am working on Kafka Producer API to produce compressed messages on wire. In Apache kafka wiki document https://cwiki.apache.org/confluence/display/KAFKA/Compression, it is mentioned that we have to use two producer properties as;
compression.codec
compressed.topics
Problem:
After reading the above I straight went to send compressed messages from kafka-producer-API. After a lot of tries I was unable to send a compressed message. When I dig the producer API in depth, I came to know that the API is using ProducerConfig.java class. In this class I found the following compression property;
public static final String COMPRESSION_TYPE_CONFIG = "compression.type";
It is very obvious that this property is compression.type not compression.codec.
Secondly I was unable to find second property "compressed.topics" that was mentioned in document. So I am unable to specify compression topics.
What is the wrong, API or document and how I can specify list of topics for compression using API.
Related
I'm trying to do few quick prototypes of using RabbitMQ as message broker for internal services as well as messages from external clients received by gateway over websocket connection.
I decided it would be best (and probabaly only) option for client to publish messages as json, and then for gateway to simply send the unaltered json messages forward.
I've seen that RawRabbit have the ability to take raw Json as message and then deserialize it to C# class.
What I can't find is some example and/or documentation of how the process should look like. Also cannot find documentation of how the Json message should be formatted.
I have to implement REST endpoints to update a resource.
I will use the methods PUT and PATCH (the latter used to send a json that has only the attributes to modify).
The payload of the calls will be a json that is parsed by Jackson.
Using custom deserializers and converter,s the parser will create the correct instances of java beans.
I know that usually these endpoints will have in their URL the ID of the resource to update.
For the PATCH endpoint, I would prefer to send the ID of the resource in the json, together with all the other resources.
I can see PROS and CONS.
CONS: the URL looks like something that is updating the collection of resources and not a single resource,
PROS: the json contains all the necessary information and the parser can find the resource in the database and add the information that wasn't sent with the request.
This would simplify the code because the parser will return an object ready to use.
As you pointed out, it would be better to pass the ID in the URL.
One of the main constraint of a REST API is the "Resource identification in requests": it should be possible to easily identify the resource modified looking the call.
It is not enough that the representation passed has the information necessary to identify the resource: it is also important that the identification is easy and does not require the knowlodege of internal details.
I am trying to write a JSON exporter in GoLang using client_golang
I could not find any useful example for this. I have a service ABC that produces JSON output over HTTP. I want to use the client-golang to export this metric to prometheus.
Take a look at the godoc for the Go client, it is very detailed and contains plenty of examples. The one for the Collector interface is probably the most relevant here:
https://godoc.org/github.com/prometheus/client_golang/prometheus#example-Collector
Essentially, you would implement the Collector interface, which contains two methods: describe and collect.
describe simply sends descriptions for the possible Metrics of your Collector over the given channel. This includes their name, possible label values and help string.
collect creates actual metrics that match the descriptions from describe and populates them with data. So in your case, it would GET the JSON from your service, unmarshal it, and write values to the relevant metrics.
In your main function, you then have to register your collector, and start the HTTP server, like this:
prometheus.MustRegister(NewCustomCollector())
http.Handle("/metrics", promhttp.Handler())
log.Fatal(http.ListenAndServe(":8080", nil))
You mean you want to write an exporter for your own service using golang? The exporters listed on prometheus exporter page are all good examples, many of which are written in golang, you could pick a simple one like redis exporter to see how it's implemented.
Basically what you need to do is:
Define your own Exporter type
Implement interface prometheus.Collector, you can poll the json data from your service and build metrics based on it
Register your own Exporter to prometheus by prometheus.MustRegister
Start a HTTP server and expose metrics endpoint for prometheus to poll metrics
I've started using Apache Nifi and I'm still learning it and experimenting with it. I really want to use Nifi to get JSON documents from API's and put them in my Elasticsearch database. So far using the built-in getTwitter and putElasticsearch controllers this works.
However now I want to do this with other APIs than Twitter, and I'm kinda stuck here. First off I really don't even know which controller to use? I would think getHttp or invokeHttp even with 'GET' as http verb then but it doesn't seem to work. If I use the getHttp I have to give an SSL service with keystore and truststore .. like why would I have to do that?
Apache Nifi is still quite new so hard to find decent guides / information about these kinds of things. I have read and searched the documentation but haven't gotten the wiser.
An example JSON to pick up from an API is:
https://api.ssllabs.com/api/v2/getEndpointData?host=www.bnpparibasfortis.be&s=193.58.4.82
Thanks in advance for anyone that can offer some help / insight.
What processor you use to get the JSON data is entirely dependent on the API you want to hit. The GetHttp or InvokeHttp processors should work to grab the data from a URL. If you'll notice, the SSL service is an optional property for both GetHttp and InvokeHttp so you only need to you use it when you want to communicate via HTTPS. Also, from the UI you can right click on a processor and then click "usage" to bring up the documentation for that processor.
At this link[1] you can find a NiFi template that uses GetHttp to get JSON data from randomuser.me and does various processing on it. It's primarily a template to show-case the different Avro processors but the method of grabbing the JSON should be relevant.
[1] https://github.com/hortonworks-gallery/nifi-templates/blob/master/templates/Convert_To_Avro_From_CSV_and_JSON.xml
I'm writing a RESTful web application where I need to provide the service of uploading images for a user. Currently, I have been able to upload an image from my current machine but I need to send it as JSON data over the web through the REST protocol.
In the server, there is a Java application running Jax-RS to manage the RESTful service. I was planning to save the JSON data that contains the image in the server and then provide a URL to the user for him to be able to locate it's image on the server.
Can someone provide some ideas on how can I do this?
If you want to send the image in a JSON object, then the image should be Base64 encoded it, or some other form of encoding. Then on the server side you will need to unmarshal the JSON and then decode back the image. You can get some ideas here on how that can be done.
Optionally, instead of doing all the converting inside the resource method (as in the example linked above), you could write a custom MessageBodyReader, where you can do the unmarshalling and decoding there.
If you decide you don't want to work with JSON, you can go the normal route and use Multipart. Depending on the implementation of JAX-RS you are using, multipart support will be different. You can see some examples (all examples have links to the official documentation)
Jersey example
Resteasy example
CXF example
There are other implementations, but I don't have examples for those. You will need to search for the documentation if you're using an implementation other than listed above.