I am experimenting with Hadoop Map-Reduce and in my tests I am able to store output of reducers to HBase. However, I want to write data a mysql database instead of HBase. Mappers would still be reading their input data from HBase. I have found this but it requires to use MySQL at both input and output while I need it at only output. Also, above link uses some deprecated classes from org.apache.hadoop.mapred package for which a new package org.apache.hadoop.mapreduce is available now, however I am not able to find any tutorial using this new package till now.
I have found this but it requires to use MySQL at both input and output while I need it at only output.
The InputFormat (DBInputFormat) is independent of the OutputFormat (DBOutputFormat). It should be possible be possible to read from HBase in the Mapper and write to a DB in the Reducer.
With the new MR API set the Job#setInputFormat and Job#setOutputFormat, with the old MR API set the JobConf#setInputFormat and JobConf#setOutputFormat appropriately to what input/output format is required. Both these formats need not be same. It should be possible to read from an XML in a mapper and write to a Queue in the Reducer also if required.
Also, above link uses some deprecated classes from org.apache.hadoop.mapred package for which a new package org.apache.hadoop.mapreduce is available now, however I am not able to find any tutorial using this new package till now.
If you are comfortable with the old API, then go ahead and use it. There is not much difference in the functionality between the new and the old API. There are two DBInputFormat for the old and the new API. Make sure you don't mix the old/new InputFormats with the old/new MR API.
Here is a tutorial on the new API.
Related
In my karate tests i need to write response id's to txt files (or any other file format such as JSON), was wondering if it has any capability to do this, I haven't seen otherwise in the documentation. In the case of no, is there a simple JavaScript function to do so?
Try the karate.write(value, filename) API but we don't encourage it. Also the file will be written only to the current "build" directory which will be target for Maven projects / stand-alone JAR.
value can be any data-type, and Karate will write the bytes (or plain-text) out. There is no built-in support for any other format.
Here is an example.
EDIT: for others coming across this answer in the future the right thing to do is:
don't write files in the first place, you never need to do this, and this question is typically asked by inexperienced folks who for some reason think that the only way to "save" a response before validation is to write it to a file. No, please don't waste your time - and please just match against the response. You can save it (or parts of it) to variables while you make other HTTP requests. And do not write your tests so that scenarios (or features) depend on other scenarios, this is a very bad practice. Also note that by default, Karate will dump all HTTP requests and responses in the log file (typically in target/karate.log) and also in the HTML report.
see if karate.write() works for you as per this answer
write a custom Java (or JS function that uses the JVM) to do what you want using Java interop
Also note that you can use karate.toCsv() to convert JSON into CSV if needed.
My justification for writing to a file is a different one. I am using karate explicitly to implement a mock. I want to expose an endpoint wherein the upstream system will send some basic data through json payload using POST/PUT method and karate will construct the subsequent payload file and stores it the specific folder, and this newly created payload file will be exposed through another GET call.
I'm working on flutter mobile application which should be attached (connected) with the web platform that I developed with laravel, I want to generate JSON file from Postgres dynamically , I mean when I update any thing in the database it will be updated in the mobile also and I need to display the data in the mobile application.
I followed this tutorial and I understood that I must convert the database or the tables into json file. How I'm going to do it please, it's the first time I'm working with Flutter and json.
https://www.youtube.com/watch?v=m7b7_Nq7XSs&list=PLK7ZDJTUghFAmRR4mueiai7zq1RJfMQ62&index=11&t=1s
If you're just getting started, please take your time and get yourself familiar with the basics and how flutter treats the data coming from a database.
Also something you should have been reading and understood is JSON and serialization.
Based on that it is not advisable that you retrieve JSON right from the database. Instead, JSON serialization happens in one way or another inside flutter using one of the recommended approaches.
Specifically for working with PostgreSQL, there seems to be a decent tutorial.
Please keep in mind that what you actually have asked for here ("... database to JSON file") indicates you really want a file output, which is completely contrary to the API that you're going to provide with flutter.
Of course it is possible to query PostgreSQL and get the result already in JSON format, but that then also means you won't be able to work with the data model inside flutter.
However, if you finally know what you are doing, here is a way to get the result of any PostgreSQL query directly as JSON:
SELECT json_agg(t) FROM (
SELECT ...whatever you can think of...
) AS t;
If you are using laravel version greater than 5, you can use API resources to create API and connection to PostgreSQL(https://laravel.com/docs/5.7/database). It is so simple to create API using laravel API resources. Then in Flutter only thing you have to do is request endpoints you have created using laravel.
I'd like to write an object to gcp object store, while using the x-goog-if-generation-match feature. Using #google-cloud/storage npm library, the file object does not seem to have an option for setting the required object generation.
What are the alternatives?
As you noticed, the #google-cloud/storage npm library doesn't support generation and metageneration preconditions.
As an alternative, you may use either the Storage XML API or the Storage JSON API which do support it. Depending on if you want to use one or the other, you'll be able to use preconditions via HTTP Headers or query string parameters. You'll find the whole list of those here.
Another alternative is to use some kind of optimistic locking:
get the generation id
write object
get the generation id again
repeat until generation after = generation before + 1
What is the difference between Couchbase java-client and couchbase-client?
I can see bulk get operation in java-client but not in couchbase-client
Is it possible to do bulk get operation in if we use couchbase-client?
For couchbase-client you refer to REST API Couchbase?
if this its true SDK java client create connection with database directly and there are many functions to get, create and update documents
you can map document in java class and even if you want put your Example.java class directly in document in spring there are documentarion for implement this.
In other way with API you can send n1ql query the response contain the document in json format.
Depends on the implementation you want.
I recomended that you will use SDK JAVA
Good luck.
My program stack is ReactiveMongo 0.11.0, Scala 2.11.6, Play 2.4.2.
I'm adding PATCH functionality support to my Controllers. I want it to be type safe, so that PATCH would not mess the data in Mongo.
Current dirty solution of doing this, is
Reading object from Mongo first,
Performing JsObject.deepMerge with provided patch,
Checking that value can still be deserialized to target type.
Serializing merged object back to JsObject, and check, that patch contains only fields that are present in merged Json (So that there is no trash added to the stored object)
Call actual $set on mongo
This is obviously not perfect, but works fine. I would write macros to generate appropriate format generalization, but it might take too much time, which I currently lack of.
Is there a way to use Playframework Json macro generated format for partial entity validation like this?
Or any other solution, that can be easily integrated in Playframework for that matters.
With the help of #julien-richard-foy made a small library, to do exactly what I wanted.
https://github.com/clemble/scala-validator
Need to add some documentation, and I'll publish it to repository.