I need to create a view with a map function ex:
function(doc, meta) {
if(doc.docType == "testDoc")
emit(meta.id, doc)
}
I have to create this view using couchbase java client 2.1.I could not find any thing in the documentation thanks
Here is some code to create a design document:
List<View> viewsForCurrentDesignDocument = new ArrayList<View>();
DesignDocument designDocument = DesignDocument.create("my_design_doc", viewsForCurrentDesignDocument);
And add a view:
String mapFunction =
"function (doc, meta) {\n" +
" if(doc.docType == \"testDoc\") {\n" +
" emit(meta.id, doc);\n" +
" }\n" +
"}";
String reduceFunction = "..." // or null if no reduce
View v = DefaultView.create("my_view", mapFunction, reduceFunction);
viewsForCurrentDesignDocument.add(v);
bucket.bucketManager().insertDesignDocument(designDocument);
You can check the API reference for more options(development mode, timeout, ...).
Related
Please help me here,
I have a spark dataframe which does API call and returns the output in JSON format which is then parsed/exploded for further processing. Let's say I have 1000 records in a dataframe and invoke the API call like in below code, then all 1000 records will try to hit the API and get the response or would that hit be sequential from spark side ?
What are the possible ways to improve the API call step and get the response much faster. Right now, API call step is taking a lot of time. Please suggest any improvements.
API call UDF -->
val apiCall = udf((url : String, DPID : String,KAFKA_DTS : String) => {
val client = IgnoreSSLOkHttpClient()
val headerBuilder = new Headers.Builder
val headers = headerBuilder.add("content-type", "application/json").add("Authorization", authStr)build()
try {
val request = new Request.Builder().url(url).headers(headers).build()
val response = client.newCall(request).execute()
if (response.code() == 200) {
response.body().string()
}
else {
"API_Call" + "#~#" + DPID + "|~|" + response.code() + "|~|" + response.body().string() + "|~|" + KAFKA_DTS + "#~#error:record has response code <> 200" + "#~#" + KAFKA_DTS + "#~#" + "Y"
}
}
catch {
case e: Exception =>
e.printStackTrace()
val sw = new StringWriter()
e.printStackTrace(new PrintWriter(sw))
print(e.toString)
logger.info("exception: " + e.toString)
"API_Call" + "#~#" + DPID + "|~|" + KAFKA_DTS + "#~#error:record did not complete API call" + "#~#" + KAFKA_DTS + "#~#" + "Y"
}
}
)
val Json = urls2
.withColumn("purchaseJson", apiCall(urls3("Url"),urls3("ID"),urls3("DTS")))
.filter("purchaseJson is not null ") .select("purchaseJson","ID","SRC_ENV_NM","CD","DTS","KAFKA_DTS")
I am using java to extract a list of node ids from an elastic search running tasks response.
The response looks like this
{
"nodes": {
"ZKUuxQZpRCCcJ0njBM1P0A": {
"name": "ZKUuxQZ",
"transport_address": "127.0.0.1:9300",
"host": "127.0.0.1",
"ip": "127.0.0.1:9300",
"roles": [
"master",
"data",
"ingest"
],
"tasks": {
"ZKUuxQZpRCCcJ0njBM1P0A:118": {
"node": "ZKUuxQZpRCCcJ0njBM1P0A",
"id": 118,
"type": "transport",
"action": "indices:data/write/delete/byquery",
"start_time_in_millis": 1527808643421,
"running_time_in_nanos": 154234724059,
"cancellable": true,
"headers": {}
}
}
}
}
}
In this example, I want to exact ZKUuxQZpRCCcJ0njBM1P0A:118. Can someone give me an example how to extract this information?
Option 1, java json parser. I wanted to write a response class and parse the json string, but ZKUuxQZpRCCcJ0njBM1P0A:118 is not in the key:value format. I am not sure how to extract it.
Option 2, using regex. The node id part may also have - or _. Can someone provide a neat java regex solution? So far my regex is [a-zA-Z0-9\-\_]+\:[0-9]+, I am not sure if this is safe enough.
Thanks.
One way you can do this is by using org.json library and create JSONObject and using keys() you can get all the keys and use this to achieve your task
Add the following dependency in your pom.xml
<dependency>
<groupId>org.json</groupId>
<artifactId>json</artifactId>
<version>20180130</version>
</dependency>
Then you can do something like below:
String jsonString = "{\n" +
" \"nodes\": {\n" +
" \"ZKUuxQZpRCCcJ0njBM1P0A\": {\n" +
" \"name\": \"ZKUuxQZ\",\n" +
" \"transport_address\": \"127.0.0.1:9300\",\n" +
" \"host\": \"127.0.0.1\",\n" +
" \"ip\": \"127.0.0.1:9300\",\n" +
" \"roles\": [\n" +
" \"master\",\n" +
" \"data\",\n" +
" \"ingest\"\n" +
" ],\n" +
" \"tasks\": {\n" +
" \"ZKUuxQZpRCCcJ0njBM1P0A:118\": {\n" +
" \"node\": \"ZKUuxQZpRCCcJ0njBM1P0A\",\n" +
" \"id\": 118,\n" +
" \"type\": \"transport\",\n" +
" \"action\": \"indices:data/write/delete/byquery\",\n" +
" \"start_time_in_millis\": 1527808643421,\n" +
" \"running_time_in_nanos\": 154234724059,\n" +
" \"cancellable\": true,\n" +
" \"headers\": {}\n" +
" }\n" +
" }\n" +
" }\n" +
" }\n" +
"}";
JSONObject jsonObject = new JSONObject(jsonString);
Set<String> topLevelKeys = jsonObject.keySet();
for (String key : topLevelKeys) {
Object value = jsonObject.get(key);
if (value instanceof JSONObject) {
JSONObject valueObject = jsonObject.getJSONObject(key);
System.out.println(valueObject.toString());
}
}
Extend this as per your requirement.
So regex is too hacky and I figured it out using gson. I wish ElasticSearch can offer us some standard library to extract all kinds of responses. Here is my solution using gson.
import com.google.gson.*;
import org.json.JSONObject;
public class test {
public static void main(String[] args) throws Exception {
String jsonString = "json_format elasticsearch reponse for listing running tasks"
JsonParser parser = new JsonParser();
JsonObject jsonObject = parser.parse(content).getAsJsonObject();
jsonObject.getAsJsonObject("nodes").entrySet().forEach(
s -> {
s.getValue().getAsJsonObject().getAsJsonObject("tasks").entrySet().forEach( s2 -> {
System.out.println(s2.getKey());
});
});
}
It prints all the running task ids like the following. It's technically nodeId:taskId, but ES has a very vague description in their task API docs(It says you can use TASK_ID to check a status of a task, well that TASK_ID is nodeId:taskId).
Mw-3i39gTHGxu5c8z9viQQ:503209021
DZ29LMsWR0aW9guWZTYe2Q:482931604
6CAbDZSWR8SfwZgnRT0qNg:494351185
Use the following regex:
[a-zA-Z0-9-_]*[0-9]+[a-zA-Z]+:[0-9]*
You can find the test result here:
https://regexr.com/3qdug
I am writing a MySQL INSERT ... ON DUPLICATE KEY UPDATE implementation via a raw SQL command in EF Core 2.0. I am very close to a working solution, but the only problem that I have is determining which PropertyInfos read via reflection are primary keys. In the CreateUpdates() method below, how can I filter primary keys out from columnProperties so they are not a part of the update SQL statement?
I have tried using EntityFramework.PrimaryKey
, but I cannot seem to get it to work with generics (TEntity).
I have included all of my relevant code, but the piece I am focused on in this question is the TODO in the last method, CreateUpdates().
private static void InsertOnDuplicateKeyUpdate<TEntity>(DbContext dbContext) where TEntity : class
{
var columnProperties = GetColumnPropertiesLessBaseEntityTimestamps<TEntity>();
var tableName = GetTableName<TEntity>();
var columns = string.Join(", ", columnProperties.Select(x => x.Name));
var values = CreateValues<TEntity>(columnProperties);
var updates = CreateUpdates(columnProperties);
var rawSqlString = "INSERT INTO " + tableName + " (" + columns + ") VALUES " + values +
" ON DUPLICATE KEY UPDATE " + updates;
dbContext.Set<TEntity>().FromSql(rawSqlString);
dbContext.SaveChanges();
}
private static string GetTableName<TEntity>()
{
return typeof(TEntity).Name.Pluralize().ToLower();
}
private static List<PropertyInfo> GetColumnPropertiesLessBaseEntityTimestamps<TEntity>()
{
return typeof(TEntity).GetProperties().Where(x =>
x.PropertyType.Namespace != "System.Collections.Generic" &&
!new List<string> {"CreatedDateUtc", "ModifiedDateUtc"}.Contains(x.Name)).ToList();
}
private static string CreateValues<TEntity>(IReadOnlyCollection<PropertyInfo> columnProperties)
{
return GetSeedRows<TEntity>().Select(row => CreateRowValues(columnProperties, row)).Aggregate("",
(current, rowValues) => current == "" ? rowValues : current + ", " + rowValues);
}
private static string CreateRowValues<TEntity>(IEnumerable<PropertyInfo> columnProperties, TEntity row)
{
return (from property in columnProperties
let value = row.GetType().GetProperty(property.Name).GetValue(row)
select WrapStringPropertyValueInSingleQuotes(property, value)).Aggregate("",
(current, value) => current == "" ? "(" + value : current + ", " + value) + ")";
}
private static object WrapStringPropertyValueInSingleQuotes(PropertyInfo property, object value)
{
if (property.PropertyType == typeof(string))
value = "'" + value + "'";
return value;
}
private static string CreateUpdates(IEnumerable<PropertyInfo> columnProperties)
{
//TODO: filter out primary keys from columnProperties
return columnProperties.Select(property => property.Name).Aggregate("", (current, column) => current == ""
? column + " = VALUES(" + column + ")"
: current + ", " + column + " = VALUES(" + column + ")");
}
In ef-core it has become much easier to retrieve meta data from the mapping model. You can get the PropertyInfos of the primary key by this line:
var keyPropertyInfos = dbContext.Model.FindEntityType(typeof(TEntity))
.FindPrimaryKey()
.Properties
.Select(p => p.PropertyInfo);
By the way, you can get all (mapped) properties by replacing FindPrimaryKey().Properties by GetProperties();
I have been understanding this example for Xamarin cross-platform mobile development:
https://msdn.microsoft.com/en-us/library/dn879698.aspx
I made an error by copying two times the the API key in the code:
using System;
using System.Threading.Tasks;
namespace XWeatherApp
{
public class Core
{
public static async Task<Weather> GetWeather(string zipCode)
{
//Sign up for a free API key at http://openweathermap.org/appid
string key = "40aabb59f41e9e88db7be4bab11f49f8";
string queryString = "http://api.openweathermap.org/data/2.5/weather?zip="
+ zipCode + ",us&appid=" + key + "&units=imperial";
//Make sure developers running this sample replaced the API key
if (key == "40aabb59f41e9e88db7be4bab11f49f8")
{
throw new ArgumentException("You must obtain an API key from openweathermap.org/appid and save it in the 'key' variable.");
}
dynamic results = await DataService.getDataFromService(queryString).ConfigureAwait(true);
if (results["weather"] != null)
{
Weather weather = new Weather();
weather.Title = (string)results["name"];
weather.Temperature = (string)results["main"]["temp"] + " F";
weather.Wind = (string)results["wind"]["speed"] + " mph";
weather.Humidity = (string)results["main"]["humidity"] + " %";
weather.Visibility = (string)results["weather"][0]["main"];
DateTime time = new System.DateTime(1970, 1, 1, 0, 0, 0, 0);
DateTime sunrise = time.AddSeconds((double)results["sys"]["sunrise"]);
DateTime sunset = time.AddSeconds((double)results["sys"]["sunset"]);
weather.Sunrise = sunrise.ToString() + " UTC";
weather.Sunset = sunset.ToString() + " UTC";
return weather;
}
else
{
return null;
}
}
}
}
Specifically, in the lines after the two comments.
I deployed the app to a physical Android phone. Obviously I got an exception (this was not so obvious after some minutes looking for the failing code).
That exception wasn't displayed in the Output window (in Visual Studio 2017). I just only got this message on screen:
Why don't the custom message for the exception (i.e., You must obtain an API key from openweathermap.org/appid and save it in the 'key' variable.).
Have you tried to use a try/catch?
something like
try{
await GetWeather(string zipCode);
}
catch(Exception ex) {
// here you should have your exception
}
I am using MongoDB Driver Java API to convert BSON to JSON.
I have test code like this.
String input = "{ \"timestamp\" : 1486064586641 }";
org.bson.Document doc = org.bson.Document.parse(input);
System.out.println("input = " + input);
System.out.println("output = " + doc.toJson());
The output is:
input = { "timestamp" : 1486064586641 }
output = { "timestamp" : { "$numberLong" : "1486064586641" } }
Is there an easy way to make the output look like the input?
BSON Documnet's toJson method supports only output to MongoDB Extended JSON (STRICT or SHELL format). If you want to have regular JSON, you can use com.mongodb.util.JSON class:
String input = "{ \"timestamp\" : 1486064586641 }";
org.bson.Document doc = org.bson.Document.parse(input);
System.out.println("input = " + input);
System.out.println("output (SHELL) = " + doc.toJson(new JsonWriterSettings(JsonMode.SHELL)));
System.out.println("output (STRICT) = " + doc.toJson(new JsonWriterSettings(JsonMode.STRICT)));
System.out.println("output (JSON) = " + com.mongodb.util.JSON.serialize(doc));
This will generate following output:
input = { "timestamp" : 1486064586641 }
output (SHELL) = { "timestamp" : NumberLong("1486064586641") }
output (STRICT) = { "timestamp" : { "$numberLong" : "1486064586641" } }
output (JSON) = { "timestamp" : 1486064586641}
Natalja's answer is excellent, but if you are using the Mongo Java driver 3.8.2 upwards you will notice some deprecation warnings. If you want the output to look like the input you can use RELAXED JsonWriterSettings mode.
Below you can see an example with the possible modes and how the JSON will looks like. There are also some deprecation warnings and alternatives to the deprecated code:
String input = "{ \"timestamp\" : 1486064586641 }";
org.bson.Document doc = org.bson.Document.parse(input);
System.out.println("input = " + input);
JsonWriterSettings shellMode = JsonWriterSettings.builder().outputMode(JsonMode.SHELL).build();
System.out.println("output (SHELL) = " + doc.toJson(shellMode));
JsonWriterSettings strictMode = JsonWriterSettings.builder().outputMode(JsonMode.STRICT).build();
System.out.println("output (STRICT) = " + doc.toJson(strictMode)); // deprecated - use extended like below
JsonWriterSettings extendedMode = JsonWriterSettings.builder().outputMode(JsonMode.EXTENDED).build();
System.out.println("output (EXTENDED) = " + doc.toJson(extendedMode));
JsonWriterSettings relaxed = JsonWriterSettings.builder().outputMode(JsonMode.RELAXED).build();
System.out.println("output (RELAXED) = " + doc.toJson(relaxed));
System.out.println("output (JSON) = " + com.mongodb.util.JSON.serialize(doc)); // deprecated - use relaxed like above
Also note that the JsonWriterSettings constructor is deprecated and you can use as an alternative the builder method like e.g:
JsonWriterSettings.builder().outputMode(JsonMode.SHELL).build()