Please help me here,
I have a spark dataframe which does API call and returns the output in JSON format which is then parsed/exploded for further processing. Let's say I have 1000 records in a dataframe and invoke the API call like in below code, then all 1000 records will try to hit the API and get the response or would that hit be sequential from spark side ?
What are the possible ways to improve the API call step and get the response much faster. Right now, API call step is taking a lot of time. Please suggest any improvements.
API call UDF -->
val apiCall = udf((url : String, DPID : String,KAFKA_DTS : String) => {
val client = IgnoreSSLOkHttpClient()
val headerBuilder = new Headers.Builder
val headers = headerBuilder.add("content-type", "application/json").add("Authorization", authStr)build()
try {
val request = new Request.Builder().url(url).headers(headers).build()
val response = client.newCall(request).execute()
if (response.code() == 200) {
response.body().string()
}
else {
"API_Call" + "#~#" + DPID + "|~|" + response.code() + "|~|" + response.body().string() + "|~|" + KAFKA_DTS + "#~#error:record has response code <> 200" + "#~#" + KAFKA_DTS + "#~#" + "Y"
}
}
catch {
case e: Exception =>
e.printStackTrace()
val sw = new StringWriter()
e.printStackTrace(new PrintWriter(sw))
print(e.toString)
logger.info("exception: " + e.toString)
"API_Call" + "#~#" + DPID + "|~|" + KAFKA_DTS + "#~#error:record did not complete API call" + "#~#" + KAFKA_DTS + "#~#" + "Y"
}
}
)
val Json = urls2
.withColumn("purchaseJson", apiCall(urls3("Url"),urls3("ID"),urls3("DTS")))
.filter("purchaseJson is not null ") .select("purchaseJson","ID","SRC_ENV_NM","CD","DTS","KAFKA_DTS")
Related
I am working on a fully automatic pipeline for my company where we automatically set up projects, add users and upload files with the different APIs on BIM360. On the stage of adding a user I get a 500 internal server error:
{"code":2000,"message":"no implicit conversion of String into Integer"}
We are using a two-legged authentication approach and as such the header looks like this:
Authorization: Bearer <token> (It has account:write rights)
x-user-id: ************ (uid of my admin account)
Content-Type: application/json
The request content is this:
#"{
""email"": """ + ***#********.** + #""",
""services"": {
""document_management"": {
""access_level"": """ + admin+ #"""
},
""project_administration"": {
""access_level"": """ + admin+ #"""
}
},
""industry_roles"": []}";
I just can't quite seem to figure out what I am doing wrong. Hope someone can help me.
EDIT: Full code for this request
public async static Task<HttpStatusCode> AddUserToProjectEmail(string projectId, string accountId, string accessToken, string userToAddEmail, string userPrivilege, string adminUserId)
{
using (HttpClient httpClient = new HttpClient())
{
using (HttpRequestMessage request = new HttpRequestMessage())
{
//Documentation for what to put in the Http POST: https://forge.autodesk.com/en/docs/bim360/v1/reference/http/projects-project_id-users-import-POST/
request.Method = new HttpMethod("POST");
request.RequestUri = new Uri("https://developer.api.autodesk.com/hq/v2/regions/eu/accounts/" + accountId + "/projects/" + projectId + "/users/import");
//Make the request payload
string jsonPayload = AddPayloadToUserAddEmail(userToAddEmail, userPrivilege);
request.Content = new StringContent(jsonPayload);
request.Content.Headers.ContentType = MediaTypeHeaderValue.Parse("application/json");
request.Headers.Add("x-user-id", adminUserId);
request.Headers.Authorization = new AuthenticationHeaderValue("Bearer", accessToken);
//Send request
var response = await httpClient.SendAsync(request);
return response.StatusCode;
}
}
}
And the request payload method:
private static string AddPayloadToUserAddEmail(string userToAddEmail, string userPrivilege)
{
string payload = #"{
""email"": """ + userToAddEmail + #""",
""services"": {
""project_administration"": {
""access_level"": """ + userPrivilege + #"""
},
""document_management"": {
""access_level"": """ + userPrivilege + #"""
}
},
""industry_roles"": []
}";
return payload;
}
I have checked all the IDs through the URL on BIM360, however it's not possible to check the Uid of my account I think.
EDIT 2: I should note that I was getting a different error before I added the x-user-id header, where it just said forbidden which makes sense. This lead me to think it had something to do with the x-user-id header, but I can't figure it out.
Don't be like me and forget to wrap the payload into an array as stated on the docs that it has to be. Using this as payload worked
#"[{
""email"": """ + userToAddEmail + #""",
""services"": {
""project_administration"": {
""access_level"": """ + userPrivilege + #"""
},
""document_management"": {
""access_level"": """ + userPrivilege + #"""
}
},
""industry_roles"": []
}]";
I am working on a Flutter app. we have a PSQL database, Node server on the background. On the Flutter app I am displaying some geometry which I fetch from the database successfully. Now after a modification on the geometry, such as lines, I want to be able to update the database via a put request.
Server goes like:
app.put('/api/shape/:id', async (req,res) =>{
let answer;
if( req.body.shape_type == "line"){
answer = await db.db.modify_line(req.params.id, req.body.info_shape);
}
res.send(answer);
});
And db.js file goes like:
modify_line : async function(id_shape, info_shape){
console.log(info_shape);
const result = await send_query("UPDATE line SET line = $2 WHERE id_shape = $1", [id_shape, info_shape]);
return(result);
},
On the Flutter app I do this:
_makeUpdateRequest() async {
var url = globals.URL + 'api/shape/' + globals.selectedShapeID.toString();
Map data;
if (globals.selectedType == globals.Type.line) {
String lseg = "(" + globals.pLines[globals.selectedLineIndex].p1.dx.toString() + "," +
globals.pLines[globals.selectedLineIndex].p1.dy.toString() + "," +
globals.pLines[globals.selectedLineIndex].p2.dx.toString() + "," +
globals.pLines[globals.selectedLineIndex].p2.dy.toString() + ")";
data = {
'shape_type': 'line',
'info_shape': {
'id_shape': globals.selectedShapeID.toString(),
'line': lseg,
}
};
}
http.Response response;
try {
//encode Map to JSON
print("encode Map to JSON");
var body = json.encode(data);
print(body);
response =
await http.put(url,
headers: {
"Content-Type": "application/json"
},
body: body
).catchError((error) => print(error.toString()));
} catch (e) {
print(e);
}
return response;
}
Database "line" table contains a "shapeID" and "lseg" information on each row.
Currently I am getting an error when I try this code:
{ id_shape: '619',
line: '(19.5,100.6,20.5,50.9)' }
fail____error: invalid input syntax for type lseg: "{"id_shape":"619","line":"(-19.5,100.6,20.5,50.9)"}"
How shall I shape my lseg json?
Thanks
Well, it looks like to me you are passing the whole input_shape object to the SQL query, which looks like this, as per your console.log:
{
id_shape: '619',
line: '(19.5,100.6,20.5,50.9)'
}
Obviously, this is invalid for PostgreSQL.
I would say that your backend code should be more like this:
modify_line : async function(id_shape, info_shape){
console.log(info_shape);
const result = await send_query(
"UPDATE line SET line = $2 WHERE id_shape = $1",
// Reference "line" sub-object
[id_shape, info_shape.line],
);
return(result);
},
You should also pay attention to the Geometric types format for lines:
[ ( x1 , y1 ) , ( x2 , y2 ) ]
( ( x1 , y1 ) , ( x2 , y2 ) )
( x1 , y1 ) , ( x2 , y2 )
x1 , y1 , x2 , y2
I'm not 100% sure by reading this that your format (with leading and trailing parenthesis) is correct.
As the issue is solved, following is the answer:
DB.js is like:
modify_line : async function(id_shape, info_shape){
const result = await send_query("UPDATE line SET line = $2 WHERE id_shape = $1", [info_shape['id_shape'], info_shape['line']]);
return(result);
},
and Flutter app is like:
_makeUpdateRequest() async {
var url = globals.URL + 'api/shape/' + globals.selectedShapeID.toString();
Map data;
if (globals.selectedType == globals.Type.line) {
String lseg =
"[" + globals.pLines[globals.selectedLineIndex].p1.dx.toString() + "," +
globals.pLines[globals.selectedLineIndex].p1.dy.toString() + "," +
globals.pLines[globals.selectedLineIndex].p2.dx.toString() + "," +
globals.pLines[globals.selectedLineIndex].p2.dy.toString() + "]";
data = {
'shape_type': 'line',
'info_shape': {
'id_shape': globals.selectedShapeID.toString(),
'line': lseg,
}
};
}
http.Response response;
try {
//encode Map to JSON
print("encode Map to JSON");
var body = json.encode(data);
print(body);
response =
await http.put(url,
headers: {
"Content-Type": "application/json"
},
body: body
).catchError((error) => print(error.toString()));
} catch (e) {
print(e);
}
return response;
}
I'm developing a project that uses the ESP8266 with ArduinoJson library.
On my webserver I need create a http response with a JSON (read from a file) inside another JSON.
Something like this:
String data = "";
String success = "0";
File loadFile = SPIFFS.open(filename, "r");
if (!loadFile){
Serial.println("Il file non esiste: " + filename);
} else {
size_t size = loadFile.size();
if ( size == 0 ) {
Serial.println("File vuoto: " + filename);
} else {
while (loadFile.available()){
data += char(loadFile.read());
}
success = "1";
}
loadFile.close();
}
String json;
json = "{\"success\":\"" + String(success) + "\",";
json += "\"form\":\"" + data + "\"}";
server->send(200, "application/json", json);
The content of "data" variable is correct but the on client I get:
{
"success": 1,
"data": { }
}
Data is empty.
What is the right way to add a JSON string inside another JSON string in "arduino and esp8266"?
I have a problem for a long time that I would like to solve:
I have a fragment with buttons, that when I press a button I have the variable * data that happened to the url I want to open.
val button01 = view.findViewById(R.id.tv_01) as Button
button01.setOnClickListener{
dato = "01"
miTexto.setText("Jornada - 01")
requestJsonObject()
}
My url
val url = Directions.JORNADAS + Directions.CMP + "&jor=$dato&tmp=" + Directions.TMP
This url is clear that he opens it to me, it is a json which I pause and I already pass to the adapter and shows it to me.
From the adapter I pass data, which come in the Json with a put.extras intent, to the activity details.kt which depends on the item that pulse shows me the data of another url.
val intent = Intent(context, Detalles::class.java
holder.itemView.setOnClickListener{
intent.putExtra("nomLocal", jornada.nomLocal)context.startActivity(intent)......
Up here all good.
My problem: I need to pass the variable data to the activity Details.kt to be able to access the url, since * data is a piece of url of what I am going to parse in the activity Details
I had thought about adding an item to the Json
private fun requestJsonObject() {
val queue = newRequestQueue(activity)
//http://www.ffcv.es/ncompeticiones/server.php?action=getResultados&cmp=328&jor=1&tmp=2018/2019
val url = Directions.JORNADAS + Directions.CMP + "&jor=$dato&tmp=" + Directions.TMP
val stringRequest = StringRequest(Request.Method.GET, url, Response.Listener { response ->
val builder = GsonBuilder()
val mGson = builder.create()
val items: List<ModelJor>
items = Arrays.asList(*mGson.fromJson(response, Array<ModelJor>::class.java))
items.add(ModelJor("\"jornada\":" + $dato)) // dato en rojo
Log.d("Resultado", items.toString())
recyclerView !!.layoutManager = GridLayoutManager(activity!!, 1)
val adapter = AdapJor(activity !!, items)
recyclerView !!.adapter = adapter
}, Response.ErrorListener { error -> Log.d(TAG, "Error " + error.message) })
queue.add(stringRequest)
}
Any solution?
I need to create a view with a map function ex:
function(doc, meta) {
if(doc.docType == "testDoc")
emit(meta.id, doc)
}
I have to create this view using couchbase java client 2.1.I could not find any thing in the documentation thanks
Here is some code to create a design document:
List<View> viewsForCurrentDesignDocument = new ArrayList<View>();
DesignDocument designDocument = DesignDocument.create("my_design_doc", viewsForCurrentDesignDocument);
And add a view:
String mapFunction =
"function (doc, meta) {\n" +
" if(doc.docType == \"testDoc\") {\n" +
" emit(meta.id, doc);\n" +
" }\n" +
"}";
String reduceFunction = "..." // or null if no reduce
View v = DefaultView.create("my_view", mapFunction, reduceFunction);
viewsForCurrentDesignDocument.add(v);
bucket.bucketManager().insertDesignDocument(designDocument);
You can check the API reference for more options(development mode, timeout, ...).