I'm a new learned and follow AWS example here: https://docs.aws.amazon.com/elasticsearch-service/latest/developerguide/es-gsg-upload-data.html to build my first elasticsearch application.
I'm using Postman for curl command POST to append new document and index as shown here. However it seems the syntax is incorrect highlight by Postman so see if support to correct, thank you!
If you're posting to the _bulk URL, then the lines 1 through 6 should be on a single line without newlines. Same for lines 7 and 8. And you need to add a single newline at the end. Like this:
{ "index": {"_index": "movies", "_type": "movie", "_id": "2" }}
{ "director": "...", ...} + add a new line at the end of this line
Related
In my pipeline I reach through REST API using GET request to a 3rd party database. As an output I receive a bunch of JSON files. The number of JSON files I have to download (same as number of iterations I will have to use) is in one of the fields in JSON file. The problem is that the field's name is 'page-count' which contains "-".
#activity('Lookup1').output.firstRow.meta.page.page-count
Data Factory considers dash in field's name as a minus sign, so I get an error instead of value from that field.
{"code":"BadRequest","message":"ErrorCode=InvalidTemplate, ErrorMessage=Unable to parse expression 'activity('Lookup1').output.firstRow.meta.page.['page-count']'","target":"pipeline/Product_pull/runid/f615-4aa0-8fcb-5c0a144","details":null,"error":null}
This is how the structure of JSON file looks like:
"firstRow": {
"meta": {
"page": {
"number": 1,
"size": 1,
"page-count": 7300,
"record-count": 7300
},
"non-compliant-record-count": 7267
}
},
"effectiveIntegrationRuntime": "intergrationRuntimeTest1",
"billingReference": {
"activityType": "PipelineActivity",
"billableDuration": [
{
"meterType": "SelfhostedIR",
"duration": 0.016666666666666666,
"unit": "Hours"
}
]
},
"durationInQueue": {
"integrationRuntimeQueue": 1
}
}
How to solve this problem?
The below syntax works when retrieving the value for a json element with a hyphen. It is otherwise treated as a minus sign by the parser. It does not seem to be documented by Microsoft however I managed to get this to work through trial and error on a project of mine.
#activity('Lookup1').output.firstRow.meta.page['page-count']
This worked for us too. We had the same issue where we couldn't reference an output field that contained a dash(-). We referenced this post and used the square brackets and single quote and it worked!
Example below.
#activity('get_token').output.ADFWebActivityResponseHeaders['Set-Cookie']
I'm using Solr v8.0.0 and im trying to split a json at the indexing time using the method described at the solr official documentation about Transforming JSON, but it is not working as expected, and I'm getting flat jsons at the end.
Here is how I'm doing it:
First I create a single core named C2
bin/solr create_core -c c2
Then, it's solrconfig.xml is automatically created and left as default.
then I try to index some data using the example URL. the only difference is that I added a ?commit=true to the end of the URL so we can see whats happening
curl 'http://localhost:8983/solr/c2/update/json/docs'\
'?commit=true'\
'?split=/'\
'&f=first:/first'\
'&f=last:/last'\
'&f=grade:/grade'\
'&f=subject:/exams/subject'\
'&f=test:/exams/test'\
'&f=marks:/exams/marks'\
-H 'Content-type:application/json' -d '
{
"first": "John",
"last": "Doe",
"grade": 8,
"exams": [
{
"subject": "Maths",
"test" : "term1",
"marks" : 90},
{
"subject": "Biology",
"test" : "term1",
"marks" : 86}
]
}'
but at the end, I got this kind of indexing, and not the one that was shown in the example:
What I Got:
{
{
"first":["John"],
"last":["Doe"],
"grade":[8],
"subject":["Maths",
"Biology"],
"test":["term1",
"term1"],
"marks":[90,
86],
"id":"284878be-1339-43b5-8a1e-adb7a4be95fb",
"_version_":1664059760532520960}]
}
What I was supposed to get:
{
"first":"John",
"last":"Doe",
"marks":90,
"test":"term1",
"subject":"Maths",
"grade":8
}
{
"first":"John",
"last":"Doe",
"marks":86,
"test":"term1",
"subject":"Biology",
"grade":8
}
My fields where flatten as it would usually do in a normal indexation, without the ?split=/ command in the url. Can anyone help me figure out why this behavior is happening?
Thanks.
No, that's not the only difference. In your request you have:
'?split=/'\
In the example from the manual it is:
'?split=/exams'\
And since you're not splitting on /exams in your request, the result differs.
I want to import data of type JSON in MongoDB compass,
the import function gives this error
" unexpected end of JSON input "
there is a some of my JSON file
[
{
"id":4,
"user":"test#example.com",
"date1":"2019-03-01",
"date2":"2019-04-01",
"statut":"Good",
"guest_number":4
}
]
the solution is to write all JSON in one line, but if we have a big doc !!
I just found a solution that I can import data with this command in terminal :
mongoimport --jsonArray --db YourDatabase --collection YourCollection --file Yourfile.json
I had this issue 6 month ago, the solution is write all JSON in one line.
[{"id":4,"user":"test#example.com","date1":"2019-03-01","date2":"2019-04-01","statut":"Good","guest_number":4}]
MongoDB Compass will told you:
Import success!
But definitely the document will not appear in your collection, so better use Robo3T if you gonna insert json. Then you can use again Compass like I do.
It is weird, yes, but I didnt found other solution yet.
[UPDATE]
I achieve import data with Compass, but I achieve exporting first a document from Compass to see how it write the json.
{"_id":{"$oid":"5e4cf105c9ba1a21143d04a2"},"tPreguntas":["Pregunta 1","Pregunta 2","Pregunta 3","Pregunta 4","Pregunta 5"],"tCategorias":[],"tPublico":true,"tFechaCreacion":{"$date":{"$numberLong":"1582100741716"}},"tCodigo":"test1","tTitulo":"Test 1","tDescripcion":"Test de muestreo número uno para comprobar.","tCreadoPor":"eoeo#eoeo.com"}
It look to different to the json online I have post in my first post. (look that objectId "$oid" for example). So if you follow that pattern Compass will import you fine.
This parsing error can be solved using minification. So, minify json like this. Although, it is quite a hectic process to do this for each object.
And this kind of minification like this worked for me.
{
"_id" : ObjectId("5b9ecf9a64f634289ca895bb"),
"name" : "Mark"
}
{
"_id" : ObjectId("5b9edd9064f634289ca895e4"),
"name" : "David"
}
To :
{"_id":"ObjectId(\"5b9ecf9a64f634289ca895bb\")","name":"Mark"}
{"_id":"ObjectId(\"5b9edd9064f634289ca895e4\")","name":"David"}
Just copy the contents of your json file then in Mongodb Compass select your database then click on Add Data which will drop down then click on insert document a dialog pops up then paste it in there and click insert.
This parsing error can be solved using minification. So, minify json like this. Although, it is quite a hectic process to do this for each object.
{
"_id" : "123456",
"name" : "stackoverflow"
}
change to :
{"_id":"123456","name":"stackoverflow"}
This answer here Solution solved the issue for me. It seems to be a formatting issue.
It's an issue with the end-of-line characters (EOL).
In a Windows environment line terminations are normally CR NL (\r\n), while MongoDB Compass seems to only support CR (\r).
You can open the file in Notepad++, enable the "Show all characters" toggle in the toolbar and inspect your current end-of-line character.
To fix the issue, select Edit > EOL Conversion > Macintosh (CR).
The structure of your JSON is incorrect, you might want to read info regarding JSON standards
A value can be a string in double quotes, or a number, or true or false or null, or an object or an array. These structures can be nested.
try using double quotes instead of single ones:
JSON validators could help you aswell
[
{
"id" : 4,
"user" : "test#example.com",
"date1" : "2019-03-01",
"date2" : "2019-04-01",
"statut" : "Good",
"guest_number" : 4
}
]
I had a similar issue but it turned out to be additional line feeds at the end of the file. Removing these fixed the issue. I suggest opening your file in an editor that shows line feeds e.g. Notepad++
Add --jsonFormat=canonical to your mongoexport script:
mongoexport --db=quotes --collection=quotes --jsonFormat=canonical --out=data/quotes.json
JSON can only directly represent a subset of the types supported by BSON. To preserve type information, MongoDB adds the following extensions to the JSON format.
Source
You can also use the command line of mongodb like this :
db.user.insert(
[
{
"id" : 4,
"user" : "test#example.com",
"date1" : "2019-03-01",
"date2" : "2019-04-01",
"statut" : "Good",
"guest_number" : 4
},
{
"id" : 5,
"user" : "test2#example.com",
"date1" : "2019-03-01",
"date2" : "2019-04-01",
"statut" : "Good",
"guest_number" : 4
}
]
Run this command in cmd and the cmd path should be in the same folder where the JSON file occurs.
mongoimport --jsonArray --db YourDatabase --collection YourCollection --file Yourfile.json
I'm trying to use the hbase rest API. I've read the docs here (http://wiki.apache.org/hadoop/Hbase/Stargate), but they seem to be incomplete (doesn't cover json) and out of date.
I've also run across this gist, which is mostly what I'm going off of.
Here is the (insert) request I'm trying to send:
URI: http://server:8070/table/row1/data
Headers:
Content-Type : application/json
Accept : application/json
HTTP verb - POST
and the json payload (where key, column, and $ are all base64 encoded values):
{
"Row": {
"key": "NjQ=",
"Cell": [{
"column": "NjQ=",
"$": "NjQ="
}]
}
}
Here is the error I get:
Error 500 Can not deserialize instance of java.util.List out of START_OBJECT token
at [Source: org.mortbay.jetty.HttpParser$Input#3da0b822; line: 2, column: 5]
(through reference chain: org.apache.hadoop.hbase.rest.model.CellSetModel["Row"])
The error seems to suggest that there is something that should be an array and isn't. I've tried putting square brackets just about everywhere, and I can't get it to do anything other than change the nature of the error message slightly.
I had a look through the source code and that seems to suggest that the placement for the square brackets that I have is correct. However, as far as I can tell the entire request looks to be correct. I'm not very fluent in Java though, so maybe I'm missing something.
What is the proper JSON syntax for inserting a record with the hbase rest gateway?
This is now the correct format:
{
"Row": [{
"key": "NjQ=",
"Cell": [{
"column": "NjQ=",
"$": "NjQ="
}]
}]
}
Where:
key - is the row key
column - is the column name
$ - is the data that you are storing in the given table, column, and row.
I find this kinda strange. it works on older versions of iQuery (ie 1.2.x to 1.3). However, i am getting a parserror when using jQuery 1.4.x
Any help will be appreciated. Thank you.
This is what i get from XMLHttpRequest.responseText
({count: 5, success: true, error: '', cache: false, data: [{column: ['Mike','Mike','Steve','Steve','Steve']}]})
Jquery 1.4 updated their JSON parser, so that it no longer accepts invalid JSON that it did before. Your server is outputting invalid JSON that was previously tolerated, but no longer is. This change was mentioned (briefly) in the release notes and documentation, and there's been lots of discussion about it if you google "jquery 1.4 JSON"
But to sum up: the keys in JSON, to be valid, must have quotes around them, the string can't be surrounded by those parentheses, and nothing can use single quotes as delimiters, so your JSON would need to be:
{"count": 5, "success": true, "error": "", "cache": false, "data": [{"column": ["Mike","Mike","Steve","Steve","Steve"]}]}
or, more readably:
{
"count": 5,
"success": true,
"error": "",
"cache": false,
"data": [
{
"column": [
"Mike",
"Mike",
"Steve",
"Steve",
"Steve"
]
}
]
}
The specs are very nicely and clearly spelled out at http://www.json.org/ with pictures and diagrams. You can also check your JSON at JSONLint - the JSON above checks out there just fine, but your original JSON raises all kinds of errors.
So as to what you need to do: if you have the ability to change what your server sends out, do that - make it output valid JSON. Please. Only if you are unable to change the server's behavior, you'll have to use a workaround from one of the posts linked above, or use some other JSON library.
General principle: in all cases, doing it the Right Way™ is preferable - fixing the server's output (even if that's filing a bug on an external server/API) is the long-term solution, but may not be immediately practical. So hackish workarounds are sometimes necessary, but not optimal.