Related
I have to update dictionary with new value for existing key in a JSON file.
I need no write new line after existing string using regex
Current file:
{
"id": "aaaa",
"desc": "Service aaa",
"boss":"user#company.de",
"email": [
"user#company.de"
],
desired file:
{
"id": "aaaa",
"desc": "Service aaa",
"boss":"user#company.de",
"email": [
"user#company.de"
"another_user#company.de"
]
I have this ansible lineinfile module playbook, but I struggle with decent regex. Everything I try just adds new line in the very end of file.
---
- hosts: localhost
gather_facts: no
tasks:
- name: insert line
lineinfile:
path: /home/file.json
state: present
insertafter: " ^ "email": [ "
line: 'another_user#company.de'
How should I write correct regex in this case to write line after the string "email": [ ?
quick comment :
JSON spec mandates an ASCII comma (",") between values of arrays (plus your choice of whitespace(s)), so to make the proposed solutions compliant, they would have to resemble this instead
—— (snippet directly from jq):
{
"id": "aaaa",
"desc": "Service aaa",
"boss": "user#company.de",
"email": [
"user#company.de",
"newline"
]
}
Let's say current file is aa.txt as follows
{
"id": "aaaa",
"desc": "Service aaa",
"boss":"user#company.de",
"email": [
"user#company.de"
],
Use sed command
sed '/"email.*\[/!{p;d;};n;a new line' aa.txt
Output
{
"id": "aaaa",
"desc": "Service aaa",
"boss":"user#company.de",
"email": [
"user#company.de"
new line
],
Alternatively use AWK
awk '1;/"email.*\[/{c=2}c&&!--c{print "new text"}' aa.txt
Output
{
"id": "aaaa",
"desc": "Service aaa",
"boss":"user#company.de",
"email": [
"user#company.de"
new line
],
I'm struggling to generate a proper JSON POST request using cURL. For that purpose I'm writing a short shell script to do so but apparently there seems to be a problem with my JSON-string (according to the error message I listed below).
If I write the CSR directly into the JSON string, it will work just fine:
authToken="Here'sMyAuthToken"
curl --data-binary '{"authToken" : "'$authToken'", "order" : {"type" : "DomainValidatedCertificateOrder", "csr" : "-----BEGIN CERTIFICATE REQUEST-----
certificaterequestkey
-----END CERTIFICATE REQUEST-----", "adminContact" : {"title" : "mytitle", "firstName" : "myfirstname", "lastName" : "mylastname", "phoneNumber" : "X0000000", "emailAddress" : "test#example.com"}, "techContact" : {"title" : "mytitle", "firstName" : "myfirstname", "lastName" : "mylastname", "phoneNumber" : "000000000", "emailAddress" : "test#example.com"}, "productCode" : "ssl-geotrust-rapidssl-12m", "validationType" : "validateViaDns", "approverEmailAddress" : "postmaster#example.com", "autoRenew" : false}}' -i -X POST https://partner.http.net/api/ssl/v1/json/orderCreate
However, if I pass the CSR over by reading the csr file directly like this
authToken="Here'sMyAuthToken"
csr=$(<csr.csr)
curl --data-binary '{"authToken" : "'$authToken'", "order" : {"type" : "DomainValidatedCertificateOrder", "csr" : "'$csr'", "adminContact" : {"title" : "mytitle", "firstName" : "myfirstname", "lastName" : "mylastname", "phoneNumber" : "X0000000", "emailAddress" : "test#example.com"}, "techContact" : {"title" : "mytitle", "firstName" : "myfirstname", "lastName" : "mylastname", "phoneNumber" : "000000000", "emailAddress" : "test#example.com"}, "productCode" : "ssl-geotrust-rapidssl-12m", "validationType" : "validateViaDns", "approverEmailAddress" : "postmaster#example.com", "autoRenew" : false}}' -i -X POST https://partner.http.net/api/ssl/v1/json/orderCreate
it will give me the following error.
curl: option -----END: is unknown
curl: try 'curl --help' or 'curl --manual' for more information
I've already found a case where someone had the exact same problem like me here:
POST request containing CSR fails in Bash
The user accomplished solving this problem using the jq package. Unfortunately I can't install this package on the machine the script is supposed to run since I'm not allowed to install any packages at all.
Could someone give an advice how to solve this problem?
Many thanks in advance!
I have been having a very hard time posting. I kept getting Errors when saving.
This is the request and response with a Content-Type: application/json in the HTTP request header:
Content-Length: 540
Content-Type: application/json
Accept: application/json
Accept-Encoding: deflate, gzip, br
Host: eatled.com
BODY={"authToken": "$authToken","order": {"type": "DomainValidatedCertificateOrder","csr": "$csr","adminContact": {"title": "mytitle","firstName": "myfirstname","lastName": "mylastname","phoneNumber": "X0000000","emailAddress": "test#example.com"},"techContact": {"title": "mytitle","firstName": "myfirstname","lastName": "mylastname","phoneNumber": "000000000","emailAddress": "test#example.com"},"productCode": "ssl-geotrust-rapidssl-12m","validationType": "validateViaDns","approverEmailAddress": "postmaster#example.com","autoRenew": false}}
I removed the Content-Type and this is the request and response.
Notice how the header changed to Content-Type: application/x-www-form-urlencoded and how the JSON is in the KEY of the $_POST data as it was received by the Server. When the server sees the form data header it will may process try to process the request as if it were a form. It depends upon how well the API was written.
Content-Length: 540
Content-Type: application/x-www-form-urlencoded
Accept: application/json
Accept-Encoding: deflate, gzip, br
Host: eatled.com
$_POST
array (
'{"authToken":_"$authToken","order":_{"type":_"DomainValidatedCertificateOrder","csr":_"$csr","adminContact":_{"title":_"mytitle","firstName":_"myfirstname","lastName":_"mylastname","phoneNumber":_"X0000000","emailAddress":_"test#example_com"},"techContact":_{"title":_"mytitle","firstName":_"myfirstname","lastName":_"mylastname","phoneNumber":_"000000000","emailAddress":_"test#example_com"},"productCode":_"ssl-geotrust-rapidssl-12m","validationType":_"validateViaDns","approverEmailAddress":_"postmaster#example_com","autoRenew":_false}}' => '',
)
This is your curl code with all the escape codes and the options reorganized because the you had them generated an error. This is tested, and works.
curl -i -H "Content-Type: application/json" -X POST http://eatled.com/receiveheader.php --data-binary "{\"authToken\": \"$authToken\",\"order\": {\"type\": \"DomainValidatedCertificateOrder\",\"csr\": \"$csr\",\"adminContact\": {\"title\": \"mytitle\",\"firstName\": \"myfirstname\",\"lastName\": \"mylastname\",\"phoneNumber\": \"X0000000\",\"emailAddress\": \"test#example.com\"},\"techContact\": {\"title\": \"mytitle\",\"firstName\": \"myfirstname\",\"lastName\": \"mylastname\",\"phoneNumber\": \"000000000\",\"emailAddress\": \"test#example.com\"},\"productCode\": \"ssl-geotrust-rapidssl-12m\",\"validationType\": \"validateViaDns\",\"approverEmailAddress\": \"postmaster#example.com\",\"autoRenew\": false}}"
You are misquoting things in your command line. You're making frequent use of this sort of structure:
'{"somekey": "'$variable'"}'
That means that you're not quoting $somevariable, so if it contains whitespace you're going to end up with a command other than what you expect. You need to quote all your variables, so the above becomes:
'{"somekey": "'"$variable"'"}'
And your full command line is:
curl --data-binary '
{
"authToken" : "'"$authToken"'",
"order" : {
"type" : "DomainValidatedCertificateOrder",
"csr" : "'"$csr"'",
"adminContact" : {
"title" : "mytitle",
"firstName" : "myfirstname",
"lastName" : "mylastname",
"phoneNumber" : "X0000000",
"emailAddress" : "test#example.com"
},
"techContact" : {
"title" : "mytitle",
"firstName" : "myfirstname",
"lastName" : "mylastname",
"phoneNumber" : "000000000",
"emailAddress" : "test#example.com"
},
"productCode" : "ssl-geotrust-rapidssl-12m",
"validationType" : "validateViaDns",
"approverEmailAddress" : "postmaster#example.com",
"autoRenew" : false
}
}
' -i -X POST https://partner.http.net/api/ssl/v1/json/orderCreate
You could simplify things by using a here document instead of trying to embed everything on the command line. That would look like:
curl -i -X POST --data-binary #- https://partner.http.net/api/ssl/v1/json/orderCreate <<EOF
{
"authToken": "$authToken",
"order": {
"type": "DomainValidatedCertificateOrder",
"csr": "$csr",
"adminContact": {
"title": "mytitle",
"firstName": "myfirstname",
"lastName": "mylastname",
"phoneNumber": "X0000000",
"emailAddress": "test#example.com"
},
"techContact": {
"title": "mytitle",
"firstName": "myfirstname",
"lastName": "mylastname",
"phoneNumber": "000000000",
"emailAddress": "test#example.com"
},
"productCode": "ssl-geotrust-rapidssl-12m",
"validationType": "validateViaDns",
"approverEmailAddress": "postmaster#example.com",
"autoRenew": false
}
}
EOF
Now you don't need all those quoting tricks.
Here's how I've tested the above solution:
#!/bin/bash
# Use some sort of http debugging service to verify the content
# of the request.
url="https://eny65dku43a4g.x.pipedream.net"
# Create an example CSR
openssl req new -nodes \
-keyout req.key \
-out req.csr \
-subject '/O=Example$Organization+Inc,CN=example.com'
csr=$(<req.csr)
authToken='example+password$here'
curl -i -X POST "$url" --data-binary #- <<EOF
{
"authToken": "$authToken",
"order": {
"type": "DomainValidatedCertificateOrder",
"csr": "$csr",
"adminContact": {
"title": "mytitle",
"firstName": "myfirstname",
"lastName": "mylastname",
"phoneNumber": "X0000000",
"emailAddress": "test#example.com"
},
"techContact": {
"title": "mytitle",
"firstName": "myfirstname",
"lastName": "mylastname",
"phoneNumber": "000000000",
"emailAddress": "test#example.com"
},
"productCode": "ssl-geotrust-rapidssl-12m",
"validationType": "validateViaDns",
"approverEmailAddress": "postmaster#example.com",
"autoRenew": false
}
}
EOF
I am currently working on a solution using Mongo database.
We must use MongoExport for this one.
Is there a way to enhance my MongoExport command to include a filter on the fields?
For instance:
-q:"{"Status": "COMPLETE"}"
-f:Id,EanCode,Channels:{"$elemMatch":{"ChannelId":"Test"}
Update:
Hello again, by suggestion in the comments I am adding am example document.
Document:
{
"Id": "1"
"Status": "COMPLETE",
"EanCode": "1234567890",
"Barcode": "100100100",
"Name": "Test Product",
"Channels": [
{
"ChannelId": "Test"
},
{
"ChannelId": "Dumbeldore"
}
]
}
Result:
{
"Id": "1"
"Status": "COMPLETE",
"EanCode": "1234567890",
"Channels": [
{
"ChannelId": "Test"
}
]
}
I can use any tool you suggest, might space out from the MongoExport.
But the important thing is that It must be optimized and export JSON Objects.
Thank you for your attention and help :)
Option 1.
I think you cannot do with mongoexport directly , but you can do via aggregation:
echo 'db.test.aggregate([{$match:{Status:"COMPLETE","Channels.ChannelId":"Test"} },{$addFields:{Channels:{ $filter: { input: "$Channels", as: "ch", cond: {$eq:["$$ch.ChannelId","Test"] } } }} } ])' | mongo --authenticationDatabase admin --port 27017 -u test -p xxxx --quiet > out.json
playground1
Option 2:
You output from aggregation to a output collection via $out and then you mongoexport.
playground2
I have a JSON file and I need to index it on ElasticSearch server.
JSON file looks like this:
{
"sku": "1",
"vbid": "1",
"created": "Sun, 05 Oct 2014 03:35:58 +0000",
"updated": "Sun, 06 Mar 2016 12:44:48 +0000",
"type": "Single",
"downloadable-duration": "perpetual",
"online-duration": "365 days",
"book-format": "ePub",
"build-status": "In Inventory",
"description": "On 7 August 1914, a week before the Battle of Tannenburg and two weeks before the Battle of the Marne, the French army attacked the Germans at Mulhouse in Alsace. Their objective was to recapture territory which had been lost after the Franco-Prussian War of 1870-71, which made it a matter of pride for the French. However, after initial success in capturing Mulhouse, the Germans were able to reinforce more quickly, and drove them back within three days. After forty-three years of peace, this was the first test of strength between France and Germany. In 1929 Karl Deuringer wrote the official history of the battle for the Bavarian Army, an immensely detailed work of 890 pages; First World War expert and former army officer Terence Zuber has translated this study and edited it down to more accessible length, to produce the first account in English of the first major battle of the First World War.",
"publication-date": "07/2014",
"author": "Deuringer, Karl",
"title": "The First Battle of the First World War: Alsace-Lorraine",
"sort-title": "First Battle of the First World War: Alsace-Lorraine",
"edition": "0",
"sampleable": "false",
"page-count": "0",
"print-drm-text": "This title will only allow printing of 2 consecutive pages at a time.",
"copy-drm-text": "This title will only allow copying of 2 consecutive pages at a time.",
"kind": "book",
"fro": "false",
"distributable": "true",
"subjects": {
"subject": [
{
"-schema": "bisac",
"-code": "HIS027090",
"#text": "World War I"
},
{
"-schema": "coursesmart",
"-code": "cs.soc_sci.hist.milit_hist",
"#text": "Social Sciences -> History -> Military History"
}
]
},
"pricelist": {
"publisher-list-price": "0.0",
"digital-list-price": "7.28"
},
"publisher": {
"publisher-name": "The History Press",
"imprint-name": "The History Press Ireland"
},
"aliases": {
"eisbn-canonical": "1",
"isbn-canonical": "1",
"print-isbn-canonical": "9780752460864",
"isbn13": "1",
"isbn10": "0750951796",
"additional-isbns": {
"isbn": [
{
"-type": "print-isbn-10",
"#text": "0752460862"
},
{
"-type": "print-isbn-13",
"#text": "97807524608"
}
]
}
},
"owner": {
"company": {
"id": "1893",
"name": "The History Press"
}
},
"distributor": {
"company": {
"id": "3658",
"name": "asc"
}
}
}
But when I try to index this JSON file using command
curl -XPOST 'http://localhost:9200/_bulk' -d #1.json
I get this error:
{"error":{"root_cause":[{"type":"action_request_validation_exception","reason":"Validation Failed: 1: no requests added;"}],"type":"action_request_validation_exception","reason":"Validation Failed: 1: no requests added;"},"status":400}
I don't know where I am making a mistake.
The bulk API of Elasticsearch use a special syntax, which is actually made of json documents written on single lines. Take a look to the documentation.
The syntax is pretty simple. For indexing, creating and updating you need 2 single line json documents. The first lines tells the action, the second gives the document to index/create/update. To delete a document, it is only needed the action line. For example (from the documentation):
{ "index" : { "_index" : "test", "_type" : "type1", "_id" : "1" } }
{ "field1" : "value1" }
{ "create" : { "_index" : "test", "_type" : "type1", "_id" : "3" } }
{ "field1" : "value3" }
{ "update" : {"_id" : "1", "_type" : "type1", "_index" : "index1"} }
{ "doc" : {"field2" : "value2"} }
{ "delete" : { "_index" : "test", "_type" : "type1", "_id" : "2" } }
Don't forget to end your file with a new line.
Then, to call the bulk api use the command:
curl -s -XPOST localhost:9200/_bulk --data-binary "#requests"
From the documentation:
If you’re providing text file input to curl, you must use the --data-binary flag instead of plain -d
Adding next line(Enter in case of postman or "\n" if you are using json as body in client API) did my work
I had a similar issue in that I wanted to delete a specific document of a specific type and via the above answer I managed to get my simple bash script working finally!
I have a file that has a document_id per line (document_id.txt) and using the below bash script I can delete documents of a certain type with the mentioned document_id's.
This is what the file looks like:
c476ce18803d7ed3708f6340fdfa34525b20ee90
5131a30a6316f221fe420d2d3c0017a76643bccd
08ebca52025ad1c81581a018febbe57b1e3ca3cd
496ff829c736aa311e2e749cec0df49b5a37f796
87c4101cb10d3404028f83af1ce470a58744b75c
37f0daf7be27cf081e491dd445558719e4dedba1
The bash script looks like this:
#!/bin/bash
es_cluster="http://localhost:9200"
index="some-index"
doc_type="some-document-type"
for doc_id in `cat document_id.txt`
do
request_string="{\"delete\" : { \"_type\" : \"${doc_type}\", \"_id\" : \"${doc_id}\" } }"
echo -e "${request_string}\r\n\r\n" | curl -s -XPOST "${es_cluster}/${index}/${doc_type}/_bulk" --data-binary #-
echo
done
The trick, after lots of frustration, was to use the -e option to echo and append \n\n to the output of echo before I piped it into curl.
And then in curl I then have the --data-binary option set to stop it stripping out the \n\n needed for the _bulk endpoint followed by the #- option to get it to read from stdin!
Was a weird mistake in my case. I was creating the bulkRequest Object and clearing it before inserting into ElasticSearch.
The line that was creating the issue.
bulkRequest.requests().clear();
My issue was also the missing \n. If you print it, it will parse it & interpret it as a newline (so it looks like you're missing a \n). In case that helps anyone
pseudocode:
document = '{"index": {"_index": "users", "_id": "1"}} \n {"first_name": "Bob"}'
print(document)
will print
{"index": {"_index": "users", "_id": "1"}}
{"first_name": "Bob"}
but that's ok -- as long as the string contains the \n separator then it should work
I want to index and search mysql database using elasticsearch & I followed this tutorial
https://github.com/jprante/elasticsearch-river-jdbc/wiki/Quickstart
At first I downloaded elasticsearch and installed river-jdbc in its plugin folder. then added mysql-jdbc inside ES_HOME/plugins/river-jdbc/ Then started elasticsearch and Started another terminal window, and created a new JDBC river with name my_jdbc_river with this curl command
curl -XPUT 'localhost:9200/_river/my_jdbc_river/_meta' -d '{
"type" : "jdbc",
"jdbc" : {
"driver" : "com.mysql.jdbc.Driver",
"url" : "jdbc:mysql://localhost:3306/bablool",
"user" : "root",
"password" : "babloo",
"sql" : "select * from details"
},
"index" : {
"index" : "jdbc",
"type" : "jdbc"
}
}'
I'm getting the following error:-
then when I run this command: curl -XGET 'localhost:9200/jdbc/jdbc/_search?pretty&q=*'
and I'm getting following error:
"error": "IndexMissingException[[jdbc] missing]", "status" : 404
And when I give this in my browser:
http://localhost:9201/_search?q=*
Im getting like this:
{
"took": 51,
"timed_out": false,
"_shards": {
"total": 1,
"successful": 1,
"failed": 0
},
"hits": {
"total": 1,
"max_score": 1.0,
"hits": [
{
"_index": "_river",
"_type": "my_jdbc_river",
"_id": "_meta",
"_score": 1.0,
"_source": {
"type": "jdbc",
"jdbc": {
"driver": "com.mysql.jdbc.Driver",
"url": "jdbc:mysql://localhost:3306/bablool",
"user": "root",
"password": "babloo",
"sql": "select * from details"
},
"index": {
"index": "jdbc",
"type": "jdbc"
}
}
}
]
}
}
Is mysql dB indexed? How can I Search in my Db?
I encountered similar problem and this is how I managed to solve the issue:
Firstly, I checked all the indices via http://localhost:9200/_cat/indices?v
Deleted all indices with health status as red (there was just one index _river with health status red)
This is how you delete in index curl -XDELETE 'localhost:9200/_river/'
Redo step 7 in link you mentioned https://github.com/jprante/elasticsearch-river-jdbc/wiki/Quickstart
Hope it solves your problem as well :) Good luck!!