curl API calls with array of hashes - json

We are using a server software offering called FreezerPro (https://www.freezerpro.com/product-tour) with an API that can be called programmatically. There are simple methods like freezers that work with curl calls like this:
freezers -- Retrive a list of freezers
Returned objects: Freezers
Required parameters: None
Optional query parameters: None
Optional control parameters: None
curl -g --insecure 'https://username:password#demo-usa.freezerpro.com/api?method=freezers' | jq . | head -n 12
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 8697 0 8697 0 0 15980 0 --:--:-- --:--:-- --:--:-- 15987
{
"Freezers": [
{
"rfid_tag": "355AB1CBC00000700000075A",
"barcode_tag": "7000001882",
"boxes": 0,
"subdivisions": 1,
"access": 0,
"description": "[1000000000]",
"name": "[1000000000]",
"id": 1882
},
Then there is a search_samples method that searches for any fields in samples given a query. E.g.:
search_samples -- search for samples:
Returned objects: Samples
Required parameters: None
Optional query parameters:
query = <filter text> optional search string to filter the results.
Optional control parameters:
start = <staring record>
limit = <limit number of records to retrieve>
sort = <sort_field>
dir = <ASC / DESC>
curl -g --insecure 'https://username:password#demo-usa.freezerpro.com/api?method=search_samples&query=111222333' | jq .
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 347 0 347 0 0 977 0 --:--:-- --:--:-- --:--:-- 977
{
"Samples": [
{
"created_at": "06/11/2018",
"owner_id": 45,
"owner": "<span ext:qtip=\"username\">username</span>",
"description": "test",
"sample_id": 53087,
"id": 53087,
"loc_id": 54018,
"type": "cfDNA",
"scount": 1,
"name": "123456AB",
"location": "ER111→Level 1→Level 2→test001 (1)",
"icon": "images/box40/i53.png"
}
],
"Total": 1
}
So far so good. The problem comes when trying to run the advanced_search query, which takes an array of hashes in the query section. Given the sample above, which has a udf called patient_id with value 111222333, and advanced_search query for udf patient_id value=111222333 should return something, but it just gives a blank result:
Example command:
curl -g --insecure 'https://username:password#demo-usa.freezerpro.com/api?method=advanced_search&subject_type=Sample&query=[{type="udf",field="patient_id",value=111222333}]'
I am using:
curl --version
curl 7.35.0 (x86_64-pc-linux-gnu) libcurl/7.35.0 OpenSSL/1.0.1f zlib/1.2.8 libidn/1.28 librtmp/2.3
Protocols: dict file ftp ftps gopher http https imap imaps ldap ldaps pop3 pop3s rtmp rtsp smtp smtps telnet tftp
Features: AsynchDNS GSS-Negotiate IDN IPv6 Largefile NTLM NTLM_WB SSL libz TLS-SRP
Is this something to do with the way curl works on interpreting/passing the query section of the URL?
Any ideas about how to construct the query? Is this a curl specific issue?
EDIT: tried curl urlencode, it complains about the query not being setup:
curl -g -G --insecure 'https://username:password#demo-usa.freezerpro.com/api' --data-urlencode 'method=advanced_search' --data-urlencode 'query=[{type="udf",field="patient_id",value=111222333}]'
{"error":true,"message":"Query or search conditions must be specified","success":false}

You must URL-encode values of your URL parameters. e.g.
curl -g --insecure 'https://username:password#demo-usa.freezerpro.com/api?method=advanced_search&subject_type=Sample&query=%5B%7Btype%3D%22udf%22%2Cfield%3D%22patient_id%22%2Cvalue%3D111222333%7D%5D'
Also please run curl with -v parameter to make it verbose, so we could at least know what HTTP status is returned.

I've found a solution using the --data flag together with the -k flag:
curl -k --header "Content-Type: application/json" --request GET --data '{"username":"user", "password":"password", "method":"advanced_search", "query":[{"type":"udf","field":"patient_id","op":"eq","value":"111222333"}], "udfs":["patient_id","other"]}' https://demo-usa.freezerpro.com/api | jq .

Related

Extract data from unix log file, construct JSON and perform post request using curl

My overall task is constantly to collect data from UNIX system log file, filter it, prepare a json payload based on the filtered data and process the data by sending a post api call to another server.
I wonder if that can be done using let's say shell script to monitor the log file with tail, filter with grep to get the specific lines dumpted in another file. With cronjob to run another script which contruct a .json and send curl request with the json to external server.
Some details:
In the log file - connector.log I am interested in lines like:
2020-09-16T15:14:37,337 INFO (tomcat-http--131) [tenant-test;-;138.188.247.4;] com.vmware.horizon.adapters.passwordAdapter.PasswordIdpAdapter - Login: user123 - SUCCESS
These lines, I can collect by the below command:
tailf connector.log | grep 'PasswordIdpAdapter - Login\|FAILURE\|SUCCESS'
and probably dump them into a file:
tailf connector.log | grep 'PasswordIdpAdapter - Login\|FAILURE\|SUCCESS' > log_data.txt
I wonder at this point, is it possible to extract only specific fields from a line(not the whole line) from the connector.log , so one line in log_data.txt to look like(1, 4, 6, 7, 8):
1 2020-09-29T07:15:13,881 [tenant1;usrname#tenant1;10.93.231.5;] - username - SUCCESS
From that point, I need to write a script(maybe could be run by cronjob every minute)/or a command to construct the below json and send the request. One line - one request.
This is the example of the json:
{
"timestamp": "2020-09-16T15:24:35,377",
"tenant_name": "tenant-test",
"log_type": "SERVICE",
"log_entry": "Login: user123 - SUCCESS"
}
The field values that should be replaced already exist in the log line: timestamp(the 1st field, e.g. 2020-09-16T15:14:37,337), tenant_name(the 1st part of the 4th field, tenant-test) and the log_entry(the last four fields, e.g. Login: user123 - SUCCESS).
When the json is constructed, I'll send it by:
curl --header "Content-Type: application/json" --request POST --data \
$payload http://myservert:8080/api/requests
What is not clear to me, this script to get the data line by line from log_data.txt e.g.
and populate some of the fields to create the .json and send it to the server.
Thanks for your answers in advance,
Petko
Thanks #shellter for the awk idea. So, bash, awk, grep, cat, cut and curl did the job.
I've created a cronjob to execute the bash script on 5 min interval.
The script gets the last 5mins of log data, dump it to another file, reads the filtered data, prepare the payload and then executes the API call. Maybe it is stupid but it works.
#!/bin/bash
MONITORED_LOG="/var/logs/test.log"
FILTERED_DATA="/tmp/login/login_data.txt"
REST_HOST="https://rest-host/topics/logs-"
# dump the last 5 mins of log data(date format: 2020-09-28T10:52:28,334)
# to a file, filter for keywords FAILURE\|SUCCESS and NOT having 'lookup|SA'
# an example of data record taken: 1 2020-09-29T07:15:13,881 [tenant1;usrname#tenant1;10.93.231.5;] - username - SUCCESS
awk -v d1="$(date --date="-5 min" "+%Y-%m-%dT%H:%M:%S")" -v d2="$(date "+%Y-%m-%dT%H:%M:%S")" '$0 > d1 && $0 < d2' $MONITORED_LOG | grep 'FAILURE\|SUCCESS' | grep -v 'lookup\|SA-' | awk '{ print $2, $3, $5, $7}' | uniq -c > $FILTERED_DATA
## loop through all the filtered records and send an API call
cat $FILTERED_DATA | while read LINE; do
## preparing the variables
timestamp=$(echo $LINE | cut -f2 -d' ')
username=$(echo $LINE | cut -f5 -d' ')
log_entry=$(echo $LINE | cut -f7 -d' ')
# get the tenant name, split by ; and remove the first char [
tenant_name=$(echo $tenant_name | cut -f1 -d';')
tenant_name="${tenant_name:1}"
# preparing the payload
payload=$'{"records":[{"value":{"timestamp":"'
payload+=$timestamp
payload+=$'","tenant_name":"'
payload+=$tenant_name
payload+=$'","log_entry":"'
payload+=$log_entry
payload+=$'"}}]}'
echo 'payload: ' $payload
# send the api call to the server with dynamic construction of tenant name
curl -i -k -u 'api_user:3494ssdfs3' --request POST --header "Content-type:application/json" --data "$payload" "$REST_HOST$tenant_name"
done

Parsing curl output [duplicate]

This question already has answers here:
Parsing JSON with Unix tools
(45 answers)
Find the value of key from JSON
(5 answers)
Closed 2 years ago.
I'm trying to parse a curl request and parse the output and store it on a file called res.txt
Here is my bash cmd line:
curl --request POST --url 'https://www.virustotal.com/vtapi/v2/url/scan' --data 'apikey=XXXXXXXXXXXXXXX' --data 'url=abcde.xyz' >> grep -Po '"scan_id":.*?[^\\]",' res.txt
The output is something like this:
{"permalink": "https://www.virustotal.com/gui/url/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/detection/u-17f485d68047604e61b4067310ab716ae6fddc774bb46ffab06d081613b28e49-1595992331", "resource": "http://abcde.xyz/", "url": "http://abcde.xyz/", "response_code": 1, "scan_date": "2020-07-29 03:12:11", "scan_id": "000000000000000000000000000000000000000", "verbose_msg": "Scan request successfully queued, come back later for the report"}`
I want to store scan_id code on res.txt, but it is not working, no errors! And I do not know if my regex is correct
Can you help me?
The core of the question is about extracting values from JSON data (created by curl, in this specific case).
While it is possible to parse specific JSON data using regular expressiosns (assuming specific structure of while spaces/line breaks), it is very hard (impossible ?) to write regular expression that will cover all possible formatting. This is similar to parsing XML data - some formats can be parsed with regex, but extremely hard to write generic parser.
Instead of regex, consider using JSON specific tool, e.g., jq
Also, there construction of the pipe (curl to grep) should use '|' and not '>>', and the '>' should be used to specify the name of the file result. See below:
curl --request POST --url 'https://www.virustotal.com/vtapi/v2/url/scan' --data 'apikey=XXXXXXXXXXXXXXX' --data 'url=abcde.xyz' |
jq .scan_id > res.txt
To remove the quotes from the res.txt ,use the 'raw-output format of jq (jq -r .scan_id`)
If not possible to use jq for any reason, consider the following modification. It is using 'sed' (instead of grep) to extract the scan_id value (0000... in this case). It assumes that that the "scan_id" tag and value are on the same line.
curl --request POST --url 'https://www.virustotal.com/vtapi/v2/url/scan' --data 'apikey=XXXXXXXXXXXXXXX' --data 'url=abcde.xyz' |
sed -n -e 's/.*"scan_id": *"\([^"]*\)".*/\1/p' > res.txt
Try
curl --request POST --url 'https://www.virustotal.com/vtapi/v2/url/scan' --data 'apikey=XXXXXXXXXXXXXXX' --data 'url=abcde.xyz'| tr ',' '\n' | grep scan_id
Demo:
$"http://abcde.xyz/", "url": "http://abcde.xyz/", "response_code": 1, "scan_date": "2020-07-29 03:12:11", "scan_id": "000000000000000000000000000000000000000", "verbose_msg": "Scan request successfully queued, come back later for the report"}' <
{"permalink": "https://www.virustotal.com/gui/url/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/detection/u-17f485d68047604e61b4067310ab716ae6fddc774bb46ffab06d081613b28e49-1595992331", "resource": "http://abcde.xyz/", "url": "http://abcde.xyz/", "response_code": 1, "scan_date": "2020-07-29 03:12:11", "scan_id": "000000000000000000000000000000000000000", "verbose_msg": "Scan request successfully queued, come back later for the report"}
$: "000000000000000000000000000000000000000", "verbose_msg": "Scan request successfully queued, come back later for the report"}' | tr ',' '\n' | grep scan_id <
"scan_id": "000000000000000000000000000000000000000"
$

Paypal Subscription Setup

I am trying to setup my three subscriptions using curl and im making sure the values are correct. Trying to do a request and get a response but my curl is not functioning corectly.
https://developer.paypal.com/docs/api/subscriptions/v1/#plans_create
My intention:
3 Plans to choose from: Elite (149 PHP/MONTH), Premium (349 PHP/MONTH), Luxury (549 PHP/MONTH)
1 month is 30 days as paypal said
id like to set it to auto renew monthly until the customer chooses to cancel it.
only one will be active at a time, if a user chooses another of the three while one is running, they will automatically be stopped getting billed and will be charged with the new one chosen. (eg: currently elite149, the subscription and renewal will change to 549luxury once chosen).
curl -v -X POST https://api.sandbox.paypal.com/v1/billing/plans \
-H "Content-Type: application/json" \
-H "Authorization: Basic account_clientid:account_secretcode" \
-H "PayPal-Request-Id: EPL-25012019-001" \
-d '{
"product_id": "MWC-2019EPL",
"name": "My White Card Subscription Plans",
"description": "MyWhiteCard Membership Levels",
"status": "ACTIVE",
"billing_cycles": [
{
"frequency": {
"interval_unit": "MONTH",
"interval_count": 1
},
"tenure_type": "REGULAR",
"sequence": 1,
"total_cycles": 999,
"pricing_scheme": {
"fixed_price": {
"value": "149",
"currency_code": "PHP"
}
}
},
{
"frequency": {
"interval_unit": "MONTH",
"interval_count": 1
},
"tenure_type": "REGULAR",
"sequence": 2,
"total_cycles": 999,
"pricing_scheme": {
"fixed_price": {
"value": "349",
"currency_code": "PHP"
}
}
},
{
"frequency": {
"interval_unit": "MONTH",
"interval_count": 1
},
"tenure_type": "REGULAR",
"sequence": 3,
"total_cycles": 999,
"pricing_scheme": {
"fixed_price": {
"value": "549",
"currency_code": "PHP"
}
}
}
],
"payment_preferences": {
"auto_bill_outstanding": true,
"setup_fee": {
"value": "0",
"currency_code": "PHP"
},
"setup_fee_failure_action": "CONTINUE",
"payment_failure_threshold": 3
},
"taxes": {
"percentage": "10",
"inclusive": false
}
}'
I just took out my account's client_id:secret
Live credentials are being used
My questions and concerns:
In the access token, do I need to put the "access_token$production$" and then the code given?
Can I manually create the Paypal request id and the product id?
Will the billing cycles be all on the same command or do I have to trigger this three times?
Is the setup fee the charge if a customer subscribes? I first assumed that is the case and set it to 0.
My intention is to have the monthly subscription (30 Days) auto renew until the user unsubscribes. I set my example to 12 but will "total_cycles": NEVER be the correct input?
I am not sure how the tax part works, why am I the one that gets to modify it?
Additional Concerns:
The document does not show the live equivalent of the link https://api.sandbox.paypal.com/v1/billing/plans is it just simply https://api.live.paypal.com/v1/billing/plans?
I tried to add the above code using git bash and curl but as I run it it shows 1008{"error":"invalid_client","error_description":"Client credentials are missing"}
Do I try to run the curl code in Git bash and not change my directory? I just start git and run the curl here:
Any help will be appreciated. I just making sure everything is what needs to be because these three subscriptions will go to a live website. I have to be certain only one subscription runs at a time.
UPDATE: I tried making it to curl -v -X POST https://api.production.paypal.com/v1/billing/plans and this one is not functionning either.
UPDATE: Can anyone show an example of a working curl sample request?
UPDATE: at the -H "Authorization: Basic account_clientid:account_secretcode", just to be clear I took out my id code there since I cant just show it in public. An example that I placed there is
-H "Authorization: Basic JAKRc85nJy2eMLq3aIV:01PvLC934xMAwLHqU4JqA89as4N"
UPDATE I tried to run this curl in git after reading the answers so far and somehow I still get and error. I made sure that the api is in api.paypal.com and the client id and secret id is the live version.
curl -v -X POST https://api.paypal.com/v1/billing/plans \
-H "Content-Type: application/json" \
-H "Authorization: Basic AR7nnwwotKOt4YdcGHZc0P2RVsRT67_Gf2hyrKyDl3ZgCKsikeKbXdQ9Fj-_21v4RulkXsgAASe7_VKv:EKwsdDo1ehtOOOSZCGMu1C9903qr4cQOOZI2rgFYhvugh2SO1V04q9MWY9SXwa352zBt1mGglLuWgR4D" \
-H "PayPal-Request-Id: MWC-2501E-001" \
-d '{
"product_id": "ELITE-2501149",
"name": "Elite Membership",
"description": "Elite Membership Monthly Plan",
"status": "ACTIVE",
"billing_cycles": [
{
"frequency": {
"interval_unit": "MONTH",
"interval_count": 1
},
"tenure_type": "REGULAR",
"sequence": 1,
"total_cycles": 999,
"pricing_scheme": {
"fixed_price": {
"value": "149",
"currency_code": "PHP"
}
}
}
],
"payment_preferences": {
"auto_bill_outstanding": true,
"setup_fee": {
"value": "0",
"currency_code": "PHP"
},
"setup_fee_failure_action": "CONTINUE",
"payment_failure_threshold": 3
},
"taxes": {
"percentage": "12",
"inclusive": false
}
}'
I get this error:
100 921 100 159 100 762 99 478 0:00:01 0:00:01 --:--:-- 578{"name":"INTERNAL_SERVICE_ERROR","debug_id":"1975a4fe9232","links":[{"href":"https://developer.paypal.com/docs/api/overview/#error","rel":"information_link"}]}
UPDATE
Read all the links and followed the steps, I think this is the only issue left
{"name":"NOT_AUTHORIZED","message":"Authorization failed due to insufficient permissions.","debug_id":"484a9d7460069","details":[{"issue":"PERMISSION_DENIED","description":"You do not have permission to access or perform operations on this resource"}],"links":[{"href":"https://developer.paypal.com/docs/api/v1/billing/subscriptions#NOT_AUTHORIZED","rel":"information_link","method":"GET"}]}
I checked the Paypal help center and it seems that I need someone from Paypal itself to authorize my REST.
Q1 - In the access token, do I need to put the "access_token$production$" and then the code given?
In your example, you are using Basic authentication scheme. You can use:
-H "Authorization: Bearer Access-Token" \
But to answer your question, no you don't need to specifically include the access_token text string.
Q2 - Can I manually create the Paypal request id and the product id?
Yes, request id is used so you can retry your API calls.
https://developer.paypal.com/docs/api/reference/api-requests/#http-request-headers
HOWEVER product_id should come from https://developer.paypal.com/docs/api/catalog-products/v1/#products_create
so you will need to do it first. See my update at the bottom part of this answer.
Q3 - Will the billing cycles be all on the same command or do I have to trigger this three times?
You will need to call each plan configuration to create three subscription plans. Basically, in the API doc example, it shows you it has a billing cycle created for a Trial period and the regular plan it self. It also says:
https://developer.paypal.com/docs/api/subscriptions/v1/#plans_create
An array of billing cycles for trial and regular billing. A plan can
have multiple billing cycles but only one regular billing cycle.
You also mentioned:
only one will be active at a time, if a user chooses another of the
three while one is running, they will automatically be stopped getting
billed and will be charged with the new one chosen. (eg: currently
elite149, the subscription and renewal will change to 549luxury once
chosen).
You'll need to do this programmatically on your end.
If a user unsubscribes to a plan, then you will need to cancel his subscription by calling:
https://developer.paypal.com/docs/api/subscriptions/v1/#subscriptions_cancel
If you plan to change subscription, then first you need to cancel existing, then subscribe the user to the new subscription plan using: https://developer.paypal.com/docs/api/subscriptions/v1/#subscriptions_create
Q4 - Is the setup fee the charge if a customer subscribes? I first assumed that is the case and set it to 0.
The setup fee is just an add-on fee you can charge your customers. You will need to manually declare this or make it optional by making setup fee value 0.
https://developer.paypal.com/docs/api/subscriptions/v1/#definition-payment_preferences
Q5 - My intention is to have the monthly subscription (30 Days) auto renew until the user unsubscribes. I set my example to 12 but will "total_cycles": NEVER be the correct input?
The maximum value is 999. It only accepts integer.
https://developer.paypal.com/docs/api/subscriptions/v1/#definition-billing_cycle
Q6 - I am not sure how the tax part works, why am I the one that gets to modify it?
Because tax is dependent on what region or country you are in. You are using PHP or Philippine Peso as currency so it could mean that you are in the Philippines. You will need to setup your tax percentage as 12% as that is what is used for taxing goods and services
Source: https://www.full-suite.com/blog/whats-difference-vat-percentage-tax/
Paypal tax object definition: https://developer.paypal.com/docs/api/subscriptions/v1/#definition-taxes
Q7 - The document does not show the live equivalent of the link https://api.sandbox.paypal.com/v1/billing/plans is it just simply https://api.live.paypal.com/v1/billing/plans?
You can find the addresses here:
https://developer.paypal.com/docs/api/overview/#get-an-access-token
It says:
Sandbox: https://api.sandbox.paypal.com
Live: https://api.paypal.com
Q8 - I tried to add the above code using git bash and curl but as I run it it shows 1008{"error":"invalid_client","error_description":"Client credentials are missing"}
It could mean either mean that you are passing invalid credentials. Make sure that you are sending them via the headers using
Bearer <Access-Token>
or
Basic <client_id>:<secret>
And verify that your string input represents the actual values.
Also, make sure you are using Sandbox credentials as it has a different set of credentials from the production or live
https://developer.paypal.com/docs/classic/lifecycle/sb-create-accounts/#create-a-sandbox-account
Q9 - Do I try to run the curl code in Git bash and not change my directory? I just start git and run the curl here:
Do not misinterpret Git Bash as the command line interface, it is simply a versioning tool for your projects. However the Git Package for windows has built-in components that should allow you to run CURL. Since Curl is installed in the Bin directory, you should be able to run it on any directory.
You can run the curl command using different tools ideally using a scripting or programming language like PHP, Phyton, Java or even Node which has better support for curl and should allow you to write and test your program easier in a neater way.
UPDATE
I've investigated on this further. I thought I'd share it with you because it seems you haven't read the whole API document yet.
You are creating Subscription plans however you'll need to Create the products first. (I have updated my answer to Question #2)
First - Create the Product
This will create the product_id you need to create your subscription plans.
https://developer.paypal.com/docs/api/catalog-products/v1/#products_create
To go back and check the product you created an get its product id, you can use this api call:
https://developer.paypal.com/docs/api/catalog-products/v1/#products_list
Then - Create The Subscription Plan
After you have created your products, you can then create multiple subscription plans for it.
https://developer.paypal.com/docs/api/subscriptions/v1/#plans_create
Moving forward, if you want to subscribe / unsubscribe users you'll need to programatically do it as per my answer to Question #3.
About your problem with invalid credentials
Try getting the access token first. To do that follow instructions here:
https://developer.paypal.com/docs/api/get-an-access-token-curl/
curl -v https://api.sandbox.paypal.com/v1/oauth2/token \
-H "Accept: application/json" \
-H "Accept-Language: en_US" \
-u "client_id:secret" \
-d "grant_type=client_credentials"
Then get the token returned and use that to make your api calls.
It will return something like this:
{
"scope": "scope",
"access_token": "Access-Token",
"token_type": "Bearer",
"app_id": "APP-80W284485P519543T",
"expires_in": 31349,
"nonce": "nonce"
}
Here is a modified version of your code that I used to test, notice it uses Bearer token instead basic client_id:secret
curl -v -X POST https://api.sandbox.paypal.com/v1/billing/plans \
-H "Content-Type: application/json" \
-H "Authorization: Bearer Access-Token-goeshere" \
-H "PayPal-Request-Id: MWC-helper" \
-d '{
"product_id": "PROD-ELITETEST", //<---- this product id should be taken from the real product id when you created your product through product_create api.
"name": "Elite Membership",
"description": "Elite Membership Monthly Plan",
"status": "ACTIVE",
"billing_cycles": [
{
"frequency": {
"interval_unit": "MONTH",
"interval_count": 1
},
"tenure_type": "REGULAR",
"sequence": 1,
"total_cycles": 999,
"pricing_scheme": {
"fixed_price": {
"value": "149",
"currency_code": "PHP"
}
}
}
],
"payment_preferences": {
"auto_bill_outstanding": true,
"setup_fee": {
"value": "0",
"currency_code": "PHP"
},
"setup_fee_failure_action": "CONTINUE",
"payment_failure_threshold": 3
},
"taxes": {
"percentage": "12",
"inclusive": false
}
}'
It worked for me!

Access Next Offset from GET http api request

I'm unable to understand pagination of Chargebee (https://apidocs.chargebee.com/docs/api) , I need to create a request in where I can add next offset to the request to get further data (without setting limit other then by default, which is 10). But i'm unable to understand how http request will be formed with this given next_offset like attached image.
Screenshot of request and response
I have had success submitting the identical request with the offset tacked onto the end. Here is an example:
First Request
curl -s https://xxyyzz.chargebee.com/api/v2/events -G -K /home/xxyyzz/.cb_curl_key.cfg --data-urlencode limit=2 --data-urlencode occurred_at[between]="[1554076800,1554077099]"
JSON results:
{
"list": [
{"event": {
"id": "ev_xxyyzz1",
"occurred_at": 1554077022,
...
"next_offset": "[\"1556428868000\",\"364450353\"]"
}
The tool I used to display the json is trying to be helpful with the backslashes.
The real next_offset value is
["1556428868000","364450353"]
Second Request
Identical to the first with
offset="["1554077017000","345017569"]"
tacked on to the end:
curl -s https://xxyyzz.chargebee.com/api/v2/events -G -K /home/xxyyzz/.cb_curl_key.cfg --data-urlencode limit=2 --data-urlencode --data-urlencode occurred_at[between]="[1554076800,1554077099]" offset="["1554077017000","345017569"]"
JSON results:
{
"list": [
{"event": {
"id": "ev_xxyyzz3",
"occurred_at": 1556429028,
}
Keep repeating the process until the "next_offset" key does not appear in the JSON result.

Failure parsing JSON with mongoimport

I get an Assertion: 10340:Failure parsing JSON string error, running mongoimport in pipe over Github API, like the following:
lsoave#ubuntu:~/rails/github/gitwatcher$ curl https://api.github.com/users/lgs/repos | mongoimport -h localhost -d gitwatch_dev -c repo -f repositories
connected to: localhost
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0Mon Jun 20 00:56:01 Assertion: 10340:Failure parsing JSON string near: [
100 22303 100 22303 0 0 31104 0 --:--:-- --:--:-- --:--:-- 111k
0x816d8a1 0x8118814 0x84b357a 0x84b5bb8 0x84adc65 0x84b2ee1 0x60bbd6 0x80f5bc1
mongoimport(_ZN5mongo11msgassertedEiPKc+0x221) [0x816d8a1]
mongoimport(_ZN5mongo8fromjsonEPKcPi+0x3b4) [0x8118814]
mongoimport(_ZN6Import9parseLineEPc+0x7a) [0x84b357a]
mongoimport(_ZN6Import3runEv+0x1a98) [0x84b5bb8]
mongoimport(_ZN5mongo4Tool4mainEiPPc+0x1ce5) [0x84adc65]
mongoimport(main+0x51) [0x84b2ee1]
/lib/tls/i686/cmov/libc.so.6(__libc_start_main+0xe6) [0x60bbd6]
mongoimport(__gxx_personality_v0+0x3f1) [0x80f5bc1]
exception:Failure parsing JSON string near: [
[
...
...
Mon Jun 20 00:45:20 Assertion: 10340:Failure parsing JSON string near: "name": "t
0x816d8a1 0x8118814 0x84b357a 0x84b5bb8 0x84adc65 0x84b2ee1 0x126bd6 0x80f5bc1
mongoimport(_ZN5mongo11msgassertedEiPKc+0x221) [0x816d8a1]
mongoimport(_ZN5mongo8fromjsonEPKcPi+0x3b4) [0x8118814]
mongoimport(_ZN6Import9parseLineEPc+0x7a) [0x84b357a]
mongoimport(_ZN6Import3runEv+0x1a98) [0x84b5bb8]
mongoimport(_ZN5mongo4Tool4mainEiPPc+0x1ce5) [0x84adc65]
mongoimport(main+0x51) [0x84b2ee1]
/lib/tls/i686/cmov/libc.so.6(__libc_start_main+0xe6) [0x126bd6]
mongoimport(__gxx_personality_v0+0x3f1) [0x80f5bc1]
exception:Failure parsing JSON string near: "name": "t
"name": "tentacles"
...
...
see the full trace here: http://pastie.org/2093486. Anyway, the json format I get back from the Github API seems ok ( curl https://api.github.com/users/lgs/repos ):
[
{
"open_issues": 0,
"watchers": 3,
"homepage": "http://scrubyt.org",
"language": null,
"forks": 1,
"pushed_at": "2009-02-25T22:49:08Z",
"created_at": "2009-02-25T22:22:40Z",
"fork": true,
"url": "https://api.github.com/repos/lgs/scrubyt",
"private": false,
"size": 188,
"description": "A simple to learn and use, yet powerful web scraping toolkit!",
"owner": {
"avatar_url": "https://secure.gravatar.com/avatar/9c7d80ebc20ab8994e51b9f7518909ae?d=https://a248.e.akamai.net/assets.github.com%2Fimages%2Fgravatars%2
Fgravatar-140.png",
"login": "lgs",
"url": "https://api.github.com/users/lgs",
"id": 1573
},
"name": "scrubyt",
"html_url": "https://github.com/lgs/scrubyt"
},
...
...
]
here it is a snippet: http://www.pastie.org/2093524.
If I try specifying csv format it works:
lsoave#ubuntu:~/rails/github/gitwatcher$ curl https://api.github.com/users/lgs/repos | mongoimport -h localhost -d gitwatch_dev -c repo -f repositories --type csv
connected to: localhost
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 22303 100 22303 0 0 23914 0 --:--:-- --:--:-- --:--:-- 106k
imported 640 objects
lsoave#ubuntu:~/rails/github/gitwatcher$
It worked for me by using "mongoimport --jsonArray ..."
Alright here is what could be going on. First off, I removed all the newlines in the JSON to reduce the number of errors from n (where n = number of lines) to 1. Then it turns out, I had to wrap the JSON Array in another variable and it worked thereafter. I think mongoimport is designed to work with mongoexport, so most likely you cannot use it to import any arbitrary JSON. However, if you want to, what I did woud be something you'd have to do in code before calling the import utility.
I used only 1 record while I was testing. Here is the record with no newlines.
[{"url":"https://api.github.com/repos/lgs/scrubyt", "pushed_at": "2009-02-25T22:49:08Z","homepage": "http://scrubyt.org", "forks": 1,"language": null,"fork": true,"html_url": "https://github.com/lgs/scrubyt","created_at": "2009-02-25T22:22:40Z", "open_issues": 0,"private": false,"size": 188,"watchers": 3,"owner": {"url": "https://api.github.com/users/lgs","login": "lgs","id": 1573,"avatar_url": "https://secure.gravatar.com/avatar/9c7d80ebc20ab8994e51b9f7518909ae?d=https://a248.e.akamai.net/assets.github.com%2Fimages%2Fgravatars%2Fgravatar-140.png"},"name": "scrubyt","description": "A simple to learn and use, yet powerful web scraping toolkit!"}]
Then I wrapped it with somedata (you can use any name here):
{somedata:[{"url":"https://api.github.com/repos/lgs/scrubyt", "pushed_at": "2009-02-25T22:49:08Z","homepage": "http://scrubyt.org", "forks": 1,"language": null,"fork": true,"html_url": "https://github.com/lgs/scrubyt","created_at": "2009-02-25T22:22:40Z", "open_issues": 0,"private": false,"size": 188,"watchers": 3,"owner": {"url": "https://api.github.com/users/lgs","login": "lgs","id": 1573,"avatar_url": "https://secure.gravatar.com/avatar/9c7d80ebc20ab8994e51b9f7518909ae?d=https://a248.e.akamai.net/assets.github.com%2Fimages%2Fgravatars%2Fgravatar-140.png"},"name": "scrubyt","description": "A simple to learn and use, yet powerful web scraping toolkit!"}]}
And I was able to see the record in Mongo.
> db.repo.findOne()
{
"_id" : ObjectId("4dff91d29c73f72483e82ef2"),
"somedata" : [
{
"url" : "https://api.github.com/repos/lgs/scrubyt",
"pushed_at" : "2009-02-25T22:49:08Z",
"homepage" : "http://scrubyt.org",
"forks" : 1,
"language" : null,
"fork" : true,
"html_url" : "https://github.com/lgs/scrubyt",
"created_at" : "2009-02-25T22:22:40Z",
"open_issues" : 0,
"private" : false,
"size" : 188,
"watchers" : 3,
"owner" : {
"url" : "https://api.github.com/users/lgs",
"login" : "lgs",
"id" : 1573,
"avatar_url" : "https://secure.gravatar.com/avatar/9c7d80ebc20ab8994e51b9f7518909ae?d=https://a248.e.akamai.net/assets.github.com%2Fimages%2Fgravatars%2Fgravatar-140.png"
},
"name" : "scrubyt",
"description" : "A simple to learn and use, yet powerful web scraping toolkit!"
}
]
}
Hope this helps!
this worked fine with me after I removed any '\n'. You can use tr in linux
cat file.json | tr -d '\n' > file.json
Using both the answers provided by #Daniel and #lobster1234 I created a script which I use to import the json entries into mongo.
#!/bin/sh
if [ -z "$1" ] ;
then
echo "missing argument"
exit -1
fi
FILE=${1%%.json}
echo $FILE
cat $FILE.json | tr -d '\n' > $FILE.import.json
mongoimport --collection collection --db main --file $FILE.import.json --jsonArray --upsert