This is the error:
curl: (1) Protocol "https" not supported or disabled in libcurl
!! Submission failed: unexpected error: input file does not exist
!! Please try again later.
I am using Windows 10.
I see a possibly relevant answer here, but I don't know where this code would be added within Octave.
The URL is changed. Use the new one in submissionUrl() function in lib/submitWithConfiguration.m file.
function submissionUrl = submissionUrl()
%submissionUrl = 'https://www-origin.coursera.org/api/onDemandProgrammingImmediateFormSubmissions.v1';
submissionUrl = 'https://www.coursera.org/api/onDemandProgrammingImmediateFormSubmissions.v1';
end
For check URL you can use curl in terminal.
curl -k 'https://www.coursera.org/api/onDemandProgrammingImmediateFormSubmissions.v1'
You must get something like {"message":"","statusCode":404}
With wrong URL you dose't get anything.
Try to use the patch that changes following lines in the response function of submitWithConfiguration.m:
params = {'jsonBody', body};
%responseBody = urlread(submissionUrl, 'post', params); OLD CODE
[code, responseBody] = system(sprintf('echo jsonBody=%s | curl -k -X POST -d #- %s', body, submissionUrl));
d #- takes data in a file on the current stdin (the echo fills in).
-k allows curl to perform "insecure" SSL
(see curl --help)
HTH
==================
your code is the one i have, but i'm W7.
Do another try by setting quotes around the url in :
function submissionUrl = submissionUrl()
submissionUrl =
'"https://www-origin.coursera.org/api/onDemandProgrammingImmediateFormSubmissions.v1"';
end
(caution use : ' " and " ' that will quote the "https://.." on the command line.)
If it doesn't work, do a direct call to coursera with a command line (cmd) :
curl -k "https://www-origin.coursera.org/api/onDemandProgrammingImmediateFormSubmissions.v1"
This will call coursera and, as there is no sent form , the site will respond with an html page with near the end ... Action not found ....
if this works, the pb is probably not inside curl, but somewhere else. let us know.
There was a typo in Answer #1, which was corrected in Answer #2.
The change is:
In the function,
function response = submitParts(conf, email, token, parts) Apply the following changes
Comment the line responseBody = urlread(submissionUrl, 'post', params);
Type the following in place of it,
[code, responseBody] = system(sprintf('echo jsonBody=%s | curl -k -XPOST -d #- %s', body, submissionUrl));
So the final code of the function looks like
function response = submitParts(conf, email, token, parts)
body = makePostBody(conf, email, token, parts);
submissionUrl = submissionUrl();
params = {'jsonBody', body};
#responseBody = urlread(submissionUrl, 'post', params);
[code, responseBody] = system(sprintf('echo jsonBody=%s | curl -k -XPOST -d #- %s', body, submissionUrl));
response = loadjson(responseBody);
end
Change the following in submitWithConfiguration.m:
curl -k -X POST
to
curl -k -XPOST
and try again.
I just ran into this issue on Windows 10 today. In my case the request was performing correctly but the curl command was outputting timing information by default which was throwing off the validation logic in the submission script.
The submission was succeeding, but if I printed the response string, it looked something like this:
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0
100 1562 100 548 100 1014 548 1014 0:00:01 --:--:-- 0:00:01 2082
100 1562 100 548 100 1014 548 1014 0:00:01 --:--:-- 0:00:01 2082
{"id":"Blablablabla","courseId":"Blablabla","itemId":"Blabla",...}
I noticed that it was using the curl command to make the request, so I added the --silent flag to the code that creates the curl command to execute in submitWithConfiguration.m (in my case on line 134).
% use urlread or curl to send submit results to the grader and get a response
function response = getResponse(url, body)
% try using urlread() and a secure connection
params = {'jsonBody', body};
[response, success] = urlread(url, 'post', params);
if (success == 0)
% urlread didn't work, try curl & the peer certificate patch
if ispc
% testing note: use 'jsonBody =' for a test case
json_command = sprintf('echo jsonBody=%s | curl --silent -k -X POST -d #- %s', body, url);
% ^^^^^^^^ this right here!!
else
% it's linux/OS X, so use the other form
json_command = sprintf('echo ''jsonBody=%s'' | curl --silent -k -X POST -d #- %s', body, url);
end
% get the response body for the peer certificate patch method
[code, response] = system(json_command);
% test the success code
if (code ~= 0)
fprintf('[error] submission with curl() was not successful\n');
end
end
end
Now the response looked like a more reasonable:
{"id":"Blablablabla","courseId":"Blablabla","itemId":"Blabla",...}
And the submittion completed successfully.
I was having the same problem. All I did to overcome this problem is, I changed the path to ex2 folder and it worked for me.
Related
I'm trying to build a job by passing JSON parameter for Jenkins through Linux CLI. But I'm not able to pass the parameters in JSON. I have used -g to turn off globbing. Still the issue persists. Any help would be appreciated.
curl -X POST -u username:password -g JENKINS_URL/job/Bulk_Job/buildWithParameters?serviceBranchJson="{"key1":"value1","key2":"value2"}"
Error message
Processing provided DSL script
{key1:value1,key2:value2}
groovy.json.JsonException: expecting '}' or ',' but got current char 'k' with an int value of 107
The current character read is 'k' with an int value of 107
expecting '}' or ',' but got current char 'k' with an int value of 107
line number 1
index number 1
{key1:value1,key2:value2}
Code snippet
import groovy.json.JsonSlurper
def serviceBranchJson = serviceBranchJson
println(serviceBranchJson)
Map servicesMap = new JsonSlurper().parseText(serviceBranchJson)
The problem is that your JSON is missing quotes around the keys and values. From your error message, the json is {key1:value1,key2:value2} and it should be: {"key1":"value1","key2":"value2"} to work properly.
Try to escape the quotes in your curl command, for example:
curl -u user:token "JENKINS_URL/job/Bulk_Job/buildWithParameters?serviceBranchJson={\"key1\":\"value1\",\"key2\":\"value2\"}"
Then it should work correctly with your pipeline code (I added println servicesMap to the end of the code):
Started by remote host ...
[Pipeline] Start of Pipeline
[Pipeline] echo
{"key1":"value1","key2":"value2"}
[Pipeline] echo
{key1=value1, key2=value2}
[Pipeline] End of Pipeline
Finished: SUCCESS
My overall task is constantly to collect data from UNIX system log file, filter it, prepare a json payload based on the filtered data and process the data by sending a post api call to another server.
I wonder if that can be done using let's say shell script to monitor the log file with tail, filter with grep to get the specific lines dumpted in another file. With cronjob to run another script which contruct a .json and send curl request with the json to external server.
Some details:
In the log file - connector.log I am interested in lines like:
2020-09-16T15:14:37,337 INFO (tomcat-http--131) [tenant-test;-;138.188.247.4;] com.vmware.horizon.adapters.passwordAdapter.PasswordIdpAdapter - Login: user123 - SUCCESS
These lines, I can collect by the below command:
tailf connector.log | grep 'PasswordIdpAdapter - Login\|FAILURE\|SUCCESS'
and probably dump them into a file:
tailf connector.log | grep 'PasswordIdpAdapter - Login\|FAILURE\|SUCCESS' > log_data.txt
I wonder at this point, is it possible to extract only specific fields from a line(not the whole line) from the connector.log , so one line in log_data.txt to look like(1, 4, 6, 7, 8):
1 2020-09-29T07:15:13,881 [tenant1;usrname#tenant1;10.93.231.5;] - username - SUCCESS
From that point, I need to write a script(maybe could be run by cronjob every minute)/or a command to construct the below json and send the request. One line - one request.
This is the example of the json:
{
"timestamp": "2020-09-16T15:24:35,377",
"tenant_name": "tenant-test",
"log_type": "SERVICE",
"log_entry": "Login: user123 - SUCCESS"
}
The field values that should be replaced already exist in the log line: timestamp(the 1st field, e.g. 2020-09-16T15:14:37,337), tenant_name(the 1st part of the 4th field, tenant-test) and the log_entry(the last four fields, e.g. Login: user123 - SUCCESS).
When the json is constructed, I'll send it by:
curl --header "Content-Type: application/json" --request POST --data \
$payload http://myservert:8080/api/requests
What is not clear to me, this script to get the data line by line from log_data.txt e.g.
and populate some of the fields to create the .json and send it to the server.
Thanks for your answers in advance,
Petko
Thanks #shellter for the awk idea. So, bash, awk, grep, cat, cut and curl did the job.
I've created a cronjob to execute the bash script on 5 min interval.
The script gets the last 5mins of log data, dump it to another file, reads the filtered data, prepare the payload and then executes the API call. Maybe it is stupid but it works.
#!/bin/bash
MONITORED_LOG="/var/logs/test.log"
FILTERED_DATA="/tmp/login/login_data.txt"
REST_HOST="https://rest-host/topics/logs-"
# dump the last 5 mins of log data(date format: 2020-09-28T10:52:28,334)
# to a file, filter for keywords FAILURE\|SUCCESS and NOT having 'lookup|SA'
# an example of data record taken: 1 2020-09-29T07:15:13,881 [tenant1;usrname#tenant1;10.93.231.5;] - username - SUCCESS
awk -v d1="$(date --date="-5 min" "+%Y-%m-%dT%H:%M:%S")" -v d2="$(date "+%Y-%m-%dT%H:%M:%S")" '$0 > d1 && $0 < d2' $MONITORED_LOG | grep 'FAILURE\|SUCCESS' | grep -v 'lookup\|SA-' | awk '{ print $2, $3, $5, $7}' | uniq -c > $FILTERED_DATA
## loop through all the filtered records and send an API call
cat $FILTERED_DATA | while read LINE; do
## preparing the variables
timestamp=$(echo $LINE | cut -f2 -d' ')
username=$(echo $LINE | cut -f5 -d' ')
log_entry=$(echo $LINE | cut -f7 -d' ')
# get the tenant name, split by ; and remove the first char [
tenant_name=$(echo $tenant_name | cut -f1 -d';')
tenant_name="${tenant_name:1}"
# preparing the payload
payload=$'{"records":[{"value":{"timestamp":"'
payload+=$timestamp
payload+=$'","tenant_name":"'
payload+=$tenant_name
payload+=$'","log_entry":"'
payload+=$log_entry
payload+=$'"}}]}'
echo 'payload: ' $payload
# send the api call to the server with dynamic construction of tenant name
curl -i -k -u 'api_user:3494ssdfs3' --request POST --header "Content-type:application/json" --data "$payload" "$REST_HOST$tenant_name"
done
I'm following this tutorial on how to set up a local gcloud functions environment. I have this script set up and deployed:
exports.helloGET = (req, res) => {
console.log(req.body);
// Example input: {“name”: “Junior”}
if (req.body.name === undefined) {
// This is an error case, as “name” is required.
res.status(400).send('No name defined!');
} else {
console.log(req.body.name);
res.status(200).send('Hello ' + req.body.name);
}
};
I tried testing this function on the Cloud GUI (the website) and it works when you test it with {"name": "Junior"}. It returns Hello Junior as expected.
I now want to be able to run the test locally from the command line, so referring to the docs, they give the example:
gcloud functions call helloWorld --data='{"message": "Hello World!"}'
However, when I try to run the following:
gcloud functions call helloGET --data='{"name": "Junior"}' --region=northamerica-northeast1
I keep getting
ERROR: (gcloud.functions.call) Invalid value for [--data]: Is not a valid JSON: Expecting property name enclosed in double quotes: line 1 column 2 (char 1)
Why is this happening? I'm doing exactly what the docs & the tutorial are doing.
I am unable to repro your issue (on Linux).
I wonder whether you're:
using a non-Linux OS and this an incompatibility
inadvertently including non-standard ' or "
Deployed:
gcloud functions deploy hello \
--entry-point=hello \
--runtime=nodejs12 \
--trigger-http \
--allow-unauthenticated \
--region=${REGION} \
--project=${PROJECT}
Then:
gcloud functions call hello \
--data='{"name":"Freddie"}' \
--region=${REGION} \
--project=${PROJECT}
executionId: 7igxh5sfvjvk
result: Hello Freddie
And:
gcloud functions call hello \
--data='{"name":"Freddie"}' \
--region=${REGION} \
--project=${PROJECT} \
--log-http
=======================
==== request start ====
uri: https://cloudfunctions.googleapis.com/v1/projects/.../functions/hello:call?alt=json
method: POST
== headers start ==
b'Authorization': --- Token Redacted ---
b'accept': b'application/json'
b'accept-encoding': b'gzip, deflate'
b'content-length': b'34'
b'content-type': b'application/json'
== headers end ==
== body start ==
{"data": "{\"name\":\"Freddie\"}"}
== body end ==
==== request end ====
---- response start ----
status: 200
-- headers start --
...
-- body start --
{
"executionId": "7igx3sfj3ate",
"result": "Hello Freddie"
}
-- body end --
total round trip time (request+response): 0.289 secs
---- response end ----
----------------------
executionId: 7igx3sfj3ate
result: Hello Freddie
NOTE curl encodes the value of data as "{\"name\":\"Freddie\"}
And:
ENDPOINT=$(\
gcloud functions describe hello \
--project=${PROJECT} \
--format="value(httpsTrigger.url)")
TOKEN=$(gcloud auth print-access-token)
curl \
--request POST \
--header "Content-Type: application/json" \
--header "Authorization: Bearer ${TOKEN}" \
--data '{"name":"Freddie"}' \
${ENDPOINT}
Hello Freddie
I have a very simple 'hello world' Mochiweb server (I just started my introduction into it), which takes a JSON request and sends it back:
'POST' ->
case Path of
"dummy" ->
Data = Req:parse_post(),
Json = proplists:get_value("json", Data),
Struct = mochijson2:decode(Json),
Action_value = struct:get_value(<<"action">>, Struct),
Action = list_to_atom(binary_to_list(A)),
Result = [got_json_request, Action],
DataOut = mochijson2:encode(Result),
Req:ok({"application/json",[],[Result]});
The thing is that when I try to make a request to it with cURL it fails:
curl -i -H "Content-Type: application/json" -H "Accept: application/json" -X POST -d '{"action":"dummy"}' http://localhost:8080/dummy
The Mochiweb log is quite difficult to read, but as I understand the error happens at this line:
Json = proplists:get_value("json", Data)
I have put a couple of io:formats in the code and found out that both Data and Json variables are [] after I make a request with cURL.
On the other hand, when I do a very simple request with cURL:
curl -d '{"action":"dummy"}' http://localhost:8080/dummy
both Data and Json are [{"{\"action\":\"dummy\"}",[]}], but in that case the line Struct = mochijson2:decode(Json) fails.
For some strange reason Mochiweb does not see the JSON data in the POST request in case the header has the "application/json" value.
So, the question is: How do I make a correct POST request with JSON data to a Mochiweb server?
EDIT: Json variable has the undefined value.
Try something along the lines of
Data = Req:recv_body(),
Json = mochijson2:decode(Data),
...
You should at least ensure method post and the content type ahead of this.
This is not about POST nor get.
It's about how you post your data to send to your server
When you send a json data to server, you need to make it as key=value
curl -d "key=value" "http://your.domain.com/path"
Therefore, if you want to post json as '{"action":"dummy"}', for GET request
curl -d "json='{\"action\":\"dummy\"}'" http://localhost:8080/dummy
For POST request as a file,
curl -F "json=#filename.json" http://localhost:8080/dummy
of course, when you send as a file, you need to read the posted file from the server side.
Well, that would be a rather obscure topic but I'll give it a try, maybe someone will know the answer.
I am writing a little remote Node.js client for the Transmission BitTorrent Client. Communication is handled via RPC using JSON objects.
Here is the specification.
And my code (written in CoffeeScript, if that is a problem I can provide the equivalent JavaScript code also, just didn't want to make this question too long to read), the important part:
runRemoteCommand: (params) ->
# convert JSON object to string
params = JSON.stringify params #, null, " "
# set request options
options =
host: #config.host
port: #config.port
path: #config.rpcPath
auth: "#{#config.username}:#{#config.password}"
headers:
'Content-Type': 'application/json'
'Content-Length': params.length
method: 'GET'
# we don't know the session id yet
sessionId = false
# wrapped in a function so it could be run two times, not really a finished solution
run = () =>
# If the session id is provided, set the header, as in 2.3.1 of the specs
if sessionId
options.headers["X-Transmission-Session-Id"] = sessionId
# define the request object and a callback for the response
request = #http.get options, (response) =>
# log everything for debug purposes
console.log "STATUS: #{response.statusCode}"
console.log "HEADERS: #{JSON.stringify response.headers}"
response.setEncoding 'utf8'
response.on "data", (data) =>
console.log "BODY: #{data}"
# if status code is 409, use provided session id
if response.statusCode == 409
sessionId = response.headers["x-transmission-session-id"]
console.log "sessionId: #{sessionId}"
# running it immediately sometimes caused the remote server to provide a 501 error, so I gave it a timeout
setTimeout run, 5000
# no output here
request.on "error", (e) =>
console.log "ERROR: #{e}"
# actually send the request
request.write params
request.end()
# run our function
run()
The params variable is defined as:
params =
"arguments":
"filename": link
"method": "torrent-add"
"tag": 6667
Everything works fine until I set a valid session id. On the first time the run function is called, I get the following output (formatted it a little to be more eye-friendly):
STATUS: 409
HEADERS:
{
"server":"Transmission",
"x-transmission-session-id":"io4dOLm8Q33aSCEULW0iv74SeewJ3w1tP21L7qkdS4QktIkR",
"date":"Wed, 04 Apr 2012 08:37:37 GMT",
"content-length":"580",
"content-type":"text/html; charset=ISO-8859-1"
}
sessionId:
io4dOLm8Q33aSCEULW0iv74SeewJ3w1tP21L7qkdS4QktIkR
BODY: 409:
ConflictYour request had an invalid session-id
header.To fix this, follow these steps: When reading a
response, get its X-Transmission-Session-Id header and remember it
Add the updated header to your outgoing requests When you get this
409 error message, resend your request with the updated
headerThis requirement has been added to help prevent CSRF
attacks.X-Transmission-Session-Id:
io4dOLm8Q33aSCEULW0iv74SeewJ3w1tP21L7qkdS4QktIkR
Which is exactly what should be returned by the remote server when no session id is provided. However, after setting the session id in the header, the server doesn't respond. The second run call is fired and the request is sent (confirmed by placing some useful console.logs), but the response callback is never fired. I receive no response from the remote server and my application freezes waiting.
I'm pretty sure the error is on my side, not on the server's, because an out-of-the-box remote client for android works just fine when connecting to the same remote session.
Am I performing the request correctly? Especially the JSON part?
EDIT: A little test
I have written a little php script to test if the JSON-encoded request is ok and used it as a "fake" remote transmission. Here it is:
$headers = apache_request_headers();
// Simulate transmission's behavior
if (!isset($headers['X-Transmission-Session-Id'])) {
header("HTTP/1.0 409 Conflict");
header("X-Transmission-Session-Id: test");
}
print_r($headers);
// Is there a nicer way to get the raw request?
print_r(file_get_contents('php://input'));
And, personally, I don't see anything wrong in the data outputted by this test. After returning the 409 status code, the Node.js app properly assigns the session id for the request. The first print_r prints an array:
Array
(
[Content-type] => application/json
[Content-length] => 152
[X-Transmission-Session-Id] => test
[Host] => tp.localhost
[Connection] => keep-alive
)
The second one prints a string, which is a properly formatted JSON string (nothing more in it):
{
"arguments": {
"filename": "http://link-to-torrent"
},
"method": "torrent-add",
"tag": 6667
}
I really can't see what am I doing wrong. Some third-party clients which I tested with the same remote server work properly.
Havng the same issue i've done this class. I'm thinking better way do a getData, method. But it works.
http = require "http"
_ = require "underscore"
class Connect
constructor: (#login, #password, #host='127.0.0.1', #port=9091, #headers={}) ->
getData: (params)->
key = "x-transmission-session-id"
options = {
host: #host
port: #port
path: '/transmission/rpc',
method: 'POST',
headers: #headers || {},
auth: "#{ #login }:#{ #password }"
}
_.extend options, params || {}
req = http.request(options, (res)=>
if res.statusCode == 401
console.log "Auth errror"
else if res.statusCode == 409
auth_header={}
auth_header[key] = res.headers[key]
_.extend #headers, auth_header
#getData(params)
else if res.statusCode == 200
res.setEncoding 'utf8'
res.on('data', (chunk)->
#here should be an emmit of data
console.log chunk
)
else
console.log "Error #{ res.statusCode }"
)
req.write('data\n')
req.write('data\n')
req.end()
connector = new Connect "transmission", "password"
connector.getData()
Well, I was able to circumvent - but not solve - the problem, using mikeal's request, which also simplified my code. The most recent version looks like this:
runRemoteCommand: (params, callback = false) =>
options =
uri: #uri
method: "POST"
json: params
if #sessionId
options.headers =
"X-Transmission-Session-Id": #sessionId
request options, (error, response, body) =>
retVal =
success: false
end = true
if error
retVal.message = "An error occured: #{error}"
else
switch response.statusCode
when 409
if response.headers["x-transmission-session-id"]
#sessionId = response.headers["x-transmission-session-id"]
end = false
#.runRemoteCommand params, callback
else
retVal.message = "Session id not present"
when 200
retVal.success = true
retVal.response = body
else retVal.message = "Error, code: #{response.statusCode}"
callback retVal if end && callback
I'll leave this answer unaccepted for the time being because I still don't know what was wrong with the "raw" version.
#!/bin/bash
#-----------------------------------------------------------------------
#
DEBUG=0
HOST="192.168.1.65"
PORT="8181"
TRURL="http://$HOST:$PORT/transmission/rpc"
USER="admin"
PASSWORD="password1"
XTSID=""
#-------------------------------------
#
function getSID ()
{
local S="$1"
S=${S##*X-Transmission-Session-Id: }
S=${S%%</code>*}
echo $S
}
#-------------------------------------
function getData ()
{
local REQUEST="$1"
local RET=$(curl --silent -H "X-Transmission-Session-Id: $XTSID" \
-H "Content-type: application/json" \
-X POST \
-d "$REQUEST" \
--user $USER:$PASSWORD $TRURL)
((DEBUG)) && echo $XTSID
if [[ "$RET" =~ "409: Conflict" ]]
then
XTSID=$(getSID "$RET")
((DEBUG)) && echo "XTSID $XTSID"
RET=$(curl --silent -H "X-Transmission-Session-Id: $XTSID" \
-H "Content-type: application/json" \
-X POST \
-d "$REQUEST" \
--user $USER:$PASSWORD $TRURL)
fi
echo $RET
}
#-------------------------------------
R='{"method":"session-stats"}'
RET=$(getData "$R")
echo $RET