I'm trying to use the go sdk to access coginto, and i'm not sure how to get a token, i'm creating my cogintoidentity with:
creds := credentials.NewStaticCredentials("-id-", "-secret-", "")
sess := session.Must(session.NewSession())
sess.Config.Credentials = creds
sess.Config.Region = aws.String("us-east-1")
sess.Config.CredentialsChainVerboseErrors = aws.Bool(true)
svc := CP.New(sess) // CP == "github.com/aws/aws-sdk-go/service/cognitoidentityprovider"
log.Println(svc.ListUserPools(&CP.ListUserPoolsInput{MaxResults: aws.Int64(10)}))
But it returns The security token included in the request is invalid.
I was using the pool's client id/secret, not my AWS's.
Related
I have just started learning ruby and trying to learn extracting web data. I am using samplewebapp code given in a sites API documentation
On executing the test.rb file as given below it generates a token.
The token.json file contains the following
{"access_token":"eyJ0eXAiOiJKV1QiLCJhbGciOiJSUzI1NiJ9.eyJhdWQiOiIzY2U1NjFhMC01ZmE5LTQ5NjUtYTQ2Mi01NmI2MzEzNjRlZGMiLCJqdGkiOiJmMjc0ZTgwM2FmZjVlMTFiMTAzNzIwMzAwNzRlNDRhZDE3ODg3ZjAzOGQ2ODk0OWIwNzVkMmQ5N2E2MjAwYzc5ZWRhNzU4MmQ3MTg1MTc1YiIsImlhdCI6MTY1NDU2OTc0MS41MjgxNTUsIm5iZiI6MTY1NDU2OTc0MS41MjgxNTcsImV4cCI6MTY1NDU3MzM0MS41MjgwMzksInN1YiI6IjE5NGFlNTU5LTU5OWMtNDQ2ZC1hOTRhLWM2MTIyYjZkMTA2ZiIsInNjb3BlcyI6W10sImNyZWRpdHNfcmVtYWluaW5nIjo0NzAwLCJyYXRlX2xpbWl0cyI6W3sicmF0ZSI6NSwiaW50ZXJ2YWwiOjYwfV19.XoN5A-H8Z7d8p_0e2rCcyV4yWO9MAF3TZlHk5VP5NjUEqtTXTVYKfKuzidYDs7K8ZzAtWzyt3aR5VSUG0_nw-uPd7Z3_Y75alQMqa2b5rHGn5JFWH7nuAriskr26WSCqAdj4cNjccPORryRr5sYKwpE4Y4kTG0IX_8frvwYxwp-8wFpLLj98K3axwN7CCpWdnPDSzuMqmH6tSpF0XhdYxB5LTVH0AyH7lN0S0_Lftq0b3sOLIvEaTfNkuRGNqwLkBYFkHFuPqwrKd8RJhC2W1QZhrmUw3eYnh-0iQABGk2V0skIqDlb6BbQ5GFX6MXgiXAc-h5Ndda7pZ5N5UCQU3g","expires_at":1654573341}
However it also generates a JSON parser error as under
/Users/mm/.rbenv/versions/3.0.4/lib/ruby/3.0.0/json/common.rb:216:in `parse': 809: unexpected token at '' (JSON::ParserError)
from /Users/mm/.rbenv/versions/3.0.4/lib/ruby/3.0.0/json/common.rb:216:in `parse'
from /Users/mm/Desktop/prokerala/client.rb:21:in `parseResponse'
from /Users/mm/Desktop/prokerala/client.rb:99:in `get'
I googled around for a while and some of the responses talk about the issues between double and single quotes.. I tried changing those also but it doesn't work.
I have also tried running it with system ruby - which gives a different error
Would appreciate very much if someone could explain what the error is / logic behind this.. Also I am not clear about the error message itself. The message talks about parse error 809:"unexpected token" on column no 809 of the token there is nothing.. Am i reading it wrong?
code used in test.rb file
client = ApiClient.new('YOUR CLIENT_ID', 'YOUR CLIENT_SECRET');
result = client.get('v2/astrology/thirumana-porutham/advanced', {
:girl_nakshatra => 4,
:girl_nakshatra_pada => 2,
:boy_nakshatra => 26,
:boy_nakshatra_pada => 3
})
puts JSON.pretty_generate(result)
code used in client.rb file
require 'net/http'
require 'json'
class ApiError < StandardError
end
class ApiClient
BASE_URL = "https://api.prokerala.com/"
# Make sure that the following file path is set to a location that is not publicly accessible
TOKEN_FILE = "./token.json"
def initialize(clientId, clientSecret)
# Instance variables
#clientId = clientId
#clientSecret = clientSecret
end
def parseResponse(response)
content = response.body
res = JSON.parse(content)
if res.key?('access_token')
return res
end
if res['status'] == "error"
raise ApiError, res['errors'].map {|e| e['detail']}.join("\n")
end
if res['status'] != "ok"
raise "HTTP request failed"
end
return res
end
def saveToken(token)
# Cache the token until it expires.
File.open(ApiClient::TOKEN_FILE,"w") do |f|
token = {
:access_token => token['access_token'],
:expires_at => Time.now.to_i + token['expires_in']
}
f.write(token.to_json)
end
end
def getTokenFromCache
if not File.file?(ApiClient::TOKEN_FILE)
return nil
end
begin
# Fetch the cached token, and return if not expired
text = File.read(ApiClient::TOKEN_FILE)
token = JSON.parse(text)
if token['expires_at'] < Time.now.to_i
return nil
end
return token['access_token']
rescue JSON::ParserError
return nil
end
end
def fetchNewToken
params = {
:grant_type => 'client_credentials',
:client_id => #clientId,
:client_secret => #clientSecret
}
res = Net::HTTP.post_form(URI(ApiClient::BASE_URL + 'token'), params)
token = parseResponse(res)
saveToken(token)
return token['access_token']
end
def get(endpoint, params)
# Try to fetch the access token from cache
token = getTokenFromCache
# If failed, request new token
token ||= fetchNewToken
uri = URI(ApiClient::BASE_URL + endpoint)
uri.query = URI.encode_www_form(params)
req = Net::HTTP::Get.new(uri.to_s, {'Authorization' => 'Bearer ' + token})
res = Net::HTTP.start(uri.hostname) do |http|
http.request(req)
end
return parseResponse(res)
end
end
running the test file using ruby 3.0.4 gives the a JSON parse error. Full error reproduced as below
/Users/mm/.rbenv/versions/3.0.4/lib/ruby/3.0.0/json/common.rb:216:in `parse': 809: unexpected token at '' (JSON::ParserError)
from /Users/mm/.rbenv/versions/3.0.4/lib/ruby/3.0.0/json/common.rb:216:in `parse'
from /Users/mm/Desktop/prokerala/client.rb:21:in `parseResponse'
from /Users/mm/Desktop/prokerala/client.rb:99:in `get'
from test.rb:11:in `<main>'
I executed the above code and from this, we are getting
#<Net::HTTPMovedPermanently 301 Moved Permanently readbody=true> as response
and response body as empty string '', which results in error in parsing the body.
Since, API endpoint is secure with https, we need set use_ssl: true before starting the session. Try below code and it will fix the issue.
res = Net::HTTP.start(uri.hostname, use_ssl: true) do |http|
http.request(req)
end
I am beginner in LUA. I have written the following code in lua file that mysql proxy run with it.
function read_query(packet)
if string.byte(packet) == proxy.COM_QUERY then
local command = string.lower(packet)
if string.find(command, "select") ~= nil and string.find(string.lower(packet), "from") ~= nil then
local socket = require('socket')
local conn, err = socket.connect('localhost', 5050)
print(conn, err)
proxy.response.type = proxy.MYSQLD_PACKET_OK
//proxy.response.resultset get json from web service (url)
proxy.response.resultset = {
fields = {
{ type = proxy.MYSQL_TYPE_INT, name = "id", },
},
rows = {
{ 9001 }
}
}
return proxy.PROXY_SEND_RESULT
end
end
end
I want to connect to the web service on Port 5050 that return the JSON file and save the json that it returns in the proxy.response.resultset.
Another question, How can i add socket module. I paste files like following image
socket module files
but give an error : can not find /socket/core.lua.
local socket = require('socket')
You are using luasocket and it comes as a mix of lua (socket.lua) and binary (socket/core.so) files. You need to set (if it's not set already) to point to .so files; something like this may work: package.cpath=package.cpath..';./?.so'
I am using http.get() to get a JSON from an API I am using, but it's not getting the data. I have the suspicion that this JSON is too big for the NodeMCU. I only need the information in the subpart "stats:". Is it possible to only http.get() that part of the JSON?
EDIT:
This is my code
function getstats()
http.get("https://api.aeon-pool.com/v1/stats_address?address=WmsGUrXTR7sgKmHEqRNLgPLndWKSvjFXcd4soHnaxVjY3aBWW4kncTrRcBJJgUkeGwcHfzuZABk6XK6qAp8VmSci2AyGHcUit", nil, function(code, pool)
if (code < 0) then
print("can't get stats")
else
h = cjson.decode(pool)
hashrate = h[1]["hashrate"]
print(hashrate)
dofile('update_display.lua')
end
end)
end
I also have another function getting data from another api above getstats()
function getaeonrate()
http.get("https://api.coinmarketcap.com/v1/ticker/aeon/?convert=EUR", nil, function(code, dataaeon)
if (code < 0) then
print("can't get aeon")
else
-- Decode JSON data
m = cjson.decode(dataaeon)
-- Extract AEON/EUR price from decoded JSON
aeonrate = string.format("%f", m[1]["price_eur"]);
aeonchange = "24h " .. m[1]["percent_change_24h"] .. "% 7d " .. m[1]["percent_change_7d"] .. "%"
dofile('update_display.lua')
end
end)
end
But now the weird thing is, when I want to access 'pool' from getstats() I get the json data from getaeonrate(). So "hashrate" isn't even in the json because I am getting the json from another function.
I tried making a new project only with getstats() and that doesn't work at all I always get errors like this
HTTP client: Disconnected with error: -9
HTTP client: Connection timeout
HTTP client: Connection timeout
Yesterday I thought that the response was too big from api.aeon-pool.com, I if you look at the json in your webbrowser you can see that the top entry is 'stats:' and I only need that, none of the other stuff. So If the request is to big It would be nice to only http.get() that part of the json, hence my original question. At the moment I am not even sure what is not working correctly, I read that the nodemcu firmware generally had problems with http.get() and that it didn't work correctly for a long time, but getting data from api.coinmarketcap.com works fine in the original project.
The problems with the HTTP module are with near certainty related to https://github.com/nodemcu/nodemcu-firmware/issues/1707 (SSL and HTTP are problematic).
Therefore, I tried with the more bare-bone TLS module on the current master branch. This means you need to manually parse the HTTP response including all headers looking for the JSON content. Besides, you seem to be on an older NodeMCU version as you're still using CJSON - I used SJSON below:
Current NodeMCU master branch
function getstats()
buffer = nil
counter = 0
local srv = tls.createConnection()
srv:on("receive", function(sck, payload)
print("[stats] received data, " .. string.len(payload))
if buffer == nil then
buffer = payload
else
buffer = buffer .. payload
end
counter = counter + 1
-- not getting HTTP content-length header back -> poor man's checking for complete response
if counter == 2 then
print("[stats] done, processing payload")
local beginJsonString = buffer:find("{")
local jsonString = buffer:sub(beginJsonString)
local hashrate = sjson.decode(jsonString)["stats"]["hashrate"]
print("[stats] hashrate from aeon-pool.com: " .. hashrate)
end
end)
srv:on("connection", function(sck, c)
sck:send("GET /v1/stats_address?address=WmsGUrXTR7sgKmHEqRNLgPLndWKSvjFXcd4soHnaxVjY3aBWW4kncTrRcBJJgUkeGwcHfzuZABk6XK6qAp8VmSci2AyGHcUit HTTP/1.1\r\nHost: api.aeon-pool.com\r\nConnection: close\r\nAccept: */*\r\n\r\n")
end)
srv:connect(443, "api.aeon-pool.com")
end
Note that the receive event is fired for every network frame: https://nodemcu.readthedocs.io/en/latest/en/modules/net/#netsocketon
NodeMCU fails to establish a connection to api.coinmarketcap.com due to a TLS handshake failure. Not sure why that is. Otherwise your getaeonrate() could be implemented likewise.
Frozen 1.5.4 branch
With the old branch the net module can connect to coinmarketcap.com.
function getaeonrate()
local srv = net.createConnection(net.TCP, 1)
srv:on("receive", function(sck, payload)
print("[aeon rate] received data, " .. string.len(payload))
local beginJsonString = payload:find("%[")
local jsonString = payload:sub(beginJsonString)
local json = cjson.decode(jsonString)
local aeonrate = string.format("%f", json[1]["price_eur"]);
local aeonchange = "24h " .. json[1]["percent_change_24h"] .. "% 7d " .. json[1]["percent_change_7d"] .. "%"
print("[aeon rate] aeonrate from coinmarketcap.com: " .. aeonrate)
print("[aeon rate] aeonchange from coinmarketcap.com: " .. aeonchange)
end)
srv:on("connection", function(sck, c)
sck:send("GET /v1/ticker/aeon/?convert=EUR HTTP/1.1\r\nHost: api.coinmarketcap.com\r\nConnection: close\r\nAccept: */*\r\n\r\n")
end)
srv:connect(443, "api.coinmarketcap.com")
end
Conclusion
The HTTP module and TLS seem a no-go for your APIs due to a bug in the firmware (1707).
The net/TLS module of the current master branch manages to connect to api.aeon-pool.com but not to api.coinmarketcap.com.
With the old and frozen 1.5.4 branch it's exactly the other way around.
There may (also) be issues with cipher suits that don't match between the firmware and the API provider(s).
-> :( no fun like that
What I have:
GNU/Linux host
nginx is up and running
there is a cron-job scheduled to run immediately after a specific file has been removed (similar to run-crons)
GitHub sends a webhook when someone pushes to a repository
What I want:
I do now want to run either lua or anything comparable to parse GitHub's request and validate it and then delete a file (if the request was valid of course).
Preferably all of this should happen without the hassle to maintain an additional PHP installation as there is currently none, or the need to use fcgiwrap or similar.
Template:
On the nginx side I have something equivalent to
location /deploy {
# execute lua (or equivalent) here
}
To read json body of GH webhook you nead use JSON4Lua lib, and to validate HMAC signature use luacrypto.
Preconfigure
Install required modules
$ sudo luarocks install JSON4Lua
$ sudo luarocks install luacrypto
In Nginx define location for deploy
location /deploy {
client_body_buffer_size 3M;
client_max_body_size 3M;
content_by_lua_file /path/to/handler.lua;
}
The max_body_size and body_buffer_size should be equal to prevent error
request body in temp file not supported
https://github.com/openresty/lua-nginx-module/issues/521
Process webhook
Get request payload data and check is correct
ngx.req.read_body()
local data = ngx.req.get_body_data()
if not data then
ngx.log(ngx.ERR, "failed to get request body")
return ngx.exit (ngx.HTTP_BAD_REQUEST)
end
Verify GH signature with use luacrypto
local function verify_signature (hub_sign, data)
local sign = 'sha1=' .. crypto.hmac.digest('sha1', data, secret)
-- this is simple comparison, but it's better to use a constant time comparison
return hub_sign == sign
end
-- validate GH signature
if not verify_signature(headers['X-Hub-Signature'], data) then
ngx.log(ngx.ERR, "wrong webhook signature")
return ngx.exit (ngx.HTTP_FORBIDDEN)
end
Parse data as json and check is master branch, for deploy
data = json.decode(data)
-- on master branch
if data['ref'] ~= branch then
ngx.say("Skip branch ", data['ref'])
return ngx.exit (ngx.HTTP_OK)
end
If all correct, call deploy function
local function deploy ()
-- run command for deploy
local handle = io.popen("cd /path/to/repo && sudo -u username git pull")
local result = handle:read("*a")
handle:close()
ngx.say (result)
return ngx.exit (ngx.HTTP_OK)
end
Example
Example constant time string compare
local function const_eq (a, b)
-- Check is string equals, constant time exec
getmetatable('').__index = function (str, i)
return string.sub(str, i, i)
end
local diff = string.len(a) == string.len(b)
for i = 1, math.min(string.len(a), string.len(b)) do
diff = (a[i] == b[i]) and diff
end
return diff
end
A complete example of how I use it in github gist https://gist.github.com/Samael500/5dbdf6d55838f841a08eb7847ad1c926
This solution does not implement verification for GitHub's hooks and assumes you have the lua extension and the cjson module installed:
location = /location {
default_type 'text/plain';
content_by_lua_block {
local cjson = require "cjson.safe"
ngx.req.read_body()
local data = ngx.req.get_body_data()
if
data
then
local obj = cjson.decode(data)
if
# checksum checking should go here
(obj and obj.repository and obj.repository.full_name) == "user/reponame"
then
local file = io.open("<your file>","w")
if
file
then
file:close()
ngx.say("success")
else
ngx.exit(ngx.HTTP_INTERNAL_SERVER_ERROR)
end
else
ngx.exit(ngx.HTTP_UNAUTHORIZED)
end
else
ngx.exit(ngx.HTTP_NOT_ALLOWED)
end
}
}
I've been reading the NodeMCU documentation and several closed issues about the change of SDK that previouly allowed to send multiple data streams (acting like a queued net.socket:send).
It seems a huge debate grew here (#730) and there (#993) or even here (#999). However, I did not find any convincing example of a webserver code that would allow me to read multiple html files (e.g. head.html and body.html) to display a page. Here's the example from TerryE that I tried to adapt, but with no success:
srv=net.createServer(net.TCP)
srv:listen(80,function(conn)
conn:on ("receive", function(sck, req)
local response = {}
local f = file.open("head.html","r")
if f ~= nil then
response[#response+1] = file.read()
file.close()
end
local f = file.open("body.html","r")
if f ~= nil then
response[#response+1] = file.read()
file.close()
end
local function sender (sck)
if #response>0 then sck:send(table.remove(response,1))
else sck:close()
end
end
sck:on("sent", sender)
sender(sck)
end )
end )
When connecting to the ESP8266, nothing loads and I get no error from the lua terminal.
For your information, head.html contains:
<html>
<head>
</head>
And body.html contains:
<body>
<h1>Hello World</h1>
</body>
</html>
I am very new to NodeMCU, please be tolerant.
Here is my solution without using tables, saving some memory:
function Sendfile(sck, filename, sentCallback)
if not file.open(filename, "r") then
sck:close()
return
end
local function sendChunk()
local line = file.read(512)
if line then
sck:send(line, sendChunk)
else
file.close()
collectgarbage()
if sentCallback then
sentCallback()
else
sck:close()
end
end
end
sendChunk()
end
srv = net.createServer(net.TCP)
srv:listen(80, function(conn)
conn:on("receive", function(sck, req)
sck:send("HTTP/1.1 200 OK\r\n" ..
"Server: NodeMCU on ESP8266\r\n" ..
"Content-Type: text/html; charset=UTF-8\r\n\r\n",
function()
Sendfile(sck, "head.html", function() Sendfile(sck, "body.html") end)
end)
end)
end)
And this is for serving single files:
function Sendfile(client, filename)
if file.open(filename, "r") then
local function sendChunk()
local line = file.read(512)
if line then
client:send(line, sendChunk)
else
file.close()
client:close()
collectgarbage()
end
end
client:send("HTTP/1.1 200 OK\r\n" ..
"Server: NodeMCU on ESP8266\r\n" ..
"Content-Type: text/html; charset=UTF-8\r\n\r\n", sendChunk)
else
client:send("HTTP/1.0 404 Not Found\r\n\r\nPage not found")
client:close()
end
end
srv = net.createServer(net.TCP)
srv:listen(80, function(conn)
conn:on ("receive", function(client, request)
local path = string.match(request, "GET /(.+) HTTP")
if path == "" then path = "index.htm" end
Sendfile(client, path)
end)
end)
Thank you for the reply. I actually added the header you mentioned, I didn't know that was necessary and I also removed the sck argument in the sender function. My first code was actually working, I don't know what was wrong last time.
Anyway, it helped me understanding what was happening: the following code seems to concatenate the response array, since the event sent of the socket calls back the sender function (sck:on("sent", sender))
sck:send(table.remove(response,1))
In fact, table.remove(array, 1) returns the first item of the array, and removes this item of the array. Calling this line multiple times has the effect to read through it, item by item.
For the sake of simplicity, here is the code of a simple webserver able to serve multiple files:
header = "HTTP/1.0 200 OK\r\nServer: NodeMCU on ESP8266\r\nContent-Type: text/html\r\n\r\n"
srv=net.createServer(net.TCP)
srv:listen(80,function(conn)
conn:on ("receive", function(sck, req)
local response = {header}
tgtfile = string.sub(req,string.find(req,"GET /") +5,string.find(req,"HTTP/") -2 )
if tgtfile == "" then tgtfile = "index.htm" end
local f = file.open(tgtfile,"r")
if f ~= nil then
response[#response+1] = file.read()
file.close()
else
response[#response+1] = "<html>"
response[#response+1] = tgtfile.." not Found - 404 error."
response[#response+1] = "<a href='index.htm'>Home</a>"
end
collectgarbage()
f = nil
tgtfile = nil
local function sender ()
if #response>0 then sck:send(table.remove(response,1))
else sck:close()
end
end
sck:on("sent", sender)
sender()
end)
end)
This example was taken from this instructables and fixed to work with the new SDK (which do not allow multiple :send anymore). Please let me know if this code has some issues.
I don't know what is the size limit of the files though. Nevertheless, I manage to append more than 2Ko to the response variable and send it at once without any issue.