Encoding JSON as Base64 Before HTTP POSTing It (in Haskell) - json

Problem: I am attempting to POST some JSON to an HTTP endpoint that only accepts Base64 encoding.
Code Sample: Here is a code sample which successfully posts without Base64 encoding:
{-# LANGUAGE OverloadedStrings #-}
module Lib where
import Data.Aeson (encode, object, (.=))
import qualified Data.ByteString.Lazy.Char8 as L8
import Network.HTTP.Client
import Network.HTTP.Client.TLS
import Network.HTTP.Types.Status (statusCode)
import qualified Data.ByteString.Base64 as B64
postJSON :: IO ()
postJSON = do
manager <- newManager tlsManagerSettings
-- Nested JSON object to POST:
let requestObject = object
[ "event" .= ("App launched." :: String)
, "properties" .= object [ "distinct_id" .= ("user" :: String)
, "token" .= ("f793bae9548d8e123cef251fd81df487" :: String)
]
]
initialRequest <- parseRequest "http://api.mixpanel.com/track"
let request = initialRequest
{ method = "POST"
, requestBody = RequestBodyLBS $ encode requestObject
, requestHeaders =
[ ("Content-Type", "application/json; charset=utf-8")
]
}
response <- httpLbs request manager
putStrLn $ "The status code was: "
++ show (statusCode $ responseStatus response)
L8.putStrLn $ responseBody response
Attempt: In order to send the JSON as Base64 encoded, I tried replacing requestBody = RequestBodyLBS $ encode requestObject with requestBody = RequestBodyLBS $ Data.Bytestring.Base64.encode (encode requestObject), but I get a type error. So how do I encode the JSON as Base64 for this HTTP POST?

B64.encode here is a function from strict ByteString to strict ByteString (you'll need to hover over the type name in the haddocks to see this if you're just browsing), while Aeson.encode returns a lazy bytestring (from the Data.ByteString.Lazy module). These are two distinct types although they have the same name.
You probably have to do something like:
...
requestBody = RequestBodyLBS $ L8.fromStrict $ Data.Bytestring.Base64.encode (L8.toStrict $ encode requestObject)

I have two things to add to jberryman's answer.
First, if (as it now appears) you're going to be putting this in a query string, you need to make sure you don't just use a base64 encoded bytestring, but instead use a base64 url-encoded bytestring. So don't use Data.Bytestring.Base64 (as jberryman linked to), but rather Data.Bytestring.Base64.URL (here).
Second, while he pointed you in the right direction on the Base64 encoding part, it seems you're still hung up on setting the querystring. For that, you should check out the setQueryString function in the http-client library you're already using (link here).
That function has the signature:
setQueryString :: [(ByteString, Maybe ByteString)] -> Request -> Request
So if you're base64 encoded bytestring is built like this
let urlEncodedBytestring = Data.Bytestring.Base64.URL.encode . L8.toStrict $ encode requestObject
and if you're attemtping to set the data key in the querystring of your request, then you'll probably want:
let requestWithQueryStringSet = setQueryString [("data", (Just urlEncodedBytestring))] request

Related

How to read body as any valid json?

I haven't found the docs for that use case. How can I get the request body, ensure it's a valid JSON (any valid JSON, including numbers, string, booleans, and nulls, not only objects and arrays) and get the actual JSON.
Using Pydantic forces the JSON to have a specific structure.
You can find nearly everything inside the Request object
You are able to get request body with request.json(), which will give you the parsed JSON as dictionary.
from fastapi import Request, FastAPI
#app.post("/dummypath")
async def get_body(request: Request):
return await request.json()
If you want access the body as string, you can use request.body()
The accepted answer is valid as well, but FastAPI provides a built-in way to do that - check the Singular values in body section in docs.
A parameter with the default Body gets all the payload that doesn't match passed Pydantic-typed parameters (the whole payload in our case) and converts it to the dict. In case of invalid JSON, a standard validation error would be produced.
from fastapi import Body, FastAPI
app = FastAPI()
#app.post('/test')
async def update_item(
payload: dict = Body(...)
):
return payload
UPD: Note on ... (Ellipsis) - it allows marking a value as required. Read more in the Required with Ellipsis docs section
If you are confident that the incoming data is "a valid JSON", you can create a simple type annotation structure to receive the arbitrary JSON data.
from fastapi import FastAPI
from typing import Any, Dict, AnyStr, List, Union
app = FastAPI()
JSONObject = Dict[AnyStr, Any]
JSONArray = List[Any]
JSONStructure = Union[JSONArray, JSONObject]
#app.post("/")
async def root(arbitrary_json: JSONStructure = None):
return {"received_data": arbitrary_json}
Examples
1. JSON object
curl -X POST "http://0.0.0.0:6022/" -H "accept: application/json" -H "Content-Type: application/json" -d "{\"test_key\":\"test_val\"}"
Response:
{
"received_data": {
"test_key": "test_val"
}
}
2. JSON array
curl -X POST "http://0.0.0.0:6022/" -H "accept: application/json" -H "Content-Type: application/json" -d "[\"foo\",\"bar\"]"
Response:
{
"received_data": [
"foo",
"bar"
]
}
If you are not sure about the content type of the incoming data, better to parse the request body.
It can be done as,
from fastapi import FastAPI, Request
app = FastAPI()
#app.post("/")
async def root(request: Request):
return {"received_request_body": await request.body()}
The advantage of this method is that the body will contain any kind of data, JSON, form-data, multipart-form-data, etc.
from fastapi import Request
async def synonyms__select(request: Request):
return await request.json()
will return a JSON object.
This is an example to print the content of a Request, it will print the json body (if it is json parsable) otherwise just print the raw bytes of the body.
async def print_request(request):
print(f'request header : {dict(request.headers.items())}' )
print(f'request query params : {dict(request.query_params.items())}')
try :
print(f'request json : {await request.json()}')
except Exception as err:
# could not parse json
print(f'request body : {await request.body()}')
#app.post("/printREQUEST")
async def create_file(request: Request):
try:
await print_request(request)
return {"status": "OK"}
except Exception as err:
logging.error(f'could not print REQUEST: {err}')
return {"status": "ERR"}
FastAPI has a JSON encoder.
There are some cases where you might need to convert a data type (like
a Pydantic model) to something compatible with JSON
from fastapi import FastAPI
from fastapi.encoders import jsonable_encoder
from pydantic import BaseModel
import simplejson as json
class SubmitGeneral(BaseModel):
controllerIPaddress: str
readerIPaddress: str
ntpServer: str
#app.post("/submitGeneral")
async def submitGeneral(data: SubmitGeneral):
data = jsonable_encoder(data)
#data = json.loads(data.json()) # same as above line
print(f"data = {json.dumps(data)}")
# you have to access the properties with brackets, not by dot notation
query = f"update LocalPLC set ControllerIpAddress = '{data['controllerIPaddress']}', ReaderIPAddress = '{data['readerIPaddress']}'"
return {"status": "OK"}
For those of you using BaseModel and want to have a JSON field, you can import Json from pydantic
from fastapi import FastAPI
from pydantic import BaseModel, Json, Field
app = FastAPI()
class MockEndpoint(BaseModel):
endpoint: str = Field(description="API endpoint to mock")
response: Json = Field(description="Example response of the endpoint")
#app.get("/")
async def root():
return {"message": "Hello World"}
#app.post("/mock")
async def mock_request(mock_endpoint: MockEndpoint):
return mock_endpoint

Send request as content type of x-www-form-urlencoded with wreq

I'm learning to use wreq this weekend and I've run into some strange behavior.
I have a module AuthRequest
{-# LANGUAGE OverloadedStrings #-}
{-# LANGUAGE DeriveGeneric #-}
module AuthRequest where
import Data.Aeson
import GHC.Generics
import Data.Monoid
data AuthRequest = AuthRequest {
client_id :: String
, client_secret :: String
, grant_type :: String
} deriving (Generic, Show)
instance ToJSON AuthRequest where
toJSON (AuthRequest id_ secret grant) =
object [ "client_id" .= id_
, "client_secret" .= secret
, "grant_type" .= grant
]
toEncoding(AuthRequest id_ secret grant) =
pairs ("client_id" .= id_ <> "client_secret" .= secret <> "grant_type" .= grant)
and a module HttpDemo
{-# LANGUAGE OverloadedStrings #-}
{-# LANGUAGE DeriveGeneric #-}
module HttpDemo where
import Control.Lens
import Network.Wreq
import AuthRequest
import Data.Aeson
clientId = "some_id"
clientSecret = "some_secret"
url = "http://localhost:5000"
opts :: Options
opts = defaults
& header "Content-Type" .~ ["application/x-www-form-urlencoded"]
req :: AuthRequest
req = AuthRequest clientId clientSecret "credentials"
postIt = postWith opts url (toJSON req)
On the other end, I have a simple python flask server that listens to this request with a breakpoint so I can see the value that comes through.
When I look at the request.form on the server side, I see this: ImmutableMultiDict([('{"client_secret":"some_secret","client_id":"some_id","grant_type":"whatever"}', '')])
The key is what should be my post body!
But if I make a similar request using the requests python library
requests.post('http://localhost:5000', data={'client_id': clientId, 'client_secret': clientSecret, 'grant_type': grant_type}, headers={'content-type': 'application/x-www-form-urlencoded'})
I see what I expect: ImmutableMultiDict([('grant_type', 'whatever'), ('client_id', 'some_id'), ('client_secret', 'some_secret')])
What I think I want is to send this request as x-www-form-urlencoded. I see there are some docs here around this, but not clear on how to proceed. Maybe I need a FormValue instance? An example would be helpful.
As per your discussion with #Alexis, your client seems to be sending JSON, while the server is expected urlencoded. The documentation of Post shows how to send urlencoded data, using the := constructor. In this case this would be
postIt = post url ["client_id" := clientId, "client_secret" := clientSecret, "grant_type" := grantType]
I've given the example using post rather than postWith since the default appears to be that it uses application/x-www-form-urlencoded.
There seems to be a slight complication with OverloadedStrings. To make a compilable program, I had to remove the AuthRequest module and explicitly give the types of the constants as below
{-# LANGUAGE OverloadedStrings #-}
module Main where
import Network.Wreq
import Data.ByteString as B
clientId = "some_id" :: ByteString
clientSecret = "some_secret" :: ByteString
grantType = "credentials" :: ByteString
url = "http://localhost:8080"
postIt = post url ["client_id" := clientId, "client_secret" := clientSecret, "grant_type" := grantType]
main = postIt

Custom JSON errors for Servant-server

When using servant, I'd like to return all errors as JSON. Currently, if a request fails to parse, I see an error message like this, returned as plain text
Failed reading: not a valid json value
Instead I would like to return this as application/json
{"error":"Failed reading: not a valid json value"}
How can I do this? The docs say ServantErr is the default error type, and I can certainly respond with custom errors inside my handlers, but if parsing fails I don't see how I can return a custom error.
First, some language extensions
{-# LANGUAGE FlexibleContexts #-}
{-# LANGUAGE FlexibleInstances #-}
{-# LANGUAGE MultiParamTypeClasses #-}
{-# LANGUAGE OverloadedStrings #-}
{-# LANGUAGE ScopedTypeVariables #-}
{-# LANGUAGE TypeFamilies #-}
{-# LANGUAGE TypeOperators #-}
{-# LANGUAGE UndecidableInstances #-}
{-# LANGUAGE ViewPatterns #-}
Now then
Unfortunately this is more difficult than it should be. Servant, while well-designed and the composition of small logical parts, is very opinionated about how HTTP services should operate. The default implementation of ReqBody, which you are probably using, is hard-coded to spit out a text string.
However, we can switch out ReqBody for our own data type:
module Body where
import Control.Monad.Trans (liftIO)
import Data.Proxy (Proxy(..))
import Network.Wai (lazyRequestBody)
import Data.Aeson
import Servant.API
import Servant.Server
import Servant.Server.Internal
data Body a
instance (FromJSON a, HasServer api context) => HasServer (Body a :> api) context where
type ServerT (Body a :> api) m = a -> ServerT api m
route Proxy context subserver =
route (Proxy :: Proxy api) context (addBodyCheck subserver (withRequest bodyCheck))
where
bodyCheck request = do
body <- liftIO (lazyRequestBody request)
case eitherDecode body of
Left (BodyError -> e) ->
delayedFailFatal err400 { errBody = encode e }
Right v ->
return v
In this very brief amount of code a lot is happening:
We are teaching the servant-server package on how to handle our new datatype when it appears in the type resolution for serve (Proxy :: Proxy (Body foo :> bar)) server.
We have ripped most of the code from the v0.8.1 release of ReqBody.
We are adding a function to the pipeline that processes request bodies.
In it, we attempt to decode to the a parameter of Body. On failure, we spit out a JSON blob and an HTTP 400.
We are entirely ignoring content-type headers here, for brevity.
Here is the type of the JSON blob:
newtype BodyError = BodyError String
instance ToJSON BodyError where
toJSON (BodyError b) = object ["error" .= b]
Most of this machinery is internal to servant-server and underdocumented and rather fragile. For example, already I see that the code diverges on master branch and the arity of my addBodyCheck has changed.
Though the Servant project is still quite young and remarkably ambitious, I have to say that the aesthetics and robustness of this solution are definitely underwhelming.
To test this
We will need a Main module:
{-# LANGUAGE DataKinds #-}
{-# LANGUAGE TypeOperators #-}
module Main where
import Data.Proxy (Proxy(..))
import Network.Wai.Handler.Warp (run)
import Servant.API
import Servant.Server
import Body
type API = Body [Int] :> Post '[JSON] [Int]
server :: Server API
server = pure
main :: IO ()
main = do
putStrLn "running on port 8000"
run 8000 (serve (Proxy :: Proxy API) server)
And a shell:
~ ❯❯❯ curl -i -XPOST 'http://localhost:8000/'
HTTP/1.1 400 Bad Request
Transfer-Encoding: chunked
Date: Fri, 20 Jan 2017 01:18:57 GMT
Server: Warp/3.2.9
{"error":"Error in $: not enough input"}%
~ ❯❯❯ curl -id 'hey' -XPOST 'http://localhost:8000/'
HTTP/1.1 400 Bad Request
Transfer-Encoding: chunked
Date: Fri, 20 Jan 2017 01:19:02 GMT
Server: Warp/3.2.9
{"error":"Error in $: Failed reading: not a valid json value"}%
~ ❯❯❯ curl -id '[1,2,3]' -XPOST 'http://localhost:8000/'
HTTP/1.1 200 OK
Transfer-Encoding: chunked
Date: Fri, 20 Jan 2017 01:19:07 GMT
Server: Warp/3.2.9
Content-Type: application/json
[1,2,3]%
Ta-da!
You should be able to run all this code on LTS-7.16.
What did we learn
(1) Servant and Haskell are fun.
(2) The typeclass machinery of Servant allows for a kind of plug-and-play when it comes to the types you specify in your API. We can take out ReqBody and replace it with our own; on a project I did at work we even replaced the Servant verbs (GET, POST, ...) with our own. We wrote new content types and we even did something similar with ReqBody like you saw here.
(3) It is the remarkable ability of the GHC compiler that we can destructure types during compile-time to influence runtime behavior in a safe and logically sound way. That we can express a tree of API routes at the type-level and then walk over them using typeclass instances, accumulating a server type using type families, is a wonderfully elegant way to build a well-typed web service.
Currently right now I just handle this in middleware. I do something like the following:
{-# LANGUAGE OverloadedStrings #-}
{-# LANGUAGE FlexibleContexts #-}
module Lib.ErrorResponse where
import Data.Text.Lazy.Encoding (decodeUtf8)
import Data.ByteString.Lazy (toStrict)
import Blaze.ByteString.Builder (toLazyByteString)
import Blaze.ByteString.Builder.ByteString (fromByteString)
import Network.Wai
import Network.Wai.Internal
import Network.HTTP.Types
import Data.Text
import Data.Aeson
import qualified Data.Text.Lazy as TL
customError :: Application -> Application
customError = modifyResponse responseModifier
responseModifier :: Response -> Response
responseModifier r
| responseStatus r == status400 && not (isCustomMessage r "Bad Request") =
buildResponse status400 "Bad Request" (customErrorBody r "BadRequest") 400
| responseStatus r == status403 =
buildResponse status403 "Forbidden" "Forbidden" 400
| responseStatus r == status404 =
buildResponse status404 "Not Found" "Not Found" 404
| responseStatus r == status405 =
buildResponse status405 "Method Not Allowed" "Method Not Allowed" 405
| otherwise = r
customErrorBody :: Response -> Text -> Text
customErrorBody (ResponseBuilder _ _ b) _ = TL.toStrict $ decodeUtf8 $ toLazyByteString b
customErrorBody (ResponseRaw _ res) e = customErrorBody res e
customErrorBody _ e = e
isCustomMessage :: Response -> Text -> Bool
isCustomMessage r m = "{\"error\":" `isInfixOf` customErrorBody r m
buildResponse :: Status -> Text -> Text -> Int -> Response
buildResponse st err msg cde = responseBuilder st
[("Content-Type", "application/json")]
(fromByteString . toStrict . encode $ object
[ "error" .= err
, "message" .= msg
, "statusCode" .= cde
]
)
And then I can use just like any other middleware:
run 8000 . customError $ serve api server
Taking inspiration from #codedmart I also use a middleware, but it does not construct the json, it only changes the content type of the response when there is an error, and keep the original error message.
startApp :: IO ()
startApp = run 8081 . (modifyResponse errorHeadersToJson) $ serve api server
errorHeadersToJson :: Response -> Response
errorHeadersToJson r
| responseStatus r == status200 = r
| otherwise = mapResponseHeaders text2json r
text2json :: ResponseHeaders -> ResponseHeaders
text2json h = Map.assocs (Map.fromList [("Content-Type", "application/json")] `Map.union` Map.fromList h)
The json is built beforehand with a function overriding the Servant throwError function.
data ServerError = ServerError
{ statusCode :: Int
, error :: String
, message :: String
} deriving (Eq, Show)
$(deriveJSON defaultOptions ''ServerError)
throwJsonError :: ServantErr -> String -> Servant.Handler b
throwJsonError err "" = throwError $ err { errBody = encode $ ServerError (errHTTPCode err) ("Server error"::String) (show $ errBody err) }
throwJsonError err message = throwError $ err { errBody = encode $ ServerError (errHTTPCode err) ("Server error"::String) message }
then I can throw any error with a custom message, it will be served as a json with the correct content-type :
throwJsonError err500 "Oh no !"

How to get JSON data in an Odoo controller using type='json'?

A few days ago I did a similar question here: How to get JSON data in an Odoo controller?
But now, I need to create a controller which receives only JSON data. So, I am doing the request from a Python console, this way:
import requests
import json
url = 'http://localhost:8069/odoo/test'
headers = {'Content-Type': 'application/json'}
data = {
'name': 'Jane',
'email': 'jane.doe#gmail.com',
}
data_json = json.dumps(data)
r = requests.post(url=url, data=data_json, headers=headers)
I have created a controller which listens to http://localhost:8069/odoo/test, this way:
import openerp.http as http
from openerp.http import Response
import logging
_logger = logging.getLogger(__name__)
class WebFormController(http.Controller):
#http.route('/odoo/test', type='json',
auth='public', methods=['POST'], website=True)
def index(self, **args):
_logger.info('CONNECTION SUCCESSFUL')
_logger.info(args)
name = args.get('name', False)
email = args.get('email', False)
_logger.info(name)
_logger.info(email)
if not name:
Response.status = '400 Bad Request'
return '{"response": "OK"}'
The problem is that I am receiving an empty JSON in the controller. I can read CONNECTION SUCCESFUL in the log, with no error, but when I show args, I get {}, and obviously due to that, False when writing name and email.
If I pass the data as a Python dictionary or as a string, I get the following error:
Invalid JSON data: 'name=Jane&email=jane.doe%40gmail.com' or
Invalid JSON data: "{'name': 'Jane', 'email': 'jane.doe#gmail.com'}" respectively.
If I modify type='json' and I write type='http' instead, I get the following error:
Function declared as capable of handling request of type 'http' but called with a request of type 'json'.
I have read that may be this could be solved if the request is sent using the parameter json instead of data, this way:
r = requests.post(url=url, json=data_json, headers=headers)
Unfortunately, the server which is going to make the request has an old operating system which cannot update the python-requests package, so I cannot use json parameter since it did not exist at the version installed in that server.
Please, can anyone help me? I need get JSON data in the controller, not a string neither Python dictionaries.
You have just forgotten to put your data inside the params keywords:
Use this correct syntax :
data = {"params": dict(key="value")}
data = {
"params": {
"name":"prakashsharma",
"email":"prakashsharmacs24#gmail.com",
"phone":"+917859884833"
}
}
Please don't forget to use json.dumps(data) and 'Content-Type': 'application/json' while requesting a resource in json format.
I am damn sure your issue will be solved after using this one my friend... cheers :)!!
You can use below format for a POST request
{
"params" : {
"name" : "Order/1/18",
"session_id" : 1,
"customer_count" : 2,
"partner_id" : 9,
"lines": [
{
"product_id": 37,
"qty" : 2,
"price_unit" : 2,
"discount" : 10
}
],
"pos_reference" : 2,
"note" : "This is a test note"
}
}
Content type must be application/json
How odoo route will handle request ?
Route will help creating a POS order in odoo [POST]
#http.route(['/api/v1/resources/<string:api_key>/pos_order'],
auth="public",
website=False,
type="json",
csrf=False,
methods = ['POST'])
def create_update_pos_order(self, api_key=None, **kwargs):
print(kwargs.get('name')) -> Order/1/18
God Bless Forvas::
But for more clearity:
if you want to test through cURL:
curl -i -X POST -H "Content-Type: application/json" -d '{"params": {"name":"prakashsharma","email":"prakashsharmacs24#gmail.com","phone":"+917859884833"}}' 'http://localhost:8069/web/yourlistoner/'
if you want to test through python request:
import requests
headers = {
'Content-Type': 'application/json',
}
data = '{"params": {"name":"prakashsharma","email":"prakashsharmacs24#gmail.com","phone":"+917859884833"}}'
requests.post('http://localhost:8069/web/yourlistoner/', headers=headers, data=data)
the function in odoo will be something like
from odoo import http
import json
class YourClass(http.Controller):
#http.route('/web/yourlistoner/', type='json', auth="none", methods=['POST'],cors="*", csrf=False)
def listoner(self, **kw):
print http.request.params
print "lllllllllllllllllllll"
return json.dumps({"result":"Success"})

How to validate JSON with schema in Haskell?

I want to validate JSON with a schema. hjsonschema seemed like a good choice as it is fairly new and supports the latest draft.
But the plotly json schema always gives me valid responses.
I may be misunderstanding something here but this should not be valid JSON
bad.json
{
"fjsdklj" : 5
}
even though it is considered valid by the following code
module Main where
import Control.Applicative
import Data.Aeson
import Data.HashMap.Strict (HashMap)
import qualified Data.HashMap.Strict as H
import Data.Monoid
import qualified Data.ByteString.Lazy as B
import qualified Data.JsonSchema as JS
import Data.Maybe
main :: IO ()
main = do
schemaJSON <- (fromJust . decode) <$> B.readFile "simple-schema.json"
bad <- (fromJust . decode) <$> B.readFile "bad.json"
let schemaData = JS.RawSchema {
JS._rsURI = Nothing,
JS._rsData = schemaJSON
}
schema <- compileSchema (JS.SchemaGraph schemaData H.empty) schemaData
checkResults (JS.validate schema bad)
compileSchema :: JS.SchemaGraph -> JS.RawSchema -> IO (JS.Schema JS.Draft4Failure)
compileSchema graph rs =
case JS.compileDraft4 graph rs of
Left failure -> error $ "Not a valid schema: " <> show failure
Right schema -> return schema
checkResults :: [JS.ValidationFailure JS.Draft4Failure] -> IO ()
checkResults [] = putStrLn "Just fine"
checkResults x = putStrLn $ "ERROR: " ++ show x
simple-schema.json is the plotly schema and bad.json the snippet I posted above.
It's nothing about Haskell.
Your schema doesn't have a required property at top level so an empty json object is acceptable.
Also it contains no "additionalProperties": false property so anything that does not fit into one of the defined patterns are ignored.
BTW I doubt whether it's a valid Draft4 json schema. It passed the validation of the Draft4 meta-schema but the syntax is a little bit different, maybe something Python-specific. You'd better run the test suites which came along with the hjsonscheme package to see if everything works fine.