I am trying to get list of wall post from Vk.com using Vontakte, mongoengine, Django
Here is my view, where wallposts = vk.get('wall.get', owner_id=237897731, offset=0, count=10) is Vk.com API call:
import vkontakte
vk = vkontakte.API(token=access_token)
class VkWallPostListView(ListView):
model = VkWallPost
context_object_name = "vk_list"
def get_template_names(self):
return ["blog/vk_list.html"]
def get_queryset(self):
wallposts = VkWallPost.objects
if 'all_posts' not in self.request.GET:
wallposts = vk.get('wall.get', owner_id=237897731, offset=0, count=10)
for wallpost in wallposts:
wallpost.save()
#wallposts = wallposts.filter(text__startswith='RT')
tag = self.request.GET.get('tag', None)
if tag:
wallposts = wallposts.filter(tags=tag)
return wallposts
Also in this view i am trying to save results of API call to MongoDB right after the actual call:
for wallpost in wallposts:
wallpost.save()
But in browser i see an error:
Exception Value:
'int' object has no attribute 'save'
Exception Location: c:\Users\JOOMLER\BitNami_DjangoStack\django_mongo_test\blog\views.py in get_queryset, line 109
If i remove this two strings for cycle all works fine and shows on the fly data from Vk.com in browser. But i want to save it for later use. So, i assume the problem is how to save JSON response to MongoDB ?
That was simple and i am surprised that no answers:
i need to use raw Pymongo:
wallpost.save()
is not correct, because save django model with predefined fields
VkWallPost._get_collection().insert(vk_post)
this is correct - we use raw pymongo insert
Well, did you see what do you have in wallposts variable?
Cause, I think, error message is pretty clear: in cycle variable wallpost you have int value. And, of course, trying to call save() will throw exception. May be, objects in wallposts have integer indices, do you think?
Try to print what you have in wallposts.
Related
I store a blob of Json in the datastore using JsonProperty.
I don't know the structure of the json data.
I am using endpoints proto datastore in order to retrieve my data.
The probleme is the json property is encoded in base64 and I want a plain json object.
For the example, the json data will be:
{
first: 1,
second: 2
}
My code looks something like:
import endpoints
from google.appengine.ext import ndb
from protorpc import remote
from endpoints_proto_datastore.ndb import EndpointsModel
class Model(EndpointsModel):
data = ndb.JsonProperty()
#endpoints.api(name='myapi', version='v1', description='My Sample API')
class DataEndpoint(remote.Service):
#Model.method(path='mymodel2', http_method='POST',
name='mymodel.insert')
def MyModelInsert(self, my_model):
my_model.data = {"first": 1, "second": 2}
my_model.put()
return my_model
#Model.method(path='mymodel/{entityKey}',
http_method='GET',
name='mymodel.get')
def getMyModel(self, model):
print(model.data)
return model
API = endpoints.api_server([DataEndpoint])
When I call the api for getting a model, I get:
POST /_ah/api/myapi/v1/mymodel2
{
"data": "eyJzZWNvbmQiOiAyLCAiZmlyc3QiOiAxfQ=="
}
where eyJzZWNvbmQiOiAyLCAiZmlyc3QiOiAxfQ== is the base64 encoded of {"second": 2, "first": 1}
And the print statement give me: {u'second': 2, u'first': 1}
So, in the method, I can explore the json blob data as a python dict.
But, in the api call, the data is encoded in base64.
I expeted the api call to give me:
{
'data': {
'second': 2,
'first': 1
}
}
How can I get this result?
After the discussion in the comments of your question, let me share with you a sample code that you can use in order to store a JSON object in Datastore (it will be stored as a string), and later retrieve it in such a way that:
It will show as plain JSON after the API call.
You will be able to parse it again to a Python dict using eval.
I hope I understood correctly your issue, and this helps you with it.
import endpoints
from google.appengine.ext import ndb
from protorpc import remote
from endpoints_proto_datastore.ndb import EndpointsModel
class Sample(EndpointsModel):
column1 = ndb.StringProperty()
column2 = ndb.IntegerProperty()
column3 = ndb.StringProperty()
#endpoints.api(name='myapi', version='v1', description='My Sample API')
class MyApi(remote.Service):
# URL: .../_ah/api/myapi/v1/mymodel - POSTS A NEW ENTITY
#Sample.method(path='mymodel', http_method='GET', name='Sample.insert')
def MyModelInsert(self, my_model):
dict={'first':1, 'second':2}
dict_str=str(dict)
my_model.column1="Year"
my_model.column2=2018
my_model.column3=dict_str
my_model.put()
return my_model
# URL: .../_ah/api/myapi/v1/mymodel/{ID} - RETRIEVES AN ENTITY BY ITS ID
#Sample.method(request_fields=('id',), path='mymodel/{id}', http_method='GET', name='Sample.get')
def MyModelGet(self, my_model):
if not my_model.from_datastore:
raise endpoints.NotFoundException('MyModel not found.')
dict=eval(my_model.column3)
print("This is the Python dict recovered from a string: {}".format(dict))
return my_model
application = endpoints.api_server([MyApi], restricted=False)
I have tested this code using the development server, but it should work the same in production using App Engine with Endpoints and Datastore.
After querying the first endpoint, it will create a new Entity which you will be able to find in Datastore, and which contains a property column3 with your JSON data in string format:
Then, if you use the ID of that entity to retrieve it, in your browser it will show the string without any strange encoding, just plain JSON:
And in the console, you will be able to see that this string can be converted to a Python dict (or also a JSON, using the json module if you prefer):
I hope I have not missed any point of what you want to achieve, but I think all the most important points are covered with this code: a property being a JSON object, store it in Datastore, retrieve it in a readable format, and being able to use it again as JSON/dict.
Update:
I think you should have a look at the list of available Property Types yourself, in order to find which one fits your requirements better. However, as an additional note, I have done a quick test working with a StructuredProperty (a property inside another property), by adding these modifications to the code:
#Define the nested model (your JSON object)
class Structured(EndpointsModel):
first = ndb.IntegerProperty()
second = ndb.IntegerProperty()
#Here I added a new property for simplicity; remember, StackOverflow does not write code for you :)
class Sample(EndpointsModel):
column1 = ndb.StringProperty()
column2 = ndb.IntegerProperty()
column3 = ndb.StringProperty()
column4 = ndb.StructuredProperty(Structured)
#Modify this endpoint definition to add a new property
#Sample.method(request_fields=('id',), path='mymodel/{id}', http_method='GET', name='Sample.get')
def MyModelGet(self, my_model):
if not my_model.from_datastore:
raise endpoints.NotFoundException('MyModel not found.')
#Add the new nested property here
dict=eval(my_model.column3)
my_model.column4=dict
print(json.dumps(my_model.column3))
print("This is the Python dict recovered from a string: {}".format(dict))
return my_model
With these changes, the response of the call to the endpoint looks like:
Now column4 is a JSON object itself (although it is not printed in-line, I do not think that should be a problem.
I hope this helps too. If this is not the exact behavior you want, maybe should play around with the Property Types available, but I do not think there is one type to which you can print a Python dict (or JSON object) without previously converting it to a String.
I have a WS using Flask/python 2.7. I have 1 JSON object passed to the WS. I have been successful in capturing the object and returning the whole JSON.
I have looked all over for examples (many use print of test dataset in python) and have tried json.dumps, json.loads, json.dump, json.load, for loops, etc.
What I would like to do seems simple and I know it is me, but I get errors no matter what I try. I am trying to parse the JSON, put the values in to variables, and do "stuff".
This works:
#app.route('/v1/test', methods = ['POST'])
def api_message():
if request.headers['Content-Type'] == 'application/json':
return "JSON Message: " + json.dumps(request.json, separators=(',',':'))
else:
return "415 Unsupported Media Type"
This does not (and many variations of this using different things):
jsonobject = json.dumps(request.json)
pstring = json.loads(jsonobject)
for key, value in pstring.iteritems():
return value
What I want to do (pseudo code):
for each JSON
get the name value pairs in to a place where I can do something like this (which was done on a flat file)
input_data = pd.read_csv(sio, delimiter=',', names=columns)
probs = model.predict_proba(input_data)
I am sure I didn't make this as clear as I could but it is a challenge because I get errors like below (examples -- not all at once of course) with all the different things I try:
AttributeError: 'dict' object has no attribute 'translate'
TypeError: 'dict' object is not callable
AttributeError: 'str' object has no attribute 'iteritems'
So after all that, what is the right way to do this?
I ma new to writing python unit tests. I have a method in a class returning a Json response from an API. The JSON response contains attributes such as data, token, object name and status. The method hits API and returns response with different values each time, so I can't hard code a response and test it. How can I write a unit test for this type of method.
One thing, I thought of is to check whether the response is not null. Is there any other type of checks I can do here.
Each time it returns a different token, date(timestamp). The status will be same.
def Json_get_status(self):
get_url = "xxxx" #URL to hit API
r = requests.get(get_url)
self.get_json = json.loads(r.text)
self.get_token=self.get_json["token"]
self.get_date=self.get_json["date"]
self.get_status=self.get_json["status"]
return self.get_json
If your method under test is supposed to "read the status correctly", then you might want to specifically test that.
So assuming your app is something like
def read_status(response):
parsed = json.loads(response)
# does other stuff
return something
Then in your test_unit.py
def test_read_status(self):
mock_input_val = {'something': 'good val'}
expected_positive_return_val = something
self.assertEqual(read_status(json.dumps(mock_input_val)),
expected_positive_return_val)
Also good to do a negative test for the condition where read_status either fails to parse the json object or finds an error in the response.
I'm using SOAPUI to mock out a web-api service, I'm reading the contents of a static json response file, but changing the contents of a couple of the nodes based on what the user has passed through in the request.
I can't create multiple responses as surcharge is calculated from the amount passed.
The toString() method of the object that gets return by the slurper is replacing { with [ with invalidates my JsonResponse. I've included the important bits of the code below, has anyone got a way around this or is JsonSlurper not the right thing to use here?
def json=slurper.parseText(new File(path).text)
// set the surcharge to the two credit card nodes
// these are being set fine
json.AvailableCardTypeResponse.PaymentCards[0].Surcharge="${sur_charge}"
json.AvailableCardTypeResponse.PaymentCards[1].Surcharge="${sur_charge}"
response.setContentType("application/json;charset=utf-8" );
response.setContentLength(length);
Tools.readAndWrite( new ByteArrayInputStream(json.toString().getBytes("UTF-8")), length,response.getOutputStream() )
return new com.eviware.soapui.impl.wsdl.mock.WsdlMockResult(mockRequest)
You're slurping the json into a List/Map structure, then writing this List/Map structure out.
You need to convert your lists and maps back to json.
Change the line:
Tools.readAndWrite( new ByteArrayInputStream(json.toString().getBytes("UTF-8")), length,response.getOutputStream() )
to
Tools.readAndWrite( new ByteArrayInputStream( new JsonBuilder( json ).toString().getBytes("UTF-8")), length,response.getOutputStream() )
I have code that looks like this:
def client = new groovyx.net.http.RESTClient('myRestFulURL')
def json = client.get(contentType: JSON)
net.sf.json.JSON jsonData = json.data as net.sf.json.JSON
def slurper = new JsonSlurper().parseText(jsonData)
However, it doesn't work! :( The code above gives an error in parseText because the json elements are not quoted. The overriding issue is that the "data" is coming back as a Map, not as real Json. Not shown, but my first attempt, I just passed the parseText(json.data) which gives an error about not being able to parse a HashMap.
So my question is: how do I get JSON returned from the RESTClient to be parsed by JsonSlurper?
The RESTClient class automatically parses the content and it doesn't seem possible to keep it from doing so.
However, if you use HTTPBuilder you can overload the behavior. You want to get the information back as text, but if you only set the contentType as TEXT, it won't work, since the HTTPBuilder uses the contentType parameter of the HTTPBuilder.get() method to determine both the Accept HTTP Header to send, as well was the parsing to do on the object which is returned. In this case, you need application/json in the Accept header, but you want the parsing for TEXT (that is, no parsing).
The way you get around that is to set the Accept header on the HTTPBuilder object before calling get() on it. That overrides the header that would otherwise be set on it. The below code runs for me.
#Grab(group='org.codehaus.groovy.modules.http-builder', module='http-builder', version='0.6')
import static groovyx.net.http.ContentType.TEXT
def client = new groovyx.net.http.HTTPBuilder('myRestFulURL')
client.setHeaders(Accept: 'application/json')
def json = client.get(contentType: TEXT)
def slurper = new groovy.json.JsonSlurper().parse(json)
The type of response from RESTClient will depend on the version of :
org.codehaus.groovy.modules.http-builder:http-builder
For example, with version 0.5.2, i was getting a net.sf.json.JSONObject back.
In version 0.7.1, it now returns a HashMap as per the question's observations.
When it's a map, you can simply access the JSON data using the normal map operations :
def jsonMap = restClientResponse.getData()
def user = jsonMap.get("user")
....
Solution posted by jesseplymale workes for me, too.
HttpBuilder has dependencies to some appache libs,
so to avoid to add this dependencies to your project,
you can take this solution without making use of HttpBuilder:
def jsonSlurperRequest(urlString) {
def url = new URL(urlString)
def connection = (HttpURLConnection)url.openConnection()
connection.setRequestMethod("GET")
connection.setRequestProperty("Accept", "application/json")
connection.setRequestProperty("User-Agent", "Mozilla/5.0")
new JsonSlurper().parse(connection.getInputStream())
}