I have a Pyramid web-application that I am trying to unit-test.
In my tests file I have this snippet of code:
anyparam = {"isApple": "True"}
#parameterized.expand([
("ParamA", anyparam, 'success')])
def test_(self, name, params, expected):
request = testing.DummyRequest(params=params)
request.session['AI'] = ''
response = dothejob(request)
self.assertEqual(response['status'], expected, "expected response['status']={0} but response={1}".format(expected, response))
Whereas in my views:
#view_config(route_name='function', renderer='json')
def dothejob(request):
params = json.loads(request.body)
value = params.get('isApple') #true or false.
However, when I'm trying to unit-test it, I am getting this error:
JSONDecodeError: Expecting value: line 1 column 1 (char 0)
However, when I make the same request via POST via web-browser it works perfectly fine.
By doing testing.DummyRequest(params=params) you are only populating request.params, not request.body.
You probably want to do something like:
request = testing.DummyRequest(json_body=params)
Also, you may want to use directly request.json_body in your code instead of json.loads(request.body).
Related
I am new around here, and I need some help
so, I am trying to make a report in odoo with Base report CSV, in the table, I have 2 relational field, and I don't know how to combine those table, I tried combining the function from the module Base report CSV like below, but it give an error, a blank error which only make me confused, anyone got any idea how I could do this?
from odoo import models
import csv
class csvreport(models.AbstractModel):
_name = 'report.hr_timesheet.report'
_inherit = 'report.report_csv.abstract'
def generate_csv_report(self, writer, data, partners):
writer.writeheader()
for obj in partners:
employee = self.env.cr.execute("""select hr_employee.name where hr_employee.id = %s;""", (obj.employee_id))
task = self.env.cr.execute("""select project_task.name where project_task.id = %s;""", (obj.project_id))
writer.writerow({
'name': obj.name,
'date': obj.date,
'unit_amount': obj.unit_amount,
'responsible': employee.fetchall(),
'task': task.fetchall(),
})
def csv_report_options(self):
res = super().csv_report_options()
res['fieldnames'].append('name')
res['fieldnames'].append('date')
res['fieldnames'].append('unit_amount')
res['fieldnames'].append('responsible')
res['fieldnames'].append('task')
res['delimiter'] = ';'
res['quoting'] = csv.QUOTE_ALL
return res
The Error :
Since i can't post picture, i'll just post a gdrive link
You should see the following error message in the error log:
ValueError: SQL query parameters should be a tuple, list or dict; ...
To fix that error pass query arguments in a tuple:
employee = self.env.cr.execute(query_str, (obj.employee_id.id, ))
You can't pass obj.employee_id (a record), because psycopg2 can't adapt type hr.employee.
To get the employee name just use dot notation:
employee_name = obj.employee_id.name
The from clause is missing for both queries and you can't call fetchall on employee or task because self.env.cr.execute will return None, to fetch the result, use the cursor fetch*() methods.
self.env.cr.fetchall()
I develop django website on cpanel with MySQL database. I have a function that pull feeds from this website https://travelcommunication.net/feed/ and create an object from that (web scraping using beautifulsoup4).
But when I try to grab the content section, the error appears. But that only happens with certain items, not all.
I try on my local (with sqlite database) and all working fine. I have also tried on heroku (with PostgreSQL database) and all working fine.
Here is my code:
#views.py
def pull_feeds(request, pk):
if request.user.is_superuser:
source = Autoblogging.objects.get(pk=pk)
url = requests.get(source.url)
soup = BeautifulSoup(url.content, "html.parser")
length = source.items
items = soup.find_all('item')[:length]
contents = soup.find_all('content:encoded')[:length]
for i in range(length-1, -1, -1):
content = contents[i].text
title = items[i].title.text
body = content[content.find('<p>'):] #the problem is here .. when i comment this, everything works fine
category = Category.objects.get(pk=source.category.id)
if not Post.objects.filter(title=title).exists():
post = Post(title=title,
body=body, #the problem is here .. when i comment this, everything works fine
category=category)
link = content[content.find('src=')+5:content.find('alt')-2]
img_data = requests.get(link).content
with open('temp_image.jpg', 'wb') as handler:
handler.write(img_data)
with open('temp_image.jpg', 'rb') as handler:
file_name = link.split("/")[-1]
post.cover.save(file_name, files.File(handler))
os.remove("temp_image.jpg")
return redirect("news:autoblogging")
else:
return HttpResponse("Sorry you are not allowed to access this page")
Does anyone know how to fix this error? Thanks.
I am trying to extract the text part from the request that I made through grequest library but I am unable to figure out how can I do so.
If we use Requests Library I would do
r = requests.get('www.google.com')
htmls.append(r.text)
Now if I am using grequests I can only get a list of response code and not text.
rs = (grequests.get(u) for u in urls)
result = grequests.map(rs)
What I've tried
result = grequests.map(rs.text)
I get an error using above piece of code AttributeError: 'generator' object has no attribute 'text'
My desired output is a list of html text where response code is 200 else the value should be None.
How can I achieve that?
Desired Output:
response_code = [<Response [200]>,<Response [404]>,<Response [200]>]
htmls = ['html1', None, 'html2']
You can use something like below
rs = (grequests.get(u) for u in urls)
responses = grequests.map(rs)
text = list(map(lambda d : d.text if d else None, responses))
print(text)
What you are getting back is a response array after you call the map. And then you can process this data using native map function
I am using an API to pull data from a url, however the API has a pagination limit. It goes like:
Page (default is 1 and it's the page number you want to retrieve)
Per_page (default is 100 and it's the maximum number of results returned in the response(max=500))
I have a script which I can get the results of a page or per page but I want to automate it. I want to be able to loop through all the pages or per_page(500) and load it in to a json file.
Here is my code that can get 500 results per_page:
import json, pprint
import requests
url = "https://my_api.com/v1/users?per_page=500"
header = {"Authorization": "Bearer <my_api_token>"}
s = requests.Session()
s.proxies = {"http": "<my_proxies>", "https": "<my_proxies>" }
resp = s.get(url, headers=header, verify=False)
raw=resp.json()
for x in raw:
print(x)
The output is 500 but is there a way to keep going and pull the results starting from where it left off? Or even go by page and get all the data per page until there's no data in a page?
It will be helpful, if you present a sample response from your API.
If the API is equipped properly, there will be a next property in a given response that leads you to the next page.
You can then keep calling the API with the link given in the next recursively. On the last page, there will be no next in the Link header.
resp.links["next"]["url"] will give you the URL to the next page.
For example, the GitHub API has next, last, first, and prev properties.
To put it into code, first, you need to turn your code into functions.
Given that there is a maximum of 500 results per page, it implies you are extracting a list of data of some sort from the API. Often, these data are returned in a list somewhere inside raw.
For now, let's assume you want to extract all elements inside a list at raw.get('data').
import requests
header = {"Authorization": "Bearer <my_api_token>"}
results_per_page = 500
def compose_url():
return (
"https://my_api.com/v1/users"
+ "?per_page="
+ str(results_per_page)
+ "&page_number="
+ "1"
)
def get_result(url=None):
if url_get is None:
url_get = compose_url()
else:
url_get = url
s = requests.Session()
s.proxies = {"http": "<my_proxies>", "https": "<my_proxies>"}
resp = s.get(url_get, headers=header, verify=False)
# You may also want to check the status code
if resp.status_code != 200:
raise Exception(resp.status_code)
raw = resp.json() # of type dict
data = raw.get("data") # of type list
if not "url" in resp.links.get("next"):
# We are at the last page, return data
return data
# Otherwise, recursively get results from the next url
return data + get_result(resp.links["next"]["url"]) # concat lists
def main():
# Driver function
data = get_result()
# Then you can print the data or save it to a file
if __name__ == "__main__":
# Now run the driver function
main()
However, if there isn't a proper Link header, I see 2 solutions:
(1) recursion and (2) loop.
I'll demonstrate recursion.
As you have mentioned, when there is pagination in API responses, i.e. when there is a limit of maximum number of results per page, there is often a query parameter called page number or start index of some sort to indicate which "page" you are querying, so we'll utilize the page_number parameter in the code.
The logic is:
Given a HTTP request response, if there is less than 500 results, it means there is no more pages. Return the results.
If there are 500 results in a given response, it means there's probably another page, so we advance page_number by 1 and do a recursion (by calling the function itself) and concatenate with previous results.
import requests
header = {"Authorization": "Bearer <my_api_token>"}
results_per_page = 500
def compose_url(results_per_page, current_page_number):
return (
"https://my_api.com/v1/users"
+ "?per_page="
+ str(results_per_page)
+ "&page_number="
+ str(current_page_number)
)
def get_result(current_page_number):
s = requests.Session()
s.proxies = {"http": "<my_proxies>", "https": "<my_proxies>"}
url = compose_url(results_per_page, current_page_number)
resp = s.get(url, headers=header, verify=False)
# You may also want to check the status code
if resp.status_code != 200:
raise Exception(resp.status_code)
raw = resp.json() # of type dict
data = raw.get("data") # of type list
# If the length of data is smaller than results_per_page (500 of them),
# that means there is no more pages
if len(data) < results_per_page:
return data
# Otherwise, advance the page number and do a recursion
return data + get_result(current_page_number + 1) # concat lists
def main():
# Driver function
data = get_result(1)
# Then you can print the data or save it to a file
if __name__ == "__main__":
# Now run the driver function
main()
If you truly want to store the raw responses, you can. However, you'll still need to check the number of results in a given response. The logic is similar. If a given raw contains 500 results, it means there is probably another page. We advance the page number by 1 and do a recursion.
Let's still assume raw.get('data') is the list whose length is the number of results.
Because JSON/dictionary files cannot be simply concatenated, you can store raw (which is a dictionary) of each page into a list of raws. You can then parse and synthesize the data in whatever way you want.
Use the following get_result function:
def get_result(current_page_number):
s = requests.Session()
s.proxies = {"http": "<my_proxies>", "https": "<my_proxies>"}
url = compose_url(results_per_page, current_page_number)
resp = s.get(url, headers=header, verify=False)
# You may also want to check the status code
if resp.status_code != 200:
raise Exception(resp.status_code)
raw = resp.json() # of type dict
data = raw.get("data") # of type list
if len(data) == results_per_page:
return [raw] + get_result(current_page_number + 1) # concat lists
return [raw] # convert raw into a list object on the fly
As for the loop method, the logic is similar to recursion. Essentially, you will call the get_result() function a number of times, collect the results, and break early when the furthest page contains less than 500 results.
If you know the total number of results in advance, you can simply the run the loop for a predetermined number of times.
Do you follow? Do you have any further questions?
(I'm a little confused by what you mean by "load it into a JSON file". Do you mean saving the final raw results into a JSON file? Or are you referring to the .json() method in resp.json()? In that case, you don't need import json to do resp.json(). The .json() method on resp is actually part of the requests module.
On a bonus point, you can make your HTTP requests asynchronous, but this is slightly beyond the scope of your original question.
P.s. I'm happy to learn what other solutions, perhaps more elegant ones, that people use.
I am trying to compile monthly data in to an existing JSON file that I loaded via import json. Initially, my json data just had one property which is 'name':
json_data['features'][1]['properties']
>>{'name':'John'}
But the end result with the monthly data I want is like this:
json_data['features'][1]['properties']
>>{'name':'John',
'2016-01': {'x1':0, 'x2':0, 'x3':1, 'x4':0},
'2016-02': {'x1':1, 'x2':0, 'x3':1, 'x4':0}, ... }
My monthly data are on separate tsv files. They have this format:
John 0 0 1 0
Jane 1 1 1 0
so I loaded them via import csv and parsed through a list of urls and set about placing them in a collective dictionary like so:
file_strings = ['2016-01.tsv', '2016-02.tsv', ... ]
collective_dict = {}
for i in strings:
with open(i) as f:
tsv_object = csv.reader(f, delimiter='\t')
collective_dict[i[:-4]] = rows[0]:rows[1:5] for rows in tsv_object
I checked how things turned out by slicing collective_dict like so:
collective_dict['2016-01']['John'][0]
>>'0'
Which is correct; it just needs to be cast into an integer.
For my next feat, I attempted to assign all of the monthly data to the respective json members as part of their external properties:
for i in file_strings:
for j in range(len(json_data['features'])):
json_data['features'][j]['properties'][i[:-4]] = {}
json_data['features'][j]['properties'][i[:-4]]['x1'] = int(collective_dict[i[:-4]][json_data['features'][j]['properties']['name']][0])
json_data['features'][j]['properties'][i[:-4]]['x2'] = int(collective_dict[i[:-4]][json_data['features'][j]['properties']['name']][1])
json_data['features'][j]['properties'][i[:-4]]['x3'] = int(collective_dict[i[:-4]][json_data['features'][j]['properties']['name']][2])
json_data['features'][j]['properties'][i[:-4]]['x4'] = int(collective_dict[i[:-4]][json_data['features'][j]['properties']['name']][3])
Here I got an arrow pointing at the last few characters:
Syntax Error: unexpected EOF while parsing
It is a pretty complicated slice, I suppose user error is not to be ruled out. However, I did double and triple check things. I also looked up this error. It seems to come up with input() related calls. I'm left a bit confused, I don't see how I made a mistake (although I'm already mentally prepared to accept that).
My only guess was that something somewhere was not a string. When I checked collective_dict and json_data, everything that was supposed to be a string was a string ('John', 'Jane' et all). So, I guess it's something else.
I made the problem as simple as I could while keeping the original structure of the data and for loops and so forth. I'm using Python 3.6.
Question
Why am I getting the EOF error? How can I build my external properties data without encountering such an error?
Here I have rewritten your last code block to:
for i in file_strings:
file_name = i[:-4]
for j in range(len(json_data['features'])):
name = json_data['features'][j]['properties']['name']
file_dict = json_data['features'][j]['properties'][file_name] = {}
for x in range(4):
x_string = 'x{}'.format(x+1)
file_dict[x_string] = int(collective_dict[file_name][name][x])
from:
for i in file_strings:
for j in range(len(json_data['features'])):
json_data['features'][j]['properties'][i[:-4]] = {}
json_data['features'][j]['properties'][i[:-4]]['x1'] = int(collective_dict[i[:-4]][json_data['features'][j]['properties']['name']][0])
json_data['features'][j]['properties'][i[:-4]]['x2'] = int(collective_dict[i[:-4]][json_data['features'][j]['properties']['name']][1])
json_data['features'][j]['properties'][i[:-4]]['x3'] = int(collective_dict[i[:-4]][json_data['features'][j]['properties']['name']][2])
json_data['features'][j]['properties'][i[:-4]]['x4'] = int(collective_dict[i[:-4]][json_data['features'][j]['properties']['name']][3])
That is just to make it a bit more readable, but that shouldn't change anything.
A thing I noticed in your other part of code is the following:
collective_dict[i[:-4]] = rows[0]:rows[1:5] for rows in tsv_object
The thing I refer to is the = rows[0]:rows[1:5] for rows in tsv_object part. In my IDE, that does not work, and I'm not sure if that is a typo in your question or of that is actually in your code, but I imagine you want it to actually be
collective_dict[i[:-4]] = {rows[0]:rows[1:5] for rows in tsv_object}
or something like that. I'm not sure if that could confuse the parser think that there is an error at the end of the file.
The ValueError: Invalid literal for int()
If your tsv-data is
John 0 0 1 0
Jane 1 1 1 0
Then it should be no problem to do int() of the string value. E.g.: int('42') will become an int with value 42. However, if you have an error in one, or several, lines of your files, then use something like this block of code to figure out which file and line it is:
file_strings = ['2016-01.tsv', '2016-02.tsv', ... ]
collective_dict = {}
for file_name in file_strings:
print('Reading {}'.format(file_name))
with open(file_name) as f:
tsv_object = csv.reader(f, delimiter='\t')
for line_no, (name, *x_values) in enumerate(tsv_object):
if len(x_values) != 4:
print('On line {}, there is only {} values!'.format(line_no, len(x_values)))
try:
intx = [int(x) for x in x_values]
except ValueError as e:
# Catch "Invalid literal for int()"
print('Line {}: {}'.format(line_no, e))