I am trying to extract the text part from the request that I made through grequest library but I am unable to figure out how can I do so.
If we use Requests Library I would do
r = requests.get('www.google.com')
htmls.append(r.text)
Now if I am using grequests I can only get a list of response code and not text.
rs = (grequests.get(u) for u in urls)
result = grequests.map(rs)
What I've tried
result = grequests.map(rs.text)
I get an error using above piece of code AttributeError: 'generator' object has no attribute 'text'
My desired output is a list of html text where response code is 200 else the value should be None.
How can I achieve that?
Desired Output:
response_code = [<Response [200]>,<Response [404]>,<Response [200]>]
htmls = ['html1', None, 'html2']
You can use something like below
rs = (grequests.get(u) for u in urls)
responses = grequests.map(rs)
text = list(map(lambda d : d.text if d else None, responses))
print(text)
What you are getting back is a response array after you call the map. And then you can process this data using native map function
Related
I have a Pyramid web-application that I am trying to unit-test.
In my tests file I have this snippet of code:
anyparam = {"isApple": "True"}
#parameterized.expand([
("ParamA", anyparam, 'success')])
def test_(self, name, params, expected):
request = testing.DummyRequest(params=params)
request.session['AI'] = ''
response = dothejob(request)
self.assertEqual(response['status'], expected, "expected response['status']={0} but response={1}".format(expected, response))
Whereas in my views:
#view_config(route_name='function', renderer='json')
def dothejob(request):
params = json.loads(request.body)
value = params.get('isApple') #true or false.
However, when I'm trying to unit-test it, I am getting this error:
JSONDecodeError: Expecting value: line 1 column 1 (char 0)
However, when I make the same request via POST via web-browser it works perfectly fine.
By doing testing.DummyRequest(params=params) you are only populating request.params, not request.body.
You probably want to do something like:
request = testing.DummyRequest(json_body=params)
Also, you may want to use directly request.json_body in your code instead of json.loads(request.body).
Via the modules urllib and re I am attempting to scrape a web page for it's text content. I'm following a guide provided by "SentDex" on youtube found here ( https://www.youtube.com/watch?v=GEshegZzt3M ) and the documentation with the official Python site to cobble together a quick solution. The information that comes back has plenty of HTML markup and special characters that I am trying to remove. My end result is successful but I feel that it is hard coded solution and is only useful for this one scenario.
The code is as follows :
url = "http://someUrl.com/dir/doc.html" #Target URL
values = {'s':'basics',
'submit':'search'} #Set parameters for later use
data = urllib.parse.urlencode(values) #Really not sure...
data = data.encode('utf-8') #set to UTF-8
req = urllib.request.Request(url,data)#Arrange the request parameters
resp = urllib.request.urlopen(req)#Get the document's contents matching that data type from that URL
respData = resp.read() #read the content into a variable
#BS4 method
soup = BeautifulSoup(respData, 'html.parser')
text = soup.find_all("p")
#end BS4
#re method
text = re.findall(r"<p>(.*?)</p>",str(respData)) #get all paragraph tag contents
text = str(text) #convert it to a string
#end re
conds = ["<b>","</b>","<i>","</i>","\\","[","]","\'"] #things to remove from text
for case in conds:#for each of those things
text = text.replace(case,"") #remove string AKA replace with nothing
Are there more effective ways to achieve the end goal of removing all "Markup" from a string further than explicit definitions of each condition?
This code seems to work fine when I used Django console and just print it.
reference = FloodHazard.objects.filter(hazard='High')
ids = reference.values_list('id', flat=True)
for myid in ids:
getgeom = FloodHazard.objects.get(id=myid).geom
response = BuildingStructure.objects.filter(geom__intersects=getgeom).values(
'brgy_locat').annotate(counthigh=Count('brgy_locat'))
print response
I was able to show all the values, but when using HttpResponse, it returns an empty set. What is the proper way of returning JSON data from a queryset? So far, tried this:
reference = FloodHazard.objects.filter(hazard='High')
ids = reference.values_list('id', flat=True)
response = {}
for myid in ids:
getgeom = FloodHazard.objects.get(id=myid).geom
response['high'] = BuildingStructure.objects.filter(geom__intersects=getgeom).values(
'brgy_locat').annotate(counthigh=Count('brgy_locat'))
json_post = ujson.dumps(list(response))
return HttpResponse(json_post, content_type='application/json')
There is no much sense in your code. You assign all querysets to the single key in the response dict. You should use a list for this purpose:
As far as I understand the code should be something like this:
response = []
for myid in ids:
getgeom = FloodHazard.objects.get(id=myid).geom
response.extend(BuildingStructure.objects.filter(geom__intersects=getgeom)
.values('brgy_locat')
.annotate(counthigh=Count('brgy_locat')))
json_post = ujson.dumps(response)
If you want to return a hazard level as well as the list of buildings then you can return a dict:
json_post = ujson.dumps({'hazard': 'high', 'buildings': response})
I have written the following code to send my gps locations saved in a Lua table to parse.com; the display.newText is a test to see if the Lua table data works properly which does so. The problem is that the POST network request doesn't work with my "message" table when used as the JSON parameter. Although exact what you see is working when I remove the -- from the following code. This defines the "message" table variables again. I use Corona SDK.
function post ( message )
test12 = display.newText( message["x1"] .. message["y1"] .. message["x2"] .. message["y2"]..message["v"], 150, time/100, "Verdana-Bold", 10 )
--message = { x1 = 22.312456, y1 = 22.312456, x2 = 22.312456, y2 = 22.312456, v = 10 }
params.body = json.encode ( message )
network.request( baseUrl .. objectClass, "POST", networkListener, params)
end
I think you need to have a better understanding of what Parse is looking for. I don't know Parse well enough to tell you what it is, but HTTP POST requires a body that is a set of HTTP Query String style key-value pairs. We've seen GET use things like:
http:/somesite.com/someurl.php?key=value&key2=value&key3=value
HTTP GET is limited to like 255 characters and the values have to be URLencoded.
HTTP POST is just like GET but it allows longer blocks of data to be sent on the input stream of the device These blocks of data are still Query String key-value pairs. I would expect the body statement (though not for your usage of Parse) to be something like:
params.body = "key=value&key2=value&key3=value" where the values of course could be much longer and any values properly URL encoded. Just shoving a block of JSON data in there isn't going to be something a POST script would parse. Perhaps something like:
params.body = "query=" .. urlencode( json.encode( someLuaTable ) )
You can use this URL encoding function:
function urlencode(str)
if (str) then
str = string.gsub (str, "\n", "\r\n")
str = string.gsub (str, "([^%w ])",
function (c) return string.format ("%%%02X", string.byte(c)) end)
str = string.gsub (str, " ", "+")
end
return str
end
But you need to figure out what key Parse wants for its input.
Your message has not the right format.
try this:
message = {"x1":22.312456,"y1":22.312456,"x2":22.312456,"y2":22.312456,"v":10}
cheers,
Felix
The issue was the Corona changes the variable to string while parse.com expected numbers. I simply multiplied all the table variables by 1 and then the whole things works ok.
function post ( message )
test12 = display.newText( message["x1"] .. message["y1"] .. message["x2"] .. message["y2"]..message["v"], 150, time/100, "Verdana-Bold", 10 )
--message = { x1 = 22.312456, y1 = 22.312456, x2 = 22.312456, y2 = 22.312456, v = 10 }
message["x1"] = message["x1"]*1
message["y1"] = message["y1"]*1
message["y2"] = message["y2"]*1
message["x2"] = message["x2"]*1
message["v"] = message["v"]*1
params.body = json.encode ( message )
network.request( baseUrl .. objectClass, "POST", networkListener, params)
end
If you have similar issues with Sqlite in corona that's the way to fix it.
How do I access what has been posted by a client to my classic ASP server?
I know that there is the Request.Forms variable, but the client's request was not made using a Form.
The client request's body is just a string made using a standard POST statement.
Thanks
You need to read request bytes if content type of request sent by client is not form data. In this case, request is not a form-data that is accessible through name-value pairs so you cannot use Request.Form collection. I suggest investigate the BinaryRead method.
Reading posted data and convert into string :
If Request.TotalBytes > 0 Then
Dim lngBytesCount
lngBytesCount = Request.TotalBytes
Response.Write BytesToStr(Request.BinaryRead(lngBytesCount))
End If
Function BytesToStr(bytes)
Dim Stream
Set Stream = Server.CreateObject("Adodb.Stream")
Stream.Type = 1 'adTypeBinary
Stream.Open
Stream.Write bytes
Stream.Position = 0
Stream.Type = 2 'adTypeText
Stream.Charset = "iso-8859-1"
BytesToStr = Stream.ReadText
Stream.Close
Set Stream = Nothing
End Function
Hope it helps.
Update #1:
With using JScript
if(Request.TotalBytes > 0){
var lngBytesCount = Request.TotalBytes
Response.Write(BytesToStr(Request.BinaryRead(lngBytesCount)))
}
function BytesToStr(bytes){
var stream = Server.CreateObject("Adodb.Stream")
stream.type = 1
stream.open
stream.write(bytes)
stream.position = 0
stream.type = 2
stream.charset = "iso-8859-1"
var sOut = stream.readtext()
stream.close
return sOut
}
To get the JSON string value just use CStr(Request.Form)
Works a treat.
In Classic ASP, Request.Form is the collection used for any data sent via POST.
For the sake of completeness, I'll add that Request.QueryString is the collection used for any data sent via GET/the Query String.
I would guess based on the above that even though the client is not a web browser, the Request.Form collection should be populated.
note: all of this is assuming the data being sent is textual in nature, and that there are no binary uploads (e.g. pictures or files) being sent. Update your question body if this is an incorrect assumption.
To test, write out the raw form data and see what you have - something along the lines of:
Response.Write(Request.Form)
Which with a regular web page will output something like
field=value&field2=value2
If you get something along those lines, you could then use that as a reference for a proper index.
If you do not get something like that, update your question with what you tried and what you got.