Is it possible somehow export Response information from Network tab from Chrome Dev Tools with python?
Thank you.
You can do this by right-clicking on the request to be exported and there will be
"Save all as HAR with content",Or directly copy the string that returns the result
If you're using Python then you can get the same information that you see in the Response tab, using the requests library,
>>> import requests
>>> response = requests.get("http://www.google.com")
>>> print response.content
Related
Requirement:
To get the data from a URL into S3 and build a data pipeline. I am trying to get the dataset using python requests.get().
However, the URL changes every few minutes, and it becomes difficult to fetch the data using requests.get() without manual changes to the python script.
My Approach:
I am able to get the data to S3 and build the data pipeline. However, the URL is dynamic I am kinda stuck there.
Code:
import json
import requests
import boto3
import csv
import re
def lambda_handler(event, context):
response= requests.get("https://calcat.covid19.ca.gov/cacovidmodels/_w_d58f724d/session/d37479fcceb6b02080b6b7c23dd05046/download/dl.custom?w=d58f724d",verify=False)
#print(response.content)
s3_client = boto3.client('s3',region_name='us-west-2')
with open('/tmp/data.csv', 'w') as datacsvfile:
writer = csv.writer(datacsvfile, delimiter=",")
for line in response.iter_lines():
writer.writerows([line.strip().split(',')])
Query:
How can I get the href from the "Inspect element" section , store it in a variable and create a dynamic URL to make a REST call to. Below is the "Inspect element" code I am referring to :
This href only appears in the "inspect element" and not in the "View Source Code".
I'm working on a second house pricing project, so I need to scrape information from one of the largest second house trading platform in China. Here's my problem, the info on the page and the corresponding element using Chrome 'inspect' function are as follows:
my code:
>>>from lxml import etree
>>>import requests
>>>url = 'http://bj.lianjia.com/chengjiao/101101498110.html'
>>>r = requests.get(url)
>>>xiaoqu_avg_price = tree.xpath('//[#id="resblockCardContainer"]/div/div/div[2]/div/div[1]/span/text()')
>>>xiaoqu_avg_price
[]
The returned empty list is not desirable (ideally it should be 73648). Furthermore, I viewed its HTML source code, which shows:
So how should I do to get what I want? And what is the resblockCard means? Thanks.
This site like many others uses ajax for populating content. If you make a similar request you can get desired value in json format.
import requests
url = 'http://bj.lianjia.com/chengjiao/resblock?hid=101101498110&rid=1111027378082'
# Get json response
response = requests.get(url).json()
print(response['data']['resblock']['unitPrice'])
# 73648
Note two group of numbers in request url. The first group from original page url, second you can find under script tag in the original page source: resblockId:'1111027378082'.
That XPath query is not working as expected because you are running it against the source code of the page as it is served by the server, not as it looks on a rendered browser page.
One solution for this is to use Selenium in conjunction with PhantomJS or some other browser driver, which will run the JavaScript on that page and render it for you.
from selenium import webdriver
from lxml import html
driver = webdriver.PhantomJS(executable_path="<path to>/phantomjs.exe")
driver.get('http://bj.lianjia.com/chengjiao/101101498110.html')
source = driver.page_source
driver.close() # or quit() if there are no more pages to scrape
tree = html.fromstring(source)
price = tree.xpath('//div[#id="resblockCardContainer"]/div/div/div[2]/div/div[1]/span/text()')[0].strip()
The above returns 73648 元/㎡.
I have an API who is generarted using loopback / strongloop and its running as follows.
and i want to export the the generated api into yaml or json so that i want to reuse it in another application. i m looking for swagger.json file.
SO in sawgger you get your json from your running api by going to localhost:3300/api-docs. How do i get it from here ?
You can do that with
localhost:3300/explorer/resources to get a list of all resources and localhost:3300/explorer/resource/ModelPluralName to get swagger for specific resource.
Also you can click on "Raw" link in your API explorer.
*Assuming of course that your application is running on localhost port 3300.
you can do that simply by typing
localhost:3300/explorer/swagger.json ,
you need to download the json file and load that into editor.swagger.io using file import then download it as YAML to feed into your restAPi ppliaction.
I am trying to send some JSON data to an Odoo controller, but when I send the request, I always get 404 as response.
This is the code of my controller:
import openerp.http as http
import logging
_logger = logging.getLogger(__name__)
class Controller(http.Controller):
#http.route('/test/result', type='json', auth='public')
def index(self, **args):
_logger.info('The controller is called.')
return '{"response": "OK"}'
Now, I type the URL (http://localhost:8069/test/result) on the browser to check if it is available, and I get function index at 0x7f04a28>, /test/result: Function declared as capable of handling request of type 'json' but called with a request of type 'http'. This way I know that the controller is listening at that URL and is expecting JSON data.
So I open a Python console and type:
import json
import requests
data = {'test': 'Hello'}
data_json = json.dumps(data)
r = requests.get('http://localhost:8069/test/result', data=data_json)
When I print r in the console, it returns <Response [404]>, and I cannot see any message in the log (I was expecting The controller is called.).
There is a similar question here, but it is not exactly the same case:
OpenERP #http.route('demo_json', type="json") URL not displaying JSON Data
Can anyone help me? What am I doing wrong?
I have just solved the problem.
Firstly, as #techsavvy told, I had to modify the decorator, to write type='http' instead of type='json'.
And after that, the request from the console returned a 404 error because it did not know which database it was sending data to. In localhost:8069 I had more than one database. So I tried to have only one at that port. And that is, now it works great!
To manage that without removing any of the other databases, I have just modified the config file to change the parameter db_filter and put there a regular expression which only included my current database.
I have just gone through your issue and I noticed that you have written JSON route which is call from javascript. if you want to call it from browser url hit then you have to define router with type="http" and auth="public" argument in route:
#http.route('/', type='http', auth="public", website=True)
I'm trying Jmeter tool for load testing where i'm feeding the data through a csv file which has all the emails and passwords for login request. But while passing the parameter, Jmeter is encoding '#' sign with '%40' and if i put %40 in place of # in my csv, its not getting decoded to # in Jmeter. For other special characters, the encoding and decoding is happening properly. Please help.
It should be totally expected.
If you're logging in via GET request %40 is correct way of encoding # symbol.
If you're sending a POST request, JMeter should automatically send # symbol (at least my JMeter 2.10 does)
You might wish to try one of following:
Add View Results Tree listener, switch to HTTP tab and see what's actually being sent.
Make sure that Encode? box is unchecked for email parameter
Explicitly tell JMeter to decode email via __urldecode() function
Use a Beanshell Pre Processor to properly encode/decode your email
import java.net.URLDecoder;
import java.net.URLEncoder;
String email = "someone#example.com";
String encoded = URLEncoder.encode(email, "UTF-8");
String decoded = URLDecoder.decode(encoded, "UTF-8");
This is coming when we do via Parameters,
If we do using "Body Data" that would work fine.
I used this way.
{"password":"${password}","emailId":"${emailId}"}
For the HTTP Request, Change the Client implementation to Java
Select the Advanced tab from the HTTP Request
In Client Implementation > Choose Java in Implementation