I´m calling:
events: {
url: '/CondominioVip/evento/evento_json.json',
error: function() {
alert('there was an error while fetching events!');
}
}
I've also tried to add type: 'POST but it didn't work either.
My controller:
def evento_json():
events= "[{'title':'event1','start':'2010-01-01'},{'title':'event3','start':'2010-01-09 12:30:00','allDay':False}]"
return events
Test call from browser (http://localhost:8000/CondominioVip/evento/evento_json.json):
[{'title':'event1','start':'2010-01-01'},{'title':'event3','start':'2010-01-09 12:30:00','allDay':False}]
Request from web2py ajax function:
Accept:application/json, text/javascript, /; q=0.01
Accept-Charset:ISO-8859-1,utf-8;q=0.7,*;q=0.3
Accept-Encoding:gzip,deflate,sdch
Accept-Language:pt-BR,pt;q=0.8,en-US;q=0.6,en;q=0.4
Cache-Control:max-age=0
Connection:keep-alive
Content-Length:31
Content-Type:application/x-www-form-urlencoded
Cookie:session_id_admin=127.0.0.1-f9db7d99-4e7e-4bae-a229-d6614e9599f0; cvip_language=pt-BR; session_id_condominiovip=127.0.0.1-b820cbe3-9a55-40bb-8618-8d3f9a43f7c2
Host:localhost:8000
Origin:http://localhost:8000
Referer:http://localhost:8000/CondominioVip/evento/index/2
User-Agent:Mozilla/5.0 (Windows NT 6.1) AppleWebKit/536.5 (KHTML, like Gecko) Chrome/19.0.1084.46 Safari/536.5
X-Requested-With:XMLHttpRequest
Response headers:
Cache-Control:no-store, no-cache, must-revalidate, post-check=0, pre-check=0
Connection:keep-alive
Content-Length:105
Content-Type:application/json
Date:Tue, 29 May 2012 02:54:04 GMT
Expires:Tue, 29 May 2012 02:54:03 GMT
Pragma:no-cache
Server:Rocket 1.2.4 Python/2.7.3
Set-Cookie:cvip_language=pt-BR; expires=Wed, 30-May-2012 02:54:04 GMT; Path=/, session_id_condominiovip=127.0.0.1-b820cbe3-9a55-40bb-8618-8d3f9a43f7c2; Path=/
X-Powered-By:web2py
Response:
[{'title':'event1','start':'2010-01-01'},{'title':'event3','start':'2010-01-09 12:30:00','allDay':False}]
I'm out of options. Any help would be apprecianted.
Update:
I've installed the JsonView extension on Chrome, so returning a string is not considered a json response.
I made some changes:
def evento_json():
rows = db(evento.id>0).select(evento.id,evento.titulo,evento.data_hora_inicio,evento.data_hora_fim)
events = []
for row in rows:
event = {'title': row['titulo'],
'start': row['data_hora_inicio'],
'end': row['data_hora_fim'],
'allDay': False,
'url': URL(c='evento', f='index', args=[row['id']], extension=False)}
events.append(event)
return events
But web2py throws an error. I´ve printed "events" and "json(events)" before send it to generic.json, and the format is exactly what fullcalendar expects.
The way I found to stop the error is:
In controller:
def evento_json():
import datetime
start = datetime.datetime.fromtimestamp(int(request.vars['start'])).strftime('%Y-%m-%d %H:%M:%S')
end = datetime.datetime.fromtimestamp(int(request.vars['end'])).strftime('%Y-%m-%d %H:%M:%S')
set = db((evento.id>0) &
(evento.data_hora_inicio >= start) &
(evento.data_hora_fim <= end))
if (not auth.has_membership(auth.id_group(role='site_admin'), auth.user.id)) and \
(not auth.has_membership(auth.id_group(role='cond_admin'), auth.user.id)):
set = set(evento.flag_disp==True)
rows = set.select(evento.id,
evento.titulo,
evento.data_hora_inicio,
evento.data_hora_fim,
evento.flag_disp)
events = []
for row in rows:
event = {'title': row['titulo'],
'start': row['data_hora_inicio'],
'end': row['data_hora_fim'],
'allDay': False,
'url': URL(c='evento', f='index', args=[row['id']], extension=False),
'color': 'blue' if row['flag_disp'] is True else 'red'}
events.append(event)
if events:
from gluon.serializers import json
return XML(json(events))
else:
return '{}'
And create a view evento/evento_json.json with the following:
{{=response._vars)}}
It works! But seems a bug to me. I'm not sure if I'm doing something wrong.
Maybe try:
events= "[{'title':'event1','start':'2010-01-01'},{'title':'event3','start':'2010-01-09T12:30:00Z','allDay':false}]"
(Changed the event3 start date/time to ISO8601 format as per the API, and changed "False" to "false".)
Related
I was unable to use the requests library and use the get() function to scrape data from this specific website as running the below code block will result in a status code of 403 (unsuccessful)
import requests
#using headers in order to emulate a browser
headers = {'user-agent': 'Chrome/55.0.2883.87'}
url = "https://www.rumah.com/properti-dijual"
# Make a request to the website and retrieve the data
response = requests.get(url, headers=headers)
# Check if the request was successful
if response.status_code == 200:
print("Request was successful", response.status_code)
# Save the source code as a text file
print(response.text)
else:
print("Request was not successful", response.status_code)
However, when I tried the same source code trying to scrape a different website, the request was successful (status code 200).
import requests
#using headers in order to emulate a browser
headers = {'user-agent': 'Chrome/55.0.2883.87'}
url = "https://www.subscene.com"
# Make a request to the website and retrieve the data
response = requests.get(url, headers=headers)
# Check if the request was successful
if response.status_code == 200:
print("Request was successful", response.status_code)
# Save the source code as a text file
print(response.text)
else:
print("Request was not successful", response.status_code)
I'm trying to scrape housing data from the website by getting a successful request to the website. I realized that some websites prevent scraping and those specific pages are listed in the robots.txt file. However, I can't find the specific page that I want to scrape in the robots.txt file, therefore I thought that I should be able to scrape this website.
Here is the robots.txt file for the specific webpage:
enter image description here
This is my first question in StackOverflow. Any help would be appreciated!
Your url https://www.rumah.com/properti-dijual is using cloudfare protection, and https://www.subscene.com as well.
But maybe, https://www.subscene.com has a more strict policy.
In case your getting error 403:
provide all headers as following:
import requests
headers = {
'authority': 'www.rumah.com',
'accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9',
'accept-language': 'de,de-DE;q=0.9,en;q=0.8,en-GB;q=0.7,en-US;q=0.6,fr;q=0.5,de-CH;q=0.4',
'cache-control': 'no-cache',
'dnt': '1',
'pragma': 'no-cache',
'sec-ch-ua': '"Not_A Brand";v="99", "Microsoft Edge";v="109", "Chromium";v="109"',
'sec-ch-ua-mobile': '?0',
'sec-ch-ua-platform': '"Windows"',
'sec-fetch-dest': 'document',
'sec-fetch-mode': 'navigate',
'sec-fetch-site': 'none',
'sec-fetch-user': '?1',
'upgrade-insecure-requests': '1',
'user-agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/109.0.0.0 Safari/537.36 Edg/109.0.1518.70',
}
response = requests.get('https://www.rumah.com/properti-dijual', headers=headers)
If that doesn't work, try using javascript:
fetch("https://www.rumah.com/properti-dijual", {
"headers": {
"accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9",
"accept-language": "de,de-DE;q=0.9,en;q=0.8,en-GB;q=0.7,en-US;q=0.6,fr;q=0.5,de-CH;q=0.4",
"cache-control": "no-cache",
"pragma": "no-cache",
"sec-ch-ua": "\"Not_A Brand\";v=\"99\", \"Microsoft Edge\";v=\"109\", \"Chromium\";v=\"109\"",
"sec-ch-ua-mobile": "?0",
"sec-ch-ua-platform": "\"Windows\"",
"sec-fetch-dest": "document",
"sec-fetch-mode": "navigate",
"sec-fetch-site": "none",
"sec-fetch-user": "?1",
"upgrade-insecure-requests": "1"
},
"body": null,
"method": "GET",
"mode": "cors",
});
You can also initiate javascript with python using Selenium or Selenium-Profiles (undetected, uses Chrome)
I am getting a bad request response to my request. I have checked with an online JSON validator my dictionary data to be correct, and everything seems fine.
My code is the following:
// Parse datetime to timestamp and include data in a dict
let data_dict = {
"stop_date": Date.parse(sup_limit.value),
"start_date": Date.parse(inf_limit.value)
}
// Send the Ajax request
let request = $.ajax({
url: url,
type: 'POST',
data: data_dict,
contentType: 'application/json;charset=UTF-8',
});
Backend receive endpoint:
#dashboard_bp.route('/download_last_test_influx<mode>', methods=['GET', 'POST'])
#login_required
def download_last_test_influx(mode: str):
# Check if request comes from a custom or test event
if mode == 'custom':
start_date = int(request.json.get('start_date'))
stop_date = int(request.json.get('stop_date'))
# Check if time range is valid, if not return server internal error
if stop_date - start_date <= 0:
return jsonify({'message': 'Time range must be grater than 0'}), 500
# Create response header
response = make_response(send_file(spock_comm_mgr
.test_backup_influx_manager
.get_last_test_influx_record(start_date=start_date, stop_date=stop_date)))
response.headers['Content-Type'] = 'application/gzip'
response.headers['Content-Encoding'] = 'gzip'
return response
Request header:
POST /download_last_test_influxcustom HTTP/1.1
Host: 0.0.0.0:5000
User-Agent: Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:89.0) Gecko/20100101 Firefox/89.0
Accept: */*
Accept-Language: en-US,en;q=0.5
Accept-Encoding: gzip, deflate
Content-Type: application/json;charset=UTF-8
X-Requested-With: XMLHttpRequest
Content-Length: 48
Origin: http://0.0.0.0:5000
Connection: keep-alive
Referer: http://0.0.0.0:5000/influx_management
Cookie: *********************
Request payload:
stop_date=1623758400000&start_date=1623708000000
Response message:
Bad Request
The browser (or proxy) sent a request that this server could not understand.
You are telling your server, you are sending JSON data, but the request body is not a JSON string but a url-encoded string (because that's the default behaviour of $.ajax() when you pass an object as data).
Use JSON.stringify, to pass a correct JSON body
let request = $.ajax({
url: url,
type: 'POST',
data: JSON.stringify(data_dict),
contentType: 'application/json;charset=UTF-8',
});
I got a problem which I dont understand.
I try to post data to my API in a form using the following code
formSubmit() {
const req =this.http.post('http://[ip]/api/login', {
id: '7',
username: 'PostTest',
password: 'studp123lan',
matrikelnr: 'winf303666',
email: 'winf303666#example.de',
email_verified: '1'
})
.subscribe(
res => {
console.log(res);
},
err => {
console.log("Error occured");
}
When I inspect it in the Chrome Developter tools, this is what I get:
Failed to load http://[ip]/api/login: Response for preflight has
invalid HTTP status code 404
register.component.ts:42 Error occured
And this is what I get in the network tab:
General:
Request URL:http://[ip]/api/login
Request Method:OPTIONS
Status Code:404 Not Found
Remote Address:[ip]:80
Referrer Policy:no-referrer-when-downgrade
Response Header:
Access-Control-Allow-Headers:Content-Type
Access-Control-Allow-Methods:GET, POST, PUT, DELETE, OPTIONS
Access-Control-Allow-Origin:*
Content-Length:0
Date:Thu, 21 Dec 2017 09:00:35 GMT
Server:Kestrel
X-Powered-By:ASP.NET
Request Header:
Accept:*/*
Accept-Encoding:gzip, deflate
Accept-Language:de-DE,de;q=0.9,en-US;q=0.8,en;q=0.7
Access-Control-Request-Headers:content-type
Access-Control-Request-Method:POST
Connection:keep-alive
Host:[ip]
Origin:http://localhost:4200
User-Agent:Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36
(KHTML, like Gecko) Chrome/63.0.3239.108 Safari/537.36
Somehow, this doesn't work. But when I Post the same data via Postman Post Request to the same URL, it works like a charm.
Can anyone explain and help?
Thanks.
404 is a page not found error : this means your endpoint isn't available at this address.
Sure, you make a postman call and it works : but did you create your postman call by hand, or used the interceptor to make it ? (The interceptor is a Chomr plugin that allows you to register all calls made by Chrome, into postman).
There must be something you have forgotten. Could you post your postman call, and if you can, try with the interceptor ?
i get request from android and IOS clients. I store the request in an array.
then i encode it to JSON. Now i want to sent only the JSON data in my HTTP header. How to do that? Currently its sending the html page in response i want to send only json encoded data in 200 OK header.
Below is my code:
public function index()
{
$request = array(
'request' => $this->input->get('request'),
'device_id' => $this->input->get('device_id'),
'launch_date'=> $this->input->get('launch_date'),
'allowed_hours'=>$this->input->get('allowed_hours'),
'site_id'=> $this->input->get('site_id'),
'product'=>$this->input->get('product'),
'software_version'=> $this->input->get('software_version'),
'platform_os'=> $this->input->get('platform_os'),
'platform'=> $this->input->get('platform'),
'platform_model'=> $this->input->get('platform_model')
);
$response = array(
'response_code' =>200 ,
'device_id'=> $this->input->get('device_id'),
'allowed_hours'=> $this->input->get('allowed_hours'),
'product'=>'mlc',
'prov_ur'=>NULL
);
return $this->output
->set_content_type('Content-Type: application/json')
->set_output(json_encode($response));
}
}
MY response:
A PHP Error was encountered
Severity: Notice
Message: Undefined index: request
Filename: admin/license.php
Line Number: 22
A PHP Error was encountered
Severity: Notice
Message: Undefined index: allowed_hours
Filename: admin/license.php
Line Number: 25
A PHP Error was encountered
Severity: Notice
Message: Undefined index: allowed_hours
Filename: admin/license.php
Line Number: 40
{"response_code":200,"device_id":"70D0D01FBAD2","allowed_hours":null,"product":"mlc","prov_ur":null}array(10) { ["Host"]=> string(14) "192.168.50.123" ["Connection"]=>
string(10) "keep-alive" ["Cache-Control"]=> string(9) "max-age=0" ["Accept"]=> string(74) "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,/;q=0.8" ["Upgrade-Insecure-Requests"]=> string(1) "1" ["User-Agent"]=>
string(110) "Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/47.0.2526.111 Safari/537.36"
["Accept-Encoding"]=> string(19) "gzip, deflate, sdch"
["Accept-Language"]=> string(14) "en-US,en;q=0.8" ["Cookie"]=>
string(518) "cisession=OhhBBhVodwwf7Tb55AVsU32ClMS5cgmxBl15WHA%2BrGnvo1kiK%2B67BWeAuJVSV2MY25zZd0riHC9cyx9fiigiBuqkPMT%2FKE9d6et%2FXaE3F7I59P9%2FEzy5byQ5nEkJq5xwXoH1I7%2B7v62cQL21%2Bjfnk3AwIy4luM7N51IUuTqg7TxunoZFD1gJO84r8degY1imNpmDk2W%2FjsQPn9bQpkWJ9KVMxxViFDaELEU0rIfYmif%2BdvXjK9W%2Fj7iWQxZYE9ZGazgBTKlLO%2BJZHNdPrdmGPFTzTUROZdffpF%2Bb25bRMPEJsZ9CE2mdVuSn%2FEu678utd0lcd9bh%2BDbTDikrHP4jBFOLbZfWKT%2F9r5GkMBrLBl%2BlvPx9RbAq%2FIsjeA1V7c6JYf41TO1bG2XKT14QFHm8m0qY8HCal%2B%2BR8tZe9i3zy24%3Dcfc459942e4ef82a5554257216a19d621f446a25" ["If-Modified-Since"]=> string(29) "Thu, 01 Jan 1970 00:00:00 GMT" }
this is not a header problem your array do not have "allowed_hours"
Message: Undefined index: allowed_hours
you can change your code like this
$allowed_hours = "";
if(!empty($this->input->get('allowed_hours'))){
$allowed_hours = $this->input->get('allowed_hours');
}
$response = array(
'response_code' =>200 ,
'device_id'=> $this->input->get('device_id'),
'allowed_hours'=> $allowed_hours,
'product'=>'mlc',
'prov_ur'=>NULL
);
I trying to understand how to deal with HTTP error codes using web.py as a REST framework. I can easily use try/catch blocks to return HTTP 404, 400, 500, etc... but I am having a hard time sending a custom JSON message with it.
import web
import json
urls = (
'/test/(.*)', 'Test'
)
app = web.application(urls, globals())
def notfound():
return web.notfound(json.dumps({'test': 'test'}))
class Test:
def GET(self, id):
web.header('Content-Type', 'application/json')
return self.get_resource(str(id))
def get_resource(self, id):
result = {}
if id == '1':
result = {'1': 'one'}
elif id == '3':
return web.notfound()
return json.dumps(result)
if __name__ == '__main__':
web.config.debug = False
app.notfound = notfound
app.run()
This works fine, but when id == 3, I cannot override the behaviour, and the Content-Type header is duplicated:
# curl -i -H "Accept: application/json" http://localhost:8080/test/3
HTTP/1.1 404 Not Found
Content-Type: application/json
Content-Type: text/html
Transfer-Encoding: chunked
Date: Mon, 09 Sep 2013 23:59:28 GMT
Server: localhost
404 Not Found
How can I return a JSON Content-Type, with a custom message?
HTTP Errors in web.py should be raised as exceptions.
I used this class to decorate json error (it has specific output format in my app, so you may adopt it to your needs):
class NotFoundError(web.HTTPError):
'''`404 Not Found` error.'''
headers = {'Content-Type': 'application/json'}
def __init__(self, note='Not Found', headers=None):
status = '404 Not Found'
message = json.dumps([{'note': note}])
web.HTTPError.__init__(self, status, headers or self.headers,
unicode(message))
I have created a simple json api with web.py you may want to check, believe me it has some interesting ideas.