I am trying to scrape the table on the following website: https://bookli.app/resource-bookings-timeline/?locationId=cl3smb6ek150150ds67q3dvtnw
When I go to inspect > network, the data seems to come from the following API:
https://api.bookli.app/graphql
Is there any way to receive the information here or do I need some sort of key?
I tried to get the proper information from the API but it does not seem to work.
Related
Currently, I'm working with Beautiful Soup to extract information from websites. I've managed to scrape data from multiple pages from a certain apartment renting website with it - let's call it “Website A”. Now, I'd like to obtain data from another renting websites (“Website B”). I tried to follow a similar procedure as previously, but it failed because Website B requires a login.
I did manage to scrape the first page of apartments of Website B by means of Adelin's answer. His/her approach is based on the usage of Curl Trillworks (link). In principle, this approach could work for Website B as well. However, then one would need to manually repeat the procedure for the 800 or so pages on which the apartments are listed, and afterwards do the same for each of the 15 apartments per page.
This is too much work for me, so I try to automate the process. For instance, I tried adapting this to my situation, but I haven't succeeded so far. The dictionary I get is empty. I also tried making a new header for each page by putting a new referer each time in the original header. Then I'd put these referers in the the header dictionary. However, this failed - probably because websiteB recognized I was using the same cookie everytime I sent a request (the same one I used for the original apartment page for Website B).
So my question is:
Suppose one would have a list of pages of Website B that all have the same format (www.websiteB.com/PageNumber/ ). How would one
quickly/automatically obtain a header for each page by means of your
own login credentials for the website, with which one can create an appropriate response?
I could share the code I have so far, but I'm somewhat hesitant as this is a large commercial website and I suspect they aren't particularly happy with me sharing code that allows their website to be scraped and names the website itself as well.
I'm using an API to receive information from a server. I'm authorising using a token. I can query using curl or various api testers online. But now I'm trying to learn how to implement (and request) the results in practice.
I'd like to query, parse and display the json result on a web page (preferably a Wordpress site).
So I need to make a get request, authorise myself using my token, and receive the result and then display it in a nice way.
How do I do that?
This is a vague question, not really sure if you are making requests from just one side but you have 2 options, albeit one being better than the other.
You can create add multiple functions inside your themes function.php file to extend its capabilities but again these are limited to only the theme you are using.
I would recommend that you build a plugin, this will give you the flexibility to use both JS and PHP to do whatever you need to do. With this method you can create a shortcode to display the results from your API query.
You will need to learn to use the Wordpress function wp_remote_get() to perform external API calls.
Here is a tutorial with more information:
Create Plugin and Create External API Calls
Parse JSON from Remote API
Hope this helps
I'm trying to build a super simple Google Sheet dashboard comparing the prices at D+7 and D+30 in real-time of specific listings/rooms that are both on Airbnb and Booking.com.
On the Booking.com side, it was super easy : I just created a formula concatenating the URL with the check-in/check-out dates, number of guests and trip duration as parameters and using the =IMPORTXML function and the proper class, I was able to automatically retrieve the price.
It is more difficult on Airbnb, as the price is dynamic(see here: https://www.airbnb.com/rooms/25961741). When I use what I think is the proper class, I get a "N/A Error, Imported content is empty" on Google Sheet.
I also tried using the Airbnb API with REGEX functions to extract the price, but the price set in the listing info is a default price, and does not reflect reality:
"price":1160,"price_formatted":"$1160"
https://api.airbnb.com/v2/listings/25961741?client_id=d306zoyjsyarp7ifhu67rjxn52tv0t20&_format=v1_legacy_for_p3&number_of_guests=1
Do you now if there are any other possible way to access this dynamic price and have it automatically parsed into a spreadsheet? It seems that the data I'm looking for in within meta tags on the HTML code and I don't know if it's possible to scrape it into Google sheet using =IMPORT functions.
Maybe with of a script ?
Thanks a lot !
I'm curious if you were unable to yank direct with the ABNB API; what if you tried to directly pull off the site's service? Have a look at this URL:
https://www.airbnb.com/api/v2/explore_tabs?version=1.3.9&satori_version=1.0.7&_format=for_explore_search_web&experiences_per_grid=20&items_per_grid=18&guidebooks_per_grid=20&auto_ib=false&fetch_filters=true&has_zero_guest_treatment=false&is_guided_search=true&is_new_cards_experiment=true&luxury_pre_launch=false&query_understanding_enabled=true&show_groupings=true&supports_for_you_v3=true&timezone_offset=-240&client_session_id=8e7179a2-44ab-4cf3-8fb8-5cfcece2145d&metadata_only=false&is_standard_search=true&refinement_paths%5B%5D=%2Fhomes&selected_tab_id=home_tab&checkin=2018-09-15&checkout=2018-09-27&adults=1&children=0&infants=0&click_referer=t%3ASEE_ALL%7Csid%3A61218f59-cb20-41c0-80a1-55c51dc4f521%7Cst%3ALANDING_PAGE_MARQUEE&allow_override%5B%5D=&price_min=16&federated_search_session_id=5a07b98f-78b2-4cf9-a671-cd229548aab3&screen_size=medium&query=Paris%2C%20France&_intents=p1&key=d306zoyjsyarp7ifhu67rjxn52tv0t20¤cy=USD&locale=en
This is a GET request to ABNB's live page search; now I don't know much about ABNB but I can see from the listings portion of the JSON feed it does have a few pricing factors that differ from the API results you provided; I'm not sure what you need to pull exactly but this may lead you in the right direction; check the 'Listings' array and see if there's something you can possibly use.
Keep in mind if you are looking to automate scraping this data you would want to generate new search sessions; but first you want to see if this is the type of data you're looking for.
Another option, Google CSE's API; I've pulled data in the page headers of sites as they appear in Google based on the Schema.org's tags; but this may be delayed data and it appears you need real-time; the best route would be reserach the above example or try to make sure of ABNB's natural API (they provide its functionality for a reason right?; there must be a way to get what you need).
Hope my answer helped lead you in the right direction!
I'm currently working on an application in titanium appcelerator, alloy - where I'm pulling data from wordpress via rest api.
My problem now is that I need to get the data from the the users of the of the wordpress site. Info like name, email ect.
Anyone ever tried or know how it's done? The end tag: /users? Doesn't seems to be enough
In accordance with official documentation on WP-API docs data that you need is not accessable through standart API requests and endpoints (it is only "visible" during editing).
But you can create your own route and using internal API/hooks to get all data and return it in json format in prefered way (see Adding Endpoints section on provided before site).
I am trying to write a very basic jQuery Mobile app that will allow someone to select a city and view the current temperature using the OpenWeatherMap API. The data I want to use will be returned as json.
As per the landing page I can see that doing the query explicitly with the city is fine...
http://api.openweathermap.org/data/2.5/weather?q=London,uk
But what I am unable to find is how do I query the API to simply return all the cities, which will then subsequently allow me to populate my list view, then allowing the user to see the temperature.
I have read some people talking about using a 'callback', but I am unable to see how this would help.
I have written a basic app previously, but again, that was using a fixed location, where the user didnt have to select it, so I never ran into this problem before.
I am having problems posting my code, but will attempt to post it later this eve.
Any pointers/tutorial links would be great.
You have to contact them via email to get the latest list of countries.