I'm working with Twitter's API to retrieve a list of tweets from a Twitter account.
I'm using this: https://dev.twitter.com/rest/reference/get/statuses/user_timeline
I get the JSON response (http://pastebin.com/raw/zqyUuXcG) but in the text field (at the end of it) I also have the url to the tweet itself.
I'd like to avoid that because I want to keep the text clean and put the url in an hyperlink (like on date or on the container div).
I couldn't find a way to avoid url to be included in the text field. Is there a way to do it?
Thanks
Depends on the method you want to use after you obtain a JSON response, but doesn't look like you have direct control over URL placement straight from the API.
You could use grep to either extract the URL or filter it out from the text field.
For example, in R, this chunk of code would remove everything after and including "https":
gsub( "https.*$", "", textfield)
Related
I've been messing with YouTube lately, but the search bar confuses me.
If I search something long, like "a;sldjf;asbybytyeu2430572735gljahflg", then control-f in the DOM, the only reference I can find to the search term is in the page title.
Even when I look at the 'input' element of the search bar, it doesn't have a 'value' property.
My question is this: How can the browser display text that doesn't appear anywhere in the DOM? Where is the data stored?
It is in the URL query string. For example, if I search "dank memes", this is the URL:
https://www.youtube.com/results?search_query=dank+memes
It can be accessed by JavaScript with window.location.search which in this case would return "?search_query=dank+memes"
But most likely YouTube's JavaScript code does not access it and it is only used to send the information for a GET request to the server. Then the server will display all the needed information.
I am working on a html page with an input bar. How can I get the form data from this page to another html page with a iframe? Because I need to embed the search result from other site but I want to show on my site new page. I just guess should use iframe. But I don't know how to get data and add in page address field.
You have several options here:
1) If your form uses GET method, then all the text data from the form will be passed to next page (which is set up as an "action" attribute of the form) in URL (like www.example.com/?name=John&lastname=Doe). After that, you can extract data from this URL using code from this StackOverflow question and put it inside HTML element using Javascript as well like it is done here
2) Whatever server language you use, you can get query variables from there and put them to the page before sending it to the client. In case of PHP, check this example. NB: don't forget about security.
3) You can temporarily store text in localStorage API of the browser. Simply set the value before submitting the form and then get it on the next page.
iFrame would be inefficient and not needed here.
I try to do some Web Scraping
The objective is to collect all remedials according to the postal code. The problem is when I try my code, my list is empty because the url did't change according to the postal code. This is why I want to change the HTML value during the scrape.
I'm not sure how to do this. I tried using Selenium and XPATH however I wasn't able to find anything.
Here's the HTML Code: (in red is what I need to change.)
EDIT : Indeed, the goal is to collect the pagination with the name and the type of remedial according to the postal code, this is why I want to change the HTML content during the scrap.
This is the best that I can do for the moment, I hope u will see the error
This input is in a form, which is good because Selenium has special functionalities to handle forms.
from selenium import webdriver
url = "https://www.maif.fr/services-en-ligne/consultationreparateurs/geolocaliserReparateur.action?view"
query = "whatever you want to put into the search box"
driver = webdriver.Chrome()
driver.get(url)
webform_input = driver.find_element_by_xpath("//input[#id='adresseInternaute']")
webform_input.send_keys(query)
webform_input.submit()
The key here is submit(). It will walk the HTML tree until it finds a button within the current form, meaning you don't have to write an extra two lines just to click the search button.
I have no clue whether MongoDB provides such an option or not.
I have a local MongoDB in my meteor application connected to RoboMongo. So, I have my collections with many docs inside. Inside a doc, there are many fields with much plane text in it, like in
Collection1:
{
"_id": ObjectID("123456789"),
"thing": firstThing,
"type": text,
"langEn"{
"content": "This is so much senseless text like in (**X**), you don´t want to read ",
"content2": "This is even more text like in (**Y**)...seriously?"
"content3": "Ok...I´m out because of (**Z**)"
}
}
So, X, Y and Z should be links like a href="#" or any reference, with that I can see the content of another collection on that site.
Because the user always decides by himself what he wants to see and therefore what is loaded onto the page from the collection, I can´t just write fix html-hashes on the site itself. It depends on which docs and fields are currently loaded on the site.
I tried to use html in MongoDB:
..."content": "This is so much senseless text like in <a href='#(**X**)'> Link to X </a>, ..."
But this fails...
Is there any way to make text of a MongoDB field a clickable link?
Edit
The second code example is just a try (just leave it there, in case it is nonsense).
I want to be able to programmatically (without it displaying in the browser) send an URL such as http://www.amazon.com/s/ref=nb_sb_noss_1?url=search-alias%3Daps&field-keywords=platypi&sprefix=platypi%2Caps&rh=i%3Aaps%2Ck%3Aplatypi" and get back in a string (or some more appropriate data type?) the html results of the page (the interesting part, anyway) so that I could parse that and reformat selected parts of it as matched text and images (which link to the appropriate page). I want to do this with Razor/Web Pages, if that makes any difference.
IOW, this is sort of a screen-scraping question, but really a "behind-the-screen" scraping.
Is it possible? How? A 100 point post-answer-bonus will be awarded to the (or the most helpful) answer.
Use the WebClient class (or .Net 4.5's better HttpClient class) to download the HTML, then use HTML AgilityPack to parse it