I've a page with subpages, it looks like this (just an example, not my real data) :
-page
↳ apple
↳ iPhone
↳ iPad
↳ samsung
↳ Galaxy
I want to retrieve on the root page all the subpages of apple. I find a way to retrieve all subpages {{Special:PrefixIndex/{{FULLPAGENAME}}/ |hideredirects=1 |stripprefix=1}} but not a way to retrieve all in apple.
The expected response should be
• Iphone
• Ipad
The Special page Special:Prefixtext allows any level of prefixes, so it should be as easy as:
{{Special:PrefixIndex/{{FULLPAGENAME}}/apple/ |hideredirects=1 |stripprefix=1}}
(note the part, where /apple/ is appended as a subpage to the fullpagename variable)
Related
I am attempting to get a list of all links of satellite data for a year/month on the European Space Agency's Cryosat-2 website (https://science-pds.cryosat.esa.int/#Cry0Sat2_data%2FSIR_SAR_L2%2F2013%2F02). No matter what web scraping or html reading package I use, the links are never included. Below is an example of such an attempt with the url provided, but it is by no means my only attempt. I am looking for an explanation as to why the links that initiate the download of individual files aren't extracted, and what the solution is to obtaining them.
library(textreadr)
html_string<- 'https://science-pds.cryosat.esa.int/#Cry0Sat2_data%2FSIR_SAR_L2%2F2013%2F02'
html_read<- read_html(html_string)
html_read
[1] "Layer 1" "European Space Agency"
[3] "CryoSat-2 Science Server" "The access and use of CryoSat-2 products are regulated by the"
[5] "ESA's Data Policy" "and subject to the acceptance of the specific"
[7] "Terms & Conditions" "."
[9] "Users accessing CryoSat-2 products are intrinsically acknowledging and accepting the above." "Name"
[11] "Modified" "Size"
[13] "ESA European Space Agency"
Ok, here is a solution. In this kind of case where you can't get the info with regular scraping (rvest and so on), two solutions:
or you get the info with RSelenium and it can be tedious
or you inspect the page and monitor the XHR request (with the element inspector of firefox for example: network, and XHR).
You will find out that the data are loaded from urls looking like:
https://science-pds.cryosat.esa.int/?do=list&maxfiles=500&pos=5500&file=Cry0Sat2_data/SIR_SAR_L2/2021/02
These pages appear like html, but if you open them, they are not. They are JSON. The webpage just display dynamically the info requested as JSON.
So you can simply get the info as follow:
url <- 'https://science-pds.cryosat.esa.int/?do=list&maxfiles=500&pos=5500&file=Cry0Sat2_data/SIR_SAR_L2/2021/02'
library(jsonlite)
fromJSON(url)
$success
[1] TRUE
$is_writable
[1] FALSE
$results
mtime size name
1 1616543429 37713 CS_OFFL_SIR_SAR_2__20210223T041611_20210223T042157_D001.HDR
2 1616543428 845594 CS_OFFL_SIR_SAR_2__20210223T041611_20210223T042157_D001.nc
3 1616543364 37713 CS_OFFL_SIR_SAR_2__20210223T043539_20210223T043844_D001.HDR
4 1616543363 528578 CS_OFFL_SIR_SAR_2__20210223T043539_20210223T043844_D001.nc
5 1616543321 37713 CS_OFFL_SIR_SAR_2__20210223T044915_20210223T045113_D001.HDR
6 1616543322 387650 CS_OFFL_SIR_SAR_2__20210223T044915_20210223T045113_D001.nc
7 1616543360 37713 CS_OFFL_SIR_SAR_2__20210223T045156_20210223T045427_D001.HDR
8 1616543359 456414 CS_OFFL_SIR_SAR_2__20210223T045156_20210223T045427_D001.nc
9 1616543328 37713 CS_OFFL_SIR_SAR_2__20210223T045551_20210223T045749_D001.HDR
10 1616543327 385998 CS_OFFL_SIR_SAR_2__20210223T045551_20210223T045749_D001.nc
path
1 Cry0Sat2_data/SIR_SAR_L2/2021/02/CS_OFFL_SIR_SAR_2__20210223T041611_20210223T042157_D001.HDR
2 Cry0Sat2_data/SIR_SAR_L2/2021/02/CS_OFFL_SIR_SAR_2__20210223T041611_20210223T042157_D001.nc
3 Cry0Sat2_data/SIR_SAR_L2/2021/02/CS_OFFL_SIR_SAR_2__20210223T043539_20210223T043844_D001.HDR
4 Cry0Sat2_data/SIR_SAR_L2/2021/02/CS_OFFL_SIR_SAR_2__20210223T043539_20210223T043844_D001.nc
5 Cry0Sat2_data/SIR_SAR_L2/2021/02/CS_OFFL_SIR_SAR_2__20210223T044915_20210223T045113_D001.HDR
6 Cry0Sat2_data/SIR_SAR_L2/2021/02/CS_OFFL_SIR_SAR_2__20210223T044915_20210223T045113_D001.nc
7 Cry0Sat2_data/SIR_SAR_L2/2021/02/CS_OFFL_SIR_SAR_2__20210223T045156_20210223T045427_D001.HDR
8 Cry0Sat2_data/SIR_SAR_L2/2021/02/CS_OFFL_SIR_SAR_2__20210223T045156_20210223T045427_D001.nc
9 Cry0Sat2_data/SIR_SAR_L2/2021/02/CS_OFFL_SIR_SAR_2__20210223T045551_20210223T045749_D001.HDR
10 Cry0Sat2_data/SIR_SAR_L2/2021/02/CS_OFFL_SIR_SAR_2__20210223T045551_20210223T045749_D001.nc
These should give you all the info you need. If you tweak a bit the url, you should be able to get all the info from the date you want.
The problem seems to be that the page is dynamic. It probably has some JS code and only loads the links after it runs. So when you get the HTML from the link, you're only getting the base page (before the JS runs).
I can think of two possible solutions:
You can try to use selenium, which emulates an user in the browser so it will load the page completely, but the set-up might be a bit complicated. See for an intro https://www.r-bloggers.com/2014/12/scraping-with-selenium/
The page probably sends an HTTP request to get the links from an API, you can try to figure out the exact request. The network tab on you browser is a good place to start.
iOS 12 first time cannot set cookies, but its work after 2nd tried.
iOS 13 cannot set cookies even refresh the page many times.
Anyone face this new issue?
or have the solution for setcookies
My situation as below
step 1 :Site A > Site B
- site B will set cookie
step 2 : site B return a response link for site A
- site A use the cookies set at site B and then use the value to append in response link
im trying to launch my app when user taps on link on email .. i've seen many posts like this but i can't seem to get it to work .. there's a problem i've been encountering, if the link i send is https://www.google.com (or any other valid websites), when i tap on the link, it works. It launches safari then goes to the site.. but if i replace it with my urlScheme, the body in the email appears to be not read as a link .. by the way.. im creating the email message in the client side .. here is sample code..
Nsstring *htmlBodyWebsite = #“<a href=\”https://www.google.com\”>Click here</a>" < working
Nsstring *htmlBodyUrlScheme = #“<a href=\”myApp://\”>Click here</a>" <not working
// when i enter in safari browser - myApp:// << working
so it seems that i've setup urlscheme correctly since when i entered myApp:// in safari, it prompted me to launch my app.
so the process here is, i create the email message in the client side... then our server receives it and sends it via email to target recipient..
now when the recipient receives the email, here are some observations i've made..
when i send htmlBodyWebsite, it works fine. Using google chrome browser and in the email message itself, when i right click, then inspect, i would be able to find the referenced link with proper tagging
but with htmlBodyUrlScheme it doesn't have proper html tagging
thanks
it seems to work if i tap on the link using the ios built-in Mail app.. neither using safari to open gmail.com nor gmail app doesnt make it resolve this issue..
I'm creating a Django site that hosts a large set of long, transcribed debates.
I have two main views: a Haystack search view which indexes each individual speech, and a full view which indexes each transcript (containing hundreds of individual speeches). Both views use django endless pagination to display results.
I'm trying to link between these two views so that any search result (speech) can be viewed in context within its parent transcript, and I want the page to jump to the anchor of that speech ID on page load.
I calculate the page where the individual result appears and redirect to that URL while storing the pk of the result in messages so I can highlight the result:
def full_view_redirect(request, year, month, day, pk):
y=str(year)
m=str(month)
d=str(day)
qs = transcripts.objects.filter(speechdate__year=year).filter(
speechdate__month=month).filter(speechdate__day=day).order_by('basepk').all()
firstpk = int(qs[0].basepk)
pageNo = ((int(pk)-firstpk)//15)+1
messages.add_message(request, messages.INFO, pk)
if pageNo == 1:
return redirect("/full/"+y+"/"+m+"/"+d+"/"+"#"+str(pk))
else: ## this doesn't work
return redirect("/full/"+y+"/"+m+"/"+d+"/"+"?page="+str(pageNo)+"#"+str(pk))
My question is similar to http://htmlasks.com/how_to_make_this_link_work_page2reviews_reload_the_page_and_jump_to_the_anchor but the suggestion here to switch around the anchor and the ?page doesn't work.
I can't get the anchor to work so the page jumps to the desired result on page load. Am I missing something obvious?
Edit:
I verified the div I want to jump to has a proper id, eg.
<div class="panel panelhighlight" id="54969">
A url like /full/1903/04/29/?page=7#54969 loads the proper page but does not jump to the div.
A url like /full/1903/04/29/#54969?page=7 loads the first page and not page 7.
Edit 2:
I have switched from django-endless-pagination to django-digg-paginator so that pagination is handled within my view, not at the template level.
Then, I had to ensure the redirect reloads by omitting the slash between the page number and the anchor. /full/1903/04/29/7#54969 repositions the page successfully on load.
As detailed in my edit, I had two problems going on that I needed to fix:
I switched from django-endless-pagination to
django-digg-paginator so that pagination is handled within my view/URL pattern,
not at the template level.
I had to ensure the redirect reloads by omitting the slash
between the page number and the anchor, ie. /full/1903/04/29/7#54969
repositions the page successfully on load.
I need to do dynamic link opening based on each of 3 cases. How do I write this into a link ?
Case 1: user is on desktop or Android. Send user to website link with
reference to content example.com/content/123
Case 2: user is on
iOS app is not installed. Send to App Download, post download open
content article that user tapped on
Case 3: user is on iOS app is
installed. Open app with activity link and link to that activity in
the app myapp://content/123
Any ideas how to write this into an HTML source that I can add to email? On click it will have to figure out what it wants to do.
Here is what the source looks like in mailchimp email template per content item:
<span>A brief funny description about this place, in one or two phrases...
Read More
</span>
Any help is greatly appreciated.
You are looking for the concept of deep linking. I can help you with 2 of your cases
First I'll explain how I would attempt to make this work. This sample will open the Facebook app and navigate to the Wikipedia page. You can try this out by opening this link from your Android or iOS device.
Instead of having the direct link in the email itself, I'd use one link that would then, on the server's side, determine the action to take.
In your email template, do this
<span>A brief funny description about this place, in one or two phrases...
Read More
</span>
This should then take the user to a script named 'my_server_script.php' (my example is PHP, but you can achieve this in any language).
In this script, you can then check the User Agent by using PHP's $_SERVER['HTTP_USER_AGENT']. This will give you something like Mozilla/5.0 (Macintosh; U; PPC Mac OS X; en) AppleWebKit/418 (KHTML, like Gecko) Safari/417.9.3. Then by parsing this string, you can determine the user's OS and take an action based on that. Please do note that User Agent strings can be modified and is not always 100%, but it should be more than enough for your use.
Once you have determined the action to take, you could redirect the user to the app by sending them to the deeplink.
Here is PHP code I tested on Android earlier, this should open the Facebook app and take you to the Wikipedia page. As far as I know, it should work on iOS as well.
<?php
// Set App deeplink
$app_url = 'fb://profile/33138223345';
// Try to redirect the device to the URL
header('Location: ' . $app_url);
?>
I don't have a solution for detecting if the app is installed, but if you do find a solution for
that, you should be able to redirect them to the appropriate app store using the code below
<?php
// iTunes link
$app_install_link = 'https://itunes.apple.com/za/app/facebook/id284882215?mt=8';
// Then redirect the user to the app store location
header('Location: ' . $app_install_link);
?>
If you determine they should be sent to the desktop version, simply do
<?php
// Browser link
$link = 'https://example.com/content/123';
header('Location: ' . $link );
?>
This is personally how I would make this work since you won't have the capability in the email itself to check what device you are on and modify the link based on that.
You should however still be able to use the deeplink fb://profile/33138223345 by doing
<span>A brief funny description about this place, in one or two phrases...
Read More
</span>
and that should open the Facebook app on iOS and Android.
Hope this helped a bit!