I am trying to find sign in link via scrapy crawler on any website, for example www.microsoft.com. However when I get initial response from the website the sign in link isn't present. But when I visit the website I am able to see the sign in link in the "View Source".
The code is written to get all the links by checking for <a> tag in response using xpath. There are other links that are present in response, however the sign in link is not visible.
for links in response.xpath("//a"): ---> to get the links
if links.xpath("#href").extract() and links.xpath("#href").extract()[0]:
link = links.xpath("#href").extract()[0]
if(link.startswith("http")):
searched_url = link
else :
if(link.startswith("/")):
link = link[1:len(link)]
searched_url = response.url + link
all_other_urls.append(searched_url)
print(searched_url) ---> printing the links present in the response.
The response I receive has link present in this manner (refer signInUrl). :
<div id="meControl" class="c-me" data-signinsettings='{"containerId":"meControl","enabled":true,"headerHeight":48,"debug":false,"extensibleLinks":[],"userData":{"idp":"msa","firstName":"","lastName":"","memberName":"","cid":"","authenticatedState":"3"},"rpData":{"preferredIdp":"msa","msaInfo":{"signInUrl":"https://www.microsoft.com/mscomhp/onerf/signin?EEL=True\u0026pcexp=True","signOutUrl":"https://www.microsoft.com/mscomhp/onerf/signout?pcexp=True","meUrl":"https://login.live.com/me.srf?wa=wsignin1.0"},"aadInfo":{"signOutUrl":"https://www.microsoft.com/mscomhp/onerf/signout?pcexp=True","appId":"","siteUrl":"","blockMsaFed":true}}}' data-m='{"cN":"GlobalNav_Account_cont","cT":"Container","id":"c6c1c7c2m1r1a1","sN":6,"aN":"c1c7c2m1r1a1"}'>
<div class="msame_Header">
<div class="msame_Header_name st_msame_placeholder">Sign in</div>
</div>
My code will work if the link is present in <a> tag which I can see on website view source:
a class="mectrl_resetStyle mectrl_trigger" id="mectrl_main_trigger" aria-label="Sign in to your account" href="https://www.microsoft.com/mscomhp/onerf/signin?EEL=True&pcexp=True&ru=https%3A%2F%2Fwww.microsoft.com%2Fen-us%2F" target="_top"><div class="mectrl_header" aria-hidden="true"><div class="mectrl_headertext mectrl_truncate"> Sign in /div> div class="mectrl_profilepic mectrl_glyph glyph_signIn_circle" id="mectrl_headerPicture" role="presentation" aria-hidden="true"></div></div></a
You can find the signin-url without rendering the page as follows:
import json
signin_settings = response.xpath('//*[#id="meControl"]/#data-signinsettings').extract_first()
signin_d = json.loads(signin_settings)
signin_url = signin_d['rpData']['msaInfo']['signInUrl']
Related
I have a component menu which contains a few anchor tags. Each tag brings the user to that respective page section. I am trying to share the same anchor tag among the other components.
For example, I have two more HTML components called homepage.component.html and details.component.html. For each I call the menu.component.html by its selector. Both homepage and details html components have an id for the section I wanna scroll to. Here's how it looks like:
menu.component.html
Go to content
for both homepage.component.html and details.component.html
<div class="home-content" id="content"> Here comes more code </div>
It should work just like in a non-dynamic html project, however, when the anchor tag is clicked, the url redirects to '' (which is the first/default page) and then it shows the content for the first page, instead of the current componenet I am on.
I have tried creating a function where I get the current url and using the router.navigate, I pass the parameters indicating the fragment:
menu.component.ts
currentRoute: string
scrollToItem(){
this.currentRoute = this.router.url
this.router.navigate([this.currentRoute], {fragment: 'content'})
}
menu.component.html
<a (click)="scrollToItem()">Go to content</a>
However, this function adds the id #content to the url each time the anchor tag is clicked, redirecting the user to my 404 page.
I wanted to know if there is a way to use an anchor tag on the menu.componenet.html, while all the items that have "content" as their ids in different components are going to be displayed. Hopefully I made my question clear. If there is still questions about how the error occurs I can create and shate a stackblitz project. Thanks in advance :)
I know how to do using html and JavaScript
<h2 id="C4">Chapter 4</h2>
Jump to Chapter 4
This is what I am trying in SAPUI5. On click to Back to top link it should navigate to helpButton. This is not working for me.
<Button id="helpButton" icon ="sap-icon://sys-help" />
<Link text="Back to top"
press="#helpButton"/>
You can actually do this in UI5. A little differently than how you tried though.
The problem is that the UI5 ID is not the same as the HTML ID (which is what you want to use with the hash-link for the browser to jump there). Also, you cannot specify a URL inside the press "attribute" of the link. The press "attribute" is in fact an event (so you can only specify an event handler name).
So to be able to do what you want, you have to use the href property of the Link and fill it with the HTML ID of the target control. You can do this on the onAfterRendering hook of the view (that's when you are able to find the HTML ID of the target control):
onAfterRendering: function() {
var oRef = this.byId("target").getDomRef();
this.byId("link").setHref("#" + oRef.id);
}
You can find a working fiddle here: https://jsfiddle.net/93mx0yvt/26/.
I followed Captcha's Tutorial and did this:
Paste this snippet before the closing </head> tag on your HTML template:
<script src='https://www.google.com/recaptcha/api.js'></script>
Paste this snippet at the end of the <form> where you want the reCAPTCHA widget to appear:
<div class="g-recaptcha" data-sitekey="key"></div>
I want to know how I can make a form to use the captcha in. I just want a basic form that you have to solve a captcha before you can see some text. So you solve the captcha and press the button and it shows some text like hidden message. I can't find this anywhere. Help me! I prefer plain html.
When your users submit the form where you integrated reCAPTCHA, you'll get as part of the payload a string with the name "g-recaptcha-response". In order to check whether Google has verified that user, send a POST request with these parameters
You need to send POST to URL https://www.google.com/recaptcha/api/siteverify with parameters what you see on your google reCAPTCHA account.
If you want BEFORE submitting form, add data-callback attribute to your g-recaptcha-tag. Inside that attribude set name of function that show hidden content only for successfully verifed users.
For more info check reCAPTCHA documentation.
Example
In your javascript define function to show hidden content:
function alertSuccess() {
$(".hidden.message").show();
//alert("Success");
}
In reCAPTCHA
<div data-callback="alertSuccess" class="g-recaptcha" data-sitekey="__YOUR_SECRET_KEY__"></div>
I have two html pages: /algorithrms and /algorithms/add written in scala template. The route file contains following lines:
GET /algorithms controllers.Application.algorithms()
GET /algorithms/add controllers.Application.newAlgorithmForm()
I want to add a button in the page /algorithms and when I click that button, it simply redirects to the second page /algorithms/add. I know how to do this in JavaScript. I just want to call an action from the button click and then let the action redirects me to the landing page.
So I added the following code in the first page's html template:
#form(action=routes.Application.newAlgorithmForm()){
<input type="submit" value="Add">
}
It worked, but the landing url is: http://localhost:9000/algorithms/add?
I don't want that question mark. I want to know 1) what I did wrong to cause the question mark to generate and 2) how to remove it?
I do not know if you use Twitter bootstrap, but hyperlinks can look like buttons too, and the redirect to another page sounds to me like a plain hyperlink:
<a class="btn" href="#controllers.routes.Application.newAlgorithmForm()" >
#Messages("add.newAlgorithmForm")
</a>
I'm writing an application, a reporter with heirarchy of folders and files, in the lower heirarchy level there are 2 types of reports: the simple one is a flat (non link) report that being presented as a single simple line.
the second type is a link with a general description in the header and if you press the link you get a full report.
example: if I run a telnet command, I will see the command in the header and if I want to see the entire session with the device I will press the link and it will be presented.
My problem is that most of this lined-files are small but the OS reserve a minimum space for every file so I loss alot of disk space for nothing.
The solution I want to implement is a "dummy" links, which will be presented and will behave like a regular links but actually will be stored in the same file like their "parent" (probably with alot of other links like them).
The solutions I saw so far are only for "jumping" inside a page but this is not what I'm looking for, I want it to be presented like a seperated file and I dont want the "parent" file to present this information at all (the only way to see it will be by pressing the link and even then it will present only this information and not the other file content).
any idea guys?
To link to a specific part in a web page, place an anchor link where you want the user to go to when they click a link with:
<a name="anchor"></a>
and link to it with:
Click here
You can replace "anchor" with a more descriptive name if needed.
To hide/show a div (the following code is untested, but should work)
JQuery solution (If you're using JQuery):
function toggle(divname) {
$(divname).toggle();
}
The corresponding HTML:
<div id="content">some content</div>
<a onclick="toggle('content')">Click here to show/hide the content div!</a>
Non-JQuery Solution:
function toggle(divname) {
var adiv = document.getElementById(divname);
if (adiv.style.display === 'block' || adiv.style.display === '') {
adiv.style.display = 'none';
} else {
adiv.style.display = 'block'
}
}
The HTML:
<div style="display:hidden" id="content">Content</div>
<a onclick="toggle('content')">Click here to show/hide the content div!</a>