Logging into websites using JSoup, is it possible? - html

Is it possible to log into websites that use csrf tokens and such using JSOUP? The website I am trying to log into is aliexpress.com. Which seems to have a lot of input values, and I noticed that the csrf token changes on every attempt. I am guessing that these are protective measures to block out spam. I am quite new to html, and was wondering, is it even possible to login to sites like these? Thanks
Here is my code in case I did something wrong:
public static void main(String[] args) throws IOException {
Connection.Response loginForm = Jsoup.connect("http://login.aliexpress.com")
.method(Method.POST)
.data("cookieexists", "false")
.data("loginId", "xxxxxx#gmail.com")
.data("password", "xxxxxx")
.data("event_submit_do_login", "submit")
.data("submit-btn", "Sign In")
.data("appName", "aebuyer")
.data("appEntrance", "default")
.data("_csrf_token", "vdspQZH4cMoQT6GxLyU0a7")
.data("rdsToken", "")
.data("cid", "a832dd6d-990f-44eb-8bdb-9ec49d1c0a99")
.data("umidToken", "4e6219e38c34346dc2bb7914a54794aac7645e7b")
.data("lang", "en_us")
.data("hsid", "VP4zHOZfVs1Ec4qqEsI1mA")
.data("isRDSReady", "false")
.data("isUMIDReady", "false")
.data("umidGetStatusVal", "")
.data("bizParams", "")
.data("isRequiresHasTimeout", "false")
.data("loginHost", "https://passport.aliexpress.com/")
.data("scene", "")
.data("isMobile", "false")
.data("modulus", "d3bcef1f00424f3261c89323fa8cdfa12bbac400d9fe8bb627e8d27a44bd5d59dce559135d678a8143beb5b8d7056c4e1f89c4e1f152470625b7b41944a97f02da6f605a49a93ec6eb9cbaf2e7ac2b26a354ce69eb265953d2c29e395d6d8c1cdb688978551aa0f7521f290035fad381178da0bea8f9e6adce39020f513133fb")
.data("exponent", "10001")
.data("ua","099#KAFEx7E+TEGE6YTLEEEEE6twSXLoZ6NHDu3YS6tqDywMC6t3gR9YLfD1SryM4MDFYuJoV6t1YRB7n6f1Ych6n6g1DRJo+xqTETFUEcZt288mby1PYPrfds1gLjdTEEi5DEEErGFEHCLlhRaTG9llEJIm/KRlbpZV8MixZy164GFEJFwlsyaDMOcmby1TSV3WPy1tY0s3YuaHLyZTvbd9E7EFD67EEKqTETAEluZdtkH7NyhGYfciuOkG/MhXaQ3bAMZtY0s3JGFET6i5EE1iE7Eo73lP/3m4NVkAbIJi4leDbRZTvbuwsqk6/MhXaCqTETfEEcZt9CXmby1PYPrfds1tYo8XCV/zLybTETDElapdLIallGEeOOCxls1Pr08FSP8zYovKZMRc41qTETFUEcZt9Scmby1PYPrfds1gLioTETzElsXdLUQEWECmYKhMtV8gPy1fnM9SZyvHLioTETYQldXd7FEEVcCmYKhMtV8gPy1XC0ysUoSnE7EiK3l6/wqaEcExOMRyv9cGNyvHYp5TEEi5D7EE/GFETJDovl8JE7hMDwGEmba3U61vv0XBaYevDpiRciUt1OLjbo4Qc7rczQ8nZR2Bw7CWSQ/vPsqqi9exUqZ96Yz6bQWAnWuWLLrDA2iebRMREyB7Pbcdb4QsPok4QioWiYeWZyhqqOKR1eJ6UI2Vcgw6bXkPnypX1YuYalp7Pwnh1eyJroeXFSZ6UcCsZLpzLUZ1WsJhvLqpa291dzCncU7RbohMtscSi7hYSeJFFsxcTy/0/fhCv4bprqrANa2n+GCWds166OIp+WJFrtSSPFnRrVs8SAUM+GZDnR+kbyo6cWyIbISMHG1uVPybZz4M/vl4DXym8jvqaWw7zVZBNdZ6bteWNREZwErbnpywry4t+O/kuV8W/EPcuzpYa0uVaYnzZuB5hg9ulrimV65MJwnuV6i4DTY03BlWaccJ8ypu9u2A8WWzPMIszbcYDi4nEoCYDpiB4gvl9zkGVPq0IlbuQfcxVyrs37eQSyJKcw4RLeXKPt+Z4LslVewCYREnCQZVgcx+hz+l/ri5Vfx+8IRuNP94S0Yo3/sLDVwjvK4l7uyxOGFETYilssn1dASTEELlluaLom7wxeSTE1LlluZdt3illllls3aSt3t3llle33iSw6alluUdt37q33llWLaStEGde7FE5YHKJ7ThVa6o3kskwmyV0HUC6Df39yzU0xEG1DN7Vafo3kikLw+IRhj2qbAB6OT2qago1DN7VaTo3kikawWIRhj2qoAB6OT2qafo1DN7VmGtI8EJE7EhlAaP/3iSJGFET6i5EE1mE7EFlllbrxqTETFUluZdDWkmby1PYPrfds1gLioTETYQltCdBHGEV38mYKhMtV8gPy1XC0ysUoSnE7EB63l7/h6LEc7JNM1j83X0UykG+qxEdUF9E7EFD67EE1qTETFUluZt9CZmby1PYPrfds1gLjdTEEi5DEEE4GFEJFwlsyaD0Lxmby1TSV3WPy1tY0s3YuaHLyZTvbuiE7Eo73lP/3m+q0kAbIJi4leDbRZTvbuwsqk6/MhXaVqTETAEluZdtkpCNyhGYfciuOkG/MhXaQ3bAMZtY0s3IGFEHSwlsyDe2VkAbpZ1ZVnabRZTC0sR8u86")
.followRedirects(true)
.execute();
// TODO code application logic here
Document doc = loginForm.parse();
System.out.print("website title:" + doc);
}
}

Since you know which parametrers to send, I assume that you know how to use your browser's developer's tools, so it will be easy for you:
In order to login to the site, you have to take two steps. The first step is sending a GET request and parse the result. The second step is to send a POST request, with the needed parameters, including the ones you've obtained from step 1.
I've seen that when sending the first get request to https://login.aliexpress.com/, the result does not contain the values of _csrf_token etc. The browser sends another request to https://passport.aliexpress.com/mini_login.htm?lang=en_us&appName=aebuyer&appEntrance=default&styleType=auto&bizParams=&notLoadSsoView=false&notKeepLogin=true&isMobile=false&rnd=0.9476178801629621 so you must do the same, and parse the result (notice that the last parameter of the get request is some random number. I think you should also randomize the string and not use the same number again and again, it might be some protection measure):
String firstURL = "https://passport.aliexpress.com/mini_login.htm?lang=en_us&appName=aebuyer&appEntrance=default&styleType=auto&bizParams=&notLoadSsoView=false&notKeepLogin=true&isMobile=false&rnd=0.9476178801629621";
Connection.Response loginForm = Jsoup.connect(firstURL)
.userAgent("Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:56.0) Gecko/20100101 Firefox/56.0")
.method(Method.GET);
After that you'll have to parse the response and extract the parameters, something like this -
Element e = doc.select("input[id=fm-cid]").first();
String cid = e.attr("value");
After parsing all the needed values, you can send the POST request.

Related

VBA and json objects

I want to create meail merge letters to members of a community, reminding them of the new year and to pay their fees. There is a nice API on a server that can create QR-codes that the a phone pay app can read. The server responds to HTTP POST rewuests. This API provides the possibility to pre-fill in the payment parameters like the payment receiver, the amount an a message to the payment receiver.
I am starting from an Excel-sheet that keeps the list of members amongst other items. I can create mail merge letters from this member list, but fails to get the creation of the prefilled QR-codes work correctly.
Since the base is Excel (and Word for the mail merge) the language used is VBA. With the code below i can get response from the server with a prefilled QR-code that contains the recieiver number of the payment:
Sub DownloadQRCode(ByVal myURL As String, ByVal LocalFileName As String)
Dim msXML As New MSXML2.ServerXMLHTTP60
Dim myStream As New ADODB.Stream
msXML.Open "POST", myURL, False
msXML.setRequestHeader "Content-Type", "application/json"
msXML.setRequestHeader "User-Agent", "Mozilla/5.0 (Windows NT 10.0; Win64; x64)"
msXML.send "{""payee"":""987654321""}"
If msXML.Status = 200 Then
myStream.Open
myStream.Type = adTypeBinary
myStream.Write (msXML.responseBody)
myStream.SaveToFile LocalFileName, adSaveCreateOverWrite
myStream.Close
End If
Set msXML = Nothing
Set myStream = Nothing
End Sub
If I change the msXML.send statement to
msXML.send "{""message"":{""value"":""Sven_Svensson"",""editable"":""false""}}"
I get the response code 400 (Bad request)
The speicification for the message object looks like this
My questions are: Do I use the wrong value in "Content-Type" header or is the json representation of the message object faulty?
SOLVED!
A couple of mistakes from my side:
Phone/account numbers must be valid in this application
Booleans must not be quoted
Numbers must not be quoted
json uses brackets for arrays, not as outer paranthesis
I have this working fine now
This works for me:
msXML.send "{""payee"":""987654321"",""message"":{""value"":""Sven_Svensson"",""editable"":true}}"
Note booleans in JSON are not quoted.
You used an array ([]) in your message 'msXML.send "{""message"":[{""value"":""Sven_Svensson"",""editable"":""false""}]}"' while message's specification doesn't state it. I guess it should look like 'msXML.send "{""message"":{""value"":""Sven_Svensson"",""editable"":""false""}}"'

InDesign Socket HTTP response is missing sections

I am automating Adobe InDesign to create documents using JSON data gathered from a web API with a SQL Server backend. I am using the Sockets object to make an HTTP 1.0 call to our server. Sometimes the response received is missing about 1700 characters from various points within the JSON string, yet when I call the same API endpoint using curl or Postman I get a complete and valid response.
The response should be about 150k characters long, and I'm using conn.read(99999999) to read it. In addition, the appearance of the end of the string looks correct, so I don't believe it's any kind of truncation problem.
The problem only seems to occur when I request a UTF-8 encoding. If I request ASCII I get a complete and valid response, but missing various Unicode characters. If I request BINARY I get a complete and valid response but the JavaScript/ExtendScript seems to be handling any multi-byte Unicode characters received as individual bytes, rather than as the Unicode characters we want to display.
Here is an illustration of the behavior I'm seeing, using bogus data...
"Expected" response...
[{"Id":1, "name":"Random Name", "Text":"A bunch of text", "AnotherId": 1}]
"Actual" response...
[{"Id":1, "name":"Random Name", "Text":"A bunc": 1}]
The problem first manifested itself as a JSON2 parsing error, for obvious reasons, but the root of it seems to be the fact that parts of the data are going missing in-transit.
So far we've only seen this problem when making the call using the InDesign Sockets object, and not every response exhibits this behavior.
Any help or insights you could offer would be appreciated.
Here is the function I'm using to call for data...
function httpRequest(url, encoding) {
try {
var response = "";
var hostName = getHostFromUrl(url);
var pathAndQuery = getPathAndQueryFromUrl(url);
var httpGet = "GET ";
httpGet += pathAndQuery;
httpGet += " HTTP/1.0\r\nHost: ";
httpGet += hostName;
httpGet += "\r\n";
var conn = new Socket;
conn.timeout = 30;
//conn.encoding = encoding || "UTF-8";
//conn.charset = "UTF-16";
if (conn.open(hostName + ":80", encoding || "UTF-8")) {
// send a HTTP GET request
conn.writeln(httpGet);
// and read the server's response
response = conn.read(99999999);
conn.close();
}
return parseHttpResponse(response);
}
catch (e) {
$.writeln(e);
$.global.alert("There was a problem making an HTTP Request: " + e);
return null;
}
}
It turns out my handling of the HTTP response was too simplistic and needed extra logic to handle Unicode characters properly.
The solution, in my case, was to use the GetURL method made available by Kris Coppieter here.

HMRC MTD Hello World with Qt

I have a Qt program that stores all my small (tiny) company information on a sql database and I have over the years tailored it to do all my accounting stuff, invoices, BOMs etc.
At the push of a button I can get all of the necessary sql data to calculate a quarterly VAT return, but we're going to have to electronically submit all the data now, not just calculate it. I have all the data needed, it's just a case of submitting over HTTP using json (of which I know a little/nothing about respectively).
I'm small enough so that I don't have to do this submission at the moment, but the time will likely come, so I'm trying the most basic of requests in the HMRC's sandbox as a starting point.
On this page it shows you how to do an hello world request in Java, so I'm trying to do the same with Qt with C++.
I've tried the following which responds to the push of a button and I have of course, set up a slot to deal with a response:
void MainWindow::hello()
{
QJsonObject json;
QString rs("https://test-api.service.hmrc.gov.uk/hello/world");
QNetworkRequest request
{
QUrl(rs)
};
request.setHeader(QNetworkRequest::ContentTypeHeader,"application/vnd.hmrc.1.0+json");
request.setUrl(QUrl(rs));
manager->get(request);
}
and the main window init:
manager = new QNetworkAccessManager();
QObject::connect
(manager, &QNetworkAccessManager::finished, this, [=](QNetworkReply *reply)
{
if (reply->error())
{
ui->debugText->appendHtml(reply->errorString());
return;
}
QString answer = reply->readAll();
ui->debugText->appendHtml(answer);
}
);
To which I get the reply:
Error transferring https://test-api.service.hmrc.gov.uk/hello/world -
server replied: Not Acceptable
I assume that means I am communicating with the sever now, but I do not know what this terse error message means!
The Java on the HMRC web page is as follows:
// construct the GET request for our Hello World endpoint
HttpClient client = HttpClientBuilder.create().build();
HttpGet request = new HttpGet(
"https://test-api.service.hmrc.gov.uk/hello/world");
request.addHeader("Accept", "application/vnd.hmrc.1.0+json");
// execute the request
HttpResponse response = client.execute(request);
// extract the HTTP status code and response body
int statusCode = response.getStatusLine().getStatusCode();
String responseBody = EntityUtils.toString(response.getEntity());
Is that enough information for someone to point me in the right direction of what I'm doing wrong please? Suspect I am missing a fundamental point here.
In your Java example, you are setting the HTTP header "Accept". In your C++/Qt snippet, your are setting the "Content-Type" header.
You may want to adapt your code like this to match your Java working example:
QNetworkRequest request { QUrl(rs) };
request.setRawHeader(QByteArray("Accept"), QByteArray("application/vnd.hmrc.1.0+json"));
manager->get(request);

HTTP Compression of JSON Data on Windows Azure Web Role

I've got compression propperly configured for my Azure web role. Both .aspx pages and static pages like *.css are being compressed correctly.
<urlCompression doStaticCompression="true" doDynamicCompression="true" dynamicCompressionBeforeCache="true" />
I've got several different [System.Web.Services.WebMethod]'s though, that are not returning GZIP'd data. The size of each request is around 350KB, so I'm thinking it should be quite a bit faster if I can get this to work.
Within my webMethod, I create a list of objects, return the objects, and I assume some type of built in serializer turns this into JSON?
Is there anyway to force this content to be compressed?
Thanks so much!
I've seen people have issues with built in Compression for numerous reasons
The simplest way is to use a third party component such as Telerik's RadCompression to enforce compression on the response to AJAX calls.
Alternatively, you can override the application's BeginRequest method or write your own handler
to pack up the responses on the fly. A basic VB version of how to do this is here:
Sub Application_BeginRequest(...)
If Request.RawUrl.Contains(".aspx") And _
Not Request.Headers("Accept-Encoding") Is Nothing Then
If Request.Headers("Accept-
encoding").ToLower().Contains("gzip") Then
Response.Filter = New GZipStream(Response.Filter,CompressionMode.Compress, True)
Response.AppendHeader("Content-encoding", "gzip")
' Else...attempt deflate if GZip is not allowed
End If
End If
End Sub
I've done a method with the handler as well (and that's what I believe Telerik's RadCompression uses), but it is a good bit more complicated as you have to modify the response size, etc.
Here's what I ended up with, a variation Yak's answer.
HttpApplication app = (HttpApplication)sender;
HttpRequest request = app.Request;
HttpResponse response = app.Response;
System.Web.HttpApplication Appl = (System.Web.HttpApplication)sender;
HttpContext context = Appl.Context;
string origpath = context.Request.Url.AbsolutePath;
//Ajax Web Service request is always starts with application/json
if (request.ContentType.ToLower(CultureInfo.InvariantCulture).StartsWith("application/json"))
{
//User may be using an older version of IE which does not support compression, so skip those
if (!((request.Browser.IsBrowser("IE")) && (request.Browser.MajorVersion <= 6)))
{
string acceptEncoding = request.Headers["Accept-Encoding"];
if (!string.IsNullOrEmpty(acceptEncoding))
{
acceptEncoding = acceptEncoding.ToLower(CultureInfo.InvariantCulture);
if (acceptEncoding.Contains("gzip"))
{
response.Filter = new GZipStream(response.Filter, CompressionMode.Compress);
response.AddHeader("Content-encoding", "gzip");
}
else if (acceptEncoding.Contains("deflate"))
{
response.Filter = new DeflateStream(response.Filter, CompressionMode.Compress);
response.AddHeader("Content-encoding", "deflate");
}
}
}
}

Submitting an HTML form in Android without WebView

I was given an assignment to develop a very simple weather app in Android using the data given from
http://forecast.weather.gov/zipcity.php
This would be my first actual android app, my only other experience is watching/reading many tutorials on getting started.
To give some background information of the app, it simply needs to have one activity that is a
text field for the user to enter a city, state combination, or just a zip code. This then needs to be submitted to the website's (listed above) form.
After this has been submitted, my app needs to retrieve page sent in response and display it in a simple layout (possibly a listView of the days of the week).
My first question is, how would I go about submitting a text input to the above website's form? Would I have to know the name of the form's field to submit to it precisely?
In general, how should I start this process?
Edited version:
Using this code:
HttpClient httpClient = new DefaultHttpClient();
HttpPost post = new HttpPost("http://forecast.weather.gov/zipcity.php?inputstring=04419");
HttpResponse httpResp;
try {
httpResp = httpClient.execute(post);
StatusLine status = httpResp.getStatusLine();
Toast.makeText(getApplicationContext(), status.getStatusCode(), Toast.LENGTH_SHORT).show();
} catch (ClientProtocolException e) {
e.printStackTrace();
} catch (IOException e) {
e.printStackTrace();
}
I'm getting a:
10-26 15:29:56.686: DEBUG/SntpClient(59): request time failed: java.net.SocketException: Address family not supported by protocol
error in the debug view. Is this error related? Is my use of toast reasonable for testing if this http interaction is successful?
When you want to see what a post request in passing use Firefox + tamperdata, that way you can look at how to use the webservice.
I took a look at it, I searched for "33129" and when you enter text on the left hand box to start a search it simply pases 2 parameters:
inputstring = "33129"
Go2 = "Go"
That would be one way to do it, on the other hand, once the request is finished it transforms it into another request, thats more specific. You can search by city state or zip, not both.
If you request with a zip you get redirected to:
http://forecast.weather.gov/MapClick.php?CityName=Miami&state=FL&site=MFL&lat=25.77&lon=-80.2
So there is probably a redirection going on there.
You are going to have to take a close look at how to work with this.
Now, to do a post request in android use a name value pair like this.
List<NameValuePair> nameValuePairs = new ArrayList<NameValuePair>();
nameValuePairs.add(new BasicNameValuePair("inputstring","33129));
Ramp's answer shows the rest.
Also, ALL comunication should be done on a thread that is not the main UI thread or you will get a ANR error.
When I checked out your URL, I entered city and state.This is the URL it generates in HTTP POST :
http://forecast.weather.gov/MapClick.php?CityName=Edison&state=NJ&site=PHI&textField1=40.5288&textField2=-74.3693
So your will do a HTTP POST with the CityName and state URL parameters. It will be something like this if you use apache HttpClient in your android program :
( It should be OK to use HTTP GET too I guess)
HttpPost post = new HttpPost(url);//construct url with cityName and State
HttpResponse httpResp = httpClient.execute(post);
StatusLine status = httpResp.getStatusLine();
Hope this helps.
Something like this:
HttpClient hc = new DefaultHttpClient();
List<NameValuePair> l = new ArrayList<NameValuePair>(6);
l.add(new BasicNameValuePair("fieldname", "fieldvalue"));
//More fields...
HttpPost pr = new HttpPost(TheURL);
pr.setEntity(new UrlEncodedFormEntity(l, "the-proper-encoding"));
HttpResponse Resp = hc.execute(pr);
if(Resp.getStatusLine().getStatusCode() != 200)
//Fail...