I'm trying to setup JMeter to do some stress testing on one of our applications.
So far I've been able to make it go to the login page, login to see the list of projects but when I try to make it go into a project the URL gets messed up and gives me a 500 error.
After loggin in, the URL has parameters like
?Client_Id='CLIENT'&Project_Id='PROJECT'&System_Id=2
When it tries to go into the project, it opens a page index.asp which builds some frames dynamically.
When building the frames, the response and the URL for each frame look fine. But, when making the actual call for each of the frames the HTTP request gets called like this:
?Client_Id='CLIENT'<<oject_Id='PROJECT'
Resulting in invalid query and 500 error from the server.
Related
The following code used to work but not anymore and I'm seeing junk HTML with success code of 200 returned.
response = urlopen('https://www.tipranks.com/api/stocks/stockAnalysisOverview/?tickers='+symbol)
data = json.load(response)
If you open the page in chrome you will see the JSON file format. But when opened in python I'm now getting:
f1xx.v1xx=v1xx;f1xx[374148]=window;f1xx[647467]=e8NN(f1xx[374148]);f1xx[125983]=n3EE(f1xx[374148]);f1xx[210876]=(function(){var
P6=2;for(;P6 !== 1;){switch(P6){case 2:return {w3:(function(v3){var
v6=2;for(;v6 !== 10;){switch(v6){case 2:var O3=function(W3){var
u6=2;for(;u6 !== 13;){switch(u6){case 2:var o3=[];u6=1;break;case
14:return E3;break;case 8:U3=o3.H8NN(function(){var Z6=2;for(;Z6 !==
1;){switch(Z6){case 2:return 0.5 - B8NN.P8NN();break;}}.....
What should I be doing to adapt to the new backend change so that I can parse the JSON again.
It is a bot protection, to prevent people from doing what you are doing. This API endpoint is supposed to be used only by the website itself, not by some Python script!
If you delete your site data and then freshly access the page in the browser, you'll see it first loads the HTML page that you see which loads some JavaScript, which then executes a POST to another URL with some data. Somewhere in the process a number of cookies get set and finally the code refreshes the page which then loads the JSON data. At this point visiting the URL directly returns the data because the correct cookies are already set.
If you look at those requests, you'll see the server returns a header server: rhino-core-shield. If you google that, you can see that it's part of the Reblaze DDoS Protection Platform.
You may have luck with a headless browser like ghost.py or pyppetteer but I'm not sure how effective it will be, you'll have to try. The proper way to do this would be to find an official (probably paid) API for getting the information you need instead of relying on non-public endpoints.
I'm trying to get data from this site: [1] https://www.eurobet.it/it/scommesse/#!/calcio/?temporalFilter=TEMPORAL_FILTER_OGGI_DOMANI
I found this link where I can get the data in JSON format: [2] https://www.eurobet.it/detail-service/sport-schedule/services/discipline/calcio?prematch=1&live=0&temporalFilter=TEMPORAL_FILTER_OGGI_DOMANI
But there is a problem:
The JSON link Doesn't work every time in fact sometimes I get a 404 error.
I noticed that if I open the first link [1] before opening the second [2] it works perfectly.
This error is also more frequent when I try to scrape other data on the same site: [3] https://www.eurobet.it/detail-service/sport-schedule/services/discipline/calcio/piu-giocate/u-o-goal?prematch=1&live=0&temporalFilter=TEMPORAL_FILTER_OGGI_DOMANI
In this link [3] I try to get all "u-o-goal" odds but this link works only if (before starting my program to scrape data) in the main link [1] I press the "U/O GOAL" button -> https://i.stack.imgur.com/Nei5u.png
In my code, I'm using Java and htmlunit to scrape the data.
My question is: how this webpage works, why couldn't I open directly the links [2]/[3], I know that there is a sort of request and approval system behind but I can't see where.
You cannot directly open these URLs since the website (and many like it) will use cookies and bot-prevention techniques/session tracking so they can gather data about usage of their website. eg. they set a "Referer".
I'm not going to code a solution for you but I can at least help you understand what you need to do to get to where you want...
I've attempted to summarise how I'd typically unpick a request like this to recreate it, but in its essence, you need to understand the sequence of HTTP requests being made (this is how the web works - HTTP requests).
First you typically start with no session cookies and you access the site directly (no referer).
Once you access a website, typically the server responds with a session cookie for you to communicate back to the server a unique session ID so it has some sort of record of your browser having already been in contact.
Your browser may make more requests (asynchronously) and in doing so typically sends the cookies and the referring URL (usually the base Url will work... just don't use something that starts with something other than "https://www.eurobet.it"
anything else you're going to need to figure it out. Lots of headers are optional. Lots of query params have defaults.
https://stackoverflow.com/a/64671815/7619034 - here's an answer I've given before that answers this type of question which comes up often enough.
so to explain a bit further, for your specific scenario...
When you access https://www.eurobet.it/it/scommesse/#!/calcio/?temporalFilter=TEMPORAL_FILTER_OGGI_DOMANI, the server responds with HTTP headers:
...
set-cookie: __cfduid=dd38d***********41125; ...
...
The rest doesn't look that relevant:
Going straight to the other request: https://www.eurobet.it/detail-service/sport-schedule/services/discipline/calcio?prematch=1&live=0&temporalFilter=TEMPORAL_FILTER_OGGI_DOMANI
This HTTP request takes (as input):
cookie: __cfduid=dd38d***********41125; mbox=session#6661556c.....b6e8cc1fa6f03#1608242987; at_check=true; s_ecid=MCMID%***********2021453010; AMCVS_45F10C3A53DAEC9F0A490D4D%40AdobeOrg=1; AMCV_45F10C3A53DAEC9F0A490D4D%40AdobeOrg=1075005958%7CMCIDTS%7C18614%7CMCMID%7C91883906030825914429183258312021453010%7CMCAID%7CNONE%7CMCOPTOUT-1608248327s%7CNONE%7CvVersion%7C4.4.1; s_cc=true
...
referer: https://www.eurobet.it/it/scommesse/
...
x-eb-accept-language: it_IT
x-eb-marketid: 5
x-eb-platformid: 1
Cookies are set in an initial request (typically) using Set-Cookie header and then are passed back to the server in subsequent requests using the cookie header.
I'm not certain how many of these values are relevant but you'd need to figure out where each came from in the chain of HTTP requests between the initial one and this one and you'd need to replicate them (see url above of my previous answer - warning this can be time consuming).
The other headers can be set statically most likely since they probably aren't due to change.
If you have access to curl on the command line, you can attempt to reconstruct some of these requests by hand. Some will be time sensitive since cookies do expire after an amount of time (see set-cookie header details for exactly when). Once you've reconstructed a working request, you can then start coding it in your application.
If you can work all this out you should be able to re-construct the chain of HTTP GET requests to get the JSON data you want. Good luck!
I am having trouble finding a way to perform a simple operation without making it way more complex than it has to be.
Example: I want to say "Alexa, What is the status of my website?"
I want it to know I'm referring to http://refindustry.com/index1.php
And I want it to read the single line on that page that at current says"Our website is under construction"
It is a really simple operation each time I want it to request the defined page and read the single line of html text on the page.
Keep in mind I don't want to have to host a server or pay amazon to do this I just want her to simply request the page and read the single line.
Tried going to amazon developer and it looked insane account linking lamda requests just way more difficult that it should be.
You can get the free tier of AWS to host a lambda that can fetch the text from your website and return the response to Alexa.
Then you need to setup your skill with a few sample utterances for how to ask for the status of your website and use this lambda.
You have to have a service that will fulfil the intent from Alexa - this could be a server somewhere (so the skill would send the request via HTTP), or a Lambda function.
As Josep says, you can get a free AWS tier - write some code to fetch the page and "grab" the status, before using it in the response to Alexa. It should be pretty straight forward to do this.
If you're a little daunted about AWS, you could look at Hosted Skills - https://developer.amazon.com/blogs/alexa/post/ebc1c777-2cb2-4210-8c89-2a70dd1a0248/get-your-head-out-of-the-cloud-with-alexa-hosted-skills-preview
Background
laravel currently allows you to easily define views for specific HTTP status code responses. For example, an HTTP status code of 404 will display the resources/views/errors/404.blade.php view automatically if it exists. it works the same for other codes like 500 errors.
Problem
All of the routes on my site are processed through controllers, all of which extend a base Controller. This base controller sometimes initializes the user, sets the current timezone, and other random stuff. The master template often relies on these variables. For example, if a user is logged in, the Controller figures that out and passes that user to the view. My master template looks for that user, and shows certain functionality if present. When a 404 is hit, I want to still be able to show the user specific menus, and continue using that 404 view.
Question
I submitted an issue on Github to see if we could route HTTP exceptions through controllers, but they did not agree with the proposal. So now I'm looking to see if there is a way to render a controller method when I catch an HTTP exception. I do not want to simply redirect to the appropriate route, but rather catch the exception and render the controller method.
I don't know enough about the internals of the routing, so I'm curious, is this possible? And if so, how do I do it?
Thanks!
It's definitely possible - look into the ExceptionHandler (under app\Exceptions), or look at implementing some Middleware to catch unknown routes.
Alternatively you can also extend the 404 view from your base template, save data in the session to present on the view or even call a ->back() on the 404 with a flash error.
I'm trying to standardize the way I handle exceptions in my web application (homemade framework) but I'm not certain of the "correct" way to handle various situations. I'm wondering if there is a best practice from UI/ user-friendly point of view.
User logs into application and opens two tabs showing the same screen. On one tab they issue a delete command on object FOO. Then, in the other tab they then click the edit command on FOO (which no longer exists); e.g. a GET request for editObject.php?object_id=FOO. What should I do when they issue the edit request for this nonexistent object?
-Currently I am redirecting these "missing" objects to the previous page with an error message like "object does not exist".
User issues a GET request to search for Objects with color=Red, e.g. searchObjects.php?color=Red. The query returning these results blew up because somebody dropped the OBJECTS table. This is an unexpected exception and isn't quite the same as 1).
-Currently I am redirecting to errorPage.php with a message "Unexpected error"
In general, what should I do if GET/POST parameters that should be there are instead mysteriously missing. Perhaps somebody is trying to inject something?
-Currently I am treating these the same as 2)
What should I be doing in each of the above 3 cases?
Render a "Object does not exist" view at the url editObject.php?object_id=FOO
Redirect to a controller that displays an error view: header('Location: errorPage.php')
Serve a 404: not sure of the syntax for doing this in PHP/Apache
Other
I'd say render it and serve a 404. That way, the user has the chance to see where they went wrong in the URL, or copy & paste it. If you redirect to a generic error page, they don't have that chance.
The PHP way to serve a 404 is
header("HTTP/1.0 404 not found");