My company is hosting an ecom shop based on Infinity Shop System. Our logs say that there are HTTP calls to this path which lead to 404 errors since the file does not exist:
http://{domain}/{somePath}/skin/default/images/tb-collectingalarm-red-low_inject.png
However, this reference is not made by us as I cannot find this path in any line of our source code.
The logs also state that only (some?) Firefox users do this call:
User Agent Mozilla/5.0 (Windows NT 6.3; rv:35.0) Gecko/20100101
Firefox/35.0
So, since this does cause quite some 404 errors, does anyone know what could cause these requests?
We already followed the referrer URL which lead to one of our sites but within its html markup we could not find any reference.
Related
I'm writing an application which will run on a microcontroller (arduino or Raspberry Zero) with wifi and a web server which will be configurable by a web browser without any client side scripts. This will use a string of HTML forms for the purpose of creating a number of small files on the microcontroller which will be interpreted by the microcontroller to perform its tasks.
I'm writing it initially on a Slackware Linux system but when it gets close to completion, will move it all to a Raspberry Pi running a customised version of Ubuntu Linux for final tuning.
I'm using lighttpd with mod_fastcgi and libfcgi and I am writing forms handler software in C.
Now, ideally, the responses returned to the server by each form would be processed by its individual handler daemon started by mod_fcgi, however I have not been able to figure out how to configure fastcgi to load more than one handler daemon. My fcgi.conf file is pointed at by a link later in this missive.
I could live with this restriction but another problem arises. In using just one handler, the action="handlerProgram" field at the top of every form has to point at that one handler, each form is unique and must be handled differently so how do I tell the formsHandler program which form is being handled? I need to be able to embed another label into each HTML form somewhere so that the web client will send this back to the server which will pass its value to the forms handler via the environment - or some such mechanism. Any clues on how to do this? Pleaase?
Peter.
PS. Here's a link to the related config and html data. HTML Problem
Maybe one of these solutions may help :
In the html code, add informations about the form to handle after the handler program name in the action tag, like :
action="/cgi-bin/handlerProgram/id/of/form/to/handle"
In your CGI handlerProgram you'll have the PATH_INFO environment variable valued to "/id/of/form/to/handle". Use it to know what form to handle.
In the html code add a hidden input field to your form like :
<input type="hidden" id="form_to_hanlde" value="form_id"/>
Just use the form_to_handle field's value in you handlerProgram to know what form to handle.
Joe Hect Posted an answer which completely solves this question.
The information which needed to be sent for the form called 'index.htm' is the name of the form. I used the action field "ACTION=/formsHandler.fcgi/index.htm" and below is the contents of the environment returned as reported by echo.fcgi (renamed to formsHandler.fcgi to avoid having to change anything else in my config.). If you can decipher the listing after this page has scrambled it, you will see that the required information is now present in a number of places, including PATH_INFO as suggested. Thank you, Joe.
Now all I have to do is figure out how to vote for you properly.
{
Request number 1
CONTENT_LENGTH: 37
DOCUMENT_ROOT: /home/lighttpd/htdocs
GATEWAY_INTERFACE: CGI/1.1
HTTP_ACCEPT: text/html, application/xhtml+xml, */*
HTTP_ACCEPT_ENCODING: gzip, deflate
HTTP_ACCEPT_LANGUAGE: en-AU
HTTP_CACHE_CONTROL: no-cache
HTTP_CONNECTION: Keep-Alive
HTTP_HOST: 192.168.0.16:6666
HTTP_PRAGMA:
HTTP_RANGE:
HTTP_REFERER: http://192.168.0.16:6666/
HTTP_TE:
HTTP_USER_AGENT: Mozilla/5.0 (Windows NT 6.1; WOW64; Trident/7.0; rv:11.0) like Gecko
HTTP_X_FORWARDED_FOR:
PATH:
PATH_INFO: /index.htm
PATH_TRANSLATED: /home/lighttpd/htdocs/index.htm
QUERY_STRING:
CONTENT_LENGTH: 37
CONTENT:
REMOTE_ADDR: 192.168.0.19
REMOTE_HOST:
REMOTE_PORT: 54159
REQUEST_METHOD: POST
REQUEST_ACTION:
ACTION:
REQUEST_URI: /formsHandler.fcgi/index.htm
REDIRECT_URI:
SCRIPT_FILENAME: /home/lighttpd/htdocs/formsHandler.fcgi
SCRIPT_NAME: /formsHandler.fcgi
SERVER_ADDR: 192.168.0.16
SERVER_ADMIN:
SERVER_NAME: 192.168.0.16
SERVER_PORT: 6666
SERVER_PROTOCOL: HTTP/1.1
SERVER_SIGNATURE:
SERVER_SOFTWARE: lighttpd/1.4.41
}
When a request arrives to my java servlet I'm checking its UserAgent:
protected void service(HttpServletRequest request, HttpServletResponse response){
final String UA = request.getHeader("User-Agent");
eu.bitwalker.useragentutils.Browser browser = UserAgent.parseUserAgentString(UA).getBrowser();}
Most requests has UA (User Agent) with information in it, e.g. Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/43.0.2357.130 Safari/537.36.
Some requests (about 10%) has only Mozilla/5.0 or Mozilla/4.0.
Does it means they are bots?
Is it possible that something before the servlet removes the relevant part in the UA?
I'm using HaraldWalker User Agent Utils to identify the UA and it returns Mozilla for those UA's.But this online tool returns unknown.
Can someone please explain?
It looks very likely that these are some sort of bot, as that user agent is not used by any mainstream browser.
It will be worth you filtering your logs to extract just these entries, and checking if they are following any sort of obvious bot-like pattern. For instance, you may see:
A request every X seconds exactly
That they all happen at a specific time of day
That they all happen within a very short period of time
That they request URLs in alphabetical order
That all the requests come from a single IP address, or limited range of IPs
I want to get html code from windows phone market pages. So far I have not run into any problems but today following error is displayed every time I retrieve data.
[...] Your request appears to be from an automated process.
If this is incorrect, notify us by clicking here to be redirected [...].
I tried to use proxy in case to many requests are called from one IP but this does not bring any progression. Do you happen to know why this problem takes place, any ideas about possible way outs? Any help would be very much appreciated. The main goal is to somehow get information about windows phone app from market.
It seems that they detect the user agent and block the request if it is not valid / known for a device.
I managed to make it work with curl with eg.
curl -A 'Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1.9) Gecko/20071025 Firefox/2.0.0.9' http://www.windowsphone.com/en-us/store/app/pinpoint-by-foundbite/ff9fdf41-aabd-4cac-9086-8710bd327da9
For asp.net, if you use HttpRequest to get the html content, try the following:
HttpWebRequest request = (HttpWebRequest)WebRequest.Create(url);
request.UserAgent = "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1.9) Gecko/20071025 Firefox/2.0.0.9";
For PHP you can set your user agent as well via curl_setopt.
I was not able to find out, whether there is an IP-based block after several requests.
This seems to only happen in Chrome (latest version 31.0.1650.48 m, but also earlier), but since it doesn't always happen it's hard to say for sure.
When streaming audio stored in Azure Blob storage, Chrome will occasionally play about 30-50% of the track and then stop. It's hard to reproduce, but if I clear the cache and play the file over and over again, it eventually happens. An example file can be found here.
The error is pretty much the same as what's described here, but I've yet to see the problem on any files hosted elsewhere.
Update:
The Azure Blog log only gives AnonymousSuccess messages, no error messages. This is what I get:
1.0;2013-11-14T12:10:10.6629155Z;GetBlob;AnonymousSuccess;200;3002;269;anonymous;;p3urort;blob;"http://p3urort.blob.core.windows.net/tracks/bd2fd171-b3c5-4e1c-97ba-b5109cf15098";"/p3urort/tracks/bd2fd171-b3c5-4e1c-97ba-b5109cf15098";c377a003-ea6b-4982-9335-15ebfd3cf1b1;0;160.67.18.120:54132;2009-09-19;419;0;318;7663732;0;;;"0x8D09A26E7479CEB";Friday, 18-Oct-13 14:38:53 GMT;;"Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/31.0.1650.48 Safari/537.36";"http://***.azurewebsites.net/";
Apparently you have to set the content type to audio/mpeg3
Here's how I do it:
CloudBlockBlob blockBlob = container.GetBlockBlobReference(fileName);
blockBlob.UploadFromStream(theStream);
theStream.Close();
blockBlob.Properties.ContentType = "audio/mpeg3";
blockBlob.SetProperties();
From here: https://social.msdn.microsoft.com/Forums/azure/en-US/0139d27a-0325-4be1-ae8d-fbbaf1710629/unable-to-load-audio-in-html5-audio-tag-from-storage?forum=windowsazuredevelopment
[edit] - This didn't actually work for me, I'm trying to troubleshoot, but I don't know what's wrong, going to ask a new question.
This mp3 only plays for 1.5 min and then stops. When downloaded, the file plays fully...
https://orator.blob.core.windows.net/mycontainer/zenhabits.net.unsolved.mp3
We are noticing a lot of duplicate records are being created in various tables in our database, but are at a loss as to why this is happening. Interestingly, while the records are otherwise duplicate (down to even the created_at stamps!), on our users table, the password salt and hash are different on each record -- which leads me to believe that somehow Rails is somehow running transactions/save operations twice. Obviously, we are not calling save or create multiple times in the application code.
This duplication does not seem to happen with every record saved in the database, and we cannot seem to infer a pattern yet. There is also a validates_uniqueness_of validation on the User model (though not a unique key on the table yet; we need to clean up all the duplicates to be able to do that) -- so Rails should stop itself if a record already exists, but if the requests are firing simultaneously that's a race condition.
We are currently running Rails 3.2.2 behind Passenger 3.0.11/nginx on our app servers (currently 2 of them), and have one central nginx webserver which sends requests upstream to an app server. Could this setup somehow cause processes to be duplicated or something? Would it matter that requests aren't locked to one upstream server (ie. if one user requests a page that includes static content like images, one or both app servers may be used)? (I feel like that's grasping at straws but I want to cover every possibility)
What else could cause this to happen?
Update: As an example, a user was created today which got duplicate records. Both have the created_at stamp of 2012-03-28 16:48:11, and all columns except for hashed_password and salt are identical. From the request log, I can see the following:
App Server 1:
Started POST "/en/apply/create_user" for 1.2.3.4 at 2012-03-28 12:47:19 -0400
[2012-03-28 12:47:19] INFO : Processing by ApplyController#create_user as HTML
[2012-03-28 12:47:20] INFO : Rendered apply/new_user.html.erb within layouts/template (192.8ms)
Started POST "/en/apply/create_user" for 1.2.3.4 at 2012-03-28 12:48:10 -0400
[2012-03-28 12:48:10] INFO : Processing by ApplyController#create_user as HTML
[2012-03-28 12:48:11] INFO : Redirected to apply/initialize_job_application/3517
[2012-03-28 12:48:11] INFO : /app/controllers/apply_controller.rb:263:in `block (2 levels) in create_user'
App Server 2:
Started POST "/en/apply/create_user" for 1.2.3.4 at 2012-03-28 12:48:10 -0400
[2012-03-28 12:48:10] INFO : Processing by ApplyController#create_user as HTML
Web Server:
1.2.3.4 - - [28/Mar/2012:12:48:10 -0400] "POST /en/apply/create_user HTTP/1.1" 499 0 "en/apply/create_user" "Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; Win64; x64; Trident/5.0)" "-"
1.2.3.4 - - [28/Mar/2012:12:48:11 -0400] "POST /en/apply/create_user HTTP/1.1" 302 147 "en/apply/create_user" "Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; Win64; x64; Trident/5.0)" "-"
So the create action was hit three times (returning to the form the first time due to an error, probably), and at least once on each server. The latter two both are registered by the webserver as separate requests, but the first gets status code 499 Client Closed Request (an nginx extension according to wikipedia), and the second gets a 302 as expected. Could the 499 be causing the problems here?
Two possibilities come to mind.
The first one is an odd (and against the RFC) behavior of Nginx when used as a load balancer. It will retry any failed requests against the next backend. The RFC allows that only for safe methods (e.g. GET or HEAD). The result of this is that if your nginx considers a request failed for some reason, it might be that it is re-send to the next server. If both servers complete their transaction though, you have a duplicate record. Judging from your webservers log (and the 499 status code which Nginx uses to denote a user clicking abort in their browser) this looks like the most probable cause.
The second possibility is that your users double-click on the send button. With the right timing, their browsers could send two complete requests nearly at the same time.
To make sure that your user records are really unique, you should create unique indexes on your database. These are then actually ensured (albeit with a worse error message compared to the ActiveRecord check. Because of that, you should always define your uniqueness constraint on both the database schema and your models.
Also, you could look into replacing your frontend nginx with a more conformant loadbalancer. I'd recommend haproxy for that.
It really seems like a race condition. Make sure to lock between the requests. It could easily happen that one or two requests are duplicated every now and then. The same can happen when exchanging items without transactions, so please make sure that you don't have a race between your requests.