How do I prevent google web crawler from reading a single page as two different pages - html

I have a web page say example.com/blog/news.php which I used this code:
RewriteEngine on
RewriteCond %{REQUEST_FILENAME} !-d
RewriteCond %{REQUEST_FILENAME}\.php -f
RewriteRule ^(.*)$ $1.php [NC,L]
in the .htaccess file to hide the file extension. But google crawler reads example.com/blog/news.php and example.com/blog/news as two different pages. Please how do I prevent this, I tried to redirect the news.php page to news but it returned an error message saying to much redirect or something like that.

You can 301 redirect your .php URLs to the new (non-php ) format using the following rule and this way Google will only index your new URL.
Put the following right bellow RewriteEngine on and before your existing rules :
RewriteCond %{ENV:REDIRECT_STATUS} ^$
RewriteRule ^([^.]+)\.php$ /$1 [L,R=301]

Related

Why is ".html" appearing on the end of my urls in browser? [duplicate]

How to remove .html from the URL of a static page?
Also, I need to redirect any url with .html to the one without it. (i.e. www.example.com/page.html to www.example.com/page ).
I think some explanation of Jon's answer would be constructive. The following:
RewriteCond %{REQUEST_FILENAME} !-f
RewriteCond %{REQUEST_FILENAME} !-d
checks that if the specified file or directory respectively doesn't exist, then the rewrite rule proceeds:
RewriteRule ^(.*)\.html$ /$1 [L,R=301]
But what does that mean? It uses regex (regular expressions). Here is a little something I made earlier...
I think that's correct.
NOTE: When testing your .htaccess do not use 301 redirects. Use 302 until finished testing, as the browser will cache 301s. See https://stackoverflow.com/a/9204355/3217306
Update: I was slightly mistaken, . matches all characters except newlines, so includes whitespace. Also, here is a helpful regex cheat sheet
Sources:
http://community.sitepoint.com/t/what-does-this-mean-rewritecond-request-filename-f-d/2034/2
https://mediatemple.net/community/products/dv/204643270/using-htaccess-rewrite-rules
To remove the .html extension from your urls, you can use the following code in root/htaccess :
RewriteEngine on
RewriteCond %{THE_REQUEST} /([^.]+)\.html [NC]
RewriteRule ^ /%1 [NC,L,R]
RewriteCond %{REQUEST_FILENAME}.html -f
RewriteRule ^ %{REQUEST_URI}.html [NC,L]
NOTE: If you want to remove any other extension, for example to remove the .php extension, just replace the html everywhere with php in the code above.
Also see this How to remove .html and .php from URLs using htaccess` .
This should work for you:
#example.com/page will display the contents of example.com/page.html
RewriteCond %{REQUEST_FILENAME} !-f
RewriteCond %{REQUEST_FILENAME} !-d
RewriteCond %{REQUEST_FILENAME}.html -f
RewriteRule ^(.+)$ $1.html [L,QSA]
#301 from example.com/page.html to example.com/page
RewriteCond %{THE_REQUEST} ^[A-Z]{3,9}\ /.*\.html\ HTTP/
RewriteRule ^(.*)\.html$ /$1 [R=301,L]
With .htaccess under apache you can do the redirect like this:
RewriteEngine On
RewriteCond %{REQUEST_FILENAME} !-f
RewriteCond %{REQUEST_FILENAME} !-d
RewriteRule ^(.*)\.html$ /$1 [L,R=301]
As for removing of .html from the url, simply link to the page without .html
page
You will need to make sure you have Options -MultiViews as well.
None of the above worked for me on a standard cPanel host.
This worked:
Options -MultiViews
RewriteEngine On
RewriteCond %{REQUEST_FILENAME} !-d
RewriteCond %{REQUEST_FILENAME} !-f
RewriteRule ^([^\.]+)$ $1.html [NC,L]
For those who are using Firebase hosting none of the answers will work on this page. Because you can't use .htaccess in Firebase hosting. You will have to configure the firebase.json file. Just add the line "cleanUrls": true in your file and save it. That's it.
After adding the line firebase.json will look like this :
{
"hosting": {
"public": "public",
"cleanUrls": true,
"ignore": [
"firebase.json",
"**/.*",
"**/node_modules/**"
]
}
}
Thanks for your replies. I have already solved my problem. Suppose I have my pages under http://www.yoursite.com/html, the following .htaccess rules apply.
<IfModule mod_rewrite.c>
RewriteEngine On
RewriteCond %{THE_REQUEST} ^[A-Z]{3,9}\ /html/(.*).html\ HTTP/
RewriteRule .* http://localhost/html/%1 [R=301,L]
RewriteCond %{THE_REQUEST} ^[A-Z]{3,9}\ /html/(.*)\ HTTP/
RewriteRule .* %1.html [L]
</IfModule>
Good question, but it seems to have confused people. The answers are almost equally divided between those who thought Dave (the OP) was saving his HTML pages without the .html extension, and those who thought he was saving them as normal (with .html), but wanting the URL to show up without. While the question could have been worded a little better, I think it’s clear what he meant. If he was saving pages without .html, his two question (‘how to remove .html') and (how to ‘redirect any url with .html’) would be exactly the same question! So that interpretation doesn’t make much sense. Also, his first comment (about avoiding an infinite loop) and his own answer seem to confirm this.
So let’s start by rephrasing the question and breaking down the task. We want to accomplish two things:
Visibly remove the .html if it’s part of the requested URL (e.g. /page.html)
Point the cropped URL (e.g. /page) back to the actual file (/page.html).
There’s nothing difficult about doing either of these things. (We could achieve the second one simply by enabling MultiViews.) The challenge here is doing them both without creating an infinite loop.
Dave’s own answer got the job done, but it’s pretty convoluted and not at all portable. (Sorry Dave.) Łukasz Habrzyk seems to have cleaned up Anmol’s answer, and finally Amit Verma improved on them both. However, none of them explained how their solutions solved the fundamental problem—how to avoid an infinite loop. As I understand it, they work because THE_REQUEST variable holds the original request from the browser. As such, the condition (RewriteCond %{THE_REQUEST}) only gets triggered once. Since it doesn’t get triggered upon a rewrite, you avoid the infinite loop scenario. But then you're dealing with the full HTTP request—GET, HTTP and all—which partly explains some of the uglier regex examples on this page.
I’m going to offer one more approach, which I think is easier to understand. I hope this helps future readers understand the code they’re using, rather than just copying and pasting code they barely understand and hoping for the best.
RewriteEngine on
# Remove .html (or htm) from visible URL (permanent redirect)
RewriteCond %{REQUEST_URI} ^/(.+)\.html?$ [nocase]
RewriteRule ^ /%1 [L,R=301]
# Quietly point back to the HTML file (temporary/undefined redirect):
RewriteCond %{REQUEST_FILENAME} !-d
RewriteCond %{REQUEST_FILENAME}.html -f
RewriteRule ^ %{REQUEST_URI}.html [END]
Let’s break it down…
The first rule is pretty simple. The condition matches any URL ending in .html (or .htm) and redirects to the URL without the filename extension. It's a permanent redirect to indicate that the cropped URL is the canonical one.
The second rule is simple too. The first condition will only pass if the requested filename is not a valid directory (!-d). The second will only pass if the filename refers to a valid file (-f) with the .html extension added. If both conditions pass, the rewrite rule simply adds ‘.html’ to the filename. And then the magic happens… [END]. Yep, that’s all it takes to prevent an infinite loop. The Apache RewriteRule Flags documentation explains it:
Using the [END] flag terminates not only the current round of rewrite
processing (like [L]) but also prevents any subsequent rewrite
processing from occurring in per-directory (htaccess) context.
Resorting to using .htaccess to rewrite the URLs for static HTML is generally not only unnecessary, but also bad for you website's performance. Enabling .htaccess is also an unnecessary security vulnerability - turning it off eliminates a significant number of potential issues. The same rules for each .htaccess file can instead go in a <Directory> section for that directory, and it will be more performant if you then set AllowOverride None because it won't need to check each directory for a .htaccess file, and more secure because an attacker can't change the vhost config without root access.
If you don't need .htaccess in a VPS environment, you can disable it entirely and get better performance from your web server.
All you need to do is move your individual files from a structure like this:
index.html
about.html
products.html
terms.html
To a structure like this:
index.html
about/index.html
products/index.html
terms/index.html
Your web server will then render the appropriate pages - if you load /about/, it will treat that as /about/index.html.
This won't rewrite the URL if anyone visits the old one, though, so it would need redirects to be in place if it was retroactively applied to an existing site.
I use this .htacess for removing .html extantion from my url site, please verify this is correct code:
RewriteEngine on
RewriteBase /
RewriteCond %{http://www.proofers.co.uk/new} !(\.[^./]+)$
RewriteCond %{REQUEST_fileNAME} !-d
RewriteCond %{REQUEST_fileNAME} !-f
RewriteRule (.*) /$1.html [L]
RewriteCond %{THE_REQUEST} ^[A-Z]{3,9}\ /([^.]+)\.html\ HTTP
RewriteRule ^([^.]+)\.html$ http://www.proofers.co.uk/new/$1 [R=301,L]
Making my own contribution to this question by improving the answer from #amit-verma (https://stackoverflow.com/a/34726322/2837434) :
In my case I had an issue where RewriteCond %{REQUEST_FILENAME}.html -f was triggering (believing the file existed) even when I was not expecting it :
%{REQUEST_FILENAME}.html was giving me /var/www/example.com/page.html for all these cases :
www.example.com/page (expected)
www.example.com/page/ (also quite expected)
www.example.com/page/subpage (not expected)
So the file it was trying to load (believing if was /var/www/example.com/page.html) were :
www.example.com/page => /var/www/example/page.html (ok)
www.example.com/page/ => /var/www/example/page/.html (not ok)
www.example.com/page/subpage => /var/www/example/page/subpage.html (not ok)
Only the first one is actually pointing to an existing file, other requests were giving me 500 errors as it kept believing the file existed and appending .html repeatedly.
The solution for me was to replace RewriteCond %{REQUEST_FILENAME}.html -f with RewriteCond %{DOCUMENT_ROOT}%{REQUEST_URI}.html -f
Here is my entire .htaccess (I also added a rule to redirect the user from /index to /) :
# Redirect "/page.html" to "/page" (only if "/pages.html" exists)
RewriteCond %{REQUEST_FILENAME} -f
RewriteCond %{THE_REQUEST} /(.+)\.html [NC]
RewriteRule ^(.+)\.html$ /$1 [NC,R=301,L]
# redirect "/index" to "/"
RewriteRule ^index$ / [NC,R=301,L]
# Load "/page.html" when requesting "/page" (only if "/pages.html" exists)
RewriteCond %{DOCUMENT_ROOT}%{REQUEST_URI}.html -f
RewriteRule ^ /%{REQUEST_URI}.html [QSA,L]
Here is a result example to help you understand all the cases :
Considering I have only 2 html files on my server (index.html & page.html)
www.example.com/index.html => redirects to www.example.com
www.example.com/index => redirects to www.example.com
www.example.com => renders /var/www/example.com/index.html
www.example.com/page.html => redirects to www.example.com/page
www.example.com/page => renders /var/www/example.com/page.html
www.example.com/page/subpage => returns 404 not found
www.example.com/index.html/ => returns 404 not found
www.example.com/page.html/ => returns 404 not found
www.example.com/test.html => returns 404 not found
No more 500 errors 🚀
Also, just to help you debug your redirections, consider disabling the network cache in your browser (as old 301 redirections my be in cache, wich may cause some headaches 😅):
first create a .htaccess file and set contents to -
RewriteEngine on
RewriteCond %{REQUEST_FILENAME} !-d
RewriteCond %{REQUEST_FILENAME}\.html -f
RewriteRule ^(.*)$ $1.html
next remove .html from all your files eg. test.html became just test and also if you wanna open a file from another file then also remove .html from it and just file name
Use a hash tag.
May not be exactly what you want but it solves the problem of removing the extension.
Say you have a html page saved as about.html and you don't want that pesky extension you could use a hash tag and redirect to the correct page.
switch(window.location.hash.substring(1)){
case 'about':
window.location = 'about.html';
break;
}
Routing to yoursite.com#about will take you to yoursite.com/about.html. I used this to make my links cleaner.
To remove the .html extension from your URLs, you can use the following code in root/htaccess :
#mode_rerwrite start here
RewriteEngine On
# does not apply to existing directores, meaning that if the folder exists on server then don't change anything and don't run the rule.
RewriteCond %{REQUEST_FILENAME} !-d
#Check for file in directory with .html extension
RewriteCond %{REQUEST_FILENAME}\.html !-f
#Here we actually show the page that has .html extension
RewriteRule ^(.*)$ $1.html [NC,L]
Thanks
For this, you have to rewrite the URL from /page.html to /page
You can easily implement this on any extension like .html .php etc
RewriteRule ^(.*).html$ $1.html [L]
RewriteCond %{REQUEST_FILENAME} !-f
RewriteRule ^([^\.]+)$ $1.html [NC,L]
You will get a URL something like this:
example.com/page.html to example.com/page
Please note both URLs below will be accessible
example.com/page.html and example.com/page
If you don't want to show page.html
Try this
RewriteRule ^(.*).html$ $1 [L]
RewriteCond %{REQUEST_FILENAME} !-f
RewriteRule ^([^\.]+)$ $1.html [NC,L]
More info here
If you have a small static website and HTML files are in the root directory.
Open every HTML file and make the next changes:
Replace href="index.html" with href="/".
Remove .html in all local links. For example: "href="about.html"" should look like "href="about"".
RewriteEngine On
RewriteCond %{THE_REQUEST} ^[A-Z]{3,9}\ /html/(.*).html\ HTTP/
RewriteRule .* https://example.com/html/%1 [R=301,L]
RewriteCond %{THE_REQUEST} ^[A-Z]{3,9}\ /html/(.*)\ HTTP/
RewriteRule .* %1.html [L]
it might work because its working in my case
RewriteRule /(.+)(\.html)$ /$1 [R=301,L]
Try this :) don't know if it works.

Need to redirect missing URLs to new host

I'm moving a blog from one site to another and repurposing the original site. I want to maintain all existing links that point to the site and hopefully maintain SEO page ranking.
Old URL: http://www.companyabc.com/2010/04/test.html
New URL: http://blog.companyabc.com/2010/04/test.html
The way I'd like to do it is to use a custom 404 error page on www.companyabc.com like this:
<html><meta http-equiv="REFRESH" content="0;url=http://blog.companyabc.com/%1"></html>
where %1 is the original URI (/2010/04/test.html), but I don't know if that's possible.
Another option is to use an .htaccess file that redirects if the URL is not found, but I haven't gotten that to work either. I'm sure I'm doing something wrong in the rewrite condition:
RewriteEngine On
RewriteCond %{REQUEST_URI} !-l
RewriteRule ^(.*)\.html$ https://blog.companyabc.com/$1.html [R=301,L]
Any suggestions? Thanks for the help.
I got it working using the following .htaccess configuration:
RewriteEngine On
RewriteCond %{REQUEST_FILENAME} !-f
RewriteCond %{REQUEST_FILENAME} !-d
RewriteRule ^(.*)$ http://blog.companyabc.com/$1 [R=301,L]
I never tried the REQUEST_FILENAME value because I mistakenly thought it only applied to files that can be downloaded from the website. I didn't realize it also applies to the .html files in the blog.
With this solution all URLs that don't exist at www.companyabc.com will be redirected to blog.companyabc.com instead of showing a 404 error page, which is what I'm looking for.

Redirect according to the URL

I have the following scenario:
I'd like to do a redirection system like this:
/redirect/place1 -> /page.php?place=place1
/redirect/testabc -> /page.php?place=testabc
/redirect/xyzw123 -> /page.php?place=xyzw123
The only way I can see now is to do subfolders in /redirect which have a index.php who redirects it. But I'd like that to be fully automatic.
Thanks for any answer!
If you want to redirect /redirect/(Anything) to /page.php?place=(Same as Anything)
You can use redirect command from mod_alias:
Redirect 301 /redirect/ /page.php?place=
or
you wan use the RewriteRule comand from mod_rewrite
RewriteRule /redirect/(.*) /page.php?place=$1 [L]
More informations:
[L] is the kind of redirect, you can add a status code if you want to do a redirect condition, like [L,R=301] will redirect sending 301 redirect permanent
here you can find all http code for the redirection
https://fr.wikipedia.org/wiki/Liste_des_codes_HTTP
here you can find all the flags from the rewrite condition
https://httpd.apache.org/docs/2.4/rewrite/flags.html
If you want to redirect only if page or folder doesn't exist:
RewriteCond %{DOCUMENT_ROOT}%{REQUEST_URI} !-f
RewriteCond %{DOCUMENT_ROOT}%{REQUEST_URI} !-d
RewriteRule /redirect/(.*) /page.php?place=$1 [L]
!-d mean not found as a folder
!-f mean not found as a file

Shortten URL's with htaccess that works with Google indexing

Ok, let's explain first what I have in the server and in the htaccess file:
In the server I have the following files:
www.mydomain.com/provincias/madrid/town1.html
www.mydomain.com/provincias/madrid/town2.html
www.mydomain.com/provincias/madrid/town3.html
...
www.mydomain.com/provincias/barcelona/town1.html
www.mydomain.com/provincias/barcelona/town2.html
...
as you see, the folder 'provincias' is repeated in all urls, is necessary for get ordered the internal files but not for navigate and for users, for this reason I have added the following rule in the htaccess file that works fine:
RewriteRule ^([a-zA-Z]+)/([a-zA-Z]+)\.html$ provincias/$1/$2\.html [L,NC]
so with this rule you can access, for example, the next url: www.mydomain.com/provincias/barcelona/town2.html with this other url that users see in the address bar of their browsers: www.mydomain.com/barcelona/town2.html because internally the htaccess redirect it to the real url with 'provincias'.
But I have a problem, I have seen that Google has indexed the technical and real url, the first one, with 'provincias' folder, and if you click on it people navigate watching that url in their address bar. How can I redirect the people and google traffic from www.mydomain.com/provincias/barcelona/town2.html to www.mydomain.com/barcelona/town2.html taking into consideration that internally the url without 'provincias' doesn't exist?
Try this rule in your .htaccess
#redirect real url to new one
RewriteCond %{THE_REQUEST} [A-Z]{3,}\ /+provincias/([^&\ ]+)/([^&\ ]+)\.html
RewriteRule ^ /%1/%2.html? [R=301,L]
RewriteCond %{REQUEST_FILENAME} !-f
RewriteCond %{REQUEST_FILENAME} !-d
RewriteRule ^([a-zA-Z]+)/([a-zA-Z]+)\.html$ provincias/$1/$2\.html [L,NC]

Automatically Rewrite html,php,css or any file extension using htaccess?

is there a way to automatically remove the file extension from the url?
for ex:
the user types
www.website.com/page.html
it will automatically convert the page to:
www.website.com/page
and if possible with a /:
www.website.com/page/
it must be automatic that the .htaccess will force the url to rewrite the file extension.
You try to do this wrong way. You do not want to remove extensions. You want to rewrite extensionless requests to real files. So if user type
http://foo.bar/file
your server would serve as it was requested
http://foo.bar/file.html
To do this you need mod_rewrite (or equivalent) and set rewriting rules according to your needs. The same module is also used to make URLs looking nicer, so instead of
http://foo.bar/script.php?id=34&smth=abc
you can have
http://foo.bar/script/id/34/smth/abc
or even
http://foo.bar/script/34/abc
Read more on mod rewrite.
I use this in .htaccess (index.php is handling the pages):
RewriteRule ^home/$ index.php?page=home
// Result: http://domain.com/home/
If you want it more dynamic:
RewriteRule ^(.*)/$ index.php?page=$1
// Result: http://domain.com/any-page/
For more data you might need when requesting a page:
RewriteRule ^(.*)/(.*)/$ index.php?page=$1&id=$2
// Result: http://domain.com/data-page/15/
If you want to just get the called page without the index part:
RewriteRule ^(.*)/$ $1.php
To remove extensions the user adds:
RewriteCond %{REQUEST_FILENAME} !-d
RewriteCond %{REQUEST_FILENAME}\.php -f
RewriteCond %{REQUEST_FILENAME}\.html -f
You need to do 2 things. First you need to change all of your links to the URLs without extensions, then you need to create a 301 redirect to redirect browsers and bots that still may have the old URLs with extensions to the new nice looking ones. This a server response of a new location, it's not a rewrite (which is an on-the-server-only internal URI rewrite).
RewriteEngine On
RewriteCond %{THE_REQUEST} ^(GET|HEAD)\ /(.*)\.(html?|php)
RewriteRule ^ /%2/ [L,R=301]
This makes it so when someone types http://www.website.com/page.html in their browser, they get redirected to http://www.website.com/page/ and the new URL will appear in the browser's URL address bar.
Now the second thing you need to do is internally change it back to the valid resource, since /page/ doesn't exist, it'll return a 404.
# need these so we don't clobber legit requests, just pass them through
RewriteCond %{REQUEST_FILENAME} -f [OR]
RewriteCond %{REQUEST_FILENAME} -d
RewriteRule ^ - [L]
# now check the extensions
RewriteCond %{REQUEST_URI} ^/(.*?)/?$
RewriteCond %{DOCUMENT_ROOT}/%1.php -f
RewriteRule ^ /%1.php [L]
RewriteCond %{REQUEST_URI} ^/(.*?)/?$
RewriteCond %{DOCUMENT_ROOT}/%1.html -f
RewriteRule ^ /%1.html [L]
RewriteCond %{REQUEST_URI} ^/(.*?)/?$
RewriteCond %{DOCUMENT_ROOT}/%1.htm -f
RewriteRule ^ /%1.htm [L]
You'll just need to put these in the htaccess file in your document root.