The question might be a bit basic – considering I'm not what the vast majority would consider a newcomer to front end web development.
I am teaching an 8 year old html, css and javascript. I'm taking the opportunity to also teach about utf-8 encoding, in particular the way HTML uses it to allow non-English characters to be encoded and displayed.
I want to show him how accented characters do not appear properly without including <meta charset="UTF-8"/>.
Surprisingly I was able to display "Á" in the test webpage when in theory this shouldn't have been possible as the utf-8 charset meta tag was missing.
After some research I came to the conclusion that in modern IDE's the encoding system comes "built in", hence there's no real need to write down <meta charset />. If this is wrong please correct me as I am currently confused as to what exactly happened and I don't want to teach wrong information to an 8 year old.
After some research I came to the conclusion that in modern IDE's the encoding system comes "built in", hence there's no real need to write down . If this is wrong please correct me
Yes, that is wrong!
Surprisingly I was able to display "Á" in the test webpage when in theory this shouldn't have been possible as the utf-8 charset meta tag was missing.
This is also wrong, let me explain!
UTF-8 is an encoding system. This means it describes how to map bytes into textual characters. It's certainly possible to display "Á" without using utf-8.
The letter A (normal, no accents) is encoded with the number 65 in both ASCII and UTF-8. In fact, all english characters and punctuation are encoded the same way across virtually all encodings, so encoding problems rarely become apparent in English-only text.
However, accented letters, non-english characters and emojis (😁) are encoded differently in different encoding systems. What causes "corrupt" text to be displayed is an encoding mismatch: your web browser thinks the encoding used is X while the file was actually encoded with system Y, so byte values no longer map to correct characters. For example, system X uses number 250 to encode 😁, while system Y uses number 190, and under system Y 250 is mapped to "Ë". So now my 😁 appear as "Ë".
<meta charset="utf-8"/> specifies the encoding used for the HTML file. It is absolutely needed. Your webpage worked without because browsers may use other ways to get it, including educated guesses, but it should always be explicitly written in the HTML to avoid problems down the line.
You should specify the encoding for several reasons:
Even if the encoding system would come buit-in, you cannot know which is the default encoding chosen for the IDE.
HTML5 specification says that the default encoding should be taken from the transport layer when not specified which will be the default encoding charset for HTTP1.1: ISO-8859-1.
See the full explaination here: Why it's necessary to specify the character encoding in an HTML5 document if the default character encoding for HTML5 is UTF-8?
In order to define charset for HTML5 Doctype, which notation should I use?
Short:
<meta charset="utf-8" />
Long:
<meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
In HTML5, they are equivalent. Use the shorter one, as it is easier to remember and type. Browser support is fine since it was designed for backwards compatibility.
Both forms of the meta charset declaration are equivalent and should work the same across browsers. But, there are a few things you need to remember when declaring your web files character-set as UTF-8:
Save your file(s) in UTF-8 encoding without the byte-order mark (BOM).
Declare the encoding in your HTML files using meta charset (like above).
Your web server must serve your files, declaring the UTF-8 encoding in the Content-Type HTTP header.
Apache servers are configured to serve files in ISO-8859-1 by default, so you need to add the following line to your .htaccess file:
AddDefaultCharset UTF-8
This will configure Apache to serve your files declaring UTF-8 encoding in the Content-Type response header, but your files must be saved in UTF-8 (without BOM) to begin with.
Notepad cannot save your files in UTF-8 without the BOM. A free editor that can is Notepad++. On the program menu bar, select "Encoding > Encode in UTF-8 without BOM". You can also open files and re-save them in UTF-8 using "Encoding > Convert to UTF-8 without BOM".
More on the Byte Order Mark (BOM) at Wikipedia.
Another reason to go with the short one is that it matches other instances where you might specify a character set in markup. For example:
<script type="javascript" charset="UTF-8" src="/script.js"></script>
<p><a charset="UTF-8" href="http://example.com/">Example Site</a></p>
Consistency helps to reduce errors and make code more readable.
Note that the charset attribute is case-insensitive. You can use UTF-8 or utf-8, however UTF-8 is clearer, more readable, more accurate.
Also, there is absolutely no reason at all to use any value other than UTF-8 in the meta charset attribute or page header. UTF-8 is the default encoding for Web documents since HTML4 in 1999 and the only practical way to make modern Web pages.
Also you should not use HTML entities in UTF-8. Characters like the copyright symbol should be typed directly. The only entities you should use are for the five reserved markup characters: less than, greater than, ampersand, prime, double prime.
Entities need an HTML parser, which you may not always want to use going forward. They introduce errors, make your code less readable, increase your file sizes, and sometimes decode incorrectly in various browsers depending on which entities you used. Learn how to type/insert copyright, trademark, open quote, close quote, apostrophe, em dash, en dash, bullet, Euro, and any other characters you encounter in your content, and use those actual characters in your code.
The Mac has a Character Viewer that you can turn on in the Keyboard System Preference, and you can find and then drag and drop the characters you need, or use the matching Keyboard Viewer to see which keys to type. For example, trademark is Option + 2. UTF-8 contains all of the characters and symbols from every written human language.
So there is no excuse for using -- instead of an em dash. It is not a bad idea to learn the rules of punctuation and typography also ... for example, knowing that a period goes inside a close quote, not outside.
Using a <meta> tag for something like content-type and encoding is highly
ironic, since without knowing those things, you couldn't parse the file
to get the value of the meta tag.
No, that is not true. The browser starts out parsing the file as the browser's default encoding, either UTF-8 or ISO-8859-1. Since US-ASCII is a subset of both ISO-8859-1 and UTF-8, the browser can read <html><head> just fine either way ... it is the same. When the browser encounters the meta charset tag, if the encoding is different than what the browser is already using, the browser reloads the page in the specified encoding.
That is why we put the meta charset tag at the top, right after the head tag, before anything else, even the title. That way you can use UTF-8 characters in your title.
You must save your file(s) in UTF-8 encoding without BOM
That is not strictly true. If you only have US-ASCII characters in your document, you can Save it as US-ASCII and serve it as UTF-8, because it is a subset. But if there are Unicode characters, you are correct, you must Save as UTF-8 without BOM.
If you want a good text editor that will save your files
in UTF-8, I recommend Notepad++.
On the Mac, use Bare Bones TextWrangler (free) from Mac App Store, or Bare Bones BBEdit which is at Mac App Store for $39.99 ... very cheap for such a great tool.
In either app, there is a menu at the bottom of the document window where you specify the document encoding and you can easily choose "UTF-8 no BOM". And of course you can set that as the default for new documents in Preferences.
But if your Webserver serves the encoding in the HTTP header,
which is recommended, both [meta tags] are needless.
That is incorrect. You should of course set the encoding in the HTTP header, but you should also set it in the meta charset attribute so that the page can be saved by the user, out of the browser onto local storage and then opened again later, in which case the only indication of the encoding that will be present is the meta charset attribute.
You should also set a base tag for the same reason ... on the server, the base tag is unnecessary, but when opened from local storage, the base tag enables the page to work as if it is on the server, with all the assets in place and so on, no broken links.
AddDefaultCharset UTF-8
Or you can just change the encoding of particular file types like so:
AddType text/html;charset=utf-8 html
A tip for serving both UTF-8 and Latin-1 (ISO-8859-1) files is to give the UTF-8 files a "text" extension and Latin-1 files "txt."
AddType text/plain;charset=iso-8859-1 txt
AddType text/plain;charset=utf-8 text
Finally, consider saving your documents with Unix line endings, not legacy DOS or (classic) Mac line endings, which don't help and may hurt, especially down the line as we get further and further from those legacy systems.
An HTML document with valid HTML5, UTF-8 encoding, and Unix line endings is a job well done. You can share and edit and store and read and recover and rely on that document in many contexts. It's lingua franca. It's digital paper.
<meta charset="utf-8"> was introduced with/for HTML5.
As mentioned in the documentation, both are valid. However, <meta charset="utf-8"> is only for HTML5 (and easier to type/remember).
In due time, the old style is bound to become deprecated in the near future. I'd stick to the new <meta charset="utf-8">. There's only one way, but up. In tech's case, that's phasing out the old (really, REALLY fast)
Documentation: HTML meta charset Attribute—W3Schools
While not contesting the other answers, I think the following is worthy of mentioning.
The “long” (http-equiv) notation and the “short” one are equal. Whichever comes first wins;
Web server headers will override all the <meta> tags;
BOM (byte order mark) will override everything, and in many cases it will affect HTML 4 (and probably other stuff, too);
If you don't declare any encoding, you will probably get your text in “fallback text encoding” that is defined your browser. Neither in Firefox nor in Chrome it's UTF-8;
In absence of other clues the browser will attempt to read your document as if it was in ASCII to get the encoding, so you can't use any weird encodings (UTF-16 with BOM should do, though);
While the specifications say that the encoding declaration must be within the first 512 bytes of the document, most browsers will try reading more than that.
You can test by running echo 'HTTP/1.1 200 OK\r\nContent-type: text/html; charset=windows-1251\r\n\r\n\xef\xbb\xbf<!DOCTYPE html><html><head><meta http-equiv="Content-Type" content="text/html; charset=utf-8"><meta charset="windows-1251"><title>привет</title></head><body>привет</body></html>' | nc -lp 4500 and pointing your browser at localhost:4500. (Of course you will want to change or remove parts. The BOM part is \xef\xbb\xbf. Be wary of the encoding of your shell.)
Please mind that it's very important that you explicitly declare the encoding. Letting browsers guess can lead to security issues.
Use <meta charset="utf-8" /> for web browsers when using HTML5.
Use <meta http-equiv="Content-Type" content="text/html; charset=utf-8" /> when using HTML4 or XHTML, or for outdated DOM parsers, like DOMDocument in PHP 5.3.
To embed a signature in an email, I would use the long version:
<meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
The reason is that not many email readers use HTML5, so it's always better use old HTML styles. Actually, it's better to use tables than divs + CSS as well.
There is some news based on Mozilla Foundation, and SitePoint:
Do not use this value (http-equiv=content-type) as it is obsolete.
Prefer the charset attribute on the <meta> element.
I had an issue earlier today where someone couldn't compile a static site due to some non-ASCII characters in a kramdown file. While writing a small script that finds these characters in our content, I ran across a large number of non-HTML encoded special characters.
What are the implications in including these characters directly in the HTML? Take the © character.
If I include the character directly in HTML, it seems to render correctly in my browser. That being said, I don't know the side-effects for those who don't have fonts installed that support these characters.
What are the side effects of leaving these non-ASCII characters in the HTML? I know in some situations it can lead to strange (?) characters showing up, but I'd like more specific information on how these special characters get rendered.
If I HTML encode these special characters and a client doesn't have a font that supports them, does it show the same (?) character? Is there any meaningful difference between using the HTML-encoded vs non encoded characters?usign
Is there any meaningful difference between using the HTML-encoded vs non encoded characters?
Not in terms of the browser being able to display them in general.
If you want to use these as you call them "non-standard" characters (which are very much standard characters, just not ASCII characters), you should specify an encoding, preferably utf-8. The HTML5 way of doing this (which is backwards compatible and supported by pretty much all browsers) is
<meta charset="utf-8">
That said, some tools compiling static HTML from markdown etc. might have problems with it, but that depends on the tool. You're safer using the entities like © there; which you can also always use without specifying an encoding.
This is not the full story, as the way a browser is decoding a file can also be influenced by other factors, like HTTP Response Headers. Also, even if you omit it, as you could observe, browsers do everything they can to still parse it correctly, there's just no guarantee.
I printed some UTF-16 encoded characters and tried to display it in Firefox and it displayed it as �.
So I went to Tools->Encoding and changed the encoding from UTF-8 to UTF-16 (I also tried changing charset directly in the HTML) However, when I did that, my page was completely flooded with symbols:
ℼ佄呃偙⁅瑨汭ാ㰊瑨汭ാഊ㰊敨摡ാ †ഠ †㰠楴汴㹥楬畮⁸楆敲潦⁸楤灳慬獹朠牡慢敧挠慨慲瑣牥湩氠敩⁵景眠扥 瀠条畓数獕牥⼼楴汴㹥††氼湩敲㵬猢潨瑲畣⁴捩湯•牨晥∽瑨灴⼺振湤献瑳瑡捩渮瑥猯灵牥獵牥椯杭是癡捩湯椮潣㸢††氼湩敲㵬愢灰敬琭畯档椭潣≮栠敲㵦栢瑴㩰⼯摣獳慴楴敮............
How can web browsers display UTF-16 characters without wrecking the page?
The “flooded with symbols” excerpt looks like an HTML document that is UTF-8 encoded but treated as if it were UTF-16 encoded. Or it might contain mostly UTF-8 data with some UTF-16 encoded data thrown in, which won’t work.
If you save your data as properly UTF-16 encoded and declare the encoding in HTTP headers and/or meta tags, then some browsers will display it OK, some won’t. Search engines generally fail to process UTF-16, and UTF-16 is mostly not used and should not be used on the web, except by mutual agreement between consenting well-informed partners.
Firefox could not figure the correct charset in your document.
For web pages head meta tag should be used to indicate the content's charset.
It should be placed in the beginning of the HTML file indicating which charset the browser should use for the rest of the file.
<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
So the browser is charset blind until it reads that line. But using utf-8 is no problem. Because every character up to that point is encoded in utf-8 the same way it would be in ASCII (same goes for latin-1 and others). That's not the case in utf-16.
W3C says:
There are three different Unicode character encodings: UTF-8, UTF-16
and UTF-32. Of these three, only UTF-8 should be used for Web content.
So you should use utf-8. But if you still want to try something with utf-16 use the BOM in the begging of your file. You're going to give your browser a better chance of figuring it out and properly decode the content.
This other answer is very succinct about utf-16 usage.
While Joel gives a full lesson on character encoding and why HTML uses it declaration inside the content and not as a header information.
Sending UTF-16 data as a Web page to browsers is an XSS risk in older browsers. (See another answer.) Don’t do it. Instead, convert the data to UTF-8 on the server and send UTF-8 over HTTP.
The way to make this work is for the page to say what encoding it's in. In the case of UTF-16, it also helps to include a BOM. The "flooded with Chinese" effect is most likely because your page is UTF-16LE but the browser treated it as UTF-16BE or vice versa...
I found a website that contains the string "don’t". The obvious intent was the word "don't". I looked at the source expecting to see some character references, but didn't (it just shows the literal string "don’t". A Google search yielded nothing (expect lots of other sites that have the same problem!). Can anyone explain what's happening here?
Edit: Here's the meta tag that was used:
<meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1">
Would this not cause the page to be served up as Latin-1 in the HTTP header?
In your browser, switch the page encoding to "UTF-8". You're seeing a right single quote character, which is encoded by the octets 0xE2 0x80 0x99 in UTF-8. In your charset, windows-1252, those 3 octets render as "’". The page should be explicitly specifying UTF-8 as its charset either in the HTTP headers or in an HTML <meta> tag, but it probably isn't.
According to Character encondings in HTML a lemme in wikipedia:
HTML (Hypertext Markup Language) has
been in use since 1991, but HTML 4.0
(December 1997) was the first
standardized version where
international characters were given
reasonably complete treatment. When an
HTML document includes special
characters outside the range of
seven-bit ASCII two goals are worth
considering: the information's
integrity, and universal browser
display.
I suppose the site you checked, isn't impelemented with this in mind.
This has all got to do with encoding. Take a look back at the source, is there a tag at the top specifying it (charset)? My guess is it'll be UTF8 - although it could be something completely different.
This thread explains all. A combination of using a weird UTF-8 apostrophe character (probably originating from a Word Document), on a server that probably reports its encoding as non-UTF-8, despite the page having UTF characters (and possible even correctly reporting its own encoding).