In trying to scrape some data from the website of a housing agency in China (the name of the agency is Anjuke) to gather data for a small personal project I realized that all of the numbers on the website are visually displayed as numbers, but are digitally read as obscure Chinese characters.
Is there a name for this kind of a font or this kind of a technique more specific than "anti-scraping measures"?
Additional information about this specific case:To see this in action you can click on any of the listings from the Anjuke website, and then attempt to copy-and-paste the price (or any HTML element that has the "strongbox" class), and you will see that instead of pasting the number is pastes and obscure Chinese character (such as 驋, 齤, 麣, 龤, or 龒).
Looking at the CSS revealed that these numbers have a font of "fangchan-secret", and a bit of quick googling linked to a blog post in Chinese by zhyuzh3d. I read some Chinese, although not loads. This blog post appears to be a Chinese explanation of how how fangchan-secret is a method to prevent to prevent webscraping, and also an explanation of how to get around around this preventative measure.
I have looked into the previous postings to do with this area but haven't found any relevant answers as perhaps I am asking the wrong question.
On the popular design site Dribbble, there seem to be interesting formatting changes in profile names that break from the conventions of the site's styling.
Alot of people have been adding special characters (ΔδΓ etc.) that can be achieved by pasting into their profile form and saving changes, yet some users have somehow managed to enter formatted versions of their name, despite the profile form not supporting HTML or Markdown. You can see an example in the images below.
An example of copying the font to Google with maintained formatting
When opening in inspector, it also shows the formatted type
How could this be done in a simple text input form that doesn't support HTML/Markdown?
These are almost certainly Unicode characters, just like these characters that you reference in your question: ΔδΓ.
For example, Unicode's mathematical alphanumeric symbols section includes symbols that look like the ones in your screenshot. Since these are separate Unicode characters there is no need for additional formatting.
Users will need to have a font that supports those characters installed locally to view them.
I'm wondering if it's a bad idea to use weird characters in my code. I recently tried using them to create little dots to indicate which slide you're on and to change slides easily:
There are tons of these types of characters, and it seems like they could be used in place of icons/images in many cases, they are style-able and scale-able, and screen readers would be able to make sense of them.
But, I don't see anyone doing this, and I've got a feeling this is a bad idea, I just can't decide why. I guess it seems too easy to be true. Could someone tell me why this is or isn't okay? Here are some more examples of the characters i'm talking about:
↖ ↗ ↙ ↘ ㊣ ◎ ○ ● ⊕ ⊙ ○ △ ▲ ☆ ★ ◇ ◆ ■ □ ▽ ▼ § ￥ 〒 ￠ ￡ ※ ♀ ♂ &⁂ ℡ ↂ░ ▣ ▤ ▥ ▦ ▧ ✐✌✍✡✓✔✕✖ ♂ ♀ ♥ ♡ ☜ ☞ ☎ ☏ ⊙ ◎ ☺ ☻ ► ◄ ▧ ▨ ♨ ◐ ◑ ↔ ↕ ♥ ♡ ▪ ▫ ☼ ♦ ▀ ▄ █ ▌ ▐ ░ ▒ ▬ ♦ ◊
PS: I would also welcome general information about these characters, what they're called and stuff (ASCII, Unicode)?
There are three things to deal with:
1. As characters in a sentence/text:
The problem is that some fonts simply do not have them. However since CSS can control font use you probably will not run into this problem. As long as you use a web safe font, and know that that character is available in that font, you should probably be okay.
You can also use an embedded font, though be sure to fall back on a web safe font that contains the character you need as many browser will not support embedded fonts.
However sometimes certain devices will not have multiple fonts to choose from. If that font does not support your character you will run into problems. However depending on what your site does and the audience you are targeting this may not be a problem for you. Not to mention that devices like that are very old, and uncommon.
All in all it was probably not a good idea a handful of years ago, but now you are not likely to have problems as long as you cover all your bases.
It is important however to point out that you should never hard code those characters, instead use HTML entities. Just inserting those characters into your code can lead to unpredictable results. I recently copied some text from Word directly into my code, Word used smart quotes (quote marks that curve inwards properly). They showed up fine in Notepad++, but when I viewed the page I did not get quotes, I got some weird symbol.
I could have either replaced them with normal quotes " or with HTML entities to keep the style “ and ” (“ and ”).
Any Unicode character can be inserted this way (even those without special names).
Wikipedia has a good reference:
http://en.wikipedia.org/wiki/List_of_XML_and_HTML_character_entity_references
2. As UI elements:
While it may be safe to use them in many cases, it is still better to use HTML elements where possible. You could simply style some div elements to be round and filled/not filled for your example.
As far as design goes they are really limiting, finding one that fits with the style of your page can be a hassle, and may mean that you will definitely need to embed a font, which is still only supported by the latest browsers.
Plus many devices do not support heavy font manipulation, and will often display them poorly. It works in the flow of your text, but as a vital part of the UI there can be major problems. Any possible issue one of those characters can bring will be multiplied by the fact that it is part of your UI.
From an artistic stand point they simply limit your abilities too much.
3. What are you doing?
Finaly you need to consider this:
Text is for telling
Image is for showing
HTML is for organizing
CSS is for making things look good while you show them
JavaScript is for functionality
Those characters are text, they are for telling someone something. So ask the question: "What am I doing?" and then use what was designed for that task. If you are telling use them, if you are showing use Image, or CSS.
I've seen this done before (the stars) and I think it's an awesome idea! It's also becoming quite popular to use a font (with #font-face) full of icons, like this one: http://fortawesome.github.com/Font-Awesome/
I can't see any downside to using a font like "font awesome" (only the upsides you mention like scalabilty and the ability to change color with CSS). Perhaps there's a downside to using the special characters you mention but none that I know of.
The problem with using those characters is that not all of them are available in all fonts used by all users, which means your application may look strange, or in the worst case be unusable. That said, it is becoming more common to assume the characters available in certain common fonts (Apple/Microsoft's Arial, Bitstream Vera). You can't even assume that you can download a font, as some users may capture content for offline reading with a service like Instapaper or Read It Later.
There are a number of problems:
Portability: using anything other than the 7-bit ASCII characters in code can make your code less portable, as recipients may use the wrong encoding. You can do a lot to mitigate this (eg. use UTF16 or at least UTF-8 encoded files). Most languages allow you to specify strings in characters using some form of escape notation (eg. "\u1234" in C#), which will avoid the problem, but loses some of the advantages.
Font-dependency: user interface elements that depend on special characters being available in a font may be harder to internationalize, since those glyphs might not be in the font that you want/need to use for a particular audience.
No color, limited choice of art: while font glyphs might seem useful to a coder, they probably look pretty poor to a UI designer.
The question is very broad; it could be split to literally thousands of questions of the type “why shouldn’t I use character ... in HTML documents?” This seems to be what the question is about—not really about code. And it’s about characters, seen as “weird” or “uncommon” or “special” from some perspective, not about character encodings. (None of the characters mentioned are encoded in ASCII. Some are encoded in ISO-8895-1. All are encoded in Unicode.)
The characters are used in HTML documents. There is no general reason against not using them, but loads of specific reasons why some specific characters might not be the best approach in a specific situation.
For example, the “little dots” you mention in your example (probably not dots at all but circles or bullets), when used as control elements as you describe, would mean poor usability and poor accessibility. Making them significantly larger would improve the situation, but this more or less proves that such text characters are not suitable for controls.
Screen readers could make sense of special characters if they used a database of various properties of characters. Well, they don’t, and they often fail to read properly even the most common special characters. Just reading the Unicode name of a character can be cryptic or outright misleading. The proper reading would generally depend on meaning and context.
The main issue, however, is that people do not generally recognize characters in the meanings that you would assign to them. How many people know what the circled plus symbol “⊕” stands for? Maybe 1 out of 1,000, optimistically thinking. It might be all right to use in on a page about advanced mathematics or physics, especially if the notation is defined there. But used in general text, it would be just… a weird character, and people would read different meanings into it, or just get puzzled.
So using special characters just because they look cool isn’t a good idea. Even when there is time and place for a special character, there are technical issues with them. How many fonts do you expect to contain “⊕”? How many of those fonts do you expect Joe Q. Public to have in his computer? In this specific case, you would find the font coverage reasonably good, but you would still have to analyze it and write a longish list of font names in your CSS code to cover most platforms. In the pile of poo case (♨), it would be unrealistic to expect most people to see anything but a symbol for unrepresentable character. Regarding the methods of finding out such things, check out my Guide to using special characters in HTML.
I've run into problems using unusual characters: the tools editor, compiler, interpreter etc.) often complain and report errors. In the end, it wasn't worth the hassle. Darn western hegemony, or homogeneity, or, well, something!
I have been trying to validate my web page for the last two hours, I only have one error remaining before it is successfully validated but I keep on getting the character decoding problem, I cannot get round it.....
The whole document is fine except it says...
Sorry, I am unable to validate this document because on line 77 it contained one or more bytes that I cannot interpret as utf-8 (in other words, the bytes found are not valid values in the specified Character Encoding). Please check both the content of the file and the character encoding indication.
The error was: utf8 "\x85" does not map to Unicode
The only thing on line 77 is some text inside some <p> tags, I have tried changing them to <a>, or <span> and taking the <p> away so it is just loose inside the div but the error only goes away when I delete the text inside the tags.
I am using the utf-8 encoding:
<meta http-equiv="Content-type" content="text/html;charset=utf-8" />
I am sorry if this is simple to resolve, my knowledge is extremely basic, I am only a first year computing student.
EDIT: the text inside the <p> tags are as follows:
<p>Our team thrives on the latest
political news as we do you. We work
around the clock to bring you the
latest, most important news as soon as
it happens. What do we ask in return…
nothing! This site is funded by us!
Your satisfaction is as much a pay
packet to us then a wad of untraceable
counterfeit notes.<br/><br/>
Sign up
to our newsletter to get regular
updates on news as soon as it happens
without having to navigate to our
site. For your security we only sell
the details you input to our site to
companies who “pinky promise” they
won’t be naughty with
them.<br/><br/>
StudentPolitics.Now
– Trading in satisfying others since
2011</p>
The problem is that the document only claims to be UTF-8 but isn't really.
Configure your editor to save in that format (the W3C has a guide for a number of them).
If you modify the HTML programatically, then check the program (and/or database if one is in play) aren't munging the data or storing non-UTF-8 data.
If that doesn't work, then try deleting the text and retyping it. You might have a zero width character that can't be represented properly in there.
Save your document in a UTF format. If it already is, try copy-paste the source code to a new file and save it in UTF format (sometimes it can get stuck during edits in some programs).
What editor are you using?
EDIT: There are some non-standard characters in your text: … (three dots in a single character, “” (curly braces), ’ (curly apostrophe), – (dash).
I guess you've copied your text from Word or a similar text processor, I get that often too. Either change those characters to their ASCII counterparts or HTML entities or be sure to save the file with UTF encoding.
validator did not like the three full stops after the word "return" three full stops after one another must mean something else...
Thank you for all your help guys.
Sometimes when you generate query from a database the encoding of the characters may not be UTF-8 in that case you should make sure that the values returned in the queries match UTF-8, also sometimes when making a substring you can cut a character in Spanish as tildes and las ñ and to show incomplete the character.
For example check the source code in your browser
I would like to print some kind of ASCII "art" on a web page in pre-tags. These graphics use DOS characters to show a map like old maze games did. I didn't find anything in the HTML special character reference. Is there a way to use these characters in HTML ?
Thanks in advance.
With the right Unicode characters, the old character encodings shouldn't make much odds. The tricky bit may be converting existing ASCII art into Unicode - at which point you need to know the original encoding.
The relevant code charts will be listed on the Unicode "symbols" charts page. In particular, I suspect you'll find the box drawing and block elements charts useful.
You'll need to make sure that your page uses a font which contains the right characters, of course...
As an example, you can render this:
┌┐
└┘
With:
<pre>┌┐
└┘</pre>
Not quite a proper box, but getting there...
You can send them in the <pre> tags, although in XHTML you'll need to encapsulate it in <![CDATA[[]> I think. Be careful though, not all encodings render this correctly. For example, a lot of ASCII art designed for DOS code page 430 (US) fails over here in the UK (830). Eastern Europe suffers especially.
I think the best approach here would be to render images.
EDIT: Oh. You could try , but I'm not sure if that would work.