Articles in wikipedia get edited. They can grow/shrink/updated etc. What file system/database storage layout etc is used underneath to support it. In database course, I had read a bit on variable length record, but that seemed like more for small strings and not for whole document. Like in file system, files can grow/shrink etc, and I think its done by chaining blocks together. each time, we update a file, not the whole file is rewritten. Perhaps something similar would be done here.
I am looking for specific names,terminologies, may be even how the schema in mysql is defined. (I think wikipedia uses mysql).
Below are links to some writeup on wikipedia architecture, but I am not being able to answer my question from these:
http://swe.web.cs.unibo.it/twiki/pub/WikiFactory/AntonelloDiMuroThesis/Wikipedia-cheapandexplosivescalingwithLAMP.pdf
http://dom.as/uc/workbook2007.pdf
Thanks,
See:
http://www.mediawiki.org/wiki/Manual:Database_layout
I develop an online, Flash-based multiplayer game. It is a complex game, and requires a lot of documentation to fully explain it to our users. Ideally, I would like to find MySQL-based wiki software that can provide these editable documentation pages outside of Flash (in the HTML realm) but also within Flash for convenience, and so that players can refer to the information without interrupting their game or having to switch back-and-forth between browser tabs. I am expecting that I would need to do a lot of the work on the Flash side myself, as far as formatting, for example, but I would like to feel comfortable in querying the wiki's database to get info directly. I guess this means that I need a wiki that is structured relatively "flat" or intuitively so that I can do things like:
Run a MySQL query that returns a list of all the articles (their titles and IDs) in the wiki
For each article ID in the wiki, return the associated content
This may mean that I have to limit the kinds of formatting I put into the wiki -- things like tables would probably be omitted since they would be very difficult, if not impossible, for me to do on the Flash side. And that is fine!
Basically I am just looking for suggestions for wiki software that is pretty easy to use, but mostly is technically simple enough on the back-end that interfacing with it directly via MySQL is not difficult. When interfacing with the database directly, I only need to READ data. Any time the wiki would be edited or added to would be done via the wiki's actual front-end application.
Thanks for any suggestions!
MediaWiki is the best-known and best-supported MySQL-based Wiki, used for plenty of complex game documentation projects like MinecraftWiki. The database is not all that simple, but it's well documented and basic read operations aren't too hard. For example, here's how to fetch the current content of the page "MyPage":
SELECT old_text FROM page,revision,text WHERE page.page_title="MyPage" AND
page.page_id=revision.rev_page AND revision.rev_text_id=text.old_id;
(And yes, old_text is the current content of the page. Don't ask me why!)
Your main problem will be figuring out how to parse MediaWiki markup, there are plenty of parsers for it but I'm not aware of anything that would work in Flash.
Can anybody suggest a way to process the information and analyze the data from the comments users post on a article in my website.
I exactly want to process the comments as follows:
Example: Like on a article on computerization may get the following comments:
I love computerization as it makes the work easier.
Computerization is spreading unemployment as 1 computer can work better than 4 people.
How I process this information -
: I take the comments and try to recognize some predefined[and extensible] keywords in it.
Assuming that you are trying to extract some useful information from the comments, you could apply some machine learning to the comments to classify or categorize the data contained within, the sentiments etc.
There are number of different types of learning you can do on the text, however I personally recommend using support vector machines or a naive bayes classifier to be able to categorize and analyze the comments. You could also possibly use clustering, but there needs to be an element of natural language processing in the solution you choose. There are number of different libraries that you can use to implement the code to use either, i.e. svmlight, javaml, etc. I have personally used javaml and it is a good library.
As a student in computer engineering I have been pressured to type up very detailed comments for everything I do. I can see this being very useful for group projects or in the work place but when you work on your own projects do you spend as much time commenting?
As a personal project I am working on grows more and more complicated I sometimes feel as though I should be commenting more but I also feel as though it's a waste of time since I will probably be the only one working on it. Is it worth the time and cluttered code?
Thoughts?
EDIT: This has given me a lot to think about. Thanks for all your input! I never expected this large of a response.
Well considered comments illuminate when the code cannot. Well considered function and variable names eliminate the need for copious comments. If you find it necessary to comment everything, consider simplifying your code.
If you ever look at code you wrote 6 months before, you will be wondering why you did not comment better.
If the code is well written, with short methods (see Composed Method pattern) and meaningful names, then the code needs very little comments. Only comments which explain the "why" are useful - the comments explaining "what" should be replaced by improving the code so much that it's obvious what it does. Comments should not be used as an excuse for writing bad code.
Public APIs, especially in closed-source apps, are perhaps the only place where thorough javadocs are recommended - and then you need to take the effort to maintain them and keep them always accurate and up-to-date. A misleading or outdated comment is worse than no comment. It's better to document what the code does by writing tests for it and using good names for the tests.
The book Clean Code has a good chapter about comments.
Unit tests and the like are the best forms of code documentation. Some testing frameworks write out a spec of what the class under test should do, giving people a great introduction to how a piece of code works in pure english while also providing very clean way to implement the tests itself.
Examples of that are Scala's ScalaTest or RSpec for Ruby.
I find that unless some weird hacky thing is required by the code in question, it is usually not beneficial to comment it. Also, it adds a lot of overhead because you have to maintain the comments... and maintaining the code and tests is already enough work.
Remember, code with out-of-date comments is worse than no comments at all!
A lot of the time, comments just says what the code does anyway, which is a waste of human effort. And if it doesn't, your code probably sucks and you should refactor it.
Just use testing frameworks.
Comment -- or better yet, recode -- anything that is non-obvious now. Later it will be completely non-obvious. You might think, "but it's going to be me," but if your way of thinking (and ways of coding) changes as you grow what's obvious to you now might not be obvious to you later.
Have you read Code Complete yet? Recommended as a very good read, and a great way to figure out some of the things CS profs drill down your throat.
Code comments come in two variety:
Comments to explain logic, making
sure that the code matches the
intent. Often people will write high
level pseudocode and will use that
in comment form to fill in the
actual code of what the module will
do. Then they leave the comments as
a read-along which can be used
during later review.
Comments to
explain usage of a module. Think
javadocs. Your intent here is for
the consumers to understand why your
code is important. One use of
javadocs is in the Visual Studio
Intellisense (since I don't use
Eclipse idk). It shows the comments
from the javadoc in the intellisense
hover. Becomes VERY handy later on.
When professors ask you to document everything in your code, I have found the usage of psuedocode translated to actual code to be sufficient. However, in practice I've not found that many devs need it, as usually the code is sufficient to explain itself (when simple, well written, not relying on tricks, and when using descriptive variable names).
But I still put in comments about intent if I think it's the least bit unclear. This is just a bit of best practice.
But I definitely say read that book. ;)
If you ever decide to open-source your personal project, people will thank you for your comments (unless they're terrible). If you hit upon a spectacularly great idea and your personal project turns into a business, then you'll be hiring more developers, and again your comments will be valuable. If you suffer a mild head injury, then when you return to work you'll be thankful for your comments.
Some people treat comments as a code smell, a sign that the code could use more descriptive names and a better structure. They will fix the code so it does not need comments.
This works in a lot of cases. However one type of comment that is useful is 'why' something is being done. Sometimes fixes are made for obscure reasons that would not be obvious when reviewing the code later. The comments should not express what the code does (that should be covered by naming) or how it does that (again, the code tells you that), so save your comments for 'why'.
I find that nothing serves as better documentation as to how something works then unit tests.
Whenever i'm doing something that isn't self-documenting, i'll put a comment. I will forget what i was doing unless i do. But i prefer to write code that's so obvious that comments don't help much. Wherever possible, the code should be clear enough that thousands of lines of comments would be unnecessary.
Whatever you do, do NOT write comments like this...
// add 1 to i
++i;
That's noise. You're actually worse off with comments like that than with none at all.
A hard-core stance is: "if you have to write a comment for your code, your code is broken". Rather than writing explanatory comments, refactor your code so that the comments become less necessary. This applies especially to function names (including their parameters), since they tend to be modified the most, and the comments seldom are updated to match.
Instead of:
// Compute average for the two times
int a = t1 + (t2 - t1) / 2;
write
int averageTime = AverageOfTimes(t1, t2);
int AverageOfTimes(int t1, int t2) {
return t1 + (t2-t1);
}
Stale comments are one of the leading causes of WTF's when I'm reading other people's code.
Overcommenting has been cited as a "code smell" by several authors, including the authors of "Clean Code".
Personally, I write an explanatory comment for each class (I code in C# and C++ mostly), and occasionally when I am using an algorithm I want to refer to.
To be honest, if code is clear its not necessary, but comments are best when a specific logic breaks given certain data (which may not be obvious). Leaving comments of issues that may occur is a great way to help prevent accidental bugs due to misunderstandings of what data to expect (or specifically not).
I used to be in the exact same situation as you. When I first started I never commented anything because everything was extremely small and I always knew how it worked. But as I continued to expand my code and everything started pointing to each other, I found myself not knowing what certain things did anymore and got lost. I had to rewrite a lot of things so I knew what they did again, and I started commenting everything so I knew exactly how it worked and how to use it in the future. You may think you know how everything works now, but in the future you'll look back at something and say 'HUH?' It's better to comment things now and save yourself the trouble later.
The way I comment things:
Always add this at the top of any function, so you know what it does.
/**
* What the function is supposed to do, a brief description of it.
*
* #param paramter_name Object-type A description of it, do for each parameter.
*
* #return Object-type - A brief description of what is being returned.
**/
Then throughout your code make sure you comment things that look complicated. When you run checks, put a quick comment like 'make sure this is valid'. For any long lines of code or large blocks of code, add a comment of what that specific section does, to easily find it later on.
Commenting is useful for individual project as well as group projects especially when you will need to maintain or enhance the code over an extended period of time. This scenario may not be applicable for school projects, but in the workplace it is definitely applicable. If you have ever had to look at code that you wrote 6 months in the past then it might as well have been written by somebody else.
I have found that--as everyone will tell you--if you are coming back to code after several months you will have forgotten everything. That said, I hardly comment my own code except for comments like // ew.. hacked because X-beyond-my-control doesn't do Y correctly. But this is because the code itself is written very cleanly and is very easy to read. For instance, all variable and function names are completely descriptive. I don't use abbreviations except for rather long words which are obvious such as 'info'. All functions are extremely brief. All functions do one thing if possible. Nesting is avoided if possible. Logically related functions are grouped via classes or modules.
When I read other people's code, I don't read the comments. I read the code. And the difference between clear and well written code and spaghetti is far more important than any comments. Usually I don't care what their intent is/was; I want to change the code. And to do that it easily it needs to be well organized. So the comments are peripheral. But perhaps this is just me.
Technically speaking the code is perfectly self documenting. I like to comment anything that is non-obvious or especially complex. I tend to like to have at minimum summary on a class, and maybe a short blurb for each member. When you have to write a huge amount of doc on a very large and complex method that is usually a good sign that it needs to be looked at for a refactor.
foreach(var a in b.Items) //Enumerate the items in "b"
{
if(a.FirstName == "Jones") //See if first name is Jones
....
You want something in the middle of the above and no commenting whatsoever.
commenting is always useful!And I dont thik that commenting is a waste of time!It helps other developeres understand your code and it helps you when you haven't work on a project for months.
University software courses often push people towards excessive commenting as a way of forcing students to think twice about what they have typed. A nasty rule-of-thumb that was put my way when I was marking undergrad coursework a few years ago suggested a comment every 5-10 lines. Think most Java courses tell you to limit methods to circa 30 lines, so lots of comments within a function is a sign you should break it up.
My personal preference is to limit comments to documenting function purpose/inputs/outputs and data structures. Otherwise comments ought to be limited to things that require particular attention (eg things that looks out of place, perhaps bugs, at first glance).
The first comments are the variables and methods names. A lot of attention shall be paid to choose them. Do not hesitate to rename them if you can think of a better name.
Consistency and convention also help. Try to avoid NumberOfStuff, ThingCount, FooSize, always use the same form, possibly something in line with the language (does it have Array.length, or Vector.size). Always name similar things with similar names.
"Code never lies. Comments sometimes do". One day or the other, someone will modify the code and not the associated comment. Better spenf more time writing self-explanatory code complemented with some clever comment, than splitting out code and then explaining things with a lot of comments.
Rule #1
Comments should indicate WHY not 'what' (the code already tells you 'what' is happening)
Rule #2
After writing the comment - rewrite your code to read like the comment (then remove the comment)
While good variable and function names etc. may lessen the need for comments, any sufficiently complex program is going to be hard to understand merely by looking at the code alone, no matter how carefully it was written.
In general, the larger the body of code and the longer you expect it to be in use, the more important it will be to write detailed comments. It is perfectly reasonable for teachers to demand detailed comments because they don't want to have to spend a lot of time trying to understand large amounts of code from different students, but you don't necessarily have to maintain that same standard outside the classroom.
Regarding commenting functions & classes and the like, I find that only the most trivial functions are so self explanatory that a comment won't save the reader some time. Also, when you are using a smart IDE for a language such as Java or Curl writing properly formatted documentation comments will allow developers to see documentation for functions and methods simply by hovering over a call signature. This can save you a lot of time when working with large systems.
When screen-scraping, what are the "gotcha"s to look out for?
The inspiration for this is: my spouse's co-worker asked me to scrape all the pages from a Blogger-hosted blog that her friend with cancer kept in her final months and this lady wanted to keep all of the posts in case the blog were ever deleted. I eventually found a free tool that was barely good enough.
One issue with scraping many Blogger pages is that there's often a navigation menu where you can click on the triangles to expand the post lists by year or month. These little buggers created insane amounts of duplicate content because you'd have the same page over and over again with different combinations of the menus being expanded/collapsed. In Blogger's case I'm not sure this is avoidable since the links are all formatted as real http links and not obvious JavaScript calls. Still, it got me thinking:
If you were to scrape a website, what kinds of potentially non-obvious things would you compensate for?
Do not use regex to scrape
While regular expressions can be good for a large variety of tasks, I find it usually falls short when parsing HTML DOM. The problem with HTML is that the structure of your document is so variable that it is hard to accurately (and by accurately I mean 100% success rate with no false positive) extract a tag.
What I recommend you do is use a DOM parser such as BeautifulSoup or equivalent (SimpleHTMLDom in PHP).
Some may think this is overkill, but in the end, it will be easier to maintain and also allows for more extensibility.
A regular expression could be devised to achieve the same goal but would be limited. For example, developing a regex to get the src and alt tag would force the alt attribute to be after the src or the opposite, and to overcome this limitation would add more complexity to the regular expression.
Also, consider the following. To properly match an <img> tag using regular expressions and to get only the src attribute (captured in group 2), you need the following regular expression:
<\s*?img\s+?[^>]*?\s*?src\s*?=\s*?(["'])((\\?+.)*?)\1[^>]*?>
And then again, the above can fail if:
The attribute or tag name is in capital and the i modifier is not used.
Quotes are not used around the src attribute.
Another attribute then src uses the > character somewhere in their value.
Some other reason I have not foreseen.
So again, simply don't use regular expressions to parse a dom document.
I screen scrape a lot. Some advice:
Emulate a User-Agent string for some browser you want to use. Different websites frequently return very different results depending on what your user agent is. If they don't recognize the User-Agent they will often revert to lowest common denominator, so it's usually best to start with some recent browser. (For example the World of Warcraft Armory returns beautiful, easy to parse XML if it thinks you're a recent Firefox. If it doesn't know what you are it sends terrible HTML).
Be polite to the site you're scraping; don't hit it too hard. Your scraper will go faster if you multi-thread it, making many requests at once, but that will annoy the site owner.
Be smart about error handling. Do not write code like while (1) { makeRequest(); }. If your code or the server throws an error a loop like this will immediately fetch another request, generating another error. It can get ugly quickly. Handle errors well and consider putting in sleeps or exits if you see a lot of errors.
When developing your parsing code, test against a cached version rather than hitting the server every time. Will make your development go faster and is the basis of a simple test suite.
First, I'd check for an RSS feed. On blogger, you just have to add /rss to the root url, if I remember correctly.
Then I'd check if there isn't already some tool to scrape blogger.
Then if there's no RSS feed, and no existing tool, I'd give up and do it by hand with copy/paste. Unless we're talking 5000 pages, it's much faster and easier that way. Take it from someone who's tried.
If you have access to the actual account, blogger has an export function.
edit: Or of course, you could try mechanical turk.
As far as gotchas are concerned..It's usually a good idea to limit the amount of requests made over a certain period of time. Smashing a site with alot of requests in a short space of time is a good way to have your requests rejected.
Aside from the technical considerations, make sure your not putting yourself at legal risk. Most large sites have specific legal language in their terms of use that disallows programmatic access to their services via an automated computer program, and also, the obvious copyright concerns.
From a technical standpoint, definitely use a DOM parser library and you'll save loads of time. Many provide the ability to read HTML into an XML structure that can be queried using XPath to find exactly what you need.
If you know someone who has access to the account, they can use Blogger's export "Export blog" feature.