Google has described a novel framework for distributed processing on Massive Graphs.
http://portal.acm.org/citation.cfm?id=1582716.1582723
I wanted to know if similar to Hadoop (Map-Reduce) are there any open source implementations of this framework?
I am actually in process of writing a Pseudo distributed one using python and multiprocessing module and thus wanted to know if someone else has also tried implementing it.
Since public information about this framework is extremely scarce. (A link above and a blog post at Google Research)
Apache Giraph http://giraph.apache.org
Phoebus https://github.com/xslogic/phoebus
Bagel https://github.com/mesos/spark/pull/48
Hama http://hama.apache.org/
Signal-Collect http://code.google.com/p/signal-collect/
HipG http://www.cs.vu.nl/~ekr/hipg/
The main Hadoop project for distributed graph processing is the Hama project. Its still in incubation though.
The project has broken its work into two areas; a matrix package and a graph package.
Update:
A better option would be the Apache Giraph project which is based on Google Pregel.
Yes, a new project called Golden Orb, which is an open-source Pregel implementation written in Java that runs on both HBASE and Cassandra.
It has been submitted to Apache incubator for approval, and Ravel, the company behind Golden Orb, said they are releasing it this month (http://www.raveldata.com/goldenorb/).
See http://www.quora.com/Graph-Databases/What-open-source-graph-databases-support-horizontal-scaling
UPDATE: GraphX is GraphLab2 on Spark implemented by Joey Gonzalez, the creator of GraphLab2.
Spark's unique primitives make GraphX-Pregel the fastest JVM-based Pregel implementation. Spark is written in Scala, but Spark has a Java and Python API.
See...
GraphX: A Resilient Distributed Graph System on Spark (PDF)
Introduction to GraphX, by Joseph Gonzalez, Reynold Xin - UC Berkeley AmpLab 2013 (YouTube)
My Hacker News comment/overview on Spark.
P.S. There is also Bagel, which was the first cut at Pregel on Spark. It works; however, GraphX will be the way forward.
Two projects from Carnegie Mellon University provide Pregel-style computation
on graphs:
GraphLab http://graphlab.org
GraphChi http://graphchi.org
The programming model is not exactly same as Pregel, as they are not based on messaging but on modifying the graph (edge, vertex) data directly. Basically, it is easy to emulate Pregel in these framework.
There is also Signal/Collect a framework written in Scala and now using Akka
http://code.google.com/p/signal-collect/
https://github.com/uzh/signal-collect
From their website:
In Signal/Collect an algorithm is written from the perspective of vertices and edges. Once a graph has been specified the edges will signal and the vertices will collect. When an edge signals it computes a message based on the state of its source vertex. This message is then sent along the edge to the target vertex of the edge. When a vertex collects it uses the received messages to update its state. These operations happen in parallel all over the graph until all messages have been collected and all vertex states have converged.
Many algorithms have very simple and elegant implementations in Signal/Collect. You find more information about the programming model and features in the project wiki. Please take the time to explore some of the example algorithms below.
I create a framework called Phoebus. It is an implementation of Pregel written in Erlang. Checkout my blog entry for applying Pregel model to path finding as well..
Apache Giraph is currently in Incubator and under very active development, with committers from LinkedIn, Twitter, Facebook and academia looking to bring it up to production scale very quickly. It is pretty directly modeled on Pregel and was originally developed at Yahoo! Research. We're looking for new contributors and have several introductory JIRA issues to help people get started with the project. We'd love to have you get involved.
Stanford Students have developed an open Source implementation of Pregel.
http://infolab.stanford.edu/gps/
As the question states, is there any point adding Dublin Core meta-tags to your HTML head? Or has sitemap.org removed the use for most of this (though it only replaces some of the tags)
I ask this as most sites I visit don't seem to use DC metatags in their source.
I'm interested in whether I need them for a site that will be used mostly for developers, however the discussion can be broader than this category.
To quote Google (from 2002):
"Currently we don't trust metadata because we are afraid of it being manipulated"
I would rather say that the time of rich metadata hasn’t come yet. In fact technologies like RFD are just on the way up. Tim Berners-Lee – you know, the guy who invented the web – quite recently spoke at TED about The next Web of open, linked data. So Dublin Core and other metadata formats are anything but out.
Dublin Core is still very important in certain industry sectors. Here in the UK, government organisations use DC to provide standardised access to tags.
META tags are not the only place you can put DC metadata. You can integrate it more with HTML using RDFa.
Now, as for proliferation — well, the only incentive it currently gives to webmasters is satisfaction for job well done, but does not yet affect SEO. As soon as this changes, you'll see outburst of sites tagged with RDF and microformats. And it will come. Yahoo already started working on that: http://ysearchblog.com/2008/03/13/the-yahoo-search-open-ecosystem/
I was looking on the web for information about the Dublin Core and if search engines used them and I came across the academic paper "Search Engines and Resource Discovery on the Web:
Is Dublin Core an Impact Factor?" by Mehdi Safari:
http://www.webology.ir/2005/v2n2/a13.html
To quote his conclusions section: "it was found that using Dublin Core elements did not improve the retrieval rank of the web pages" and that "Dublin Core metadata, as a well-known metadata schema, is not widely accepted and used by search engine designers and the spiders do not consider its elements while ranking the web pages".
This was back in 2005, but I am assuming this is still true.
Semantic web efforts are still sputtering along. I've run across a couple of research efforts to use RDF triples including the Dublin Core... but nothing close to commercialization.
However, as a general organizing principle for the world wild web? Don't bother. My guess is that folksonomies will deal with some metadata management, but that site tagging will need to be handled through ontological deduction of some sort. I get the same feeling around DC and RDF that I get around general-purpose globally open UDDI registries: nice idea, but that's not the way the world works.
It would be kinda interesting to know whether DC tags increase your Google Page Rank (and how reliably): that could be a strong incitament for many!
Closed. This question is opinion-based. It is not currently accepting answers.
Want to improve this question? Update the question so it can be answered with facts and citations by editing this post.
Closed 7 years ago.
Improve this question
A little backround:I'm a 22 yr old with just a high school degree and a lot of free time (college did not work out). I am completely new to web programming and I have taken a couple day classes in Photoshop, Illustrator, Dreamweaver, Fireworks, and InDesign. Seeing as these are only day classes, I am by no means a pro at any of them but I am getting more familiar with the Adobe programs. My teacher who was a freelance web designer told me that along with those Adobe programs, if I could learn HTML, XHTML, CSS, Flash, and Java that I would be in good shape regarding getting a job. She also told me if I could get good at both the design and programming sides, I could really get a good job.
I was just wondering if anyone has some advice or information for basically a noob who is starting from scratch and really wants to get into this profession. I've gotten on lynda.com to try and get going on the programming stuff and I'm just trying to turn some of these skills into a job. Best case scenario is that eventually I could be doing freelance and supporting myself...But that is obviously very far away. ANy advice would be greatly appreciated....
Advice You Don't Want To Hear
figure out what went wrong with college - at 22, you should have graduated by now, or close to it
fix it!
focus!
go back to college with a new focus and resolve - you now have a goal
If You Can Teach Yourself, You Can Graduate College
if the problem was boring general ed classes, look into CLEP tests to test out of them
if the problem was curriculum, take classes you are interested in first
if the problem was time, start part-time
if the problem was school choices, research different colleges - many online universities are now accredited
if the problem was self-discipline, then freelancing is not a good career choice
I'm not saying that you can't do what you asked about, but your chances of getting a good job are greatly increased with a degree, even a 2-year degree.
Obviously, this is not the only path, but it is arguably the easiest path. Marketing yourself when you have no degree and no experience is difficult, even if you are overflowing with talent and creativity. (It's easier now with the Internet, but by the same token the marketplace is that much more crowded.)
Good luck!
I'm currently a software engineering consultant. Familiarity with the following list of things helped me get an interview and an offer right out of college.
Note: HTML, XHTML, CSS... these are just markup languages, and chances are they'll barely be glanced at if you put them on a resume. Flash (not so much) and Java are more impressive, but you might want to look into the following additional topics/technologies to really spice up that resume:
Get comfortable with OO (Object-Oriented) principles (inheritance, polymorphism, abstract vs concrete classes, encapsulation, etc.)
Java is a great open-source beginner programming language. I'm primarily a .NET developer so I tend to favor that, but I started with Java in my college days and picked up on it very quickly
.NET 2.0, 3.5 -- C# and VB.NET (LINQ, lambda expressions, anonymous methods, etc.) -- You can start with the Express edition of Visual Studio, but may eventually want to get the full version
Move on to higher level programming concepts such as Design Patterns (MVC/MVP, Command, Facade, Adapter, the list goes on and on) -- I'd recommend the Gang of Four book (Google will tell you which book it is)
Database Management Systems
Learn SQL, be comfortable JOINing, using GROUP BY and HAVING clauses, and familiarize yourself with aggregate functions
Tackle DB design concepts (relational modeling especially)
Start with the free ones, like MySQL or PostgreSQL, then...
Focus on Microsoft SQL Server or Oracle (these are the big cats)
Go deeper with things like Normal Forms, data warehousing (OLAP, MOLAP, ROLAP, cubes, etc.)
Testing: look into unit testing and test-driven development
Software quality assurance - defect prevention techniques, etc. (this goes along with some of the points mentioned below)
Look into methodologies like Waterfall, Agile, and XP (eXtreme programming), perhaps even PSP and TSP
Learn to use Source/version control systems such as CVS, SVN, and VSS (Microsofts, unfortunately not free -- the first two are)
You could get really crazy and learn about static code analysis, but definitely look into code reviews and code inspections
EDIT: I thought I'd give you some books to check out (no particular order):
Introduction to Algorithms, 2nd ed.
Thomas H. Cormen, Charles E.
Leiserson, Ronald L. Rivest, and
Clifford Stein, 2002.
Artificial Intelligence: Structures
and Strategies for Complex Problem
Solving, 5th edition. George F.
Luger, 2005.
A First Book of Visual C++. Gary J.
Bronson, 2000.
An Introduction to Object-Oriented
Programming with Java, 3rd edition
(Java 1.5) update. C. Thomas Wu,
2004.
Mathematical Structures for Computer
Science, 5th edition. Judith L.
Gersting, 2003.
Mastering the Requirements Process,
2nd edition. Suzanne Robertson and
James Robertson, 2006.
Data Management: Databases and
Organizations, 5th edition. Richard
T. Watson, 2006.
Software Quality Engineering:
Testing, Quality Insurance, and
Quantifiable Improvement, 1st
edition. Jeff Tian, 2005.
Artificial Intelligence: A Modern
Approach, 2nd edition. Stuart
Russell and Peter Norvig, 2003.
Software Architecture in Practice,
2nd edition. Len Bass, Paul
Clements, and Rick Kazman, 2003.
Unit Testing in Java: How Tests
Drive the Code, 1st edition.
Johannes Link and Peter Frohlich,
2003.
Practical PostgreSQL, 1st edition.
John C. Worsley and Joshua D. Drake,
2002.
PSP: A Self-Improvement Process for
Software Engineers, 1st edition.
Watts S. Humphrey, 2005.
TSPi: Introduction to the Team
Software Process, 1st edition.
Watts S. Humphrey, 2000.
I guess that's all I have for now. If you can get these things down, your skill set should be pretty solid, and you'll be on your way towards being another member of the software engineering world. I'm not sure that anything you do on your own will give you the same level of knowledge as college courses, but I'm sure this is a good start. This is a hefty list; don't be surprised if obtaining these skills takes a couple of years.
As far as your graphic design skills, depending on the type of job you find yourself in, they may be more important than your programming skills. On top of either skill sets, make sure your soft skills are polished and that you are confident in your work.
Don't create websites by exporting to web from your design programs. If you see yourself creating slices and mouseover effects in Fireworks STOP and hit yourself in the head with something heavy and blunt.
Learn XHTML and CSS and learn them well. Try to be as semantic as possible.
Pick an all inclusive framework and build yourself a web app like a blog. As much as I love ASP.NET it is not all inclusive. It is massive. It will throw you in every direction. The same can be said of Java. Try something simple like Django or Rails.
Practice, practice, practice, and realize that all that you know is crap and that you need to get better.
Go back to 4 and do it till you die.
Ok, this may not be popular - but is drawn from my own experience at being a "self taught" programmer. The bottom line for getting a job at a company as a "programmer/web developer" is about "0%" without some type of degree/certification/on-the-job experience.
You may have noticed "the problem" with that statement - without a degree or certification, how do you get "on-the-job experience"? Welcome to "the real world".
My path to becoming a developer started with "the desire" ... and getting a job doing something else (Semiconductor manufacturing if you must know). I learned programming on my own as a "hobby" and continually looked for ways to apply it to my job tasks to improve my "job performance". I eventually applied for positions that would bring me closer to "programming" to make better use of my growing skills until I had enough "job experience examples" to apply for a programmer position.
That took "8 years". Regardless of what you think about college, getting "any" degree related to computer science at one would have cut that in half. You can do it on your own, but until you have some outstanding examples of how you used your programming skills to solve "real business" problems, you won't be considered over anyone with a degree "just out of school". When you finally do make it, you will notice that the "just out of school" folks who don't know jack about solving "business problems" will probably start with a better salary than yours.
The environment is probably better now than when I started (25 years ago - :-)) but the same general principle applies - the degree may not mean you know how to program, but it will get you past the "HR" screening process so you can get the job. :-)
Good luck ...
Create a project that would actually be useful and non-trivial for you: A forum or message board, or a job posting site, for instance.
But here's the important part: Give yourself a firm deadline. You can do quite a lot in, say, 4 weeks, and if you keep to a schedule like
database backend in week 1
login system in week 2
messaging in week 3
and so on, you can broadly cover many related subjects. Your project won't be pretty, but if you start out being a perfectionist, you'll never finish it, and will end up knowing only the first half of the technology in your project really well.
As you get more experience, you can go back and polish things up or rewrite stuff you did wrong, and then you have a portfolio that accurately reflects your current skills.
Learning HTML, XHTML, CSS, Flash, and Java plus several graphical programs is a pretty tall order. You'll overwhelm yourself trying to do that. Pick one and learn it then move onto the next. Grab a book or search the Stack Overflow archives for recommended online tutorials.
The best way to learn is to pick a project and just work on it. Then learn on demand as you find a need. The end product won't be the highest quality, but you'll learn how everything works together.
For serious web developer it is very crucial to understand how websites work inside out.
As you are starting from scratch I strongly recommend W3Schools.
With this website you can learn from Very Good Tutorials, then Try It Yourself and Test your skills.
Here are the steps for an absolute beginner:
html
css
JavaScript (Client Side Scripting)
PHP (Server Side Scripting)
SQL (DataBase applications)
DOM(Document Object Model)
AJAX
Drupal/Joomla/Plone (Content Management Systems)
NOTE: Type the code rather than cut-paste or using tools like Dreamweaver/FrontPage.Use the tools only after you are very comfortable with manual coding.(Believe me this helps a lot)
Enjoy!!!
The best I could suggest is that you create a fake need, for example create a simple file read / write app, or something that can toss information onto a database and retrieve it.
A more advanced project you could start on after would be a tagboard of sorts, with Create/Update/Read/Delete (CRUD) functionality, and add features to it in order to get practice with cookies, login, more database functionality, etc. You could also try using an image editor that just draws a clock showing the current server time the request was recieved as practice with images.
I would recommend learning html and css first. That is the cornerstone of anything you'll do on the web. For graphics, learn photoshop. Once you can make basic html websites, I would then choose to pursue either asp.net or flash. There are good jobs in both fields. I would say pick asp.net if you like programming, and flash if you like the visual aspects of web development more. As an asp.net developer I would say 90% of my day is doing database related work with MS SQL server. Really focus on databases. Finally, if you develop with asp.net, you should program in C# rather than vb.net. I started as a vb.net programmer and had to switch to C#, simply because most of the professional world uses it, hence it will be easier getting a job.
Look into local Code camps and user groups, create a project and build it, start by lerning HTML, CSS and javascript, then look at learning PHP is a great starter language for doing beginning web development the code side.
as far as getting a job without a degree, start a little lower on the food chain, I started in the QA lab, from there you can learn from the development side good practices and the do's and don't. Also as a QA person you learn in a hurry who are good developers and who are not simply by the work they produce.
W 3 Schools is a neat beginner/novice reference and tutorial site.
The site covers most technologies used in web-development.
I'm really interested in speech-to-text algorithms, but I'm not sure where to start studying up on them. A bunch of searching around led me to this, but it's from 1996 and I'm fairly certain that there have been improvements since then.
Does anyone who has any experience with this sort of stuff have any recommendations for reading / source code to examine? Or just general advice on what I should be trying to learn about if I want to get into the world of writing speech recognition programs (sometimes it's hard to know what to search for if you don't have much knowledge about the domain).
Edit: I'd like to do something cross-platform, but for the moment I'd be targeting linux.
Edit 2: Thanks csmba for the well-thought out reply. At this point in time, I'm mainly interested in being able to create applications that allow automation, or execution of different commands through voice. So, a limited amount of recognizable commands being able to be strung together. An example would be a music player that took commands like "Play the album Hello Everything by Squarepusher", or an application launcher that allowed the user to create voice-shortcuts to launch specific apps.
I realize that it's a pretty giant problem, and that I have nowhere near the level of knowledge required right now to tackle implementing an entire recognition engine, although the techniques involved with doing so fascinate me, and it is something I'd like to work myself up to doing. In all likelihood, I'll probably end up picking up a book or two on the subject and studying up / playing with "simple" implementations in my free time.
This is a HUGE questions, I wouldn't know how to begin... So let me just try giving you the right "terms" so you can refine your quest:
First, understand that Speech Recognition is a diverse and complicated subject, and it has many different applications. People tend to map this domain to the first thing that comes to their head (usually, that would be computers understanding what you are saying like in IVR systems). So first lets distinguise the concept into the main categories:
Human-to-Machine: Applications that deal with understanding what a human is saying, but the human knows he is talking to a machine and the grammar is very limited. Examples are
Computer automation
Specialized: Pilots automating some controls for example (noise a huge problem)
IVR (Interactive Voice Response) systems like Google-411 or when you call the bank and the computer on the other side says "say 'service' to get customer service"
human-to-human (Spontaneous speech): This is a bigger, more complex problem. Here we can also break it down into different applciations:
Call Center: conversation between Agent-Customer, phone quality, compressed
Intelligence: radio/phone/live conversations between 2 or more individuals
Now, Speech-To-Text is not what you should be saying that you care about. What you care about is solving a problem. Different technologies are used to solve different problems. See an overview here of some of them. to summarize, other approaches are Phonetic transcription, LVCSR and direct based.
Also, are you interested in being the PHd behind the technology? you would need a Masters equivalent involving Signal processing and probably a PHd to be cutting edge. In which case, you will work for a company that develops the actual speech engine. Companies like Nuance and IBM are the big ones, but also Phillips and other startups exist.
On the other hand, if you want to be the one implementing applications, you will not be working on the engine, but working on building application that USE the engine. A good analogy I think is form the gaming industry:
Are you developing the graphic engine (like the Cry engine), or working on one of several hundred games, all use the same graphic engine?
Don't get me wrong, there is plenty to work on the quality of the search also outside the IBM/Nuance of the world. The engine is usually very open, and there are a lot of algorithmic tweaking to be done that can dramatically affect performance. Each business application has different constraints and cost/benefit function, so you can make experiments for many years building better voice recognition based applications.
one more thing: in general, you would also want to have good statistics background the lower in the stack you want to be.
At this point in time, I'm mainly interested in being able to create applications that allow automation
Good, we are converging here... Then you have no interest in "Speech-to-Text". That buzzwords takes you to the world of full transcription, a place you do not need to go to. You should be focusing on some of the more Human-to-Machine technologies like Voice XML and the ones used in IVR systems (Nuance is the biggest player there)
I would definitely recommend picking up a book or two if you are new to the field. I've got no experience in the field, so I can't make a recommendation. If you are still in college (or still have close ties), you should find out if any of your professors can make a recommendation.
The survey you linked is probably an excellent resource, too. I'm sure there have been advancements since 1996, but the basics are unlikely to have fundamentally changed. If the survey is well-written, then it would be well worth your time to read it.
For OS X check out this: OS X Speech Technologies
For Windows check out this: Microsoft Speech API
I have worked with IBMs ViaVoice product. It has a good ASR (automated speech recognition) engine, and a nice text-to-speech engine.
The websites not very good, but this is a link for the Embedded version http://www-01.ibm.com/software/voice/support/
It is platform agnostic though, and everything works through a MVC architecture using vxml a variant of xml for voice purposes.
What platform are you targeting ?. There is Microsoft Speech APIs that you can use if its for windows.
There is also the Speech Recognition Service for Android.