How to detect hand gesture in live webcam using javascript? - html

I embedded live web cam to html page. Now i want to find hand gestures. How to do this using JavaScript, I don't have idea. I Googled lot but didn't get any good idea to complete this. So any one know about this? how to do this.

Accessing the webcam requires the HTML5 WebRTC API which is available in most modern browsers apart from Internet Explorer or iOS.
Hand gesture detection can be done in JavaScript using Haar Cascade Classifiers (ported from OpenCV) with js-objectdetect or HAAR.js.
Example using js-objectdetect in JavaScript/HTML5: Open vs. closed hand detection (the "A" gesture of the american sign language alphabet)

Here is a JavaScript hand-tracking demo -- it relies on HTML5 features which are not yet enabled in all typical browsers, it doesn't work well at all here, and I don't believe it covers gestures, but it might be a start for you: http://code.google.com/p/js-handtracking/

You need to have some motion detecting device (Camera) and you can use kinect to get the motion of different parts of the body. You will have to send data in browser telling about the body parts and position where you can manipulate the data according to your requirement
Here you can find how you can make it. Motion detection and rendering
More about kinect General info

While this is a really old question, theres some new opportunities to do handtracking using fast neural networks and images from a webcam. And in Javascript. I'd recommend the Handtrack.js library which uses Tensorflow.js just for this purpose.
Simple usage example.
<!-- Load the handtrackjs model. -->
<script src="https://cdn.jsdelivr.net/npm/handtrackjs/dist/handtrack.min.js"> </script>
<!-- Replace this with your image. Make sure CORS settings allow reading the image! -->
<img id="img" src="hand.jpg"/>
<canvas id="canvas" class="border"></canvas>
<!-- Place your code in the script tag below. You can also use an external .js file -->
<script>
// Notice there is no 'import' statement. 'handTrack' and 'tf' is
// available on the index-page because of the script tag above.
const img = document.getElementById('img');
const canvas = document.getElementById('canvas');
const context = canvas.getContext('2d');
// Load the model.
handTrack.load().then(model => {
// detect objects in the image.
model.detect(img).then(predictions => {
console.log('Predictions: ', predictions);
});
});
</script>
Demo
Running codepen
Also see a similar neural network implementation in python -
Disclaimer .. I maintain both projects.

Related

Rendering with Chromium direct to my own 'canvas' (e.g. GDI+)

First, a quick description of the end-goal:
I'm building a cross-platform, .NET Core-based, printing app. This app will be able to print all sorts of file types with custom page settings, such as headers, footers, and margins. A key feature is it supports multiple-pages-up (e.g. a landscape sheet of paper with two portrait pages rendered side by side...called "2up").
Printing HTML is important not just because of printing HTML, but I want to use all the great HTML-based syntax-highlighting out there for source code (e.g. www.prismjs.com).
The app is basically done but for one major problem: I can't get the HTML to render well enough. So far I've implemented source code printing three ways:
1) As plain text with my own line-numbering and line-wrap engine. This works wonderfully for everything I can throw at it, but it does not support syntax highlighting.
2) Using Html-Renderer (https://github.com/ArthurHub/HTML-Renderer/issues) an OSS .NET-based Html Renderer. This implementation is the weakest because Html-Renderer's CSS support is really weak. It can't handle hardly anything prismjs or highlightjs' generate.
3) Usinglitehtml' (www.litehtml.com) via LiteHtmlSharp. This was very promising and I almost have it working with some major hacks, but litehtml also does not support key, modern, HTML/CSS features.
Neither Html-Renderer or litehtml support the CSS page-break-before feature that, when combined with media print would let me ensure lines are not split between pages.
What I really want to use is the Chromium rendering engine. litehtml provides a fantastic API for this sort of problem: It calls me whenever it needs to render, and I draw (text, table borders, images, etc...) using GDI+. My dream is to find something in Chromium (CEF, Puppeteer, ???) that provides a similar API.
Or, an alternative, an API that will let me pass in a GDI+ Graphics (or HDC) and the renderer will render to that surface.
With Html-Renderer I measure calculate # pages like this.
SizeF size = HtmlRender.MeasureGdiPlus(g, html, containingSheet.GetPageWidth());
int numPages = (int)(size.Height / containingSheet.GetPageHeight());
My page rendering code (e.g. OnPaint) looks like this:
SizeF size = new SizeF(containingSheet.GetPageWidth(), containingSheet.GetPageHeight());
HtmlRender.RenderGdiPlus(g, html, PointF(0, 0), size );
With htmllite the OnPaint code looks like this:
// Set the clip such that any extraLines are clipped off bottom
g.SetClip(new Rectangle(0, 0, (int)Math.Round(PageSize.Width), (int)Math.Round(PageSize.Height - remainingPartialLineHeight)));
LiteHtmlSize size = new LiteHtmlSize(Math.Round(PageSize.Width), Math.Ceiling(PageSize.Height));
litehtml.Document.Draw((int)-0, (int)-yPos, new position {
x = 0,
y = 0,
width = (int)size.Width,
height = (int)size.Height
});
And in this case the call to litehtml.Document.Draw causes a bunch of callbacks into my app that I process using the same Graphics the OnPaint is called with.
Most discussions of CEF and Chromium point to ScreenshotAsync etc... which will not do because I need to be rendering to a PRINTERS HDC (or GDI+) and blitting bitmaps will loose quality.
I have poured over the Chromium source and I cannot find the obvious way to say to CEF/Chromium "render page 1 (defined as Height/Width) to this GDI+ Graphics object" then "render page 2..." etc... The printing support (and how pdfium is integrated come close!).
Chromium issue 311308 indicates I'm hosed until this work gets picked up again.
Note: I have full access to nodejs w/in my app. I have built a dotnet/nodejs bridge, which is how I convert the raw text file of a source code file to richly formatted, line-numbered, syntax-highlighted html via prismjs. This means I could easily use puppeteer/Headless Chrome if I could just figure out the right APIs.
Does anyone have a suggestion that might help? I'm willing to contribute to Chromium if it's not major heart surgery.

Limitations of Web Workers

Please bear in mind that I have never used Web Workers before and I'm having some trouble wrapping my head around them.
Here's an explanation of a simplified version of what I'm doing.
My page has links to various files - some are text, some are images, etc. Each file has an image showing a generic file icon.
I want the script to replace each generic icon with a preview of the file's contents.
The script will request the file from the server (thereby adding it to the cache, like a preloader), then create a canvas and draw the preview onto it (a thumbnail for images, an excerpt of text for text files, a more specific icon for media files...) and finally replace the generic icon's source with the canvas using a data URL.
I can do this quite easily. However, I would prefer to have it in the background so that it doesn't interfere with the UI while it's working.
Before I dive right in to this, I need to know: can Workers work with a canvas, and if so how would I create one? I don't think document.createElement('canvas') would work because Workers can't access the DOM, or am I misunderstanding when all the references I've found say they "can't access the DOM"?
You cannot access the DOM from web workers. You cannot load images. You cannot create canvas elements and draw to them from web workers. For now, web workers are pretty much limited to doing ajax calls and doing compute intensive things. See this related question/answer on web workers and canvas objects: Web Workers and Canvas and this article about using webworkers to speed up image manipulation: http://blogs.msdn.com/b/eternalcoding/archive/2012/09/20/using-web-workers-to-improve-performance-of-image-manipulation.aspx
Your simplest bet is to chunk your work into small chunks (without web workers) and do a chunk at a time, do a setTimeout(), then process the next chunk of work. This will allow the UI to be responsive while still getting your work done. If there is any CPU consuming computation to be done (like doing image analysis), this can be farmed out to a web worker and the result can be sent via message back to the main thread to be put into the DOM, but if not, then just do your work in smaller chunks to keep the UI alive.
Parts of the tasks like loading images, fetching data from servers, etc... can also be done asynchronously so it won't interfere with the responsiveness of the UI anyway if done properly.
Here's the general idea of chunking:
function doMyWork() {
// state variables
// initialize state
var x, y, z;
function doChunk() {
// do a chunk of work
// updating state variables to represent how much you've done or what remains
if (more work to do) {
// schedule the next chunk
setTimeout(doChunk, 1);
}
}
// start the whole process
doChunk();
}
Another (frustrating) limitation of Web Workers is that it can't access geolocation on Chrome.
Just my two cents.
So as others have stated, you cannot access the DOM, or do any manipulations on the DOM from a web worker. However, you can outsource some of the more complete calculations on the web worker. Then once you get your return message from the web worker inside of your main JS thread, you can extract the values you need and use them on the DOM there.
This may be unrelated to your question, but you mentioned canvas so i'll share this with you.
if you need to improve the performance of drawling to canvas, I highly recommend having two canvas objects. One that's rendered to the UI, the other hidden. That way you can build everything on the hidden canvas, then draw the hidden canvas on the displayed one. It may not sound like it will do much if anything, but it will increase performance significantly.
See this link for more details about improving canvas performance.

All-in-one location/hashchange history management library

First of all, I know there's libraries that provide polyfills for location.pushState/popState (History.js, Hash.js, jQuery hashchange), so please don't just link to those.
I need a more powerful library to achieve the following in a RIA:
User clicks a link
library is notified and loads context via Ajax (no complete reload!)
All <a> elements are leveraged with a click handler that
prevents page reloads in 2. (preventDefault) and
calls location.pushState instead / sets location.hash for older browsers
loaded content is inserted in page and replaces current content
Continue with 1.
Also, previously loaded content should be restored as the user navigates back.
As an example, klick through Google+ in Internet Explorer <10 and any other browser.
Is there anything that comes even close? I need support for IE8, FF10, Safari 5 and Chrome 18. Also, it should have a permissive license like MIT or Apache.
I believe Sammy.js ( http://sammyjs.org) (MIT-licenced) has the best focus on what you want to do, with its 2 main pillars being:
Routes
Events
I could quote from the docs but it's pretty straightforward:
setup clientside routes that relate to stuff to be done, e.g: update the view through ajax
link events to call routes, e.g: call the route above when I click an link. (You would have to make sure e.preventDefault is called in the defined event I believe, since this is an app decision really, so that can't be abstracted away by any library that you're going to use imho)
Some relevant docs
http://sammyjs.org/docs
http://sammyjs.org/docs/routes
http://sammyjs.org/docs/events
Example for a route: (from http://sammyjs.org/docs/tutorials/json_store_1)
this.get('#/', function(context) {
$.ajax({
url: 'data/items.json',
dataType: 'json',
success: function(items) {
$.each(items, function(i, item) {
context.log(item.title, '-', item.artist);
});
}
});
});
Or something like
this.get('#/', function(context) {
context.app.swap(''); ///the 'swap' here indicates a cleaning of the view
//before partials are loaded, effectively rerendering the entire screen. NOt doing the swap enables you to do infinite-scrolling / appending style, etc.
// ...
});
Of course other clientside MVC-frameworks could be an option too, which take away even more plumbing, but might be overkill in this situation.
a pretty good (and still fairly recent) comparison:
http://codebrief.com/2012/01/the-top-10-javascript-mvc-frameworks-reviewed/
( I use Spine.js myself ) .
Lastly, I thought it might be useful to include an answer I've written a while ago that goes into detail to the whole best-practice (as I see it) in client-side refreshes, etc. Perhaps you find it useful:
Accessibility and all these JavaScript frameworks
I currently use PathJS in one of my applications.
It has been the best decision that i have made.
For your particular usecase take a look at HTML5 Example.
The piece of code that that makes the example work (from the source):
<script type="text/javascript">
// This example makes use of the jQuery library.
// You can use any methods as actions in PathJS. You can define them as I do below,
// assign them to variables, or use anonymous functions. The choice is yours.
function notFound(){
$("#output .content").html("404 Not Found");
$("#output .content").addClass("error");
}
function setPageBackground(){
$("#output .content").removeClass("error");
}
// Here we define our routes. You'll notice that I only define three routes, even
// though there are four links. Each route has an action assigned to it (via the
// `to` method, as well as an `enter` method. The `enter` method is called before
// the route is performed, which allows you to do any setup you need (changes classes,
// performing AJAX calls, adding animations, etc.
Path.map("/users").to(function(){
$("#output .content").html("Users");
}).enter(setPageBackground);
Path.map("/about").to(function(){
$("#output .content").html("About");
}).enter(setPageBackground);
Path.map("/contact").to(function(){
$("#output .content").html("Contact");
}).enter(setPageBackground);
// The `Path.rescue()` method takes a function as an argument, and will be called when
// a route is activated that you have not yet defined an action for. On this example
// page, you'll notice there is no defined route for the "Unicorns!?" link. Since no
// route is defined, it calls this method instead.
Path.rescue(notFound);
$(document).ready(function(){
// This line is used to start the HTML5 PathJS listener. This will modify the
// `window.onpopstate` method accordingly, check that HTML5 is supported, and
// fall back to hashtags if you tell it to. Calling it with no arguments will
// cause it to do nothing if HTML5 is not supported
Path.history.listen();
// If you would like it to gracefully fallback to Hashtags in the event that HTML5
// isn't supported, just pass `true` into the method.
// Path.history.listen(true);
$("a").click(function(event){
event.preventDefault();
// To make use of the HTML5 History API, you need to tell your click events to
// add to the history stack by calling the `Path.history.pushState` method. This
// method is analogous to the regular `window.history.pushState` method, but
// wraps calls to it around the PathJS dispatched. Conveniently, you'll still have
// access to any state data you assign to it as if you had manually set it via
// the standard methods.
Path.history.pushState({}, "", $(this).attr("href"));
});
});
</script>
PathJS has some of the most wanted features of a routing library:
Lightweight
Supports the HTML5 History API, the 'onhashchange' method, and graceful degredation
Supports root routes, rescue methods, paramaterized routes, optional route components (dynamic routes), and Aspect Oriented Programming
Well Tested (tests available in the ./tests directory)
Compatible with all major browsers (Tested on Firefox 3.6, Firefox 4.0, Firefox 5.0, Chrome 9, Opera 11, IE7, IE8, IE9)
Independant of all third party libraries, but plays nice with all of them
I found the last too points most attractive.
You can find them here
I hope you find this useful.
i'd like to suggest a combination of
crossroads.js as a router
http://millermedeiros.github.com/crossroads.js/
and hasher for handling browser history and hash urls (w/ plenty of fallback solutions):
https://github.com/millermedeiros/hasher/
(based on http://millermedeiros.github.com/js-signals/)
This will still require a few lines of code (to load ajax content etc.), but give you loads and loads of other possibilities when handling a route.
Here's an example using jQuery (none of the above libraries require jQuery, i'm just lazy...)
http://fiddle.jshell.net/Fe5Kz/2/show/light
HTML
<ul id="menu">
<li>
foo
</li>
<li>
bar/baz
</li>
</ul>
<div id="content"></div>
JS
//register routes
crossroads.addRoute('foo', function() {
$('#content').html('this could be ajax loaded content or whatever');
});
crossroads.addRoute('bar/{baz}', function(baz) {
//maybe do something with the parameter ...
//$('#content').load('ajax_url?baz='+baz, function(){
// $('#content').html('bar route called with parameter ' + baz);
//});
$('#content').html('bar route called with parameter ' + baz);
});
//setup hash handling
function parseHash(newHash, oldHash) {
crossroads.parse(newHash);
}
hasher.initialized.add(parseHash);
hasher.changed.add(parseHash);
hasher.init();
//add click listener to menu items
$('#menu li a').on('click', function(e) {
e.preventDefault();
$('#menu a').removeClass('active');
$(this).addClass('active');
hasher.setHash($(this).attr('href'));
});​
Have you looked at the BigShelf sample SPA (Single Page Application) from Microsoft? It sounds like it covers how to achieve most of what you're asking.
It makes use of History.js, a custom wrapper object to easily control navigation called NavHistory and Knockout.js for click handling.
Here's an extremely abbreviated workflow of how this works: first you'll need to initialize a NavHistory object which wraps history.js and registers a callback which executes when there is a push state or hash change:
var nav = new NavHistory({
params: { page: 1, filter: "all", ... etc ... },
onNavigate: function (navEntry) {
// Respond to the incoming sort/page/filter parameters
// by updating booksDataSource and re-querying the server
}
});
Next, you'll define one or more Knockout.js view models with commands that can be bound to links buttons, etc:
var ViewModel = function (nav) {
this.search = function () {
nav.navigate({ page: 2, filter: '', ... }); // JSON object matching the NavHistory params
};
}
Finally, in your markup, you'll use Knockout.js to bind your commands to various elements:
<a data-bind="click: search">...</a>
The linked resources are much more detailed in explaining how all of this works. Unfortunately, it's not a single framework like you're seeking, but you'd be surprised how easy it is to get this working.
One more thing, following the BigShelf example, the site I'm building is fully cross-browser compatible, IE6+, Firefox, Safari (mobile and desktop) and Chrome (mobile and desktop).
The AjaxTCR Library seems to cover all bases and contains robust methods that I haven't seen before. It's released under a BSD License (Open Source Initiative).
For example, here are five AjaxTCR.history(); methods:
init(onStateChangeCallback, initState);
addToHistory(id, data, title, url, options);
getAll();
getPosition();
enableBackGuard(message, immediate);
The above addToHistory(); has enough parameters to allow for deep hash-linking in websites.
More eye-candy of .com.cookie(), .storage(), and .template() provides more than enough methods to handle any session data requirements.
The well documented AjaxTCR API webpage has a plethora of information with downloadable doc's to boot!
Status Update:
That website also has an Examples Webpage Section including downloadable .zip files with ready to use Front End(Client) and Back End(Server) project files.
Notably are the following ready-to-use examples:
One-way Cookie
HttpOnly Cookies
History Stealing
History Explorer
There are quite a bit other examples that rounds out the process to use many of their API methods, making any small learning curve faster to complete.
Several suggestions
ExtJs, see their History Example, and here are the docs.
YUI Browser History Manager.
jQuery BBQ seem to provide a more advanced feature-set over jQuery.hashcode.
ReallySimpleHistory may also be of help, though it's quite old and possibly outdated.
Note: ExtJs History has been extended to optimize duplicate (redundant) calls to add().
PJAX is the process you're describing.
The more advanced pjax techniques will even start to preload the content, when the user hovers over the link.
This is a good pjax library.
https://github.com/MoOx/pjax
You mark the containers which need will be updated on the subsequent requests:
new Pjax({ selectors: ["title", ".my-Header", ".my-Content", ".my-Sidebar"] })
So in the above, only the title, the .my-header, .my-content, and .my-sidebar will be replaced with the content from the ajax call.
Somethings to look out for
Pay attention to how your JS loads and detects when the page is ready. The javascript will not reload on new pages. Also pay attention to when any analytics calls get called, for the same reason.

Generating a tone using pure javascript with Chromium WebAudio API

How can I generate a tone (pure sine wave, for instance) using only javascript and Chromium's WebAudio API?
I would like to accomplish something like the Firefox equivalent.
The Chromium WebAudio demos here appear to all use prerecorded <audio> elements.
Thanks!
The Web Audio API has what's known as the Oscillator Interface to generate the tones you're talking about. They're pretty straight forward to get going...
var context = new webkitAudioContext(),
//Call function on context
oscillator = context.createOscillator(); // Oscillator defaults to sine wave
oscillator.connect(context.destination);
oscillator.start();
You can change the type of wave by doing:
oscillator.type = 1; // Change to square wave.
or alternatively:
oscillator.type = oscillator.SQUARE;
I've written an article about this very topic in more detail, so that might be of some use to you!
Probably not these best way, but I used dsp.js to generate different types of sinusoids, then passed them off to the Web Audio API in this demo: http://www.htmlfivewow.com/demos/waveform-generator/index.html

How to submit HTML and receive a bitmap?

I have an app that allows a user to use JQuery and Javascript to add images and position them in a div dynamically.
I would like to be able to submit the div with all the HTML to a WebService and receive back an image so we have a bitmap of the result of the end user's work.
I would prefer a solution in .Net as this is what I am most familiar with but am open to pretty much anything?
I would like to be able to submit the div with all the HTML to a WebService and receive back an image
Try http://browsershots.org!
Browsershots makes screenshots of your web design in different operating systems and browsers. It is a free open-source online web application providing developers a convenient way to test their website's browser compatibility in one place.
How about this. You load the html into a webbrowser control and then use the DrawToBitmap method. It doesn't show up on intellisense and this is probably not the best solution, but it works. Observe the DocumentCompleted event and add the following code:
private void webBrowser_DocumentCompleted(object sender, WebBrowserDocumentCompletedEventArgs e)
{
var bmp = new Bitmap(100, 100);
var rect = new Rectangle(webBrowser.Location.X, webBrowser.Location.Y, webBrowser.Width, webBrowser.Height);
webBrowser.DrawToBitmap(bmp, rect);
bmp.Save("test.jpg", ImageFormat.Jpeg);
}
You'll probably want to change the width and height of that bitmap object (do it in some smart way or something). Hope this helps.
EDIT: I see now that you are using a webservice for this, hence this solution probably won't work. I'll leave it here just for information's sake.
I was not able to figure out how to do this by submitting the html and receiving an image but I was able to create and ASHX handler that returns a png file based on this blog post
which was good enough for my scenario.
He uses CutyCapt to take a screen shot of an existing web page, write the image to a folder on the webserver and return it.