Can we automate Screen Reader testing for web accessibility - cross-browser

Is it possible to automate screen reader testing using Jaws and NVDA across all the browser platforms for checking web accessibility ?

Short answer is no, you cannot effectively do this.
However there are lots of automated tools that will identify around 40% of accessibility errors, such as empty links etc.
The reason you need to test manually is that they cannot test for things like logical tab order, keyboard traps etc.
Axe do a decent plugin for Google Chrome that is reasonably good as a starting point. Look in the Play store for that.
Other than that, learn how to use a screen reader (it doesn't take long) and test the site manually, you will pick up accessibility issues far quicker and learn more than trying to follow rules etc.

It's a common mistake to think that accessibility only targets screenreader users and blind or partially blind people. Screen reader testing will only test for a small part of what accessibility guidelines focus on.
Manual screenreader testing is encouraged to test for human interaction according to the 4 WCAG principles (perceivable, operable, understandable, robust). Automating screenreader testing is equivalent to suppress screenreader testing.

Related

Testing accessibility using Mac VoiceOver for web pages

I am working on developing/testing accessibility of a web page using VoiceOver [Mac Tool].
Typical workflow I am using is trial and error,
Do changes to HTML
Open page in browser
Turn on VoiceOver and test and again #1
What I understand is VoiceOver convert/see HTML/Webpage in text form and starts reading it. I hope how it does it would make it easy for me to understand how to make process easy.
So I want to know if there is any tool or utility or guidelines which could assist me to see HTML/Webpage the way VoiceOver does. I mean tool which take html as input and then output text file like VoiceOver (JUST to understand what changes affects VoiceOver in what way quickly).
Any inputs will be really helpful.
WebAIM has a great article titled Using VoiceOver to Evaluate Web Accessibility
It helps by telling you how to get set up, what keys are used, how users navigate, how to test images, tables, and other elements.
Testing with any screen reader is about more than tabbing through the page as you watch. Ideally you want to get to the point where you can turn off your screen and test. You must also recognize that most VO users do not tab through a page, but instead tend to jump around by landmarks, headings, links, lists, and so on.
I am not pasting the entire article, but here is the outline:
Introduction
Getting Started
Reading
Navigation
Other useful shortcut keys
Images
Data Tables
Forms
Practice
VoiceOver on the iPhone, iPod Touch, and iPad
Chrome Dev tools has an Accessibility Properties panel, which will show you the accessible name of an item the way it is exposed to the VoiceOver API. It would be quicker to keep that panel open and only test in VoiceOver when you are happy with what you're seeing in the panel.
To answer your question, it sounds like you're looking for the Accessibility inspector tool, which exposes the accessibility API (OSX Safari uses NSAccessibility and iOS Safari uses UIAccessibility for iOS) bits (which VoiceOver consumes). This is part of Xcode, so you'll need Xcode on your Mac to get to it

Aria-live attribute not working on chrome

I have multiple sets of dynamic content on my page that get updated following some user interaction. I want the updated content to be read out by a screen reader but the live region is not being read by NVDA on chrome. It's working on Firefox though. Please help. Jaws and chromevox on chrome are also not working.
Edit: December 2020 This answer is not so accurate anymore. Chrome has made some great strides in improving accessibility and this answer is likely not relevant if you are still experiencing this problem. I would suggest looking into some of the aria-live specs to better understand how it should be used, as most issues with it not being announced are likely due to things that can be fixed in the implementation rather than an NVDA+Chrome issue.
Original Answer:
Chrome is not commonly used by screenreaders and is lacking in some key accessibility areas. NVDA is most commonly used with Firefox as it's the one of the best browser-screenreader pairings.
This resource from a leading accessibility firm doesn't list chrome as supporting it (although Chrome is significantly far behind in terms of adoption by users with assistive technology - they aren't even mentioned on the page): https://www.paciellogroup.com/blog/2014/03/screen-reader-support-aria-live-regions/
If you're interested in common browser pairings (eg. which screenreader/browser combinations are worth supporting to accommodate the majority of users), please keep an eye on the webaim screenreader surveys - they're the best resource I've come across for statistics.
http://webaim.org/projects/screenreadersurvey6/
apologies for not being able to directly answer why Chrome doesn't seem to work well with NVDA - It's a dangerous road to go down as there are numerous areas where they don't work well together. Considering the usage of Chrome/NVDA users it might not be worth it.

How do you test if your HTML page or application is accessible?

After adding accessibility to my HTML page, how would one test if it is working correctly? I would like to hear how it sounds and navigates for someone who is using a screen reader.
Let me put it another way, when you build a web page you preview it in a browser(s) to see if it looks and behaves correctly. When you add accessible to that same web page how to do you know it's working correctly?
If you are looking for a quick spot check, you can use WebAIM's WAVE, which tells you the big errors. If you wish to dig deeper, you both need an understanding of WCAG 2.0, and assistive technology. WebAIM has a checklist for WCAG 2.0, which would be the next step, along-side using assistive technology. People tend to only think assistive technology = screen readers, but this is incorrect. There are many different types of assistive technology. I recommend reading through the note at the top of the checklist.
NVDA is a free screen reader
Looks like you might be SOL:
Site accessibility: what screen readers, etc to test against, and how?
Screen Readers For Testing Website Accessibility
If you're really looking for a semi-automated solution, you might be able to stitch together Selenium (or similar) and a screen reader emulator (as described in the second linked post) to pull down an audio file of the screen reader, then submit the text to Mechanical Turk for transcription, then diff your site text against the transcript you get to see how confused the writer was. It would cost a few bucks, but potentially save you more-expensive programmer hours.

WCAG vs real users' opinions

To make websites accessible, my policy so far has been to follow the WCAG--no more, no less. I assumed the WCAG was thorough and complete. Thus I assumed that if I followed every one of its recommendations, my sites would be irreproachable from an accessibility standpoint. Having spoken to a number of web users with disabilities, I've wondered if I should reconsider this policy.
These real users have told me that the following techniques are absolutely essential for any website, based on their experience:
JavaScript-based style switcher interface. E.g. to switch between black-on-white and white-on-black color schemes.
Alternate stylesheets. Same purpose as JS switcher, but a different implementation.
High-contrast CSS border around all buttons. (High-contrast background color not sufficient.)
Don't let the DOM update without user intervention (e.g. due to remote event), even with aria-live="polite".
No CSS background colors anywhere. (Seriously.)
Not every user cited every one of these techniques. This is just the union of the sets of suggestions made by all users. There have been other surprising suggestions as well; these are just the ones I remember.
What's notable about all of these suggestions is that, as far as I can tell, they're not WCAG recommendations. What do you make of this dissonance?
One the one hand, I have no reason to doubt that these techniques are important for the users with whom I spoke. They have no reason to be untruthful, or to exaggerate their needs. They seem to have a solid understanding of the assistive technologies they use, so I don't think they're merely miscommunicating, either.
On the other hand, if some people need these techniques, why aren't they in the WCAG? Is the WCAG incomplete? Or are these demands so idiosyncratic, so unique to these particular users, that it would be unwise to include them in the WCAG and implement them on every site?
Can the WCAG be relied upon as a correct and complete checklist for web accessibility? If so, do you have any thoughts on how that could be reconciled with the users' comments, and what might be an appropriate response when they request such techniques?
Edit based on conversation with Dryden Long
I should clarify that I don't expect the WCAG to be absolutely complete and to offer a solution for each and every last disabled user. Rather, I'm getting at whether a web developer can reasonably rely on the WCAG to make a site that's as accessible as practically possible. And, whether one needs to look outside the WCAG to achieve that goal.
1. WCAG 2.0 Drawbacks
WCAG 2.0 are great. They are the result of the work of approx. 200 people working on them during several years, including most of the best experts in web accessibility, representatives from people with disabilities, etc. They have long been discussed to reach a consensus that reflects the views from the different stakeholders (so that, say, accessible content can be created in an effective fashion, without sacrificing aesthetics, without putting a large burden on browsers, etc.) They synthesize many years of research and overcome many of the limitations of the older WCAG 1.0. They have also got the input from the community beyond W3C, receiving hundreds (and even thousands) of public comments every year through the public comments mailing list.
That said, WCAG 2.0 themselves recognize they are not perfect:
Note that even content that conforms at the highest level (AAA) will not be accessible to individuals with all types, degrees, or combinations of disability, particularly in the cognitive, language, and learning areas. Authors are encouraged to [...] seek relevant advice about current best practice to ensure that Web content is accessible, as far as possible, to this community.
From my perspective, maybe people with cognitive, language and learning disabilities have not been able to lobby and be represented at the relevant forums. Also, current research is proceeding to reach conclusive results regarding e.g. dyslexia.
2. Success criteria vs techniques
As a result of that work in WCAG 2.0, a few dozens of success criteria were written, which establish the testable requirements an accessible content must abide by ("accessible" is here to be understood as "conforming to WCAG 2.0"). These sucess criteria are the core, normative content of WCAG 2.0, do not depend on the technology evolution, and thus are quite stable along time.
Then, there are several hundreds of techniques to implement these sucess criteria. Techniques are technology-dependent. They depend on the specific content type (e.g. you use different techniques to provide an alternate text in an HTML page and in a PDF document), but also on the evolution of browsers and assistive technologies, and, what not, on the evolution of web design itself.
That means several things:
Techniques are a living document, they have been updated more or less once every year and a half since WCAG 2.0 were approved. Hint: you may use the Techniques for WCAG 2.0 Submission form to provide your input if you think a useful technique is not currently considered.
Techniques aren't normative but informative. A WCAG Note on Understanding WCAG 2.0 explains that (boldface not in the original)
Note that all techniques are informative. The "sufficient techniques" are considered sufficient by the WCAG Working Group to meet the success criteria. However, it is not necessary to use these particular techniques. If techniques are used other than those listed by the Working Group, then some other method for establishing the technique's ability to meet the Success Criteria would be needed.
Besides, each and every technique includes a footer stating that
If this is a sufficient technique for a success criterion, failing this test procedure does not necessarily mean that the success criterion has not been satisfied in some other way, only that this technique has not been successfully implemented and can not be used to claim conformance.
It makes sense that techniques are not normative, given that there are (maybe) other ways to reach conformance with success criteria. Imagine that, e.g. user agents (browsers and assistive technologies) start supporting picture descriptions embedded in picture files using IPTC IIM (International Press Telecommunications Council Information Interchange Model). They could perfectly become an acceptable way to satisfy Success Criterion 1.1.1 but would not be reflected (at least for now) in any technique.
3. Conformance vs accessibility-in-use
Testing with users
Having a standardized recommendation to check and resort to is great, as it distills the knowledge of the industry on a single place, and provides a common aggreement on what is deemed to be considered "accessible" (hey, that what standards are for). However, Involving Users in Evaluating Web Accessibility is key to ensure universal accessibility. That makes sense: as other answers stated, different people with a broad range of abilities may have different needs which are only reflected in practice. I would add that different websites have different requirements which are not captured by a single, multi-purpose document. That said, you would not normally evaluate your site with people with disabilities if you are creating, for instance, the web page for your neighbourhood soccer team, but you would (ideally) do, for sure, if you are developing the website of a Premier League team.
All this does not make much difference with usual web development practices for usability: even though your client hands in a request-for-quotation document with the requirements and specification of the website, you would hold an interview with them, and have some usability testing done, conduct focus groups, etc. But you would neither do that for your neighbourhood team.
Barriers and tricks in practice
People with disabilities are used to dealing with inaccessible sites. The techier they are, the more used they are to resorting to "tricks" so as to gain access to otherwise inaccessible contents: "Yeah, you need to alt-space and then down and the menu will pop-out", "if you activate that hidden option in JAWS and activate virtual cursor it will read the fields in that page". Less tech-savvy users maybe do not know that sort of tricks, but they learn some tricks by heart on the sites they use most: "I need to type 'k' twice to select King's Station".
Research has found that results of evaluations by people with disabilities consistently exceed those of conformance testing. That is, users overcome some accessibility barriers by resorting to these "tricks". However, that should not divert the focus of conformance. Many people will not be knowleadgeable of those tricks, and even those who are will encounter poorer usability, will take more time to complete their tasks, and would be more prone to leave (or choose a competitor's site next time).
On the other hand, some techniques rely on the support of assistive technologies, whose use may require the user's expertise. For instance, technique C22 deals with using CSS to control visual presentation of text. The idea is quite simple (boldface added):
The objective of this technique is to demonstrate how CSS can be used to control the visual presentation of text. This will allow users to modify, via the user agent, the visual characteristics of the text to meet their requirement. The text characteristics include aspects such as size, color, font family and relative placement.
However, how can users modify via the user agent the visual characteristics of the text? This is a setting generally available, through different ways on different platforms. Some users may not be familiar enough with that, and find more convenient having an in-page button to switch to large-print, high-contrast text. Or some users prefer having that option readily available within the page.
4. Techniques that support your user requirements
All of the above present explanations of why WCAG 2.0 may not be sufficient. However, many of the cases you present are indeed considered by current WCAG 2.0 techniques
JavaScript-based style switcher interface, and alternate stylesheets (served by the server) are both considered by technique C29 on Using a style switcher to provide a conforming alternate version, and also supported by technique G174 on Providing a control with a sufficient contrast ratio that allows users to switch to a presentation that uses sufficient contrast
No CSS background colors anywhere is related to technique G148 on not specifying background color, not specifying text color, and not using technology features that change those defaults. I would not personally consider CSS background colors should be avoided. Sometimes, the problem is that you have only set background, but not foreground colors. Make sure that you both specify foreground and background, or none of them, to avoid mingling your style with user's.
Don't let the DOM update without user intervention (even with ARIA polite) is related to techniques G75 on providing a mechanism to postpone any updating of content and G76 on Providing a mechanism to request an update of the content instead of updating automatically
High-contrast CSS border around all buttons, background contrast not being enough, is the only thing that puzzles me and could deserve further investigation.
Take-home message
After that long-but-needed explanation, my summary advice is:
Get most familiar with WCAG 2.0 success criteria and understand why they are necessary. Get familiar with how people with disabilities use the web (I note you have already done that).
Apply WCAG 2.0 techniques judiciously. You'll mostly be on the safe side if you stick to WCAG 2.0 techniques. However, always take into account the sucess criteria they respond to, and ask yourself if they are the best way to meet them. Choose the techniques that suit best to the specific website, audience, and your experience.
Involve users with different abilities in web site development, testing and maintenance (I also note you are already doing that). You will probably have already tackled most potential accessibility barriers, but user evaluation will probably stick out a few more. After all, if you get user feedback detecting a problem, correct it.
Test as you would do with any other usability test. That is, do not focus on the users reporting whether CSS background colors should not be specified, but check whether they can easily perform the tasks they are deemed to do on your website, or if they get stuck at some point. Same as you do not ask whether psychometric properties of a specific color are deemed to be atractive, but do test whether red buttons achieve a higher conversion rate in A/B tests.
Can the WCAG be relied upon as a correct and complete checklist for web accessibility? Clearly it is not the ultimate, theorem-provable, way to ensure everyone can access your website on equal footing. This is a matter of human-computer interaction, and as such it cannot be wholly captured by a limited set of guidelines. However, it is a reliable source that clearly covers most of the real-world use scenarios, and great advantage can be taken of it, if used as it is intended.
To the best of my knowledge, complying with WCAG 2 is considered sufficient for accessibility for legal purposes in most countries. But as you note, it can never cover every unique combination and circumstance of disability.
If you are following WCAG2 correctly, site visitors will be able to use assistive technology and tools to customise their experience. For example, using relative units like ems and percentages for font-sizes means that IE users can rely on the browser's text-resize tool to make the text the exact size they like (since IE hates pixel-sizes); making sure you don't put valuable information in background images or text in images means that the content is preserved if the visitor uses Window's high-contrast mode or makes their own user-stylesheet to get nice big highlights on focus. I do think the WCAG guideline for 'visible focus' is a bit soft - the default dotted outline is considered sufficient, but often gets lost on complex or colourful designs. I usually use the same effect for both :hover and :focus as I find it gives a more distinct result when tabbing through a page.
A lot of WCAG2 is actually about removing anything that would prevent customisation. On European sites I've seen the term 'barrier-free' used as a translation of 'accessibility', which I think is a good way to think about it. The content should be able to be poked and prodded to the user's liking without any loss of meaning.
Having some disabled customers/clients participate in your user-testing is a good idea for catching things that are not covered by WCAG2 but might still affect your specific target audience. For example, if you're making a site targeted at senior citizens you might want to check the size of buttons and links, since mobility can be restricted by age-related issues like arthritis.
As a developer, it's your responsibility to make sure all the content is exposed to the browser's accessibility engine; from there it gets exposed to either the user, the operating system, or assistive technology. You can't fix every problem for every user, but you can make sure your site is open and flexible.

WAI-ARIA, IE7, and JAWS

One of the projects I am currently working on has customer requirements for IE7 and 508 compliance. This customer currently uses JAWS version 9 and is making requests which I am told are impossible without a newer version of JAWS which supports WAI-ARIA. For example:
Accessible, 508 compliant Ajax Loading Indicator
Accessible, 508 compliant Ajax Pagination
Before I come back and tell them that they need to upgrade to a newer version of JAWS (from Freedom Scientific's website, WAI-ARIA is supported starting with version 10), I would like to be armed with as much information as possible to make a convincing argument. First of all, is IE7 capable of conveying ARIA information to JAWS (I've seen claims of ARIA support starting with IE8, but nothing explicitly stating it doesn't work with IE7)? If so, is it only to a limited extent? Supposing the answer to this question is yes and the degree of support is sufficient, are there big differences in support between JAWS versions 10, 11, and 12 which I should know about? Perhaps this will is a good bar to meet: given the requirement for IE7, what is the minimum version of JAWS required to apply ARIA in order to solve the two questions linked above.
I know there are a lot of questions jammed in here, but I believe they worked together as a single question: please give me an overview of the varying degrees of support for WAI-ARIA between IE7 and different versions of JAWS; and to what extent can these solve the customer requests linked above.
Thanks! And even if you can't address all of the elements of this question, I would be grateful for any information you can share.
You seem to be equating 508 compliance with support for WAI-ARIA? AFAIK 508 doesn't dictate the use of ARIA.
While ARIA has been designed to overcome the many difficulties of making dynamic web apps accessible, some old hacks that it's designed to replace still more or less work. Also because it is a relatively new technology there is limited support in anything but the latest browsers/Assistive tech.
To have an AJAX progress indicator and pagination accessible with-out ARIA is not straightforward but usually you can get it working one way or another by forcing Focus to the updated parts of the page and using hidden (off-left) text to give screen reader users some verbal directions or hints. The trick being this has to be done is such a way that JAWS will go back and reload it's DOM rather that it's cached version of the page. JAWS 9 is better at this than previous versions.
If you google around you should be able to find various ways to do this.
My adice to you is hire an Accessibility Consultant for a couple of sessions otherwise you are going to be stumbling around in the dark, they'll also be able help you talk to your client and suggest Design changes if it is all looking too complicated to be done in an accessible way.
"First of all, is IE7 capable of
conveying ARIA information to JAWS"..
etc
Browsers don't send ARIA information directly to JAWS, they notify the accessibility layer of Operating System which just interpret the messages as standard operating system accessibility messages. The Assistive technology (JAWS or whatever) are just (hopefully) listening for these system messages.
I can’t speak to Chris Bentley’s suggestion of using off screen text to provide hints to screen readers. While I assume this would work I’ve never seen it done on a website. I’ve also never dealt with internal apps designed to be 508 compliant and have next to no web development experience. I second Chris’ suggestion of hiring a consultant who has experience in this area. I don’t know what the differences are between Jaws versions and WAI-ARIA support. I know Jaws 10 does at least offer some WAI-ARIA support under Internet Explorer 7. I just went to http://news.yahoo.com with Jaws 10 and was able to use the ; key to move from landmark to landmark. You can run multiple versions of Jaws without a problem on the same machine, I currently have Jaws 10, 11, and 12 installed. This should allow you to test different versions of Jaws if you can't find documentation that spells out the differences.
WAI-ARIA support is available starting in IE8 and Firefox 2+. Concerning screenreaders, use NVDA instead of JAWS. It's free and works great with Firefox. In fact, the Mozilla foundation has granted funding to NVDA to improve the NVDA screen reader. They work great together.