EPUB Reading System conformance test suite project kicks off

October 25, 2012

The IDPF today makes an initial call for participation in a new EPUB conformance test suite project. This newly started open source project is developing materials and methods for the evaluation of feature coverage in EPUB Reading Systems.  The goal is to develop a testing paradigm that is comprehensive, objective, and tests all the features in EPUB 3 and its dependent Web Standards that are relevant to content creators. A means to publicly communicate results is also envisioned.

As EPUB 3 support rolls out in the industry, there is an increasing demand for this level of detailed Reading System conformance information to promote interoperability of EPUB content. Ultimately, the project intends to deliver
  • a collection of EPUB documents that comprise an extensive set of tests for both required and optional Reading System functionality and features
  • a documented process for executing the tests and compiling results
  • a web service for the submission, storage and comparison of results
At this inception stage, development of tests is the primary activity, and IDPF seeks contributors to work to extend the test coverage. Contributions from Reading System developers, service providers, content creators, and others are welcome.
In developing the IDPF EPUB Reading System conformance  test suite, we will apply a combination of testing methodologies to provide a solution that balances completeness, robustness, and practicality. The scope of conformance  testing of even individual features of Web Standards is potentially unbounded, given the combinatorics of intersections among features (e.g. CSS multi-column layout intersects with CSS writing modes and CSS text). Therefore we will develop in an agile and incremental manner, aiming in the first instance for complete feature coverage in a testing methodology that requires no more than 2 to 3 hours of manual effort and inspection to test an individual reading system. Automated testing will be preferred but not at the expense of applicability to all reading systems. Reading Systems may not support JavaScript, and may be dedicated software or devices that do not have APIs or network connectivity. Since we know that every EPUB reading system can consume EPUB 3 files and render a representation, inspection of rendering results will be the baseline methodology, with automation as an accelerator where appropriate. Note that while testing of accessibility features is in scope for the test suite, the test suite itself will initially to a large extent be predicated on visual inspection of results, i.e. will not itself be full accessible.
Relationship of the IDPF EPUB Reading System conformance  test suite to other projects:
  • The IDPF EPUB 3 Samples open source project will be a primary source for testfiles for the IDPF EPUB Reading System conformance  test suite. However, these vetted samples are intended to showcase best authoring practices, so may not in all cases be suitable as conformance test elements, and correspondingly not all test files will make good samples, but these projects will aim to maximize alignment.EPUB 3 Samples open source project 
  • The IDPF Readium open source EPUB 3 renderer will be a primary initial testbed for the EPUB Reading System conformance test suite, and we may consider measures to optimize automation of testing of Readium. But the EPUB Reading System conformance test suite will be designed to work on any EPUB 3 Reading System, not just Readium.
  • HTML5 tests (Modernizr, CanIUse, Ringmark, W3C test files) will be leveraged wherever possible especially when testing features amenable to automated determination via JavaScript.
  • BISG EPUB 3 Support Grid: IDPF expects that detailed systematic test results will complement and inform higher-level information about EPUB 3 feature support that BISG is reporting via the EPUB 3 Support Grid. IDPF will continue to be contribute to this activity through the BISG Content Structure Committee, and BISG has volunteered to become involved in the development and potentially hosting of the web service for the submission, storage and comparison of results.
More information is available in the source code repository as well as in the project wiki. For questions and/or offers to contribute, please contact IDPF CTO Markus Gylling via email or submit a subscribe request to the epub-testsuite discussion list

Secondary menu