Toward QIIME 2

As some of our users may be aware, we’re starting to think about our transition from QIIME 1 to QIIME 2. We want to briefly share our thoughts about this transition so the QIIME user community has an idea of what to expect as we start this process.

First, QIIME 1.9.x is a long-term support release. We will continue supporting users of QIIME 1.9.1 and later through 2017. This will include support on the QIIME forum, education in QIIME Workshops (which will increasingly focus on QIIME 2 as it becomes available), and bug fix releases as necessary. We will not  be adding new features to QIIME 1 in the interest of focusing our development efforts on QIIME 2. Starting on January 1, 2016, we will not support QIIME versions less than 1.9.1 any more, so we suggest upgrading to the latest version of QIIME before then.

Changes coming with QIIME 2

QIIME 2 will be a nearly complete rewrite of QIIME 1 with several key areas of focus. First, we will be providing access to QIIME through graphical user interfaces, including the web-based Qiita and a new Jupyter-based interface with interactive visualization and automated analysis documentation for simplified method reporting. We currently have a working prototype of the Jupyter-based interface available on GitHub, and we’ll be releasing a video demo of that system in the next few weeks. QIIME 2 will have an improved command line interface for power users, and a stable well-documented API for bioinformatics developers (this will look a lot like scikit-bio’s API). The latter will facilitate the integration of QIIME 2 as a component in other systems. One immediate application for this will be expanding the functionality that is available through the QIIME BaseSpace app.

QIIME 2 will provide better documentation of input and output file formats and will support a plugin architecture, making it easier for developers of other tools to integrate them with QIIME. This means that new microbiome analysis tools can be made available to our users faster. We’ll be working with developers of popular microbiome analysis tools, including phyloseq, PICRUSt, vsearch, HUMAnN2, metagenomeSeq, and the RDP tools to develop support for these packages as QIIME plugins.

QIIME 2 will have improved support for multi -omics data analysis. QIIME 1 supports analysis of any type of data that can be represented in a BIOM file (e.g., shotgun metagenomics and metatranscriptomics profiles, metabolite profiles, genome composition profiles) but the documentation for working with these data, including loading them into BIOM files, is sparse and the functionality is not widely used. QIIME 2 will have first-class support for working with and integrating these data in combined analyses.

QIIME and its dependencies will be much easier to install and update. We will primarily distribute software through Anaconda, and will also distribute virtual machines, possibly in the form of Docker containers.

We’ll provide technical support through a new QIIME 2 Forum based on a Stack Overflow-like system. We’re exploring using BioStars 4 for this. This will allow forum users to better track and report their contributions, so for example, people who answer a lot of questions on the forum can advertise that on their websites, CVs, cover letters, etc. The system will also support upvoting and downvoting of questions and answers, so it’ll be easier to quickly find the correct answers to questions and reduce duplicate questions. We’ll continue to support QIIME 1 users on the existing Google Groups-based QIIME Forum, which we’re now calling the QIIME 1 Forum, through 2017.

QIIME 2 will be released under the BSD three-clause license and will be free for all use (including commercial). Since we’ll support an open plugin architecture, it is possible that third-party developers could release commercial plugins. We’ll discourage that, but can’t (and don’t want to) prohibit it.

Starting now, we’re transitioning to Twitter for making QIIME-related announcements, so you should follow @qiime_ on Twitter for updates. We’ll also continue posting announcements on the QIIME Blog and the QIIME Forum for the short-term. (Note that we won’t provide support on Twitter – all questions should be directed to the QIIME Forum.)

Technical Details

For developers and technical users, here are some additional details. QIIME 2 will be developed in Python 3.5. Our API will follow the scikit-bio API lifecycle, and will be fully documented and doc-tested using numpydoc. We’ll be making extensive use of pandas, NumPy, SciPy, and scikit-learn throughout the project and aligning with the Python scientific computing ecosystem (as discussed in Jake VanderPlas’s SciPy 2015 presentation) as much as possible. We’ll be using a pre-existing framework for parallelization – possibly IPython’s parallel framework, but that is still to be decided. Our command line interfaces will be click-based wrappers around workflows and public APIs. We have not decided on the interactive visualization framework at this time, but we are exploring Jupyter’s interactive widgets, Bokeh, Seaborn, and others. Since QIIME 2 will be a re-write of QIIME 1, we will start with a new GitHub repository (the revision history of QIIME 1 won’t be relevant). The QIIME 2 repository doesn’t exist yet, but it will soon.

We’ll be opening a formal request for comments (RFC) system within the next few months where we can solicit developer community input on key design decisions, and we’ll advertise our RFCs on Twitter. We plan to follow the RFC structure in place with scikit-bio, which is similar to the approach of Python’s Enhancement Proposals (PEPs) and NumPy’s Enhancement Proposals (NEPs).

What to expect over the next few months

We’ll be soliciting input from our user community on features that should be included and on the interfaces we’re developing. You also might notice some activity on GitHub as we start updating existing feature requests and bug reports to reflect our transition to QIIME 2. Our user community is integral to QIIME, and we aim to design and develop QIIME 2 in collaboration with our users.

In the first quarter of 2016 we expect to release public prototypes that will let our users start experimenting with the ideas and tools we’re developing for QIIME 2.

In the first half of 2016, we expect to release a first alpha version of QIIME 2, which will include a subset of features, mostly focused on improved diversity and taxonomic composition analysis, differential abundance testing, regression modeling, and interactive visualizations (i.e., downstream of the BIOM table and phylogenetic tree generation). You can think of this first alpha release as vastly improved results of core_diversity_analyses.py.

During the first half of 2016, we’ll also be releasing recommendations on how to get from raw sequence data to the BIOM table and tree (for example, as illustrated in these vsearch-based notes that were posted last week). Those steps will be integrated in QIIME 2 later in 2016.

 

Thanks, as always, for your interest in QIIME! We’re looking forward to working closely with our user community to support the next generation of microbiome and multi-omics analysis!

Greg Caporaso, on behalf of the QIIME Developers

Advertisement