Hello QIIME Users,
This month we’re starting to transition from prototyping to developing QIIME 2. To mark this transition, I put together a demo video that illustrates the end point of one of our QIIME 2 prototypes, q2d2, as a follow-up to my previous blog post, Toward QIIME 2.
This is an approximately 18-minute video that illustrates ideas for the QIIME 2 API, and our jupyter (i.e., IPython Notebook) based “documentation-driven” interface. The YouTube page linked above has a table of contents that will let you skip to specific sections of the video.
q2d2 can be easily installed and run on your laptop. As I note in the video, you should use q2d2 for testing purposes only – it’s not sufficiently tested for real data analysis.
In an effort to improve the QIIME 1 Forum for our users and to make it easier for us to moderate, we’re making a few changes to the Forum effective now.
First, new posts will be of type “Question” rather than “Discussion”. This enables replies to be up/down-voted, assigned to specific moderators, and a (single) response can be marked as the best answer, at which point the question is noted as having been answered. See this post, which is a test question. Notice that it is “answered” because we’ve selected a one reply as the “best answer”. We think this will help our users find the correct answers to questions in threads that have a lot of back-and-forth discussion. Be sure to make use of this new up/down voting functionality – by up-voting replies that are helpful to you and down-voting ones that aren’t, you’ll help others find answers to their questions quicker.
Next, we now disallow posting by email. This means that you’ll have to use the web interface to post new questions and reply to messages. You will still get email notifications of activity on threads that you’re involved with, and these emails will have links that can take you directly to the thread. While we realize that this is a little less convenient, it will help users identify duplicate questions when a post is created (see Figure 1), which overall will improve the forum by reducing duplicated discussions. We also hope that it will reduce the time that developers spend moderating the forum, which means we can spend more time working on QIIME itself (including QIIME 2!).
Finally, we now disallow replying to topics that are over a year old. We hope that this will help keep Q&A focused, and therefore easier for other users to follow and learn from.
We hope these changes will make the Forum easier to use, and even more useful to our users. Talk with you on the Forum!
As some of our users may be aware, we’re starting to think about our transition from QIIME 1 to QIIME 2. We want to briefly share our thoughts about this transition so the QIIME user community has an idea of what to expect as we start this process.
First, QIIME 1.9.x is a long-term support release. We will continue supporting users of QIIME 1.9.1 and later through 2017. This will include support on the QIIME forum, education in QIIME Workshops (which will increasingly focus on QIIME 2 as it becomes available), and bug fix releases as necessary. We will not be adding new features to QIIME 1 in the interest of focusing our development efforts on QIIME 2. Starting on January 1, 2016, we will not support QIIME versions less than 1.9.1 any more, so we suggest upgrading to the latest version of QIIME before then.
QIIME 2 will be a nearly complete rewrite of QIIME 1 with several key areas of focus. First, we will be providing access to QIIME through graphical user interfaces, including the web-based Qiita and a new Jupyter-based interface with interactive visualization and automated analysis documentation for simplified method reporting. We currently have a working prototype of the Jupyter-based interface available on GitHub, and we’ll be releasing a video demo of that system in the next few weeks. QIIME 2 will have an improved command line interface for power users, and a stable well-documented API for bioinformatics developers (this will look a lot like scikit-bio’s API). The latter will facilitate the integration of QIIME 2 as a component in other systems. One immediate application for this will be expanding the functionality that is available through the QIIME BaseSpace app.
QIIME 2 will provide better documentation of input and output file formats and will support a plugin architecture, making it easier for developers of other tools to integrate them with QIIME. This means that new microbiome analysis tools can be made available to our users faster. We’ll be working with developers of popular microbiome analysis tools, including phyloseq, PICRUSt, vsearch, HUMAnN2, metagenomeSeq, and the RDP tools to develop support for these packages as QIIME plugins.
QIIME 2 will have improved support for multi -omics data analysis. QIIME 1 supports analysis of any type of data that can be represented in a BIOM file (e.g., shotgun metagenomics and metatranscriptomics profiles, metabolite profiles, genome composition profiles) but the documentation for working with these data, including loading them into BIOM files, is sparse and the functionality is not widely used. QIIME 2 will have first-class support for working with and integrating these data in combined analyses.
QIIME and its dependencies will be much easier to install and update. We will primarily distribute software through Anaconda, and will also distribute virtual machines, possibly in the form of Docker containers.
We’ll provide technical support through a new QIIME 2 Forum based on a Stack Overflow-like system. We’re exploring using BioStars 4 for this. This will allow forum users to better track and report their contributions, so for example, people who answer a lot of questions on the forum can advertise that on their websites, CVs, cover letters, etc. The system will also support upvoting and downvoting of questions and answers, so it’ll be easier to quickly find the correct answers to questions and reduce duplicate questions. We’ll continue to support QIIME 1 users on the existing Google Groups-based QIIME Forum, which we’re now calling the QIIME 1 Forum, through 2017.
QIIME 2 will be released under the BSD three-clause license and will be free for all use (including commercial). Since we’ll support an open plugin architecture, it is possible that third-party developers could release commercial plugins. We’ll discourage that, but can’t (and don’t want to) prohibit it.
Starting now, we’re transitioning to Twitter for making QIIME-related announcements, so you should follow @qiime_ on Twitter for updates. We’ll also continue posting announcements on the QIIME Blog and the QIIME Forum for the short-term. (Note that we won’t provide support on Twitter – all questions should be directed to the QIIME Forum.)
For developers and technical users, here are some additional details. QIIME 2 will be developed in Python 3.5. Our API will follow the scikit-bio API lifecycle, and will be fully documented and doc-tested using numpydoc. We’ll be making extensive use of pandas, NumPy, SciPy, and scikit-learn throughout the project and aligning with the Python scientific computing ecosystem (as discussed in Jake VanderPlas’s SciPy 2015 presentation) as much as possible. We’ll be using a pre-existing framework for parallelization – possibly IPython’s parallel framework, but that is still to be decided. Our command line interfaces will be click-based wrappers around workflows and public APIs. We have not decided on the interactive visualization framework at this time, but we are exploring Jupyter’s interactive widgets, Bokeh, Seaborn, and others. Since QIIME 2 will be a re-write of QIIME 1, we will start with a new GitHub repository (the revision history of QIIME 1 won’t be relevant). The QIIME 2 repository doesn’t exist yet, but it will soon.
We’ll be opening a formal request for comments (RFC) system within the next few months where we can solicit developer community input on key design decisions, and we’ll advertise our RFCs on Twitter. We plan to follow the RFC structure in place with scikit-bio, which is similar to the approach of Python’s Enhancement Proposals (PEPs) and NumPy’s Enhancement Proposals (NEPs).
We’ll be soliciting input from our user community on features that should be included and on the interfaces we’re developing. You also might notice some activity on GitHub as we start updating existing feature requests and bug reports to reflect our transition to QIIME 2. Our user community is integral to QIIME, and we aim to design and develop QIIME 2 in collaboration with our users.
In the first quarter of 2016 we expect to release public prototypes that will let our users start experimenting with the ideas and tools we’re developing for QIIME 2.
In the first half of 2016, we expect to release a first alpha version of QIIME 2, which will include a subset of features, mostly focused on improved diversity and taxonomic composition analysis, differential abundance testing, regression modeling, and interactive visualizations (i.e., downstream of the BIOM table and phylogenetic tree generation). You can think of this first alpha release as vastly improved results of core_diversity_analyses.py.
During the first half of 2016, we’ll also be releasing recommendations on how to get from raw sequence data to the BIOM table and tree (for example, as illustrated in these vsearch-based notes that were posted last week). Those steps will be integrated in QIIME 2 later in 2016.
Thanks, as always, for your interest in QIIME! We’re looking forward to working closely with our user community to support the next generation of microbiome and multi-omics analysis!
Greg Caporaso, on behalf of the QIIME Developers
The QIIME 1.9.1 bug fix release is now live, and the virtual machines (Virtual Box and EC2 image) will be available by Friday, 5/29. This release contains several serious bug fixes, detailed in the 1.9.1 ChangeLog, so we recommend that all users update to QIIME 1.9.1. See the instructions for upgrading to the latest version of QIIME.
If you have any questions about what’s changed in this version of QIIME, or about the bugs that were fixed, don’t hesitate to get in touch on the QIIME Forum.
Finally, several of the QIIME developers will be at ASM next week – come find us to say hello, and we’ll give you one of our new QIIME stickers.
Thanks for using QIIME!
Greg (on behalf of the QIIME developers)
We have now added a bug fix for qiime-default-reference issue #14, which is the bug mentioned in my most recent blog post. If you’ve install QIIME with pip, or are working in the AWS instance or QIIME VirtualBox, you can fix this bug by running:
pip install --upgrade qiime-default-reference
You may need to preface this command with sudo, depending on how you installed QIIME. For example:
sudo pip install --upgrade qiime-default-reference
This will update the default template alignment used by PyNAST to be the PyNAST-aligned (rather than SSU-aligned) Greengenes 13_8 85% OTU representative sequences.
To confirm that the update was successful, you should run print_qiime_config.py. This should list the version of qiime-default-reference as 0.1.2. For example:
Comparisons of PyNAST aligning against both the Greengenes core set (as used in QIIME 1.8.0 and earlier) and the PyNAST-aligned Greengenes 13_8 85% OTU representative sequences (packaged in qiime-default-reference 0.1.2) are available linked from here.
Again, apologies for any inconvenience this may have caused.
We discovered an issue in QIIME 1.9.0 with alignment of 16S amplicons that are generated with non-515F/806R primers when using PyNAST with the default template alignment. This bug will only be a problem for your analysis if it meets those exact criteria.
This bug stems from an issue with the template alignment that is packaged in the qiime-default-reference project, and would cause many of your sequences to fail to align with PyNAST. The issue is discussed in more detail, including a description of a work-around that we’re currently testing, on qiime-default-reference issue #14. Please follow that issue to track progress on this. We’ll follow-up on the blog and QIIME forum when a fix is ready – we expect that this will be this week.
We know that this is very inconvenient, and we apologize for any issues this may have caused. If you have any questions about this, please don’t hesitate to follow up on the forum. If you’d like to know if this issue affected one of your runs, please post to the forum with the output of running print_qiime_config.py.