QIIME 2 has succeeded QIIME 1

As of 1 January 2018, QIIME 2 has succeeded QIIME 1. QIIME 1 is no longer officially supported, as our development and support efforts are now focused entirely on QIIME 2. We’re looking forward to helping our users transition to QIIME 2, and the best place to get started is in the QIIME 2 documentation.

Our design of QIIME 2 has been heavily influenced by interactions with our users at workshops and on the QIIME 1 and QIIME 2 forums, and we think you’ll like the new system. New features include improved microbiome analysis methods (see the QIIME 2 tutorials for details), different interface types, integrated provenance tracking (so you’ll always be able to tell how your data, visualizations, or statistics were generated), the plugin architecture (which makes it easier for method developers to get their tools to QIIME users), and the new interactive visualizations (see a few examples in the Gallery at https://view.qiime2.org/).

Some users have asked what it means for QIIME 1 to no longer be officially supported. This means that QIIME 1 developers will no longer be making updates to the code (for example, bug fixes, new features, documentation, etc.), nor providing new QIIME 1 releases. We will also no longer provide official support on the QIIME 1 Forum.

While official support is being dropped, the software itself and existing releases will remain available for use, and the website will remain live. The source code will also continue to be available at https://github.com/biocore/qiime. We will also keep the QIIME 1 Forum around in case anyone wants to ask questions and receive unofficial support for QIIME 1 (e.g., from community members). QIIME 1 developers will not be monitoring the forum, however, and we make no guarantees that existing QIIME 1 releases will continue to function into the future (for example, if an upstream dependency breaks an existing QIIME 1 release, we won’t be there to fix it).

The QIIME 1 Amazon EC2 and VirtualBox images will also continue to be available, so even if a release of QIIME 1 isn’t natively installable anymore at some point (for example, due to a change in a dependency), you should still be able to use those virtual machines for as long as the images are supported by Amazon or VirtualBox.

Finally, we will no longer use this blog (http://blog.qiime.org) to communicate announcements to our users. Instead, we will use the Announcements category on the QIIME 2 Forum. This is where you’ll be able to find information about QIIME releases, upcoming workshops, software bugs, etc. The most important announcements also go to the @qiime2 Twitter account.

If you haven’t started migrating over to QIIME 2 yet, now is the time to do so. To obtain official support for QIIME 2, please post questions on the QIIME 2 Forum.

Thanks to all of our users of QIIME 1 over the past seven and a half years. We look forward to continuing to work with you on QIIME 2!

Greg (on behalf of the QIIME 1 developers)

QIIME 2 will succeed QIIME 1 on 1 Jan 2018

As many of our users are aware, we’re very actively developing QIIME 2, which offers many benefits over QIIME 1. As of 1 January 2018, we will no longer support QIIME 1, in order to focus our effort entirely on QIIME 2. This means that, beginning in January, we will no longer monitor the QIIME 1 forum, provide bug fixes for QIIME 1, or have any future releases of QIIME 1. We will also discontinue the QIIME 1 Blog at that time, in favor of the QIIME 2 Forum Announcements. As of the QIIME 1.9.0 release, we’re no longer adding new features to QIIME 1, as we focus our development effort on QIIME 2.

Our goal is to make this transition as smooth as possible for our user community. To prepare for this change, we recommend that our users start familiarizing themselves with QIIME 2 now. New QIIME users should begin learning QIIME 2, not QIIME 1.

The documentation for QIIME 2 is available at https://docs.qiime2.org. Start with our Getting Started document for a description of how we think you should work through the documentation to most effectively learn QIIME 2. You can get help with QIIME 2 on the QIIME 2 Forum, where you should start by reading our forum Code of Conduct. We’re also in the early stage of preparing a document to help translate QIIME 1 commands to QIIME 2 commands. We plan to re-implement the QIIME 1 commands that we believe are the most used, but not the commands that we don’t think are used (for example, because they provide outdated methods).

We’re looking forward to helping our users transition to QIIME 2. We think you’ll like the new features, including different interface types (including a prototype graphical user interface!), integrated provenance tracking, improved microbiome analysis methods, a plugin architecture, and the new interactive visualizations.

As always, thanks for using QIIME!

QIIME 2.0.6 is live!

Hi all,

The QIIME 2.0.6 release is now live! QIIME 2 is still in alpha (i.e., early testing) release stage, but with this release we’ve overhauled the QIIME 2 website which includes installation and usage tutorials, and created a new QIIME 2 user forum, so it’s a great time to try it out.

As always, thanks for your interest in QIIME!

Greg

QIIME 2 updates on the American Gut Blog

It’s official: QIIME 2 is now NSF-funded! Find my most recent QIIME 2 blog post on the American Gut blog here.

Enjoy!

Greg

q2d2: A public QIIME 2 prototype

Hello QIIME Users,
This month we’re starting to transition from prototyping to developing QIIME 2. To mark this transition, I put together a demo video that illustrates the end point of one of our QIIME 2 prototypes, q2d2, as a follow-up to my previous blog post, Toward QIIME 2.

This is an approximately 18-minute video that illustrates ideas for the QIIME 2 API, and our jupyter (i.e., IPython Notebook) based “documentation-driven” interface. The YouTube page linked above has a table of contents that will let you skip to specific sections of the video.

q2d2 can be easily installed and run on your laptop. As I note in the video, you should use q2d2 for testing purposes only – it’s not sufficiently tested for real data analysis.

Enjoy!
Greg

Changes to the QIIME 1 Forum

In an effort to improve the QIIME 1 Forum for our users and to make it easier for us to moderate, we’re making a few changes to the Forum effective now.

First, new posts will be of type “Question” rather than “Discussion”. This enables replies to be up/down-voted, assigned to specific moderators, and a (single) response can be marked as the best answer, at which point the question is noted as having been answered.  See this post, which is a test question. Notice that it is “answered” because we’ve selected a one reply as the “best answer”. We think this will help our users find the correct answers to questions in threads that have a lot of back-and-forth discussion. Be sure to make use of this new up/down voting functionality – by up-voting replies that are helpful to you and down-voting ones that aren’t, you’ll help others find answers to their questions quicker.

Next, we now disallow posting by email. This means that you’ll have to use the web interface to post new questions and reply to messages. You will still get email notifications of activity on threads that you’re involved with, and these emails will have links that can take you directly to the thread. While we realize that this is a little less convenient, it will help users identify duplicate questions when a post is created (see Figure 1), which overall will improve the forum by reducing duplicated discussions. We also hope that it will reduce the time that developers spend moderating the forum, which means we can spend more time working on QIIME itself (including QIIME 2!).

Figure 1: Posting a question now suggests posts where your question may have already been answered.

Finally, we now disallow replying to topics that are over a year old. We hope that this will help keep Q&A focused, and therefore easier for other users to follow and learn from.

We hope these changes will make the Forum easier to use, and even more useful to our users. Talk with you on the Forum!

Greg

Toward QIIME 2

As some of our users may be aware, we’re starting to think about our transition from QIIME 1 to QIIME 2. We want to briefly share our thoughts about this transition so the QIIME user community has an idea of what to expect as we start this process.

First, QIIME 1.9.x is a long-term support release. We will continue supporting users of QIIME 1.9.1 and later through 2017. This will include support on the QIIME forum, education in QIIME Workshops (which will increasingly focus on QIIME 2 as it becomes available), and bug fix releases as necessary. We will not  be adding new features to QIIME 1 in the interest of focusing our development efforts on QIIME 2. Starting on January 1, 2016, we will not support QIIME versions less than 1.9.1 any more, so we suggest upgrading to the latest version of QIIME before then.

Changes coming with QIIME 2

QIIME 2 will be a nearly complete rewrite of QIIME 1 with several key areas of focus. First, we will be providing access to QIIME through graphical user interfaces, including the web-based Qiita and a new Jupyter-based interface with interactive visualization and automated analysis documentation for simplified method reporting. We currently have a working prototype of the Jupyter-based interface available on GitHub, and we’ll be releasing a video demo of that system in the next few weeks. QIIME 2 will have an improved command line interface for power users, and a stable well-documented API for bioinformatics developers (this will look a lot like scikit-bio’s API). The latter will facilitate the integration of QIIME 2 as a component in other systems. One immediate application for this will be expanding the functionality that is available through the QIIME BaseSpace app.

QIIME 2 will provide better documentation of input and output file formats and will support a plugin architecture, making it easier for developers of other tools to integrate them with QIIME. This means that new microbiome analysis tools can be made available to our users faster. We’ll be working with developers of popular microbiome analysis tools, including phyloseq, PICRUSt, vsearch, HUMAnN2, metagenomeSeq, and the RDP tools to develop support for these packages as QIIME plugins.

QIIME 2 will have improved support for multi -omics data analysis. QIIME 1 supports analysis of any type of data that can be represented in a BIOM file (e.g., shotgun metagenomics and metatranscriptomics profiles, metabolite profiles, genome composition profiles) but the documentation for working with these data, including loading them into BIOM files, is sparse and the functionality is not widely used. QIIME 2 will have first-class support for working with and integrating these data in combined analyses.

QIIME and its dependencies will be much easier to install and update. We will primarily distribute software through Anaconda, and will also distribute virtual machines, possibly in the form of Docker containers.

We’ll provide technical support through a new QIIME 2 Forum based on a Stack Overflow-like system. We’re exploring using BioStars 4 for this. This will allow forum users to better track and report their contributions, so for example, people who answer a lot of questions on the forum can advertise that on their websites, CVs, cover letters, etc. The system will also support upvoting and downvoting of questions and answers, so it’ll be easier to quickly find the correct answers to questions and reduce duplicate questions. We’ll continue to support QIIME 1 users on the existing Google Groups-based QIIME Forum, which we’re now calling the QIIME 1 Forum, through 2017.

QIIME 2 will be released under the BSD three-clause license and will be free for all use (including commercial). Since we’ll support an open plugin architecture, it is possible that third-party developers could release commercial plugins. We’ll discourage that, but can’t (and don’t want to) prohibit it.

Starting now, we’re transitioning to Twitter for making QIIME-related announcements, so you should follow @qiime_ on Twitter for updates. We’ll also continue posting announcements on the QIIME Blog and the QIIME Forum for the short-term. (Note that we won’t provide support on Twitter – all questions should be directed to the QIIME Forum.)

Technical Details

For developers and technical users, here are some additional details. QIIME 2 will be developed in Python 3.5. Our API will follow the scikit-bio API lifecycle, and will be fully documented and doc-tested using numpydoc. We’ll be making extensive use of pandas, NumPy, SciPy, and scikit-learn throughout the project and aligning with the Python scientific computing ecosystem (as discussed in Jake VanderPlas’s SciPy 2015 presentation) as much as possible. We’ll be using a pre-existing framework for parallelization – possibly IPython’s parallel framework, but that is still to be decided. Our command line interfaces will be click-based wrappers around workflows and public APIs. We have not decided on the interactive visualization framework at this time, but we are exploring Jupyter’s interactive widgets, Bokeh, Seaborn, and others. Since QIIME 2 will be a re-write of QIIME 1, we will start with a new GitHub repository (the revision history of QIIME 1 won’t be relevant). The QIIME 2 repository doesn’t exist yet, but it will soon.

We’ll be opening a formal request for comments (RFC) system within the next few months where we can solicit developer community input on key design decisions, and we’ll advertise our RFCs on Twitter. We plan to follow the RFC structure in place with scikit-bio, which is similar to the approach of Python’s Enhancement Proposals (PEPs) and NumPy’s Enhancement Proposals (NEPs).

What to expect over the next few months

We’ll be soliciting input from our user community on features that should be included and on the interfaces we’re developing. You also might notice some activity on GitHub as we start updating existing feature requests and bug reports to reflect our transition to QIIME 2. Our user community is integral to QIIME, and we aim to design and develop QIIME 2 in collaboration with our users.

In the first quarter of 2016 we expect to release public prototypes that will let our users start experimenting with the ideas and tools we’re developing for QIIME 2.

In the first half of 2016, we expect to release a first alpha version of QIIME 2, which will include a subset of features, mostly focused on improved diversity and taxonomic composition analysis, differential abundance testing, regression modeling, and interactive visualizations (i.e., downstream of the BIOM table and phylogenetic tree generation). You can think of this first alpha release as vastly improved results of core_diversity_analyses.py.

During the first half of 2016, we’ll also be releasing recommendations on how to get from raw sequence data to the BIOM table and tree (for example, as illustrated in these vsearch-based notes that were posted last week). Those steps will be integrated in QIIME 2 later in 2016.

 

Thanks, as always, for your interest in QIIME! We’re looking forward to working closely with our user community to support the next generation of microbiome and multi-omics analysis!

Greg Caporaso, on behalf of the QIIME Developers

QIIME 1.9.1 is live

The QIIME 1.9.1 bug fix release is now live, and the virtual machines (Virtual Box and EC2 image) will be available by Friday, 5/29. This release contains several serious bug fixes, detailed in the 1.9.1 ChangeLog, so we recommend that all users update to QIIME 1.9.1. See the instructions for upgrading to the latest version of QIIME.

If you have any questions about what’s changed in this version of QIIME, or about the bugs that were fixed, don’t hesitate to get in touch on the QIIME Forum.

Finally, several of the QIIME developers will be at ASM next week – come find us to say hello, and we’ll give you one of our new QIIME stickers.

Thanks for using QIIME!

Greg (on behalf of the QIIME developers)

Bug fix: QIIME 1.9.0 PyNAST default reference alignment

Hi all,

We have now added a bug fix for qiime-default-reference issue #14, which is the bug mentioned in my most recent blog post. If you’ve install QIIME with pip, or are working in the AWS instance or QIIME VirtualBox, you can fix this bug by running:

pip install --upgrade qiime-default-reference

You may need to preface this command with sudo, depending on how you installed QIIME. For example:

sudo pip install --upgrade qiime-default-reference

This will update the default template alignment used by PyNAST to be the PyNAST-aligned (rather than SSU-aligned) Greengenes 13_8 85% OTU representative sequences.

To confirm that the update was successful, you should run print_qiime_config.py. This should list the version of qiime-default-reference as 0.1.2. For example:

Comparisons of PyNAST aligning against both the Greengenes core set (as used in QIIME 1.8.0 and earlier) and the PyNAST-aligned Greengenes 13_8 85% OTU representative sequences (packaged in qiime-default-reference 0.1.2) are available linked from here.

Again, apologies for any inconvenience this may have caused.

Greg

QIIME 1.9.0 bug affecting PyNAST alignment of 16S amplicons generated with non-515F/806R primers

Hi all,

We discovered an issue in QIIME 1.9.0 with alignment of 16S amplicons that are generated with non-515F/806R primers when using PyNAST with the default template alignment. This bug will only be a problem for your analysis if it meets those exact criteria.

This bug stems from an issue with the template alignment that is packaged in the qiime-default-reference project, and would cause many of your sequences to fail to align with PyNAST. The issue is discussed in more detail, including a description of a work-around that we’re currently testing, on qiime-default-reference issue #14. Please follow that issue to track progress on this. We’ll follow-up on the blog and QIIME forum when a fix is ready – we expect that this will be this week.

We’d like to thank forum user Slave Trajanoski for pointing this out, and for the helpful example data which let us narrow down the problem quickly.

We know that this is very inconvenient, and we apologize for any issues this may have caused. If you have any questions about this, please don’t hesitate to follow up on the forum. If you’d like to know if this issue affected one of your runs, please post to the forum with the output of running print_qiime_config.py.

Best,

Greg