This release contains a fix for the make_distance_histograms.py bug that we announced last week. Again, we’re sorry for any inconvenience that that may have caused. This release is additionally packed with a lot of new features – some key ones are:

* A lot of new tutorials including retraining of the RDP classifier, working with Amazon Web Services, coverage of basic unix/linux commands, and others.

* Addition of the OTUPIPE workflow for chimera detection, quality filtering, and OTU picking. This is now available via the pick_otus.py module, and will require you to install the usearch software (even in EC2 and VirtualBox, due to licensing restrictions).

* Addition of code to support plotting comparisons of raw distance data in QIIME. This is available in the new scripts make_distance_comparison_plots.py and make_distance_boxplots.py, and covered in a new tutorial which includes some examples of the plots that can be generated.

* Added new script nmds.py to support Non-Metric Multidimensional Scaling analysis.

* Support in the pick_otus_through_otu_table.py scipt for running uclust_ref in parallel with creation of new clusters (i.e., open-reference OTU picking with the reference step running in parallel and the de novo step running serially).

* assign_taxonomy_reference_seqs_fp and assign_taxonomy_id_to_taxonomy_fp are new qiime_config values, allowing users to set defaults for the dataset they’d like to perform taxonomy assignment against. This works for the serial and parallel versions of assign_taxonomy for both BLAST and RDP.

* Added option (-e/–max_rare_depth) to the command line of alpha_rarefaction.py. This provides a convenient way for users to specify the maximum rarefaction depth on the command line, and is useful for when it needs to be set to something other than the median rarefaction depth. Also added option to control minimum rarefaction depth from the alpha_rarefaction.py command line.

* Added support for 5- and 10-fold and leave-one-out cross-validation to supervised_learning.py.

* Added subsample_fasta.py module for randomly subsampling fasta files.

Plus lots of additional new features: the list continues in the ChangeLog for this release.

Thanks, and have fun!

Greg

We discovered a serious bug last night in the make_distance_histograms.py script. This bug results in incorrect t statistics and p values being generated under some circumstances. We’re still investigating this now, and we’re working on a fix.

This bug only affects the output from the make_distance_histograms.py script. If you’re using this script through the beta_diversity_through_plots.py workflow, these will be the files in the “<metric>_histograms” directory where <metric> refers to the beta diversity metric that you’re using. It does look like this bug affects all versions of QIIME.

We’re very sorry for any inconvenience this has caused. We take these issues very seriously, and we’d like to thank the QIIME Forum user “Poom” for bring this to our attention. We’re going to follow up in the QIIME Forum with additional details on this bug. Our plan for getting a fix to users is that we’re going to bump up the priority on the QIIME 1.4.0 release, which will incorporate this fix (as well as a lot of new features). Since we are already close to a release, we decided that this will be a more efficient solution than patching the 1.3.0 release. We’ll keep you posted with details on QIIME 1.4.0, but we expect that this will come out within the next two weeks.

Greg

We posted a new home page at www.qiime.org today. This is in response to users who told us that there are too many QIIME sites to keep track of (the documentation, the blog, the forums, our YouTube help videos page). All of these independent sites are still in the same locations, but you can now easily access them all (plus some new stuff that will be going up soon) from the home page at www.qiime.org. The previous homepage is now the top-level of the QIIME documentation. Enjoy!

Greg

This a major release of QIIME, packed with too many new features to cover them all here. Some highlights are:

* Added support for the RDP Classifier version 2.2 in response to several feature requests. QIIME now supports RDP 2.0 and RDP 2.2.

* Overhauled support for Illumina data in QIIME with a new script, split_libraries_fastq.py. fastq is now the default format for incoming Illumina data to QIIME, and several scripts have been provided to convert other file formats to fastq (process_qseq.py and process_iseq.py). Check out the “Processing Illumina Data” tutorial for an overview of how to work with Illumina data in QIIME.

* Full integration of the QIIME Denoiser (Reeder and Knight, 2010) into Qiime: Denoiser is no longer a stand-alone package. Some enhancements were made to the Denoiser in the process, including check-pointing for failure recovery.

* Added support for AmpliconNoise via the ampliconnoise.py script. We’ve also dropped support for PyroNoise as AmpliconNoise is the successor to that package.

* Added the core_qiime_analyses.py script. This plugs together many components of QIIME (split libraries.py, pick_otus_through_otu_table.py, beta_diversity_through_plots.py, alpha_rarefaction.py, and several others) in an effort to combine the core functionality in a single command. The output of this script includes an html page that serves as an index into the results, to facilitate sharing of results. This script should be considered to be in BETA status. One known issue is no failure recovery – if an analysis fails at some point in the analysis it’s not possible to continue where the failed run left off, possibly resulting in wasted compute time.

* Parameter files have been made optional for all workflow scripts. This greatly simplifies the interface for the workflow scripts when users want to work with default parameters.

* Added the plot_taxa_summary.py workflow script, which includes (optionally) summarizing the OTU table by category, then summarizing taxa for an OTU table, and generating area, bar, and pie charts.

* A lot of updated and new documentation: the QIIME Overview tutorial, the Illumina data processing tutorial, and the Denoiser tutorial have all been overhauled, and a new tutorial has been added that covers processing 18S data with QIIME. A lot of minor documentation changes have been pushed in through-out the code base.

* The beta_diversity_through_3d_plots.py has been renamed to beta_diversity_through_plots.py to reflect additional functionality. It now also generates 2d plots and distance histograms, and any of the plots can be disabled by passing the options –suppress_distance_histograms, –suppress_2d_plots, and –suppress_3d_plots.

* Added inflate_denoiser_output.py script to simplify the integration of denoiser results into the QIIME pipeline. See the Denoiser tutorial for details. To reduce the number of possible pathways through QIIME with denoising (which were difficult to support), support for denoising was removed from pick_otus_through_otu_table.py in favor of working with the pipeline presented in the tutorial.

* Reorganized and renamed output from the workflow scripts for clarity. You’ll notice changes in your results from pick_otus_through_otu_tables.py, pick_reference_otus_through_otu_tables.py, and beta_diversity_through_plots.py. The same files are still created, but with less complex naming of files and a simpler directory structure.

* Added plot_semivariogram.py to plot semivariograms using two distance matrices.

* Added filter_tree.py which prunes a list of tips from a tree. Lists of tips can be provided in similar ways as to filter_fasta.py.

* Added make_tep.py, which creates a TopiaryExplorer project file (.tep) from an otu table, a metadata mapping file, and a tree.

* We’ve posted help videos for getting setup with the QIIME VirtualBox and EC2 which you can find here.

Check out the ChangeLog for a more comprehensive list of the changes in QIIME 1.3.0.

A few additional notes:

* For QIIME VB users: we now recommend making at least 2GB of RAM available to your VirtualBox.

* We’d like to thank Paul Marshall for his work on the app_deploy.py script. This is a very useful tool for installing/upgrading QIIME (and other software) in Linux environments, and was used for building the QIIME 1.3.0 VirtualBox and EC2 instances. It is available for our users to upgrade/install QIIME, but should be considered to be in BETA status – this has been tested in the QIIME VirtualBox, but is not recommended for use on production systems at this stage.

* Due to popular demand, we’ll be adding a page soon that lists courses and workshops that will cover QIIME. We’ll post to the blog when that page goes live.

Thanks to all of the QIIME developers for the hard work that went into QIIME 1.3.0, and to our users for all the feedback that helps us improve QIIME. As always, get in touch on the forums if you run into questions. Have fun!

-Greg

Some of the notable new features are:

• In response to NCBI’s announcement to drop support for SRA, we no longer support submission of data to the SRA using QIIME. We now support submission of data to MG-RAST using the new submit_to_mgrast.py script.
• The make_pie_charts.py script has been replaced with a new script, plot_taxa_summary.py, which in addition to creating pie charts will also create area charts and bar charts. These new plots are particularly useful for looking at how taxonomy changes across time/space gradients. We’ve created a new taxa summary tutorial showing how to use this new script. You can find some example bar plots there.
• Added binary_otu_gain as a new beta diversity metric to compute non-phylogenetic gain (G), or the amount of new OTUs in a sample (or samples) with respect to another sample (or samples). This complements the phylogenetic variant of this metric, unifrac_g.
• Added a reference-based OTU picking workflow script, pick_reference_otus_through_otu_table.py, which performs strict reference-based OTU picking where a pre-existing tree and taxonomy will be used (allowing users to bypass the slow steps where these are created in the pick_otus_through_otu_table.py workflow). This can also be used for applying the Shotgun UniFrac pipeline.
• Changed defaults for uclust and uclust_ref OTU pickers, as described in this blog post.
• Added support for generating inVUE plots in make_3d_plots.py.
• Changed the method for p-value calculation in Procrustes analysis Monte Carlo in response to SF bug # 3189200.

As of QIIME 1.2.1, we moved the EC2 image out of beta testing status, and have now released it as an EBS which means that it is possible to save the state of the machine and (manually) pause it to save money when not actively running jobs. We encourage users to try this out as our initial uses of this have been very successful. This is a great way to get a lot of compute power, relatively cheaply, for your QIIME runs. Information on running QIIME on EC2 is here. You can find the official image by searching for AMI ami-0c12e165. After booting the image in your own amazon account, you’ll then log in as user ‘ubuntu’.

Updated versions of the QIIME VirtualBox and the EC2 image are now available, as is a new version of the QIIME VirtualBox update script. We recommend updating to the latest version of the Oracle VirtualBox software as they seem to have fixed some issues that were present in the previous versions.

Also note that we’ll be maintaining links to the most recent versions of dependencies, as well as our most recent build of the Greengenes references OTUs on the top right corner of the blog homepage so you can always find that information there.

Have fun!

Greg

We were noticing in certain cases (notably the gg_otus_6oct2010) that we had representative sequences that were greater than 97% similar. This led to generation of multiple OTUs that should have been a single OTU, or ‘split OTUs’. The changes we’ve made to address this are: increase max_accepts from 8 to 20; increase max_rejects from 32 to 500; increase stepwords from 8 to 20; and increase word_length from 8 to 12. This appears to get around the ‘split OTU’ problem.

Performance comparisons based on 50,000 to 250,000 sequences (median length: 235) are below (Figures 1 and 2). The increase in run time is not much different for uclust (de novo), but is quite a bit higher for uclust_ref because there is more extensive testing against a very large collection of seeds. These comparisons were performed using an Amazon Web Services m2.4xlarge instance with the QIIME EC2 image (modified to use the SVN version of QIIME).

We’ve also changed the default representative sequence picking method from most_abundant to first. This ensures that the uclust seed is taken as the representative sequence for each OTU. Sequence abundance is taken into account upstream in pick_otus.py now (as it has been since QIIME 1.1.0) using the presort by abundance feature which ensures that abundant sequences are more likely to be used as OTU seeds than less abundant sequences. In practice, the difference in what sequence is chosen as the representative is minimal as the most abundant sequence is always the first in each cluster, but can be thrown off if there is a tie for the most abundant sequence within an OTU.

In this process we’ve also re-picked our greengenes OTUs. The new version, gg_otus_29nov2010, is available from the Greengenes site here.

If you’re using QIIME 1.2.0, you can call pick_otus.py –max_accepts 20 –max_rejects 500 –uclust_stable_sort; pick_rep_set.py -m first to get close to the new defaults in QIIME 1.2.0-dev. The –stepwords and –word_length options are currently available only in the svn version of QIIME, and will go into QIIME 1.3.0. If you’re using SVN QIIME and you’d prefer uclust to run faster, you can also set these values back to the 1.2.0 defaults (which are listed below) on the command line or in a parameters file.

Finally, thanks to Les Dethlefsen at Stanford for initial investigations on what these parameter settings should be for OTU picking!
Greg

Figure 1: Run time of pick_otus.py -m uclust with QIIME 1.2.0 (blue) and 1.2.0-dev (red) parameters.

Figure 2: Run time of pick_otus.py -m uclust -C with QIIME 1.2.0 (blue) and 1.2.0-dev (red) parameters.

In summary:
QIIME 1.2.0-dev defaults:
pick_otus:max_accepts     20
pick_otus:max_rejects     500
pick_otus:stepwords     20
pick_otus:word_length     12
pick_rep_set:rep_set_picking_method first

QIIME 1.2.0 defaults:
pick_otus:max_accepts     8
pick_otus:max_rejects     32
pick_otus:stepwords     8
pick_otus:word_length     8
pick_rep_set:rep_set_picking_method most_abundant

New features include:

* Addition of the new PyCogent SFF tools. Converting SFF files into fasta, qual and flowgram files is now supported through process_sff.py without dependence on Roche’s sff tools.

* Ability to add jackknife support to 2D and 3D PCoA plots as seen in Figure 2 of this article. This feature is now accessible via the jackknifed_beta_diversity.py script, which has replaced jackknifed_upgma.py.

* Added a parallel version of the uclust_ref OTU picker (parallel_pick_otus_uclust_ref.py).

* Added ability to run beta diversity calculations in parallel at the single OTU table level to improve performance when computing diversity on very large collections of samples. This functionality is now hooked up to the beta_diversity_through_3d_plots.py workflow script.

* Added several new pages to the documentation including a Procrustes analysis/plotting tutorial; a tutorial for identifying and removing chimeric sequences; and improved documentation on common Qiime input file formats.

* Added new script (shared_phylotypes.py) for computing shared OTUs between pairs of samples.

* Add script (plot_rank_abundance_graph.py) to draw rank abundance graphs.

* Added script (quality_scores_plot.py) to plot quality score by position given a .qual file. This is useful with another new script (truncate_fasta_qual_files.py) to truncate fasta and qual files at the point where quality begins to decrease. This has been useful in controlling for quality issues on 454 Ti runs.

* Added capability to perform supervised classification of metadata categories using the Random Forests classifier (supervised_learning.py). Outputs include a ranking of OTUs by discriminatory power, and the estimated probability of each metadata category for each sample. The latter may be useful for detecting potentially mislabeled samples. The technique was recently used here.

We’re also excited to announce that we’ve created a QIIME EC2 image that will be going live this week. This will allow Amazon Web Services run their QIIME analyses in the cloud. This has been particularly useful so far for denoising data as this is typically the performance bottleneck in QIIME analyses. We hope that this will generally be useful for users who need additional compute resources to support their big analyses.

As the majority of our users are working on 64-bit machines, we’ve dropped support for the 32-bit QIIME VirtualBox in QIIME 1.2.0. Supporting both versions was becoming too time-consuming. Some notes on this are available here. We’re sorry for any inconvenience this may cause.

For QIIME VB users, we’ve created a VB upgrade script for upgrading the 64-bit Qiime VirtualBox. You can find information on obtaining and running this script here.

For this release of the QIIME VB and EC2 image we used the CloVR package to build our virtual machine images. We’d like to thank Samuel Angiuoli and James White of the Institute for Genome Sciences for their help in getting us up and running with CloVR! We’d also like to thank our users for the continued feedback on QIIME: your input is critical to making QIIME a useful and usable toolkit.

As always, get in touch on the QIIME forum with any questions or comments. Have fun!

Greg

Bug in reference-based uclust (i.e., uclust_ref)

Most importantly, we identified an issue with uclust that resulted in some sequences being lost when picking OTUs against a reference set with the serial version of uclust_ref, when allowing for new clusters and there are greater than 100k seeds. The call that would possibly have been affected is pick_otus.py -m uclust_ref; but not parallel_pick_otus_uclust_ref.py;  pick_otus.py -m uclust; or pick_otus.py -m uclust_ref –suppress_new_clusters.

If you suspect that a run may have been affected, get in touch on the QIIME forum and we’ll give you a script to check your OTU map. A new version of uclust has been posted to fix this, and as of QIIME 1.2.0 (and the current svn version) this will be the minimum supported version of uclust. Here are links to the uclust 1.2.21q binaries. You should install this ASAP.

QIIME-compatible Greengenes files

Next, we’re excited to announce that Greengenes is now hosting QIIME-compatible reference sequences and taxonomy files. These are frequently requested for use with QIIME’s BLAST taxon assigner (they get passed as  -r and -t, respectively). The reference file is based on 97% OTUs that we picked from greengenes. This is also useful for reference-based OTU picking (e.g., with uclust_ref or blast), in which case the included tree can be used in phylogenetic diversity calculations. Our 6oct2010 build of the greengenes OTUs is available here, and additional notes on that build are available here.

Coming soon: QIIME 1.2.0

Finally, we’re about to make the push toward QIIME 1.2.0, which we hope to release within the first couple of weeks in November. We’ll be releasing a new VB image, but also providing an upgrade script for existing VB users so you won’t have to start fresh with a new VB image if you don’t want to. There are lots of exciting changes coming in 1.2.0 — more on that soon.

-Greg

http://groups.google.com/group/qiime-forum

The QIIME Forum will be the fastest way to get help with QIIME. We’ll be aiming to respond to questions within one working day. The qiime.help@colorado.edu address will still be monitored, but should be used only for questions that would reveal data that is still private. As we’re focusing our support efforts on the forum, we’ll aim to respond to qiime.help e-mail questions within three to five working days.

Anyone can post at the QIIME Forum: you don’t need to be a member. If you do sign up as a member of the forum, we suggest that you set your message delivery settings to ‘No e-mail’ to avoid getting messages from the list that are not relevant to your question. You can sign up for e-mail notification on a per-message basis to monitor threads that are of interest to you.

Post to the forum if you have any questions, and thanks for your interest in QIIME!

http://www.drive5.com/uclust/downloads1_2_15.html

All users who use PyNAST for aligning sequences (the default) or uclust or uclust_ref for picking OTUs (uclust is the default for pick_otus.py) should install this update as soon as possible.

The primary change addresses an issue where sequences would cluster independently based on the case of the sequence. So for example, in uclust 1.1.579, the sequences ‘ACCGT’ and ‘accgt’ would cluster independently.

Users of the Qiime releases may notice a unit test failure after the upgrade. This has to do with a test for a valid version of uclust. If you get the following error when running test/test_pycogent_backports/test_uclust.py:

==========================
ERROR: uclust version is supported
----------------------------------------
Traceback (most recent call last):
File "tests/test_pycogent_backports/test_uclust.py", line 445, in test_uclust_supported_version
version = tuple(map(int,version_string.split('.')))
ValueError: invalid literal for int() with base 10: 'exportedq'

You don’t need to worry about it. For svn users, a fix has been added to the repository so this test will pass after your next svn update.

As always, get in touch with qiime.help@colorado.edu if you have any questions.

-Greg