Automating Assay Data Registration in the Vincent Lab

Scientists work on a wide variety of subjects and techniques, generating heterogeneous data that often requires custom registration through the import wizard. Projects involving the systematic processing of many samples with a stable workflow (fixed protocols, fixed equipment, and assay setup) offer the possibility to automate data and assay registration.

Here we demonstrate how data registration can be automated with a concrete project conducted with the Vincent Lab at EMBL Heidelberg.

Scientific background

Diatoms are microscopic marine organisms that play a crucial role in the Earth’s ecosystems by conducting about 20% of the planet’s annual photosynthesis and supporting marine food chains. They have various tight relationships with other organisms, some of which are mutually beneficial, while others are harmful, all of which are vital for their growth and with an overall impact on the global environment. Surprisingly, even though we know of around 20,000 different diatom species, we have only discovered fewer than 200 of these relationships, indicating that there are many more waiting to be found. Recent research has pointed out that our methods for uncovering these diatom interactions have limitations. So, we believe that these diatom symbioses are more common than previously thought and are significant for diatom ecology and therefore our ecosystem’s functioning, especially in changing environments.

The Vincent Lab uses cutting-edge techniques to systematically unravel these tiny partnerships in natural marine samples, starting with diatoms. They use a treasure trove of samples collected during scientific expeditions on the Tara schooner, preserved in a way that allows them to study the organisms and their partnerships at the single-cell level without altering them. The Vincent Lab combines high-throughput feedback microscopy, image-enabled cell sorting, and downstream single-cell omics to resolve organismal taxonomy and thereby reveal the breadth of microbial symbioses in the open ocean.

To achieve their goals, the Vincent Lab screens many Tara samples using the same workflow and protocols. Such a setup is the perfect use case for automated data registration in LabID: it enforces a data structure that matches the data management plan, guarantees data traceability, and reduces the time devoted to data management operations, thereby relieving the team of the burden of manually registering data.

Data storage strategy

We opted for the following strategy:

• The original Tara samples are registered as Sample and the sorted plates as Sequencing Library, with each Sequencing Library plate being a child of a Tara Sample
• The image-enabled sorting assays and their digital outputs (experimental metadata, images…) should be loaded as Light Microscopy Assay. Each assay takes a single plate as input and defines the different digital outputs (experimental metadata, images…) as output datasets.
• The single-cell sequencing should be loaded as Illumina Sequencing Assay with one or more plates as input, and the demultiplexed FastQ files as output datasets.

Samples and Plates creation

All the Tara samples have been preloaded in LabID using Excel-based batch import. Each time a sample is sorted into a plate, a new sequencing library reflecting this plate should be manually created. This operation is easily done using the Create Child button available on the Tara sample page.

Automated registration of the sorting assays

To automatically register the image-enabled sorting assays and their digital outputs, we use the CLI register batch command with a custom sniffer plug-in whose role is to parse the folder containing all the files created by the particle sorter. This implies that the content of this folder is always organised the same way, and that files/folders follow a predefined naming convention. Such requirements allow for automated data validation (all expected data found and named according to expectations) before registration.

The implementation of the copas_vision_loader sniffer is visible in the LabID CLI project (plugins module).

The copas_vision_loader sniffer specifications

This sniffer expects to run on a folder containing the COPAS Vision results for one plate with the following structure:

• A COPAS_Exp folder containing the original COPAS Vision files (.bxr4, .lmd, .txt, .dcimg)
• A Dispensed_TIFFS folder containing the expected number of images. All images must be of the same format (e.g. tiff) as indicated by their file extension.
• A All_profiles folder containing the exported txt channel profiles

Naming convention:

• The run folder (given as input) name must be the plate name and is validated using the sample_name_regex (plate_name = sample_name_sorting-date_SCreplicate-number), where the sample_name is the plate’s parent sample name, sorting-date is the date of the plate sorting (YYYYMMDD) and replicate-number is an incremental integer. The corresponding SequencingLibrary (same name) must be found in LabID.
• The COPAS Vision file (.bxr4, .lmd, .txt, .dcimg) names are expected to start with the plate_name (plate_name.*.bxr4)
• The image file names must end with the well position (e.g F4) i.e. .+_well_position.Tiff
• The channel profile file names have the .txt extension i.e. ch_number.txt

Setting up the periodic task

The automation aspect is achieved by automatically running the labid register batch command periodically (e.g. every hour) using a job scheduler like cron. A few aspects must be considered:

Which user should be used to connect to LabID?

This user is used to submit the data and therefore ends up as the data owner. Here we recommend creating a technical user with a permanent API key. This helps avoid potential permission issues when the user owning the data leaves the lab.

→ We created a new member of the Vincent Group named vincentagent.

Where should the monitored directory be created?

As explained in the CLI register batch command use case, the assay run folders must be placed in a monitored directory that is given to the labid register batch command using the --root-dir option. LabID requires this monitored directory to be in the dropbox of the user submitting the data.

→ We created a copasvision directory in the vincentagent dropbox.

Which user should be used to execute the periodic task?

The user running the periodic task must be able to read the files dropped in the copasvision directory in the vincentagent dropbox. This user should therefore be a lab member. It makes sense to use a technical user to avoid issues when lab members leave the lab.

→ In our setup, we have a unique labidloader user to execute all the periodic tasks for all the groups.

Finally, the labid register batch periodic task should be installed on a stable server (you may ask your IT department for this) and run with the vincentagent user credentials using the --config-path option.

Automated registration of the sequencing assays

A similar approach (using the CLI register batch command) may be used to register the sequencing assays and data with a custom sniffer. However, at EMBL we already integrated the Genomics Core Facility with LabID such that sequencing assays are automatically pre-registered in LabID.

Briefly, the Genomics Core Facility transfers the demultiplexed sequencing data together with assay metadata (as a JSON document) to the LabID data repository; and an Illumina Sequencing Assay in initialized state is automatically created in LabID on behalf of the data owner. The data owner receives an email containing a link to the initialized assay page where the user can finalize the assay registration using the interactive data registration wizard.

This means that, at EMBL, the plate sequencing results and details are automatically registered in LabID; still the registration of these initialized assays must be completed i.e. validated. This validation can also be automated using the CLI validate batch command and a custom sniffer. The CLI code base contains the PooledIlluminaInitializedAssayValidator sniffer that is extended to achieve the needed behaviour:

• Define the regular expression to validate the sample name
• Enforce that the SequencingLibrary exists in LabID
• Provide a custom implementation of the dir_qualifies method used to first assess if the initialized assay should be processed by this sniffer

→ The custom TRECSequencingPlateAssayValidator sniffer class is available as a plugin (trec_scplate_validator) in the LabID CLI project (plugins module).

Here again, the automation aspect is achieved by automatically running the labid validate batch command periodically as the labidloader user and using the vincentagent user credentials to connect to LabID.

Enforcing naming convention and other prerequisites

A sniffer extracts metadata about the assay, the sample names and their relationships with the datasets and represents this information using the LabID object model. Many options are possible when a sniffer inspects the data. For example, a sniffer may check that:

• The data is organized in the expected folder structure
• The folders contain the expected number of files
• The files are named according to the expected convention
• Items such as samples or instruments already exist in LabID, and reuse them instead of creating new ones

Final Set-Up