Your sequencing data will made available in the standard FASTQ file format. We also provide some smaller files giving information about those files, and a report for each lane of sequencing.
Processing the sequenced run folders is done using Illumina's bcl2fastq program. Further details on this tool can be found on the Illumina web site.
Your data will be demultiplexed according to the information supplied in your submission for sequencing. What you receive will depend on the type of sequencing done.
If one is downloading files outside of the CRUK-CI institute, it is strongly recommended that one checks the files have no corruption during transfer (usually truncation rather than corruption). We provide checksums for the FASTQ files, which can be used to make sure all the files have transferred properly.
You will normally receive one (single read) or two (paired end) FASTQ files per sample for each lane of sequencing, plus one or two additional FASTQ files that contain the reads that the demultiplexer (Illumina's bcl2fastq) cannot assign to any sample (what we call "lost reads"). The files will be named according to the pattern:
<SLX>.<barcode>.<flowcell>.s_<lane>.r_<read>.fq.gz
A small subset of kits (library types) also return an index read: a separate FASTQ file or pair of files containing the index reads with quality scores. These files will be named with an "i" instead of the "r" for the index read number:
<SLX>.<barcode>.<flowcell>.s_<lane>.i_<index read>.fq.gz
The lost reads file will have a different naming pattern:
<SLX>.<flowcell>.s_<lane>.lostreads.fq.gz
There will also be checksum files for each sample and the lost reads.
<SLX>.<barcode>.<flowcell>.s_<lane>.md5sums.txt <SLX>.<flowcell>.s_<lane>.lostreads.md5sums.txt
External users of our service in particular should use the checksums to make sure your data files have copied to your local systems without corruption or truncation. See the section below for instructions on how to do this.
If your submission requested custom indexing, you will receive one set of files containing all your reads. The barcode will be "UnspecifiedIndex". You will always receive one or two index FASTQ files with custom indexing. There is no lost reads file (all the reads are in the sample FASTQ). The files you can expect are:
<SLX>.UnspecifiedIndex.<flowcell>.s_<lane>.r_<read>.fq.gz <SLX>.UnspecifiedIndex.<flowcell>.s_<lane>.i_<index read>.fq.gz <SLX>.UnspecifiedIndex.<flowcell>.md5sums.txt
Similar to custom indexing, you will receive one set of files containing all your reads. The barcode will be "NoIndex" or "INLINE". There is no index information so there will be no index reads, nor lost reads. The files you can expect are:
<SLX>.NoIndex.<flowcell>.s_<lane>.r_<read>.fq.gz <SLX>.NoIndex.<flowcell>.md5sums.txt
As of November 2020 we supply 10x data as a set of FASTQ files in the same manner as any other regular multiplexed library type, for both single index (SI-GA and SI-NA) and dual index (TT/NT/TN) kits. If you have received your single cell data files as compressed FASTQ files, refer to the regular sequencing section. Also refer to the section on file name conversion if these files need to be renamed for 10x downstream pipelines.
The FASTQ files you receive for SI-GA and SI-NA indexing will have all four index sequences for each barcode combined in the set of FASTQ files for the sample. A quick run through a demultiplexer will confirm the presence of the four indexes at approximately 25% of reads for each one.
The rest of this section refers to single cell sequencing before November 2020, when single cell data was delivered as a TAR file containing twelve or sixteen FASTQ files. If you have received a TAR file per sample, continue reading.
With the single index SI-GA and SI-NA indexing (and indeed previous, now discontinued, 10x kits), each barcode is actually four individual barcode sequences, so to associate what will be four sets of FASTQ files with the single sample given in submission the FASTQ is put into a tar archive. There will be a checksum file for the archive. The files will be:
<SLX>.<barcode>.<flowcell>.s_<lane>.tar <SLX>.<barcode>.<flowcell>.s_<lane>.md5sums.txt <SLX>.<flowcell>.s_<lane>.lostreads.tar <SLX>.<flowcell>.s_<lane>.lostreads.md5sums.txt
Inside each archive there will be twelve or sixteen FASTQ files. These will be named as they were produced by bcl2fastq, which should allow 10x's Cell Ranger and Long Ranger to read the files as if they had been produced by these tools' demux pipelines. There will be FASTQ files for the index reads too.
For example, for a sample that was labelled with the 10x SINA H9 barcode, the tar archive will contain:
SIGAH9_S5_L001_I1_001.fastq.gz SIGAH9_S5_L001_R1_001.fastq.gz SIGAH9_S5_L001_R2_001.fastq.gz SIGAH9_S6_L001_I1_001.fastq.gz SIGAH9_S6_L001_R1_001.fastq.gz SIGAH9_S6_L001_R2_001.fastq.gz SIGAH9_S7_L001_I1_001.fastq.gz SIGAH9_S7_L001_R1_001.fastq.gz SIGAH9_S7_L001_R2_001.fastq.gz SIGAH9_S8_L001_I1_001.fastq.gz SIGAH9_S8_L001_R1_001.fastq.gz SIGAH9_S8_L001_R2_001.fastq.gz
The sample number (the second part of the file name, "S5" etc) is an internal sample number generated by bcl2fastq that has no significance.
There will be some additional files delivered with the FASTQ data. These hold some statistics and QC test results that may be useful.
We deliver with the data a report that we also use to ensure the sequencing has gone as expected. As of summer 2020, this is a MultiQC report containing the individual reports previously delivered separately.
This report will be named <SLX>.<flowcell>.s_<lane>.html.
Some tools require the FASTQ files to be named as they would be when delivered by bcl2fastq, not as we provide them. 10x's pipelines are a notable example. To this end, the script crukci_to_illumina.py will convert a directory containing our FASTQ files to names as they would have been immediately after bcl2fastq.
It changes our file names to the pattern:
<barcode>_S<number>_L00<lane>_[IR]<read>_001.fastq.gz
barcode is the barcode as it appears in our file name; number is an arbitrary sample number that bcl2fastq adds to the file name (it corresponds to a row in the sample sheet but really it counts for little); lane is the lane number; [IR] is either 'R' for regular reads or 'I' for index reads; read is the read number.
The tool can be run with the command:
python3 crukci_to_illumina.py [<fastq directory>]
If fastq directory is not given, the script will look at files in the current directory. It does not recurse into subdirectories.
There are two methods we employ to allow users of the sequencing service to retrieve their sequencing data. Persons working inside CRUK-CI should use our data download tool; everyone else must fetch their data from our FTP site.
We provide a tool for downloading files for projects, libraries and runs that you can use from the command line. The full user manual for the download tool can be found on this internal web page. Please visit the page to download the tool, find instructions for how to use it and also how to install Java on your personal machine (link requires you to be in the building or running the VPN).
Outside of CRUK-CI, data is delivered through the FTP site: ftp1.cruk.cam.ac.uk.
This is a vanilla FTP site that should be accessible by any FTP client you choose. Your group will have been provided a user name and password for accessing the site when your group arranged to use the CRUK-CI sequencing service. Your data will be in a private region of the server only accessible with your group's credentials. The site is read only.
Files are available on the FTP site for a guaranteed thirty (30) days after sequencing. You MUST fetch the files in this time period. The files will be removed from the FTP site once this time has elapsed.
It is your responsibility thereafter to look after the files and store them as you see fit. We do not hold external groups' data in the CRUK-CI archives.
It is possible that the files do not transfer correctly from the FTP site on the first go. We've never really known of corruption during a transfer but truncated files do happen, where the connection is dropped before the transfer is complete. External users of the service must check the transfer is accurate after taking a copy. We provide checksums for the FASTQ files, and one can make sure that the local copy of the files match the original by running the md5sum command once the download has finished.
The checksum files have the suffix ".md5sums.txt" and can be used to check the FASTQ by running:
md5sum -c *.md5sums.txt
Any files that are not reported as "OK" should be downloaded again and rechecked.
There are many FTP clients available on the web one can use to fetch files from our FTP site. Some clients are:
On Linux, you might find that these programs are available through the platform's package management system. Most web browsers will also allow you to navigate the FTP site if you use the FTP protocol in the address bar: ftp://ftp1.cruk.cam.ac.uk/. The browser will prompt for your user name and password. The web browser is handy for having a look at your area of the server and previewing the reports but a proper FTP program is recommended for fetching the files.
We have become aware of some users using the Mac's Finder application to connect to the FTP server and copy the files. While convenient, it appears that Finder can silently truncate files while copying if the connection to the FTP server drops. Thus we do not recommend using Finder or Windows Explorer to copy the files: use a proper FTP program that will report errors. Above all, and regardless of the program used, you must check your files against the checksums after downloading as described above.
Our FTP server is pretty much as vanilla as they come and should not cause any problems. Nonetheless, occasionally people do tell us they cannot connect, and so far this has always been problems at the client end. Here are some things to check.