Develop Reference

Snakemake pipeline using directories and files

I am building a snakemake pipeline with python scripts.
Some of the python scripts take as input a directory, while others take as input files inside those directories.
I would like to be able to do have some rules which take as input the directory and some that take as input the files. Is this possible?
Example of what I am doing showing only two rules:
FILES = glob.glob("data/*/*raw.csv")
FOLDERS = glob.glob("data/*/")
rule targets:
input:
processed_csv = expand("{files}raw_processed.csv", files =FILES),
normalised_csv = expand("{folders}/normalised.csv", folders=FOLDERS)
rule process_raw_csv:
input:
script = "process.py",
csv = "{sample}raw.csv"
output:
processed_csv = "{sample}raw_processed.csv"
shell:
"python {input.script} -i {input.csv} -o {output.processed_csv}"
rule normalise_processed_csv:
input:
script = "normalise.py",
processed_csv = "{sample}raw_processed.csv" #This is input to the script but is not parsed, instead it is fetched within the code normalise.py
params:
folder = "{folders}"
output:
normalised_csv = "{folders}/normalised.csv" # The output
shell:
"python {input.script} -i {params.folder}"
Some python scripts (process.py) take all the files they needed or produced as inputs and they need to be parsed. Some python scripts only take the main directory as input and the inputs are fetched inside and the outputs are written on it.
I am considering rewriting all the python scripts so that they take the main directory as input, but I think there could be a smart solution to be able to run these two types on the same snakemake pipeline.
Thank you very much in advance.
P.S. I have checked and this question is similar but not the same: Process multiple directories and all files within using snakemake

I would like to be able to do have some rules which take as input the directory and some that take as input the files. Is this possible?
I don't see anything special with this requirement... What about this?
rule one:
output:
d=directory('{sample}'),
a='{sample}/a.txt',
b='{sample}/b.txt',
shell:
r"""
mkdir -p {output.d}
touch {output.a}
touch {output.b}
"""
rule use_dir:
input:
d='{sample}',
output:
out='outdir/{sample}.out',
shell:
r"""
cat {input.d}/* > {output.out}
"""
rule use_files:
input:
a='{sample}/a.txt',
b='{sample}/b.txt',
output:
out='outfiles/{sample}.out',
shell:
r"""
cat {input.a} {input.b} > {output.out}
"""
rule use_dir will use the content of directory {sample}, whatever it contains. Rule use_files will use specifically files a.txt and b.txt from directory {sample}.

How to handle ftp links provided in config file in snakemake?

I am attempting to build a snakemake workflow that will provide a symlink to a local file if it exists or if the file does not exist will download the file and integrate it into the workflow. To do this I am using two rules with the same output with preference given to the linking rule (ln_fastq_pe below) using ruleorder.
Whether the file exists or not is known before execution of the workflow. The file paths or ftp links are provided in tab-delimited config file that is used by the workflow to read in samples.
e.g. the contents of samples.txt:
id sample_name fq1 fq2
b test_paired resources/SRR1945436_1.fastq.gz resources/SRR1945436_2.fastq.gz
c test_paired2 ftp://ftp.sra.ebi.ac.uk/vol1/fastq/SRR194/005/SRR1945435/SRR1945435_1.fastq.gz ftp://ftp.sra.ebi.ac.uk/vol1/fastq/SRR194/005/SRR1945435/SRR1945435_2.fastq.gz
relevant code from the workflow here:
import pandas as pd
from snakemake.remote.FTP import RemoteProvider as FTPRemoteProvider
FTP = FTPRemoteProvider()
configfile: "config/config.yaml"
samples = pd.read_table("config/samples.tsv").set_index("id", drop=False)
all_ids=list(samples["id"])
ruleorder: ln_fastq_pe > dl_fastq_pe
rule dl_fastq_pe:
"""
download file from ftp link
"""
input:
fq1=lambda wildcards: FTP.remote(samples.loc[wildcards.id, "fq1"], keep_local=True),
fq2=lambda wildcards: FTP.remote(samples.loc[wildcards.id, "fq2"], keep_local=True)
output:
"resources/fq/{id}_1.fq.gz",
"resources/fq/{id}_2.fq.gz"
shell:
"""
mv {input.fq1} {output[0]}
mv {input.fq2} {output[1]}
"""
rule ln_fastq_pe:
"""
link file
"""
input:
fq1=lambda wildcards: samples.loc[wildcards.id, "fq1"],
fq2=lambda wildcards: samples.loc[wildcards.id, "fq2"]
output:
"resources/fq/{id}_1.fq.gz",
"resources/fq/{id}_2.fq.gz"
shell:
"""
ln -sr {input.fq1} {output[0]}
ln -sr {input.fq2} {output[1]}
"""
When I run this workflow, I receive the following error pointing to the line describing the ln_fastq_pe rule.
WorkflowError in line 58 of /path/to/Snakefile:
Function did not return str or list of str.
I think the error is in how I am describing the FTP links in the samples.txt config file in the dl_fastq_pe rule. What is the proper way to describe FTP links given in a tabular config file so that snakemake will understand them and can download and use the files in a workflow?
Also, is it possible to do what I am trying to do and will this method get me there? I have tried other solutions (e.g. using python code to check if file exists and executing one set of shell commands if it does and the other if it doesn't) to no avail.

You are trying to pass the objects from pandas to Snakemake. The latter expects the values of types str or list[str] in the input section of the rule, but the values you provide (samples.loc[wildcards.id, "fq1"]) are of type pandas.core.frame.DataFrame or pandas.core.series.Series. You need to convert them to what Snamemake expects. For example, this may help: samples.loc[wildcards.id, "fq1"].tolist().

I figured out how to do this by omitting input and instead reading in the fields from samples.tsv through params and merging the two rules into one rule. Snakemake is not picky about what is read in through params unlike input. I then use test command to ask if a file exists. If it exists, proceed with the symlink, if not, download with wget.
Solution is as follows:
import os
import pandas as pd
samples = pd.read_table("config/samples.tsv").set_index("id", drop=False)
all_ids=list(samples["id"])
rule all:
input:
expand("resources/fq/{id}_1.fq.gz", id=all_ids),
expand("resources/fq/{id}_2.fq.gz", id=all_ids)
rule dl_fastq_pe:
"""
if file exists, symlink. If file doesn't exist, download to resources
"""
params:
fq1=lambda wildcards: samples.loc[wildcards.id,"fq1"],
fq2=lambda wildcards: samples.loc[wildcards.id,"fq2"]
output:
"resources/fq/{id}_1.fq.gz",
"resources/fq/{id}_2.fq.gz"
shell:
"""
if test -f {params.fq1}
then
ln -sr {params.fq1} {output[0]}
ln -sr {params.fq2} {output[1]}
else
wget --no-check-certificate -O {output[0]} {params.fq1}
wget --no-check-certificate -O {output[1]} {params.fq2}
fi
"""

MissingOutputException with Snakemake

I'm having the following problem:
My Snakemake program does not recognise the output my python script generated. I tried both, writing the ouput to the stdout and from there into the correct output file and directly writing from the python script (which is the version below).
Setting --latency-wait to 600 did not help either.
Other users reported that running ls helped which I tried while waiting for the latency but that didn't help either.
Additionally, when running again,snakemake wants to run all input files again, despite some output files already existing.
Does anyone have a suggestion what else I could try?
This is the snakemake command I'm using:
snakemake -j 2 --use-conda
Below is my snakefile:
import os
dir = "my/data/dir"
cell_lines = os.listdir(dir)
files = os.listdir(dir+"GM12878/25kb_resolution_intrachromosomal")
chromosomes = [i.split("_")[0] for i in files]
rule all:
input:
expand("~/TADs/{cell_lines}_{chromosomes}_TADs.tsv", cell_lines = cell_lines, chromosomes = chromosomes)
rule tad_calling:
input:
"my/data/dir/{cell_lines}/25kb_resolution_intrachromosomal/{chromosomes}_25kb.RAWobserved"
output:
"~/TADs/{cell_lines}_{chromosomes}_TADs.tsv"
benchmark:
"benchmarks/{cell_lines}_{chromosomes}.txt"
conda:
"my_env.yaml"
shell:
"""
python ~/script.py {input} {output}
"""

I think the problem is with the tilde (~), snakemake does not expand those (or e.g. $HOME). It takes those as the literal path. You could do something like:
from pathlib import Path
home = str(Path.home())
rule tad_calling:
...
output:
f"{home}/TADs/{cell_lines}_{chromosomes}_TADs.tsv"
...

Missing input files for rule all in snakemake

I am trying to construct a snakemake pipeline for biosynthetic gene cluter detection but am struggling with the error:
Missing input files for rule all:
antismash-output/Unmap_09/Unmap_09.txt
antismash-output/Unmap_12/Unmap_12.txt
antismash-output/Unmap_18/Unmap_18.txt
And so on with more files. As far as I can see the file generation in the snakefile should be working:
workdir: config["path_to_files"]
wildcard_constraints:
separator = config["separator"],
extension = config["file_extension"],
sample = config["samples"]
rule all:
input:
expand("antismash-output/{sample}/{sample}.txt", sample = config["samples"])
# merging the paired end reads (either fasta or fastq) as prodigal only takes single end reads
rule pear:
input:
forward = "{sample}{separator}1.{extension}",
reverse = "{sample}{separator}2.{extension}"
output:
"merged_reads/{sample}.{extension}"
conda:
"~/miniconda3/envs/antismash"
shell:
"pear -f {input.forward} -r {input.reverse} -o {output} -t 21"
# If single end then move them to merged_reads directory
rule move:
input:
"{sample}.{extension}"
output:
"merged_reads/{sample}.{extension}"
shell:
"cp {path}/{sample}.{extension} {path}/merged_reads/"
# Setting the rule order on the 2 above rules which should be treated equally and only one run.
ruleorder: pear > move
# annotating the metagenome with prodigal#. Can be done inside antiSMASH but prefer to do it out
rule prodigal:
input:
"merged_reads/{sample}.{extension}"
output:
gbk_files = "annotated_reads/{sample}.gbk",
protein_files = "protein_reads/{sample}.faa"
conda:
"~/miniconda3/envs/antismash"
shell:
"prodigal -i {input} -o {output.gbk_files} -a {output.protein_files} -p meta"
# running antiSMASH on the annotated metagenome
rule antiSMASH:
input:
"annotated_reads/{sample}.gbk"
output:
touch("antismash-output/{sample}/{sample}.txt")
conda:
"~/miniconda3/envs/antismash"
shell:
"antismash --knownclusterblast --subclusterblast --full-hmmer --smcog --outputfolder antismash-output/{wildcards.sample}/ {input}"
This is an example of what my config.yaml file looks like:
file_extension: fastq
path_to_files: /home/lamma/ABR/Each_reads
samples:
- Unmap_14
- Unmap_55
- Unmap_37
separator: _
I can not see where i am going wrong within the snakefile to produce such an error. Apologies for the simple question, I am new to snakemake.

The problem is that you setup your global wildcard constraints wrong:
wildcard_constraints:
separator = config["separator"],
extension = config["file_extension"],
sample = '|'.join(config["samples"]) # <-- this should fix the problem
Then immediatly another problem follows with extension and seperator wildcards. Snakemake can only infer what these should be from other filenames, you can not actually set these through wildcard constraints. We can make use of f-string syntax to fill in what the values should be:
rule pear:
input:
forward = f"{{sample}}{config['separator']}1.{{extension}}",
reverse = f"{{sample}}{config['separator']}2.{{extension}}"
...
and:
rule prodigal:
input:
f"merged_reads/{{sample}}.{config['file_extension']}"
...
Take a look at snakemake regex if the wildcard constraints confuse you, and find a blog about f-strings if you are confused about the f"" syntax and when to use single { and when to use double {{ to escape them.
Hope that helps!

(Since I can't comment yet ...)
You might have a problem with your relative paths, and we cannot see where your files actually are found.
A way to debug this is to use config["path_to_files"] to create absolute paths in input:
That would give you better error message on where Snakemake expects the files - input/output files are relative to the working directory.

Input Wildcard Constraints Snakemake

I am new to snakemake, and I am using it to merge steps from two pipelines into a single larger pipeline. The issue that several steps create similarly named files, and I cannot find a way to limit the wildcards, so I am getting a Missing input files for rule error on one step that I just cannot resolve.
The full snakefile is long, and is available here: https://mdd.li/snakefile
The relevant sections are (sections of the file are missing below):
wildcard_constraints:
sdir="[^/]+",
name="[^/]+",
prefix="[^/]+"
# Several mapping rules here
rule find_intersecting_snps:
input:
bam="hic_mapping/bowtie_results/bwt2/{sdir}/{prefix}_hg19.bwt2pairs.bam"
params:
hornet=os.path.expanduser(config['hornet']),
snps=config['snps']
output:
"hic_mapping/bowtie_results/bwt2/{sdir}/{prefix}_hg19.bwt2pairs.remap.fq1.gz",
"hic_mapping/bowtie_results/bwt2/{sdir}/{prefix}_hg19.bwt2pairs.remap.fq2.gz",
"hic_mapping/bowtie_results/bwt2/{sdir}/{prefix}_hg19.bwt2pairs.keep.bam",
"hic_mapping/bowtie_results/bwt2/{sdir}/{prefix}_hg19.bwt2pairs.to.remap.bam",
"hic_mapping/bowtie_results/bwt2/{sdir}/{prefix}_hg19.bwt2pairs.to.remap.num.gz"
shell:
dedent(
"""\
python2 {params.hornet}/find_intersecting_snps.py \
-p {input.bam} {params.snps}
"""
)
# Several remapping steps, similar to the first mapping steps, but in a different directory
rule wasp_merge:
input:
"hic_mapping/bowtie_results/bwt2/{sdir}/{prefix}_hg19.bwt2pairs.keep.bam",
"hic_mapping/wasp_results/{sdir}_{prefix}_filt_hg19.remap.kept.bam"
output:
"hic_mapping/wasp_results/{sdir}_{prefix}_filt_hg19.bwt2pairs.filt.bam"
params:
threads=config['job_cores']
shell:
dedent(
"""\
{module}
module load samtools
samtools merge --threads {params.threads} {output} {input}
"""
)
# All future steps use the name style wildcard, like below
rule move_results:
input:
"hic_mapping/wasp_results/{name}_filt_hg19.bwt2pairs.filt.bam"
output:
"hic_mapping/wasp_results/{name}_filt_hg19.bwt2pairs.bam"
shell:
dedent(
"""\
mv {input} {output}
"""
)
This pipeline is essentially doing some mapping steps in one directory structure that looks like hic_mapping/bowtie_results/bwt2/<subdir>/<file>, (where subdir is three different directories) then filtering the results, and doing another almost identical mapping step in hic_remap/bowtie_results/bwt2/<subdir>/<file>, before merging the results into an entirely new directory and collapsing the subdirectories into the file name: hic_mapping/wasp_results/<subdir>_<file>.
The problem I have is that the wasp_merge step breaks the find_intersecting_snps step if I collapse the subdirectory name into the filename. If I just maintain the subdirectory structure, everything works fine. Doing this would break future steps of the pipeline though.
The error I get is:
MissingInputException in line 243 of /godot/quertermous/PooledHiChip/pipeline/Snakefile:
Missing input files for rule find_intersecting_snps:
hic_mapping/bowtie_results/bwt2/HCASMC5-8_HCASMC-8-CAGAGAGG-TACTCCTT_S8_L006/001_hg19.bwt2pairs.bam
The correct file is:
hic_mapping/bowtie_results/bwt2/HCASMC5-8/HCASMC-8-CAGAGAGG-TACTCCTT_S8_L006_001_hg19.bwt2pairs.bam
But it is looking for:
hic_mapping/bowtie_results/bwt2/HCASMC5-8_HCASMC-8-CAGAGAGG-TACTCCTT_S8_L006/001_hg19.bwt2pairs.bam
Which is not created anywhere, nor defined by any rule. I think it is somehow getting confused by the existence of the file created by the wasp_merge step:
hic_mapping/wasp_results/HCASMC5-8_HCASMC-8-CAGAGAGG-TACTCCTT_S8_L006_001_filt_hg19.bwt2pairs.filt.bam
Or possibly a downstream file (after the target that creates this error):
hic_mapping/wasp_results/HCASMC5-8_HCASMC-8-CAGAGAGG-TACTCCTT_S8_L006_001_filt_hg19.bwt2pairs.bam
However, I have no idea why either of those files would confuse the find_intersecting_snps rule, because the directory structures are totally different.
I feel like I must be missing something obvious, because this error is so absurd, but I cannot figure out what it is.

The problem is that both the directory name and the file name contain underscores, and in the final file name I separate the two components by underscores.
By either changing that separation character, or replacing the rule with a python function that get the names from elsewhere, I can solve the issue.
This works:
rule wasp_merge:
input:
"hic_mapping/bowtie_results/bwt2/{sdir}/{prefix}_hg19.bwt2pairs.keep.bam",
"hic_mapping/wasp_results/{sdir}{prefix}_filt_hg19.remap.kept.bam"
output:
"hic_mapping/wasp_results/{sdir}{prefix}_filt_hg19.bwt2pairs.filt.bam"
params:
threads=config['job_cores']
shell:
dedent(
"""\
{module}
module load samtools
samtools merge --threads {params.threads} {output} {input}
"""
)