I'm having the following problem:
My Snakemake program does not recognise the output my python script generated. I tried both, writing the ouput to the stdout and from there into the correct output file and directly writing from the python script (which is the version below).
Setting --latency-wait to 600 did not help either.
Other users reported that running ls helped which I tried while waiting for the latency but that didn't help either.
Additionally, when running again,snakemake wants to run all input files again, despite some output files already existing.
Does anyone have a suggestion what else I could try?
This is the snakemake command I'm using:
snakemake -j 2 --use-conda
Below is my snakefile:
import os
dir = "my/data/dir"
cell_lines = os.listdir(dir)
files = os.listdir(dir+"GM12878/25kb_resolution_intrachromosomal")
chromosomes = [i.split("_")[0] for i in files]
rule all:
input:
expand("~/TADs/{cell_lines}_{chromosomes}_TADs.tsv", cell_lines = cell_lines, chromosomes = chromosomes)
rule tad_calling:
input:
"my/data/dir/{cell_lines}/25kb_resolution_intrachromosomal/{chromosomes}_25kb.RAWobserved"
output:
"~/TADs/{cell_lines}_{chromosomes}_TADs.tsv"
benchmark:
"benchmarks/{cell_lines}_{chromosomes}.txt"
conda:
"my_env.yaml"
shell:
"""
python ~/script.py {input} {output}
"""
I think the problem is with the tilde (~), snakemake does not expand those (or e.g. $HOME). It takes those as the literal path. You could do something like:
from pathlib import Path
home = str(Path.home())
rule tad_calling:
...
output:
f"{home}/TADs/{cell_lines}_{chromosomes}_TADs.tsv"
...
Related
I am building a snakemake pipeline with python scripts.
Some of the python scripts take as input a directory, while others take as input files inside those directories.
I would like to be able to do have some rules which take as input the directory and some that take as input the files. Is this possible?
Example of what I am doing showing only two rules:
FILES = glob.glob("data/*/*raw.csv")
FOLDERS = glob.glob("data/*/")
rule targets:
input:
processed_csv = expand("{files}raw_processed.csv", files =FILES),
normalised_csv = expand("{folders}/normalised.csv", folders=FOLDERS)
rule process_raw_csv:
input:
script = "process.py",
csv = "{sample}raw.csv"
output:
processed_csv = "{sample}raw_processed.csv"
shell:
"python {input.script} -i {input.csv} -o {output.processed_csv}"
rule normalise_processed_csv:
input:
script = "normalise.py",
processed_csv = "{sample}raw_processed.csv" #This is input to the script but is not parsed, instead it is fetched within the code normalise.py
params:
folder = "{folders}"
output:
normalised_csv = "{folders}/normalised.csv" # The output
shell:
"python {input.script} -i {params.folder}"
Some python scripts (process.py) take all the files they needed or produced as inputs and they need to be parsed. Some python scripts only take the main directory as input and the inputs are fetched inside and the outputs are written on it.
I am considering rewriting all the python scripts so that they take the main directory as input, but I think there could be a smart solution to be able to run these two types on the same snakemake pipeline.
Thank you very much in advance.
P.S. I have checked and this question is similar but not the same: Process multiple directories and all files within using snakemake
I would like to be able to do have some rules which take as input the directory and some that take as input the files. Is this possible?
I don't see anything special with this requirement... What about this?
rule one:
output:
d=directory('{sample}'),
a='{sample}/a.txt',
b='{sample}/b.txt',
shell:
r"""
mkdir -p {output.d}
touch {output.a}
touch {output.b}
"""
rule use_dir:
input:
d='{sample}',
output:
out='outdir/{sample}.out',
shell:
r"""
cat {input.d}/* > {output.out}
"""
rule use_files:
input:
a='{sample}/a.txt',
b='{sample}/b.txt',
output:
out='outfiles/{sample}.out',
shell:
r"""
cat {input.a} {input.b} > {output.out}
"""
rule use_dir will use the content of directory {sample}, whatever it contains. Rule use_files will use specifically files a.txt and b.txt from directory {sample}.
I need to run Bppancestor with multiple config files, I have tried different approaches but none of them worked. I have around 150 files, so doing it one by one is not an efficient solution.
The syntax to run bppancestor is the following one:
bppancestor params=config_file
I tried doing:
bppancestor params=directory_of_config_files/*
and using a Snakefile to try to automatize the workflow:
ARCHIVE_FILE = 'bpp_output.tar.gz'
# a single config file
CONFIG_FILE = 'config_files/{sims}.conf'
# Build the list of input files.
CONF = glob_wildcards(CONFIG_FILE).sims
# pseudo-rule that tries to build everything.
# Just add all the final outputs that you want built.
rule all:
input: ARCHIVE_FILE
# run bppancestor
rule bpp:
input:
CONF,
shell:
'bppancestor params={input}'
# create an archive with all results
rule create_archive:
input: CONF,
output: ARCHIVE_FILE
shell: 'tar -czvf {output} {input}'
Could someone give me advice on this?
You're very close. Rule bpp should use as input a specific config file and specify concrete output (not sure if the output is a file or a folder). If I understand the syntax correctly, this link suggests that output files can be specified using output.sites.file and output.nodes.file:
rule bpp:
input:
CONFIG_FILE,
output:
sites='sites.{sims}',
nodes='nodes.{sims}',
shell:
'bppancestor params={input} output.sites.file={output.sites} output.nodes.file={output.nodes}'
Rule create_archive will collect all the outputs and archive them:
rule create_archive:
input: expand('sites.{sims}', CONF), expand('nodes.{sims}', CONF)
output: ARCHIVE_FILE
shell: 'tar -czvf {output} {input}'
I am writing a snakemake file that has input files on a specific location, with specific folder names (for this example, barcode9[456]). I need to change the naming conventions in these directories so I now want to add a first rule to my snakemake, which would link the folders in the original location (FASTQ_PATH) to an output folder in the snakemake working directory. The names of the link folders in this directory come from a python dictionay d, defined in the snakemake. I would then use these directories as input for the downstream rules.
So the first rule of my snakemake is actually a python script (scripts/ln.py) that maps the naming convention in the original directory to the desired naming conventions, and links the fastqs:
The snakemake looks like so:
FASTQ_PATH = '/path/to/original_location'
# dictionary that maps the subdirectories in FASTQ_PATH (keys) with the directory names that I want in the snakemake working directory (values)
d = {'barcode94': 'IBC_UZL-CV5-04',
'barcode95': 'IBC_UZL-CV5-42',
'barcode96': 'IBC_UZL-CV5-100'}
rule all:
input:
expand('symLinkFq/{barcode}', barcode = list(d.values())) # for each element in list(d.values()) I want to create a subdirectory that would point to the corresponding path in the original location (FASTQ_PATH)
rule symlink:
input:
FASTQ_PATH,
d
output:
'symLinkFq/{barcode}'
script:
"scripts/ln.py"
The python script that I am calling to make the links is shown below
import pandas as pd
import subprocess as sp
import os
# parsing variables from Snakefile
d_sampleNames = snakemake.input[1]
fastq_path = snakemake.input[0]
os.makedirs('symLinkFq')
for barcode in list(d_sampleNames.keys()):
idx = list(d_sampleNames.keys()).index(barcode)
sampleName = list(d_sampleNames.values())[idx]
sp.run(f"ln -s {fastq_path}/{barcode} symLinkFq/{sampleName}", shell=True) # the relative path with respect to the working directory should suffice for the DEST in the ln -s command
But when I call snakemake -np -s Snakefile I get
Building DAG of jobs...
MissingInputException in line 15 of /nexusb/SC2/ONT/scripts/SnakeMake/minimalExample/renameFq/Snakefile:
Missing input files for rule symlink:
barcode95
barcode94
barcode96
The error kind of makes sense to me. The only 'input' files that I have are python variables instead of being files that do exist in my system.
I guess the issue that I am having comes down to the fact that the wildcards that I want to use for all rules are not present in any file that can be used as input, so what I can think of using is the dictionary with the correspondence, though it is not working as I tried.
Does anyone know how to get around this, any other different approach is welcome.
If I understand correctly, I think it could be easier...
I would reverse the key/value mapping (here with dict(zip(...))) than use a lambda input function to get the source directory for each output key:
FASTQ_PATH = '/path/to/files'
d = {'barcode94': 'IBC_UZL-CV5-04',
'barcode95': 'IBC_UZL-CV5-42',
'barcode96': 'IBC_UZL-CV5-100'}
d = dict(zip(d.values(), d.keys())) # Values become keys and viceversa
rule all:
input:
expand('symLinkFq/{barcode}', barcode = d.keys())
rule symlink:
input:
indir= lambda wc: os.path.join(FASTQ_PATH, d[wc.barcode]),
output:
outdir= directory('symLinkFq/{barcode}'),
shell:
r"""
ln -s {input.indir} {output.outdir}
"""
As an aside, in a python script I would use os.symlink() instead of spawning a subprocess and call ln -s - I think it's easier to debug if something goes wrong.
I want to download files from a password protected FTP server in a Snakemake rule. I have seen the answer of Maarten-vd-Sande on specifying it using wildcards. Is it also possible using inputs without running into the MissingInputException?
FILES = ['file1.txt',
'file2.txt']
#remote file retrieval
rule download_file:
# replacing input by output would download all files in one job?
input:
file = expand("{file}", file=FILES)
shell:
# #this assumes your runtime has the SSHPASS env variable set
"sshpass -e sftp -B 258048 server<< get {input.file} data/{input.file}; exit"
I have seen the hint on the SFTP class in snakemake, but I am unsure how to use it in this context.
Thanks in advance!
I haven't tested this, but I am guessing something like this should work! We say that all the output we want is in rule all. Then we have the download rule to download those. I have no experience with using snakemake.remote, so I might be completely wrong in this though.
from snakemake.remote.SFTP import RemoteProvider
SFTP = RemoteProvider()
FILES = ['file1.txt',
'file2.txt']
rule all:
input:
FILES
rule download_file:
input:
SFTP.remote("{filename}.txt")
output:
"{filename}.txt"
# shell: # I am not sure if the shell keyword is required, if not, then you can remove these two lines.
# The : does nothing, just for the sake of having something there
# ":"
So I ended up using the following. The trick was how to pass the command to sftp using <<< "command". The envvars let snakemake check that the SSHPASSis set for sshpass to pick up.
envvars:
"SSHPASS"
#remote file retrieval
# #Idea: Replace using SFTP class
rule download_file:
output:
raw = temp(os.path.join(config['DATADIR'], "{file}", "{file}.txt"))
params:
file="{file}.txt"
resources:
walltime="300", nodes=1, mem_mb=2048
threads:
1
shell:
"sshpass -e sftp -B 258048 server <<< \"get {params.file} {output.raw} \""
I am trying to construct a snakemake pipeline for biosynthetic gene cluter detection but am struggling with the error:
Missing input files for rule all:
antismash-output/Unmap_09/Unmap_09.txt
antismash-output/Unmap_12/Unmap_12.txt
antismash-output/Unmap_18/Unmap_18.txt
And so on with more files. As far as I can see the file generation in the snakefile should be working:
workdir: config["path_to_files"]
wildcard_constraints:
separator = config["separator"],
extension = config["file_extension"],
sample = config["samples"]
rule all:
input:
expand("antismash-output/{sample}/{sample}.txt", sample = config["samples"])
# merging the paired end reads (either fasta or fastq) as prodigal only takes single end reads
rule pear:
input:
forward = "{sample}{separator}1.{extension}",
reverse = "{sample}{separator}2.{extension}"
output:
"merged_reads/{sample}.{extension}"
conda:
"~/miniconda3/envs/antismash"
shell:
"pear -f {input.forward} -r {input.reverse} -o {output} -t 21"
# If single end then move them to merged_reads directory
rule move:
input:
"{sample}.{extension}"
output:
"merged_reads/{sample}.{extension}"
shell:
"cp {path}/{sample}.{extension} {path}/merged_reads/"
# Setting the rule order on the 2 above rules which should be treated equally and only one run.
ruleorder: pear > move
# annotating the metagenome with prodigal#. Can be done inside antiSMASH but prefer to do it out
rule prodigal:
input:
"merged_reads/{sample}.{extension}"
output:
gbk_files = "annotated_reads/{sample}.gbk",
protein_files = "protein_reads/{sample}.faa"
conda:
"~/miniconda3/envs/antismash"
shell:
"prodigal -i {input} -o {output.gbk_files} -a {output.protein_files} -p meta"
# running antiSMASH on the annotated metagenome
rule antiSMASH:
input:
"annotated_reads/{sample}.gbk"
output:
touch("antismash-output/{sample}/{sample}.txt")
conda:
"~/miniconda3/envs/antismash"
shell:
"antismash --knownclusterblast --subclusterblast --full-hmmer --smcog --outputfolder antismash-output/{wildcards.sample}/ {input}"
This is an example of what my config.yaml file looks like:
file_extension: fastq
path_to_files: /home/lamma/ABR/Each_reads
samples:
- Unmap_14
- Unmap_55
- Unmap_37
separator: _
I can not see where i am going wrong within the snakefile to produce such an error. Apologies for the simple question, I am new to snakemake.
The problem is that you setup your global wildcard constraints wrong:
wildcard_constraints:
separator = config["separator"],
extension = config["file_extension"],
sample = '|'.join(config["samples"]) # <-- this should fix the problem
Then immediatly another problem follows with extension and seperator wildcards. Snakemake can only infer what these should be from other filenames, you can not actually set these through wildcard constraints. We can make use of f-string syntax to fill in what the values should be:
rule pear:
input:
forward = f"{{sample}}{config['separator']}1.{{extension}}",
reverse = f"{{sample}}{config['separator']}2.{{extension}}"
...
and:
rule prodigal:
input:
f"merged_reads/{{sample}}.{config['file_extension']}"
...
Take a look at snakemake regex if the wildcard constraints confuse you, and find a blog about f-strings if you are confused about the f"" syntax and when to use single { and when to use double {{ to escape them.
Hope that helps!
(Since I can't comment yet ...)
You might have a problem with your relative paths, and we cannot see where your files actually are found.
A way to debug this is to use config["path_to_files"] to create absolute paths in input:
That would give you better error message on where Snakemake expects the files - input/output files are relative to the working directory.