Whenever I use snakemake with singularity and would like create a directory or file, I run into read-only errors if I try to write outside of the snakemake directory.
For example, if my rule contains the following:
container:
docker://some_image
shell:
"touch ~/hello.txt"
then everything runs fine. hello.txt is created inside the snakemake directory. However if my touch command tries to create a file outside of the snakemake directory:
container:
docker://some_image
shell:
"touch /home/user/hello.txt"
Then I get the following error:
touch: cannot touch '/home/user/hello.txt': Read-only file system
Is there anyway to give snakemake the ability to create files anywhere it wants when using singularity?
Singularity by default mounts certain directories including user's home directory. In your first command (touch ~/hello.txt), file gets written to home directory, where singularity has read/write permissions. However in your second command (touch /home/user/hello.txt), singularity doesn't have read/write access to /home/user/; you will need to bind that path manually using singularity's --bind and supply it to snakemake via --singularity-args.
So the snakemake command would look something like
snakemake --use-singularity --singularity-args "--bind /home/user/"
Related
So I have a build environment that gets setup when I do a . setup or source setup.
There are a bunch of commands (lets say command abc, command xyz) that become available once the setup file is sourced which I need to use inside a bash script. So I have to do something like this :
#!/bin/bash
cd build_dir
. setup
command abc && command xyz
And then on calling my script, I expect command abc and command xyz to have been executed.
But instead I see an error saying command abc not Found.
The setup environment is complex enough that I wouldn't want to bog my script with adding all the commands and env variables manually, I'd rather ditch the script completely.
Why is this happening and is there anyway of doing this with either shell scripting or python?
command abc means to ignore any alias or function named abc and instead searches an executable file inside the directories listed in $PATH. Obviously there is no such file on your system.
So, you need to know where your file abc is located. Then you have two possibilities: Either add this directory to your PATH,
PATH=$PATH:/foo/bar/abcdir
or you don't use command, but execute your program using the explicit path, i.e.
/foo/bar/abcdir/abc && command xyz
I recently started using snakemake and would like to run a shell script in my snakefile. However, I'm having trouble accessing input, output and params. I would appreciate any advice!
Here the relevant code snippets:
from my snakefile
rule ..:
input:
munged = 'results/munged.sumstats.gz'
output:
ldsc = 'results/ldsc.txt'
params:
mkdir = 'results/ldsc_results/',
ldsc_sumstats = '/resources/ldsc_sumstats/',
shell:
'scripts/run_gc.sh'
and the script:
chmod 770 {input.munged}
mkdir -p {params.mkdir}
ldsc=$(ls {params.ldsc_sumstats})
for i in $ldsc; do
...
I get the following error message:
...
chmod: cannot access '{input.munged}': No such file or directory
ls: cannot access '{params.ldsc_sumstats}': No such file or directory
...
The syntax of using {} statements applies only to shell scripts defined within Snakefile, while in the example you provide, the script is defined externally.
If you want to use the script as an external script you will need to pass the relevant arguments (and parse them inside the shell script). Otherwise, it should be possible to copy-paste the script content inside the shell directive and let snakemake substitute the {} variables.
From v7.14.0 Snakemake now supports executing external Bash scripts, with access to snakemake objects inside. See the docs for example usage.
I am trying to run a very simple python file that simply prints "woof" within a docker container. As far as I know I have created a docker container called:
c5d3c4c383d1
I then run the following command, in an attempt to tell myself what directory I am running things from in docker:
sudo docker run c5d3c4c383d1 pwd
This returns the following value:
/
Which I assume to be my root directory, so I go to my root directory. Typing pwd shows:
/
I then create a file called meow.py via the nano command and enter in this a single line that is:
print("Woof!")
I save this and confirm this is in the / directory with an ls command.
I then enter the following:
sudo docker run c5d3c4c383d1 python meow.py
Which returns:
python: can't open file 'meow.py': [Errno 2] No such file or directory
I don't understand this. Obviously I am not in the root directory when running a command with the docker as the meow.py file is DEFINETLY in the root directory but it is saying this file cannot be found. What the heck... As i said when I run pwd within the docker container it says i am in the / directory, but I cannot be given this file not found error.
docker is a container ... thats its root directory ... think of it like a totally different machine that you would normally ssh into... try something like this
docker run -it c5d3c4c383d1 bash
thats basically like you have just ssh'd into your remote machine
go ahead and try some commands (ls,pwd,etc)
now run echo print("hello world")>test.py
now run ls you should see your test.py ... go ahead and run it with python test.py
now you can exit you container ... if you launch the same container again you should still have your test.py file there... although i think its more common that people write a dockerfile that sets up their environment and then they just treat each session as disposable, as opposed to keeping the same container
I have a simple Python script, inside of a Docker container, that:
-Makes a call to an external API
-A license file is returned from the API
I want to be able to save the returned license file, to my directory inside of the docker container.
This is what I have done:
r = requests.post("http://api/endpoint", headers=headers, data=data, auth=("", "")
f = open('test.lic', 'w+')
f.write(r.content)
f.close()
My understanding is that this would create a test.lic file if one did not already exist, or open the existing test.lic file, and write the content of the request object to the test.lic. However this is not working. No file is being saved to my directory inside of the Docker container. If I run these lines from a python shell it works so I'm guessing it has something to do with being inside of a Docker container.
It could be that the file is getting saved, but it is in the working directory in the container.
The docker docs say:
The default working directory for running binaries within a container is the root directory (/), but the developer can set a different default with the Dockerfile WORKDIR command.
So the file may be ending up in the root directory / of your container.
You could add a print of os.getcwd() to your script (if you are able to see the output... which you might need to use the docker logs command) to verify this. Either way, the file will likely be in the location returned by os.getcwd().
In my case the file was saved to a different container. Check it if you have multiple containers in the docker-compose.
I run a cronjob-script as root user. The script is located in /root/scripts/myscript.py
In this python script I write to a file like this
log = open(os.path.basename(__file__) + ".log", "w+")
print("some text", file=log)
This is supposed to create a file with the name myscript.py.log and it does. But this file is located in /root/myscript.py.log. Why is that? I expected it to be created in the script directory.
I use Debian Squeeze and add the job via crontab -e as root user.
This is because it depends on the cwd when the cron task was started, cron could change directory too so Always use absolute pathnames for jobs run by cron, or pass in an absolute stub from the command line.
In your case crontab -e as root, cron will run your script as root and therefore use the home directory for that user. ie /root