I have a .coveragerc file in the root of my project. It tells coverage.py to omit my project's migrations directories:
[run]
omit = *migrations*
When I run coverage.py at the command line, the config I put in .coveragerc is obeyed.
However, PyCharm does not recognize it. Is there a setting that I'm missing?
If it turns out there's no way for PyCharm to recognize .coveragerc, I'd be happy with even just a way to omit those directories.
You can make PyCharm use the .coveragerc by putting it into the working directory you run your tests from.
The feature request from https://youtrack.jetbrains.com/issue/PY-16945 was implemented and works in version 2018.1.
There is a feature request for this at https://youtrack.jetbrains.com/issue/PY-16945
There is a different way to make PyCharm ignore certain files and folders:
In the Settings choose Project: ... - Project Structure. Here you can mark folders as Excluded or exclude files specifically.
PyCharm`s code coverage reports ignores all those exluded files, too.
I found myself in the situation where I needed this badly. My travis run were running correctly and so did coveralls but I was unable to get things work in PyCharm.
The thing is a bit hacky, but hopefully it will help people :
In my root project directory, I got a .coveragerc
[run]
omit = ./venv
concurrency = multiprocessing
parallel = True
source = HookTest
data_file = .cache/.coverage
And I "hacked" a little run_coverage.py of PyCharm : (pycharm-2016.3.2/helpers/coverage_runner/run_coverage.py )
Starting at
argv = []
Replace everything with :
argv = []
for arg in sys.argv:
if arg.startswith('-m') and arg[2:]:
argv.append(arg[2:])
else:
argv.append(arg)
cwd = os.getcwd()
rcfile = cwd + "/.coveragerc"
if os.path.exists(rcfile):
print("Loading rcfile")
i = argv.index("run")+1
argv = argv[:i] + ["--rcfile={}".format(rcfile)] + argv[i:]
sys.argv = argv
try:
main()
finally:
if run_cov:
os.chdir(cwd)
if os.getenv('COVERAGE_COMBINE'):
main(["combine"])
main(["xml", "-o", coverage_file + ".xml", "--ignore-errors"])
To make this run with python setup.py test, I created a script in PyCharm that uses said setup.py, had test has parameter, and COVERAGE_COMBINE as global env. It might not be the best of all time solutions but at least it allows me not to use HTML output anymore while working locally :)
It does not work in the last Pycharm version 2018.3.4.
The only way I succeeded to make it works is to heck the run_coverage.py as #PonteIneptique
Here is the only modification I had to do:
run_xml = os.getenv('PYCHARM_RUN_COVERAGE_XML')
argv = ["xml", "-o", coverage_file + ".xml", "--ignore-errors"]
rcfile = cwd + "/.coveragerc"
if os.path.exists(rcfile):
print("Loading rcfile\n")
argv += ["--rcfile", rcfile]
if run_xml:
os.chdir(cwd)
main(argv)
else:
try:
main()
finally:
if run_cov:
os.chdir(cwd)
main(argv)
Be sure to set the working dir of of the .coveragerc file too in your configuration.
Pycharm coders should update their code for supporting this .coveragerc file from the GUI.
Related
I am fairly new to programming and currently working on a python script. It is supposed to gather all the files and directories that are given as paths inside the program and copy them to a new location that the user can choose as an input.
import shutil
import os
from pathlib import Path
import argparse
src = [ [insert name of destination directory, insert path of file/directory that
should be copied ]
]
x = input("Please choose a destination path\n>>>")
if not os.path.exists(x):
os.makedirs(x)
print("Directory was created")
else:
print("Existing directory was chosen")
dest = Path(x.strip())
for pfad in src:
if os.path.isdir(pfad[1]):
shutil.copytree(pfad[1], dest / pfad[0])
elif os.path.isfile(pfad[1]):
pfad1 = Path(dest / pfad[0])
if not os.path.exists(pfad1):
os.makedirs(pfad1)
shutil.copy(pfad[1], dest / pfad[0])
else:
print("An error occured")
print(pfad)
print("All files and directories have been copied!")
input()
The script itself is working just fine. The problem is that I want write a test that automatically test the code each time I push it to my GitLab repository. I have been browsing through the web for quite some time now but wasnt able to find a good explanation on how to approach creating a test for a script like this.
I would be extremely thankful for any kind of feedback or hints to helpful resources.
First, you should write a test that you can run in command line.
I suggest you use the argparse module to pass source and destination directories, so that you can run thescript.py source_dir dest_dir without human interaction.
Then, as you have a test you can run, you need to add a .gitlab-ci.yml to the root of the project so that you can use the gitlab CI.
If you never used the gitlab CI, you need to start here: https://docs.gitlab.com/ee/ci/quick_start/
After that, you'll be able to add a job to your .gitlab-ci.yml, so that a runner with python installed will run the test. If you don't understad the bold terms of the previous sentence, you need to understant Gitlab CI first.
Working with scientific data, specifically climate data, I am constantly hard-coding paths to data directories in my Python code. Even if I were to write the most extensible code in the world, the hard-coded file paths prevent it from ever being truly portable. I also feel like having information about the file system of your machine coded in your programs could be security issue.
What solutions are out there for handling the configuration of paths in Python to avoid having to code them out explicitly?
One of the solution rely on using configuration files.
You can store all your path in a json file like so :
{
"base_path" : "/home/bob/base_folder",
"low_temp_area_path" : "/home/bob/base/folder/low_temp"
}
and then in your python code, you could just do :
import json
with open("conf.json") as json_conf :
CONF = json.load(json_conf)
and then you can use your path (or any configuration variable you like) like so :
print "The base path is {}".format(CONF["base_path"])
First off its always good practise to add a main function to go with each class to test that class or functions in the file. Along with this you determine the current working directory. This becomes incredibly important when running python from a cron job or from a directory that is not the current working directory. No JSON files or environment variables are then needed and you will obtain interoperation across Mac, RHEL and Debian distributions.
This is how you do it, and it will work on windows also if you use '\' instead of '/' (if that is even necessary, in your case).
if "__main__" == __name__:
workingDirectory = os.path.realpath(sys.argv[0])
As you can see when you run your command, the working directory is calculated if you provide a full path or relative path, meaning it will work in a cron job automatically.
After that if you want to work with data that is stored in the current directory use:
fileName = os.path.join( workingDirectory, './sub-folder-of-current-directory/filename.csv' )
fp = open( fileName,'r')
or in the case of the above working directory (parallel to your project directory):
fileName = os.path.join( workingDirectory, '../folder-at-same-level-as-my-project/filename.csv' )
fp = open( fileName,'r')
I believe there are many ways around this, but here is what I would do:
Create a JSON config file with all the paths I need defined.
For even more portability, I'd have a default path where I look for this config file but also have a command line input to change it.
In my opinion passing arguments from command line would be best solution. You should take a look at argparse . This allows you to create nice way to handle arguments from the command line. for example:
myDataScript.py /home/userName/datasource1/
Idea
Basically, what my script does is checking C:/SOURCE for .txt files and add a timestamp to it. To replicate it you can basically make that folder and put some txt files in there. Then, it's supposed to run a .vbs file, which then runs a .bat files with some rclone commands which don't matter here. I did it like this because there wont be a CMD window opening when running the rclone command through the .vbs file.
Python code
import time, os, subprocess
while True:
print("Beginning checkup")
print("=================")
timestamp = time.strftime('%d_%m_%H_%M') # only underscores: no naming issues
the_dir = "C:/SOURCE"
for fname in os.listdir(the_dir):
if fname.lower().endswith(".txt"):
print("found " + fname)
time.sleep(0.1)
new_name = "{}-{}.txt".format(os.path.splitext(fname)[0], timestamp)
os.rename(os.path.join(the_dir, fname), os.path.join(the_dir, new_name))
time.sleep(0.5)
else:
subprocess.call(['cscript.exe', "copy.vbs"])
time.sleep(60)
VBScript code
Set WshShell = CreateObject("WScript.Shell" )
WshShell.Run Chr(34) & "copy.bat" & Chr(34), 0
Set WshShell = Nothing
The only important part for the Python script is below the very last else, where the subprocess.call() is supposed to run the .vbs file. What happens when running the script is it shows the first two lines that always come up when running CMD, but then nothing.
How could I fix that? I tried:
subprocess.call("cscript copy.vbs")
subprocess.call("cmd /c copy.vbs")
both with the same outcome, it doesn't do anything.
Anyone have an idea?
Why are you invoking a VBScript to invoke a batch script from Python? You should be able to simple run whatever the batch script is doing directly from your Python code. But even if you wanted to keep the batch script, something like this should do just fine without VBScript as an intermediary.
subprocess.call(['cmd', '/c', 'copy.bat'])
You may want to give the full path of the batch file, though, to avoid issues like the working directory not being what you think it is.
If your batch script resides in the same directory as the Python script, you can build the path with something like this:
import os
import subprocess
scriptdir = os.path.dirname(__file__)
batchfile = os.path.join(scriptdir, 'copy.bat')
subprocess.call(['cmd', '/c', os.path.realpath(batchfile)])
It seems there is no such an operation that could not be done using plain Python. Scan a directory, copy a file -- Python has it all in the standard library. See os.path and shutil modules.
Adding VB scripts and launching subprocesses make your code complex and difficult to debug.
I have a python script which takes a config file on command line and gives an output.
I am trying to see how I can use nosetests to run all these files.
I read through the nosetests info on google but i could not follow how to run them with the config file.
Any ideas on where I could get started?
Something like this should work:
import sys
import nose
def test_me():
assert True
if __name__ == '__main__':
module_name = sys.modules[__name__].__file__
config_name = 'nose.cfg'
result = nose.run(
argv=[sys.argv[0],
module_name,
'--config=' + config_name]
)
You can also pass your config instance, as described in the docs for nose.run() arguments here.
I didnt have to do any of those. Just nosetests by themselves execute any test beginning with "test_....py" and make sure you use "--exe" if they are executable if not you can skip that option. The nosetests help page on wiki really helps.
I want to make a site using sphinx document generator. But my machine is windows. I cant find sufficient resources for windows. If anyone could suggest me a way to implement sphinx in windows would be a great help.
Thanks.
Sphinx works just fine on Windows. To get started, go to the Quickstart tutorial and follow the instructions. One thing I would add is make sure you answer "y" to the question that asks if you want to separate the build and source folders. This will make things more simple later.
If you want to use apidoc (I think you do), then can use the command-line tools in the Scripts folder of your Python install. Or you can write your own script. Below is one I wrote to get .rst files for some target modules:
files = [u'C:\\Work\\Scripts\\Grapher\\both_pyplot.py',
u'C:\\Work\\Scripts\\Grapher\\colors_pyplot.py',
u'C:\\Work\\Scripts\\Grapher\\dataEditor.pyw',
u'C:\\Work\\Scripts\\Grapher\\grapher.pyw']
for d in ('pyfiles', 'rst_temp'):
try:
shutil.rmtree(d)
except WindowsError:
pass
os.mkdir(d)
#copy, rename .pyw files to .py so sphinx will pick them up
for fn in files:
fn2 = fn
if fn.lower().endswith('.pyw'):
fn2 = fn[:-1]
shutil.copy2(fn, os.path.join('pyfiles', os.path.basename(fn2)))
#now send to apidoc
lst = [fn, '-o', 'rst_temp', 'pyfiles']
from sphinx.apidoc import main
main(lst)
msg = ('Now copy the rst files you want documentation for from the '
'"rst_temp" dir to the the "source" dir, edit the index.html file '
'in the "source" dir, and run builder.py')
print msg
The apidoc extension does not recognize .pyw files, so this script copies target modules to a temporary location and renames them with a .py extension so apidoc can use them.
To build your project, you can run the make.bat file in your project folder (created when quickstart runs) or you can write your own script. Here's a sample (builder.py):
import sys, os
fn = __file__
sys.path.append(os.path.normpath('C:\\Work\\Scripts\\Grapher'))
lst = [fn, '-b', 'html', 'source', 'build']
from sphinx import main
main(lst)