When running manage.py celeryd for async processes, everything works as expected but i'm getting a weird warning at every start-up. It's not causing any errors, but i'm having trouble making it disappear or understanding its meaning.
Here it is:
/home/user/lib/python2.7/billiard-2.7.3.15-py2.7-linux
-x86_64.egg/billiard/forking.py:455:
UserWarning: Will add directory '/home/user/webapps/django/proj' to path!
This is necessary to accommodate pre-Django 1.4 layouts using setup_environ.
You can skip this warning by adding a DJANGO_SETTINGS_MODULE=settings
environment variable.
W_OLD_DJANGO_LAYOUT % os.path.realpath(project_dir)
This is unusual because the django settings module has been added to my wsgi, and it works for everything but this. Does it want me to add the settings to httpd.conf or something?
Thanks
Oddly enough, I just ran into the same problem! Looking up the file in question (billiard/forking.py), I found this function:
def _Django_old_layout_hack__save():
if 'DJANGO_PROJECT_DIR' not in os.environ:
try:
settings_name = os.environ['DJANGO_SETTINGS_MODULE']
except KeyError:
return # not using Django.
try:
project_name, _ = settings_name.split('.', 1)
except ValueError:
return # not modified by setup_environ
project = __import__(project_name)
try:
project_dir = os.path.normpath(_module_parent_dir(project))
except AttributeError:
return # dynamically generated module (no __file__)
warnings.warn(UserWarning(
W_OLD_DJANGO_LAYOUT % os.path.realpath(project_dir)
))
os.environ['DJANGO_PROJECT_DIR'] = project_dir
This function apparently does some sanity checks on os.environ. Notice that, after retrieving DJANGO_SETTINGS_MODULE, it tries to split the module name by a period. This code seems to assume that, if your DJANGO_SETTINGS_MODULE is a top-level module (as it is, by default), then your environment hasn't been modified.
Unfortunately, if it isn't a top-level module, it seems to assume that you used setup_environ, and that it must now add the project directory to the path.
In my case, I had simply moved the simple settings.py module out into its own settings package, splitting it up into the common, and development/production files. Of course, I had to modify manage.py and wsgi.py to point to the correct settings module. Which, of course, started to trigger this warning.
The way I worked around it was by adding the DJANGO_PROJECT_DIR variable directly in my manage.py. I'm not sure if I'll need to add it elsewhere (e.g. in production environments), but that's all I've run into so far.
Here's the relevant line in manage.py:
if __name__ == "__main__":
os.environ.setdefault("DJANGO_SETTINGS_MODULE", "settings.current")
# Add the project directory to the path, to appease billiard
os.environ.setdefault("DJANGO_PROJECT_DIR",
os.path.dirname(os.path.realpath(__file__)))
I'm using django 1.4 and celery 3.0.11.
thank's #voithos. I tried yours solution. It's work for me.
But I came across another solution, much more simple. no need of creating a new module settings.
Juste add this piece of code into your manage.py file
# Add the project directory to the path, to appease billiard
os.environ.setdefault("DJANGO_PROJECT_DIR", os.path.dirname(os.path.realpath(__file__)))
finally my manage.py file looks like this
if __name__ == "__main__":
os.environ.setdefault("DJANGO_SETTINGS_MODULE", "sample.settings")
from django.core.management import execute_from_command_line
# Add the project directory to the path, to appease billiard
os.environ.setdefault("DJANGO_PROJECT_DIR",os.path.dirname(os.path.realpath(__file__)))
execute_from_command_line(sys.argv)
Related
I have flask web app and its structure is as follows:
/app
/__init__.py
/wsgi.py
/app
/__init__.py
/views.py
/models.py
/method.py
/common.py
/db_client.py
/amqp_client.py
/cron
/__init.py__
/daemon1.py
/daemon2.py
/static/
/main.css
/templates/
/base.html
/scripts
/nginx
/supervisor
/Dockerfile
/docker-compose.yml
In app/app/cron i have written standalone daemons which i want to call outside the docker. e.g.
python daemon1.py
daemon1.py code
from ..common import stats
from ..method import msapi, dataformater
from ..db_client import db_connection
def run_daemon():
......
......
......
if name =="main":
run_daemon()
So when i am trying to run this daemon1.py its throwing ValueError: Attempted relative import in non-package
Please suggest the right approach for import as well as to structure these daemons.
Thanks in advance.
I ran into the exact same problem with an app that was running Flask and Celery. I spent far too many hours Googling for what should be an easy answer. Alas, there was not.
I did not like the "python -m" syntax, as that was not terribly practical for calling functions within running code. And on account of of my seemingly small brain, I was not able to come to grips with any of the other answers out there.
So...there is the wrong way and the long way. Both of them work (for me) and I'm sure I'll get a tongue lashing from the community.
The Wrong Way
You can call a module directly using the imp package like so:
import imp
common = imp.load_source('common', os.path.dirname(os.path.abspath('__file__')) + '/common.py')
result = common.stats() #not sure how you call stats, but you hopefully get the idea
I had a quick search for the references that said that is a no-no, but I can't find them...sorry.
The Long Way
This method involves temporarily appending each of your modules to you PATH. This has worked for me on my Docker deploys and works nicely regardless of the container's directory structure. Here are the steps:
1) You must import the relevant modules from the parent directories in your __init__ files. This is really the whole point of the __init__ - allowing the modules in its package to be callable. So, in your case, cron/__init__ should contain:
from . import common
It doesn't look like your directories go any higher than that, but you would do the same for any other packages levels up as well.
2) Now you need to append the path of the module to the PATH variable. You can see what is in there right now by running:
sys.path
As expected, you probably won't see any of your modules in there. That means, that Python can't figure out what you want when you call the common module. In order to add the path, you need to figure out the directory structure. You will want to make this dynamic to account for changing directories.
It's worth noting that this will need to run each time your module runs. I'm not sure what your cron module is, but in my case it is Celery. So, this runs only when I fire up workers and the initial crontabs.
Here is the hack I threw together (I'm sure there is a cleaner way to do it):
curr_path = os.getcwd() #current path where cron is running
parrent_path = os.path.abspath(os.path.join(os.getcwd(), '..')) #the parent directory path
parrent_dir = os.path.basename(os.path.abspath(parrent_path)) #the parent directory name
while parrent_dir <> 'project_name': #loop until you get to the top directory - should be the project name
parrent_path = os.path.abspath(os.path.join(par_path, '..'))
parrent_dir = os.path.basename(os.path.abspath(parrent_path))
In your case, this might be a challenge since you have two directories named 'app'. Your top level 'app' is my 'project_name'. For the next step, let's assume you have changed it to 'project_name'.
3) Now you can append the path for each of your modules to the PATH variable:
sys.path.append(parrent_dir + '/app')
Now if you run sys.path again, you should see the path to /app in there.
In summary: make sure all of your __init__'s have imports, determine the paths to the modules you want to import, append the paths to the PATH variable.
I hope that helps.
#greenbergé Thank you for your solution. i tried but didn't worked for me.
So to make things work I have changed my code little bit. Apart from calling run_daemon() in main of daemon1.py, i have called function run_daemon() directly.
python -m 'from app.cron.daemon1 import run_daemon(); run_daemon()'
As it is not exact solution of the problem but things worked for me.
I've been getting na error in PyCharm and I can't figure out why I'm getting it:
No tests were found
This is what I have for my point_test.py:
import unittest
import sys
import os
sys.path.insert(0, os.path.abspath('..'))
from ..point import Point
class TestPoint(unittest.TestCase):
def setUp(self):
pass
def xyCheck(self,x,y):
point = Point(x,y)
self.assertEqual(x,point.x)
self.assertEqual(y,point.y)
and this point.py, what I'm trying to test:
import unittest
from .utils import check_coincident, shift_point
class Point(object):
def __init__(self,x,y,mark={}):
self.x = x
self.y = y
self.mark = mark
def patched_coincident(self,point2):
point1 = (self.x,self.y)
return check_coincident(point1,point2)
def patched_shift(self,x_shift,y_shift):
point = (self.x,self.y)
self.x,self,y = shift_point(point,x_shift,y_shift)
Is it something wrong with my run configuration? I looked at this SO post but I'm still utterly confused. My run configuration currently looks like this:
In order to recognize test functions, they must be named test_. In your case, rename xyCheck to test_xyCheck :)
I know it's more than a year since the question was asked, but I had the same issue and that post was the first result in search.
As I understood PyCharm (or Intellij Idea Python plugin) needs your test to meet the following criteria if you want it to be launched when you run all the tests in directory.
test functions should start with "test" (underscore is not necessary)
the file, containing the test should also start with "test". "Test" (with capital T doesn't work in my case
I'm using Intellij IDEA 2016.3.5 with Python plugin
If you want to run you tests with command line
python -m unittest
Then you should add __init__.py to test directory. Python still wants your test function names to start with "test", and you test file name to start with "test", but in case of files it doesn't care if the first "t" is capital or not. TestCase and test_case is equally fine.
One thing that can also cause this problem is if you have not selected the right testing framework in your settings:
settings > Python Integrated Tools > Testing > Default test runner
All my tests are in pytest, but it was set to unit test
Don't use dashes ("-") in your filename. These files were being ignored on my machine. renaming them from project-tests.py to project_tests.py solved the problem.
Another gotcha that has just bitten me.
I had a test file within my test package called test_queue.py which followed all of the above advice, however when I selected "Run UnitTests" in PyCharm the console reported no tests found.
The issue in my case was that I had a non-unit test file in the root of the project also called test_queue.py which had been used for some other purpose early on in the project and forgotten about.
Even though I was specifically selecting the test file in my tests folder, and the path was set to absolutely point at the unit test version of the file, it seems the file in the root of the project was being used.
So, one more thing to check, make sure there are no other files in your project with the same name.
Another issue you may want to check is if you see in the console output "## tests deselected by '-k XXXX'". This means you've added a Keyword to the Run/Debug Configuration, and Pycharm will only run tests whose name contains the keyword.
Adding another answer in the hopes that it helps someone down the road. I've been fighting this problem and just figured out the answer (I think). I originally deleted the standard Django file tests.py out of my application folder. I also created a subdirectory of my project called tests, which contains separate test scripts. In this situation, Pycharm failed to find the tests. I corrected this simply by creating an empty file called tests.py in my application folder.
So:
Make sure you have a file called tests.py in your application director (it can be an empty file)
It seems that a folder called tests, in your project, can contain separate test scripts and Pycharm seems to find and run these.
Here's a picture of the directory structure that's working for me:
I had this exception when running individual tests in a Django 1.8 project in PyCharm 2018.1. I could run all the tests together, but individual tests in one file crashed.
The exception was happening in unittest's loader.py
It was getting an ImportError trying to import test_admin_views.py, though the exception was hiding the details of that error.
To see the details of the ImportError, I opened a Python Console and ran:
import my_app.users.tests.test_admin_views
This gave me:
Traceback (most recent call last):
[...]
File "my_app/my_app/users/tests/model_factories.py", line 42, in <module>
from my_app.applications.tests.model_factories import ApplicationFactory
ImportError: cannot import name ApplicationFactory
I noticed that some of the other factories in that file are imported without using the full path, so I tried adding ApplicationFactory to the relative path import list:
from .model_factories import UserFactory, UserGroupFactory, ApplicationFactory
Which worked!
None of these helped me, but I was able to fix the issue by setting the working directory to the project directory.
I debugged this by first creating a new Django test configuration witj a target of one of the app names with tests.
That ran successfully. No idea why working directory should need to be specified.
I was trying to do a bit of testing on a pieace of code but I get the ImportError: Start directory is not importable. Below is my code. I would really appreciate if someone could help. I am using Python 2.7.5 and also pycharm.
This is the command I execute in virenv
manage.py test /project/tests.py
from django.test import TestCase
# Create your tests here.
from project.models import Projects
from signup.models import SignUp
class ProjectTestCase(TestCase):
def SetUp(self):
super(ProjectTestCase,self).SetUp()
self.john = SignUp.objects.get(email='john#john.com')
self.project = Projects.objects.get(pk=1)
def test_project_permission(self):
self.assertFalse(self.john.has_perm('delete',self.project))
self.assertTrue(self.john.has_perm('view',self.project))
self.assertTrue(self.john.has_perm('change',self.project))
You should verify that you have an __init__.py in your project directory (it can be an empty file)
You're not calling test correctly. Assuming you have an app known as project, the way to call the tests in that app is manage.py test project
[Edit for additional comment which is really getting into separate question territory]
I would suggest adding this to settings.py (at the bottom)
import sys
if 'test' in sys.argv:
DATABASES['default'] = {'ENGINE': 'django.db.backends.sqlite3'}
SOUTH_TESTS_MIGRATE = False
This:
A. Will use sqlite (sort of, I'm not sure if it's actually sqlite or an in-memory db that works like sqlite) ... I've noticed issues if I use postgres and am trying to have my permissions not be excessively liberal AND I've had a test abort)
B. Disables south migrations (so it will just start with a clean database built against whatever models are currently saying)
I am following the polls app tutorial part 3 and I cannot get the templates to be found
This is the exact error
polls/index.html
Request Method: GET
Request URL: http://localhost:8000/polls/
Django Version: 1.4.3
Exception Type: TemplateDoesNotExist
Exception Value:
polls/index.html
Exception Location: c:\Python27\lib\site-packages\django\template\loader.py in find_template, line 138
Python Executable: c:\Python27\python.exe
Python Version: 2.7.2
So in my settings.py I have put the directory there. "C:/Scripts/mysite/template" and created /polls/index.html
TEMPLATE_DIRS = (
# Put strings here, like "/home/html/django_templates" or "C:/www/django/templates".
# Always use forward slashes, even on Windows.
# Don't forget to use absolute paths, not relative paths.
"C:/Scripts/template"
"C:/Scripts/mysite/template"
)
It for some reason cannot find it though. Any ideas?
This is the exact instruction from the tutorial.
create a directory, somewhere on your filesystem, whose contents Django can access. (Django runs as whatever user your server runs.)
You're missing a comma at the end of the first line:
"C:/Scripts/template", # <--- HERE
"C:/Scripts/mysite/template"
without that, Python automatically concatenates the two strings (because they're inside parens) so that it just becomes "C:/Scripts/templateC:/Scripts/mysite/template".
Debug
from your applications main folder, run the following command: python manage.py shell this is going to bootstrap your app with the settings file.
type from django.conf import settings and hit enter
type settings.TEMPLATE_DIRS and check the output. Do you see the template directories that you specified?
Absolute paths relative to the settings.py file
I generally use absolute paths relative to the settings.py file. That way collaborators can share a main settings file, and no matter what system/environment you deploy to your paths will be correct.
to do this:
# settings.py
import os
# ... other settings
TEMPLATE_DIRS = (
os.path.join(os.path.normpath(os.path.dirname(__file__)), 'templates'),
)
let me know if the debug step didn't help, and i'll try and supply some more help.
I have a fair number of Python scripts that contain reusable code that are used and referenced by other Python scripts. However, these scripts tend to be scattered across different directories and I find it to be somewhat tedious to have to include (most often multiple) calls to sys.path.append on my top-level scripts. I just want to provide the 'import' statements without the additional file references in the same script.
Currently, I have this:
import sys
sys.path.append('..//shared1//reusable_foo')
import Foo
sys.path.append('..//shared2//reusable_bar')
import Bar
My preference would be the following:
import Foo
import Bar
My background is primarily in the .NET platform so I am accustomed to having meta files such as *.csproj, *.vbproj, *.sln, etc. to manage and contain the actual file path references outside of the source files. This allows me to just provide 'using' directives (equivalent to Python's import) without exposing all of the references and allowing for reuse of the path references themselves across multiple scripts.
Does Python have equivalent support for this and, if not, what are some techniques and approaches?
The simple answer is to put your reusable code in your site-packages directory, which is in your sys.path.
You can also extend the search path by adding .pth files somewhere in your path.
See https://docs.python.org/2/install/#modifying-python-s-search-path for more details
Oh, and python 2.6/3.0 adds support for PEP370, Per-user site-packages Directory
If your reusable files are packaged (that is, they include an __init__.py file) and the path to that package is part of your PYTHONPATH or sys.path then you should be able to do just
import Foo
This question provides a few more details.
(Note: As Jim said, you could also drop your reusable code into your site-packages directory.)
You can put the reusable stuff in site-packages. That's completely transparent, since it's in sys.path by default.
You can put someName.pth files in site-packages. These files have the directory in which your actual reusable stuff lives. This is also completely transparent. And doesn't involve the extra step of installing a change in site-packages.
You can put the directory of the reusable stuff on PYTHONPATH. That's a little less transparent, because you have to make sure it's set. Not rocket science, but not completely transparent.
In one project, I wanted to make sure that the user could put python scripts (that could basically be used as plugins) anywhere. My solution was to put the following in the config file for that project:
[server]
PYPATH_APPEND: /home/jason:/usr/share/some_directory
That way, this would add /home/jason and /usr/share/some_directory to the python path at program launch.
Then, it's just a simple matter of splitting the string by the colons and adding those directories to the end of the sys.path. You may want to consider putting a module in the site-packages directory that contains a function to read in that config file and add those directories to the sys.path (unfortunately, I don't have time at the moment to write an example).
As others have mentioned, it's a good idea to put as much in site-packages as possible and also using .pth files. But this can be a good idea if you have a script that needs to import a bunch of stuff that's not in site-packages that you wouldn't want to import from other scripts.
(there may also be a way to do this using .pth files, but I like being able to manipulate the python path in the same place as I put the rest of my configuration info)
The simplest way is to set (or add to) PYTHONPATH, and put (or symlink) your modules and packages into a path contained in PYTHONPATH.
My solution was to package up one utility that would import the module:
my_util is in site packages
import my_util
foo = myutil.import_script('..//shared1//reusable_foo')
if foo == None:
sys.exit(1)
def import_script(script_path, log_status = True):
"""
imports a module and returns the handle
"""
lpath = os.path.split(script_path)
if lpath[1] == '':
log('Error in script "%s" in import_script' % (script_path))
return None
#check if path is already in sys.path so we don't repeat
npath = None
if lpath[0] == '':
npath = '.'
else:
if lpath[0] not in sys.path:
npath = lpath[0]
if npath != None:
try:
sys.path.append(npath)
except:
if log_status == True:
log('Error adding path "%s" in import_script' % npath)
return None
try:
mod = __import__(lpath[1])
except:
error_trace,error_reason = FormatExceptionInfo()
if log_status == True:
log('Error importing "%s" module in import_script: %s' % (script_path, error_trace + error_reason))
sys.path.remove(npath)
return None
return mod