So I was trying to modify an existing library and instead of doing it the smart way and using pip -e I instead just installed the libraries, then swapped the modified files for whatever changes I wanted. For example if I had:
Library A/
---doSomethingA.py
---otherFiles.py
I just deleted doSomethingA.py and replaced it with my version of doSomethingA.py. Theoretically I figured, because im editing the file locally, it should still work as planned for my library with whatever extra functionality I want.
HOWEVER.... its basically going crazy. While I can see my edited changes in the file, when I run the library its obviously not running that file. I did things like:
commenting out the whole file (still runs somehow)
Actually uninstalling the library and part of another script using doSomethingA.py it still runs?? (i.e Something like import libraryA works on JupyerHub, but not on the putty terminal... ?)
I've obviously come to the conclusion that its not running the file that it says that it is (and trust me I've checked the path of the file like 10 times).
My question is:
How is this possible? What are the places that python would store another copy of the file etc?
I've also deleted the __pychache__, but I can't think of anything else to do. Is my best option to just give up and create a new virtual environment, etc?
I understand that you are running on jupyter hub.
This means that your python is running remotly on the server and the framework is taking care to synch your local project (but not the installed libraries).
The python on the server is not aware of your local change.
As a temporary mitigation you can copy the installed library to you project root.
Related
I created a Python package that is actually a command line interface. Everything works fine, but I would like to be able to store simple user settings (a few values) in some file (json, yaml, or whatever). Handling such files itself is not a problem, but I don't know where I could store them. My program can be installed using pip install https://github.com/repo and works fully off-line.
It seems that Python does not allow me to store settings files in the same folder, where the compiled program itself is located (Programs\Python\Python38\Scripts) which makes sense, but I don't know how to do it any other way.
Inspired by the comments under the question, I searched a little bit more and found the answer to my problem.
I store my settings file in C:\users\me\appdata\local\programs\python\python38\lib\site-packages\my_package\config.json which can be accessed with
os.path.join(os.path.dirname(os.path.abspath(__file__)), 'config.json')
I haven't tested this on Linux or macOS yet, but I think it will work.
EDIT: Unfortunately it does not work on Linux without sudo. For this reason, I used the following:
os.path.join(os.path.expanduser("~"), '.my_package_name', 'config.json')
I want to run a python script as part of a jenkins pipline triggered from a github repo. If I store the script directly in the repo itself, I can just do sh 'python path/to/my_package/script.py' which work perfectly. However since I want to use this from multiple pipelines from multiple repos, I want to put this in a jenkins shared library.
I found this question which suggested storing the python file in the resources directory and copying it to a temp file before use. That only works if the script is one standalone file. Unfortunately, mine is a package with multiple python files and imports between them, so thats a no go. I also tried to copy the entire folder containing the python package from the answer to this question, which suggests getting the location of the library with
import groovy.transform.SourceURI
import java.nio.file.Path
import java.nio.file.Paths
class ScriptSourceUri {
#SourceURI
static URI uri
}
but its gives me the following error:
Scripts not permitted to use staticMethod java.net.URI create java.lang.String. Administrators can decide whether to approve or reject this signature.
It seems that some additional permissions are required, which I don't think I'll be able to acquire (its a shared machine).
So? Does anyone know how I can run a python package from jenkins shared library? Right now the only solution I can think of is to manually recreate the directory structure of the python package, which is obviously very messy and non-generic.
PS: There is no particular reason for using the python script over writing the same script in groovy. Its just that the python script is well tested, well understood and well supported. Rewriting the whole thing in groovy just isn't feasible right now.
You can go to http://host:8080/jenkins/scriptApproval/ page of your Jenkins installation and approve the request for your scripts, please see below:-
And follow the link for more information.
I work on two servers, and on one server .pydistutils.cfg looks like:
install_scripts=~/opt_old/bin
install_data=~/opt_old/share
install_lib=~/usr/lib/python2.6/site-packages
I think, this creates problems with using pip and linking libraries on local versions of python.
On the second server, the file is non existent, and I don't have any issues.
Why do we need this file and why is PYTHONPATH not sufficient?
I installed a local version of Python and renamed the file .pydistutils.cfg. Hence, it seems that the file is not that important.
A pretty good write-up is here:
http://bouktin.blogspot.com/2012/04/configure-pydistutilscfg-python.html
I don't immediately see a reason why an average developer should use it, it seems a bit kludgy to me, perhaps it makes sense if you make your own distro, target docker or similar distribution system or target an embedded system?
Here's a super-simple usage example:
https://github.com/amolenaar/gaphor/wiki/Custom-Python-Installation-Location
This is probably a question that has a very easy and straightforward answer, however, despite having a few years programming experience, for some reason I still don't quite get the exact concepts of what it means to "build" and then to "install". I know how to use them and have used them a lot, but have no idea about the exact processes which happen in the background...
I have looked across the web, wikipedia, etc... but there is no one simple answer to it, neither can I find one here.
A good example, which I tried to understand, is adding new modules to python:
http://docs.python.org/2/install/index.html#how-installation-works
It says that "the build command is responsible for putting the files to install into a build directory"
And then for the install command: "After the build command runs (whether you run it explicitly, or the install command does it for you), the work of the install command is relatively simple: all it has to do is copy everything under build/lib (or build/lib.plat) to your chosen installation directory."
So essentially what this is saying is:
1. Copy everything to the build directory and then...
2. Copy everything to the installation directory
There must be a process missing somewhere in the explanation...complilation?
Would appreciate some straightforward not too techy answer but in as much detail as possible :)
Hopefully I am not the only one who doesn't know the detailed answer to this...
Thanks!
Aivoric
Building means compiling the source code to binary in a sandbox location where it won't affect your system if something goes wrong, like a build subdirectory inside the source code directory.
Install means copying the built binaries from the build subdirectory to a place in your system path, where they become easily accessible. This is rarely done by a straight copy command, and it's often done by some package manager that can track the files created and easily uninstall them later.
Usually, a build command does all the compiling and linking needed, but Python is an interpreted language, so if there are only pure Python files in the library, there's no compiling step in the build. Indeed, everything is copied to a build directory, and then copied again to a final location. Only if the library depends on code written in other languages that needs to be compiled you'll have a compiling step.
You want a new chair for your living-room and you want to make it yourself. You browse through a catalog and order a pile of parts. When they arrives at your door, you can't immediately use them. You have to build the chair at your workshop. After a bit of elbow-grease, you can sit down in it. Afterwards, you install the chair in your living-room, in a convenient place to sit down.
The chair is a program you want to use. It arrives at your house as source code. You build it by compiling it into a runnable program. You install it by making it easier to use.
The build and install commands you are refering to come from setup.py file right?
Setup.py (http://docs.python.org/2/distutils/setupscript.html)
This file is created by 3rd party applications / extensions of Python. They are not part of:
Python source code (bunch of c files, etc)
Python libraries that come bundled with Python
When a developer makes a library for python that he wants to share to the world he creates a setup.py file so the library can be installed on any computer that has python. Maybe this is the MISSING STEP
Setup.py sdist
This creates a python module (the tar.gz files). What this does is copy all the files used by the python library into a folder. Creates a setup.py file for the module and archives everything so the library can be built somewhere else.
Setup.py build
This builds the python module back into a library (SPECIFICALLY FOR THIS OS).
As you may know, the computer that the python library originally came from will be different from the library that you are installing on.
It might have a different version of python
It might have a different operating system
It might have a different processor / motherboard / etc
For all the reasons listed above the code will not work on another computer. So setup.py sdist creates a module with only the source files needed to rebuild the library on another computer.
What setup.py does exactly is similar to what a makefile would do. It compiles sources / creates libraries all that stuff.
Now we have a copy of all the files we need in the library and they will work on our computer / operating system.
Setup.py install
Great we have all the files needed. But they won't work. Why? Well they have to be added to Python that's why. This is where install comes in. Now that we have a local copy of the library we need to install it into python so you can use it like so:
import mycustomlibrary
In order to do this we need to do several things including:
Copy files to their library folders in our version of python.
Make sure library can be imported using import command
Run any special install instructions for this library. (seting up paths, etc)
This is the most complicated part of the task. What if our library uses BeautifulSoup? This is not a part of Python Library. We'd have to install it in a way such that our library and any others can use BeautifulSoup without interfering with each other.
Also what if python was installed someplace else? What if it was installed on a server with many users?
Install handles all these problems transparently. What is does is make the library that we just built able to run. All you have to do is use the import command, install handles the rest.
At work we are using Trac on several internal wiki's and an external wiki. REcently we found the need for a new plugin. After we going through a few tutorials we went to install a plugin to make sure it would work. It didn't. We've been going through trying to figure out. Below I will list the steps and various things I did while trying to get it to work.
1) I went to trac-hacks website and downloaded their hellow world plugin, figured I couldn't make a mistake using their code.
2) I compiled and made an egg using python setup.py bdist_egg on the machine where trac is installed, to make sure it's the same Python version being used.
3) I then copied it over to /directory/where/trac/is/plugins/ folder and chmod 755 the file egg file.
4) I then restarted http, unable to find a better way of restaring trac so this may be where my problem is. It didn't work. So I deleted the egg folder in plugins
5) Uploaded it via trac->administration->plugins and restarted httpd again. Nothing.
6) I realized I had to edit the trac.ini file and added helloworld.* = enabled under component and restarted the web server.
It's quite possible it's me but any help would be greatly appreciated!
Its the helloworld plugin from trachack, essentially displays hello world and theres a button. There are no error messages provided, hence why googling was hard.
I'm assuming that it's using root and that's the user I built it with. I will look into seeing if it's anybody else, just taking a quick look though I don't see anything else that could be using it. I only copied the egg file to the plugins folder, I set up another folder elsewhere and built it and cp to the plugins folder. I'm glad to know I was doing that right because looking up documentation on how to restart trac turns up practically nothing, they just say restart trac or restart apache. I will look into the logs later on tomorrow. Thanks for the replies! Also we are using trac .12.1.
So after looking at the log files it seems that it doesn't even load the plugin, can't find anywhere that says it's loading or any errors with it. Now we have a few trac sites for various projects and one of the sites already has plugins installed so I went there and and put the test plugin there and checked logs and it didn't work either. So I'm just going to conclude it's the plugin or something we already have in place and it's not me. I believe I'm going to try and make one and test it. Thanks for the help!
It sounds like you built the egg correctly. After you copy it into your plugins folder, change the file's owner and group (I'm assuming you're on Linux since you mentioned chmod) to match the account that your webserver uses. I'm not sure if that's strictly necessary, but it's what's always worked for me.
I may be misreading your #4, but it sounds like you copied the whole egg folder to your plugins directory. Only the .egg file needs to be copied over, it's a self-contained package. I don't think Trac looks for .egg files in subdirectories.
Restarting your webserver is the easiest way to restart Trac. Actually, I'm not aware of any other way to do it.
When it comes to plugin problems, Trac's log is usually a very good source of information. I recommend setting Trac's log level to DEBUG, then shut down the web server. Clear out the contents of Trac's log file, then start the web server and make a copy of Trac's log file after the server has completely come back online. Do this process twice: once with the plugin installed and once without it installed. The difference in the logfiles should give you a good indication of what the problem is. Once you get accustomed to what your logs normally look like, you'll be able to read the log in place without clearing it out and generating two versions of it.
By the way, what Trac version are you using?
Check the Trac version and downloaded plugin
instead python setup.py bdist_egg try python setup.py install
Quite an old thread, but since I ran into the same problem at one point:
Make sure you build the .egg with the same Python version that you use to run Trac with!Backwards compatibility between Python versions does not matter here, because Trac reads information about the Python version out of the .egg file before it even loads it, to make sure it is compatible.
(Small version numbers should not matter, so you should be able to run a .egg with Python 2.7.10 when it was built with 2.7.3, but not when it was built with 2.6.x. Look at the Version number that is written into the .egg file name.)