EMFILE: Too many files open - python

I'm trying to set up an api on azure's web app service using bottle + anaconda packages.
I can't simply use a copy of the site-packages folder because numpy is involved. Instead, in addition to the site-packages folder I must also give numpy access to the mkl binaries. So I copy the Anaconda\envs\{ENV_NAME}\Library\bin folder into the app and add it to %PATH%. That folder has less than 200 files in it, so I'm surprised seeing the following error during the deployment:
2020-10-29T04:34:21.3218237Z ##[error]Error: EMFILE: too many open files, open 'D:\a\_temp\temp_web_package_058969368946595324\site-packages\statsmodels\tsa\arima\datasets\__init__.py'
Everything builds and runs as long as I don't include the bin folder to %PATH%
No, I'm not close to my file size limit on the azure web app service. Has anyone run into this before?

This error happens because of the XDT Transform.
During an XDT Transform, all contents of the original package are transformed and then zipped up. This error is thrown if the deployment is significantly large.

Related

How to bundle Python apps with asset dependencies for end users?

I have a Tkinter app that uses images included in the same folder as the .py file. pyinstaller script.py produces an executable that runs but does not open any windows. This is because it is looking for images that don't exist in the same subdirectory. When I copy the important images to the dist folder Pyinstaller creates, the application runs correctly.
However, I would like to have a single executable that I can share with other users that doesn't also require them to have the images stored. The images should be bundled with the software somehow, like how commercial software (usually) doesn't require you to download assets separately from the program itself.
Is there a way to bundle Python programs and the assets they use into single-click applications?
Note that I am using Python 3 on Linux Mint. I am also something of a novice, so don't be surprised if I'm missing something obvious here.
It appears I've solved my own problem.
Instead of having the images included in the same folder as main.py and using the resulting short relative filepath to reach them, install the images in an appropriate space in the system directory tree (I used /home/$USER$/.$PROGRAMNAME$/) and have the program access the files using the absolute path to that directory. This will allow you to copy the program anywhere on your computer you want and have it run without a problem.
However, if you want to share it with someone else, you'll need to also include an installation script that places the assets in the correct directory on their computer.

How to package a Pycharm python project for upload to AWS Lambda?

Using pycharm on Windows.
I have created a zip file for upload to AWS Lambda the manual way:
1) Install the modules manually into a directory other than the default directory.
2) Create my .py code file
3) Zip the contents of the project folder
4) Upload that zip folder to Lambda
I am new to Pycharm and with a project I see that there are a whole bunch of files and folders that I do not understand.
I tried to zip the entire Pycharm project contents and upload - that did not work. It looks like I need to run some kind of setup that creates the proper folder structure and files that have the correct content.
Any help would be appreciated.
For all those still stuck with this, I have a few suggestions which could possibly resolve the issue altogether:
Use pip's -t option to specify the Application Directory
Using Pip's -t option, one can specify the Application directory. It's better than using the pycharm's package installer, as we can specify the installation directory with this.
Zip the complete Application directory (Answer's your question)
Go inside your Pycharm project directory -> select all -> Right Click -> send to compressed (zip). This may result in the inclusion of some unneeded directories (__pycache__, .idea), but would not affect the program execution. If needed, you may skip those two directories while creating the zip.
I believe you were zipping the project directory, rather than compressing the contents of the Project directory.
As I also answered here Jetbrains now offers the AWS Toolkit which allows local and remote development of Lambda functions.
Despite some lingering issues it works quite well. Still finding my way with it.
It includes packaging and deploying.
Toolkit page on Jetbrains website

Can't create a virtual environment in the Google Drive folder

I'm using Google Drive to keep a copy of my code projects in case my computer dies (I'm also using GitHub, but not on some private projects).
However, when I try to create a virtual environment using virtualenv, I get the following error:
PS C:\users\fchatter\google drive> virtualenv env
New python executable in C:\users\fchatter\google drive\env\Scripts\python.exe
ERROR: The executable "C:\users\fchatter\google drive\env\Scripts\python.exe" could not be run: [Error 5] Access is denied
Things I've tried:
I thought it was because the path to the venv included blank spaces, but the command works in other paths with blank spaces. I also tried installing the win32api library, as recommended in the virtualenv docs, but it didn't work.
running the PowerShell as an administrator.
Any ideas on how to solve this? My workaround at the moment is to create the venv outside of the Google Drive, which works but is inconvenient.
After running into the same issue and playing with it for several hours, it doesn't seem possible. It has nothing to do with spaces in file/folder names. I've tested that. It seems that Google Drive Stream perform some action to the file/folder after a period of time that makes python loose its path the files. For instance, you can take clone a python module into a Google Drive Stream folder, do a "pip install -e ./", and it will work in the virtevn for a few minutes, for instance importing it into a python shell. But after a few minutes you can no longer import the module. I suspect that Google Drive Stream is simply not fully compatible with all filesystem system calls, one of which is being used by python.
Don't set up a virtual env in a cloud synced folder, nor should you run a python script from such a folder. It's a bad idea. They are not meant for version control. Write access (modifying files) to the folder is limited because in your case Google drive will periodically sync the folder which will prevent exclusive write access to the folder almost always.
TLDR; One cant possibly modify files while they are being synced.
I suggest you stick to git for version control.

Why the parameter --auto-reload is not working if the addons path content are links to modules?

Description
Normally if you change your python code means, you need to restart the server in order to apply the new changes.
If the --auto-reload parameter is enabled means, you don't need to restart the server. It enables auto-reloading of python files and xml files without having to restart the server. It requires pyinotify. It is a Python module for monitoring filesystems changes.
Previous Problem
I got the error:
ERROR ? pyinotify: add_watch: cannot watch /home/user/.local/share/Odoo/addons/8.0 WD=-1, Errno=No space left on device (ENOSPC)
But I followed the advice of this link and now I don't get that error anymore:
sysctl -n -w fs.inotify.max_user_watches=16384
After this I got this in the server log:
openerp.service.server: Watching addons folder /opt/odoo_8/src/linked-addons
openerp.service.server: AutoReload watcher running
That means that's working properly. And in fact I tested it with a physical addon path and it worked.
Current Problem
I have all my modules en several folders but I only use one addons path: /opt/odoo_8/src/linked-addons. This folder contains all the links of the modules that I'm using. All modules are working well when I run Odoo.
But the problem is that pyinotify is not able to check the files beyond the links and it doesn't reload the files well. What I should do to fix this? Is there a way that pyinotify can recognise the content of the links?
PD: I don't want to change my way of managing the modules folders in Odoo.
The only solution I found is to isolate the custom modules in a different folder. I have added the new folder to the addons path in the configuration file and now everything works fine
addons_path = /opt/odoo_8/src/linked-addons,/opt/odoo_8/src/custom
PD: I saved the module folders directly in the custom folder instead of saving the links to the folders.

MongoDB window closes automatically when I try to open

I'm trying to install MangoDB in my windows machine 64-bit. I'm following official documentation to install the same. I have installed successfully as stated and stuck at Run MongoDB section at step 2.
http://docs.mongodb.org/manual/tutorial/install-mongodb-on-windows/
When I run the file "C:\Program Files\MongoDB\bin\mongod.exe" window opens and automatically closes. Not sure why this is happening, I Googled and tried troubleshooting the same but no luck. I even added MongoDB path to windows environment variables as well. But still it doesn't work.
Anyone faced this issue? Any suggestions would be helpful.
Thanks!
from doc:
MongoDB requires a data directory to store all data. MongoDB’s default
data directory path is \data\db. Create this folder using the
following commands from a Command Prompt:
md \data\db
You can specify an alternate path for data files using the
--dbpath option to mongod.exe, for example:
C:\mongodb\bin\mongod.exe --dbpath d:\test\mongodb\data
The reason of your problem is that you don't have that data directory.
Create a folder named 'data' with sub-folder 'db' in the root of the folder where your MongoDB Server is installed.
Like in my case, it was installed on C:/Program Files/MongoDB/ and the mongod.exe, mongo.exe resides in C:/Program Files/MongoDB/Server/4.0/bin/.
So, I created a folder namely "data" and a sub-folder in it namely "db".
C:/data/db
Sometimes, there might be a chance that "data" folder already exists in the root drive. Then, just create a sub-folder named "db" in it.
After all of this, close mongod.exe & mongo.exe if running. And open them again in order to see if the problem is gone! I hope that solves the problem.
As laike9m said, MongoDB requires the directory data/db to be created. I just solved the same problem and I did it by creating the directory in "C:/ " (working on windows).
Yep I ran into the same problem. go to the root of the drive that mongo is installed 'I had mongo installed on drive I:' so I clicked on drive I: and then I created a folder and called it data then I created another folder and called it db. note it has to be done right at the root so as soon as you click onto the drive if you try to put it in another folder on the drive Mongo will not be able to find it.

Categories

Resources