I have a large project that runs on an application server. It does pipelined processing of large batches of data and works fine on one Linux system (the old production environment) and one windows system (my dev environment).
However, we're upgrading our infrastructure and moving to a new linux system for production, based on the same image used for the existing production system (we use AWS). The python version (2.7) and libraries should be identical because of this, we're verifying this on our own using file hashes, also.
Our issue is that when we attempt to start processing on the new server, we receive a very strange output written to standard out followed by hanging of the server, "Removing descriptor: [some number]". I cannot duplicate this on the dev machine.
Has anyone ever encountered behavior like this in python before? Besides modules in the python standard library we are also using eventlet and beautifulsoup. In the standard library we lean heavily on urllib2, re, cElementTree, and multiprocessing (mostly the pools).
wberry was correct in his comment, I was running into a max descriptors per process issue. This seems highly dependent on operating system. Reducing the size of the batches I was having each processor handle to below the file descriptor limit of the process solved the problem.
Related
I have a python script that works fine on my main computer without problems. But when I uploaded it to the Ubuntu server it started crashing. I thought for a long time what the problem was and looked at the system logs. It turned out that ubuntu automatically forcibly terminates the script due to lack of memory (server configuration is 512 MB of RAM), how can I debug the program on the consumed memory in different work options?
Have a look at something like Guppy3, which includes heapy, a 'heap analysis toolset' that can help you find where the memory's being used/held. Some links to information on how to use it are in the project's README.
If you have a core, consider using https://github.com/vmware/chap, which will allow you to look at both python and native allocations.
Once you have opened the core, probably "summarize used" is a good place to start.
I have been working with PyCharm for quite some time now and I recently upgraded my storing system with a NAS.
Everything is working fine except one : PyCharm scans through my files to reindex them very very often. This makes me losing a lot of time waiting for it to end.
When the reindexing occurs:
When a script ends
When a debugging session ends
When PyCharm loses the focus, i.e. I use another application
So it happens basically ALL the time, taking quite a long time (several minutes sometimes).
Misc.:
Windows 10
PyCharm Community Edition 2018.1
Netgear - ReadyNas 422
Do you have any ideas to solve this issues ?
So I have contacted the IntelliJ support and here is their response:
Working with network drives/folders is not supported officially yet.
Using remote development features is recommended (remote interpreter,
deployment etc). Here is more detailed answer
https://intellij-support.jetbrains.com/hc/en-us/community/posts/207069145/comments/207464249.
What I end up doing, which is really not ideal, is to create a local copy of my projects environment and syncing it with a folder in my NAS. To do so I used the SyncBackPro software.
I'm using PyCharm both at home and at work with code stored on a Samba share (using its remote interpreter feature). I don't encounter consistent reindexing but by default it does not support file system notifications to know when a file changed.
However, as a programmer this shouldn't discourage you! You can drop in your own file system notifier that connects to your remote system (assuming your NAS runs Linux and supports SSH) and thus avoid the performance drop.
I actually wrote such a proxy to run the fsnotifier on a remote system a few years ago and I'm still using it. If you are interested, check out https://github.com/ThiefMaster/fsnotifier-remote
Some things in the repo are outdated (JetBrains removed this stupid file size check for example), but it should still provide you a good basis to start from if you are interested in using it.
The Issue
I am using my laptop with Apache to act as a server for a local project involving tensorflow and python which uses an API written in Flask to service GET and POST requests coming from an app and maybe another user on the local network.The problem is that the initial page keeps loading when I specifically import tensorflow or the object detection folder within the research folder in the tensorflow github folder, and it never seems to finish doing so, effectively getting it stuck. I suspect the issue has to do with the packages being large in size, but I didn't have any issue with that when running the application on the development server provided with Flask.
Are there any pointers that I should look for when trying to solve this issue? I checked the memory usage, and it doesn't seem to be rising substantially, as well as the CPU usage.
Debugging process
I am able to print basic hello world to the root page quite quickly, but I isolated the issue to the point when the importing takes place where it gets stuck.
The only thing I can think of is to limit the number of threads that are launched, but when I limited the number of threads per child to 5 and number of connections to 5 in the httpd-mpm.conf file, it didn't help.
The error/access logs don't provide much insight to the matter.
A few notes:
Thus far, I used Flask's development server with multi-threading enabled to serve those requests, but I found it to be prone to crashing after 5 minutes of continuous run, so I am now trying to use Apache using the wsgi interface in order to use Python scripts.
I should also note that I am not servicing html files, just basic GET and POST requests. I am just viewing them using the browser.
If it helps, I also don't use virtual environments.
I am using Windows 10, Apache 2.4 and mod_wsgi 4.5.24
The tensorflow module being a C extension module, may not be implemented so it works properly in Python sub interpreters. To combat this, force your application to run in the main Python interpreter context. Details in:
http://modwsgi.readthedocs.io/en/develop/user-guides/application-issues.html#python-simplified-gil-state-api
I'm trying to watch for changes in .py files, in a directory.
I went through existing solution.
I'm curious on how django library solves this problem. The development server is restarted on file changes.
The code can be found in django.utils.autoreload. The autoreloader uses a separate thread that watches any python module that has been imported, and any translation file.
If inotify is available, Django uses that to listen to change events. Otherwise, it checks the timestamps of every file every second. If there are any changes, the process is restarted.
Django's autoreloader may not be the best source of inspiration. Better options may be Watchman (with the appropriate python bindings) or the pure-python alternative Watchdog.
Fast forward to April 2019:
With django 2.2 pywatchman as part of Watchman will be supported and pyinotify (being unmaintained since mid 2015) is dropped:
If you're using Linux or MacOS and install both pywatchman and the
Watchman service, kernel signals will be used to autoreload the server
(rather than polling file modification timestamps each second). This offers
better performance on large projects, reduced response time after code changes,
more robust change detection, and a reduction in power usage.
source: django-admin
When using Watchman with a project that includes large non-Python
directories like node_modules, it's advisable to ignore this directory
for optimal performance.
See the watchman documentation for information on how to do this.
I am writing a piece of code that uses the Box.com Python SDK. The SDK uses the requests module to communicate with Box.com as per the API documentation. For my purposes, I need to make several GET and POST requests in a row, some of which could be used to transfer files. The issue that I'm running into is this:
On Linux (Ubuntu 13.10), each request takes a relatively long time (5 to 15 seconds) to get through, though transfer speeds for file transfers are as expected in the context of my network connection.
On Windows 8.1, running the exact same code, the requests go through really fast (sub-second fast).
On both platforms I am using the same version of iPython (1.1.0) and of the requests module (1.2.3) under Python 2.7. This is particularly problematic for me because the code I'm working on will eventually be implemented on Linux machines.
Is this problem someone has encountered before? I would love to hear from anyone with some ideas on what the issue might be. I have yet to try it on a different Linux installation to see if it is a problem with the specific setup.
Thanks.
EDIT 1
So, I decided to check this using virtual machines. Using the same Debian virtual machine under Windows all the responses were fast, but under Ubuntu they were slow. I then made a Ubuntu 12.04 live USB and ran the code on that, and the responses were fast there as well.
So, it's not Python or Linux in general, it's my particular installation and I have no idea how to diagnose the problem :(
Use a tool such as wireshark (which needs to be run with sudo on most distributions) to log the individual network packets when your code makes the API requests, to determine what is taking so long.
My guess is the following possibilities are most likely:
For some reason your Ubuntu installation is picking up the wrong DNS server list, and DNS lookups are timing out.
IPv6 issue (which may appear to be a DNS issue, too). Disable IPv6.