I'm trying Python for the first time and am having a slowness issue. I wrote the program using Python 3.7 on Windows 10 64-bit. The program reads a 2.5GB text file on a server and splits it into 3 separate files on the local hard drive. It splits the file line by line based on the first character of each line.
On my Windows 10 machine it takes 25 minutes to complete, which I thought was long but it is a very large file.
I used pyinstaller to create an .exe file so users can run the program without installing Python (although I don't think this is part of the problem as it takes 25 minutes just running the .py from VS2017).
Two different users on Windows 7 machines can run the program in under 5 minutes, which is a vast difference from the 25 minutes on my Windows 10 box (and now verified on a 2nd Win10 box).
I've done extensive searching online but haven't found anything that helps me determine the cause yet. Some people say to exclude Python directories from Windows Defender scans but my Defender is turned off (we use McAfee).
Could someone maybe point me in the right direction to look why it is taking so much longer on Windows 10? I'm happy to post the code but I do not think it is a coding issue as it still works perfectly on both OS. I've been developing professionally for ~15 years now in a multitude of languages but never had a discrepancy this big.
EDIT:
I have tested on two Win7 machines and two Win10 machines now. Both Win7's run the program in 5 minutes. Both Win10 machines run the program in 25 minutes.
The Win7 PC has an Intel i5-3470 CPU # 3.20GHz with 8GB RAM
My Win10 PC has an Intel i7-6700 CPU # 3.40GHz with 16GB RAM
I can't check Win7 PC at this time, but my Win10 HD is Solid State Drive Non-Encrypted 256 GB FLASH - NAND (TLC) 120 TBW SATA-3.2 6 Gb/s 2.5 Inch
I have run the same exe on both machines but it takes 5x longer on Win10
We are both on the same network and I ran AT&T's online network speed test. The Win7 and Win10 machines both registered ~100MB down.
I also turned the Win10 to High Performance power option and no improvement.
The Python code is not multithreaded as I'm only iterating through one file once.
Thanks.
I have Python program which uses multiprocess module. I am trying to to check how to memory and process it uses over a time.
for checking memory I am using memory_profiler and it works perfectly fine. it gives me exactly what I want, a graph of memory usage over time.
is there any module I could try to check the CPU usage in similar fas
The psutil library is able to give system information (CPU / Memory usage)
psutil is a module providing an interface for retrieving information on running processes and system utilization (CPU, memory) in a portable way by using Python, implementing many functionalities offered by tools like ps, top and Windows task manager.
It currently supports Linux, Windows, OSX, Sun Solaris, FreeBSD,
OpenBSD and NetBSD, both 32-bit and 64-bit architectures, with Python
versions from 2.6 to 3.5 (users of Python 2.4 and 2.5 may use 2.1.3
version).
https://pypi.python.org/pypi/psutil
Is there a way to limit the memory consumption of Python on Windows?
I'm running some experiments with TensorFlow 1.3.0, Python 3.6.3, Windows 7, all 64-bit. Problem: even a very small error in a TensorFlow program can cause runaway memory consumption. Unfortunately, Windows tries to use virtual memory to keep the program running, with the result that it goes so deep into disk thrashing that the machine locks up for several minutes.
I'd like to solve the problem by switching to 32-bit Python, but it seems TensorFlow only supports 64-bit.
Does any element of the stack provide a way to limit the memory consumption? Of course I'm not expecting a buggy program to work; what I want it to do once memory consumption exceeds a certain threshold is crash, so it doesn't go into disk thrashing and lock up the machine.
I was trying to probe the Python 32bit memory limit. So I wrote the little program
a=[]
while 1:
a.append(chr(65))
and watched the Windows task manager for the memory consumption of python.exe.
Firstly, I was surprised that it is occationally reduced (almost halved sometimes). Second, the amount only goes up to about 500MB (I believe on another 64bit machine it rose endlessly).
The computer has 4GB memory, windows boot>3GB is supposingly active and I patched the python.exe with imagecfg.exe -l. No other relevant processes are running and total memory does not exceed 40%. I believe the very same procedure worked on another computer though.
Any suggestions how I can check if my python can go up to 3GB now?
Is Python generally slower on Windows vs. a *nix machine? Python seems to blaze on my Mac OS X machine whereas it seems to run slower on my Window's Vista machine. The machines are similar in processing power and the vista machine has 1GBs more memory.
I particularly notice this in Mercurial but I figure this may simply be how Mercurial is packaged on windows.
I wanted to follow up on this and I found something that I believe is 'my answer'. It appears that Windows (vista, which is what I notice this on) is not as fast in handling files. This was mentioned by tony-p-lee.
I found this comparisons of Ubuntu vs Vista vs Win7. Their results are interesting and like they say, you need to take the results with a grain of salt. But I think the results lead me to the cause. Python, which I feel was indirectly tested, is about equivalent if not a tad-bit faster on Windows.. See the section "Richards benchmark".
Here is their graph for file transfers:
(source: tuxradar.com)
I think this specifically help address the question because Hg is really just a series of file reads, copies and overall handling. Its likely this is causing the delay.
http://www.tuxradar.com/content/benchmarked-ubuntu-vs-vista-vs-windows-7
No real numbers here but it certainly feels like the start up time is slower on Windows platforms. I regularly switch between Ubuntu at home and Windows 7 at work and it's an order of magnitude faster starting up on Ubuntu, despite my work machine being at least 4x the speed.
As for runtime performance, it feels about the same for "quiet" applications. If there are any GUI operations using Tk on Windows, they are definitely slower. Any console applications on windows are slower, but this is most likely due to the Windows cmd rendering being slow more than python running slowly.
Maybe the python has more depend on a lot of files open (import different modules).
Windows doesn't handle file open as efficiently as Linux.
Or maybe Linux probably have more utilities depend on python and python scripts/modules are more likely to be buffered in the system cache.
I run Python locally on Windows XP and 7 as well as OSX on my Macbook. I've seen no noticable performance differences in the command line interpreter, wx widget apps run the same, and Django apps also perform virtually identically.
One thing I noticed at work was that the Kaspersky virus scanner tended to slow the python interpreter WAY down. It would take 3-5 seconds for the python prompt to properly appear and 7-10 seconds for Django's test server to fully load. Properly disabling its active scanning brought the start up times back to 0 seconds.
With the OS and network libraries, I can confirm slower performance on Windows, at least for versions =< 2.6.
I wrote a CLI podcast-fetcher script which ran great on Ubuntu, but then wouldn't download anything faster than about 80 kB/s (where ~1.6 MB/s is my usual max) on either XP or 7.
I could partially correct this by tweaking the buffer size for download streams, but there was definitely a major bottleneck on Windows, either over the network or IO, that simply wasn't a problem on Linux.
Based on this, it seems that system and OS-interfacing tasks are better optimized for *nixes than they are for Windows.
Interestingly I ran a direct comparison of a popular Python app on a Windows 10 x64 Machine (low powered admittedly) and a Ubuntu 14.04 VM running on the same machine.
I have not tested load speeds etc, but am just looking at processor usage between the two. To make the test fair, both were fresh installs and I duplicated a part of my media library and applied the same config in both scenarios. Each test was run independently.
On Windows Python was using 20% of my processor power and it triggered System Compressed Memory to run up at 40% (this is an old machine with 6GB or RAM).
With the VM on Ubuntu (linked to my windows file system) the processor usage is about 5% with compressed memory down to about 20%.
This is a huge difference. My trigger for running this test was that the app using python was running my CPU up to 100% and failing to operate. I have now been running it in the VM for 2 weeks and my processor usage is down to 65-70% on average. So both on a long and short term test, and taking into account the overhead of running a VM and second operating system, this Python app is significantly faster on Linux. I can also confirm that the Python app responds better, as does everything else on my machine.
Now this could be very application specific, but it is at minimum interesting.
The PC is an old AMD II X2 X265 Processor, 6GB of RAM, SSD HD (which Python ran from but the VM used a regular 5200rpm HD which gets used for a ton of other stuff including recording of 2 CCTV cameras).