I am currently working on RnD in TensorFlow (CPU Version), but unable to decide on the basic requirement for my system for training on large datasets or may be I stumbled upon a possible bug in TensorFlow library.
The Official TensorFlow documentation, nowhere suggests any specific requirement for the system to be building and running TensorFlow programs on. From what I can understand, if that can be run over Windows, Linux, Mac along with Android, iOS and also over embedded systems like RaspberryPi, I suppose there should not be any such hardware requirement for the same.
However, while in the process of initial research, I tried running the TensorFlow Seq2Seq model (translating English to French https://www.tensorflow.org/tutorials/seq2seq), where the training and test datasets end up taking around 7-8 GB of diskspace initially and 20-22Gb on a whole. Once the translate.py python script is executed, it ends up choking the memory and pushing disk utilization to 98% and 100% respectively.
My current system runs Windows 8.1 64 bit OS, Core i5 5200U clocking at 2.2 GHz, 8GB RAM and around 70GB free space on HDD (specifically allotted for TensorFlow usage). But even after allowing my system to run over 7-8 hours (with no other application running) it got stuck multiple times and usually after the memory utilization peeks to around 100% after tokenizing the datasets.
Though I am not sure, but I suppose the TensorFlow learning graph is being created inside the RAM and once it expands to around all the memory space, the program ends up in un-ending loop waiting for memory to get cleared and then increase the learning graph.
So the whole drills down to 3 questions:
Does TensorFlow uses RAM for building and saving Learning Graph? If so, is it possible to get choked in a similar fashion?
From a business perspective, is there a minimum hardware requirement for training such a system?
If it is not the system requirement, can this be a possible bug in TensorFlow library which pushes it into an unending loop waiting for memory to get cleared?
Update
After running the python script for over 30 hours continuously, the process seems to have stuck at the same place for past 14 hours while "Reading development and training data". Refer image below for further investigation:
As soon as I was about to shut down the program, the same started responding again and I waited for another 15-20 minutes and finally I got the answer from the OS itself. It was indeed low RAM that was causing the problem. Attaching the screen grab of the Windows Alert of system running low on memory for reference, incase anyone gets caught in same situation.
UPDATE
I tried taking a VM instance on Google Cloud Platform. This machine had 2 x Intel Xeon (R) each running at 2.23 GHZ, with 13GB RAM and 50GB storage. But the result was same in this situation also even though the application was utilising more than 10.5 GB RAM. Seems like this tutorial script needs a very intense system probably a Super Computer with atleast 32 GB RAM to run and execute completely. I might look to write/arrange my own dataset now. However, this must be taken as future enhancement to use Persistent Storage (HDD/SSD) to create the Graph instead of RAM so as to avoid chocking of Memory.
Related
I have a problem I'm not sure how to tackle.
I currently am trying to run a program with JupyterLab on my Raspberry Pi 4, but when trying to load a Tensorflow Model, I get the following warning in the terminal:
Allocation of 360087552 exceeds 10% of free system memory
Now, this is confounding to me. The model it's trying to load is only about 900mb. The Raspberry Pi model I have has 8gb of RAM, same as my laptop. It's using a 64gb SD card with 42.8gb free space (more than my laptop). Yet, despite having the same amount of RAM and more free space than my laptop (which runs everything without issue), it is unable to load the model, and the kernel crashes.
I've already done everything I could think of to free up memory, including Expanding the Filesystem and increasing the Memory Split to 256 in the raspi-config, and increasing the CONF_SWAPSIZE to 1024.
So, is there anything further I can do to try to resolve this issue, or is this merely a limitation of the Raspberry Pi 4, and should I look into alternative and more powerful single board computers?
Thanks for the help,
Sam
My suspicion is that you're using the 32 bit os with pae, that only allows 3gb per process. The allocation given would exceed that. Did you try with the 64 bit os?
I have a new Surface Book 2 with Windows Build 18.09 on it. The processor is an i7 8.th generation (8 cores) and it has 16 GB of RAM.
When I run any type of Python Code, the performance is unbearibly slow. I really do not think it is normal Python performance on this Laptop due to the following reasons:
the resource monitor shows 5% processor usage for any python code I run. Considering 8 cores being 100%, the python process should definitely use 12,5%.
I have another Windows 2-1 tablet (Miix 520) that has an i7 7th generation processor and that is normally throattling a lot. Still this tablet runs the same python code with the same python interpreter around 60% faster - not to speak of my Linux laptop with i7 7th generation running the code around 4-5 times faster.
I have no clue what I can do to get appropriate python performance. One comment I found elsewhere was the explanation that Windows Defender is slowing down python processes. I can not deactivate it because it is a working computer that is partially managed by IT. However, I can blacklist folders and files which I did for the whole Anaconda folder - I use Anaconda in order to manage python environments on Windows - and for python.exe. Unfortunately, this did not bring any improvements.
Does anyone has any experiences / explanations for such low performance of python on Windows (or the Surface Book 2 in particular)? Does any one have suggestions what could be done in order to get "normal" python performance?
It turned out that Windows Defender is slowing down the execution of of python processes.
Blacklisting python.exe and the folder from where I execute my script in Windows Defender leads to a significant performance boost.
Another reason, I found out about, is that Windows seems to have lower disc access rates than Linux. This was significant in my case because I processed 50.000 images.
Is there a way to limit the memory consumption of Python on Windows?
I'm running some experiments with TensorFlow 1.3.0, Python 3.6.3, Windows 7, all 64-bit. Problem: even a very small error in a TensorFlow program can cause runaway memory consumption. Unfortunately, Windows tries to use virtual memory to keep the program running, with the result that it goes so deep into disk thrashing that the machine locks up for several minutes.
I'd like to solve the problem by switching to 32-bit Python, but it seems TensorFlow only supports 64-bit.
Does any element of the stack provide a way to limit the memory consumption? Of course I'm not expecting a buggy program to work; what I want it to do once memory consumption exceeds a certain threshold is crash, so it doesn't go into disk thrashing and lock up the machine.
I am running a program that does simple data processing:
parses text
populates dictionaries
calculates some functions over the resulting data
The program only uses CPU, RAM, and HDD:
run from Windows command line
input/output to the local hard drive
nothing displayed on or printed to screen
no networking
The same program is run on:
desktop: Windows 7, i7-930 CPU overclocked #3.6 GHz (with matching memory speed), Intel X-25M SSD
laptop: Windows XP, Intel Core2 Duo T9300 #2.5GHz, 7200 rpm HDD
The CPU is 1.44 faster frequency, HDD is 4 times higher benchmark score (Passmark - Disk Mark). I found the program runs just around 1.66 times faster on the desktop. So apparently, the CPU is the bottleneck.
It seems there's only 15% benefit from the i7 Core vs Intel Core2 Duo architecture (most of the performance boost is due to the straight CPU frequency). Is there anything I can do in the code to increase the benefit of the new architecture?
EDIT: forgot to mention that I use ActivePython 3.1.2 if that matters.
The increasing performance of hardware brings in most cases automatically results in benefit to user applications. The much maligned "GIL" means that you may not be able to take advantage of multicores with CPython unless you design your program to take advantage via various multiprocessing modules / libraries.
SO discussion on the same : Does python support multiprocessor/multicore programming?
A related collation of solutions on python wiki: http://wiki.python.org/moin/ParallelProcessing
Split your processing into multiple threads. Your particular i7 should be able to support up to 8 threads in parallel.
Consider repeating on regular HDD's - that SSD could well result in a substantial performance difference depending on caches, and the nature of that data.
Is Python generally slower on Windows vs. a *nix machine? Python seems to blaze on my Mac OS X machine whereas it seems to run slower on my Window's Vista machine. The machines are similar in processing power and the vista machine has 1GBs more memory.
I particularly notice this in Mercurial but I figure this may simply be how Mercurial is packaged on windows.
I wanted to follow up on this and I found something that I believe is 'my answer'. It appears that Windows (vista, which is what I notice this on) is not as fast in handling files. This was mentioned by tony-p-lee.
I found this comparisons of Ubuntu vs Vista vs Win7. Their results are interesting and like they say, you need to take the results with a grain of salt. But I think the results lead me to the cause. Python, which I feel was indirectly tested, is about equivalent if not a tad-bit faster on Windows.. See the section "Richards benchmark".
Here is their graph for file transfers:
(source: tuxradar.com)
I think this specifically help address the question because Hg is really just a series of file reads, copies and overall handling. Its likely this is causing the delay.
http://www.tuxradar.com/content/benchmarked-ubuntu-vs-vista-vs-windows-7
No real numbers here but it certainly feels like the start up time is slower on Windows platforms. I regularly switch between Ubuntu at home and Windows 7 at work and it's an order of magnitude faster starting up on Ubuntu, despite my work machine being at least 4x the speed.
As for runtime performance, it feels about the same for "quiet" applications. If there are any GUI operations using Tk on Windows, they are definitely slower. Any console applications on windows are slower, but this is most likely due to the Windows cmd rendering being slow more than python running slowly.
Maybe the python has more depend on a lot of files open (import different modules).
Windows doesn't handle file open as efficiently as Linux.
Or maybe Linux probably have more utilities depend on python and python scripts/modules are more likely to be buffered in the system cache.
I run Python locally on Windows XP and 7 as well as OSX on my Macbook. I've seen no noticable performance differences in the command line interpreter, wx widget apps run the same, and Django apps also perform virtually identically.
One thing I noticed at work was that the Kaspersky virus scanner tended to slow the python interpreter WAY down. It would take 3-5 seconds for the python prompt to properly appear and 7-10 seconds for Django's test server to fully load. Properly disabling its active scanning brought the start up times back to 0 seconds.
With the OS and network libraries, I can confirm slower performance on Windows, at least for versions =< 2.6.
I wrote a CLI podcast-fetcher script which ran great on Ubuntu, but then wouldn't download anything faster than about 80 kB/s (where ~1.6 MB/s is my usual max) on either XP or 7.
I could partially correct this by tweaking the buffer size for download streams, but there was definitely a major bottleneck on Windows, either over the network or IO, that simply wasn't a problem on Linux.
Based on this, it seems that system and OS-interfacing tasks are better optimized for *nixes than they are for Windows.
Interestingly I ran a direct comparison of a popular Python app on a Windows 10 x64 Machine (low powered admittedly) and a Ubuntu 14.04 VM running on the same machine.
I have not tested load speeds etc, but am just looking at processor usage between the two. To make the test fair, both were fresh installs and I duplicated a part of my media library and applied the same config in both scenarios. Each test was run independently.
On Windows Python was using 20% of my processor power and it triggered System Compressed Memory to run up at 40% (this is an old machine with 6GB or RAM).
With the VM on Ubuntu (linked to my windows file system) the processor usage is about 5% with compressed memory down to about 20%.
This is a huge difference. My trigger for running this test was that the app using python was running my CPU up to 100% and failing to operate. I have now been running it in the VM for 2 weeks and my processor usage is down to 65-70% on average. So both on a long and short term test, and taking into account the overhead of running a VM and second operating system, this Python app is significantly faster on Linux. I can also confirm that the Python app responds better, as does everything else on my machine.
Now this could be very application specific, but it is at minimum interesting.
The PC is an old AMD II X2 X265 Processor, 6GB of RAM, SSD HD (which Python ran from but the VM used a regular 5200rpm HD which gets used for a ton of other stuff including recording of 2 CCTV cameras).