I need to get current bandwidth usage on given interface in my Python code. How can I achieve this?
I tried to extract the value from /proc/net/dev twice with short sleep interval (e.g. 0.3s) between two consecutive calls and compute the difference in bytes divided by time. It works but I do not know if I can trust this results. I'm looking for some more elegant solutions. Any tools, libraries, or simple algorithms?
I will appreciate all suggestions!
There are libraries for this, like PSutil, but if you are on *Nix I would opt for using the underlying system.
Although you could use /proc/net/dev, rather use /sys/class/net/eth0/statistics since you will not have to parse anything since there are individual files for each statistic
Related
In a programming language like Python Which will have better efficiency? if i use a sorting algorithm like merge sort to sort an array or If I use a built in API like sort() to sort the array? If Algorithms are independent of programming languages, then what is the advantage of algorithms over built in methods or API's
Why to use public APIs:
The built in methods were written and reviewed by very experienced and many coders, and a lot of effort was invested to optimize them to be as efficient as it gets.
Since the built in methods are public APIs, it is also means they are constantly used, which means you get a massive "free" testing. You are much more likely to detect issues in public APIs than in private ones, and once something is discovered - it will be fixed for you.
Don't reinvent the wheel. Someone already programmed it for you, use it. If your profiler says there is a problem, think about replacing it. Not before.
Why to use custom made methods:
That said, the public APIs are general case. If you need something
very specific for your scenario, you might find a solution that will
be more efficient, but it will take you quite some time to actually
achieve better than the already optimize general purpose public API.
tl;dr: Use public APIs unless you:
Need it and can afford a lot of time to replace it.
Know what you are doing pretty well.
Intend to maintain it and do robust testing for it.
The libraries normally use well tested and correctly optimized algorythms. For example Python uses Timsort which:
is a stable sort (order of elements that compare equal is preserved)
in the worst case takes O( n log n ) comparisons to sort an array of n elements
in the best case (when the input is already sorted) runs in linear time
Unless you have special requirements that make you know that for your particular data sets one sort algorythm will give best result you can use the standard library implementation.
The other reason to build a sort by hand, is evidently for academic purposes...
I am little bit new to python and I have a large code base written in python 3.3.2 (32 bit). It uses numpy 1.7.1 and takes a very long time to run because of computationally intensive calculations.
I need to parallelize the code to increase the performance. I am thinking about using pypy to parallelize but am unsure how to use it with existing code.
I have search Google but couldn't find and appropriate or satisfactory answer. I also read about using cython but I am unsure how to use that as well.
Could anyone provide pointers on increasing the performance of my code?
Since you're new to Python, I highly recommend taking the time to survey all of the possibilities before jumping into something pypy which may not be appropriate for your needs. There are lots of ways to speed up NumPy code, and the best way really depends on exactly what you're doing.
A great starting off point is Ian Oszvald's High Performance Python tutorial. Don't just watch it: follow along and try out the examples!
From there, you should think about whether parallelization will help. There are several options for parallelization, like the stdlib's multiprocessing module, but a lot of people in the scientific space are using IPython's parallelization capabilities. For learning about this, check out Min Ragan-Kelley's IPython Parallel tutorial (pt 1, pt 2, pt 3).
Once you have some sense of what Python is capable of, pick a method for speeding up your code and try it out. When you run into more specific problems, StackOverflow will be able to provide more concrete answers than just some links to tutorials ;)
Recently, i am looking through some python modules to understand their behavior and how optimized their implementation are. Can any one tell what algorithm does python use to perform the set difference operations. One possible way to achieve set difference is by using hash tables which will involve an extra N space. I tried to find the source code of set operations but i am not able to find out the code location. Please help.
A set in python is a hash itself. So implementing difference for it is not as hard as you imagine. Looking from a higher level how does one implement set difference? Iterate over one of the collections and add to the result all elements that are not present in the other sequence.
I wrote a program that calls a function with the following prototype:
def Process(n):
# the function uses data that is stored as binary files on the hard drive and
# -- based on the value of 'n' -- scans it using functions from numpy & cython.
# the function creates new binary files and saves the results of the scan in them.
#
# I optimized the running time of the function as much as I could using numpy &
# cython, and at present it takes about 4hrs to complete one function run on
# a typical winXP desktop (three years old machine, 2GB memory etc).
My goal is to run this function exactly 10,000 times (for 10,000 different values of 'n') in the fastest & most economical way. following these runs, I will have 10,000 different binary files with the results of all the individual scans. note that every function 'run' is independent (meaning, there is no dependency whatsoever between the individual runs).
So the question is this. having only one PC at home, it is obvious that it will take me around 4.5 years (10,000 runs x 4hrs per run = 40,000 hrs ~= 4.5 years) to complete all runs at home. yet, I would like to have all the runs completed within a week or two.
I know the solution would involve accessing many computing resources at once. what is the best (fastest / most affordable, as my budget is limited) way to do so? must I buy a strong server (how much would it cost?) or can I have this run online? in such a case, is my propritary code gets exposed, by doing so?
in case it helps, every instance of 'Process()' only needs about 500MB of memory. thanks.
Check out PiCloud: http://www.picloud.com/
import cloud
cloud.call(function)
Maybe it's an easy solution.
Does Process access the data on the binary files directly or do you cache it in memory? Reducing the usage of I/O operations should help.
Also, isn't it possible to break Process into separate functions running in parallel? How is the data dependency inside the function?
Finally, you could give some cloud computing service like Amazon EC2 a try (don't forget to read this for tools), but it won't be cheap (EC2 starts at $0.085 per hour) - an alternative would be going to an university with a computer cluster (they are pretty common nowadays, but it will be easier if you know someone there).
Well, from your description, it sounds like things are IO bound... In which case parallelism (at least on one IO device) isn't going to help much.
Edit: I just realized that you were referring more to full cloud computing, rather than running multiple processes on one machine... My advice below still holds, though.... PyTables is quite nice for out-of-core calculations!
You mentioned that you're using numpy's mmap to access the data. Therefore, your execution time is likely to depend heavily on how your data is structured on the disc.
Memmapping can actually be quite slow in any situation where the physical hardware has to spend most of its time seeking (e.g. reading a slice along a plane of constant Z in a C-ordered 3D array). One way of mitigating this is to change the way your data is ordered to reduce the number of seeks required to access the parts you are most likely to need.
Another option that may help is compressing the data. If your process is extremely IO bound, you can actually get significant speedups by compressing the data on disk (and sometimes even in memory) and decompressing it on-the-fly before doing your calculation.
The good news is that there's a very flexible, numpy-oriented library that's already been put together to help you with both of these. Have a look at pytables.
I would be very surprised if tables.Expr doesn't significantly (~ 1 order of magnitude) outperform your out-of-core calculation using a memmapped array. See here for a nice, (though canned) example. From that example:
I need to measure the performance of a hard disk using python. What is the best/fastest/shortest/easiest approach to do it? It doesn't have to be overly accurate, just a ballpark value.
My actual goal is to write a small utility which will adjust the postgres settings to the best configuration for the given hardware.
My naive approach would be to write some files and measure the time how long it would take. I would try it for several block sizes, and then I would try to access some random positions within a large file. Any other ideas?
I would think your best bet would be using an external tool, Bonnie++ for example, and parse the program output. Even if you're not that concerned with precision there's no reason to reinvent the wheel. Why rewrite what's already there?
Start here: http://www.acnc.com/benchmarks.html
Get the source for one you like write something like it in Python.