If I get the current time using either:
datetime.datetime.now()
or,
time.time()
Does it make a system call to get the time? Or does the Python process have its own time service that is called to get the time?
The reason I am asking this question is to understand whether Python has to make a context switch to get the current time. Because if it makes a system call to get the current time, then that's quite an expensive operation. But if it has its own time service that, then the operation is not as expensive.
Related
More out of curiosity, I was wondering how might I make a python script sleep for 1 second without using the time module?
Is there a computation that can be conducted in a while loop which takes a machine of n processing power a designated and indexable amount of time?
As mentioned in comments for your second part of question:
The processing time is depends on the machine(computer and its configuration) you are working with and active processes on it. There isnt fixed amount of time for an operation.
It's been a long time since you could get a reliable delay out of just trying to execute code that would take a certain time to complete. Computers don't work like that any more.
But to answer your first question: you can use system calls and open a os process to sleep 1 second like:
import subprocess
subprocess.run(["sleep", "1"])
I have implemented kalman filter. I want to find out how much of cpu energy is being consumed by my script. I have checked other posts on Stackoverflow and following them I downloaded psutil library. Now, I am unaware of where to put the statements to get the correct answer. Here is my code:
if __name__ == "__main__":
#kalman code
pid = os.getpid()
py = psutil.Process(pid)
current_process = psutil.Process();
memoryUse = py.memory_info()[0]/2.**30 # memory use in GB...I think
print('memory use:', memoryUse)
print(current_process.cpu_percent())
print(psutil.virtual_memory()) # physical memory usage
Please inform whether I am headed in the right direction or not.
The above code generated following results.
('memory use:', 0.1001129150390625)
0.0
svmem(total=6123679744, available=4229349376, percent=30.9, used=1334358016, free=3152703488, active=1790803968, inactive=956125184, buffers=82894848, cached=1553723392, shared=289931264, slab=132927488)
Edit: Goal: find out energy consumed by CPU while running this script
I'm not sure whether you're trying to find the peak CPU usage during execution of that #kalman code, or the average, or the total, or something else?
But you can't get any of those by calling cpu_percent() after it's done. You're just measuring the CPU usage of calling cpu_percent().
Some of these, you can get by calling cpu_times(). This will tell you how many seconds were spent doing various things, including total "user CPU time" and "system CPU time". The docs indicate exactly which values you get on each platform, and what they mean.
In fact, if this is all you want (including only needing those two values), you don't even need psutils, you can just use os.times() in the standard library.
Or you can just use the time command from outside your program, by running time python myscript.py instead of python myscript.py.
For others, you need to call cpu_percent while your code is running. One simple solution is to run a background process with multiprocessing or concurrent.futures that just calls cpu_percent on your main process. To get anything useful, the child process may need to call it periodically, aggregating the results (e.g., to find the maximum), until it's told to stop, at which point it can return the aggregate.
Since this is not quite trivial to write, and definitely not easy to explain without knowing how much familiarity you have with multiprocessing, and there's a good chance cpu_times() is actually what you want here, I won't go into details unless asked.
I have spent a long time profiling some of my Django code (trying to find the root cause of a performance issue), which converts a queryset to a list of HTML tags.
The queryset in question has around 8000 records, but profiling has shown that the database/SQL is not the issue, as the actual query executes in around 1 millisecond.
After many hours of trying to find the problem I accidentally stumbled upon the fact that running the specific code via Apache/WSGI seems to play a big role in the performance problem.
To narrow the issue down to a single line of Python code, I have wrapped it in a "clock" to measure it's performance, like so:
import time
start_time = time.clock()
records = list(query_set) # <--- THIS LINE IS THE CULPRIT
total_time = time.clock() - start_time
with open('some_path/debug.txt', 'a') as f:
f.write('%f\n'%total_time)
Ignoring the fact that I am converting an 8000+ record queryset to a Python list (I have reasons for doing so), the part I want to focus on is the timing differences:
Apache/WSGI: 1.184240000000000403446165365 (seconds)
From a shell: 0.6034849999999999381472548521 (seconds)
The code in question lives in a function that I can conveniently call from a Django management shell and run manually, which allows me to get the second timing metric.
So can someone please tell me what the heck is causing that doubling in execution time of that single line. Obviously a fair bit of built-in Django code is invoked by that line, but how can the way the code is executed make such a massive difference?
How can I get the same performance in Apache/WSGI as I get in the shell?
Note that I am running the Django application in daemon mode (not embedded mode) and the rest of my application is not suffering from performance issues. It's just that I never thought the difference between shell and Apache/WSGI would be a cause of performance difference, for this particular code, let alone any.
Update 1
I forgot to mention that I tried running the same code in nginx/uWSGI and the same problem occurred, so it doesn't seem to be Apache that's at fault. Possibly the way WSGI works itself?
Update 2
So the plot thickens...
That list(query_set) line of code is not multi-threaded by me, but if I alter my implementation slightly and do the actual iteration of the querset in multi-threaded code:
def threaded_func(query_set_slice): # Gets invoked by multiple threads created by me
start_time = time.clock()
for record in query_set_slice:
... do stuff ...
total_time = time.clock() - start_time
with open('some_path/debug.txt', 'a') as f:
f.write('%f\n'%total_time)
... there is still an execution time difference (between shell and Apache/WSGI), but this time the ratio is reversed.
I.e. The numbers are:
Apache/WSGI: 0.824394 (seconds)
Shell: 1.890437 (seconds)
Funnily enough, as the ratio of the threaded function is the reverse of the previous ratio, the execution time of the function that invokes the threaded function is the same for the two types of invocation (Apache/WSGI and Shell).
Update 3
After doing some research on time.clock() it seems that it can't be trusted all the time, so I tried time.time() and that produced roughly a 1:1 ratio.
I also tried LineProfiler and that gave a 1:1 ratio, so my guess is that time.clock() is not taking into account context switches and things like that.
I have a "healthchecker" program, that calls a "prober" every 10 seconds to check if a service is running. If the prober exits with return code 0, the healthchecker considers the tested service fine. Otherwise, it considers it's not working.
I can't change the healthchecker (I can't make it check with a bigger interval, or using a better communication protocol than spawning a process and checking its exit code).
That said, I don't want to really probe the service every 10 seconds because it's overkill. I just wanna probe it every minute.
My solution to that is to make the prober keep a "cache" of the last answer valid for 1 minute, and then just really probe when this cache expires.
That seems fine, but I'm having trouble thinking on a decent approach to do that, considering the program must exit (to return an exit code). My best bet so far would be to transform my prober in a daemon (that will keep the cache in memory) and create a client to just query it and exit with its response, but it seems too much work (and dealing with threads, and so on).
Another approach would be to use SQLite/memcached/redis.
Any other ideas?
Since no one has really proposed anything I'll drop my idea here. If you need an example let me know and I'll include one.
The easiest thing to do would be to serialize a dictionary that contains the system health and last time.time() it was checked. At the beginning of your program unpickle the dictionary, check the time, if it's less then your 60 second time interval, quit. Otherwise check the health like normal and cache it (with the time).
I want to set multiple alarm in python. What is the recommended way of setting it up? My use-case is that I've threshold time for N variables. When the current time reaches the threshold value, I want all the variables with that threshold values.
Here's my apprach:-
threshold_time_list = [get list all times from the DB]
current_time = datetime.now()
[i for i in threshold_time_list if i==current_time]
But this is very inefficient way of doing it since I might have 250+ variables like a/b/c.
And also I have to check this condition every second(cronjob). Is there a better way of doing it?
I found on SO, this can be done using threading and making the thread to go to sleep for threshod - current_time. But running 250 threads parallely is again an issue, since I've been facing an issue in my production where Django gets hanged (dont know why) and I need to restart the server to make it work again. We're asssuming that Django might get out of threads for processing, hence making 250 more threads is cumbersome.
Also if someone knows , why does Django gets hang in b/w the running live product it will be beneficial.
Can this alarm question be done in celery?
Use the sched module. This lets you create any number of scheduled tasks, then run them when they kick off, one at a time.