I'm currently trying to model a service counter with SimPy but I'm running into difficulties with using yield to hold the resources. Under the Counter.arrive() function, if the line "yield req" exists, the entire function skips execution (at least I think that's what happens since I don't get any of the print output). However, if I comment out that line, the code executes like nothing happens. This is a problem because without the yield the code is not blocked until the request is approved and the entire simulation fails because everyone gets to use the resource.
Code snippet as follows:
import simpy
class Counter:
def __init__(self, env, name, staff):
self.env = env
self.staff = simpy.Resource(env, staff)
self.name = name
self.dreq = []
def arrive(self, name):
...
req = self.staff.request()
yield req
output = "Req: %s\n" % req
self.dreq.append(req)
...
print(output)
...
def customer(env, counter, name):
print("Customer %s arrived at %s" %(name,env.now))
counter.arrive(name)
yield env.timeout(5)
print("Customer %s left at %s" %(name,env.now))
...
env = simpy.Environment()
counter = Counter(env, "A", 1)
def setup(env, counter, MAX_CUST):
for i in range(MAX_CUST):
env.process(customer(env,counter, 1))
yield env.timeout(1)
env.process(setup(env,counter,5))
env.run(until=100)
Edit: I understand that using yield should pause the function until the request gets approved but the very first request does not go through as well which does not make sense as there is 1 unit of the resource available at the start.
Docs for convenience: https://simpy.readthedocs.io/en/3.0.6/topical_guides/resources.html
Requests (and timeouts and everything which you need to yield) gets processed by simpy, so it needs to arrive at simpy to get processed. You tell simpy to process customer with env.process:
env.process(customer(env,counter, 1))
In customer you call counter.arrive(name). Because arrive is a generator (because of yield) it does nothing until something calls next on it. Simpy needs to know of it to process it properly. You should be able to do this by:
env.process(counter.arrive(name))
which should fix your problem.
Note that also in this code you never release the ressource, so only one customer can actually arrive.
Related
I'm quite new with ARI scripting for Asterisk, and I've been trying to make some script to handle a 1 to 1 communication with ari-py in python. I've been following the example that provided in the asterisk wiki and so far so good. But when I try to create a call, the recipient always keep ringing, even if I have answered it. Is there something wrong with how I handle the call? Here's my script
def stasis_start_cb(self, channel, ev):
"""Handler for StasisStart event"""
chan = channel.get('channel')
chan.answer()
print "Channel %s has entered the application" % chan.json.get('name')
outgoing = client.channels.originate(endpoint="SIP/1002", extension='1002', callerId='Tes', app='channel-dump', appArgs='dialed')
I tried using OOP to simplify the function usage, are there anything wrong with that script? And here's another script trying to make a call by using a bridge:
def outgoing_call(self,channel):
try:
outgoing = client.channels.originate(endpoint="SIP/1002", app='channel-dump', appArgs='dialed')
except requests.HTTPError:
channel.hangup()
return
def outgoing_start(self, bri, channel):
channel.answer()
self.addChan(channel, bridge)
def stasis_start(self, channel, ev):
chan = channel.get('channel')
name = chan.json.get('name')
"""ars = ev.get('args')
if not ars:
print "Error: {} didn't provide any arguments!".format(name)
return
if ars and ars[0] != 'inbound':
return
if len(ars) != 2:
print "Error: {} didn't tell us who to dial".format(name)
chan.hangup()"""
print "Channel {} entered the application".format(name)
chan.ring()
self.outgoing_call(chan)
self.outgoing_start(bridge, chan)
Both the client is able to be added in the bridge, and I can make a call, but the problem still persist, the recipient keep saying they are ringing despite I have answered the call
Turns out, the problem is in here
def outgoing_call(self,channel):
try:
outgoing = client.channels.originate(endpoint="SIP/1002", app='channel-dump', appArgs='dialed')
except requests.HTTPError:
channel.hangup()
return
As the dialed number answer the call, they uses the same script, so they ended up calling themselves again. A simple if condition to make the dialed number not call to itself again is all that is needed
I'm trying to restrict the API calls in my code. I already found a nice python library ratelimiter==1.0.2.post0
https://pypi.python.org/pypi/ratelimiter
However, this library can only limit the rate in local scope. i.e) in function and loops
# Decorator
#RateLimiter(max_calls=10, period=1)
def do_something():
pass
# Context Manager
rate_limiter = RateLimiter(max_calls=10, period=1)
for i in range(100):
with rate_limiter:
do_something()
Because I have several functions, which make API calls, in different places, I want to limit the API calls in global scope.
For example, suppose I want to limit the APIs call to one time per second. And, suppose I have functions x and y in which two API calls are made.
#rate(...)
def x():
...
#rate(...)
def y():
...
By decorating the functions with the limiter, I'm able to limit the rate against the two functions.
However, if I execute the above two functions sequentially, it looses track of the number of API calls in global scope because they are unaware of each other. So, y will be called right after the execution of x without waiting another second. And, this will violate the one time per second restriction.
Is there any way or library that I can use to limit the rate globally in python?
I had the same problem, I had a bunch of different functions that calls the same API and I wanted to make rate limiting work globally. What I ended up doing was to create an empty function with rate limiting enabled.
PS: I use a different rate limiting library found here: https://pypi.org/project/ratelimit/
from ratelimit import limits, sleep_and_retry
# 30 calls per minute
CALLS = 30
RATE_LIMIT = 60
#sleep_and_retry
#limits(calls=CALLS, period=RATE_LIMIT)
def check_limit():
''' Empty function just to check for calls to API '''
return
Then I just call that function at the beginning of every function that calls the API:
def get_something_from_api(http_session, url):
check_limit()
response = http_session.get(url)
return response
If the limit is reached, the program will sleep until the (in my case) 60 seconds have passed, and then resume normally.
After all, I implemented my own Throttler class. By proxying every API request to the request method, we can keep track of all API requests. Taking advantage of passing function as the request method parameter, it also caches the result in order to reduce API calls.
class TooManyRequestsError(Exception):
def __str__(self):
return "More than 30 requests have been made in the last five seconds."
class Throttler(object):
cache = {}
def __init__(self, max_rate, window, throttle_stop=False, cache_age=1800):
# Dict of max number of requests of the API rate limit for each source
self.max_rate = max_rate
# Dict of duration of the API rate limit for each source
self.window = window
# Whether to throw an error (when True) if the limit is reached, or wait until another request
self.throttle_stop = throttle_stop
# The time, in seconds, for which to cache a response
self.cache_age = cache_age
# Initialization
self.next_reset_at = dict()
self.num_requests = dict()
now = datetime.datetime.now()
for source in self.max_rate:
self.next_reset_at[source] = now + datetime.timedelta(seconds=self.window.get(source))
self.num_requests[source] = 0
def request(self, source, method, do_cache=False):
now = datetime.datetime.now()
# if cache exists, no need to make api call
key = source + method.func_name
if do_cache and key in self.cache:
timestamp, data = self.cache.get(key)
logging.info('{} exists in cached # {}'.format(key, timestamp))
if (now - timestamp).seconds < self.cache_age:
logging.info('retrieved cache for {}'.format(key))
return data
# <--- MAKE API CALLS ---> #
# reset the count if the period passed
if now > self.next_reset_at.get(source):
self.num_requests[source] = 0
self.next_reset_at[source] = now + datetime.timedelta(seconds=self.window.get(source))
# throttle request
def halt(wait_time):
if self.throttle_stop:
raise TooManyRequestsError()
else:
# Wait the required time, plus a bit of extra padding time.
time.sleep(wait_time + 0.1)
# if exceed max rate, need to wait
if self.num_requests.get(source) >= self.max_rate.get(source):
logging.info('back off: {} until {}'.format(source, self.next_reset_at.get(source)))
halt((self.next_reset_at.get(source) - now).seconds)
self.num_requests[source] += 1
response = method() # potential exception raise
# cache the response
if do_cache:
self.cache[key] = (now, response)
logging.info('cached instance for {}, {}'.format(source, method))
return response
Many API providers constrain developers from making too many API calls.
Python ratelimit packages introduces a function decorator preventing a function from being called more often than that allowed by the API provider.
from ratelimit import limits
import requests
TIME_PERIOD = 900 # time period in seconds
#limits(calls=15, period=TIME_PERIOD)
def call_api(url):
response = requests.get(url)
if response.status_code != 200:
raise Exception('API response: {}'.format(response.status_code))
return response
Note: This function will not be able to make more then 15 API call within a 15 minute time period.
Adding to Sunil answer, you need to add #sleep_and_retry decorator, otherwise your code will break when reach the rate limit:
#sleep_and_retry
#limits(calls=0.05, period=1)
def api_call(url, api_key):
r = requests.get(
url,
headers={'X-Riot-Token': api_key}
)
if r.status_code != 200:
raise Exception('API Response: {}'.format(r.status_code))
return r
There are lots of fancy libraries that will provide nice decorators, and special safety features, but the below should work with django.core.cache or any other cache with a get and set method:
def hit_rate_limit(key, max_hits, max_hits_interval):
'''Implement a basic rate throttler. Prevent more than max_hits occurring
within max_hits_interval time period (seconds).'''
# Use the django cache, but can be any object with get/set
from django.core.cache import cache
hit_count = cache.get(key) or 0
logging.info("Rate Limit: %s --> %s", key, hit_count)
if hit_count > max_hits:
return True
cache.set(key, hit_count + 1, max_hits_interval)
return False
Using the Python standard library:
import threading
from time import time, sleep
b = threading.Barrier(2)
def belay(s=1):
"""Block the main thread for `s` seconds."""
while True:
b.wait()
sleep(s)
def request_something():
b.wait()
print(f'something at {time()}')
def request_other():
b.wait()
print(f'or other at {time()}')
if __name__ == '__main__':
thread = threading.Thread(target=belay)
thread.daemon = True
thread.start()
# request a lot of things
i = 0
while (i := i+1) < 5:
request_something()
request_other()
There's about s seconds between each timestamp printed. Because the main thread waits rather than sleeps, time it spends responding to requests is unrelated to the (minimum) time between requests.
So I hope this isn't a duplicate, however I either haven't been able to find the adequate solution or I just am not 100% on what I'm looking for. I've written a program to thread lots of requests. I create a thread to
Fetch responses from a number of api's such as this: share.yandex.ru/gpp.xml?url=MY_URL as well as scraping blogs
Parse the responses of all requests from the example above/ json/ using python-goose to extract articles
Return the parsed results back to the primary thread and insert into a database.
It's all been going well until it needs to pull back larger amounts of data which i haven't tested before. The primary reason for this is that it takes me over my shared memory limit on a shared Linux server (512mb) initiating a kill. This should be enough as it's only a few thousand requests, although i could be wrong. I'm clearing all large data variables/ objects within the main thread but that doesn't seem to help either.
I ran a memory_profile on the primary function which creates the threads with a thread class which looks like this:
class URLThread(Thread):
def __init__(self,request):
super(URLThread, self).__init__()
self.url = request['request']
self.post_id = request['post_id']
self.domain_id = request['domain_id']
self.post_data = request['post_params']
self.type = request['type']
self.code = ""
self.result = ""
self.final_results = ""
self.error = ""
self.encoding = ""
def run(self):
try:
self.request = get_page(self.url,self.type)
self.code = self.request['code']
self.result = self.request['result']
self.final_results = response_handler(dict(result=self.result,type=self.type,orig_url=self.url ))
self.encoding = chardet.detect(self.result)
self.error = self.request['error']
except Exception as e:
exc_type, exc_obj, exc_tb = sys.exc_info()
fname = os.path.split(exc_tb.tb_frame.f_code.co_filename)[1]
errors.append((exc_type, fname, exc_tb.tb_lineno,e,'NOW()'))
pass
#profile
def multi_get(uris,timeout=2.0):
def alive_count(lst):
alive = map(lambda x : 1 if x.isAlive() else 0, lst)
return reduce(lambda a,b : a + b, alive)
threads = [ URLThread(uri) for uri in uris ]
for thread in threads:
thread.start()
while alive_count(threads) > 0 and timeout > 0.0:
timeout = timeout - UPDATE_INTERVAL
sleep(UPDATE_INTERVAL)
return [ {"request":x.url,
"code":str(x.code),
"result":x.result,
"post_id":str(x.post_id),
"domain_id":str(x.domain_id),
"final_results":x.final_results,
"error":str(x.error),
"encoding":str(x.encoding),
"type":x.type}
for x in threads ]
And the results look like this on the first batch of requests i pump through it (FYI it's a link as the output text isn't readable in here, i can't paste a html table or embed an image until i get 2 more points ):
http://tinypic.com/r/28c147d/8
And it doesn't seem to drop any of the memory in subsequent passes (I'm batching 100 requests/ threads through at 1 time). By this i mean once a batch of threads is complete they seem to stay in memory ad every time it runs another, memory is added as below:
http://tinypic.com/r/nzkeoz/8
Am I doing something really stupid here?
Python will generally free the memory taken up by an object when there are no references to that object left. Your multi_get function returns a list that contains references to every thread that you have created. So it's unlikely that Python would free that memory. But we would need to see what the code that is calling multi_get is doing in order to be sure.
To start freeing the memory you will need to stop returning references to the threads from this function. Or if you want to continue to do that, at least delete them somewhere del x.
I'm using ZODB as a persistent storage for objects that are going to be modified through a webservice.
Below is an example to which I reduced the issue.
The increment-function is what is called from multiple threads.
My problem is, that when increment is called simultaneously from two threads, for different keys, I'm getting the conflict-error.
I imagine it should be possible to resolve this, at least as long different keys are modified, in a proper way?
If so, I didn't manage to find an example on how to... (the zodb-documentation seems to be somewhat spread across different sites :/ )
Glad about any ideas...
import time
import transaction
from ZODB.FileStorage import FileStorage
from ZODB.DB import DB
from ZODB.POSException import ConflictError
def test_db():
store = FileStorage('zodb_storage.fs')
return DB(store)
db_test = test_db()
# app here is a flask-app
#app.route('/increment/<string:key>')
def increment(key):
'''increment the value of a certain key'''
# open connection
conn = db_test.open()
# get the current value:
root = conn.root()
val = root.get(key,0)
# calculate new value
# in the real application this might take some seconds
time.sleep(0.1)
root[key] = val + 1
try:
transaction.commit()
return '%s = %g' % (key, val)
except ConflictError:
transaction.abort()
return 'ConflictError :-('
You have two options here: implement conflict resolution, or retry the commit with fresh data.
Conflict resolution only applies to custom types you store in the ZODB, and can only be applied if you know how to merge your change into the newly-changed state.
The ZODB looks for a _p_resolveConflict() method on custom types and calls that method with the old state, the saved state you are in conflict with, and the new state you tried to commit; you are supposed to return the merged state. For a simple counter, like in your example, that'd be a as simple as updating the saved state with the change between the old and new states:
class Counter(Persistent):
def __init__(self, start=0):
self._count = start
def increment(self):
self._count += 1
return self._count
def _p_resolveConflict(self, old, saved, new):
# default __getstate__ returns a dictionary of instance attributes
saved['_count'] += new['_count'] - old['_count']
return saved
The other option is to retry the commit; you want to limit the number of retries, and you probably want to encapsulate this in a decorator on your method, but the basic principle is that you loop up to a limit, make your calculations based on ZODB data (which, after a conflict error, will auto-read fresh data where needed), then attempt to commit. If the commit is successful you are done:
max_retries = 10
retry = 0
conn = db_test.open()
root = conn.root()
while retry < max_retries:
val = root.get(key,0)
time.sleep(0.1)
root[key] = val + 1
try:
transaction.commit()
return '%s = %g' % (key, val)
except ConflictError:
retry += 1
raise CustomExceptionIndicatingTooManyRetries
Just found the Queue module which is helping me adapt the pyftpdlib module. I'm running an very strict FTP server, and my goal is to restrict the filenames available to upload. This is to prevent people from uploading whatever they want (it's actually the backend of an upload client, not a complete FTP server).
I have this in the ftpserver Authorizer:
def fetch_worlds(queue, username):
while queue.empty():
worlds = models.World.objects.filter(is_synced=True, user__username=username)
print worlds
queue.put(worlds, timeout=1)
class FTPAuthorizer(ftpserver.DummyAuthorizer):
def __init__(self):
self.q = Queue.Queue()
self.t = None # Thread
self.world_item = None
def has_perm(self, username, perm, path=None):
print "Checking permission\n"
if perm not in ['r','w']:
return False
# Check world name
self.t = threading.Thread(target=fetch_worlds, args=(self.q, username))
self.t.daemon = True
self.t.start()
self.world_item = self.q.get()
print "WORLDITEM: %s" % self.world_item
if path is not None:
path = os.path.basename(path)
for world in self.world_item:
test = "{0}_{1}.zip".format(username, world.name)
if path == test:
print "Match on %s" % test
return True
return False
My issue is, after the server starts, the first time I STOR a file, it does an initial db call and gets all the worlds properly. But when I then add another world (for example, set is_synced=True on one, it still returns the old data from self.q.get(). has_perm() is called every time a file is uploaded, and it needs to return live data (to check if a file is allowed).
For example, brand new server:
STOR file.zip, self.q.get() returns <World1, World2>
Update the database via other methods etc
STOR file2.zip, inside fetch_worlds, print worlds returns <World1, World2, World3> but self.q.get() returns <World1, World2>
Just found the Queue module and it seemed like it would be helpful but I can't get the implementation right.
(also couldn't add tag pyftpdlib)
i think this is what could be happening here:
when has_perm is called, you create a Thread that will query a database (?) to add elements to the queue
after calling start the call to the database will take some time
meanwhile in your main thread you entered q.get which will block.
the db call finishes and the result is added to the queue
and is immediately removed from the queue again by the blocking q.get
the queue is now empty, your thread enters the while-loop again and executes the same query again and puts the result onto the queue.
the next call to q.get will return that instead of what it expects.
you see, you could have a race condition here, that already is aparent from the fact that you're adding something to a queue in a loop while you don't have a loop when pulling.
also you assume the element you get from the queue is the result to what you have put onto it before. that doesn't have to be true. if you call has_perm two times this will result in two calls to fetch_worlds with the possibility that the queue.empty() check for one of the calls fails. so only one result will be put onto the queue. now you have two threads waiting on q.get, but only one will get a result, while the oter one waits until one becomes ready...
has_perm looks like it should be a blocking call anyway.