I have created a flask based IOT application where the devices send data regularly via a REST API and that data is stored in a DB.
I want to develop a notification system which send notification to the Mobile APP whenever the thresholds for a particular device exceeds.
The threshold and the time window for each device is stored in DB
Example:
if the temperature of device x for the last 5 minutes is greater that 30 deg C then send a notification to the user.
What would be the best approach to solve this using Python ?
Currently I am using celery beat and running a worker every 1 sec which reads the device data and threshold configured by user from the database and based on the value sends the notification to the APP via PYFCM.
I don't feel this method would be scalable in the long run.
That is not particularly "python" or "flask" thing you're asking about.
That's architectural thing. And, after it, that's a design thing.
So, architecturally the typical IoT data are timeseries of measurement values tagged with some attributes for distinction by origins, groups, flavors, you-name-it.
As timeseries are being established (received from devices & stored in DB), you typically want to process them with statistical methods to build some functions ready for analysis.
That, in a nutshell, is the architectural disposition.
Now, having all that we can assume what the best design solution could be.
Ideally, that should be the tool that is suited for storing timeseries & equipped with statistical analysis tools as well, in perfect case - having notification adapters/transport too (or that could be a bunch of separate tools that coulkd easily bound together).
Fortunately, they have already exist: look up for Timeseries Databases.
My personal prefrence is InfluxDB, but there exist others, like Prometheus, this recently launched AWS service, etc. Google it all.
Virtually all of them are equipped with stat analysis tools of different kind (in InfluxDB, for instance, you can do it with query language right away, plus there's a powerful stream/batch processor that goes even beyond).
Most of them are equipped with notification instruments as well (e.g. aforementioned Influx stream processor have built-in notification actions on statistical events).
So if you're looking for long term/scalable solution (what's the expected scale, BTW?) - that's one of the best options you've got. Will take some effort/refactor though.
And of course you always can implement it with the tools you've got.
Your "polling workers" solution seems to be perfectly fine for now, next you go for statistical libs specific to your platforms (not familiar with Python, sorry, and your DB isn't known either - can't give a specific advice which to use) and you'll be fine.
Related
I have a simple python script that I would like to run thousands of it's instances on GCP (at the same time). This script is triggered by the $Universe scheduler, something like "python main.py --date '2022_01'".
What architecture and technology I have to use to achieve this.
PS: I cannot drop $Universe but I'm not against suggestions to use another technologies.
My solution:
I already have a $Universe server running all the time.
Create Pub/Sub topic
Create permanent Compute Engine that listen to Pub/Sub all the time
$Universe send thousand of events to Pub/Sub
Compute engine trigger the creation of a Python Docker Image on another Compute Engine
Scale the creation of the Docker images (I don't know how to do it)
Is it a good architecture?
How to scale this kind of process?
Thank you :)
It might be very difficult to discuss architecture and design questions, as they usually are heavy dependent on the context, scope, functional and non functional requirements, cost, available skills and knowledge and so on...
Personally I would prefer to stay with entirely server-less approach if possible.
For example, use a Cloud Scheduler (server less cron jobs), which sends messages to a Pub/Sub topic, on the other side of which there is a Cloud Function (or something else), which is triggered by the message.
Should it be a Cloud Function, or something else, what and how should it do - depends on you case.
As I understand, you will have a lot of simultaneous call on a custom python code trigger by an orchestrator ($Universe) and you want it on GCP platform.
Like #al-dann, I would go to serverless approach in order to reduce the cost.
As I also understand, pub sub seems to be not necessary, you will could easily trigger the function from any HTTP call and will avoid Pub Sub.
PubSub is necessary only to have some guarantee (at least once processing), but you can have the same behaviour if the $Universe validate the http request for every call (look at http response code & body and retry if not match the expected result).
If you want to have exactly once processing, you will need more tooling, you are close to event streaming (that could be a good use case as I also understand). In that case in a full GCP, I will go to pub / sub & Dataflow that can guarantee exactly once, or Kafka & Kafka Streams or Flink.
If at least once processing is fine for you, I will go http version that will be simple to maintain I think. You will have 3 serverless options for that case :
App engine standard: scale to 0, pay for the cpu usage, can be more affordable than below function if the request is constrain to short period (few hours per day since the same hardware will process many request)
Cloud Function: you will pay per request(+ cpu, memory, network, ...) and don't have to think anything else than code but the code executed is constrain on a proprietary solution.
Cloud run: my prefered one since it's the same pricing than cloud function but you gain the portability, the application is a simple docker image that you can move easily (to kubernetes, compute engine, ...) and change the execution engine depending on cost (if the load change between the study and real world).
Here is an interesting problem. I am building an application in AI, that is due to be fundamentally agnostic to computational architectures(hopefully to include mobile). We have built a python application with a machine learning and deep-learning data structure that connects to despaired systems, and combines data with the new systems usually running a windows app/sometimes mac.
We have been working with net-workers and pen testers to determine the best approach to maintaining multiple client-server connections with asynchronicity and a heavy load, and maintain a positive security posture.
Here are the Upside's:
1. Data from the producers is very fast.
2. Programmatically we have increased the ability to load-balance.
The downsides:
1. The controllers / are not very fast. Here we are using languages (Node.js, go-Lang, cherrypy, rabbitmq) to speed up the connections and multi/Hyper thread. (Raspberry pi). Our goal is to use minimal frameworks pizero, raspberry, Arduino, to install and use our tech in an "instance" (See Tony Stark in avengers, connect the chips to the rail and create a command center).
2.point machines are not always quick, AND networking can suck. Using some fabric (with some hopes and dreams) we are nearly to the point of being able to deploy a system in seconds. Currently, data is slow on the back-end, and DB's are not helping that case. Some of our clients would like to test, and we are not interested in making a massive distro. With the amount of data, we only hope to achieve a "lightweight" client application that will leverage (dissimilar) technologies.
as we connect to "permanent" despaired systems we will pass our computational loads, and use for cold storage.
-We are having issues with application persistence / Surviving reboot
-With the multi-client control, commands are getting dropped.(some get them 2 times +)
-We are running many scripts and programs that get, write, read, put. We have them spread between multiple languages with all different uses and requirements.
- After the application is loaded updates are not always making it to each unit.
Thank you to all whom reply
I have a simple webservice that I need to scale up substantially.
I'm trying to decide where to go amongst the various web frameworks, load balancers, app servers (e.g Mongrel2, Tornado, and nginx, mod_proxy).
I have an existing Python app (currently exposed via BaseHTTPServer) that accepts some JSON data (about 900KB per request), and returns some JSON data (about 1k). The processing is algorithmic and done in a mixture of Python and some C (via Cython).
This is heavily optimized already (down to 1.1 seconds per-job from >1hour). But I can optimise that no further. While I rewrite in something a bit more thread-friendly, I need to scale things out horizontally (ec2 maybe).
There is no session or state, but the startup time of the app is quite slow (even with pickling and cashing). It takes about 3 seconds to load all the source data. Once running it takes about 1.1 seconds per request. I
Maybe I could spin up a number of copies and then reverse proxy them? Maybe I could do some funky worker pool in one of those frameworks? But I'm still in the unknown unknowns here.
First, you should decouple your webservice layer from number crunching. Use external job queue (for example http://celeryproject.org/), to offload web frontend. Then you can scale each part interdependently.
You should look for IaaS-type cloud providers (EC2, Rackspace, Linode, Softlayer etc), where you should be able to add nodes automatically (preferred way would be to spin up some preconfigured image to minimize node setup time).
Python seems to have many different packages available to assist one in parallel processing on an SMP based system or across a cluster. I'm interested in building a client server system in which a server maintains a queue of jobs and clients (local or remote) connect and run jobs until the queue is empty. Of the packages listed above, which is recommended and why?
Edit: In particular, I have written a simulator which takes in a few inputs and processes things for awhile. I need to collect enough samples from the simulation to estimate a mean within a user specified confidence interval. To speed things up, I want to be able to run simulations on many different systems, each of which report back to the server at some interval with the samples that they have collected. The server then calculates the confidence interval and determines whether the client process needs to continue. After enough samples have been gathered, the server terminates all client simulations, reconfigures the simulation based on past results, and repeats the processes.
With this need for intercommunication between the client and server processes, I question whether batch-scheduling is a viable solution. Sorry I should have been more clear to begin with.
Have a go with ParallelPython. Seems easy to use, and should provide the jobs and queues interface that you want.
There are also now two different Python wrappers around the map/reduce framework Hadoop:
http://code.google.com/p/happy/
http://wiki.github.com/klbostee/dumbo
Map/Reduce is a nice development pattern with lots of recipes for solving common patterns of problems.
If you don't already have a cluster, Hadoop itself is nice because it has full job scheduling, automatic data distribution of data across the cluster (i.e. HDFS), etc.
Given that you tagged your question "scientific-computing", and mention a cluster, some kind of MPI wrapper seems the obvious choice, if the goal is to develop parallel applications as one might guess from the title. Then again, the text in your question suggests you want to develop a batch scheduler. So I don't really know which question you're asking.
The simplest way to do this would probably just to output the intermediate samples to separate files (or a database) as they finish, and have a process occasionally poll these output files to see if they're sufficient or if more jobs need to be submitted.
Closed. This question needs to be more focused. It is not currently accepting answers.
Want to improve this question? Update the question so it focuses on one problem only by editing this post.
Closed 5 years ago.
Improve this question
I develop a client-server style, database based system and I need to devise a way to stress / load test the system. Customers inevitably want to know such things as:
• How many clients can a server support?
• How many concurrent searches can a server support?
• How much data can we store in the database?
• Etc.
Key to all these questions is response time. We need to be able to measure how response time and performance degrades as new load is introduced so that we could for example, produce some kind of pretty graph we could throw at clients to give them an idea what kind of performance to expect with a given hardware configuration.
Right now we just put out fingers in the air and make educated guesses based on what we already know about the system from experience. As the product is put under more demanding conditions, this is proving to be inadequate for our needs going forward though.
I've been given the task of devising a method to get such answers in a meaningful way. I realise that this is not a question that anyone can answer definitively but I'm looking for suggestions about how people have gone about doing such work on their own systems.
One thing to note is that we have full access to our client API via the Python language (courtesy of SWIG) which is a lot easier to work with than C++ for this kind of work.
So there we go, I throw this to the floor: really interested to see what ideas you guys can come up with!
Test 1: Connect and Disconnect clients like mad, to see how well you handle the init and end of sessions, and just how much your server will survive under spikes, also while doing this measure how many clients fail to connect. That is very important
Test 2: Connect clients and keep them logged on for say a week, doing random actions (FuzzTest). Time the round-trip of each action. Also keep record of the order of actions, because this way your "clients" will find loopholes in your usecases (very important, and VERY hard to test rationally).
Test 3 & 4: Determine major use cases for your system, and write up scripts that do these tasks. Then run several clients doing same task(test 3), and also several clients doing different tasks(test 4).
Series:
Now the other dimension you need here is amount of clients.
A nice series would be:
5,10,50,100,500,1000,5000,10000,...
This way you can get data for each series of tests with different work loads.
Also congrats on SWIGing your clients api to Python! That is a great way to get things ready.
Note: IBM has a sample of fuzz testing on Java, which is unrealted to your case, but will help you design a good fuzztest for your system
If you are comfortable coding tests in Python, I've found funkload to be very capable. You don't say your server is http-based, so you may have to adapt their test facilities to your own client/server style.
Once you have a test in Python, funkload can run it on many threads, monitoring response times, and summarizing for you at the end of the test.
For performance you are looking at two things: latency (the responsiveness of the application) and throughput (how many ops per interval). For latency you need to have an acceptable benchmark. For throughput you need to have a minimum acceptable throughput.
These are you starting points. For telling a client how many xyz's you can do per interval then you are going to need to know the hardware and software configuration. Knowing the production hardware is important to getting accurate figures. If you do not know the hardware configuration then you need to devise a way to map your figures from the test hardware to the eventual production hardware.
Without knowledge of hardware then you can really only observe trends in performance over time rather than absolutes.
Knowing the software configuration is equally important. Do you have a clustered server configuration, is it load balanced, is there anything else running on the server? Can you scale your software or do you have to scale the hardware to meet demand.
To know how many clients you can support you need to understand what is a standard set of operations. A quick test is to remove the client and write a stub client and the spin up as many of these as you can. Have each one connect to the server. You will eventually reach the server connection resource limit. Without connection pooling or better hardware you can't get higher than this. Often you will hit a architectural issue before here but in either case you have an upper bounds.
Take this information and design a script that your client can enact. You need to map how long your script takes to perform the action with respect to how long it will take the expected user to do it. Start increasing your numbers as mentioned above to you hit the point where the increase in clients causes a greater decrease in performance.
There are many ways to stress test but the key is understanding expected load. Ask your client about their expectations. What is the expected demand per interval? From there you can work out upper loads.
You can do a soak test with many clients operating continously for many hours or days. You can try to connect as many clients as you can as fast you can to see how well your server handles high demand (also a DOS attack).
Concurrent searches should be done through your standard behaviour searches acting on behalf of the client or, write a script to establish a semaphore that waits on many threads, then you can release them all at once. This is fun and punishes your database. When performing searches you need to take into account any caching layers that may exist. You need to test both caching and without caching (in scenarios where everyone makes unique search requests).
Database storage is based on physical space; you can determine row size from the field lengths and expected data population. Extrapolate this out statistically or create a data generation script (useful for your load testing scenarios and should be an asset to your organisation) and then map the generated data to business objects. Your clients will care about how many "business objects" they can store while you will care about how much raw data can be stored.
Other things to consider: What is the expected availability? What about how long it takes to bring a server online. 99.9% availability is not good if it takes two days to bring back online the one time it does go down. On the flip side a lower availablility is more acceptable if it takes 5 seconds to reboot and you have a fall over.
If you have the budget, LoadRunner would be perfect for this.
On a related note: Twitter recently OpenSourced their load-testing framework. Could be worth a go :)