I am new to Locust Load testing framework and in process of migrating my existing Azure cloud based Performance testing C# scripts to Locust's Python based scripts. Our team almost completed migration of scripts. But during our load tests, we are getting errors as below, which fails to create new requests from the machine due to high CPU utilization or because of so many exception on Locust. We are running with Locust web based mode - details are indicated below. These scritps are working fine on smaller loads of 50 to 100 users
"Error 1 -('Connection aborted.', RemoteDisconnected('Remote end closed connection without response',))"
"Error 2 : Connection pool is full, discarding connection"
"** **Error 3 :urllib3.exceptions.NewConnectionError: : Failed to establish a new connection: [Errno 110] Connection timed out****"
Yes, we are using UrlLibs on the utility classes . But first 2 error does seems to be of Locust.
Our Load testing configurations are : "3500 users at a hatch rate of 5 users per second". Running natively(no docker container) on a 8 Core , 16 Gb Linux Ubuntu Virtual machine on Azure. ulimit set as 50,000 on Linux machine.
Please help us with your thoughts
Sample test is as below
import os
import sys
sys.path.append(os.environ.get('WORKDIR', os.getcwd()))
from locust import HttpLocust, TaskSet, task
from locust.wait_time import between
class ContactUsBehavior(TaskSet):
wait_time = AppUtil.get_wait_time_function(2)
#task(1)
def post_load_test_contact(self):
data = { "ContactName" : "Mane"
, "Email" : "someone#someone.com"
, "EmailVerifaction" : "someone#someone.com"
, "TelephoneContact" : ""
, "PhoneNumber" : ""
, "ContactReason" : "Other"
, "OtherComment" : "TEST Comments 2019-12-30"
, "Agree" : "true"
}
self.client.post("app/contactform", self.client, 'Contact us submission', post_data = data)
class UnauthenticatedUser(HttpLocust):
task_set = ContactUsBehavior
# host is override-able
host = 'https://app.devurl.com/'
Locust’s default HTTP client uses python-requests which internally use urllib3 .
If you are working on a large scale tests, you should consider another HTTP client. The connection pool of urllib 3 (PoolManager ) will reuse connections and limit how many connections are allowed per host at any given time to avoid accumulating too many unused sockets.
So you have option to tweak the pool : https://urllib3.readthedocs.io/en/latest/advanced-usage.html#customizing-pool-behavior
Or you can try any other high performance HTTP client . Eg: gevenhttp
Locust also provides a built-int client which is faster than the default python-requests:
https://docs.locust.io/en/stable/increase-performance.html
You should consider to run Locust in cluster mode in different nodes if the client still couldn't handle the big load.
Related
I want an Expert Advisor to open an Trade triggerd by a Telegram-Message.
I succesfully set up an Hello-World application using MQ4 as Server and Python/Telegram-bot as Client.
When the Telegram-Bot recieves a Message, he will send a request to MQ4 and gets a simple response without executing a trade.
Running Code below.
# Hello World client in Python
# Connects REQ socket to tcp://localhost:5555
import zmq
context = zmq.Context()
# Socket to talk to server
print("Connecting to trading server…")
socket = context.socket(zmq.REQ)
socket.connect("tcp://localhost:5555")
print("Connecting to trading server succeed")
#################################################################################
# Use your own values from my.telegram.org
api_id = ######
api_hash = '#####'
bot_token = '#####'
#################################################################################
from telethon import TelegramClient, events
client = TelegramClient('anon', api_id, api_hash)
#client.on(events.NewMessage)
async def my_event_handler(event):
if "Ascending" in event.raw_text:
if "AUDUSD" in event.raw_text:
await event.reply("AUDUSD sell")
# Do 1 request, waiting for a response
for request in range(1):
print("Telegram: AUDUSD sell execution requested %s …" % request)
socket.send(b"AUDUSD Sell execute")
#Send 2 variables (Ordertype // Symbol)
# Get the reply. -> Not neccesary for final application
# Apülication just needs to send 2 Varianles to MQ4 and trigger the open_order()
message = socket.recv()
print("Received reply %s [ %s ]" % (request, message))
client.start()
client.run_until_disconnected()
// Hello World server in MQ4
#include <Zmq/Zmq.mqh>
//+------------------------------------------------------------------+
void OnTick()
{
Context context("helloworld");
Socket socket(context,ZMQ_REP);
socket.bind("tcp://*:5555");
while(!IsStopped())
{
ZmqMsg request;
// Wait for next request from client
// MetaTrader note: this will block the script thread
// and if you try to terminate this script, MetaTrader
// will hang (and crash if you force closing it)
socket.recv(request);
Print("Receive: AUDUSD Sell execute");
Sleep(1000);
ZmqMsg reply("Trade was executed");
// Send reply back to client
socket.send(reply);
Print("Feedback: Trade was executed");
}
}
//+------------------------------------------------------------------+
Now I want to send 2 variables from Python to MQ4.
1. Ordertype: buy/sell
2. Symbol: EURUSD, AUDUSD,...
Send "Sell" if message contains "Ascending" -
Send "Buy" if message contains "Descending"
Send "AUDUSD" if message contains "AUDUSD",...
To do so I found a Library from Darwinex and want to combine it (interpretation of message, sending value as an array) with my already functioning telegram-bot.
For testing I wanted to try the example-code from Darwinex by itself.
I found the Code v2.0.1:
Python:
https://github.com/darwinex/DarwinexLabs/blob/master/tools/dwx_zeromq_connector/v2.0.1/Python/DWX_ZeroMQ_Connector_v2_0_1_RC8.py
MQ4: (Note: This Library code may replace the whole MQ4 code above in final app.)
https://github.com/darwinex/DarwinexLabs/blob/master/tools/dwx_zeromq_connector/v2.0.1/MQL4/DWX_ZeroMQ_Server_v2.0.1_RC8.mq4
When I copy the Code without changing I get an error in Python:
NameError: name '_zmq' is not defined
After running: _zmq._DWX_ZeroMQ_Connector() - in the Kernel of Spyder.
What can I do to fix that error?
In the final state I want to run the Python-Code and the Expert Advisor on the same Windows Server 2012 R2.
Is it enough if I run the .py file in the powershell from the server or should I host the file with the Webside?
I expect to get the whole system/examplecode running on my VPS or Webside-Host-Server and get an testing environment for further coding action, but currenty I cant get the Library Code in Python to run properly.
Also the MT4 ceeps crashing with the current code - but should be fixed if I combine my application with the Library-Codeexample.
(running everything on my local PC with WIN 10).
Q : I think it is a connection-problem between MT4 and Python.
Without a fully reproducible MCVE-code this is undecideable.
Having used a ZeroMQ-based bidirectional signalling/messaging between a QuantFX in python and trading ecosystem MetaTrader 4 Terminal implemented in MQL4, there is a positive experience of using this architecture.
Details decide.
The Best Next Step :
Start with a plain PUSH/PULL archetype python-PUSH-es, MQL4-script-PULL-s, preferably using tcp:// transport-class ( win platforms need not be ready to use an even simpler, protocol-less, ipc:// transport-class.
Once you have posack'd this trivial step, move forwards.
Q : How do I need to setup my Server to get a connection betwen those two - since it should be the same as on my local PC?
It is normal to use ZeroMQ on the same localhost during prototyping, so you may test and debug the integration. For details on ZeroMQ, feel free to read all details in other posts.
Q : Is it enough if I run the .py file in the powershell from the server or should I host the file with the Webside I already have and use that as "Python-Server"?
Yes, in case the .py file was designed that way. No code, no advice. That simple.
Possible issues :
Versions - ZeroMQ, since 2.11.x till the recent 4.3.+, has made a lot of changes
Installation DLL-details matter.
MQL4 has similarly gone through many changes ( string ceased to be a string and become struct to name a biggest impacting one ), so start with simple scenarios and integrate the target architecture in steps / phases with due testing whether the completed phases work as expected.
to fix that problem you need this:
from DWX_ZeroMQ_Connector_v2_0_1_RC8 import DWX_ZeroMQ_Connector
_zmq = DWX_ZeroMQ_Connector()
(adjust your version of the connector as appropriate).
should fix that problem.
I try to connect to Pusher Websocket API using the following code :
https://github.com/nlsdfnbch/Pysher/
import pysher
# Add a logging handler so we can see the raw communication data
import logging
root = logging.getLogger()
root.setLevel(logging.INFO)
ch = logging.StreamHandler(sys.stdout)
root.addHandler(ch)
pusher = pysher.Pusher('de504dc5763aeef9ff52')
# We can't subscribe until we've connected, so we use a callback handler
# to subscribe when able
def connect_handler(data):
channel = pusher.subscribe('live_trades')
channel.bind('trade', callback)
pusher.connection.bind('pusher:connection_established', connect_handler)
pusher.connect()
while True:
# Do other things in the meantime here...
time.sleep(1)
instead of some valid response, i get this every few seconds :
Connection: Error - [WinError 10042] An unknown, invalid, or
unsupported option or level was specified in a getsockopt or
setsockopt call Connection: Connection closed Attempting to connect
again in 10 seconds.
what is the problem ?
I saw the same error using a different library that uses websockets. I can see from your description (and link) that Pysher uses websockets.
I found (yet another) websocket client for Python that reported an issue with websockets, specifically with Python 3.6.4: [https://github.com/websocket-client/websocket-client/issues/370]
It references the bug in Python tracker as well [https://bugs.python.org/issue32394]
Upgrading to Python 3.6.5 worked for me. Alternatively, they suggest that upgrading to Windows 10 1703+ should work too (just for completeness; I have not verified this).
I am trying to integrate snmptrapd and RabbitMQ for delivering traps notifications to an exterior system.
My system is composed of 3 components:
A Linux virtual machine with snmptrapd and RabbitMQ (Publisher);
A Linux virtual machine with RabbitMQ (Consumer);
A Linux bare metal with docker so I can have a lot of containers sending traps (using nping)
The snmptrapd part is simple:
authCommunity execute mycom
traphandle default /root/some_script
In my first attempts the some_script was written in Python, but the performance was not perfect (20 containers sending 1 trap per second during 10 seconds, I only received 160 messages in the consumer).
#!/usr/bin/env python
import pika
import sys
message = ""
for line in sys.stdin :
message += (line)
credentials = pika.PlainCredentials('test', 'test')
parameters = pika.ConnectionParameters('my_ip', 5672, '/', credentials)
connection = pika.BlockingConnection(parameters)
channel = connection.channel()
channel.queue_declare(queue='snmp')
channel.basic_publish(exchange='',
routing_key='snmp',
body=message)
connection.close()
I switched to Perl and now I can get 200 traps/messages.
My Perl script uses Net::AMQP::RabbitMQ
#!/usr/bin/perl
use Net::AMQP::RabbitMQ;
foreach my $line ( <STDIN> ) {
chomp( $line );
$message = "$message\n$line";
}
my $mq = Net::AMQP::RabbitMQ->new();
$mq->connect("my_ip", {
user => "test",
password => "test",
vhost => "/"
});
$mq->channel_open(1);
$mq->publish(1, "snmp", $message);
$mq->disconnect();
But I want better. I tried 200 containers sending 1 trap per second and it failed miserably, receiving only around 10% of messages in the consumer.
I think this has to do with the overhead of always have to open, publish and close the channel in RabbitMQ per trap received, because at the network level I receive all the messages (checked trough a tcpdump).
Is there a way to keep an always-open publish channel so I don't have to reopen/create a connection to the queue?
Asking if you can talk to RabbitMQ server without connecting to it first is like asking if you can talk to someone on the telephone without connecting to their phone first (by dialing and answering).
You really should reuse your connection if you're going to send multiple messages, but you do need a connection first!
Anyway, the problem isn't with the publisher. It's the consumer that's buggy if it's losing messages.
I am using a single node Cassandra and I intend to run some queries in order to check the response time. In some queries, after 10s of execution occurs to me the following error:
OperationTimedOut: errors = {}, last_host = 127.0.0.1
So I ran the following command:
sudo gedit /usr/bin/cqlsh.py
And changed cqlsh.py file:
# cqlsh should run correctly when run out of a Cassandra source tree,
# out of an unpacked Cassandra tarball, and after a proper package install.
cqlshlibdir = os.path.join(CASSANDRA_PATH, 'pylib')
if os.path.isdir(cqlshlibdir):
sys.path.insert(0, cqlshlibdir)
from cqlshlib import cql3handling, cqlhandling, pylexotron, sslhandling
from cqlshlib.displaying import (ANSI_RESET, BLUE, COLUMN_NAME_COLORS, CYAN,
RED, FormattedValue, colorme)
from cqlshlib.formatting import (DEFAULT_DATE_FORMAT, DEFAULT_NANOTIME_FORMAT,
DEFAULT_TIMESTAMP_FORMAT, DateTimeFormat,
format_by_type, format_value_utype,
formatter_for)
from cqlshlib.tracing import print_trace, print_trace_session
from cqlshlib.util import get_file_encoding_bomsize, trim_if_present
DEFAULT_HOST = '127.0.0.1'
DEFAULT_PORT = 9042
DEFAULT_CQLVER = '3.3.1'
DEFAULT_PROTOCOL_VERSION = 4
DEFAULT_CONNECT_TIMEOUT_SECONDS = 240
DEFAULT_FLOAT_PRECISION = 5
DEFAULT_MAX_TRACE_WAIT = 300
However, when I try to run the query again, cql return the same error after 10s:
OperationTimedOut: errors = {}, last_host = 127.0.0.1
What I have to do so that the query has no answer timeout?
The latest version of cassandra allows you to specify cqlsh timeout when you use it, instead of having to edit your cqlshrc file.
cqlsh --request-timeout <your-timeout>
Are you executing these queries in cqlsh?
If so, you are hitting the client request timeout (not the connect timeout, nor the server-side read request timeout).
You can change the default timeout by setting one in ~/.cassandra/cqlshrc:
[connection]
client_timeout = 20
# Can also be set to None to disable:
# client_timeout = None
See https://issues.apache.org/jira/browse/CASSANDRA-7516 for more detail.
I see from another comment you are already aware of paging. This will be the best approach because it does not require you to marshal the entire result set in memory at the data and app tiers.
You'll see a handful of responses telling you how to raise the various timeouts, but the real answer is that you almost never want to raise those timeouts, because if you have a real data set, you will kill your server (or drop requests/mutations) with lots of long-running queries. You are better off using paging and more short-running queries than huge, long running queries.
You have to change the read_request_timeout_in_ms parameter in the cassandra.yaml file. And then restart Cassandra.
Why are my dynamodb requests via boto:get_item so slow and too frequently very slow? The AWS console reports that my get latency has hit a high of 12.5ms. None of my requests are anywhere near that low.
Python 2.7.5
AWS region us-west-1
boto 2.31.1
dynamodb table size ~180k records
Code:
from boto.dynamodb2.fields import HashKey
from boto.dynamodb2.table import Table
from boto.dynamodb2.types import STRING
import boto.dynamodb2
import time
REGION = "us-west-1"
AWS_KEY = "xxxxx"
AWS_SECRET = "xxxxx"
start = time.time()
peeps = ("cefbdadf518f44da8a68e35b2321bb1f", "7e3a691df6134a4f83d381a5507cbb18")
connection = boto.dynamodb2.connect_to_region(REGION, aws_access_key_id=AWS_KEY, aws_secret_access_key=AWS_SECRET)
users = Table("users-test", schema=[HashKey("id", data_type=STRING)], connection=connection)
for peep in peeps:
user = users.get_item(consistent=True, id=peep)
print time.time() - start
Results:
(botot)➜ ~ python test2.py
0.056941986084
0.0681240558624
(botot)➜ ~ python test2.py
1.05709600449
1.06937909126
(botot)➜ ~ python test2.py
0.048614025116
0.0575139522552
(botot)➜ ~ python test2.py
0.0553398132324
0.064425945282
(botot)➜ ~ python test2.py
3.05251288414
3.06584000587
(botot)➜ ~ python test2.py
0.0579640865326
0.0699849128723
(botot)➜ ~ python test2.py
0.0530469417572
0.0628390312195
(botot)➜ ~ python test2.py
1.05059504509
1.05963993073
(botot)➜ ~ python test2.py
1.05139684677
1.0603158474
update 2014-07-11 08:03 PST
The actual use-case is looking up a user for each web request. As #gamaat said, the cost for DynamoDB is on the first lookup because thats when the HTTPS connection is made. So it seems if I can store the DynamoDB connection between requests and reuse it, things would go faster. So I used werkzeug.contrib.cache.FileSystemCache to store the connection but it never seems to actually store the connection for retrieval. Other values get stored fine, just not this connection object. Any ideas? And if this is not a good way to store the connection between requests, then what is?
update 2014-07-11 15:30 PST
Since I'm using supervisor and uwsgi to manage my Flask app, it seems that the problem is actually how can I share the connection object between requests for my Flask app.
The solution to the question that appears to be yielding better response times (before average response time was ~500ms, and after it is ~50ms) was to do two things:
1) put the Boto DynamoDB connection object in default_settings.py so that it gets loaded in once into app.config["DYNDB_CONN"] per application load; and
2) configure uwsgi to have a cheaper value of num_proccesses - 1, and cheaper-initial of num_proccesses - 1. This tells uwsgi to always to have num_processes - 1 uwsgi processes running at all times with the option of starting up one more process if load requires it.
I did this to minimize the number of uwsgi processes that would restart and therefore create a new Boto DynamoDB connection object (incurring HTTP connection setup costs).