I need to make a chat between two clients. But I don't know when a socket is closed or not, is there a way to check if it is?
Here is the part of the code that I need to fix:
def main():
"""Implements the conversation with server."""
# Open client socket, Transport layer: protocol TCP, Network layer: protocol IP
client_socket = socket.socket()
client_socket.connect((HOST_IP, PORT))
# start conversation with new client in parallel thread
name = input("enter your name ")
protocol.send_request(client_socket, name)
thread_for_responses = threading.Thread(target=get_responses,
args=(client_socket, ))
thread_for_responses.start()
while True:
# Get request from keyboard
client_request_str = input()
if client_request_str: # if client_request_str not empty string
# send request according to the protocol
protocol.send_request(client_socket, client_request_str)
# Get response from server
Instead of while True, I need to check if the socket is closed so it won't go into a loop where it will crush for using a closed socket.
Python programmers usually say it is easier to ask for forgiveness than for permission. By this they mean "handle the exception".
For example, you cannot divide by zero. Here are two ways you could deal with this fact:
def print_quotient(a, b):
if b == 0:
print("quotient is not a number")
else:
print("quotient is {}".format(a / b))
vs
def print_quotient(a, b):
try:
print("quotient is {}".format(a / b))
except ZeroDivisionError:
print("quotient is not a number")
These functions behave the same way so it doesn't make a lot of difference which approach you take. This is because b cannot change. This differs from your example where the socket can change. External factors influence its state and this changes the behavior of trying to use it (for example, to send bytes with it). In this case, exception handling is superior because it does not have to try to make sure nothing is going to go wrong, it just handles things when they go wrong. There is no case where the code thinks it has set everything up to work correctly and then it turns out it has missed something.
So, when you use socket operations, apply exception handling for any exceptions that might arise from those operations.
Related
I've tried looking around online through different python docs, forums, and other people's questions but I haven't found anyone with this same question.
What my scripts typically look like is I'll create a socket connection that tries connecting to ports 1-9999 and will only tell me when a port is open. When I run this on windows it takes 1 second to scan a port before moving on to the next one (60 ports/m. ~16.5m for 1000 ports). When I run the same scripts on linux, it'll cycle through all 9999 ports very quickly, while still returning the same desired results.
I was hoping to be able to build cross-compatible tools, but it appears linux
is just the better operating system when it comes to my networking needs? I have both at my disposal so I don't mind using one over the other. I'd just like to know if there's anything that could be done to make port scanning almost as equally fast on both operating systems, otherwise I won't spend as much time building on/for windows.
The difference in speed is the same regardless of which network I'm on.
My questions are:•Why is the performance so different on windows compared to linux when given the same functions?•Is there anything that can be done to make port scanning with sockets faster like it is on linux?
--edit--
here's the piece I use to check ports
def whole_scan(Host_):
service = ''
host = Host_
max_port = 9999
min_port = 1
def scan_host(host, port, r_code = 1):
try:
s = socket(AF_INET, SOCK_STREAM)
code = s.connect_ex((host, port))
if code == 0:
r_code = code
s.close()
except Exception, e:
pass
return r_code
hostip = gethostbyname(host)
for port in range(min_port, max_port):
try:
response = scan_host(host,port)
if response == 0:
try:
service = getservbyport(port)
except Exception, e:
service = 'n/a'
print(" |--port: %d\t%s" % (port,service.upper()))
except Exception, e:
pass
I've also verified my firewall is disabled and adding the value to my registry to disable the limit on connections had no change on performance. I'm on windows 10.
Windows limits the concurrent number of half-open connections and that may be at play here if you are opening that many connection requests at a time. For example, on Windows 7 try setting this key value to 0 (to disable it)
HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Tcpip\Parameters\EnableConnectionRateLimiting
I doubt that this is causing the performance problem, however, there is a bug in your scan_host() function.
This function attempts to return r_code, however, r_code is only set if connect_ex() returns 0. Should connect_ex() return a non-zero value, or an exception were to occur in the same block of code, r_code would not be set, and the return statement would raise a NameError exception. This exception will propagate to the calling code, which catches it then and ignores it and all other exceptions.
It's not a good idea to ignore exceptions; perhaps you might learn something relevant to the problem, perhaps not, but I suggest that you log the exceptions that are occurring.
Also, it would be useful if you added some debug print statements into your code. This will help you locate the part of your code where the majority of time is spent.
There is also the line:
hostip = gethostbyname(host)
which never seems to be executed - I can't tell because perhaps the indentation in your post is not quite right.
Another thing to consider is DNS. Possibly the DNS server used by Windows is slower, or there is some issue there. You could eliminate that by using the IP address instead of host name:
response = scan_host(gethostbyname(host), port)
I have a multi-threaded Python 3 application that on thread #1 accepts TCP socket communications. Thread #2 will check all current connections if they have anything to receive, then act accordingly.
So, currently I have a list called all_connections which is a list of accepted socket connection objects.
Using for connection in all_connections: I can loop through all the connection objects. I know I use conn.recv(256) to check if there is anything ready to recive on this socket. Will this block the loop though untill there is something to receive? I have set conn.setblocking(1) beforehand although Im unsure if this is the best way to get around it:
Here is some example code:
Thread 1
self.all_connections = [] # init a list to hold connection objs
while 1:
try:
conn, address = self.socket.accept()
conn.setblocking(1) # non blocking
except Exception as e:
continue
self.all_connections.append(conn) # Save the connection object
Thread 2
while True:
for connection in self.all_connections:
received = connection.recv(256)
return
So, I'm only interested in connections that have actually sent something, as I will be sending them something back most likely.
I know I can use select.select in order to check if there is anything to receive on the socket, but that wouldn't help me reference the specific connection.
Yes, read() will block; this is the default behaviour. Calling socket.setblocking(1) actually enables blocking, which is opposite of what you wanted. setblocking(False) will set non-blocking mode. I/O on non-blocking sockets requires that you use exception handling.
A better way, and you are already headed in the right direction, is to use select(). You do in fact know which socket sent data because select() returns a list of sockets that are available for reading, writing, or that have an error status. You pass to select() a list of the sockets that you are interested in and it returns those that are available for I/O. Here is the function signature:
select(...)
select(rlist, wlist, xlist[, timeout]) -> (rlist, wlist, xlist)
So the code in thread 2 would look something like this:
from select import select
while True:
rlist, wlist, xlist = select(self.all_connections, [], [])
for connection in rlist:
received = connection.recv(256)
The above code only checks for readable sockets in the list of all connections and reads data from those that are ready. The read will not block.
I'm a bit new at Python and I am working on a robotics project. The short form of my question is that I am trying to find the best way (for my situation) to run multiple python programs at once.
A little bit of context, my robot is a platform for a service robot that is capable of following markers and paths using image algorithms and also receive commands from a remote computer. I want to have separate programs for the image processing, the driving, and so on, and then manage all of them through a main program. I know I can't use anything basic like functions or classes, because each of these processes must be looping continuously, and I don't want to combine all the code to run in a single while loop, because it runs very slowly and it is significantly harder to manage.
So, in short, how do I make two separate, looping programs "talk"? Like I want the imaging program to send information about what it sees to the driving and steering program, etc.
I did some research and I found some information on multithreading and API's and stuff like that, though I can't really tell which one is actually the thing I'm looking for.
To clarify, I just need to be pointed in the right direction. This doesn't seem like a very high-level thing, and I know there are definitely tutorials out there, I'm just really confused as to where to start as I am teaching myself this as I go.
After some sniffing around, I found that using IPC was a good solution. The process I used wasn't too difficult, I just made some very simple server and client classes and had them communicate over the Localhost IP. There's undoubtedly a better way to do this, but for a beginner like myself, it was a simple way to make two programs talk without modifying code too much. For those who are trying to do a similar thing as I did, here's the classes I made for myself. Fair warning, they're not exactly pristine or even very complex, but they got the job done.
Here's the class I made for the server:
import socket
from random import random
from time import sleep
class ServerObject:
def __init__(self,host_address,port):
self._host_address = host_address
self._s = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
self._s.bind((self._host_address,port))
def handshake(self):
print "Server Started. Awaiting Connection"
while True:
_data, _addr = self._s.recvfrom(1024)
if str(self._s.recvfrom(1024)[0]) == 'marco':
break
print 'marco recieved. sending polo...'
while True:
self._s.sendto('polo',_addr)
if str(self._s.recvfrom(1024)[0]) == 'confirm':
break
sleep(.5)
print 'connection verified'
self._addr = _addr
return True
def send(self,data):
self._s.sendto(str(data),self._addr)
def recieve(self,mode = 0):
_data, _addr = self._s.recvfrom(1024)
if mode == 0:
return str(_data)
if mode == 1:
return int(_data)
if mode == 2:
return float(_data)
if mode == 3:
return tuple(_data)
def change_port(self,port):
self._s.bind((self._host_address,port))
def close(self):
self._s.close()
print '_socket closed_'
if __name__ == '__main__':
host = '127.0.0.1'
talk = ServerObject(host,6003)
talk.handshake()
And here's the class I made for the client:
import socket
from time import sleep
class ClientObject:
def __init__(self,host_address,server_port,port = 0):
self._server = (host_address,server_port)
self._s = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
self._s.bind((host_address,port))
def handshake(self):
print ' sending marco'
self._s.sendto('marco',self._server)
sleep(.1)
self._s.sendto('marco',self._server)
while True:
if str(self._s.recvfrom(1024)[0]) == 'polo':
break
#self._s.sendto('marco',self._server)
#self._s.sendto('marco',self._server)
print ' connection verified'
self._s.sendto('confirm',self._server)
self._s.setblocking(0)
return True
def recieve(self,mode = 0):
_data, _addr = self._s.recvfrom(1024)
if mode == 0:
return str(_data)
if mode == 1:
return int(_data)
if mode == 2:
return float(_data)
if mode == 3:
return tuple(_data)
def send(self,data):
self._s.sendto(str(data),self._server)
def close(self):
self._s.close()
print '_socket closed_'
if __name__ == '__main__':
host = '127.0.0.1'
port = 0
talk = ClientObject(host,24603,port)
talk.handshake()
#while True:
#print talk.recieve()
Use the ServerObject class on the program that will primarily send data and the ClientObject class on the program that will primarily recieve data. These can be flipped around in many situations, but I found it's best to do it this way to take advantage of UDP. The client class has an optional port variable that is set to 0 by default. This is because for UDP the client needs another port to establish itself on. 0 means it will pick an available port, but if you specify one, it's possible to re-establish a connection if the client goes offline without needing to restart both programs.
Use the handshake first on both programs being sure to use the same IP and port (not referring to the last variable on the client) and then use the send and receive functions to pass data back and forth.
again, these aren't that good, in fact there's many problems that cab arise with using this method, but for a simple task, they got the job done. I set up the handshake to print verifications of what is happening, but if those get annoying, you can just remove those lines.
Hope this helps!
I think multiproccessing library could be a solution.
You will be able to run several processes in parallel when each process could perform it specific work, while sending data to each other.
You can check this example
This is generic directory walker, which have process that scans directory tree and passes the data to other process, which scans files in already discovered folders. All this done in parallel.
This is probably a little bit outside the scope of your project, but have you considered using ROS? It lets you run a bunch of different nodes (can be Python scripts) at the same time that communicate by publishing and subscribing to topics. They can be on the same system (i.e. one or more nodes on the robot) or different systems (i.e. one node on the robot, multiple nodes on the PC). ROS also has a lot of awesome built in tools and libraries that are specifically made for robotic systems such as visualization, mapping, odometry, etc. Here's a bit of starting info:
https://en.wikipedia.org/wiki/Robot_Operating_System
http://wiki.ros.org/ROS/StartGuide
It's usually used for much larger frameworks than you seem to be describing, and beware that it takes quite a bit of time (in my experience) to implement, but it is very easy to expand once its up and running. Like I said, it all depends on the scope of your project!
Good luck!
I've been looking around all day, but I haven't been able to fix the problem I've got with my chat client here.
Here's the issue: I recently decided to change the client so that it would allow the user to input any message they wanted without having to wait for a reply first (blocking or something stops my program until a reply is in)
I decided to use the select.select module to do so, but after writing a couple different versions of my client today trying to get it to work, I keep getting stuck at this one point.
Whenever I enter a message, the loop gets stuck somewhere, (probably at .recv data)
how can I fix this? Nothing I try gets it to go by that.
Edit: To be more clear, when I run, I get to the point where I input the message, hit enter and then nothing happens at all. It just stays running like that.
from socket import *
import select
import sys #because why not?
print("New Chat Client Using Select Module")
HOST = input("Host: ")
PORT = int(input("Port: "))
s = socket(AF_INET,SOCK_STREAM)
print("Trying to connect....")
s.connect((HOST,PORT))
s.setblocking(0)
# Not including setblocking(0) because select handles that.
print("You just connected to",HOST,)
# Lets now try to handle the client a different way!
while True:
Incoming_data = [s]
Exportable_data = []
Exceptions = []
User_input = input("Your message: ")
rlist,wlist,xlist = select.select(Incoming_data,Exportable_data,Exceptions)
if User_input == True:
Exportable_data += [User_input]
for i in rlist:
data = i.recv(1024)
if data == "":
continue
for i in wlist:
if Exportable_data is True:
i.send(Exportable_data)
continue
Is there any way I can override the blocking (that I presume is the problem) when it's set to receive data? Won't s.setblocking(0) make it so it won't block(?) (With or without it still gets stuck)
Thanks for taking a look
I think you should have separate thread or process which will interact with your socket and another thread, which will accept user input and print chat messages. Interaction between the threads you can do using 2 queues: for incoming and outgoing messages.
Have a look at threading and queue modules.
I have client-server architecture build in python, unfortunately the original design was made that each request to server is represented by one TCP connection and I have to send requests in large groups (20 000+) and sometimes there occurs socket error #10055.
I've already found out how to handle it in python:
>>> errno.errorcode[10055]
'WSAENOBUFS'
>>> errno.WSAENOBUFS
10055
And build a code that is able to handle that error and reconnect (of course with little time delay to give server time to do whatever it has to do):
class MyConnect:
# __init__ and send are not important here
def __enter__(self):
self.sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
# Try several reconnects
for i in range(0,100):
try:
self.sock.connect((self.address, self.port))
break
except socket.error as e:
if e.errno == errno.WSAENOBUFS:
time.sleep(1)
else:
raise
return self
def __exit__(self, type, value, traceback):
self.sock.close()
# Pseudocode
for i in range(0,20000):
with MyConnect(ip,port) as c:
c.send(i)
My questions are:
is there any "good practice" way to do this?
is e.errno == errno.WSAENOBUFS multi-platform? If not so, how to make it multi-platform?
Note: I've tested in just on Windows yet, I need it to work on Linux too.
You are clogging your TCP stack with outgoing data and all the connection establishment and termination packets.
If you have to stick to this design, then force each connection to linger until its data has been successfully sent. That is to say, that by default, close() on the socket returns immediately and further delivery attempts and connection tear-down happen "in the background". You can see that doing so over 20000+ times in a tight loop can easily overwhelm the OS network stack.
The following will force your socket close() to hang on for up to 10 seconds trying to deliver the data:
import struct
s.setsockopt(socket.SOL_SOCKET, socket.SO_LINGER, struct.pack('ii', 1, 10))
Note that this is not the same as Python socket.sendall() - that one just passes all the bytes to the kernel .
Hope this helps.