Ok, I have a socket and I'm handling one line at a time and logging it. The code below works great for that. The cout function is what I use to send data to the log. The for loop I use so I can process one line at a time.
socket.connect((host, port))
readbuffer = ""
while True:
readbuffer = readbuffer+socket.recv(4096).decode("UTF-8")
cout("RECEIVING: " + readbuffer) #Logs the buffer
temp = str.split(readbuffer, "\n")
readbuffer=temp.pop( )
for line in temp:
#Handle one line at a time.
The I ran into a problem where when the server disconnected me, all of the sudden I had a massive file full of the word "RECEIVING: ". I know this is because when a python socket disconnects the socket starts receiving blank data constantly.
I have tried inserting:
if "" == readbuffer:
print("It disconnected!")
break
All that did was immediately break the loop, and say that it disconnected, even on a successful connection.
I also know that I can detect a disconnection by sending data, but I can't do that, because anything I send gets broadcasted to all the other clients on the server, and this is meant to debug those clients so I it would interfere.
What do I do. Thank you in advanced.
You need to check the result of the recv() separately to the readbuffer
while True:
c = socket.recv(4096)
if c == '': break # no more data
readbuffer = readbuffer + c.decode("UTF-8")
...
Related
I'm trying to scrape really hectic twitch chats for keywords but sometimes the socket stops for a split second, but in that split second, 5 messages can go by. I thought of implementing some multithreading but no luck in the code below. It seems like they all fail to catch a keyword, or all succeed. Any help is appreciated. Code below:
import os
import time
from dotenv import load_dotenv
import socket
import logging
from emoji import demojize
import threading
# loading environment variables
load_dotenv()
# variables for socket
server = "irc.chat.twitch.tv"
port = 6667
nickname = "frankied003"
token = os.getenv("TWITCH_TOKEN")
channel = "#xqcow"
# creating the socket and connecting
sock = socket.socket()
sock.connect((server, port))
sock.send(f"PASS {token}\n".encode("utf-8"))
sock.send(f"NICK {nickname}\n".encode("utf-8"))
sock.send(f"JOIN {channel}\n".encode("utf-8"))
while True:
consoleInput = input(
"Enter correct answer to the question (use a ',' for multiple answers):"
)
# if console input is stop, the code will stop ofcourse lol
if consoleInput == "stop":
break
# make array of all the correct answers
correctAnswers = consoleInput.split(",")
correctAnswers = [answer.strip().lower() for answer in correctAnswers]
def threadingFunction():
correctAnswerFound = False
# while the correct answer is not found, the chats will keep on printing
while correctAnswerFound is not True:
while True:
try:
resp = sock.recv(2048).decode(
"utf-8"
) # sometimes this fails, hence retry until it succeeds
except:
continue
break
if resp.startswith("PING"):
sock.send("PONG\n".encode("utf-8"))
elif len(resp) > 0:
username = resp.split(":")[1].split("!")[0]
message = resp.split(":")[2]
strippedMessage = " ".join(message.split())
# once the answer is found, the chats will stop, correct answer is highlighted in green, and onto next question
if str(strippedMessage).lower() in correctAnswers:
print(bcolors.OKGREEN + username + " - " + message + bcolors.ENDC)
correctAnswerFound = True
else:
if username == nickname:
print(bcolors.OKCYAN + username + " - " + message + bcolors.ENDC)
# else:
# print(username + " - " + message)
t1 = threading.Thread(target=threadingFunction)
t2 = threading.Thread(target=threadingFunction)
t3 = threading.Thread(target=threadingFunction)
t1.start()
time.sleep(.3)
t2.start()
time.sleep(.3)
t3.start()
time.sleep(.3)
t1.join()
t2.join()
t3.join()
First, it makes not much sense to let 3 threads in parallel read on the same socket, it only leads to confusion and race conditions.
The main problem though is that you are assuming that a single recv will always read a single message. But this is not how TCP works. TCP has no concept of a message, but only is a byte stream. A message is an application level concept. A single recv might contain a single message, multiple messages, parts of messages ...
So you have to actually parse the data you get according to the semantics defined by the application protocol, i.e.
initialize some buffer
get some data from the socket and add them to the buffer - don't decode the data
extract all full messages from the buffer, decode and process each of the message separately
leave remaining incomplete messages in the buffer
continue with #2
Apart from that don't blindly throw away errors during recv(..).decode(..). Given that you are using a blocking socket recv will usually only fail if there is a fatal problem with the connection, in which case a retry will not help. The problem is most likely because you are calling decode on incomplete messages which might also mean invalid utf-8 encoding. But since you simply ignore the problem you essentially lose the messages.
I'm trying to write a Python program that can browse directories and grab files w/ sockets if the client connects to the server. The browsing part works fine, it prints out all directories of the client.
Here's a part of the code:
with clientsocket:
print('Connected to: ', addr)
while True:
m = input("Command > ")
clientsocket.send(m.encode('utf-8'))
data = clientsocket.recv(10000)
if m == "exit":
clientsocket.close()
if m.split()[0] == 'get':
inp = input("Filename > ")
while True:
rbuf = clientsocket.recv(8192)
if not rbuf:
break
d = open(inp, "ab")
d.write(rbuf)
d.close()
elif data.decode('utf-8').split()[0] == "LIST":
print(data.decode('utf-8'))
if not data:
break
However, the problem lies in here:
if m.split()[0] == 'get':
inp = input("Filename > ")
while True:
rbuf = clientsocket.recv(8192)
if not rbuf:
break
It seems to be stuck in an infinite loop. What's more interesting is that the file I'm trying to receive is 88.3kb, but what the file returns is 87kb while it's in the loop, which is very close...
I tried receiving a python script at one time as well (without the loop) and it works fine.
Here's some of the client code:
while True:
msg = s.recv(1024).decode('utf-8')
if msg.split()[0] == "list":
dirs = os.listdir(msg.split()[1])
string = ''
for dira in dirs:
string += "LIST " + dira + "\n"
s.send(string.encode('utf-8'))
elif msg == "exit":
break
else:
#bit that sends the file
with open(msg.split()[1], 'rb') as r:
s.sendall(r.read())
So my question is, why is it getting stuck in an infinite loop if I have it set up to close when there is no data, and how can I fix this?
I'm sort of new to network programming in general, so forgive me if I miss something obvious.
Thanks!
I think I know what's the problem, but I may be wrong. It happened to me several times, that the entire message is not received in one recv call, even if I specify the correct length. However, you don't reach the end of stream, so your program keeps waiting for remaining of 8192 bytes which never arrives.
Try this:
Sending file:
#bit that sends the file
with open(msg.split()[1], 'rb') as r:
data = r.read()
# check data length in bytes and send it to client
data_length = len(data)
s.send(data_length.to_bytes(4, 'big'))
s.send(data)
s.shutdown(socket.SHUT_RDWR)
s.close()
Receiving the file:
# check expected message length
remaining = int.from_bytes(clientsocket.recv(4), 'big')
d = open(inp, "wb")
while remaining:
# until there are bytes left...
# fetch remaining bytes or 4094 (whatever smaller)
rbuf = clientsocket.recv(min(remaining, 4096))
remaining -= len(rbuf)
# write to file
d.write(rbuf)
d.close()
There are several issues with your code.
First:
clientsocket.send(m.encode('utf-8'))
data = clientsocket.recv(10000)
This causes the file to be partially loaded to data variable when you issue get statement. That's why you don't get full file.
Now this:
while True:
rbuf = clientsocket.recv(8192)
if not rbuf:
break
...
You indeed load full file but the client never closes the connection (it goes to s.recv() after sending the file) so if statement is never satisfied. Thus this loop gets blocked on the clientsocket.recv(8192) part after downloading the file.
So the problem is that you have to somehow notify the downloader that you've sent all the data even though the connection is still open. There are several ways to do that:
You calculate the size of the file and send it as a first few bytes. For example, say the content of the file is ala ma kota. These are 11 bytes and thus you send \x11ala ma kota. Now receiver knows that first byte is size and it will interpret it as such. Of course one byte header isn't much (you would only be able to send max 256 byte files) so normally you go for for example 4 bytes. So now your protocol between client and server is: first 4 bytes is the size of the file. Thus our initial file would be sent as \x0\x0\x0\x11ala ma kota. The drawback of this solution is that you have to know the size of the content before sending it.
You mark the end of the stream. So you pick a particular character, say X and you read the straem until you find X. If you do then you know that the other side sent everything it has. The drawback is that if X is in the content then you have to escape it, i.e. additional content processing (and interpretation on the other side) is needed.
Hey so I decided to create an IRC ChatBot whose sole purpose it is to read incoming messages from Twitch Chat and if a giveaway is recognized by a keyword it's supposed to enter the giveaway by sending !enter in Chat.
I build the Bot upon this source: https://github.com/BadNidalee/ChatBot. I only changed things in the Run.py so thats the only Code I'm going to post. The unaltered ChatBot does work but it has no reconnect ability and regularly stops receiving data because the socket closes or other reasons.
All I wanted to change was make it so that the ChatBot is stable and can just stay in the IRC Chat constantly without disconnecting. I tried to achieve this by setting a timeout of 8 seconds for my socket and catching timeout exceptions that would occur and reconnect after they occur.
And all in all it does seem to work, my Bot does what it's supposed to even when alot of messages are coming in, it recognizes when a Giveaway starts and answers acordingly. IRC Server PING Messages are also handled and answered correctly. If there is no message in Chat for over 8 seconds the Exception gets thrown correctly and the Bot also reconnects correctly to IRC.
BUT heres my Problem: After seemingly random times the socket will literally just Stop working. What I find strange is it will sometimes work for 20 minutes and sometimes for an hour. It doesn't occur when special events, like lots of messages or something else happens in Chat, it really seems random. It will not timeout there's just nothing happening anymore. If I cancel the program with CTRL-C at this point the console sais the last call was "readbuffer = s.recv(1024)" But why is it not throwing a timeout exception at that point? If s.recv was called the socket should timeout if nothing is received after 8 seconds but the program just stops and there is no more output until you manually abort it.
Maybe I went about it the wrong way completely. I just want a stable 24/7-able ChatBot that scans for one simple keyword and answers with one simple !enter.
This is also my first Time programming in Python so If I broke any conventions or made any grave mistakes let me know.
The getUser Method returns the username of the line of chat that is scanned currently.
The getMessage Method returns the message of the line of chat that is scanned.
The openSocket Method opens the Socket and sends JOIN NICK PASS etc to the IRC
#!/usr/bin/python
import string
import socket
import datetime
import time
from Read import getUser, getMessage
from Socket import openSocket, sendMessage
from Initialize import joinRoom
connected = False
readbuffer = ""
def connect():
print "Establishing Connection..."
irc = openSocket()
joinRoom(irc)
global connected
connected = True
irc.settimeout(8.0)
print "Connection Established!"
return irc
while True:
s = connect()
s.settimeout(8.0)
while connected:
try:
readbuffer = s.recv(1024)
temp = string.split(readbuffer, "\n")
readbuffer = temp.pop()
for line in temp:
if "PING" in line:
s.send(line.replace("PING", "PONG"))
timern = str(datetime.datetime.now().time())
timern = timern[0:8]
print timern + " PING received"
break
user = getUser(line)
message = getMessage(line)
timern = str(datetime.datetime.now().time())
timern = timern[0:8]
print timern +" " + user + ": " + message
if "*** NEW" in message:
sendMessage(s, "!enter")
break
except socket.timeout:
connected = False
print "Socket Timed Out, Connection closed!"
break
except socket.error:
connected = False
print "Socket Error, Connection closed!"
break
I think you've missunderstood how timeout work on the socket.
s.settimeout(8.0)
Will only set s.connect(...) to timeout if it can't reach the destination host.
Further more, usually what you want to use instead if s.setblocking(0) however this alone won't help you either (probably).
Instead what you want to use is:
import select
ready = select.select([s], [], [], timeout_in_seconds)
if ready[0]:
data = s.recv(1024)
What select does is check the buffer to see if any incoming data is available, if there is you call recv() which in itself is a blocking operation. If there's nothing in the buffer select will return empty and you should avoid calling recv().
If you're running everything on *Nix you're also better off using epoll.
from select import epoll, EPOLLIN
poll = epoll()
poll.register(s.fileno(), EPOLLIN)
events = poll.poll(1) # 1 sec timeout
for fileno, event in events:
if event is EPOLLIN and fileno == s.fileno():
data = s.recv(1024)
This is a crude example of how epoll could be used.
But it's quite fun to play around with and you should read more about it
I start learning socket programming and I created a simple server that should replay everything I send to him back to me.
And its work work and I manage to telnet to it and write things.
but I have two problems with my code:
First, every sigh I hit on the keyboard get immediately send and I get it back. it doesn't wait until I press enter.
Second, when I print to the client 'Welcome to the server. Type something and hit enter\n\n' the client see the cmd imput cursor in the middle of the new line, and not in the start of it.
here is part of my code:
#server in local host
class ClientConnection(threading.Thread):
def __init__(self,conn):
threading.Thread.__init__(self)
self.conn = conn
def run(self):
clientthread(self.conn)
#create a new thread for each connection
def clientthread(conn):
conn.send('Welcome to the server. Type something and hit enter\n\n'.encode())
while True:
data = conn.recv(2048).decode()
replay = 'OK....' + data
if not data:
break
conn.sendall(replay.encode())
conn.close()
... #socket get close in the end
How do I make the server response only if I the client press enter? I try to check if the recive = '\n' but it doesn't seem to work.
I will appreciate any help and tips
edit: my second question answard simply
Yes, the client will often send you data as the user types rather then only when the user hits return. Thus, it is up to you to check if the data has a return, and only respond if it does.
data = ""
while True:
chunk = conn.recv(2048).decode()
if not data:
break
# try to separate the chunk by newline, to see if you got one.
while True:
split = chunk.split("\r\n", num=1)
data += split[0]
if len(split) == 1:
break
# Now we have a completed newline, so send the response
replay = 'OK....' + data
conn.sendall(replay.encode())
data = ""
conn.close()
i am trying to create a gui client for my command line server. However, i am running into some annoying problems i cant seem to fix.
I'm not 100 % sure of what the actual problem is as sometimes the code will work, other times it wont. I think the main problem is that originally i tried the
while 1:
self.data = s.recv(1024)
if not self.data():
break
else:
print self.data()
Then i was sending to it with this
for f in files:
s.send(f)
Each f was a string of a filename. I expected it to come out on the recv side as one file name recieved for each recv call but instead on one recv call i got a big chunk of filenames i assume 1024 chars worth
Which made it impossible to check for the end of the data and thus the loop never exited.
This is the code i have now
def get_data(self,size = 1024):
self.alldata = ""
while 1:
while gtk.events_pending():
gtk.main_iteration()
self.recvdata = self.s.recv(size)
self.alldata += self.recvdata
if self.alldata.find("\r\n\r\nEOF"):
print "recieved end message"
self.rdata = self.alldata[:self.alldata.find("\r\n\r\nEOF")]
break
print "All data Recieved: " + str(len(self.rdata)) + "Bytes"
print "All data :\n" + self.rdata + "\n-------------------------------------------------"
self.infiles = self.rdata.split("-EOS-")
for nf in self.infiles:
if len(nf) > 2:
self.add_message(self.incomingIcon,nf)
At the minute im trying to get the client to read correctly from the server. What i want to happen is when the command list is typed in and sent to the client the server sends back the data and each file is appended to the list store
some times this works ok, other times only one of 1200 files gets returned, if it executes ok, if i try to type another command and send it , the whole gtk window geys out and the program becomes unresponsive.
Sorry i cant explain this question better, ive tried alot of different solutions all of which give different errors.
if someone could explain the recv command and why it may be giving the errors this is how im sending data to the client
if(commands[0] == 'list'):
whatpacketshouldlooklike=""
print "[Request] List files ", address
fil = list_files(path)
for f in fil:
sdata = f
whatpacketshouldlooklike += sdata + "-EOS-"
newSock.send(sdata +"-EOS-")
#print "sent: " + sdata
newSock.send("\r\n\r\nEOF")
whatpacketshouldlooklike += "\r\n\r\nEOF"
print "---------------------------------"
print whatpacketshouldlooklike
print "---------------------------------"
The problem you had in the first part is that sockets are stream based not message based. You need to come up with a message abstraction to layer on top of the stream. This way the other end of the pipe knows what is going on(how much data to expect as a part of one command) and isn't guessing at what is supposed to happen.
Use an abstraction layer (Pyro, XML-RPC, zeromq) or define your own protocol to distinguish messages.
For example as own protocol you can send the length of a message as a "header" before each string. In this case you should use the struct module to parse the length into a binary format. Ask again, if you want to go this way, but I strongly recommend choosing one of the mentioned abstraction layers.
There are different problems with your code.
Let's start with the fundamental that some people already commented, there is no relation between sends() and recv(), you do not control which part of the data is returned on a recv(call), you need some kind of protocol, on your case it could be just as simple as terminating command strings with "\n", and checking for "\n" on the server to consume the data.
Now other problems:
You are using send without checking it's return size, a send() does not guarantee that the data is completely written, if you need that please use sendall().
By using recv(1024) in a blocking socket (default), your server code may wait for 1024 bytes to be received, this will not allow you to process messages until you get the full chunk, you need to use a non blocking socket, and the select module.
My source code:
def readReliably(s,n):
buf = bytearray(n)
view = memoryview(buf)
sz = 0
while sz < n:
k = s.recv_into(view[sz:],n-sz)
sz += k
# print 'readReliably()',sz
return sz,buf
def writeReliably(s,buf,n):
sz = 0
while sz < n:
k = s.send(buf[sz:],n-sz)
sz += k
# obj = s.makefile(mode='w')
# obj.flush()
# print 'writeReliably()',sz
return sz
Usage of these functions:
# Server
s = socket.socket(socket.AF_INET,socket.SOCK_STREAM)
s.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
s.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEPORT, 1)
s.setsockopt(socket.IPPROTO_TCP, socket.TCP_NODELAY, 1)
s.bind((host,port))
s.listen(10) # unaccepted connections
while True:
sk,skfrom = s.accept()
sz,buf = io.readReliably(sk,4)
a = struct.unpack("4B",buf)
print repr(a)
# ...
io.writeReliably(sk,struct.pack("4B",*[0x01,0x02,0x03,0x04]))
See also official docs about recv_into(...), https://docs.python.org/2/library/socket.html#socket.socket.recv_into