Develop Reference

Multithreading for a socket connection in python

I'm trying to scrape really hectic twitch chats for keywords but sometimes the socket stops for a split second, but in that split second, 5 messages can go by. I thought of implementing some multithreading but no luck in the code below. It seems like they all fail to catch a keyword, or all succeed. Any help is appreciated. Code below:
import os
import time
from dotenv import load_dotenv
import socket
import logging
from emoji import demojize
import threading
# loading environment variables
load_dotenv()
# variables for socket
server = "irc.chat.twitch.tv"
port = 6667
nickname = "frankied003"
token = os.getenv("TWITCH_TOKEN")
channel = "#xqcow"
# creating the socket and connecting
sock = socket.socket()
sock.connect((server, port))
sock.send(f"PASS {token}\n".encode("utf-8"))
sock.send(f"NICK {nickname}\n".encode("utf-8"))
sock.send(f"JOIN {channel}\n".encode("utf-8"))
while True:
consoleInput = input(
"Enter correct answer to the question (use a ',' for multiple answers):"
)
# if console input is stop, the code will stop ofcourse lol
if consoleInput == "stop":
break
# make array of all the correct answers
correctAnswers = consoleInput.split(",")
correctAnswers = [answer.strip().lower() for answer in correctAnswers]
def threadingFunction():
correctAnswerFound = False
# while the correct answer is not found, the chats will keep on printing
while correctAnswerFound is not True:
while True:
try:
resp = sock.recv(2048).decode(
"utf-8"
) # sometimes this fails, hence retry until it succeeds
except:
continue
break
if resp.startswith("PING"):
sock.send("PONG\n".encode("utf-8"))
elif len(resp) > 0:
username = resp.split(":")[1].split("!")[0]
message = resp.split(":")[2]
strippedMessage = " ".join(message.split())
# once the answer is found, the chats will stop, correct answer is highlighted in green, and onto next question
if str(strippedMessage).lower() in correctAnswers:
print(bcolors.OKGREEN + username + " - " + message + bcolors.ENDC)
correctAnswerFound = True
else:
if username == nickname:
print(bcolors.OKCYAN + username + " - " + message + bcolors.ENDC)
# else:
# print(username + " - " + message)
t1 = threading.Thread(target=threadingFunction)
t2 = threading.Thread(target=threadingFunction)
t3 = threading.Thread(target=threadingFunction)
t1.start()
time.sleep(.3)
t2.start()
time.sleep(.3)
t3.start()
time.sleep(.3)
t1.join()
t2.join()
t3.join()

First, it makes not much sense to let 3 threads in parallel read on the same socket, it only leads to confusion and race conditions.
The main problem though is that you are assuming that a single recv will always read a single message. But this is not how TCP works. TCP has no concept of a message, but only is a byte stream. A message is an application level concept. A single recv might contain a single message, multiple messages, parts of messages ...
So you have to actually parse the data you get according to the semantics defined by the application protocol, i.e.
initialize some buffer
get some data from the socket and add them to the buffer - don't decode the data
extract all full messages from the buffer, decode and process each of the message separately
leave remaining incomplete messages in the buffer
continue with #2
Apart from that don't blindly throw away errors during recv(..).decode(..). Given that you are using a blocking socket recv will usually only fail if there is a fatal problem with the connection, in which case a retry will not help. The problem is most likely because you are calling decode on incomplete messages which might also mean invalid utf-8 encoding. But since you simply ignore the problem you essentially lose the messages.

Python IRC ChatBot hangs on socket.recv after seemingly random time even though socket.settimeout is 8

Hey so I decided to create an IRC ChatBot whose sole purpose it is to read incoming messages from Twitch Chat and if a giveaway is recognized by a keyword it's supposed to enter the giveaway by sending !enter in Chat.
I build the Bot upon this source: https://github.com/BadNidalee/ChatBot. I only changed things in the Run.py so thats the only Code I'm going to post. The unaltered ChatBot does work but it has no reconnect ability and regularly stops receiving data because the socket closes or other reasons.
All I wanted to change was make it so that the ChatBot is stable and can just stay in the IRC Chat constantly without disconnecting. I tried to achieve this by setting a timeout of 8 seconds for my socket and catching timeout exceptions that would occur and reconnect after they occur.
And all in all it does seem to work, my Bot does what it's supposed to even when alot of messages are coming in, it recognizes when a Giveaway starts and answers acordingly. IRC Server PING Messages are also handled and answered correctly. If there is no message in Chat for over 8 seconds the Exception gets thrown correctly and the Bot also reconnects correctly to IRC.
BUT heres my Problem: After seemingly random times the socket will literally just Stop working. What I find strange is it will sometimes work for 20 minutes and sometimes for an hour. It doesn't occur when special events, like lots of messages or something else happens in Chat, it really seems random. It will not timeout there's just nothing happening anymore. If I cancel the program with CTRL-C at this point the console sais the last call was "readbuffer = s.recv(1024)" But why is it not throwing a timeout exception at that point? If s.recv was called the socket should timeout if nothing is received after 8 seconds but the program just stops and there is no more output until you manually abort it.
Maybe I went about it the wrong way completely. I just want a stable 24/7-able ChatBot that scans for one simple keyword and answers with one simple !enter.
This is also my first Time programming in Python so If I broke any conventions or made any grave mistakes let me know.
The getUser Method returns the username of the line of chat that is scanned currently.
The getMessage Method returns the message of the line of chat that is scanned.
The openSocket Method opens the Socket and sends JOIN NICK PASS etc to the IRC
#!/usr/bin/python
import string
import socket
import datetime
import time
from Read import getUser, getMessage
from Socket import openSocket, sendMessage
from Initialize import joinRoom
connected = False
readbuffer = ""
def connect():
print "Establishing Connection..."
irc = openSocket()
joinRoom(irc)
global connected
connected = True
irc.settimeout(8.0)
print "Connection Established!"
return irc
while True:
s = connect()
s.settimeout(8.0)
while connected:
try:
readbuffer = s.recv(1024)
temp = string.split(readbuffer, "\n")
readbuffer = temp.pop()
for line in temp:
if "PING" in line:
s.send(line.replace("PING", "PONG"))
timern = str(datetime.datetime.now().time())
timern = timern[0:8]
print timern + " PING received"
break
user = getUser(line)
message = getMessage(line)
timern = str(datetime.datetime.now().time())
timern = timern[0:8]
print timern +" " + user + ": " + message
if "*** NEW" in message:
sendMessage(s, "!enter")
break
except socket.timeout:
connected = False
print "Socket Timed Out, Connection closed!"
break
except socket.error:
connected = False
print "Socket Error, Connection closed!"
break

I think you've missunderstood how timeout work on the socket.
s.settimeout(8.0)
Will only set s.connect(...) to timeout if it can't reach the destination host.
Further more, usually what you want to use instead if s.setblocking(0) however this alone won't help you either (probably).
Instead what you want to use is:
import select
ready = select.select([s], [], [], timeout_in_seconds)
if ready[0]:
data = s.recv(1024)
What select does is check the buffer to see if any incoming data is available, if there is you call recv() which in itself is a blocking operation. If there's nothing in the buffer select will return empty and you should avoid calling recv().
If you're running everything on *Nix you're also better off using epoll.
from select import epoll, EPOLLIN
poll = epoll()
poll.register(s.fileno(), EPOLLIN)
events = poll.poll(1) # 1 sec timeout
for fileno, event in events:
if event is EPOLLIN and fileno == s.fileno():
data = s.recv(1024)
This is a crude example of how epoll could be used.
But it's quite fun to play around with and you should read more about it

socket programming for beginner - server creating

I start learning socket programming and I created a simple server that should replay everything I send to him back to me.
And its work work and I manage to telnet to it and write things.
but I have two problems with my code:
First, every sigh I hit on the keyboard get immediately send and I get it back. it doesn't wait until I press enter.
Second, when I print to the client 'Welcome to the server. Type something and hit enter\n\n' the client see the cmd imput cursor in the middle of the new line, and not in the start of it.
here is part of my code:
#server in local host
class ClientConnection(threading.Thread):
def __init__(self,conn):
threading.Thread.__init__(self)
self.conn = conn
def run(self):
clientthread(self.conn)
#create a new thread for each connection
def clientthread(conn):
conn.send('Welcome to the server. Type something and hit enter\n\n'.encode())
while True:
data = conn.recv(2048).decode()
replay = 'OK....' + data
if not data:
break
conn.sendall(replay.encode())
conn.close()
... #socket get close in the end
How do I make the server response only if I the client press enter? I try to check if the recive = '\n' but it doesn't seem to work.
I will appreciate any help and tips
edit: my second question answard simply

Yes, the client will often send you data as the user types rather then only when the user hits return. Thus, it is up to you to check if the data has a return, and only respond if it does.
data = ""
while True:
chunk = conn.recv(2048).decode()
if not data:
break
# try to separate the chunk by newline, to see if you got one.
while True:
split = chunk.split("\r\n", num=1)
data += split[0]
if len(split) == 1:
break
# Now we have a completed newline, so send the response
replay = 'OK....' + data
conn.sendall(replay.encode())
data = ""
conn.close()

How do I react on user input during reactor.run() in twisted python?

Im currently working on a client/server multi player cardgame solution in python using twisted libraries. Im pretty new to python, and this game actually is a proof of concept to myself, as i just want to show myself, that i can do such stuff. but im stuck ~8 hours on this problem now, and i need some advice.
following lines and the following description should let you know, what im trying to do here:
Description:
Server ist started
as soon as the demo client connects, server does some stuff in the background (registering, creating pile of cards for the player)
the demo client now starts() the cardgame
remote_start is called and calls function hoststart if its started by certain player (to be implemented)
the cards from the players pile are sent to the client and print out
the client should now be able to press a number (number of card), which then should be sent back to the server to print out the number (only printing doesnt make sense, but its the only thing that came to my mind right now :P)
now here starts the problem:
as soon as the reactor.run() is started, it seems as whole script is executed (user registered, game started) and the input at the client is done after the whole script is executed. so the sending of the entered number is done, before the number is entered within the input function and therefore returns 0 to the server.
I think this is very complex to explain, so please ask any questions which may come up.
How can i resolve this problem?
Can i somehow make the server wait an amount of seconds so that the client is able to input the Number during that time?
Do i need to open a new connection which allways opens if have to wait for input on the client?
client.py:
PICK = 0
factory = pb.PBClientFactory()
reactor.connectTCP("localhost", PORT, factory)
def1 = factory.getRootObject()
def1.addCallbacks(got_obj1, err_obj1)
ret = reactor.run()
def got_obj1(obj1):
def2 = obj1.callRemote("register","User A")
def2 = obj1.callRemote("start","User A")
def2.addCallback(show)
obj1.callRemote("pick",PICK) # Problem
def show(*var)
data_string = json.dumps(var)
decoded = json.loads(data_string)
for card in decoded[0]:
print(str(card['name'])
PICK = Input("Which card do you want to pick?") # Problem
server.py:
print("Starting server on ", PORT)
rpc = RPCs()
reactor.listenTCP(PORT, pb.PBServerFactory(rpc))
reactor.run()
class RPCs(pb.Root):
sessions = []
def remote_pick(self,pick):
print("picked: ", pick)
def remote_start(self, username):
for session in self.sessions :
if (session.getName() == username):
ret = self.hoststart(username)
return ret
def hoststart(self,username):
pile4client = []
card4pile {}
for session in self.sessions:
if (session.getName() == username):
ret = session.showpile(0)
for card in ret:
card4pile = { "name" : card.getName()}
pile4client .append(card4pile)
return pile4client
class Session():
piles = []
def showpile(self, num):
return self.piles[num]

Pass obj1 to show and call its callRemote in show. Something like this:
def got_obj1(obj1):
...
def2.addCallback(show, obj1)
def show(value, obj1):
...
PICK = Input("Which card do you want to pick?")
obj1.callRemote("pick",PICK)

Windows Process won't communicate across a socket

I've been writing an IRC chatbot that checks Google Voice for SMS messages then relays them to the chatroom through a socket on port 6667. The chat message processing runs in the main sub, in an infinite loop, while the GV checking is done in a separate process. The actual checking and fetching works fine, but it's the socket that doesn't work; no message is posted to the channel. What's strange is that this works flawlessly under OS X, so I don't believe the problem is the Voice processing logic:
def checkVoice()
while 1:
print "Update voice!"
#voice processing... edited for brevity
sendPrivateMessage(CHANNEL,message) #This doesn't work
#more code
time.sleep(10)
#main block
if __name__ == '__main__':
sendPrivateMessage(CHANNEL,"Voice checking started") #This works
p = Process(target=checkVoice)
p.start()
I assumed that the problem was with the way Windows is with Python, seeing as it works with other platforms.
The full code for the chatbot can be seen here:
bot.py
As asked, here is the sendPrivateMessage method:
def sendPrivateMessage(channel, message):#private message send function
global mute
if mute == 0:
IRC.send("PRIVMSG " + channel + " :" + message + "\r\n")