Develop Reference

Send/Read data to/from a socket SQL Server/Python

Using SQL 2016 & Python 3.7
I currently have two python programs:
A server that receives input from a socket and returns output
A client that sends a prompt to the socket, reads the response and then outputs.
I then have a script in SQL Server that uses xp_cmdshell to run the client and read the results. I'd like to be able to remove the client from the process by using SQL Server to directly access the socket. Unfortunately since you can access SQL Server as a socket, my searches are receiving a high level of noise and not giving me the results I need.
If I upgrade to SQL2017 I can use the internal python option to run the client locally, however upgrading will not be an option for some of our cients and I need a one size fits all solution.
Example Server: SimpleSocket.py:
import socket
from _thread import *
HEADER_LENGTH = 10
def threaded(local_client_socket):
command_header = local_client_socket.recv(HEADER_LENGTH)
command_length = int(command_header.decode("utf-8"))
command = local_client_socket.recv(command_length).decode("utf-8")
if not command:
print('Connection closed by client')
print("Received from client:", command)
reply = f"You sent me '{command}' This is my reply.".encode("utf-8")
reply_header = str(len(reply)).zfill(HEADER_LENGTH).encode("utf-8")
local_client_socket.send(reply_header + reply)
local_client_socket.close()
def open_sockets():
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.bind(("127.0.0.1", 9878))
print("Socket bound to port", 9878)
s.listen(16)
print("Socket is listening")
print("-" * 80)
while True:
client_socket, address = s.accept()
print('Connected to :', address[0], ':', address[1])
start_new_thread(threaded, (client_socket,))
if __name__ == '__main__':
open_sockets()
Example Client: SimpleClient.py
import socket
import sys
HEADER_LENGTH = 10
client_socket = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
client_socket.connect(("127.0.0.1", 9878))
command = "Text I'm sending to server".encode("utf-8")
command_header = str(len(command)).zfill(HEADER_LENGTH).encode("utf-8")
client_socket.send(command_header + command)
reply_header = client_socket.recv(HEADER_LENGTH)
if not len(reply_header):
print("Done")
sys.exit()
reply_length = int(reply_header.decode("utf-8"))
reply = client_socket.recv(reply_length).decode(("utf-8"))
print("Reply from server:", reply)
client_socket.close()
Example SQL Call:
DECLARE #Python_Location NVARCHAR(255) = N'c:\ProgramData\Anaconda3\envs\Search\python.exe'
, #Python_Script NVARCHAR(255) = N'c:\PythonScripts\Search\SimpleClient.py'
, #Command NVARCHAR(255)
SET #Command = N'"' + #Python_Location + N' ' + #Python_Script + '"'
PRINT #Command;
EXECUTE AS USER = 'cmdshell';
EXEC xp_cmdshell #stmt = #Command;
REVERT;
I have used OLE Automation in the past to pull data from a webservice, so was wondering if that may be an option. It's not really an area I know well and my searches haven't turned up anything relevant.
I'd be interested in knowing any options available to me, and all offerings gratefully received. Even if you don't have a full answer, any help may send me in a good direction, or help me structure my search queries to be more relevant. I may even be able to adjust the python code to output the result set to a different format if there's an efficient option.
Many thanks.

"NO IDENT RESPONSE" error when connecting to an IRC server

Usually I would attempt something like this with the twisted library, but that isn't available for python 3 - so I attempted it using the sockets library. The code does establish a connection, but the server quickly responds with "no ident response". I don't have much network programming experience, so I would appreciate it if someone could point out the error I'm making here. I'm also quite aware that there are functions/other code that aren't used, or that Ive used inconsistently. I just thought I would paste the entirety of my code here in case its relevant.
import socket
server = "irc.freenode.net"
channel = "put channel here"
nickname = "put nickname here"
def encode(text):
return text.encode("ascii")
def ping():
irc_socket.send(encode("PONG :pingis\n"))
def send_message(chan, msg):
irc_socket.send(encode("PRIVMSG " + chan + " :" + msg + "\n"))
def join_channel(chan):
irc_socket.send(encode("JOIN " + chan + "\n"))
def login(username='user', realname='Pythonist', hostname='Helena', servername='Server'):
irc_socket.send(encode("USER %s %s %s %s" % (username, hostname, servername, realname)))
irc_socket.send(encode("NICK " + nickname))
irc_socket = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
irc_socket.connect((server, 6667))
login()
join_channel(channel)
while True:
buffer = irc_socket.recv(1024)
msg = str.split(str(buffer))
if msg[0] == "PING":
irc_socket.send("PONG %s" % msg[1])
print(msg)
The code was originally from: http://wiki.shellium.org/w/Writing_an_IRC_bot_in_Python and Ive made minor changes.

Teensy tiny little problem that’s causing all the trouble: you’re missing newlines in login. "USER %s %s %s %s" should be "USER %s %s %s %s\n" and "NICK " + nickname should be "NICK " + nickname + "\n". The server looks up your ident and doesn’t find it, and the next step after that is for you to register, but you never send it a complete line, so it keeps waiting… and waiting…

Ident is UNIX service which nobody has been using for twenty years or so.
It was used to identify the remote user when doing terminal server to remote server connections in terminal applications. Ident is no way safe for modern internet, so nobody is using it anymore.
So you have hit the ghost of the past. Also, as mentioned in the above answer, if you send correct NICK command the IRC server is happy with your client.

Unable To Connect Python Sockets On Different Computers

I recently wrote a code for a small chat program in Python. Sockets connect fine when I connect them from different terminals on the same system. But the same doesn't seem to happen when I connect them from different computers which are connected over the same Wifi network.
Here's the server code:
#!/usr/bin/env python
print "-"*60
print "WELCOME TO DYNASOCKET"
print "-"*60
import socket, os, sys, select
host = "192.168.1.101"
port = 8888
connlist = []
try:
s = socket.socket(socket.AF_INET,socket.SOCK_STREAM)
print "Socket Successfully Created."
connlist.append(s)
s.bind((host,port))
print "Socket Successfully Binded."
s.listen(10)
print "Socket is Now Listening."
except Exception, e:
print "Error : " + str(e)
sys.exit()
def air(sock,message):
for socket in connlist:
if socket != sock and socket != s:
try:
socket.sendall(message)
except:
connlist.remove(socket)
while 1:
read_sockets,write_sockets,error_sockets = select.select(connlist,[],[])
for sock in read_sockets:
if sock == s:
conn, addr = s.accept()
connlist.append(conn)
print "Connected With " + addr[0] + " : " + str(addr[1])
else:
try:
key = conn.recv(1024)
print "<" + str(addr[1]) + ">" + key
data = raw_input("Server : ")
conn.sendall(data + "\n")
air(sock, "<" + str(sock.getpeername()) + ">" + key)
except:
connlist.remove(sock)
print "Connection Lost With : " + str(addr[1])
conn.close()
s.close()
Here's the client script:
#!/usr/bin/env python
print "-"*60
print "WELCOME TO DYNASOCKET"
print "-"*60
import socket, os, sys
host = "192.168.1.101"
port = 8888
try:
s = socket.socket(socket.AF_INET,socket.SOCK_STREAM)
print "Socket Successfully Created."
s.connect((host,port))
print "Connected With " + host + " : " + str(port)
except socket.error, e:
print "Error : " + str(e)
while 1:
reply = raw_input("Client : ")
s.send(reply)
message = s.recv(1024)
print "Server : " + message
s.close()
When I try to connect The client From a different computer I get this error :
Error : [Errno 10060] A Connection attempt failed because the connected party
did not respond after a period of time, or established connection failed
because connected host has failed to respnd.

Your are binding your server only to the local host, so that connections from other hosts are blocked.
Try:
s.bind(("0.0.0.0",port))

I experienced this problem and it took me a many hours to figure this out and I found that (like many others said #Cld) it is your firewall blocking the connection. How I fixed this:
Try to run the server onto the machine that you trying to connect from.
(For example, if you want to run the server on machine A and connect from machine B, run the server on machine B).
If you are on windows (I am not sure about Mac or Linux) it will popup with with the firewall pop-up, which will allow you to give permission to your program to access your private network.
Simply tick the box that says:
"Private networks, such as my home or work network"
and Press allow access
That's it! You've fixed that particular issue. Now feel free to test the server on that machine or close the server and go back to your main machine, which will host that server and run it. You should see that it is now working.
I hope this has helped you, as it is my first post!
EDIT: I also did what #Daniel did in his post with changing the s.bind to include '0.0.0.0'.

I had this same problem for quite sometime, and creating tcp tunnels with ngrok worked for me. You can check it out here
For simple sockets application on your pc, just expose the port you're using by ngrok tcp <port_number>, bind the server socket to localhost and port exposed, and use the url of the tunnel with the port number at client side (typically looks like 0.tcp.us.ngrok.io and a port number).
You can even make multiple tunnels on the free account (needed in my case) by specifying the --region flag: https://ngrok.com/docs#global-locations

How do i program a simple IRC bot in python?

I need help writing a basic IRC bot that just connects to a channel.. is anyone able to explain me this? I have managed to get it to connect to the IRC server but i am unable to join a channel and log on. The code i have thus far is:
import sockethost = 'irc.freenode.org'
port = 6667
join_sock = socket.socket()
join_sock.connect((host, port))
<code here>
Any help would be greatly appreciated.

To connect to an IRC channel, you must send certain IRC protocol specific commands to the IRC server before you can do it.
When you connect to the server you must wait until the server has sent all data (MOTD and whatnot), then you must send the PASS command.
PASS <some_secret_password>
What follows is the NICK command.
NICK <username>
Then you must send the USER command.
USER <username> <hostname> <servername> :<realname>
Both are mandatory.
Then you're likely to see the PING message from server, you must reply to the server with PONG command every time the server sends PING message to you. The server might ask for PONG between NICK and USER commands too.
PING :12345678
Reply with the exact same text after "PING" with PONG command:
PONG :12345678
What's after PING is unique to every server I believe so make sure you reply with the value that the server sent you.
Now you can join a channel with JOIN command:
JOIN <#channel>
Now you can send messages to channels and users with PRIVMSG command:
PRIVMSG <#channel>|<nick> :<message>
Quit with
QUIT :<optional_quit_msg>
Experiment with Telnet! Start with
telnet irc.example.com 6667
See the IRC RFC for more commands and options.
Hope this helps!

I used this as the MAIN IRC code:
import socket
import sys
server = "server" #settings
channel = "#channel"
botnick = "botname"
irc = socket.socket(socket.AF_INET, socket.SOCK_STREAM) #defines the socket
print "connecting to:"+server
irc.connect((server, 6667)) #connects to the server
irc.send("USER "+ botnick +" "+ botnick +" "+ botnick +" :This is a fun bot!\n") #user authentication
irc.send("NICK "+ botnick +"\n") #sets nick
irc.send("PRIVMSG nickserv :iNOOPE\r\n") #auth
irc.send("JOIN "+ channel +"\n") #join the chan
while 1: #puts it in a loop
text=irc.recv(2040) #receive the text
print text #print text to console
if text.find('PING') != -1: #check if 'PING' is found
irc.send('PONG ' + text.split() [1] + '\r\n') #returnes 'PONG' back to the server (prevents pinging out!)
Then, you can start setting commands like: !hi <nick>
if text.find(':!hi') !=-1: #you can change !hi to whatever you want
t = text.split(':!hi') #you can change t and to :)
to = t[1].strip() #this code is for getting the first word after !hi
irc.send('PRIVMSG '+channel+' :Hello '+str(to)+'! \r\n')
Note that all irc.send texts must start with PRIVMSG or NOTICE + channel/user and the text should start with a : !

It'd probably be easiest to base it on twisted's implementation of the IRC protocol. Take a look at : http://github.com/brosner/bosnobot for inspiration.

This is an extension of MichaelvdNet's Post, which supports a few additional things:
Uses SSL wrapper for socket
Uses server password authentication
Uses nickserv password authentication
Uses nonblocking sockets, to allow other events to trigger
Logs changes to text files to channel
#!/usr/local/bin/python
import socket
import ssl
import time
## Settings
### IRC
server = "chat.freenode.net"
port = 6697
channel = "#meLon"
botnick = "meLon-Test"
password = "YOURPASSWORD"
### Tail
tail_files = [
'/tmp/file-to-tail.txt'
]
irc_C = socket.socket(socket.AF_INET, socket.SOCK_STREAM) #defines the socket
irc = ssl.wrap_socket(irc_C)
print "Establishing connection to [%s]" % (server)
# Connect
irc.connect((server, port))
irc.setblocking(False)
irc.send("PASS %s\n" % (password))
irc.send("USER "+ botnick +" "+ botnick +" "+ botnick +" :meLon-Test\n")
irc.send("NICK "+ botnick +"\n")
irc.send("PRIVMSG nickserv :identify %s %s\r\n" % (botnick, password))
irc.send("JOIN "+ channel +"\n")
tail_line = []
for i, tail in enumerate(tail_files):
tail_line.append('')
while True:
time.sleep(2)
# Tail Files
for i, tail in enumerate(tail_files):
try:
f = open(tail, 'r')
line = f.readlines()[-1]
f.close()
if tail_line[i] != line:
tail_line[i] = line
irc.send("PRIVMSG %s :%s" % (channel, line))
except Exception as e:
print "Error with file %s" % (tail)
print e
try:
text=irc.recv(2040)
print text
# Prevent Timeout
if text.find('PING') != -1:
irc.send('PONG ' + text.split() [1] + '\r\n')
except Exception:
continue

That will open a socket, but you also need to tell the IRCd who you are. I've done something similar in perl ages ago, and I found the IRC RFCs to be very helpful.
Main RFC: http://irchelp.org/irchelp/rfc/rfc.html
Other RFCs: http://irchelp.org/irchelp/rfc/index.html

Check if remote host is up in Python

How would I check if the remote host is up without having a port number? Is there any other way I could check other then using regular ping.
There is a possibility that the remote host might drop ping packets

This worked fine for me:
HOST_UP = True if os.system("ping -c 1 " + SOMEHOST) is 0 else False

A protocol-level PING is best, i.e., connecting to the server and interacting with it in a way that doesn't do real work. That's because it is the only real way to be sure that the service is up. An ICMP ECHO (a.k.a. ping) would only tell you that the other end's network interface is up, and even then might be blocked; FWIW, I have seen machines where all user processes were bricked but which could still be pinged. In these days of application servers, even getting a network connection might not be enough; what if the hosted app is down or otherwise non-functional? As I said, talking sweet-nothings to the actual service that you are interested in is the best, surest approach.

HOST_UP = True if os.system("ping -c 5 " + SOMEHOST.strip(";")) is 0 else False
to remove nasty script execution just add .strip(";")
-c 5
to increase the number of ping requests, if all pass than True
PS. Works only on Linux, on Windows always returns True

The best you can do is:
Try and connect on a known port (eg port 80 or 443 for HTTP or HTTPS); or
Ping the site. See Ping a site in Python?
Many sites block ICMP (the portocol used to ping sites) so you must know beforehand if the host in question has it enabled or not.
Connecting to a port tells you mixed information. It really depends on what you want to know. A port might be open but the site is effectively hung so you may get a false positive. A more stringent approach might involve using a HTTP library to execute a Web request against a site and see if you get back a response.
It really all depends on what you need to know.

Many firewalls are configured to drop ping packets without responding. In addition, some network adapters will respond to ICMP ping requests without input from the operating system network stack, which means the operating system might be down, but the host still responds to pings (usually you'll notice if you reboot the server, say, it'll start responding to pings some time before the OS actually comes up and other services start up).
The only way to be certain that a host is up is to actually try to connect to it via some well-known port (e.g. web server port 80).
Why do you need to know if the host is "up", maybe there's a better way to do it.

What about trying something that requires a RPC like a 'tasklist' command in conjunction with a ping?

I would use a port scanner. Original question states that you don't want to use a port. Then you need to specify which Protocol (Yes, this needs a port) you want to monitor: HTTP, VNC, SSH, etc. In case you want to monitor via ICMP you can use subprocess and control ping parameters, number of pings, timeout, size, etc.
import subprocess
try:
res = subprocess.Popen(['ping -t2 -c 4 110.10.0.254 &> /dev/null; echo $?'],shell=True,stdout=subprocess.PIPE,stderr=subprocess.PIPE)
out, err = res.communicate()
out = out.rstrip()
err = err.rstrip()
print 'general.connectivity() Out: ' + out
print 'general.connectivity() Err: ' + err
if(out == "0"):
print 'general.connectivity() Successful'
return True
print 'general.connectivity() Failed'
return False
except Exception,e:
print 'general.connectivity() Exception'
return False
In case you want a port scanner
import socket
from functools import partial
from multiprocessing import Pool
from multiprocessing.pool import ThreadPool
from errno import ECONNREFUSED
NUM_CORES = 4
def portscan(target,port):
try:
# Create Socket
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
socketTimeout = 5
s.settimeout(socketTimeout)
s.connect((target,port))
print('port_scanner.is_port_opened() ' + str(port) + " is opened")
return port
except socket.error as err:
if err.errno == ECONNREFUSED:
return False
# Wrapper function that calls portscanner
def scan_ports(server=None,port=None,portStart=None,portEnd=None,**kwargs):
p = Pool(NUM_CORES)
ping_host = partial(portscan, server)
if portStart and portStart:
return filter(bool, p.map(ping_host, range(portStart, portStart)))
else:
return filter(bool, p.map(ping_host, range(port, port+1)))
# Check if port is opened
def is_port_opened(server=None,port=None, **kwargs):
print('port_scanner.is_port_opened() Checking port...')
try:
# Add More proccesses in case we look in a range
pool = ThreadPool(processes=1)
try:
ports = list(scan_ports(server=server,port=int(port)))
print("port_scanner.is_port_opened() Port scanner done.")
if len(ports)!=0:
print('port_scanner.is_port_opened() ' + str(len(ports)) + " port(s) available.")
return True
else:
print('port_scanner.is_port_opened() port not opened: (' + port +')')
return False
except Exception, e:
raise
except Exception,e:
print e
raise