I decided to play around with the SMTP and wrote a very simple test script in Python:
import socket
sock = socket.socket()
sock.bind(("",25))
sock.listen(1)
print("Server started...")
while True:
conn,addr = sock.accept()
print(addr,"connected")
print(conn.recv(4096))
conn.close()
The script does exactly what I want it to do when I connect to it via ncat: it prints out the data it gets, then closes. I also set up my router to forward connections on port 25 to my computer, so it should be able to be accessed externally.
However, when I send an email via Gmail to test#<my ip address>, I never receive it. Why does this happen?
I understand that there is a SMTP library for Python, but I'm not trying to make a serious mail server, I just want to learn how it works.
As I have said before, I don't understand SMTP very well, but I was under the assumption that it would work pretty much like any other network protocol and I would be able to at least print out the request I would get using this program.
The text, test#<my ip address> isn't a valid syntax for an email address.
Assuming that your ip address is, say, 198.51.100.43, try this email address:
test#[198.51.100.43]
References:
https://www.ietf.org/rfc/rfc2822.txt
https://www.ietf.org/rfc/rfc2821.txt , section 4.1.3
As I have said before, I don't understand SMTP very well, but I was under the assumption that it would work pretty much like any other network protocol and I would be able to at least print out the request I would get using this program.
I suggest that your understanding of "pretty much like any other network protocol" is too much influenced by HTTP. Because all the classic protocols like SMTP, FTP, IMAP, POP, NNTP ... consist of several handshakes between client and server and the first one for all of these protocols is a message from the server. Which means that the SMTP client will connect to the server and time out after a while because the initial greeting was not received.
For the basic specification of SMTP see RFC 2821, but there will be others to read if you want to support TLS and other extensions.
Related
Currently I am trying to reverse engineer how the key exchange and encryption works for a program. I mainly used wire shark to figure out how the packets where being encrypted. Of what I have found they are using ECDHE-RSA-AES128-GCM-SHA256 with TLS1.2
This is currently what I have.
import socket, ssl
tcpSocket = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
tcpSocket.bind(('', 443))
tcpSocket.listen(1)
while True:
newsocket, fromaddr = tcpSocket.accept()
newsocket = ssl.wrap_socket(newsocket,
server_side=True,
do_handshake_on_connect=True,
certfile="cert.pem",
ssl_version=ssl.PROTOCOL_TLSv1_2,
ciphers="ECDHE-RSA-AES128-GCM-SHA256")
try:
#Later add stuff
pass
finally:
newsocket.shutdown(socket.SHUT_RDWR)
newsocket.close()
I have tried to visualize how this works using a simple picture here.
I made this using the wireshark packets that I saw here
Currently with this code, the server never replies with a client key exchange and instead just crashes leading to https://hastebin.com/wiqojapule.sql
So the main question im trying to ask is why does the python program never aw acknowledge the client and then send a server hello. Currently it just sends a Client Hello back.
Also, here is the wire shark for my python socket and the program: here
Comparing the two pcap files (with real server and with test server) two things stand out:
In the working case (real server) the client is sending a server_name extension (SNI) with the target hostname and the server sends a certificate back which is signed by a public CA.
In the failing case (test server) the client is sending no server_name extension and the server is sending only a self-signed certificate back. The connection is then closed by the client, i.e. the handshake is not completed.
The missing SNI extension against the test server is probably due to given the destination as IP address and not a hostname. And this is probably also not the problem.
Very likely instead is that the client simply does not like the self-signed certificate send by the server and is therefore closing the connection. A nicer behaving client would probably send first a TLS alert unknown CA before the close but it is not uncommon that TLS stacks don't send these alerts and instead just close the connection.
... I am trying to reverse engineer how the key exchange and encryption works for a program
There does not seem anything special to reverse engineer here. The client seems to use TLS as specified in the standard, likely by using some of the many available TLS stacks and not by implementing its own. And contrary to the assumption of the OP that only a specific cipher is used the client is actually offering several ciphers (19 ciphers offered inside ClientHello) and the server can choose any of these.
The behavior one is seeing regarding self-signed certificates is actually expected: a proper client should not accept connections to a server with an untrusted certificate.
I'm new to the world of networking and wanted to clarify some of my thoughts regarding a problem I am facing at the moment. I saw this post which makes me think what I'm doing may be impossible, but I thought it would be worth a shot to ask on here and see what more qualified people think about it.
I am a TA for an intro computer science course, and I am writing a final project for students to complete at the end of the semester. Essentially, the project would be to fill in the holes in the implementation of a messaging client. I have set it up so each client would run two threads (one to listen for incoming messages, and one to wait for input to send messages to the other client). I have gotten this to work successfully on localhost with communication between two different port numbers, and am trying to find a way to have this work over the network so the two clients do not necessarily have to be on the same machine.
After struggling through a few methods, I came up with this solution: I would host a server on Heroku that would keep track of the clients' IPs and port numbers, and use a rest API so that one client could easily get the IP and port of the other client they are trying to communicate with. I have tested this, and the API seems to work. Thus, a client can create a socket endpoint and send it to this server to be entered into its database, and when the communication is terminated, it is removed from the database (this JSON would store a username as the primary key and internally manage an IP and port number) as the connection is now closed.
So, what I have is each client with an IP and port number knowing the IP and port number it is trying to communicate with. My last struggle is to actually form the connection. I understand there is a distinction between localhost (127.0.0.1) and the public IP for an internet endpoint. Upon searching, I found a way to find the public IP for the current user to share with the database, but I cannot bind to it. Whenever I try to, I get sockets error code 13: permission denied. I would imagine that if I tried connecting to the public IP of the other machine, I would get a similar error (but I cannot test the client until I can get a server running!).
I read online that some router work would be needed to actually form this connection between two machines. I guess I'm struggling to understand the practicality of socket programming if such a simple operation (connecting two socket endpoints on two different computers) requires so much tweaking. Is there something I am missing?
For reference, here is a general outline of my code thus far. The server:
# Server thread
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.bind((LOCAL_IP, AVAILABLE_PORT))
s.listen(1)
# In my code, there is a quitting mechanism which closes s as well.
while True:
client_socket, addr = s.accept()
data = client_socket.recv(1024)
print "Received: " + data
client_socket.close()
...and the client:
# Client thread
# It is an infinite loop so I am always waiting for another potential message to send
while True:
x = raw_input()
sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
sock.connect((OTHER_MACHINE_LOCAL_IP, OTHER_PORT))
sock.sendall(x)
sock.close()
Right now, I cannot make progress with these permission denied errors. Is there any way I could get this to work? Understand that, being that this would be for around 250ish students to use who are all intro CS students, I would want to avoid having to instruct them to do anything with their routers.
If there is another method to do this which would make this easier that I am missing, I would also love to hear any suggestions :) Thanks in advance!
I have a program (.exe) that gets certain info from a server, and I wanted to be able to get that info from the command line too. I started netcat and listened on the port the program uses to communicate with its target, because I wanted to know what "requests" to make from a script, and I received this over netcat, plain text:
net.tcp://[my ip address]:41012/Lapis.Btouch/ServerInfo
I tried sending exactly that to the target (replacing my IP for its IP) using socket.send() but it wouldn't return anything. What does that mean and how can I get the data?
import socket
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.connect(('[server ip address]',41012))
while True:
s.send('net.tcp://[server ip address]:41012/Lapis.Btouch/ServerInfo')
response = s.recv(1024)
print response
s.close()
That's the code I'm using to send the request to the target server. It won't return anything, so I think I'm not making the request in the right way. Thanks for any help.
Try capture your network packet on port 41012, via tcpdump or wireshark or ...
Then check:
does your code send the request?
does the response returned to you?
If the answer for question (1) is False, problem is side of you and your machine. so go and solve it (review your code, check your firewall and ...)
If the answer of question (1) is True, but server doesn't send any response (this case often not happened) review your request format (check you URL or test it via browser or ...). also based on the server you connect to, maybe needed to your machine's IP recognized for server (I mean some server only give response to requests, that come from known IP addresses, in this case you need introduce yourself to server (e.g. add your IP to trusted list on server), before sending request)
Last case, if answer of both questions are correct (I guess it's your problem), you most correct your code (response = s.recv(1024)) why do you used 1024 for your response length?. Use correct and accurate parameters.
In python exist several methods for receiving response via socket (from server), search stackoverflow and you can find useful tips and commands. commands like use non-blocking ways, ascync ways and ...
There're basically two issues I'd like to resolve:
Client side send query string when initializing the connection to server
Server side validate user token in handshake (not after the connection is established and then validate streaming message that contains the token) and set user session accordingly.
I read a article (https://auth0.com/blog/2014/01/15/auth-with-socket-io/) that talks about this process implemented in nodejs, just wonder if the same function can be achieved by using python. (Currently I'm doing some research on twisted but haven't found anything similar)
PS: guess it's helpful to demo the use case as well. A user may login to your server over normal http then server will issue him/her an valid accessToken. Then this user may need to establish a socket connection with the server (or some other server), then the server needs to figure out who the user is and validate before establishing the socket connection.
Query strings are part of HTTP URLs.
If you're building a TCP socket server instead of an HTTP server, you don't get URLs—or headers, or anything else out-of-band.* All you get is a stream of data. You need to come up with a protocol for your data that you can fit the token into.
This means the server can't "figure out who the user is and validate before establishing the socket connection". It has to establish the socket connection, read the first message, parse it, validate the token, and then drop or continue the connection. (You can, of course, put up a front-end server that accepts connections, validates them, and then migrates or proxies them to the real back-end server. But someone has to accept, read, and parse.)
Note that this is exactly what HTTP does—it can't see the query string until it accepts the connection and reads the first line of data.
Meanwhile, the example you're looking at appears to be using WebSockets. A WebSockets client can't talk to a socket server (well, unless you build a WebSockets server on top of your socket server, or a proxy in front of it) in the first place.
* This isn't quite true. You can cram 40 bytes of options into TCP header extensions. But then you have to go below the level people are usually talking about when they say "socket server"—and there's a good chance it won't make it through the internet. Also, TCP does have a concept of "out-of-band" data, but that isn't relevant here; you still have to accept the connection and read from it to get an OOB data.
I'd like to elaborate a little bit on this question and, particularly, on this answer. Let's suppose I have a fixed list of services to check, say: ('ftp', 'ssh', 'http'). Opening a socket to port 22 of a remote server:
import socket
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
result = s.connect_ex(('my_remote_server', 22))
if(result == 0) :
print s.recv(256)
s.close()
I get the following output:
'SSH-2.0-OpenSSH_6.1p1 Debian-4\r\n'
So I can guess there's a ssh service listening on that port. Therefore my question is: where or how can I find the welcome messages (or similar) for different kind of services? Are these regulated in some way? In the case of ssh, does it always start with SSH-?
The response you get from a server depens both on the exact protocol and server configuration.
HTTP has a Server header for example, but many server out on the internet lies about what server is responding, to throw off malicious hackers trying to exploit server-specific flaws. Moreover, the server remains silent until the client has sent a request; your code has to send something before there is any response.
The best you can do is try out different protocols and see if the other side responds sensibly. Trying to send an SSH handshake to a HTTP port is likely to result in a HTTP/1.0 400 Bad Request response, for example.
In other words, there is no ready-made solution for this, you'll have to create your own heuristics based on what protocols you are trying to detect.
The best you can do is probably search for the string ssh(/ftp/http) and conclude based on that.
Even that would be a guess, of course, as this can usually be changed(see here) with a config parameter and it is considered good security practice to change it so as to not expose the server program/version.