I'm running Python 2.6.5 on ec2 and I've replaced the old ftplib with the newer one from Python2.7 that allows importing of FTP_TLS. Yet the following hangs up on me:
from ftplib import FTP_TLS
ftp = FTP_TLS('host', 'username', 'password')
ftp.retrlines('LIST') (Times out after 15-20 min)
I'm able to run these three lines successfully in a matter of seconds on my local machine, but it fails on ec2. Any idea as to why this is?
Thanks.
It certainly sounds like a problem related to whether or not you're in PASSIVE mode on your FTP connection, and whether both ends of the connection can support it.
The ftplib documentations suggests that it is on by default, which is a shame, because I was going to suggest that you turn it on. Instead, I'll suggest that you set_debuglevel to where you can see the lower levels of the protocol happening and see what mode you're in. That should give you information on how to proceed. Either you're in passive mode and the other end can't deal with it properly, or (hopefully) you'd not, but you should be.
FTP and FTPS (but not SFTP) can be configured so that the server makes a backwards connection to the client for the actual transfers or so that the client makes a second forward connection to the server for the transfers. The former, especially, is prone to complications whenever network address translation is involved. Without the TLS, some firewalls can actually rewrite the FTP session traffic to make it magically work, but with TLS that's impossible due to encryption.
The fact that are presumably authenticating and then timing out when you try to transfer data (LIST requires a 2nd connection in one direction or the other) is the classic symptom, usually, of a setup that either needs passive mode, OR, there's this:
Connect as usual to port 21 implicitly securing* the FTP control connection before authenticating. Securing the data connection requires the user to explicitly ask for it by calling the prot_p() method.
ftps.prot_p() # switch to secure data connection
ftps.retrlines('LIST') # list directory content securely
I don't work with FTPS often, since SFTP is so much less problematic, but if you're not doing that, the far end server might not be cooperating.
*note, I suspect this sentence is trying to say that FTP_TLS "implicitly secures the FTP control connection" in contrast with the explicit securing of the data connection.
If you're still having trouble could you try ruling out Amazon firewall problems. (I'm assuming you're not using a host based firewall.)
If your EC2 instance is in a VPC then in the AWS Management Console could you:
ensure you have an internet gateway
ensure that the subnet your EC2 instance is in has a default route (0.0.0.0/0) configured pointing at the internet gateway
in the Security Group for both inbound and outbound allow All Traffic from all sources (0.0.0.0/0)
in the Network ACLs for both inbound and outbound allow All Traffic from all sources (0.0.0.0/0)
If your EC2 instance is NOT in a VPC then in the AWS Management Console could you:
in the Security Group for inbound allow All Traffic from all sources (0.0.0.0/0)
Only do this in a test environment! (obviously)
This will open your EC2 instance up to all traffic from the internet. Hopefully you'll find that your FTPS is now working. Then you can gradually reapply the security rules until you find out the cause of the problem. If it's still not working then the AWS firewall is not the cause of the problem (or you have more than one problem).
Related
I am attempting to create a basic chatroom in python, and I would like to know how I could transmit data from one script to another, preferably without using google drive. If needed to, I could create a webserver on Replit, but I don't do well with HTML or PHP.
Side note: I can't port forward, as my google wifi doesn't accept any level of port forwarding.
I would send messages of about 50 characters every couple seconds
Since you mention port forwarding, I assume you want two chat clients that run on different local networks to talk to each other, for example your own and the chat client of a friend in a remote location, over the internet.
If you (or your counterpart) cannot set up port forwarding, then direct communication between the script on your computer and theirs is hard, if not impossible. The solution is to set up a third computer or service on the internet that can be reached by both clients and use it for relaying messages between them.
A network is typically protected by a firewall of sorts and will typically be behind a router that performs network address translation (NAT) to help multiple devices on a network to simultaneously access services on the internet, whilst all using the same IP address on the internet. Port forwarding fits into that by connecting a specific port from the outside directly to a port on a machine on the inside - without that, an outside computer might be able to reach your IP address, but they could never connect to a computer or program on the inside of the network, as the router wouldn't know what computer to contact, also the firewall might disallow the connection to begin with.
But if your computer on the inside establishes a connection with an accessible server on the internet, expecting a response, that creates a temporary conduit through the router and firewall that can be used by the server to send messages (look up 'hole punching' for more information). And if both computers do this, the server can relay message between both clients. Only the server then needs to run in an environment that doesn't have firewall restrictions or NAT that prevent this.
You could write a simple Python server, that accepts incoming connections and can send several responses and a simple client that connects to it, identifying itself and joining a chatroom, or having a direct conversation with another connected client. There are many techniques that would allow you to do this, but I think web sockets might be a good starting point, as long as you don't plan to do advanced fast or high volume stuff that would require something like a UDP connection.
A library like websockets could be a good starting point, but you may want to start out by figuring out where you would have this service hosted first, since there may be limitations on what you're able and allowed to do.
Also, if all you're looking to do is send simple messages, you may want to stay away from writing your own server an protocols at all - have a look around for open source message servers written in a language you are comfortable with, or that just work out of the box without any development, in which case the language doesn't even really matter, as long as you can connect to it and exchange messages from Python.
I’ve been using the Luigi visualizer for pipelining my python code.
Now I’ve started using an aws instance, and want to access the visualizer from my own machine.
Any ideas on how I could do that?
We had the very same problem today on GCP, and solved with the following steps:
setting firewall rules for incoming TCP connections on port used by the service (which by default is 8082);
installing apache2 server on the instance with a site.conf configuration that resolve incoming requests on ip-of-instance:8082.
That's it. Hope this can help.
Good question and I'm amazed I can't find a duplicate on StackOverflow. You broadly need to do two things:
Ensure the luigi webserver is hosting content correct. You can probably do this via site.conf, or you can probably do it via luigi's default-scheduler-host property. This corresponds to #PierluigiPuce second point.
Correctly expose and secure your EC2 instance. This is a VPC exercise (see docs) and is an entire area to learn, but in short you need to configure your VPC so that the valid requests are routed to the instance on the correct port, and invalid requests are blocked. This corresponds to #PierluigiPuce first point.
Your primary consideration is whether it is okay for this to be public facing. Probably not. Then you can secure the instance via IP address ranges, via a VPN, or even via SSH port forwarding through a jump host.
Having it completely open is the easiest and worst solution. Putting the instance in a public subnet, and restricting access based on IP address, is probably the second easiest solution, and might be a reasonable compromise for you.
Gooday to All, Im writing down a very sensitive web application that functions like a file browser, Instead of using sftp/ftp or ssh. It uses purely http/https. Im using request.remote_addr to determine the client's IP address. and reject if the ip isn't on the list.
good_ips = ['127.0.0.1','192.168.1.10','192.168.1.1']
if request.remote_addr in good_ips:
pass
else:
sys.exit()
It works fine, but I just would like to ask how reliable and safe this is :).
This would be the result if the ip is not on the list. other wise the site would run fine :D.
Thank you and good day!
No, that's not sufficient to build a "sensitive" service as you are using it.
See https://en.wikipedia.org/wiki/IP_address_spoofing for just a start on what could go wrong.
You should use authentication, such as with a public key (which SSH supports), or a password. Kerberos is also a possibility.
Whether source IP filtering is an adequate measure for you depends on your exact scenario, your threat model. #JohnZwinck is right in that it usually is not enough by itself, but for some applications, it can be.
While it is easy to fake source IP in individual IP packets, http goes over tcp, and a modern implementation of tcp is protected against address spoofing. Older implementations of TCP (in older operating systems) are vulnerable though. So if your server has a recent OS, it is not at all straightforward to spoof a source IP. It's probably easier to compromise a client with an address in the allowed list.
Another thing that can go wrong is related to network address translation (nat). Suppose you restrict access in your application to internal IP addresses (192.168.0.0/24). All works well, but your security department decides there need to be reverse proxies for all web applications. A proxy is deployed and all works fine. However, now your application receives all requests from the proxy wuth an internal address, so the restriction in the application doesn't make much sense anymore. Something simlar can also happen with clients, in some scenarios, clients may be behind nat, meaning they will all have the same apparent client IP address - which can be good or bad.
The best practice is of course to have proper authentication (via passwords, client certificates, or multi-factor, etc., however secure you want it to be), with IP restriction being an additional layer to provide more security.
I'm developing a client-server game in python and I want to know more about port forwarding.
What I'm doing for instance is going to my router (192.168.0.1) and configure it to allow request for my real IP-adress to be redirected to my local adress 192.168.0.X. It works really well. But I'm wondering if I can do it by coding something automatically ?
I think skype works like a kind of p2p and I can see in my router that skype is automatically port forwarded to my pc adress. Can I do it in Python too?
There are different solutions here, but most are not trivial, and you'll have to do some reading, and you'll need some kind of fallback.
UPnP/IGD is the simplest. If your router supports it, and is configured to allow it, and you know how to write either low-level networking code or old-fashioned SOAP web service code, you can ask the router to assign you a port. If it responds with success, start using that port and you're basically done.
If you can run a (very low-bandwidth) server with a public address for all of your users, Hole punching may solve the problem.
Think about how a behind-the-NAT client talks to a public server. You make a request to some IP and port, but the server is seeing your router's IP, not yours (which is a good thing, because yours isn't accessible). When it replies, your router has to know to forward it to you—which it does just by remembering that you're the behind-the-NAT client that just sent a request to that server.
What if, instead of talking to a public server, you talk to some other peer behind his own separate NAT? Well, Your router doesn't know the difference; as long as you get a response from the same place, it'll get through. But how do you get a response, when your message isn't going to get through his NAT? He does the same thing, of course. One of the messages may get lost, but the other one will get through, and then you're both set and can communicate. You will need to send keep-alives regularly so the router doesn't forget you were communicating, but other than that, there's really nothing tricky.
The only problem is that you need to know the public IP address for the other peer, and the port that he's expecting you to come from, and he needs to know the same about you. That's why you need a server—to act as an introducer between peers.
Hole punching will work with UDP from most home networks. It won't work with TCP from many home networks, or with either UDP or TCP from many corporate networks. (Also, in corporate networks, you may have multiple layers of NATs, which means you need introducers at every interface rather than just one on the internet, or symmetric NATs, which can't be punched.)
You can use STUN (or similar) services like ICE or TURN. This only works if there is an ICE, TURN, etc. service to use—which is generally not the case for two peers on different home NATs, unless you deploy your own server and build an introducer to help out, and if you're going to do that, you can just use hole punching. But in a corporate environment, this can be the best way to provide connectivity for P2P apps.
Finally, you can make the user configure port-forwarding manually and enter the forwarded port number into your problem. This is not ideal, but you should always provide it as a fallback (except maybe for apps meant only for corporate deployment), because nothing else is going to work for all of your users.
I believe Skype uses all of these. If you enable UPnP, it tries to IGD your router. Or you can configure it to use a TURN server. Or you can just enter a specific port that you've forwarded manually. If you do none of the above, it tries to use UDP hole punching, with an introducer that Skype runs.
So your application needs to do TCP/UDP networking if I understand correctly. That means that at least one of the connecting clients needs a properly open port, and if both of them is behind NAT (a router) and have no configured open ports, your clients cannot connect.
There are several possible solutions for this, but not all are reliable: UPnP, as suggested here, can open ports on demand but isn't supported (or enabled) on all routers (and it is a security threat), and P2P solutions are complex and still require open ports on some clients.
The only reliable solution is to have a dedicated server that all clients can connect to that will negotiate the connections, and possibly proxy between them.
You could look at something like this (assuming your router supports it): http://en.wikipedia.org/wiki/Universal_Plug_and_Play#NAT_traversal
For implementing port forwarding using python, there's a fantastic ActriveState recipe that does asynchronous port forwarding server using only Python standard library (socket, syncope). Look at
http://code.activestate.com/recipes/483732-asynchronous-port-forwarding/
Imagine you have a HTTP server on your local machine, this is a typical Python/Twisted application. This server is used to access your local data, server is used just as a GUI interface. So user can use his web browser or special application ( acts like a web browser ) to access his local data.
Now you want to be sure that only local user who physically sit near this machine get access to the HTTP server.
Also I will have FTP server and it must be protected the same way too.
At the moment I am running such code for my HTTP server:
class LocalSite(server.Site):
def buildProtocol(self, addr):
if addr.host != '127.0.0.1':
print 'WARNING connection from ' + str(addr)
return None
try:
res = server.Site.buildProtocol(self, addr)
except:
res = None
return res
So I am just check the IP address at the moment and I am not sure this is enough.
Is there any ways to emulate local IP from remote machine.?
Well, If a bad guy get access over my OS I have no way to protect - but this is not my deal. My firewall and antivirus should care about this, right?
Anyway, I would like to listen any extra ideas about increase security of such HTTP server.
May be we can use MAC address to verify connection.?
Check the processes on local machine and detect which is actually executes connection?
We can use HTTPS, but in my understanding this acts in opposite direction: this is for user to trust to the server, not server to trust to the user.
Using CAPTCHA is a kind of solution. But I do not like this at all (it strains users) and this will not work for FTP server.
I am also use random port number every time application starts.
The type of internet connection is not defined - this is a p2p application. Any user in the WEB can use my software and it must be protected against remote access.
I believe the way you handled it is good enough. About it being cross-platform, I believe it is as Windows(starting from windows 7) too maps localhost to 127.0.0.1 but for previous versions, you have to define localhost in the main hosts file.