I want to access a ftp server, anonymous, just for download. My company have a proxy, and ftp ports (21) are blocked. I can't access the ftp server directly.
What I whant to do is to write some code that behaves exactly the same way browsers do. The idea is that, if I can download the files with my browser, there is way to do it with code.
My code works when I try to access a web site outside the company, but is still not working for ftp servers.
proxy = urllib2.ProxyHandler({'https': 'proxy.mycompanhy.com:8080',
'http': 'proxy.mycompanhy.com:80',
'ftp': 'proxy.mycompanhy.com:21' })
auth = urllib2.HTTPBasicAuthHandler()
opener = urllib2.build_opener(proxy, auth, urllib2.HTTPHandler)
urllib2.install_opener(opener)
urlAddress = 'https://python.org'
# urlAddress = 'ftp://ftp1.cptec.inpe.br'
conn = urllib2.urlopen(urlAddress)
return_str = conn.read()
print return_str
When I try to access python.org, it works fine. If I remove the install_opener part, it does not work anymore, proving that the proxy is required.
When I use the ftp url, it blocks (or timeout if I choose to use these parameters).
I understand that ftp and http are two very different protocols.
What I don't understand is the mechanism that browsers use to access these ftp servers.
I mean, I don't know if there is a layer on server side that interfaces between http and ftp, retriveing a html; or if browser, in some other maner, access the ftp and builds the page.
There also might be a confusion with the ftp domain (or the url) and the connection mode. It seems to me that when urllib2 reads the ftp://... it automatically uses the port 21.
I found a solution using wget. This package handles with proxies, but documentation was very ubscure. You need to setup an environment variable with proxy name.
import wget
import os
import errno
# setup proxy
os.environ["ftp_proxy"] = "proxy.mycompanhy.com"
os.environ["http_proxy"] = "proxy.mycompanhy.com"
os.environ["https_proxy"] = "proxy.mycompanhy.com"
src = "http://domain.gov/data/fileToDownload.txt"
out = "C:\\outFolder\\outFileName.txt" # out is optional
# create output folder if it doesn't exists
outFolder, _ = os.path.split( out )
try:
os.makedirs(outFolder)
except OSError as exc: # Python >2.5
if exc.errno == errno.EEXIST and os.path.isdir(outFolder):
pass
else: raise
# download
filename = wget.download(src, out)
Related
I have this running in a simple python script that runs a http server, its supposed to retrieve data from a post call from my utility company, ive noticed though that as soon as i run it, and navigate to https://www.yougetsignal.com/tools/open-ports/ and check if my port is open for 443, it says closed (even though its open before)
i put this in a test.py and run it so i can wait for the post request and download the xml data, but for whatever it turns off my port when i go and check it
from pgesmd_self_access.api import SelfAccessApi
from pgesmd_self_access.server import SelfAccessServer
from pgesmd_self_access.helpers import save_espi_xml
pge_api = SelfAccessApi.auth( < i changed this to reflect my auth file > )
SelfAccessServer(pge_api, save_file=save_espi_xml)
i thought the port should stay open on my router? it doesnt seem like normal behavior
i think i need to implement SO_REUSEADDR in his code according to some docket documentation, but im not sure how to do that
class SelfAccessServer:
"""Server class for PGE SMD Self Access API."""
def __init__(
self, api_instance, save_file=None, filename=None, to_db=True, close_after=False
):
"""Initialize and start the server on construction."""
PgePostHandler.api = api_instance
PgePostHandler.save_file = save_file
PgePostHandler.filename = filename
PgePostHandler.to_db = to_db
server = HTTPServer(("", 7999), PgePostHandler)
server.socket = ssl.wrap_socket(
server.socket,
certfile=api_instance.cert[0],
keyfile=api_instance.cert[1],
server_side=True,
)
if close_after:
server.handle_request()
else:
server.serve_forever()
I'm currently utilising Paramiko and SCPClient in Python to transfer a directory from one server to another as a means of backup. This works well however I do not want it to copy hidden files (.file_name) or symbolic links. Is this possible?
Unfortunately rsync isn't an option for me as it's not available on either of the remote servers I connect to. My script is below (sensitive info replaced with dummy data). Note I need to connect to a jump host before being able to connect to target_1 or target_2.
import os
import shutil
import time
import paramiko
from scp import SCPClient
#set up ssh variables
j_host = '00.00.00.00'
target_host_1 = '00.00.00.001'
target_host_2 = '00.00.00.002'
port_no = 22
username = ''
passw = ''
#set up temporary folder on local machine to store files
path = "/local_path/"
os.mkdir(path)
#create SSH Client for jump server
jump_host=paramiko.SSHClient()
jump_host.set_missing_host_key_policy(paramiko.AutoAddPolicy())
jump_host.connect(j_host, username=username, password=passw)
#set up channel to connect to 1 via jump server
jump_host_transport_1 = jump_host.get_transport()
src_addr = (j_host, port_no)
dest_addr_1 = (target_host_1, port_no)
jump_host_channel_1 = jump_host_transport_1.open_channel("direct-tcpip", dest_addr_1, src_addr)
#set up channel to connect to 2 via jump server
jump_host_transport_2 = jump_host.get_transport()
dest_addr_2 = (target_host_2, port_no)
jump_host_channel_2 = jump_host_transport_2.open_channel("direct-tcpip", dest_addr_2, src_addr)
#function which sets up target server, either 1 or 2
def create_SSHClient(server, port, user, password, sock):
target=paramiko.SSHClient()
target.set_missing_host_key_policy(paramiko.AutoAddPolicy())
target.connect(server, port, user, password, sock=sock)
return target
#invoke above function to set up connections for 1 & 2
ssh_1 = create_SSHClient(target_host_1, port_no, username, passw, jump_host_channel_1)
ssh_2 = create_SSHClient(target_host_2, port_no, username, passw, jump_host_channel_2)
#delete old files in backup folder
command = "rm -rf /filepath/{*,.*}"
stdin, stdout, stderr = ssh_2.exec_command(command)
lines = stdout.readlines()
#print(lines)
#pause to ensure old directory is cleared
time.sleep(5)
#SCPCLient takes a paramiko transport as an argument, sets up file transfer connection
scp_1 = SCPClient(ssh_1.get_transport())
scp_2 = SCPClient(ssh_2.get_transport())
#get files from 1, store on local machine, put on 2
scp_1.get('/filepath/.', '/target_folder_local/', recursive=True)
scp_2.put('/target_folder_local/.', '/filepath/', recursive=True)
#remove temporary folder
shutil.rmtree(path)
#close connections
ssh_1.close()
ssh_2.close()
jump_host.close()
There's no API in SCPClient to skip hidden files or symbolic links.
For upload, it's easy, if you copy the SCPClient's code and modify it as you need. See the os.walk loop in _send_recursive function.
If you do not want to modify the SCPClient's code, you will have to iterate the files on your own, calling SCPClient.put for each. It will be somewhat less efficient, as it will start new SCP server for each file.
For download, you might be able to modify the SCPClient code to respond with non-zero code to C commands fed by the server for the files you do not want to download.
Check the _recv_file function. There where name is resolved, check for names or attributes of files you are not interested in downloading and do chan.send('\x01') and exit the function.
Though why do you want to use SCP? Use SFTP. It is much better suited for custom rules you need.
Paramiko does not have recursive SFTP transfer functionality (But pysftp does, see pysftp vs. Paramiko). But you won't be able to use it anyway, for the same reason you cannot use it with SCP. For your specific needs.
But check my answer to Python pysftp get_r from Linux works fine on Linux but not on Windows. It shows a simple recursive SFTP download code. Just modify it slightly to skip the files you do not want to download.
Something like
if (not S_ISLNK(mode)) and (not entry.filename.startswith(".")):
(see Checking if a file on SFTP server is a symbolic link, and deleting the symbolic link, using Python Paramiko/pysftp)
I have a samba directory smb://172.16.0.10/public_pictures/ and I would like to know if it is accessible.
try something like the following:
import urllib
if open("smb://172.16.0.10/public_pictures/"):
print("accessible")
else:
print("no accessible")
but obviously it does not work for me
Using pysmb (docs):
from smb.SMBConnection import SMBConnection
remote_address = "172.16.0.10"
share_name = "public_pictures"
conn = SMBConnection(username, password, name, remote_name)
conn.connect(remote_address)
accessible = share_name in conn.listShares()
One way of handling samba is to use pysmb. If so, then it goes something like the following:
# we need to provide localhost name to samba
hostname = socket.gethostname()
local_host = (hostname.split('.')[0] if hostname
else "SMB{:d}".format(os.getpid()))
# make a connection
cn = SMBConnection(
<username>, <password>, local_host, <netbios_server_name>,
domain=<domain>, use_ntlm_v2=<use_ntlm_v2>,
is_direct_tcp=<self.is_direct_tcp>)
# connect
if not cn.connect(<remote_host>, <remote_port>):
raise IOError
# working connection ... to check if a directory exists, ask for its attrs
attrs = cn.getAttributes(<shared_folder_name>, <path>, timeout=30)
Some notes:
in your example above, public_pictures is the shared folder, while path would be simply /
you'll need to know if you are using SMB on port 139 or 445 (or a custom port). If the latter you will usually want to pass is_direct_tcp=True (although some servers will still serve NetBIOS samba on 445)
if you expect not to need a username or password, then probably you are expecting to connect as username="guest" with an empty password.
I have a python code that would need to make use of various ShadowSocks proxy server that I have set up in order to use the IP of those servers.
Say for example I would like to use:
1.1.1.1:5678
2.2.2.2:5678
3.3.3.3:5678
i.e., all these servers have the same remote port and the local ports are all 1080.
My preference is to have the 3 proxies to rotate randomly so that each time I send a urlopen() request (in urllib2), my code randomly connect to one of the proxies and send the request via that proxy, and disconnect when the request is complete.
The IP could be hard coded or could be stored in some config files.
The problem is at the moment, all the sample online that I have found seems all require the connection to be pre-established and the Python code should simply use whatever that is on localhost:1080 instead of actively making connections.
I am just wondering if anyone could lend me a helping hand to accomplish this in the code.
Thanks!
If you have a look at the source of urllib2, you can see that when a default opener is installed, it is really just takes an object with an open method. So you really just need to create an object whose open method returns a random opener. Something like the following (untested) should work:
import urllib2
import random
class RandomOpener(object):
def __init__(self, ip_list)
self.ip_list = ip_list
def open(self, *args, **kwargs):
proxy = random.choice(self.ip_list)
handler = urllib2.ProxyHandler({'http': 'http://' + proxy})
opener = urllib2.build_opener(handler)
return opener(*args, **kwargs)
my_opener = RandomOpener(['1.1.1.1:5678',
'2.2.2.2:5678',
'3.3.3.3:5678'])
urllib2.install_opener(my_opener)
I'm writing a program in Python that has to make a http request while being forced onto a direct connection in order to avoid a proxy. Here is the code I use which successfully manages this:
print "INFO: Testing API..."
proxy = urllib2.ProxyHandler({})
opener = urllib2.build_opener(proxy)
urllib2.install_opener(opener)
req = urllib2.urlopen('http://maps.googleapis.com/maps/api/geocode/json?address=blahblah&sensor=true')
returneddata = json.loads(req.read())
I then want to add a try statement around 'req', in order to handle a situation where the user is not connected to the internet, which I have tried like so:
try:
req = urllib2.urlopen('http://maps.googleapis.com/maps/api/geocode/json?address=blahblah&sensor=true')
except urllib2.URLError:
print "Unable to connect etc etc"
The trouble is that by doing that, it always throws the exception, even though the address is perfectly accessible & the code works without it.
Any ideas? Cheers.