I have been trying to automate SFTP transfer from a Windows client via a python script to a CentOS machine running an Apache server. I have created a user account on the CentOS server that can only access SFTP, similar to the instructions listed here: https://www.digitalocean.com/community/tutorials/how-to-enable-sftp-without-shell-access-on-centos-7
I then used the following code in an attempt to transfer the file
transport.connect(username = username, password = password)
sftp = paramiko.SFTPClient.from_transport(transport)
sftp.put(base_dir + '\\report', '/var/www/html/reports/' + host_name, confirm = False)
However this results in the following error:
Traceback (most recent call last):
File "noschedule_make_report.py", line 74, in <module>
main()
File "noschedule_make_report.py", line 62, in main
sftp.chdir('/var/www/html/reports')
File "C:\Python27\lib\site-packages\paramiko\sftp_client.py", line 626, in chdir
if not stat.S_ISDIR(self.stat(path).st_mode):
File "C:\Python27\lib\site-packages\paramiko\sftp_client.py", line 460, in stat
t, msg = self._request(CMD_STAT, path)
File "C:\Python27\lib\site-packages\paramiko\sftp_client.py", line 780, in _request
return self._read_response(num)
File "C:\Python27\lib\site-packages\paramiko\sftp_client.py", line 832, in _read_response
self._convert_status(msg)
File "C:\Python27\lib\site-packages\paramiko\sftp_client.py", line 861, in _convert_status
raise IOError(errno.ENOENT, text)
IOError: [Errno 2] No such file
This code worked when I didn't set the restrictions on the upload user account as described in the Digital Ocean post, and instead had much more liberal permissions and shell login. Is there a way for me to have both the locked out login for the upload user and to use the Paramiko funcitonality?
Please note that using a sftp.chdir('/var/www/html/reports') command before the put command produced the same error, occurring at the chdir line instead.
Also I understand that similar questions have been asked (IOError: [Errno 2] No such file - Paramiko put()), but I am specifically asking if I can relegate these two sets of functionality.
There is a concept I think you have perhaps overlooked when configuring the sftp part, This is ChrootDirectory.
A Chroot in Unix world is a way to execute a command or an environment inside a system directory, so this directory appears the root of the system you're into. This is primary used as security feature because there is no way to escape this chroot. For instance imagine you have a path /opt/server/ftp/users/ and a ftp daemon is chrooted in /opt/server/ftp/ a client will see the users directory when he will do a ls -al and it will be impossible to access files on the system like /etc/
So this problem has nothing to do with the Paramiko code per-se but with the sftp configuration you set and the comprehension of what is a Chroot environment.
ChrootDirectory in you setup define the sftp user will be dropped into this directory when connection it created AND that he'll be impossible to see the full path of the system when it is logged, so when you upload the files you don't have to chdir /var/www/html/reports because you can't see this directory. Considering you set ChrootDirectory /var/www/html/reports
Check first the ChrootDirectory value you set, if you put /var/sftp/ but you want to access the system path (not the chroot one) /var/www/html/reports/ this is wrong. Correct to /var/www/html/reports/ seems legit, then change your code to
sftp.put(base_dir + '\\report', '.' + host_name, confirm = False)
the character . as second parameter means the current directory
Related
I'm trying to execute script from zabbix ui. I put my script to '/usr/lib/zabbix/externalscripts' folder. The script's name is "check_ssl.py". When I connect to server and go to that folder and execute the script manually - it works, but when I try to execute it from zabbix's ui - it throws an error :
"Traceback (most recent call last):
File "/usr/lib/zabbix/externalscripts/check_ssl.py", line 14, in <module>
ACCESS_KEY = conf('ACCESS_KEY')
File "/usr/local/lib/python3.8/dist-packages/decouple.py", line 243, in __call__
return self.config(*args, **kwargs)
File "/usr/local/lib/python3.8/dist-packages/decouple.py", line 105, in __call__
return self.get(*args, **kwargs)
File "/usr/local/lib/python3.8/dist-packages/decouple.py", line 90, in get
raise UndefinedValueError('{} not found. Declare it as envvar or define a default value.'.format(option))
decouple.UndefinedValueError: ACCESS_KEY not found. Declare it as envvar or define a default value."
ACCESS_KEY variable is declared and set as env. variable. Does anyone know why it throws this error?
The service environment is different from other users' env, see https://serverfault.com/questions/413397/how-to-set-environment-variable-in-systemd-service
Edit the service with systemctl edit zabbix-server and add Environment="ACCESS_KEY=your_access_key" in the [Service] section.
Create the script with Scope: Action Operation, Type: Script, Execute on: Zabbix agent (i've done it like that and works fine)
On commands insert the command, example python3 /home/suppor/python_scripts/customer/restart_apache.py
On Actions create one, select the condition (may be the trigger when to run the script), on Operation tab add the operation to run the script created
It works fine. You must allow remote commands on the zabbix agent
So, up until about a year ago I had several scripts that I ran via Windows Task Scheduler then all of a sudden I started getting 0x1 "errors".
Fast forward to present and I decided to try to figure out what was going on (because I really need some stuff to run when I am away). I have one python script that imports several others to run. Based on numerous other posts about problems with Task Scheduler,: I decided to use a .bat file
SET logfile="C:\Reports\batch.log"
#echo off
#echo Starting Script at %date% %time% >> %logfile%
call C:\Users\Me\Anaconda3\condabin\conda.bat activate C:\Users\Me\Anaconda3
C:\Users\Me\Anaconda3\python.exe C:\Users\Me\RUN_daily_notifications.py
pause
#echo finished at %date% %time% >> %logfile%
In many of the scripts that are executed are paths to different files here and there and the command line execution crashes at the first one (so I would assume I would need to fix them all). For example, in the first script I am setting up an Excel file:
writer = pd.ExcelWriter(r'P:\1MatData\Query Output\Hold Report.xlsx')
I run this via the Task Scheduler, the command line opens up and gives me this error:
C:\WINDOWS\system32>SET logfile="C:\Reports\batch.log"
Running Eng_Que
Traceback (most recent call last):
File "C:\Users\Me\RUN_daily_notifications.py", line 13, in <module>
import Eng_Que #G8
File "C:\Users\Me\Eng_Que.py", line 140, in <module>
writer = pd.ExcelWriter(r'P:\1MatData\Query Output\Hold Report.xlsx')
File "C:\Users\Me\Anaconda3\lib\site-packages\pandas\io\excel\_xlsxwriter.py", line 182, in __init__
super().__init__(
File "C:\Users\Me\Anaconda3\lib\site-packages\pandas\io\excel\_base.py", line 810, in __init__
self.handles = get_handle(
File "C:\Users\Me\Anaconda3\lib\site-packages\pandas\io\common.py", line 651, in get_handle
handle = open(handle, ioargs.mode)
FileNotFoundError: [Errno 2] No such file or directory: 'P:\\1MatData\\Query Output\\Hold Report.xlsx'
I've tried rewriting the line with the path in the script numerous ways, but never can seen to get a single backslash to pass.
For example (changes to the script and command line error):
writer = pd.ExcelWriter('P:\\1MatData\\Query Output\\Hold Report.xlsx')
still passes double backslashes:
FileNotFoundError: [Errno 2] No such file or directory: 'P:\\1MatData\\Query Output\\Hold Report.xlsx'
This:
writer = pd.ExcelWriter('P:\1MatData\Query Output\Hold Report.xlsx')
Gives:
FileNotFoundError: [Errno 2] No such file or directory: 'P:\x01MatData\\Query Output\\Hold Report.xlsx'
All of these (except for the last one of course) work fine in my terminal (Sy
pder).
I've also tried os.path.normpath and .realpath and still can't get single backslashes. Any advice, wisdom or suggestions would be greatly appreciated.
Edit to provide additional info requested:
The P: drive is a network drive that I have read/write access to.
Task properties:
General--
Selected Run only when user is logged on and have Run with highest privileges checked.
Actions--
Program/script: C:\Users\Me\Desktop\BatchFiles\daily_reports.bat
Add arguments: blank
Start in: blank
Settings--
Allow task to be run on demand -- checked
Stop the task if it runs longer than: 3 days
If the running task does not end when requested force it to stop -- checked.
Having trouble getting the pysftp.connection.cd() working – not sure what I am doing wrong or what the issue is. I can achieve what I need by passing the full remote path into the relevant methods. But if try to change the remote active directory I get error. Relevant code is below:
with ps.Connection('hostname or IP address', username='username', password='secret', cnopts=cnopts) as conn:
print("Connection successful....")
print(conn.listdir()) # this works
print(conn.listdir(remote_location)) # this works
with conn.cd(remote_location): # this throws an error
print(conn.listdir())
The error I get is below here:
File "C:\Users\username\.conda\envs\data_wrangling\lib\site-packages\pysftp\__init__.py", line 508, in cd
self.cwd(remotepath)
File "C:\Users\username\.conda\envs\data_wrangling\lib\site-packages\pysftp\__init__.py", line 524, in chdir
self._sftp.chdir(remotepath)
File "C:\Users\username\.conda\envs\data_wrangling\lib\site-packages\paramiko\sftp_client.py", line 659, in chdir
if not stat.S_ISDIR(self.stat(path).st_mode):
OverflowError: mode out of range
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "c:\Users\username\my_code\python\mears_to_er_interface\sftp_chdir_test.py", line 41, in <module>
with conn.cd("remote_location"):
File "C:\Users\username\.conda\envs\data_wrangling\lib\contextlib.py", line 112, in __enter__
return next(self.gen)
File "C:\Users\username\.conda\envs\data_wrangling\lib\site-packages\pysftp\__init__.py", line 511, in cd
self.cwd(original_path)
File "C:\Users\username\.conda\envs\data_wrangling\lib\site-packages\pysftp\__init__.py", line 524, in chdir
self._sftp.chdir(remotepath)
File "C:\Users\username\.conda\envs\data_wrangling\lib\site-packages\paramiko\sftp_client.py", line 659, in chdir
if not stat.S_ISDIR(self.stat(path).st_mode):
OverflowError: mode out of range
The output for conn.stat(remote_location).st_mode is 83448.
Paramiko uses Python stat module in the SFTPClient.chdir implementation to check if the path is a folder (and not a file).
The problem is that the Python stat module can work with 16-bit permissions only. Your SFTP server returns permissions with 17th bit set. That results in the OverflowError.
There's not much you can do about it. See this Python issue: stat.S_ISXXX can raise OverflowError for remote file modes.
Just avoid using the SFTPClient.chdir. Simply use absolute paths everywhere in the Paramiko API, instead of relying on paths relative to the working directory set using chdir.
The only other place where Paramiko currently uses the stat module is the SFTPAttributes.__str__. Otherwise you should be safe.
I've reported this to the Paramiko project: https://github.com/paramiko/paramiko/issues/1802
If you want to analyze the permission bits in your own code, just mask out the excess bits. See:
"OverflowError: mode out of range" in Python when working with file mode/attributes returned by Paramiko
I am trying to sync two folders via FTP, yes I know there are better or different ways but for now I need to implement it this way, I was trying the example code from pyftpsync since well, a sample code should work easily right? I am just trying to connect between some test folders I made, one is empty(local) and the remote has a single text file that I want to fetch. It tries to connect but after about 2 minutes I get this error.
Well, my FTP does work outside of python. I can connect over WinSCP just fine.
Some places mentioned that a proxy could possibly cause this, but it seems I am not behind a proxy currently, but maybe I did not set that properly and it believes there should be a proxy somehow?
Here is my code, just using commands on the prompt for pyftpsync produces the same errors for me. So it is possible some input parameter is off causing all of this.
import time
import os
import re
import shutil
import string
import sys
from ftpsync.targets import FsTarget
from ftpsync.ftp_target import FtpTarget
from ftpsync.synchronizers import DownloadSynchronizer
#synchronize a local folder with ftp
local = FsTarget( "C:\\testfolder\\" )
user = "login"
passwd = "password"
remote = FtpTarget("/my/folder/location/testfold/", "126.0.0.1",port=22, username=user,password=passwd,tls=False,timeout=None,extra_opts=None)
opts = {}
s=DownloadSynchronizer(local, remote, opts)
s.run()
This is the output I am getting, I have edited out the folder names and IP addresses.
INFO:keyring.backend:Loading KWallet
INFO:keyring.backend:Loading SecretService
INFO:keyring.backend:Loading Windows
INFO:keyring.backend:Loading chainer
INFO:keyring.backend:Loading macOS
INFO:pyftpsync:Download to C:\testfolder
from ftp://126.0.0.1/.../testfold
INFO:pyftpsync:Encoding local: utf-8, remote: utf-8
Traceback (most recent call last):
File "c:\..\.py", line 30, in <module>
s.run()
File "C:\\AppData\Local\Programs\Python\Python37-32\lib\site-
packages\ftpsync\synchronizers.py", line 1268, in run
res = super(DownloadSynchronizer, self).run()
File "C:\\AppData\Local\Programs\Python\Python37-
32\lib\site-packages\ftpsync\synchronizers.py", line 827, in run
res = super(BiDirSynchronizer, self).run()
File "C:\\AppData\Local\Programs\Python\Python37-
32\lib\site-packages\ftpsync\synchronizers.py", line 211, in run
self.remote.open()
File "C:\\AppData\Local\Programs\Python\Python37-
32\lib\site-packages\ftpsync\ftp_target.py", line 141, in open
self.ftp.connect(self.host, self.port)
File "C:\\AppData\Local\Programs\Python\Python37-
32\lib\ftplib.py", line 155, in connect
self.welcome = self.getresp()
File "C:\\Local\Programs\Python\Python37-
32\lib\ftplib.py", line 236, in getresp
resp = self.getmultiline()
File "C:\\AppData\Local\Programs\Python\Python37-
32\lib\ftplib.py", line 226, in getmultiline
nextline = self.getline()
File "C:\\AppData\Local\Programs\Python\Python37-
32\lib\ftplib.py", line 210, in getline
raise EOFError
EOFError
Anyways any possible troubleshooting ideas would help. Thank you.
Pyftpsync uses FTP protocol.
You are connecting to port 22, which is used for SSH/SFTP.
So if your server is actually SFTP server, not FTP server, you cannot use Pyftpsync with it.
Hi I am using paramiko 1.7.6 "fanny" on microsoft windows xp v2002 service pack3 with python 2.4.2
I have the follwing script:
import paramiko
hostname='blah'
port=22
username='blah'
password='blah'
fullpath='\\\\root\\path\\file.xls'
remotepath='/inbox/file.xls'
self.client= paramiko.SSHClient()
self.client.load_system_host_keys()
self.client.connect(hostname,port,username,password)
sftp = self.client.open_sftp()
sftp.put(fullpath,remotepath)
the error I get is:
sftp.put(fullpath,remotepath))
File "build\bdist.win32\egg\paramiko\sftp_client.py", line 577, in put
File "build\bdist.win32\egg\paramiko\sftp_client.py", line 337, in stat
File "build\bdist.win32\egg\paramiko\sftp_client.py", line 628, in _request
File "build\bdist.win32\egg\paramiko\sftp_client.py", line 675, in _read_response
File "build\bdist.win32\egg\paramiko\sftp_client.py", line 701, in _convert_status
IOError: [Errno 2] /inbox/file.xls is not a valid file path
but the path definitely exists (I can move into it using sftp.chdir('inbox')) I
have also tried moving into the folder and using put but I get the exact same
error (did take out inbox prefix)
Has anyone had this issue?
Cheers
matt
IOError: [Errno 2] /inbox/file.xls is not a valid file path
This is your error, which means that /inbox isn't a valid path. You probably meant to use
remotepath='inbox/file.xls'
I had the same issue.
The signature specifies sftp_client.py
def put(self, localpath, remotepath, callback=None, confirm=True):
most of the forums answered referred the first argument as remotepath.
if we change the first one as local path and the second one as remote path,
it works fine.
No issues with this.