I'm trying to get a Python script to work that downloads a file and then loads the data into a mariadb. My best option seems to be using subprocess.run, as I cannot update the Python version to a version that includes the new mariadb connector.
I've tried different options:
subprocess.run(["mysql", "-u" + mdb_usr, "-p" + mdb_pwd, "database", " < " + file.sql])
subprocess.run(["mysql", "-u" + mdb_usr, "-p" + mdb_pwd, "database < " + file.sql])
subprocess.run(["mysql -u" + mdb_usr + " -p" + mdb_pwd + "database < " + file.sql], shell = True)
But none of them seems to work.
Without the < file.sql, I can get the script to work, but I can't seem to get the sql to execute. Can anyone point me in the right direction?
Related
I'm trying to get the vendorId, the productCode and the SerialNumber of a usb device using python.
Therefore I created the following script which runs quite well in a linux environment.
import sys
import usb
devs = usb.core.find(find_all=True)
nCount=0
for dev in devs:
for cfg in dev:
intf = usb.util.find_descriptor(cfg,bInterfaceClass=0x8)
if intf is not None:
nCount += 1
try:
sys.stdout.write("USB device " + usb.util.get_string(dev,dev.iProduct,None) + '\n')
except:
sys.stdout.write("USB device " + str(nCount) + '\n')
sys.stdout.write("------" + '\n')
sys.stdout.write("VendorId = " + hex(dev.idVendor) + '\n')
sys.stdout.write("ProductId = " + hex(dev.idProduct) + '\n')
if not dev.iSerialNumber == 0:
try:
sys.stdout.write("SerialNbr = " + usb.util.get_string(dev,dev.iSerialNumber ) + '\n')
except NotImplementedError:
sys.stdout.write("Sreial-Number not readable")
else:
sys.stdout.write("SerialNbr = none" + '\n')
sys.stdout.write('\n')
As soon as a start the script in windows, the script returns an error "NotImplementedError" when calling the function usb.util.get_string(dev,dev.iSerialNumber)
I found that there is a problem with the windows drivers and the libusb.
The solution wI0m heading to, does not allow to change system drivers with Zadig first. IIt must be callable without changing drivers first.
So my question is, if it's possible to get the serial number with python on a windows system without changing the drivers first.
I want to monitor training progress of a CNN which is trained via a slurm process on a server (i.e., the Python script is executed through a bash script whenever the server has resources available; the session is not interactive. Hence, I cannot simply open a terminal and run Tensorboard dev).
So far, I have tried the following without finding a new experiment on my Tensorboard dev site:
mod = "SomeModelType"
logdir = "/some/directory/used/in/Tensorboard/callback"
PARAMETERS = "Some line of text describing the training settings"
subprocess.Popen(["tensorboard", "dev upload --logdir '" + logdir + \
"' --name Myname_" + mod + " --description '" + \
PARAMETERS + "'"])
If I insert the text string "tensorboard dev upload --logdir 'some/directory..." in a terminal, Tensorboard will start as expected.
If I include the code showed above, no new Tensorboard experiment will be started.
I also tried this:
subprocess.run(["/pfs/data5/home/kit/ifgg/mp3890/.local/bin/tensorboard", \
"dev", "upload", "--logdir", "'" + logdir + \
"'", "--name", "LeleNet" + mod#, "--description" + "'" + \
#PARAMETERS + "'"
], \
capture_output = False, text = False)
which starts Tensorboard, but it will not continue the Python script. Hence, Tensorboard, will be listening to output that never comes, because the Python session is listening to its own output instead of training the CNN.
Edit
This:
subprocess.Popen(["/pfs/data5/home/kit/ifgg/mp3890/.local/bin/tensorboard", \
"dev", "upload", "--logdir", "'" + logdir + \
"'", "--name", "LeleNet" + mod#, "--description" + "'" + \
#PARAMETERS + "'"
])
led to some message "Listening for new data in the log dir..." popping up all the time in interactive mode and led to cancellation of the slurm job (job disappeared). Moreover, Tensorboard does not work correcty this way. The experiment is created, but never receives any data.
I got it to work as follows:
logdir = "/some/directory"
tbn = "some_name"
DESCRIPTION = "some description of the experiment"
subprocess.call("tensorboard dev upload --logdir '" + logdir + \
"' --name " + tbn + " --description '" + \
DESCRIPTION + "' &", shell = True)
I am calling from python an R script that is very simple called R Code.R:
args <- commandArgs(TRUE)
print(args)
source(args[1])
setwd("<YOUR PATH>")
output <- head(mtcars, n = n)
write.table(output, "output.txt")
using the following script:
import subprocess
pth = "<YOUR PATH>"
subprocess.call(" ".join(["C:/R/R-3.6.0/bin/x64/R.exe", "-f", '"' + pth + '/R Code.R"', "--args",
'"' + pth + '/arguments.txt"',"1>", '"' + pth + '/log.txt"', "2>&1"]))
subprocess.call(" ".join(["C:/R/R-4.0.3/bin/x64/R.exe", "-f", '"' + pth + '/R Code.R"', "--args",
'"' + pth + '/arguments.txt"',"1>", '"' + pth + '/log.txt"', "2>&1"]))
where arguments.txt contain:
n <- 10
The problem is that when I am using R-4.0.3 the log.txt file is not generating and I need to dump a log file because it is automatically looking for it in a posterior process I have.
When I am executing in CMD (Windows) the following command:
C:/R/R-4.0.3/bin/x64/R.exe -f "<YOUR PATH>/R Code.R" --args "<YOUR PATH>/arguments.txt" 1> "<YOUR PATH>/log.txt" 2>&1'
It does work perfectly, it is only when embedded in another software.
Also, I have tried without white space in the name and calling the scripts from root folder without having to specify the path.
Any idea of why it doesn't work for R-4.* or even better, how to solve it?
Thank you!
PD: Thank you, Martin, for your tips and for making me formulate a better question
Rhelp people got this solved, thank you, Duncan Murdoch!
Solution 1:
import os
pth = "<YOUR PATH>"
os.system(" ".join(["C:/R/R-4.0.3/bin/x64/R.exe", "-f", '"' + pth + '/RCode.R"', "--args",
'"' + pth + '/arguments.txt"',"1>", '"' + pth + '/log.txt"']))
Solution 2:
import subprocess
pth = "<YOUR PATH>"
subprocess.call(" ".join(["1>", '"' + pth + '/log.txt"', "2>&1",
"C:/R/R-4.0.3/bin/x64/R.exe", "-f", '"' + pth + '/RCode.R"', "--args",
'"' + pth + '/arguments.txt"']), shell = True)
well, in one case (3.6.0) you use R.exe , in the other (4.0.3) Rscript.exe.
Both R and Rscript have existed for a long time, and they have always had slightly different behavior.
You really should not confuse them with each other (even though, on Windows, I see, they look like the same file .. they will not behave the same).
Ok, now you use R.exe for both.
Just to find out more / see more where the problem may happen, can you try
all of
using a minimal reproducible example, i.e. one we can use directly ourselves, i.e., not using "<YOUR PATH>" (nor setwd(.))
not using file names with a ' ' (space), i.e., e.g., use code.R
calling this from a "terminal"/shell instead of as python subprocess ?
Last but not least: Yes, for R 4.0.0, a completely updated toolset ("brandnew toolchain", e.g. much newer C compiler) was used to build R for windows, "Rtools 4.0" or rtools40: https://cran.r-project.org/bin/windows/Rtools/ . So changes are expected but should typically only have been to the better, not the worse ..
I am trying to execute the following command via popen but not sure why it is not working and not error message thrown too.
import os
class Pabot():
def run_pabot(self, folderOrSuiteName, tags=None):
print("pabot --testlevelsplit -r " + folderOrSuiteName + " --i " + tags + " " + folderOrSuiteName + "")
os.popen("pabot --testlevelsplit -r " + folderOrSuiteName + " --i " + tags + " " + folderOrSuiteName + "")
run = Pabot()
run.run_pabot("o/boo/test.robot", "Sequence_TC1")
From print statement:
pabot --testlevelsplit -r foo/boo/test.robot --i Sequence_TC1 foo/boo/test.robot
Right after execution, the window console gets disappeared.
Note: The same command (from print statement) works fine in commandline.
Any idea why popen does not function in this case?
I have an issue putting files to a server that contains hyphens ("-"), and I think that it may be because of how Linux is treating the file, but I am in no way sure. The script is scanning a folder for pictures/items, puts them in a list and then transferring all items to the server.
This is a part of the script:
def _transferContent(locale):
## Transferring images to server
now = datetime.datetime.now()
localImages = '/home/bcns/Pictures/upload/'
localList = os.listdir(localImages)
print("Found local items: ")
print(localList)
fname = "/tmp/backup_images_file_list"
f = open(fname, 'r')
remoteList = f.read()
remoteImageLocation = "/var/www/bcns-site/pics/photos/backup_" + locale + "-" + `now.year` + `now.month` + `now.day` + "/"
print("Server image location: " + remoteImageLocation)
## Checking local list against remote list (from the server)
for localItem in localList:
localItem_fullpath = localImages + localItem
if os.path.exists(localItem_fullpath):
if localItem in remoteList:
print("Already exists: " + localItem)
else:
put(localItem_fullpath, remoteImageLocation)
else:
print("File not found: " + localItem)
And this is the out put:
Directory created successfully
/tmp/bcns_deploy/backup_images_file_list
[<server>] download: /tmp/backup_images_file_list <- /tmp/bcns_deploy/backup_images_file_list
Warning: Local file /tmp/backup_images_file_list already exists and is being overwritten.
Found local items:
['darth-vader-mug.jpg', 'gun-dog-leash.jpg', 'think-safety-first-sign.jpg', 'hzmp.jpg', 'cy-happ-short-arms.gif', 'Hogwarts-Crest-Pumpkin.jpg']
Server image location: /var/www/bcns-site/pics/photos/backup_fujitsu-20131031/
[<server>] put: /home/bcns/Pictures/upload/darth-vader-mug.jpg -> /var/www/bcns-site/pics/photos/backup_fujitsu-20131031/
Fatal error: put() encountered an exception while uploading '/home/bcns/Pictures/upload/darth-vader-mug.jpg'
Underlying exception:
Failure
I have tried to remove the hyphons, and then the transfer works just fine.
Server runs Ubuntu 12.04 and client runs Debian 7.1 on ext3 disks.
Irritating error, but anyone out here that has a clue on what might make this error?
Dashes in command line options in Linux matter, but dashes in the middle of filenames are file.
Check file permissions -- it's possible that in transferring one file manually, the perms are set differently than if Fabric transfers.
I suggest using put() to transfer a directory at a time. This will help to make sure all the files (and permissions) are what they should be.
Example (untested):
def _transferContent(locale):
## Transferring images to server
now = datetime.datetime.now()
localImageDir = '/home/bcns/Pictures/upload/'
remoteImageDir = "/var/www/bcns-site/pics/photos/backup_" + locale + "-" + `now.year` + `now.month` + `now.day` + "/"
print("Server image location: " + remoteImageDir)
put( localImagesDir, remoteImageDir)