I'm trying to write a json dumps string using linux bash shell echo in a text file. My problem is it removes all double quotes.
example code.
d = {"key": "value"}
"echo %s > /home/user/%s" % (simplejson.dumps(d), 'textfile'))
Output in textfile
{key: value}
It removes all double quotes so I can't load it to json because it is not a valid json anymore.
Thanks
You need to escape quotes for Bash usage:
("echo %s > /home/user/%s" % (simplejson.dumps(d), 'textfile')).replace('"', '\\"')
Since you said you're using paramiko, writing to the file directly is perfect. Edited code to reflect paramiko:
You can write to the file directly after logging onto the server, no need to pass in the bash command (which is hackish as is).
You will need two try-catch's: one to catch any error in opening the file, the other to catch any write in the file. If you'd prefer an exception to be thrown up in either of those cases, remove the try-catch.
import paramiko
*do your ssh stuff to establish an SSH session to server*
sftp = ssh.open_sftp()
try:
file = sftp.file('/home/user/textfile', 'a+')
try:
file.write(simplejson.dumps(d))
except IOError:
...*do some error handling for the write here*
except IOError:
...*do some error handling for being unable to open the file here*
else:
file.close()
sftp.close()
Related
I'm in need of some assistance. I'm attempting to perform SQLLDR from within python. The best method I found was to use subprocess.call. Using the params for another function, I duplicated it within this code.
When I run it, I get the appropriate fields, as expected.
But, the process returns a 1, which is a failure.
I have no additional information and can't locate what could be the problem.
I have verified the data.csv loads into my table from BASH, however python doesn't.
def load_raw():
DATA_FILE='data.csv'
CONTROL_FILE='raw_table.ctl'
LOG_FILE='logfile.log'
BAD_FILE='badfile.log'
DISCARD_FILE='discard.log'
connect_string = os.environ['CONNECT_STRING']
sqlldr_parms='rows=1000 readsize=50000 direct=true columnarrayrows=100 bindsize=500000 streamsize=500000 silent=(HEADER,FEEDBACK)'
parms = {}
parms['userid'] = connect_string
parms['sqlldr'] = sqlldr_parms
parms['data'] = DATA_FILE
parms['control'] = CONTROL_FILE
parms['log'] = LOG_FILE
parms['bad'] = BAD_FILE
parms['discard'] = DISCARD_FILE
cmd = "userid=%(userid)s %(sqlldr)s data=%(data)s control=%(control)s log=%(log)s bad=%(bad)s discard=%(discard)s" % parms
print "cmd is: %s" % cmd
with open('/opt/app/workload/bfapi/bin/stdout.txt', 'wb') as out:
process = call(cmd, shell=True, stdout=out, stderr=out)
print process
cmd is: sqlldr userid=usr/pass rows=1000 readsize=50000 direct=true columnarrayrows=100 bindsize=500000
streamsize=500000 silent=(HEADER,FEEDBACK) data=data.csv control=raw_table.ctl
log=logfile.log bad=badfile.log discard=discard.log
process returns 1
The log files for log, bad and discard are not created
stdout.txt contains
/bin/sh: -c: line 0: syntax error near unexpected token ('
/bin/sh: -c: line 0:sqlldr userid=usr/pass rows=1000 readsize=50000 direct=true columnarrayrows=100
bindsize=500000 streamsize=500000 silent=(HEADER,FEEDBACK) data=data.csv control=raw_table.ctl
log=logfile.log bad=badfile.log discard=discard.log'
data.csv contains
id~name~createdby~createddate~modifiedby~modifieddate
6~mark~margaret~"19-OCT-16 01.03.23.966000 PM"~kyle~"21-OCT-16 03.11.22.256000 PM"
8~jill~margaret~"27-AUG-16 12.10.12.214000 PM"~kyle~"21-OCT-16 04.16.01.171000 PM"
raw_table.ctl
OPTIONS ( SKIP=1)
LOAD DATA
CHARACTERSET UTF8
INTO TABLE RAW_TABLE
FIELDS TERMINATED BY '~' OPTIONALLY ENCLOSED BY '"' TRAILING NULLCOLS
(ID,
NAME,
CREATED_BY,
CREATED_DATETIME TIMESTAMP,
MODIFIED_BY,
MODIFIED_DATETIME TIMESTAMP)
The error was caused by the silent param. Wrapping it in single quotes allowed the code to work, as here: silent='(HEADER,FEEDBACK)'
I'm working with Django and I'd created two database. Everything seems to work fine, but then I had to edit one of the two and add a column.. From that moment the db wouldn't work anymore, so I exported in a text file the first database and thinking "now I recreate the two db and run a python script to refill the first one". The problem is that whene I try to run the script I get errors, because I can't run the command like bash using os.system, and I don't really know any other way... So, here's my code:
import os
def func ():
try:
FILE=open ("./languagedb.txt", "r")
except IOError:
print 'Can\'t open db file'
exit (1)
for line in FILE:
if (line.startswith('INSERT')):
values=line[43:-1]
language=values[1:3]
values=values[6:]
field=""
fieldBool=True
i=0
while fieldBool:
try:
c=values[i]
except:
print ''
if c != '\'':
field=field+str(c)
i=i+1
else:
fieldBool=False
values=values [(i+3):]
text=""
textBool=True
i=0
while textBool:
try:
c=values[i]
except:
print ''
if c != '\'':
text=text+str(c)
i=i+1
else:
textBool=False
comand="Language.objects.create(language=\""+language+"\", text=\""+text+"\", campo=\""+field+"\")"
os.system(comand)
This is the way I call the shell:
python manage.py shell
and the commands I give it:
import django
from languageMods.models import *
import mymigration #The name fo the file containing the above code
mymigration.func()
And I get the following error, for example
sh: -c: line 0: syntax error near unexpected token `language="en",'
Which is shell's error.
Does someone know how to execute a command from a python script in a python shell?
If you start your script the way you describe it you can just call the django DB API directly in your code:
Language.objects.create(language=language, text=text, campo=field)
I have created this below script and it works fine. But the output is not friendly (see below). I want the first line to display only the hostname and IP and remove (,'[], please suggest
('testhostname', [], ['10.10.10.10'])
cannot resolve hostname: 10.10.10.11
import socket
pfile = open ('C:\\Python27\\scripts\\test.txt')
while True:
IP = pfile.readline()
if not IP:
break
try:
host = socket.gethostbyaddr(IP.rstrip())
print host
except socket.herror, err:
print "cannot resolve hostname: ", IP
pfile.close()
Rather than printing all of the host tuple that is returned by gethostbyaddr, I suggest unpacking into separate variables that you can then print as you see fit:
hostname, alias_list, ip_addr_list = gethostbyaddr(IP.rstrip())
print hostname, ip_addr_list # or ip_addr_list[0] if you only want the one IP
If you want more control over the formatting, I suggest using the str.format method:
print "hostname: {}, IP(s): {}".format(hostname, ", ".join(ip_addr_list))
Also, a few other code suggestions (not directly related to your main question):
Use a with statement rather than manually opening and closing your file.
Iterate on the file object directly (with for IP in pfile:), rather than using while True: and calling pfile.readline() each time through.
Use the syntax except socek.herror as err rather than the older form with commas (which is deprecated in Python 2 and no longer exists in Python 3).
Is there a way to execute a sql script file using cx_oracle in python.
I need to execute my create table scripts in sql files.
PEP-249, which cx_oracle tries to be compliant with, doesn't really have a method like that.
However, the process should be pretty straight forward. Pull the contents of the file into a string, split it on the ";" character, and then call .execute on each member of the resulting array. I'm assuming that the ";" character is only used to delimit the oracle SQL statements within the file.
f = open('tabledefinition.sql')
full_sql = f.read()
sql_commands = full_sql.split(';')
for sql_command in sql_commands:
curs.execute(sql_command)
Another option is to use SQL*Plus (Oracle's command line tool) to run the script. You can call this from Python using the subprocess module - there's a good walkthrough here: http://moizmuhammad.wordpress.com/2012/01/31/run-oracle-commands-from-python-via-sql-plus/.
For a script like tables.sql (note the deliberate error):
CREATE TABLE foo ( x INT );
CREATE TABLER bar ( y INT );
You can use a function like the following:
from subprocess import Popen, PIPE
def run_sql_script(connstr, filename):
sqlplus = Popen(['sqlplus','-S', connstr], stdin=PIPE, stdout=PIPE, stderr=PIPE)
sqlplus.stdin.write('#'+filename)
return sqlplus.communicate()
connstr is the same connection string used for cx_Oracle. filename is the full path to the script (e.g. 'C:\temp\tables.sql'). The function opens a SQLPlus session (with '-S' to silence its welcome message), then queues "#filename" to send to it - this will tell SQLPlus to run the script.
sqlplus.communicate sends the command to stdin, waits for the SQL*Plus session to terminate, then returns (stdout, stderr) as a tuple. Calling this function with tables.sql above will give the following output:
>>> output, error = run_sql_script(connstr, r'C:\temp\tables.sql')
>>> print output
Table created.
CREATE TABLER bar (
*
ERROR at line 1:
ORA-00901: invalid CREATE command
>>> print error
This will take a little parsing, depending on what you want to return to the rest of your program - you could show the whole output to the user if it's interactive, or scan for the word "ERROR" if you just want to check whether it ran OK.
Into cx_Oracle library you can find a method used by tests to load scripts: run_sql_script
I modified this method in my project like this:
def run_sql_script(self, connection, script_path):
cursor = connection.cursor()
statement_parts = []
for line in open(script_path):
if line.strip() == "/":
statement = "".join(statement_parts).strip()
if not statement.upper().startswith('CREATE PACKAGE'):
statement = statement[:-1]
if statement:
try:
cursor.execute(statement)
except Exception as e:
print("Failed to execute SQL:", statement)
print("Error:", str(e))
statement_parts = []
else:
statement_parts.append(line)
The commands into script file must be separated by "/".
I hope it can be of help.
Is there any way to start up the Python interpreter from within a script , in a manner similar to just using python -i so that the objects/namespace, etc. from the current script are retained? The reason for not using python -i is that the script initializes a connection to an XML-RPC server, and I need to be able to stop the entire program if there's an error. I can't loop until there's valid input because apparently, I can't do something like this:
#!/usr/bin/python -i
# -*- coding: utf-8 -*-
import xmlrpclib
# Create an object to represent our server.
server_url = str(raw_input("Server: "))
while not server = xmlrpclib.Server(server_url):
print 'Unable to connect to server. Please try again'
else:
print 'Xmlrpclib.Server object `__main__.server\' of URL `', server_url, "' created"
break
# Python interpreter starts...
because:
% chmod u+x ./rpcclient.py
% ./rpclient.py
Traceback (most recent call last):
File "./rpcclient.py", line 8
while not server = xmlrpclib.Server(server_url):
^
SyntaxError: invalid syntax
>>>
Unfortunately, python -i starts the interpreter just after it prints out the traceback, so I somehow have to call the interactive interpreter - replacing the execution of the script so it retains the server connection - from within the script
Have you tried reading the error message? :)
= is assignment, you want the comparison operator == instead.
Well, I finally got it to work.
Basically, I put the entire try/except/else clause in a while True: loop, with the else suite being a break statement and the end of the except suite being a continue statement. The result is that it now continually loops if the user puts in an address that doesn't have a fully compliant XML-RPC2 server listening. Here's how it turned out:
#!/usr/bin/python -i
# -*- coding: utf-8 -*-
import xmlrpclib, socket
from sys import exit
# Create an object to represent our server.
#server = xmlrpclib.Server(server_url) and print 'Xmlrpclib.Server object `__main__.server\' of URL `', server_url, "' created"
server_url = str(raw_input("Server: "))
server = xmlrpclib.ServerProxy(server_url)
while True:
try:
server.system.listMethods()
except xmlrpclib.ProtocolError, socket.error:
print 'Unable to connect to server. Please try again'
server_url = str(raw_input("Server: "))
server = xmlrpclib.ServerProxy(server_url)
continue
except EOFError:
exit(1)
else:
break
print 'Xmlrpclib.Server object `__main__.server\' of URL `', server_url, "' created"
# Python interpreter starts...
Thank you very much!
...and I have to wait another day to accept this...