Develop Reference

Python JSON import to MongoDB from ZIP files

I have a script that gets all of the .zip files from a folder, then one by one, opens the zip file, loads the content of the JSON file inside and imports this to MongoDB.
The error I am getting is the JSON object must be str, bytes or bytearray, not 'TextIOWrapper'
The code is:
import json
import logging
import logging.handlers
import os
from logging.config import fileConfig
from pymongo import MongoClient
def import_json():
try:
client = MongoClient('5.57.62.97', 27017)
db = client['vuln_sets']
coll = db['vulnerabilities']
basepath = os.path.dirname(__file__)
filepath = os.path.abspath(os.path.join(basepath, ".."))
archive_filepath = filepath + '/vuln_files/'
filedir = os.chdir(archive_filepath)
for item in os.listdir(filedir):
if item.endswith('.json'):
file_name = os.path.abspath(item)
fp = open(file_name, 'r')
json_data = json.loads(fp)
for vuln in json_data:
print(vuln)
coll.insert(vuln)
os.remove(file_name)
except Exception as e:
logging.exception(e)
I can get this working to use a single file but not multiple, i.e. to do one file I wrote:
from zipfile import ZipFile
import json
import pymongo
archive = ZipFile("vulners_collections/cve.zip")
archived_file = archive.open(archive.namelist()[0])
archive_content = archived_file.read()
archived_file.close()
connection = pymongo.MongoClient("mongodb://localhost")
db=connection.vulnerability
vuln1 = db.vulnerability_collection
vulners_objects = json.loads(archive_content)
for item in vulners_objects:
vuln1.insert(item)

From my comment above:
I have no experience with glob, but from skimming the doc I get the impression your archive_files is a simple list of file-paths as strings, correct? You can not perform actions like .open on string (thus your error), so try changing your code to this:
...
archive_filepath = filepath + '/vuln_files/'
archive_files = glob.glob(archive_filepath + "/*.zip")
for file in archive_files:
with open(file, "r") as currentFile:
file_content = currentFile.read()
vuln_content = json.loads(file_content)
for item in vuln_content:
coll.insert(item)
...
file is NOT a file object or anything but just a simple string. So you cant perform methods on it that are not supported by string.

You are redefining your iterator by setting it to the result of the namelist method. You need a for loop within the for to go through the contents of the zip file and of course a new iterator variable.

Isn't file.close wrong and the correct call is file.close().
U can use json.load() to load file directly, instead of json.loads()
fp = open(file_name, 'r')
json_data = json.load(fp)
fp.close()

Read Value from Config File Python

I have a file .env file contain 5 lines
DB_HOST=http://localhost/
DB_DATABASE=bheng-local
DB_USERNAME=root
DB_PASSWORD=1234567890
UNIX_SOCKET=/tmp/mysql.sock
I want to write python to grab the value of DB_DATABASE
I want this bheng-local
I would have use
import linecache
print linecache.getline('.env', 2)
But some people might change the order of the cofigs, that's why linecache is not my option.
I am not sure how to check for only some strings match but all the entire line, and grab the value after the =.
I'm kind of stuck here :
file = open('.env', "r")
read = file.read()
my_line = ""
for line in read.splitlines():
if line == "DB_DATABASE=":
my_line = line
break
print my_line
Can someone please give me a little push here ?

Have a look at the config parser:
https://docs.python.org/3/library/configparser.html
It's more elegant than a self-made solution

Modify your .env to
[DB]
DB_HOST=http://localhost/
DB_DATABASE=bheng-local
DB_USERNAME=root
DB_PASSWORD=1234567890
UNIX_SOCKET=/tmp/mysql.sock
Python code
#!/usr/local/bin/python
import configparser
config = configparser.ConfigParser()
config.read('test.env')
print config.get('DB','DB_DATABASE')
Output:
bheng-local
Read https://docs.python.org/3/library/configparser.html

This should work for you
#!/usr/local/bin/python
file = open('test.env', "r")
read = file.read()
for line in read.splitlines():
if 'DB_DATABASE=' in line:
print line.split('=',1)[1]

#!/usr/local/bin/python
from configobj import ConfigObj
conf = ConfigObj('test.env')
print conf['DB_DATABASE']

from os import environ, path
from dotenv import load_dotenv
basedir = path.abspath(path.dirname(__file__))
load_dotenv(path.join(basedir, '.env'))
DB_DATABASE = environ.get('DB_DATABASE')
print(DB_DATABASE)
This could be another option

How do I load data from a text file in python?

I'm basically writing text to a file for example
data = ("save.data", "a+")
data.write(u"name = 'zrman'")
I wan't to be able to load that file and allow me to do this in python
print name
Any help would be great
-Thx

Use ConfigParser Python module. It's exactely what you're looking for.
Write :
import ConfigParser
config = ConfigParser.RawConfigParser()
config.set('main', 'name', 'zrman')
with open('conf.ini', 'wb') as configfile:
config.write(configfile)
Read :
from ConfigParser import ConfigParser
config = ConfigParser()
config.read('conf.ini')
print config.sections()
# ['main']
print config.items('main')
# [('name', 'zrman')]

Take a look at the docs here, this will walk you through the process of reading and writing files in python.

You could read in the lines and then exec the code:
f = open('workfile', 'w')
for line in f:
exec(line)
print name

python search for a string and append to it using regular expression

I need to search for a certain parameter known as jvm_args in a configuration file known as config.ini
**contents of config.ini:
first_paramter=some_value1
second_parameter=some_value2
jvm_args=some_value3**
I need to know how to find this parameter in my file and append something to its value, (i.e append a string to the string some_value3).

If you "just" want to find keys and values in an ini file, I think the configparser module is a better bet than using regexps. The configparser asserts that the file has "sections", though.
Documentation for configparser is here: http://docs.python.org/library/configparser.html - useful examples at the bottom. The configparser can also be used for setting values and writing out a new .ini-file.
Input file:
$ cat /tmp/foo.ini
[some_section]
first_paramter = some_value1
second_parameter = some_value2
jvm_args = some_value3
Code:
#!/usr/bin/python3
import configparser
config = configparser.ConfigParser()
config.read("/tmp/foo.ini")
jvm_args = config.get('some_section', 'jvm_args')
print("jvm_args was: %s" % jvm_args)
config.set('some_section', 'jvm_args', jvm_args + ' some_value4')
with open("/tmp/foo.ini", "w") as fp:
config.write(fp)
Output file:
$ cat /tmp/foo.ini
[some_section]
first_paramter = some_value1
second_parameter = some_value2
jvm_args = some_value3 some_value4

You can use re.sub
import re
import os
file = open('config.ini')
new_file = open('new_config.ini', 'w')
for line in file:
new_file.write(re.sub(r'(jvm_args)\s*=\s*(\w+)', r'\1=\2hello', line))
file.close()
new_file.close()
os.remove('config.ini')
os.rename('new_config.ini', 'config.ini')
also check ConfigParser

As both avasal and tobixen have suggested, you can use the python ConfigParser module to do this. For example, I took this "config.ini" file:
[section]
framter = some_value1
second_parameter = some_value2
jvm_args = some_value3**
and ran this python script:
import ConfigParser
p = ConfigParser.ConfigParser()
p.read("config.ini")
p.set("section", "jvm_args", p.get("section", "jvm_args") + "stuff")
with open("config.ini", "w") as f:
p.write(f)
and the contents of the "config.ini" file after running the script was:
[section]
framter = some_value1
second_parameter = some_value2
jvm_args = some_value3**stuff

without regex you can try:
with open('data1.txt','r') as f:
x,replace=f.read(),'new_entry'
ind=x.index('jvm_args=')+len('jvm_args=')
end=x.find('\n',ind) if x.find('\n',ind)!=-1 else x.rfind('',ind)
x=x.replace(x[ind:end],replace)
with open('data1.txt','w') as f:
f.write(x)

how to open up a child process to tail the files present in a config file using python on a mac machine?

I am trying to tail the files in a config file using python on mac
I am able to get the values from the config file but not able to open up child processes for the same
a sample config file has
[section1]
host_prefix = true
timestamp_prefix = true
[section2]
host = localhost
port = 1463
pids = /var/run/harvester
[files]
apache.access = /var/log/apache2/access.log
apache.errors = /var/log/apache2/errors.log
mail = /var/log/mail.log
mysql.log = /var/log/mysql.log
I am opening up the config file and trying to get the filepaths and I need to tail them in new child processes in separate terminals
#! /bin/env python
import StringIO
import os
import re
from multiprocessing import Process
COMMENT_CHAR = '#'
OPTION_CHAR = '='
def parse_config(filename):
options = {}
f = open(filename)
for line in f:
if COMMENT_CHAR in line:
line, comment = line.split(COMMENT_CHAR, 1)
if OPTION_CHAR in line:
option, value = line.split(OPTION_CHAR, 1)
option = option.strip()
value = value.strip()
options[option] = value
f.close()
return options
try:
f = open("/etc/harvest.conf", 'r')
print 'found'
options = parse_config('/etc/harvest.conf')
print options.values()
os.system('tail -f options.values')
except:
try:
f = open("/usr/local/etc/harvest.conf", 'r')
print 'found'
options = parse_config('/usr/local/etc/harvest.conf')
print options.values()
os.system('tail -f options.values')
except IOError:
print 'cannot find file'
the above code gives me all the values from the config file that includes 'localhost','1463'
but I want only the paths from the file and need to tail them in separate child processes

Try ConfigParser. It can work with INI files.

use os.path.exists to check if a file exists
use ConfigParser to parse an ini-type config file