Develop Reference

How do I fix my code so that it is automated?

I have the below code that takes my standardized .txt file and converts it into a JSON file perfectly. The only problem is that sometimes I have over 300 files and doing this manually (i.e. changing the number at the end of the file and running the script is too much and takes too long. I want to automate this. The files as you can see reside in one folder/directory and I am placing the JSON file in a differentfolder/directory, but essentially keeping the naming convention standardized except instead of ending with .txt it ends with .json but the prefix or file names are the same and standardized. An example would be: CRAZY_CAT_FINAL1.TXT, CRAZY_CAT_FINAL2.TXT and so on and so forth all the way to file 300. How can I automate and keep the file naming convention in place, and read and output the files to different folders/directories? I have tried, but can't seem to get this to iterate. Any help would be greatly appreciated.
import glob
import time
from glob import glob
import pandas as pd
import numpy as np
import csv
import json
csvfile = open(r'C:\Users\...\...\...\Dog\CRAZY_CAT_FINAL1.txt', 'r')
jsonfile = open(r'C:\Users\...\...\...\Rat\CRAZY_CAT_FINAL1.json', 'w')
reader = csv.DictReader(csvfile)
out = json.dumps([row for row in reader])
jsonfile.write(out)
****************************************************************************
I also have this code using the python library "requests". How do I make this code so that it uploads multiple json files with a standard naming convention? The files end with a number...
import requests
#function to post to api
def postData(xactData):
url = 'http link'
headers = {
'Content-Type': 'application/json',
'Content-Length': str(len(xactData)),
'Request-Timeout': '60000'
}
return requests.post(url, headers=headers, data=xactData)
#read data
f = (r'filepath/file/file.json', 'r')
data = f.read()
print(data)
# post data
result = postData(data)
print(result)

Use f-strings?
for i in range(1,301):
csvfile = open(f'C:\Users\...\...\...\Dog\CRAZY_CAT_FINAL{i}.txt', 'r')
jsonfile = open(f'C:\Users\...\...\...\Rat\CRAZY_CAT_FINAL{i}.json', 'w')

import time
from glob import glob
import csv
import json
import os
INPATH r'C:\Users\...\...\...\Dog'
OUTPATH = r'C:\Users\...\...\...\Rat'
for csvname in glob(INPATH+'\*.txt'):
jsonname = OUTPATH + '/' + os.basename(csvname[:-3] + 'json')
reader = csv.DictReader(open(csvname,'r'))
json.dump( list(reader), open(jsonname,'w') )

How to combine YAML files in python?

I got some Kubernetes YAML files which I need to combine.
For that, I tried using Python.
The second file, sample.yaml, should be merged to the first file, source.yaml.
The source.yaml file has one section sample:, where the complete sample.yaml should be added.
I tried using the below code:
#pip install pyyaml
import yaml
def yaml_loader(filepath):
#Loads a yaml file
with open(filepath,'r')as file_descriptor:
data = yaml.load(file_descriptor)
return data
def yaml_dump(filepath,data):
with open(filepath,"w") as file_descriptor:
yaml.dump(data, file_descriptor)
if __name__ == "__main__":
file_path1 = "source"
data1 = yaml_loader(file_path1)
file_path2 = "sample.yaml"
with open(file_path2, 'r') as file2:
sample_yaml = file2.read()
data1['data']['sample'] = sample_yml
yaml_dump("temp.yml", data1)
This is creating a new file temp.yml but instead of line breaks, it is saving \n as strings:
How to fix this?

Your original YAML may have issues. If you use VS Code, format your YAML file. Click on the bottom of vscode(if using the same) [Spaces]
and select convert indentation to spaces
also, you can check if YAML module has any indentation property to be configured ,when loading the file

creating and using a preferences file in python

Brand new to stack and python; hopefully someone wiser than myself can help. I have searched up and down and can't seem to find an actual answer to this, apologies if there is an exact answer and I've missed it :( (the few that I've found are either old or don't seem to work).
Closest I've found is
Best way to retrieve variable values from a text file?
Alas, imp seems to be depreciated and tried figuring out importlib but little above my current brain to figure out how to adapt it as errors throw up left and right on me.
This is very close to what I want and could potentially work if someone can help update with new methods, alas still doesn't have how to overwrite the old variable.
= - - Scenario - - =
I would like to create a preferences file (let's call it settings.txt or settings.py: doesn't need to be cross-compatible with other languages, but some reason I'd prefer txt - any preference/standards coders can impart would be appreciated?).
\\\ settings.txt\
water_type = "Fresh"\
measurement = "Metric"\
colour = "Blue"\
location = "Bottom"\
...
I am creating a script main_menu.py which will read variables in settings.txt and write to this file if changes are 'saved'
ie.
"select water type:"
Fresh
Salt
if water_type is the same as settings.txt, do nothing,
if water_type different, overwrite the variable in the settings.txt file
Other scripts down the line will also read and write to this settings file.
I've seen:
from settings import *
Which seems to work for reading the file if I go the settings.py path but still leaves me on how do I overwrite this.
also open to any better/standard/ideas you guys can think of.
Appreciate any help on this!

Here are some suggestions that may help you:
Use a json file:
settings.json
{
"water_type": "Fresh",
"measurement": "Metric",
"colour": "Blue",
"location": "Bottom",
...
}
then in python:
import json
# Load data from the json file
with open("settings.json", "r") as f:
x = json.load(f) # x is a python dictionary in this case
# Change water_type in x
x["water_type"] = "Salt"
# Save changes
with open("settings.json", "w") as f:
json.dump(x, f, indent=4)
Use a yaml file: (edit: you will need to install pyyaml)
settings.yaml
water_type: Fresh
measurement: Metric
colour: Blue
location: Bottom
...
then in python:
import yaml
# Load data from the yaml file
with open("settings.yaml", "r") as f:
x = yaml.load(f, Loader=yaml.FullLoader) # x is a python dictionary in this case
# Change water_type in x
x["water_type"] = "Salt"
# Save changes
with open("settings.yaml", "w") as f:
yaml.dump(x, f)
Use a INI file:
settings.ini
[Preferences]
water_type=Fresh
measurement=Metric
colour=Blue
location=Bottom
...
then in python:
import configparser
# Load data from the ini file
config = configparser.ConfigParser()
config.read('settings.ini')
# Change water_type in config
config["Preferences"]["water_type"] = "Salt"
# Save changes
with open("settings.ini", "w") as f:
config.write(f)

For .py config files, it's usually static options or settings.
Ex.
# config.py
STRINGTOWRITE01 = "Hello, "
STRINGTOWRITE02 = "World!"
LINEENDING = "\n"
It would be hard to save changes made to the settings in such a format.
I'd recommend a JSON file.
Ex. settings.json
{
"MainSettings": {
"StringToWrite": "Hello, World!"
}
}
To read the settings from this file into a Python Dictionary, you can use this bit of code.
import json # Import pythons JSON library
JSON_FILE = open('settings.json','r').read() # Open the file with read permissions, then read it.
JSON_DATA = json.loads(JSON_FILE) # load the raw text from the file into a json object or dictionary
print(JSON_DATA["MainSettings"]["StringToWrite"]) # Access the 'StringToWrite' variable, just as you would with a dictionary.
To write to the settings.json file you can use this bit of code
import json # import pythons json lib
JSON_FILE = open('settings.json','r').read() # Open the file with read permissions, then read it.
JSON_DATA = json.loads(JSON_FILE) # load the data into a json object or dictionary
print(JSON_DATA["MainSettings"]["StringToWrite"]) # Print out the StringToWrite "variable"
JSON_DATA["MainSettings"]["StringToWrite"] = "Goodnight!" # Change the StringToWrite
JSON_DUMP = json.dumps(JSON_DATA) # Turn the json object or dictionary back into a regular string
JSON_FILE = open('settings.json','w') # Reopen the file, this time with read and write permissions
JSON_FILE.write(JSON_DUMP) # Update our settings file, by overwriting our previous settings
Now, I've written this so that it is as easy as possible to understand what's going on. There are better ways to do this with Python Functions.

You guys are fast! I'm away from the computer for the weekend but had to log in just to say thanks.
I'll look into these more next week when I'm back at it and have some time to give it the attention needed. A quick glance could be a bit of fun to implement and learn a bit more.
Had to answer as adding comment only is on one of your guys solutions and wanted to give a blanket thanks to all!
Cheers

Here's a python library if you choose to do it this way.
If not this is also a good resource.
Creating a preferences file example
Writing preferences to file from python file
import json
# Data to be written
dictionary ={
"name" : "sathiyajith",
"rollno" : 56,
"cgpa" : 8.6,
"phonenumber" : "9976770500"
}
# Serializing json
json_object = json.dumps(dictionary, indent = 4)
# Writing to sample.json
with open("sample.json", "w") as outfile:
outfile.write(json_object)
Reading preferences from .json file in Python
import json
# open and read file content
with open('sample.json') as json_file:
data = json.load(json_file)
# print json file
print(data)

Python script to save bookmarks into a json file

I actually wanted my bookmarks for a text classifier .It needs data in .json format .So i want to know a python script which will retrieve data from the bookmarks directory and store it in a .json file.(I am using ubuntu)

Google Chrome already saves bookmarks in a form of JSON. Your question does not define what is desired outcome so here is a simple code to access and print the whole file of your saved bookmarks on Google Chrome Windows operating system. You will need to do some adjustments to the code as it is designed to run on Windows rather than Ubuntu as I do not have access to it at this moment.
import getpass
import json
user = getpass.getuser()
loc = "C:/Users/{}/AppData/Local/Google/Chrome/User Data/Default/Bookmarks.bak".format(user)
f = open(loc, encoding="utf8")
data = json.load(f)
print(data)
Edit:
import getpass
import json
user = getpass.getuser()
loc = "C:/Users/{}/AppData/Local/Google/Chrome/User Data/Default/Bookmarks.bak".format(user)
with open(loc, encoding="utf8") as f:
data = json.load(f)
for y in range(0,100):
try:
for x in data["roots"]["bookmark_bar"]["children"][y]["children"]:
print(x["url"])
except:
pass

How to download a CSV file from the World Bank's dataset

I would like to automate the download of CSV files from the World Bank's dataset.
My problem is that the URL corresponding to a specific dataset does not lead directly to the desired CSV file but is instead a query to the World Bank's API. As an example, this is the URL to get the GDP per capita data: http://api.worldbank.org/v2/en/indicator/ny.gdp.pcap.cd?downloadformat=csv.
If you paste this URL in your browser, it will automatically start the download of the corresponding file. As a consequence, the code I usually use to collect and save CSV files in Python is not working in the present situation:
baseUrl = "http://api.worldbank.org/v2/en/indicator/ny.gdp.pcap.cd?downloadformat=csv"
remoteCSV = urllib2.urlopen("%s" %(baseUrl))
myData = csv.reader(remoteCSV)
How should I modify my code in order to download the file coming from the query to the API?

This will get the zip downloaded, open it and get you a csv object with whatever file you want.
import urllib2
import StringIO
from zipfile import ZipFile
import csv
baseUrl = "http://api.worldbank.org/v2/en/indicator/ny.gdp.pcap.cd?downloadformat=csv"
remoteCSV = urllib2.urlopen(baseUrl)
sio = StringIO.StringIO()
sio.write(remoteCSV.read())
# We create a StringIO object so that we can work on the results of the request (a string) as though it is a file.
z = ZipFile(sio, 'r')
# We now create a ZipFile object pointed to by 'z' and we can do a few things here:
print z.namelist()
# A list with the names of all the files in the zip you just downloaded
# We can use z.namelist()[1] to refer to 'ny.gdp.pcap.cd_Indicator_en_csv_v2.csv'
with z.open(z.namelist()[1]) as f:
# Opens the 2nd file in the zip
csvr = csv.reader(f)
for row in csvr:
print row
For more information see ZipFile Docs and StringIO Docs

import os
import urllib
import zipfile
from StringIO import StringIO
package = StringIO(urllib.urlopen("http://api.worldbank.org/v2/en/indicator/ny.gdp.pcap.cd?downloadformat=csv").read())
zip = zipfile.ZipFile(package, 'r')
pwd = os.path.abspath(os.curdir)
for filename in zip.namelist():
csv = os.path.join(pwd, filename)
with open(csv, 'w') as fp:
fp.write(zip.read(filename))
print filename, 'downloaded successfully'
From here you can use your approach to handle CSV files.

We have a script to automate access and data extraction for World Bank World Development Indicators like: https://data.worldbank.org/indicator/GC.DOD.TOTL.GD.ZS
The script does the following:
Downloading the metadata data
Extracting metadata and data
Converting to a Data Package
The script is python based and uses python 3.0. It has no dependencies outside of the standard library. Try it:
python scripts/get.py
python scripts/get.py https://data.worldbank.org/indicator/GC.DOD.TOTL.GD.ZS
You also can read our analysis about data from World Bank:
https://datahub.io/awesome/world-bank

Just a suggestion than a solution. You can use pd.read_csv to read any csv file directly from a URL.
import pandas as pd
data = pd.read_csv('http://url_to_the_csv_file')