Develop Reference

How to get an unknown substring between two known substrings, within a giant string/file

I'm trying to get all the substrings under a "customLabel" tag, for example "Month" inside of ...,"customLabel":"Month"},"schema":"metric...
Unusual issue: this is a 1071552 characters long ndjson file, of a single line ("for line in file:" is pointless since there's only one).
The best I found was that:
How to find a substring of text with a known starting point but unknown ending point in python
but if I use this, the result obviously doesn't stop (at Month) and keeps writing the whole remaining of the file, same as if using partition()[2].
Just know that Month is only an example, customLabel has about 300 variants and they are not listed (I'm actually doing this to list them...)
To give some details here's my script so far:
with open("file.ndjson","rt", encoding='utf-8') as ndjson:
filedata = ndjson.read()
x="customLabel"
count=filedata.count(x)
for i in range (count):
if filedata.find(x)>0:
print("Found "+str(i+1))
So right now it properly tells me how many occurences of customLabel there are, I'd like to get the substring that comes after customLabel":" instead (Month in the example) to put them all in a list, to locate them way more easily and enable the use of replace() for traductions later on.
I'd guess regex are the solution but I'm pretty new to that, so I'll post that question by the time I learn about them...

If you want to search for all (even nested) customLabel values like this:
{"customLabel":"Month" , "otherJson" : {"customLabel" : 23525235}}
you can use RegEx patterns with the re module
import re
label_values = []
regex_pattern = r"\"customLabel\"[ ]?:[ ]?([1-9a-zA-z\"]+)"
with open("file.ndjson", "rt", encoding="utf-8") as ndjson:
for line in ndjson:
values = re.findall(regex_pattern, line)
label_values.extend(values)
print(label_values) # ['"Month"', '23525235']
# If you don't want the items to have quotations
label_values = [i.replace('"', "") for i in label_values]
print(label_values) # ['Month', '23525235']
Note: If you're only talking about ndjson files and not nested searching, then it'd be better to use the json module to parse the lines and then easily get the value of your specific key which is customLabel.
import json
label = "customLabel"
label_values = []
with open("file.ndjson", "rt", encoding="utf-8") as ndjson:
for line in ndjson:
line_json = json.loads(line)
if line_json.get(label) is not None:
label_values.append(line_json.get(label))
print(label_values) # ['Month']

Vigenere Cipher in Python bug

I'm trying to implement Vigenere's Cipher. I want to be able to obfuscate every single character in a file, not just alphabetic characters.
I think I'm missing something with the different types of encoding. I have made some test cases and some characters are getting replaced badly in the final result.
This is one test case:
,.-´`1234678abcde^*{}"¿?!"·$%&/\º
end
And this is the result I'm getting:
).-4`1234678abcde^*{}"??!"7$%&/:
end
As you can see, ',' is being replaced badly with ')' as well as some other characters.
My guess is that the others (for example, '¿' being replaced with '?') come from the original character not being in the range of [0, 127], so its normal those are changed. But I don't understand why ',' is failing.
My intent is to obfuscate CSV files, so the ',' problem is the one I'm mainly concerned about.
In the code below, I'm using modulus 128, but I'm not sure if that's correct. To execute it, put a file named "OriginalFile.txt" in the same folder with the content to cipher and run the script. Two files will be generated, Ciphered.txt and Deciphered.txt.
"""
Attempt to implement Vigenere cipher in Python.
"""
import os
key = "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa"
fileOriginal = "OriginalFile.txt"
fileCiphered = "Ciphered.txt"
fileDeciphered = "Deciphered.txt"
# CIPHER PHASE
if os.path.isfile(fileCiphered):
os.remove(fileCiphered)
keyToUse = 0
with open(fileOriginal, "r") as original:
with open(fileCiphered, "a") as ciphered:
while True:
c = original.read(1) # read char
if not c:
break
k = key[keyToUse]
protected = chr((ord(c) + ord(k))%128)
ciphered.write(protected)
keyToUse = (keyToUse + 1)%len(key)
print("Cipher successful")
# DECIPHER PHASE
if os.path.isfile(fileDeciphered):
os.remove(fileDeciphered)
keyToUse = 0
with open(fileCiphered, "r") as ciphered:
with open(fileDeciphered, "a") as deciphered:
while True:
c = ciphered.read(1) # read char
if not c:
break
k = key[keyToUse]
unprotected = chr((128 + ord(c) - ord(k))%128) # +128 so that we don't get into negative numbers
deciphered.write(unprotected)
keyToUse = (keyToUse + 1)%len(key)
print("Decipher successful")

Assumption: you're trying to produce a new, valid CSV with the contents of cells enciphered via Vigenere, not to encipher the whole file.
In that case, you should check out the csv module, which will handle properly reading and writing CSV files for you (including cells that contain commas in the value, which might happen after you encipher a cell's contents, as you see). Very briefly, you can do something like:
with open("...", "r") as fpin, open("...", "w") as fpout:
reader = csv.reader(fpin)
writer = csv.writer(fpout)
for row in reader:
# row will be a list of strings, one per column in the row
ciphered = [encipher(cell) for cell in row]
writer.writerow(ciphered)
When using the csv module you should be aware of the notion of "dialects" -- ways that different programs (usually spreadsheet-like things, think Excel) handle CSV data. csv.reader() usually does a fine job of inferring the dialect you have in the input file, but you might need to tell csv.writer() what dialect you want for the output file. You can get the list of built-in dialects with csv.list_dialects() or you can make your own by creating a custom Dialect object.

eliminate text after certain character in python pipeline- with slice?

This is a short script I've written to refine and validate a large dataset that I have.
# The purpose of this script is the refinement of the job data attained from the
# JSI as it is rendered by the `csv generator` contributed by Luis for purposes
# of presentation on the dashboard map.
import csv
# The number of columns
num_headers = 9
# Remove invalid characters from records
def url_escaper(data):
for line in data:
yield line.replace('&','&')
# Be sure to configure input & output files
with open("adzuna_input_THRESHOLD.csv", 'r') as file_in, open("adzuna_output_GO.csv", 'w') as file_out:
csv_in = csv.reader( url_escaper( file_in ) )
csv_out = csv.writer(file_out)
# Get rid of rows that have the wrong number of columns
# and rows that have only whitespace for a columnar value
for i, row in enumerate(csv_in, start=1):
if not [e for e in row if not e.strip()]:
if len(row) == num_headers:
csv_out.writerow(row)
else:
print "line %d is malformed" % i
I have one field that is structured like so:
finance|statistics|lisp
I've seen ways to do this using other utilities like R, but I want to ideally achieve the same effect within the scope of this python code.
Maybe I can iterate over all the characters of all the columnar values, perhaps as a list, and if I see a | I can dispose of the | and all the text that follows it within the scope of the column value.
I think surely it can be achieved with slices as they do here but I don't quite understand how the indices with slices work- and I can't see how I could include this process harmoniously within the cascade of the current script pipeline.
With regex I guess it's something like this
(?:|)(.*)

Why not use string's split method?
In[4]: 'finance|statistics|lisp'.split('|')[0]
Out[4]: 'finance'
It does not fail with exception when you do not have separator character in the string too:
In[5]: 'finance/statistics/lisp'.split('|')[0]
Out[5]: 'finance/statistics/lisp'

Creating a table which has sentences from a paragraph each on a row with Python

I have an abstract which I've split to sentences in Python. I want to write to 2 tables. One which has the following columns: abstract id (which is the file number that I extracted from my document), sentence id (automatically generated) and each sentence of this abstract on a row.
I would want a table that looks like this
abstractID SentenceID Sentence
a9001755 0000001 Myxococcus xanthus development is regulated by(1st sentence)
a9001755 0000002 The C signal appears to be the polypeptide product (2nd sentence)
and another table NSFClasses having abstractID and nsfOrg.
How to write sentences (each on a row) to table and assign sentenceId as shown above?
This is my code:
import glob;
import re;
import json
org = "NSF Org";
fileNo = "File";
AbstractString = "Abstract";
abstractFlag = False;
abstractContent = []
path = 'awardsFile/awd_1990_00/*.txt';
files = glob.glob(path);
for name in files:
fileA = open(name,'r');
for line in fileA:
if line.find(fileNo)!= -1:
file = line[14:]
if line.find(org) != -1:
nsfOrg = line[14:].split()
print file
print nsfOrg
fileA = open(name,'r')
content = fileA.read().split(':')
abstract = content[len(content)-1]
abstract = abstract.replace('\n','')
abstract = abstract.split();
abstract = ' '.join(abstract)
sentences = abstract.split('.')
print sentences
key = str(len(sentences))
print "Sentences--- "

As others have pointed out, it's very difficult to follow your code. I think this code will do what you want, based on your expected output and what we can see. I could be way off, though, since we can't see the file you are working with. I'm especially troubled by one part of your code that I can't see enough to refactor, but feels obviously wrong. It's marked below.
import glob
for filename in glob.glob('awardsFile/awd_1990_00/*.txt'):
fh = open(filename, 'r')
abstract = fh.read().split(':')[-1]
fh.seek(0) # reset file pointer
# See comments below
for line in fh:
if line.find('File') != -1:
absID = line[14:]
print absID
if line.find('NSF Org') != -1:
print line[14:].split()
# End see comments
fh.close()
concat_abstract = ''.join(abstract.replace('\n', '').split())
for s_id, sentence in enumerate(concat_abstract.split('.')):
# Adjust numeric width arguments to prettify table
print absID.ljust(15),
print '{:06d}'.format(s_id).ljust(15),
print sentence
In that section marked, you are searching for the last occurrence of the strings 'File' and 'NSF Org' in the file (whether you mean to or not because the loop will keep overwriting your variables as long as they occur), then doing something with the 15th character onward of that line. Without seeing the file, it is impossible to say how to do it, but I can tell you there is a better way. It probably involves searching through the whole file as one string (or at least the first part of it if this is in its header) rather than looping over it.
Also, notice how I condensed your code. You store a lot of things in variables that you aren't using at all, and collecting a lot of cruft that spreads the state around. To understand what line N does, I have to keep glancing ahead at line N+5 and back over lines N-34 to N-17 to inspect variables. This creates a lot of action at a distance, which for reasons cited is best to avoid. In the smaller version, you can see how I substituted in string literals in places where they are only used once and called print statements immediately instead of storing the results for later. The results are usually more concise and easily understood.

Transform hard coded initialization values to CSV

Using Python I wanted to extract data rows shown below to a csv file from a bunch of javascript files which contain hardcoded data as shown below:
....html code....
hotels[0] = new hotelData();
hotels[0].hotelName = "MANHATTAN";
hotels[0].hotelPhone = "";
hotels[0].hotelSalesPhone = "";
hotels[0].hotelPhone = 'Phone: 888-350-6432';
hotels[0].hotelStreet = "787 11TH AVENUE";
hotels[0].hotelCity = "NEW YORK";
hotels[0].hotelState = "NY";
hotels[0].hotelZip = "10019";
hotels[0].hotelId = "51543";
hotels[0].hotelLat = "40.7686";;
hotels[0].hotelLong = "-73.992645";;
hotels[1] = new hotelData();
hotels[1].hotelName = "KOEPPEL";
hotels[1].hotelPhone = "";
hotels[1].hotelSalesPhone = "";
hotels[1].hotelPhone = 'Phone: 718-721-9100';
hotels[1].hotelStreet = "57-01 NORTHERN BLVD.";
hotels[1].hotelCity = "WOODSIDE";
hotels[1].hotelState = "NY";
hotels[1].hotelZip = "11377";
hotels[1].hotelId = "51582";
hotels[1].hotelLat = "40.75362";;
hotels[1].hotelLong = "-73.90366";;
var mykey = "AlvQ9gNhp7oNuvjhkalD4OWVs_9LvGHg0ZLG9cWwRdAUbsy-ZIW1N9uVSU0V4X-8";
var map = null;
var pins = null;
var i = null;
var boxes = new Array();
var currentBox = null;
var mapOptions = {
credentials: mykey,
enableSearchLogo: false,
showMapTypeSelector: false,
enableClickableLogo: false
}
.....html code .....
Hence the required csv output would be like rows of the above data:
MANHATTAN,,,Phone: 888-350-6432 ...
KOEPPEL,,,Phone: 718-721-9100 ...
Should I use code generation tool to directly parse the above statements to get the data ? Which is the most efficient Python method to transform such data contained in thousands of Javascript files into csv tabular format?
Update:
Ideally I would like the solution to parse the JavaScript statements as Python objects and then store it to CSV to gain maximum independence from ordering and formatting of the input script code

I'd recommend using a regular expression to pick out all "hotel[#]. ..." lines, and then add all of the results to a dictionary. Then, with the dictionary, output to a CSV file. The following should work:
import re
import csv
src_text = your_javascript_text
p = re.compile(r'hotels\[(?P<hotelid>\d+)\].(?P<attr>\w+) = ("|\')(?P<attr_val>.*?)("|\');', re.DOTALL)
hotels = {}
fieldnames = []
for result in [m.groupdict() for m in p.finditer(src_text)]:
if int(result['hotelid']) not in hotels:
hotels[int(result['hotelid'])] = {}
if result['attr'] not in fieldnames:
fieldnames.append(result['attr'])
hotels[int(result['hotelid'])][result['attr']] = result['attr_val']
output = open('hotels.csv','wb')
csv_writer = csv.DictWriter(output, delimiter=',', fieldnames=fieldnames, quoting=csv.QUOTE_ALL)
csv_writer.writerow(dict((f,f) for f in fieldnames))
for hotel in hotels.items():
csv_writer.writerow(hotel[1])
You now have a dictionary of Hotels w/ attributes, grouped by the ID in the Javascript, as well as the output file "hotels.csv" (with header row & proper escaping). I did do things like named groups which really aren't necessary, but find it to be more self-commenting.
It should be noted that if the same group is provided in the Javascript twice, like hotelPhone, the last is the only one stored.
When dealing with this type of problem, it falls to you and your judgment how much tolerance and sanitation you need. You may need to modify the regular expression to handle examples not int he small sample provided (ie. change in capture groups, restrict matches to those at the start of a line, etc.); or escape newline characters, like those in the phone number); or strip out certain text (eg. "Phone: " in the phone numbers). There's no real way for us to know this, so keep that in mind.
Cheers!

If this is something you will have to do routinely and you want to make the process fully automatic I think the easiest would be just to parse the files using Python and then write to csv using the csv Python module.
Your code could look somewhat like this:
with open("datafile.txt") as f:
hotel_data = []
for line in f:
# Let's make sure the line not empty
if line:
if "new hotelData();" in line:
if hotel_data:
write_to_csv(hotel_data)
hotel_data = []
else:
# Data, still has ending quote and semi colon
data = line.split("= ")[1]
# Remove ending quote and semi colon
data = data[:-2]
hotel_data.append(data)
def write_to_csv(hotel_data):
with open('hotels.csv', 'wb') as csvfile:
spamwriter = csv.writer(csvfile, delimiter=',',
quotechar='""', quoting=csv.QUOTE_MINIMAL)
spamwriter.writerow(hotel_data)
Beware that I have not tested this code, it is just meant to help you and point you in the right direction, it is not the complete solution.

If each hotel has every field declared in your files (i.e. if all of the hotels have the same amount of lines, even if some of them are empty), you may try to use a simple regular expression to extract every value surrounded by quotes ("xxx"), and then group them by number (for example, group every 5 fields into a single line and then add a line break).
A simple regex that would work would be ["'][^"']*["'] (EDIT: this is because I see that some fileds (i.e. Phone) use single quotes and the rest use quotes).
To make the search, use findall:
compPattern = re.compile(pattern)
results = compPattern.findall(compPattern)