Develop Reference

I am looking to create an API endpoint route that returns txt in a json format -Python

I'm new to developing and my question(s) involves creating an API endpoint in our route. The api will be used for a POST from a Vuetify UI. Data will come from our MongoDB. We will be getting a .txt file for our shell script but it will have to POST as a JSON. I think these are the steps for converting the text file:
1)create a list for the lines of the .txt
2)add each line to the list
3) join the list elements into a string
4)create a dictionary with the file/file content and convert it to JSON
This is my current code for the steps:
import json
something.txt: an example of the shell script ###
f = open("something.txt")
create a list to put the lines of the file in
file_output = []
add each line of the file to the list
for line in f:
file_output.append(line)
mashes all of the list elements together into one string
fileoutput2 = ''.join(file_output)
print(fileoutput2)
create a dict with file and file content and then convert to JSON
json_object = {"file": fileoutput2}
json_response = json.dumps(json_object)
print(json_response)
{"file": "Hello\n\nSomething\n\nGoodbye"}
I have the following code for my baseline below that I execute on my button press in the UI
#bp_customer.route('/install-setup/<string:customer_id>', methods=['POST'])
def install_setup(customer_id):
cust = Customer()
customer = cust.get_customer(customer_id)
### example of a series of lines with newline character between them.
script_string = "Beginning\nof\nscript\n"
json_object = {"file": script_string}
json_response = json.dumps(json_object)
get the install shell script content
replace the values (somebody has already done this)
attempt to return the below example json_response
return make_response(jsonify(json_response), 200)
my current Vuetify button press code is here: so I just have to ammend it to a POST and the new route once this is established
onClickScript() {
console.log("clicked");
axios
.get("https://sword-gc-eadsusl5rq-uc.a.run.app/install-setup/")
.then((resp) => {
console.log("resp: ", resp.data);
this.scriptData = resp.data;
});
},
I'm having a hard time combining these 2 concepts in the correct way. Any input as to whether I'm on the right path? Insight from anyone who's much more experienced than me?

You're on the right path, but needlessly complicating things a bit. For example, the first bit could be just:
import json
with open("something.txt") as f:
json_response = json.dumps({'file': f.read()})
print(json_response)
And since you're looking to pass everything through jsonify anyway, even this would suffice:
with open("something.txt") as f:
data = {'file': f.read()}
Where you can pass data directly through jsonify. The rest of it isn't sufficiently complete to offer any concrete comments, but the basic idea is OK.
If you have a working whole, you could go to https://codereview.stackexchange.com/ to ask for some reviews, you should limit questions on StackOverflow to actual questions about getting something to work.

Parsing tweets in json format to find tweeter users

I am reading a tweeter feed in json format to read the number of users.
Some lines in the input file might not be tweets, but messages that the Twitter server sent to the developer (such as limit notices). I need to ignore these messages.
These messages would not contain the created_at field and can be filtered out accordingly.
I have written the following piece of code, to extract the valid tweets, and then extract the user.id and the text.
def safe_parse(raw_json):
try:
json_object = json.loads(raw_json)
if 'created_at' in json_object:
return json_object
else:
return
except ValueError as error:
return
def get_usr_txt (line):
tmp = safe_parse(line)
if(tmp != None):
return ((tmp.get('user').get('id_str'),tmp.get('text')))
else:
return
My challenge is that I get one extra user called "None"
Here is a sample output (it is a large file)
('49838600', 'This is the temperament you look for in a guy who would
have access to our nuclear arsenal. ), None, ('2678507624', 'RT
#GrlSmile: #Ricky_Vaughn99 Yep, which is why in 1992 I switched from
Democrat to Republican to vote Pat Buchanan, who warned of all of
t…'),
I am struggling to find out, what I am doing wrong. There is no None in the tweeter file, hence I am assuming that I am reading the
{"limit":{"track":1,"timestamp_ms":"1456249416070"}} but the code above should not include it, unless I am missing something.
Any pointers? and thanks for the your help and your time.

Some lines in the input file might not be tweets, but messages that the Twitter server sent to the developer (such as limit notices). I need to ignore these messages.
That's not exactly what happens. If one of the following happens:
raw_json is not a valid JSON document
created_at is not in the parsed object.
you return with default value, which is None. If you want to ignore these, you can add filter step between two operations:
rdd.map(safe_parse).filter(lambda x: x).map(get_usr_txt)
You can also use flatMap trick to avoid filter and simplify your code (borrowed from this answer by zero323):
def safe_parse(raw_json):
try:
json_object = json.loads(raw_json)
except ValueError as error:
return []
else:
if 'created_at' in json_object:
yield json_object
rdd.flatMap(safe_parse).map(get_usr_txt)

Twitter Steam analysis issue

I am trying to take a twitter stream, save it to a file and then analyze the contents. However I am having an issue with files generated in the program as opposed to create by CLI
Twitter Analysis program:
import json
import pandas as pd
import matplotlib.pyplot as plt
tweets_data = []
tweets_file = open(“test.txt”, “r”)
for line in tweets_file:
try:
tweet = json.loads(line)
tweets_data.append(tweet)
except:
continue
tweets = pd.DataFrame()
tweets[‘text’] = map(lambda tweet: tweet[‘text’], tweets_data)
However with the last line I keep getting “KeyError: ‘text’ “ which I understand to be related to the fact it can’t find the key.
When I first run the twitter search program, if I output the results to a file from the CLI, it works fine with no issues. If I save the output to a file from inside the program, it gives me the error?
Twitter Search program:
class Output(StreamListener):
def on_data(self, data):
with open(“test.txt”, “a”) as tf:
tf.write(data)
def on_error(self, status):
print status
L = Output()
auth = OAuthHandler(consKey, consSecret)
auth.set_access_token(Tok1, Tok2)
stream = Stream(auth, L)
stream.filter(track=[´cyber´])
If I run the above as is, analyzing the test.txt will give me the error. But if I remove the line and instead run the program as:
python TwitterSearch.py > test.txt
then it will work with no problem when running text.txt through the analysis program
I have tried changing the file handling from append to write which was of no help.
I also added the line:
print tweet[‘text’]
tweets[‘text’] = map(lambda tweet: tweet[‘text’], tweets_data)
This worked and showing that the program can see a value for the text key. I also compared the output file from the program and the CLI and could not see any difference. Please help me to understand and resolve the problem?

File pointer - sending and reception in a function

Python 2.7
Download a test file from: www.py4inf.com/code/mbox.txt
Briefly, I need list all lines that start with 'From' and take only the mail address. Selecting line by line.
If the condition is true, write in other file the ( only) mail address (result). I could wrote the code and it is working. But It would be better if I use functions. I crashed when I tried to pass the parameters. I have a problem when I have a function that receive a parameter and send one or two.
The result is: copy line by line ALL input file in output file almost like a recursion. No search nothing and the file is very big.
At last, have you any page to read about funtions, paramt, passing paramt, pass reference, and other. Ask is easy and I prefer read and try to understand and if I have a problem, light a cande in the middle of the night!.
#Li is the input paramet. Line from fileRead(the pointer of the file).
#if the condition is true select all lines that start with From:
def funFormat(li):
if li.startswith('From:'):
li = li.rstrip('')
li = li.replace('From: ',' \n')
return li?
fileRead=open('mbox_small.txt','r')
for eachline in fileRead:
result=funFormat(eachline)
fileWrite =open('Resul73.txt', 'a')
fileWrite.write( result )
fileRead.close()
fileWrite.close()

You're opening a file every time you need to write, and closing it just at the end. Maybe that's what's messing it up? Try this and let me know if it works -
#Li is the input paramet. Line from fileRead(the pointer of the file).
#if the condition is true select all lines that start with From:
def funFormat(li):
if li.startswith('From:'):
li = li.rstrip('')
li = li.replace('From: ',' \n')
return li
else:
return None
fileRead = open('mbox_small.txt', 'r')
fileWrite = open('Resul73.txt', 'a')
for eachline in fileRead:
result=funFormat (eachline)
if result:
fileWrite.write (result)
fileRead.close()
fileWrite.close()
Also, I suggest you to read up on with blocks. This will help you work with files more efficiently. As for functions, there are enough resources online.

Python Syntax Error "Expected indented block"

I would just like to ask for help regarding my code as it is giving me an error which I cannot see. Normally, IDLE would highlight the error but this time, it's not giving me any at all so I am quite confused on where my problem is located.
Also just a headsup, I am quite new to python and have just recently tried it out about 2 days ago so I would be thankful if anyone could help this noob(me) with this problem.
import time as t
from os import path
##dest is the string
def createFile(dest):
'''
The script creates a text at the passed in location, names file based on date
'''
date = t.localtime(t.time())
##name=month/day/year
name = '%d_%d_%d.txt'%(date[1],date[2],(date[0]%100))
##if file does not exist
if not(path,isfile(dest+name)):
f = open(dest + name, 'w')
f.write('\n'*30)
f.close()
if __name__=='__main__':
destination = 'C:\\Python34\\My Projects in Python\\'
createFile(destination)
input("done!")

You had couple of issues in your code:
You need to indent the comment as well
You need to move back the name
You need to put path.isfile instead of path,isfile
You need to indent 3 lines after your first if
Here's working code:
import time as t
from os import path
##dest is the string
def createFile(dest):
'''
The script creates a text at the passed in location, names file based on date
'''
date = t.localtime(t.time())
##name=month/day/year
name = '%d_%d_%d.txt'%(date[1],date[2],(date[0]%100))
##if file does not exist
if not(path.isfile(dest+name)):
f = open(dest + name, 'w')
f.write('\n'*30)
f.close()
if __name__=='__main__':
destination = 'C:\Python34\My Projects in Python\'
createFile(destination)
input("done!")