So there's two things that we do on a certain site, which is download new files and post files, and they're easily definable (and tedious), so I've been trying to script it.
So anyways, I have the file downloading script done. I'm able to re use the login code
#Cookies
cookies = http.cookiejar.LWPCookieJar()
opener = urllib.request.build_opener(urllib.request.HTTPCookieProcessor(cookies))
urllib.request.install_opener(opener)
#Authenticate user
print("logging in")
url = "http://someurl.com/index.php?app=core&module=global§ion=login&do=process"
values = {"username" : USERNAME,
"password" : PASSWORD}
data = urllib.parse.urlencode(values)
req = urllib.request.Request(url, data)
urllib.request.urlopen(req)
But to post a thread with said file you have to attach it. I know how to send it the proper title and text for the thread from what I learned on how to log in. My problem is that to attach files you have to send a request to a form not a page request.
E.G. You select what file you want from file dialog and then click attach files, which it then uploads, you then finish writing up you're thread and THEN submit page.
Here's relevant html
<fieldset class='attachments'>
<script type='text/javascript'>
//<![CDATA[
ipb.lang['used_space'] = "Used <strong>[used]</strong> of your <strong>[total]</strong> global upload quota (Max. single file size: <strong>256MB</strong>)";
//]]>
</script>
<h3 class='bar'>Attachments</h3>
<!--SKINNOTE: traditional uploader needs this. -->
<div id='attach_error_box' class='message error' style='display:none'></div>
<input type='file' id='nojs_attach_0_1' class='input_upload' name='FILE_UPLOAD' tabindex='1' />
<input type='file' id='nojs_attach_0_2' class='input_upload' name='FILE_UPLOAD' tabindex='1' />
<ul id='attachments'><li style='display: none'></li></ul>
<br />
<span id='buttonPlaceholder'></span>
<input type='button' id='add_files_attach_0' class='input_submit' value='Attach This File' style='display: none; clear: both' tabindex='1' /> <span class='desc' id='space_info_attach_0'>Used <strong>9.45MB</strong> of your <strong>976.56MB</strong> global upload quota (Max. single file size: <strong>256MB</strong>)</span>
I have no idea how to code this, so I'm looking for direction.
Also on a sidenote if this sorta script might've been easier in other languages tell me which? I only used python because I knew it.
Thanks a lot.
Related
I'm trying to make a script that will auto-upload images that I place in a folder on my PC, but I'm having trouble getting my POST request to work.
Here is the form I'm trying to submit to...
<form method="POST" action="/uploadeventthumb.php?id=0" enctype="multipart/form-data">
<input type="file" name="image"><p>
<input type="hidden" name="id2" value="3kkggi1618601391"></p><p>
<input type="hidden" name="id3" value="1113887"></p><p>
<input type="Submit" name="Submit" value="Submit">
<br></p><p>Please only upload JPG images. <br>The dimensions should be <b>1280x720</b> pixels Wide and High.
</p></form>
Here is my code...
for f in glob.glob(upload+'*-t.jpg'):
f = f.replace("\\", "/")
sessionObj = requests.session()
r = sessionObj.post(TheSportsDB.login, TheSportsDB.login_data)
if r.status_code == 200:
print("Logged in!")
file = {'file' : open(f, 'rb')}
fn = f.replace(upload, "")
fn = fn.replace("-t.jpg", "")
TSDB = TheSportsDB.thumb+fn
r = sessionObj.post(TSDB, files=file)
print(r.text)
When I run the script I get a successful Logged in! message and then it prints out the text from the upload page, but never actually uploads the image, can somebody tell me what I am doing wrong, please!
What I was intending to use this script for just wasn't possible as the id2 field changed on every request, so I'm guessing it is a CSRF token as mentioned by #NobbyNobbs in the comments.
I managed to easily create the script using Selenium rather than Requests and BeautifulSoup.
I am quite inexperienced in both HTML, JS and flask but I am working on a chatbot that able to detect sentimental analysis of the sender.
My HTML code:
<div class="bottom_wrapper clearfix">
<div class="message_input_wrapper">
<form action = "{{ url_for('reply') }}" method = "POST">
<input
class="message_input"
id="text_message"
name = "sentimental_name"
placeholder="Tell me how you feel today..."
onkeydown="if (event.keyCode == 13)document.getElementById('send').click()">
</div>
<!--div class = "send_message1" id = 'audio' onclick = "start_dictation()">
<span style="font-size: 32px; color:black;">
<i class="fas fa-microphone"></i>
</span>
</div-->
<div class="send_message" id="send" onclick="get_message()">
<!--<div class="icon"></div>-->
<div class="text">Send</div>
</div>
</form>
</div>
This is my python-flask code:
#app.route('/senti', methods = ['POST'])
def reply():
if request.method == 'POST':
message = request.form['text_message']
a = TextBlob(message).sentiment.polarity
b = TextBlob(message).sentiment.subjectivity
My js that links to the onlick =
function get_message(){
var message = document.getElementById("text_message").value;
var json_data = {"msg":message}
var sender = JSON.stringify(json_data)
console.log(sender)
console.log(message);
insert_chat('me',message);
interact(sender);
}
Console log:
POST http://127.0.0.1:5000/senti 500 (INTERNAL SERVER ERROR)
send # jquery-3.4.1.js:9837
ajax # jquery-3.4.1.js:9434
interact # chat.js:34
get_message # chat.js:55
onclick # chat:58
It seems really simple but it is like I miss something. Thank you so much!
You would have to use "sentimental_name" in
request.form["sentimental_name"]
because you have
But it uses JavaScript function get_message() to get data when you click ENTER
<input ... onkeydown="if (event.keyCode == 13)document.getElementById('send').click()">
<div class="send_message" id="send" onclick="get_message()">
and converts to JSON with field "msg" so it sends it as data or json, not form.
function get_message(){
var message = document.getElementById("text_message").value;
var json_data = {"msg":message}
var sender = JSON.stringify(json_data)
console.log(sender)
console.log(message);
insert_chat('me',message);
interact(sender);
In flask reply() you can check this using:
print(request.args)
print(request.data)
print(request.form)
print(request.json)
JavaScript may expect that reply() returns also JSON - ie.
return jsonify(list_or_dictionary).
In JavaScript I see interact(sender); so you would have to find this function and see what it sends and what result it may expect.
BTW: you can also use requests.data.get("msg") and request.form.get("msg") instead of ["msg"] becauses .get() returns None when it can't find "msg" and you can use if not message: to catch this problem. And ["msg"] raises error when there is no "msg" and you would have to use try:/except: to catch it.
I have a problem submitting a HIT to Amazon Mechanical Turk sandbox.
I'm using the following code to submit a HIT:
external_content = """"
<ExternalQuestion xmlns="http://mechanicalturk.amazonaws.com/AWSMechanicalTurkDataSchemas/2006-07-14/ExternalQuestion.xsd">
<ExternalURL>https://MY_HOST_GOES_HERE/</ExternalURL>
<FrameHeight>400</FrameHeight>
</ExternalQuestion>
"""
import boto3
import os
region_name = 'us-east-1'
aws_access_key_id = 'MYKEY'
aws_secret_access_key = 'MYSECRETKEY'
endpoint_url = 'https://mturk-requester-sandbox.us-east-1.amazonaws.com'
# Uncomment this line to use in production
# endpoint_url = 'https://mturk-requester.us-east-1.amazonaws.com'
client = boto3.client('mturk',
endpoint_url=endpoint_url,
region_name=region_name,
aws_access_key_id=aws_access_key_id,
aws_secret_access_key=aws_secret_access_key,
)
# This will return $10,000.00 in the MTurk Developer Sandbox
print(client.get_account_balance()['AvailableBalance'])
response = client.create_hit(Question=external_content,
LifetimeInSeconds=60 * 60 * 24,
Title="Answer a simple question",
Description="Help research a topic",
Keywords="question, answer, research",
AssignmentDurationInSeconds=120,
Reward='0.05')
# The response included several helpful fields
hit_group_id = response['HIT']['HITGroupId']
hit_id = response['HIT']['HITId']
# Let's construct a URL to access the HIT
sb_path = "https://workersandbox.mturk.com/mturk/preview?groupId={}"
hit_url = sb_path.format(hit_group_id)
print(hit_url)
The error message I get is:
botocore.exceptions.ClientError: An error occurred (ParameterValidationError) when calling the CreateHIT operation: There was an error parsing the XML question or answer data in your request. Please make sure the data is well-formed and validates against the appropriate schema. Details: Content is not allowed in prolog. (1493572622889 s)
What might be the reason here? The xml fully agrees with xml schema located on amazon servers.
The html returned by the external host is:
<!DOCTYPE html>
<head>
<meta http-equiv='Content-Type' content='text/html; charset=UTF-8'/>
<script src='https://s3.amazonaws.com/mturk-public/externalHIT_v1.js' type='text/javascript'></script>
</head>
<body>
<!-- HTML to handle creating the HIT form -->
<form name='mturk_form' method='post' id='mturk_form' action='https://workersandbox.mturk.com/mturk/externalSubmit'>
<input type='hidden' value='' name='assignmentId' id='assignmentId'/>
<!-- This is where you define your question(s) -->
<h1>Please name the company that created the iPhone</h1>
<p><textarea name='answer' rows=3 cols=80></textarea></p>
<!-- HTML to handle submitting the HIT -->
<p><input type='submit' id='submitButton' value='Submit' /></p></form>
<script language='Javascript'>turkSetAssignmentID();</script>
</body>
</html>
Thank you
This message "Details: Content is not allowed in prolog." is the clue. It turns out that what this is saying is that you can't have content outside of where it is expected. This is what usually happens when a junk character (think smart-quotes or non-printable ASCII value) appears in there. These can be a real pain in the butt to diagnose.
In your case, it's a little easier to debug but still just as frustrating. Check out this line:
external_content = """"
It turns out that Python only needs three quotes (""") in order to acknowledge a multi-line string definition. Thus your fourth " was actually rendering as part of the XML. Change that line to this:
external_content = """
And you're golden. I just tested it and it works. Sorry for all the frustration, but hopefully this unblocks you. Happy Sunday!
I'm doing pages in html and python (I'm novice in python), I would like to have IP client address, but I don't know if it is possible. I saw it is possible with PHP language.
So, I execute my code in command line (with Linux) like that:
./code.py client_server app_name app_version
infos.py
def main( client_server, app_name, app_version):
template = open('infoHTML.py').read()
c = string.Template(template).substitute(
app_name = app_name,
app_version = app_version,
os = user,
user = login)
f = tempfile.NamedTemporaryFile(prefix='/tmp/info.html', mode='w', delete=False)
f.write(contenu)
f.close()
webbrowser.open(f.name)
if __name__ == "__main__":
client_server = sys.argv[1]
app_name = sys.argv[2]
app_version = sys.argv[3]
user = sys.platform
sys.argv.append(user)
login = getpass.getuser()
sys.argv.append(login)
main(client_server, app_name, app_version)
I have an html code into python code here: infoHTML.py
<html>
App: ${app_name}<br/><br/>
Version: ${app_version}<br/><br/>
User: ${user}<br/><br/>
<form name="sendData" method="get" action="http://localhost:8000/cgi/display.py">
Project: <input type="text" name="pro"><br/><br/>
Number: <input type="text" name="num"/><br/><br/>
<input type="submit" value="OK"/>
</form>
</body>
</html>
It's possible. You need to do it either by rendering the address on the response body or by requesting it with ajax after the response has already been rendered.
It would be hard to give you a code solution without seeing what web server you are using, but here are a couple of pointers for the first approach. To obtain the address, on the server side (python handler):
import socket
ip = socket.gethostbyname(socket.gethostname())
or if you are using something like Google App Engine:
ip = self.request.remote_addr
you should then write the IP to the response. For example, if you are using a templating engine to render your HTML, your HTML can look like something similar to this:
<html>
<script>
var ip = {{ip}}
</script>
and on the python code that renders the template you should do something like that:
htmlContent = template.render(ip=ip)
self.response.write(htmlContent)
I'm having some issues getting AJAX communication working using the Bottle framework. This is my first time using AJAX, so it's likely I just have the basics wrong. Hopefully a Bottle/AJAX guru can point this novice in the right direction. Here is the code I'm using:
#!/usr/bin/env python
from bottle import route, request, run, get
# Form constructor route
#route('/form')
def construct_form():
return '''
<html>
<head>
<script type="text/javascript">
function loadXMLDoc()
{
xmlhttp = new XMLHTTPRequest();
xmlhttp.onReadyStateChange = function()
{
if(xmlhttp.readyState == 4 && xmlhttp.status == 200)
{
document.getElementById("responseDiv").innerHTML = xmlhttp.responseText;
}
}
xmlhttp.open("GET", "/ajax", true);
xmlhttp.send();
}
</script>
</head>
<body>
<form>
<input name="username" type="text"/>
<input type="button" value="Submit" onclick="loadXMLDoc()"/>
</form>
<div id="responseDiv">Change this text to what you type in the box above.</div>
</body>
</html>
'''
# Server response generator
#route('/ajax', method='GET')
def ajaxtest():
inputname = request.forms.username
if inputname:
return 'You typed %s.' % (inputname)
return "You didn't type anything."
run(host = 'localhost', port = '8080')
There are a few issues here.
Javascript is case sensitive. XMLHTTPRequest should be XMLHttpRequest. You should have seen an error about this in your Javascript console.
onReadyStateChange should be onreadystatechange.
If you fix the above two issues your AJAX call will work, but you will only ever get the 'You didn't type anything.' response. This is because you are using GET. You need to change your code so the form values are posted using the POST method.
Also, why aren't you using jQuery to do AJAX? It would make your life much easier. :)