Develop Reference

Trying to run a simple web server to receive post data from my utility company that uses greenbuttondata

I have this running in a simple python script that runs a http server, its supposed to retrieve data from a post call from my utility company, ive noticed though that as soon as i run it, and navigate to https://www.yougetsignal.com/tools/open-ports/ and check if my port is open for 443, it says closed (even though its open before)
i put this in a test.py and run it so i can wait for the post request and download the xml data, but for whatever it turns off my port when i go and check it
from pgesmd_self_access.api import SelfAccessApi
from pgesmd_self_access.server import SelfAccessServer
from pgesmd_self_access.helpers import save_espi_xml
pge_api = SelfAccessApi.auth( < i changed this to reflect my auth file > )
SelfAccessServer(pge_api, save_file=save_espi_xml)
i thought the port should stay open on my router? it doesnt seem like normal behavior
i think i need to implement SO_REUSEADDR in his code according to some docket documentation, but im not sure how to do that
class SelfAccessServer:
"""Server class for PGE SMD Self Access API."""
def __init__(
self, api_instance, save_file=None, filename=None, to_db=True, close_after=False
):
"""Initialize and start the server on construction."""
PgePostHandler.api = api_instance
PgePostHandler.save_file = save_file
PgePostHandler.filename = filename
PgePostHandler.to_db = to_db
server = HTTPServer(("", 7999), PgePostHandler)
server.socket = ssl.wrap_socket(
server.socket,
certfile=api_instance.cert[0],
keyfile=api_instance.cert[1],
server_side=True,
)
if close_after:
server.handle_request()
else:
server.serve_forever()

Python CGI accessing mySQL: end of script output before headers

I am having trouble getting a Python CGI script to execute properly on an apache2 webserver that is running on a virtual Ubuntu 18 server. The hosting provider is DreamCompute, if that matters. I have gotten CGI scripts working under var/www/html/apps/cgi-bin/. Both helloworld.py and helloworld.pl execute fine both in the SSH terminal and in browser (Microsoft Edge).
The script giving me trouble is a Python script that accesses a MySQL database, reads data from it, then uses that data to fill some lists and generate some simple output at random (a magical spell with random effects for a tabletop RPG).
The spell generator script also executes fine in SSH, but when I try to view it in a browser it throws a 500 internal server error. The apache error logs tell me that the problem is end of script output before headers. I looked around, but couldn't find anyone with a similar problem and configuration.
EDIT: The full entry in the error log is:
[Tue Apr 20 00:20:25.324101 2021] [cgid:error] [pid 17275:tid 140105176721152] [client 2607:fea8:1d41:b800:7dca:305:b11a:447f:62505] End of script output before headers: spell_generator.py, referer: http://my.website/apps/cgi-bin/
After adding and removing parts of the Python script to see how it behaves in a browser, I believe I have isolated the problem: the part of the script that connects to the MySQL database. That database is also hosted on the same Ubuntu virtual machine, and it definitely has the right kind of data in it (just strings and IDs; nothing fancy).
Here's the Python code. I've removed some documentation comments but it should be pretty straightforward:
#!/usr/bin/python3
print("Content-type: text/html\n\n");
# The above two lines are boilerplate to ensure no print statements cause
# problems later in the program.
# Import statements
import cgi
import cgitb
cgitb.enable();
from random import choice
import mysql.connector
import sys
from enum import Enum
# Initialize enums for system comparison
class Ruleset(Enum):
GENERIC = "GENERIC"
GENERIC_DB = "generic_attributes"
DND_5 = "DND5e"
DND_5_DB = "dnd5e_attributes"
PATHFINDER_1 = "PF1e"
PATHFINDER_1_DB = "pathfinder1e_attributes"
# Determine what system to use, generic by default
spellSystem = Ruleset.GENERIC.value;
if (len(sys.argv)) > 1:
if (sys.argv[1] == Ruleset.DND_5.value):
spellSystem = Ruleset.DND_5.value;
if (sys.argv[1] == Ruleset.PATHFINDER_1.value):
spellSystem = Ruleset.PATHFINDER_1.value;
# === PROBLEM HERE ===
# Initialize SQL cursor, defaulting to generic
if spellSystem == Ruleset.DND_5.value:
spellDB = mysql.connector.connect(
host = "localhost",
user = "RemoteUser",
password = "password",
database = Ruleset.DND_5_DB.value
)
elif spellSystem == Ruleset.PATHFINDER_1.value:
spellDB = mysql.connector.connect(
host = "localhost",
user = "RemoteUser",
password = "password",
database = Ruleset.PATHFINDER_1_DB.value
)
else:
spellDB = mysql.connector.connect(
host = "localhost",
user = "RemoteUser",
password = "password",
database = Ruleset.GENERIC_DB.value
)
spellCursor = spellDB.cursor();
spellCursor.execute("SELECT ElementName FROM Element");
listHolder = spellCursor.fetchall();
# === CODE BELOW DOES NOT CAUSE PROBLEMS ===
#
# [logic that uses the SQL data]
#
# Output HTML page
print("<html> <head> <title>TEST - Magic Spell Generator</title> <link rel='stylesheet' href='../../css/style.css'> </head>");
print("<body> body contents");
print("</body>");
print("</html>");
The RemoteUser is a user account on the SQL server, which is used by "external" (non-root) programs to access the databases. password in the actual deployed script is a cryptographically secure password.
I'm not a Python expert, but the code runs with no problems when I execute it from the SSH terminal, so I don't think that bad code is to blame (although I could certainly be wrong). Here's a list of things I've tried already:
Checking for too many active SQL connections (there appears to be only one, the root user).
Making sure the script file has the correct privileges (chmod 755, same as the rest).
Making sure the necessary Python modules are installed (they are, and the Python-MySQL connector is the up to date one that works with the version of Python I'm using).
Restarting apache2.
I've spent most of today trying to find an answer. Any help or potential leads are welcome.

Turns out the problem wasn't actually the script trying to access the SQL database, which I figured out when I ran it properly in the SSH terminal (./spell_generator.py instead of python3 spell_generator.py). It crashed due to segmentation fault (core dumped), which implied that it was a code problem rather than a configuration problem.
After a bit of searching, I found Database connectivity through MySQL connector Python with CGI is not working, which pointed me to the real culprit: I was importing modules in the wrong order. I changed the import statements so that import mysql.connector is now the first one, and the script runs perfectly.
It shouldn't make a difference, but it does. I'm just happy.

jython script: Mail Session attribute is set but when i restart the console it is not able to detect the mail seesion

I have written Jython script to create a mail session in IBM websphere.
Jython Script :
import sys
nodeName =sys.argv[0]
serverName =sys.argv[1]
def createSession(nodeName,serverName):
print "Creating mailsession"
ds =AdminConfig.getid('/Node:'+nodeName+'/Server:'+serverName+'/MailProvider:Built-in Mail Provider/')
print ds
print AdminConfig.required('MailSession')
name = ['name','MailSession']
jndi = ['jndiName','mail/Session']
host = ['mailTransportHost','mailhost.misys.global.ad']
storehost = ['mailStoreHost','mailhost.misys.global.ad']
mailAttrs=[name,jndi,host,storehost]
print mailAttrs
ss = AdminConfig.create('MailSession',ds,mailAttrs)
AdminConfig.save()
After running the script i am able to see mail session created by script in console. but it is throwing an error on server as below :
[Root exception is javax.naming.NameNotFoundException: Context: MyServer20Cell/nodes/MyServer20Node/servers/MyServer20,
name: mail/Session: First component in name mail/Session not found.
But the strange thing is when i opened the IBM Console and go to mail Session , without modifying any value in mail session, click on apply changes ,save it and restart the server .It Works fine and server is not throwing any error.
Can any one tell did i have done anything wrong in Script. How i can resolve this issue.

This problem is solved. need to add this property - mailTransportProtocol and add it in attributes

Connecting to splunk server using python script then use this a in robotframework test

I am new to python and have run in to a problem with the following.
This is a code snippet from the Splunk api, thats used to connect to a splunk server then print the installed apps.
import splunklib.client as client
HOST = "server.splunk"
PORT = 8089
USERNAME = "UserABC"
PASSWORD = "Passw000rd"
# Create a Service instance and log in
service = client.connect(
host=HOST,
port=PORT,
username=USERNAME,
password=PASSWORD)
# Print installed apps to the console to verify login
for app in service.apps:
print app.name
I've tried it on my system in cmd and it works fine. However I intend to use this function in a Robot Framework test so the function needs to be defined in order to have a keyword I can use. I'm guessing something like the following:
import splunklib.client as client
def setServer(HOST, PORT, USERNAME, PASSWORD):
HOST = "server.splunk"
PORT = 8089
USERNAME = "UserABC"
PASSWORD = "Passw000rd"
service = client.connect(host=HOST,port=PORT,username=USERNAME,password=PASSWORD)
for app in service.apps:
print app.name
print ("\n")
My problem is when I run this nothing is printed to CMD at all. Any ideas
Thanks

A print in Python library is not displayed on the console of Robot Framework, that is the expected behaviour. If you want to check that the piece of code was run and the print was done, check the log.html produced by Robot. It should contain your print. Then if you really want to display something on Robot Console, then you have to use Log To Console keyword from your Robot Test. But as your print is in the python lib, you will have to import BuiltIn lib within your Python code. With all that, you should be fine.

Connecting to MongoHQ from heroku console (heroku run python)

I'm getting a 'need to login' error when trying to interact with my MongoHQ database through python console on heroku:
...
File "/app/.heroku/venv/lib/python2.7/site-packages/pymongo/helpers.py", line 128, in _check_command_response
raise OperationFailure(msg % response["errmsg"])
pymongo.errors.OperationFailure: command SON([('listDatabases', 1)]) failed: need to login
My applicable code
app/init.py:
from mongoengine import connect
import settings
db = connect(settings.DB, host=settings.DB_HOST, port=settings.DB_PORT, username=settings.DB_USER, password=settings.DB_PASS)
app/settings.py:
if 'MONGOHQ_URL' in os.environ:
url = urlparse(os.environ['MONGOHQ_URL'])
DB = url.path[1:]
DB_HOST = url.hostname
DB_PORT = url.port
DB_USER = url.username
DB_PASS = url.password
os.environ['MONGOHQ_URL'] looks like:
'mongodb://[username]:[password]#[host]:[port]/[db-name]'
This code works (connects and can read and write to mongodb) both locally and from the heroku web server.
According to the docs (http://www.mongodb.org/display/DOCS/Connections), it should at make a 'login' attempt on connection to the server as long as the username and password params are passed to Connection or parseable from the URI. I couldn't think of a way to see if the login attempt was being made and failing silently.
I've tried bypassing mongoengine and using pymongo.Connection and got the same result. I tried all of the several patterns of using the Connection method. I created a new database user, different from the one mongoHQ creates for heroku's production access -> same same.
It's a flask app, but I don't think any app code is being touched.
Update
I found a solution, but it will cause some headaches. I can manually connect to the database by
conn = connect(settings.DB, host=settings.DB_HOST, port=settings.DB_PORT, username=settings.DB_USER, password=settings.DB_PASS)
db = conn[settings.DB]
db.authenticate(settings.DB_USER, settings.DB_PASS)
Update #2
Mongolab just worked out of the box.

Please use the URI method for connecting and pass the information to via the host kwarg eg:
connect("testdb_uri", host='mongodb://username:password#localhost/mongoenginetest')

MongoHQ add-on uses password hashes not actual passwords and that's perhaps the error.
You should change the environment variable MONGOHQ_URL to a real password with the following command:
heroku config:set MONGOHQ_URL=mongodb://...
Once set, you may restart your applications (heroku apps) so the change gets picked up. If you're in the directory of the failing application, config:seting the config var will restart the application.