Develop Reference

How do I get the GCE vm instance id using python?

I am using python. I have the correct project and vm instance names. So I can query Google Cloud metrics just fine. But now I need to query some Agent metrics, but it needs the instance id of my vm instead of the name. What is the simplest way for me to get the instance id of my vm with a query?
Sorry, I should be more clear. Here is my sample code:
results = client.list_time_series(
request={
"name": project_name,
"filter": filter,
"interval": interval,
"view": monitoring_v3.ListTimeSeriesRequest.TimeSeriesView.FULL,
}
)
I want to make a query similar to this. Any simple filter I can use, or something else, that will get me the instance_id of a particular instance name?

If you are inside the gce vm you use the Metadata server
import requests
metadata_server = "http://metadata/computeMetadata/v1/instance/"
metadata_flavor = {'Metadata-Flavor' : 'Google'}
gce_id = requests.get(metadata_server + 'id', headers = metadata_flavor).text
gce_name = requests.get(metadata_server + 'hostname', headers = metadata_flavor).text
gce_machine_type = requests.get(metadata_server + 'machine-type', headers = metadata_flavor).text
If you are looking to list gce vms check the example in the GCP documentation for using client libraries: Listing Instances

cx_Oracle SessionPool root of all Flask problems

I created a web service in Flask over uwsgi. I thought I would follow good practice and create a SessionPool with 20 connections to be safe. Each call to a web service endpoint, I acquire a connection from the pool, and at the end I release it.
When using Locust to swarm test the API, I was getting hundreds of failures, nearly 100% on some of the longer responses (30Mb JSON response). Smaller payloads were much better, but with intermittent failures.
The minute I switched back to bad practice and created a brand new connection and cursor within the method itself, all my problems vanished. 100% success on 1000s of stress test calls.
My errors were varied. TNS Bad Packet, incorrect number of connections from pool, request cancelled by user....you name it, it was there.
So I can't use Oracle connection pooling with flask it seems, or have a single connection at the Flask application level (this generated errors, not sure why, which is why I switched to connection pooling).
Any advice on creating scalable apps using cx_Oracle in flask.
My original code was:
pool = cx_Oracle.SessionPool("user", "password", "myserver.company.net:1521/myservice", min=10, max=10, increment=0, getmode=cx_Oracle.SPOOL_ATTRVAL_WAIT, encoding="UTF-8")
def read_products_search(search=None):
"""
This function responds to a request for /api/products
with the complete lists of people
:return: json string of list of people
"""
conn_ariel = pool.acquire()
cursor_ariel = conn_ariel.cursor()
search=search.lower()
print("product search term is: ", search)
# Create the list of products from our data
sql = """
SELECT DRUG_PRODUCT_ID, PREFERRED_TRADE_NAME, PRODUCT_LINE, PRODUCT_TYPE, FLAG_PASSIVE, PRODUCT_NUMBER
FROM DIM_DRUG_PRODUCT
WHERE lower(PREFERRED_TRADE_NAME) LIKE '%' || :search1 || '%' or lower(PRODUCT_LINE) LIKE '%' || :search2 || '%' or lower(PRODUCT_NUMBER) LIKE '%' || :search3 || '%'
ORDER BY PREFERRED_TRADE_NAME ASC
"""
cursor_ariel.execute(sql, {"search1":search,"search2":search, "search3":search })
products = []
for row in cursor_ariel.fetchall():
r = reg(cursor_ariel, row, False)
product = {
"drug_product_id" : r.DRUG_PRODUCT_ID,
"preferred_trade_name" : r.PREFERRED_TRADE_NAME,
"product_line" : r.PRODUCT_LINE,
"product_type" : r.PRODUCT_TYPE,
"flag_passive" : r.FLAG_PASSIVE,
"product_number" : r.PRODUCT_NUMBER
}
# logging.info("Adding Product: %r", product)
products.append(product)
if len(products) == 0:
products = None
pool.release(conn_ariel)
return products

When you create the pool, use threaded=True.
See How to use Python Flask with Oracle Database.

LDAP mocking in Python (ldap3 module)

My current task is to do QA Automation on our web app, but I don't want to use real credentials for it (for which we use a LDAP server). My idea was to mock the LDAP server when the web app is in TEST_MODE, and to my luck I found out that 'ldap3' (the python module) that we use for authentication, also supports a mocking function. An example code is here:
from ldap3 import Server, Connection, ALL, ALL_ATTRIBUTES, MOCK_SYNC
REAL_SERVER = 'my_real_server'
REAL_USER = 'cn=my_real_user,ou=test,o=lab'
REAL_PASSWORD = 'my_real_password'
# Retrieve server info and schema from a real server
server = Server(REAL_SERVER, get_info=ALL)
connection = Connection(server, REAL_USER, REAL_PASSWORD, auto_bind=True)
# Store server info and schema to json files
server.info.to_file('my_real_server_info.json')
server.schema.to_file('my_real_server_schema.json')
# Read entries from a portion of the DIT from real server and store them in a json file
if connection.search('ou=test,o=lab', '(objectclass=*)', attributes=ALL_ATTRIBUTES):
connection.response_to_file('my_real_server_entries.json', raw=True)
# Close the connection to the real server
connection.unbind()
# Create a fake server from the info and schema json files
fake_server = Server.from_definition('my_fake_server', 'my_real_server_info.json', 'my_real_server_schema.json')
# Create a MockSyncStrategy connection to the fake server
fake_connection = Connection(fake_server, user='cn=my_user,ou=test,o=lab', password='my_password', client_strategy=MOCK_SYNC)
# Populate the DIT of the fake server
fake_connection.strategy.entries_from_json('my_real_server_entries.json')
# Add a fake user for Simple binding
fake_connection.strategy.add_entry('cn=my_user,ou=test,o=lab', {'userPassword': 'my_password', 'sn': 'user_sn', 'revision': 0})
# Bind to the fake server
fake_connection.bind()
I followed this example with our code, and was successful halfway through it. I extracted the real_server_entries in a json file, and now is part to do the fake connection. So to summarise everything is done until this part:
# Create a MockSyncStrategy connection to the fake server
fake_connection = Connection(fake_server, user='cn=my_user,ou=test,o=lab', password='my_password', client_strategy=MOCK_SYNC)
I am not really sure what to put in the place of user and password.
A part of my code:
_USER_SEARCH_FILTER = "(&(objectClass=user)(cn={}))"
_ALL_USERS_SEARCH_FILTER = "(&(objectCategory=person)(objectClass=user))"
_EMAIL_ATTRIBUTE = "mail"
_DISPLAY_NAME_ATTRIBUTE = "displayName"
_USERNAME_ATTRIBUTE = "cn"
_SEARCH_ATTRIBUTES = (
_EMAIL_ATTRIBUTE, _DISPLAY_NAME_ATTRIBUTE, _USERNAME_ATTRIBUTE)
_LDAP_CONNECTION_ERROR = "Connection to LDAP server %s:%s failed: %s"
_LDAP_SERVER = Server(host=LDAP.host, port=int(LDAP.port), get_info='ALL')
server = _LDAP_SERVER
_CONNECTION = Connection(
server,
LDAP.manager_dn, LDAP.manager_password,
auto_bind=True, client_strategy=RESTARTABLE
)
server.info.to_file('my_real_server_info.json')
server.schema.to_file('my_real_server_schema.json')
if _CONNECTION.search(LDAP.root_dn, _ALL_USERS_SEARCH_FILTER, attributes=_SEARCH_ATTRIBUTES):
_CONNECTION.response_to_file('my_real_server_entries.json', raw=True)
_CONNECTION.unbind()
mock_server = Server.from_definition('mock_server', 'my_real_server_info.json', 'my_real_server_schema.json')
mock_connection = Connection(mock_server, user='???', password='???', client_strategy=MOCK_SYNC)
mock_connection.strategy.entries_from_json('my_real_server_entries.json')
mock_connection.strategy.add_entry('LDAP.root_dn', { #My guess is that here I mock the attributes, but this is also the other problem I am having (check below) })
The other problem I have is that I am not even sure if I can add a fake user, because when I took a look into the real_entries.json file the password is not stored there as an attribute (not even an encrypted version of it) and the only attributes we have are:
`cn` - username
`displayName` - LASTNAME, FIRSTNAME
`mail` - example#mail.com

SoftLayer API Nessus Scan Status / Report via python

I want to use python client to create a Nessus Security Scanner and check the status by getStatus and get the result by getReport method. While, I have read these helps by php(SoftLayer API Nessus Scan Status / Report via PHP). But how can i use these by python client?
When I call setInitParameter(scan_id) in by python, the exception as flows:
SoftLayerAPIError(Client): Function ("setInitParameter") is not a valid method for this service

i recomend you to read documentation of the client first:
https://github.com/softlayer/softlayer-python
https://softlayer-api-python-client.readthedocs.io/en/latest/
the init parameters are set like this:
clientService.getObject(id=myInitParameter)
here you can find more examples using the client:
https://softlayer.github.io/python/
Here you can find additional documentation:
http://sldn.softlayer.com/blog
And renember that with the Softlayer's python client unlike the php client the data are sending in json format so the request:
$client = SoftLayer_SoapClient::getClient('SoftLayer_Account', null, $apiUsername, $apiKey);
$accountInfo = $client->getObject();
$hardware = $client->getHardware();
foreach ($hardware as $server){
$scanclient = SoftLayer_SoapClient::getClient('SoftLayer_Network_Security_Scanner_Request', '', $apiUsername, $apiKey)
$scantemplate = new stdClass();
$scantemplate->accountId = $accountInfo->id;
$scantemplate->hardwareId = $server->id;
$scantemplate->ipAddress = $server->primaryIpAddress;
try{
// Successfully creates new scan
$scan = $scanclient->createObject($scantemplate);
} catch (Exception $e){
echo $e->getMessage() . "\n\r";
}
would be like this:
clientAccount = client['SoftLayer_Account']
accountInfo = clientAccount.getObject() #for this case we do not need init parameter
hardware = clientAccount.getHardware() #for this case we do not need init parameter
for server in hardware:
scanclient = client['SoftLayer_Network_Security_Scanner_Request']
scantemplate = {
"accountId": accountInfo["id"],
"hardwareId": server["id"],
"ipAddress": server["primaryIpAddress"]
}
scanclient.createObject(scantemplate)

Test Environment with Mocked REST API

Lets say I have a very simple web app which is presented as blue if the current president is a democrat and red if they are a republican. A REST API is used to get the current president, via the endpoint:
/presidents/current
which currently returns the json object:
{name: "Donald Trump", party: "Republican"}
So when my page loads I call the endpoint and I show red or blue depending on who is returned.
I wish to test this HTML/javascript page and I wish to mock the back-end so that I can control from within the test environment the API responses. For example:
def test_republican():
# configure the response for this test that the web app will receive when it connects to this endpoint
configure_endpoint(
"/presidents/current",
jsonify(
name="Donald Trump",
party="Republican"
)
)
# start the web app in the browser using selenium
load_web_app(driver, "http://localhost:8080")
e = driver.find_element_by_name("background")
assert(e.getCssValue("background-color") == "red")
def test_democrat():
# configure the response for this test that the web app will receive when it connects to this endpoint
configure_endpoint(
"/presidents/current",
jsonify(
name="Barack Obama",
party="Democrat"
)
)
# start the web app in the browser using selenium
load_web_app(driver, "http://localhost:8080")
e = driver.find_element_by_name("background")
assert(e.getCssValue("background-color") == "blue")
So the question is how should I implement the function configure_endpoint() and what libraries can you recommend me?

As #Kie mentioned, configure_endpoint implementation won't be enough, if you're going to stub the whole server-side within Selenium Python code. You would need a web server or whatever that will response via HTTP to requests from within testing environment.
It looks like the question is partially about testing of client-side code. What I see is that you're trying to make unit-test for client-side logic, but use integration testing suite in order to check this logic (it's strange).
The main idea is as follows.
You're trying to test client-side code. So, let's make mocks client-side too! Because this part of code is completely client-side related stuff.
If you actually want to have mocks, not stubs (watch the difference here: https://stackoverflow.com/a/3459491/882187) it is a better way to mock out HTTP requests inside your Javascript code. Just because you're testing a client-side piece of code, not some parts of server-side logic.
Having it isolated from whatever server-side is - is a great idea that you would love when your project become grow, while more and more endpoints will be appearing.
For example, you can use the following approach:
var restResponder = function() { // the original responder your client-side app will use
this.getCurrentPresident = function(successCallback) {
$.get('/presidents/current', callback);
}
};
var createMockResponder = function(president, party){ // factory that creates mocks
var myPresident = president;
var myParty = party;
return function() {
this.getCurrentPresident = function (successCallback) {
successCallback({"name": myPresident, "party": myParty});
}
};
}
// somewhere swap the original restResponder with new mockResponder created by 'createMockResponder'
// then use it in your app:
function drawColor(restResponder, backgroundEl) {
restResponder.getCurrentPresident(function(data){
if (data.party == "Democrat") $(backgroundEl).style('background-color', 'blue')
else if (data.party == "Republican") $(backgroundEl).style('background-color', 'red')
else console.info('Some strange response from server... Nevermind...');
});
}
Practically, this implementation depends on what do you have at the client-side as a framework. If jQuery, then my example is enough, but it looks very wordy. In case you have something more advanced, like AngularJS, you can do the same in 2-3 lines of code:
// Set up the mock http service responses
$httpBackend = $injector.get('$httpBackend');
// backend definition common for all tests
authRequestHandler = $httpBackend.when('GET', '/auth.py')
.respond({userId: 'userX'}, {'A-Token': 'xxx'});
Check out the docs: https://docs.angularjs.org/api/ngMock/service/$httpBackend
If you're still stick to the idea, that you need mocks inside Selenium tests, please
try this project: https://turq.readthedocs.io/en/latest/
It serves with Python DSL for describing REST responders.
Using turq your mocks will look as follows:
path('/presidents/current').json({'name':'Barack Obama', 'party': 'Democrat'}, jsonp=False)
Also, I would recommend to try stubs instead of mocks and use this Python module: mock-server https://pypi.python.org/pypi/mock-server/0.3.7
You are required to create the directory layout containing corresponding pre-populated JSON responses and to add some boilerplate code in order to make the mock-server respond on 'localhost:8080'. The directory layout for your example will look like this:
stub_obama/
presidents/
current/
GET_200.json # will contain {"name": "Barack Obama", "party": "Democrat"}
stub_trump/
presidents/
current/
GET_200.json # will contain {"name": "Donald Trump", "party": "Republican"}
But the mock_server is based on Tornado, it is very heavy solution for using in tests I think.
I hope, my answer is helpful and informative. Welcome to discuss it! I made tons of projects with Selenium, big and small tests, tested client-side and server-side.

I would use tornado web framework.
import json
import functools
import operator
from tornado import ioloop, web, gen
from tornado.options import define, options
define("data_file", default='default/mock.json', type=str)
class Handler(web.RequestHandler):
def data_received(self, chunk):
pass
def initialize(self, data):
self.data = data
#gen.coroutine
def get(self, *args, **kwargs):
path = self.request.path.split("/")[1:]
path = functools.reduce(
operator.add,
[[k, v[0].decode("utf-8")] for k, v in self.request.query_arguments.items()],
path
)
try:
self.write(functools.reduce(operator.getitem, path, self.data))
except KeyError:
self.set_status(404)
class Application(web.Application):
def __init__(self):
data = {}
with open(options.data_file) as data_file:
data = json.load(data_file)
handlers = [
('(.*)', Handler, {"data": data})
]
settings = dict(
gzip=True,
static_hash_cache=True,
)
web.Application.__init__(self, handlers, **settings)
def main():
io_loop = ioloop.IOLoop.instance()
backend_application = Application()
backend_application.listen(8001)
io_loop.start()
if __name__ == "__main__":
main()
This is a code I used for mocking a REST-API which is a standalone script, but it can be embedded into your test environment as well.
I defined a JSON file which defines the different path components and what should be returned. Like this:
{
"presidents": {
"current": {
"name": "Donald Trump",
"party": "Republican"
}
}
}
I saved this to a mock.json and called the script with a parameter mock_rest.py --data-file="./mock.json".
I hope that gives you a starting point and a good example.

If your load_web_app function uses the requests library to access the REST API, using requests-mock is a convenient way to fake that library's functionality for test purposes.

For those who stumble upon this question, and do not want to end up writing the code to create their own mock server implementations of the API, you can use Mocktastic, which is a downloadable desktop application for Windows, MacOS and Linux, which provides an easy to use GUI to setup your mock API servers.