Develop Reference

Using QGIS Model Designer to get Python scripts, not working?

I just realised that I could decrease the time needed to create my scripts by using the built-in model tool then extract the scripts. However, when extracting and running the script in the Python console, nothing happens. There are no files created, nor are there any errors reported. Running the script from the model builder works fine.
I might just be missing something, since this i totally new to me.
This is the code:
"""
Model exported as python.
Name : model
Group :
With QGIS : 32403
"""
from qgis.core import QgsProcessing
from qgis.core import QgsProcessingAlgorithm
from qgis.core import QgsProcessingMultiStepFeedback
from qgis.core import QgsProcessingParameterRasterLayer
from qgis.core import QgsProcessingParameterRasterDestination
import processing
class Model(QgsProcessingAlgorithm):
def initAlgorithm(self, config=None):
self.addParameter(QgsProcessingParameterRasterLayer('rasterinput', 'raster_input', defaultValue=None))
self.addParameter(QgsProcessingParameterRasterDestination('Raster_aspect', 'raster_aspect', createByDefault=True, defaultValue='C:/Users/tlind/AppData/Roaming/QGIS/QGIS3/profiles/default/processing/outputs/raster_aspect_output.tif'))
def processAlgorithm(self, parameters, context, model_feedback):
# Use a multi-step feedback, so that individual child algorithm progress reports are adjusted for the
# overall progress through the model
feedback = QgsProcessingMultiStepFeedback(1, model_feedback)
results = {}
outputs = {}
# Aspect
alg_params = {
'INPUT': parameters['rasterinput'],
'Z_FACTOR': 1,
'OUTPUT': parameters['Raster_aspect']
}
outputs['Aspect'] = processing.run('native:aspect', alg_params, context=context, feedback=feedback, is_child_algorithm=True)
results['Raster_aspect'] = outputs['Aspect']['OUTPUT']
return results
def name(self):
return 'model'
def displayName(self):
return 'model'
def group(self):
return ''
def groupId(self):
return ''
def createInstance(self):
return Model()

How to pickle a class instance if the class has another instance of different class?

So I am trying to build a chatbot using the famous "chatterbot" library in python. I made a class called Trainer which trains my chatbot. So in this class I am initializing the instance for the class "chatterbot" and then training it. So, to avoid retraining it again and again I though to pickle the instance of my Trainer class. So if it already exist, I don't need to retrain it and hence I am trying to pickle the Trainer class instance. I am using dill library to pickle the class instance but as it tries to pickle my model, it shows me the following error:
_pickle.PicklingError: Can't pickle >'sqlalchemy.orm.session.Session'>: it's not the same object as >>sqlalchemy.orm.session.Session
Now, I don't see anywhere in my code that I have created any kind of session. But I believe the chatterbot library that I am using in my Trainer class must be using any kind of session in it. Infact I checked the source code and it is using the logger. So it might be possible that's causing the pain. I have no clue how to proceed with this problem. I tried to change the source code of chatterbot library and removed every occurrence of logger from it but I did nothing but broke the code. Can anyone help me how to fix this issue. I am attaching required code here.
utils:
import logging
from pathlib import Path
import pickle
import dill
import os
from .trainer import Trainer
# Method returns the directories in which model objects are stored/saved.
def model_base_dir():
directory = 'MLModel/pickled_models'
parent_directory = os.pardir
return os.path.abspath(os.path.join(parent_directory,directory))
def picked_model(base_dir=None):
if base_dir == None:
logging.exception("Base Directory does not exist !!!")
raise AssertionError("Base Directory does not exist !!!")
model_path = base_dir+'/version1.pickle'
if Path(model_path).is_file():
with open(model_path,'rb') as handle:
model = dill.load(handle)
return model
else:
return None
def save_model(model_obj):
dir = model_base_dir() # Get the directory where model needs to be saved
with open(dir+'/version1.pickle','wb') as f:
dill.dump(model_obj,f)
f.close()
def train_model():
mod_obj = Trainer()
save_model(mod_obj)
return mod_obj
Trainer
from chatterbot import ChatBot
from chatterbot.trainers import ChatterBotCorpusTrainer
class Trainer():
def __init__(self):
self.chatbot = ChatBot('Dexter')
self.create_chatbot_trainer(language="chatterbot.corpus.english")
def train_chatbot(self,trainer,language):
return trainer.train(language)
def create_chatbot_trainer(self,language):
self.trainer = ChatterBotCorpusTrainer(self.chatbot)
self.trainer = self.train_chatbot(self.trainer,language)
return self.trainer
def __getstate__(self):
d = self.__dict__.copy()
d.pop('_parents',None)
return d
def response(self,text=""):
if text is None:
return "Sorry, your query is empty"
else:
return self.chatbot.get_response(text)
The train_model() gets trigger from my django view.
Any help appreciated.

How to output model pickle file to s3 in luigi?

I have a task which trains the model eg:
class ModelTrain(luigi.Task):
def output(self):
client = S3Client(os.getenv("CONFIG_AWS_ACCESS_KEY"),
os.getenv("CONFIG_AWS_SECRET_KEY"))
model_output = os.path.join(
"s3://", _BUCKET, exp.version + '_model.joblib')
return S3Target(model_output, client)
def run(self):
joblib.dump(model, '/tmp/model.joblib')
with open(self.output().path, 'wb') as out_file:
out_file.write(joblib.load('/tmp/model.joblib'))
FileNotFoundError: [Errno 2] No such file or directory: 's3://bucket/version_model.joblib'
Any pointers in this regard would be helpful

A few suggestions-
First, make sure you're using the actual self.output().open() method instead of wrapping open(self.output().path). This loses the 'atomicity' of the luigi targets, plus those targets are supposed to be swappable, so if you changed back to a a LocalTarget your code should work the same way. You let the specific target class handle what it means to open the file. The error you get looks like python is trying to find a local path, which obviously doesn't work.
Second, I just ran into the same issue, so here's my solution plugged into this code:
from luigi import format
class ModelTrain(luigi.Task):
def output(self):
client = S3Client(os.getenv("CONFIG_AWS_ACCESS_KEY"),
os.getenv("CONFIG_AWS_SECRET_KEY"))
model_output = os.path.join(
"s3://", _BUCKET, exp.version + '_model.joblib')
# Use luigi.format.Nop for binary files
return S3Target(model_output, client, format=format.Nop)
def run(self):
# where does `model` come from?
with self.output().open('w') as s3_f:
joblib.dump(model, s3_f)
My task is using pickle so I had to follow something similar to this post to re-import.
class MyNextTask(Task):
...
def run(self):
with my_pickled_task.output().open() as f:
# The S3Target implements a read method and then I can use
# the `.loads()` method to import from a binary string
results = pickle.loads(f.read())
... do more stuff with results ...
I recognize this post is stale, but putting the solution I found out there for the next poor soul trying to do this same thing.

Could you try to remove .path in your open statement.
def run(self):
joblib.dump(model, '/tmp/model.joblib')
with open(self.output(), 'wb') as out_file:
out_file.write(joblib.load('/tmp/model.joblib'))

How do I include the setup method in python-appium base page?

I am just a beginner when it comes to python or coding in general. I'm trying to setup an automation framework using python-appium and the page object model. My question is, How do I include the setup method into my base page? When I call the method from my test script it says 'driver' is unresolved and throws an exception. I know I am just missing something simple but my google-fu has failed me and now I have posted here.
Here is my setup method:
def setUp(self):
"Setup for the test"
desired_caps = {}
desired_caps['platformName'] = 'Android'
desired_caps['platformVersion'] = '6.0.1'
desired_caps['deviceName'] = '05157df532e5e40e'
# Returns abs path relative to this file and not cwd
desired_caps['app'] = os.path.abspath(
os.path.join(os.path.dirname(__file__),
'/Users/tyler/Desktop/Instasize_20172109_3.9.9_118_google.apk'))
desired_caps['appPackage'] = 'com.jsdev.instasize'
desired_caps['appActivity'] = '.activities.MainActivity'
self.driver = webdriver.Remote('http://localhost:4723/wd/hub',
desired_caps)
I want to call this method to all my tests or a variation of it depending on the device being used. Here is how I am trying to call it from my base page:
def driverSetup(self):
driverSetup = DriverBuilderAndroid(driver)
driverSetup.setUp()
All of my import statements are present. Let me know if there is any other info you need. If you could refer to a python appium POM tutorial that would be much appreciated as well. This is my first post on stackoverflow.

I created a separate python file called driverBuilder with my setup shown above. I then import it into each of my test files and call the method like this:
from DriverBuilder import DriverBuilder """<--class from DriverBuilder file"""
def test(self):
driver = DriverBuilder.driver

how to unittest and mock for open funtion

I have read many article over the last 6 hours and i still don't understand mocking and unit-testing. I want to unit test a open function, how can i do this correctly?
i am also concerned as the bulk of my code is using external files for data import and manipulation. I understand that i need to mock them for testing, but I am struggling to understand how to move forward.
Some advice please. Thank you in advance
prototype5.py
import os
import sys
import io
import pandas
pandas.set_option('display.width', None)
def openSetupConfig (a):
"""
SUMMARY
Read setup file
setup file will ONLY hold the file path of the working directory
:param a: str
:return: contents of the file stored as str
"""
try:
setupConfig = open(a, "r")
return setupConfig.read()
except Exception as ve:
ve = (str(ve) + "\n\nPlease ensure setup file " + str(a) + " is available")
sys.exit(ve)
dirPath = openSetupConfig("Setup.dat")
test_prototype5.py
import prototype5
import unittest
class TEST_openSetupConfig (unittest.TestCase):
"""
Test the openSetupConfig function from the prototype 5 library
"""
def test_open_correct_file(self):
result = prototype5.openSetupConfig("Setup.dat")
self.assertTrue(result)
if __name__ == '__main__':
unittest.main()

So the rule of thumb is to mock, stub or fake all external dependencies to the method/function under test. The point is to test the logic in isolation. So in your case you want to test that it can open a file or log an error message if it can't be opened.
import unittest
from mock import patch
from prototype5 import openSetupConfig # you don't want to run the whole file
import __builtin__ # needed to mock open
def test_openSetupConfig_with_valid_file(self):
"""
It should return file contents when passed a valid file.
"""
expect = 'fake_contents'
with patch('__builtin__.open', return_value=expect) as mock_open:
actual = openSetupConfig("Setup.dat")
self.assertEqual(expect, actual)
mock_open.assert_called()
#patch('prototype5.sys.exit')
def test_openSetupConfig_with_invalid_file(self, mock_exit):
"""
It should log an error and exit when passed an invalid file.
"""
with patch('__builtin__.open', side_effect=FileNotFoundError) as mock_open:
openSetupConfig('foo')
mock_exit.assert_called()