Develop Reference

Running script using local Python and packages but executing certain commands on remote server

I know how to run a Python script made locally on a remote server and have seen a lot of questions in that regard. But I am in a situation where I cannot install python packages on the remote server I am accessing. Specifically, I need to use pypostal, which requires libpostal to be installed and I cannot do so. Moreover, I need pyspark to play with Hive tables.
Therefore, I need the script to run locally, where I can manage my packages and everything executes fine, but certain commands need to access the server in order to grab data. For example, using pyspark to get Hive tables on the server into a local dataframe. Essentially, I need all the Python to be executed using my local distribution with my local packages but perform its actions on the remote server.
I have looked into things like paramiko. But as far as I can workout, is just like an SSH client, which would use the Python distro on the remote server and not locally. Though, perhaps I don't understand how to use it properly.
I am running python 3.6 on Ubuntu 18.04 using WSL. The packages I am using are pandas, numpy, pyspark, and postal (subsequently libpostal).
TLDR;
Is it possible to run a script locally, have parts of it execute remotely but using my local Python? Or if there are other possible solutions, I would be grateful.

How to run python script on multiple remote machine? without installing the script?

I created a simple python script which can give what kind of operating system it is using. But I want this script to run multiple remote machine at a same time without installing it on other machine on the same network. Is it possible to run this script on remote machine without installing this script on remote machine? . I saw with ssh python can do this .

Based on the way your question is worded, I think the answer is no.
If you want to run a python script on a remote machine, you will need to have a python interpreter of some kind available on the remote machine to execute the code.

Run Spyder /Python on remote server

So there are variants of this question - but none quite hit the nail on the head.
I want to run spyder and do interactive analysis on a server. I have two servers , neither have spyder. They both have python (linux server) but I dont have sudo rights to install packages I need.
In short the use case is: open spyder on local machine. Do something (need help here) to use the servers computation power , and then return results to local machine.
Update:
I have updated python with my packages on one server. Now to figure out the kernel name and link to spyder.
Leaving previous version of question up, as that is still useful.
The docker process is a little intimidating as does paramiko. What are my options?

(Spyder maintainer here) What you need to do is to create an Spyder kernel in your remote server and connect through SSH to it. That's the only facility we provide to do what you want.
You can find the precise instructions to do that in our docs.

I did a long search for something like this in my past job, when we wanted to quickly iterate on code which had to run across many workers in a cluster. All the commercial and open source task-queue projects that I found were based on running fixed code with arbitrary inputs, rather than running arbitrary code.
I'd also be interested to see if there's something out there that I missed. But in my case, I ended up building my own solution (unfortunately not open source).
My solution was:
1) I made a Redis queue where each task consisted of a zip file with a bash setup script (for pip installs, etc), a "payload" Python script to run, and a pickle file with input data.
2) The "payload" Python script would read in the pickle file or other files contained in the zip file. It would output a file named output.zip.
3) The task worker was a Python script (running on the remote machine, listening to the Redis queue) that would would unzip the file, run the bash setup script, then run the Python script. When the script exited, the worker would upload output.zip.
There were various optimizations, like the worker wouldn't run the same bash setup script twice in a row (it remembered the SHA1 hash of the most recent setup script). So, anyway, in the worst case you could do that. It was a week or two of work to setup.
Edit:
A second (much more manual) option, if you just need to run on one remote machine, is to use sshfs to mount the remote filesystem locally, so you can quickly edit the files in Spyder. Then keep an ssh window open to the remote machine, and run Python from the command line to test-run the scripts on that machine. (That's my standard setup for developing Raspberry Pi programs.)

pyqt test suite fails mysteriously with Jenkins, works flawlessly from Windows command shell

(note I doubt this is specific to PyQt so I've tagged with Qt too)
We have a 2 test suites (call them A and B) that we run with pytest on our dev workstations:
python -m pytest -c configfile -s -v A
python -m pytest -c configfile -s -v B
Suite B (and only that one) tests our PyQt components; A doesn't have any PyQt in it. We defined project A in Jenkins (version 1.658 btw) to run Suite A: it runs without issue in Jenkins. We did the same, defined a project B in Jenkins to run Suite B: this one fails intermittently after many tests, with a SYSTEM log message and a WARNING log message from Qt (caught by setting a handler via QtCore.qInstallMessageHandler()). The Jenkins log that captures the test suite B's stdout is:
SYSTEM log message from Qt: WindowCreationData::create: CreateWindowEx failed (Not enough storage is available to process this command.)
WARNING log message from Qt: Failed to create platform window for QWidgetWindow(0x705d260, name="FramedPartWidgetWindow") with flags QFlags<Qt::WindowType>(Window|WindowTitleHint|WindowSystemMenuHint|WindowMinMaxButtonsHint|WindowCloseButtonHint|WindowFullscreenButtonHint) (context: category=default)
Build step 'Execute Windows batch command' marked build as failure
The last line is output by Jenkins script running test suite B.
On the Jenkins machine (a Windows 7 Pro 64-bit platform, btw) that runs the test suites, I can open a Windows command shell and if I run the test suites from there, both test suites run without issue. Then I open a web browser, go to the Jenkins project page for suite B and click "Build now": this runs the same thing I ran from command shell, but I get the above error. If I do 10 builds, the above will happen for a different test every time, although always in the same "area". If I filter out the tests in the vicinity of where the failure occurred, the test runs further, but after removing 4 test classes this way, this no longer helps.
The issue is not desktop, because I am logged in. One difference between the command shell run vs the Jenkins run is that from shell, test suite B opens many (PyQt) windows and closes them. From Jenkins, I can't see any windows open so they seem to be opening in some "virtual" desktop. So maybe the issue the desktop. Do I need to somehow configure that virtual desktop to have larger graphics capacity?
The error seems to indicate that the process started by Jenkins is running out of some resource, but it's not clear what: there is plenty of drive space and memory.
Anyone have any idea where to go from here? I did a google search and all I could find is these, they don't look too promising although I will try the suggestions:
Developer Central
Not enough storage is available to process this command
Win32Exception Not enough storage is available to process this command
I'm not familiar with the intricacies of how the Jenkins service on Windows runs processes, so I'm at a loss.
Update 20161219: Apparently this is a known issues with GUI testing from Windows services, see my post on Bitnami Jenkins forum.

Apparently this is a known issues with GUI testing from Windows services, see my post on Bitnami Jenkins forum, as is the case with the bitnami jenkins stack we use. As I mention in that post, the bottom of the page https://wiki.jenkins-ci.org/display/JENKINS/Tomcat says GUI testing in Windows is not likely to work when Jenkins is installed using Tomcat as a container installed as a service. The only option seems to be to setup Tomcat to run using the Windows Scheduler (instead of a service), but unfortunately the bitnami stack we use for jenkins does not seem to allow this, so the only solution for us is to install jenkins from scratch and tomcat as a scheduled task.
It appears that on Windows (based on docs for setting up Jenkins to test GUI via Squish),
Install Jenkins master (this should be doable via bitnami stack in a Linux VM)
Install Windows slave. Make sure to not start the slave as Windows Service at Launch method. Windows Services are intended to run command line applications but not to run applications which consist of a GUI. Starting the Jenkins slave as JNLP via Launch slave agents via Java Web Start works fine.
Setup a node inside Jenkins at Manage Jenkins|Manage Nodes|New Node.
Read https://kb.froglogic.com/display/KB/Automation+on+Windows
This page seems to aggregate several posts related to this issue.

Running remote windows GUI application through ssh

I'm using py.test to run a series of python scripts that test an external windows GUI application.
I'm trying to run these tests on a remote ssh-enabled windows machines using fabric (using Bitvise SSH server on the remote end) but of course the tests which require GUI access fail.
I know that py.test has a package called xdist, but I guess it will suffer from the same problem if the channel we use is SSH.
Has anybody solved the issue of running remote GUI applications in Windows through SSH? SSH has been quite convenient for me to run remote deployment commands, and it would logically follow that py.test would be one of such commands.