Develop Reference

How to share data between multiple python environments and active scripts controlled by a master script?

I'm having trouble finding info on this. I've seen numerous posts about sharing data between scripts running in the same environment, but not between environments.
I'm using Anaconda, and 16 envs that use several different python versions—python 3.5 to 3.8-for running various scripts/apps. I'm now working on shell scripts and a master python script that will control the launching of other envs, launching scripts, opening OS native apps and automating scripted tasks, and all will be saving, moving, and accessing data from several master folders on the same machine. I suppose the master script will behave like a mini server, and it also lives in its own env.
What I'm trying to figure out is if there's a way to easily pipe data between the environments, or do I have to store things in yaml or JSON files that they can all access (such as passing custom environment variables from the master env to all the others, or one script letting another know when it has completed, or detecting when a specific terminal window has closed)?
I don't need most scripts to share data between each other directly. I need to have feedback sent from the other envs to the master script that will be in control of everything and print output in its terminal window and fire up shell scripts. I need that master script to communicate with the other envs/scripts to give them new tasks and to load up the envs themselves, and so it knows that it's time to do something else—basically event listener and even handler stuff (I assume) along with watch folders. Most of them will run consecutively. Some scripts will be run at the same time from their own shells from the same environment, processing different data, and at times the master script (or an individual script) will pause and await user feedback in the master script terminal.
It might sound more complicated than it really is as most things happen linearly and the data that needs to be shared is small events and variables. One thing starts and finishes, a watch folder sees new files and the master fires up a new env and launches a script to process them, then the next kicks off, then the next, then 4 of the same thing kick off, then an app opens and runs a task and closes, then user is prompted how to proceed to choose which task runs next, etc. etc..
I found these packages which seem promising for some of the tasks:
python-dotenv and python-dotenv[cli]
watchdog and watchdog[watchmedo]
PyYAML
libyaml
appscript
inquirer
simplejson
cliclick (mac os Terminal commands for executing keyboard commands and mouse movements)
Creating lambda functions seems to be an easy way for executing os/system commands with python. Right now I just need to find the way to get all of these things talking to the master and vice versa, and sharing some data. One app uses jupyter lab, and I'm not sure if it's easy to automate that from another env/script.
I don't need something with a GUI like jupyter lab, and I honestly don't like its UI, and prefer to use a single master terminal window with some simple user input options.
A point in the right direction would be greatly appreciated.

Seems the solution here is to use sockets, create a server, and create clients inside the scripts I need to use. Not sure why my searches weren't bringing up sockets, but it's the solution I needed and doesn't require dependencies.
Sockets are built into python so using import sockets can handle most of what I need.
On top of that, import threading so multiple threads can be used for clients, and I'm using import system to send system commands. The threads are being setup as daemons to avoid any trouble if a client doesn't disconnect cleanly.
This has the benefit of running on a local network but can also be used for more complex system to connect to remote clients and servers. Running locally, your server can use its private IPv4 address to send and receive on one machine or on the intranet.
Tutorial
I found this YouTube video by Tech With Tim going through the complete basic setup, which was a big help as I'm completely new to this.
I ended up setting up classes for the server and client because all the functionality I needed would not work right without it. This video was a good way to get my feet wet, but far from what was needed.
I also created a standalone task manager script which is working better than trying to make the server do everything.
Basic server setup
server = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
server.bind(ADDRESS)
After that you need a function to handle client messages and a startup function for the server itself. Using server.listen() to listen for messages and a while loop to handle connections and kick off new threads for each client.
I have my master controller script separate, because I found it cumbersome to have the server running in the same window where I needed user input to take place. So I just programmatically launch, size and position a new Terminal window from the master and load up the server script inside it.
Basic client setup
client = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
client.connect(ADDRESS)
As with the server, the client will need a function for sending messages. Tim had a nice approach where you send a small header from the client first letting it know how many bytes the incoming message will be before actually sending the message, to ensure things don't get truncated.
Vars
Using environment variables and an env file really helped streamline this setup. I did this using python-dotenv. Make sure to set the port as an int in the main script or it might error because it sees it as as string.
As I made my scripts more advanced, I ended up placing all my vars and dicts full of vars in a custom module that I load as needed.
HEADER = 64 # header bytes
PORT = int(os.getenv('PORT')) # make this an int
SERVER = os.getenv('SERVER') # server local IP
ADDRESS = (SERVER, PORT)
FORMAT = 'utf-8'
DISCONNECT = '!killClient'

Sending commands to and receiving their output from a running Python application (via remote connection)

I have a long-running server application written in Python 3, which I would like to interactively debug from time to time. For that I would like to run Python commands "within" the server application process, inspect the values of global variables etc., like in a REPL or the standardd Python console.
It seems the Python standard library's code module and its InteractiveConsole class seems to be what I am looking for. I was thinking of running that in a separate thread so that the main application is not blocked while I communicate with it.
However, it seems that class provides interaction via standard input and output. That might not be exactly what I need. Is there a way to make that interactive console listen / connect to a socket and send input and output through this socket, so that I can connect to the console via a TCP connection?
Or is there another, better way to implement my requirement without this code module?

Recover previous Python output to terminal

I have a bash script, in Python that runs on a Ubuntu server. Today, I mistakenly closed the Putty window after monitoring that the script ran correctly.
There is some usefull information that was printed during the scrip running and I would like to recover them.
Is there a directory, like /var/log/syslog for system logs, for Python logs?
This scripts takes 24 hours to run, on a very costly AWS EC2 instance, and running it again is not an option.
Yes, I should have printed usefull information to a log file myself, from the python script, but no, I did not do that.

Unless the script has an internal logging mechanism like e.g. using logging as mentioned in the comments, the output will have been written to /dev/stdout or /dev/stderr respectively, in which case, if you did not log the respective data streams to a file for persistent storage by using e.g. tee, your output is lost.

Starting and stopping server script

I've been building a performance test suite to exercise a server. Right now I run this by hand but I want to automate it. On the target server I run a python script that logs server metrics and halts when I hit enter. On the tester machine I run a bash script that iterates over JMeter tests, setting timestamps and naming logs and executing the tests.
I want to tie these together so that the bash script drives the whole process, but I am not sure how best to do this. I can start my python script via ssh, but how to halt it when a test is done? If I can do it in ssh then I don't need to mess with existing configuration and that is a big win. The python script is quite simple and I don't mind rewriting it if that helps.

The easiest solution is probably to make the Python script respond to signals. Of course, you can just SIGKILL the script if it doesn't require any cleanup, but having the script actually handle a shutdown request seems cleaner. SIGHUP might be a popular choice. Docs here.
You can send a signal with the kill command so there is no problem sending the signal through ssh, provided you know the pid of the script. The usual solution to this problem is to put the pid in a file in /var/run when you start the script up. (If you've got a Debian/Ubuntu system, you'll probably find that you have the start-stop-daemon utility, which will do a lot of the grunt work here.)
Another approach, which is a bit more code-intensive, is to create a fifo (named pipe) in some known location, and use it basically like you are currently using stdin: the server waits for input from the pipe, and when it gets something it recognizes as a command, it executes the command ("quit", for example). That might be overkill for your purpose, but it has the advantage of being a more articulated communications channel than a single hammer-hit.

jython killing parent process that spawns subprocess breaks subprocess stdout to file?

Let me start with what I'm really trying to do. We want a platform independent startup script for invoking a JVM with some system properties and a dynamically generated classpath. We picked Jython in particular because we only need to depend on the standalone jython.jar in our startup script. We decided we could write a jython script that uses subprocess.Popen to launch our application's jvm and then terminates.
One more thing. Our application uses a lot of legacy debug code that prints to standard out. So the startup script typically has been redirecting stdout/stderr to a log file. I attempted to reproduce that with our jython script like this:
subprocess.Popen(args,stdout=logFile,stderr=logFile)
After this line the launcher script and hosting jvm for jython terminates. The problem is nothing shows up in the logFile. If I instead do this:
subprocess.Popen(args,stdout=logFile,stderr=logFile).wait()
then we get logs. So the parent process needs to run parallel to the application process launched via subprocess? I want to avoid having two running jvms.
Can you invoke subprocess in such a way that the stdout file will be written even if the parent process terminates? Is there a better way to launch the application jvm from jython? Is Jython a bad solution anyway?

We want a platform independent startup script for invoking a JVM with some system properties and a dynamically generated classpath.
You could use a platform independent script to generate a platform specific startup script either at installation time or before each invocation. In the latter case, additionally, you need a simple static platform specific script that invokes your platform independent startup-script-generating script and then the generated script itself. In both cases you start your application by calling a static platform specific script.
Can you invoke subprocess in such a way that the stdout file will be written even if the parent process terminates?
You could open file/redirect in a child process e.g., using shell:
Popen(' '.join(args+['>', 'logFile', '2>&1']), # shell specific cmdline
shell=True) # on Windows see _cmdline2list to understand what is going on