My company make a variety of (usually) desktop apps to be used internally to aid the creation/analysis of our datasets.
Recently I created a program which acts as a sort of installation & runtime manager, with the purpose of:
Providing users with a consistent means by which to locate and install applications
Manage execution of installed apps and provide environments for various interpreted languages so the user doesnt need to know/care about python/R
Ensure users are up-to-date with latest versions etc..
The program itself is a desktop GUI which integrates with our in-house software website and grabs applications from a network location.
To run apps, it typically overwrites the users' PATH variable with it's own settings plus what ever the app defines.
Occasionally, this is not enough isolation and a user has a problem running apps.
Is there a way to easily run an app in isolation? Ideally, I could remove all environment variables other than the defaults provided by Windows, but this still leaves the possibility of odd registry values interfering with execution..
The application itself is written in Qt C++
Related
I plan to write a service for my *nix systems that can interact with a select number of GUI applications like Photoshop, Libre Office, etc. on the local machine.
The purpose for the local service is essentially to listen to a remote message and accordingly perform specific operations- for instance, changing the background of a layer in Photoshop, or adding margins to a page in Libre Office (or MS Office). You can assume that the application is active in the display environment of the operating system.
Now my question is:
Is this even possible? I personally find this task impossible unless I get to peek into the source code of these apps and basically augment these applications themselves. But since they are mostly proprietary apps, there are legal implications too.
Assuming it is somehow possible by changing the source of these apps, won't a team have to figure out the architecture and the inner workings of all these specifi applications or are there general frameworks concepts I should look into?
The UI should be updated from the main thread only. Or just consider sending Windows messages out.
The purpose for the local service is essentially to listen to a remote message and accordingly perform specific operations
Microsoft does not currently recommend, and does not support, Automation of Microsoft Office applications from any unattended, non-interactive client application or component (including ASP, ASP.NET, DCOM, and NT Services), because Office may exhibit unstable behavior and/or deadlock when Office is run in this environment.
If you are building a solution that runs in a server-side context, you should try to use components that have been made safe for unattended execution. Or, you should try to find alternatives that allow at least part of the code to run client-side. If you use an Office application from a server-side solution, the application will lack many of the necessary capabilities to run successfully. Additionally, you will be taking risks with the stability of your overall solution. Read more about that in the Considerations for server-side Automation of Office article.
I have the last six months been working on a Python GUI application that I will use at work. Specifically my GUI will run on a couple of super computer clusters that I use for work.
However, I am mostly developing the software at my personal computer, and here I do not have direct access to the commands that my GUI will call, since the GUI will use subprocess to call commands that only are available on the computing cluster.
So, in order to efficiently develop the program, I often have to copy the directory containing all files related to the GUI, to the cluster. Then I test my current version there, locate all my bugs, fix them by editing the files on the cluster, and finally copy back all files to my computer, overwriting the old version.
This just seems like a bad way of doing it, but I have to be able to test my software in the environment it is made for in order to find my bugs.
Surely this is a common problem in software development... What do actual programmers do (as opposed to hobby programmers such as myself)?
Edit:
Examples of commands that are only available on the computing cluster, that I make heavy use of, are squeue, sacct, and scontrol (SLURM related commands).
Edit2:
I could mention that I tested using ssh connections with Python, but it slowed down the commands significantly, having to establish the ssh connection for each command I wanted. Unless I could set of a lasting ssh session, as in logging in when opening my program, I don't think the ssh-ing will work.
Explore the concepts that make Vagrant a popular choice for developers
Vagrant is a tool for building and managing virtual machine
environments in a single workflow. With an easy-to-use workflow and
focus on automation, Vagrant lowers development environment setup
time, increases production parity, and makes the "works on my machine"
excuse a relic of the past.
Your use case is covered by a couple of vagrant boxes that create a slurm cluster for development purposes. A good starting point might be
Example slurm cluster on your laptop (multiple VMs via vagrant)
If you understand and can setup your development environment with tools like Vagrant, you might explore next which options modern code editors or integrated development environments (IDE) offer for remote development. Remote development covers some other use cases, that might fit into your developer toolbox as well.
A "good enough", free and open source code editor for Python development is Visual Studio Code. According to the docs it has powerful features for remote development.
Visual Studio Code Remote Development allows you to use a container, remote machine, or the Windows Subsystem for Linux (WSL) as a full-featured development environment.
Read the docs
VS Code Remote Development
I am trying to restructure a large project so that I can have one core set of code reused among a web frontend application written in flask and an automated back end that uses luigi tasks.
so I have two clients that I'd like to share code between in order to access a database and perform some automated tasks from either the web or luigi.
What is a good way to keep those three things organized and structured so that I can easily import core modules into either project.
I've had some issues getting Luigi to recognize modules that are parallel to it.
Technically, the only requirement to re-use the code is that the Python files of the shared modules ("core set of code") are found via sys.path of the executing interpreter, e.g. by adding the according directory to the PYTHONPATH environment variable. This enables you to use them as modules. However, you have to assess the changes to your project layout depending on your version control scheme and how you deploy the web application and clients. Usually, I would assume that you have to use the following rules of thumb.
Manage the shared code modules as a separate project. Especially, if the lifecycles of the depending clients and web applications are not tied together.
Treat the shared core modules as a library or framework. I should not rely on implementation details of the web application or the clients. Usually, you need to provide some entry points and configuration hooks.
Change your deployment in a way that the shared core modules are pulled as a dependency of the according client/application. There is a magnitude of possibilites depending on your setup and use case. For example, you could build pip-installable packages using setuptools.
So my team and I have bought into Docker - it is fantastic for deployment and testing. My real question is how to set up a great developer experience, specifically around writing Python apps, but this question could be generalized to nodejs, Java, etc.
The problem: When writing a Python app, I really like having decent linting/autocomplete functionality, there are some really good editors out there (Atom, VSCode, PyCharm) that provide these, but most really want a Python install on the local disk. The real advantage of Docker is that all of the core language and any project libraries can all be in the container, so reproducing all of that on the host machine just for developing is a pain.
I know that PyCharm pro does support Docker and docker-compose, but I found it quite sluggish and a lot of the test running capabilities were busted. On top of that, I really would like something that I can commit to version control so that the team can share dev setup and people don't have to repeat all of the steps for their own system.
A few Ideas that I had were:
Install an editor (like Atom) in a sidecar Docker container and use X11 forwarding
Use a browser based editor such as https://c9.io/ in a container - this seems most promising
Install some agent in a dev container that could handle autocomplete/linting, etc. and connect to it from a locally running editor - I think this would be the best solution, but I also think that right now it actually doesn't exist.
Has anyone had luck setting up a more productive development environment besides just mounting volumes and editing text?
You should use an 'advanced' IDE like IntelliJ (Pycharm) and configure a remote Python SDK using SSH-Access to your App-Docker-Container (using a shared ssh-key to auth against the app-container with a preinstalled openssh server and preconfigured authorized_keys file).
You can share this SDK information in your project file with all devs, so they wlll have this setup out of the box
1) This will ensure, your IDE knows about all the python libs/symbols available/installed in your docker-container during runtime. It will also enable you to properly debug remotely at the same time
2) This ensures, you have an IDE at your hand including a lot of important additional features like the inspector, 3way duff, search in path.. . hardly any of the Browser-Based IDEs will catch up with Pycharm at this point IMHO
Of course, as already mentioned in the comments, you need to share aka mount your code into the container. On linux, you plainly use host-volume-mounts from your local src folder to the container.
On OSX, you will run into performance issues when using host mounts. You might use something like http://docker-sync.io ( i am biased - there are also a lot of other similar tools )
I know this is an old question, but as I stumbled across it while trying to see what other editors might offer in this space, I would like to point out Visual Studio Code's notion of a Dev Container, which seems to provide the best level of integration I've seen for this so far. I'm hoping to see this turn into an industry trend myself.
Could use x11docker
x11docker allows to run graphical desktop applications (and entire desktops) in Docker Linux containers.
Docker allows to run applications in an isolated container environment. Containers need much less resources than virtual machines for similar tasks.
Docker does not provide a display server that would allow to run applications with a graphical user interface.
x11docker fills the gap. It runs an X display server on the host system and provides it to Docker containers.
Additionally x11docker does some security setup to enhance container isolation and to avoid X security leaks. This allows a sandbox environment that fairly well protects the host system from possibly malicious or buggy software.
https://github.com/mviereck/x11docker
https://github.com/mviereck/x11docker/wiki (extensive! knowledge)
https://dev.to/brickpop/my-dream-come-true-launching-gui-docker-sessions-with-dx11-in-seconds-1a53
So I am creating a local Python script which I plan to export as an executable. However, this script is in need of a MongoDB instances that runs in the background as a service or daemon. How could one possibly include this MongoDB service along with their own ported application?
I have this configuration manually installed on my own computer with a MongoDB database installed as a local Windows service, and Python where my script adds and removes to the database as some events are fired. Is there any possible way to distribute this setup without manual installation of Python and MongoDB?
If you want to include installations of all your utilities, I recommend pynsist. It'll allow you to make a Windows installer that will make your code launchable as an app on the clients system, and include any other files and/or folders that you want.
Py2exe converts python scripts and their dependencies into Windows executable files. It has some limitations, but may work for your application.
You might also get away with not installing mongo, by embedding something like this in your application: https://github.com/Softmotions/ejdb. This may require you to rewrite your data access code.
If you can't or don't do that, then you could have all your clients share a multi-tenant mongo that you host someplace in the cloud.
Finally, if you can't or won't convert your python script to a standalone exe with an embedded database, and you don't want to host a shared mongo instance for your clients, there are legions of software installation makers that make deploying mongo, python, setting up an execution environment, creating services, etc, pretty easy. Some are free, some cost money. A long list can be found here: https://en.m.wikipedia.org/wiki/List_of_installation_software