Develop Reference

Submitting data to Siebel using Python

I am looking for advice on how to automate submitting data directly to a Siebel application at work using Python. Currently I enter the data into an Autohotkey GUI and when a button is selected it enters the data into Siebel for me using mouse moves and mouse clicks to select the right entries for each piece of data. Obviously this is prone to errors and I would like to make the application better if possible. Using an object oriented programming language would improve this greatly. Just to clarify, this is NOT for automation testing. The data and account/page that I am submitting too changes quite often. So, modules like Selenium, Mechanize, and BeautifulSoup won't work for this as far as I can tell. Since not everything has a form or a friendly label that I can submit data to. If anyone has experience with Siebel and knows a way to copy data from and submit data directly to different entries that would be great.
Right now my best option is to use modules like Pyautogui and Pywinauto to perform mouse moves and clicks to copy what my Autohotkey script does. But this seems inefficient and potentially prone to errors. There has to be a better way to accomplish the same thing using Python. I am just not certain how and I would appreciate any advice you guys may have. Even if that is "no there is no other way" it would help me figure out what to do next. Thanks in advance!

Interacting with the Siebel CRM Application can be done in a large number of ways (SOAP, REST, COM, Java, UI, to name a few supported) and the use-case and environment typically define the preferred approach. A decent Siebel developer/consultant will be able to help you make the right choice.
The ease and available tooling for automating the UI is largely dependent on the version of Siebel you run. Prior to OpenUI this was mainly the domain of large Test Automation vendors (HP, Mercury, Oracle) and required a separate license module to be purchased.
Post Open UI the web UI itself became a single DOM object and much more suitable to automation using open source test tooling like Selenium. With the Test Automation license module activate it will also introduce additional HTML attributes that help to create stable locators.
If interacting with the the UI is just a means to change data, then I would advise an alternate approach: directly interact with the business layer. The added advantage is that there is much more information in the data objects than is typically available in a single UI screen and is more structure.
The easiest approach is probably using the web services. The older versions support mainly SOAP but the latest version also support REST. Most programming languages have support for these approaches and will allow you to import their WSDL files. Keep in mind that you're dependent on the DEV team to extend these interfaces too when they add fields to the UI.
Another approach that gives the most flexibility is to directly interact with the business layer using Java (Bean) or COM. The java approach only requires two JAR files and Google has enough examples on this approach to explain how to use it. When Python is your preferred approach, then the COM interface is an interesting approach. This GitHub project has some good examples to get you on your way.

Django / Geodjango / Raster / geotiff: Problems with practice task

I got a practice task, which I can not get any further.
The task is the following:
Project Description The goal of the projects is to create a map view
similar to the Google Maps, where the user can see some imagery data
captured by drones.
User should be able to move around the map freely, as well as zoom in
and zoom out to take a closer look at the captured imagery data.
It is strongly desired that the served imagery data will support
transparency while minimizing the file size and bandwidth usage. This
does not have to be implemented, but solution ideas are welcomed.
The raw imagery data will be provided as GeoTIFF files. Imagery
visible on the map can be added by placing a file inside a directory
that is read by the server. Project Delivery Method
Project should be delivered as a Git repository with documentation
required to setup and run the project.
Requirements
1. Server implementation in Python 3.5+
2. Project must be able to run on Ubuntu Server 16.04
3. Optimal disk space usage for imagery data displayed to the user (as the app may be processing terabytes of satellite imagery data)
4. Relatively conservative bandwidth usage
Notes:
1. The project will be deployed on a machine that is already running other Python software. Dependency conflicts must be avoided.
(virtualenv, Docker)
2. The UI can be a simple HTML page with embedded libraries and inline scripts.
In addition, it was specified in an e-mail:
"The test task is not code but just the approach and rough app
architecture
```I'm attaching a tank spec. Like I've mentioned. I'm more interested
in problem-solving and your ideas. I expect a working prototype tough.
Use any libraries you wish to use. Create an elegant, easy to
understand the solution. You can use as much time as you want. Would be
great if you could deliver the code by git.... ```"
So far I have done:
Ubuntu as VM
Venv
Postgres and PostGIS installed (Django writes error-free in a database)
Django project and app created
Documentation up to this point
I have now integrated the geotiff via console and that seems to work too:
from django.contrib.gis.gdal import GDALRaster
raster = GDALRaster('base/static/base/geotiff/xto-site3-rgb.tif')
raster.name
Out[4]: 'base/static/base/geotiff/xto-site3-rgb.tif'
raster.width, raster.height
Out[5]: (23001, 9668)
In the models.py is so far:
from django.contrib.gis.db import models
class RasterBase(models.Model):
raster = models.RasterField()
name = models.TextField()
How does it work that I install the grid so that I can portray this in an html similar to google maps? If I understand correctly, I must now write the geotiff in the database, and read on from there, right?
Unfortunately, I find in the network largely only outdated stuff, or often examples, which is assumed by shapefiles. Should I convert the grid to a shapefile and continue like that?
So far, I only make small things in Django, like my own blog and a few statistics, but this with Geodjango is a bit fierce because I have to give it up, as it were tomorrow. Latest Tuesday morning.
I would be very grateful if someone could give me some tips. All in all, that's pretty important to me, and it would be a shame if I messed up half of the task (or the last third) of the task.
Django is version 2.0
The GeoTIFF ~900mb
Thanks for all. :-)

Late to the party, but maybe some people are searching for a solution here.
When you want to display geodata on a map, you can use a WebGIS framework like Openlayers or Leaflet. They provide all the functionality to move the map and zoom in / out.
I would not recommend to store large raster data in a database. You can serve it directly from a file server via a TileLayer or use a XYZ tiling structure to minimize the bandwidth usage.
Openlayers has a lot of examples on how to server GeoTiff files.

Is there a Python library to create thumbnails for various document file formats?

I'd like to generate thumbnails from various "document" file formats such as odt, doc(x) and ppt(x) but also mp4, psd, tiff (and possibly others) from a Python application. As far as I know for each of these formats there is at least one open source application which can generate preview images/thumbnails (e.g. LibreOffice, ffmpeg) or at least extract embedded thumbnails (e.g. imagemagick).
My main problem is that each of these applications/libraries use different command line options so I'm looking for a Python library (or a unified CLI tool) which provides a high-level API to generate a thumbnail with specified dimensions, quality level given a filename and calls the appropriate external tool (ideally including catching exceptions, segfaults and timeouts). Bonus points if it can generate multiple thumbnails if requested (e.g. one per page, page X-Y, every Z seconds but at most N images).
Does anyone know such a library/utility? (Boundary condition: The files may contain sensitive material or might be quite big so this must work without any network communication, using an external web service is not possible.)
If there is no such thing in Python, a locally installable web service would be fine as well.

I ended up writing my own library (named anythumbnailer, MIT license) which worked well enough for my immediate needs. The library is not what I envisioned (only basic thumbnailing, no support for dimensions, …) but it can generate thumbnails for doc(x), xls(x), ppt(x), videos and pdf on Linux with the help of ffmpeg, LibreOffice and ffmpeg.

you can look at Preview generator. preview-generator is a library for generating preview - thumbnails, pdf, text and json overview for all your file-based content. This module gives you access to jpeg, pdf, text, htlm and json preview of virtually any kind of file. It also includes a cache mechanism so you do not have to care about preview storage.

Using Sphinx to create context-sensitive help files in HTML

I am currently using AsciiDoc for documenting my software projects because it supports PDF and HTML help generation. I am currently running it through Cygwin so that the a2x toolchain functions properly. This works well for me but is a pain to setup on other Windows computers. I have been looking for alternative methods and recently revisited Sphinx. Noticing that it now produces HTML help files I gave it a try and it seems to work well in the small tests I performed.
My question is, is there a way to specify map id's for context sensitive help in the text so that my Windows programs can call the proper help API and the file is launched and opened to the desired location?
In AsciiDoc I am using pass::[<?dbhh topicname="_about" topicid="801"?>]. By using these constructs a context.h and alias.h are generated along with the other HTML help files (context sensitive help information).

I do not know about AcsiiDoc much, but in Sphinx you can reference arbitrary locations by placing anchors where you need them. See :ref: role.

Reporting charts and data for MS-Office users

We have lots of data and some charts repesenting one logical item. Charts and data is stored in various files. As a result, most users can easily access and re-use the information in their applications.
However, this not exactly a good way of storing data. Amongst other reasons, charts belong to some data, the charts and data have some meta-information that is not reflected in the file system, there are a lot of files, etc.
Ideally, we want
one big "file" that can store all
information (text, data and charts)
the "file" is human readable,
portable and accessible by
non-technical users
allows typical office applications
like MS Word or MS Excel to extract
text, data and charts easily.
light-weight, easy solution. Quick
and dirty is sufficient. Not many
users.
I am happy to use some scripting language like Python to generate the "file", third-party tools (ideally free as in beer), and everything that you find on a typical Windows-centric office computer.
Some ideas that we currently ponder:
using VB or pywin32 to script MS Word or Excel
creating html and publish it on a RESTful web server
Could you expand on the ideas above? Do you have any other ideas? What should we consider?

I can only agree with Reef on the general concepts he presented:
You will almost certainly prefer the data in a database than in a single large file
You should not worry that the data is not directly manipulated by users because as Reef mentioned, it can only go wrong. And you would be suprised at how ugly it can get
Concerning the usage of MS Office integration tools I disagree with Reef. You can quite easily create an ActiveX Server (in Python if you like) that is accessible from the MS Office suite. As long as you have a solid infrastructure that allows some sort of file share, you could use that shared area to keep your code. I guess the mess Reef was talking about mostly is about keeping users' versions of your extract/import code in sync. If you do not use some sort of shared repository (a simple shared folder) or if your infrastructure fails you often so that the shared folder becomes unavailable you will be in great pain. Note what is also somewhat painful if you do not have the appropriate tools but deal with many users: The ActiveX Server is best registered on each machine.
So.. I just said MS Office integration is very doable. But whether it is the best thing to do is a different matter. I strongly believe you will serve your users better if you build a web-site that handles their data for them. This sort of tool however almost certainly becomes an "ongoing project". Often, even as an "ongoing project", the time saved by your users could still make it worth it. But sometimes, strategically, you want to give your users a poorer experience to control project costs. In that case the ActiveX Server I mentioned could be what you want.

Instead of using one big file, You should use a database. Yes, You can store various types of files like gifs in the database if You like to.
The file would not be human readable or accessible by non-technical users, but this is good.
The database would have a website that Your non-technical users would use to insert, update and get data from. They would be able to display it on the page or export it to csv (or even xls - it's not that hard, I've seen some csv->xls converters). You could look into some open standard document formats, I think it should be quite easy to output data with in it. Do not try to output in "doc" format (but You could try "docx"). You should be able to easily teach the users how to export their data to a CSV and upload it to the site, or they could use the web interface to insert the data if they like to.
If You will allow Your users to mess with the raw data, they will break it (i have tried that, You have no idea how those guys could do that). The only way to prevent it is to make a web form that only allows them to perform certain actions that You exactly know how that they should suppose to perform.
The database + web page solution is the good one. Using VB or pywin32 to script MSOffice will get You in so much trouble I cannot even imagine.
You could use gnuplot or some other graphics library to draw (pretty straightforward to implement, it does all the hard work for You).
I am afraid that the "quick" and dirty solution is tempting, but I only can say one thing: it will not be quick. In a few weeks You will find that hacking around with MSOffice scripting is messy, buggy and unreliable and the non-technical guys will hate it and say that in other companies they used to have a simple web panel that did that. Then You will find that You will not be able to ask about the scripting because everyone uses the web interfaces nowadays, as they are quite easy to implement and maintain.
This is not a small project, it's a medium sized one, You need to remember this while writing it. It will take some time to do it and test it and You will have to add new features as the non-technical guys will start using it. I knew some passionate php teenagers who would be able to write this panel in a week, but as I understand You have some better resources so I hope You will come with a really reliable, modular, extensible solution with good usability and happy users.
Good luck!